# Can Self-Supervised Representation Learning Methods Withstand Distribution
Shifts and Corruptions?

Prakash Chandra Chhipa1,*, Johan Rodahl Holmgren1, Kanjar De1,2,**, Rajkumar
Saini1 and Marcus Liwicki1
1Machine Learning Group, EISLAB, Luleå Tekniska Universitet, Luleå, Sweden
2Video Coding Systems, Fraunhofer Heinrich-Hertz-Institut, Berlin, Germany
<EMAIL_ADDRESS>
*corresponding author -<EMAIL_ADDRESS>
###### Abstract

Self-supervised representation learning (SSL) in computer vision aims to
leverage the inherent structure and relationships within data to learn
meaningful representations without explicit human annotation, enabling a
holistic understanding of visual scenes. Robustness in vision machine learning
ensures reliable and consistent performance, enhancing generalization,
adaptability, and resistance to noise, variations, and adversarial attacks.
Self-supervised representation learning paradigms, namely contrastive
learning, knowledge distillation, mutual information maximization, and
clustering, have been considered to have shown advances in invariant learning
representations. This work investigates the robustness of learned
representations of SSL approaches focusing on distribution shifts and image
corruptions in computer vision. Detailed experiments have been conducted to
study the robustness of SSL methods on distribution shifts and image
corruptions. The empirical analysis demonstrates a clear relationship between
the performance of learned representations within SSL paradigms and the
severity of distribution shifts and corruptions. Notably, higher levels of
shifts and corruptions are found to significantly diminish the robustness of
the learned representations. These findings highlight the critical impact of
distribution shifts and image corruptions on the performance and resilience of
SSL methods, emphasizing the need for effective strategies to mitigate their
adverse effects. The study strongly advocates for future research in the field
of self-supervised representation learning to prioritize the key aspects of
safety and robustness in order to ensure practical applicability. The source
code and results are available on GitHub.
111https://github.com/prakashchhipa/Robsutness-Evaluation-of-Self-supervised-
Methods-Distribution-Shifts-and-Corruptions. **Work performed at Machine
Learning Group, EISLAB, Luleå Tekniska Universitet, Luleå, Sweden

##  1 Introduction

Safety and robustness are crucial in computer vision as they ensure the
accurate and reliable perception of the visual world, enabling applications
such as autonomous driving [31], and surveillance systems to make informed and
trustworthy decisions, reduce environmental noise [22], ultimately enhancing
overall human safety and well-being. In recent years self-supervised
representation learning (SSL) methods [18] have garnered interest in computer
vision applications. Its current state-of-the-art is prominent even against
supervised examples where invariant representation learning has been the core,
as stated in [17]. SSL is a well-explored representation learning approach,
with many studies on its performance in large datasets such as ImageNet-2012
and also on multi-modality [41]. In addition, SSL has also been well-explored
with other learning approaches, including active learning [3], graphs [35],
life-long learning [36], and many more. Recent advances in self-supervised
representation learning can be broadly categorized into multiple paradigms,
namely contrastive learning [9, 23], Knowledge Distillation [21, 7, 12],
Mutual Information Maximization [39, 2], and Clustering [5]. Despite these
advancements, the robustness and safety aspects of SSL paradigms have not been
extensively explored, which hinders their applicability in real-world use
cases. This study is one of the early attempts highlighting the above-stated
research gap on a large-scale dataset [25] focusing on the distribution shifts
and image corruptions.

Representation learning from self-supervised representation learning paradigms
for computer vision can be categorized majorly as (i) Joint Embedding
Architecture & Method (JEAM) ([10], [20], [6], [40]), (ii) Prediction Methods
([37], [33], [16]), and loosely (iii) Reconstruction Methods ([30], [19]).
Specifically, JEAM can be divided further with each subdivision providing many
interesting works; (i) Contrastive Methods (PIRL [32], SimCLR [10], SimCLRv2
[11], MoCo [24]), (ii) Distillation (BYOL [20], SimSiam [13]), (iii)
Quantization (SwAV [6], DeepCluster [4]), and (iv) Information Maximization
(Barlow Twins [40], VICReg [1]). Robustness is critical in real-life computer
vision applications as there will be a shift in distribution for the deployed
models with time. Understanding the behavior of existing models to the
distribution shift is a crucial consideration in developing newer, more robust
models.

Ericsson et al. [17] explore the impact of different augmentation strategies
on the transferability of self-supervised representation learning models to
downstream tasks. The authors show that CNNs trained contrastively do learn
invariances corresponding to the augmentations used, and specializing CNNs to
particular appearance/spatial augmentations can lead to greater corresponding
invariances. Furthermore, learning invariances to synthetic transforms does
provide a degree of invariance to corresponding real-world transforms. This
work establishes the correspondence between synthetic transforms and learning
invariances for knowledge transfer limited to [15] without focusing on
robustness and distribution shift.

Another significant work by Jiang et al. [28] focuses on improving the
robustness of self-supervised pre-training by learning representations that
are consistent under both data augmentations and adversarial perturbations. It
leverages contrastive learning to enhance adversarial robustness via self-
supervised pre-training. They discuss several options to inject adversarial
perturbations to reduce adversarial fragility. Through experiments in both
supervised fine-tuning and semi-supervised learning settings, they demonstrate
that the proposed adversarial contrastive learning can lead to models that are
both label-efficient and robust. The paper does not specifically focus on
corruption, but rather on improving the model’s ability to handle adversarial
attacks. This work shows notable improvement in robustness performance but
remains limited to a small-scale CIFAR dataset, subject to limited
generalizability.

Research is needed to learn invariant SSL representations capable of handling
distribution shifts and corruptions; this study provides a ground in this
direction by sharing insights into the robustness performance of a large-scale
dataset. The identified research gap(s), raises several research questions
addressed in later sections. For the detailed investigation, we considered the
most popular SSL paradigms, namely contrastive learning, knowledge
distillation, mutual information maximization, and clustering. Next, we
exhaustively evaluated the corruptions and their severity levels present in
ImageNet-C dataset [25] to understand the resilience of each method. Further,
compare the robustness performance across multiple metrics, including
qualitative analysis. To the best of our knowledge, this is one of the early
works in this direction.

Q1: How do self-supervised representation learning (SSL) paradigms
(contrastive learning, knowledge distillation, mutual information
maximization, clustering) perform in terms of robustness when exposed to
distribution shifts and image corruptions? A1: Distribution shifts and image
corruptions have an effect on the robustness performance of the well-known SSL
paradigms. The empirical analysis in this study shows that the error rates
(averaged over all distribution shifts and image corruptions) increase with an
increase in the severity levels of the distribution shifts and image
corruptions. (Figure 1, and Section 3.Q1).

Q2: To what extent can self-supervised representation learning methods
maintain their robustness in the presence of distribution shifts, and what are
the factors that limit their ability to do so? A2: Extensive experiments
reveal that SSL methods sustain robustness performance when subjected to lower
levels of corruptions, and subsequently, the performance reduces when
subjected to higher levels of corruptions. Higher corruptions may lead to
massive distribution shifts, which may affect the robustness performance of
learned representations. (Figure 2, Table 3, and Section 3.Q2).

Q3: What is the relationship between the robustness of different SSL paradigms
and common categories of corruptions? A3: Generally, robustness performance
decreases for increased severity of corruptions; specifically, the weather
group’s robustness performance is poorer than that of other groups. (Figure 5,
and Section 3.Q3).

Q4: Do self-supervised representation learning methods deviate from the
observed trend of error increase for certain corruptions, and what factors
contribute to their robustness in the face of these corruptions? A4: Yes; a
few corruptions, namely, snow, elastic transform, and saturate, deviate from
the observed trend supported by visual quality analysis. (Table 4, and Section
5.Q4).

Q5: To what extent does the presence of corruptions shift the focus of
classifiers from overall representation to specific features? A5: GradCam [34]
analysis reveals that there is a significant shift in the attention maps when
the image is subjected to higher levels of corruption. (Figure 4&6, and
Section 3.Q5).

Q6: Do different backbones, such as Convolutional Neural Networks (CNNs) and
Transformers, influence the behavior and robustness? A6. Yes; the self-
attention mechanism in transformer, in contrast to CNNs, does not embed any
visual inductive bias of spatial locality [27]. (Figure 4, and Section 3.Q6).

##  2 Methodology

Comparative performance evaluation against robustness is carried out in two
steps. In the first step, self-supervised representation learning method(s)
are chosen from each potential self-supervised representation learning
approach (based on JEAM), including contrastive learning, knowledge
distillation, mutual information maximization, and clustering. In the second
step, evaluation measures are chosen, indicating quantitative and qualitative
comparisons on distribution shift and corrupted data samples from ImageNet-C.

Reason for measuring robustness of learnt representations with corruptions and
severity \- This study focuses specifically on robustness of representations
where domain shifts is simulated in controlled manner through corruption and
their varying severity level. Corruptions and perturbations in ImageNet-C [25]
are meticulously curated and carefully designed to closely simulate natural
phenomena in vision, related to geometric distortions, visual noises, and
other explicit factors. Five severity levels further resembles the increased
difficulty level, aiding to study robustness at scale. Corruptions across
multiple severity levels, thereby altering the original data distribution in a
controlled manner [25]. Each corruption severity level shifts the distribution
progressively. The corruptions cause variations in texture, color, and spatial
coherence, effectively expanding the data manifold towards shift.

###  2.1 Self-supervised Representation Learning Methods

Methods from different self-supervised representation learning approaches are
considered for analysis on the ImageNet-C dataset. The self-supervised
representation learning techniques considered for this work are categorized
into four main categories based on their methodology.

Contrastive Learning: It is a self-supervised representation learning approach
in computer vision and other machine learning domains. The principle behind
contrastive learning is to learn valuable representations by encouraging
similarity between semantically similar data points while maintaining
dissimilarity between unrelated or contrasting data points. In computer
vision, this approach helps in learning features and representations from
images without relying on labeled data. Instead, it exploits the inherent
structure in the data to learn meaningful representations that can be used for
various downstream tasks. Specifically, SimCLR method [9] minimizes the
temperature-scaled loss function. This contrastive loss penalizes the network
when positive pair similarity is low and negative pair similarity is high.

Knowledge Distillation: Distillation-based self-supervision is where student
and teacher style encoders are structured and share the learning weights with
specific arrangements such as exponential moving averages. Typically,
similarity learning is performed by inducing architectural dissimilarity, such
as adding a prediction MLP network on only one of the branches. In this work,
SimSiam [12], a self-distillation method, and BYOL [21] & DINO (with ResNet
encoder) [7] dual encoder style knowledge distillation methods are employed.

Mutual Information Maximization: This principle is used in self-supervised
representation learning to learn valuable and meaningful representations from
data without explicit labels. The principle is to maximize the mutual
information between different views or transformations of the input data,
assuming that the learned representations should be invariant or robust to
these transformations. Barlow Twins [39], and VICReg [2] are two self-
supervised representation learning methods employed for the work to follow the
principle of mutual information maximization to learn visual representations
by applying redundancy reduction.

Clustering: SwAV [5] combines contrastive learning and clustering-based
approaches to learning meaningful and invariant features from images. The main
idea behind SwAV is to use a clustering mechanism to enforce consistency
between different views of the same image while promoting diversity in the
learned representations.

Robustness Evaluation Criteria: The error rate metrics, namely corruption
error ($CE$), mean corruption error ($mCE$), clean error, average error, and
average relative error, were introduced as a standardized measure to benchmark
the robustness of machine learning models on Imagenet-C. The two-step
evaluation is described by Hendrycks et al. [25]. The same procedure has been
followed in this study.

###  2.2 Dataset and Experimental Setup

ImageNet-C dataset [25] contains 19 types of corruptions with five severity
levels, each algorithmically generated. The main objective is to analyze the
performance of different self-supervised representation learning methods
across these corruptions and severity levels. By conducting detailed
experiments, this research aims to gain insights into how self-supervised
representations handle various types of corruptions. In this paper, we have
performed detailed experiments considering all the corruptions and severity
levels to gain a deeper understanding of how different self-supervised
representations work on different types of corruptions and present our
findings in Section 3.

Table 1: Configuration used (refer [14, 7] for implementation details).

| Barlow Twins | BYOL | SimSiam | SimCLR | DINO | SWaV
---|---|---|---|---|---|---
Batch Size | 2048 | 4096 | 256 | 4096 | 1024 | 256
Epochs | 300 | 200 | 100 | 200 | 800 | 200
Linear-Eval% | 71.8 | 71.8 | 68.3 | 66.9 | 75.3 | 70.5
Epoch | 90 | 90 | 90 | 90 | 100 | 100
Batch Size | 256 | 512 | 512 | 512 | 256 | 256

Experimental details for evaluating the robustness of self-supervised
representations are as follows. We extracted the encoder from a ImageNet pre-
trained self-supervised representation learning model and added a
classification layer at the end of the network. This allows the model to be
fine-tuned on ImageNet 2012 dataset for a classification task. Evaluations is
performed on ImageNet-C dataset [25]. For this work, we have considered six of
the state-of-the-art SSL algorithms, and the configurations are shown in Table
1. We first initialized the classifier layer randomly and froze all the
parameters of the pre-trained encoder. Next, we trained the classifier using
the labeled training set. The models used were trained by mmsetup[14] except
for DINO, which came from its original repository [7]. ResNet-50 was chosen
across all different methods to keep the analysis uniform, and all experiments
subsequently were conducted using this architecture. The SSL models were
tested on ImageNet-C [25], and $mCE$ [25] is used as a performance measure.
The results are shown in Table 2 and 3.

##  3 Can SSL methods endure shifts in data distribution and image
corruptions?

The raised research questions are discussed in this section.

Q1: How do self-supervised representation learning (SSL) paradigms
(contrastive learning, knowledge distillation, mutual information
maximization, clustering) perform in terms of robustness when exposed to
distribution shifts and image corruptions?

The average error rates against all corruptions (per severity level) of all
the SSL methods are depicted in Figure 1. The general trend is that SimCLR and
SimSam have higher error rates as compared to other methods. While learning
has reported good performance previously on ImageNet-C [29], we noticed that
SimCLR is not comparably robust against these corruptions.

Figure 1: Error rates vs. severity levels across ImageNet-C [25] corruptions.

A pattern observed (Figure 1) is that, in general, Knowledge distillation
methods seem to outperform contrastive learning. Clustering outperforms other
methods indicating robust representations. From Figure 1, one important
observation is that for corruptions with lower severity levels, the six SSL
methods form three sets where SwaV and DINO perform best, followed by BYOL and
Barlow twins; finally, SimCLR and SimSiam have relatively lower performance.
However, at the highest severity level, all the methods have similar and high
error rates. This is likely because most images in this group are heavily
distorted and challenging even for the human visual system to comprehend. From
Table 2, we observe that SwaV outperforms all the competing methods in terms
of corruption error and mean corruption error; however, DINO has a better
robustness performance.

Table 2:  Results for each method calculated over the corruption metric [25]. | Barlow Twin | BYOL | SimCLR | SimSam | SWaV | DINO
---|---|---|---|---|---|---
clean error | 28.2 | 28.2 | 33.1 | 31.7 | 29.5 | 24.7
average error | 73.8 | 73.8 | 75.99 | 75.8 | 70.5 | 71.5
average relative error | 74.7 | 74.6 | 76.0 | 76.0 | 70.7 | 72.9

Q2: To what extent can self-supervised representation learning methods
maintain their robustness in the presence of distribution shifts, and what are
the factors that limit their ability to do so?

Table 3 presents a detailed analysis using mean corruption error $mCE$ for
each corruption. Here, we report the average $mCE$ for each corruption in the
ImageNet-C dataset. One of the findings is that glass blur significantly
impacts the robustness of these models, specifically at higher severity
levels. Most of these models have demonstrated good robustness to brightness-
based corruptions. As corroborated by Figure 2 for most corruptions, the model
robustness suffers with the increase in severity levels.

Figure 2: Model performance against specific corruptions by severity. For
corruptions, namely, snow, saturate, and elastic (last row), SSL models
perform poorly at severity level 2 than at severity level 3. Table 3: $mCE$
for each corruption type against the baseline. The error rates in each column
of corruption types are average values of all severity levels.

|  | Weather | Noise | Extra | Digital | Blur
---|---|---|---|---|---|---
| mCE | Snow | Frost | Fog | Bright | Gauss. | Shot | Impulse | Speckle | Gauss. | Spatter | Saturate | Pixelate | Contrast | Elastic | JPEG | Zoom | Defocus | Motion | Glass
barlow | 73.8 | 84 | 78 | 87 | 46 | 73 | 75 | 83 | 70 | 73 | 69 | 47 | 66 | 85 | 72 | 62 | 86 | 79 | 80 | 88
byol | 73.8 | 85 | 78 | 86 | 46 | 73 | 76 | 81 | 71 | 73 | 70 | 48 | 66 | 84 | 73 | 63 | 86 | 78 | 80 | 86
simclr | 76.0 | 85 | 78 | 89 | 48 | 76 | 79 | 85 | 76 | 77 | 70 | 50 | 67 | 80 | 75 | 68 | 88 | 83 | 82 | 88
simsam | 75.8 | 85 | 80 | 90 | 50 | 74 | 76 | 81 | 71 | 74 | 73 | 50 | 65 | 88 | 75 | 63 | 88 | 82 | 84 | 89
swav | 70.5 | 80 | 73 | 79 | 41 | 71 | 73 | 82 | 68 | 70 | 65 | 44 | 67 | 73 | 71 | 60 | 83 | 77 | 77 | 85
dino | 72.9 | 83 | 75 | 82 | 41 | 74 | 75 | 84 | 68 | 70 | 65 | 44 | 68 | 79 | 70 | 60 | 82 | 76 | 76 | 85
supervised [25] | 76.7 | 78 | 75 | 66 | 57 | 80 | 82 | 83 | 76 | 74 | 76 | 58 | 77 | 71 | 85 | 77 | 80 | 75 | 78 | 89

Q3: What is the relationship between the robustness performance of different
SSL paradigms and common categories of corruptions?

As the severity levels of corruptions increase, all self-supervised
representation learning (SSL) methods demonstrate a decline in their
robustness, as shown in Figure 5. While the noise and blur groups have a
similar trend, whereas digital group shows comparatively strong resilience for
intermediate severity level. SSL methods are least robust against weather
group.

Figure 3: Glass blur on dogs; markers in the images show correct (green) and
incorrect (red) classifications. In ImageNet, with many dog breed classes,
misclassification doesn’t necessarily indicate a bad model if the
representation is adequate. In the twin dog example, with low blur severity,
both dogs have good activations for all models, suggesting good
representations. However, at high blur severity, the model struggles to
classify, resulting in distorted activations and difficulty in distinguishing
between the dogs, leading to poor results.

Figure 4: Comparison between different backbones, ResNet50 and ViT-s8 for DINO
SSL method over ImageNet-C [25] corruptions. Severity levels (left),
corruptions (right).

Figure 5: Group-wise comparison. (a) Noise (b) Blur (c) Weather (d) Digital
(left to right).

Q4: Do self-supervised representation learning methods deviate from the
observed trend of error increase for certain corruptions, and what factors
contribute to their robustness in the face of these corruptions?

We observed (Figure 2) that for three corruptions, namely, snow, saturate, and
elastic transform (last row), there is a deviation from the expected trend;
the expected trend is that the error increases with an increase in severity
level. However, SSL models are performing low at severity level 2 than at
severity level 3. Given the intriguing deviations displayed by (snow, elastic,
saturate) from their anticipated behavior, we delved deeper into our inquiry,
employing a renowned perceptual measure known as Structural Similarity Index
measure (SSIM) [38] to investigate further, as one of the metrics popularly
used by image quality researchers for reference image-based quality
assessment.

We computed the SSIM between the original image (from ImageNet) and the
corresponding corrupted image (from ImageNet-C) for all test images and
averaged at each severity level (Table 4); this gives an estimate of the
visual quality.

Snow corruption occludes the object by adding whitish pixels as snowflakes
with motion blur. It has more visually challenging images at severity level 2
than other levels; therefore, SSIM at level 2 is lower than the SSIM at other
severity levels. Similarly, for elastic transform, the SSIM at level 2 is
lower than the SSIM at other severity levels. At low severity (levels 1 and
2), the affine transform is more noticeable in some cases, causing artifacts,
which can also be seen in figure 6 on its elastic transform. The saturation
corruptions have very low saturation at low severity levels, causing it to be
a grayscale image. This might lead to some classes not being accurately
predicted, where color information is crucial. In nutshell, only for snow,
elastic and saturate, increased severity level (2 to 4) by increasing
respective artifacts, does not reflect increased noise in image examples which
mitigates above stated behaviour from all SSL methods.

Table 4: SSIM metric for snow, elastic and saturate-based corruptions. Severity | Snow | Elastic | Saturate
---|---|---|---
1 | 0.218 | 0.276 | 0.288
2 | 0.179 | 0.237 | 0.283
3 | 0.194 | 0.315 | 0.273
4 | 0.186 | 0.312 | 0.234
5 | 0.189 | 0.305 | 0.210

Q5: To what extent does the presence of corruptions shift the focus of
classifiers from overall representation to specific features?

To gain more insight into how different self-supervised representation
learning methods for classification task pick a label, we have used gradcams
[34] to compare the different methods qualitatively. Gradcams are used to
explain the model’s decision as they provide heatmaps on where in the image
the model is focusing. In Figure 6, we show the grad cams of an image for all
SSL methods under different corruptions of varying severity levels.

The difference among Gradcams gives an understanding of how the model behavior
changes in the presence of a particular corruption. From Table 3, we noticed
that glass blur corruption had caused the highest misclassification for all
competing SSL methods; to understand how different methods respond to
different severity of glass blur, we provide the corresponding gradcams in
Figure 4.

Figure 6: Random Corruptions on a Cobra; the markers in the images show
correct and incorrect classifications. Cobra, a reptile with multiple classes
in ImageNet, may confuse classifiers. However, cobras are generally
distinguishable from other reptiles due to their distinctive neckband. At
severity level 2 of elastic transform, there are artifacts causing a distorted
doublet, making it a challenging case. Overall, models perform well with good
representation. The classifier in these examples shows a bias toward the
neckband, while the original un-cropped image emphasizes the edges for
classification by DINO and SwaV. However, due to corruptions, the focus shifts
more towards the neckband.

Q6: Do different backbones, such as Convolutional Neural Networks (CNNs) and
Transformers, influence the behavior and robustness?

There has been analysis [26] on adversarial robustness for transformer and CNN
architectures but to specifically analyze the robustness against corruptions
and distribution shifts, we chose the most robust SSL method from the previous
analysis (i.e., DINO), and compared the backbone ViT-s8 [8] transformer with
standard CNN ResNet-50. Undoubtedly, transformer architecture outperformed CNN
backbone across the severity levels and also for each image corruption. A
detailed trend is shown in Figure 4.

##  4 Conclusion

The primary objective of this investigation was to conduct an in-depth
analysis of diverse paradigms employed in current self-supervised
representation learning paradigms, focusing on their robustness
characteristics when subjected to varying corruptions present in the
ImageNet-C database. The aim was to gain a comprehensive understanding of how
these self-supervised representation learning paradigms perform and behave in
the face of diverse corruptions, thereby contributing to the advancement of
robust representation learning in the computer vision domain. Through
empirical analysis, we have presented various analytical trends and
demonstrated that self-supervised representation learning methods exhibit
decreased robustness as distributional shifts intensify. Notably, our findings
indicate that the DINO method employing the distillation approach and the SwAV
method utilizing clustering exhibit relatively higher levels of robustness
compared to the other methods investigated in this study. While DINO is
associated with knowledge distillation, SwAV employs a contrastive assignment
quantization approach, indicating their dissimilarity in methodology. These
results suggest that multiple SSL methods originating from diverse SSL
paradigms display enhanced robustness when evaluated on ImageNet-C. However,
it is essential to view these empirical findings as a starting point for
further exploration rather than definitive conclusions. The comparative study
conducted in this research serves to enhance the comprehension of the computer
vision community regarding the strengths and limitations of various self-
supervised representation learning approaches. Furthermore, it facilitates
researchers in developing robust representations in future endeavors. A
significant finding from our analysis is that the SwaV method, which employs a
clustering approach, exhibits higher robustness compared to popular methods
such as SimCLR and Barlow Twins. This result offers valuable insights for
future research directions aimed at further improving self-supervised
representation learning methodologies. Considering the findings of this study,
it becomes imperative to address the challenges associated with the
performance degradation of self-supervised representation learning methods
under distribution shifts and image corruptions. By prioritizing safety and
robustness, researchers can contribute to the development of more reliable and
trustworthy self-supervised representation learning techniques that can
effectively handle real-world scenarios and enhance the practical utility of
these methods. In this work, we dedicate to the methodical revelation of
empirical evidence, rather than hypothesizing. Our endeavor remains steadfast
in illuminating numerous enigmas through a rigorous examination. We firmly
hold the conviction that this pioneering work shall pave the way for future
inquiries, enabling the formulation and evaluation of cogent hypotheses.

## References

  * [1] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  arXiv preprint arXiv:2105.04906, 2021.
  * [2] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  In ICLR, 2022.
  * [3] Razvan Caramalau, Binod Bhattarai, D Stoyanov, and Tae-Kyun Kim.  Mobyv2: Self-supervised active learning for image classification.  In The 33rd British Machine Vision Conference (BMVC). BMVA, 2022\.
  * [4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 132–149, 2018.
  * [5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in neural information processing systems, 2020.
  * [6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  * [8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations, 2020.
  * [10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [11] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton.  Big self-supervised models are strong semi-supervised learners.  Advances in Neural Information Processing Systems, 33:22243–22255, 2020.
  * [12] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In CVPR, 2021.
  * [13] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750–15758, 2021.
  * [14] MMSelfSup Contributors.  MMSelfSup: Openmmlab self-supervised learning toolbox and benchmark.  https://github.com/open-mmlab/mmselfsup, 2021.
  * [15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [16] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In Proceedings of the IEEE international conference on computer vision, pages 1422–1430, 2015.
  * [17] Linus Ericsson, Henry Gouk, and Timothy M Hospedales.  Why do self-supervised models transfer? on the impact of invariance on downstream tasks.  In The 33rd British Machine Vision Conference, 2022, page 509. BMVA Press, 2022.
  * [18] Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales.  Self-supervised representation learning: Introduction, advances, and challenges.  IEEE Signal Processing Magazine, 39(3):42–62, 2022.
  * [19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [20] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in neural information processing systems, 33:21271–21284, 2020.
  * [21] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  In NeurIPS, 2020.
  * [22] Jianfeng He, Bei Xiao, Xuchao Zhang, Shuo Lei, Shuhui Wang, and Chang-Tien Lu.  Reducing noise pixels and metric bias in semantic inpainting on segmentation map.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1876–1885, 2021.
  * [23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning, 2020\.
  * [24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [25] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  Proceedings of the International Conference on Learning Representations, 2019.
  * [26] Jaehyuk Heo, Seungwan Seo, and Pilsung Kang.  Exploring the differences in adversarial robustness between vit-and cnn-based models using novel metrics.  Computer Vision and Image Understanding, page 103800, 2023.
  * [27] Samy Jelassi, Michael Sander, and Yuanzhi Li.  Vision transformers provably learn spatial structure.  Advances in Neural Information Processing Systems, 35:37822–37836, 2022.
  * [28] Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang.  Robust pre-training by adversarial contrastive learning.  Advances in neural information processing systems, 33:16199–16210, 2020.
  * [29] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning, 2021.
  * [30] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [31] Jiale Li, Hang Dai, and Yong Ding.  Self-distillation for robust lidar semantic segmentation in autonomous driving.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII, pages 659–676. Springer, 2022.
  * [32] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707–6717, 2020.
  * [33] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI, pages 69–84. Springer, 2016.
  * [34] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [35] Amir Shirian, Krishna Somandepalli, and Tanaya Guha.  Self-supervised graphs for audio representation learning with limited labeled data.  IEEE Journal of Selected Topics in Signal Processing, 16(6):1391–1401, 2022.
  * [36] Mamatha Thota, Dewei Yi, and Georgios Leontidis.  Lleda–lifelong self-supervised domain adaptation.  arXiv preprint arXiv:2211.09027, 2022.
  * [37] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling.  Rotation equivariant cnns for digital pathology.  In International Conference on Medical image computing and computer-assisted intervention, pages 210–218. Springer, 2018.
  * [38] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [39] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning, 2021.
  * [40] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.
  * [41] Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales.  Self-supervised multimodal learning: A survey.  arXiv preprint arXiv:2304.01008, 2023.


# Improving Cross-Domain Hate Speech Generalizability with Emotion Knowledge

Shi Yin Hong  Susan Gauch
Department of Electrical Engineering and Computer Science
University of Arkansas, Fayetteville, AR, USA
{syhong<EMAIL_ADDRESS>
###### Abstract

Reliable automatic hate speech (HS) detection systems must adapt to the in-
flow of diverse new data to curtail hate speech. However, hate speech
detection systems commonly lack generalizability in identifying hate speech
dissimilar to data used in training, impeding their robustness in real-world
deployments. In this work, we propose a hate speech generalization framework
that leverages emotion knowledge in a multitask architecture to improve the
generalizability of hate speech detection in a cross-domain setting. We
investigate emotion corpora with varying emotion categorical scopes to
determine the best corpus scope for supplying emotion knowledge to foster
generalized hate speech detection. We further assess the relationship between
using pretrained Transformers models adapted for hate speech and its effect on
our emotion-enriched hate speech generalization model. We perform extensive
experiments on six publicly available datasets sourced from different online
domains and show that our emotion-enriched HS detection generalization method
demonstrates consistent generalization improvement in cross-domain evaluation,
increasing generalization performance up to 18.1% and average cross-domain
performance up to 8.5%, according to the F1 measure.

##  1 Introduction

Hate speech (HS) possesses justified regulatory grounds since it inflicts harm
toward a targeted individual or group based on perceived characteristics
(Gelber, 2021). The social obstruction imposed by the pervasive online HS thus
ignites counteraction from the natural language processing (NLP) community to
create machine learning-based systems to automate the HS identification
process (Poletto et al., 2021; Alkomah and Ma, 2022). Despite efforts
dedicated to the goal in the past decade, HS detection remains a challenging
task to conquer (Fortuna et al., 2022; Wiegand et al., 2021). Notably, the
lack of generalizability is a prevalent issue with current HS models (Yin and
Zubiaga, 2021).

Dataset | Original Size | Domain
---|---|---
Founta (Founta et al., 2018) | 99,799 | Twitter
Kaggle (Peller, 2014) | 312,737 | Wikipedia
Kumar (Kumar et al., 2018) | 15,000 | Facebook
Offensive Reddit (Qian et al., 2019) | 5,020 | Reddit
Razavi (Razavi et al., 2010) | 1,525 | Natural Semantic Modules, Usenet
Waseem and Hovy (Waseem and Hovy, 2016) | 16,907 | Twitter
GoEmotions (Demszky et al., 2020) | 58,009 | Reddit
Table 1:  Datasets used in the cross-domain cross-dataset generalization
evaluation. The GoEmotions (Demszky et al., 2020) dataset supplies the
auxiliary emotion knowledge in our multitask HS generalization framework.

HS models that suffer from generalizability show a discrepancy in their
performance across HS datasets (Wiegand et al., 2019). Such models are
competitive in detecting HS on the data from the same source as the data they
are trained with but show a significant performance gap when detecting HS from
varied HS sources. The mainstream approach addressing the issue utilizes
knowledge from the HS domain to improve HS generalization. Observations on the
semantic distribution of the data (e.g., implicit or explicit HS) serve as the
basis for counteractive augmentation, synthetic data generative, and sampling
techniques to bridge linguistic gaps observed in HS (Ilan and Vilenchik, 2022;
Arango et al., 2022; Wullach et al., 2021; Ludwig et al., 2022). Furthermore,
datasets may be re-annotated, combined, or created to meet the generalization
task by topics (Yoder et al., 2022; Nejadgholi et al., 2022; Toraman et al.,
2022).

In real-world applications, however, it is unrealistic to conduct
retrospective HS analysis with the constant inflow of new and changing data.
Further, HS is the byproduct of the evolving social context, culture, and
linguistic interpretation (Hilte et al., 2023), which elevates the challenge
of using static criteria in assessing hate speech. HS models that cannot
demonstrate robust generalizability cannot reliably carry out their high-stake
social responsibility in safeguarding vulnerable groups from the multifarious
HS reflected in diverse online platforms. The lack of generalizability of HS
models can even unintentionally exacerbate the proliferation of online HS by
allowing out-of-domain hateful speech to evade its consequences while
curtailing free speech when sanctioning unhateful speech (Bianchi et al.,
2022).

In this work, we utilize emotion knowledge to support the generalization of HS
detection. We find that utilizing emotion knowledge in addressing HS, which
exhibits a greater relative conceptual variability, helps to mitigate the
variance of HS language that challenges HS generalization. Specifically, we
adopt the GoEmotions dataset (Demszky et al., 2020) to provide emotion
information and investigate the effect of leveraging two variants of emotion
corpora – the dataset’s original release with 28 emotions and its Ekman
emotion corpus (Ekman, 1971) equivalent – in improving the HS detection’s
generalizability in a multitask framework. We utilize BERT (Devlin et al.,
2019) and fBERT (Sarkar et al., 2021) as the base Transformers models to
evaluate their effectiveness in enhancing emotion-driven HS generalizability
given their varying pre-trained corpora relatedness to HS. We assess the
proposed model’s performance in improving the generalizability of six popular
benchmark datasets from different domains with the cross-dataset evaluation
method. Our emotion-enriched HS detection generalization method demonstrates
consistent cross-domain generalization binary F1 performance, increasing
generalization performance up to 18.1% and average cross-domain up to 8.5%.
Our main contributions are summarized below:

  * •

We propose an emotion-integrated multitask HS generalization framework that
utilizes emotion knowledge to strengthen cross-domain HS generalization.

  * •

We study how the categorical scope of the emotion corpora – the 28-class
GoEmotions (Demszky et al., 2020) corpus and the six-class Ekman emotion
(Ekman, 1971) corpus – affects the generalizability of HS detection with our
method.

  * •

We evaluate the effect of the adopted Transformers base models’, BERT and
fBERT, varying adaptiveness to the HS domain on our cross-domain HS
generalization framework.

  * •

We perform extensive evaluations in cross-domain settings on six publicly
available benchmark datasets with varied HS forms to show our model improves
cross-domain HS generalization.

##  2 Related Works

###  2.1 Generalization of HS Detection

In studies of HS detection’s generalization, some works aim to identify
sources behind the lack of performance generalization. Fortuna et al. (2020)
analyze the homogeneity of applied categories in popular public HS datasets
and empirically support the lack of compatibility among the cross-dataset
performance. They suggest an underlying reason is the lack of consensus on
HS’s subjective definitional concept, leading to varied criteria for HS
categorization. Fortuna et al. (2021) reason that the low generalization of HS
detectors roots to the imbalance of explicit and implicit distributions of HS
across datasets and encourage HS dataset creators to identify precise
categorization (e.g., sexism, racism) to endow levels of granularity in HS.
Arango et al. (2022) argue for more transparency behind the user distribution
of existing HS datasets to prevent spurious high performance of HS classifiers
overfitted to limited users in data production. They propose to improve cross-
dataset generalization by adopting countering sampling techniques addressing
user overfitting.

$GE_{ek}$ | $GE_{go}$
---|---
Anger | Anger, annoyance, disapproval
Disgust | Disgust
Fear | Fear, nervousness
Joy | Admiration, amusement, approval,
| caring, desire, excitement,
| gratitude, joy, love,
| optimism, pride, relief
Sadness | Sadness, disappointment,
| embarrassment, grief, remorse
Surprise | Surprise, realization,
| confusion, curiosity
Table 2: Categorical emotion conversion between the version of the GoEmotions
(Demszky et al., 2020) dataset adopted with the Ekman $GE_{ek}$ (Ekman, 1971)
corpus and its original $GE_{go}$ corpus.

The dominant method in addressing the generalization of HS detection considers
analyzing underlying semantic and topical traits of HS datasets. Bourgeade et
al. (2023) sample from six HS corpora and present a re-annotated dataset
version based on topic-generic and topic-specific levels. They find that
adopting a mixture of topic-generic and topic-specific tweets in the model
fine-tuning step enhances the generalization of HS classifiers. Nejadgholi et
al. (2022) show the weakness of HS classifiers at generalizing implicit racism
from topic-centric HS datasets and propose a model based on concept activation
vector to improve the interpretability of the model in performing
generalization. Ludwig et al. (2022) adopt unsupervised domain adaptation to
improve HS models’ ability to perform generalization across HS targeting
toward a subset of categorized target groups encompassed by the HateXExplain
dataset (Mathew et al., 2021). Wullach et al. (2021) adopt a GPT-based
language model to generate synthetic HS via sequence generation using existing
HS datasets as an augmentation technique to improve the quality of HS
generalization.

###  2.2 Multitask HS Detection With Emotions

Multitask learning is a training methodology that has recently gained
popularity in NLP due to its power to integrate knowledge from related tasks
in modeling a target task (Zhang et al., 2023; Turcan et al., 2021). In the
context of this study, emotion classification is the auxiliary task modeled
jointly with the main task of binary hate speech classification to improve
hate speech detection generalization leveraging the integrated emotion
knowledge. Present works that consider emotion features in related studies
focus more on the abusive language detection domain without considering the
model’s generalizability. Rajamanickam et al. (2020) investigate abusive
language detection by incorporating emotion detection into MLP-based and
BiLSTM-based networks with a hard-sharing multitask framework. Samghabadi et
al. (2019) employ pre-trained DeepEmoji to assign textual data with relevant
emotions in capturing offensive language. Plaza-del-Arco et al. (2021) and
Plaza-del Arco et al. (2022) incorporated sentiment information in addressing
HS domain-related tasks, offensive and abusive language detections, with
emotion information. In the HS domain, Chiril et al. (2022) adopt emotion
lexicons such as SenticNet and EmoSenticNet to detect hate speech with multi-
targets from topic-generic datasets and conclude that the utilization of
affective knowledge enhances hate speech detection categorized by targets and
topics. Mnassri et al. (2023) perform hate and offensive language detection
using emotion information in a multitask setting involving cross-lingual
settings.

Diverging from previous works, we utilize emotion knowledge with varying
categorical scopes to uplift cross-domain generalizability. We further examine
Transformers models’ relative domain adaptability to the HS in cross-domain
generalizability.

##  3 Methodology

###  3.1 Experimental Datasets

We adopted the six datasets and the processing procedures used by Ilan and
Vilenchik (2022), a recent hate speech generalization work, to assess our
model’s ability to improve hate speech detection generalizability in a cross-
domain setting. These datasets are: Founta (Founta et al., 2018), Kaggle
(Peller, 2014), Kumar (Kumar et al., 2018), Waseem and Hovy (W&H) (Waseem and
Hovy, 2016), Offensive Reddit (Qian et al., 2019), and Razavi (Razavi et al.,
2010). For larger datasets such as Founta, Kaggle, and Kumar, positive and
negative HS samples were randomly sampled for 5,000 entries, totaling 10,000
samples per dataset. Negative samples in smaller datasets such as Razavi and
W&H were downsampled to the sizes of positive samples, 482 and 795,
respectively, to control the source of variance in generalization study (Swamy
et al., 2019). For the Offensive Reddit dataset, the 3,230 positive samples
were balanced with 3,230 negative samples from Washam (2019). User mentions,
hashtags, URLs, and emojis were removed for text preprocessing. Table 1 shows
an overview of the datasets and their respective domains used in our cross-
domain HS generalizability study.

(a)  Figure 1: Abstract architecture for the emotion knowledge-enriched HS
generalization framework.

We use two variants of the GoEmotions dataset (Demszky et al., 2020) to supply
the emotion information. The first variant is the original version of the
dataset with 27 emotions plus the neutral class, which we would refer to as
$GE_{go}$ in the following sections. $GE_{go}$ serves in our study by
providing fine-grained categorical emotion information in assessing the
generalizability of our emotion-enriched hate speech detection method. To
contrast the detailed emotion categories supplied by $GE_{go}$, we convert the
28 emotion classes in $GE_{go}$ into its Ekman equivalent with six emotions as
detailed in Demszky et al. (2020). We refer to the GoEmotions dataset in the
form of the traditional six-class emotion corpus as $GE_{ek}$. Table 2
provides the emotion categorical conversion between $GE_{go}$ and $GE_{ek}$.

Train/Test | Founta | Kaggle | Kumar | Off.Red. | Razavi | W&H | CD Avg
---|---|---|---|---|---|---|---
Founta | 0.922 | 0.800 | 0.470 | 0.734 | 0.672 | 0.543 | 0.669
Founta + $GE_{go}$ | 0.926 | 0.809 | 0.556 | 0.739 | 0.686 | 0.553 | 0.697
Founta + $GE_{ek}$ | 0.925 | 0.805 | 0.550 | 0.736 | 0.679 | 0.548 | 0.683
Kaggle | 0.833 | 0.902 | 0.581 | 0.718 | 0.741 | 0.632 | 0.701
Kaggle + $GE_{go}$ | 0.845 | 0.911 | 0.592 | 0.725 | 0.767 | 0.677 | 0.721
Kaggle + $GE_{ek}$ | 0.839 | 0.916 | 0.616 | 0.742 | 0.777 | 0.701 | 0.735
Kumar | 0.729 | 0.645 | 0.692 | 0.640 | 0.669 | 0.644 | 0.665
Kumar + $GE_{go}$ | 0.764 | 0.739 | 0.703 | 0.670 | 0.712 | 0.663 | 0.709
Kumar + $GE_{ek}$ | 0.779 | 0.746 | 0.711 | 0.680 | 0.741 | 0.664 | 0.722
Off. Red. | 0.671 | 0.638 | 0.530 | 0.931 | 0.627 | 0.618 | 0.617
Off. Red. + $GE_{go}$ | 0.727 | 0.694 | 0.558 | 0.932 | 0.657 | 0.650 | 0.657
Off. Red. + $GE_{ek}$ | 0.688 | 0.678 | 0.556 | 0.931 | 0.651 | 0.659 | 0.646
Razavi | 0.798 | 0.829 | 0.566 | 0.767 | 0.866 | 0.631 | 0.718
Razavi + $GE_{go}$ | 0.834 | 0.836 | 0.635 | 0.773 | 0.890 | 0.652 | 0.746
Razavi + $GE_{ek}$ | 0.844 | 0.855 | 0.663 | 0.770 | 0.871 | 0.653 | 0.757
W&H | 0.535 | 0.500 | 0.518 | 0.659 | 0.545 | 0.896 | 0.555
W&H + $GE_{go}$ | 0.597 | 0.529 | 0.570 | 0.675 | 0.557 | 0.899 | 0.583
W&H + $GE_{ek}$ | 0.603 | 0.541 | 0.555 | 0.684 | 0.584 | 0.898 | 0.591
Table 3:  BERT-based in-domain and cross-domain HS generalization performance

###  3.2 Emotion-enriched Hate Speech Generalization

We implement a multitask architecture as illustrated in Figure 1 given a set
of disjoint tasks $\Omega=\\{\Omega_{hs},\Omega_{e}\\}$, where $\Omega_{hs}$
denotes the main task of hate speech detection and $\Omega_{e}$ denotes the
auxiliary task of emotion analysis. We let $\Omega_{hs}$ and $\Omega_{e}$
share the same Transformer-based encoding layers to promote direct knowledge
transfer. This hard parameter-sharing choice prevents overfitting and
overparameterization (Ruder, 2017) (Liu et al., 2019). The respective dataset
$D_{\Omega}$ for each task is $D_{hs}$ for $\Omega_{hs}$ and $D_{e}$ for
$\Omega_{e}$, where
$D_{\Omega}=\\{(\chi_{\Omega}^{(i)},y_{\Omega}^{(i)})\\}|_{i=1}^{N}$. The
input and target output are denoted by $\chi_{\Omega}^{(i)}$ and
$y_{\Omega}^{(i)}$, and $N$ is the total data entry for each task.

Given $D_{\Omega}$, we first tokenize $\omega\in D_{\Omega}$ into their
subword representations, $s_{\omega}$. We obtain the embedding vector
$v(s_{\omega})$ via an embedding layer that transforms
$[e_{1}(s_{\omega}),e_{2}(s_{\omega}),...,e_{d}(s_{\omega})]^{T}$ into their
vectorial representation, where $d$ is the dimension of the embedding space
and $e_{i}(s_{\omega})$ denotes the $i^{th}$ element of the embedding vector.
We acquire the hidden state $h_{\Omega}^{(i)}$ as follows:

$h_{\Omega}^{(i)}=Encoder(v(s_{\omega})^{(i)},\xi_{\Omega}^{E})$ (1)

where $\xi_{\Omega}^{E}$ denotes the parameters for the encoder. We use BERT
(Devlin et al., 2019) and fBERT (Sarkar et al., 2021) for encoding, where BERT
is trained on a general corpus, and fBERT is trained on an offensive language
corpus based on the OLID dataset (Zampieri et al., 2019). We promote the joint
knowledge exchange between $\Omega_{hs}$ and $\Omega_{e}$ as their parameters
are shared in the encoder unit. In the process, $\Omega_{e}$ functions as a
regularizer, introducing an inductive bias as the two tasks share more general
representations that make the model favors prediction $\hat{y}_{hs}^{(i)}$
that explains both tasks well.

Train/Test | Founta | Kaggle | Kumar | Off.Red. | Razavi | W&H | CD Avg
---|---|---|---|---|---|---|---
Founta | 0.929 | 0.7961 | 0.377 | 0.743 | 0.600 | 0.526 | 0.629
Founta + $GE_{go}$ | 0.930 | 0.805 | 0.397 | 0.753 | 0.609 | 0.560 | 0.625
Founta + $GE_{ek}$ | 0.929 | 0.841 | 0.411 | 0.751 | 0.662 | 0.576 | 0.666
Kaggle | 0.848 | 0.922 | 0.596 | 0.745 | 0.765 | 0.756 | 0.742
Kaggle + $GE_{go}$ | 0.854 | 0.925 | 0.609 | 0.754 | 0.775 | 0.769 | 0.752
Kaggle + $GE_{ek}$ | 0.861 | 0.926 | 0.618 | 0.757 | 0.785 | 0.783 | 0.761
Kumar | 0.848 | 0.596 | 0.715 | 0.596 | 0.765 | 0.756 | 0.712
Kumar + $GE_{go}$ | 0.868 | 0.619 | 0.721 | 0.611 | 0.797 | 0.787 | 0.736
Kumar + $GE_{ek}$ | 0.857 | 0.602 | 0.733 | 0.600 | 0.797 | 0.787 | 0.729
Off. Red. | 0.644 | 0.631 | 0.537 | 0.936 | 0.621 | 0.659 | 0.618
Off. Red. + $GE_{go}$ | 0.682 | 0.688 | 0.550 | 0.933 | 0.665 | 0.683 | 0.653
Off. Red. + $GE_{ek}$ | 0.652 | 0.645 | 0.553 | 0.938 | 0.665 | 0.700 | 0.643
Razavi | 0.845 | 0.871 | 0.642 | 0.768 | 0.881 | 0.790 | 0.783
Razavi + $GE_{go}$ | 0.877 | 0.879 | 0.643 | 0.771 | 0.878 | 0.791 | 0.792
Razavi + $GE_{ek}$ | 0.859 | 0.872 | 0.644 | 0.769 | 0.885 | 0.795 | 0.788
W&H | 0.607 | 0.605 | 0.505 | 0.729 | 0.526 | 0.894 | 0.526
W&H + $GE_{go}$ | 0.663 | 0.634 | 0.574 | 0.742 | 0.607 | 0.896 | 0.638
W&H + $GE_{ek}$ | 0.630 | 0.630 | 0.577 | 0.753 | 0.603 | 0.903 | 0.640
Table 4:  fBERT-based in-domain and cross-domain HS generalization performance

Both tasks $\Omega_{hs}$ and $\Omega_{e}$ continue to share parameters at the
decoding stage but are independent with separate MLP layers for each task. For
each task, the predicted output $\hat{y}_{\Omega}^{(i)}$ is obtained as
follows:

$\hat{y}_{\Omega}^{(i)}=Decoder(h_{\Omega}^{(i)},\xi_{\Omega}^{D})$ (2)

where $\xi_{\Omega}^{D}$ denotes the parameters of the decoder for $\Omega$.

For single-class prediction when modeling $\Omega_{hs}$ and $\Omega_{e}$ using
the $GE_{ek}$ corpus, we minimize the negative log-likelihood (NLL) loss:

$\begin{split}L_{NLL}(\Omega)=&-\sum_{i=1}^{n}(y_{\Omega}^{(i)}log(\hat{y}^{(i)}_{\Omega})+\\\
&(1-y_{\Omega}^{(i)})log(1-\hat{y}^{(i)}_{\Omega}))\end{split}$ (3)

where $y_{\Omega}^{(i)}$ denotes the ground truth label, and
$\hat{y}^{(i)}_{\Omega}$ denotes the predicted label for $\Omega$. For making
predictions using the multi-labeled $GE_{go}$ corpus, we apply the binary
cross-entropy (BCE) loss:

$\begin{split}L_{BCE}(\Omega)=&-\frac{1}{N}\sum_{i=1}^{N}y_{\Omega}^{(i)}log(p(y_{\Omega}^{(i)}))+\\\
&(1-y_{\Omega}^{(i)})log(1-p(y_{\Omega}^{(i)}))\end{split}$ (4)

where $N$ is the training entry count, $y_{\Omega}^{(i)}$ denotes the ground
truth label and $p(y_{\Omega}^{(i)})$ denotes the prediction probability for
true positive prediction for $\Omega$.

##  4 Cross-Domain Generalization

###  4.1 Experimental Setup and Implementation Details

We compare the performance of our approach with the uncased base version of
BERT and fBERT fine-tuned only with HS datasets as baselines. For our emotion-
integrated HS generalization model, we adopt BERT and fBERT as the base models
as shown in Table 3 and Table 4, respectively. Emotion-integrated models are
noted with their respective emotion corpus, $+GE_{go}$ or $+GE_{ek}$. We
perform training on one dataset and evaluation on all datasets’ separate
testing sets. This includes in-dataset evaluation as we perform training and
testing on the same dataset, which also assesses the in-domain
generalizability. We assess cross-domain generalizability performance between
different training and test sets not from the same domain. We further analyze
the overall generalizability performance by providing the average cross-domain
binary F1 for individual experiments as shown in the last column (CD Avg) of
Table 3 and Table 4.

All models were implemented using PyTorch (Paszke et al., 2019), and all
experiments were conducted on NVIDIA Quadro RTX 4000. We trained all models
with 5 epochs with early stopping as we often observed that the best
validation performance is obtained in the first three epochs. We employed a
batch size of 8 and an Adam optimizer with 1E-4 as the learning rate. The
average binary F1 performance of three separate trails using seeds {0, 1, 3}
are reported. The best in-domain and cross-domain average scores are in bold.
Results that show improvement from baselines are highlighted in gradients of
purple for in-domain settings and blue for cross-domain settings based on
their relative strength of improvement.

###  4.2 Results

Table 3 shows the performance of our evaluation using BERT as the base model.
From the baseline model, we observe a general decline in performance when
models are evaluated in a cross-domain setting compared to an in-domain
setting, which supports our motivation to improve cross-domain
generalizability. The disparity in performance is the greatest for the
Offensive Reddit, W&H, and Kaggle datasets, which show a cross-domain
performance decline of 33.7% $(\frac{0.617-0.931}{0.931}\times 100)$, 22.4%,
and 22.2%, respectively, when average cross-domain performance is compared
with the respective in-domain performance for each evaluation dataset.

When we include emotion knowledge using the original GoEmotions dataset with
its $GE_{go}$ corpus, the performance of in-domain and cross-domain
generalization improves. The greatest in-domain improvement
($\mathord{\uparrow}$4.5% =
$\frac{\frac{0.597+0.899}{2}-\frac{0.535+0.896}{2}}{\frac{0.535+0.896}{2}}\times
100$) is with the W&H dataset. Adding the emotion knowledge from the $GE_{go}$
corpus leads to an average increase in cross-domain performance from 2.9%
($\frac{0.721-0.701}{0.701}\times 100$) to 6.6%
($\frac{0.709-0.665}{0.665}\times 100$) observed in the Kaggle and Kumar
datasets, respectively. Experiments that exhibit higher average out-domain
improvement used Kumar ($\mathord{\uparrow}$6.6%), Offensive Reddit
($\mathord{\uparrow}$6.6%), and W&H ($\mathord{\uparrow}$4.9%) as training
sets. The greatest generalizability uplifts for individual cross-domain
experiments are shown in Founta $\xrightarrow{}$ Kumar (i.e., Founta dataset
is the training set for the model that generalizes on the testing set from the
Kumar dataset), Kumar $\xrightarrow{}$ Kaggle, and Razavi $\xrightarrow{}$
Kumar experiments, resulting in generalizability enhancement of 18.1%, 14.5%,
and 12.2%, respectively.

When we adopt the GoEmotions dataset based on the $GE_{ek}$ corpus to supply
emotion knowledge in our HS generalization model, both in-domain and cross-
domain performances also show improvements. The in-domain performance
increases up to 4.9% with the maximum uplifts corresponding to the W&H
datasets. In cross-domain experiments, the average increase in binary F1
ranges from 2.1% to 8.5% with the least and best cross-domain performance
average corresponding to experimental settings where Founta and Kumar datasets
are used as training sets. Integrating emotion knowledge via the $GE_{ek}$
corpus also shows competitive cross-domain generalizability when W&H
($\mathord{\uparrow}$6.4%) and Kaggle ($\mathord{\uparrow}$4.8%) datasets. The
greatest generalizability uplifts for individual cross-domain experiments are
shown in Razavi $\xrightarrow{}$ Kumar ($\mathord{\uparrow}$17.1%), Kumar
$\xrightarrow{}$ Kaggle ($\mathord{\uparrow}$15.7%), and Kaggle
$\xrightarrow{}$ W&H ($\mathord{\uparrow}$10.9%) experiments. Experiments
Razavi $\xrightarrow{}$ Kumar and Kumar $\xrightarrow{}$ Kaggle are also the
top-performance individual cross-domain experiments using the $GE_{go}$
corpus.

Table 4 shows the performance of our evaluation when we used fBERT as the base
model. We also observe a general decline in performance when models are
evaluated in cross-domain settings compared to in-domain settings with the
baseline model’s performance. The same three datasets when using the BERT as
the baseline show the greatest difference in in-domain and cross-domain,
resulting in a performance decline of 33.9%, 19.6%, and 16.0% for the
Offensive Reddit, Kaggle, and W&H datasets, respectively.

(a)  Figure 2: a. Average binary F1 cross-domain generalizability performance
of the baseline average, $GE_{go}$-based model average, and $GE_{ek}$-based
model average. b. Percentage uplifts compared to the baseline average of the
$GE_{go}$-based model average and and $GE_{ek}$-based model average.

Using the $GE_{go}$ corpus to induce emotion knowledge in our model, we
consistently observe cross-domain generalizability enhancement but not always
with in-domain experiments. For the Offensive Reddit and Razavi datasets, the
in-domain performance decreases by 0.003 in binary F1. The greatest in-domain
improvement ($\mathord{\uparrow}$3.8%) is observed from the W&H dataset. The
generalizability improvement in average cross-domain performance ranges from
1.2% to 8.1%. The minimum and maximum average cross-domain performances
correspond to Razavi and W&H datasets, respectively. Emotion knowledge
provided by the $GE_{go}$ corpus also distinctly helps average cross-domain
generalizability performance when Offensive Reddit ($\mathord{\uparrow}$5.7%)
and Kumar ($\mathord{\uparrow}$3.3%) are used as the training sets. The
individual cross-domain generalizbility enhancement is most pronounced with
the W&H $\xrightarrow{}$ Razavi ($\mathord{\uparrow}$15.4%), W&H
$\xrightarrow{}$ Kumar ($\mathord{\uparrow}$13.6%), and W&H $\xrightarrow{}$
Founta ($\mathord{\uparrow}$9.1%).

When we adopt the $GE_{ek}$ corpus with fBERT as the base model in our
emotion-enriched framework, we observe performance improvement in both in-
domain and cross-domain settings. The greatest performance in-domain uplift is
3.5% with the Founta dataset. The average increase in binary F1 ranges from
0.6% to 8.3% in cross-domain settings with the least and best generalizability
corresponding to the Razavi and W&H training sets. Strong average cross-domain
generalizability enhancement also manifests in experiments where Founta
($\mathord{\uparrow}$5.9%) and Offensive Reddit ($\mathord{\uparrow}$4.0%)
datasets are the training sets. The greatest generalizability uplifts for
individual cross-domain experiments are shown in W&H $\xrightarrow{}$ Razavi
($\mathord{\uparrow}$14.6%), W&H $\xrightarrow{}$ Kumar
($\mathord{\uparrow}$14.1%), and Founta $\xrightarrow{}$ Razavi
($\mathord{\uparrow}$10.3%). Experiments W&H $\xrightarrow{}$ Razavi and W&H
$\xrightarrow{}$ Kumar are also the top-performance individual cross-domain
experiments using the $GE_{go}$ corpus.

(a)  Figure 3: a. Average binary F1 cross-domain generalizability performance
of the baseline average and emotion-enriched model average of the BERT-based
and fBERT-based models. b. Percentage uplifts compared to the baseline average
of the BERT-based model average and fBERT-based model average.

##  5 Analysis and Discussions

In this section, we study the factors in our framework that affect cross-
domain generalizability based on the experimental results.

###  5.1 Emotion Corpora and Cross-Domain HS Generalizability

Observation 1: Adopting emotion corpora with fewer categorical scopes, such as
the $GE_{ek}$, generally results in more consistent cross-domain HS
generalizability improvement.

To analyze the relationship between the categorical scope of emotion corpus
and cross-domain generalization, we visualize the average performance of the
cross-domain generalization of two variants of our emotion-enriched models
eliminating the distinction of the employed base models. We take the average
performance of the cross-domain generalizability of BERT-based and fBERT-based
baselines and emotion-enriched models separated by the use of different
emotion corpora. The average performance in binary F1 of the baseline,
$GE_{go}$-based model, and $GE_{ek}$-based model are shown in Figure 2a.
Figure 2b shows the percentage of cross-domain improvement relative to the
baseline average.

We observe that the emotion-enriched model where the emotion knowledge is
introduced by the $GE_{ek}$ exhibits better performance in all cases except
the experiment where the Offensive Reddit dataset is used as the training set.
The highest generalizability improvement by an increase in percentage employed
W&H as the training set, resulting in an improvement of 7.4%. We note that the
W&H dataset contains relatively the shortest average sentence length, and the
HS samples show a direct HS style. These characteristics are also exhibited in
experiments when the Founta dataset is employed as the training set, which
shows the second highest generalizability improvement by percentage
($\mathord{\uparrow}$5.1%). Thus, the emotion knowledge supplied by the
$GE_{ek}$ corpus is generally the better choice for improving cross-domain
generalizability when a model is trained with short, explicit HS and is
expected to generalize to HS that could be longer in length where the HS style
might also be more implicit (e.g., Razavi, Kumar). For the case where the
Offensive Reddit dataset is employed as the training set, we find that the HS
samples in this dataset gear toward explicit sexism, which is not perceived in
other datasets. In this case, the emotion knowledge supplied by the 28-class
emotion corpus $GE_{go}$ helps to mitigate the semantic variance across
contrasting HS topics more than the six-class $GE_{ek}$ corpus.

###  5.2 Domain Adaptability of Base Models and Cross-Domain HS
Generalizability

Observation 2: The strength cross-generalizability enhancement is more
pronounced with our emotion-enriched model when adopting a base model that is
not adapted to the HS domain (e.g., BERT). However, adopting a base model that
is adapted to the HS domain (e.g., fBERT) using our framework generally
results in the highest cross-domain performance.

To analyze the effect of the adopted base models’ domain adaptability on
cross-domain generalization, we visualize our framework’s performance on
cross-domain generalization using BERT-based and fBERT-based models
eliminating the distinction of emotion corpora. We take the average
performance of the cross-domain generalizability average of the variant of our
model that uses the $GE_{go}$ corpus and the variant of our model that uses
the $GE_{ek}$ for each base model. The average performance in binary F1 for
the BERT-based and fBERT-based model are shown in Figure 3a. Figure 3b shows
the percentage of cross-domain improvement relative to the baseline average.

From Figure 3, we note that adopting a non-HS domain-adapted model as the base
model like BERT with our framework results in the greatest percentage of
generalizability improvement in most cases. For the two cases where BERT-based
emotion-enriched models show a relatively weaker generalizability uplift than
fBERT-based models, we note that the training sets, Founta and W&H datasets,
are also the only two datasets that are sourced from Twitter.

Figure 3a supports that adopting a base model like fBERT that is adapted to
the HS domain leads to higher performance despite the effect of adding emotion
knowledge in uplifting cross-domain generalizability might not be as strong as
adopting a non-HS domain adapted base model. For experiments where training
sets are Kaggle, Kumar, Razavi, and W&H, our emotion-enriched model that
utilizes fBERT, which is pre-trained on a dataset that is in the HS domain,
shows the best performance.

From Figure 3b, the most distinct improvement is in cross-domain settings with
Kumar ($\mathord{\uparrow}$7.6%), Offensive Reddit ($\mathord{\uparrow}$5.7%),
and W&H ($\mathord{\uparrow}$5.6%) datasets as the training sets. As
mentioned, the Offensive Reddit dataset exhibits a topical contrast to other
datasets as its HS has a sexism focus. Hence, adopting a general base model
with emotional knowledge helps to reduce the semantic variance across
contrasting HS topics. We observe that the Kumar and W&H datasets both exhibit
relatively indirect styles of HS. This suggests that incorporating emotion
knowledge helps to bridge the gap in allowing an HS model trained with a
general non-HS domain adopted model on implicit HS to generalize on HS that
are relatively more direct (e.g. Founta, Kaggle).

##  6 Conclusion

In this work, we investigated cross-domain HS generalizability integrating
emotion analysis. We presented a multitask HS generalizability framework that
utilizes emotion knowledge to enhance cross-domain HS generalizability. We
employed the 28-class GoEmotions corpus (Demszky et al., 2020) and the
traditional six-class Ekman corpus (Ekman, 1971) to examine their effects on
improving cross-domain HS generalizability. We found that incorporating
emotion knowledge using the Ekman corpus leads to more consistent
generalizability performance. We also inspected the role of HS domain
adaptiveness in base models on cross-domain HS generalizability and noted that
the introduction of emotion knowledge has a relatively stronger strength in
bridging the cross-domain generalization gap of pre-trained models that are
not adapted to the HS domain. Results support that our emotion-enriched models
outperform baselines in all cross-domain settings.

## Limitations

We acknowledge limitations in preserving the conceptual granularity exhibited
in public HS datasets by adopting their varied categorical labels (i.e. toxic,
abusive, sexism) in a binary form. Furthermore, the analyses presented in this
work are based on the chosen datasets corresponding to their domain(s) only.
Therefore, conclusions drawn from the limited quantity of datasets from
restricted domains are not intended to be comprehensive. We also noted that
more potential insights regarding HS generalizability may be drawn by
comparing the results from evaluations against more state-of-the-art baselines
with varied domain adaptiveness to specific aspects of HS (i.e. implicitness,
sarcasm). This is left to future works.

## Acknowledgements

We thank all reviewers for their constructive feedbacks. This work is
supported by in part by NSF 1946391 "RII Track-1: Data Analytics that are
Robust and Trusted (DART): From Smart Curation to Socially Aware Decision
Making."

## References

  * Alkomah and Ma (2022) Fatimah Alkomah and Xiaogang Ma. 2022.  A literature review of textual hate speech detection methods and datasets.  _Information_ , 13(6).
  * Arango et al. (2022) Aymé Arango, Jorge Pérez, and Barbara Poblete. 2022.  Hate speech detection is not as easy as you may think: A closer look at model validation (extended version).  _Information Systems_ , 105:101584.
  * Bianchi et al. (2022) Federico Bianchi, Stefanie HIlls, Patricia Rossini, Dirk Hovy, Rebekah Tromble, and Nava Tintarev. 2022.  “it’s not just hate”: A multi-dimensional perspective on detecting harmful speech online.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8093–8099, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Bourgeade et al. (2023) Tom Bourgeade, Patricia Chiril, Farah Benamara, and Véronique Moriceau. 2023\.  What did you learn to hate? a topic-oriented analysis of generalization in hate speech detection.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 3495–3508, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Chiril et al. (2022) Patricia Chiril, Endang Wahyu Pamungkas, Farah Benamara, Véronique Moriceau, and Viviana Patti. 2022.  Emotionally informed hate speech detection: a multi-target perspective.  _Cognitive Computation_ , pages 1–31.
  * Demszky et al. (2020) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020.  GoEmotions: A dataset of fine-grained emotions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4040–4054, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ekman (1971) Paul Ekman. 1971.  Universals and cultural differences in facial expressions of emotion.  In _Nebraska symposium on motivation_. University of Nebraska Press.
  * Fortuna et al. (2022) Paula Fortuna, Monica Dominguez, Leo Wanner, and Zeerak Talat. 2022.  Directions for NLP practices applied to online hate speech detection.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11794–11805, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Fortuna et al. (2020) Paula Fortuna, Juan Soler, and Leo Wanner. 2020.  Toxic, hateful, offensive or abusive? what are we really classifying? an empirical analysis of hate speech datasets.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 6786–6794, Marseille, France. European Language Resources Association.
  * Fortuna et al. (2021) Paula Fortuna, Juan Soler-Company, and Leo Wanner. 2021.  How well do hate speech, toxicity, abusive and offensive language classification models generalize across datasets?  _Information Processing & Management_, 58(3):102524.
  * Founta et al. (2018) Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018.  Large scale crowdsourcing and characterization of twitter abusive behavior.  In _Proceedings of the international AAAI conference on web and social media_ , volume 12.
  * Gelber (2021) Katharine Gelber. 2021.  Differentiating hate speech: a systemic discrimination approach.  _Critical Review of International Social and Political Philosophy_ , 24(4):393–414.
  * Hilte et al. (2023) Lisa Hilte, Ilia Markov, Nikola Ljubešić, Darja Fišer, and Walter Daelemans. 2023.  Who are the haters? a corpus-based demographic analysis of authors of hate speech.  _Frontiers in Artificial Intelligence_ , 6:986890.
  * Ilan and Vilenchik (2022) Tal Ilan and Dan Vilenchik. 2022.  Harald: Augmenting hate speech data sets with real data.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 2241–2248, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Kumar et al. (2018) Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2018.  Benchmarking aggression identification in social media.  In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_ , pages 1–11, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Liu et al. (2019) Yifan Liu, Bohan Zhuang, Chunhua Shen, Hao Chen, and Wei Yin. 2019.  Training compact neural networks via auxiliary overparameterization.  _CoRR_ , abs/1909.02214.
  * Ludwig et al. (2022) Florian Ludwig, Klara Dolos, Torsten Zesch, and Eleanor Hobley. 2022.  Improving generalization of hate speech detection systems to novel target groups via domain adaptation.  In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_ , pages 29–39, Seattle, Washington (Hybrid). Association for Computational Linguistics.
  * Mathew et al. (2021) Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021.  Hatexplain: A benchmark dataset for explainable hate speech detection.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 14867–14875.
  * Mnassri et al. (2023) Khouloud Mnassri, Praboda Rajapaksha, Reza Farahbakhsh, and Noel Crespi. 2023.  Hate speech and offensive language detection using an emotion-aware shared encoder.  In _IEEE International Conference on Communications (ICC)_ , Roma, Italy, France.
  * Nejadgholi et al. (2022) Isar Nejadgholi, Kathleen Fraser, and Svetlana Kiritchenko. 2022.  Improving generalizability in implicitly abusive language detection with concept activation vectors.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5517–5529, Dublin, Ireland. Association for Computational Linguistics.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Peller (2014) Julian Peller. 2014.  Toxic comment classification challenge.  _"https://www.kaggle.com/datasets/julian3833/jigsaw-toxic-comment-classification-challenge"_.
  * Plaza-del-Arco et al. (2021) Flor Miriam Plaza-del-Arco, Sercan Halat, Sebastian Padó, and Roman Klinger. 2021.  Multi-Task Learning with Sentiment, Emotion, and Target Detection to Recognize Hate Speech and Offensive Language.  _arXiv e-prints_ , page arXiv:2109.10255.
  * Plaza-del Arco et al. (2022) Flor Miriam Plaza-del Arco, M Dolores Molina-González, L Alfonso Ureña-López, and María-Teresa Martín-Valdivia. 2022.  Integrating implicit and explicit linguistic phenomena via multi-task learning for offensive language detection.  _Knowledge-Based Systems_ , 258:109965.
  * Poletto et al. (2021) Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021.  Resources and benchmark corpora for hate speech detection: a systematic review.  _Language Resources and Evaluation_ , 55:477–523.
  * Qian et al. (2019) Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019\.  A benchmark dataset for learning to intervene in online hate speech.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4755–4764, Hong Kong, China. Association for Computational Linguistics.
  * Rajamanickam et al. (2020) Santhosh Rajamanickam, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2020.  Joint modelling of emotion and abusive language detection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4270–4279, Online. Association for Computational Linguistics.
  * Razavi et al. (2010) Amir H Razavi, Diana Inkpen, Sasha Uritsky, and Stan Matwin. 2010.  Offensive language detection using multi-level classification.  In _Advances in Artificial Intelligence: 23rd Canadian Conference on Artificial Intelligence, Canadian AI 2010, Ottawa, Canada, May 31–June 2, 2010. Proceedings 23_ , pages 16–27. Springer.
  * Ruder (2017) Sebastian Ruder. 2017.  An overview of multi-task learning in deep neural networks.  _arXiv preprint arXiv:1706.05098_.
  * Samghabadi et al. (2019) Niloofar Safi Samghabadi, Afsheen Hatami, Mahsa Shafaei, Sudipta Kar, and Thamar Solorio. 2019.  Attending the emotions to detect online abusive language.  _arXiv preprint arXiv:1909.03100_.
  * Sarkar et al. (2021) Diptanu Sarkar, Marcos Zampieri, Tharindu Ranasinghe, and Alexander Ororbia. 2021\.  fBERT: A neural transformer for identifying offensive content.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_.
  * Swamy et al. (2019) Steve Durairaj Swamy, Anupam Jamatia, and Björn Gambäck. 2019.  Studying generalisability across abusive language detection datasets.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 940–950, Hong Kong, China. Association for Computational Linguistics.
  * Toraman et al. (2022) Cagri Toraman, Furkan Şahinuç, and Eyup Yilmaz. 2022.  Large-scale hate speech detection with cross-domain transfer.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2215–2225, Marseille, France. European Language Resources Association.
  * Turcan et al. (2021) Elsbeth Turcan, Smaranda Muresan, and Kathleen McKeown. 2021.  Emotion-infused models for explainable psychological stress detection.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2895–2909, Online. Association for Computational Linguistics.
  * Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016.  Hateful symbols or hateful people? predictive features for hate speech detection on Twitter.  In _Proceedings of the NAACL Student Research Workshop_ , pages 88–93, San Diego, California. Association for Computational Linguistics.
  * Washam (2019) Jason Washam. 2019.  Subreddit-classification.  _https://github.com/jwasham12/Subreddit-Classification_.
  * Wiegand et al. (2021) Michael Wiegand, Josef Ruppenhofer, and Elisabeth Eder. 2021.  Implicitly abusive language – what does it actually look like and why are we not getting there? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 576–587, Online. Association for Computational Linguistics.
  * Wiegand et al. (2019) Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019.  Detection of abusive language: the problem of biased datasets.  In _Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies, volume 1 (long and short papers)_ , pages 602–608.
  * Wullach et al. (2021) Tomer Wullach, Amir Adler, and Einat Minkov. 2021.  Fight fire with fire: Fine-tuning hate detectors using large samples of generated hate speech.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4699–4705, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Yin and Zubiaga (2021) Wenjie Yin and Arkaitz Zubiaga. 2021.  Towards generalisable hate speech detection: a review on obstacles and solutions.  _PeerJ Computer Science_ , 7:e598.
  * Yoder et al. (2022) Michael Yoder, Lynnette Ng, David West Brown, and Kathleen Carley. 2022.  How hate speech varies by target identity: A computational analysis.  In _Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)_ , pages 27–39, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019.  Predicting the type and target of offensive posts in social media.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1415–1420, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Zhang et al. (2023) Zhihan Zhang, Wenhao Yu, Mengxia Yu, Zhichun Guo, and Meng Jiang. 2023.  A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 943–956, Dubrovnik, Croatia. Association for Computational Linguistics.