# Towards Objective Evaluation of
Socially-Situated Conversational Robots:
Assessing Human-Likeness through Multimodal User Behaviors

Koji Inoue<EMAIL_ADDRESS>Kyoto UniversityKyotoJapan ,  Divesh
Lala<EMAIL_ADDRESS>Kyoto UniversityKyotoJapan ,  Keiko Ochi
<EMAIL_ADDRESS>Kyoto UniversityKyotoJapan ,  Tatsuya Kawahara
<EMAIL_ADDRESS>Kyoto UniversityKyotoJapan and  Gabriel Skantze
<EMAIL_ADDRESS>KTH Royal Institute of TechnologyStockholmSweden

(2023)

###### Abstract.

This paper tackles the challenging task of evaluating socially situated
conversational robots and presents a novel objective evaluation approach that
relies on multimodal user behaviors. In this study, our main focus is on
assessing the human-likeness of the robot as the primary evaluation metric.
While previous research often relied on subjective evaluations from users, our
approach aims to evaluate the robot’s human-likeness based on observable user
behaviors indirectly, thus enhancing objectivity and reproducibility. To
begin, we created an annotated dataset of human-likeness scores, utilizing
user behaviors found in an attentive listening dialogue corpus. We then
conducted an analysis to determine the correlation between multimodal user
behaviors and human-likeness scores, demonstrating the feasibility of our
proposed behavior-based evaluation method.

Conversational Robot; Evaluation Method; Multimodal Behavior; Spoken Dialogue
System

††journalyear: 2023††copyright: rightsretained††conference: INTERNATIONAL
CONFERENCE ON MULTIMODAL INTERACTION; October 9–13, 2023; Paris,
France††booktitle: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI
’23 Companion), October 9–13, 2023, Paris, France††doi:
10.1145/3610661.3617151††isbn: 979-8-4007-0321-8/23/10††ccs: Human-centered
computing HCI design and evaluation methods

##  1\. Introduction

One of the research challenges in the field of conversational robots and
dialogue systems is the establishment of evaluation methods (Sim and Loo,
2015; Abd-Alrazaq et al., 2020; Deriu et al., 2021; Zhang et al., 2021; Ni et
al., 2023). Thanks to the emergence of large-scale language models (LLMs),
recent chatbots have become capable of carrying out highly sophisticated
conversations. The realization of such systems has been made possible by the
creation of extensive text datasets and the implementation of evaluation
methods. Since objective evaluation methods alone cannot encompass all
phenomena, a series of studies, including human subjective evaluations, have
been conducted. In the case of task-oriented dialogues, such as restaurant
searches, since the goal of the conversation is clear and objective, it is
straightforward to consider objective evaluation metrics such as task
achievement rate and the number of turns, and research and development efforts
have been made based on these objective indicators. However, this is not the
case where the target conversations are more real and sophisticated such as
ones like human-human conversations in our society.

Recent advancements have led to the development of socially-situated
conversational robots (SCRs), which are specifically designed for social
contexts. Socially-situated conversations encompass a wide range of
interactions, from brief exchanges such as reception and information guide
(Swartout et al., 2010; Iio et al., 2020) to more extended conversations such
as counseling (DeVault et al., 2014; Laranjo et al., 2018; Scoglio et al.,
2019; Rasouli et al., 2022) and interviews (Han et al., 2013; Johnston et al.,
2013; Yu et al., 2017; Inoue et al., 2020a). It is crucial to invest efforts
into the development of SCRs to enhance their practicality, address diverse
social issues, and promote harmonious symbiosis with society.

This study addresses objective evaluation methods for SCRs. Traditional
studies on SCRs have frequently depended on subjective evaluation methods such
as “satisfaction” and “effectiveness” (Casas et al., 2020) or actual system
utterances due to the lack of a clearly defined goal for the conversation.
However, relying solely on subjective evaluation diminishes research
reproducibility and constrains the growth of the research community. In this
study, as an initial step toward the objective and general evaluation method
for SCRs, we introduce an evaluation method based on observable and multimodal
user behaviors (Figure 1) and report our initial trial in an attentive
listening dialogue task.

Figure 1. Overview of the proposed evaluation scheme

The contributions of this paper are two folds:

  * •

We proposed a new evaluation approach for SCRs based on observable and
multimodal user behaviors.

  * •

As a target metric, we focused on the human-likeness of the robot and created
a dataset together with human-likeness scores annotated based on the user
behaviors.

##  2\. Proposed evaluation method

In the current study, we focus on the concept of human likeness as a target
evaluation metric that may be shared with other studies (Edlund et al., 2008;
Kawahara, 2018; Lin et al., 2022). The notion of “naturalness” has been
adopted as an indicator in numerous studies, and “human-likeness” represents a
more concrete manifestation of this concept. While previous studies have
addressed aspects such as user satisfaction (Ultes and Maier, 2021) and
miscommunication (Meena et al., 2015) in the context of automatic evaluating
of conversational systems, the concept of human-likeness pursued in this study
differs in its aspiration for natural dialogue between humans, which
necessitates more advanced conversational abilities. SCRs inherently strive
for human-to-human social interaction, thus emphasizing the importance of
evaluating human likeness rather than naturalness or satisfaction. Note that
the key point of this study is to evaluate SCRs based on multimodal user
behaviors, so the proposed evaluation frame can be applied to other evaluation
metrics.

We propose an evaluation method that lies in its focus on observable user
behaviors. While conventional evaluation metrics have primarily emphasized
system utterances, this has contributed to a heightened subjectivity. By
conducting evaluations based on observable user behavior, we create objective
indicators to the fullest extent possible. User behavior encompasses a wide
range of multimodal aspects. For instance, in addition to including speech and
linguistic features such as total utterance time and word count, it
encompasses dialogue-specific features such as backchannels, fillers, and
switching pause length (turn-taking gap), as well as non-verbal features such
as eye gaze, which is specific to embodied conversational robots.

Intuitively, those user behaviors are different depending on the human
likeness of the robot. This can be more understood by comparing conversations
between human-robot and human-human ones. For example, in the context of
human-robot conversations, if we contemplate the number of uttered words, the
user might tend to utter clearly with a simple and limited vocabulary.
Additionally, in terms of spoken dialogue-specific behaviors, empirical
observations indicate that users tend to provide fewer backchannels and have
longer turn-taking pauses when interacting with systems that are perceived as
non-humanlike. Conversely, in human-human conversations, they tend to be a
proclivity for fluent utterance of a variety of words with smooth turn-taking.
Hence, we can infer that as the number of uttered words increases, proximity
to human-human dialogue intensifies, thereby augmenting the human-likeness
score. In this study, we empirically explore multimodal user behaviors that
relate to the human likeness of the robot.

##  3\. Dataset construction

In this study, we explore the potential of the proposed evaluation framework
by utilizing an attentive dialogue corpus. Here, third-party people
subsequently annotated the corpus to give human-likeness scores with a simple
approach referring to multimodal user behaviors.

###  3.1. Attentive listening dialogue corpus

An attentive listening dialogue corpus was used in this study. In this
dialogue, the task is to attentively listen to a user’s talk, and the system
needs to utter the listener responses, such as backchannels and questions.
Several attentive listening systems have been proposed so far (Schröder et
al., 2015; Inoue et al., 2020b; Oertel et al., 2021). In this instance, we
employed an existing system (Inoue et al., 2020b) for this data collection.
The interface of the system was an android robot (Inoue et al., 2016) whose
appearance is similar to that of human beings. The role of the user was
assigned to a university student, who was asked to speak for eight minutes on
the topic of “challenges faced during the COVID-19 pandemic.” These dialogues
were made in the Japanese language. Note that the aforementioned configuration
merely represents one of the potential setups, and it is desirable to explore
various types of dialogues, systems, and interfaces in future investigations.

In order to vary the human-likeness of the robot, two scenarios were prepared
for this data collection. The first scenario entails interacting with the
aforementioned pre-existing autonomous system. The second scenario involves an
operator in a separate room engaging in direct conversation on behalf of the
system, the so-called Wizard-of-OZ (WOZ). In this case, the operator’s spoken
voice was played back directly through the android’s speaker, and nonverbal
expressions, such as the android’s gaze and gestures, were controlled by the
operator using a handheld controller. There were a total of two operators,
with one of them participating in each dialogue. With the two aforementioned
configurations, 20 dialogues were recorded using the autonomous system, and 49
dialogues were recorded with operator involvement. Thus, there were a total of
69 university students acting as users. After each dialogue concluded, the
participants were asked to answer a 19-item questionnaire evaluation created
in a previous study (Inoue et al., 2020b).

Figure 2. Sample video clip viewed by annotators

###  3.2. Annotation of human-likeness scores

Using the aforementioned dialogue data, we annotated labels to assess the
human-likeness of the system. First, we extracted segments of dialogues and
removed the system’s visual and auditory components, leaving only the user’s
visual and auditory inputs. Third-party annotators were then assigned the task
of binary classification to determine whether the dialogue partner (the
system) was human or an autonomous system. Figure 2 illustrates examples of
the visual stimuli presented to the annotators. In other words, the annotators
indirectly inferred whether the dialogue partner was a human or a system by
focusing solely on the user’s behaviors, which are the main focal point of
this study. By gathering judgments from multiple annotators, we calculated the
ratio of “human” judgments as a measure of “human-likeness.” The dialogue
segments were extracted for a duration of one minute, resulting in a total of
924 samples from the aforementioned 69 dialogues. Each sample was assessed by
five independent evaluators. In total, there were 78 annotators, with each
being randomly allocated 50 to 70 samples.

Table 1 presents the results of the annotation. This evaluation represents the
aggregation of sample quantities per numerical value of human-likeness,
carried out by the aforementioned annotators. Initially, when examining the
entirety (column “Total”), it becomes evident that the numerical values of
human-likeness exhibit variation. Next, upon observing the differences between
the two system types, it can be discerned that the autonomous system tends to
possess comparatively lower proportions of human-likeness. Meanwhile, WOZ, on
the other hand, demonstrates a tendency towards higher numerical values of
human-likeness. In other words, at present, the autonomous system is inferior
to WOZ, thus indicating a reasonable reflection of this fact.

In this study, to conduct evaluation in each dialogue, we calculated the
average score of human-likeness for each dialogue. The distribution of the
averaged scores is illustrated in Figure 3. Even in this scenario, it is
evident that the scores exhibit variation. In the subsequent analysis, which
is described in the following section, we will use these averaged scores as
the target variables.

Table 1. Annotation result of human-likeness score (HL: human-likeness, Auto.:
Autonomous system)

HL score | System type | Total
---|---|---
Auto. | WOZ
1.0 (5/5) |  004 (01.5%) |  066 (10.1%) |  070 (07.6%)
0.8 (4/5) |  026 (09.7%) | 138 (21.0%) | 164 (17.7%)
0.6 (3/5) |  040 (14.9%) | 156 (23.8%) | 196 (21.2%)
0.4 (2/5) |  059 (22.0%) | 145 (22.1%) | 204 (22.1%)
0.2 (1/5) |  082 (30.6%) | 103 (15.7%) | 185 (20.0%)
0.0 (0/5) |  057 (21.3%) |  048 (07.3%) | 105 (11.4%)
Total | 268 | 656 | 924

Figure 3. Distribution of human-likeness scores averaged per dialogue

##  4\. Analysis

We then considered the possibility of the proposed evaluation method by
investigating the relationship between the annotated human-likeness scores and
the multimodal user behaviors.

###  4.1. Evaluation of the human-likeness scores from multimodal user
behaviors

Table 2. Correlation coefficients between human-likeness scores and objective
user behaviors

Behavior | Corr. ($r$)
---|---
(Voice activity) |
Total utterance time |  -0.35
Average utterance duration | 0.17
# of utterance | 0.11
(Linguistic) |
# of words | 0.16
# of unique words |  -0.33
# of content words | 0.11
# of unique content words |  -0.30
(Gaze) |
# of gaze shift (eye contact) |  -0.21
Total gaze duration | 0.00
Average gaze duration | -0.12
(Dialogue) |
# of turns | 0.07
Average turn duration | 0.04
Average switching pause length | -0.35
# of backchannels | 0.11
# of fillers | 0.05
# of laughs | 0.15
# of disfluencies | -0.03

The aim of this study is to evaluate the human-likeness scores of SCRs from
multimodal user behaviors. Here, we examined the correlation between the
multimodal user behaviors listed in Table 2 and the human-likeness scores.
These behaviors can be categorized into four groups: voice activity,
linguistic, gaze, and dialogue. These behaviors are based on manually
annotated data, but ones such as voice activity can be extracted
automatically. Content words in the linguistic category are defined as nouns,
verbs, adjectives, adverbs, and conjunctions. The numerical value of each
behavior was calculated by averaging those across multiple dialogue segments
used in the previous section.

By investigating Spearman’s rank correlation coefficients, we observed weak
correlations in several behaviors. Figure 4 illustrates the correlations on
the top-4 user behaviors. Total utterance time and the number of uttered
unique words showed a higher correlation coefficient. Likewise, the number of
gaze shifts and the average switching pause length manifested augmented
correlation. This result indicates the potential of inferring the human-
likeness score from these multimodal behaviors.

Therefore, we explored the extent to which the aforementioned behaviors can
estimate the human-likeness scores. We conducted leave-one-out cross-
validation using support vector regression. The target variable was the human-
likeness score, and the explanatory variables were the numerical values of the
user behaviors listed in Table 2. The evaluation metric employed was the mean
absolute error (MAE). Consequently, the average MAE amounted to 0.146. Given
that the current dataset consists of values incremented by 0.2, it has been
demonstrated that estimating the human-likeness score with an error of less
than or equal to one increment is feasible.

Figure 4. Relationship between human-likeness scores and top-4 user behaviors

###  4.2. Relationship with subjective evaluation

To verify the generalizability and practicality of the human-likeness scores
used in this study, we also investigated the relationship with the
conventional subjective evaluation scores obtained in the attentive listening
dialogue. Note that the subjective evaluation items were made in the previous
study (Inoue et al., 2020b). Among the 19 evaluation items, there were five
items that exhibited weak correlations as listed in Table 3. For example, the
correlated items were “The robot understood the talk” ($r=0.39$) and “I was
satisfied with the dialogue” ($r=0.21$), which are important factors in the
attentive listening task. From these results, the human-likeness scores
demonstrate a certain degree of correlation with some subjective evaluations,
thereby confirming its generalizability and practicality.

Table 3. Correlation coefficients between human-likeness labels and subjective
evaluation scores

Question item | Corr.
---|---
(Robot behaviors)
Q1 |  The words uttered by the robot were natural |  -0.07
Q2 |  The robot responded with good timing |  -0.25
Q3 |  The robot responded diligently |  -0.07
Q4 |  The robot’s reaction was like a human’s |  -0.19
Q5 |  The robot’s reaction adequately encouraged my talk |  -0.07
Q6 |  The frequency of the robot’s reaction was adequate |  -0.05
(Impression on the robot)
Q7 |  I want to talk with the robot again |  -0.08
Q8 |  The robot was easy to talk with |  -0.12
Q9 |  I felt the robot is kind |  -0.10
Q10 |  The robot listened to the talk seriously |  -0.18
Q11 |  The robot listened to the talk with focus |  -0.15
Q12 |  The robot listened to the talk actively |  -0.13
Q13 |  The robot understood the talk |  -0.39
Q14 |  The robot showed interest for the talk |  -0.20
Q15 |  The robot showed empathy towards me |  - 0.17
Q16 |  I think the robot was being operated by a human | -0.30
Q17 |  The robot was good at taking turns |  -0.19
(Impression on the dialogue)
Q18 |  I was satisfied with the dialogue |  -0.21
Q19 |  The exchange in the dialogue was smooth |  -0.19

##  5\. Conclusion

In this paper, we proposed a method to evaluate socially-situated
conversational robots based on observable and multimodal user behaviors. We
utilized attentive listening dialogue data for annotation of the human-
likeness of the robot, revealing a correlation between the user behaviors and
the human-likeness scores. Additionally, we demonstrated the ability to
predict the scores with an average MAE of 0.146. Future work will involve
examining the proposed evaluation method in a wider range of social situations
to confirm its generalizability. For example, we are now extending this work
to job interviews and first-time meeting scenarios where the role of the
robots is different from the one in the attentive listening scenario.

## Acknowledgement

This work was supported by JST ACT-X (JPMJAX2103), JST Moonshot R&D
(JPMJPS2011), and JSPS KAKENHI (JP19H05691 and JP23K16901). The authors also
express appreciation to Dr. Masato Komuro for his insightful comments on this
research.

## References

  * (1)
  * Abd-Alrazaq et al. (2020) Alaa Abd-Alrazaq, Zeineb Safi, Mohannad Alajlani, Jim Warren, Mowafa Househ, Kerstin Denecke, et al. 2020\.  Technical metrics used to evaluate health care chatbots: scoping review.  _Journal of medical Internet research_ 22, 6 (2020).
  * Casas et al. (2020) Jacky Casas, Marc-Olivier Tricot, Omar Abou Khaled, Elena Mugellini, and Philippe Cudré-Mauroux. 2020\.  Trends & methods in chatbot evaluation. In _Companion Publication of ICMI_. 280–286.
  * Deriu et al. (2021) Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021.  Survey on evaluation methods for dialogue systems.  _Artificial Intelligence Review_ 54 (2021), 755–810.
  * DeVault et al. (2014) David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, Gale Lucas, Stacy Marsella, Fabrizio Morbini, Angela Nazarian, Stefan Scherer, Giota Stratou, Apar Suri, David Traum, Rachel Wood, Yuyu Xu, Albert Rizzo, and Louis P. Morency. 2014\.  SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support. In _AAMAS_. 1061–1068.
  * Edlund et al. (2008) Jens Edlund, Joakim Gustafson, Mattias Heldner, and Anna Hjalmarsson. 2008.  Towards human-like spoken dialogue systems.  _Speech Communication_ 50, 8 (2008), 630–645.
  * Han et al. (2013) Sangdo Han, Kyusong Lee, Donghyeon Lee, and Gary Geunbae Lee. 2013\.  Counseling Dialog System with 5W1H Extraction. In _SIGDIAL_.
  * Iio et al. (2020) Takamasa Iio, Satoru Satake, Takayuki Kanda, Kotaro Hayashi, Florent Ferreri, and Norihiro Hagita. 2020\.  Human-like guide robot that proactively explains exhibits.  _International Journal of Social Robotics_ 12 (2020), 549–566.
  * Inoue et al. (2020a) Koji Inoue, Kohei Hara, Divesh Lala, Kenta Yamamoto, Shizuka Nakamura, Katsuya Takanashi, and Tatsuya Kawahara. 2020a.  Job interviewer android with elaborate follow-up question generation. In _ICMI_. 324–332.
  * Inoue et al. (2020b) Koji Inoue, Divesh Lala, Kenta Yamamoto, Shizuka Nakamura, Katsuya Takanashi, and Tatsuya Kawahara. 2020b.  An attentive listening system with android ERICA: Comparison of autonomous and WOZ interactions. In _SIGDIAL_. 118–127.
  * Inoue et al. (2016) Koji Inoue, Pierrick Milhorat, Divesh Lala, Tianyu Zhao, and Tatsuya Kawahara. 2016.  Talking with ERICA, an autonomous android. In _SIGDIAL_. 212–215.
  * Johnston et al. (2013) Michael Johnston, Patrick Ehlen, Frederick G. Conrad, Michael F. Schober, Christopher Antoun, Stefanie Fail, Andrew Hupp, Lucas Vickers, Huiying Yan, and Chan Zhang. 2013\.  Spoken Dialog Systems for Automated Survey Interviewing. In _SIGDIAL_. 329–333.
  * Kawahara (2018) Tatsuya Kawahara. 2018\.  Spoken dialogue system for a human-like conversational robot ERICA. In _IWSDS_. 65–75.
  * Laranjo et al. (2018) Liliana Laranjo, Adam G Dunn, Huong Ly Tong, Ahmet Baki Kocaballi, Jessica Chen, Rabia Bashir, Didi Surian, Blanca Gallego, Farah Magrabi, Annie YS Lau, et al. 2018\.  Conversational agents in healthcare: a systematic review.  _Journal of the American Medical Informatics Association_ 25, 9 (2018), 1248–1258.
  * Lin et al. (2022) Ting-En Lin, Yuchuan Wu, Fei Huang, Luo Si, Jian Sun, and Yongbin Li. 2022.  Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems. In _SIGKDD_. 3299–3308.
  * Meena et al. (2015) Raveesh Meena, José Lopes, Gabriel Skantze, and Joakim Gustafson. 2015.  Automatic Detection of Miscommunication in Spoken Dialogue Systems. In _SIGDIAL_. 354–363.
  * Ni et al. (2023) Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2023.  Recent advances in deep learning based dialogue systems: A systematic survey.  _Artificial intelligence review_ 56 (2023), 3055–3155.
  * Oertel et al. (2021) Catharine Oertel, Patrik Jonell, Dimosthenis Kontogiorgos, Kenneth Funes Mora, Jean-Marc Odobez, and Joakim Gustafson. 2021.  Towards an engagement-aware attentive artificial listener for multi-party interactions.  _Frontiers in Robotics and AI_ 8 (2021).
  * Rasouli et al. (2022) Samira Rasouli, Garima Gupta, Elizabeth Nilsen, and Kerstin Dautenhahn. 2022.  Potential applications of social robots in robot-assisted interventions for social anxiety.  _International Journal of Social Robotics_ 14 (2022), 1–32.
  * Schröder et al. (2015) Marc Schröder, Elisabetta Bevacqua, Roddy Cowie, Florian Eyben, Hatice Gunes, Dirk Heylen, Mark ter Maat, Gary McKeown, Sathish Pammi, Maja Pantic, Catherine Pelachaud, Björn Schuller, Etienne de Sevin, Michel Valstar, and Martin Wöllmer. 2015\.  Building autonomous sensitive artificial listeners. In _ACII_. 456–462.
  * Scoglio et al. (2019) Arielle AJ Scoglio, Erin D Reilly, Jay A Gorman, and Charles E Drebing. 2019.  Use of social robots in mental health and well-being research: Systematic review.  _Journal of medical Internet research_ 21, 7 (2019).
  * Sim and Loo (2015) Doreen Ying Ying Sim and Chu Kiong Loo. 2015.  Extensive assessment and evaluation methodologies on assistive social robots for modelling human–robot interaction – A review.  _Information Sciences_ 301 (2015), 305–344.
  * Swartout et al. (2010) William Swartout, David Traum, Ron Artstein, Dan Noren, Paul Debevec, Kerry Bronnenkant, Josh Williams, Anton Leuski, Shrikanth Narayanan, Diane Piepol, Chad Lane, Jacquelyn Moriel, Priti Aggarwal, Matt Liewer, Jen-Yuan Chiang, Jillian Gerten, Selina Chu, and Kyle White. 2010\.  Ada and Grace: Toward realistic and engaging virtual museum guides. In _IVA_. 286–300.
  * Ultes and Maier (2021) Stefan Ultes and Wolfgang Maier. 2021.  User Satisfaction Reward Estimation Across Domains: Domain-independent Dialogue Policy Learning.  _Dialogue & Discourse_ 12, 2 (2021), 81–114.
  * Yu et al. (2017) Zhou Yu, Vikram Ramanarayanan, Patrick Lange, and David Suendermann-Oeft. 2017.  An open-source dialog system with real-time engagement tracking for job interview training applications. In _IWSDS_.
  * Zhang et al. (2021) Chen Zhang, João Sedoc, Luis Fernando D’Haro, Rafael Banchs, and Alexander Rudnicky. 2021\.  Automatic evaluation and moderation of open-domain dialogue systems.  _arXiv preprint, 2111.02110_ (2021).


# Abuse is Contextual, What about NLP?
The Role of Context in Abusive Language Annotation and Detection

Stefano Menini
Fondazione Bruno Kessler
Trento, Italy
<EMAIL_ADDRESS>
Alessio Palmero Aprosio
Fondazione Bruno Kessler
Trento, Italy
<EMAIL_ADDRESS>
Sara Tonelli
Fondazione Bruno Kessler
Trento, Italy
<EMAIL_ADDRESS>

###### Abstract

The datasets most widely used for abusive language detection contain lists of
messages, usually tweets, that have been manually judged as abusive or not by
one or more annotators, with the annotation performed at message level. In
this paper, we investigate what happens when the hateful content of a message
is judged also based on the context, given that messages are often ambiguous
and need to be interpreted in the context of occurrence. We first re-annotate
part of a widely used dataset for abusive language detection in English in two
conditions, i.e. with and without context. Then, we compare the performance of
three classification algorithms obtained on these two types of dataset,
arguing that a context-aware classification is more challenging but also more
similar to a real application scenario.

##  1 Introduction

‘Abuse is contextual’ is one of the key claims reported in Prabhakaran et al.
(2020), where the authors describe the content of a panel with NLP
practitioners and human rights experts at the Human Rights Conference
RightsCon 2020. A similar remark is made in Jurgens et al. (2019) in a recent
position paper on current NLP methods to fight online abuse, arguing that NLP
research tends to consider a narrow scope of what constitutes abuse, without
respecting, for instance, community norms in classification decisions.
Similarly, Vidgen et al. (2019) claim that one of the main research challenges
concerning abusive language detection is accounting for context, although they
focus mainly on user-level variables and network representations.

In this work, we present an analysis aimed at better understanding what
context is in abusive language detection, and what its effects are on data
annotation and classification. We focus on discourse context, i.e. the
messages preceding a given tweet, because it has been relatively understudied
in abusive language detection, while user-level and network-level features
have already been extensively discussed in past works Fehn Unsvåg and Gambäck
(2018); Del Tredici et al. (2019); Liu and Weber (2014); Sap et al. (2019);
Caselli et al. (2020).

Consider for example the following tweet:

> @User1: hell yeah bitches hate that shit

Without any context, an annotator may think that it has an abusive content,
aimed at offending someone by calling them ‘bitches’, and its offensive intent
is strengthened by the use of ‘that shit’. However, our judgement would change
if we knew that the preceding tweet is:

> @User2 Ima stop calling females bitches next week sometime. Just tryin to
> get that bitch word out of my vocabulary

Indeed, the context would show that the two users are probably friends and
that the message by @User1 is a sort of joke. The main goal of this paper is
to analyse what is the impact of cases like this when detecting abusive
language on Twitter, both manually and automatically. In particular, the main
research questions we want to address are:

  * R1:

How would abusive language annotation change if we provided annotators also
with the textual context preceding a tweet? Would abusive judgements increase
or decrease?

  * R2:

What would classification performance be like if we used abusive language
datasets annotated with context? Which algorithms would work best and how
would the performance change compared to single-tweet classification?

To address the above questions, we perform a manual annotation of around 8,000
English tweets in two different conditions, with and without context, and
analyse what linguistic phenomena contribute to different annotations of the
same tweets in the two settings (Section 4). We also compare three supervised
classification algorithms, described in Section 5 and 6, and evaluate their
different performance on the data (Section 7). Additional remarks on the
experiments and a proposal on how abusive language detection tasks should be
cast are discussed in Section 8. The annotated datasets are made available at
https://github.com/dhfbk/twitter-abusive-context-dataset.

##  2 Related work

The problem of detecting abusive language on social media has recently gained
a lot of attention in the NLP community, thanks also to shared tasks on the
topic Wiegand et al. (2018); Bosco et al. (2018); Basile et al. (2019);
Carmona et al. (2018); Zampieri et al. (2019, 2020) and the consequent
availability of benchmarks. Most of the systems participating in the above
tasks rely on deep learning and use embeddings as features, sometimes enriched
with tweet-based surface features like the presence of uppercased words or
exclamation marks. For example, OffensEval organisers at Semeval 2019 report
that 70% of participating systems use deep learning Zampieri et al. (2019),
and that the best performing one in the binary classification task Liu et al.
(2019) is based on fine-tuned Bidirectional Encoder Representation from
Transformer (BERT) Devlin et al. (2019). Also HateEval organisers at Semeval
2019, focused on hate speech detection against immigrants and women, report
that most submitted runs rely on deep learning Basile et al. (2019), although
the top-ranked system in the binary task for English uses sentence embeddings
as features to be fed into an SVM-based classifier Indurthi et al. (2019).

While most approaches to hate speech detection take into account only features
related to the single message to be classified Park and Fung (2017); Zhang et
al. (2018b); Kshirsagar et al. (2018); Lee et al. (2018), recent studies have
highlighted the importance of context in interpreting the meaning of a message
and its possible hateful content Jurgens et al. (2019); Vidgen et al. (2019);
Caselli et al. (2020). However, context can concern different aspects: some
refer to user-level features and profiles Vidgen et al. (2019), arguing that
an aggressive attitude can be identified at profile level. Some other works
suggest that incorporating information about the network structure of a user
is beneficial to the detection of hate speech and abusive behaviour Chatzakou
et al. (2017); Fehn Unsvåg and Gambäck (2018); Del Tredici et al. (2019). Some
other works suggest to consider also the annotator profile, showing that
messages written in a specific dialect may be judged differently by annotators
with a different ethnicity Sap et al. (2019), biasing the creation of datasets
and therefore of supervised systems.

Analysing the text surrounding a given message to evaluate its offensiveness,
instead, is a rather understudied problem. Some works have modeled and
classified threads from Reddit Almerekhi et al. (2019) and FoxNews Gao and
Huang (2017) to detect words and topics that trigger toxicity, or to find
early discussion features that can predict controversiality, modeling not just
the content of the initiating post, but also the content and structure of the
initial responding comments Hessel and Lee (2019). These works are however
different from ours both in terms of task, of social media platform and of
thread structure. Other works have focused on Wikipedia, analysing linguistic
cues of ongoing conversations to predict their future trajectory Zhang et al.
(2018a), or trying to assess whether a comment is toxic based only on the
preceding messages Karan and Šnajder (2019). The most similar work to ours is
probably Pavlopoulos et al. (2020), whose goal is to assess the contribution
of context in toxicity detection. Surprisingly, they come to the opposite
conclusion, showing that including the textual context increases the perceived
toxicity. However, their setting is different, in that they focus on Wikipedia
Talk Pages and include as context only the parent comment and the discussion
title. Furthermore, the corpus that they create with a setting similar to
ours, i.e. by crowd-annotating with and without context, contains only 250
comments. The same authors acknowledge that larger annotated datasets would be
needed to estimate more accurately how often context amplifies or mitigates
toxicity.

##  3 Data Extraction and Analysis

In order to analyse the role of context on data annotation and classification,
we focus on the dataset presented in Founta et al. (2018), which originally
included around 100K tweets annotated with four labels: hateful, abusive, spam
or none. The authors used a bootstrapping approach to sampling tweets, which
were labelled by several crowdsource workers and then validated. Differently
from other hate speech datasets, this was not created starting from a set of
predefined offensive terms or hashtags so as to reduce bias, a main issue in
hate speech datasets Wiegand et al. (2019). Also for this reason, the dataset
has been widely used in experiments for hate speech classification in English.

Since the dataset includes the tweet IDs, we used this information to query
the Twitter API and retrieve the complete thread including the given message,
so to leverage also the context of each tweet. We then mapped the hateful and
abusive tweets to our abusive class, while removing the spam instances. An
overview of the dataset is reported in Table 1. The API was queried between
August and September 2019 and the retrieval was problematic because some of
the tweets were not available anymore, confirming similar remarks in previous
works Klubicka and Fernández (2018).

| Original | Retrieved | At least 1
---|---|---|---
| Dataset |  | before
Abusive | 31,985 | 12,357 | 1,931
Not abusive | 67,814 | 37,091 | 6,087
Total | 99,800 | 49,448 | 8,018
Duplicates | 2,017 |  |
Table 1: Statistics on three versions of the dataset by Founta et al. (2018):
original, retrieved by us, subset having a previous context

After removing the tweets no longer available online and the ones without a
preceding context, we obtained a new dataset consisting of 8,018 annotated
tweets. As expected, the removal of messages affects more the abusive
category, for which only 38% of the tweets could be retrieved, while for the
majority non abusive class 55% of the messages could be found. This may depend
on the fact that users decided to remove messages from their account, but also
that some profiles were suspended in compliance with Twitter policy.

We also observed that the original dataset contains duplicated tweets (2,017
in total), probably because both original and re-tweeted messages were
annotated, and we therefore removed them.

##  4 Manual (Re)annotation

Since one of the goals of this work is to understand whether the textual
context plays any role in detecting abusive messages online, we perform a
manual re-annotation of the tweets with context. In particular, during the
task, annotators not only read the single tweet but also the preceding context
and are asked to judge the abusiveness of the message taking the whole
conversation into account. We use Amazon Mechanical Turk platform to perform
the annotation of 8,018 tweets with context.

The guidelines provided to crowd-workers explicitly require the tweet to be
annotated as abusive or not considering also what was written before in the
thread. To ensure a good quality of collected annotations, we introduced gold
standard examples in the task (one every five tweets) and restricted the
access to English native speakers. All requirements introduced by Mechanical
Turk for tasks containing adult content, for example adding a warning in the
task title, were implemented. For each tweet we collected 3 annotations, and
the majority label was finally assigned. Inter-annotator agreement was 0.713
multi-rater free-marginal kappa with 85.6% of labels in agreement.

In order to enable a fair comparison between annotation with and without
context, we re-annotated the same tweets using the same methodology without
providing crowd-workers with the tweet context. This second annotation was
performed selecting the same pool of crowd-workers as for the first task, so
to reduce variability due to annotators’ bias and preferences. The task was
launched around three months after the completion of the first one, making it
unlikely that crowd-workers remember the labels assigned to the same tweets
with context. This re-annotation without context was performed also to make
sure that the two annotated datasets were tagged following exactly the same
guidelines. Indeed, the original annotation provided in Founta et al. (2018)
was performed by adjusting incrementally the definition of the abusive
categories to be annotated. Although they could be mapped to a binary
classification, the way in which the annotation process was structured may
lead to different results compared to our setting.

| Annotations without context (SINGLE-TWEET) | Annotations with context (CONTEXT)
---|---|---
| At least 1 | # previous tweets: | At least 1 | # previous tweets:
| tweet before | 1 | 2-5 | 6+ | tweet before | 1 | 2-5 | 6+
Abusive | 1,472 | 699 | 608 | 165 | 800 | 404 | 324 | 72
Not Abusive | 6,546 | 2,794 | 2,747 | 1,005 | 7,218 | 3,089 | 3,031 | 1,098
Total | 8,018 | 3,493 | 3,355 | 1,170 | 8,018 | 3,493 | 3,355 | 1,170
Table 2: Comparison of re-annotated tweets providing crowd-workers with no
context (left) or preceding context (right)

In Table 2 we report the distribution of abusive and non-abusive messages in
the dataset annotated with and without context (CONTEXT and SINGLE-TWEET
respectively). When annotated with context, the abusive class contains 672
tweets less than in the annotation without context, and its weight decreases
from 18% to 10%. This shows that the main effect of context is to reduce the
perceived offensiveness of messages, enabling annotators to understand the
tone of the conversation and sometimes the relation between users. The example
reported below shows clearly this phenomenon: if we consider the last tweet
(in italics) in isolation, we may think that @User2 is aggressively
terminating a phone call. Instead, looking at the context we notice that this
tweet is just a funny reply to the call mentioned by @User1.

> @User1: My next call: ‘Hi. I sent an email. I got no response. Is that
> because you’re fucking handling it, or fucking ignoring it?’ Wish me luck.
> @User2: ‘I’m fucking hanging up now.’ $\ast$click$\ast$

Another example, reported below, shows how the ironic tweet by @User2 may be
labeled as abusive without the previous context:

> @User1: Penn State trustee says he’s not ‘totally out’ of sympathy for
> Sandusky victims.
> @User2: Yeah. Those damn victims…they’re the worst! $\ast$shakes fist in the
> air$\ast$

Interestingly, the longer the context, the more tweets are considered non
abusive by annotators: for the group of tweets having a context longer than 5
tweets, only 6% of them are annotated as abusive. This percentage increases up
to 10% when the context is between 2 and 5 tweets, and 12% when only one tweet
is in the context.

Other cases include tweets whose meaning was unclear or underspecified, for
example because of the presence of anaphoric expressions, ellipsis, incomplete
messages split across several tweets, etc. In all these cases, the presence of
context allows for a more informed decision, reducing the interpretation
effort required to annotators. A general trend that we observe is that,
whenever a profanity is present in a tweet, it is more likely to be annotated
as abusive in the SINGLE-TWEET setting, while it is often labeled as non
abusive in the CONTEXT configuration. To measure this, we compute the mean
square contingency coefficient (Phi coefficient) aimed at capturing the
association between the presence of profanities and the ‘abusive’ label. We
download the Google profanity words
list111https://github.com/RobertJGabriel/Google-profanity-
words/blob/master/list.txt and check for each tweet if a profanity is present
or not. The Phi coefficient is 0.77 for the SINGLE-TWEET setting, showing a
very strong positive correlation between profanities and abusive labels, while
it is only moderate (0.39) in the CONTEXT configuration. This finding is in
line with past works, showing that the offensiveness of swearing is context-
dependent Pamungkas et al. (2020) and that profanities are frequently used
among people with a strong social relationship without any offensive goal Bak
et al. (2012), but rather to signal an informal attitude among the members of
a community or even intimacy Pei and Jurgens (2020).

##  5 Classification algorithms

As a next step in our study, we want to analyse the classification performance
obtained with different algorithms on the newly annotated datasets: SINGLE-
TWEET was annotated by providing crowd-workers with tweets in isolation, and
the same information will be used by the classifiers, similar to the standard
setting of hate speech detection competitions Zampieri et al. (2019, 2020).
The second, CONTEXT, was annotated by showing crowd-workers the tweet to be
classified as well as all the preceding tweets retrieved from Twitter.
Likewise, the classifiers will be fed also with the context at training and
test time.

We compare two widely used deep-learning approaches, i.e. BERT Devlin et al.
(2019) and Bidirectional LSTM (BiLSTM) Graves and Schmidhuber (2005), as well
as a non-neural one based on Support Vector Machines Cortes and Vapnik (1995).
Details on the different algorithms are reported below.

BERT: This is a language representation model developed by Google Research
Devlin et al. (2019), whose deep learning architecture obtained state-of-the-
art results in several natural language processing tasks including sentiment
analysis, natural language inference, textual entailment Devlin et al. (2019)
and hate speech detection Liu et al. (2019). BERT can be fine-tuned and
adapted to specific tasks by adding just one additional output layer to the
neural network. We use the base model of BERT for English,222Uncased,
12-layer, 768-hidden, 12-heads, 110M parameters. trained on 3.3 billion words,
which is made available on the project website.333https://github.com/google-
research/bert For the CONTEXT data, we adopt the configuration proposed in
Devlin et al. (2019) for sentence pair classification tasks: the input tweets
are given to the classifier as a pair of text chunks, with the first text of
the pair containing the context preceding the tweet, and the second one
containing the target tweet. For SINGLE-TWEET data, only the string of the
tweet is given in input to BERT.

Bidirectional LSTMs: BiLSTMs Graves and Schmidhuber (2005) are made of a
combination of two Long Short-Term Memory networks, a particular case of
recurrent neural network, i.e. a classifier optimized to process sequences.
Using this combination, BiLSTM connects information both from previous and
following statuses simultaneously. For CONTEXT data, we test BiLSTM in two
configurations. In the first setting, we use a sequence of word vectors as
input. Regardless the span of the context considered, the entire conversation
(the preceding context plus the tweet) is considered as a single text, and
each word is represented in the network through word embeddings. The network
is configured with three bidirectional layers with respectively 128, 64 and 32
neurons.

In the second setting the context and the tweets are represented again as a
sequence of word embeddings. Instead of considering them as a single text, we
keep the tweet and its preceding messages separated. Both the context and the
tweet move through two bidirectional layers of respectively 64 and 32 neurons,
before being merged into one to generate a prediction. Both configurations are
tested with different values of dropout and with different optimizers (see
details in the following Section). For SINGLE-TWEET data, only the first
setting is adopted.

SVM: For the classification we use LIBSVM Chang and Lin (2011). As for the
CONTEXT data, the tweet and its context are represented by concatenating a
300-dimension sentence embeddings generated from the context and a
300-dimension sentence embeddings for the tweet (see Section 6 for more
details). For SINGLE-TWEET, the classifier is fed only with a vector of 300
dimensions.

In our experiments, we also evaluated other approaches, i.e. Feed Forward
Networks, Long Short-Term Memory networks and Gated Recurrent Units. All
performed worse compared to the classification algorithms described above,
therefore we do not include them in the paper.

##  6 Experiments

###  6.1 Preprocessing

Before running the classification experiments, we preprocess the tweets as
follows: we split hashtags using Ekphrasis Gimpel et al. (2010), which
recognises the tokens in a hashtag based on Google n-grams. The same tool is
also used to replace mentions with “username” and urls with “url”. We also
replace every instance of “money”, “time”, “date” (and in general any
“number”) with a generic label. We convert each emoji into a textual
description using the Python emoji
package,444https://github.com/carpedm20/emoji/ as suggested in previous work
Singh et al. (2019).

Then, since BERT classifier accepts as input sentences or sets of sentences,
we just pass the pre-processed tweet text or its context as is. For the two
other approaches, the text is converted into vectors in two ways, both using
FastText, a library for efficient learning of word and sentence
representations Bojanowski et al. (2017); Grave et al. (2018). In particular,
all vectors are extracted starting from the pre-trained embeddings obtained
from the Common Crawl corpus.555https://fasttext.cc/docs/en/crawl-vectors.html
Since SVM takes in input sentence embeddings, we convert the context and the
current tweet into sentence embeddings of 300 dimensions each. Since BiLSTM
takes in input word embeddings, each word is represented as a vector of size
300 and a tweet (or context) is then obtained by merging the resulting set of
vectors into a matrix, used as input for the machine learning algorithm. In
the first setting, a single matrix is generated, containing all the vectors
extracted from both the tweet and the context (if any). In the second setting,
the matrices of the tweet and the context are generated separately, two
different neural networks are created and then merged in the last (common)
layer.

###  6.2 Experimental Setup

The dataset we use contains 8,018 tweets, that in one of the two
configurations are preceded by a context. Note that the class balance of
SINGLE-TWEET and CONTEXT is different (details in Table 2). As for the CONTEXT
data, we experiment with different context lengths, ranging from 1 (i.e. only
the preceding tweet) to all. Since not all contexts have the same length, we
replace missing tweets in the context with empty strings when a configuration
requires a context length that is higher than the available one. This
simulates a real scenario, in which not every tweet has a context of the same
size.

All the configurations and algorithms are evaluated randomly dividing the data
into training (80%), validation (10%) and test (10%). Each experiment is
repeated three times with different random splits, and the results are
averaged.

For BiLSTM,666The BiLSTM is created with Keras 2.2.5, and runs with
Tensorflow-gpu 1.10.1.  different experiments with a dropout ranging from 0 to
0.5 have been compared. We report only the best results, obtained with dropout
= 0.3. The loss function is binary cross-entropy and two optimizers (i.e. SGD
and Adam) have been evaluated, although we report only the results obtained
with SGD which yields a better performance. The BiLSTM is tested with up to 20
epochs, the results in Table 3 and 4 are obtained with 10 epochs. For BERT the
best performance, reported in Table 3 and 4, is obtained with 3 epochs (we
tested up to 10 epochs), a batch size of 16, a max length of 256 and learning
rate 2e-5. Every component of the code relying on randomization uses 42 as
seed. For SVM, we test different types of kernels (i.e. linear, polynomial or
radial) and cost parameter (up to 100), reporting only the results obtained
with the best configuration, i.e. linear kernel and $C=50$.777For the
experiments we use 4 Tesla P40. The BiLSTM (single text) takes, according to
the context size, around one minute per epoch with 2.5M - 4.9M total
parameters. The BiLSTM (separate context) takes, according to the context,
from 5 to 18 minutes per epoch using 15.1M - 21.5M total parameters. BERT was
able to run in 5 to 10 minutes per epoch.

##  7 Evaluation

We report in Table 3 the classification results obtained on the SINGLE-TWEET
dataset, while in Table 4 the classifier performance on the CONTEXT dataset is
presented, with a context window ranging from one tweet to all those available
in the preceding context.

Classifier | P_noAbuse | R_noAbuse | F1_noAbuse | P_Abuse | R_Abuse | F1_Abuse | P | R | F1
---|---|---|---|---|---|---|---|---|---
BERT | 0.971 | 0.952 | 0.961 | 0.791 | 0.865 | 0.826 | 0.881 | 0.908 |  0.893 $\pm$ 0.007
BiLSTM | 0.952 | 0.958 | 0.955 | 0.796 | 0.775 | 0.785 | 0.874 | 0.866 | 0.870 $\pm$ 0.004
SVM | 0.943 | 0.970 | 0.956 | 0.836 | 0.721 | 0.773 | 0.889 | 0.845 | 0.864 $\pm$ 0,013
Majority | 0.816 | 1.000 | 0.899 | 0.000 | 0.000 | 0.000 | 0.408 | 0.500 | 0.449

Table 3: Classification results on SINGLE-TWEET dataset, average of three runs
(with $\pm$ StDev). Precision, Recall and F1 are macro-averaged.

Context | P_noAbuse | R_noAbuse | F1_noAbuse | P_Abuse | R_Abuse | F1_Abuse | P | R | F1
---|---|---|---|---|---|---|---|---|---
BERT
1 | 0.951 | 0.970 | 0.960 | 0.643 | 0.517 | 0.572 | 0.797 | 0.743 |  0.766 $\pm$ 0.017
2 | 0.952 | 0.965 | 0.958 | 0.610 | 0.528 | 0.565 | 0.781 | 0.747 | 0.761 $\pm$ 0.017
3 | 0.950 | 0.967 | 0.959 | 0.620 | 0.511 | 0.560 | 0.785 | 0.739 | 0.759 $\pm$ 0.025
All | 0.950 | 0.967 | 0.958 | 0.619 | 0.508 | 0.554 | 0.784 | 0.738 | 0.756 $\pm$ 0.018
SVM
1 | 0.926 | 0.964 | 0.945 | 0.436 | 0.265 | 0.329 | 0.681 | 0.614 | 0.637 $\pm$ 0.019
2 | 0.928 | 0.962 | 0.945 | 0.446 | 0.291 | 0.352 | 0.687 | 0.627 | 0.649 $\pm$ 0.024
3 | 0.928 | 0.966 | 0.946 | 0.466 | 0.287 | 0.355 | 0.697 | 0.626 | 0.651 $\pm$ 0.024
All | 0.928 | 0.966 | 0.947 | 0.475 | 0.291 | 0.360 | 0.702 | 0.629 | 0.654 $\pm$ 0.018
BiLSTM (single text)
1 | 0.939 | 0.956 | 0.947 | 0.494 | 0.404 | 0.439 | 0.716 | 0.680 | 0.693 $\pm$ 0.039
2 | 0.938 | 0.957 | 0.948 | 0.488 | 0.397 | 0.433 | 0.713 | 0.677 | 0.690 $\pm$ 0.045
3 | 0.939 | 0.958 | 0.948 | 0.499 | 0.403 | 0.441 | 0.719 | 0.680 | 0.695 $\pm$ 0.040
All | 0.937 | 0.960 | 0.948 | 0.500 | 0.381 | 0.428 | 0.718 | 0.671 | 0.688 $\pm$ 0.046
BiLSTM (separate context)
1 | 0.929 | 0.974 | 0.951 | 0.581 | 0.282 | 0.355 | 0.755 | 0.628 | 0.653 $\pm$ 0.040
2 | 0.931 | 0.974 | 0.952 | 0.615 | 0.302 | 0.378 | 0.773 | 0.638 | 0.665 $\pm$ 0.039
3 | 0.926 | 0.974 | 0.950 | 0.548 | 0.254 | 0.314 | 0.737 | 0.614 | 0.632 $\pm$ 0.064
All | 0.928 | 0.971 | 0.949 | 0.549 | 0.274 | 0.333 | 0.738 | 0.623 | 0.641 $\pm$ 0.050
Baseline (majority class)
| 0.900 | 1.000 | 0.947 | 0.000 | 0.000 | 0.000 | 0.450 | 0.500 | 0.474

Table 4: Classification results on CONTEXT dataset, average of three runs
(with $\pm$ StDev). Precision, Recall and F1 are macro-averaged. ‘Context’
column shows the number of preceding tweets considered as context.

Some considerations concerning the classifiers’ performance hold for both
datasets. Indeed, BERT-based classification yields the best score both in the
SINGLE-TWEET and in the CONTEXT setting, while SVM achieves the worst
performance. For SVM, we tested also a different way to model the context,
creating a single 300-dimensional FastText embedding for the context and the
tweet, but the classifier performance decreased. Concerning BiLSTM, modelling
the context and the tweet to classify as a single text is better than
separating them in two layers. In both settings, BERT yields a much better F1
on the abusive class, the minority one, showing that it can deal with
unbalanced classes better than the two other approaches. We tested other
strategies to give the tweet and the context in input to BERT, for example by
merging them in a single text rather than split in two columns, following the
setting proposed for single sentence classification tasks in Devlin et al.
(2019), but the overall performance decreased.

Concerning the differences between the two settings, we observe that with
SINGLE-TWEET all classifiers achieve a better performance than with CONTEXT,
and this result is consistent across all algorithms and configurations. Our
results suggest that the classification based only on single messages may be
easier than the context-aware one, not only because of a simpler setup of the
classification algorithms, but also because all the lexical information
necessary to interpret the abusiveness of a tweet are present in the same
message. This makes it easier for the classifier to capture the connections
between lexical elements and the corresponding class, for example between
profanities and the (supposed) abusiveness. Indeed, when including only single
tweets, F1 is 0.893, in line with the results reported by participants at the
last SemEval task on abusive language detection for English, where 60 systems
achieve F1 $>$ 0.90 Zampieri et al. (2020). Using a context-aware annotation
and classification, instead, leads to a drop in performance, affecting in
particular the abusive class.

We observe also that the context size does not show a consistent impact.
Furthermore, different context lengths do not have a statistically significant
effect, which we measure for each classification algorithm using approximate
randomization test.

Since in the CONTEXT dataset the number of abusive tweets is only 800, the
lower performance achieved compared with SINGLE-TWEET may be due to the
greater class imbalance and the lower number of abusive instances in the
training set, rather than to differences in the task. To measure this effect,
we run a further comparison using the same BERT configuration on both
datasets, but downsampling them so that they contain the same number of
abusive and non abusive instances, i.e. 800 abusive, of which 633 for
training, and 6,456 non abusive, of which 5,232 for training. The results of
this comparison are reported in Table 5 as the average over three runs. This
additional comparison confirms that on the SINGLE-TWEET dataset BERT yields a
much better performance. It also shows that this difference is inherently due
to the way in which the task is modeled, rather than to the size and balance
of the classes. Indeed, the performance on this smaller dataset is even
slightly better than on the one obtained on both datasets before downsampling.

SINGLE-TWEET dataset
---
Context Size | P_noAbuse | R_noAbuse | F1_noAbuse | P_Abuse | R_Abuse | F1_Abuse | P | R | F1
0 | 0.963 | 0.971 | 0.967 | 0.859 | 0.822 | 0.839 | 0.911 | 0.897 | 0.903 $\pm$ 0.006
CONTEXT dataset
Context Size | P_noAbuse | R_noAbuse | F1_noAbuse | P_Abuse | R_Abuse | F1_Abuse | P | R | F1
1 | 0.956 | 0.958 | 0.957 | 0.591 | 0.585 | 0.587 | 0.774 | 0.771 | 0.772 $\pm$ 0.001
2 | 0.961 | 0.954 | 0.957 | 0.590 | 0.629 | 0.608 | 0.775 | 0.792 | 0.783 $\pm$ 0.013
3 | 0.957 | 0.960 | 0.958 | 0.598 | 0.582 | 0.589 | 0.777 | 0.771 | 0.774 $\pm$ 0.026
All | 0.954 | 0.961 | 0.957 | 0.598 | 0.555 | 0.575 | 0.776 | 0.758 | 0.766 $\pm$ 0.014

Table 5: Classification results using BERT on downsampled SINGLE-TWEET and
CONTEXT datasets

##  8 Discussion

The manual re-annotation of the dataset by Founta et al. (2018) including the
preceding tweets highlighted the relevant role played by context when deciding
whether a message is abusive or not. However, additional work is needed to
find better approaches for effectively including context in the
classification. Previous analyses on Wikipedia Karan and Šnajder (2019);
Pavlopoulos et al. (2020) highlighted similar difficulties in modelling the
textual content of a thread for automated classification. Our case is even
more challenging because of a major difference between Twitter and other
platforms: while on other social networks, whose goal is to explicitly discuss
a topic, it is rather easy to extract a conversation graph, identifying posts
and related replies (see e.g. the conversation trees from Reddit discussions
in Hessel and Lee (2019), the structure of Twitter threads is less stable.
Indeed, the list of tweets that we extract to build a context is not
necessarily a sequence of replies in a coherent conversation, but could
include gaps created by deleted messages. This could make the relation between
an abusive message and its context not clear, since the next messages are not
necessarily a reaction to what was said before.

Reshaping abusive language annotation and detection as a task that should take
the context into account is in our view relevant also because it would
contribute to mitigating the problem of disembodiment in NLP Waseem et al.
(2020), i.e. the fact that both data and machine learning encode subjective
content that is however presented as universalised and objective, with the
potential to create social marginalisation. For the abusive language case,
considering tweets in isolation would oversimplify the way in which online
abuses happen in reality, ignoring the fact that context gives a more informed
perspective on the circumstances that led to a supposed abuse. Although we are
aware that abusive language annotation is affected by different types of
biases, ranging from data sampling Wiegand et al. (2019); Ousidhoum et al.
(2020) to annotators’ preferences Waseem (2016), we believe that including the
local context in the task definition could be a first step towards a fairer
approach to data annotation and classification.

##  9 Conclusions

In this work we investigate the role of textual context in abusive language
detection on Twitter. We first manually re-annotate the tweets in the dataset
from Founta et al. (2018) for which the preceding tweets could be retrieved
online. Annotation was performed by the same crowd-workers in two different
time periods, asking them to take into account the context and then to
annotate the tweets in isolation. A comparison between the two datasets shows
that the context is sometimes necessary to understand the real intent of the
user, and that it is more likely to make a tweet non abusive, even if it
contains profanities. We therefore recommend that, in future annotation
efforts for abusive language detection, the textual context of messages is
also provided to annotators. This would have an impact also on the
implementation of classification systems, which should foresee strategies to
embed contextual information: the task would be more challenging, but also
more similar to a real application scenario. Our classification experiments
suggest also that the high performance achieved with BERT-based systems in
‘standard’ abusive language detection may be overly optimistic, because when
context is given in input to the same classifiers and they are evaluated on
context-aware datasets, their performance drops dramatically.

In the future, we hope that the release of the datasets presented in this
paper and our initial experiments will foster research in the direction of
context-aware abusive language detection, including both the creation of
guidelines and annotated data, as well as novel classification approaches.

As a next step, we plan to investigate novel ways to model context, for
example by analysing the role of direct responses containing an explicit
mention of the user. We also plan to further extend this approach to other
platforms, to assess whether platform-specific differences can be observed
regarding the role of context, along the line of what has already been
observed for single messages Corazza et al. (2019).

## References

  * Almerekhi et al. (2019) Hind Almerekhi, Haewoon Kwak, Bernard J. Jansen, and Joni Salminen. 2019.  Detecting toxicity triggers in online discussions.  In _Proceedings of the 30th ACM Conference on Hypertext and Social Media_ , HT ’19, pages 291–292.
  * Bak et al. (2012) JinYeong Bak, Suin Kim, and Alice Oh. 2012.  Self-disclosure and relationship strength in twitter conversations.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 60–64, Jeju Island, Korea.
  * Basile et al. (2019) Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019\.  SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter.  In _Proceedings of SEMEVAL 2019_.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Bosco et al. (2018) Cristina Bosco, Felice Dell’Orletta, Fabio Poletto, Manuela Sanguinetti, and Maurizio Tesconi. 2018.  Overview of the EVALITA 2018 hate speech detection task.  In _Proceedings of EVALITA 2018_.
  * Carmona et al. (2018) Miguel Ángel Álvarez Carmona, Estefanía Guzmán-Falcón, Manuel Montes-y-Gómez, Hugo Jair Escalante, Luis Villaseñor Pineda, Verónica Reyes-Meza, and Antonio Rico Sulayes. 2018.  Overview of MEX-A3T at IberEval 2018: Authorship and Aggressiveness Analysis in Mexican Spanish Tweets.  In _Proceedings of IberEval 2018_ , pages 74–96.
  * Caselli et al. (2020) Tommaso Caselli, Valerio Basile, Jelena Mitrovic, Inga Kartoziya, and Michael Granitzer. 2020.  I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , Marseille, France.
  * Chang and Lin (2011) Chih-Chung Chang and Chih-Jen Lin. 2011.  Libsvm: A library for support vector machines.  _ACM transactions on intelligent systems and technology (TIST)_ , 2(3):1–27.
  * Chatzakou et al. (2017) Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, and Athena Vakali. 2017.  Mean birds: Detecting aggression and bullying on twitter.  In _Proceedings of the 2017 ACM on Web Science Conference_ , pages 13–22.
  * Corazza et al. (2019) Michele Corazza, Stefano Menini, Elena Cabrio, Sara Tonelli, and Serena Villata. 2019.  Cross-platform evaluation for italian hate speech detection.  In _Proceedings of the Sixth Italian Conference on Computational Linguistics, Bari, Italy, November 13-15, 2019_ , volume 2481 of _CEUR Workshop Proceedings_. CEUR-WS.org.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Mach. Learn._ , 20(3):273–297.
  * Del Tredici et al. (2019) Marco Del Tredici, Diego Marcheggiani, Sabine Schulte im Walde, and Raquel Fernández. 2019.  You shall know a user by the company it keeps: Dynamic representations for social media users in NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4707–4717.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4171–4186, Minneapolis, Minnesota.
  * Fehn Unsvåg and Gambäck (2018) Elise Fehn Unsvåg and Björn Gambäck. 2018.  The effects of user features on twitter hate speech detection.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , Brussels, Belgium.
  * Founta et al. (2018) Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018.  Large scale crowdsourcing and characterization of twitter abusive behavior.  In _11th International Conference on Web and Social Media, ICWSM 2018_.
  * Gao and Huang (2017) Lei Gao and Ruihong Huang. 2017.  Detecting online hate speech using context aware models.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017_ , pages 260–266, Varna, Bulgaria. INCOMA Ltd.
  * Gimpel et al. (2010) Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2010.  Part-of-speech tagging for twitter: Annotation, features, and experiments.  Technical report, Carnegie-Mellon University, School of Computer Science.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018.  Learning word vectors for 157 languages.  In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Graves and Schmidhuber (2005) Alex Graves and Jürgen Schmidhuber. 2005.  Framewise phoneme classification with bidirectional lstm and other neural network architectures.  _Neural networks_ , 18(5-6):602–610.
  * Hessel and Lee (2019) Jack Hessel and Lillian Lee. 2019.  Something’s brewing! early prediction of controversy-causing posts from discussion features.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT_ , pages 1648–1659.
  * Indurthi et al. (2019) Vijayasaradhi Indurthi, Bakhtiyar Syed, Manish Shrivastava, Nikhil Chakravartula, Manish Gupta, and Vasudeva Varma. 2019.  FERMI at SemEval-2019 task 5: Using sentence embeddings to identify hate speech against immigrants and women in twitter.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 70–74, Minneapolis, Minnesota, USA.
  * Jurgens et al. (2019) David Jurgens, Libby Hemphill, and Eshwar Chandrasekharan. 2019.  A just and comprehensive strategy for using NLP to address online abuse.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , Florence, Italy.
  * Karan and Šnajder (2019) Mladen Karan and Jan Šnajder. 2019.  Preemptive toxic language detection in Wikipedia comments using thread-level context.  In _Proceedings of the Third Workshop on Abusive Language Online_ , pages 129–134, Florence, Italy.
  * Klubicka and Fernández (2018) Filip Klubicka and Raquel Fernández. 2018.  Examining a hate speech corpus for hate speech detection and popularity prediction.  In _Workshop on Replicability and Reproducibility of Research Results in Science and Technology of Language_.
  * Kshirsagar et al. (2018) Rohan Kshirsagar, Tyrus Cukuvac, Kathy McKeown, and Susan McGregor. 2018.  Predictive embeddings for hate speech detection on twitter.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , pages 26–32.
  * Lee et al. (2018) Younghun Lee, Seunghyun Yoon, and Kyomin Jung. 2018.  Comparative studies of detecting abusive language on twitter.  _CoRR_ , abs/1808.10245.
  * Liu et al. (2019) Ping Liu, Wen Li, and Liang Zou. 2019.  NULI at SemEval-2019 task 6: Transfer learning for offensive language detection using bidirectional transformers.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , Minneapolis, Minnesota, USA.
  * Liu and Weber (2014) Zhe Liu and Ingmar Weber. 2014.  Predicting ideological friends and foes in twitter conflicts.  In _Proceedings of the 23rd International Conference on World Wide Web_ , WWW ’14 Companion, page 575–576. Ass. for Computing Machinery.
  * Ousidhoum et al. (2020) Nedjma Ousidhoum, Yangqiu Song, and Dit-Yan Yeung. 2020.  Comparative evaluation of label-agnostic selection bias in multilingual hate speech datasets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2532–2542, Online. Association for Computational Linguistics.
  * Pamungkas et al. (2020) Endang Wahyu Pamungkas, Valerio Basile, and Viviana Patti. 2020.  Do you really want to hurt me? predicting abusive swearing in social media.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , Marseille, France.
  * Park and Fung (2017) Ji Ho Park and Pascale Fung. 2017.  One-step and two-step classification for abusive language detection on twitter.  In _Proceedings of the First Workshop on Abusive Language Online_ , pages 41–45. Association for Computational Linguistics.
  * Pavlopoulos et al. (2020) John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020.  Toxicity detection: Does context really matter?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Pei and Jurgens (2020) Jiaxin Pei and David Jurgens. 2020.  Quantifying intimacy in language.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5307–5326, Online. Association for Computational Linguistics.
  * Prabhakaran et al. (2020) Vinodkumar Prabhakaran, Zeerak Waseem, Seyi Akiwowo, and Bertie Vidgen. 2020.  Online abuse and human rights: WOAH satellite session at RightsCon 2020.  In _Proceedings of the Fourth Workshop on Online Abuse and Harms_ , pages 1–6, Online. Association for Computational Linguistics.
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019.  The risk of racial bias in hate speech detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1668–1678, Florence, Italy.
  * Singh et al. (2019) Abhishek Singh, Eduardo Blanco, and Wei Jin. 2019.  Incorporating emoji descriptions improves tweet classification.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Vidgen et al. (2019) Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019.  Challenges and frontiers in abusive content detection.  In _Proceedings of the Third Workshop on Abusive Language Online_ , pages 80–93, Florence, Italy.
  * Waseem (2016) Zeerak Waseem. 2016.  Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter.  In _Proceedings of the First Workshop on NLP and Computational Social Science_ , Austin, Texas.
  * Waseem et al. (2020) Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2020.  Disembodied Machine Learning: On the Illusion of Objectivity in NLP.  Anonymous preprint under review.
  * Wiegand et al. (2019) Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019.  Detection of Abusive Language: the Problem of Biased Datasets.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 602–608.
  * Wiegand et al. (2018) Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018.  Overview of the germeval 2018 shared task on the identification of offensive language.  In _GermEval 2018_.
  * Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019.  SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval).  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 75–86.
  * Zampieri et al. (2020) Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020.  SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020).  In _Proceedings of SemEval_.
  * Zhang et al. (2018a) Justine Zhang, Jonathan Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Dario Taraborelli, and Nithum Thain. 2018a.  Conversations gone awry: Detecting early signs of conversational failure.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , pages 1350–1361.
  * Zhang et al. (2018b) Z. Zhang, D. Robinson, and J. Tepper. 2018b.  Detecting hate speech on twitter using a convolution-gru based deep neural network.  In _ESWC 2018: The semantic web Conference Proceedings_ , pages 745–760. Springer Verlag.