#  Annotators with Attitudes:
How Annotator Beliefs And Identities Bias Toxic Language Detection

Maarten Sap♡♠ Swabha Swayamdipta♠ Laura Vianna♢
Xuhui Zhou♣ Yejin Choi♡♠ Noah A. Smith♡♠
♡Paul G. Allen School of Computer Science, University of Washington, Seattle,
WA, USA
♠Allen Institute for AI, Seattle, WA, USA
♢Department of Psychology, University of Washington, Seattle, WA, USA
♣Georgia Institute of Technology, Atlanta, GA, USA
<EMAIL_ADDRESS>
###### Abstract

Warning: this paper discusses and contains content that is offensive or
upsetting.

The perceived toxicity of language can vary based on someone’s identity and
beliefs, but this variation is often ignored when collecting toxic language
datasets, resulting in dataset and model biases. We seek to understand the
who, why, and what behind biases in toxicity annotations. In two online
studies with demographically and politically diverse participants, we
investigate the effect of annotator identities (who) and beliefs (why),
drawing from social psychology research about hate speech, free speech, racist
beliefs, political leaning, and more. We disentangle what is annotated as
toxic by considering posts with three characteristics: anti-Black language,
African American English (AAE) dialect, and vulgarity. Our results show strong
associations between annotator identity and beliefs and their ratings of
toxicity. Notably, more conservative annotators and those who scored highly on
our scale for racist beliefs were less likely to rate anti-Black language as
toxic, but more likely to rate AAE as toxic. We additionally present a case
study illustrating how a popular toxicity detection system’s ratings
inherently reflect only specific beliefs and perspectives. Our findings call
for contextualizing toxicity labels in social variables, which raises immense
implications for toxic language annotation and detection.

##  1 Introduction

Figure 1:  Annotator identities and attitudes can influence how they rate
toxicity in text. We summarize the key findings from our analyses of biases in
toxicity (offensiveness or racism) ratings for three types of language: anti-
Black content, African American English (AAE), and vulgar language.

Determining whether a text is toxic (i.e., contains hate speech, abuse, or is
offensive) is inherently a subjective task that requires a nuanced
understanding of the pragmatic implications of language Fiske (1993); Croom
(2011); Waseem et al. (2021). Without this nuance, both humans and machines
are prone to biased judgments, such as over-relying on seemingly toxic
keywords (e.g., expletives, swearwords; Dinan et al., 2019; Han and Tsvetkov,
2020) or backfiring against minorities (Yasin, 2018; Are, 2020, i.a.). For
example, racial biases have been uncovered in toxic language detection where
text written in African American English (AAE) is falsely flagged as toxic Sap
et al. (2019); Davidson et al. (2019).

The crux of the issue is that not all text is equally toxic for everyone
Waseem (2016); Al Kuwatly et al. (2020). Yet, most previous research has
treated this detection as a simple classification with one correct label,
obtained by averaging judgments by a small set of human raters per post Waseem
and Hovy (2016); Wulczyn et al. (2017); Davidson et al. (2017); Founta et al.
(2018); Zampieri et al. (2019). Such approaches ignore the variance in
annotations Pavlick and Kwiatkowski (2019); Geva et al. (2019); Arhin et al.
(2021); Akhtar et al. (2021) based on who the annotators are, and what their
beliefs are.

In this work, we investigate the who, why, and what behind biases111We use the
term “bias” to denote both simple skews or variation in annotations (e.g., for
variation in detecting vulgar content as toxic) or representational harms
(e.g., AAE being over-detected as toxic or anti-Black content being under-
detected as toxic; Barocas et al., 2017; Blodgett et al., 2020). in toxicity
annotations, through online studies with demographically and politically
diverse participants. We measure the effects of annotator identities (who
annotates as toxic) and attitudes or beliefs (why they annotate as toxic) on
toxicity perceptions, through the lens of social psychology research on hate
speech, free speech, racist beliefs, altruism, political leaning, and more. We
also analyze the effect of what is being rated, by considering three text
characteristics: anti-Black or racially prejudiced meaning, African American
English (AAE), and vulgar words.

We seek to answer these questions via two online studies. In our breadth-of-
workers controlled study, we collect ratings of toxicity for a set of 15 hand
curated posts from 641 annotators of different races, attitudes, and political
leanings. Then, in our breadth-of-posts study, we simulate a typical toxic
language annotation setting by collecting toxicity ratings for $\sim$600
posts, from a smaller but diverse pool of 173 annotators.222Please contact the
authors for the anonymized study data.

Distilled in Figure 1, our most salient results across both studies show that
annotators scoring higher on our racist beliefs scale were less likely to rate
anti-Black content as toxic (§4). Additionally, annotators’ conservatism
scores were associated with higher ratings of toxicity for AAE (§5), and
conservative and traditionalist attitude scores with rating vulgar language as
more toxic (§6).

We further provide a case study which shows that PerspectiveAPI, a popular
toxicity detection system, mirrors ratings by annotators of certain attitudes
and identities over others (§7). For instance, for anti-Black language, the
system’s scores better reflect ratings by annotators who score high on our
scale for racist beliefs. Our findings have immense implications for the
design of toxic language annotation and automatic detection—we recommend
contextualizing ratings in social variables and looking beyond aggregated
discrete decisions (§8).

##  2 The Who, Why, and What of Toxicity Annotations

We aim to investigate how annotators’ ratings of the toxicity of text is
influenced by their own identities (who they are; §2.1), and their beliefs
(why they consider something toxic; §2.2) on specific categories of text (what
they consider toxic; §2.3)—namely, text with anti-Black language, presence of
African American English (AAE), and presence of vulgar or profane words. To
this end, we design two online studies (§3) and discuss who find each of these
text characteristics offensive and why as separate research questions in
Sections §4, §5, and §6, respectively.

###  2.1 Demographic Identities: Who considers something as toxic?

Prior work has extensively shown links between someone’s gender, political
leaning, and race affects how likely they are to perceive or notice harmful
speech or racism Cowan et al. (2002); Norton and Sommers (2011); Carter and
Murphy (2015); Prabhakaran et al. (2021). Grounded in this prior literature,
our study considers annotators’ race, gender, and political leaning. Since
perceptions of race and political attitudes vary vastly across the globe, we
restrict our study to participants exclusively from the United States.

###  2.2 Attitudes: Why does someone consider something toxic?

While some annotator toxicity ratings may highly correlate with demographic
factors at face value Prabhakaran et al. (2021); Jiang et al. (2021), we aim
to go beyond demographics to investigate annotator beliefs that explain these
correlations. Based on prior work in social psychology, political science, and
sociolinguistics, we select seven attitude dimensions, which we operationalize
via scales (in small caps), as described below.333We abstain from conclusions
beyond our abstractions.

#### Valuing the freedom of offensive speech (FreeOffSpeech):

the belief that any speech, including offensive or hateful speech, should be
unrestricted and free from censorship. Recently, this belief has become
associated with majority and conservative identities Cole (1996); Gillborn
(2009); White and Crandall (2017); Elers and Jayan (2020). We use the scale by
Cowan and Khatchadourian (2003); see Appendix A.1.

#### Perceiving the HarmOfHateSpeech:

the belief that hate speech or offensive language can be harmful for the
targets of that speech Soral et al. (2018); Nadal (2018). This belief is
correlated with socially-progressive philosophies (Downs and Cowan, 2012, see
also Nelson et al., 2013). We use the scale by Cowan and Khatchadourian
(2003); see Appendix A.2.

#### Endorsement of RacistBeliefs:

the beliefs which deny the existence of racial inequality, or capture
resentment towards racial minorities Poteat and Spanierman (2012). We measure
RacistBeliefs using items from the validated Modern Racism Scale McConahay
(1986); see Appendix A.3.

####  Traditionalism:

the belief that one should follow established norms and traditions, and be
respectful of elders, obedient, etc. In the US, these beliefs are associated
with generally conservative ideologies Johnson and Tamney (2001); Knuckey
(2005). We use an abridged version444This was done to reduce cognitive load on
annotators. of the Traditionalism scale Bouchard Jr. and McGue (2003) that
measures annotators’ adherence to traditional values; see Appendix A.4.

#### Language Purism (LingPurism):

the belief that there is a “correct” way of using English Jernudd and Shapiro
(1989). Typically, this belief also involves negative reactions to non-
canonical ways of using language Sapolsky et al. (2010); DeFrank and Kahlbaugh
(2019). We created and validated a four-item LingPurism scale to measure this
concept; see Appendix A.5.

####  Empathy:

one’s tendency to see others’ perspectives and feel others’ feelings. Research
in social psychology has linked higher levels of empathy to the ability and
willingness of recognizing and labeling hate speech Cowan and Khatchadourian
(2003). We measure Empathy using an abbreviated Interpersonal Reactivity Index
Pulos et al. (2004); see Appendix A.6.

####  Altruism:

one’s attitude of selfless concern about others’ well-being, which can move
people to act when harm or injustice happens Wagstaff (1998); Gavrilets and
Fortunato (2014); Riar et al. (2020), including harms through hate speech
(Cowan et al., 2002). We gathered the items to measure Altruism with an
adapted scale taken from Steg et al. (2014); see Appendix A.7.

It is worth noting that some of the above attitudes, though not all, correlate
with demographics very strongly. Table 8 in Appendix A.8 details these
correlations from our study.

###  2.3 Text Characteristics: What is considered offensive?

Not all toxic text is toxic for the same reasons. We aim to understand how
characteristics of text can affect ratings of toxicity, in addition to
annotator attitudes and identities. Specifically, we consider three dimensions
or categories of text, based on recent work on text characteristics that tend
to be over- or under-detected as toxic Dinan et al. (2019); Sap et al. (2019);
Han and Tsvetkov (2020); Zhou et al. (2021): anti-Black language, presence of
African American Engligh (AAE) dialect markers, and vulgar language (e.g.,
swearwords, slurs). We distinguish between two types of vulgarity, following
Zhou et al. (2021): swearwords or explicit words that do not reference
identities (offensive, non-identity referring; OnI), and (reclaimed) slurs or
other identity-referring vulgarity (offensive identity-referring; OI). In our
analyes, we focus on OnI vulgarity unless explicitly noted.

##  3 Data & Study Design

We design two online studies to study the effect of annotator identities and
attitudes on their toxicity ratings on posts with different characteristics.
In either study, annotators are asked to rate how offensive and how racist
they consider a post to be (see Appendix B for the exact questions).555For
both studies, we sought explicit consent from participants, paid participants
above minimum wage, and obtained approval by our institution’s ethics board
(IRB). We specifically focus on readers’ perceptions or opinions, instead of
imposing prescriptive definitions of toxicity or hate speech which previous
work has shown still suffers from large annotator disagreement Ross et al.
(2017). In the sections §4–6, we report only (near-)significant associations;
see Appendix E and F for all results.

Breadth-of-Workers study
---
Vulgar (OnI), not Anti-Black, not AAE
I sort of fuckin love how close my gf is with my family like that she knows
about my family jokes before I fuckin do
I’m hoping that Cersei gets her ass handed to her but I love Daenerys and Jon
;3; I don’t want either of em losing xD
I got mosquito bites on my foot and they fucking hurt
Personally I think it would be fucking hilarious to put Franziska or Godot in
but idk if they’re important enough characters to make the cut
I just keep telling myself that shitty things happen to me but something
really great is gonna happen bc romantic comedies have ruined me
AAE, not Vulgar, not Anti-Black
If u grown & still get thirsty for Jordans knowin erbody else gon havem & u
still feel like u accomplished something that say alot about u
Just spent a whole 20k(imaginary money) on a home studio.. And used the last
3$ 2 get some flowers n a gift card for YA BIHH.
When everything is going accordingly the enemy always try to come creep up on
you but I aint havin it !! buhh byee
You can’t slang no tea  cause he been putting me on bout everything . I could
only wish dawg the best
ain’t no problem in cutting ppl off. save that space for somebody who’ll
benefit from u and vice versa
Anti-Black, not Vulgar, not AAE
I don’t get government benefits just for being white
Where is white history month? Why can’t I have white pride day?
I don’t like black doctors, it’s just my preference
Anti-Black racism isn’t that big of a deal because people treated Italians
like this in the 60’s and 70’s
Going to Africa. Hope I don’t get AIDS. Just kidding. I’m white!
Table 1:  All 15 posts for our breadth-of-workers study.

###  3.1 Breadth-of-Workers Study

Our first study focuses on collecting toxicity ratings from a wide and diverse
set of participants for a controlled set of posts. Shown in Table 1, we hand
curated a set of 15 posts that belong exclusively to one text category (e.g.,
vulgar but non-AAE and non-anti-Black; see Appendix C.1 for more data
selection and validation details). To exclude confounding effects of offensive
identity mentions (OI; e.g., slurs) which could be both vulgar and anti-Black
(or sexist, homophobic, etc.), we only considered posts with vulgar terms that
are non-identity referring (OnI; e.g., swearwords).

We ran our study on a 641 participants that were recruited using a pre-
qualifier survey on Amazon Mechanical Turk (MTurk) to ensure racial and
political diversity. Our final participant pool spanned various racial (13%
Black, 85% White), political (29% conservative, 59% liberal), and gender
identities (54% women, 45% men, 1% non-binary). Each participant gave each of
the 15 posts toxicity ratings, after which they answered a series of questions
about their attitudes and their identity. We used three attention checks to
ensure data quality. For further details, please see Appendix C.

In our subsequent analyses, we compute associations between the toxicity
ratings and identities or attitudes by computing the effect sizes (Pearson $r$
correlation or Cohen’s $d$) between the average toxicity rating of the posts
in each category and annotator identities or attitude scores.

###  3.2 Breadth-of-Posts Study

Breadth-of-Posts study
---
cat. | Anti-Black | AAE | Vulgar (OnI) | Vulgar (OI)
count | 113 | 270 | 196 | 217
Table 2:  Counts for each text category for the 571 posts in our breadth-of-
posts study. OI: identity-referring vulgarity, OnI: non-identity referring
vulgarity; categories are explained in §2.3. Posts could belong to multiple
categories (Figure 5 in Appendix F).

Our second study focuses on collecting ratings for a larger set of posts, but
with fewer annotators per post to simulate a crowdsourced dataset on toxic
language. In contrast to the previous study, we consider anti-Black or AAE
posts that could also be vulgar, and allow this vulgarity to cover both
potentially offensive identity references (OI) as well as non-identity vulgar
words (OnI; see §2.3). We do not consider posts that are anti-Black and AAE,
since the pragmatic toxicity implications of anti-Black meaning expressed in
AAE are very complex (e.g., in-group language with self-deprecation, sarcasm,
reclaimed slurs; Greengross and Miller, 2008; Croom, 2011), and are thus
beyond the scope of this study.

We draw from two existing toxic language detection corpora to select 571 posts
(Table 2). For AAE and possibly vulgar posts, we draw from Founta et al.
(2018), using an automatic AAE detector by Blodgett et al. (2016)666The text-
only AAE detector by Blodgett et al. (2016) strongly correlates ($r$=.60) with
more race-aware AAE detectors Sap et al. (2019). and the vulgarity word list
from Zhou et al. (2021) for detecting OI and OnI terms. For anti-Black and
possibly vulgar posts, we select posts annotated as anti-Black in Vidgen et
al. (2021), using the same method by Zhou et al. (2021) for detecting vulgar
terms. See Appendix D.1 for more data selection details.

As with the previous study, we ran our annotation study on 173 participants
recruited through a pre-qualifier survey. Our annotators varied racially (20%
Black, 76% White), politically (30% conservative, 54% liberal), and in gender
(45% women, 53% men, $<$2% non-binary). Each post was annotated by 6
participants from various racial and political identities.777For each post, we
collected toxicity ratings from two white conservative workers, two from white
liberal workers, and two from Black workers. Additionally, we asked
participants one-item versions of our attitude scales, using the question from
each scale that correlated best with toxicity in our breadth-of-workers study
as explained in Appendix D.3. See Appendix D for more study design details.

In our analyses, we examine toxicity of anti-Black and potentially vulgar
posts (§4.2) and of AAE and potentially vulgar posts (§5.2), but not of vulgar
posts separately, due to confounding effects of the AAE or anti-Black
characteristics that those posts could have. Additionally, unlike the breadth-
of-workers study, here each annotator could rate a varying number of posts.
Thus, we compute associations between toxicity ratings and identities or
attitudes using a linear mixed effects model888Using the statsmodels
implementation:
https://www.statsmodels.org/stable/generated/statsmodels.formula.api.mixedlm.html
with random effects for each participant.

##  4 Who finds anti-Black posts toxic, and why?

Table 3: Associations between annotator variables and ratings of offensiveness
and racism for the anti-Black posts in the breadth-of-workers study. We use
the Holm correction for multiple comparisons for non-hypothesized associations
and only present significant Pearson $r$ or Cohen’s $d$ effect sizes (∗: $p$ <
0.05, ∗∗: $p$ < 0.001; n.s.: not significant).

Anti-Black language denotes racially prejudiced or racist content—subtle
Breitfeller et al. (2019) or overt—which is often a desired target for toxic
language detection research Waseem (2016); Vidgen et al. (2021). Based on
prior work on linking conservative ideologies, endorsement of unrestricted
speech, and racial prejudice with reduced likelihood to accept the term “hate
speech” Duckitt and Fisher (2003); White and Crandall (2017); Roussos and
Dovidio (2018); Elers and Jayan (2020), we hypothesize that conservative
annotators and those who score highly on the RacistBeliefs or FreeOffSpeech
scales will rate anti-Black tweets as less toxic, and vice-versa. Conversely,
based on findings by Cowan and Khatchadourian (2003), we hypothesize that
annotators with high HarmOfHateSpeech scores will rate anti-Black tweets are
more toxic.

###  4.1 Breadth-of-Workers Results

As shown in Table 3, we found several associations between annotator beliefs
and toxicity ratings for anti-Black posts, confirming our hypotheses. The
three most salient associations with lower racism ratings were annotators who
scored higher in RacistBeliefs, FreeOffSpeech, and those who leaned
conservative. We find similar trends for offensiveness ratings.

Conversely, we found that participants who scored higher in HarmOfHateSpeech
were much more likely to rate anti-Black posts as more offensive, and more
racist. Finally, though both white and Black annotators rated these posts very
high in offensiveness (with means $\mu_{\text{Black}}$ = 3.85 and
$\mu_{\text{white}}$ = 3.59 out of 5), our results show that Black
participants were slightly more likely than white participants to rate them as
offensive.

Our exploratory analyses unearthed other significant associations: negative
correlations with LingPurism, Traditionalism, and gender (male), and positive
correlations with high Empathy, Altruism, and gender (female).

###  4.2 Breadth-of-Posts Results

Table 4: Associations for anti-Black (and potentially also vulgar) posts from
the breadth-of-posts study, shown as the $\beta$ coefficients from a mixed
effects model with a random effect for each annotator (†: $p$ < 0.075, ∗: $p$
< 0.05, ∗∗: $p$ < 0.001; Holm-corrected for multiple comparisons; n.s.: not
significant).

Table 4 shows similar results as in the breadth-of-workers analyses, despite
the posts now potentially containing vulgarity. Specifically, we find that
annotators who scored higher in RacistBeliefs rated anti-Black posts as less
offensive, whereas those who scored higher in HarmOfHateSpeech rated them as
more offensive. Ratings of racism showed similar effects, along with a near-
significant association between higher FreeOffSpeech scores and lower ratings
of racism for anti-Black posts.

###  4.3 Perceived Toxicity of Anti-Black Language

Overall, our results from both studies corroborate previous findings that
studied associations between attitudes toward hate speech and gender and
racial identities, specifically that conservatives, white people, and men tend
to value free speech more, and that liberals, women, and non-white people
perceive the harm of hate speech more Cowan and Khatchadourian (2003); Downs
and Cowan (2012). Our results also support the finding that those who hold
generally conservative ideologies tend to be more accepting towards anti-Black
or racially prejudiced content Goldstein and Hall (2017); Lucks (2020);
Schaffner (2020).

In the context of toxicity annotation and detection, our findings highlight
the need to consider the attitudes of annotators towards free speech, racism,
and their beliefs on the harms of hate speech, for an accurate estimation of
anti-Black language as toxic, offensive, or racist (e.g., by actively taking
into consideration annotator ideologies; Waseem, 2016; Vidgen et al., 2021).
This can be especially important given that hateful content very often targets
marginalized groups and racial minorities Silva et al. (2016); Sap et al.
(2020), and can catalyze violence against them O’Keeffe et al. (2011); Cleland
(2014).

##  5 Who finds AAE posts toxic, and why?

Table 5: Associations between ratings of racism and annotator variables, for
the AAE posts from the breadth-of-workers study. As with the previous results,
we correct for multiple comparisons for non-hypothesized associations and only
show significant results (†: $p$ < 0.075, ∗: $p$ < 0.05).

African American English (AAE) is a set of well-studied varieties or dialects
of U.S. English, common among, but not limited to, African-American or Black
speakers Green (2002); Edwards (2004). This category has been shown to be
considered “worse” English by non-AAE speakers (Hilliard, 1997; Blake and
Cutler, 2003; Champion et al., 2012; Beneke and Cheatham, 2015; Rosa and
Flores, 2017), and is often mistaken as obscene or toxic by humans and AI
models Spears et al. (1998); Sap et al. (2019), particularly due to dialect-
specific lexical markers (e.g., words, suffixes).

Based on prior work that correlates racial prejudice with negative attitudes
towards AAE Gaither et al. (2015); Rosa (2019), we hypothesize that annotators
who are white and who score high in RacistBeliefs will rate AAE posts as more
toxic. Additionally, since AAE can be considered non-canonical English
Sapolsky et al. (2010); DeFrank and Kahlbaugh (2019), we hypothesize that
annotators who are more conservative and who score higher in Traditionalism
and LingPurism will rate AAE posts with higher toxicity.

###  5.1 Breadth-of-Workers Results

Table 5 shows significant associations between annotator identities and
beliefs and their ratings of toxicity of AAE posts. Partially confirming our
hypothesis, we found that ratings of racism had somewhat significant
correlations with annotators’ conservative political leaning, and their scores
on our RacistBeliefs scale. However, contrary to our expectations, we found
that white and Black annotators did not differ in how offensive they rated AAE
tweets ($d$ = 0.14, $p$ > 0.1). We found no additional hypothesized or
exploratory associations for racism ratings, and no significant associations
for offensiveness ratings.

###  5.2 Breadth-of-Posts Results

Table 6: Associations between AAE (and potentially also vulgar) post ratings
from the breadth-of-posts study and annotator variables, shown as the $\beta$
coefficients from a mixed effects model with a random effect for each
annotator. We only show significant results (†: $p$ < 0.075, ∗: $p$ < 0.05,
∗∗: $p$ < 0.001; Holm-corrected for multiple comparisons; n.s.: not
significant).

Shown in Table 6, our results for AAE and potentially vulgar breadth-of-posts
study show higher offensiveness ratings from conservative raters, and those
who scored higher in Traditionalism and, almost significantly, RacistBeliefs.
We also find that conservative annotators and those who scored higher in
FreeOffSpeech (and near-significantly, Traditionalism) rated AAE posts as more
racist.

As an additional investigation, we measure whether attitudes or identities
affects toxicity ratings of AAE posts that contain the word “n*gga,” a
(reclaimed) slur that has very different pragmatic interpretations depending
on speaker and listener identity Croom (2011). Here, we find that raters who
are more conservative tended to score those posts as significantly more racist
($\beta=0.465,p=0.003$; corrected for multiple comparisons).

###  5.3 Perceived Toxicity of AAE

Our findings suggest that annotators perceive that AAE posts are associated
with the Black racial identity Rosa (2019), which could cause those who score
highly on the RacistBeliefs scale to annotate them as racist, potentially as a
form of colorblind racism (e.g., where simply mentioning race is considered
racist; Bonilla-Silva, 2006). Moreover, specific markers of AAE could have
been perceived as obscene by non-AAE speakers Spears et al. (1998), even
though some of these might be reclaimed slurs (e.g., “n*gga”; Croom, 2011;
Galinsky et al., 2013). Contrary to expectations, annotators’ own racial
identity did not affect their ratings of AAE posts in our studies. Future work
should investigate this phenomenon further, in light of the variation in
perceptions of AAE within the Black community Rahman (2008); Johnson et al.
(2022), and the increased acceptance and usage of AAE by non-Black people in
social media Ilbury (2020); Ueland (2020).

These findings shed some light on the racial biases found in hate speech
detection Davidson et al. (2019); Sap et al. (2019), partially explaining why
AAE is perceived as toxic. Based on our results, future work in toxic language
detection should account for this over-estimation of AAE as racist. For
example, annotators could explicitly include speakers of AAE, or those who
understand that AAE or its lexical markers are not inherently toxic, or are
primed to do so Sap et al. (2019). Avoiding an incorrect estimation of AAE as
toxic is crucial to avoid upholding racio-linguistic hierarchies and thus
representational harms against AAE speakers Rosa (2019); Blodgett et al.
(2020).

##  6 Who finds vulgar posts toxic, and why?

Vulgarity can correspond to non-identity referring swearwords (e.g., f*ck,
sh*t; denoted as OnI) or identity-referring slurs (e.g., b*tch, n*gga; denoted
as OI). Both types of vulgarity can be mistaken for toxic despite also having
non-hateful usages (e.g., to indicate emotion or social belonging; Croom,
2011; Dynel, 2012; Galinsky et al., 2013).

Table 7: Associations between toxicity ratings and annotator variables for the
vulgar posts from the breadth-of-workers study. We correct for multiple
comparisons for non-hypothesized associations and only show significant
results (∗: $p$ < 0.05, ∗∗: $p$ < 0.001).

Given that vulgarity can be considered non-canonical or impolite language Jay
and Janschewitz (2008); Sapolsky et al. (2010); DeFrank and Kahlbaugh (2019),
we hypothesize that annotators who score high on LingPurism, Traditionalism,
and who are more conservative will rate vulgar posts as more offensive.
Importantly, here, we focus on the posts that are exclusively vulgar (OnI)
from only our breadth-of-workers study, to avoid confounding effects of vulgar
posts with anti-Black meaning or in AAE (both of those cases were analyzed in
§4.2 and §5.2). We refer the reader to Appendix F for the results on vulgar
posts in the breadth-of-posts study.

###  6.1 Breadth-of-Workers Results

Confirming our hypotheses, we found that offensiveness ratings of vulgar (OnI)
posts indeed correlated with annotators’ Traditionalism and LingPurism scores,
and conservative political leaning (Table 7). We found no associations between
attitudes and racism ratings for vulgar posts.

###  6.2 Perceived Toxicity of Vulgar Language

Our findings corroborate prior work showing how adherence to societal
traditional values is often opposed to the acceptability of vulgar language
Sapolsky et al. (2010). Traditional values and conservative beliefs have been
connected to finding vulgar language as a direct challenging the moral order
Jay (2018); Sterling et al. (2020); Muddiman (2021). Our results suggest that
vulgarity is a very specific form of offensiveness that deserves special
attention. Specifically, future work might consider studying the specific
toxicity of individual identity-referring vulgar (OI) words, which can carry
prejudiced meaning as well (e.g., slurs such as “n*gg*r”). Moreover,
annotators across different levels of traditionalism could be considered when
collecting ratings of vulgarity, especially since perceptions might vary with
generational and cultural norms Dynel (2012).

##  7 Toxicity Detection System Case Study: PerspectiveAPI

Our previous findings indicated that there is strong potential for annotator
identities and beliefs to affect their toxicity ratings. We are additionally
interested in how this influences the behavior of toxicity detection models
trained on annotated data. We present a brief case study to answer this
question with the PerspectiveAPI,999www.perspectiveapi.com a widely used,
commercial system for toxicity detection. Appendix G provides a more in-depth
description.

We investigate whether PerspectiveAPI scores align with toxicity ratings from
workers with specific identities or attitudes, using the 571 posts from our
breadth-of-posts study. Specifically, we compare the correlations between
PerspectiveAPI scores and ratings from annotators, broken down by annotators
with different identities (e.g., men and women) or with higher or lower scores
on attitude scales (split at the mean). See Appendix G.1 for details about
this methodology.

Our investigation shows that Perspective scores can be significantly more
aligned with ratings from certain identities or groups scoring higher or lower
on attitude dimensions (see Table 12 in Appendix G.2). Our most salient
results show that for anti-Black posts, Perspective scores are somewhat
significantly more aligned with racism ratings by annotators who score high in
RacistBeliefs ($r_{\text{high}}$ = 0.29, $r_{\text{low}}$ = 0.17, $\Delta r$ =
0.12, $p$ = 0.056; Figure 2). Additionally, for AAE posts, Perspective scores
are slightly more correlated with racism ratings by annotators who were women
($\Delta r$ = 0.22, $p$ < 0.001) or white ($\Delta r$ = 0.08, $p$ = 0.07), and
who scored higher in LingPurism ($\Delta r$ = 0.14, $p$ = 0.003) or
Traditionalism ($\Delta r$ = 0.10, $p$ = 0.030).

Figure 2:  Correlation between PerspectiveAPI toxicity scores and racism
ratings for anti-Black posts, broken down by by participants scoring high and
low in RacistBeliefs.

Overall, our findings indicate that PerspectiveAPI toxicity score predictions
align with specific viewpoints or ideologies, depending on the text category.
Particularly, it seems that the API underestimates the toxicity of anti-Black
posts in a similar way to annotators who scored higher on the RacistBeliefs
scale, and aligns more with white annotator’s perception of AAE toxicity (vs.
Black annotators). This corroborate prior findings that show that toxicity
detection models inherently encode a specific positionality Cambo (2021) and
replicate human biases Davani et al. (2021).

##  8 Discussion & Conclusion

Overall, our analyses showed that perceptions of toxicity are indeed affected
by annotators’ demographic identities and beliefs (§2). We found—via a
breadth-of-workers study and a breadth-of-posts study (§3)—several
associations when isolating specific text characteristics: anti-Black (§4),
AAE (§5), and vulgarity (§6). Finally, we showed that a popular toxicity
detection system yields toxicity scores that are more aligned with raters with
certain attitudes and identities than others (§7). We discuss implications of
our findings below.

#### Variation in toxicity ratings in hate speech datasets.

In our study we deliberately sought rating of perceptions of toxicity of posts
by racially and politically diverse participants. However, many existing hate
speech datasets instructed annotators to adhere to detailed definitions of
toxicity Davidson et al. (2017); Founta et al. (2018), and some even selected
crowdworkers for their liberal ideology Waseem (2016); Sap et al. (2020);
Vidgen et al. (2021). While those annotation setups and annotator homogeneity
could cause less variation in toxicity annotations of anti-Black, AAE, and
vulgar posts, there is still empirical evidence of anti-AAE racial biases in
some of these datasets Sap et al. (2019); Davidson et al. (2019).

Given the large variation in perceptions of toxicity that we showed and the
implicit encoding of perspectives by toxicity models, we recommend researchers
and dataset creators investigate and report annotator attitudes and
demographics; researchers could collect attitude scores based on relevant
social science research, perhaps in lightweight format as done in our breadth-
of-posts study, and report those scores along with the dataset (e.g., in
datasheets; Gebru et al., 2018).

#### Contextualize toxicity predictions in social variables.

As shown in our results and previous studies (e.g., Waseem, 2016; Ross et al.,
2017; Waseem et al., 2021), determining what is toxic is subjective. However,
given this subjectivity, the open question remains: whose perspective should
be considered when using toxicity detection models? To try answering this
question, we urge researchers and practitioners to consider all stakeholders
and end users on which toxicity detection systems might be deployed (e.g.,
through human-centered design methods; Sanders, 2002; Friedman et al., 2008;
Hovy and Yang, 2021). While currently, the decision of content moderation
often solely lies in the hands of the platforms, we encourage the exploration
of alternative solutions (e.g., community fact checkers, digital juries;
Maronikolakis et al., 2022; Gordon et al., 2022).

In general, we urge people to embrace that each design decision has socio-
political implications Green (2020); Cambo (2021), and encourage them to
develop technologies to shift power to the targets of oppression Blodgett et
al. (2020); Kalluri (2020); Birhane (2021). Finally, given the increasingly
essential role of online platforms in people’s daily lives Rahman (2017), we
echo calls for policy regulating online spaces and toxicity detection
algorithms Jiang (2020); Benesch (2020); McGuffie and Newhouse (2020);
Gillespie et al. (2020).

#### Beyond toxicity classification: modeling distributions and generating
explanations.

Our findings on the subjectivity of the toxicity detection tasks suggests that
standard approaches of obtaining binary (or even $n$-ary) labels of toxicity
and averaging them into a majority vote are inadequate. Instead, researchers
could consider modeling the distribution or variation in toxicity labels with
respect to individual annotators Geva et al. (2019); Fornaciari et al. (2021);
Davani et al. (2021) or to specific identities or beliefs.

But, perhaps more importantly, we encourage re-thinking the toxicity detection
paradigm altogether. With the goal to assist human content
moderators,101010Note that while content moderation can induce significant
psyhcological harms in moderators Roberts (2017); Steiger et al. (2021), full
automation also has significant risks. creating systems that explain biased
implications of posts could be more helpful than opaque toxicity scores Thus,
we advocate for moving away from classification frameworks, and towards more
nuanced, holistic, and explainable frameworks for inferring the desired
concepts of toxicity and social biases (e.g., Social Bias Frames; Sap et al.,
2020).

#### Limitations and open questions.

Our work had several limitations and raised several open research questions,
some of which we outline below. First, our particular choices of attitudes and
scales could affect our results; other scales (e.g., Gerdes et al., 2011, for
measuring empathy) as well as other psychological variables (e.g., propensity
to volunteer or to value dignity) could be studied in the context of toxicity
perceptions. Additionally, the automatic AAE detector in the breadth-of-posts
study could have induced data selection biases, despite being strongly
correlated with race-aware dialect detection (as noted in footnote 6).
Furthermore, our analysis of the attitudes encoded in the PerspectiveAPI in §7
was merely a pilot study; we hope future work will explore more in-depth
methods for assess model positionality.

While our study focused on racial discrimination by studying AAE and anti-
Black posts, future work should explore other axes of discrimination (e.g.,
sexism, homophobia, ableism, etc.). Additionally, our study focused only on
U.S.-centric perspectives; we hope researchers will explore variations in
toxicity perceptions in other cultural contexts (e.g., variations based on
caste in India).

## References

  * Akhtar et al. (2021) Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021.  Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection.  ArXiv preprint arXiv:2106.15896.
  * Al Kuwatly et al. (2020) Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020.  Identifying and measuring annotator bias based on annotators’ demographic characteristics.  In _Proceedings of the Fourth Workshop on Online Abuse and Harms_.
  * Are (2020) Carolina Are. 2020.  How instagram’s algorithm is censoring women and vulnerable users but helping online abusers.  _Feminist media studies_ , 20(5).
  * Arhin et al. (2021) Kofi Arhin, Ioana Baldini, Dennis Wei, Karthikeyan Natesan Ramamurthy, and Moninder Singh. 2021.  Ground-Truth, whose truth? – examining the challenges with annotating toxic text datasets.
  * Barocas et al. (2017) Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017.  The problem with bias: Allocative versus representational harms in machine learning.  In _SIGCIS_.
  * Beneke and Cheatham (2015) Margaret Beneke and Gregory A Cheatham. 2015.  Speaking up for african american english: Equity and inclusion in early childhood settings.  _Early Childhood Education Journal_ , 43(2).
  * Benesch (2020) Susan Benesch. 2020.  Proposals for improved regulation of harmful online content.  Technical report.
  * Birhane (2021) Abeba Birhane. 2021.  Algorithmic injustice: a relational ethics approach.  _Patterns (New York, N.Y.)_ , 2(2).
  * Blake and Cutler (2003) Renée Blake and Cecilia Cutler. 2003.  AAE and variation in teachers’ attitudes: A question of school philosophy?  _Linguistics and education_ , 14(2).
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016.  Demographic dialectal variation in social media: A case study of African-American English.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_.
  * Bonilla-Silva (2006) Eduardo Bonilla-Silva. 2006.  _Racism without racists: Color-blind racism and the persistence of racial inequality in the United States_.
  * Bouchard Jr. and McGue (2003) Thomas J. Bouchard Jr. and Matt McGue. 2003.  Genetic and environmental influences on human psychological differences.  _Journal of Neurobiology_ , 54(1).
  * Breitfeller et al. (2019) Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019.  Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Burnham et al. (2018) Martin J Burnham, Yen K Le, and Ralph L Piedmont. 2018.  Who is mturk? personal characteristics and sample consistency of these online workers.  _Mental Health, Religion & Culture_, 21(9-10).
  * Cambo (2021) Scott Allen Cambo. 2021.  _Model Positionality: A Novel Framework for Data Science with Subjective Target Concepts_.  Ph.D. thesis, Northwestern University.
  * Carter and Murphy (2015) Evelyn R Carter and Mary C Murphy. 2015.  Group-based differences in perceptions of racism: What counts, to whom, and why?  _Social and personality psychology compass_ , 9(6).
  * Champion et al. (2012) Tempii B Champion, Deirdre Cobb-Roberts, and Linda Bland-Stewart. 2012.  Future educators’ perceptions of african american vernacular english (AAVE).  _Online Journal of Education Research_ , 1(5).
  * Cleland (2014) Jamie Cleland. 2014.  Racism, football fans, and online message boards: How social media has added a new dimension to racist discourse in English football.  _J. Sport Soc. Issues_ , 38(5).
  * Cole (1996) D Cole. 1996.  Racist speech should be protected by the constitution.  _Hate crimes_.
  * Cowan and Khatchadourian (2003) Gloria Cowan and Désirée Khatchadourian. 2003.  Empathy, ways of knowing, and interdependence as mediators of gender differences in attitudes toward hate speech and freedom of speech.  _Psychology of Women Quarterly_ , 27(4).
  * Cowan et al. (2002) Gloria Cowan, Miriam Resendez, Elizabeth Marshall, and Ryan Quist. 2002.  Hate speech and constitutional protection: Priming values of equality and freedom.  _The Journal of social issues_ , 58(2).
  * Croom (2011) Adam M Croom. 2011.  Slurs.  _Language Sciences_ , 33(3).
  * Davani et al. (2021) Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2021.  Hate speech classifiers learn human-like social stereotypes.
  * Davidson et al. (2019) Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019.  Racial bias in hate speech and abusive language detection datasets.  In _Proceedings of the Third Workshop on Abusive Language Online_.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _ICWSM_.
  * DeFrank and Kahlbaugh (2019) Melanie DeFrank and Patricia Kahlbaugh. 2019.  Language choice matters: When profanity affects how people are judged.  _Journal of Language and Social Psychology_ , 38(1).
  * Dinan et al. (2019) Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019.  Build it break it fix it for dialogue safety: Robustness from adversarial human attack.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Downs and Cowan (2012) Daniel M Downs and Gloria Cowan. 2012.  Predicting the importance of freedom of speech and the perceived harm of hate speech.  _Journal of applied social psychology_ , 42(6).
  * Duckitt and Fisher (2003) John Duckitt and Kirstin Fisher. 2003.  The impact of social threat on worldview and ideological attitudes.  _Political Psychology_ , 24(1).
  * Dynel (2012) Marta Dynel. 2012.  Swearing methodologically : the (im)politeness of expletives in anonymous commentaries on youtube.  _Journal of English Studies_ , 10(0).
  * Edwards (2004) Walter F Edwards. 2004.  African american vernacular english: phonology.  In _A Handbook of Varieties of English: Morphology and syntax_.
  * Elers and Jayan (2020) Christine Helen Elers and Pooja Jayan. 2020.  “this is us”: Free speech embedded in whiteness, racism and coloniality in aotearoa, new zealand.  _First Amendment Studies_ , 54(2).
  * Fiske (1993) S T Fiske. 1993.  Controlling other people. the impact of power on stereotyping.  _The American psychologist_ , 48(6).
  * Fornaciari et al. (2021) Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021.  Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Founta et al. (2018) Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018.  Large scale crowdsourcing and characterization of twitter abusive behavior.  In _ICWSM_.
  * Friedman et al. (2008) Batya Friedman, Peter H Kahn, and Alan Borning. 2008.  Value sensitive design and information systems.  _The handbook of information and computer ethics_.
  * Gaither et al. (2015) Sarah E Gaither, Ariel M Cohen-Goldberg, Calvin L Gidney, and Keith B Maddox. 2015\.  Sounding black or white: Priming identity and biracial speech.  _Frontiers in Psychology_ , 6.
  * Galinsky et al. (2013) Adam D Galinsky, Cynthia S Wang, Jennifer A Whitson, Eric M Anicich, Kurt Hugenberg, and Galen V Bodenhausen. 2013.  The reappropriation of stigmatizing labels: the reciprocal relationship between power and self-labeling.  _Psychol. Sci._ , 24(10).
  * Gavrilets and Fortunato (2014) Sergey Gavrilets and Laura Fortunato. 2014.  A solution to the collective action problem in between-group conflict with within-group inequality.  _Nature communications_ , 5(1).
  * Gebru et al. (2018) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018.  Datasheets for datasets.  In _FAccT*_.
  * Gerdes et al. (2011) Karen E Gerdes, Cynthia A Lietz, and Elizabeth A Segal. 2011.  Measuring empathy in the 21st century: Development of an empathy index rooted in social cognitive neuroscience and social justice.  _Social Work Research_ , 35(2).
  * Geva et al. (2019) Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019.  Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Gillborn (2009) David Gillborn. 2009.  Risk-free racism: Whiteness and so-called free speech.  _Wake Forest law review_ , 44.
  * Gillespie et al. (2020) Tarleton Gillespie, Patricia Aufderheide, Elinor Carmi, Ysabel Gerrard, Robert Gorwa, Ariadna Matamoros-Fernandez, Sarah T Roberts, Aram Sinnreich, and Sarah Myers West. 2020.  Expanding the debate about content moderation: Scholarly research agendas for the coming policy debates.  _Internet Policy Review_ , 9(4).
  * Goldstein and Hall (2017) Donna M Goldstein and Kira Hall. 2017.  Postelection surrealism and nostalgic racism in the hands of donald trump.  _HAU: Journal of Ethnographic Theory_ , 7(1).
  * Gordon et al. (2022) Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeffrey T Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022.  Jury learning: Integrating dissenting voices into machine learning models.  In _CHI_.
  * Green (2020) Ben Green. 2020.  Data science as political action: Grounding data science in a politics of justice.  _SSRN Electronic Journal_.
  * Green (2002) Lisa Green. 2002.  _African American English: A Linguistic Introduction_ , 8.3.2002 edition edition.
  * Greengross and Miller (2008) Gil Greengross and Geoffrey F Miller. 2008.  Dissing oneself versus dissing rivals: effects of status, personality, and sex on the Short-Term and Long-Term attractiveness of Self-Deprecating and Other-Deprecating humor.  _Evolutionary Psychology_ , 6(3).
  * Han and Tsvetkov (2020) Xiaochuang Han and Yulia Tsvetkov. 2020.  Fortifying toxic speech detectors against veiled toxicity.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Hilliard (1997) A G Hilliard. 1997.  Language, culture and the assessment of african american children.  _Assessment for equity and inclusion: Embracing all our children_.
  * Hovy and Yang (2021) Dirk Hovy and Diyi Yang. 2021.  The importance of modeling social factors of language: Theory and practice.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Huff and Tingley (2015) Connor Huff and Dustin Tingley. 2015.  “who are these people?” evaluating the demographic characteristics and political preferences of mturk survey respondents.  _Research & Politics_, 2(3).
  * Ilbury (2020) Christian Ilbury. 2020.  “sassy queens”: Stylistic orthographic variation in twitter and the enregisterment of AAVE.  _Journal of sociolinguistics_ , 24(2).
  * Jay (2018) Timothy Jay. 2018.  Swearing, moral order, and online communication.  _Journal of Language Aggression and Conflict_ , 6(1).
  * Jay and Janschewitz (2008) Timothy Jay and Kristin Janschewitz. 2008.  The pragmatics of swearing.  _Journal of Politeness Research_ , 4.
  * Jernudd and Shapiro (1989) Björn H Jernudd and Michael J Shapiro. 1989.  _The politics of language purism_.
  * Jiang (2020) J A Jiang. 2020.  Identifying and addressing design and policy challenges in online content moderation.  _Extended Abstracts of the 2020 CHI Conference on_.
  * Jiang et al. (2021) Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R Brubaker. 2021\.  Understanding international perceptions of the severity of harmful content online.  _PloS one_ , 16(8).
  * Johnson et al. (2022) Darin G Johnson, Bradley D Mattan, Nelson Flores, Nina Lauharatanahirun, and Emily B Falk. 2022.  Social-Cognitive and affective antecedents of code switching and the consequences of linguistic racism for black people and people of color.  _Affective Science_ , 3(1).
  * Johnson and Tamney (2001) Stephen D Johnson and Joseph B Tamney. 2001.  Social traditionalism and economic conservatism: Two conservative political ideologies in the united states.  _The Journal of social psychology_ , 141(2).
  * Kalluri (2020) Pratyusha Kalluri. 2020.  Don’t ask if artificial intelligence is good or fair, ask how it shifts power.  _Nature_ , 583(7815).
  * Knuckey (2005) Jonathan Knuckey. 2005.  A new front in the culture war? moral traditionalism and voting behavior in us house elections.  _American Politics Research_ , 33(5).
  * Loepp and Kelly (2020) Eric Loepp and Jarrod T Kelly. 2020.  Distinction without a difference? an assessment of mturk worker types.  _Research & Politics_, 7(1).
  * Lucks (2020) Daniel S Lucks. 2020.  _Reconsidering Reagan: Racism, Republicans, and the Road to Trump_.
  * Maronikolakis et al. (2022) Antonis Maronikolakis, Axel Wisiorek, Leah Nann, Haris Jabbar, Sahana Udupa, and Hinrich Schütze. 2022.  Listening to affected communities to define extreme speech: Dataset and experiments.  In _ACL 2022 Findings_.
  * McConahay (1986) John B McConahay. 1986.  Modern racism, ambivalence, and the modern racism scale.  In _Prejudice, discrimination, and racism_ , volume 337.
  * McGuffie and Newhouse (2020) Kris McGuffie and Alex Newhouse. 2020.  The radicalization risks of gpt-3 and advanced neural language models.
  * Muddiman (2021) Ashley Muddiman. 2021.  Conservatives and incivility.  In _Conservative Political Communication_.
  * Nadal (2018) Kevin L Nadal. 2018.  _Microaggressions and traumatic stress: Theory, research, and clinical treatment_.
  * Nelson et al. (2013) Jessica C Nelson, Glenn Adams, and Phia S Salter. 2013.  The marley hypothesis: Denial of racism reflects ignorance of history.  _Psychological science_ , 24(2).
  * Norton and Sommers (2011) Michael I Norton and Samuel R Sommers. 2011.  Whites see racism as a Zero-Sum game that they are now losing.  _Perspectives on psychological science: a journal of the Association for Psychological Science_ , 6(3).
  * O’Keeffe et al. (2011) Gwenn Schurgin O’Keeffe, Kathleen Clarke-Pearson, and Council on Communications and Media. 2011.  The impact of social media on children, adolescents, and families.  _Pediatrics_ , 127(4).
  * Pavlick and Kwiatkowski (2019) Ellie Pavlick and Tom Kwiatkowski. 2019.  Inherent disagreements in human textual inferences.  _Transactions of the Association for Computational Linguistics_ , 7\.
  * Poteat and Spanierman (2012) V Paul Poteat and Lisa B Spanierman. 2012.  Modern racism attitudes among white students: The role of dominance and authoritarianism and the mediating effects of racial color-blindness.  _The Journal of Social Psychology_ , 152(6).
  * Prabhakaran et al. (2021) Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Díaz. 2021.  On releasing Annotator-Level labels and information in datasets.  In _Proc. of LAW-DMR workshop at EMNLP_.
  * Pulos et al. (2004) Steven Pulos, Jeff Elison, and Randy Lennon. 2004.  The hierarchical structure of the interpersonal reactivity index.  _Social behavior and personality_ , 32(4).
  * Rahman (2008) Jacquelyn Rahman. 2008.  Middle-class african americans: Reactions and attitudes toward african american english.  _American speech_ , 83(2).
  * Rahman (2017) K Sabeel Rahman. 2017.  The new utilities: Private power, social infrastructure, and the revival of the public utility concept.  _Cardozo law review_ , 39.
  * Riar et al. (2020) Marc Riar, Benedikt Morschheuser, Juho Hamari, and Rüdiger Zarnekow. 2020.  How game features give rise to altruism and collective action? implications for cultivating cooperation by gamification.  In _Proceedings of the 53rd Hawaii International Conference on System Sciences_.
  * Roberts (2017) Sarah T Roberts. 2017.  Social media’s silent filter.  _The Atlantic_.
  * Rosa (2019) Jonathan Rosa. 2019.  _Looking Like a Language, Sounding Like a Race_.
  * Rosa and Flores (2017) Jonathan Rosa and Nelson Flores. 2017.  Unsettling race and language: Toward a raciolinguistic perspective.  _Language In Society_ , 46(5).
  * Ross et al. (2017) Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017.  Measuring the reliability of hate speech annotations: the case of the european refugee crisis.  In _NLP 4 CMC Workshop_.
  * Roussos and Dovidio (2018) Gina Roussos and John F Dovidio. 2018.  Hate speech is in the eye of the beholder: The influence of racial attitudes and freedom of speech beliefs on perceptions of racially motivated threats of violence.  _Social psychological and personality science_ , 9(2).
  * Sanders (2002) Elizabeth Sanders. 2002.  _From user-centered to participatory design approaches_.
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019.  The risk of racial bias in hate speech detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Sap et al. (2020) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020.  Social bias frames: Reasoning about social and power implications of language.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Sapolsky et al. (2010) Barry S. Sapolsky, Daniel M. Shafer, and Barbara K. Kaye. 2010.  Rating offensive words in three television program contexts.  _Mass Communication and Society_ , 14(1).
  * Schaffner (2020) Brian F Schaffner. 2020.  The heightened importance of racism and sexism in the 2018 US midterm elections.  _British journal of political science_.
  * Silva et al. (2016) Leandro Silva, Mainack Mondal, Denzil Correa, Fabrício Benevenuto, and Ingmar Weber. 2016.  Analyzing the targets of hate in online social media.  In _Tenth international AAAI conference on web and social media_.
  * Silver and Dunlap (1987) N Clayton Silver and William P Dunlap. 1987.  Averaging correlation coefficients: should fisher’s z transformation be used?  _Journal of applied psychology_ , 72(1).
  * Soral et al. (2018) Wiktor Soral, Michał Bilewicz, and Mikołaj Winiewski. 2018.  Exposure to hate speech increases prejudice through desensitization.  _Aggressive behavior_ , 44(2).
  * Spears et al. (1998) Arthur K Spears et al. 1998.  African-american language use: Ideology and so-called obscenity.  _African-American English: Structure, history, and use_.
  * Steg et al. (2014) Linda Steg, Goda Perlaviciute, Ellen Van der Werff, and Judith Lurvink. 2014.  The significance of hedonic values for environmentally relevant attitudes, preferences, and actions.  _Environment and behavior_ , 46(2).
  * Steiger et al. (2021) Miriah Steiger, Timir J Bharucha, Sukrit Venkatagiri, Martin J Riedl, and Matthew Lease. 2021.  The psychological Well-Being of content moderators: The emotional labor of commercial moderation and avenues for improving support.  In _CHI_ , number Article 341 in CHI ’21.
  * Sterling et al. (2020) Joanna Sterling, John T Jost, and Richard Bonneau. 2020.  Political psycholinguistics: A comprehensive analysis of the language habits of liberal and conservative social media users.  _Journal of personality and social psychology_ , 118(4).
  * Ueland (2020) Ane Ueland. 2020.  _Language and identity: a study of African American Vernacular English and its status in American society_.  Ph.D. thesis, University of Stavanger, Norway.
  * Vidgen et al. (2021) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
  * Wagstaff (1998) Graham F Wagstaff. 1998.  Equity, justice, and altruism.  _Current Psychology_ , 17(2).
  * Waseem (2016) Zeerak Waseem. 2016.  Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter.  In _Proceedings of the First Workshop on NLP and Computational Social Science_.
  * Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016.  Hateful symbols or hateful people? predictive features for hate speech detection on Twitter.  In _Proceedings of the NAACL Student Research Workshop_.
  * Waseem et al. (2021) Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021.  Disembodied machine learning: On the illusion of objectivity in nlp.  Anonymous preprint under review.
  * White and Crandall (2017) Mark H White and Christian S Crandall. 2017.  Freedom of racist speech: Ego and expressive threats.  _Journal of personality and social psychology_ , 113(3).
  * Wulczyn et al. (2017) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017.  Ex machina: Personal attacks seen at scale.  In _Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017_.
  * Yasin (2018) Danyaal Yasin. 2018.  Black and banned: Who is free speech for? .  Accessed: 2018-12-6.
  * Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019.  Predicting the type and target of offensive posts in social media.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_.
  * Zevallos (2017) Zuleyka Zevallos. 2017.  Sociology of race.  .  Accessed: 2021-10-10.
  * Zhou et al. (2021) Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021.  Challenges in automated debiasing for toxic language detection.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_.

##  Appendix A Attitude Scales

Below, we list the individual items from each of our seven attitude
dimensions. In our breadth-of-workers study, we asked participants all of the
items in each scale, but in the larger-scale study, we only asked the bolded
items (one per scale). (R) denotes a reverse coded item.

###  A.1 Valuing the freedom of offensive speech (FreeOffSpeech).

Participants were instructed to rate their level of agreement with the item
statements, using a 5-point Likert scale ranging from “not at all” (1) to
“very much so” (5).

  * •

People should face consequences for saying something offensive online. (R)

  * •

Offensive posts and comments online should be removed from the platforms.

  * •

There is no such a thing as hate speech, only “speech”.

  * •

People should be allowed to say things that others may consider offensive.

  * •

It’s more important to protect free speech rights than to protect people’s
feelings when it comes to offensive language.

This scale was taken from Cowan et al. (2002).

###  A.2 Perceiving the HarmOfHateSpeech.

Participants were instructed to rate their level of agreement with the item
statements, using a 5-point Likert scale ranging from “not at all” (1) to
“very much so” (5).

  * •

When someone posts something offensive towards minorities online it
intimidates and casts fear in the heart of individuals who are part of the
minority group.

  * •

Offensive language encourages discrimination against minorities.

  * •

Offensive language is harmful to minorities.

This scale was taken from Cowan et al. (2002).

###  A.3 Endorsement of RacistBeliefs.

Participants were instructed to rate their level of agreement with the item
statements, using a 5-point Likert scale ranging from “not at all” (1) to
“very much so” (5).

  * •

Discrimination against racial minorities is no longer a problem in the United
States.

  * •

It is easy to understand the anger of racial minorities people in America. (R)

  * •

Racial minorities are getting too demanding in their push for equal rights.

  * •

Over the past few years, racial minorities have gotten more economically than
they deserve.

  * •

Over the past few years, the government and news media have shown more respect
to racial minorities than they deserve.

These items form the validated Modern Racism Scale, created by McConahay
(1986).

###  A.4 Traditionalism.

Participants were asked: “Please tell us how important each of these is as a
guiding principle in your life.” They answered each item on a 5-point Likert
scale, ranging from “not at all important to me” (1) to “extremely very
important to me” (5).

  * •

Being obedient, dutiful, meeting obligations.

  * •

Self-discipline, self-restraint, resistance to temptations.

  * •

Honoring parents and elders, showing respect.

  * •

Traditions and customs.

This is an abridged version of the traditionalism scale by Bouchard Jr. and
McGue (2003).

###  A.5 Language Purism (LingPurism).

Participants were instructed to rate their level of agreement with the item
statements, using a 5-point Likert scale ranging from “not at all” (1) to
“very much so” (5).

  * •

I dislike when people make simple grammar or spelling errors.

  * •

It is important to master the English language properly, and not make basic
spelling mistakes or misuse a common word.

  * •

I am not afraid to correct people when they make simple grammar or spelling
errors.

  * •

There exists such a thing as good proper English.

This scale was created by the authors.

Figure 3: Distributions of the attitude scores of the workers in the breadth-
of-workers study. Table 8: Pearson $r$ correlations between the attitude and
demographic variables from participants in our breadth-of-workers study. We
only show significant correlations (*: $p$ < 0.05, **: $p$ < 0.001), and
denote non-significant correlations with “n.s.”. Our demographic variables are
not correlated with each other.

###  A.6 Empathy.

Participants were instructed to rate their level of agreement with the item
statements, using a 5-point Likert scale ranging from “not at all” (1) to
“very much so” (5).

  * •

Before criticizing somebody, I try to imagine how I would feel if I were in
his/her place.

  * •

I don’t usually become sad when I see other people crying. (R)

  * •

When someone is feeling ‘down’ I can usually understand how they feel.

  * •

I have tender, concerned feelings for people or groups of people less
fortunate than me.

This scale is an abbreviated version of the widely used Interpersonal
Reactivity Index by Pulos et al. (2004).

###  A.7 Altruism.

Participants were asked: “Please tell us how important each of these is as a
guiding principle in your life.” They answered each item on a 5-point Likert
scale, ranging from “not at all important to me” (1) to “extremely very
important to me” (5).

  * •

Social justice, correcting injustice, caring for the weak.

  * •

Equality, equal opportunity for all.

These items are taken from the altruism part of the scale by Steg et al.
(2014).

###  A.8 Attitude distributions & inter-variable correlations

We plot the distributions of our breadth-of-workers participants on the seven
attitude scales in Figure 3. While most attitudes have wider distributions,
RacistBeliefs notably stands out as having a skewed distributions towards
people scoring low on the scale.

While some attitudes may correlate highly with demographic factors at face
value (e.g., Traditionalism and politically conservatism); other forms of
biases may not be easily explained by demographics alone. We examine the
relationship between our attitude measurements and annotator demographic
identity variables. Shown in Table 8, we find strong significant correlations
between several of our annotator variables.

Notably, we find that an annotator’s political orientation correlated strongly
with several variables, with liberal leaning identities being associated with
higher scores on the Empathy, HarmOfHateSpeech, and Altruism scales, whereas
conservative political leaning was associated with higher scores on the
Traditionalism, LingPurism, FreeOffSpeech and RacistBeliefs scales.

##  Appendix B Toxicity Questions

Following crowdsourcing setups in prior work Waseem (2016); Davidson et al.
(2017); Wulczyn et al. (2017); Founta et al. (2018); Sap et al. (2019), we
asked three fine-grained questions to annotators for each post in both our
studies:

  * •

“How toxic/hateful/disrespectful or offensive does this post seem to you?”

  * •

“How much could this post be seen as toxic/hateful/disrespectful or offensive
to anyone?”

  * •

“In your opinion, how racist is this post?”

Given the high correlations between the two offensiveness variables (Pearson
$r\geq.7$; Table 9), we use create an offensiveness (“off.”) score by taking
the average rating given to the “to you” and “to anyone” questions. In all our
analyses, we use that offensiveness score, along with the raw racism score.

Table 9: Correlations between different offensiveness questions for each tweet
(all $p<0.001$). Since offensiveness “to you” and “to any” are very strongly
correlated, we average them into a single offensiveness score (off.).

##  Appendix C Small-Scale Controlled Study Details

###  C.1 Data Selection & Validation

We aimed to select online posts that were very indicative of each of the above
characteristics (vulgar, AAE, anti-Black) but not indicative of the others, in
order to tease out the effect of that category. We selected vulgar and AAE
posts from a publicly available large corpus of tweets annotated for hate
speech by Founta et al. (2018).111111We only used the training subset of the
corpus. For each tweet in that corpus, we detected the presence of non-
identity related profanity or swearwords using the list from Zhou et al.
(2021), and extracted the likelihood that the tweet is in AAE using a lexical
detector by Blodgett et al. (2016). As candidates, we selected 10 vulgar
tweets that have low likelihood of being AAE, and 26 tweets that have high
likelihood of being AAE but contain no vulgarity. For anti-Black posts, we
selected 11 candidate online posts curated by Zevallos (2017).

We ran a human validation study to verify that the candidate posts are truly
indicative of their respective categories. We created an annotation scheme to
collect binary ratings for two questions per post: “does it contain vulgar
language”, and “is it offensive to minorities”; a post could belong to either
category or neither. Each post was manually annotated by three undergraduate
research assistants trained for the task. Post validation, we manually
selected 5 posts per category with perfect inter-annotator agreement. Table 1
lists the final 15 posts used for our study.

###  C.2 Participant Recruitment

We ran our study on Amazon Mechanical Turk (MTurk), a crowdsourcing platform
that is often used to collect offensiveness annotations.121212Note, this study
was approved by the author’s institutional review board (IRB). With the task
at hand, we sought a racially and politically diverse pool of participants,
which can be challenging given that MTurk workers are usually tend to be
predominantly white and skew liberal Huff and Tingley (2015); Burnham et al.
(2018); Loepp and Kelly (2020). Therefore, we ran a pre-selection survey to
collect race and political ideology of workers, noting that this pre-survey
could grant them access to a longer survey on free speech, hate speech, and
offensiveness in language.131313To better recruit for our pre-survey, we noted
in the title that “BIPOC people and conservatives were encouraged to
participate,” and also varied the title’s wording to emphasize free speech or
hate speech in different recruiting rounds.,141414We compensated workers
$0.02–$0.03 for this pre-survey. We stopped recruiting once we reached at
least 200 Black and 200 conservative participants.

###  C.3 Study Setup

We ran our study on the widely used survey platform Qualtrics, using an MTurk
HIT to recruit and compensate participants.151515Participants were compensated
$4.33 for the entire survey, equivalent to an average hourly compensation of
$22/h. Participants were first asked to consent to the task, then were shown
instructions for annotating the 15 posts (with occasional reminders of the
instructions). Then we asked participants their views on several topics using
the scales described in §2.2 and §A and finally their demographics. Throughout
the study, we added three attention checks to ensure the quality of responses.

Allowing only Black, white liberal, and white conservative workers to
participate, we ran our survey for 4 weeks from March 10 to April 5, 2021,
occasionally reminding participants from our pre-survey that they could take
the survey.

##  Appendix D Breadth-of-Posts Annotation Study Details

###  D.1 Data Selection

In this study, we draw from two existing corpora of posts labeled for
toxicity, hate, or offensiveness. First, we select posts that are
automatically detected as AAE and/or vulgar from Founta et al. (2018), using
the lexical detector of AAE by Blodgett et al. (2016) and the vulgar wordlist
by Zhou et al. (2021). Second, we select posts that are automatically detected
as vulgar and/or annotated as anti-Black from Vidgen et al. (2021).
Importantly, in this large-scale study, we consider posts that potentially
have multiple characteristics (e.g., AAE and vulgar), and thus consider both
posts with potentially offensive identity references (vulgar-OI) as well as
non-identity vulgar words (vulgar-OnI). However, to circumvent potential
racial biases in what is labelled as “racist” in the Vidgen et al. (2021)
corpus Sap et al. (2019); Davidson et al. (2019), we do not consider posts
that are annotated as anti-Black but detected as AAE.

Given an initial set of posts from our categories, we then randomly sample up
to 600 posts, stratifying by toxicity label, vulgarity, AAE, and anti-Black
meaning. Our final sample contains 571 posts, as outlined in Table 2 and
Figure 5.

###  D.2 Breadth-of-Posts Survey Details

As in the breadth-of-workers study, we recruit participants using a pre-
qualifying survey on MTurk. Then, we set up a second MTurk task to collect
toxicity ratings, and annotator attitudes and identities. For each post, we
collected two ratings from white conservative workers, two from white liberal
workers, and two from Black workers. To better mirror the crowdsourcing
setting and to reduce the annotator burden, we shorten the task to only ask
one question per attitude (listed in §A). We also asked one attention check
question to ensure data quality.

For this study, our final dataset contains 3,171 ratings from $N=173$
participants.161616As before, we discard 255 ratings where workers failed an
attention check. Our participants were 53% were men, 45% women, and $<$2% non-
binary, identified as 76% white, 20% Black, and $<$4% some other race, and
spanned the political spectrum from 54% liberal to 30% conservative, with 16%
centrists or moderates.

###  D.3 Selecting Attitude Questions

In order to simplify the annotation task for annotators, we abridged the
attitude scales to only one item. Using the data from the breadth-of-workers
study, we select the question that best correlated with all toxicity ratings.
Specifically, for each scale, we first take the tweet category with the
highest correlation with toxicity (e.g., anti-Black posts for RacistBeliefs),
and then take the item whose response scores correlated most with the toxicity
rating for those posts. Those items are bolded in §A.

##  Appendix E Further Breadth-of-Workers Results

We show all associations between attitudes and toxicity ratings in Table 10.

Table 10: Full set of results from our analyses of the breadth-of-workers
study of 15 posts, presented as Pearson $r$ or Cohen’s $d$ effect sizes, along
with significance levels (†: $p<0.075$, ∗: $p<0.05$, ∗∗: $p<0.001$). We
correct for multiple comparison for variable relationships that were
exploratory (i.e., not discussed as hypotheses in §4–6).  Figure 4: Average
ratings of offensiveness and racism for each tweet category in the breadth-of-
workers controlled study. All differences are significant ($p<0.001$) after
correcting for multiple comparisons.

Additionally, we investigate the differences in the overall toxicity ratings
of anti-Black vs. AAE vs. vulgar posts? (Figure 4). Overall, anti-Black tweets
were rated as substantially more offensive and racist than AAE or vulgar
tweets (with effect sizes ranging from $d$ = 2.4 to $d$ = 3.6). Additionally,
vulgar tweets were rated as more offensive than AAE tweets ($d$ = -0.29, $p$ <
0.001).

Surprisingly, we also found that AAE tweets were considered slightly more
racist than vulgar tweets ($d$ = 0.19, $p$ < 0.001). To further inspect this
phenomenon, we performed exploratory analyses by computing the differences in
ratings of racism for AAE and vulgar broken down by annotator gender, race,
and political leaning. We found that AAE tweets were rated as significantly
more racist than vulgar tweets only by annotators who were white or liberal
($d$ = 0.20 and $d$ = 0.22, respectively, with $p$ < 0.001 corrected for
multiple comparisons), compared to Black or conservative. There were no
significant differences when looking at men and women separately.

##  Appendix F Further Breadth-of-Posts Results

To account for the varying number of posts that each annotators could rate, we
use a linear mixed effects model171717Using the Python statsmodels
implementation. to compute associations between each post’s toxicity ratings
and identities or attitudes. Specifically, we our linear model regresses the
attitude score onto the toxicity score, with a random effect for each
worker.181818In R-like notation, toxicity$\sim$attitude+(1|WorkerId).

See Figure 5 and Table 11.

Figure 5: Venn diagram of number of tweets in each of the categories. Table
11:  Associations between the annotator demographic and attitude variables and
their ratings of offensiveness and racism on the posts from the breadth-of-
posts study. We break down the results by category, but categories are
overlapping. Only significant associations ($\beta$ coefficients from a mixed
effects model) are shown (†: $p<0.075$, ∗: $p<0.05$, ∗∗: $p<0.001$; Holm-
corrected for multiple comparisons).

##  Appendix G PerspectiveAPI Case Study: Details & Results

###  G.1 Details

We first obtain Perspective toxicity scores for all the posts in our breadth-
of-posts study (§3.2).191919the API was accessed in October 2021 Then, we
split workers into two different groups for each of our attitudes and identity
dimensions. For attitudes and political leaning, we assign each annotator to a
“high” or “low” group based on whether they scored higher or lower than the
mean score on that attitude scale. For gender and race, we use binary bins for
man/woman and white/black.

Then, for each attitude or identity dimension, we compute the Pearson $r$
correlation between the Perspective score and the toxicity ratings from the
high and low groups, considering posts from potentially overlapping categories
(e.g., AAE and potentially vulgar posts).Finally, we compare the high and low
correlations using Fisher’s $r$-to-$z$ transformation Silver and Dunlap
(1987).

###  G.2 Results

See Table 12 and Figures 6–13.

Table 12: We correlated the PerspectiveAPI toxicity scores with
offensiveness/racism ratings by our annotators, breaking them into two bins
based on their attitude scores. Then, we used Fisher’s z-to-r test to measure
whether the differences in correlations between the annotators who are
high/low were significant (†: $p<0.1$, ∗: $p<0.05$).  Figure 6: PerspectiveAPI
and ratings of offensiveness of anti-Black tweets. Figure 7: PerspectiveAPI
and ratings of racist of anti-Black tweets. Figure 8: PerspectiveAPI and
ratings of offensiveness of AAE tweets. Figure 9: PerspectiveAPI and ratings
of racist of AAE tweets. Figure 10: PerspectiveAPI and ratings of
offensiveness of vulgar-OI tweets. Figure 11: PerspectiveAPI and ratings of
racist of vulgar-OI tweets. Figure 12: PerspectiveAPI and ratings of
offensiveness of vulgar-OnI tweets. Figure 13: PerspectiveAPI and ratings of
racist of vulgar-OnI tweets.