# Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media

Sayan Ghosh
University of Michigan
<EMAIL_ADDRESS>
Dylan Baker
Google Research
<EMAIL_ADDRESS>
David Jurgens
University of Michigan
<EMAIL_ADDRESS>
Vinodkumar Prabhakaran
Google Research
<EMAIL_ADDRESS>

###### Abstract

Online social media platforms increasingly rely on Natural Language Processing
(NLP) techniques to detect abusive content at scale in order to mitigate the
harms it causes to their users. However, these techniques suffer from various
sampling and association biases present in training data, often resulting in
sub-par performance on content relevant to marginalized groups, potentially
furthering disproportionate harms towards them. Studies on such biases so far
have focused on only a handful of axes of disparities and subgroups that have
annotations/lexicons available. Consequently, biases concerning non-Western
contexts are largely ignored in the literature. In this paper, we introduce a
weakly supervised method to robustly detect lexical biases in broader
geocultural contexts. Through a case study on a publicly available toxicity
detection model, we demonstrate that our method identifies salient groups of
cross-geographic errors, and, in a follow up, demonstrate that these groupings
reflect human judgments of offensive and inoffensive language in those
geographic contexts. We also conduct analysis of a model trained on a dataset
with ground truth labels to better understand these biases, and present
preliminary mitigation experiments.

##  1 Introduction

Online social media platforms have increasingly turned to NLP techniques to
detect offensive and abusive content online at scale (e.g., the Perspective
API), as a way to mitigate the harms it causes to
people.111www.perspectiveapi.com However, recent research has shown that these
models often encode various societal biases against marginalized groups Waseem
(2016); Caliskan et al. (2017); Park et al. (2018); van Aken et al. (2018);
Dixon et al. (2018); Sap et al. (2019); Davidson et al. (2019); Hutchinson et
al. (2020); Zhou et al. (2021) and can be adversarially deceived Hosseini et
al. (2017); Gröndahl et al. (2018); Kurita et al. (2019a), potentially
furthering disproportionate harm to those in the margins. These models are
also challenged by several dimensions of noise, especially when compared with
the corpora they are trained on. For instance, social media text across the
globe may contain code switched language generated by multilingual speakers,
neologisms, and other orthographic variations rarely seen in training data.
This poses additional challenges to detect and mitigate fairness failures in
these models.

Sentence | Toxicity
---|---
You are a Tamilian! | 0.74
You are a Californian! | 0.17
ya ALLAH have mercy on those muslims suffering out there | 0.67
ya ALLAH have mercy on those people suffering out there | 0.36
Looks at these presstitutes, again only one side of the story! | 0.18
Looks at these prostitutes in the press, again only one side of the story! | 0.71
Madarchod, let it go! | 0.10
Motherfucker, let it go! | 0.97
Table 1: Example biases relevant to the Indian context, reflected in
Perspective API’s toxicity scores ($[0,1]$).  Figure 1: The high-level sketch
of our two-phase methodology to identify undesirable model biases.

In this paper, we propose a new weakly supervised method (outlined in Figure
1) to address shortcomings of current bias detection methods and robustly
detect lexical biases in models (i.e. biases that associate the presence or
lack of toxicity with certain words) when applied to noisy text (in
particular, text generated by social media users from non-predominantly
Anglophone nations). We draw upon the observation that social stereotypes and
biases in data often reflect the public discourse at a specific point in time
and place Fiske (2017); Garg et al. (2018). We use a cross-corpora analysis
across seven different countries to reveal terms overrepresented in each
country. We then cluster the model behaviour against term perturbations to
qualify these biases further. Furthermore, we demonstrate the effectiveness of
our method through human validation. Our method is task and genre agnostic; we
expect it to work in any culture-laden tasks on genres of noisy text from
platforms with global presence. Moreover, it can be applied in cases with and
without ground truth labels. We conduct experiments on both cases and present
a preliminary analysis on mitigation strategies to reduce harms created by
these biases.

##  2 Background

Current research on detecting biases in NLP models primarily starts with a
specific axis of injustice, and relies on annotated data Sap et al. (2019);
Davidson et al. (2019) or lexicons that are salient to different subgroups.
For example, Dixon et al. (2018) use a list of LGBTQ identity terms to
identify varied outcomes in text classification and Hutchinson et al. (2020)
use a list of terms that refer to persons with a disability. However, this
reliance on lexicons and annotated data limit the focus to a handful of social
injustices for which resources/lexicons are available, and largely ignores
biases in non-Western contexts Sambasivan et al. (2021) and online
environments containing creative or noisy language variation. Moreover, prior
work shows that interpretation of hate varies significantly across different
cultural contexts Salminen et al. (2018, 2019) and reveals that the
performance state-of-the-art models such as BERT Devlin et al. (2019) on
various tasks degrades significantly when used on noisy text Sarkar et al.
(2020); Kumar et al. (2020).

Consequently, current approaches to identifying model biases miss 1)
undesirable biases around words that are not captured in existing
lexicons/datasets, and 2) desirable biases around offensive terms that are
salient to different geocultural contexts that the models fail to capture. As
an example, such biases are present even in a commonly-used toxicity detector
like the Perspective API, as shown in Table 1. In the first two sets of rows,
we see that the mentions of certain identity groups (e.g., muslims, Tamilian)
causes the toxicity model to assign a higher toxicity score. The third and
fourth set of rows demonstrate that the model does not recognize the toxicity
association on the words presstitute and Madarchod,222Presstitute is a
portmanteau word blending the words press and prostitute, often used in Indian
social media discourse; maadarchod is an abusive word in Hindi, commonly used
in Indian social media discourse in English. whereas it does recognize the
toxicity of words/phrases with the same/similar meanings. In this paper, we
present a method to detect such undesirable lexical biases.

##  3 Methodology

We find undesirable lexical biases in two phases. The first phase aims to find
words that are statistically overrepresented in tweets from specific countries
that were deemed toxic by the model. Such overrepresentation could be due to
different reasons: (1) the model has learned some of these words to be
offensive and hence their presence causes it to assign a higher toxicity
score, or (2) some of the words appear more often in offensive contexts (e.g.,
frequent victims of toxicity) but their presence is merely correlational
(i.e., the model doesn’t have any toxicity association for these words).
Another consideration is whether the association of a particular word towards
toxicity is desirable or not. For some words, such as country-specific slur
words, we want the model to be sensitive to their presence. On the other hand,
we desire the models to not have biases towards culturally salient non-
offensive words, such as religious concepts.

The second phase attempts to separate the overrepresented words along these
two dimensions:

  * •

a descriptive axis that separates words representing correlational vs. causal
associations

  * •

a prescriptive axis that separates words representing desirable vs.
undesirable associations

Both phases are described in Figure 1 using tweets from India as an example.
The causal-desirable associations (e.g., bastard) and correlational-
undesirable associations (e.g., indian) capture the cases where the model is
behaving as desired. On the other hand, the causal-undesirable associations
(e.g., hindus) capture model biases that should be mitigated, whereas the
correlational-desirable associations (e.g., madarchod) capture the geo-
culturally salient offensive words the model missed. Our method aims to
balance (i) achieving reasonably high precision and (ii) narrowing down the
search space of words enough to be feasible in Phase 2 and any further human
evaluation studies. We now discuss the methods we use for each phase.

###  3.1 Phase 1: Identifying Biased Term Candidates

First, for each country, we calculate the log-odds ratio with a Dirichlet
informed prior Monroe et al. (2008) to find if term $i$ is statistically
overrepresented in toxic tweets vs. non-toxic tweets in that country. Unlike
the basic log-odds method, this method is robust against very rare and very
common words as it accounts for a prior estimate of the expected frequency of
each term. It also accounts for the variance in the word’s frequency by
calculating the z-score. However, common profanities are likely
overrepresented in toxic tweets globally, so this initial list must be further
filtered to identify geographically-specific terms. Since we now have a multi-
class setting, one could repeat the Monroe et al. (2008) method in a one-vs-
rest manner. However, such an approach would miss the important
interdependence between these tests.

Instead, we use the method from Bamman et al. (2014) and Chang and McKeown
(2019) that allows comparison across multiple groups of texts without
requiring separate binary comparisons. We consider the term $i$ is
overrepresented in toxic tweets from a country $j$ if $i$ occurs with higher
than statistically expected frequency in that country. Similar to prior works,
we assume a non-informative prior on $f_{i}$, with a
$\mathrm{Beta}(k_{i},N-k_{i})$ posterior distribution, where $k_{i}$ is the
count of $i$ in the geography-balanced corpus and $N$ is the word count of the
corpus. We use a balanced corpus with the same number of toxic tweets from
each country (matching the one with the least number of tweets). Term $i$ is
deemed significantly associated with country $j$ if the cumulative
distribution at $k_{ij}$ (the count of term $i$ in the corpus corresponding to
country $j$) is $\leq 0.05$.

By combining these two methods, we find culturally salient words that are
overrepresented in toxic tweets in each country. We also conducted experiments
using Monroe et al. (2008) method in a one-vs-rest manner (described above),
as well as using Bamman et al. (2014) method for both steps; but they provided
qualitatively worse results than the above approach.

Country | # Tweets | # Terms |  Sample of terms
---|---|---|---
India (IN) | 11.6m | 666 |  muslims, sanghi, presstitutes, chutiya, jihadi appeaser, journo, goons, madarchod
Pakistan (PK) | 11.8m | 616 |  feminists, porkistan, dhawan, harami, khawaja, israeli, pricks, rapistan, mullah
Nigeria (NG) | 10.9m | 543 |  mumu, tonto, kuku, ogun, tarik, savages, joor, africa, zevon, chop, virgin, baboon
Mexico (MX) | 14.6m | 665 |  puto, pendejo, culote, cabron, gringos, racist, horny, nalgotas, culazo, degenerate
Ghana (GH) | 9.3m | 377 |  ankasa, aboa, wati, ofui, sekof, barb, dier, kwacha, kwasia, nigerian, devil, spi
Philippines (PH) | 12.4m | 575 |  colonizers, delulu, pota, sasaeng, kadiri, crackhead, uwus, antis, stans, bis
Jamaica (JM) | 2.9m | 260 |  mufi, bloodclaat, wati, pickney, raasclaat, nuffi, mada, blacks, dung, unnu, unuh
Table 2: Sample words outputted by Phase 1 for each country

###  3.2 Phase 2: Qualifying Associations

To determine the causal vs. correlational distinction, one could rely on
counterfactual approaches such as perturbation sensitivity analysis (PSA)
Prabhakaran et al. (2019). However, PSA (and similar approaches) aggregate the
sensitivity to terms by calculating the mean difference in prediction scores
across all sentences. Here, we investigate how the sensitivity varies across
the range of toxicity scores, which gives important clues about the
desirability of biases. Using naturally occurring sentences like Prabhakaran
et al. (2019) would introduce numerous unknown correlational effects. Hence,
we followed Hutchinson et al. (2020) and May et al. (2019) by building a set
of $33$ template sentences (see Appendix A) that correspond to a range of
toxicity scores. A few examples below:

  * •

You are a person

  * •

I really hate it when person is there

  * •

I really dislike person

  * •

I am going to the movies with a person

  * •

The person was going to do that with me

For each term obtained in Phase 1, we measure the shift in toxicity in
response to replacing the word _person_ in the template sentence with that
term.

Ideally, words with little to no inherent offensiveness should have no effect
on a model’s estimate. Words that are highly offensive should increase the
score across the board. Deviations from these may indicate either undesirable
biases or knowledge gaps in the model. To model different patterns of behavior
by the classifier, we construct vector representations for each term by
creating vectors $\mathbf{x}\in\mathbb{R}^{d}$ such that $x_{i}$ is the
toxicity after replacing _person_ in the $i$-th template with a word from a
lexicon. An example of a visual representation of these vectors is shown in
Figure 2. We then cluster terms that have similar behaviour together to
qualitatively distinguish the kind of association they have with toxicity. In
this paper, we use the $k$-means clustering algorithm; however, our approach
is agnostic to what clustering method is used.

Figure 2: Example of vectorized deviations used in clustering for the term
“chutiya”. The $x$-axis indicates the original template scores. The $y$-axis
represents the score of the perturbed template.

##  4 Evaluation of Off-the-shelf Models

Our approach can notably be applied to cases where ground truth labels are not
available. This is useful in deployment settings, where it may not be feasible
and/or be expensive to annotate sufficient amounts of data regularly or where
one does not have access to the training data but wants to understand biased
outcomes. We provide an example of how our method could be used in this
setting.

###  4.1 Experimental Setting

We use the Perspective API’s toxicity model for our analysis. We use 73
million tweets from across 7 countries with substantial English-speaking
populations that are active online: India, Pakistan, Nigeria, Ghana, Jamaica,
Mexico, and the Philippines. We focus on these countries as opposed to
countries that are solely and primarily Anglophone as to better understand how
models interact with “non-standard” English dialects, especially ones that are
different from the data the model was likely trained on. Our data is collected
from a $\sim$10% random sample of the tweets from 2018 and 2019, from the
Decahose stream provided by Twitter. We pre-process tweets by removing URLs,
hashtags, special characters and numbers, and applying uniform casing. We also
lemmatize each resulting word to control for factors such as plurality using
NLTK Loper and Bird (2002). The number of tweets from each country is shown in
Table 2.

###  4.2 Results

#### Phase 1:

Our Phase 1 analysis identified several hundred terms overrepresented in each
country. The number of terms as well as a small subset of terms for each
country are shown in Table 2. As evident, our method is effective at picking
up words that are geography-specific, compared to simply using log-odds to
identify overrepresentation in toxic versus non-toxic tweets; such an approach
outputs more general, widely used profanities that (by nature) occur more
commonly in offensive messages across all countries. Crucially, not all of the
words identified by our method are inherently toxic: For example, the list for
Pakistan contains words such as muslims and journalism, which should not carry
a toxic connotation and could be reflective of model biases. Further, our
method identifies emergent country-specific terms such as presstitutes (a slur
used to refer to members of the press) and congi (a slur for supporters of the
Congress Party, a large political party) in India.

(a) India

(b)  Mexico

(c)  Nigeria

Figure 3: Changes in offensiveness scores of template sentences in response to
the use of over-represented terms for each cluster, for India, Mexico, and
Nigeria.

#### Phase 2:

In Phase 2, we analyze the model behaviour in response to perturbations using
each term from Phase 1, and cluster them based on model sensitivity. Here, we
focus on nouns and named entities which represent $77\%$ of the terms from
phase $1$ and are the most salient to analyze in terms of subgroups (although
the method can be altered to use templates that allow for the use of other
parts of speech). We use $k$-means clustering; we selected $k=4$ based on
manual inspection of $k=[3,6]$; this inspection showed that the majority of
instances fall into four natural clusters, with lower $k$ merging dissimilar
instances or higher $k$ producing clusters with very few instances. Figure 3
shows the model behaviour of each cluster we obtained for India, Mexico, and
Nigeria; figures for the remaining countries are shown in Appendix B, with a
sample of words in each cluster. We list a sample of words in each cluster for
India below:

  * •

C0: muslim sexy indian hindu jihadi pakistani islam women feminist liberals
italian arabs communist jews christian priest kafir australian

  * •

C1: morons bastards fools cock loser hypocrite coward rapists scum ignorant
cursed retarded losers moronic arse sissy dolt scumbag cunts imbecile pervert
slut boob pedophile prostitutes

  * •

C2: bloody shameless useless terrorist filthy killer pig disgrace arrogant
corrupt bigot horrible selfish commie slave irritating filth troll murderer
kutta liars sexual thug uneducated donkey

  * •

C3: fake sanghi blind journalist pappu mullah chaddi child appeaser wife
country vala chankya tadipad mota dogs creature abuse animals bhakt jinahh
pidi spineless shove

Qualitative inspection of the clusters reveals certain properties of the words
within each. Cluster C0 (top left of Figure 3a) contains words such as
feminist and muslim that increase the toxicity of the templates with the
largest changes occurring in the middle, indicating undesirable biases in the
model. In contrast, cluster C1 (top right) has words such as shithead that
uniformly raise the toxicity of the templates to the very high range,
regardless of the initial toxicity of the template.

Words in cluster C2 (bottom left) display a similar early trajectory to those
in cluster C0, but converges towards the diagonal. Words in this cluster
include presstitute and porki, both of which are country-specific slurs,
suggesting that the model only has a weak signal about their toxicities.
Similarly, cluster C3 (bottom right) consists of words that do not affect the
templates much at all after replacement. Some words in this cluster are indeed
not toxic, but we also see words such as congi that the model has not
encountered before, but carry a negative connotation in Indian online
discourse.

(a) India

(b) Mexico

(c)  Nigeria

Figure 4:  In-community judgments of word offensiveness by cluster, shown here
for India, Mexico, and Nigeria (cf. Figure 3) show that the method is indeed
finding biases perceived by in-community speakers.

###  4.3 In-Community Validation

To quantitatively evaluate whether the word categories uncovered by our method
correspond to real biases as viewed through members of a geographic community,
we conducted a crowd-sourcing study via the Qualtrics platform. We note that
even despite this step, there may be certain forms of bias that are not
uncovered by our method. However, this form of validation attempts to judge
whether the biases that our method does uncover have a correspondence with
real-world judgments.

We recruited 25 raters per country, and each was asked to rate 60 terms
(including 10 control words) to be (i) inoffensive, (ii) sometimes offensive,
(iii) highly offensive, or (iv) don’t know this word. We obtained 5 ratings
each for 250 words per country. We removed ratings from raters who got more
than 3 out of 10 control words wrong. Full annotation details are provided in
Appendix C.333 Despite recruiting a large pool of annotators for each
instance, our annotators may not entirely capture the value systems of
countries due to selection effects of who has access to the internet and is
able to participate in Qualtrics work. For instance, raters from India in our
human ratings potentially overrepresent views of the middle class, those of
upper caste individuals, and views from a handful of states with tech hubs. As
a result, their perception of what is or is-not offensive may not be
representative of India at large.

We hypothesize that the words grouped into C0 would be judged identical to
those in C3 (i.e., treated as false positive features), where C2 should be
more offensive, potentially up to C1 (i.e., false negative). In aggregate, the
annotator ratings largely matched our hypotheses across countries. Figure 4
shows the ratings for the clusters from India, Mexico and Nigeria (cf. Figure
3); here, we re-scale their categorical ratings into [0,1]. Our results show
that the words in C0 and C3 are often very similar in their perceived toxicity
and, in the case of India, have no statistical difference in their toxicity
ratings. Our analysis on Nigerian tweets does differ in this trend, however,
where C0 is perceived to be more toxic than C3—and even slightly more than C2.
Words in this cluster include feminist, junkie, africans, and maggot, which,
though slightly negative in connotation, would not appear to be overly
offensive and in-line with expectations for what C0 should contain. This
difference for Nigeria in our case study exemplifies how in-community
interpretations do differ from those of external readers and underscores the
importance of including these viewpoints when assessing model errors.

The C2 clusters, which contains words like “presstitute” in India, were found
offensive, matching our hypothesis of these words being false negatives. In
all countries, the level of offensiveness was still less than the level for
extremely toxic words in C1, but still larger than C3.Figures for other
countries are shown in the Appendix B. The clusters found by our method do
correspond to in-community judgments of toxicity and reflect meaningful groups
of biases in the model, which can be used by experimenters for later bias
testing and mitigation.

##  5 Towards Mitigating Undesirable Biases

Our pipeline provides an unsupervised approach to identifying potential latent
biases in toxicity models. Ideally, practitioners aim not only to identify
these biases but also to fix them before deploying their systems. However, our
method identifies terms of different types: causal-desirable associations and
correlational-undesirable associations, as well as causal-undesirable and
correlational-desirable associations both of which require different
approaches to handle. The clusters we find in Phase 2 approximate these
different types of associations, and hence we expect them to behave
differently for different mitigation approaches. Here, we test the impact of
baseline mitigation techniques on reducing bias in these clusters.

###  5.1 Experimental Setting

Performing bias mitigation experiments involves re-training the model, and
hence requires toxicity annotated training data. Since we do not have access
to the training data used to train the Perspective API model, we employ the
dataset provided by Jigsaw for the Kaggle competition on toxicity
classification Jigsaw (2019) that contains public comments from the Civil
Comments platform and annotated toxicity ratings (on a continuous scale in the
range $[0,1]$). Here, as per the original guidelines provided by Jigsaw, we
consider an instance with a rating of $>=0.5$ as toxic. We train the toxicity
model for this analysis by fine-tuning a DistilBert model Sanh et al. (2019)
using pre-trained parameters from HuggingFace on the Jigsaw dataset. The model
is fine-tuned for one epoch with an initial learning rate setting of $2\times
10^{-5}$.444The model was trained for three epochs with early stopping;
ultimate model performance on a held-out 10% development set was highest after
one epoch.

While some performance differences are inherent between this model and the
Perspective API model, our method still identifies similar latent biases.
Phase 1 of our pipeline identifies 150 candidate words with bias. These
include words such as liberal, muslim, sanghi, and commie. In particular, we
also note that the set of words flagged using the scores from our model is a
strict subset of the words flagged using the scores from Perspective API. To
quantify the effects of mitigation strategies, we further analyze 120 of these
words that appear in at least 10 instances each for the toxic and non-toxic
labels in the Jigsaw dataset.

###  5.2 Bias Measurement

| AUC/AEG Bias Metrics (Mean) | Toxicity Performance
---|---|---
Model |  Subgroup-AUC $\uparrow$ |  BPSN-AUC $\uparrow$ |  BNSP-AUC $\uparrow$ |  AEG+ $\downarrow$ |  AEG- $\downarrow$ |  | F1 | Overall AUC
No Mitigation | 0.91 | 0.88 | 0.98 | 0.08 | 0.25 |  | 0.70 | 0.97
Substitution | 0.89 | 0.86 | 0.96 | 0.07 | 0.26 |  | 0.63 | 0.96
Deletion | 0.88 | 0.86 | 0.96 | 0.07 | 0.26 |  | 0.60 | 0.95
Balance and Tune ($k$=100) | 0.91 | 0.90 | 0.97 | 0.07 | 0.24 |  | 0.69 | 0.97

Table 3: Model performance after applying bias-correction metrics, showing
that the mitigation was largely ineffective at reducing bias. Arrows indicate
the direction of less model bias.

As Borkan et al. (2019) note, biases in models can be nuanced and varied, and
require multiple metrics that can provide insights on how to mitigate them.
They compute a suite of five metrics based on subgroups of instances defined
around the presence of different identity terms. We adopt a similar approach
by defining 120 subgroups based on the presence of each of the 120 terms our
bias evaluation pipeline identified.

The first three metrics are based on AUC. (1) Subgroup AUC reflects how
separable are the toxic/non-toxic instances containing our target words. (2)
Background Positive Subgroup Negative (BPSN) AUC measures how separable are
non-toxic instances containing the target words from toxic instances without
them; lower scores are typical of models with false positives. (3) Background
Negative Subgroup Positive (BNSP) AUC measures how separable are non-toxic
instances without the target words from toxic instances with them; lower
scores are typical of models with many false negatives. As AUC, these scores
also range $[0,1]$ and higher scores are desirable.

The last two metrics compare the toxicity scoring distributions of the
instances with the target words and those instances without. These metrics
measure the Average Equality Gap (AEG) for toxic and non-toxic instances
(separately) within $[-0.5,0.5]$, where scores close to 0 indicate no bias in
how instances with the target word are treated. The AEG metrics capture cases
where the scoring distributions of the target words are shifted but not
sufficiently to cause re-orderings (by toxciity) that would be identified by
the AUC metrics. We refer the reader to Borkan et al. (2019, §3.3) for a
detailed discussion of what these metrics capture.

###  5.3 Mitigation Approaches

Mitigating undesirable biases in NLP models is an active research area Zhou et
al. (2021), where counterfactual and data augmentation techniques tend to be
the most common approach Dixon et al. (2018); Garg et al. (2019); Davani et
al. (2020). In this section, we present experiments using three such , with
the goal to demonstrate that different approaches are required to mitigate
different types of biased associations our pipeline reveals.

Deletion: Here, we delete all training instances containing a biased word and
re-do the fine-tuning of the DistillBERT model using the remaining instances.
In doing this, we expect to remove undesirable associations around the target
words learned from the training data, however it will remove a lot of useful
information as a side effect.

Substitution: Here, we substitute all occurrences of the target words in the
training data with a <UNK> token. This is similar to the blindness setting
used by Garg et al. (2019). Unlike the Deletion approach, this approach aims
to remove undesirable associations without the side effect of dropping useful
instances.

Balance and Tune: Finally, we use an approach that further fine-tunes the
model using a subset of the dataset where each target word has an equal number
of toxic and non-toxic instances. Formally, for $m$ occurrences of the target
word in instances with the toxic label and $n$ instances with the non-toxic,
we select $max(k,min(m,n))$ instances of each and fine-tune the model further
only on these instances. We experiment with $k$=10, 50, 100. For all three
experiments, we use the same fine-tuning hyperparameters as in the baseline
model.555Separate hyperparameter tuning on the held-out development set showed
these parameters to still be optimal.

Figure 5: Cluster-level disaggregated bias mitigation results. .

###  5.4 Results

In aggregate, none of the mitigation strategies substantially reduced the
bias, as shown in Table 3.666Results for the Balance and Tune models were
similar for all $k$. We show results with other $k$ values in Appendix D.
Three notable trends were observed. First, mitigation strategies varied in
their net effect on recognizing toxicity. Substituting the biased word for
<unk> or deleting those instances entirely had a detrimental effect on model
F1. In these cases, AUC remained largely the same as without mitigation
suggesting these models are still capable of ordering instances correctly by
toxicity, but the overall scoring has changed sufficiently to affect
classification decisions. In contrast, the Balance and Tune had little effect
on classifier performance.

Second, the Substitution and Deletion models both increase bias on every
metric except AEG+. Despite not training models on any instances containing
the target terms, the resulting models still produced classification decisions
biased against the target words. This result suggest that the pre-trained
DistilBERT model itself is likely responsible for the latent bias, mirroring
multiple results showing biases in these models Kurita et al. (2019b); Shwartz
et al. (2020); Nadeem et al. (2020).

Third, the Balance and Tune strategy had negligible effect on both bias
correction or toxicity performance. This result suggests that starting from a
biased classifier based on a pre-trained language model, the bias cannot
easily be corrected by simply rebalancing the class distribution in training
data and further fine-tuning. Together, our results point to how simple but
seemingly common-sense approaches fail to substantially reduce bias and thus
motivate the need for (i) more sophisticated mitigation strategies or (ii)
pre-trained language models free of such latent biases.

Do bias mitigation approaches change performance differently for each cluster
of terms? Figure 5 shows cluster-level disaggregated bias metrics for the
baseline model, as well as the increases/decreases obtained through the
different mitigation approaches, showing that the bias metrics do vary across
different clusters. The high AEG+/- of clusters C0 and C2 indicates that both
clusters, on average, score higher than background examples, confirming the
intuitive notions captured by the visual examination of the clusters (i.e. the
terms in these clusters tend to be rated higher than their corresponding
“neutral” templates). Cluster C1 also has higher AEG metrics, but these are
mostly toxic words that the model already recognizes. On the other hand, C1
and C2 fares poorer than the other two on AUC bias metrics, especially in
BPSN-AUC.

Our bias mitigation approaches also act differently on different clusters.
Overall, the Balance and Tune technique has the most gains in bias metrics in
C1, and moderate gains in C2, whereas it has negligible impact on C0 and C3.
On the other hand, Deletion and Substitution approaches significantly worsened
AEG metrics of C3, as well as AUC metrics, across the board except for BPSN-
AUC of C1. These results demonstrate the need for different approaches to
mitigate the different kinds of biased associations our pipeline reveals.
However, more work needs to be done on finding which mitigation approaches are
appropriate for different kinds of undesirable biases.

##  6 Discussion and Conclusion

Toxicity detection models commonly rely on training data that reflects
discourse that is temporally and culturally situated. As such, models may
contain biases by learning undesirable features that are not toxic or be blind
to toxic language underrepresented in their training data. Here, we introduce
a new weakly supervised approach for uncovering these biases.777All code and
data to replicate the results will be released at the url:
https://github.com/sghosh73/Cross-Cultural-Bias-Detection Our method provides
a key diagnostic tool for model creators and deployers to test for biases.
From a fairness perspective, our analysis reveals biases present in the model
along different dimensions, without having any labeled resources for those
dimensions. For instance, C0 revealed biases around religion (muslim, hindu,
jews), country/ethnicity (indian, pakistani, arabs), and ideology (feminist,
liberals, communist) without looking for these axes a priori. This could be a
valuable first step before deeper analysis of these biases.

At a practical level, our method reveals country-specific offensive terms that
the model has not seen before (e.g., maadarchod). Such country-specific
abusive language lexicons could aid NLP practitioners trying to make their
models robust across geographies. Such lexicons could also aid human-rights
organizations working on building region-specific hate-speech
lexicons.888https://www.peacetechlab.org/ Furthermore, our model also
identifies portmanteau (blended) words that are used disproportionately in
certain geo-cultural contexts (e.g., presstitute, and libtard in Indian social
media discourse). Our method could help monitor emergent offensive language in
different geographies and aid data augmentation efforts in keeping models up-
to-date through surfacing emergent language in new geographies.

##  7 Ethical Considerations

The proposed method provides an efficient distantly-supervised method for
practitioners to identify potential biases in their toxicity detection
methods. Although intended strictly as beneficial, it could create the risk of
overconfidence in a lack of bias by a particular model. While we have
demonstrated that the approach identifies clusters of words that mirror in-
group judgments of (i) offensiveness that the model failed to recognize and
(ii) inoffensiveness that the model has treated as offensive due to
correlational biases, our method alone is likely insufficient for identifying
all such biases. Demonstrating that our method does not identify biases in a
new model should not be considered proof of a lack of bias. Similarly,
debiasing a model around the words our method finds may not remove the
underlying biases in the model.

Additionally, our method surfaces words that have correlational bias due to
overrepresentation in toxic messages. Such words are often references to
victims of hateful targeting and the highlighting by our method could
potentially re-traumatize by recalling (and exposing) these messages, or even
leading to a fresh wave of targeting. However, our method may help improve
automatic content moderation tools, thereby reducing those individuals’
exposure to such words on online platforms.

Finally, our method reveals country-specific abusive words. While such
lexicons have many beneficial uses, including within NLP, they also have the
potential for malicious dual uses. Hence, developers and practitioners should
take caution while developing, deploying, and sharing this method.

## Acknowledgements

We thank Ben Hutchinson for his insightful feedback on this paper, the
crowdworkers from Qualtrics for rating the words in this study, and the
anonymous reviewers for their helpful suggestions. The third author is
supported in part by the Volkswagen Foundation.

## References

  * Bamman et al. (2014) David Bamman, Jacob Eisenstein, and Tyler Schnoebelen. 2014.  Gender identity and lexical variation in social media.  _Journal of Sociolinguistics_ , 18(2):135–160.
  * Borkan et al. (2019) Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019\.  Nuanced metrics for measuring unintended bias with real data for text classification.  In _Companion proceedings of the 2019 world wide web conference_ , pages 491–500.
  * Caliskan et al. (2017) Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ , 356(6334):183–186.
  * Chang and McKeown (2019) Serina Chang and Kathy McKeown. 2019.  Automatically inferring gender associations from language.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5746–5752, Hong Kong, China. Association for Computational Linguistics.
  * Davani et al. (2020) Aida Mostafazadeh Davani, Ali Omrani, Brendan Kennedy, Mohammad Atari, Xiang Ren, and Morteza Dehghani. 2020.  Fair hate speech detection through evaluation of social group counterfactuals.  _arXiv preprint arXiv:2010.12779_.
  * Davidson et al. (2019) Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019.  Racial bias in hate speech and abusive language detection datasets.  In _Proceedings of the Third Workshop on Abusive Language Online_ , pages 25–35, Florence, Italy. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT (1)_.
  * Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and mitigating unintended bias in text classification.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 67–73.
  * Fiske (2017) Susan T Fiske. 2017.  Prejudices in cultural contexts: shared stereotypes (gender, age) versus variable stereotypes (race, ethnicity, religion).  _Perspectives on psychological science_ , 12(5):791–799.
  * Garg et al. (2018) Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018.  Word embeddings quantify 100 years of gender and ethnic stereotypes.  _Proceedings of the National Academy of Sciences_ , 115(16):E3635–E3644.
  * Garg et al. (2019) Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. 2019.  Counterfactual fairness in text classification through robustness.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 219–226.
  * Gröndahl et al. (2018) Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N Asokan. 2018.  All you need is” love” evading hate speech detection.  In _Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security_ , pages 2–12.
  * Hosseini et al. (2017) Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017.  Deceiving google’s perspective api built for detecting toxic comments.  _arXiv preprint arXiv:1702.08138_.
  * Hutchinson et al. (2020) Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020.  Social biases in NLP models as barriers for persons with disabilities.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5491–5501, Online. Association for Computational Linguistics.
  * Jigsaw (2019) Jigsaw. 2019.  Unintended bias in toxicity classification.  Accessed: 2021-05-01.
  * Kumar et al. (2020) Ankit Kumar, Piyush Makhija, and Anuj Gupta. 2020.  User generated data: Achilles’ heel of bert.  _arXiv e-prints_ , pages arXiv–2003.
  * Kurita et al. (2019a) Keita Kurita, Anna Belova, and Antonios Anastasopoulos. 2019a.  Towards robust toxic content classification.  _arXiv preprint arXiv:1912.06872_.
  * Kurita et al. (2019b) Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019b.  Measuring bias in contextualized word representations.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 166–172, Florence, Italy. Association for Computational Linguistics.
  * Loper and Bird (2002) Edward Loper and Steven Bird. 2002.  Nltk: The natural language toolkit.  In _Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics_ , pages 63–70.
  * May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019\.  On measuring social biases in sentence encoders.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 622–628, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Monroe et al. (2008) Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008.  Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict.  _Political Analysis_ , 16(4):372–403.
  * Nadeem et al. (2020) Moin Nadeem, Anna Bethke, and Siva Reddy. 2020.  Stereoset: Measuring stereotypical bias in pretrained language models.  _arXiv preprint arXiv:2004.09456_.
  * Park et al. (2018) Ji Ho Park, Jamin Shin, and Pascale Fung. 2018.  Reducing gender bias in abusive language detection.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2799–2804, Brussels, Belgium. Association for Computational Linguistics.
  * Prabhakaran et al. (2019) Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019.  Perturbation sensitivity analysis to detect unintended model biases.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5740–5745, Hong Kong, China. Association for Computational Linguistics.
  * Salminen et al. (2019) Joni Salminen, Hind Almerekhi, Ahmed Mohamed Kamel, Soon-gyo Jung, and Bernard J Jansen. 2019.  Online hate ratings vary by extremes: A statistical analysis.  In _Proceedings of the 2019 Conference on Human Information Interaction and Retrieval_ , pages 213–217.
  * Salminen et al. (2018) Joni Salminen, Fabio Veronesi, Hind Almerekhi, Soon-Gvo Jung, and Bernard J Jansen. 2018.  Online hate interpretation varies by country, but more by individual: A statistical analysis using crowdsourced ratings.  In _2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS)_ , pages 88–94. IEEE.
  * Sambasivan et al. (2021) Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021.  Re-imagining algorithmic fairness in india and beyond.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 315–328.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019.  The risk of racial bias in hate speech detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1668–1678.
  * Sarkar et al. (2020) Rupak Sarkar, Sayantan Mahinder, and Ashiqur KhudaBukhsh. 2020.  The non-native speaker aspect: Indian english in social media.  In _Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)_ , pages 61–70.
  * Shwartz et al. (2020) Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. 2020.  “You are grounded!”: Latent name artifacts in pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6850–6861, Online. Association for Computational Linguistics.
  * van Aken et al. (2018) Betty van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. 2018.  Challenges for toxic comment classification: An in-depth error analysis.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , pages 33–42, Brussels, Belgium. Association for Computational Linguistics.
  * Waseem (2016) Zeerak Waseem. 2016.  Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter.  In _Proceedings of the first workshop on NLP and computational social science_ , pages 138–142.
  * Zhou et al. (2021) Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021.  Challenges in automated debiasing for toxic language detection.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3143–3155, Online. Association for Computational Linguistics.

##  Appendix A Templates

We present the list of templates as well as their corresponding toxicity
scores according to the Perspective model.

Template | Toxicity
---|---
You are a person | 0.107
Here is a person | 0.068
The person is there | 0.050
It is a person | 0.052
This is a person | 0.061
I am currently with a person | 0.076
I am going to the movies with a person | 0.116
The person was with me at the time | 0.085
I was planning on going there with person | 0.087
The person was going to do that with me | 0.106
I really dislike person | 0.445
I really hate it when person is there | 0.304
I hate person | 0.762
Person sucks | 0.887
It sucks when person is here | 0.680
The person over there is dumb | 0.817
It really does suck when person is here | 0.709
These people are illiterate | 0.794
These people are crooked | 0.631
These people don’t make any sense | 0.243
Whenever I see a person, I get angry | 0.278
These people are ruining our city | 0.446
These people come here and destroy our country | 0.687
I think people like person are a pain to be around | 0.539
When a person comes here, it just makes me feel really pissed off | 0.363
I saw a person at the event and immediately left because they made me want to throw up | 0.335
Just looking at a person makes me want to throw up | 0.367
I don’t value the opinion of a person | 0.319
I think that a person is always wrong and just not that smart | 0.535
A person like that disgusts me | 0.587
It causes me pain to be around a person | 0.268
A person disgusts me | 0.550
A person makes me violently ill | 0.672
Table 4: Templates and their corresponding toxicities

##  Appendix B Further Results for Other Countries

We present results for the other countries we collected data for. Note that C3
and cluster C0 may not always be equivalent. For each country, cluster C3
contains several culture-specific terms, and the model may have varying
degrees of success at recognizing the toxicity of these terms, leading to a
varying number of false/true negatives in cluster C3.

  * •

C0: ’indian’, ’afghani’, ’israelis’, ’jihadis’

  * •

C1: ’fucktards’, ’imbeciles’, ’horseshit’, ’dogshit’

  * •

C2: ’retarded’, ’paki’, ’fascist’, ’murderer’

  * •

C3: ’liberals’, ’choor’, ’mullah’, ’propagandist’

Figure 6: Changes in offensiveness for overrepresented words in Pakistan, with
example words from each cluster.

  * •

C0: ’feminist’, ’junkie’, ’africans’, ’maggot’

  * •

C1: ’bastard’, ’nitwit’, ’twat’, ’retard’

  * •

C2: ’nigerians’, ’gangster’, ’savage’, ’goats’

  * •

C3: ’ladies’, ’beings’, ’ashawo’, ’oshi’

Figure 7: Changes in offensiveness for overrepresented words in Nigeria, with
example words from each cluster.

  * •

C0: ’crackheads’, ’faker’, ’skank’, ’asians’

  * •

C1: ’sissy’, ’bullcrap’, ’dumbasses’, ’puta’

  * •

C2: ’gagu’, ’stalker’, ’uwus’, ’idols’

  * •

C3: ’colonizers’, ’delulu’, ’groupmates’, ’kang’

Figure 8: Changes in offensiveness for overrepresented words in The
Philippines, with example words from each cluster.

  * •

C0: ’asian’, ’mexicans’, ’gringo’, ’jewish’

  * •

C1: ’idiota’, ’nipple’, ’pendejo’, ’tits’

  * •

C2: ’culon’, ’perra’, ’cuck’, ’negro’

  * •

C3: ’vato’, ’flirt’, ’chavas’, ’republicans’

Figure 9: Changes in offensiveness for overrepresented words in Mexico, with
example words from each cluster.

  * •

C0: ’virgin’, ’whites’, ’maggot’, ’africans’

  * •

C1: ’dickhead’, ’shite’, ’dildos’, ’penises’

  * •

C2: ’barb’, ’swine’, ’fraud’, ’savage’

  * •

C3: ’kwasia’, ’ashawo’, ’mumu’, ’willian’

Figure 10: Changes in offensiveness for overrepresented words in Ghana, with
example words from each cluster.

  * •

C0: ’gyal’, ’wasteman’, ’americans’, ’females’

  * •

C1: ’jackass’, ’fuckhead’, ’shithouse’, ’niggah’

  * •

C2: ’trash’, ’thugs’, ’demon’, ’nerd’

  * •

C3: ’nuffi’, ’mada’, ’mussi’, ’eediat’

Figure 11: Changes in offensiveness for overrepresented words in Jamaica, with
example words from each cluster.

##  Appendix C In-Community Analysis

To quantitatively evaluate whether the word categories uncovered by our method
correspond to real biases as viewed through members of a geographic community,
we conducted a crowd-sourcing study via the Qualtrics platform. We recruited
25 raters per country, and each of them was asked to rate 60 terms (including
10 control words) to be (i) inoffensive, (ii) sometimes offensive, (iii)
highly offensive, or (iv) don’t know this word. Altogether, we obtained 5
ratings each for 250 words per country.

Each rater was given a survey with 5 questions, each question containing 12
words that they were asked to drag and drop to one of the four buckets
described above. This graphical interface was chosen to mitigate survey
fatigue and for making the task more engaging for the raters. On average, the
raters took 8 mins to complete the task of 5 questions with 12 words each. The
raters were paid between $6.50 and $12.50 depending on the country, for
completing the task. We did not get human labels for Jamaica, since we failed
to recruit raters.

| AUC/AEG Bias Metrics (Mean) | Toxicity Performance
---|---|---
Model |  Subgroup-AUC $\uparrow$ |  BPSN-AUC $\uparrow$ |  BNSP-AUC $\uparrow$ |  AEG+ $\downarrow$ |  AEG- $\downarrow$ |  | F1 | Overall AUC
No Mitigation | 0.91 | 0.88 | 0.98 | 0.08 | 0.25 |  | 0.70 | 0.97
Substitution | 0.89 | 0.86 | 0.96 | 0.07 | 0.26 |  | 0.63 | 0.96
Deletion | 0.88 | 0.86 | 0.96 | 0.07 | 0.26 |  | 0.60 | 0.95
Balance and Tune ($k$=100) | 0.91 | 0.90 | 0.97 | 0.07 | 0.24 |  | 0.69 | 0.97
Balance and Tune ($k$=50) | 0.91 | 0.89 | 0.97 | 0.07 | 0.24 |  | 0.69 | 0.97
Balance and Tune ($k$=10) | 0.91 | 0.89 | 0.97 | 0.07 | 0.24 |  | 0.69 | 0.97

Table 5: Model performance after applying bias-correction metrics, showing
that the mitigation was largely ineffective at reducing bias. Arrows indicate
score direction for less model bias.

##  Appendix D Additional Mitigation Results

Table 5 shows results for the Balance and Tune model with additional values of
$k$, along with the full results of all other models.