# Data Bias According to Bipol: Men are Naturally Right and It is the Role of
Women to Follow Their Lead

Irene Pagliai*1, Goya van Boven*2, Tosin Adewumi*${\dagger}$, Lama
Alkhaled${\dagger}$, Namrata Gurung3,
Isabella Södergren4 and Elisa Barney${\dagger}$
*Joint first authors
1University of Göttingen, Germany, 2Utrecht University, the Netherlands,
*${\dagger}$Machine Learning Group, LTU, Sweden, 3QualityMinds GmbH, Germany,
4Digital Services and Systems, LTU, Sweden
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

We introduce new large labeled datasets on bias in 3 languages and show in
experiments that bias exists in all 10 datasets of 5 languages evaluated,
including benchmark datasets on the English GLUE/SuperGLUE leaderboards. The 3
new languages give a total of almost 6 million labeled samples and we
benchmark on these datasets using SotA multilingual pretrained models: mT5 and
mBERT. The challenge of social bias, based on prejudice, is ubiquitous, as
recent events with AI and large language models (LLMs)have shown. Motivated by
this challenge, we set out to estimate bias in multiple datasets. We compare
some recent bias metrics and use bipol, which has explainability in the
metric. We also confirm the unverified assumption that bias exists in toxic
comments by randomly sampling 200 samples from a toxic dataset population
using the confidence level of 95% and error margin of 7%. Thirty gold samples
were randomly distributed in the 200 samples to secure the quality of the
annotation. Our findings confirm that many of the datasets have male bias
(prejudice against women), besides other types of bias. We publicly release
our new datasets, lexica, models, and codes.

Caution: This paper contains examples of what some may consider as stereotypes
or offensive text.

##  1 Introduction

The problem of social bias in data is a pressing one. Recent news about social
bias of artificial intelligence (AI)systems, such as
Alexa111bbc.com/news/technology-66508514 and
ChatGPT,222bloomberg.com/news/newsletters/2022-12-08/chatgpt-open-ai-s-
chatbot-is-spitting-out-biased-sexist-results shows that the age-old problem
persists with data, which is used to train machine learning (ML)models. Social
bias is the inclination or prejudice for, or against, a person, group or idea,
especially in a way that is considered to be unfair, which may be based on
race, religion or other factors (Bellamy et al., 2018; Antoniak & Mimno, 2021;
Mehrabi et al., 2021; Alkhaled et al., 2023). It can also involve stereotypes
that generalize behavior to groups (Brownstein, 2019). It can unfairly skew
the output of ML models (Klare et al., 2012; Raji et al., 2020). Languages
with fewer resources than English are also affected (Rescigno et al., 2020;
Chávez Mulsa & Spanakis, 2020; Kurpicz-Briki, 2020). For example, in Italian,
the female gender is under-represented due to the phenomena such as the
“inclusive masculine" (when the masculine is over-extended to denote groups of
both male and female referents) (Luccioli et al., ; Vanmassenhove & Monti,
2021).

In this work, we are motivated to address the research question of how much
bias exists in the text data of multiple languages, if at all bias exists in
them. We particularly investigate 6 benchmark datasets on the English
GLUE/SuperGLUE leaderboards (Wang et al., 2018; 2019) and one dataset each for
the other 4 languages: Italian, Dutch, German, and Swedish. First, we train
SotA multilingual Text-to-Text Transfer Transformer (mT5)(Xue et al., 2021)
and multilingual Bidirectional Encoder Representations from Transformers
(mBERT) models for bias classification on the multi-axes bias dataset
(MAB)(Alkhaled et al., 2023) for each language. For the evaluations, we search
through the literature to compare different metrics or evaluation methods as
shown in Table 1 and discussed in Section 2. This motivates our choice of
bipol, the multi-axes bias metric, which we then compare in experiments with a
lexica baseline method. In addition, to confirm the unverified assumption that
toxic comments contain bias (Sap et al., 2020; Alkhaled et al., 2023), we
annotate 200 randomly-selected samples from the training set of the English
MAB.

Metric/Evaluator | Axis | Terms
---|---|---
Winogender (Rudinger et al., 2018) | 1 | 60
WinoBias (Zhao et al., 2018) | 1 | 40
StereoSet (Nadeem et al., 2021) | 4 | 321
GenBiT (Sengupta et al., 2021) | 1 | -
CrowS-Pairs (Nangia et al., 2020) | 9 | 3,016
Bipol (Alkhaled et al., 2023) |  $>$2, $<$13 |  $>$45, $<$466
Table 1:  Comparison of some bias evaluation methods.

##### Our Contributions

  * •

We make available new large labeled datasets on bias of almost 2 million
samples each for 3 languages: Italian, Dutch, and German.333github.com/LTU-
Machine-Learning/bipolmulti

  * •

We make available lexica of sensitive terms for bias detection in the 3
languages.

  * •

We confirm the unverified assumption in the underlying datasets of MAB (Social
Bias Inference Corpus v2 (SBICv2)and Jigsaw) (Alkhaled et al., 2023) that
toxic comments contain bias.

The rest of this paper is organized as follows. In Section 2, we discuss the
literature review of related work. In Section 3, we briefly discuss the bipol
metric. In Section 4, we explain the steps involved in the methodology and the
datasets we use. In Section 5, we present our findings and discuss them. In
Section 6, we end with the conclusion and possible future work.

##  2 Literature Review

Although English usually gets more support and attention in the literature,
there have been attempts at measuring and mitigating bias in other languages.
Testing for the presence of bias in Italian often has a contrastive
perspective with English, with a focus on gender bias (Gaido et al., 2021;
Rescigno et al., 2020). MuST-SHE (Bentivogli et al., 2020) and gENder-IT
(Vanmassenhove & Monti, 2021) are examples of gender bias evaluation sets.
Going beyond gender bias, Kurpicz-Briki & Leoni (2021) and Huang et al. (2020)
also identified biases related to people’s origin and speakers’ age. It is
essential to remember that the mentioned biases can be vehicles for misogynous
and hateful discourse (El Abassi & Nisioi, 2020; Attanasio et al., 2022;
Merenda et al., 2018).

Bias studies for Dutch mostly consider binary gender bias. Chávez Mulsa &
Spanakis (2020) investigate gender bias in Dutch static and contextualized
word embeddings by creating Dutch versions of the Word/Sentence Embedding
Association Test (WEAT/SEAT) (Caliskan et al., 2017; May et al., 2019). WEAT
measures bias in word embeddings and can be limited in scope, in addition to
having sensitivity to seed words. McCurdy & Serbetci (2020) perform a similar
evaluation in a multilingual setup to compare the effect of grammatical gender
saliency across languages. Several works use different NLP techniques to
evaluate bias in corpora of Dutch news articles (Wevers, 2019; Kroon et al.,
2020; Kroon & van der Meer, 2021; Fokkens et al., 2018) and literary texts
(Koolen & van Cranenburgh, 2017).

In Kurpicz-Briki (2020), bias is measured with regards to place of origin and
gender in German word embeddings using WEAT (Caliskan et al., 2017). In
Kurpicz-Briki & Leoni (2021), an automatic bias detection method (BiasWords)
is presented, through which new biased word sets can be identified by
exploring the vector space around the well-known word sets that show bias. In
the template-based study of Cho et al. (2021), on gender bias in translations,
the accuracy of gender inference was measured for multiple languages including
German. It was found that, particularly for German, the inference accuracy and
disparate impact were lower for female than male, implying that certain
translations were wrongly performed for cases that required female inference.
Since German is a grammatically gendered, morphologically rich language, Gonen
& Goldberg (2019) found that debiasing methods of Bolukbasi et al. (2016) were
ineffective on german word embeddings.

For Swedish, the main focus of bias research appears to be on gender. Sahlgren
& Olsson (2019) show with their experiments that gender bias is present in
pretrained Swedish language models. Katsarou et al. (2022) and Precenth (2019)
found that the male gender tends to be associated with higher-status
professions. A study with data from mainstream news corpora by Devinney et al.
(2020) shows that women are associated with concepts like family,
communication and relationships.

##  3 Bipol

The bipol metric uses a two-step mechanism for estimating bias in text data:
binary classification and sensitive term evaluation using lexica. It has
maximum and minimum values of 1 and 0, respectively. Bipol is expressed in
Equations 1b and 1c from the main Equation 1a, where $b_{c}$ is the
classification component and $b_{s}$ is the sensitive term evaluation
component. Details about the metric are discussed in Alkhaled et al. (2023).

$\mathit{b}=\begin{cases}\mathit{b_{c}}.\mathit{b_{s}},&\text{if $b_{s}>0$}\\\
\mathit{b_{c}},&\text{otherwise}\end{cases}$ (1a)

$\mathit{b_{c}}=\frac{tp+fp}{tp+fp+tn+fn}$ (1b)

$\mathit{b_{s}}=\frac{1}{r}\sum_{t=1}^{r}{\left(\frac{1}{q}\sum_{x=1}^{q}{\left(\frac{|\sum_{s=1}^{n}a_{s}-\sum_{s=1}^{m}c_{s}|}{\sum_{s=1}^{p}d_{s}}\right)}_{x}\right)}_{t}$
(1c)

In step 1, a trained model is used to classify all the samples. The ratio of
the biased samples to the total samples predicted is determined. The tp, fp,
tn, and fn are values of the true positives, false positives, true negatives,
and false negatives, respectively. Since there’s hardly a perfect classifier,
the positive error rate is usually reported. False positives are known to
exist in similar classification systems like spam detection and automatic hate
speech detection (Heron, 2009; Feng et al., 2018; Adewumi et al., 2023b).

Step 2 is similar to term frequency-inverse document frequency (TF-IDF) in
that it is based on term frequency (Salton & Buckley, 1988; Ramos et al.,
2003), Biased samples from step 1 are evaluated token-wise along all possible
bias axes, using all the lexica of sensitive terms. An axis is a domain such
as gender or race. Tables 2 and 3 provide the lexica sizes. For English and
Swedish, we use the same lexica released by Alkhaled et al. (2023) and Adewumi
et al. (2023c), respectively. For the other 3 languages, we create new lexica
of terms (e.g. she & her) associated with specific gender or stereotypes from
public sources.444fluentu.com/blog/italian/italian-nouns,
en.wiktionary.org/wiki/Category:Italian_offensive_terms, Dutch_profanity,
Category:German_ethnic_slurs Some of the terms in the lexica were selected
from the sources based on the topmost available. These may also be expanded as
needed, since bias terms are known to evolve (Haemmerlie & Montgomery, 1991;
Antoniak & Mimno, 2021). The non-English lexica are small because fewer terms
are usually available in other languages compared to the high-resource English
language and we use the same size across the languages to be able to compare
performance somewhat.

Equation 1c first finds the absolute difference between the two maximum summed
frequencies in the types of an axis
($|\sum_{s=1}^{n}a_{s}-\sum_{s=1}^{m}c_{s}|$), where n and m are the total
terms in a sentence along an axis. For example, in the sentence ´Women!!!
PERSON taught you better than that. Shame on you!’, female terms = 1 while
male terms = 0. This is then divided by the summed frequencies of all the
terms ($d_{s}$) in that axis ($\sum_{s=1}^{p}d_{s}$). The operation is
performed for all axes ($q$) and the average taken
($\frac{1}{q}\sum_{x=1}^{q}$). It is performed for all the biased samples
($r$) and the average taken ($\frac{1}{r}\sum_{t=1}^{r}$ ).

Axis | Type 1 | Type 2 | Type 3
---|---|---|---
Racial | 84 (black) | 127 (white) |
Gender | 76 (female) | 46 (male) |
Religious | 180 (christian) | 465 (muslim) | 179 (hindu)
Table 2:  English lexica sizes (Alkhaled et al., 2023). An example of racial-black is negro.  Axis | Type 1 | Type 2
---|---|---
Gender | 17 (female) | 17 (male)
Racial | 10 (black) | 10 (white)
Table 3:  Italian, Dutch, German and Swedish Lexica sizes. An example of
gender-female is woman.

##  4 Methodology

###  4.1 Datasets

####  4.1.1 MAB

comment_text |  label |  old_id |  id
---|---|---|---
A fitness ranking that drops each month as more and more fat people move to Colorado. |  biased |  6231681 |  167
Een fitness ranking die elke maand daalt als meer en meer dikke mensen verhuizen naar Colorado. |  |  |
Ein Fitness-Ranking, das jeden Monat fällt, wie mehr und mehr fette Menschen nach Colorado zu bewegen. |  |  |
Un ranking di fitness che scende ogni mese come sempre più persone grasse si spostano in Colorado. |  |  |
En fitness ranking som sjunker varje månad som fler och fler feta människor flyttar till Colorado. |  |  |
|  |  |
Women ! ! ! PERSON taught you better than that. Shame on you! |  biased |  6233025 |  628
Vrouwen ! ! ! PERSOON heeft je beter geleerd dan dat. Je moet je schamen! |  |  |
Frauen!!!! PERSON lehrte Sie besser als das. Schande über Sie! |  |  |
Donne ! ! ! Person ti ha insegnato meglio di così, vergognati! |  |  |
Kvinnor ! ! !- Han lärde dig bättre än så. Skäms på dig! |  |  |
|  |  |
"The article itself touches on the topic of abortion, so absolutely it’s appropriate to discuss it." |  unbiased |  451965 |  1486981
"Het artikel zelf raakt het onderwerp abortus, dus het is absoluut gepast om het te bespreken." |  |  |
"Der Artikel selbst berührt das Thema Abtreibung, also ist es absolut angebracht, es zu diskutieren." |  |  |
"L’articolo stesso tocca il tema dell’aborto, quindi è assolutamente opportuno discuterne." |  |  |
"Artikeln själv berör ämnet abort, så absolut är det lämpligt att diskutera det." |  |  |
Table 4:  English, Dutch, German, Italian, and Swedish examples from the MAB dataset. "PERSON" is the anonymization of a piece of personal identifiable information (PII)in the dataset.  Set | Biased | Unbiased | Total
---|---|---|---
Training | 533,544 | 1,209,433 | 1,742,977
Validation | 32,338 | 69,649 | 101,987
Test | 33,470 | 68,541 | 102,011
| 599,352 | 1,347,623 | 1,946,975
Table 5:  MAB dataset split

The Italian, Dutch and German datasets were machine-translated from MAB 555The
reference provides details of the annotation of the base data. (Alkhaled et
al., 2023) with the Helsinki-NLP model (Tiedemann & Thottingal, 2020). Each
translation took about 48 hours on one GPU. Examples from the data are
provided in Table 4. Table 5 provides statistics about the datasets. For
quality control (QC), we verified translation by back-translating some random
samples using Google NMT. Personal identifiable information (PII) was removed
from the MAB dataset using the spaCy library. The 3 datasets are used to train
new bias classifiers. We also train on the original English and the Swedish.

##### Machine-Translation issues:

Culture-specific biases may not be represented in the MAB versions for the
translated languages because the original dataset is in English. This is a
limitation. However, bias is also a universal concern, such that there are
examples that span across cultures. For instance, examples 1 and 2 in Table 4
are of universal concern because individuals with non-conforming bodies and
women should be respected, regardless of culture or nationality. Hence, the
MAB versions are relevant for bias detection, though they were translated.

####  4.1.2 Evaluation datasets

Ten datasets are evaluated for bias in this work. All are automatically
preprocessed before evaluation, the same way the training data were
preprocessed. This includes removal of IP addresses, emojis, URLs, special
characters, emails, extra spaces, numbers, empty text rows, and duplicate
rows. All texts are then lowercased.

We selected datasets that are available on the HuggingFace (Wolf et al., 2020)
Datasets. We evaluated the first 1,000 samples of each training split due to
resource constraints. For English, we evaluated the sentence column of Corpus
of Linguistic Acceptability (CoLA)(Warstadt et al., 2019), the sentence column
of Question-Answering Natural Language Inference (QNLI)(Wang et al., 2018),
the sentennce1 column of Microsoft Research Paraphrase Corpus (MRPC)(Dolan &
Brockett, 2005), the premise column of Multi-Genre Natural Language Inference
(MNLI)(Williams et al., 2018), the premise column of the CommitmentBank
(CB)dataset (De Marneffe et al., 2019), and the passage column of Reading
Comprehension with Commonsense Reasoning Dataset (ReCoRD)(Zhang et al., 2018).
For Italian, we evaluated the context column of the Stanford Question
Answering Dataset (SQuAD)(Croce et al., 2018; Rajpurkar et al., 2016); for
Dutch, the sentence1 column of the Semantic Textual Similarity Benchmark
(STSB)(Cer et al., 2017); for German, the text column of the German News
Articles Datasets 10k (GNAD10)(Schabus et al., 2017); for Swedish, the premise
of the CB.

###  4.2 Annotation for the assumption confirmation

To verify the assumption that toxic comments contain bias, we randomly
selected 200 samples from the training set of MAB-English for annotation on
Slack, an online platform. The selection of 200 samples is based on an error
margin of 7% and a confidence level of 95%. To ensure high-quality annotation,
we use established techniques for this task: 1) the use of gold (30) samples,
2) multiple (i.e. 3) annotators, and 3) minimum qualification of undergraduate
study for annotators. Each annotator was paid 25 U.S. dollars and the it took
about 2 hours to complete the annotation on average. We mixed the 30 gold
samples with the 200, to verify the annotation quality of each annotator, as
they were required to get, at least, 16 correctly for their annotation to be
accepted. The 30 gold samples are samples with unanimous agreement in the
original Jigsaw or SBICv2 data, which make up the MAB. We provide inter-
annotator agreement (IAA)using Jaccard similarity coefficient (intersection
over union) and credibility unanimous score (CUS)(Adewumi et al., 2023a)
(intersection over sample size).

###  4.3 Experiments

We selected two state-of-the-art (SotA)pre-trained, multilingual models for
experiments to compare their macro F1 performance: mT5-small and mBERT-base.
These are from the HuggingFace hub. We further report the mT5 positive error
rate of predictions. The mT5-small has 300 million parameters (Xue et al.,
2021) while mBERT-Base has 110 million parameters. We trained only on the MAB
datasets and evaluated using only the mT5 model, the better model of the 2, as
will be observed in Section 5. For the CB and ReCoRD datasets, we evaluate all
samples since they contain only about 250 and 620 entries, respectively. We
used wandb (Biewald, 2020) for hyper-parameter exploration, based on Bayesian
optimization. For mT5, we set the maximum and minimum learning rates as 5e-5
and 2e-5 while the maximum and minimum epochs are 20 and 4, respectively. One
epoch is equivalent to the ratio of the total number of samples to the batch
size (i.e. the steps). We used a batch size of 8 because higher numbers easily
resulted in memory challenges.

For mBERT, we set the learning rates and epochs as with mT5. However, we
explore over batch sizes of 8, 16 and 32. For both models, we set the maximum
input sequence length to 512. Training took, on average, about 7.3 hours per
language per epoch for mBERT while it was 6 hours for mT5. For all the
experiments, we limit the run counts to 2 per language because of the long
training time each takes on average. The average scores of the results are
reported. The saved models with the lowest losses were used to evaluate the
datasets. All the experiments were performed on two shared Nvidia DGX-1
machines that run Ubuntu 20.04 and 18.04. One machine has 8 x 40GB A100 GPUs
while the other has 8 x 32GB V100 GPUs.

The lexica baseline, compared in experiments, is similar to the equation of
the second step in bipol. It does not consider bias semantically and uses term
frequencies, similarly to TF-IDF. It uses the same lexica as bipol. Its
maximum and minimum values are 1 and 0, respectively.

##  5 Results and Discussion

From Table 6, we observe that all mT5 results are better than those of mBERT
across the languages. The two-sample t-test of the difference of means between
all the corresponding mT5 and mBERT scores have p values $<$ 0.0001 for alpha
of 0.05, showing the results are statistically significant. It appears better
hyper-parameter search may be required for the mBERT model to converge and
achieve better performance. The best macro F1 result is for English mT5 at
0.787. This is not surprising, as English has the largest amount of training
data for the pre-trained mT5 model (Xue et al., 2021). This occurred at the
learning rate of 2.9e-5 and step 1,068,041.

|  macro F1 $\uparrow$ (s.d.) |  mT5 error $\downarrow$
---|---|---
MAB version | mBERT | mT5 |  fp/(fp +tp)
English | 0.418 (0.01) | 0.787 (0) | 0.261
Italian | 0.429 (0) | 0.768 (0) | 0.283
Dutch | 0.419 (0.01) | 0.768 (0) | 0.269
German | 0.418 (0.01) | 0.769 (0) | 0.261
Swedish | 0.418 (0.01) | 0.768 (0) | 0.274
Table 6:  Average F1 scores on the validation sets.  |  bipol scores $\downarrow$ (s.d.) |
---|---|---
English | $\mathit{b_{c}}$ | $\mathit{b_{s}}$ | bipol (b) |  baseline $\downarrow$
CB | 0.096 | 0.875 | 0.084 (0) | 0.88
CoLA | 0.101 | 0.943 | 0.095 (0) | 0.958
ReCoRD | 0.094 | 0.852 | 0.025 (0) | 0.829
MRPC | 0.048 | 0.944 | 0.045 (0) | 0.957
MNLI | 0.063 | 0.833 | 0.053 (0) | 0.965
QNLI | 0.03 | 0.933 | 0.028 (0) | 0.945
Italian |  |  |  |
SQuAD | 0.014 | 0 | 0.014 (0) | 0.989
Dutch |  |  |  |
STSB | 0.435 | 0.992 | 0.432 (0) | 0.987
German |  |  |  |
GNAD10 | 0.049 | 0.502 | 0.025 (0) | 1
Swedish |  |  |  |
CB | 0.08 | 0.938 | 0.075 (0) | 0.97
Table 7: Average bipol & lexica baseline scores.

Figures 1 and 2 depict the validation sets macro F1 and loss line graphs for
the 2 runs for the 5 languages, respectively. From Table 7, we observe that
all the evaluated datasets have biases, though seemingly little (but
important) when compared to the maximum of 1. We say important because many of
the datasets contain small number of samples yet they can be detected.
Furthermore, a low value does not necessarily diminish the weight of the
effect of bias in society or the data but we leave the discussion about what
amount should be tolerated open for the NLP community. Our recommendation is
to have a bias score as close to zero as possible. On the other hand, the
lexica baseline appears overly confident of much more bias, which is incorrect
because the method fails to exclude unbiased text in its evaluation, which is
a shortcoming of methods based solely on it. The Dutch STSB is higher than the
other bipol scores because of the higher bipol classifier component score of
0.435, which may be because of the nature of the dataset.

Figure 1: Macro F1 of the validation set for the 5 languages, as generated by
wandb. Figure 2: Loss on the validation set for the 5 languages, as generated
by wandb. Figure 3: Error matrix of mT5 on MAB-English

###  5.1 Error analysis & qualitative results

According to the error matrix in Figure 3, the mT5 model is better at
correctly predicting unbiased samples. This is because of the higher unbiased
samples in the training data of MAB. In Table 8, the first example for the
English CB contains a stereotypical statement "men are naturally right and it
is the role of women to follow their lead", leading to the correct biased
prediction by the model. Similarly, this correct prediction is made in the
Swedish CB. We notice over-generalization (May et al., 2019; Nadeem et al.,
2021) in the correct examples for the CoLA predictions, where "every" is used.
The table also shows some incorrect predictions.

comment_text |  prediction |  csv id
---|---|---
Correct predictions from the English CB
nicky approached her with the assumption that men are naturally right and it is the role of women to follow their lead. constance whose confidence was growing daily was not prepared to give in to nickys wishes merely because of his sex. if she felt he was right then she agreed with him. |  biased |  85
b: thats true. a: so. b: uh the other argument is that the death penalty is a deterrent and i really dont uh agree with that. i dont think anyone who would commit uh a crime that would get them the death penalty would stop at the moment and say well i was about to kill and dismember this person but oh if they catch me theyre going to kill me so i better not do it. i just dont think uh that it works that way. a: yeah. i dont think its done. |  biased |  133
Incorrect prediction
b: yeah and the ground will filter some of it but not all of it. a: no not when you figure i didnt realize one cow produces that much manure |  biased |  137
|  |
Correct predictions from the English CoLA
if you give him enough opportunity every senator will succumb to corruption. |  biased |  266
every senator becomes more corrupt the more lobbyists he talks to. |  biased |  277
Incorrect prediction
bill squeezed the puppet through the hole. |  biased |  82
|  |
Table 8:  Qualitative examples of apparently correct and incorrect predictions
in some of the English datasets, based on the mT5 model.

###  5.2 Consistent prediction with perturbation

An interesting property of relative consistency that we observed with the
model predictions, as demonstrated with the CoLA dataset, is that when
sentences are perturbed, the model mostly maintains its predictions, as long
as the grounds for prediction (in this case - over-generalization) remain the
same. The perturbations are inherent in the CoLA dataset itself, as the
dataset is designed that way. Some examples are provided in Table 9, where 6
out of 8 are correctly predicted. This property is repeated consistently in
other examples not shown here.

comment_text |  prediction |  csv id
---|---|---
if you give him enough opportunity every senator will succumb to corruption. |  biased |  266
you give him enough opportunity and every senator will succumb to corruption. |  biased |  267
we gave him enough opportunity and sure enough every senator succumbed to corruption. |  unbiased |  268
if you give any senator enough opportunity he will succumb to corruption. |  biased |  269
you give any senator enough opportunity and he will succumb to corruption. |  biased |  270
you give every senator enough opportunity and he will succumb to corruption. |  biased |  271
we gave any senator enough opportunity and sure enough he succumbed to corruption. |  biased |  272
we gave every senator enough opportunity and sure enough he succumbed to corruption. |  unbiased |  273
Table 9:  Mostly consistent correct prediction with perturbation in the CoLA
dataset.

###  5.3 Explainability by graphs

We show explainability by visualization using graphs. Bipol produces a
dictionary of lists for every evaluation and we represent 2 in top-5 frequent
terms bar graph. Many of the datasets display overall male bias. As examples,
figures 4 and 5 show that the GNAD10 and STSB datasets have overall male and
female bias, respectively.

Figure 4: Top 5 frequent terms in the GNAD10 dataset (paired terms are only
for comparison). Figure 5: Top 5 frequent terms in the STSB dataset (paired
terms are only for comparison).

###  5.4 Assumption confirmation through annotation

The results of the annotation of the 200 MAB samples reveal that toxic
comments do contain bias. This is shown in Figure 6. The Jaccard similarity
coefficient and CUS of IAA are 0.261666Not to be interpreted using Kappa for 2
annotators on 2 classes. Ours involved 3 annotators and 0.515, respectively,
given that over 50% is the intersection of unanimous decision.

Figure 6: Annotation confirms assumption about toxic comments.

##  6 Conclusion

The findings of this work show that bias besets Natural Language Processing
(NLP)datasets regardless of language, including benchmark datasets on the
GLUE/SuperGLUE leaderboards. We introduced MAB datasets in 3 languages for
training models in bias detection. Each has about 2 million labeled samples.
We also contribute lexica of bias terms for the languages. In addition, we
verified the assumption that toxic comments contain bias. It may be impossible
to completely remove bias from data or models, since they reflect the real
world, but resources for estimating bias can provide insight into mitigation
strategies for reducing bias. Future work may explore ways of minimising false
positives in classifiers to make them more effective. One may also explore how
this work scales to other languages or how multilignual models compare to
language-specific monolingual models or large language models (LLM s).
Regarding culture-specific biases in datasets, one solution will be to collect
data from the specific cultures/nationalities to capture these biases.

## Ethics statement & limitation

The authors took care while providing examples of data samples, despite some
containing stereotypes or toxic content. The classifiers trained for
estimating the biases in this work are limited in effectiveness, as shown in
the results, hence a result of 0 on any dataset does not necessarily indicate
a bias-free dataset. The original MAB was annotated by humans who may have
their personal biases based on cultural or demographic backgrounds. This means
the final annotations may not be seen as absolute ground truth of social
biases.

#### Acknowledgments

This work was partially supported by the Wallenberg AI, Autonomous Systems and
Software Program (WASP), funded by the Knut and Alice Wallenberg Foundation
and counterpart funding from Luleå University of Technology (LTU). The authors
also thank Björn Backe for his insightful comments during the writing of this
paper.

## References

  * Adewumi et al. (2023a) Tosin Adewumi, Mofetoluwa Adeyemi, Aremu Anuoluwapo, Bukola Peters, Happy Buzaaba, Oyerinde Samuel, Amina Mardiyyah Rufai, Benjamin Ajibade, Tajudeen Gwadabe, Mory Moussou Koulibaly Traore, Tunde Oluwaseyi Ajayi, Shamsuddeen Muhammad, Ahmed Baruwa, Paul Owoicho, Tolulope Ogunremi, Phylis Ngigi, Orevaoghene Ahia, Ruqayya Nasir, Foteini Liwicki, and Marcus Liwicki.  Afriwoz: Corpus for exploiting cross-lingual transfer for dialogue generation in low-resource, african languages.  In _2023 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–8, 2023a.  doi: 10.1109/IJCNN54540.2023.10191208.
  * Adewumi et al. (2023b) Tosin Adewumi, Sana Sabah Sabry, Nosheen Abid, Foteini Liwicki, and Marcus Liwicki.  T5 for hate speech, augmented data, and ensemble.  _Sci_ , 5(4), 2023b.  ISSN 2413-4155.  doi: 10.3390/sci5040037.  URL https://www.mdpi.com/2413-4155/5/4/37.
  * Adewumi et al. (2023c) Tosin Adewumi, Isabella Södergren, Lama Alkhaled, Sana Sabah Sabry, Foteini Liwicki, and Marcus Liwicki.  Bipol: Multi-axes evaluation of bias with explainability in benchmark datasets.  In _Proceedings of Recent Advances in Natural Language Processing (RANLP)_ , Varna, Bulgaria, 2023c.
  * Alkhaled et al. (2023) Lama Alkhaled, Tosin Adewumi, and Sana Sabah Sabry.  Bipol: A novel multi-axes bias evaluation metric with explainability for nlp.  _Natural Language Processing Journal_ , 4:100030, 2023.  ISSN 2949-7191.  doi: https://doi.org/10.1016/j.nlp.2023.100030.  URL https://www.sciencedirect.com/science/article/pii/S2949719123000274.
  * Antoniak & Mimno (2021) Maria Antoniak and David Mimno.  Bad seeds: Evaluating lexical methods for bias measurement.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1889–1904, 2021.
  * Attanasio et al. (2022) Giuseppe Attanasio, Debora Nozza, Dirk Hovy, and Elena Baralis.  Entropy-based attention regularization frees unintended bias mitigation from lists.  _arXiv preprint arXiv:2203.09192_ , 2022.
  * Bellamy et al. (2018) Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang.  Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias, 2018.
  * Bentivogli et al. (2020) Luisa Bentivogli, Beatrice Savoldi, Matteo Negri, Mattia A. Di Gangi, Roldano Cattoni, and Marco Turchi.  Gender in danger? evaluating speech translation technology on the MuST-SHE corpus.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 6923–6933, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.619.  URL https://aclanthology.org/2020.acl-main.619.
  * Biewald (2020) Lukas Biewald.  Experiment tracking with weights and biases, 2020.  URL https://www.wandb.com/.  Software available from wandb.com.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  _Advances in neural information processing systems_ , 29, 2016.
  * Brownstein (2019) Michael Brownstein.  Implicit Bias.  In Edward N. Zalta (ed.), _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, Fall 2019 edition, 2019.
  * Caliskan et al. (2017) Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ , 356(6334):183–186, 2017.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pp. 1–14, Vancouver, Canada, August 2017. Association for Computational Linguistics.  doi: 10.18653/v1/S17-2001.  URL https://aclanthology.org/S17-2001.
  * Chávez Mulsa & Spanakis (2020) Rodrigo Alejandro Chávez Mulsa and Gerasimos Spanakis.  Evaluating bias in Dutch word embeddings.  In _Proceedings of the Second Workshop on Gender Bias in Natural Language Processing_ , pp. 56–71, Barcelona, Spain (Online), December 2020. Association for Computational Linguistics.  URL https://aclanthology.org/2020.gebnlp-1.6.
  * Cho et al. (2021) Won Ik Cho, Jiwon Kim, Jaeyeong Yang, and Nam Soo Kim.  Towards cross-lingual generalization of translation gender bias.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pp. 449–457, 2021.
  * Croce et al. (2018) Danilo Croce, Alexandra Zelenanska, and Roberto Basili.  Neural learning for question answering in italian.  In Chiara Ghidini, Bernardo Magnini, Andrea Passerini, and Paolo Traverso (eds.), _AI*IA 2018 – Advances in Artificial Intelligence_ , pp. 389–402, Cham, 2018. Springer International Publishing.  ISBN 978-3-030-03840-3.
  * De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _proceedings of Sinn und Bedeutung_ , volume 23, pp. 107–124, 2019.
  * Devinney et al. (2020) Hannah Devinney, 1974 Björklund, Jenny, and Henrik Björklund.  Semi-supervised topic modeling for gender bias discovery in english and swedish.  _EQUITBL Proceedings of the Second Workshop on Gender Bias in Natural Language Processing_ , pp. 79 – 92, 2020.  URL https://proxy.lib.ltu.se/login?url=https://search.ebscohost.com/login.aspx?direct=true&db=edsswe&AN=edsswe.oai.DiVA.org.uu.430350&lang=sv&site=eds-live&scope=site.
  * Dolan & Brockett (2005) William B. Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_ , 2005.  URL https://aclanthology.org/I05-5002.
  * El Abassi & Nisioi (2020) Samer El Abassi and Sergiu Nisioi.  Mdd@ ami: Vanilla classifiers for misogyny identification.  _EVALITA Evaluation of NLP and Speech Tools for Italian-December 17th, 2020_ , pp. 55, 2020.
  * Feng et al. (2018) Bo Feng, Qiang Fu, Mianxiong Dong, Dong Guo, and Qiang Li.  Multistage and elastic spam detection in mobile social networks through deep learning.  _IEEE Network_ , 32(4):15–21, 2018.
  * Fokkens et al. (2018) Antske Fokkens, Nel Ruigrok, Camiel Beukeboom, Gagestein Sarah, and Wouter van Atteveldt.  Studying muslim stereotyping through microportrait extraction.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan, May 2018. European Language Resources Association (ELRA).  URL https://aclanthology.org/L18-1590.
  * Gaido et al. (2021) Marco Gaido, Beatrice Savoldi, Luisa Bentivogli, Matteo Negri, and Marco Turchi.  How to split: the effect of word segmentation on gender bias in speech translation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pp. 3576–3589, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-acl.313.  URL https://aclanthology.org/2021.findings-acl.313.
  * Gonen & Goldberg (2019) Hila Gonen and Yoav Goldberg.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  _arXiv preprint arXiv:1903.03862_ , 2019.
  * Haemmerlie & Montgomery (1991) Frances M Haemmerlie and Robert L Montgomery.  Goldberg revisited: Pro-female evaluation bias and changed attitudes toward women by engineering students.  _Journal of Social Behavior and Personality_ , 6(2):179, 1991.
  * Heron (2009) Simon Heron.  Technologies for spam detection.  _Network Security_ , 2009(1):11–15, 2009.
  * Huang et al. (2020) Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and Michael J Paul.  Multilingual twitter corpus and baselines for evaluating demographic bias in hate speech recognition.  _arXiv preprint arXiv:2002.10361_ , 2020.
  * Katsarou et al. (2022) Styliani Katsarou, Borja Rodríguez-Gálvez, and Jesse Shanahan.  Measuring gender bias in contextualized embeddings.  _Computer Sciences and Mathematics Forum_ , 3(1), 2022.  ISSN 2813-0324.  doi: 10.3390/cmsf2022003003.  URL https://www.mdpi.com/2813-0324/3/1/3.
  * Klare et al. (2012) Brendan F Klare, Mark J Burge, Joshua C Klontz, Richard W Vorder Bruegge, and Anil K Jain.  Face recognition performance: Role of demographic information.  _IEEE Transactions on Information Forensics and Security_ , 7(6):1789–1801, 2012.
  * Koolen & van Cranenburgh (2017) Corina Koolen and Andreas van Cranenburgh.  These are not the stereotypes you are looking for: Bias and fairness in authorial gender attribution.  In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_ , pp. 12–22, Valencia, Spain, April 2017. Association for Computational Linguistics.  doi: 10.18653/v1/W17-1602.  URL https://aclanthology.org/W17-1602.
  * Kroon & van der Meer (2021) Anne C Kroon and Toni GLA van der Meer.  Who’s to fear? implicit sexual threat pre and post the “refugee crisis”.  _Journalism Practice_ , pp. 1–17, 2021.
  * Kroon et al. (2020) Anne C Kroon, Damian Trilling, Toni GLA van der Meer, and Jeroen GF Jonkman.  Clouded reality: News representations of culturally close and distant ethnic outgroups.  _Communications_ , 45(s1):744–764, 2020.
  * Kurpicz-Briki (2020) Mascha Kurpicz-Briki.  Cultural differences in bias? origin and gender bias in pre-trained german and french word embeddings.  2020\.
  * Kurpicz-Briki & Leoni (2021) Mascha Kurpicz-Briki and Tomaso Leoni.  A world full of stereotypes? further investigation on origin and gender bias in multi-lingual word embeddings.  _Frontiers in big Data_ , 4:20, 2021.
  * (35) Alessandra Luccioli, Silvia Bernardini, and Raffaella Baccolini.  Stereotipi di genere e traduzione automatica dall’inglese all’italiano: uno studio di caso sul femminile nelle professioni.
  * May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger.  On measuring social biases in sentence encoders.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 622–628, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1063.  URL https://aclanthology.org/N19-1063.
  * McCurdy & Serbetci (2020) Katherine McCurdy and Oguz Serbetci.  Grammatical gender associations outweigh topical gender bias in crosslinguistic word embeddings.  _arXiv preprint arXiv:2005.08864_ , 2020.
  * Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  _ACM Computing Surveys (CSUR)_ , 54(6):1–35, 2021.
  * Merenda et al. (2018) Flavio Merenda, Claudia Zaghi, Tommaso Caselli, and Malvina Nissim.  Source-driven representations for hate speech detection.  _Computational Linguistics CLiC-it 2018_ , pp. 258, 2018.
  * Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy.  StereoSet: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 5356–5371, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.416.  URL https://aclanthology.org/2021.acl-long.416.
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman.  CrowS-pairs: A challenge dataset for measuring social biases in masked language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1953–1967, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.154.  URL https://aclanthology.org/2020.emnlp-main.154.
  * Precenth (2019) Rasmus Precenth.  Word embeddings and gender stereotypes in swedish and english.  2019\.
  * Raji et al. (2020) Inioluwa Deborah Raji, Timnit Gebru, Margaret Mitchell, Joy Buolamwini, Joonseok Lee, and Emily Denton.  Saving face: Investigating the ethical concerns of facial recognition auditing.  In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’20, pp. 145–151, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450371100.  doi: 10.1145/3375627.3375820.  URL https://doi.org/10.1145/3375627.3375820.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pp. 2383–2392, Austin, Texas, November 2016. Association for Computational Linguistics.  doi: 10.18653/v1/D16-1264.  URL https://aclanthology.org/D16-1264.
  * Ramos et al. (2003) Juan Ramos et al.  Using tf-idf to determine word relevance in document queries.  In _Proceedings of the first instructional conference on machine learning_ , volume 242, pp. 29–48. Citeseer, 2003.
  * Rescigno et al. (2020) Argentina Anna Rescigno, Eva Vanmassenhove, Johanna Monti, and Andy Way.  A case study of natural gender phenomena in translation. a comparison of google translate, bing microsoft translator and deepl for english to italian, french and spanish.  In _CLiC-it_ , 2020.
  * Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme.  Gender bias in coreference resolution.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pp. 8–14, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-2002.  URL https://aclanthology.org/N18-2002.
  * Sahlgren & Olsson (2019) Magnus Sahlgren and Fredrik Olsson.  Gender bias in pretrained swedish embeddings.  In Mareike Hartmann and Barbara Plank (eds.), _Proceedings of the 22nd Nordic Conference on Computational Linguistics, NoDaLiDa 2019, Turku, Finland, September 30 - October 2, 2019_ , pp. 35–43. Linköping University Electronic Press, 2019.  ISBN 978-91-7929-995-8.  URL https://aclweb.org/anthology/W19-6104/.
  * Salton & Buckley (1988) Gerard Salton and Christopher Buckley.  Term-weighting approaches in automatic text retrieval.  _Information processing & management_, 24(5):513–523, 1988.
  * Sap et al. (2020) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi.  Social bias frames: Reasoning about social and power implications of language.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 5477–5490, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.486.  URL https://www.aclweb.org/anthology/2020.acl-main.486.
  * Schabus et al. (2017) Dietmar Schabus, Marcin Skowron, and Martin Trapp.  One million posts: A data set of german online discussions.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_ , pp. 1241–1244, Tokyo, Japan, August 2017.  doi: 10.1145/3077136.3080711.
  * Sengupta et al. (2021) Kinshuk Sengupta, Rana Maher, Declan Groves, and Chantal Olieman.  Genbit: measure and mitigate gender bias in language datasets.  _Microsoft Journal of Applied Research_ , 16:63–71, October 2021.  URL https://www.microsoft.com/en-us/research/publication/genbit-measure-and-mitigate-gender-bias-in-language-datasets/.
  * Tiedemann & Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal.  OPUS-MT — Building open translation services for the World.  In _Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT)_ , Lisbon, Portugal, 2020.
  * Vanmassenhove & Monti (2021) Eva Vanmassenhove and Johanna Monti.  gender-it: An annotated english-italian parallel challenge set for cross-linguistic natural gender phenomena.  _arXiv preprint arXiv:2108.02854_ , 2021.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pp. 353–355, Brussels, Belgium, November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-5446.  URL https://aclanthology.org/W18-5446.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file4496bf24afe7fab6f046bf4923da8de6-Paper.pdf.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641, 2019.
  * Wevers (2019) Melvin Wevers.  Using word embeddings to examine gender bias in dutch newspapers, 1950-1990.  _arXiv preprint arXiv:1907.08922_ , 2019.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pp. 1112–1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1101.  URL https://aclanthology.org/N18-1101.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pp. 38–45, Online, October 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-demos.6.  URL https://aclanthology.org/2020.emnlp-demos.6.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 483–498, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.41.  URL https://aclanthology.org/2021.naacl-main.41.
  * Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.  Record: Bridging the gap between human and machine commonsense reading comprehension.  _arXiv preprint arXiv:1810.12885_ , 2018.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pp. 15–20, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-2003.  URL https://aclanthology.org/N18-2003.