11institutetext: Department of Statistics, LMU Munich, Germany
11email<EMAIL_ADDRESS>
11email: {chris,esteban.garcesarias,marius.roger,bernd.bischl,matthias}
@stat.uni-muenchen.de 22institutetext: Smart Data Analytics (SDA), University
of Bonn, Germany 22email<EMAIL_ADDRESS>33institutetext:
Munich Center for Machine Learning (MCML), LMU Munich, Germany

# How Different Is Stereotypical Bias Across Languages?

Ibrahim Tolga Öztürk 11 Rostislav Nedelchev 22 Christian Heumann 11 Esteban
Garces Arias 11 Marius Roger 11 Bernd Bischl 1133 Matthias Aßenmacher 1133

###### Abstract

Recent studies have demonstrated how to assess the stereotypical bias in pre-
trained English language models. In this work, we extend this branch of
research in multiple different dimensions by systematically investigating (a)
mono- and multilingual models of (b) different underlying architectures with
respect to their bias in (c) multiple different languages. To that end, we
make use of the English StereoSet data set [17], which we semi-automatically
translate into German, French, Spanish, and Turkish. We find that it is of
major importance to conduct this type of analysis in a multilingual setting,
as our experiments show a much more nuanced picture as well as notable
differences from the English-only analysis. The main takeaways from our
analysis are that mGPT-2 (partly) shows surprising anti-stereotypical behavior
across languages, English (monolingual) models exhibit the strongest bias, and
the stereotypes reflected in the data set are least present in Turkish models.
Finally, we release our codebase alongside the translated data sets and
practical guidelines for the semi-automatic translation to encourage a further
extension of our work to other languages.

###### Keywords:

Stereotypes Bias Fairness Natural Language Processing Pre-Trained Language
Models Transformer Benchmarking

##  1 Introduction

Stereotypical bias in pre-trained language models (PLMs) has been an actively
researched topic in contemporary natural language processing, with the concept
of gender likely being the most prominent one among the examined demographic
biases [7, 8, 6, 12]. Since PLMs primarily learn from the data gathered from
pages and websites open to and created by the public, they also inevitably
memorize the stereotypes111A generalized belief about a particular category of
people [4]. present in this data. On one hand, it is infeasible to inspect
individual entries one-by-one in a data set to ensure it does not possess any
stereotypes, due to typically large data set sizes; on the other hand, the
data set cannot be considerably downsized, as this would limit the performance
of the machine learning model. Stereotypical decisions driven by predictions
derived from deep learning models can render companies or engineers to be
liable for the stereotypical bias. Hence, the likelihood of producing
stereotypical outputs must be minimized, and before that, a generic
methodology to measure and evaluate the stereotypical bias in the models is
essential. To this day, various approaches for stereotypical bias measurement
exist in the literature. An inspired approach to measure stereotypical bias in
the pre-trained language models was proposed by Nadeem et al. [17], where an
English data set and a methodology to measure the stereotypical bias in
English language models was constructed. However, this methodology is
significantly limited, as it supports only one language, whereas the current
state-of-the-art multilingual models support more than 90 languages [10].

##### Contribution

In this work, we evaluate the stereotypical bias in mono- and multilingual
models by creating new data sets via semi-automated translation of the
StereoSet data [17] to four different languages. This enables us to draw
comparisons across multiple different dimensions and obtain a more nuanced
picture. We determine to which extent pre-trained language models exhibit
stereotypical biases by carefully considering multiple different combinations:
We 1) examine both mono- and multilingual models, while 2) considering the
different commonly used transformer architectures (encoder, decoder, encoder-
decoder) and 3) perform our experiments for languages of different families
(Indo-European vs. Ural-Altaic). In a series of experiments, we extend the
code222https://github.com/moinnadeem/StereoSet published by Nadeem et al. [17]
to a more generic version allowing for easier application to other languages
and models. Additionally, we noticed and corrected some inconsistencies in
this code, which we will further discuss in Section 3.4. We publish our
codebase 333https://github.com/slds-lmu/stereotypes-multi to nurture further
research with respect to stereotypical bias.

##  2 Related work

Detecting and mitigating bias and stereotypes in PLMs represents an active and
relevant research field, especially since these stereotypes might actually
lead to negative real-world consequences for humans. Thus, it has become
common practice, to at least try to measure biases and stereotypes when pre-
training a new model. The word embedding association tests (WEAT [2]) is one
important example, showing that European-American names have more positive
valence than African-American names in state-of-the-art sentiment analysis
applications. Caliskan et al. [3] claim that this issue pertains to a much
broader context than having intentional bias among different groups of people,
as it is more challenging to analyze the underlying reasons for this behavior.
Nadeem et al. [17] measure the stereotypical bias (for the English language)
by creating their own data set, with WEAT being the inspiration for their so-
called Context Association Test (CAT). Although this (as well as most other)
work is conducted on English PLMs, there is also a notable amount of research
on multi-lingual models. For instance, Stanovsky et al. [25] conduct an
experiment on the comparison of gender bias in some of the widely used
translation services. They discovered that Amazon Translate performs second
best in the German language among the chosen systems. Moreover, three out of
four systems attain the most satisfactory performance for German among eight
different languages. A rationale for that might be German’s similarity to the
English source language. Lauscher and Glavas [13] measure different types of
cross-lingual biases in seven languages from various language families. They
come to the unanticipated finding that the Wikipedia corpus is more biased
than a corpus of tweets. Further, their results indicate that FastText is the
most biased method among the four examined embedding models. Névéol et al.
[19] extend the CrowS-Pairs data set [18] to the French language and measure
the bias while providing the possibility to extend to different languages.

Other than that, there is also work on the sources of bias and on mitigation
(i.e., debiasing). Mehrabi et al. [16] divide the sources of bias into two
categories: originating from the data and originating from the model. The
behavior of a model overly focusing on data-related biases is called bias
amplification [29]. Hall et al. [11] report a correlation between the strength
of bias amplification and measures such as accuracy, model capacity, or model
overconfidence. This also implies that this issue is more substantial when
recognizing group membership (e.g., gender) is easier than class membership
(e.g., positive). Besides introducing WEAT, Bolukbasi et al. [2] also propose
debiasing techniques. Bartl et al. [1] apply counterfactual data substitution
to the GAP corpus [27] and fine-tune BERT [9] to mitigate gender bias,
achieving promising results for English. However, the same method yielded
unsatisfactory performance for German – possibly due to grammar, since German
is a gender-marking language, in contrast to English. This shows once more
that bias detection and mitigation depend on the language, stressing the
importance of our work. Going beyond gender, Meade et al. [15] also apply
debiasing techniques for racial and religious biases.

##  3 Materials and Methods

###  3.1 StereoSet data

The StereoSet data set, created by Nadeem et al. [17], is designed to have two
association tests (intra- and inter-sentence) for the evaluation of pre-
trained models. For the intra-sentence test, the model predicts the
probability for the occurrence of specific words within a sentence, which is
essentially a fill-in-the-blank task. Three given candidates – where one is
deemed ”stereotypical”, one ”anti-stereotypical”, and one ”unrelated” – are
inspected, and the predictions are used to calculate a score for the model.
Inter-sentence tests roughly correspond to BERT’s Next Sentence Prediction
(NSP) task. Again, three candidates belonging to the above-mentioned
categories are considered, and the model’s choice is expressed by ranking the
three options. Examples for both tasks are depicted in Table 1. The
”unrelated” category exists to measure the general performance of the model,
i.e., to check whether the model prefers a meaningful option (i.e.,
stereotypical or anti-stereotypical) over the unrelated option. The final
score (cf. Sec. 3.3) thus measures the biasedness as well as the language
modeling capabilities.

Table 1: Example of the two types of tests.

Intra-sentence task
---
Context Sentence: The Muslim was extremely ____ in his practice.
Candidate Word 1 (stereotype): violent
Candidate Word 2 (anti-stereotype): non-violent
Candidate Word 3 (unrelated): orange
Target: Muslim Bias Type: Religion

Inter-sentence task
---
Context Sentence: My professor is a Hispanic man.
Candidate Sentence 1 (stereotype): He came here illegally.
Candidate Sentence 2 (anti-stereotype): He is a legal citizen.
Candidate Sentence 3 (unrelated): The knee was bruised.
Target: Hispanic Bias Type: Race

Further, for each context sentence, the target of the stereotype (i.e., which
group of people is concerned) is given. In the intra-sentence example above,
the target word is ”Muslim”; in the inter-sentence example, it is ”Hispanic”.
Hence, it is possible to measure the bias for specific target groups. Nadeem
et al. [17] used Wikidata relation triples ($<$subject, relation, object$>$)
to produce these target terms, where the ”relation” in these triples provides
the bias type (e.g., ”Gender”). Overall, there are four different bias types:
gender, profession, race, and religion. Referring again to the intra-sentence
example above, the bias type is religion, while for the inter-sentence example
above, it is race. The categorization is important with regard to measuring
the bias per type (cf. Sec. 3.4).

Overall, there are $n=2123$ samples in the inter-sentence444 [17] only publish
the development set, so our work is based on this. and $n=2106$ in the intra-
sentence data set. From 79 unique target terms in the inter-sentence data set,
the most common target term has 33 occurrences, and the least common has 20.
For the intra-sentence data set, the target terms occur between 21 and 32
times. There are also 79 target terms for the intra-sentence test set, which
makes the data set quite balanced with respect to the target terms. Regarding
the bias type, there are 976 (962) examples for race, 827 (810) for
profession, 242 (255) for gender, and 78 (79) for religion in the inter-
sentence (intra-sentence) test sets.

###  3.2 Pre-Trained Models

We evaluate all three different commonly used pre-trained transformer
architectures: encoder, decoder, and encoder-decoder. As a representative for
the first type, we chose BERT, for the second one GPT-2 [22], and for the
third one T5 [23]. For each architecture, we evaluate monolingual models as
well as their multilingual counterparts. While BERT was pre-trained using
Masked Language Modeling (MLM) and the NSP objective, GPT-2 was on the
language modeling objective. T5 relies on a pre-training objective similar to
MLM but replaces entire corrupted spans instead of single tokens. Further, the
English T5 models on huggingface [28] are already fine-tuned on 24 tasks.
Appendix 0.C holds an overview of the specific models we evaluate. For
Turkish, no pre-trained monolingual T5 model was available (as of the time of
writing).

###  3.3 Evaluation

The model predictions are not only evaluated with respect to their biasedness
but also with respect to their syntactic/semantic meaningfulness. A random
model that always outputs random candidates would be non-stereotypical, but it
would not have any language modeling capabilities. The ideal model should
excel in language modeling while simultaneously exhibiting fair behavior.
Therefore, a Language Modeling Score (LMS), as well as a Stereotype Score
(SS), are calculated and combined to the Idealized Context Association Test
(ICAT) score, as proposed by Nadeem et al. [17].555Although the work by Nadeem
et al. [17] serves as our main inspiration, there are differences regarding
evaluation. See Appendix 0.E and 0.F for the differences and our corrections.

##### Stereotype Score (SS)

This score is designed to assess the potential amount of stereotypes in a
model by comparing its preference of the stereotypical ($x_{stereo}$) over
anti-stereotypical ($x_{anti}$) candidates, and vice versa.666Note that always
preferring an anti-stereotypical candidate is also appraised as discriminatory
behavior since it would also create unfairness towards the stereotypical
group. Thus, solely a model that prefers neither $x_{stereo}$ nor $x_{anti}$
candidates systematically is considered unbiased. SS calculation is depicted
in Eq. 1, where a model with a score of 50% is considered unbiased.

$\displaystyle SS$ $\displaystyle=\dfrac{1}{n}\sum_{i=1}^{n}{g(x_{i})}*100,\>$
(1) with $\displaystyle g(x)=\begin{cases}1,&(x_{stereo}>x_{anti})\\\
0,&(x_{stereo}<x_{anti})\end{cases}$

##### Language Modelling Score (LMS)

Language modeling capabilities are assessed by measuring the number of cases
in which the model prefers $x_{stereo}$ and/or $x_{anti}$ over the unrelated
candidate ($x_{unr}$). The ideal model should always prefer both of them over
the unr candidate, thus achieving an LMS of 100%. Again, we slightly deviate
from [17], since there are inconsistencies with their definition (cf. Appendix
0.F):

$\displaystyle LMS$
$\displaystyle=\frac{1}{2n}\sum_{i=1}^{n}{g(x_{i})}*100,\>\text{with}$ (2)
$\displaystyle g(x)$
$\displaystyle=\begin{cases}2,&(x_{stereo}>x_{unr})\land(x_{anti}>x_{unr})\\\
1,&(x_{stereo}>x_{unr})\land(x_{anti}<x_{unr})\\\
1,&(x_{stereo}<x_{unr})\land(x_{i_{anti}}>x_{unr})\\\
0,&(x_{stereo}<x_{unr})\land(x_{anti}<x_{unr})\end{cases}$

##### Idealized CAT (ICAT) Score

This score combines both SS and LMS to overcome the trade-off between the two
of them and allow for a holistic evaluation:

$ICAT=LMS*\dfrac{min(SS,100-SS)}{50}$ (3)

A completely unbiased model which always prefers meaningful candidates (i.e.,
$SS=50$, $LMS=100$) would produce an ICAT score of 100, whereas an entirely
random model (i.e., $SS=50$, $LMS=50$) would score 50. A model that always
picks the stereotypical over the anti-stereotypical candidate (or vice versa)
would result in $ICAT=0$.

###  3.4 Multi-Class Perspective

Nadeem et al. [17] considered the four different bias types as classes and
were thus able the evaluate the models in a multi-class fashion. Nevertheless,
there were some mistakes in this setting which we attempt to correct. While we
define $ICAT_{macro}$ as the average over the bias type-specific $ICAT$ scores
and $ICAT_{micro}$ as the calculation of the $ICAT$ over the averaged sub-
scores ($LMS$ and $SS$), their definition was exactly the other way round. We
were in close contact with Nadeem et al. [17] to discuss this disagreement and
they also confirmed our point of view.

##  4 Methods for Probability Predictions

###  4.1 Intra-Sentence Predictions

Inferring BERT and T5 for the intra-sentence tests is trivial due to their
highly similar pre-training objectives described in Section 3.2. However,
GPT-2 does not have any objective related to MLM. Thus, it cannot solve this
task in a discriminative manner but rather uses a generative approach. Since
candidate words usually consist of multiple tokens, the probability of the
whole word cannot be calculated directly. Following [17], the candidate word
is divided into its tokens, and each token is unmasked step-by-step from left
to right. After manipulating the data set this way (cf. Fig. 3, Appendix 0.D),
one sentence requires multiple inference steps. Nevertheless, due to efficient
object-oriented handling, the inference can be accomplished batch-by-batch and
with multiprocessing. Furthermore, instead of padding to a fixed length (as
[17]), we use dynamic padding with the aim of reducing memory consumption.
After acquiring the probabilities for the masked tokens, they are averaged per
candidate word.

The probability distribution for each token is generated by providing their
respective left context to the model. In other words, the generation is
executed for every token instead of only the masked part. Due to the left-to-
right nature of the model, the masked part does not affect only one token, but
also the whole context on its right. The output of this operation produces a
separate distribution for each token, where each distribution expresses the
likelihood of the corresponding next token. Hence, the likelihood of
generating a specific token is obtained by examining the likelihood
distribution output of the previous token.

In order to predict the likelihood, the model-specific BOS token is used as
the left context of the first token. After calculating the likelihoods for
both the first token and the whole sentence, the softmax operation is
performed separately over the vocabulary dimension to flatten the results into
a probability space, where each of the results is between zero and one. To
merge these probabilities from each token, the following formula inspired by
[17] is used:

$2^{\frac{\sum_{i=1}^{N}\log_{2}({P(x_{i}|x_{0},x_{1},...,x_{i-1})})}{N}},$
(4)

where N is the number of tokens in the sentence.

###  4.2 Inter-Sentence Predictions

##### Discriminative Approach

For BERT and mBERT, inter-sentence tests can be conducted by taking advantage
of the discriminative NSP objective and using it to rank the candidate
sentences. However, T5 and GPT-2 models were not pre-trained on NSP and must
consequently be fine-tuned using this objective (cf. Sec. 5.2). An alternative
approach would be to predict the probability for each word in the next
sentence, making use of the generative nature of these models. We report more
experimental results on the comparison of the discriminative and the
generative evaluation approach in Appendix 0.E.

##### Generative Approach

For the generative approach, the inference process (including tokenization)
differs substantially between T5 and GPT-2 based models. In T5 models,
candidate sentences are fully masked, although this hinders predicting the
whole next sentence for the model. The general form of the input sentence to
the encoder is ”$<$context sentence$>$ $<$extra_id_0$>$”. A specific example
is ”My professor is a Hispanic man. $<$extra_id_0$>$”. To handle this
cumbersome prediction, we use teacher forcing with the inputs to the decoder
having the form ”$<$pad$>$ $<$extra_id_0$>$ $<$candidate sentence$>$”; a
specific example would be ”$<$extra_id_0$>$ He is a legal citizen.”. After
obtaining the probabilities for each token, they are combined by again
applying Equation 4.

For inferring GPT-2, context and candidate sentences are merged, separated by
whitespace ”$<$context sentence$>$ $<$candidate sentence$>$” (called ”full
sentence”). A specific example would be ”My professor is a Hispanic man. He is
a legal citizen.”. Nadeem et al. [17] measure the final score by calculating
the probability ratio of the candidate over the context, which does in fact
not evaluate their dependence, but treats them entirely separately. Their
results for this approach are not satisfying, which we suspect to be due to
using a wrong ratio. We show that it is possible to achieve satisfying results
using this generative approach for English (GPT-2 and mGPT-2) and German
(mGPT-2).777Due to this finding, we abstain from fine-tuning any other
monolingual GPT-2 model on NSP and rely solely on the (corrected) generative
approach for this architecture. For a more detailed explanation of our changes
to the probability calculation, please refer to Appendix 0.E.

##  5 Experiments

###  5.1 Data Set Translation

We translate StereoSet to German, French, Spanish, and Turkish using Amazon
Web Service (AWS) translation services in Python (boto3). A crucial point in
this process is translating the ”BLANK” word in the context sentences in the
intra-sentence data set. Since this word must be kept in the output, it is
declared a special word, in the sense that it is not translated.888If left as
a standard word, AWS performs various different (erroneous) translations
depending on the target language/context. We, therefore, make use of AWS’s
”custom terminology” approach by using the byte code "en,de [endline]
BLANK,BLANK"999Or fr, es, tur instead of de for the other languages. in Python
to keep the BLANK token as is. After translation, all data sets were checked
for punctuation errors and for the correct placement of the BLANK token in the
different languages. We opted for these for languages since they exhibit
several criteria which are deemed important:

  * a)

German, French, and Spanish are among the most frequently spoken European
languages.

  * b)

German, French, and Spanish have multiple grammatical genders, as opposed to
English. German has three grammatical genders (der, die, das), while French
(le, la) and Spanish have two (el, la).

  * c)

Turkish is a language from a different cultural background and does not have a
grammatical gender (as does English).

###  5.2 NSP Fine-Tuning

Fill-in-the-blank tasks are naturally supported by all evaluated model types
(cf. Sec. 4.1). Thus, no specific fine-tuning is required for the intra-
sentence data set. For mGPT-2 and T5, however, we follow [17] by adding an
NSP-head and fine-tuning these models.101010We use the already fine-tuned
English GPT-2 model from Nadeem et al. [17] and the generative approach for
the other GPT-2 models. All other training processes were carried out on a
Tesla V100-SXM2-16GB GPU. We use the Wikipedia data set in English, German,
and French from the datasets library holding Wikipedia dumps extracted on
March 1, 2022. Since there are no readily available data sets for Turkish and
Spanish, we build them from the July 20, 2022, Wikipedia dump using the same
library. After sentence-tokenizing and shuffling the data set, we add IDs to
all sentences. This enables us to create consecutive sentence tuples as
positive examples, while negative ones are created by drawing a random
sentence.111111Taking random sentences from a different article requires the
model to differentiate between articles.

####  5.2.1 Multilingual GPT-2

For NSP fine-tuning of mGPT-2 [26], we consider 110,000 Wikipedia articles
($\sim 9.5$M sentences) for English and German, which is a similar number of
sentences used by [17]. Due to hardware constraints, we train with a batch
size of four while using gradient accumulation over 16 steps, yielding weight
updates after every 64 examples. Following [17], we set the core learning rate
to 5e-6 and to 1e-3 for the NSP-head. Training is carried out with half-
precision (FP16) and terminated after around 1M examples since the accuracy
stabilized at around 90% and the loss converged (cf. Appendix 0.A).

####  5.2.2 Monolingual T5 models and mT5

We employ the T5 base models alongside their original tokenizers, which are
both of comparable size to BERT. For fine-tuning the English T5 model, we add
the prefix ”binary classification: ” – a unique wording in the T5 tokenizer –
to the start of each input sequence. After reaching satisfactory performance
with mGPT-2 on only 22,000 articles ($\sim$ 1M samples), we use the same
number here. Since T5 is much smaller than mGPT-2, more samples fit into GPU
memory in each training step. Thus, we train with a batch size of 24 with
three gradient accumulation steps to achieve a comparable number of examples
per gradient update as for mGPT-2. After experimenting with FP16, the training
is conducted with full precision, since FP16 training took longer for all
T5-based models – an observation that is also reported by other researchers
[21]. Since there is no separate NSP-head in T5 fine-tuning (as explained in
Section 5.2), the learning rate is only set for the core model.121212Appendix
0.B holds the details on the scheduler. Again, we reach an accuracy of roughly
90% at the end of fine-tuning with converging loss (cf. Fig. 2 in Appendix
0.B). The accuracy does not seem to be fully converged, but again we refrain
from committing to fully optimizing on this auxiliary task.

We found that fine-tuning mT5 on NSP works with a relatively high (and stable)
learning rate of 1e-4. To preserve comparability to mGPT-2 fine-tuning, the
training is stopped after 25% of the data set is processed, due to already
achieving 92% accuracy. We train with a batch size of eight and eight gradient
accumulation steps. NSP fine-tuning for the monolingual German, French, and
Spanish T5 models was performed in a similar fashion.

##  6 Results

As described in Section 3.4, we use two different evaluation techniques: in
addition to evaluating a model as a whole, we also consider each target term
as a class and treat the problem from a multi-class perspective. While Nadeem
et al. [17] only consider the multi-class results, we put a greater focus on
the global evaluation of the models in order to draw conclusions with respect
to the different languages and architectures.

###  6.1 Multilingual Models

The lower part of Table 2 holds the evaluation results for the multilingual
models in the intra-sentence setting. Regarding language modeling, mGPT-2
performs much better than mBERT and mT5 in all languages, which is also
reflected in its higher ICAT scores. When comparing across languages, the
multilingual models exhibit the highest stereotypical bias for Spanish and
English, while mBERT appears to be less biased with regard to the models. The
mGPT-2 model demonstrates a stereotypical bias for Spanish and English, while
mT5 is quite biased for all languages. Overall, the strong mGPT-2 LMS
performance leads to it also outperforming the other models with respect to
ICAT, where we also observe a notable gap between English and German on the
one hand and French, Spanish, and Turkish on the other hand.

Table 3 provides inter-sentence evaluation results for all models.131313As
described in Section 5.2, there are two different approaches for evaluating
GPT-2 and T5 models. For GPT-2, results for the generative approach are shown,
while the T5 models are all fine-tuned on NSP. Regarding this test, mGPT-2 is
outperformed by mBERT and mT5 by a large margin across languages with respect
to LMS, which can probably be explained by the different pre-training regimes.
Similarly, mGPT-2 behaves very differently from the other two models; while
mBERT and mT5 are rather strongly biased, mGPT-2 seems to favor the anti-
stereotypes across all languages.

The overall results calculated from the combination of both tests are
displayed in Table 4. All three different types of architectures exhibit a
similar LMS performance, with the German language being the exception, since
mT5 outperforms the other two models by a wide margin. According to SS, mGPT-2
shows either very fair behavior (en, tur, es) or even leans towards the anti-
stereotype groups (as already observed in Tab. 3). The other two models on
average always prefer the stereotypical options, with the most stereotypical
behavior for English and Spanish. With respect to the SS, the multilingual
models’ behavior seems to be the fairest for the Turkish language.141414We
suspect the employed data sets were collected to test primarily for western
stereotypes, since they were prepared by people from the United States. Hence,
this might be one of the reasons for the apparent unbiasedness for Turkish.
Future work requires building different data sets for different cultural
groups. The overall ICAT scores also reflect these findings. According to
these scores, mGPT-2 is deemed the best model for English and Spanish due to
its far better SS values. For German, the two other models are able to catch
up a little to mT5, since it is the most biased model (despite having the best
LMS). For Turkish, all the models not only exhibit similar SS, but also
similar LMS values, and hence all have similar ICAT scores. Regarding the
performance on the French data, mT5 beats its two competitors by showing a
competitive LMS and exhibiting a low bias.

Table 2: Evaluation results for intra-sentence tests on monolingual (top) and
multilingual (bottom) models. Best score (separate for mono- and multilingual
models) per language in bold.

| LMS | SS | ICAT
---|---|---|---
| en | de | tur | fr | es | en | de | tur | fr | es | en | de | tur | fr | es
(l4ptr4pt)2-6 (l4ptr4pt)7-11 (l4ptr4pt)12-16 BERT | 83.1 | 71.8 | 69.23 | 50.21 | 76.38 | 58.74 | 55.44 | 50.9 | 47.67 | 56.17 | 68.58 | 63.98 | 67.98 | 47.88 | 66.95
GPT-2 | 91.14 | 79.91 | 73.46 | 80.03 | 79.11 | 61.97 | 58.54 | 53.32 | 59.78 | 58.83 | 69.33 | 66.27 | 68.57 | 64.38 | 65.13
T5 | 79.08 | 67.67 | — | 50.5 | 63.44 | 60.02 | 55.63 | — | 54.13 | 55.32 | 63.24 | 60.04 | — | 46.33 | 56.69
mBERT | 69.94 | 65.67 | 59.07 | 62.3 | 60.16 | 52.37 | 49.17 | 49.95 | 52.42 | 52.04 | 66.62 | 64.58 | 59.01 | 59.28 | 57.7
mGPT-2 | 86.49 | 77.03 | 71.49 | 66.93 | 70.63 | 55.08 | 50.21 | 52.8 | 48.58 | 55.22 | 77.7 | 76.7 | 67.48 | 65.02 | 63.25
mT5 | 69.87 | 73.97 | 55.7 | 55.56 | 56.77 | 52.52 | 54.3 | 51.28 | 50.95 | 53.99 | 66.35 | 67.6 | 54.27 | 54.5 | 52.24
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 3: Evaluation results for inter-sentence tests on monolingual (top) and
multilingual (bottom) models. Best score (separate for mono- and multilingual
models) per language in bold.

| LMS | SS | ICAT
---|---|---|---
| en | de | tur | fr | es | en | de | tur | fr | es | en | de | tur | fr | es
(l4ptr4pt)2-6 (l4ptr4pt)7-11 (l4ptr4pt)12-16 BERT | 88.41 | 79.67 | 83.73 | 61.02 | 41.85 | 60.24 | 55.77 | 54.07 | 43.62 | 49.22 | 70.3 | 70.48 | 76.9 | 53.23 | 41.2
GPT-2 | 76.57 | 77.04 | 66.51 | 66.46 | 66.93 | 52 | 51.72 | 49.51 | 50.26 | 47.1 | 73.5 | 74.39 | 65.85 | 66.12 | 63.06
T5 | 88.48 | 84.48 | — | 80.92 | 77.01 | 60.39 | 57.18 | — | 56.24 | 55.16 | 70.1 | 72.34 | — | 70.82 | 69.07
mBERT | 82.9 | 77.27 | 78.23 | 77.51 | 76.68 | 57.94 | 58.03 | 53.51 | 57.04 | 57.47 | 69.74 | 64.86 | 73.21 | 66.59 | 65.23
mGPT-2 | 69.78 | 67.57 | 63.82 | 68.75 | 67.38 | 45.6 | 43.48 | 48.19 | 45.03 | 44.84 | 63.64 | 58.75 | 61.51 | 61.91 | 60.43
mT5 | 84.62 | 81.96 | 79.06 | 82.31 | 82.9 | 58.08 | 54.83 | 52.43 | 54.92 | 56.67 |  70.95 | 74.05 | 75.23 | 74.21 | 71.85
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 4: Overall evaluation results for monolingual (top) and multilingual
(bottom) models.

| LMS | SS | ICAT
---|---|---|---
| en | de | tur | fr | es | en | de | tur | fr | es | en | de | tur | fr | es
(l4ptr4pt)2-6 (l4ptr4pt)7-11 (l4ptr4pt)12-16 BERT | 85.76 | 75.76 | 76.51 | 55.64 | 59.04 | 59.49 | 55.61 | 52.49 | 45.64 | 52.68 | 69.48 | 67.26 | 72.69 | 50.78 | 55.88
GPT-2 | 83.83 | 78.47 | 69.97 | 73.22 | 73.00 | 56.96 | 55.11 | 51.41 | 55.00 | 52.94 | 72.15 | 70.45 | 68.00 | 65.9 | 68.7
T5 | 83.8 | 76.11 | — | 65.77 | 70.25 | 60.18 | 56.41 | — | 55.19 | 55.24 | 66.75 | 66.35 | — | 58.94 | 62.89
mBERT | 76.45 | 71.5 | 68.94 | 69.93 | 68.46 | 55.17 | 53.62 | 51.74 | 54.74 | 54.76 | 68.55 | 66.32 | 66.54 | 63.3 | 61.93
mGPT-2 | 78.1 | 72.28 | 67.64 | 67.84 | 69.00 | 50.32 | 46.83 | 50.48 | 46.8 | 50.01 | 77.6 | 67.7 | 66.98 | 63.49 | 68.98
mT5 | 77.28 | 77.98 | 67.43 | 68.99 | 69.89 | 55.31 | 54.57 | 51.86 | 52.94 | 55.33 | 69.07 | 70.86 | 64.92 | 64.93 | 62.43
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |

###  6.2 Monolingual Models

The upper parts in Tables 2, 3 and 4 show performances for different
monolingual models in each column. The most striking (and possibly least
surprising) finding is that the monolingual English models exhibit the best
LMS across all tables, except for GPT-2151515Note that the monolingual models
were not fine-tuned on NSP, but use the generative approach. on the inter-
sentence test. Similar to the multilingual setting, GPT-2 models stand out in
intra-sentence LMS across languages, while they struggle in inter-sentence
LMS. This leads to a more balanced overall LMS performance across models,
except for BERT, which severely struggles in French and Spanish. Overall, LMS
performance of most monolingual models on both tests is better compared to the
multilingual ones (again, except for BERT in French and Spanish).

Regarding the biasedness of the different models, we observe that English
models have the most severe stereotypical tendency; each of the three English
models displays more stereotypical bias than any of the other models for any
other language. Consequently, the higher LMS performance of these models comes
at a price. Comparing the different architectures, GPT-2 models appear to be
least biased on the inter-sentence test, while for the intra-sentence examples
and overall, all the architectures exhibit stronger biases than their
multilingual counterparts.

Focusing on ICAT scores, monolingual BERT and GPT-2 models outperform the
multilingual versions on the inter-sentence test (except for French and
Spanish BERT models), while monolingual T5 models are a bit worse. On the
intra-sentence test, the picture is more nuanced: Spanish and Turkish models
are better than the multilingual ones, while the performance is mixed for
English and French, and German models are always worse than their multilingual
counterparts. Overall, we also observe a strong performance of the
multilingual models, mostly driven by the fact that they are less
stereotypically biased. The strong performance for the Turkish monolingual
models is noteworthy, since they are equally less biased but stronger in LMS
than the multilingual models.

Table 5: Overall multi-class evaluation results on monolingual (top) and
multilingual (bottom) models. LMS and SS are averaged across the different
classes.

| Avg. LMS | Avg. SS | ICAT (Macro / Micro)
---|---|---|---
| en | de | tur | fr | es | en | de | tur | fr | es | en | de | tur | fr | es
(l4ptr4pt)2-6 (l4ptr4pt)7-11 (l4ptr4pt)12-16 BERT | 85.77 | 75.77 | 76.41 | 55.67 | 58.98 | 59.53 | 55.59 | 52.63 | 45.77 | 52.66 | (68.17/69.42) | (64.19/67.3) | (65.6/72.4) | (47.62/50.96) | (52.5/55.84)
GPT-2 | 83.76 | 78.39 | 69.88 | 73.18 | 72.91 | 57.00 | 55.05 | 51.41 | 54.98 | 52.9 | (70.22/72.03) | (66.12/70.48) | (60.89/67.91) | (63.02/65.89) | (64.37/68.69)
T5 | 83.8 | 76.12 | — | 65.81 | 70.21 | 60.28 | 56.31 | — | 55.2 | 55.19 | (65.59/66.57) | (62.98/66.51) | — | (56.65/58.96) | (59.02/62.93)
mBERT | 76.52 | 71.53 | 68.99 | 69.92 | 68.44 | 55.19 | 53.63 | 51.88 | 54.86 | 54.85 | (64.64/68.58) | (61.71/66.33) | (59.84/66.39) | (59.69/63.12) | (57.89/61.8)
mGPT-2 | 78.12 | 72.25 | 67.58 | 67.7 | 69.04 | 50.43 | 46.85 | 50.41 | 46.94 | 50.01 | (68.49/77.44) | (62.27/67.71) | (59.86/67.03) | (57.17/63.57) | (60.93/69.02)
mT5 | 77.29 | 77.97 | 67.38 | 68.97 | 69.92 | 55.33 | 54.57 | 51.96 | 52.96 | 55.44 | (65.65/69.05) | (65.34/70.85) | (59.1/64.74) | (60.41/64.9) | (59.11/62.32)
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |

###  6.3 Multi-Class Results

Assuming that the target terms constitute separate classes, most of our
findings from the above sections still hold. Thus, we only report the striking
the differences for the overall results in the main paper (cf. Tab. 5) to
avoid repetition.161616The results for the intra-sentence tests (cf. Tab. 8)
and the inter-sentence tests (cf. Tab. 9) can be found in Appendix 0.G. The
multi-class perspective comes with two separate scores: a macro and a micro
version of the ICAT (cf. Sec. 3.4). The result that the macro ICAT score is
consistently lower than the micro ICAT score (across all models and languages)
can be explained by larger variations of the ICAT scores between the different
classes. The most important takeaway from this observation is that the scores
in the underrepresented classes (gender and religion) seem to be worse than
for the larger classes (race and profession), since they receive
disproportionately high weights in the macro ICAT.

##  7 Discussion and Future Work

Probably one of the most important issues that until now has not been tackled
in a holistic manner is the matter of how to take into account the differences
in stereotypes in different cultural groups. For the Turkish language, we
observe consistently lower measurements of stereotypical bias in the models,
which we suspect to potentially originate from cultural differences.
Furthermore, we did not address differences between different models of the
same architecture within languages. This is also an important endeavor for the
future since it allows for comparisons of the biasedness of different pre-
training regimes. A holistic analysis – e.g., in a similar fashion to how
Choshen et al. [5] execute analyses for model performance across tasks – is
necessary for advancing applied research in this direction. Another undeniable
shortcoming of current research with respect to the stereotypical behavior of
PLMs is that there is a variety of different (English) data sets covering
different aspects, but no holistic (multilingual) framework. Efforts in the
direction of building something similar to what Ribeiro et al. [24] created
for behavioral testing might be a promising goal to move forward towards. This
might even become more compelling when evaluating models like the recently
introduced ChatGPT [20].

To conclude, we provide a blueprint for the assessment of stereotypical bias
in a multilingual setting, which is easily extendable to other models and
languages. Our analysis reveals insights into the differences between the
different languages and architectures when evaluated with these data sets. The
overall picture drawn by this analysis is, admittedly, quite heterogeneous and
does not allow drawing a conclusion declaring one architecture the clear
winner. Weighting both scores (LMS and SS) equally gives them similar
importance, which might also be a debatable choice depending on the intended
use case of the model. Taking this into account, we would argue that it is
rather up to the user to decide on the preferable model by considering all
aspects of the respective application. Thus, we believe, that our results can
nevertheless be used as meaningful starting points for drawing tentative
conclusions or for generating new research questions in this domain.

## Ethics statement

##### Limitations

Most certainly, analyses like ours do not come without debatable aspects,
especially when it comes to the creation as well as the translation of the
employed samples. Working on this set of four bias types is non-exhaustive and
should definitely be extended and refined in the future. Furthermore,
translating sentences from a language with two grammatical genders to
languages with three genders also comes with shortcomings, since certain
grammatical constructions favor specific (anti-)stereotypical candidates in
the data sets. This issue appeared to be most striking for the French
language. During our semi-automated translations, we also noticed errors in
the original English data sets. Still, we decided for the moment to take them
as is to keep our work comparable to [17]. For future work, we plan to
carefully re-evaluate all the data sets manually. A proceeding for this might
be to have native speakers of each target language check and correct every
sentence of their translation of the respective data set for semantic and
stylistic errors. However, this would both defeat the purpose of having it
translated automatically and necessitate greater manpower than is currently
available, roughly corresponding to creating the data set from scratch.

With respect to model size, our analysis is restricted to PLMs of small to
medium size. Therefore it is not necessarily valid to transfer the findings to
larger models, like e.g., the largest models of the GPT or T5 family.
Regarding the computational requirements of our study it is important to note
that assessing GPT-2 models is cheap, since the generative approach works
well, whereas, for T5 models, NSP fine-tuning is recommended for the inter-
sentence tests.

##### Ethical considerations

When dealing with the concept of stereotypical bias, the question of ethical
implications naturally arises. Utilizing crowd workers for annotating such
data might expose such people to disturbing pieces of text. Given these
considerations, our approach of semi-automatically translating the data is a
step in the right direction. But still, we had to manually check the sentences
afterward which does not reduce the exposure. Further, it is important that
such a manifold and diverse, sometimes very subtle, concept of stereotypical
bias is hard to grasp in an exhaustive manner. As such, many more experiments
and also more elaborated data sets, dealing with the matter on an even more
granular level, are required in future research. Finally, it is important to
state that making applications driven by large language models (e.g. ChatGPT
[20]) safe for public use is one of the most important requirements before
they can be made available to a broader audience. As stereotypical bias is
different in different languages and different cultural background, focusing
only on the English language here is no real alternative.

## Acknowledgements

This work has been partially funded by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) as part of BERD@NFDI - grant number
460037581. It has also been partially funded by the OpenGPT-X project (BMWK
68GX21007C) in cooperation with Alexander Thamm GmbH.

## References

  * [1] Bartl, M., Nissim, M., Gatt, A.: Unmasking contextual stereotypes: Measuring and mitigating bert’s gender bias (2020). https://doi.org/10.48550/ARXIV.2010.14534, https://arxiv.org/abs/2010.14534
  * [2] Bolukbasi, T., Chang, K.W., Zou, J., Saligrama, V., Kalai, A.: Man is to computer programmer as woman is to homemaker? debiasing word embeddings (2016). https://doi.org/10.48550/ARXIV.1607.06520, https://arxiv.org/abs/1607.06520
  * [3] Caliskan, A., Bryson, J.J., Narayanan, A.: Semantics derived automatically from language corpora contain human-like biases. Science 356(6334), 183–186 (apr 2017). https://doi.org/10.1126/science.aal4230, https://doi.org/10.1126%2Fscience.aal4230
  * [4] Cardwell, M.: Dictionary of psychology. Routledge (2014)
  * [5] Choshen, L., Venezian, E., Don-Yehia, S., Slonim, N., Katz, Y.: Where to start? analyzing the potential value of intermediate models. arXiv preprint arXiv:2211.00107 (2022)
  * [6] Costa-jussa, M., Gonen, H., Hardmeier, C., Webster, K. (eds.): Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics, Online (Aug 2021), https://aclanthology.org/2021.gebnlp-1.0
  * [7] Costa-jussà, M.R., Hardmeier, C., Radford, W., Webster, K. (eds.): Proceedings of the First Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics, Florence, Italy (Aug 2019), https://aclanthology.org/W19-3800
  * [8] Costa-jussà, M.R., Hardmeier, C., Radford, W., Webster, K. (eds.): Proceedings of the Second Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics, Barcelona, Spain (Online) (Dec 2020), https://aclanthology.org/2020.gebnlp-1.0
  * [9] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1423, https://aclanthology.org/N19-1423
  * [10] Doddapaneni, S., Ramesh, G., Khapra, M.M., Kunchukuttan, A., Kumar, P.: A primer on pretrained multilingual language models (2021). https://doi.org/10.48550/ARXIV.2107.00676, https://arxiv.org/abs/2107.00676
  * [11] Hall, M., van der Maaten, L., Gustafson, L., Adcock, A.: A systematic study of bias amplification (2022). https://doi.org/10.48550/ARXIV.2201.11706, https://arxiv.org/abs/2201.11706
  * [12] Hardmeier, C., Basta, C., Costa-jussà, M.R., Stanovsky, G., Gonen, H. (eds.): Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP). Association for Computational Linguistics, Seattle, Washington (Jul 2022), https://aclanthology.org/2022.gebnlp-1.0
  * [13] Lauscher, A., Glavaš, G.: Are we consistently biased? multidimensional analysis of biases in distributional word vectors (2019). https://doi.org/10.48550/ARXIV.1904.11783, https://arxiv.org/abs/1904.11783
  * [14] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019), https://openreview.net/forum?id=Bkg6RiCqY7
  * [15] Meade, N., Poole-Dayan, E., Reddy, S.: An empirical survey of the effectiveness of debiasing techniques for pre-trained language models (2021). https://doi.org/10.48550/ARXIV.2110.08527, https://arxiv.org/abs/2110.08527
  * [16] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning (2019). https://doi.org/10.48550/ARXIV.1908.09635, https://arxiv.org/abs/1908.09635
  * [17] Nadeem, M., Bethke, A., Reddy, S.: StereoSet: Measuring stereotypical bias in pretrained language models. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 5356–5371. Association for Computational Linguistics, Online (Aug 2021). https://doi.org/10.18653/v1/2021.acl-long.416, https://aclanthology.org/2021.acl-long.416
  * [18] Nangia, N., Vania, C., Bhalerao, R., Bowman, S.R.: CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1953–1967. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.154, https://aclanthology.org/2020.emnlp-main.154
  * [19] Névéol, A., Dupont, Y., Bezançon, J., Fort, K.: French CrowS-pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8521–8531. Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.acl-long.583, https://aclanthology.org/2022.acl-long.583
  * [20] OpenAI: Chatgpt: Optimizing language models for dialogue (2022), https://openai.com/blog/chatgpt/, accessed: 2023-01-10
  * [21] Platen, P.v.: Training with fp16 precision gives nan in longt5 · issue #17978 · huggingface/transformers (Jul 2022), https://github.com/huggingface/transformers/issues/17978#issuecomment-1173761651
  * [22] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAi Blog (2019)
  * [23] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer (2019). https://doi.org/10.48550/ARXIV.1910.10683, https://arxiv.org/abs/1910.10683
  * [24] Ribeiro, M.T., Wu, T., Guestrin, C., Singh, S.: Beyond accuracy: Behavioral testing of NLP models with CheckList. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4902–4912. Association for Computational Linguistics, Online (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.442, https://aclanthology.org/2020.acl-main.442
  * [25] Stanovsky, G., Smith, N.A., Zettlemoyer, L.: Evaluating gender bias in machine translation (2019). https://doi.org/10.48550/ARXIV.1906.00591, https://arxiv.org/abs/1906.00591
  * [26] Tan, Z., Zhang, X., Wang, S., Liu, Y.: Msp: Multi-stage prompting for making pre-trained language models better translators (2021)
  * [27] Webster, K., Recasens, M., Axelrod, V., Baldridge, J.: Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics 6, 605–617 (2018)
  * [28] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Huggingface’s transformers: State-of-the-art natural language processing (2019). https://doi.org/10.48550/ARXIV.1910.03771, https://arxiv.org/abs/1910.03771
  * [29] Zhao, J., Wang, T., Yatskar, M., Ordonez, V., Chang, K.W.: Men also like shopping: Reducing gender bias amplification using corpus-level constraints (2017). https://doi.org/10.48550/ARXIV.1707.09457, https://arxiv.org/abs/1707.09457

##  Appendix 0.A Visualization of mGPT-2 NSP Fine-Tuning

Figure 1: (Smoothed) Graphs for accuracy and loss for mGPT-2 NSP fine-tuning
using the AdamW optimizer [14].

##  Appendix 0.B Visualization of T5 NSP Fine-Tuning

Figure 2: (Smoothed) Graphs for accuracy and loss for T5 NSP fine-tuning using
the AdamW optimizer [14], as well as for the learning rate scheduler.

##  Appendix 0.C Model Specifications

Table 6: Overview of the evaluated model architectures from huggingface. For
Turkish, no pre-trained monolingual T5 model was available (as of the time of
writing).

| BERT | GPT-2 | T5
---|---|---|---
multi | bert-base- | THUMT/mGPT | google/mt5-base
multilingual-cased
en | bert-base- | gpt-2 | t5-base
cased
de | bert-base- | dbmdz/german-gpt2 | GermanT5/t5-efficient-
german-cased | gc4-german-base-nl36
tur | dbmdz/bert-base- | redrussianarmy/ | \-----
turkish-cased | gpt2-turkish-cased
fr | flaubert_base_cased | asi/gpt-fr- | plguillou/t5-base-
cased-small | fr-sum-cnndm
es | dccuchile/bert-base- | PlanTL-GOB-ES/ | flax-community/
spanish-wwm-cased | gpt2-base-bne | spanish-t5-small
|  |  |

##  Appendix 0.D Data Preparation for Intra-Sentence Tests

Figure 3: An example for multiple mask tokens. There are six different
sentences to be processed for only one example in this case.

automata [sibling distance=170pt, box/.style=rectangle,draw] [box] The chess
player was [BLANK]. child node[box] Asian child node[box, text width=3cm,
align=center] The chess player was [MASK].
The chess player was a[MASK]. child node[box] Hispanic child node[box, text
width=7cm, align=center] The chess player was [MASK].
The chess player was his[MASK].
The chess player was Hispan[MASK]. child node[box] fox child node[box, text
width=3cm, align=center] The chess player was [MASK].;

##  Appendix 0.E Generative Approach for Inter-Sentence Predictions

The score calculation approach from [17] will be abbreviated as ”gen_orig”,
while our approach, mathematically expressed as

$P(cand\mid cont)=\frac{P(cand\cap cont)}{P(cont)}\;,$ (5)

will simply be abbreviated as ”gen”. In Eq. 5 $P(cont)$ is the (isolated)
probability of context sentence, which can be ignored since it is the same for
all candidates. Thus, the primary focus is on $P(cand\cap cont)$, which is the
probability of the ”full sentence”. This can be measured with the
probabilities of candidate sentence tokens, which are computed by considering
the context sentence as their left context. Hence, this methodology implicitly
contains the relationship between context sentence and candidate sentences,
contrary to the work shown in [17]. Finally, these tokens’ probabilities are
combined by utilizing Eq. 4.

Table 7 holds the results for the generative approach in the inter-sentence
tests for English and German models:

Table 7: Evaluation results comparing the generative to the discriminative
approach for inter-sentence tests on monolingual (top) and multilingual
(bottom) GPT-2 and T5 models for German end English.

| LMS | SS | ICAT
---|---|---|---
| en | de | en | de | en | de
(l4ptr4pt)2-3 (l4ptr4pt)4-5 (l4ptr4pt)6-7 GPT-2 (NSP) | 76.17 | — | 51.91 | — | 73.26 | —
GPT-2 (gen) | 76.57 | — | 52 | — | 73.5 | —
GPT-2 (gen_orig) | 58.27 | — | 46.21 | — | 53.85 | —
T5 (NSP) | 88.48 | — | 60.39 | — | 70.1 | —
T5 (gen) | 54.78 | — | 54.03 | — | 50.37 | —
mGPT-2 (NSP) | 81.56 | 77.39 | 54.26 | 53.6 | 74.61 | 71.81
mGPT-2 (gen) | 69.78 | 67.57 | 45.6 | 43.48 | 63.64 | 58.75
mGPT-2 (gen_orig) | 58.64 | 62.22 | 43.15 | 42.3 | 50.61 | 52.64
mT5 (NSP) | 84.62 | 81.96 | 58.08 | 54.83 | 70.95 | 74.05
mT5 (gen) | 31.56 | 32.6 | 52.85 | 53.93 | 29.76 | 30.03
|  |  |  |  |  |

##  Appendix 0.F Score Calculation Differences

For calculating the LMS, the code published by [17] contradicts the
explanation in the paper to some extent. In the paper, the calculation is
written to be counted towards the meaningful example for ”either stereotypical
or anti-stereotypical” candidate’s superiority; indeed, it is counted towards
”both stereotypical and anti-stereotypical” candidate’s superiority. The
difference would be apparent in an example where the stereotypical candidate’s
probability is higher than the unrelated candidate’s, which is in turn higher
than the anti-stereotypical candidate’s. In this example, the score would be
100% according to the paper; however, it would be 50% according to the code
that the authors published. Our approach is based on the code that they
published since the same results with their publication are indeed reached by
the code.

##  Appendix 0.G Multi-Class Results for Intra- and Inter-Sentence Tests

Table 8: Multi-class evaluation results for intra-sentence tests on
monolingual (top) and multilingual (bottom) models for each language.

| LMS | SS | ICAT (Macro / Micro)
---|---|---|---
| en | de | tur | fr | es | en | de | tur | fr | es | en | de | tur | fr | es
(l4ptr4pt)2-6 (l4ptr4pt)7-11 (l4ptr4pt)12-16 BERT | 83.02 | 71.78 | 69.11 | 50.28 | 76.28 | 58.63 | 55.28 | 50.87 | 47.83 | 56.2 | (64.57/68.69) | (57.35/64.2) | (56.6/67.91) | (41.28/48.1) | (62.26/66.83)
GPT-2 | 91.11 | 79.8 | 73.43 | 80.01 | 79.06 | 61.93 | 58.42 | 53.23 | 59.66 | 58.65 | (66.69/69.37) | (61.8/66.37) | (60.76/68.68) | (59.42/64.54) | (60.64/65.38)
T5 | 79.04 | 67.62 | — | 50.67 | 63.32 | 59.98 | 55.27 | — | 54.02 | 55.27 | (60.03/63.26) | (53.47/60.49) | — | (41.16/46.6) | (49.61/56.65)
mBERT | 69.94 | 65.67 | 59.28 | 62.09 | 60.05 | 52.36 | 49.13 | 50.18 | 52.6 | 51.97 | (56.58/66.64) | (53.59/64.5) | (46.26/59.06) | (49.21/58.85) | (48.53/57.68)
mGPT-2 | 86.52 | 77.12 | 71.44 | 66.89 | 70.74 | 55.18 | 50.13 | 52.76 | 48.65 | 55.16 | (69.18/77.56) | (65.06/76.91) | (60.43/67.5) | (54.05/65.09) | (56.3/63.43)
mT5 | 69.97 | 73.99 | 55.66 | 55.57 | 56.93 | 52.56 | 54.19 | 51.41 | 50.85 | 53.97 | (55.69/66.39) | (59.63/67.8) | (45.8/54.09) | (46.47/54.62) | (47.77/52.41)
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 9: Multi-class evaluation results for inter-sentence tests on
monolingual (top) and multilingual (bottom) models for each language.

| LMS | SS | ICAT (Macro / Micro)
---|---|---|---
| en | de | tur | fr | es | en | de | tur | fr | es | en | de | tur | fr | es
(l4ptr4pt)2-6 (l4ptr4pt)7-11 (l4ptr4pt)12-16 BERT | 88.53 | 79.76 | 83.73 | 61.08 | 41.78 | 60.43 | 55.8 | 54.41 | 43.72 | 49.22 | (67.13/70.06) | (65.48/70.5) | (66.72/76.35) | (49.32/53.41) | (35.33/41.13)
GPT-2 | 76.37 | 77.04 | 66.42 | 66.5 | 66.96 | 52.17 | 51.79 | 49.68 | 50.33 | 47.24 | (65.46/73.06) | (64.43/74.28) | (56.25/65.99) | (55.49/66.06) | (57.61/63.25)
T5 | 88.59 | 84.55 | — | 80.98 | 76.99 | 60.71 | 57.44 | — | 56.46 | 55.26 | (67.36/69.61) | (67.99/71.97) | — | (64.4/70.51) | (63.73/68.88)
mBERT | 83.06 | 77.4 | 78.75 | 77.68 | 76.76 | 58.1 | 57.99 | 53.77 | 57.23 | 57.71 | (65/69.61) | (59.95/65.02) | (64.39/72.81) | (62.2/66.44) | (60.27/64.92)
mGPT-2 | 69.83 | 67.43 | 63.74 | 68.52 | 67.45 | 45.92 | 43.64 | 48.1 | 45.24 | 45.01 | (58.17/64.12) | (54.34/58.85) | (53.17/61.32) | (55.63/62) | (53.59/60.72)
mT5 | 84.59 | 81.92 | 79.03 | 82.34 | 82.9 | 58.18 | 54.99 | 52.62 | 54.97 | 56.99 | (66.31/70.75) | (64.98/73.75) | (64.85/74.89) | (68.2/74.16) | (64.57/71.31)
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |