theoremTheorem lemmaLemma

# Evaluating Metrics for Bias in Word Embeddings

Sarah Schröder  Alexander Schulz  Philip Kenneweg  Robert Feldhans  Fabian
Hinder  Barbara Hammer
CITEC, Machine Learning Group
Bielefeld University - Faculty of Technology
{saschroeder, aschulz, pkenneweg, rfeldhans, fhinder<EMAIL_ADDRESS>bielefeld.de

###### Abstract

Over the last years, word and sentence embeddings have established as text
preprocessing for all kinds of NLP tasks and improved the performances
significantly. Unfortunately, it has also been shown that these embeddings
inherit various kinds of biases from the training data and thereby pass on
biases present in society to NLP solutions.
Many papers attempted to quantify bias in word or sentence embeddings to
evaluate debiasing methods or compare different embedding models, usually with
cosine-based metrics. However, lately some works have raised doubts about
these metrics showing that even though such metrics report low biases, other
tests still show biases. In fact, there is a great variety of bias metrics or
tests proposed in the literature without any consensus on the optimal
solutions. Yet we lack works that evaluate bias metrics on a theoretical level
or elaborate the advantages and disadvantages of different bias metrics.
In this work, we will explore different cosine based bias metrics. We
formalize a bias definition based on the ideas from previous works and derive
conditions for bias metrics. Furthermore, we thoroughly investigate the
existing cosine-based metrics and their limitations to show why these metrics
can fail to report biases in some cases. Finally, we propose a new metric,
SAME, to address the shortcomings of existing metrics and mathematically prove
that SAME behaves appropriately.

##  1 Introduction

Word embeddings have been widely adopted in NLP tasks over the last decade,
and have been further developed to embed the context of words, whole sentences
or even longer texts into high dimensional vector representations. These are
applied in a wide variety of downstream tasks, including translation and text
generation as in question-answering systems (gpt) or next sentence prediction
(bert), or sentiment and similarity analysis (use; word2vec; glove;
sentencebert) of texts.
However, many works have shown that these models capture various kinds of
biases present in humans and society. Using biased word or text
representations poses a great risk to produce unfair language models, which in
turn can further amplify the biases in society.
The first works investigating bias in word embeddings focused on geometric
relations between embeddings of stereotypically associated words. From there,
several metrics based on cosine similarity have been proposed like the "Direct
Bias" by bolukbasi and the Word Embedding Association Test (WEAT) by weat.
Though initially introduced to prove the presence of bias in word embeddings,
these tests have also been widely used to compare biases in different
embedding models ((e.g. seat; use) or validate debiasing method (e.g.
sentdebias; karve2019conceptor; kaneko2021dictionary). Due to its popularity,
there have been several variants introduced in the literature like the
Sentence Encoder Association Test (SEAT) by seat, the "Generalized WEAT"
(weatgeneralized) or WEAT for gendered languages (weatgender). Another cosine
based metric is the Mean Average Cosine Similarity (MAC) score (mac), though
it has a different intuition than WEAT and the Direct Bias.
However, there is literature criticizing WEAT and its variants. For instance
seat found that SEAT produced rather inconsistent results on sentence
embeddings and emphasized the positive predictability of the test. This
prompted many other papers to doubt the meaningfulness or usability of cosine
based metrics in general. Hence, methods to quantify bias in downstream tasks
have been proposed and increasingly replace cosine based metrics (coreference;
lipstick).
On the one hand, this is desirable since it bridges the gap to fairness
evaluation in general and allows to apply other fairness definitions like
statistical parity or counterfactual fairness. On the other hand, it poses the
risk to confound biases from word representations with biases introduced by
the downstream task itself. Hence, when investigating bias in word embedding
models, apart from specific downstream tasks, metrics directly applicable to
the embeddings are beneficial.
Nevertheless, the doubts regarding existing metrics show the need to evaluate
these metrics on a theoretical level. To our knowledge, the only work doing
this is ripa. The authors showed that WEAT has theoretical flaws that can
cause it to overestimate biases and demonstrated that one could make up
attribute sets in order to make the test appear statistically insignificant.
While this alone is highly worrying, we find even more theoretical flaws that,
on one hand, cause WEAT to underestimate biases under certain conditions. On
the other hand, these flaws make WEAT results incomparable between different
embeddings. Moreover, these flaws extend to other versions like SEAT or the
generalized WEAT, and we also find flaws for MAC and the Direct Bias.
Noteworthy, there exist also works criticising that cosine based bias scores
do not capture biases entirely. However, they rely on different definitions of
bias. As we discuss in Chapter 2.1, these definitions do not necessarily
contradict the definition by bolukbasi or weat.
Since there are applications like document clustering or text similarity
estimations, where the cosine similarity is directly queried (sentencebert;
use; gao2021simcse; subramanian2018learning; liu2021transencoder), quantifying
bias in terms of cosine similarity is highly relevant and thus further
evaluation of cosine based metrics remains necessary.

The main contributions in this work are the following: (i) We define formal
requirements for cosine based score functions which formalize a notion of
meaningfulness in this context. (ii) We analytically demonstrate that the
state of the art metrics WEAT, the Direct Bias and MAC are only partially
meaningful, corresponding to the previous definitions. (iii) We propose a
novel bias score function _Scoring Association Means of Word Embeddings_
(SAME), that is based on some principles of WEAT, but reformulates potentially
problematic parts. (iv) We experimentally investigate SAME and the state of
the art score functions and substantiate the theoretic claims with real word
embeddings.

The remainder of this paper is structured as follows: In Chapter 2 we give an
overview over bias definitions and specifically cosine based bias metrics from
the literature. The formal requirements for bias score functions are explained
and applied to the existing score functions in Chapter 3.2. Next, in Chapter
4, we propose SAME with extensions for multi-attribute biases and skew and
stereotype distinction. Chapter 5 describes our experiments, where we retrain
BERT on biased data. Thereby we induce specific biases as a ground truth to
evaluate the different bias score functions. Lastly, we conclude our findings
and give an overview over the bias score functions in Chapter 6.

##  2 Related Work for Bias in Text Embeddings

In the following, we summarize the related work on bias in word embeddings.
First, we focus on definitions and discuss our choice of cosine based scores.
Then we revisit the different cosine based metrics from the literature.

###  2.1 Bias Definition

In this section, we describe several definitions of bias in word embeddings
mentioned in the literature. The focus of this work lies on geometrical bias
definitions, especially those regarding cosine similarity, and the skew and
stereotype definitions. For the sake of clarity, we also discuss other bias
definitions regarding downstream tasks. Lately, measuring bias in downstream
tasks increasingly replaces geometrical bias scores in the literature. Yet, we
argue that they cannot replace the geometrical definitions and scores entirely
and thereby emphasize the usability of cosine based scores.

####  2.1.1 Geometrical Bias Definitions

Word embedding vectors are expected to reflect word relations by their
geometrical relations, whereby distance in the vector space is said to
correlate with the semantic similarity of words (word2vec; glove). Due to this
intuition, biases in word embeddings are often considered as flawed vector
representations, where words are embedded inappropriately close or far from
other words. Since the cosine similarity is often used as a distance measure
to estimate word or document similarity (thongtan2019sentiment;
shahmirzadi2019text; sentencebert; use), it has also been used to measure bias
in terms of geometric relations.

Figure 1: Simplified (2D) visualization of gender stereotypes in occupation
embedding vectors. The occupation vectors in dotted lines show a less biased
representation of the occupations. Ideally, all gender neutral terms should be
represented on the neutral axis, thus being equidistant to $\bm{he}$ and
$\bm{she}$. The red dotted line indicates the bias direction
$\bm{g}=\pm(\bm{she}-\bm{he})$

Based on this idea, bolukbasi defined bias as the correlation between neutral
words and a bias subspace or bias direction, that describes the relations
between groups of words. For the simple case illustrated in Figure 1 this
would be the gender direction $\bm{g}=\bm{she}-\bm{he}$. To get a more robust
estimation of group relations (e.g. male and female terms), one would use
several word pairs and determine a low dimensional subspace that captures most
of the variance of individual bias directions.
weat define bias as the difference in angular distance between a neutral word
and two different groups of words. In the example depicted in Figure 1, they
would consider $\bm{secretary}$ biased since it has a lower angular distance
to $\bm{she}$ than $\bm{he}$ and $\bm{engineer}$ vice versa.
Both definitions follow a similar intuition and use the cosine similarity as
correlation or distance measure. For the simple case illustrated in Figure 1
correlation with the bias direction matches up with the difference in angular
distances. An important difference is that weat contrast over two groups of
theoretically neutral words to determine a stereotype (see Section 2.2 for
details) while bolukbasi take the mean bias correlation over all neutral words
(see Section 2.2.3).

Another work by ripa follows the definition from bolukbasi. However, they
argue that the vector length holds important information and thus biases
should rather be measured using the inner product of word vectors instead of
cosine similarity. Yet, their definition of bias mostly concerns word co-
occurrences in the training data and how these are represented in word vectors
instead of the fairness implications in different use cases of word
embeddings. Hence, it is not clear to what extent the information encoded in
the vector length influences fairness in downstream tasks. On the other hand,
there are downstream tasks such as document clustering that might rely
directly on the cosine similarity or use it as a dissimilarity measure.
Consequently, using the cosine similarity (as a modification of the inner
product that removes the influence of vector lengths) is a valid option in
cases where the vector length can be neglected. Moreover, they only consider
single word biases. When computing the bias over a set of words, it seems
preferable to normalize the word biases by word vector length to have all
words contribute with the same weight to the overall bias, resulting in the
same notion as if using cosine based scores.

####  2.1.2 Classification and Clustering Bias

lipstick found that stereotypical groups persisted after removing correlations
of neutral words with a bias subspace or retraining on debiased text sources.
This raised doubts on the usability of cosine based metrics. Instead they
proposed to use clustering or classification tasks to investigate bias. In
short, if a classifier or clustering algorithm can reconstruct stereotypical
groups (according to previous correlation with the bias subspace), then there
is still bias in the data, according to their definition. However, there are
two possible explanations why they found this persisting bias: Either the
debiasing methods were not performed thoroughly (leaving some geometrical bias
associations in the embeddings or text sources) or as ripa described it:
"Because we define unbiasedness with respect to a set of word pairs, we cannot
make any claims about word pairs outside that set". Another possibility is
that these stereotypical groups simply reflect other relations independent of
the bias attributes, but different enough to reconstruct stereotypical groups
with a flawed classification task (e.g. different types of occupations that
coincidentally relate to gender stereotypes). Since it is not clear whether
one of these effects lead to their claims, we cannot take lipstick’s work as
evidence against cosine based metrics.

####  2.1.3 Bias in other Downstream Tasks

Lately, more and more works started to measure bias in downstream tasks as
opposed to geometrical metrics. An example are coreference and
DBLP:journals/corr/abs-1904-03310-zhao1, who used a co-reference resolution
task. For instance, the language model has to decide which word a gendered
pronoun was associated with. Confronting the model with pro- and anti-
stereotypical sentences can be used to determine whether the decisions are
based on gender biases as opposed to semantic meaning.
Another work to be emphasized is kurita, who proposed a bias test based on the
masked language objective of BERT. Essentially, they query BERT for the
probability to insert certain bias attributes in a masked sentence when
associated with other neutral words. For instance they would compute the
association between "programmer" and the male gender by probing BERT for the
probability to replace "[MASK]" with "he" in "[MASK] is a programmer".
They show that this is an accurate way to measure biases in BERT and can show
a correlation with bias in the GPR task by gap.
However, the bias measured by kurita is model-specific for BERT’s masked-
language modeling (MLM) models and measuring bias in downstream tasks is
always task specific. Both might be influenced by fine-tuning for MLM or the
specific task. This is fine as long as one wants to quantify bias with respect
to this specific model or task, but might not be reliable when comparing
embedding models or probing embedding models without a specific use case in
mind. On the other hand, cosine based metrics work for any kind of embedding
models, from classical word embeddings to contextualized and sentence
embeddings, and hence are worth further investigating.

####  2.1.4 Skew and Stereotype

Recently, skew introduced another definition of bias. They also used the GPR
task by gap to obtain word biases, but distinguish between two types of bias:
Skew and stereotype. While stereotype shows the presence of groups stronger
associated with different bias attributes than other groups, which is also the
intuition behind weat’s work, the skew describes the effect that all words
from a set are (on average) biased towards the same attribute. For instance,
occupations could be stereotypically associated with male/female terms, but
it’s also possible that (on average) all occupations are rather biased towards
one gender. They further suggest that there is a certain trade-off between
these two forms of bias.
Distinguishing these kinds of biases seems to be beneficial for investigating
bias in embeddings more thoroughly. Hence, we apply their definition to
construct cosine based skew and stereotype metrics (see Section 4.4) and show
their benefit over general bias metrics in the experiments.

###  2.2 Cosine-based measure

Cosine-based proximity measures are the most used measures in terms of
geometric relations. While it is possible to refer to other
similarity/dissimilarity measures, mostly the cosine similarity

$\displaystyle cos(\bm{u},\bm{v})$ $\displaystyle=$
$\displaystyle\frac{\bm{u}\cdot\bm{v}}{||\bm{u}||\cdot||\bm{v}||}.$ (1)

is used to determine the association between two vector representations of
words $\bm{u}$ and $\bm{v}$.
Usually the similarity of words $\bm{w}$ towards groups of words (e.g. terms
specific for one gender/ religion/ race…), so called attribute sets, is
measured. The similarity of $\bm{w}$ and one set of attributes $A$ is then

$\displaystyle s(\bm{w},A)$ $\displaystyle=$ $\displaystyle mean_{\bm{a}\in
A}\cos(\bm{w},\bm{a}).$ (2)

Based on this, different bias metrics are defined in literature. We detail the
most prominent ones in the following.

####  2.2.1 WEAT

The Word Embedding Association Test (WEAT), as defined by weat, is based on
the Implicit Association Test (greenwald), which is used in psychological
studies to measure reaction times regarding pro and anti stereotypical
associations.
The test compares two sets of target words $X$ and $Y$ with two sets of bias
attributes $A$ and $B$ of equal size $n$. When observing gender stereotypes in
occupations, which might be $A=\\{he,man,male,...\\}$,
$B=\\{she,woman,female,...\\}$, $X=\\{engineer,doctor,policeman,...\\}$ and
$Y=\\{secretary,nurse,teacher,...\\}$, anticipating that the occupations in
$X$ would be closer to words in $A$ than $B$, and vice versa for occupations
in $Y$.
The association of a single word $\bm{w}$ with the bias attribute sets $A$ and
$B$ including $n$ attributes each, is given by

$\displaystyle s(\bm{w},A,B)$ $\displaystyle=$
$\displaystyle\frac{1}{n}\sum_{\bm{a}\in
A}\cos(\bm{w},\bm{a})-\frac{1}{n}\sum_{\bm{b}\in B}\cos(\bm{w},\bm{b}).$ (3)

To quantify bias, the effect size is used, which is a normalized measure for
the association difference between the target sets

$\displaystyle d$ $\displaystyle=$ $\displaystyle\frac{mean_{\bm{x}\in
X}s(\bm{x},A,B)-mean_{\bm{y}\in Y}s(\bm{y},A,B)}{stddev_{\bm{w}\in X\cup
Y}s(\bm{w},A,B)},$ (4)

where $mean_{\bm{x}\in X}s(\bm{x},A,B)$ refers to the mean of $s(\bm{x},A,B)$
with $\bm{x}$ in $X$ and $stddev_{\bm{x}\in X}s(\bm{x},A,B)$ to the standard
deviation over all word biases of $\bm{x}$ in $X$.
A positive effect size confirms the hypothesis that words in $X$ are rather
stereotypical for the attributes in $A$ and words in $Y$ stereotypical for
words in $B$, while a negative effect size indicates that the stereotypes
would be counter-wise.
Based on the Implicit Association Tests, the WEATs include several tests
probing for associations between pleasant/unpleasant words and race as well as
tests probing for gender stereotypes comparing career and family, math and
arts as well as science and arts. To determine the statistical significance of
biases measured in those tests, the authors use the test statistic

$\displaystyle s(X,Y,A,B)$ $\displaystyle=$ $\displaystyle\sum_{\bm{x}\in
X}s(\bm{x},A,B)-\sum_{\bm{y}\in Y}s(\bm{y},A,B)$ (5)

for a permutation test with partitions $(X_{i},Y_{i})$ of $X\cup Y$:

$\displaystyle p$ $\displaystyle=$ $\displaystyle
P_{r}[s(X_{i},Y_{i},A,B)>s(X,Y,A,B)].$ (6)

There are several extensions to WEAT: For instance WEAT for gendered languages
by weatgender and the "Generalized WEAT" by weatgeneralized, which allows to
apply WEAT to more than two different attribute and target groups.
seat propose the Sentence Encoder Association Test (SEAT), which basically
applies WEAT to sentence representations. The sentences are obtained by
inserting WEAT terms into simple templates such as "this is a <term>". They
also propose two additional tests: The Angry Black Women Stereotype and the
Double Bind tests.

####  2.2.2 MAC

The Mean Average Cosine Similarity (MAC) score was introduced by mac to
provide a metric for multi-class fairness problems. They compute the bias of a
word $\bm{t}$ towards an attribute set $A_{j}$ using the cosine distance as
the reciprocal of the cosine similarity:

$\displaystyle S(\bm{t},A_{j})=\frac{1}{N}\sum_{\bm{a}\in
A_{j}}1-cos(\bm{t},\bm{a}).$ (7)

In contrast to WEAT, only one set of target words $T$ is utilized. The bias of
words in $T$ towards several bias attribute sets $A=\\{A_{1},...,A_{n}\\}$ is
then given by the MAC score as

$\displaystyle MAC(T,A)=\frac{1}{|T||A|}\sum_{\bm{t}\in T}\sum_{A_{j}\in
A}S(\bm{t},A_{j}).$ (8)

Equation (8) applies a different intuition to WEAT and the Direct Bias (see
Figure 1 for the bias definition). However, MAC uses the direct associations
instead of contrasting terms like weat or measuring the correlation with a
contrastive bias direction (or subspace) like bolukbasi. Most works use the
MAC score only for multi-class problems that (the original) WEAT cannot be
applied to. For instance, sentdebias use it in addition to WEAT to evaluate
their debiasing method.

####  2.2.3 Direct Bias (Bolukbasi)

bolukbasi define the Direct Bias as the correlation of neutral words
$\bm{w}\in N$ with a bias direction (in their example gender direction
$\bm{g}$):

$\displaystyle DirectBias$ $\displaystyle=$
$\displaystyle\frac{1}{|N|}\sum_{\bm{w}\in N}|\cos(\bm{w},\bm{g})|^{c}$ (9)

with $c$ determining the strictness of bias measurement. The gender direction
is either obtained by a gender word-pair e.g. $\bm{g}=\bm{he}-\bm{she}$ or -
to get a more robust estimate - it is obtained by computing the first
principal component over a set of individual gender directions from different
word-pairs.
The authors used it as a preliminary experiment to show bias present in
embeddings, but not to evaluate their debiasing algorithm (which follows the
same intuition). The Direct Bias is applied e.g. in
genderbias_underrepresentation and costa2019evaluating.
In terms of their debiasing algorithm bolukbasi describe how to obtain a bias
subspace given defining sets $D_{1}$, …, $D_{n}$. A defining set $D_{i}$
includes words $\bm{w}$ that only differ by the bias relevant topic e.g. for
gender bias $\\{\bm{man},\bm{woman}\\}$ could be used as a defining set. Given
these sets, the authors construct individual bias directions
$\bm{w}-\bm{\mu_{i}}\;\forall\bm{w}\in D_{i},i\in\\{1,...,n\\}$ and
$\bm{\mu_{i}}=\sum_{\bm{w}\in D_{i}}\frac{\bm{w}}{|D_{i}|}$. To obtain a
k-dimensional bias subspace $B$ they compute the $k$ first principal
components over these samples. mac applied this debiasing algorithm to
mitigate multi-class bias. For this purpose they determined the bias subspace
using defining sets $D_{i}$ with more than two elements. This could be also
applied to the Direct Bias to measure multi-class bias.

##  3 Requirements for Bias Metrics

In this section, we formalize requirements for bias score functions and
analyze these properties for bias scores from the literature. Thereby, we
focus on cosine-based score functions, in particular the WEAT, Direct Bias and
MAC.

###  3.1 Motivation

To emphasize the need of formal requirements for bias metrics, we start with
an example where the effect size of WEAT behaves counter-intuitive: Consider
the example of gender bias with $A$ containing embeddings of male terms and
$B$ containing those of female terms, and 6 occupations, half of those
stereotypically male/female. All words were embedded with BERT. We computed
the word-wise biases of 6 occupations according to WEAT’s $s(\bm{w},A,B)$. The
results are shown in Figure 2 (the y-axis is added for better visibility and
does not imply any difference).

Figure 2: Bias of stereotypical male and female jobs as indicated by WEAT’s
$s(\bm{w},A,B)$ calculated on BERT embeddings. Differences along the y-axis
are arbitrary shift for better visibility.

We then used groups $X=\\{\bm{footballer},\bm{bishop}\\}$ and
$Y=\\{\bm{hairdresser},\bm{sociologist}\\}$ as target words and measured the
effect size $d(X,Y,A,B)=1.7299$, which indicates a high stereotype regarding
the groups $X$ and $Y$.
Furthermore, we constructed groups
$X^{\prime}=\\{\bm{footballer},\bm{lawmaker}\\}$ and
$Y^{\prime}=\\{\bm{hairdresser},\bm{nurse}\\}$, using the more extremely
stereotyped occupations according to the word-wise bias scores. Now, when
computing the effect size using $X^{\prime}$ and/or $Y^{\prime}$ instead of
their counterparts, one would expect a bias metric to report an even larger
bias with respect to these groups.
However, we found $d(X,Y^{\prime},A,B)=-0.0865$ and
$d(X^{\prime},Y^{\prime},A,B)=1.6448e-05$, which would be interpreted as
significantly lower biases, although looking at individual words we know that
this is not the case.
This is just one example to illustrate how WEAT can be misleading when
quantifying biases. Hence, in the following sections, we derive formal
requirements for bias scores to ensure their results are reliable and
comparable.

###  3.2 Requirements for Bias Metrics

While WEAT is frequently used in the literature (weatgender; weatgeneralized;
kaneko2021dictionary; sentdebias; karve2019conceptor; nullitout), there does
exist literature documenting its behaviour to be not always as expected, e.g.
seat, who emphasize that WEAT cannot prove the absence of bias, or ripa, who
already showed theoretical flaws of WEAT. Apart from this, we show further
examples where WEAT, the Direct Bias and MAC fail to report biases.

In this context we aim to formulate formal properties which a bias score
function should fulfill. One goal of the following properties is to ensure
that resulting bias scores are comparable between different embeddings. A
necessary condition for such comparability is the possibility to reach the
extreme values of $b$ (the minimum and maximum value) in different embeddings
depending on the input target word only.

More formally, assume a word embedding is fixed with embedding space E.
Considering $W$, a set of words in $E$ and $A_{i}\in A$ sets of attributes in
$E$, a bias score function $b(W,A)$ maps $W$ and the set of attribute sets $A$
to a real number. The set of words $W$ could also be divided into subsets $X$
and $Y$, corresponding to two attribute sets $A$ and $B$ (as done in WEAT). We
propose the following requirements:

######  Definition 1 (magnitude-comparable).

We call the bias score function $b(W,A)$ magnitude-comparable if, for a fixed
number of target words in set $W$, the maximum bias score $b_{max}$ and the
minimum bias score $b_{min}$ are independent of the attribute sets in A:

$\displaystyle\max_{W,|W|=const}b(W,A)=b_{max},\quad\min_{W,|W|=const}b(W,A)=b_{min}\;\forall\;A.$
(10)

In addition to the previous definition, a requirement for the consistency of
the minimal bias score $b_{0}$ (indicating fairness) can be formulated in a
straight forward way using the similarities to the attribute sets $A_{i}$.

######  Definition 2 (unbiased-trustworthy).

Let $b_{0}$ be the minimal bias score of a bias score function, that is
equivalent to no bias being measured. We call the bias score function
$b(\bm{w},A)$ unbiased-trustworthy if

$\displaystyle b(\bm{w},A)=b_{0}\iff
s(\bm{w},A_{i})=s(\bm{w},A_{j})\;\forall\;A_{i},A_{j}\in A.$ (11)

Analogously for scores that use at set of words
$W=\\{\bm{w}_{1},...,\bm{w}_{m}\\}$, we say $b(W,A)$ is unbiased-trustworthy
if

$\displaystyle b(W,A)=b_{0}\iff
s(\bm{w}_{k},A_{i})=s(\bm{w}_{k},A_{j})\;\forall\;A_{i},A_{j}\in
A,k\in\\{1,...,m\\}.$ (12)

###  3.3 Sensitivity to Skew and Stereotype

We formulate two additional criteria in alignment to the skew and stereotype
definitions. Since the requirements introduced in 3.2 are critical to achieve
comparable and trustworthy measurements in terms of a bias magnitude, the
following definitions are simply meant to distinguish what type of biases a
metric is sensitive towards.

######  Definition 3 (skew-sensitive).

Let $b_{0}$ be the minimal bias score of a bias score function, that is
equivalent to no bias being measured. We call the bias score function $b(W,A)$
skew-sensitive if

$\displaystyle\exists A_{i},A_{j}\in A:mean_{\bm{w}\in
W}s(\bm{w},A_{i})>mean_{\bm{w}\in W}s(\bm{w},A_{j})$
$\displaystyle\Rightarrow\ $ $\displaystyle b(W,A)\neq b_{0}.$ (13)

In lay terms, skew-sensitivity reflects the fact that a bias score should
increase if neutral words are (on average) more similar to one bias-relevant
concept than another.

######  Definition 4 (stereotype-sensitive).

Let $b_{0}$ be the minimal bias score of a bias score function, that is
equivalent to no bias being measured. We call the bias score function $b(W,A)$
stereotype-sensitive if

$\displaystyle\exists\bm{w_{1}},\bm{w_{2}}\in W,A_{i},A_{j}\in
A:s(\bm{w_{1}},A_{i})-s(\bm{w_{1}},A_{j})\neq
s(\bm{w_{2}},A_{i})-s(\bm{w_{2}},A_{j})$ $\displaystyle\Rightarrow\ $
$\displaystyle b(W,A)\neq b_{0}.$ (14)

In lay terms, this definition refers to the fact that an embedding, which
places words differently with respect to its relative distance as regards to
(opposing) concepts is biased as regards the axes which is described by these
two concepts.

As stated in the following theorem, bias score functions that are unbiased-
trustworthy are both skew-sensitive and stereotype-sensitive as stated in the
following theorem. From there follows that a metric that is not skew-sensitive
or not stereotype-sensitive cannot be unbiased-trustworthy.

A bias score function $b(W,A)$ that is unbiased-trustworthy is also skew-
sensitive and stereotype-sensitive. For the proof see Appendix 3.3.

###### Proof.

Let $W$ be skewed with regard to attributes in $A$, then

$\displaystyle\exists A_{i},A_{j}\in A:mean_{\bm{w}\in
W}s(\bm{w},A_{i})>mean_{\bm{w}\in W}s(\bm{w},A_{j}).$ (15)

In that case

$\displaystyle\exists w\in W,A_{i},A_{j}\in A:s(\bm{w},A_{i})\neq
s(\bm{w},A_{j}).$ (16)

Further, let $b(W,A)$ be a unbiased-trustworthy bias score function. From
Definition 2 directly follows that $b(W,A)\neq b_{0}$ given a skewed set of
words $W$.

Analogously, for $W=W_{1}\cup W_{2}$ with $W_{1}$ and $W_{2}$ reflecting
different stereotypes with regard to (at least) two attributes $A_{i}$,
$A_{j}\in A$ and $b(W,A)$ an unbiased-trustworthy bias score function:

$\displaystyle\exists\bm{w_{1}}\in W_{1},\bm{w_{2}}\in
W_{2}:s(\bm{w_{1}},A_{i})-s(\bm{w_{1}},A_{j})\neq
s(\bm{w_{2}},A_{i})-s(\bm{w_{2}},A_{j})$ (17) $\displaystyle\Rightarrow$
$\displaystyle\exists w\in\\{\bm{w_{1}},\bm{w_{2}}\\}:s(\bm{w},A_{i})\neq
s(\bm{w},A_{j})$ (18) $\displaystyle\Rightarrow$ $\displaystyle b(W,A)\neq
b_{0}$ (19)

∎

###  3.4 Analysis of the WEAT Score

In the following, we detail properties of the WEAT score in light of the
definitions stated above. First, we focus on the word-wise biases as reported
by $s(\bm{w},A,B)$.

######  Theorem 1.

The bias score function $s(\bm{w},A,B)$ of WEAT is not magnitude-comparable.

###### Proof.

We consider an example with $A=\\{\bm{she}\\},B=\\{\bm{he}\\}$ and
$\bm{w}=\bm{secretary}$ as depicted exemplarily in Figure 3. Thereby, let
$\theta$ be the angle between $\bm{he}$ and $\bm{she}$ and $\alpha$ the angle
between $\bm{she}$ and $\bm{secretary}$ (compare to Figure). Now, the extrema
of the function

$\displaystyle|s(\bm{w},A,B)|$ $\displaystyle=$
$\displaystyle|cos(\alpha)-cos(\theta-\alpha)|$ (20)

depend on $\theta$, i.e. $max_{\alpha}|cos(\alpha)-cos(\theta-\alpha)|$ will
differ depending on $\theta$, which in turn is given by the choice of A. ∎

Figure 3: Simplified visualization of gender stereotypes in embeddings. The
occupation vectors are biased if $0<\alpha<\frac{\theta}{2}$ or
$0<\beta<\frac{\theta}{2}$.

The bias score function $s(\bm{w},A,B)$ of WEAT is unbiased-trustworthy. For
the proof see Appendix 3.4.

###### Proof.

This follows directly from the definition of $s(\bm{w},A,B)$ (equation (3)):

$\displaystyle s(\bm{w},A,B)=s(\bm{w},A)-s(\bm{w},B)=0$ (21)
$\displaystyle\iff$ $\displaystyle s(\bm{w},A)=s(\bm{w},B)$ (22)

∎

Next, we focus on the properties of the effect size $d(X,Y,A,B)$. Note that
the effect size is not specified for cases, where
$s(\bm{w},A,B)=s(\bm{w^{\prime}},A,B)\forall\bm{w},\bm{w^{\prime}}\in X\cup Y$
due to its denominator. This is highly problematic considering Definition 2,
which states that a bias score should be $0$ in that specific case (which
implies perfect fairness). Furthermore, the following theorem shows that WEAT
can report no bias even if the embeddings contain associations with the bias
attributes.

######  Theorem 2.

The effect size $d(X,Y,A,B)$ of WEAT is not unbiased-trustworthy.

###### Proof.

For the WEAT score $b_{0}=0$.

For four words $\bm{w}_{1},\bm{w}_{2},\bm{w}_{3},\bm{w}_{4}$ and
$s(\bm{w}_{1},A,B)=s(\bm{w}_{3},A,B)$ and
$s(\bm{w}_{2},A,B)=s(\bm{w}_{4},A,B)$, the effect size

$\displaystyle
d(\\{\bm{w}_{1},\bm{w}_{2}\\},\\{\bm{w}_{3},\bm{w}_{4}\\},A,B)=$ (23)
$\displaystyle\frac{(s(\bm{w}_{1},A,B)+s(\bm{w}_{2},A,B))-(s(\bm{w}_{3},A,B)+s(\bm{w}_{4},A,B))}{2\cdot\mathrm{stddev}_{\bm{w}\in\\{\bm{w}_{1},\bm{w}_{2},\bm{w}_{3},\bm{w}_{4}\\}}s(\bm{w},A,B)}$
(24)

is $0$, if $s(\bm{w}_{1},A,B)\neq s(\bm{w}_{2},A,B)$ (otherwise $d$ is not
defined). Now, for the simple case $A=\\{\bm{a}\\},B=\\{\bm{b}\\}$ and
assuming all vectors having length $1$, we see

$\displaystyle s(\bm{w}_{1},A,B)=s(\bm{w}_{3},A,B)$ $\displaystyle\iff$
$\displaystyle\bm{a}\cdot\bm{w}_{1}-\bm{b}\cdot\bm{w}_{1}=\bm{a}\cdot\bm{w}_{3}-\bm{b}\cdot\bm{w}_{3}$
$\displaystyle\iff$
$\displaystyle\bm{a}\cdot(\bm{w}_{1}-\bm{w}_{3})-\bm{b}\cdot(\bm{w}_{1}-\bm{w}_{3})=0$
$\displaystyle\iff$
$\displaystyle(\bm{a}-\bm{b})\cdot(\bm{w}_{1}-\bm{w}_{3})=0.$ (25)

This implies that, if the two vectors $\bm{a}-\bm{b}$ and
$\bm{w}_{1}-\bm{w}_{3}$ are orthogonal (and e.g. $s(\bm{w}_{2},A,B)=0$), the
WEAT score returns 0. In this case, there exist
$\bm{a},\bm{b},\bm{w}_{1},\bm{w}_{3}$ with
$s(\bm{w}_{1},A,B)=s(\bm{w}_{3},A,B)\neq 0$ and accordingly
$s(\bm{w}_{1},A)\neq s(\bm{w}_{1},B)$. See Figure 4 for an example, with
$\bm{a}=$ he, $\bm{b}=$ she, $\bm{w}_{1}=$ engineer, $\bm{w}_{3}=$ secretary.
∎

Figure 4: This illustrates a case, where WEAT would report no bias since
secretary and engineer have the same relative similarity towards he and she.

In order to show that the effect size of WEAT is magnitude-comparable, we need
the following lemma.

Let $x_{1},...,x_{n}\in\mathbb{R}$ be real numbers. Let
$\hat{\mu},\hat{\sigma}$ denote the empirical estimate of mean and standard
deviation of the $x_{i}$. Then, for any selection of indices
$i_{1},...,i_{m}$, with $i_{j}\neq i_{k}$ for $j\neq k$, the following bound
holds

$\displaystyle\left|\sum_{j=1}^{m}\frac{x_{i_{j}}-\hat{\mu}}{\hat{\sigma}}\right|\leq\sqrt{m\cdot(n-m)}.$

Furthermore, for $0<m<n$ the bound is obtained if and only if all selected
resp. non-selected $x_{i}$ have the same value, i.e.
$x_{i_{j}}=\hat{\mu}+s\sqrt{\frac{n-m}{m}}\hat{\sigma}$ and all other
$x_{k}=\hat{\mu}-s\sqrt{\frac{m}{n-m}}\hat{\sigma}$ with $s\in\\{-1,1\\}$. For
the proof see Appendix 3.4.

###### Proof.

For cases $m=0$ or $m=n$ the statement is trivial. So assume $0<m<n$. Let
$f(x)=ax+b$ be an affine function. Then the images of $x_{i}$ under $f$ have
mean $a\hat{\mu}+b$ and standard deviation $|a|\hat{\sigma}$. On the other
hand, we have

$\displaystyle\frac{f(x_{i})-(a\hat{\mu}+b)}{|a|\hat{\sigma}}$
$\displaystyle=\frac{(ax_{i}+b)-(a\hat{\mu}+b)}{|a|\hat{\sigma}}=\text{sgn}(a)\frac{x_{i}-\hat{\mu}}{\hat{\sigma}}.$

Thus, applying $f$ does not change the bound and therefore we may reduce to
case of $\hat{\mu}=0$ and $\hat{\sigma}=1$. This allows us to rephrase the
problem of finding the maximal bound as an quadratic optimization problem:

$\displaystyle\min\quad$ $\displaystyle\bm{s}^{\top}\bm{x}$ s.t.
$\displaystyle\bm{x}^{\top}\bm{x}=n$ $\displaystyle\bm{1}^{\top}\bm{x}=0,$

where $\bm{s}=(1,...,1,0,...,0)^{\top}$, $\bm{x}=(x_{1},...,x_{n})^{\top}$ and
$\bm{1}$ denotes the vector consisting of ones only. Notice, that we assumed
w.l.o.g. that $i_{1},...,i_{m}=1,...,m$. Furthermore, we made use of the
symmetry properties to replace $\max|\bm{s}^{\top}\bm{x}|$ by the minimizing
statement above, $\hat{\mu}=0$ is expressed by the last and $\hat{\sigma}=1$
by the first constrained (recall that
$\hat{\sigma}=\sqrt{1/n\bm{x}^{\top}\bm{x}-\hat{\mu}^{2}}$). Notice, that
$\nabla_{x}\bm{x}^{\top}\bm{x}-n=2\bm{x}$ and
$\nabla_{x}\bm{1}^{\top}\bm{x}=\bm{1}$ are linear dependent if and only if
$\bm{x}=a\bm{1}$ for some $a\in\mathbb{R}$, thus, as
$0=a\bm{1}^{\top}\bm{1}=an$ if and only if $a=0$ and
$(0\bm{1})^{\top}(0\bm{1})=0$, there is no feasible $\bm{x}$ for which the
KKT-conditions do not hold and we may therefore use them to determine all the
optimal points.

The Lagrangien of the problem above and its first two derivatives are given by

$\displaystyle L(\bm{x},\lambda_{1},\lambda_{2})$
$\displaystyle=\bm{s}^{\top}\bm{x}-\lambda_{1}(\bm{x}^{\top}I\bm{x}-n)-\lambda_{2}\bm{1}^{\top}\bm{x}$
$\displaystyle\nabla_{x}L(\bm{x},\lambda_{1},\lambda_{2})$
$\displaystyle=\bm{s}-2\lambda_{1}\bm{x}-\lambda_{2}\bm{1}$
$\displaystyle\nabla^{2}_{x,x}L(\bm{x},\lambda_{1},\lambda_{2})$
$\displaystyle=-2\lambda_{1}I.$

We can write $\nabla_{x}L(\bm{x},\lambda_{1},\lambda_{2})=0$ as the following
linear equation system:

$\displaystyle\begin{bmatrix}2x_{1}&1\\\ 2x_{2}&1\\\ \vdots&\vdots\\\
2x_{n}&1\end{bmatrix}\begin{bmatrix}\lambda_{1}\\\ \lambda_{2}\end{bmatrix}$
$\displaystyle=\underbrace{\begin{bmatrix}1\\\ 1\\\ \vdots\\\
0\end{bmatrix}}_{=\bm{s}}.$

Subtracting the first row from row $2,...,m$ and row $m+1$ from row
$m+2,...,n$ we see that $2(x_{k}-x_{1})\lambda_{1}=0$ for $k=1,...,m$ and
$2(x_{k}-x_{m+1})\lambda_{1}=0$ for $k=m+2,...,n$, which either implies
$\lambda_{1}=0$ or $x_{1}=x_{2}=...=x_{m}$ and $x_{m+1}=x_{m+2}=...=x_{n}$.
However, assuming $\lambda_{1}=0$ would imply that $\lambda_{2}=1$ from the
first row and $\lambda_{2}=0$ from the $m+1$th row, which is a contradiction.
Thus, we have $x_{1}=x_{2}=...=x_{m}$ and $x_{m+1}=x_{m+2}=...=x_{n}$. But the
second constraint from the optimization problem can then only be fulfilled if
$mx_{1}+(n-m)x_{m+1}=0$ and this implies $x_{m+1}=-\frac{m}{n-m}x_{1}$. In
this case the first constraint is equal to
$n=mx_{1}^{2}+(n-m)\left(\frac{m}{n-m}x_{1}\right)^{2}$, which has the
solution $x_{1}=\pm\sqrt{\frac{n-m}{m}}$.

Set
$\bm{x}^{*}=(-\sqrt{\frac{n-m}{m}},...,-\sqrt{\frac{n-m}{m}},\sqrt{\frac{m}{n-m}},...,\sqrt{\frac{m}{n-m}})$.
Then $\bm{x}^{*}$ and $-\bm{x}^{*}$ are the only possible KKT points as we
have just seen. Plugging $\bm{x}^{*}$ into the equation system above and
solving for $\lambda_{1/2}$ we obtain

$\displaystyle\lambda_{1}^{*}$
$\displaystyle=-\frac{1}{2\left(\sqrt{\frac{m}{n-m}}+\sqrt{\frac{n-m}{m}}\right)}$
$\displaystyle\lambda_{2}^{*}$
$\displaystyle=\frac{\sqrt{\frac{m}{n-m}}}{\sqrt{\frac{m}{n-m}}+\sqrt{\frac{n-m}{m}}}$

Now, as
$\nabla^{2}_{x,x}L(\bm{x}^{*},\lambda_{1}^{*},\lambda_{2}^{*})=\left(\sqrt{\frac{m}{n-m}}+\sqrt{\frac{n-m}{m}}\right)^{-1}I$
is positive definite, we see that $\bm{x}^{*}$ is a global optimum, indeed.
The statement follows. ∎

The effect size $d(X,Y,A,B)$ of WEAT with
$X=\\{\bm{x}_{1},\ldots,\bm{x}_{m}\\},Y=\\{\bm{y}_{1},\ldots,\bm{y}_{m}\\}$ is
magnitude-comparable. For the proof see Appendix 3.4.

###### Proof.

With $c_{i}=s(\bm{x_{i}},A,B)$, $c_{i+m}=s(\bm{y_{i}},A,B)$, $n=2m$,
$\hat{\mu}=1/n\sum^{n}_{i=1}c_{i}$ and
$\hat{\sigma}=\sqrt{1/n\sum^{n}_{i=1}(c_{i}-\mu)^{2}}$, we have

$\displaystyle d$
$\displaystyle=\frac{1/m\sum^{m}_{i=1}c_{i}-1/m\sum^{2m}_{i=m+1}c_{i}}{\hat{\sigma}}$
(26)
$\displaystyle=\frac{\sum^{m}_{i=1}c_{i}-\sum^{2m}_{i=m+1}c_{i}+\sum^{m}_{i=1}c_{i}-\sum^{m}_{i=1}c_{i}}{m\hat{\sigma}}$
$\displaystyle=\frac{2\sum^{m}_{i=1}c_{i}-2m\hat{\mu}}{m\hat{\sigma}}$
$\displaystyle=\frac{2}{m}\sum^{m}_{i=1}\frac{c_{i}-\hat{\mu}}{\hat{\sigma}}\in[-2,2]$
(27)

where the last statement follows from Lemma 3.4 with
$\sum^{m}_{i=1}\frac{c_{i}-\hat{\mu}}{\hat{\sigma}}\in[-m,m]$. The extreme
value $\pm 2$ is reached if $c_{1}=\ldots=c_{m}=-c_{m+1}=\ldots=-c_{2m}$,
which can be obtained by setting
$\bm{x}_{1}=\ldots=\bm{x}_{m}=-\bm{y}_{1}=\ldots=-\bm{y}_{m}$, independently
of $A$ and $B$ as long as $A\neq B$ and $\sum_{\bm{a}_{i}\in
A}\bm{a}_{i}/\|\bm{a}_{i}\|\neq 0\neq\sum_{\bm{b}_{i}\in
B}\bm{b}_{i}/\|\bm{b}_{i}\|$. ∎

The proof of Theorem 3.4 shows that the effect size reaches its extreme values
only if all $x\in X$ achieve the same similarity score $s(\bm{x},A,B)$ and
$s(\bm{y},A,B)=-s(\bm{x},A,B)\;\forall\;y\in Y$, i.e. the smaller the variance
of $s(\bm{x},A,B)$ and $s(\bm{y},A,B)$ the higher the effect size. For one
thing, this limits WEAT’s predictability to the partitioning of neutral words
into $X$ and $Y$. On the other hand, the differences $mean_{\bm{x}\in
X}s(\bm{x},A,B)-mean_{\bm{y}\in Y}s(\bm{y},A,B)$ does not affect the effect
size values. The dependence on the variance explains the behavior shown in
Section 3.1. This shows that we cannot take low effect sizes as a guarantee
for low biases in the embeddings.

Finally, we probe the effect size for skew and stereotype sensitivity: The
effect size $d(X,Y,A,B)$ of WEAT is stereotype-sensitive assuming that the
groups $X$ and $Y$ correctly classify for the direction of stereotype, i.e.
all words in $X$ are closer to the attribute set $A$ (compared to $B$) than
words in $Y$:

$\displaystyle
s(\bm{x},A)-s(\bm{x},B)>s(\bm{y},A)-s(\bm{y},B)\;\forall\bm{x}\in X,\bm{y}\in
Y,$ (28)

or vice-versa:

$\displaystyle
s(\bm{x},A)-s(\bm{x},B)<s(\bm{y},A)-s(\bm{y},B)\;\forall\bm{x}\in X,\bm{y}\in
Y$ (29)

For the proof see Appendix 3.4.

###### Proof.

We transform the effect size (Equation (4)) into

$\displaystyle
d(X,Y,A,B)=\frac{mean_{(\bm{x},\bm{y})\in(X,Y)}s(\bm{x},A,B)-s(\bm{y},A,B)}{stddev_{\bm{w}\in
X\cup Y}s(w,A,B)}.$ (30)

From our assumption follows that $|d(X,Y,A,B)|\neq 0$. ∎

The bias score function $d(X,Y,A,B)$ of WEAT is not skew-sensitive. For the
proof see Appendix 3.4.

###### Proof.

Let $X\cup Y=\\{\bm{x},\bm{y}\\}$ be a skewed set with
$s(\bm{w},A)>s(\bm{w},B)\forall\bm{w}\in X\cup Y$. Assuming that
$s(\bm{x},A,B)=s(\bm{y},A,B)\neq 0$, which depicts a skew but no stereotype,
from the definition of the effect size follows:

$\displaystyle d(X,Y,A,B)=0,$ (31)

which contradicts equation (3) with $b_{0}=0$ for WEAT. ∎

To conclude this theorems, we found that the word-wise bias reported by WEAT
is in fact unbiased-trustworthy, but not magnitude-comparable. The effect
size, which is applied to measure bias of two sets of target words, is
magnitude-comparable, but not unbiased-trustworthy. This is due to the fact
that the effect size is sensitive to stereotypical differences between the
sets $X$ and $Y$ (which is exactly the intuition behind WEAT), but it is not
skew-sensitive. Furthermore the magnitude of the effect size depends on the
inner-group variance of $s(\bm{w},A,B)$ for groups $X$ and $Y$, but not one
the difference between those groups, which should be taken into account when
interpreting it.
This shows that WEAT can only be used to quantify stereotypical biases as
anticipated by the groups $X$ and $Y$, while skew and stereotypes that
contradict the partitioning will not be detected. And most importantly, while
high effect sizes are a certain indicator of bias, low effect sizes are not
that meaningful. Hence, when it comes to quantitatively comparing biases
between different embeddings or evaluating debiasing methods, we argue that
WEAT is not sufficient. At most, it could be used as an additional measure for
specific stereotypes using, for instance, the tests from weat. Of course, if
ones goal is to only prove the presence of bias instead of quantifying it,
WEAT is still useful..

###  3.5 Analysis of the MAC Score

On contrary to the bias definitions summarized in Section 2.1, the MAC score
follows a slightly different intuition. Instead of contrasting over the
association of words towards different attributes (as WEAT does) or measuring
the correlation with a bias direction (Direct Bias), it depicts the average
correlation of neutral words and bias attributes. In the following we show
several situations where the metric thereby contradicts our bias definition.
This includes detecting bias on fair embeddings and measuring no bias in
biased embeddings.

The bias score function $MAC(T,A)$ of MAC is not unbiased-trustworthy. For the
proof see Appendix 3.5.

###### Proof.

An ideal MAC score, indicating fairness, would be $b_{0}=\pm 1$. We again
consider the example case depicted in Figure 3 with
$\bm{w}=\bm{secretary},A_{1}=\\{\bm{she}\\},A_{2}=\\{\bm{he}\\}$.

First, consider $\theta=\pi$ and $0<\alpha<\frac{\theta}{2}$. In that case,
$\bm{secretary}$ would be closer to $\bm{she}$ than $\bm{he}$, which reflects
a biased representation and does not satisfy Eq. (11). Yet, if we apply the
MAC score $MAC(T,A)$ with $T=\\{\bm{w}\\},A=\\{A_{1},A_{2}\\}$, due to the
symmetry of the cosine function we get

$\displaystyle MAC(T,A)$
$\displaystyle=\frac{1}{2}((1-cos(\alpha))+(1-cos(\pi-\alpha)))$ (32)
$\displaystyle=1,$ (33)

which implies the word secretary to be perfectly fair embedded.

As a second example, if we considered $\alpha=\frac{\theta}{2},\theta\neq\pi$,
and thus $\bm{secretary}$ being equidistant to $A_{1}$ and $A_{2}$. Yet, we
would get

$\displaystyle MAC(T,A)$
$\displaystyle=\frac{1}{2}((1-cos(\alpha))+(1-cos(\theta-\alpha)))$ (34)
$\displaystyle=1-cos\left(\frac{\theta}{2}\right).$ (35)

Now, due to $\theta\neq\pi$, we still measure a bias, although based on eq.
(11) the occupation is unbiased towards both gender words. Also, the larger
$|\theta-\pi|$ the more extreme is the bias according to the MAC score. ∎

The bias score function $MAC(T,A)$ of MAC is magnitude-comparable if the
number of attributes is smaller than the embedding dimension $d$. For the
proof see Appendix 3.5.

###### Proof.

The MAC score reports biases in $[-1,1]$. Let all embeddings be represented in
a $d$-dimensional vector space
($\bm{t},\bm{a}\in\mathbb{R}^{d}\;\forall\;\bm{t}\in T,\bm{a}\in
A_{i}\;\forall\;A_{i}\in A$). For arbitrary attribute vectors $\bm{a}$ with
the number of attributes $n_{A}=\sum_{A_{i}\in A}|A_{i}|<d$, there exists a
subspace $V\subset\mathbb{R}^{d}$ that is orthogonal to all attributes
$\bm{a}$. Hence for any target word $\bm{t}\in V$ MAC will reach one of its
extreme values. ∎

The bias score function $MAC(T,A)$ of MAC is neither stereotype-sensitive nor
skew-sensitive. For the proof see Appendix 3.5.

###### Proof.

We showed in the proof of Theorem 3.5 that if the angle between both
attributes is $\pi$, the average bias of an individual word will be $b_{0}=1$.
Now consider a purely stereotypical scenario with $\bm{secretary}$ and
$\bm{engineer}$ as depicted in Figure 3 with $\theta=\pi$, and $\alpha=\beta$.
With both individual words’ MAC scores being $1$, the MAC of both words will
also be $1$.
On the other hand, a purely skewed representation with
$\bm{engineer}=\bm{secretary}$ would also result in a MAC score of $1$. ∎

Considering these theorems, we strongly argue against using the MAC score at
all, since it is neither reliable to measure skew nor stereotype. Furthermore,
it might falsely detect a bias simply because the embedding vectors are
represented closely in general.

###  3.6 Analysis of the Direct Bias

The Direct Bias is magnitude-comparable for $c\geq 0$. For the proof see
Appendix 3.6.

###### Proof.

For $c\geq 0$ the bias of one word $|\cos(\bm{w},\bm{g})|^{c}$ is in $[0,1]$.
Calculating the mean over all words in $W$ does not change this bound. ∎

######  Theorem 3.

The Direct Bias is not unbiased-trustworthy.

###### Proof.

For the Direct Bias $b_{0}=0$ indicates no bias. Consider a setup with two
attribute sets $A=\\{\bm{a_{1}},\bm{a_{2}}\\}$ and
$C=\\{\bm{c_{1}},\bm{c_{2}}\\}$ (we cannot call our attribute set $B$ as usual
because this could be confused with the bias subspace $B$ or bias direction
$b$ from the Direct Bias).
Using the notation from Section 2.2.3 this gives us two defining sets
$D_{1}=\\{\bm{a_{1}},\bm{c_{1}}\\}$ $D_{2}=\\{\bm{a_{2}},\bm{c_{2}}\\}$. Let
$\bm{a_{1}}=(-x,rx)^{T}=-\bm{c_{1}},\bm{a_{2}}=(-x,-rx)^{T}=-\bm{c_{2}}$ and
$r>1$.
The bias direction is obtained by computing the first principal component over
all $(\bm{a_{i}}-\bm{\mu}_{i})$ and $(\bm{c_{i}}-\bm{\mu}_{i})$ with
$\bm{\mu}_{i}=\frac{\bm{a_{i}}+\bm{c_{i}}}{2}=0$. Due to $r>1$,
$\bm{b}=(0,1)^{T}$ is a valid solution for the 1st principal component as it
maximizes the variance

$\displaystyle\bm{b}=argmax_{\|\bm{v}\|=1}\sum_{i}(\bm{v}\cdot\bm{a_{i}})^{2}+(\bm{v}\cdot\bm{c_{i}})^{2}.$
(36)

According to the definition in Section 2.1, any word $\bm{w}=(0,w_{y})^{T}$
would be considered neutral to groups $A$ and $C$ with
$s(\bm{w},A)=s(\bm{w},C)$ and being equidistant to each word pair
$\\{a_{i},c_{i}\\}$.
But with the bias direction $\bm{b}=(0,1)^{T}$ the Direct Bias would report a
maximal bias of $1$ instead of $b_{0}=0$, which contradicts Definition 2.
On the other hand, we would consider a word $\bm{w}=(w_{x},0)^{T}$ maximally
biased, but the Direct Bias would report no bias. ∎

Note that the example shown in Theorem 3 is an extreme case. Yet, it shows
that a bias direction obtained by the PCA does not necessarily represent
individual bias directions appropriately. This can lead to both over- and
underestimation bias. If the variance inside the attribute groups is higher
than the differences between the attributes, the bias direction (or subspace)
is likely to be misleading. (Note that this critique of the Direct Bias does
not affect the debiasing algorithm of bolukbasi, due to the equalize step.)

##  4 Proposed Metric

To address the shortcomings of the existing bias scores, we propose a new
score based on the before mentioned bias definition: Scoring Association Means
of Word Embeddings (SAME). It has a similar intuition to WEAT, in terms of
measuring polarity between attribute sets, but fulfills the criteria set up in
the last chapter. In a first step we detail SAME for a setting with binary
attributes and then extend it to the multiple attributes case.

###  4.1 The binary case

We use one set of target words $W$ and measure the association with two
attribute sets $A_{i}$ and $A_{j}$. Furthermore, we assume that each attribute
vector $\bm{a_{i}}\in A_{i}$ is normalized to unit length, so that
$cos(\bm{w},\bm{a_{i}})=\frac{\bm{w}}{|\bm{w}|}\cdot\bm{a_{i}}$. Then, we can
write the similarity of a word $\bm{w}$ towards attribute set $A_{i}$ as

$\displaystyle s(\bm{w},A_{i})$ $\displaystyle=$
$\displaystyle\frac{1}{|A_{i}|}\sum_{\bm{a_{i}}\in
A_{i}}\frac{\bm{w}}{\|\bm{w}\|}\cdot\bm{a_{i}}$ (37) $\displaystyle=$
$\displaystyle\frac{\bm{w}}{\|\bm{w}\|}\cdot\left(\frac{1}{|A_{i}|}\sum_{\bm{a_{i}}\in
A_{i}}\bm{a_{i}}\right)$ (38) $\displaystyle=$
$\displaystyle\frac{\bm{w}}{\|\bm{w}\|}\cdot\hat{\bm{a_{i}}}$ (39)

with $\hat{\bm{a_{i}}}=\left(\frac{1}{|A_{i}|}\sum_{\bm{a_{i}}\in
A_{i}}\bm{a_{i}}\right)$.

Similar to WEAT, we define a pairwise bias comparing the similarity of a word
$w$ towards two attribute sets $A_{i}$ and $A_{j}$ with
$\hat{\bm{a_{i}}}\neq\hat{\bm{a_{j}}}$:

$\displaystyle b(\bm{w},A_{i},A_{j})$ $\displaystyle=$
$\displaystyle\frac{s(\bm{w},A_{i})-s(\bm{w},A_{j})}{\|\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}}\|}$
(40) $\displaystyle=$
$\displaystyle\frac{\frac{\bm{w}}{\|\bm{w}\|}\cdot\hat{\bm{a_{i}}}-\frac{\bm{w}}{\|\bm{w}\|}\cdot\hat{\bm{a_{j}}}}{\|\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}}\|}$
(41) $\displaystyle=$
$\displaystyle\frac{\bm{w}\cdot(\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}})}{\|\bm{w}\|\;\|\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}}\|}$
(42) $\displaystyle=$ $\displaystyle
cos(\bm{w},\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}}).$ (43)

Contrary to equation (3) of WEAT we normalize the term, resulting in bias
scores in $[-1,1]$ independent of the attributes. By transforming the
equation, we can show that it has a similar notion to the Direct Bias from
bolukbasi, measuring the correlation of words with a bias direction between
two attributes. The only difference is that we obtain the pairwise bias
direction by averaging over individual directions instead of using the PCA.

The term $b(\bm{w},A_{i},A_{j})$ could be contrasted over pairs $\bm{w}_{1}\in
W_{1}$, $\bm{w}_{2}\in W_{2}$, similarly as done in the WEAT score, but this
would results in the potential problems stated in Theorem 2 / Figure 4. Hence,
to achieve a trustworthy and comparable metric for binary bias attributes, we
propose to take the mean absolute values of word-wise biases. This results in
the following bias score for a set of target words $W$:

$\displaystyle b(W,A_{i},A_{j})$ $\displaystyle=$
$\displaystyle\frac{1}{|W|}\sum_{\bm{w}\in W}|b(\bm{w},A_{i},A_{j})|.$ (44)

###  4.2 The multiple attributes case

Now we extend SAME to cases with multiple attributes. Let
$A=\\{A_{0},...,A_{n}\\}$ contain more than $2$ attribute sets. To measure the
bias with respect to all attributes in $A$, we summarize the pairwise biases
for all attribute pairs $(A_{0},A_{i}),i\geq 1$, measuring the correlation
with each pairwise bias direction. Since these pairwise bias directions might
correlate with each other, for $i>1$, we remove the parts of $w$ correlating
with previous bias directions and thus only measure additional biases
independent of previous bias directions. This results in the following
iterative approach for $n+1$ attribute groups:

$\displaystyle b(\bm{w},A)$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{n}|b(\bm{w_{i}},A_{0},A_{i})|\;\|\bm{w_{i}}\|$ (45)

with $\bm{w_{1}}=\frac{\bm{w}}{|\bm{w}|}$ and
$\bm{w_{i+1}}=\bm{w_{i}}-(\bm{w_{i}}\cdot\frac{\hat{\bm{a_{0}}}-\hat{\bm{a_{i}}}}{\|\hat{\bm{a_{0}}}-\hat{\bm{a_{i}}}\|})\frac{\hat{\bm{a_{0}}}-\hat{\bm{a_{i}}}}{\|\hat{\bm{a_{0}}}-\hat{\bm{a_{i}}}\|}$.
Other attribute pairs $(A_{j},A_{k})$ with $j,k>0$ can be ignored since their
bias directions can be fully described by a linear combination of bias
directions $\hat{\bm{a_{0}}}-\hat{\bm{a_{i}}}$.

The overall bias of all target words in $W$ is then:

$\displaystyle b(W,A)$ $\displaystyle=$
$\displaystyle\frac{1}{|W|}\sum_{\bm{w}\in W}b(\bm{w},A).$ (46)

###  4.3 Analysis of the metric

The following theorems and their proofs detail properties of SAME in light of
the above stated definitions. Due to the changes mentioned above, the scores
are both unbiased-trustworthy and magnitude-comparable. Hence, both the binary
and multi attribute scores are reliable metrics to quantify bias in embeddings
and can be compared between different embeddings. Since we use the mean
absolute values, the magnitude of bias reported by these scores is easily
interpretable.

The bias score function $b(W,A)$ is unbiased-trustworthy. For the proof see
Appendix 4.3.

###### Proof.

The bias score indicating no bias is $b_{0}=0$. First, we can state that

$\displaystyle b(\bm{w},A_{i},A_{j})=0\iff s(\bm{w},A_{i})=s(\bm{w},A_{j}),$
(47)

which directly follows from the definition of $b(\bm{w},A_{i},A_{j})$.
Further, for $\|\bm{w}\|\neq 0$, it follows

$\displaystyle b(\bm{w},A)=\sum_{A_{j}\in
A,j>0}|b(\bm{w},A_{0},A_{j})|\|\bm{w}\|=0$ (48) $\displaystyle\iff$
$\displaystyle b(\bm{w},A_{0},A_{j})=0\;\forall A_{j}\in A.$ (49)

and

$\displaystyle b(W,A)=0\iff b(\bm{w},A)=0\;\forall\bm{w}\in W.$ (50)

∎

According to Theorem 3.3, this shows that SAME is sensitive to both skew and
stereotype.

The bias score function $b(W,A)$ is magnitude-comparable. For the proof see
Appendix 4.3.

###### Proof.

The minimal score is $0$, the maximum $1$. We can state that

$\displaystyle max(b(W,A))=max(b(\bm{w},A)),$ (51) $\displaystyle
min(b(W,A))=min(b(\bm{w},A))$ (52)

which is derived directly from the definition of $b(W,A)$. So, we need to
prove that $b(\bm{w},A)$ is magnitude-comparable.
In 4.3, we stated that

$\displaystyle b(\bm{w},A)=0\iff s(\bm{w},A_{0})=s(\bm{w},A_{j})\;\forall
A_{j}\in A,$ (53)

which only depends on the relation of $\bm{w}$ to the attribute sets, but not
the relation between attribute sets. Furthermore, we can state that

$\displaystyle b(\bm{w},A)\geq 0$ (54)

purely based on its definition.

Regarding the maximum score, we can state that

$\displaystyle
b(\bm{w},A_{i},A_{j})=cos(\bm{w},\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}})\in[-1,1]$
(55)

with

$\displaystyle|b(\bm{w},A_{i},A_{j})|=1\iff\bm{w}=\pm\frac{\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}}}{\|\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}}\|}.$
(56)

Hence, we need to show that $b(\bm{w},A)\leq 1$. For a binary setting with
$A=\\{A_{0},A_{1}\\}$ this is trivial, since

$\displaystyle
b(\bm{w},A)=|b(\bm{w},A_{0},A_{1})|=|cos(\bm{w},\hat{\bm{a_{0}}}-\hat{\bm{a_{1}}})|\leq
1.$ (57)

Now we show that adding another attribute does not increase the maximum value
of $b(\bm{w},A)$, i.e. if we have $A=\\{A_{0},...,A_{i}\\}$ with
$b(\bm{w},A)\leq 1$ and add another attribute $A_{i+1}$ with
$\hat{\bm{a_{i+1}}}\neq\hat{\bm{a_{k}}}\;\forall k\in\\{1...i\\}$, then

$\displaystyle b(\bm{w},A\cup\\{A_{i+1}\\})\leq 1.$ (58)

From the definition of $b(\bm{w},A)$, we know that

$\displaystyle
b(\bm{w},A\cup\\{A_{i+1}\\})=b(\bm{w},A)+|b(\bm{w_{i+1}},A_{0},A_{i+1})|*\|\bm{w}_{i+1}\|$
(59)

with

$\displaystyle\bm{w_{i+1}}=\bm{w_{i}}-cos(\bm{w_{i}},\hat{\bm{a_{0}}}-\hat{\bm{a_{i}}})\frac{\hat{\bm{a_{0}}}-\hat{\bm{a_{i}}}}{|\hat{\bm{a_{0}}}-\hat{\bm{a_{i}}}|}$
(60) $\displaystyle\iff$
$\displaystyle\|\bm{w_{i+1}}\|=\|\bm{w_{i}}\|-cos(\bm{w_{i}},\hat{\bm{a_{0}}}-\hat{\bm{a_{i}}})*\|\bm{w_{i}}\|$
(61) $\displaystyle\iff$ $\displaystyle\|\bm{w_{i+1}}\|=1-b(\bm{w},A)$ (62)
$\displaystyle\iff$
$\displaystyle|b(\bm{w_{i+1}},A_{0},A_{i+1})|*\|\bm{w_{i+1}}\|\leq
1-b(\bm{w},A)$ (63)

From there we can directly conclude that

$\displaystyle b(\bm{w},A\cup\\{A_{i+1}\\})\leq 1$ (64)

∎

###  4.4 Measuring Skew and Stereotype

Using SAME as defined in the last subsections, we can reliably quantify and
compare biases in different embeddings or domains. However, we only get one
measurement of the average or word-wise bias magnitude. To get a closer look
on how a set of target words is biased, we propose two additional metrics to
determine the skew and stereotype.
The basic intuition is that the skew is given by the mean of the word-wise
bias distribution and the stereotype by the standard deviation. In the
following we detail how to calculate this in the case of binary and multiple
attributes.

####  4.4.1 The binary case

The Skew of words in $W$ given the two attribute sets $A_{i}$, $A_{j}$ is
simply the mean of word-wise biases. Contrary to Equation 44 we obtain a sign
indicating the direction of bias.

$\displaystyle b_{skew}(W,A_{i},A_{j})$ $\displaystyle=$
$\displaystyle\frac{1}{|W|}\sum_{\bm{w}\in W}b(\bm{w},A_{i},A_{j}).$ (65)

Respectively, the Stereotype is given by the standard deviation:

$\displaystyle b_{stereo}(W,A_{i},A_{j})$ $\displaystyle=$
$\displaystyle\frac{1}{|W|}\sqrt{\sum_{\bm{w}\in
W}\left(b(\bm{w},A_{i},A_{j})-b_{skew}(W,A_{i},A_{j})\right)^{2}}.$ (66)

####  4.4.2 The multiple attributes case

To obtain a meaningful and interpretable Skew and Stereotype in terms of $n>2$
attribute sets, we suggest looking at all pairwise bias directions, i.e. all
pairs ($A_{i}$, $A_{j}$) $A_{i},A_{j}\in A,i\neq j$ or contrasting over each
group $A_{i}$ compared to the union of all other sets
$\bigcup\limits_{A_{j}\in A,j\neq i}A_{j}$. This allows to identify the major
directions of skew or stereotype, indicating attribute groups that are
particularly prone to biases.

####  4.4.3 Analysis of Skew and Stereotype Extensions

The Skew extension $b_{skew}(W,A_{i},A_{j})$ is skew-sensitive, but not
stereotype-sensitive. For the proof see Appendix 4.4.3.

###### Proof.

The bias score indicating no bias is $b_{0}=0$. Let $A_{i},A_{j}$ be attribute
sets with

$\displaystyle mean_{\bm{w}\in W}s(\bm{w},A_{i})>mean_{\bm{w}\in
W}s(\bm{w},A_{j}),$ (67)

i.e. the set of words $W$ is skewed towards $A_{i}$. We can reshape Eq. 65 to

$\displaystyle b_{skew}(W,A_{i},A_{j})$ $\displaystyle=$
$\displaystyle\frac{1}{|W|}\sum_{\bm{w}\in W}b(\bm{w},A_{i},A_{j})$ (68)
$\displaystyle=$ $\displaystyle mean_{\bm{w}\in
W}\frac{s(\bm{w},A_{i})-s(\bm{w},A_{j})}{\|\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}}\|}$
(69) $\displaystyle=$ $\displaystyle mean_{\bm{w}\in
W}\frac{s(\bm{w},A_{i})}{\|\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}}\|}-mean_{\bm{w}\in
W}\frac{s(\bm{w},A_{j})}{\|\hat{\bm{a_{i}}}-\hat{\bm{a_{j}}}\|}>0$ (70)

Hence, Equation 65 meets the condition of Definition 3.

Regarding the stereotype, let $W=\\{\bm{w_{1}},\bm{w_{2}}\\}$ be a word pair
with
$s(\bm{w_{1}},A_{i})-s(\bm{w_{1}},A_{j})=-(s(\bm{w_{2}},A_{i})-s(\bm{w_{2}},A_{j}))$,
i.e. one word $\bm{w_{i}}$ is stereotypical for $A_{i}$ and the other for
$A_{j}$, both with the same magnitude of bias. However, according to Equation
65, $b_{skew}(W,A_{i},A_{j})=0$, which contradicts Definition 4. ∎

The Stereotype extension $b_{stereo}(W,A_{i},A_{j})$ is stereotype-sensitive,
but not skew-sensitive.

For the proof see Appendix 4.4.3.

###### Proof.

The bias score indicating no bias is $b_{0}=0$. Let $\bm{w_{1}},\bm{w_{2}}\in
W$ and $A_{i},A_{j}$ attribute sets with
$s(\bm{w_{1}},A_{i})-s(\bm{w_{1}},A_{j})\neq(s(\bm{w_{2}},A_{i})-s(\bm{w_{2}},A_{j})$,
i.e. $\bm{w_{1}},\bm{w_{2}}$ have different biases w.r.t. $A_{i}$ and $A_{j}$.
Hence we consider one stereotypical for $A_{i}$ and the other stereotypical
for $A_{j}$. From there we can follow that

$\displaystyle b(\bm{w},A_{i},A_{j})\neq
b_{skew}(W,A_{i},A_{j})\;\forall\;\bm{w}\in W$ (71)

and hence

$\displaystyle b_{stereo}(W,A_{i},A_{j})$ $\displaystyle=$
$\displaystyle\frac{1}{|W|}\sqrt{\sum_{\bm{w}\in
W}\left(b(\bm{w},A_{i},A_{j})-b_{skew}(W,A_{i},A_{j})\right)^{2}}>0,$ (72)

which meets the condition of Definition 4.

For the Skew let
$s(\bm{w},A_{i})-s(\bm{w},A_{j})=s(\bm{w^{\prime}},A_{i})-s(\bm{w^{\prime}},A_{j})>0\;\forall\;\bm{w},\bm{w^{\prime}}\in
W$. From there follows

$\displaystyle mean_{\bm{w}\in W}s(\bm{w},A_{i})>mean_{\bm{w}\in
W}s(\bm{w},A_{j}),$ (73)

i.e. the set of words $W$ is skewed towards $A_{i}$. However, it also follows
that

$\displaystyle b_{stereo}(W,A_{i},A_{j})$ $\displaystyle=$
$\displaystyle\frac{1}{|W|}\sqrt{\sum_{\bm{w}\in
W}\left(b(\bm{w},A_{i},A_{j})-b_{skew}(W,A_{i},A_{j})\right)^{2}}=0$ (74)

since $b(\bm{w},A_{i},A_{j})=b_{skew}(W,A_{i},A_{j})\forall\;\bm{w}\in W$,
which contradicts Definition 3.

∎

Since both extensions are either not skew-sensitive or not stereotype-
sensitive, they are not unbiased-trustworthy (see Theorem 3.3). In this
regard, one of these score functions cannot replace SAME in its general form
as explained in Sections 4.2 and 4.1. However, one could use the Skew and
Stereotype score functions jointly.

The Skew $b_{skew}(W,A_{i},A_{j})$ is magnitude-comparable. For the proof see
Appendix 4.4.3.

###### Proof.

The minimal score is $-1$, the maximum $1$. As shown in Theorem 4.3 (Eq. 55)
$b(\bm{w},A_{i},A_{j})\in[-1,1]$, independent of $\bm{w}$. Taking the mean
over $\bm{w}\in W$ does not change the extrema, hence Equation 65 meets the
condition of Definition 1. ∎

######  Lemma 1.

Given values $x\in[-1,1]\;\forall\;x\in X$ (or more general a random variable
$X$ taking on values in $[-1,1]$), the standard deviation $\sigma(X)$ is in
bounds $[0,1]$.

###### Proof.

The standard deviation is defined as

$\sigma(X):=\sqrt{\textnormal{var}(X)}=\sqrt{\operatorname*{\mathbb{E}}(X^{2})-\operatorname*{\mathbb{E}}(X)^{2}}.$
(75)

Thus, by Jensens inequality and the fact that the bounds of $X$ also bound the
expectation, it follows

$0\leq\operatorname*{\mathbb{E}}(X)^{2}\leq\operatorname*{\mathbb{E}}(X^{2})\leq
1\qquad|-\operatorname*{\mathbb{E}}(X)^{2}\\\
\Rightarrow\underbrace{\operatorname*{\mathbb{E}}(X)^{2}-\operatorname*{\mathbb{E}}(X)^{2}}_{=0}\leq\underbrace{\operatorname*{\mathbb{E}}(X^{2})-\operatorname*{\mathbb{E}}(X)^{2}}_{=\textnormal{var}(X)}\leq
1-\operatorname*{\mathbb{E}}(X)^{2}\overset{\operatorname*{\mathbb{E}}(X)^{2}\geq
0}{\leq}1.$ (76)

Hence, the statement follows. ∎

The Stereotype $b_{stereo}(W,A_{i},A_{j})$ is magnitude-comparable. For the
proof see Appendix 4.4.3.

###### Proof.

The minimal score is $0$, the maximum $1$. As shown in Theorem 4.3 (Eq. 55)
$b(\bm{w},A_{i},A_{j})\in[-1,1]$, independent of $\bm{w}$. Taking the standard
deviation over all $\bm{w}\in W$ results in values in $[0,1]$ as explained in
Lemma 1. Hence Equation 66 meets the condition of Definition 1. ∎

##  5 Experiments

To emphasize the drawbacks and benefits of the different score functions, we
conducted several experiments. The most important point is to highlight the
sensitivity of the score functions to different kinds of biases in the
training data (skew, stereotype, absolute bias as explained later).
Furthermore, we emphasize the important differences between our and the state
of the art score functions, in relation to the theoretical flaws discussed in
Chapter 3.
In order to achieve a ground truth for biases to measure, we constructed
biased datasets and fine-tuned BERT on these using the masked language
objective. The datasets consisted of a variety of sentences, including a
gender-neutral occupation and either male or female pronoun referring to that
occupation. Only the pronouns were masked out during training, forcing the
model to explicitly learn the gender probabilities associated with each
occupation. After training all bias score functions were applied to measure
gender bias with these occupations. We ran this procedure for a large number
of datasets, varying with regard to each occupation’s gender probability and
the overall gender distribution. This allowed us to produce a large number of
models with different expressions of bias. To confirm that the biases were
actually learned, we probed the resulting masked language models for
probabilities of inserting male/female pronouns in the training sentences.
This approach is similar to the one from kurita, who also measured bias using
the masked language objective. In the following, we explain the procedure of
generating datasets and training in detail. Then we evaluate the performance
of bias scores, first regarding individual words, then regarding sets of
words. In both cases, we conduct additional experiments, highlighting the
robustness of SAME in comparison to WEAT and the drawbacks of computing the
Bias Directions by PCA instead of our approach.

###  5.1 Training Procedure

Our goal was to test for both word-specific biases as well as biases in terms
of a set/domain of words. Therefor, we decided to observe gender bias in
occupations, similarly to the experiments of bolukbasi. We used the gender
attributes and occupations from their
implementation111https://github.com/tolga-b/debiaswe, although we removed some
occupations that were not gender neutral, resulting in 258 occupations. The
final lists can be found in Table 1 in the appendix.

#### List of occupations

caretaker | dancer | homemaker | librarian | nurse | hairdresser
---|---|---|---|---|---
housekeeper | secretary | teacher | nanny | receptionist | stylist
interior designer | clerk | educator | bookkeeper | environmentalist | fashion designer
paralegal | therapist | dermatologist | instructor | organist | planner
radiologist | singer songwriter | socialite | soloist | treasurer | tutor
violinist | vocalist | aide | artist | choreographer | lyricist
mediator | naturalist | pediatrician | performer | psychiatrist | publicist
realtor | singer | sociologist | baker | councilor | counselor
photographer | pianist | poet | flight attendant | substitute | cellist
correspondent | employee | entertainer | epidemiologist | freelance writer | gardener
guidance counselor | warrior | jurist | musician | novelist | psychologist
student | swimmer | understudy | valedictorian | writer | author
biologist | comic | consultant | parishioner | photojournalist | protagonist
researcher | servant | administrator | campaigner | chemist | civil servant
columnist | crooner | curator | envoy | graphic designer | headmaster
illustrator | lecturer | narrator | painter | pundit | restaurateur
trumpeter | attorney | bartender | cleric | comedian | filmmaker
jeweler | journalist | missionary | negotiator | pathologist | pharmacist
philanthropist | pollster | principal | promoter | prosecutor | solicitor
strategist | worker | accountant | analyst | anthropologist | assistant professor
associate dean | associate professor | barrister | bishop | broadcaster | commentator
composer | critic | editor | geologist | landlord | medic
plastic surgeon | professor | proprietor | provost | screenwriter | adjunct professor
adventurer | archbishop | astronomer | barber | broker | bureaucrat
butler | cardiologist | cartoonist | chef | cinematographer | detective
diplomat | economist | entrepreneur | financier | footballer | goalkeeper
guitarist | historian | inspector | inventor | investigator | lawyer
playwright | politician | professor emeritus | saxophonist | scientist | sculptor
shopkeeper | solicitor general | stockbroker | surveyor | archaeologist | architect
banker | cabbie | captain | chancellor | chaplain | conductor
constable | cop | director | disc jockey | economics professor | lifeguard
manager | mechanic | neurologist | parliamentarian | physician | programmer
rabbi | scholar | soldier | technician | trader | vice chancellor
welder | wrestler | ambassador | athlete | athletic director | dean
dentist | deputy | doctor | fighter pilot | firefighter | industrialist
investment banker | judge | lawmaker | legislator | lieutenant | magician
marshal | neurosurgeon | pastor | physicist | preacher | ranger
senator | sergeant | skipper | surgeon | trucker | tycoon
astronaut | ballplayer | cab driver | carpenter | coach | colonel
commander | commissioner | electrician | farmer | magistrate | mathematician
minister | officer | philosopher | plumber | sailor | sheriff deputy
bodyguard | boxer | butcher | custodian | drummer | janitor
laborer | president | sportswriter | superintendent | taxi driver | warden
Table 1:  List of occupations used in the experiments.

#### Template sentences for training

pronouns | sentences
---|---
he/she | [MASK]’s a OCCUPATION
| [MASK] is a OCCUPATION
| [MASK] will become a OCCUPATION
| [MASK] just started working as a OCCUPATION
| [MASK] is working as a OCCUPATION
| [MASK] was a OCCUPATION
| the OCCUPATION was in a hurry because [MASK] was late for work
| [MASK] is interested in becoming a OCCUPATION
| [MASK] does not like being a OCCUPATION
| [MASK] likes being a OCCUPATION
| [MASK] always wanted to be a OCCUPATION
| [MASK] never wanted to be a OCCUPATION
| [MASK] had an interview for a position as a OCCUPATION
| is [MASK] a OCCUPATION
| [MASK] is a OCCUPATION, right?
| [MASK] is proud to be a OCCUPATION
| the OCCUPATION took a week off, because [MASK] wanted to go on vacation
| " i want to be a OCCUPATION when i grow up", [MASK] said
| " being a OCCUPATION sounds boring", [MASK] said
| the OCCUPATION is very skilled. [MASK] has a lot of experience
him/her | the OCCUPATION enjoyed [MASK] lunch
| the OCCUPATION missed [MASK] bus
| the OCCUPATION arrived in [MASK] car
| the OCCUPATION asked [MASK] boss for a promotion
| the OCCUPATION collected [MASK] check
| the OCCUPATION was chatting with [MASK] colleagues
| the OCCUPATION loves [MASK] job
| the OCCUPATION does not like [MASK] job
| the OCCUPATION went on vacation with [MASK] family
| the OCCUPATION went on vacation with [MASK] friends
| it was [MASK] choice to become a OCCUPATION
| the OCCUPATION thanked [MASK] colleagues
| the OCCUPATION likes to spend [MASK] free time with friends
| the OCCUPATION likes to spend [MASK] free time with the family
| the OCCUPATION gets along really well with [MASK] colleagues
Table 2:  Template sentences used for training BERT in the experiments. The
[MASK] token was to be replaced by the respective pronouns and OCCUPATION by
one of the occupations mentioned in Table 1

#### Template sentences for testing

pronouns | sentences
---|---
he/she | [MASK] is one of the OCCUPATIONs
| [MASK] is well paid for a OCCUPATION
| [MASK] is paid less than the other OCCUPATIONs
| [MASK] thinks about becoming a OCCUPATION
| as a child [MASK] wanted to become a OCCUPATION
| the OCCUPATION was early because [MASK] overslept
| I thought [MASK] was a OCCUPATION
| the OCCUPATION was early because [MASK] could not sleep anyways
| the OCCUPATION is nice but [MASK] is not very competent
| the OCCUPATION is very knowlegeable but [MASK] is bad with people
him/her | the OCCUPATION started [MASK] new position on monday
| the OCCUPATION lost [MASK] job
| the OCCUPATION left early to pick up [MASK] children from school
| the OCCUPATION told everyone about [MASK] weekend
| the OCCUPATION likes to keep [MASK] workplace clean
| the OCCUPATION does not like [MASK] colleagues
| the OCCUPATION likes [MASK] colleagues very much
| the OCCUPATION smiled and showed [MASK] teeth
| the OCCUPATION wore [MASK] favorite jacket
| i asked the OCCUPATION about [MASK] new book
Table 3:  Template sentences used for testing the unmasking bias in the
experiments. The [MASK] token was to be replaced by the respective pronouns
and OCCUPATION by one of the occupations mentioned in Table 1

We further constructed 30 template sentences (see Table 2 in the appendix),
e.g. ’[MASK] is a OCCUPATION’ or ’the OCCUPATION enjoyed [MASK] lunch’, where
[MASK] could either be substituted by he/she or his/her. Given a gender
probability for each occupation (e.g. ’nurse’ is 60% male, 40% female), we
inserted the occupation in each of the template sentences and assigned a
pronoun, which was randomly selected based on the gender probability, as
solution for the masked language task. This results in 7770 sentences for
fine-tuning BERT, holding 30% back as validation set. We used a pre-trained
model (’bert-base-uncased’) and trained for 5 epochs.
To verify that BERT learned the gender associations implied by the training
data, we ran an unmasking task on the fine-tuned model using a separate test
set constructed with another set of 20 template sentences (see Table 3 in the
appendix), where we inserted the occupations. For each sentence, we queried
the probability of inserting the male/female pronoun (later called unmasking
bias), then took the average over all sentences with the same occupation and
measured the $R^{2}$ correlation of the results with the gender probabilities
in the training data (later called training bias). If the correlation was
above a threshold of $0.7$, the model was assumed to have learned the bias
well enough to be considered for further experiments.
We repeated this to generate a variety of biased models. For each model, we
defined a normal distribution of gender probabilities (for inserting the male
pronoun) by $\mu$ and $\sigma$, then randomly selected the occupation’s gender
probability (later referred as predefined bias) from this distribution, then
constructing the training data as described above. For $\mu$ we used values in
$\\{0.25,0.3,0.35,...,0.75\\}$ and $\sigma$ in $\\{0.1,0.15,...,0.35\\}$ to
elaborate how well the bias scores react to either shifted distributions
(skew) or variance of distributions (stereotype). For each combinations of
$\mu$ and $\sigma$, we trained 5 models using the same predefined biases, but
generating the sentences and selecting the training set each time to produce
randomness. This resulted in 330 models in total.
In the following sections, we usually refer to the training bias (the gender
probabilities in the actual training data) as opposed to the predefined
biases. we first experiment with word-wise biases. Later, we focus on the bias
distribution parameters $\mu$, $\sigma$ and the absolute amount of bias
comparing them with the bias scores over all occupation, where each model
makes up one data point.

###  5.2 Performance of word-wise bias scores

To evaluate the performance of bias scores on a word-level, we measured the
$R^{2}$ correlations of word-wise biases with the respective training biases
as illustrated in Figure 5. As explained in the last section, we consider the
unmasking task as a sanity check and baseline for the best possible outcome.
The word-wise bias scores reported here are Eq. 43 for SAME, Eq. 9 with
$N=\\{\bm{w}\\}$ and $c=1$ for the Direct Bias and Eq. 8 with $T=\\{\bm{w}\\}$
for MAC. Since Eq. 3 of WEAT only differs from our pairwise bias (Eq. 43) by a
constant factor (given fixed attribute sets), it will result in the same
correlation. Hence, both bias scores are reported in one graph. As can be seen
in the figure, WEAT and SAME outperform MAC and the Direct Bias significantly.

Figure 5: Example for correlation of word-wise biases measured with the
different bias scores compared to training biases (probability for male
pronoun) $p$. Since MAC and the Direct Bias do not measure the direction of
bias, we display $0.5-p$ on the x-axis.

We further calculated these correlations for every fine-tuned model and
display the mean and standard deviation of correlations in Figure 6. The
cosine based metrics cannot match the unmasking tasks, but with the mean
correlations of around $0.4$ for WEAT/SAME, they still outperform MAC with
mean correlation around $0$ and Direct Bias around $0.1$. However, we also see
a high standard deviation for WEAT/SAME, which indicates that for some models,
the score functions word considerably well (similar to Figure 5), while for
other models, they perform quite bad. This could be an indication that those
models represent the biases in a non-linear way that impacts the unmasking
task (and possibly other downstream tasks), but does not show with cosine
based scores.

Figure 6: The average correlation of word-wise bias scores with the training
and unmasking bias. WEAT and SAME are reported jointly since they only differ
by a constant factor and thus share the same correlation.

####  5.2.1 Approaches to compute the Bias Direction

As seen in the previous section, though similar to SAME, the Direct Bias
performs significantly worse. Comparing the Equations 9 and 43, we can see
that the word-wise biases both measure the cosine between a word $\bm{w}$ and
a bias direction. In case of the Direct Bias this direction is obtained by
PCA, in case of the proposed metric simply by the mean bias direction. Hence,
the difference in performance must be linked to the way the bias direction is
determined. To confirm this, we conducted the following experiment:

We selected a number of models where the word-wise bias scores reported by the
Direct Bias correlated similar well to those reported by SAME (difference in
correlation < $0.25$) and a number of models, where the Direct Bias correlated
worse (difference in correlation > $0.4$). In both cases we considered only
such models, where SAME correlated with at least $0.6$. For both sets of
models, we reported the mean angle between bias directions computed by PCA and
by averaging over individual bias directions.

similar correlation | 10.859
---|---
worse correlation | 17.551

According to these results, the bias direction computed by PCA differs from
the average bias direction in an actual use case and this can be linked with
lower performance of the Direct Bias. The following Figures 7 and 8 show
exemplary cases with similar bias directions and performance and one with a
greater difference between bias directions and performance.

Figure 7: In this example, the Direct Bias shows similar results to WEAT/own
metric regarding the word-wise correlation, while both bias directions are
very similar.

Figure 8: In this example, the Direct Bias performs far worse than WEAT/own
metric regarding the word-wise correlation, since the bias direction computed
by the PCA differs from the ideal bias direction.

####  5.2.2 Comparability of word-wise biases

As stated in Definition 1, an important property of a bias score is the
comparability of its results. We showed in Theorem 1 that the extrema of word-
wise biases calculated by WEAT depend on the average distance between bias
attributes in the embedding space. Hence those biases are not comparable
between different embedding models. Opposed to this, our bias score, SAME,
mitigates this effect by normalizing over said distance and is thus
comparable. To highlight this effect in a practical setting and show that our
proposed metric is more robust in this regard, we conducted the following
experiment:
As mentioned before, we generated $66$ different distributions (combinations
of $\mu$ and $\sigma$) of predefined biases, where each occupation was
assigned a certain gender probability. Based on each distribution (with fix
gender probability per occupation) we generated $5$ sets of training and
testing data and fine-tuned BERT on each of these. Since the predefined biases
were identical for each of the $5$ models (yet with random noise due to
train/test split and sentence generation), we expect a robust bias metric to
yield similar results for all of these models. To probe this, we calculated
the standard deviation of word-wise bias scores for each model. Given each set
of $5$ comparable models, we computed the percentage difference of these
standard deviations and finally took the mean over all $66$ distributions.
This resulted in the following percentage differences. Again we take the
unmasking task as a baseline/sanity check for the best possible result.

Table 4 shows that SAME is indeed more robust than WEAT and achieves nearly
the same robustness as the unmasking bias. However, the Direct Bias performs
even worse than WEAT. Considering the similarity of the Direct Bias and SAME,
a possible explanation is that the different fine-tuned models also represent
the bias attributes in a distinctive way leading to greater variations in the
bias direction obtained by PCA compared to our approach. This suggest that
using the mean bias direction is preferable to the bias direction by PCA.

Direct Bias | 0.306
---|---
WEAT | 0.150
SAME | 0.121
unmask | 0.096
Table 4:  Robustness of word-wise bias scores. We compute the standard
deviations of word-wise biases for each model and then the percentage
difference between each set of 5 models trained with the same training bias.
Lower values indicate more robust scores.

###  5.3 Performance of bias scores for sets of words

Figure 9: $R^{2}$ correlation of bias scores (measured over all occupations)
with unmasking bias. Bold values indicate a p-value below 0.05. Regarding the
mean of biases only our skew score produces a significant correlation.
Regarding the standard deviation SAME, our stereotype and WEAT correlate
significantly, though the stereotype clearly outperforms the others. Regarding
the absolute bias, SAME also surpasses WEAT, though the stereotype version
works even better.

In the next step, we evaluated the bias scores applicable to a set of words
$W$ or sets of words (e.g. $X$, $Y$ for WEAT). This is critical to distinguish
SAME from WEAT and evaluate the skew and stereotype adaptions. In our case $W$
is the list of occupations and the groups $X$, $Y$ for WEAT are selected using
the stereotype ratings provided with the occupations from bolukbasi,
reflecting actual stereotypes in society.
In Figure 9 we illustrate the correlations of all bias scores discussed in the
paper. Here we use the unmasking task as reference to measure correlation. We
expect this to be more accurate than comparison to the training bias, since
the unmasking bias undoubtedly reflects biases learned by the model. We
consider the mean and standard deviation of word-wise unmasking biases, as
well as the absolute bias ($\sum|0.5-p|$ with $p$ the probability of male
pronouns as predicted by the unmasking task).

As expected after investigating the word-wise bias correlations, MAC and the
Direct Bias perform rather bad. WEAT produces better results compared to the
unmasking standard deviation, but SAME (standard version) also outperforms
WEAT with regard to the unmasking standard deviation and absolute biases.
Other than that, our proposed stereotype metric outperforms all metrics from
the literature in terms of standard deviation and absolute bias by a large
margin. Regarding the mean of word biases, our skew implementation is the only
score the achieves a significant correlation with $0.21$. Hence, our proposed
score function and its versions are clearly preferable over the metrics from
the literature and this experiment further emphasizes the benefits of using
the skew and stereotype scores. Yet, keep in mind that these two should be
used jointly to prevent overlooking the other kind of bias. Nevertheless, we
conduct another experiment to highlight the robustness of our score function.

####  5.3.1 Robustness to Permutation and Subset Selection

When observing bias in certain domains or groups of target words, we might not
always have the resources or knowledge to measure bias over all relevant words
(or sentences). An example are the different WEATs (and their versions/
implementations in the literature), where only a limited number of words is
used to measure bias in terms of a concept (e.g. the career/family test).
Using only a small subset poses the risk to misjudge bias due to noise or
individual words’ associations rather than associations of the whole concept.
In this experiment, we investigate how robust the different bias scores are to
this problem. For each model trained, we determine the bias over all
occupations in $W$, then run a permutation test ($n=100$ iterations) and
determine the bias over subsets $W_{i}$ with half the number of occupations,
randomly selected from $W$. We report
$\frac{1}{n}\sum_{i=1}^{n}|b(W_{i})-b(W)|$ for each model, i.e. the mean
differences in biases over $W_{i}$ compared to $W$ with $b(W)$ to be replaced
by concrete bias scores. We further take the mean over all models and
normalize by the size of bias score intervals. Thereby, we obtain the results
shown in Table 5. We observe similar results for all score functions beside
WEAT, for which we report differences larger by one order of magnitude. This
is most likely due to the splitting of target words into groups $X$, $Y$,
which is unique for WEAT. Hence, WEAT is far less robust to subset selection
than the other metrics.

Score function | Mean Diff
---|---
MAC | 0.0010
WEAT | 0.0397
Direct Bias | 0.0018
SAME | 0.0016
stereo | 0.0016
skew | 0.0022
Table 5:  Robustness to permutation and subset selection of neutral words. We
report the mean difference in biases reported over subsets $W_{i}$ compared to
biases reported over all neutral words in $W$. Lower values indicate more
robust models.

##  6 Conclusions

In conclusion, we proved in Chapter 3.2 that all existing cosine based bias
scores have one or several drawbacks that make them unreliable to quantify
bias. The baseline definition for this claim is linked in the literature, e.g.
bolukbasi; weat. Nevertheless, we also showed in our experiments that the
weaknesses of the bias scores manifest in practice, which further confirms our
statement. Furthermore, we proposed three new bias scores: SAME for bias
quantification that suffices all requirements stated in Chapter 3.2, and two
versions to distinguish between skew and stereotype.
An overview over the capabilities and drawbacks of the bias scores can be
found in Table 6 for the single-word bias scores and in Table 7 for the
overall bias scores.

bias score | magnitude-comparable | unbiased-trustworthy | interpretable
---|---|---|---
WEAT | ✗ | ✓ | ✓
MAC | ✓ | ✗ | ✗
Direct Bias | ✓ | ✗ | ✗
SAME $b(w,A_{i},A_{j})$ | ✓ | ✓ | ✓
SAME $b(w,A)$ | ✓ | ✓ | ✗
Table 6:  Overview over the properties of word-wise biases scores. magnitude-comparablerefers to Definition 1 and unbiased-trustworthyto Definition 2. Interpretability refers to whether the bias score obtains useful information about the bias direction. bias score | extrema | trustworthy | partition-invariant | skew | stereo | multi-class
---|---|---|---|---|---|---
WEAT $d(X,Y,A,B)$ | ✓ | ✗ | ✗ | ✗ | ✓ | weatgeneralized
MAC | ✓ | ✗ | ✓ | ✗ | ✗ | ✓
Direct Bias | ✓ | ✗ | ✓ | ✗ | ✗ | extension possible
SAME $b(W,A)$ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Skew | ✓ | ✗ | ✓ | ✓ | ✗ | ✓
Stereotype | ✓ | ✗ | ✓ | ✗ | ✓ | ✓
Table 7:  Overview over the properties of bias scores for set of words $W$.
The first two rows regard Definitions 1 and 2. Partition-invariant regards
whether the score depends on the way neutral words were partitioned into
subgroups, skew and stereo refer to Definitions 3 and 4 and the last row
shows, whether the score handles multi-class bias or can be extended to do so.

Based on both the theoretical evaluation and our experiments we argue against
using the Direct Bias and MAC, as we don’t see advantages in them. We
acknowledge that WEAT can be useful when people are interest in a specific
kind of stereotype (aligning with groups $X$ and $Y$), but caution to treat
the results carefully in light of our findings. While high effect sizes with a
low p-value prove the presence of stereotypes, low effect sizes (and high
p-values) are not particularly meaningful. Hence WEAT cannot be used for
quantitative bias measurements, but only as an indicator.
SAME on the other hand, can be used for quantification as it will report any
bias in terms of the geometrical definition and provide comparable results.
The skew and stereotype complement this to allow better insights and proved
particularly useful in the experiments regarding both the correlation with our
baseline and robustness.
Yet, our experiments also show that none of the cosine based scores can
entirely grasp the biases manifesting in the models (both in relation to the
training data as well as the unmasking task). This shows the need to evaluate
biases also in terms of a downstream task if deploying the model in that
context or possibly considering other tests in addition. In that context, we
want to point out two works that used a multitude of tests to enrich their
bias evaluation: nullitout and costa2019evaluating. While we cannot advocate
their choice of cosine based metric, having multiple tests to allow a more
thorough evaluation of biases seems like a wise choice as long as there is no
consensus about the ideal bias metrics for word embeddings.
Of course, this again shows the importance of research in the field of bias
evaluation methods. We would highly recommend looking further into when cosine
based metrics fall short or showing whether other tests from the literature
(e.g. clustering test) can fill this gap appropriately.

## Acknowledgements

We gratefully acknowledge the funding from the German Federal Ministry of
Economic Affairs and Energy (BMWi) within the “Innovationswettbewerb
Künstliche Intelligenz" and from the Ministry of Culture and Science of the
state of North Rhine-Westphalia in the project "Bias aus KI-Modellen".

## References

  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Cer et al. [2018] Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil.  Universal sentence encoder, 2018.
  * Mikolov et al. [2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space, 2013.
  * Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543, 2014.
  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.
  * Bolukbasi et al. [2016] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  _Advances in neural information processing systems_ , 29:4349–4357, 2016.
  * Caliskan et al. [2017] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ , 356(6334):183–186, 2017.
  * May et al. [2019] Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger.  On measuring social biases in sentence encoders.  _CoRR_ , abs/1903.10561, 2019.  URL http://arxiv.org/abs/1903.10561.
  * Liang et al. [2020] Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency.  Towards debiasing sentence representations, 2020.
  * Karve et al. [2019] Saket Karve, Lyle Ungar, and João Sedoc.  Conceptor debiasing of word representations evaluated on weat, 2019.
  * Kaneko and Bollegala [2021] Masahiro Kaneko and Danushka Bollegala.  Dictionary-based debiasing of pre-trained word embeddings.  _arXiv preprint arXiv:2101.09525_ , 2021.
  * Swinger et al. [2019] Nathaniel Swinger, Maria De-Arteaga, Neil Thomas Heffernan IV, Mark DM Leiserson, and Adam Tauman Kalai.  What are the biases in my word embedding?  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 305–311, 2019.
  * Zhou et al. [2019] Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang.  Examining gender bias in languages with grammatical gender.  2019\.
  * Manzini et al. [2019] Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and Alan W. Black.  Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings.  _CoRR_ , abs/1904.04047, 2019.  URL http://arxiv.org/abs/1904.04047.
  * Zhao et al. [2018] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang.  Gender bias in coreference resolution: Evaluation and debiasing methods.  _CoRR_ , abs/1804.06876, 2018.  URL http://arxiv.org/abs/1804.06876.
  * Gonen and Goldberg [2019] Hila Gonen and Yoav Goldberg.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  _CoRR_ , abs/1903.03862, 2019.  URL http://arxiv.org/abs/1903.03862.
  * Ethayarajh et al. [2019] Kawin Ethayarajh, David Duvenaud, and Graeme Hirst.  Understanding undesirable word embedding associations.  _arXiv preprint arXiv:1908.06361_ , 2019.
  * Gao et al. [2021] Tianyu Gao, Xingcheng Yao, and Danqi Chen.  Simcse: Simple contrastive learning of sentence embeddings, 2021.
  * Subramanian et al. [2018] Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal.  Learning general purpose distributed sentence representations via large scale multi-task learning.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=B18WgG-CZ.
  * Liu et al. [2021] Fangyu Liu, Yunlong Jiao, Jordan Massiah, Emine Yilmaz, and Serhii Havrylov.  Trans-encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations, 2021.
  * Thongtan and Phienthrakul [2019] Tan Thongtan and Tanasanee Phienthrakul.  Sentiment classification using document embeddings trained with cosine similarity.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 407–414, 2019\.
  * Shahmirzadi et al. [2019] Omid Shahmirzadi, Adam Lugowski, and Kenneth Younge.  Text similarity in vector space models: a comparative study.  In _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_ , pages 659–666. IEEE, 2019.
  * Zhao et al. [2019] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang.  Gender bias in contextualized word embeddings.  _CoRR_ , abs/1904.03310, 2019.  URL http://arxiv.org/abs/1904.03310.
  * Kurita et al. [2019] Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov.  Measuring bias in contextualized word representations.  _arXiv preprint arXiv:1906.07337_ , 2019.
  * Webster et al. [2018] Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge.  Mind the GAP: A balanced corpus of gendered ambiguous pronouns.  _Transactions of the Association for Computational Linguistics_ , 6:605–617, 2018.  doi:10.1162/tacl_a_00240.  URL https://aclanthology.org/Q18-1042.
  * de Vassimon Manela et al. [2021] Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, and Pasquale Minervini.  Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models.  _CoRR_ , abs/2101.09688, 2021.  URL https://arxiv.org/abs/2101.09688.
  * Greenwald et al. [1998] Anthony G Greenwald, Debbie E McGhee, and Jordan LK Schwartz.  Measuring individual differences in implicit cognition: the implicit association test.  _Journal of personality and social psychology_ , 74(6):1464, 1998.
  * Chen et al. [2021] Yan Chen, Christopher Mahoney, Isabella Grasso, Esma Wali, Abigail Matthews, Thomas Middleton, Mariama Njie, and Jeanna Matthews.  Gender bias and under-representation in natural language processing across human languages.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’21, page 24–34, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450384735.  doi:10.1145/3461702.3462530.  URL https://doi.org/10.1145/3461702.3462530.
  * Costa-juss and Casas [2019] Christine Basta Marta R Costa-juss and Noe Casas.  Evaluating the underlying gender bias in contextualized word embeddings.  _GeBNLP 2019_ , page 33, 2019.
  * Ravfogel et al. [2020] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg.  Null it out: Guarding protected attributes by iterative nullspace projection.  _CoRR_ , abs/2004.07667, 2020.  URL https://arxiv.org/abs/2004.07667.