# GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying
Biases in Explanations

Rick Wilming1 Artur Dox1,2 Hjalmar Schulz1,3
Marta Oliveira2 Benedict Clark2 Stefan Haufe1,2,3
1Technische Universität Berlin, Germany
2Physikalisch-Technische Bundesanstalt, Berlin, Germany
3Charite – Universitätsmedizin, Berlin, Germany Equal contribution.

###### Abstract

Large pre-trained language models have become popular for many applications
and form an important backbone of many downstream tasks in natural language
processing (NLP). Applying ‘explainable artificial intelligence’ (XAI)
techniques to enrich such models’ outputs is considered crucial for assuring
their quality and shedding light on their inner workings. However, large
language models are trained on a plethora of data containing a variety of
biases, such as gender biases, affecting model weights and, potentially,
behavior. Currently, it is unclear to what extent such biases also impact
model explanations in possibly unfavorable ways. We create a gender-controlled
text dataset, GECO, in which otherwise identical sentences appear in male and
female forms. This gives rise to ground-truth ‘world explanations’ for gender
classification tasks, enabling the objective evaluation of the correctness of
XAI methods. We also provide GECOBench, a rigorous quantitative evaluation
framework benchmarking popular XAI methods, applying them to pre-trained
language models fine-tuned to different degrees. This allows us to investigate
how pre-training induces undesirable bias in model explanations and to what
extent fine-tuning can mitigate such explanation bias. We show a clear
dependency between explanation performance and the number of fine-tuned
layers, where XAI methods are observed to particularly benefit from fine-
tuning or complete retraining of embedding layers. Remarkably, this
relationship holds for models achieving similar classification performance on
the same task. With that, we highlight the utility of the proposed gender-
controlled dataset and novel benchmarking approach for research and
development of novel XAI methods. All code including dataset generation, model
training, evaluation and visualization is available at:
https://github.com/braindatalab/gecobench

##  1 Introduction

Machine Learning (ML) models, in particular modern large neural network
architectures, are complex, making it difficult to understand the mechanisms
by which model outputs are generated. This has led to the development of
dedicated post-hoc analysis tools that are commonly referred to as
‘explainable artificial intelligence’ (XAI). In many cases, XAI methods
provide so-called feature attributions, which assign an ‘importance’ score to
each feature of a given input [41, 27, 50].

For XAI methods to be useful, it needs to be shown that they can answer well-
defined questions about a model, its training data, and/or the way it
processes given test data with reasonable accuracy. Although applications of
XAI have proliferated in the past years [e.g. 28, 22, 52, 57], the problems to
be addressed by XAI have rarely been formally defined [32]. Theoretical or
non-anectodal empirical evidence for the utility of XAI methods is, therefore,
scarce. In particular, the widely-used metaphor of identifying features ‘used’
by a model, measured through ‘faithfulness’ or ‘fidelity’ metrics [e.g. 44,
18, 42], can lead to fundamental misinterpretations, as such a notion depends
strongly on the structure of the underlying generative model and the resulting
distribution of the (training) data [17, 55]. Wilming et al. [55] investigate
such metrics, showing that many perturbation and pixel-flipping methods fail
to detect statistical dependencies or other feature effects like suppressor
variables [14, 17], and are therefore unsuitable to directly measure certain
meaningful notions explanation ‘correctness’.

We follow previous work [17, 35, 9] by adopting the minimal requirement that
‘important’ features need to be statistically related to the predicted target
variable, which we refer to here as the _statistical association_ property of
XAI methods. Thus, we are seeking ‘true-to-the-data’, or, ‘world’ explanations
[32, 5, 8, 15]. To objectively evaluate whether an XAI method possesses this
property, the availability of ground-truth data is instrumental. Ground truth
data for explanation methods in domains such as image, tabular, and time
series data have been developed in the last few years [e.g. 23, 19, 20, 51, 1,
3]. However, most of these benchmarks do not present realistic correlations
between class-dependent and class-agnostic features (e.g., the foreground or
object of an image versus the background) [9], and often use surrogate metrics
like faithfulness instead of directly measuring explanation performance.
Moreover, all of these approaches leverage a continuous data generation
process, yielding synthetic numerical data unsuitable for the NLP domain,
where, by nature, the data generation process is discrete and not
straightforwardly defined.

For the NLP domain, feature attributions methods in supervised settings are
expected to highlight parts of an input text (e.g., words or sentences) that
are related to the predicted target, e.g., a sentiment score. Several
benchmarks have been presented [12, 43]; however, these again have several
limitations. In the case of DeYoung et al. [12], faithfulness of the model is
measured in alignment with human-annotated rationales, which do not
necessarily align with statistical association – opening the door to cognitive
biases. Rychener et al. [43] present a benchmark dataset consisting of a
question-answering task, where the ground truth explanations originate from a
text context providing the answer. However, as the authors emphasize, defining
a ground truth for question-answering cannot only depend on one word but
rather a context of words providing the prediction models with sufficient
information. This work, therefore, does not provide ground-truth explanation
in the sense of statistical association.

As models of human languages require substantial complexity, overfitting and
poor generalization can be the result when trying to train such models from
scratch for specific tasks using limited training corpora. Better results are
typically achieved by adapting large pre-trained general purpose
(‘foundation’) language models (LLMs) to the task at hand using selective re-
training or fine-tuning of certain parts of the pre-trained network,
techniques also known as transfer learning. Examples of foundational models
include BERT and its variants [11, 26, 38], which leverage the transformer
architecture [53]. Foundation LLMs are commonly trained on large corpora of
text scraped from public and non-public sources including Wikipedia, Project
Gutenberg111https://www.gutenberg.org or
OpenWebText222https://github.com/jcpeterson/openwebtext. This is typically
done in an unsupervised way by training the model to efficiently represent
words or complete documents as vector space embeddings [29, 24].

Figure 1: Overview of the benchmarking approach for evaluating the correctness
of XAI methods. Starting from a clear definition of discriminative features
inducing statistical associations between features/words and prediction
target, we specify ground truth explanations. With that, we craft a gender-
focused dataset GECO, with text sourced from Wikipedia, by labeling and
altering the grammatical gender of specfic words. The resulting training and
validation datasets are used to train the language model BERT. The test
dataset, together with the trained model, serves as input to the XAI method,
which outputs explanations for the test set samples. The word-based ground-
truth explanations, provided by the former labeling process, are then used to
measure the correctness of each sentence’s generated explanations using the
Mass Accuracy metric [3, 9, 10].

These large corpora contain a variety of biases, such as biases against
demographic groups [4, 16, 40], which may negatively affect not only
prediction but also explanation performance in downstream tasks, besides
raising issues of fairness. It has been shown that such biases affect model
weights [30, 31]. However, it is currently unclear to what extent biases
contained in pre-training corpora are reflected in explanations, possibly
preventing XAI methods from fulfilling the statistical association requirement
on correct world explanations. Using the example of grammatical gender, we can
imagine one particular way in which pre-training biases might lead to
incorrect explanations or point to residual bias in fine-tuned models.

In a gender classification task, asymmetries in the frequencies of specific
words may be present in a pre-training corpus but not in the target domain.
For example, historical novels may be biased towards male protagonists, and
may depict females less often and in more narrowly defined roles. However, the
association between, for example, role-specific words and gender in these text
is irrelevant when it comes to distinguishing grammatical gender (as well as
for many other tasks). An explanation that highlights respective words thus
suggests the influence of pre-training biases.

To investigate and quantify such phenomena, we make two contributions: (1)
GECO – a gender-controlled dataset and (2) GECOBench – a quantitative
benchmarking framework to assess the correctness of machine learning world
explanations for language models on gender classification tasks. Both can be
used for the development and assessment of the correctness of XAI methods. An
overview is shown in Figure 1.

GECO333Available on OSF:
https://osf.io/74j9s/?view_only=8f80e68d2bba42258da325fa47b9010f is a gender-
controlled dataset in which each sentence $\mathbf{x}\in\mathbb{R}^{d}$
appears in four different gendered variants. In variants $\mathbf{x}_{S}^{M}$
and $\mathbf{x}_{S}^{F}$, the (human) subject (S) is either male (M) or female
(F), while in variants $\mathbf{x}_{A}^{M}$ and $\mathbf{x}_{A}^{F}$, all (A)
human protagonists are either male or female. All four variants are identical
apart from gender-specific words such as names and pronouns. This dataset
gives rise to two binary gender-classification downstream tasks with labels M
vs. F, namely the discrimination of variant $\mathbf{x}_{S}^{M}$ from variant
$\mathbf{x}_{S}^{F}$, and the discrimination of variant $\mathbf{x}_{A}^{M}$
from variant $\mathbf{x}_{A}^{F}$ (see Table 1). Importantly, in both cases,
ground truth explanations on a word-level basis are available by construction.

GECOBench is a workflow to quantitatively benchmark the correctness of
explanations with respect to NLP classification tasks induced by GECO or
similar datasets. We here showcase the usage of GECOBench by analysing BERT
[11], a large language model pre-trained on Wikipedia.

Gender-controlled datasets like GECO cannot possibly induce gender biases in
explanations by construction. On the other hand, it is known that BERT suffers
from gender biases [33, 2]. Thus, when using GECO as a test set, any residual
asymmetry in explanations can be traced back to biases induced by pre-
training. We quantify this effect for different stages of re-training or fine-
tuning distinct layers of BERT’s architecture to investigate to what extent
re-training or fine-tuning BERT on gender-controlled data can mitigate gender
bias. By ensuring that the distinct trained models have equivalent
classification accuracy throughout the considered fine-tuning stages, we can
assess how these training regimes impact explanation performance with the
proposed dataset. Generally, we do not expect any XAI method to perform
perfectly, as correctness is only one goal of interpreting machine learning
models and not necessarily the primary purpose of each explanation approach.
Here, our focus is on comparing the performance of XAI methods to the null
performance of random explanations. However, we hypothesize that bias
contained in BERT will propagate into the model’s explanation, and a
consistent difference in explanation performance will be apparent depending on
which layer of BERT’s architecture we train or fine-tune.

##  2 Methods

The GECO dataset is comprised of a set of manipulated sentences in which
grammatical subjects and objects assume either their male or female forms. In
the following, we give further details describing the data selection, pre-
processing, and generation, resulting in the dataset $\mathcal{D}_{S}$
representing sentences with controlled and altered grammatical subjects and
the dataset $\mathcal{D}_{A}$ consisting of alterations of all grammatical
gender-encoding words.

###  2.1 Data Sourcing & Generation

Version | Sentence |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
Original | She | touches | the | heart | of | her | Aunt | Ophelia | .
Subj. Female ($\mathbf{x}_{S}^{F}$) | She | touches | the | heart | of | her | Aunt | Eleanor | .
Subj. Male ($\mathbf{x}_{S}^{M}$) | He | touches | the | heart | of | his | Aunt | Eleanor | .
All Female ($\mathbf{x}_{A}^{F}$) | She | touches | the | heart | of | her | Aunt | Eleanor | .
All Male ($\mathbf{x}_{A}^{M}$) | He | touches | the | heart | of | his | Uncle | Zachary | .
Table 1: Example labeling of a sentence, showing the original sentence and the
four manipulated versions. Words marked as the ground truth for explanations
are written in bold and color coded depending on the grammatical gender.

For the dataset, we restrict ourselves to source sentences with a human
subject, such that each sentence of our manipulated dataset is guaranteed to
have a well-defined gender label. This type of sentence naturally occurs in
books and novels. The Gutenberg444https://www.gutenberg.org archive provides a
plethora of classical titles that allow one to identify relevant text content
from well-known novels or nonfiction titles. To comply with licensing
surrounding the listed books, we collect the content of their corresponding
Wikipedia pages, and for GECO, we only use text pieces related to the plot of
the story. We query the list of the top $100$ popular books on the Gutenberg
project and obtain their corresponding Wikipedia pages.

We create two ground-truth data sets $\mathcal{D}_{S}$ and $\mathcal{D}_{A}$.
Each contains $1610$ sentences in a male and a female version, comprising
$3220$ sentences in total. $\mathcal{D}_{S}$ contains sentences in which only
words specifying the gender of the grammatical subject are manipulated to be
either in male ($\mathbf{x}_{S}^{M}$) or female ($\mathbf{x}_{S}^{F}$) form,
while $\mathcal{D}_{A}$ contains sentences in which all gender-related words
are manipulated to be either male ($\mathbf{x}_{A}^{M}$) or female
($\mathbf{x}_{A}^{F}$). Table 1 shows an example sentence and the resulting
labeled versions. The process for creating these datasets consists of two
consecutive steps: (i) Pre-processing of scraped Wikipedia pages. (ii) A
manual labeling step detecting and adapting relevant (human) subjects and
protagonists of a sentence. More details on labeling and format are provided
in the Appendix A.1.1 and Appendix A.1.2.

###  2.2 Explanation Benchmarking

The alteration of sentences induces discriminative features by construction,
and their uniqueness automatically renders them the only viable ground truth
explanations, defining two different gender classification tasks represented
by the two datasets $\mathcal{D}_{S}$ and $\mathcal{D}_{A}$. By extension,
every word that is not grammatically gender-related, therefore not altered,
becomes a non-discriminative feature.

We train machine learning models on these tasks and apply post-hoc explanation
methods to the trained models to obtain ‘explanations’ expressing the
importance of features according to each XAI method’s intrinsic criteria. When
evaluating the XAI methods, ground truth explanations are adduced to measure
if their output highlights the correct features. An overview is shown in
Figure 1.

##### Ground truth explanations.

We consider a supervised learning task, where a model
$f:\mathbb{R}^{d}\to\mathbb{R}$ learns a function between an input
$\mathbf{x}^{(i)}\in\mathbb{R}^{d}$ and a target $y^{(i)}\in\\{-1,1\\}$, based
on training data $\mathcal{D}=\\{(\mathbf{x}^{(i)},y^{(i)})\\}_{i=1}^{N}$.
Here, $\mathbf{x}^{(i)}$ and $y^{(i)}$ are realizations of the random
variables $\mathbf{X}$ and $Y$, with joint probability density function
$p_{\mathbf{X},Y}(\mathbf{x},y)$. Formally, we cast the problem of finding an
explanation or ‘important’ features as a decision problem
$(\mathcal{P}(\mathbf{x}),\mathcal{F})$ with the power set of all feature
combinations and the set of important features
$\mathcal{F}\subseteq\mathcal{P}(\mathbf{x})$. Then we are interested in
finding a test $h:\mathbb{R}^{d}\to\\{0,1\\}^{d}$, defined by
$h_{j}(\mathbf{x})=1$ for $\mathbf{x}_{j}\in\mathcal{F}$ and
$h_{j}(\mathbf{x})=0$ for
$\mathbf{x}_{j}\in\mathcal{P}(\mathbf{x})\setminus\mathcal{F}$. For defining a
set of important features, we adopt the approach of Wilming et al. [54, 55]
and give the following

######  Definition 2.1 (Statistical Association Property).

Let the training data $\mathcal{D}$ represent a supervised learning task as
described above with realizations $\mathbf{x}^{(i)}$ and $y^{(i)}$ of the
random variables $\mathbf{X}$ and $Y$, where
$\mathbf{X}=(X_{1},\dots,X_{d})^{\top}$. We say that an XAI method has the
Statistical Association property if any feature $X_{j}$ with non-zero (or,
significantly larger than zero) importance also has a statistical dependency
to the target $Y$, i.e., $X_{j}\not\\!\perp\\!\\!\\!\perp Y$.

This definition is based on the observation that the discussion of most AI
explanations implicitly or explicitly assumes that such a statistical
association exists [54]. Now, we can define the set of _potentially_ important
features via their univariate statistical dependence with the target
$\mathcal{F}=\\{X_{j}\mid X_{j}\not\\!\perp\\!\\!\\!\perp
Y,\,\,\text{for}\,\,j\in[d]\\}$ with the set of feature indices
$[d]=\\{1,\dots,d\\}$.

Each sentence of the corpus and its corresponding token sequence
$\mathbf{x}^{(i)}$ has a matching ground truth map
$h^{(i)}(\mathbf{x}^{(i)})\in\\{0,1\\}^{d}$. In Figure 3 in the appendix, we
visualize the Pearson correlation of the term frequency–inverse document
frequency (tf-idf) [48] representation of words and the target, clearly
showing how we infuse dependency through the word alteration procedure.

##### Classifiers.

In our analysis, we focus on the popular BERT model [11], though one can
expand this work using other common language models such as RoBERTa [26],
XLNet [56] or GPT models [38, 39]. For all experiments, we use the pre-trained
uncased BERT model [11]555Hosted by Hugging Face:
https://huggingface.co/google-bert/bert-base-uncased. BERT runs on sequences
of length $d$ consisting of input tokens $\mathbf{x}_{j}$ for $j=1,\dots,d$,
which are processed as corresponding embeddings
$\mathbf{e}_{j}\in\mathbb{R}^{m}$ with embedding size $m$ and we represent a
sequence $\mathbf{x}^{(i)}\in\mathbb{R}^{d}$ via its embedding representation
$\mathbf{E}\in\mathbb{R}^{m\times d}$.

To investigate the impact that fine-tuning or re-training of different parts
of BERT’s architecture can have on explanation performance, we consider four
different training paradigms: (i) We roughly split BERT’s architecture into
three parts: Embedding, Attention, and Classification. The standard approach
adopting BERT for a new downstream task is to train the last classification
layer, which we call Classification while fixing the weights for all remaining
parts of the model, here Embedding and Attention. We thereby only train a
newly initialized classification layer and call the resulting model BERT-C.
(ii) We additionally train the Embedding layer from scratch resulting in a
model called BERT-CE. (iii) In the third model, BERT-CEf, the embeddings are
fine-tuned as opposed to newly initialized. In training paradigm (iv), we
fine-tune the Embedding and Attention parts of BERT’s architecture, resulting
in model BERT-CEfAf. Moreover, a vanilla one-layer attention model, OLA-CEA,
comprising a lower dimensional embedding layer, one attention layer, and a
classification layer, was trained from scratch only on the GECO dataset. We
refer to Table 2 for a summary of these models and training schemes. More
details on the model training and performance are given in the appendix A.2.1.

All models, except the OLA-CEA model on Dataset $\mathcal{D}_{s}$, achieve an
accuracy above 80% on the test set, which we consider as evidence that the
model has successfully generalized on the given downstream task. In the
following analysis, we consider all models, including the OLA-CEA model
trained on $\mathcal{D}_{s}$.

Models | Embedding | Attention | Classification | Acc. $\mathcal{D}_{A}^{test}$ (%) | Acc. $\mathcal{D}_{S}^{test}$ (%)
---|---|---|---|---|---
BERT-C | fix | fix | re-trained | 99.2 $\pm$ 0.2 | 89.7 $\pm$ 0.8
BERT-CE | re-trained | fix | re-trained | 98.7 $\pm$ 0.9 | 86.3 $\pm$ 1.3
BERT-CEf | fine-tuned | fix | re-trained | 98.0 $\pm$ 1.0 | 86.6 $\pm$ 3.3
BERT-CEfAf | fine-tuned | fine-tuned | re-trained | 99.4 $\pm$ 0.3 | 98.1 $\pm$ 0.3
OLA-CEA | re-trained | re-trained | re-trained | 95.7 $\pm$ 2.7 | 77.7 $\pm$ 0.4
Table 2: Overview of BERT transfer learning paradigms and the performance of
the resulting models on the test datasets $\mathcal{D}_{A}^{test}$ and
$\mathcal{D}_{S}^{test}$. We divided the BERT architecture into three parts:
Embedding, Attention, and Classification. Depending on the training scheme, we
re-traine specific parts from scratch with newly initialized weights while
keeping all other model parts fix. Instead of re-training, some schemes also
just fine-tune specific parts, which refers to re-training using pre-trained
weights as initializations. Re-training and fine-tuning was performed on
balanced GECO data only.

##### Explanation methods.

Most of the most popular feature attribution methods can be applied post-hoc,
given a trained model. Here, we focus on post-hoc attribution methods, which
can be broadly divided into gradient-based methods and local sampling or
surrogate approaches. Generally, these methods produce an explanation
$\mathbf{s}(f,\mathbf{x}^{*})\in\mathbb{R}^{d}$, which is a mapping that
depends on the model $f$ and an instance $\mathbf{x}^{*}$ to be explained.
Gradient-based methods locally approximate a differentiable model $f$ around a
given input sequence of word embeddings $\mathbf{E}\in\mathbb{R}^{m\times d}$.
From this class, we consider Saliency [47], InputXGradient [45], DeepLift
[46], Guided Backpropagation [49], and Integrated Gradients [50]. Surrogate
models, on the other hand, sample around the input $\mathbf{x}^{*}$ and use a
model’s output $f(\mathbf{x})$ to train a simple, usually linear, model and
interpret $f$ through this local approximation. In this work, we consider the
surrogate methods LIME [41] and Kernel SHAP [27]. Additionally, our study
includes Gradient SHAP [27], an approximation of Shapley value sampling.

We also consider two baselines as a point of comparison for calculating
explanation performance of the above methods. Firstly, we set the explanation
for a particular input sequence $\mathbf{x}^{*}$ to uniformly distributed
noise $\mathbf{s}(f,\mathbf{x}^{*})\sim\mathcal{U}[(0,1)^{d}]$. This serves as
a null model correspoding to the hypothesis that the XAI method has no
knowledge of the informative features. Secondly, we employ the Pattern
approach [17, 54]. This approach requires a linear classifier, yet we can
adopt it by using the covariance between input features and target
$\mathbf{s}_{j}(f,\mathbf{x}^{*})=\mathrm{Cov}(\mathbf{x}_{j}^{*},y)$. We call
this the Pattern Variant, for which we utilized the tf-idf [48] representation
of each input sequence $\mathbf{x}^{(i)}$. Clearly, the explanation
$\mathbf{s}$ is independent of both the model $f$ and an instance
$\mathbf{x}^{*}$; therefore, it yields the same explanations for all input
sequences.

We apply these XAI methods to all fine-tuning variants of the BERT model and
compute explanations on all test data sentences using the default parameters
of each method. For all XAI methods except LIME, we use their Captum [25]
implementation. For LIME, we use the author’s original
code666https://github.com/marcotcr/lime.

##### Explanation performance measures.

For a given instance $\mathbf{x}^{*}\in\mathcal{D}^{test}$ we aim to
quantitatively assess the correctness of its explanation
$\mathbf{s}(f,\mathbf{x}^{*})$. The ground truth $h(\mathbf{x}^{*})$ defines a
set of potentially important tokens based on how we generated or altered words
in sentences of the corpus; however, a model might only use a subset of such
tokens for its predictions. Hence, we employ the Mass Accuracy metric
$\mathrm{MA}\left(h(\mathbf{x}^{*}),\mathbf{s}(f,\mathbf{x}^{*})\right)$[3, 9,
10], which limits the penalization of false negatives. A detailed discussion
can be found in the appendix A.3.

##  3 Experiments and Results

Figure 2: Explanation performance of different post-hoc XAI methods applied to
language models that were adapted from BERT using five different transfer
learning schemes. XAI evaluations were carried out only on correctly
classified sentences in two gender-classification tasks, represented by
datasets $\mathcal{D}_{S}$ and $\mathcal{D}_{A}$. Baseline performance for
uniformly drawn random explanations is denoted by Uniform Random. Pattern
Variant denotes a model- and pretraining-agnostic global explanation method.
Fine-tuning or retraining of the embedding layers of BERT leads to consistent
improvements in explanation performance even when model performance is held
constant for all models. Applying XAI methods to the OLA model is leading to
overall higher explanation performance, with InputXGradient becoming on par
with Pattern Variant.

We conduct experiments to study the influence of biased models on explanation
performance using the unbiased dataset GECO. Therefore, we fine-tune and re-
train four BERT configurations and one simple one-layer attention model with
five repetitions each, each configuration defined by a different choice of
random seeds. Afterwards, we apply XAI methods and choose only correctly
classified samples to assess explanation performance. Table 2 lists the
training and fine-tuning configurations with corresponding model performances.

###  3.1 Explanation Performance on GECO

Figure 2 shows the explanation performance of individual sample-base
explanation maps $\mathbf{s}(f,\mathbf{x}^{(i)})$ produced by the selected XAI
and baseline methods. These results demonstrate how asymmetries in
explanations are related to biases in pre-trained models and how popular XAI
methods perform under different transfer learning regimes.

Comparing explanation performance between datasets $\mathcal{D}_{A}$ and
$\mathcal{D}_{S}$, we observe a general difference in mass accuracy. While,
for the majority of XAI methods, the performance for experiments on dataset
$\mathcal{D}_{S}$ stays on a level lower than $0.25$, experiments on dataset
$\mathcal{D}_{A}$ are often able to offset these results into levels above
$0.25$. However, dataset $\mathcal{D}_{S}$ has fewer altered gender words,
thus fewer discriminative tokens, leading to an overall degradation of
classification accuracy across all models, which also impacts explanation
performance. For all BERT models and both datasets, Integrated Gradients
consistently outperforms other XAI methods and the Uniform random baseline.
Looking closer into the results for dataset $\mathcal{D}_{S}$, LIME and
Gradient SHAP are on par with Integrated Gradients in almost every fine-tuning
stage, yet both still have low explanation performance. All other methods
repeatedly perform similarly or only slightly better than the random baseline.
For dataset $\mathcal{D}_{A}$ with a richer set of discriminative tokens,
Integrated Gradients, together with LIME and Gradient SHAP, are the highest
performing XAI methods for all BERT models, compared to the Pattern Variant
baseline. For both data scenarios $\mathcal{D}_{S}$ and $\mathcal{D}_{A}$, it
is hard to see any reliable trend with respect to the fine-tuning stages and
explanation performance. However, for the scenario $\mathcal{D}_{A}$, it is
clear that the models BERT-CE and BERT-CEf, where the embedding layer was
trained or fine-tuned respectively, outperform BERT-C and BERT-CEfAf (see
Figure 9(a) in the appendix). This shows that the embeddings encode a lot of
biased information and, indeed, influence explanation performance. Although no
XAI method achieves the correctness score of Pattern Variant, fine-tuning a
biased embedding layer for a downstream task does impact the output for some
XAI methods drastically. The Pattern Variant is a model-independent global
explanation method only relying on the intrinsic structure of the data itself,
that performs optimally when feature and target relation is governed by a
linear relationship, which is mainly the case for GECO (see Figure 3 in the
appendix).

Comparing the OLA-CEA model to all BERT models, we observe a stark contrast in
explanation performance. Recall that the OLA-CEA model was purely trained from
scratch on the gender-controlled dataset GECO; hence, it does not suffer from
any gender bias. The mass accuracy between both datasets is similar, with
higher variance for some methods, such as Guided Backprop and DeepLift, yet
the overall performance of almost all XAI methods significantly increases
compared to the random baseline. In addition to relatively well-performing
methods like Integrated Gradients, LIME, and Gradient SHAP, the mass accuracy
of InputXGradient comes very close to the covariance baseline, making it the
best-performing method.

We observe lower performance for all attribution methods when assessing
explanation performance for all sentences, including missclassified sentences.
Respective results are shown in the appendix in Figure 9(b).

In Figure 6 in the appendix, a sentence labeled as ‘female’ is shown together
with its word-based explanations as bar plots for each fine-tuning stage. We
observe a high variability in token attribution between differently fine-tuned
BERT models and that the pronoun ‘she’ receives relatively high importance
compared to other words. However, not all XAI methods agree on the importance
of the token ‘she’; for example, for model BERT-CE InputXGradient attributes
high importance to it, yet for model BERT-CEfAf, it attributes rather high
importance to the word ‘Bella.’

By construction, GECO includes a male and female version of each original
sentence. This allows us to investigate the attribution difference
$\Delta\mathbf{s}=|\mathbf{s}(f,\mathbf{x}^{F})-\mathbf{s}(f,\mathbf{x}^{M})|$
for attributions of semantically equivalent words in the male and female
version of a sentence. Figure 7 in the appendix shows noticeable differences
for ground truth words $h(\mathbf{x}^{*})$, especially for the $D_{S}$
dataset. Generally we observe that XAI methods with high variability also have
high differences in word-based importance attribution depending on gender. We
applied the Wilcoxon signed-rank test to determine if the difference is
significant, which is the case for most training schemes and methods for
different training repetitions, as shown in Figure 8 in the appendix.

###  3.2 Bias Analysis

A bias analysis was conducted to assess the impartiality of the models
themselves, examining whether the application of the unbiased GECO dataset
resulted in truly neutral outputs, as well as identifying any discernible
differences between the five distinct training schemes employed. As
demonstrated in Table 4 of section A.2.2 in the appendix, the examined models
exhibit comparable performance across male and female input sentences, yet the
outcomes still indicate the presence of gender biases in their predictions. A
pattern emerges across the five distinct training schemes, wherein the
difference in output probabilities between male- and female-sentences
diminishes as the number of retrained or finetuned layers increases. As a
quality control measure, we conducted a bias analysis on the GECO dataset
itself, as detailed in in the appendix A.1.3, which confirmed that GECO
exhibits no significant bias with respect to gender.

##  4 Discussion

With GECO and GECOBench, we propose a rigorous open framework for benchmarking
the correctness of explanations of pre-trained language models as well as
aspects of fairness. Our initial results demonstrate (a) differences in
explanation performance between XAI approaches, (b) a general dependency of
explanation performance on the amount of re-training/fine-tuning of BERT
models, and, (c) residual gender biases as contributors to sub-par explanation
performance.

Top-performing XAI methods notably include Pattern Variant, a global and
model-independent approach. In cases where features and target are linearly
correlated, as in GECO, Pattern Variant indeed offers strong theoretical
justification for detecting important features according to statistical
associations [17]. Since Pattern Variant is model-independent, it cannot be
impacted by gender biases in pre-training corpora, establishing a solid
baseline for the upper bound of explanation performance in our benchmark.
Compared to the random baseline, we observe two further high-performing XAI
methods in the transfer learning regime, Integrated Gradients and Gradient
SHAP. Yet these methods still do not reach the Mass Accuracy level of Pattern
Variant. The reasons can be two-fold: (i) As shown by Clark et al. [9] and
Wilming et al. [55], XAI methods consistently attribute importance to
suppressor variables, features not statistically associated with the target
but nevertheless utilized by machine learning model to increase accuracy. And
(ii), model bias impacts explanations. We show that the gender bias contained
in BERT leads to residual asymmetries in explanations and forms a consistent
pattern of deviation depending on which layer of BERT was fine-tuned or re-
trained, while still achieving equivalent classification accuracy. As a
result, updating embedding layers has the strongest impact on explanations.

While this is, to our knowledge, the first XAI benchmark addressing a well-
defined notion of explanation correctness in the NLP domain, we do not
consider it an exhaustive evaluation analysis of XAI methods but rather as a
first step towards this. A possible limitation of our approach is that the
criterion of univariate statistical association used here to define XAI does
not include non-linear feature interactions that are present in many real-
world applications. However, for analyzing the fundamental behaviors of XAI
methods, this simplistic characteristic allows for straightforward evaluation
strategies, permitting us to embed these statistical properties into the
proposed corpus and to establish a ground truth of word relevance. Designing
metrics for evaluating explanation performance – especially for measuring
correctness – is another subject that requires further research. Future work
will also focus on enriching GECO with more labels, one of which will be a
sentiment for each sentence of the corpus. This will also allow for fairness
analyses investigating the interplay between protected attributes like labeled
gender words and explanations. Finally, collecting sufficient data and
inducing an artificially constructed bias which allows for pre-training
language models with known bias under more controllable conditions.

##  5 Conclusion

We have introduced GECO – a novel gender-controlled ground-truth text dataset
designed for the development and evaluation of XAI methods – and GECOBench – a
quantitative benchmarking framework to perform objective assessments of XAI
explanation performance for language models. We showcased the use of both
tools by investigating the pre-trained language model BERT, which suffers from
gender biases. We demonstrated that residual biases affect explanations of
models tuned for downstream tasks, and that the fine-tuning and re-training of
different layers of BERT can positively impact explanation performance.

## Acknowledgements

This result is part of a project that has received funding from the European
Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation programme (Grant agreement No. 758985), and the German Federal
Ministry for Economy and Climate Action (BMWK) in the frame of the QI-Digital
Initiative.

##  6 The ML Paper Reproducibility Checklist (as per [37], v2.0)

  1. i.

For all models and algorithms presented, check if you include:

    1. (a)

A clear description of the mathematical setting, algorithm, and/or model.
[Yes]

    2. (b)

A clear explanation of any assumptions. [Yes]

    3. (c)

An analysis of the complexity (time, space, sample size) of any algorithm
[Yes]

  2. ii.

For any theoretical claim, check if you include:

    1. (a)

A clear statement of the claim. [N/A]

    2. (b)

A complete proof of the claim. [N/A]

  3. iii.

For all datasets used, check if you include:

    1. (a)

The relevant statistics, such as number of examples. [Yes]

    2. (b)

The details of train / validation / test splits. [Yes]

    3. (c)

An explanation of any data that were excluded, and all pre-processing step
[Yes]

    4. (d)

A link to a downloadable version of the dataset or simulation environment
[Yes]

    5. (e)

For new data collected, a complete description of the data collection process,
such as instructions to annotators and methods for quality control [Yes]

  4. iv.

For all shared code related to this work, check if you include:

    1. (a)

Specification of dependencies. [Yes]

    2. (b)

Training code. [Yes]

    3. (c)

Evaluation code. [Yes]

    4. (d)

(Pre-)trained model(s). [Yes]

    5. (e)

README file includes table of results accompanied by precise command to run to
produce those results [Yes]

  5. v.

For all reported experimental results, check if you include:

    1. (a)

The range of hyper-parameters considered, method to select the best hyper-
parameter configuration, and specification of all hyper-parameters used to
generate results. [Yes]

    2. (b)

The exact number of training and evaluation runs [Yes]

    3. (c)

A clear definition of the specific measure or statistics used to report
results. [Yes]

    4. (d)

A description of results with central tendency (e.g. mean) and variation (e.g.
error bars). [Yes]

    5. (e)

The average runtime for each result, or estimated energy cost. [Yes]

    6. (f)

A description of the computing infrastructure used. [Yes]

## References

  * Agarwal et al. [2022] C. Agarwal, S. Krishna, E. Saxena, M. Pawelczyk, N. Johnson, I. Puri, M. Zitnik, and H. Lakkaraju.  OpenXAI: Towards a transparent evaluation of model explanations.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.
  * Ahn and Oh [2021] J. Ahn and A. Oh.  Mitigating Language-Dependent Ethnic Bias in BERT.  In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 533–549, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
  * Arras et al. [2022] L. Arras, A. Osman, and W. Samek.  CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations.  _Information Fusion_ , 81:14–40, May 2022.  ISSN 1566-2535.
  * Beukeboom [2014] C. J. Beukeboom.  Mechanisms of linguistic bias: How words reflect and maintain stereotypic expectancies.  In _Social cognition and communication_ , pages 313–330. Psychology Press, 2014.
  * Borgonovo et al. [2023] E. Borgonovo, V. Ghidini, R. Hahn, and E. Plischke.  Explaining classifiers with measures of statistical association.  _Computational Statistics & Data Analysis_, 182:107701, 2023.
  * Bugliarello et al. [2021] E. Bugliarello, R. Cotterell, N. Okazaki, and D. Elliott.  Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs.  _Transactions of the Association for Computational Linguistics_ , 9:978–994, 2021.  Place: Cambridge, MA Publisher: MIT Press.
  * Cabello et al. [2023] L. Cabello, E. Bugliarello, S. Brandl, and D. Elliott.  Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models, Oct. 2023.  URL http://arxiv.org/abs/2310.17530.  arXiv:2310.17530 [cs].
  * Chen et al. [2020] H. Chen, J. D. Janizek, S. Lundberg, and S.-I. Lee.  True to the Model or True to the Data?  2020\.
  * Clark et al. [2023] B. Clark, R. Wilming, and S. Haufe.  Xai-tris: Non-linear benchmarks to quantify ml explanation performance.  _arXiv preprint arXiv:2306.12816_ , 2023.
  * Clark et al. [2024] B. Clark, R. Wilming, A. Dox, P. Eschenbach, S. Hached, D. J. Wodke, M. T. Zewdie, U. Bruila, M. Oliveira, H. Schulz, L. M. Cornils, D. Panknin, A. Boubekki, and S. Haufe.  Exact: Towards a platform for empirically benchmarking machine learning model explanation methods, 2024.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of naacL-HLT_ , volume 1, page 2, 2019.
  * DeYoung et al. [2020] J. DeYoung, S. Jain, N. F. Rajani, E. Lehman, C. Xiong, R. Socher, and B. C. Wallace.  ERASER: A Benchmark to Evaluate Rationalized NLP Models.  In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4443–4458, Online, 2020. Association for Computational Linguistics.
  * Dodge et al. [2020] J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _arXiv preprint arXiv:2002.06305_ , 2020.
  * Friedman and Wall [2005] L. Friedman and M. Wall.  Graphical Views of Suppression and Multicollinearity in Multiple Linear Regression.  _The American Statistician_ , 59(2):127–136, 2005.
  * Fryer et al. [2020] D. V. Fryer, I. Strümke, and H. Nguyen.  Explaining the data or explaining a model? shapley values that uncover non-linear dependencies.  _arXiv preprint arXiv:2007.06011_ , 2020.
  * Graells-Garrido et al. [2015] E. Graells-Garrido, M. Lalmas, and F. Menczer.  First women, second sex: Gender bias in Wikipedia.  In _Proceedings of the 26th ACM conference on hypertext & social media_, pages 165–174, 2015.
  * Haufe et al. [2014] S. Haufe, F. Meinecke, K. Görgen, S. Dähne, J.-D. Haynes, B. Blankertz, and F. Bießmann.  On the interpretation of weight vectors of linear models in multivariate neuroimaging.  _NeuroImage_ , 87:96–110, 2014.
  * Hooker et al. [2019] S. Hooker, D. Erhan, P.-J. Kindermans, and B. Kim.  A Benchmark for Interpretability Methods in Deep Neural Networks.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Ismail et al. [2019] A. A. Ismail, M. Gunady, L. Pessoa, H. C. Bravo, and S. Feizi.  Input-Cell Attention Reduces Vanishing Saliency of Recurrent Neural Networks.  pages 10814–10824, 2019.
  * Ismail et al. [2020] A. A. Ismail, M. Gunady, H. Corrada Bravo, and S. Feizi.  Benchmarking deep learning interpretability in time series predictions.  2020\.
  * Jentzsch and Turan [2022] S. Jentzsch and C. Turan.  Gender Bias in BERT - Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task.  In _Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)_ , pages 184–199, Seattle, Washington, 2022. Association for Computational Linguistics.
  * Jiménez-Luna et al. [2020] J. Jiménez-Luna, F. Grisoni, and G. Schneider.  Drug discovery with explainable artificial intelligence.  _Nature Machine Intelligence_ , 2(10):573–584, 2020.
  * Kim et al. [2018] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. Sayres.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).  In _International Conference on Machine Learning_ , pages 2668–2677. PMLR, 2018.
  * Kiros et al. [2015] R. Kiros, Y. Zhu, R. R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba, and S. Fidler.  Skip-Thought Vectors.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc., 2015.  URL https://papers.nips.cc/paper_files/paper/2015/hash/f442d33fa06832082290ad8544a8da27-Abstract.html.
  * Kokhlikyan et al. [2020-09-16] N. Kokhlikyan, V. Miglani, M. Martin, E. Wang, B. Alsallakh, J. Reynolds, A. Melnikov, N. Kliushkina, C. Araya, S. Yan, and O. Reblitz-Richardson.  Captum: A unified and generic model interpretability library for PyTorch.  2020-09-16.
  * Liu et al. [2019] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov.  RoBERTa: A Robustly Optimized BERT Pretraining Approach, July 2019.  URL http://arxiv.org/abs/1907.11692.  arXiv:1907.11692 [cs].
  * Lundberg and Lee [2017] S. M. Lundberg and S.-I. Lee.  A Unified Approach to Interpreting Model Predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 4765–4774. Curran Associates, Inc., 2017.
  * Lundberg et al. [2018] S. M. Lundberg, B. Nair, M. S. Vavilala, M. Horibe, M. J. Eisses, T. Adams, D. E. Liston, D. K.-W. Low, S.-F. Newman, J. Kim, and S.-I. Lee.  Explainable machine-learning predictions for the prevention of hypoxaemia during surgery.  _Nature Biomedical Engineering_ , 2(10):749–760, 2018.
  * Mikolov et al. [2013] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean.  Distributed Representations of Words and Phrases and their Compositionality.  In _Advances in Neural Information Processing Systems_ , volume 26. Curran Associates, Inc., 2013.  URL https://papers.nips.cc/paper_files/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html.
  * Mitchell [2007] T. M. Mitchell.  The need for biases in learning generalizations.  _In Rutgers CS tech report_ , CBM-TR-117, 2007.
  * Montañez et al. [2019] G. D. Montañez, J. Hayase, J. Lauw, D. Macias, A. Trikha, and J. Vendemiatti.  The futility of bias-free learning and search.  In _Australasian Joint Conference on Artificial Intelligence_ , pages 277–288. Springer, 2019.
  * Murdoch et al. [2019] W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu.  Definitions, methods, and applications in interpretable machine learning.  116(44):22071–22080, 2019.
  * Nadeem et al. [2021] M. Nadeem, A. Bethke, and S. Reddy.  StereoSet: Measuring stereotypical bias in pretrained language models.  In C. Zong, F. Xia, W. Li, and R. Navigli, editors, _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5356–5371, Online, Aug. 2021. Association for Computational Linguistics.
  * Nangia et al. [2020] N. Nangia, C. Vania, R. Bhalerao, and S. R. Bowman.  CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models.  In B. Webber, T. Cohn, Y. He, and Y. Liu, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1953–1967, Online, Nov. 2020. Association for Computational Linguistics.
  * Oliveira et al. [2023] M. Oliveira, R. Wilming, B. Clark, C. Budding, F. Eitel, K. Ritter, and S. Haufe.  Benchmark data to study the influence of pre-training on explanation performance in mr image classification, 2023.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _CoRR_ , abs/2303.08774, 2023.  doi: 10.48550/ARXIV.2303.08774.  URL https://doi.org/10.48550/arXiv.2303.08774.
  * Pineau et al. [2021] J. Pineau, P. Vincent-Lamarre, K. Sinha, V. Larivière, A. Beygelzimer, F. d’Alché Buc, E. Fox, and H. Larochelle.  Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program).  _The Journal of Machine Learning Research_ , 22(1):7459–7478, 2021.
  * Radford et al. [2018] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, and others.  Improving language understanding by generative pre-training.  2018\.  Publisher: OpenAI.
  * Radford et al. [2019] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Reagle and Rhue [2011] J. Reagle and L. Rhue.  Gender bias in wikipedia and britannica.  _International Journal of Communication_ , 5:21, 2011.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin.  " Why should I trust you?" Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 1135–1144, 2016.
  * Rong et al. [2022] Y. Rong, T. Leemann, V. Borisov, G. Kasneci, and E. Kasneci.  A consistent and efficient evaluation strategy for attribution methods.  In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, _Proceedings of the 39th international conference on machine learning_ , volume 162 of _Proceedings of machine learning research_ , pages 18770–18795. PMLR, July 2022.
  * Rychener et al. [2020] Y. Rychener, X. Renard, D. Seddah, P. Frossard, and M. Detyniecki.  QUACKIE: A NLP Classification Task With Ground Truth Explanations.  _arXiv:2012.13190 [cs, stat]_ , Dec. 2020.  arXiv: 2012.13190.
  * Samek et al. [2016] W. Samek, A. Binder, G. Montavon, S. Lapuschkin, and K.-R. Müller.  Evaluating the visualization of what a deep neural network has learned.  _IEEE Transactions on Neural Networks and Learning Systems_ , 28(11):2660–2673, 2016.
  * Shrikumar et al. [2016] A. Shrikumar, P. Greenside, A. Shcherbina, and A. Kundaje.  Not just a black box: Learning important features through propagating activation differences, 2016.
  * Shrikumar et al. [2017] A. Shrikumar, P. Greenside, and A. Kundaje.  Learning important features through propagating activation differences.  In _International conference on machine learning_ , pages 3145–3153. PMLR, 2017.
  * Simonyan et al. [2014] K. Simonyan, A. Vedaldi, and A. Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In _Workshop at International Conference on Learning Representations_ , 2014.
  * Sparck Jones [1972] K. Sparck Jones.  A statistical interpretation of term specificity and its application in retrieval.  _Journal of documentation_ , 28(1):11–21, 1972.
  * Springenberg et al. [2015] J. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller.  Striving for simplicity: The all convolutional net.  In _ICLR (workshop track)_ , 2015.
  * Sundararajan et al. [2017] M. Sundararajan, A. Taly, and Q. Yan.  Axiomatic attribution for deep networks.  In D. Precup and Y. W. Teh, editors, _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3319–3328. PMLR, 06–11 Aug 2017.
  * Tjoa and Cuntai [2022] E. Tjoa and G. Cuntai.  Quantifying explainability of saliency methods in deep neural networks with a synthetic dataset.  2022\.
  * Tran et al. [2021] K. A. Tran, O. Kondrashova, A. Bradley, E. D. Williams, J. V. Pearson, and N. Waddell.  Deep learning in cancer diagnosis, prognosis and treatment selection.  _Genome Medicine_ , 13(1):152, 2021.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is All you Need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 5998–6008. Curran Associates, Inc., 2017.
  * Wilming et al. [2022] R. Wilming, C. Budding, K.-R. Müller, and S. Haufe.  Scrutinizing xai using linear ground-truth data with suppressor variables.  _Machine learning_ , 111(5):1903–1923, 2022.
  * Wilming et al. [2023] R. Wilming, L. Kieslich, B. Clark, and S. Haufe.  Theoretical behavior of XAI methods in the presence of suppressor variables.  In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 37091–37107. PMLR, 23–29 Jul 2023.
  * Yang et al. [2019] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Zhang et al. [2022] Y. Zhang, Y. Weng, and J. Lund.  Applications of Explainable Artificial Intelligence in Diagnosis and Surgery.  _Diagnostics_ , 12(2):237, 2022.
  * Zhao et al. [2017] J. Zhao, T. Wang, M. Yatskar, V. Ordonez, and K.-W. Chang.  Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints.  In M. Palmer, R. Hwa, and S. Riedel, editors, _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2979–2989, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 GECO Dataset

In the following we describe the data generation and format in more detail and
perform a bias analysis as a sanity check to verify the dataset is unbiased.

####  A.1.1 Data Generation

We accessed the top 100 list of popular books on Project Gutenberg on 17th of
March 2022 and to obtain the corresponding wikipedia articles we ran google
queries. For the task of web scraping, we use the software
Selenium777https://www.selenium.dev/. After scraping the sentences from the
wikipedia articles of the books, we preprocess the sentences using the Python
library Spacy888https://spacy.io.

In the following, we provide addtional details of our data processing rules as
part of the data generation process. We employ Spacy to only include sentences
with root verbs in the 3rd person singular.

By applying a set of filtering criteria to the raw sentences, we remove
sentences that are overly long (> 30 tokens), where the subject is neutral,
usually expressed via the word ’it’, lack punctuation (no period at the end),
mention author names, or contain duplicate information. We also exclude
sentences without common nouns related to humans, those where the subject is
not part of the plot, and those containing citations or proper nouns that
appear only once, as these elements may not contribute significantly to the
story’s narrative.

We ensure that the subject of a sentence either corresponds to proper nouns,
pronouns ‘he’ and ‘she’, or common nouns referring to a human beeing.
Furthermore, we make certain that sentences are grammatically consistent and
that the content of a sentence is part of the plot and does not contain other
trivia about the author or book interpretations. The labeling step consists of
locating the subject and other protagonists of a sentence and changing them to
their male or female version, respectively.

We attempted to employ fully automated sentence labeling using GPT-4 [36], but
encountered inconsistencies in identifying names, genders and gendered terms,
as well as detecting human subjects, particularly in dataset
$\mathcal{D}_{S}$. Due to the need for precise ground truth labels to
benchmark various explanation methods, we opted for a manual labeling approach
instead.

Figure 3: Pearson correlation between tf-idf representation of words and the
target. Here, we see the top ten words by correlation, and labeled words such
as the pronounce he and she or his and her are consistently ranked highest in
both datasets $\mathcal{D}_{A}$ and $\mathcal{D}_{S}$, indicating how the
labeling introduced dependency between target and words.

####  A.1.2 Data Format

The datasets are available in the following folder structure.

.1 GECO. .2 data_config.json. .2 gender_all. .3 test.jsonl (644 Sentences). .3
train.jsonl (2576 Sentences). .2 gender_subj. .3 test.jsonl (644 Sentences).
.3 train.jsonl (2576 Sentences).

The sentences are available in jsonl files, with each line representing a
sentence in either male or female form. An example sentence can be seen in
Listing 1. Each line contains the input sentence as a list of words as well as
the explanation ground truth for each word. The ‘target’ field indicates
whether the sentence is in female form (0) or male form (1). Lastly, the
‘sentence_idx’ field identifies which original sentence was altered and can be
used to match the male and female form of a sentence.

⬇

{

"sentence": ["Paul", "loves", "his", "dog"],

"ground_truth": [1.0, 0.0, 1.0, 0.0],

"target": 1,

"sentence_idx": 0

}

Listing 1: Example sentence of the class $\mathbf{x}_{A}^{M}$.

####  A.1.3 Bias Measures

We employ a co-occurrence metric [58], a rudimentary bias measure,
highlighting the unbiasedness of GECO. We adopt the co-occurrence metric
proposed by Cabello et al. [7] to measure the gender bias of the datasets
$\mathcal{D}_{S}$ and $\mathcal{D}_{A}$. For a given sentence
$S\in\mathcal{D}$, we approximately measure the bias induced by grammatical
gender when considering the co-occurrence between the sentence’s gender terms
and the remaining words. First, we define a set of grammatical gender terms
$A\coloneqq\\{"she","her","he","his",\dots\\}$ and second a word vocabulary
without grammatical gender terms $V\coloneqq W\setminus A$, where the
vocabulary $W$ contains all words available in a corpus of $\mathcal{D}$,
then, the co-occurrence metric is defined as

$C(\mathcal{D})=\sum_{S\in\mathcal{D}}\sum_{w\in V}\sum_{a\in
A}\mathbf{1}_{S}(a,b).$ (1)

We denote subset of male sentences in a given dataset $\mathcal{D}$ with
$\mathcal{D}^{M}$ and the subset of female sentences with $\mathcal{D}^{F}$.
For each subset we apply the the co-occurence metric. We define the bias in
the dataset $\mathcal{D}$ as

$\operatorname{bias}_{C}(\mathcal{D})=\frac{C(\mathcal{D}^{F})}{C(\mathcal{D}^{F})+C(\mathcal{D}^{M})}.$
(2)

A perfect balance is achieved when $\operatorname{bias}_{C}(\mathcal{D})$
equals 0.5, signifying that the dataset is evenly distributed between males
and females. Deviations from this value indicate the presence of biases:
values closer to 0 suggest a male bias, while those approaching 1 indicate a
female bias. Our bias analysis shows that there is no gender bias present in
the $\mathcal{D}_{A}$ dataset in terms of the co-occurence measure (2) with
$\operatorname{bias}_{C}(\mathcal{D}_{A})=0.499$. The small difference to a
perfect score can be attributed to labeling errors. As expected, we see a
larger difference in the $\mathcal{D}_{S}$ dataset, with
$\operatorname{bias}_{C}(\mathcal{D}_{S})=0.476$. The difference is expected
due to the construction of the dataset, as we only change the human subject of
the sentence and other gender terms, refering to other humans in the sentence
are kept unchanged.

###  A.2 Models

In the following we provide further details about the model training and
provide detailed metrics about the models as accuracy, confusion matrices and
roc curves for all training schemes and datasets. All models are implemented
in PyTorch.

####  A.2.1 Training

For model training, we split the two datasets $\mathcal{D}_{S}$ and
$\mathcal{D}_{A}$ into training and test sets
$\mathcal{D}_{S}^{train},\mathcal{D}_{S}^{test}$ and
$\mathcal{D}_{A}^{train},\mathcal{D}_{A}^{test}$. For each model, we train
multiple repetitions with a different random seed for each repetition. This is
done not only to compensate variations in model performance [13, 6] but also
to capture the resulting variance in model explanations.

We optimize the learning rate and keep the remaining hyperparameters fixed. A
full overview of all hyperparameters and values we use is shown in Table 3.
All models were trained on an Nvidia A100 (40GB) GPU. For hyperparameter
optimization and the final model training for the 5 different training schemes
and the two datasets we performed 464 training runs with an average running
time of 68 seconds.

An overview of the performance of the models on the training, validation and
test set is shown in Table 4.

Models | Batch Size | Embedding Dimension | Epochs | Learning Rate
---|---|---|---|---
BERT-C | 32 | 768 | 20 | 0.01
BERT-CE | 32 | 768 | 20 | 0.0001
BERT-CEf | 32 | 768 | 20 | 0.01
BERT-CEfAf | 32 | 768 | 20 | 0.000005
OLA-CEA | 64 | 64 | 200 | 0.01
Table 3: Overview of BERT training schemes and values of the hyperparameters
used to train them. Figure 4: Average accuracy of the different training
schemes on the training, validation and test data of the two datasets
$\mathcal{D}_{A}$ and $\mathcal{D}_{S}$.

####  A.2.2 Bias Analysis

Metrics (%) for $\mathcal{D}_{S}$ |  BERT-C |  BERT-CE |  BERT-CEf |  BERT-CEfAf |  OLA-CEA
---|---|---|---|---|---
True Positive Rate | 91.12$\pm$ 0.01 | 87.52$\pm$ 0.04 | 84.97$\pm$ 0.06 | 98.51$\pm$ 0.00 | 79.57 $\pm$ 0.06
True Negative Rate | 88.20$\pm$ 0.02 | 85.09$\pm$ 0.05 | 88.20$\pm$ 0.02 | 97.76$\pm$ 0.01 | 75.78 $\pm$ 0.05
Avg. Prediction Diff. | 9.68 $\pm$ 0.37 | 4.13 $\pm$ 0.79 | 4.46 $\pm$ 0.59 | 0.62 $\pm$ 0.38 | 2.48 $\pm$ 1.64

Metrics (%) for $\mathcal{D}_{A}$ |  BERT-C |  BERT-CE |  BERT-CEf |  BERT-CEfAf |  OLA-CEA
---|---|---|---|---|---
True Positive Rate | 100.00$\pm$ 0.00 | 99.50$\pm$ 0.01 | 98.82$\pm$ 0.01 | 99.75$\pm$ 0.00 | 97.52$\pm$ 0.02
True Negative Rate | 98.39$\pm$ 0.00 | 97.89$\pm$ 0.02 | 97.08$\pm$ 0.02 | 99.01$\pm$ 0.00 | 93.91$\pm$ 0.06
Avg. Prediction Diff. | 1.78 $\pm$ 0.07 | 1.60 $\pm$ 0.87 | 1.07 $\pm$ 0.52 | 0.17 $\pm$ 0.10 | 0.81 $\pm$ 0.09

Table 4: Classification metrics and average prediction difference for the
different training schemes. The True Positive Rate (TPR) measures the
proportion of accurately identified male sentences of all male sentences,
whereas the True Negative Rate (TNR) quantifies the proportion of correctly
classified female sentences from the entire set of female sentences.
Differences between the TPR and TNR indicate the model is biased towards one
class. For the $\mathcal{D}_{S}$ dataset, BERT-CEfAf shows the smallest
difference indicating a low gender-bias, while OLA-CEA and BERT-CEf show the
highest difference, indicating a high gender-bias. For the $\mathcal{D}_{A}$
dataset, we observe the same trend with OLA-CEA now showing a significant
difference between TPR and TNR compared to all other model variants. The
difference could be attributed to the lower performance of the model and the
size of GECO it was trained on. The Average Prediction Difference [21] is a
bias measure, defined as
$\frac{1}{N}\sum_{i=0}^{N}|\operatorname{pred}(\mathbf{x}_{i}^{M},M)-\operatorname{pred}(\mathbf{x}_{i}^{F},F)|$,
where $\operatorname{pred}(\mathbf{x}_{i}^{T},T)$ is the softmax output
probability of a model for the class $T\in\\{M,F\\}$ given sample $i$ with
class $T$. Here, a pattern emerges for BERT models on both datasets. As the
number of fine-tuned layers increases, the average prediction difference
decreases, with BERT-CEfAf exhibiting the best performance among all models.
For Dataset $\mathcal{D}_{S}$, this trend can be attributed to the superior
classification accuracy of BERT-CEfAf compared to other models. Interestingly,
a similar albeit less pronounced pattern is observed for Dataset
$\mathcal{D}_{A}$, where all models display comparable accuracies. Figure 5:
Overview of the Masked Language Modeling (MLM) Score [34]. We leverage the
BERT models trained on the GECO dataset, replacing the classification layer
with the pre-trained masked language modeling (MLM) layer from BERT. This
transformation effectively transforms BERT-C into the raw BERT model for MLM,
while other versions of BERT models incorporate fine-tuned or retrained
embedding and/or attention layers. Strong differences between ‘male’ and
‘female’ sentences are not apparent, yet the MLM score increases for fine-
tuned and re-trained embedding layers, showing a change in model bias.

###  A.3 Explanation Performance Measures.

Let $\mathbf{x}^{*}\in\mathcal{D}^{test}$ be an instance for which we aim to
quantitatively assess the correctness of its explanation
$\mathbf{s}(f,\mathbf{x}^{*})$. The ground truth $h(\mathbf{x}^{*})$ defines a
set of potentially important tokens based on how we generated or altered words
in sentences of the corpus; however, a model might only use a subset of such
tokens for its predictions. Hence, making use of only a subset of tokens
corresponding to ground truth tokens, compared to considering all tokens
corresponding to the ground truth, must be considered equally correct.
Expressed via information retrieval terms - we are interested in mitigating
the impact of false-negatives and emphasizing the impact of false-positives on
explanation performance. False-negatives occur when a feature flagged as a
‘ground truth feature’ receives a low importance score, and false-positives
occur when a feature flagged as not part of the ground truth receives a high
importance score. The Mass Accuracy metric ($\mathrm{MA}$) [3, 9] provides
such properties. Consider the input sequence of embeddings
$\mathbf{E}^{*}\in\mathbb{R}^{m\times d}$ for each instance of interest
$\mathbf{x}^{*}$, for which we receive explanations of equal size
$\mathbf{S}^{m\times d}$. We aggregate those explanations via the embedding
dimension $\hat{\mathbf{s}}(f,\mathbf{x}^{*})=|\sum_{j}^{m}\mathbf{S}_{ji}|$
to obtain a token-based summary explanation. We also normalize the sequence
explanation by its sum
$\mathbf{s}(f,\mathbf{x}^{*})=\hat{\mathbf{s}}(f,\mathbf{x}^{*})/\sum_{j}^{d}\hat{\mathbf{s}}_{j}(f,\mathbf{x}^{*})$,
ensuring it has a cumulative value of one and
$\mathbf{s}(f,\mathbf{x}^{*})\in[0,1]^{d}$. With that, we define the Mass
Accuracy

$\mathrm{MA}\left(h(\mathbf{x}^{*}),\mathbf{s}(f,\mathbf{x}^{*})\right)=\sum_{j=1}^{d}\mathbf{s}_{j}(f,\mathbf{x}^{*})h_{j}(\mathbf{x}^{*}).$
(3)

A $\mathrm{MA}$ score of 1 shows a perfect explanation, marking only ground
truth tokens as important. Let us consider the case where we have two ground
truth tokens, and an explanation method attributes high importance only to one
of them and neglects the other tokens. The $\mathrm{MA}$ metric still produces
a high score, in this case, de-emphasizing false-negatives. With respect to
false-positives, high attributions to non-ground-truth tokens do not directly
contribute to the $\mathrm{MA}$, yet, through the normalization of
$\mathbf{s}$, all other tokens get assigned a relatively low (non-zero)
importance, leading to an overall low $\mathrm{MA}$ score, effectively
penalizing false-positive attributions.

###  A.4 Explanations

Figure 6: Explanations by popular XAI methods for one sample sentence, broken
down into input tokens as given to the respective model, with the ground truth
manipulations highlighted in green. The majority of importance by many methods
is correctly attributed to the word ‘she’, however all tokenized words show
non-zero attribution for multiple methods, including the period character ‘.’.
Figure 7: Attribution difference of male and female explanations
$\Delta\mathbf{s}=|\mathbf{s}(f,\mathbf{x}_{female}^{*})-\mathbf{s}(f,\mathbf{x}_{male}^{*})|$
for terms in the ground truth, averaged over the model repetitions. Indicating
that XAI methods with relatively high variability in mass accuracy display
increased differences in attributing importance depending on gender. Figure 8:
Heatmaps showing P-values of a wilcoxon signed-rank test applied on the
attribution difference
$\Delta\mathbf{s}=\mathbf{s}(f,\mathbf{x}_{female}^{*})-\mathbf{s}(f,\mathbf{x}_{male}^{*})$
per ground truth word of the male and female version of a sentence.

(a)

(b)

Figure 9:  (a) Explanation Performance grouped by method. For all methods,
leaving out baselines, the OLA-CEA model achieves the best mass accuracy
score, followed by BERT-CEf and BERT-CE. The full finetuned model BERT-CEfAf
and only classification model BERT-C both achieve similary low scores. (b)
Mass Accuracy for different post-hoc XAI methods applied on the five training
schemes for all sentences in the test set. Compared to the Mass Accuracy
computed only on correctly classified sentences, we see a clear decline of
explanation performance, especially for the dataset $\mathcal{D}_{s}$.