# Fair Text Classification with Wasserstein Independence

Thibaud Leteno1∗ Antoine Gourru1∗ Charlotte Laclau2
Rémi Emonet1 Christophe Gravier1
Laboratoire Hubert Curien, UMR CNRS 5516, Saint-Etienne, France
Télécom Paris, Institut Polytechnique de Paris, Paris, France
{thibaud.leteno, antoine.gourru, remi.emonet, christophe.gravier}@univ-st-
etienne.fr
<EMAIL_ADDRESS>

###### Abstract

††∗Equal contribution.

Group fairness is a central research topic in text classification, where
reaching fair treatment between sensitive groups (e.g. women vs. men) remains
an open challenge. This paper presents a novel method for mitigating biases in
neural text classification, agnostic to the model architecture. Considering
the difficulty to distinguish fair from unfair information in a text encoder,
we take inspiration from adversarial training to induce Wasserstein
independence between representations learned to predict our target label and
the ones learned to predict some sensitive attribute. Our approach provides
two significant advantages. Firstly, it does not require annotations of
sensitive attributes in both testing and training data. This is more suitable
for real-life scenarios compared to existing methods that require annotations
of sensitive attributes at train time. Second, our approach exhibits a
comparable or better fairness-accuracy trade-off compared to existing methods.
Our implementation is available on
GitHub111https://github.com/LetenoThibaud/wasserstein_fair_classification.

##  1 Introduction

Machine learning algorithms have become increasingly influential in decision-
making processes that significantly impact our daily lives. One of the major
challenges that has emerged in research, both academic and industrial,
concerns the fairness of these models, i.e. their ability to prevent
predictions related to individuals to be based on sensitive attributes such as
gender or ethnicity. In this article, we focus on the problem of fairness in
the domain of Natural Language Processing (NLP) Bender et al. (2021); Osoba
and Welser IV (2017); Schwemmer et al. (2020). While many studies already
report biases in NLP systems Sun et al. (2019); Hutchinson et al. (2020); Tan
and Celis (2019); Liang et al. (2021); Bender et al. (2021), these issues
become even more significant with the advent of public-ready AI-powered NLP
systems such as ChatGPT OpenAI  or Google Bard Pichai , making the need for
fair NLP solutions even more compelling. As more researchers work to overcome
these shortcomings, the first problem is to define what _fairness_ is. Such a
definition may hardly be consensual or is at least difficult to establish, as
it depends on situational and cultural contexts Fiske (2017). In this work, we
adopt the most common definition of group fairness, and the one adopted by
laws in several countries, which is based on the notion of disparate impact: a
prediction model is considered to have a disparate impact if its results
disproportionately harm (or benefit) people with certain sensitive attribute
values (e.g., women, black people).

Demonic Pred. Predictor Legend: Sensitive attributes: Target
labels$\perp\\!\\!\\!\\!\perp$

Figure 1: Our method, WFC, modifies the representation space of documents such
that it is fairer when training a classifier on top. To do that, it makes it
independent of a ”demonic” model that predicts the sensitive attribute.

In this work, we focus on group fairness for neural text classification as it
is one of the most ubiquitous tasks in our society, with prominent examples in
medical and legal domains Demner-Fushman et al. (2009) or human resources
Jatobá et al. (2019), to name a few. Neural text classification relies on text
encoders, which are parameterized and learned functions that map tokens
(arbitrary text chunks) into a latent space of controllable dimension, usually
followed by a classification layer. Built upon the Transformers architecture
Vaswani et al. (2017), popular Pre-trained Language Models (PLMs) such as BERT
Devlin et al. (2019), GPT3 Radford et al. (2019) or Llama Touvron et al.
(2023) leverage self-supervised learning to train the text encoder parameters.
In modern NLP pipelines, these PLMs are further fine-tuned on the supervised
task at hand. Ultimately, PLMs accumulate uncontrolled levels of unfairness
due to unbalanced learning data or algorithmic biases, for instance. This
results in observable biases in predictions but also in the latent space as
studied in Zhao et al. (2019); May et al. (2019).

We propose a novel approach (see Figure 1), to mitigate bias in text encoders,
that aims to tackle bias directly in the latent space on which documents are
projected, thus making our model applicable to any text encoder or decoder
(e.g. BERT or LLAMA). To proceed, we disentangle the neural signals encoding
bias from the neural signals used for prediction. The proposed architecture is
based on three components. First, two Multi-Layer Perceptrons (MLPs): the
first one whose objective is to predict the sensitive attribute, and the
second one is dedicated to the prediction task at hand. Then, a third MLP,
referred to as a critic, approximates the Wasserstein distance that acts as a
regularizer in our objective function. Our proposition overcomes a major
shortcoming of prior studies: they rely on the availability of the sensitive
attributes at train time. A constraint that is incompatible with recent
regulations, as the new European ones, that enforce more stringent
requirements for the collection and utilization of protected attributes. Prior
studies are thus more difficult to use in practical settings. In the
following, we will show that our approach can address this limitation by
avoiding the use of this information during both testing and training.

The rest of this paper is organized as follows. Section 2 presents recent
advances regarding fairness in NLP. Section 3 argues about our motivation and
provides the background knowledge to understand our contribution. Section 4
proceeds with the description of the proposed approach, its theoretical
analysis, and algorithmic implementation. Section 5 introduces the setting of
our experiments, and Section 6 presents the results and interpretation. The
last section concludes the paper and gives a couple of hints for possible
future research.

##  2 Related Works

Numerous studies have been conducted on how to tackle bias in machine learning
systems. Approaches to reinforce fairness can be divided between pre-, in-,
and post-processing methods. In the NLP literature, common pre-processing
techniques consist of data rebalancing (Park et al., 2018) or embedding
debiasing (Wang et al., 2020; Bolukbasi et al., 2016). Yet, despite the
efficiency of those methods, Kaneko et al. (2022) and Tokpo et al. (2023)
showed that other biases can be learned during the training or fine-tuning
processes. On the other hand, post-processing procedures aim at correcting
biases after the learning step, through model calibration (Zhao et al., 2017;
Jia et al., 2020) or data projection (Ravfogel et al., 2020) (INLP). We refer
to (Sun et al., 2019; Blodgett et al., 2020) for more exhaustive surveys of
bias in NLP.

Recently, adversarial methods Beutel et al. (2017); Zhang et al. (2018);
Elazar and Goldberg (2018) have been investigated to mitigate biases. Han et
al. (2021c, b) respectively suggest using several discriminators where each
learns different hidden representations and applying an adversarial approach
in- and cross-domain to train the adversary on a different dataset, with
methods called Adv and GATE. Differently, recent contributions focus on
directly constraining the objective to improve fairness (Shen et al., 2022b;
Han et al., 2021a). For instance, by adding some fairness metric, such as the
Equal opportunity that we will define later in this paper, to the objective
function.

Our approach is at the crossroad of these two philosophies: on the one hand,
we propose to train a biased model whose sole purpose is to predict the
sensitive attribute and use this latter to enforce fairness in our main
prediction model. On the second hand, we minimize a classifier loss with a
regularization term measuring the dependence between the latent
representations of the classifier and some unfair representations, using
Wasserstein distance. While many works use the Kullback–Leibler (KL)
divergence to measure the mutual information between representations, Ozair et
al. (2019) show several limitations: the KL-divergence is sensitive to small
perturbations in the data, and exploiting it for estimating the mutual
information does not scale up properly. Thus, they suggest an improvement
thanks to the Wasserstein distance. Other methods based on this distance
suggest focusing on the distance between the distributions of predictions to
enforce fairness (Jiang et al., 2020; Risser et al., 2022). Finally, most
approaches aforementioned depend on the availability of sensitive attribute
annotations in the training data, and as Kenfack et al. (2023) recently
emphasized, employing proxy-sensitive attributes often worsens the fairness-
accuracy trade-off. They also propose a proxy model to retrieve the missing
sensitive attributes, adapted to improve the model’s fairness.

#### Limits of existing approaches and positioning

Compared to the adversarial approaches previously mentioned, ours is
conceptually closer to (Nam et al., 2020), while their methodology is
conducted on images rather than textual data. We also distinguish from their
research by the use of the Wasserstein distance to evaluate the mutual
information between the two models’ representations instead of focusing the
learning of the main model on samples going against the prejudice of the
biased network. Like Ozair et al. (2019), we exploit the Wasserstein distance
as an estimator of the mutual information. However, while they use it to
measure mutual information to improve representational learning on images, we
consider sensitive attributes in the mutual information estimation and use it
to improve the model fairness. Our proposition is related to Risser et al.
(2022), yet, we do not use the model outputs directly as we use hidden
representations, of fixed-size and task-independent dimensions, of Pre-trained
Language Models that encode information on sensitive attributes. Additionally,
we do not use the Wasserstein distance to compute the distance between each
group’s prediction probability but to enforce the independence of the
representation from unfair representations. By using those latent
representations in the Wasserstein-regularization term, the model is
encouraged not to encode the sensitive information in the representation
during inference. Similarly, in the field of NLP, a related approach is
proposed by Cheng et al. (2021). Their method maximizes the mutual information
between pairs of sentence representations and their augmented versions, which
vary based on the sensitive attribute. These representations go through the
same encoder, ensuring that the input is independent of the sensitive
information. However, this does not ensure independence between the prediction
and the sensitive attribute Shen et al. (2022a); Cabello et al. (2023). In
contrast, our theoretically grounded approach minimizes the mutual information
between representations of the same sentence processed by two different
encoders to make the predictions independent of the sensitive attributes.
Moreover, their approach depends on identifying biased attribute words,
limiting its applicability to cases where substitute words are accessible.
This is a constraint we avoid. Lastly, while previous methods primarily
targeted classification issues in images or categorical and numerical data, we
introduce an approach well-suited for Natural Language Processing. It can be
applied to less-explored scenarios, including continuous sensitive attributes
and regression tasks.

##  3 Preliminaries

In this section, we introduce the notations used throughout this paper. We
also present the definitions of the necessary fairness metrics, and the main
concepts, mostly related to the Wasserstein distance which are essential for
understanding the rest of the paper.

###  3.1 Motivation

We consider a corpus of $n$ triplets $\\{(x_{i},y_{i},s_{i})\\}_{i=1}^{n}$,
where $x_{i}\in\mathcal{X}$ is a short document or a sentence,
$y_{i}\in\mathcal{Y}$ is a label and $s_{i}\in\mathcal{S}$ corresponds to one
or multiple variables, referred to as sensitive attributes, such as gender,
ethnicity or age. Let us consider binary classification for illustrative
purposes. The goal is to predict the outcomes $y_{i}$ by estimating the
conditional probability $p(y=1|x=x_{i})$ through a scoring function
$f:\mathcal{X}\rightarrow\\{0,1\\}$. The prediction associated with $f$ is
noted $\hat{y}$. For example, in the context of a social network, a classifier
can use the descriptors of a message (e.g., a bag of word representation),
$x_{i}$, to predict whether a message is toxic or not, leading to the decision
to ban the message and/or the user who wrote it from the social platform.

###  3.2 Measuring Fairness

In this context, of particular relevance is group-based fairness, which
examines how well outcome ($\hat{y}$) consistency is preserved across
sensitive groups ($s$). Returning to our example, when determining whether a
message is toxic, fairness here implies that the decision is consistent for
all users, regardless of gender or ethnicity.

A commonly used group-based metric used to quantify the (un)fairness of a
given classifier is the Equality of Opportunity EO Hardt et al. (2016) that is
satisfied if the prediction made by the classifier is conditionally
independent of the protected attribute, given that the true value is positive
(e.g. $y=1$). In effect, it means that the same proportion of each group
receives a positive outcome. For binary sensitive attribute
($s\in\\{a,\bar{a}\\}$) and multi-class objectives, the consensual way of
aggregating EO score over classes is the GAP score (De-Arteaga et al., 2019;
Ravfogel et al., 2020) defined as follows

$\textbf{GAP}=\sqrt{\frac{1}{|\mathcal{C}|}\sum_{c\in\mathcal{C}}(EO_{c})^{2}},$
(1)

where the EO for a given class $c\in\mathcal{C}$ is defined by

$\begin{split}\textbf{EO}_{c}=&p(\hat{y}=c|y=c,s=a)\\\
&-p(\hat{y}=c|y=c,s=\bar{a}).\end{split}$ (2)

Additionally, as fairness often requires determining a trade-off such that
reaching equity does not degrade the general classification performance, Han
et al. (2021a) proposed the Distance To Optimum (DTO) score. This latter
measures the accuracy-fairness trade-off by computing the Euclidean distance
from a model to an _Utopia point_ (point corresponding to the best Accuracy
and best Fairness values across all the baselines). The goal is to minimize
the DTO.

Finally, we consider the Leakage metric that corresponds to the accuracy of a
classification model trained to predict the sensitive attribute from the
latent representations. It measures the fairness _of the latent
representations themselves_ and demonstrates unfairness when close to 100 of
accuracy.

###  3.3 Fairness as Mutual Information minimization

Mutual Information (MI) is an information-theory-based metric meant to measure
the statistical dependence or the amount of information shared between two
variables. In our context, given the class of a document predicted by our
model along with the value of the sensitive attribute of the document, one can
use MI to evaluate the strength of their relationship. Formally, the mutual
information is defined as the Kullback-Leibler (KL) divergence between the
joint distribution $p(x,y)$ and the product of the marginal distributions:

$\text{MI}(x,y)=\text{KL}(p(x,y)\|p(x)p(y)).$ (3)

Fairness can therefore be cast as MI minimization between $\hat{y}$, our
prediction (conditioned on $y$, the ground-truth or not), and $s$, the
sensitive attribute, as it will make the prediction and the sensitive
attribute less and less dependent. Nevertheless, MI is generally intractable
for most real-life scenarios and has strong theoretical limitations as
outlined by Ozair et al. (2019). Notably, it requires a number of samples
exponential in the value of the Mutual Information to build a high confidence
lower bound and it is sensitive to small perturbations in the data sample.
Consequently, Ozair et al. (2019) proposed a theoretically sound dependency
measure, the Wasserstein Dependency Measure, based on Wasserstein-1 distance :

$\text{MI}_{W}(x,y)=W_{1}(p(x,y),p(x)p(y)).$ (4)

A key feature of the Wasserstein distance is that it can act as a smooth
objective function, as shown in the WGAN approach Arjovsky et al. (2017). More
precisely, the Kantorovich-Rubinstein duality expresses
$W_{1}(p(x,y),p(x)p(y))$ as :

$\begin{split}\sup_{||C||_{L}\leq 1}&\mathbb{E}_{x,y\sim p(x,y)}[C(x,y)]\\\
&-\mathbb{E}_{x\sim p(x),y\sim p(y)}[C(x,y)],\end{split}$ (5)

where $||C||_{L}\leq 1$ is the set of all 1-Lipschitz functions. Arjovsky et
al. (2017) propose to approximate this measure by using a parameterized
function, defined as follows :

$\begin{split}\max_{\omega,||C_{w}||_{L}\leq 1}&\mathbb{E}_{x,y\sim
p(x,y)}[C_{\omega}(x,y)]\\\ &-\mathbb{E}_{x\sim p(x),y\sim
p(y)}[C_{\omega}(x,y)],\end{split}$ (6)

where $C_{\omega}$ is called the critic and is usually a neural network.
Wasserstein distance has been efficiently used in many machine learning fields
Frogner et al. (2015); Courty et al. (2014); Torres et al. (2021) and a
particularly interesting application is that of fair machine learning Jiang et
al. (2020); Silvia et al. (2020); Gordaliza et al. (2019); Laclau et al.
(2021). See Appendix A.1 for further theoretical details on the Wasserstein
Distance. The role of this measure in our contribution is detailed in the
subsequent sections.

##  4 Our contribution

$z_{s}$$\hat{s}$pre-
trainedzy1zs1zy2zs2zy3zs3zy4zs4$p(z_{y},z_{s})$zy1zs3zy2zs1zy3zs4zy4zs2$p(z_{y})p(z_{s})$$z_{y}$$\hat{y}$

$\displaystyle\mathcal{L}(y,\hat{y})$

$+$

$\displaystyle W(p(z_{y},z_{s}),p(z_{y})p(z_{s}))$

critic

Figure 2: Architecture for a batch of size 4 at train time. The data
representation on the left shows how we create dependency or independence
between $z_{y}$ and $z_{s}$. At inference, only the trained classifier (green)
is kept to predict $\hat{y}$.

We are now ready to show how one can cast the problem of group fairness as an
independence constraint in the intermediate latent space of the MLPs and
derive a theoretically sound approach based on the Wasserstein distance to
solve it.

###  4.1 Definition of the Objective Function

In most recent NLP applications, deep classification is performed as a two-
step approach: the scoring function $f$ is a composition of two parameterized
functions such that $f=g\circ h$, where $g(x)=z\in\mathbb{R}^{d}$ projects $x$
in low dimensional space and $h$ is a simple, usually one linear layer neural
network with softmax activation followed by an argmax referred to as a
classification layer. The objective of $g$ is to produce an embedding $z$
linearly separable with respect to the target of interest. For the deep model
predicting $y$, we write $f_{c}=g_{c}\circ h_{c}=h_{c}(z_{c})$, where the
index $c$ stands for classification.

As stated earlier, fairness can be defined as minimizing $\text{
MI}(\hat{y},s)$. As $s$ is neither observable nor allowed to be observed in
most real-life scenarios, we make use of a surrogate for $s$ that we call the
demonic model. This deep model is a composition of $f_{s}=g_{s}\circ
h_{s}=h_{s}(z_{s})$, where $s$ stands for sensitive attribute. Therefore, in
the absence of the sensitive attribute, we can use :

$\min\text{ MI}(\hat{y},\hat{s}).$ (7)

As the argmax operation producing the hard predictions following the
classification layer is not differentiable, we propose to minimize the MI
between the latent representations instead of the network final output,
leading to optimizing an upper bound of the latter equation (see details in
Appendix):

$\min\text{ MI}(z_{y},z_{s})\geq\text{ MI}(\hat{y},\hat{s}).$ (8)

Lastly, we replace MI with MIW for the reasons explained earlier. We propose
to optimize simultaneously the Wasserstein dependency measure and the more
traditional classification loss (e.g. the cross-entropy). Our objective
function writes as follows

$\begin{split}\arg\min~{}&\mathcal{L}(y,h(z_{y}))\\\
+&\beta~{}W_{1}(p(z_{y},z_{s}),p(z_{y})p(z_{s})),\end{split}$ (9)

where $\mathcal{L}$ is the loss function aiming at maximizing the accuracy of
the model for predicting $y$ while the role of the second term is to constrain
the encoder part of the language model to learn fair representations. The
hyper-parameter $\beta\in\mathbb{R^{+}}$ is a weight allowing some control on
the impact of the penalty as the speed of convergence of the two sub-
objectives may be different. In the following, we refer to the approach
minimizing Equation 9 as WFC for Wasserstein Fair Classification (WFC).

###  4.2 Optimization of WFC

The overall architecture of WFC is presented in Figure 2. Given a batch of
documents along with their sensitive attribute, we start by generating a
representation of each document using a PLM. Then, taking these vectors as
input, we train two MLPs to predict $s$ and $y$, respectively. The former is
referred to as the demonic model in the remained of this paper. Now, assuming
that the MLPs consist of one hidden layer, for the sake of simplicity, we can
extract two embedding vectors for all documents, denoted by $z_{s}$ and
$z_{y}$. Note that the prediction $\hat{y}$ made by the second MLP (in green
in Figure 2) can be directly used to compute the first part of our objective
function (see Equation 9).

Now for the second term of the loss, which is given by
$W_{1}(p(z_{y},z_{s}),p(z_{y})p(z_{s}))$, we use the approximation proposed by
Arjovsky et al. (2017):

$\begin{split}\max_{\omega,||C_{w}||_{L}\leq 1}&\mathbb{E}_{z_{y},z_{s}\sim
p(z_{y},z_{s})}[C_{\omega}(z_{y},z_{s})]\\\ -&\mathbb{E}_{z_{y}\sim
p(z_{y}),z_{s}\sim p(z_{s})}[C_{\omega}(z_{y},z_{s})].\end{split}$ (10)

In this context, $C_{\omega}$ is a MLP, referred to as the critic in Figure 2.
To enforce the Lipschitz constraint, we clamp the weights to given values
($[-0.01,0.01])$ at each optimization step222We also tested some more recent
improvements of Lipschitz constraint enforcement Gulrajani et al. (2017); Wei
et al. (2018). Interestingly, all lead to poor performance. For a batch of
documents, the critic takes as input the concatenation of $z_{y}$ and $z_{s}$
on the one hand, and the concatenation of $z_{y}$ and $z_{s}$ randomly drawn
from the dataset (equivalent to $z_{y}\sim p(z_{y}),z_{s}\sim p(z_{s})$), on
the other hand. We then follow the training procedure introduced by Arjovsky
et al. (2017) which alternate maximizing Equation 10 in the critic parameters
for $n_{c}$ iterations and minimizing Equation 9 for $n_{d}$ iterations in the
$f_{y}$ classifier parameters. The overview of the training process is
detailed in the appendix B.3

#### Training the demonic model

We pre-train the demonic model to predict the sensitive attributes and do not
update the demonic weights during the training phase of the main model. The
benefits are twofold. Firstly, unlike previous works Caton and Haas (2020), we
require only limited access to sensitive attributes label at training and we
do not need access to the labeling of sensitive attributes in the inference
regime. As a result, WFC is highly compatible with recent regulations (e.g.,
US Consumer Financial Protection Bureau). Secondly, the demonic model can now
be trained in a few-shot fashion if some examples of the training set are
annotated with sensitive attributes. However, when no sensitive attributes are
available in the training set, we replace the training data of the demonic
part of the architecture with data from another domain (e.g. another dataset)
containing sensitive information for the same attribute. For example, for
gender, we can leverage generated datasets, like the EEC dataset Kiritchenko
and Mohammad (2018). Thus, we transfer this knowledge from one dataset to
another, working towards fairness autonomy regardless of the inclusion of
sensitive attributes within the data.

##  5 Experimental Protocol

In our experiments, we intensively use the FairLib package Han et al. (2022),
which provides an access to many state-of-the-art methods and datasets.

###  5.1 Dataset

We employ two widely-used datasets to evaluate fairness in the context of text
classification, building upon prior research Ravfogel et al. (2020); Han et
al. (2021c); Shen et al. (2022a). Both datasets are readily available in
FairLib.

#### Bias in Bios De-Arteaga et al. (2019).

This dataset, referred to as ‘Bios dataset’ in the rest of the paper, consists
of brief biographies associated with occupations (a total of $28$) and genders
(male or female). As per the partitioning prepared by Ravfogel et al. (2020),
the training, validation, and test sets comprise $257,000$, $40,000$ and
$99,000$ samples, respectively.

#### Moji Blodgett et al. (2016).

This dataset contains tweets written in either "Standard American English"
(SAE) or "African American English" (AAE), annotated with positive or negative
polarity. We use the dataset prepared by Ravfogel et al. (2020), which
includes $100,000$ training examples, $8,000$ validation examples, and $8,000$
test examples. The target variable $y$ represents the polarity, while the
protected attribute corresponds to the ethnicity, indicated by the AAE/SAE
attribute.

###  5.2 Baselines

Except for the classical cross-entropy loss without fairness constraint (CE)
that we run ourselves, we report the results from Shen et al. (2022a); Han et
al. (2022) on these two datasets. The considered baselines are INLP Ravfogel
et al. (2020), the ADV method Han et al. (2021c), FairBatch Roh et al. (2021),
GATE Han et al. (2021a) and Con, displaying the $dp$ and $eo$ versions Shen et
al. (2022a).

###  5.3 Evaluation Tasks

For training a vanilla text classification model, we follow the protocol
proposed by Han et al. (2022): a frozen BERT encoder followed by a 3-layer
MLP. We use accuracy to assess the classification performance. Fairness for
all models is evaluated against three metrics presented earlier: GAP, referred
to as “Fairness” in previous works Han et al. (2022); Shen et al. (2022a), and
the Distance To Optimum (DTO) proposed by Han et al. (2021a) (we follow the
methodology of Shen et al. (2022a) and evaluate the DTO on the average
fairness and accuracy of the best empirical results for each metric over all
models to build the utopia point). Finally, we consider the Leakage score.

#### Task 1: Fair Classification

We first compare our method against state-of-the-art approaches. We use the
representation generated by a base BERT model as an input to the MLPs. For
Bios, the demonic MLP is trained on 1% of the training set and obtains 99%
accuracy for predicting the sensitive attributes on the test set. Similarly,
the demonic MLP obtains 88.5% accuracy on Moji.

#### Task 2: Demonic transfer

We conduct these experiments for Bios and train a demonic MLP either on the
EEC dataset Kiritchenko and Mohammad (2018) or the Marked Personas dataset
Cheng et al. (2023). We then evaluate the performance of the demonic MLP to
predict gender on the Bios test dataset. When training on the EEC dataset we
obtain 98.1% of accuracy, and on the Marked Personas dataset, we obtain 98.4%
of accuracy. We repeat Task 1, with those variants of the demonic MLP. We
focus on Bios in this experiment. For Moji, it would require to have access to
other datasets with the same protected attribute (ethnicity).

#### Task 3: Use of representations from different layers

In the previous experiments, following approaches presented in Han et al.
(2022), the Wasserstein distance is approximated using the last hidden
representations of the 3-layer MLP. We compare this approach, on both
datasets, with the use of the first hidden representations of the MLP and with
the output logits. For the latter, the Wasserstein is estimated between
distributions of different dimensions: for example, in the case of Bios, 2 for
the demonic MLP corresponding to the sensitive attributes and 28 for the
classification MLP corresponding to the labels.

#### Task 4: Independence with predicted hard sensitive attributes

To evaluate the impact of using the representation $z_{s}$, we reproduce Task
1, but replace $z_{s}$ with the sensitive attributes predicted by the demonic
MLP: $\hat{s}$ and concatenate $z_{y}$ and $\hat{s}$ dependently and
independently when computing the Wasserstein distance. Note that we do not
encounter a problem with the non-differentiability for $\hat{y}$ (with the
argmax operation as for $\hat{s}$ as mentioned in Section 4.1) since the
demonic model is pre-trained.

Figure 3: Visualization of the fairness-accuracy trade-off on (a) Moji and (b)
Bios. Values correspond to the average results.

###  5.4 Implementation Details

Our architecture is composed of three components: two classifiers and a
critic. The details of the MLPs used to parameterize each component are given
in Appendix B. We find the best hyperparameters for our models by grid-search
cross-validation over the MLP and Critic learning rates, the value of $n_{d}$
(number of batches used to train the main MLP), the layers producing $z_{s}$
and $z_{y}$, the value of $\beta$ and the value used to clamp the weights to
enforce the Lipschitz constraint. The values allowing us to obtain the lower
DTO during this process are presented in Appendix B. The architecture details
of the MLP for the leakage are provided in Shen et al. (2022a) as we use the
same configuration.

##  6 Results

Table 1: Results on Moji (top) and Bios (bottom). For baselines, results are
drawn from Shen et al. (2022a). We report the mean $\pm$ standard deviation
over 5 runs. * indicates the model without fairness consideration.

Model | Accuracy $\uparrow$ | Fairness $\uparrow$ | DTO $\downarrow$ | Leakage $\downarrow$
---|---|---|---|---
*CE | $72.3\pm 0.5$ | $61.2\pm 1.4$ | $31.0$ | $87.9\pm 3.3$
INLP | $73.3\pm 0.0$ | $85.6\pm 0.0$ | $7.02$ | $86.7\pm 0.6$
Adv | $75.6\pm 0.4$ | $90.4\pm 1.1$ | $1.71$ | $78.8\pm 6.0$
Gate | $\bm{76.2\pm 0.3}$ | $90.1\pm 1.5$ | $1.90$ | $100.0\pm 0.0$
FairBatch | $75.1\pm 0.6$ | $90.6\pm 0.5$ | $1.78$ | $88.4\pm 0.4$
EOGLB | $75.2\pm 0.2$ | $90.1\pm 0.4$ | $2.15$ | $85.7\pm 1.2$
Condp | $75.8\pm 0.3$ | $88.1\pm 0.6$ | $3.92$ | $\bm{54.2\pm 0.9}$
Coneo | $74.1\pm 0.7$ | $84.1\pm 3.0$ | $8.17$ | $80.1\pm 4.2$
WFC | $75.2\pm 0.1$ | $\bm{91.4\pm 0.3}$ | $\bm{1.17}$ | $86.9\pm 0.2$

Model | Accuracy $\uparrow$ | Fairness $\uparrow$ | DTO $\downarrow$ | Leakage $\downarrow$
---|---|---|---|---
*CE | $82.3\pm 0.2$ | $85.1\pm 0.8$ | $5.67$ | $98.0\pm 0.0$
INLP | $82.3\pm 0.0$ | $88.6\pm 0.0$ | $2.44$ | $97.6\pm 0.1$
Adv | $81.9\pm 0.2$ | $\bm{90.6\pm 0.5}$ | $1.80$ | $88.6\pm 4.6$
Gate | $\bm{83.7\pm 0.2}$ | $90.4\pm 0.9$ | $\bm{0.20}$ | $100.0\pm 0.0$
FairBatch | $82.2\pm 0.1$ | $89.5\pm 1.3$ | $1.86$ | $98.0\pm 0.3$
EOGLB | $81.7\pm 0.4$ | $88.4\pm 1.0$ | $2.97$ | $97.2\pm 0.5$
Condp | $82.1\pm 0.2$ | $84.3\pm 0.8$ | $6.50$ | $\bm{76.3\pm 1.5}$
Coneo | $81.8\pm 0.3$ | $85.2\pm 0.4$ | $5.72$ | $84.9\pm 3.4$
WFC | $82.4\pm 0.1$ | $89.0\pm 0.3$ | $2.06$ | $96.5\pm 0.5$

Table 2: Evaluation of the impact of parameter $\beta$ (Equation 9) on the
model’s leakage. Results on Moji (top) and Bios (bottom).

$\beta$ | DTO $\uparrow$ | Accuracy $\uparrow$ | Fairness $\uparrow$ | Leakage $\downarrow$
---|---|---|---|---
1 | 1.2 | 75.2 | 91.4 | 86.9
5 | 4.2 | 72.1 | 93.4 | 81.1
10 | 5.8 | 70.5 | 92.1 | 81.6
20 | 7.6 | 68.6 | 92.1 | 84.1

$\beta$ | DTO $\uparrow$ | Accuracy $\uparrow$ | Fairness $\uparrow$ | Leakage $\downarrow$
---|---|---|---|---
1 | 2.1 | 82.4 | 89.0 | 96.5
5 | 2.3 | 81.8 | 89.3 | 87.4
10 | 2.8 | 81.8 | 88.5 | 88.8
20 | 3.6 | 81.3 | 88.0 | 81.8

Table 3: Summary of the results for tasks 2, 3 and 4. We report the mean $\pm$
standard deviation over 5 runs for all tasks. Boldface numbers are the best
results, and a star indicates that the difference is statistically significant
according to a signed-rank Wilcoxon test (i.e. with a p-value lower than
$0.01$).

Data | Accuracy $\uparrow$ | Fairness $\uparrow$ | DTO $\downarrow$ | Leakage $\downarrow$
---|---|---|---|---
Bios 1% | $\textbf{82.4}\pm\textbf{0.1}$ | $\textbf{89.0}\pm\textbf{0.3}$ | 2.06 | $96.5\pm 0.5$
EEC | $82.2\pm 0.4$ | $88.9\pm 0.4$ | $2.26$ | $97.5\pm 0.3$
MP | $\textbf{82.4}\pm\textbf{0.3}$ | $88.9\pm 0.4$ | $2.14$ | $\textbf{96.4}\pm\textbf{0.5}$

(a) Task 2: comparison between several scenarios for training the demonic
model for prediction on Bios.

Layer | Accuracy $\uparrow$ | Fairness $\uparrow$ | DTO $\downarrow$ | Leakage $\downarrow$
---|---|---|---|---
|  | Bios |  |
Last hid. | $\textbf{82.4}\pm\textbf{0.1}^{*}$ | $\textbf{89.0}\pm\textbf{0.3}^{*}$ | $\textbf{2.06}^{*}$ | $96.5\pm 0.5$
First hid. | $81.9\pm 0.2$ | $86.7\pm 0.4$ | $4.29$ | $96.5\pm 0.6$
Last lay. | $82.1\pm 0.6$ | $87.5\pm 0.3$ | $3.49$ | $\textbf{87.0}\pm\textbf{1.1}^{*}$
|  | Moji |  |
Last hid. | $\textbf{75.2}\pm\textbf{0.1}^{*}$ | $\textbf{91.4}\pm\textbf{0.3}^{*}$ | $\textbf{1.17}^{*}$ | $86.9\pm 0.2$
First hid. | $74.3\pm 0.1$ | $80.8\pm 1.0$ | $11.4$ | $85.6\pm 0.6$
Last lay. | $73.5\pm 0.0$ | $70.2\pm 0.2$ | $21.9$ | $\textbf{64.5}\pm\textbf{0.1}^{*}$

(b) Task 3: comparison between the use of representations of different MLP
layers to compute the Wasserstein.

Labels | Accuracy $\uparrow$ | Fairness $\uparrow$ | DTO $\downarrow$ | Leakage $\downarrow$
---|---|---|---|---
|  | Bios |  |
Representations | $82.4\pm 0.1$ | $\textbf{89.0}\pm\textbf{0.3}^{*}$ | $\textbf{2.06}^{*}$ | $96.5\pm 0.5$
Hard labels | $\textbf{82.6}\pm\textbf{0.2}$ | $87.5\pm 0.2$ | $3.28$ | $\textbf{92.0}\pm\textbf{0.2}^{*}$
|  | Moji |  |
Representations | $\textbf{75.2}\pm\textbf{0.1}^{*}$ | $\textbf{91.4}\pm\textbf{0.3}^{*}$ | $\textbf{1.17}^{*}$ | $86.9\pm 0.2$
Hard labels | $72.2\pm 0.1$ | $65.0\pm 0.0$ | $27.3$ | $\textbf{81.0}\pm\textbf{0.8}^{*}$

(c) Task 4: comparison between the use of representations $z_{s}$ and hard
sensitive attributes to compute the Wasserstein distance.

###  6.1 Task 1: Fair Classification

We compare WFC with text classification baselines. For Moji, (Table 1 and Fig.
3), accuracy of WFC is higher than the accuracy of CE and it is equivalent to
competitors. On the fairness metrics, we outperform all other baselines and
obtain the best DTO. For Bios (Table 1 and Fig. 3), our method is competitive
with the other baselines and ranks 4 out of 9 in terms of accuracy-fairness
trade-off. In comparison, with equivalent methods in terms of DTO (INLP,
FairBatch, and Adv), WFC improves either the performance or the fairness.
Especially, WFC has the second-best accuracy compared to baselines. Finally,
we note that WFC is more stable in terms of Fairness compared with other
approaches having on average the best results for this metric (along with a
smaller standard deviation). Eventually, even when our method does not
outperform the baselines (e.g., Bios dataset), it still exhibits noteworthy
properties, particularly its ability to achieve competitive performances
without access to the sensitive attributes in the training set. We evaluate
this capability in the next subsection. We also explore the ability of our
proposition to improve the leakage. We initially aim at improving the fairness
while maintaining the accuracy of the model. Yet, our method allows to improve
leakage by increasing the value of $\beta$ in Equation 9, in other words, we
give more importance to the Wasserstein regularization in the loss. We note in
Table 2 that with a higher $\beta$, the leakage decreases. However, on both
datasets, the accuracy, that we want to preserve, decreases and the trade-off
worsens as we get better for the leakage. To sum up, reducing leakage makes it
more challenging to retrieve sensitive attributes but could result in
unintended information loss needed for the classification task affecting the
performance. Ultimately, we want to enhance fairness while keeping a good
performance and this objective may not necessarily match with a strong leakage
improvement.

###  6.2 Task 2: Demonic Transfer

For this task, we pre-train the demonic model on other datasets. Table 3(a)
shows that we achieve similar results as when the pre-training is done using
the same dataset. The average loss of accuracy and fairness are not
significant. These results are promising for improving fairness, especially in
situations where collecting sensitive data is not feasible or when only
partial information is accessible.

###  6.3 Task 3: Use of representations from different layers

On both datasets (Table 3(b)), accuracy is rather stable regardless of the
layers used to compute the Wasserstein distance. Still, the best results are
obtained using the last hidden representations. However, while we note a
slight decrease in fairness on Bios when using representations from other
layers, the decrease becomes much more significant on Moji. Using the last
hidden layer is the best option.

###  6.4 Task 4: Independence with predicted hard sensitive attributes

We replace $z_{s}$ by the predicted $\hat{s}$ to compute the Wasserstein
distance and report the results in Table 3(c). We observe, on average, a
slight improvement of the accuracy on Bios, and a slight decrease in accuracy
on Moji. However, while the decrease in fairness is not significant for Bios,
we observe a substantial drop for Moji. As a result, using $\hat{s}$ instead
of $z_{s}$ seems to have a neutral impact at best, this may also result, in
some cases, in a reduction of both accuracy and fairness.

##  7 Conclusion

We presented WFC a novel method that enforces fairness constraints using a
pre-trained neural network on the sensitive attributes and Wasserstein
regularization. Our model is theoretically well-motivated and has interesting
properties over existing models. The most important one is the fact that it
does not require annotation of the sensitive attribute at both training and
inference time. We obtain competitive results compared to baselines on the
Bios dataset and outperform them on the fairness score with comparable
accuracy on Moji dataset. Furthermore, we present a solution for our algorithm
to be trained when sensitive attributes are not available for a given dataset,
paving the way for its use under realistic applications. In further studies,
we will focus on applying this method using different text encoders or
decoders, datasets, and downstream tasks, as this method can generalize to
tasks out of the text classification scope, notably, regression and even
unsupervised objectives.

## Limitations

The proposed approach is rather flexible as it can handle various types of
sensitive attributes. However, due to the lack of available datasets, we were
not able to assess our performance for continuous sensitive attributes, e.g.
age. In addition, we are aware that gender may embrace a $n$-ary definition,
in all our experiments, we were limited to considering only men vs women
classification, due to data availability.

For Task 2 defined in the experiments section, we were able to show
empirically that our method works well when the demonic model is pre-trained
on a different dataset when no sensitive attributes or very few of them are
available on the main training dataset. However, we do not provide sufficient
generalization guarantees to consider an out-of-the-box large-scale
deployment. The next step will be to derive theoretical guarantees inspired by
the results of domain adaptation to assess how well this idea can be
generalized to other data and under which conditions it might fail or succeed.

Finally, for Task 1 we did not perform a statistical test to assess the
significance of the observed differences. Indeed, most of the results were
reported from Shen et al. (2022a) and we were unable to retrieve the scores
for each run.

## Ethics Statement

We acknowledge the following concerns about our work. First, biases present in
the data are US-centered, and concerning the Bias in Bios dataset genders are
binary. Furthermore, to conduct our research we need to have access to the
sensitive attributes contrary to what privacy measures recommend, and the
latter are annotated with a risk of subjectivity.

## Acknowledgements

This work was funded by the French National Agency for Research (ANR) in the
context of the Diké project (ANR-21-CE23-0026).
Our experiments utilize the previously mentioned Fairlib framework. We would
like to express our gratitude to Xudong Han for his availability and
assistance in using it.

## References

  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017.  Wasserstein generative adversarial networks.  In _ICML_ , pages 214–223. PMLR.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _FAccT_ , pages 610–623.
  * Beutel et al. (2017) Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. 2017.  Data decisions and theoretical implications when adversarially learning fair representations.  _arXiv preprint arXiv:1707.00075_.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in nlp.  In _Proceedings of ACL_ , pages 5454–5476.
  * Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016.  Demographic dialectal variation in social media: A case study of african-american english.  _arXiv preprint arXiv:1608.08868_.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  _NeurIPS_ , 29.
  * Cabello et al. (2023) Laura Cabello, Anna Katrine Jørgensen, and Anders Søgaard. 2023.  On the independence of association bias and empirical fairness in language models.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’23, page 370–378, New York, NY, USA. Association for Computing Machinery.
  * Caton and Haas (2020) Simon Caton and Christian Haas. 2020.  Fairness in machine learning: A survey.  _arXiv preprint arXiv:2010.04053_.
  * Cheng et al. (2023) Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023.  Marked personas: Using natural language prompts to measure stereotypes in language models.  _arXiv preprint arXiv:2305.18189_.
  * Cheng et al. (2021) Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. 2021.  Fairfil: Contrastive neural debiasing method for pretrained text encoders.
  * Courty et al. (2014) Nicolas Courty, Rémi Flamary, and Devis Tuia. 2014.  Domain adaptation with regularized optimal transport.  In _ECML PKDD_ , pages 274–289. Springer.
  * De-Arteaga et al. (2019) Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019.  Bias in bios: A case study of semantic representation bias in a high-stakes setting.  In _FaccT_ , pages 120–128.
  * Demner-Fushman et al. (2009) Dina Demner-Fushman, Wendy W Chapman, and Clement J McDonald. 2009.  What can natural language processing do for clinical decision support?  _Journal of biomedical informatics_ , 42(5):760–772.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Elazar and Goldberg (2018) Yanai Elazar and Yoav Goldberg. 2018.  Adversarial removal of demographic attributes from text data.  In _Proceedings of EMNLP_. Association for Computational Linguistics.
  * Fiske (2017) Susan T Fiske. 2017.  Prejudices in cultural contexts: Shared stereotypes (gender, age) versus variable stereotypes (race, ethnicity, religion).  _Perspectives on psychological science_ , 12(5):791–799.
  * Frogner et al. (2015) Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. 2015.  Learning with a wasserstein loss.  _NeurIPS_ , 28.
  * Gordaliza et al. (2019) Paula Gordaliza, Eustasio Del Barrio, Gamboa Fabrice, and Jean-Michel Loubes. 2019.  Obtaining fairness using optimal transport theory.  In _ICML_ , pages 2357–2365. PMLR.
  * Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017.  Improved training of wasserstein gans.  _Advances in neural information processing systems_ , 30.
  * Han et al. (2021a) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021a.  Balancing out bias: Achieving fairness through balanced training.  _arXiv preprint arXiv:2109.08253_.
  * Han et al. (2021b) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021b.  Decoupling adversarial training for fair NLP.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 471–477, Online. Association for Computational Linguistics.
  * Han et al. (2021c) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021c.  Diverse adversaries for mitigating bias in training.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2760–2765, Online. Association for Computational Linguistics.
  * Han et al. (2022) Xudong Han, Aili Shen, Yitong Li, Lea Frermann, Timothy Baldwin, and Trevor Cohn. 2022.  Fairlib: A unified framework for assessing and improving fairness.  In _Proceedings of the The 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 60–71.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nati Srebro. 2016.  Equality of opportunity in supervised learning.  _NeurIPS_ , 29.
  * Hutchinson et al. (2020) Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020.  Social biases in NLP models as barriers for persons with disabilities.  In _ACL_ , pages 5491–5501.
  * Jatobá et al. (2019) Mariana Jatobá, Juliana Santos, Ives Gutierriz, Daniela Moscon, Paula Odete Fernandes, and João Paulo Teixeira. 2019.  Evolution of artificial intelligence research in human resources.  _Procedia Computer Science_ , 164:137–142.
  * Jia et al. (2020) Shengyu Jia, Tao Meng, Jieyu Zhao, and Kai-Wei Chang. 2020.  Mitigating gender bias amplification in distribution by posterior regularization.  In _Proceedings of the Annual Meeting of ACL_.
  * Jiang et al. (2020) Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. 2020.  Wasserstein fair classification.  In _Uncertainty in artificial intelligence_ , pages 862–872. PMLR.
  * Kaneko et al. (2022) Masahiro Kaneko, Danushka Bollegala, and Naoaki Okazaki. 2022.  Debiasing isn’t enough!–on the effectiveness of debiasing mlms and their social biases in downstream tasks.  In _Proceedings of the International Conference on Computational Linguistics_ , pages 1299–1310.
  * Kantorovich (1942) Leonid Kantorovich. 1942.  On the translocation of masses.  In _C.R. (Doklady) Acad. Sci. URSS(N.S.)_ , volume 37(10), pages 199–201.
  * Kenfack et al. (2023) Patrik Joslin Kenfack, Samira Ebrahimi Kahou, and Ulrich Aïvodji. 2023.  Fairness under demographic scarce regime.  _arXiv preprint arXiv:2307.13081_.
  * Kiritchenko and Mohammad (2018) Svetlana Kiritchenko and Saif M. Mohammad. 2018.  Examining gender and race bias in two hundred sentiment analysis systems.
  * Laclau et al. (2021) Charlotte Laclau, Ievgen Redko, Manvi Choudhary, and Christine Largeron. 2021.  All of the fairness for edge prediction with optimal transport.  In _AISTATS_ , pages 1774–1782. PMLR.
  * Liang et al. (2021) Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021.  Towards understanding and mitigating social biases in language models.  In _ICML_ , pages 6565–6576. PMLR.
  * May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. 2019.  On measuring social biases in sentence encoders.  In _Proceedings of NAACL-HLT_ , pages 622–628.
  * Monge (1781) Gaspard Monge. 1781.  Mémoire sur la théorie des déblais et des remblais.  _Histoire de l’Académie Royale des Sciences_ , pages 666–704.
  * Nam et al. (2020) Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. 2020.  Learning from failure: De-biasing classifier from biased classifier.  _Advances in Neural Information Processing Systems_ , 33:20673–20684.
  * (38) OpenAI.  Chatgpt.  https://openai.com/blog/chatgpt.  Accessed: 2022-03-06.
  * Osoba and Welser IV (2017) Osonde A Osoba and William Welser IV. 2017.  _An intelligence in our image: The risks of bias and errors in artificial intelligence_.  Rand Corporation.
  * Ozair et al. (2019) Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. 2019.  Wasserstein dependency measure for representation learning.  _Advances in Neural Information Processing Systems_ , 32.
  * Park et al. (2018) Ji Ho Park, Jamin Shin, and Pascale Fung. 2018.  Reducing gender bias in abusive language detection.  In _Proceedings of EMNLP_ , pages 2799–2804.
  * (42) Sundar Pichai.  Google bard.  https://blog.google/technology/ai/bard-google-ai-search-updates/.  Accessed: 2022-03-06.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020.  Null it out: Guarding protected attributes by iterative nullspace projection.  In _Proceedings of the Annual Meeting of ACL_ , pages 7237–7256. ACL.
  * Risser et al. (2022) Laurent Risser, Alberto González Sanz, Quentin Vincenot, and Jean-Michel Loubes. 2022.  Tackling algorithmic bias in neural-network classifiers using wasserstein-2 regularization.  _Journal of Mathematical Imaging and Vision_ , 64(6):672–689.
  * Roh et al. (2021) Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. 2021.  Fairbatch: Batch selection for model fairness.  In _International Conference on Learning Representations_.
  * Schwemmer et al. (2020) Carsten Schwemmer, Carly Knight, Emily D Bello-Pardo, Stan Oklobdzija, Martijn Schoonvelde, and Jeffrey W Lockhart. 2020.  Diagnosing gender bias in image recognition systems.  _Socius_ , 6.
  * Shen et al. (2022a) Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, and Lea Frermann. 2022a.  Does representational fairness imply empirical fairness?  In _Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022_ , pages 81–95.
  * Shen et al. (2022b) Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, and Lea Frermann. 2022b.  Optimising equal opportunity fairness in model training.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4073–4084. Association for Computational Linguistics.
  * Silvia et al. (2020) Chiappa Silvia, Jiang Ray, Stepleton Tom, Pacchiano Aldo, Jiang Heinrich, and Aslanides John. 2020.  A general approach to fairness with optimal transport.  In _Proceedings of AAAI_ , volume 34, pages 3633–3640.
  * Sun et al. (2019) Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019.  Mitigating gender bias in natural language processing: Literature review.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 1630–1640.
  * Tan and Celis (2019) Yi Chern Tan and L Elisa Celis. 2019.  Assessing social and intersectional biases in contextualized word representations.  _NeurIPS_ , 32.
  * Tokpo et al. (2023) Ewoenam Tokpo, Pieter Delobelle, Bettina Berendt, and Toon Calders. 2023.  How far can it go?: On intrinsic gender bias mitigation for text classification.  _arXiv e-prints_ , pages arXiv–2301.
  * Torres et al. (2021) Luis Caicedo Torres, Luiz Manella Pereira, and M Hadi Amini. 2021.  A survey on optimal transport for machine learning: Theory and applications.  _arXiv preprint arXiv:2106.01963_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _NeurIPS_ , 30.
  * Wang et al. (2020) Tianlu Wang, Xi Victoria Lin, Nazneen Fatema Rajani, Bryan McCann, Vicente Ordonez, and Caiming Xiong. 2020.  Double-hard debias: Tailoring word embeddings for gender bias mitigation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5443–5453.
  * Wei et al. (2018) Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. 2018.  Improving the improved training of wasserstein gans: A consistency term and its dual effect.  In _International Conference on Learning Representation (ICLR)_.
  * Yeung (1991) Raymond W Yeung. 1991.  A new outlook on shannon’s information measures.  _IEEE transactions on information theory_ , 37(3):466–474.
  * Zhang et al. (2018) Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018.  Mitigating unwanted biases with adversarial learning.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 335–340.
  * Zhao et al. (2019) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019.  Gender bias in contextualized word embeddings.  In _Proceedings of NAACL-HLT_ , pages 629–634.
  * Zhao et al. (2017) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017.  Men also like shopping: Reducing gender bias amplification using corpus-level constraints.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2979–2989.

Data: $D=\\{(x_{i},y_{i},s_{i})\\}_{i=1}^{n}$ the training set, $n_{e}$ the
number of epochs, $n_{c}$ and $n_{d}$ the number of training iterations per
epoch for the critic and the classifier respectively, a batch size $n_{b}$,
two neural networks $h_{s}(x)$ and $h_{c}(x;\theta)$, a Critic $C_{\omega}$, a
weight on the regularization $\beta$

for  _e = 1, …, $n_{e}$_ do

for  _t = 1, …, $n_{c}$_ do

Sample $\\{x_{i},y_{i},s_{i}\\}_{i=1}^{n_{b}}$

Encode : $z_{s}\leftarrow\\{h_{s}(x_{i})\\}_{i=1}^{n_{b}}$,
$z_{y}\leftarrow\\{h_{c}(x_{i})\\}_{i=1}^{n_{b}}$

Concatenate vectors to get $Z_{dep}\leftarrow[z_{s,i},z_{y,i}]_{i=1}^{n_{b}}$

Shuffle the $z_{s,i}$ vectors.

Concatenate vectors to get $Z_{ind}\leftarrow[z_{s,i},z_{y,i}]_{i=1}^{n_{b}}$

$grad(w)\leftarrow\nabla_{\omega}\frac{1}{n_{b}}(\sum_{i=1}^{n_{b}}C_{\omega}(Z_{dep,i})-\sum_{i=1}^{n_{b}}C_{\omega}(Z_{ind,i}))$

$\omega\leftarrow Adam(\omega;grad(w))$

end for

for  _t = 1, …, $n_{d}$_ do

Sample $\\{x_{i},y_{i},s_{i}\\}_{i=1}^{n_{b}}$

Encode : $z_{s}\leftarrow\\{h_{s}(x_{i})\\}_{i=1}^{n_{b}}$,
$z_{y}\leftarrow\\{h_{c}(x_{i})\\}_{i=1}^{n_{b}}$

Concatenate vectors to get $Z_{dep}=[z_{s,i},z_{y,i}]_{i=1}^{n_{b}}$

Shuffle the $z_{s,i}$ vectors.

Concatenate vectors to get $Z_{ind}=[z_{s,i},z_{y,i}]_{i=1}^{n_{b}}$

$\mathcal{L}\leftarrow\sum_{i=1}^{n_{b}}\mathcal{L}(y_{i},h(z_{y,i}))$

$\mathcal{L}\leftarrow\mathcal{L}+\beta(\sum_{i=1}^{n_{b}}C_{\omega}(Z_{dep,i})-\sum_{i=1}^{n_{b}}C_{\omega}(Z_{ind,i}))$

$\theta\leftarrow Adam(\theta;\nabla_{\theta}\frac{1}{{n_{b}}}\mathcal{L})$

end for

end for

Algorithm 1 WFC Algorithm

##  Appendix A Theoretical background and Proof

###  A.1 Wasserstein Distance

Finding correspondences between two sets of points is a longstanding issue in
machine learning. The optimal transport (OT) Monge (1781) problem offers an
efficient solution to this issue by calculating an optimal one-to-one
transport map between the two sets. This map is determined by considering the
geometrical proximity of the points in the sets. In practice, OT can be
expressed as a problem of aligning two empirical distributions
$\mathbb{P}_{X_{1}}$ and $\mathbb{P}_{X_{2}}$ supported on two point sets
$X_{1}=\\{x_{1}^{(i)}\in\mathbb{R}^{d}\\}_{i=1}^{N_{1}}$ and
$X_{2}=\\{x_{1}^{(j)}\in\mathbb{R}^{d}\\}_{i=1}^{N_{2}}$. We consider the
Monge-Kantorovich formulation of this problem Kantorovich (1942) where the
goal is to search for a coupling $\gamma$ defined as a joint probability
distribution over $X_{1}\times X_{2}$ with marginals $\mathbb{P}_{X_{1}}$ and
$\mathbb{P}_{X_{2}}$. This amounts to minimizing the cost of transport w.r.t.
some metric $l:X_{1}\times X_{2}\rightarrow\mathbb{R}^{+}$:

$\displaystyle\min_{\gamma\in\Pi(\mathbb{P}_{X_{1}},\mathbb{P}_{X_{2}})}\langle
M,\gamma\rangle_{F}$ (11)

where $\langle\cdot\text{,}\cdot\rangle_{F}$ is the Frobenius dot product, $M$
is a dissimilarity matrix, i.e., $M_{ij}=l(x_{1}^{(i)},x_{2}^{(j)})$, defining
the cost of associating $x_{1}^{(i)}$ with $x_{2}^{(j)}$ and
$\Pi(\mathbb{P}_{X_{1}},\mathbb{P}_{X_{2}})$ is a set of doubly stochastic
matrices. This problem admits a unique solution $\gamma^{*}$ and defines a
metric on the space of probability measures called the Wasserstein distance
(also known as the Earth-Mover Distance) as follows:

$W_{M}(\mathbb{P}_{X_{1}},\mathbb{P}_{X_{2}})=\min_{\gamma\in\Pi(\mathbb{P}_{X_{1}},\mathbb{P}_{X_{2}})}\langle
M,\gamma\rangle_{F}.$

###  A.2 Proof that $\text{ MI}(z_{y},z_{s})\geq\text{ MI}(\hat{y},\hat{s})$

We have :

$p(\hat{y},z_{y},\hat{s})=p(\hat{y}|z_{y},\hat{s})p(z_{y}|\hat{s})p(\hat{s})$
(12)

Recall that $\hat{s}=h_{s}(z_{s})$ and $\hat{y}=h_{c}(z_{y})$, therefore,
$\hat{y}$ is fully determined by $z_{y}$, hence,
$p(\hat{y}|z_{y},\hat{s})=p(\hat{y}|z_{y})$. Then :

$p(\hat{y},z_{y},\hat{s})=p(\hat{y}|z_{y})p(z_{y}|\hat{s})p(\hat{s}).$ (13)

Therefore, the three variables follow the markov property

$\hat{s}\rightarrow z_{y}\rightarrow h_{c}(z_{y}).$ (14)

In such context, it was shown that Yeung (1991):

$\text{ MI}(z_{y},\hat{s})\geq\text{ MI}(\hat{y},\hat{s}).$ (15)

Following the same logic, we have

$\text{ MI}(z_{y},z_{s})\geq\text{ MI}(z_{y},\hat{s})$ (16)

hence

$\text{ MI}(z_{y},z_{s})\geq\text{ MI}(\hat{y},\hat{s}).$ (17)

##  Appendix B Experimental details

In this section, we provide additional experimental details, notably, we
detail the MLP architectures, give the optimal hyperparameters, and describe
the full algorithm of WFC.

###  B.1 MLP architectures

In Table 4, we present the architectural details of the classifier MLP. We
grid searched over the learning rate
($lr\in\\{1e-5,1e-4,1e-3,5e-5,5e-4,5e-3\\}$, the number of training batches
for classification per epoch $n_{d}\in\\{5,10,20\\}$, the value used to clamp
the weights to enforce the Lipschitz constraint
$clamp~{}value\in\\{0.001,0.01,0.1\\}$, the parameter $\beta\in\\{0.5,1,2\\}$,
the layer used between the first hidden, last hidden, or last layer.

Dataset | BiasInBios | Moji
---|---|---
input dimension | 768 | 2304
hidden layers | 1 | 1
hidden dimension | 300 | 300
learning rate | 0.0001 | 0.00001
batch size | 128 | 128
epochs max | 10000 | 10000
activation | TanH | TanH
$\beta$ | 1 | 1
$n_{c}$ | 5 | 5
$n_{d}$ | 20 | 5
clamp value | 0.01 | 0.01
layer used | last | last
Table 4: Details on hyperparameters used for the classifying MLP.

###  B.2 Critic architecture

In Table 5, we present the architectural details of the Critic, which is a
simple multi-layer perceptron. We grid searched over the learning rate
$lr\in\\{5e-5,5e-4,5e-3\\}$.

number hidden layer | 1
---|---
hidden dimension | 512
activation | ReLU
optimizer |  | Root Mean Square
---
Propagation
learning rate | 5e-5
Table 5: Details on hyperparameters used for the Critic MLP.

###  B.3 Algorithm of WFC

In Algorithm 1, we provide the detailed algorithm for WFC used in our
experiments.


# Relativity, the Special Theory,
explained to Children
(from 7 to 107 years old)

Charles-Michel Marle
Institut de Mathématiques de Jussieu
Université Pierre et Marie Curie
Paris, France

(15 July 2005)

###### Abstract

The author thinks that the main ideas or Relativity Theory can be explained to
children (around the age of 15 or 16) without complicated calculations, by
using very simple arguments of affine geometry. The proposed approach is
presented as a conversation between the author and one of his grand-children.
Limited here to the Special Theory, it will be extended to the General Theory
elsewhere, as sketched in conclusion.

For Agathe, Florent, Basile, Mathis, Gabrielle,

Morgane, Quitterie and my future other grand-children

##  1 Prologue

Maybe one day, one of my grand-children, at the age of 15 or 16, will ask me:

— Grand-father, could you explain what is Relativity Theory? My Physics
teacher lectured about it, talking of rolling trains and of lightnings hitting
the railroad, and I understood almost nothing!

This is the discussion I would like to have with her (or him).

— Do you know the theorem: the diagonals of a parallelogram meet at their
middle point?

— Yes, I do! I even know that the converse is true: if the diagonals of a
plane quadrilateral meet at their middle point, that quadrilateral is a
parallelogram. And I believe that I know a proof!

— Good! You know all the stuff needed to understand the basic idea of
Relativity theory! However, we must first think about Time and Space.

— Time and space seem to me very intuitive, and yet difficult to understand in
deep!

— Many people feel the same. The true nature of Time and Space is mysterious.
Let us say that together, Time and Space make the frame in which all physical
phenomena take place, in which all material objects evolve, including our
bodies. We should keep a modest mind profile on such a subject. We cannot hope
to understand all the mysteries of Time and Space. We should only try to
understand some of their properties and to use them to describe physical
phenomena. We should be ready to change the way we think about Time and Space,
if some experimental evidence shows that we were wrong.

— But if we do not know what are Time and Space, how can we hope to understand
some of their properties, and to be able to use them?

— By building mental pictures of Time and Space. Unfortunately we, poor
limited human beings, cannot do better: we know the surrounding world only
through our senses (enhanced by the measurement and observation instruments we
have built) and our ability of reasoning. Our reasoning always apply to the
mental pictures we have built of reality, not to reality itself.

Let me now indicate how the mental pictures of Time and Space used by
scientists have evolved, mainly from Newton to Einstein.

##  2 Newton’s and Leibniz’s views about Time and Space

###  2.1 Newtonian Time

The great scientist Isaac Newton [2] (1642–1727) used, as mental picture of
Time, a straight line $\cal T$, going to infinity on both sides, hence without
beginning nor end, with no privileged origin. Each particular time, for
example “now”, or ”three days ago at the sunset at Paris”, corresponds to a
particular element of that straight line.

Observe that Newton considered, without any discussion, that for each event
happening in the universe, there was a corresponding well defined time
(element of the straight line $\cal T$), the time at which that event happens.

— Where is that straight line $\cal T$? Is it drawn in some plane or in space?

— Nowhere! You should not think about the straight line of Time $\cal T$ as
drawn in something of larger dimension. Newton considered Time as an abstract
straight line, because successive events are linearly ordered, like points on
a straight line. Don’t forget that $\cal T$ is a mental picture of Time, not
Time itself! However, that mental picture is much more than a confuse idea: it
has very well defined mathematical properties. In modern language, we say that
$\cal T$ is endowed with an _affine structure_ and with an _orientation_.

— What is an affine structure? and what is its use?

— An affine structure allows us to compare two time intervals and to take
their ratio, for example to say that one of these intervals is two times the
other one. Newton considered the comparison of two time intervals as possible,
even when they were many centuries or millenaries apart, and to take their
ratio. In modern mathematical language, that property determines an _affine
structure_.

For the mathematician, that property means that we can apply transforms to
$\cal T$ by sliding it along itself, without contraction nor dilation, and
that these transforms (called _translations_) do not change its properties.

For the physicist, it means that the physical laws remain the same at all
times.

Another important property of Time: it always flows from past to future. To
take it into account, we endow $\cal T$ with an _orientation_ ; it means that
we consider the two directions (from past to future and from future to past)
as different, not equivalent, for example by choosing the direction from past
to future as preferred. We then say that $\cal T$ is _oriented_.

###  2.2 Newton’s absolute space

— OK, I roughly agree with that mental picture, although it does not account
for the main property of Time: it flows continuously and we cannot stop it!
And what about Space?

— Newton identified Space with the three dimensional space of geometers,
denoted by $\cal E$ : the space in which there are various figures made of
planes, straight lines, spheres, polyhedra, which obey the theorems developed
in Euclidean geometry: Thales and Pythagoras theorems, the theorem which says
that the diagonals of a parallelogram meet at their middle point, $\ldots$

###  2.3 The concept of Space-Time

Newton used Time and Space to describe the motion of every object $A$ of the
physical world as follows. That object occupies, at each time $t$ (element of
$\cal T$) for which it exists, a position $A_{t}$ in Space $\cal E$. The
motion of of $A$ is described by its successive positions $A_{t}$ when $t$
varies in $\cal T$.

Let me introduce now a new concept, that of Space-Time [1], due to the German
mathematician Hermann Minkowski (1864–1909). That concept was not used in
Mechanics before the discovery of Special Relativity. That is very
unfortunate, since its use makes much easier the understanding of the
foundations of Classical Mechanics, as well as those of Relativistic
Mechanics. Therefore I use it now, with the absolute Time and Space of Newton,
although Newton himself did not use that concept.

Newton Space-Time is simply the product set ${\cal E}\times{\cal T}$, whose
elements are pairs (called _events_) $(x,t)$, made by a point $x$ of $\cal E$
and a time $t$ of $\cal T$.

— What is the use of that Space-Time?

— It is very convenient to describe motions. For example, the motion of a
material particle $a$ (a very small object whose position, at each time
$t\in{\cal T}$, is considered as a point $a_{t}\in{\cal E}$), is described by
a line in ${\cal E}\times{\cal T}$, made by the events $(a_{t},t)$, for all
$t$ in the interval of time during which $a$ exists. That line is called the
_world line_ of $a$.

You will see on Figure 1 (where, for simplicity, Space is represented as a
straight line, as if it were one-dimensional) the world lines of three
particles, $a$, $b$ and $c$.

Figure 1: World lines in Newton Space-Time.

  * •

The world line of $b$ est parallel to the Time axis $\cal T$: that particle is
at rest, il occupies a fixed position in the absolute Space $\cal E$.

  * •

The world line of $c$ is a slanting straight line. The trajectory of that
particle in absolute Space $\cal E$ is a straight line and its velocity is
constant.

  * •

The world line of $a$ is a curve, not a straight line. It means that the
velocity of $a$ changes with time.

###  2.4 Absolute rest and motion

For Newton, _rest_ and _motion_ were absolute concepts: a physical object is
at rest if its position in Space does not change with time; otherwise, it is
in motion.

— It seems very natural. Why should we change this view?

— Because nothing is at rest in the Universe! The Earth rotates around its
axis and around the Sun, which rotates around the center of our Galaxy. And
there are billions of galaxies in the Universe, all moving with respect to the
others! For these reasons, Newton’s concept of an absolute Space was
criticized very early, notably by his contemporary, the great mathematician
and philosopher Gottfried Wilhelm Leibniz (1647–1716).

###  2.5 Reference frames

— But without knowing what is at rest in the Universe, how Newton managed to
study the motions of the planets?

— To study the motion of a body $A$, Newton, and after him almost all
scientists up to now, used a _reference frame_. It means that he used another
body $R$ which remained approximately rigid during the motion he wanted to
study, and he made as if that body was at rest. Then he could study the
_relative motion_ of $A$ with respect to $R$.

Assuming that Newton’s absolute Space $\cal E$ exists, we recover the
description of absolute motion of $A$ by choosing, for $R$, a body at rest in
$\cal E$. The corresponding reference frame is called the _absolute fixed
frame_.

The body $R$ used to determine a reference frame can be, for example,

  * •

the Earth (if we want to study the motion of a falling apple),

  * •

the trihedron made by the straight lines which join the center of the Sun to
three distant stars (if we want to study the motions of the planets in the
solar system).

###  2.6 Galilean frames and Leibniz Space-Time

All reference frames are not equivalent. A _Galilean frame_ 111 In memory of
Galileo Galilei, (1564–1642), the founder of modern Physics., also called an
_inertial frame_ , is a reference frame in which the _principe of inertia_
holds true. That principle, first formulated for absolute motions in Newton’s
absolute space $\cal E$, says that the (absolute) motion of a free particle
takes place on a straight line, at a constant speed. But, as shown by Newton
himself, that principe remains true for the _relative motion_ of a free
particle with respect to some particular reference frames, the _Galilean
frames_.

More exactly, let us assume that the principle of inertia holds true for the
relative motion of free particles with respect to the reference frame defined
by the rigid body $R_{1}$. What happens for the relative motion of these free
particles with respect to another reference frame, defined by another rigid
body $R_{2}$? It is easy to see that the principle of inertia still holds true
_if and only if_ the relative motion of $R_{2}$ with respect to $R_{1}$ is a
motion by translation at a constant speed.

The absolute frame, if it exists, therefore appears as a Galilean frame among
an infinite number of other Gallilean frames, that no measurement founded on
mechanical properties can distinguish from the others. For this reason,
several scientists, following Leibniz, doubted about its existence.

Leibniz accepted Newton’s concept of an absolute Time, but not that of an
absolute Space. His views were not successful during his life, probably
because at that time nobody saw how to cast them in a mathematically rigorous
setting. Now we can do that; let me explain how.

We will consider that at each time $t\in{\cal T}$, there exists a _Space at
time $t$_, denoted by ${\cal E}_{t}$, whose properties are those of the three-
dimensional Euclidean space of geometers. We must consider that the Spaces
${\cal E}_{t_{1}}$ and ${\cal E}_{t_{2}}$, at two different times $t_{1}$ and
$t_{2}$, $t_{1}\neq t_{2}$, have no common element. Leibniz Space-Time, which
will be denoted by $\cal U$ (for Universe), is the disjoint union of all the
Spaces ${\cal E}_{t}$ for all times $t\in{\cal T}$. So, according to Leibniz
views, we still have a Space-Time, but no more an absolute space ! The next
picture shows,

  * •

on the left side, Newton Space-Time ${\cal E}\times{\cal T}$, with the two
projections $p_{1}:{\cal E}\times{\cal T}\to{\cal E}$ and $p_{2}:{\cal
E}\times{\cal T}\to{\cal T}$;

  * •

on the right side, Leibniz Space-Time $\cal U$, endowed with only one natural
projection onto absolute Time $\cal T$, still denoted by $p_{2}:{\cal
U}\to{\cal T}$; the horizontal lines represent the Spaces ${\cal
E}_{t}=p_{2}^{-1}(t)$, for various values of $t\in{\cal T}$.

Figure 2: Newton and Leibniz Space-Time.

— But how do you put together the Spaces at various times ${\cal E}_{t}$ to
make Leibniz Space-Time $\cal U$? Are they stacked in an arbitrary way?

— Of course no! Leibniz Space-Time $\cal U$ is a $4$-dimensional affine space,
fibered (via an affine map) over Time $\cal T$, which is itself a
$1$-dimensional affine space. Its fibres, the Spaces ${\cal E}_{t}$ at various
times $t\in{\cal T}$, are $3$-dimensional Euclidean spaces. The affine
structure of $\cal U$ is determined by the _principle of inertia_ of which we
have already spoken. That principle can be formulated in a way which does not
use reference frames, by saying:

_The world line of any free particle is a straight line._

So formulated, the principle of inertia can be applied to Newton Space-Time
${\cal E}\times{\cal T}$ and to Leibniz Space-Time $\cal U$ as well. More, it
_determines_ the affine structure of $\cal U$, since one can easily show that
the affine structure for which it holds true, if any, is unique. A physical
law, the _principle of inertia_ , is so embedded in the geometry of Leibniz
Space-Time $\cal U$.

By using a reference frame $R$, one can split Leibniz Space-Time into a
product of two factors: a space ${\cal E}_{R}$, fixed with respect to that
frame, and the absolute Time $\cal T$. But of course, the space ${\cal E}_{R}$
depends on the choice of the reference frame $R$. For that reason, it seems
that before 1905, not many scientists were aware of the fact that by dropping
Newton’s absolute Space $\cal E$, they already had completely changed the
conceptual setting in which motions are described:

  * •

according to Newton, absolute Space ${\cal E}$ and absolute Time ${\cal T}$
were directly related to reality, while Space-Time ${\cal E}\times{\cal T}$
was no more than a mathematical object, not very interesting (he did not use
it) and not directly related to reality;

  * •

but according to Leibniz’s views, when expressed as done above, it is Space-
Time $\cal U$ which is directly related to reality, as well as absolute Time
$\cal T$; absolute Space $\cal E$ no more exists.

##  3 Relativity

Einstein [1] was led to drop Leibniz Space-Time when trying to reconcile the
theories used in two different parts of Physics: Mechanics on one hand,
Electromagnetism and Optics on the other hand.

According to the theory built by the great Scotch physicist James Clerk
Maxwell (1831–1879), electromagnetic phenomena propagate in vacuum as waves,
with the same velocity in all directions, independently of the motion of the
source of these phenomena. Maxwell soon understood that light was an
elecromagnetic wave, and lots of experimental results confirmed his views.

###  3.1 The luminiferous ether, a short lived hypothesis

In Leibniz Space-Time (as well as in Newton Space-Time) _relative velocities
behave additively_. In that setting, it is with respect to _at most one
particular reference frame_ that light can propagate with the same velocity in
all directions. Physicists introduced a new hypothesis: electromagnetic waves
were considered as vibrations of an hypothetic, very subtle, but highly rigid
medium called the _luminiferous ether_ , everywhere present in space, even
inside solid bodies. They thought that it was with respect to the ether’s
reference frame that light propagates at the same velocity in all directions.
This new hypothesis amounts to come back to Newton’s absolute Space identified
with the ether. There were even physicists who introduced additional
complications, by assuming that the ether, partially drawn by the motion of
moving bodies, could deform with time!

— But if the luminiferous ether really exists, accurate measurements of the
velocity of light in all directions should allow the determination of the
Earth’s relative velocity with respect to the ether!

— Good remark! These measurements were made several times, notably by Albert
Abraham Michelson (1852–1931) and Edward Williams Morley (1838–1923), between
1880 et 1887. No relative velocity of the Earth with respect to the
luminiferous ether could be detected.

These results remained not understood until 1905, despite many attempts. The
most interesting of these attempts was that due to Hendrik Anton Lorentz
(1853–1928) and George Francis FitzGerald (1851–1901). Independently, they
proposed the following hypothesis: when a rigid body, for example a rule or
the arm of an interferometer, is moving with respect to the luminiferous
ether, that body contracts slightly in the direction of its relative
displacement.

— So that is the famous relativistic contraction my teacher spoke about!

— No! Not at all! Lorentz and FitzGerald considered that contraction as a true
physical effect of the relative motion of a body with respect to the ether.
This assumption is now completely abandoned, together with the luminiferous
ether! The relativistic contraction of lengths and dilation of times has
nothing to do with it: rather than a real phenomenon, it is only an
appearance, like the following effect of perspective. Imagine that you look at
a 20 centimeters rule, from a distance of, say two meters from its center.
That rule looks shorter when it is not perpendicular to the straight line
which joins your eye to its center than when it is. It may even seem to be
reduced to a point when it lies along that straight line. As we will soon see,
the relativistic contraction of lengths and dilation of times has a similar
origin.

###  3.2 Minkowski Space-Time

Einstein was the first 222 The great French mathematician Jules Henri Poincaré
(1854–1912) has, almost simultaneously and independently, presented very
similar ideas [3], without explicitly recommending to drop the concept of an
absolute Time. to understand (in 1905) that the results of Michelson and
Morley experiments could be explained by a deep change of the properties
ascribed to Space and Time. At that time, his idea appeared as truly
revolutionary. But now it may appear as rather natural, if we think along the
following lines:

_When we dropped Newton Space-Time in favour of Leibniz Space-Time, we
recognized that there is no absolute Space, but that Space depends on the
choice of a reference frame. Maybe Time too is no more absolute than Space,
and depends on the choice of a reference frame!_

— But if we drop absolute Time, which properties are left to our Space-Time?

— In 1905, Einstein implicitly considered that Space-Time still was a
$4$-dimensional affine space, which will be called _Minkowski Space-Time_ and
will be denoted by $\cal M$. He implicitly considered too that _translations_
of $\cal M$ leave its properties unchanged, and he assumed that the _principe
of inertia_ still holds true in $\cal M$ when expressed without the use of
reference frames:

_The world line of any free particle is a straight line._

He also kept the notion of a _Galilean frame_. In $\cal M$, a Galilean frame
is determined by a direction of straight line (not any straight line, a _time-
like_ straight line, as we will see below). Given a Galilean frame $R$,
Minkowski Space-Time $\cal M$ can be split into a product ${\cal
E}_{R}\times{\cal T}_{R}$ of a three-dimensional Space ${\cal E}_{R}$ and a
one-dimensional Time ${\cal T}_{R}$, which both depend on $R$. Let me recall
that in Leibniz Space-Time $\cal U$, a Galilean frame $R$ allowed us to split
$\cal U$ into a product ${\cal E}_{R}\times{\cal T}$ of a three-dimensional
Space ${\cal E}_{R}$, which depended on $R$, and the one-dimensional absolute
Time $\cal T$, which did not depend on $R$. That is the main difference
between Leibniz’s and Einstein’s views about Space and Time.

Under these hypotheses, the properties of Space-Time follow from two
principles:

  * •

the _Principle of Relativity:_ all physical laws have the same expression in
all Galilean frames;

  * •

the _Principle of Constancy of the velocity of light:_ the modulus of the
velocity of light is an universal constant, which depends neither on the
Galilean frame with respect to which it is calculated, nor on the motion of
the source of that light.

— You said that a direction of straight line was enough to determine a
Galilean frame. But how is that possible, since we no more have an absolute
Time?

— That determination will follow from the pinciple of constancy of the
velocity of light. Let us call _light lines_ the straight lines in $\cal M$
which are possible world lines of light signals. Given an event $A\in{\cal
M}$, the light lines through $A$ make a $3$-dimensional cone, the _light cone
with apex $A$_; the two layers of that cone are called _the past half-cone_
and _the future half-cone_ with apex $A$. Since it is assumed that
translations leave unchanged the properties of Space-Time, the light cone with
another event $B$ as apex is deduced from the light cone with apex $A$ by the
translation which maps $A$ onto $B$.

Apart from light lines, there are two other kinds of straight lines in $\cal
M$:

  * •

_time-like straight lines_ , which lie _inside_ the light cone with any one of
their elements as apex;

  * •

and _space-like straight lines_ , which lie _outside_ the light cone with any
of their element as apex.

I can now explain how the direction of a time-like straight line $\cal A$
determines a Galilean frame $R$. That frame is such that the rigid bodies at
rest in it are those whose all material points have, as world lines, straight
lines parallel to $\cal A$. The straight lines parallel to $\cal A$ will be
called the _isochorous lines_ 333 The word _isochorous_ , already used in
Thermodynamics, refers here to a set of events which all occur at the same
spatial location at various times, in similarity with the word _isochronous_
which refers to a set of events which all occur simultaneously in time at
various spatial locations. of the reference frame $R$; each of these lines is
a set of events which all happen at the same place in the Space ${\cal E}_{R}$
of our frame $R$. For each event $M\in{\cal M}$, the set of all other events
which occur at the same time as $M$, for the Time ${\cal T}_{R}$ of our
Galilean frame $R$, will be called the _isochronous subspace_ through $M$, for
the Galilean frame $R$. It is a $3$-dimensional affine subspace ${\cal
E}_{R,\,M}$ of $\cal M$ containing the event $M$, and the other isochronous
subspaces for $R$ are all the $3$-dimensional subspaces of $\cal M$ parallel
to ${\cal E}_{R,M}$. They are determined by the property: the length covered
by a light signal, calculated in the reference frame $R$, during a given time
interval, also evaluated in that reference frame, _is the same in any two
opposite directions._

In a schematic $2$-dimensional Space-Time (or in a plane section containing
$\cal A$ of the “true” $4$-dimensional Space-Time), the direction of
isochronous subspaces is easily obtained as shown on the left part of Figure
3: we take the two light lines ${\cal L}^{g}$ and ${\cal L}^{d}$ through an
event $A\in{\cal A}$ (the red lines on that figure); we take another event
$A_{1}\in A$, for example in the future of $A$, and we build the parallelogram
$A\,A_{1}^{g}\,A_{2}\,A_{1}^{d}$ with two sides supported by ${\cal L}^{g}$
and ${\cal L}^{d}$, with $A$ as one of its apices and $A_{1}$ as center. The
isochronous subspaces are all the straight lines parallel to the space-like
diagonal $A_{1}^{g}\,A_{1}^{d}$ of that parallelogram. Three of these lines
are drawn (in blue) on Figure 3, ${\cal E}_{R,\,A}$, ${\cal E}_{R,\,A_{1}}$
and ${\cal E}_{R,\,A_{2}}$.

— Why?

— A light signal starting from $A$ covers, during the time interval between
events $A$ and $A_{1}$, the lengths $A_{1}\,A_{1}^{g}$ towards the left and
$A_{1}\,A_{1}^{d}$ towards the right. These lengths are equal because
$A_{1}^{g}\,A_{1}^{d}$ is the diagonal of a parallelogram whose center is
$A_{1}$.

Figure 3: Construction of Space and Time relative to a Galilean frame

— What for the “true” $4$-dimensional Minkowski Space-Time $\cal M$ ? And what
are the Space ${\cal E}_{R}$ and the Time ${\cal T}_{R}$ of our reference
frame $R$?

— It is the same, as shown on the right side of Figure 3. Take the event
$A_{2}$ on the light line $\cal A$ such that $A_{1}$ is the middle point of
$A\,A_{2}$. Consider the future light half-cone with apex $A$ and the past
light half-cone with apex $A_{2}$. Their intersection is a $2$-dimensional
sphere $S$. The unique affine hyperplane ${\cal E}_{R,\,A_{1}}$ which contains
$S$ is an isochronous subspace for the Galilean frame determined by the
direction of $\cal A$ (in blue on Figure 3). The other isochronous subspaces
for that Galilean frame are all the hyperplanes parallel to ${\cal
E}_{R,\,A_{1}}$. The Space ${\cal E}_{R}$ is the set of all the isochorous
lines, _i.e_ the set of all straight lines parallel to $\cal A$, and the Time
${\cal T}_{R}$ the set of all isochronous subspaces. Minkowski Space-Time
$\cal M$ splits into the product ${\cal E}_{R}\times{\cal T}_{R}$, or in other
words can be identified with that product, because a pair made by an
isochorous line and an isochronous subspace determine a unique element of
$\cal M$, the event at which they meet.

— What happens if you change your Galilean frame?

— Of course, as for Galilean frames in Leibniz Space-Time, the direction of
isochorous lines (the straight world lines of points at rest with respect to
the chosen Galilean frame) is changed. Moreover, contrary to what happened in
Leibniz Space-Time, the direction of isochronous subspaces is also changed!
Therefore, the chronological order of two events can be different when it is
appreciated in two different Galilean frames!

###  3.3 Metric properties of Minkowski Space-Time

Up to now, we have compared the lengths of two straight line segments in $\cal
M$ only when they were supported by parallel straight lines. That was allowed
by the _affine structure_ of $\cal M$. We need more, because the spectral
lines of atoms allow us to build clocks and to compare time intervals measured
in two different Galilean frames.

Let $A\,A_{1}$ and $A\,B_{1}$ be two straight line segments supported by two
different time-like straight lines $\cal A$ and $\cal B$, which meet at the
event $A$. Let $R_{\cal A}$ and $R_{\cal B}$ be the Galilean frames determined
by the directions of $\cal A$ and $\cal B$, respectively. We assume that the
time intervals corresponding to $A\,A_{1}$ measured in $R_{\cal A}$, and to
$A\,B_{1}$ measured in $R_{\cal B}$, are the same. Let $B^{\prime}$ be the
event at which the time-like straight line $\cal B$ meets the isochronous
subspace ${\cal E}_{R_{\cal A},A_{1}}$ containing $A_{1}$ of the Galilean
frame $R_{\cal A}$ (figure 4). Since the events $A_{1}$ and $B^{\prime}$ are
synchronous for $R_{\cal A}$, the time interval corresponding to $A\,B_{1}$
appears longer than the time interval corresponding to $A\,A_{1}$ when both
are observed in the reference frame $R_{\cal A}$, by the ratio
$\displaystyle\frac{A\,B_{1}}{A\,B^{\prime}}$. That ratio is the _ratio of
dilation of times_ of the Galilean frame of $R_{\cal B}$, when observed in the
Galilean frame $R_{\cal A}$. Similarly,
$\displaystyle\frac{A\,A_{1}}{A\,A^{\prime}}$ is the ratio of dilation of
times of the Galilean frame $R_{\cal A}$ when observed in the Galilean frame
$R_{\cal B}$. According to the Principle of Relativity, these two Galilean
frames must play the same role with respect to the other, which implies the
equality
$\displaystyle\frac{A\,A_{1}}{A\,A^{\prime}}=\frac{A\,B_{1}}{A\,B^{\prime}}$.
By a well known property of hyperbolae, that equality holds _if and only if
$A_{1}$ and $B_{1}$ lie on the same arc of hyperbola which has the light lines
${\cal L}^{d}$ and ${\cal L}^{g}$ (which meet at $A$ and are contained in the
two-dimensional plane which contains $\cal A$ and $\cal B$) as asymptotes_. Or
more generally, on the same hyperboloid with the light cone of $A$ as
asymptotic cone.

Figure 4: Comparison of times.

The comparison of lengths on two non-parallel space-like straight lines is
similar to the comparison of time intervals. Let $A\,A^{d}$ and $A\,B^{d}$ be
two segments supported by two space-like straight lines which meet at the
event $A$. They are of equal length _if and only if $A^{d}$ and $B^{d}$ lie on
the same hyperboloid with the light cone of $A$ as asymptotic cone_.

##  4 Conclusion

The comparison of time intervals and lengths presented above allows a very
natural introduction of the pseudo-Euclidean metric of Minkowski Space-Time.
The construction of isochronous subspaces in two different Galilean frames, as
presented above, leads to the formulas for Lorentz transformations with a
minimum of calculations. The pictures we have presented allow a very easy
explanation of the apparent contraction of lengths and dilation of times
associated to a change of Galilean frames and a very simple explanation,
without complicated calculations, of the (improperly called) paradox of
Langevin’s twins.

By explaining that the affine structure of Space-Time should be questioned, a
smooth transition towards General Relativity, suitable from children from 8 to
108 years old, seems possible.

Acknowledgements. The author thanks the team “Analyse algébrique” of the
“Institut de Mathématiques de Jussieu” and his University for taking in charge
his registration fee at this International Conference.

## References

  * [1] Einstein, A., Lorentz, H.A., Weyl, H., Minkowski, H., The Principles of Relativity, a collection of original papers on the special and general theory of relativity, with notes by A. Sommerfeld. Methuen and Company, 1923. Reprinted by Dover Publications, Inc., New York.
  * [2] Newton, Isaac, Principes mathématiques de la Philosophie naturelle, tomes I et II, translated by Madame la Marquise du Chastellet, chez Desaint et Saillant, Paris, 1759. Reprinted by the Éditions Jacques Gabay, Paris, 1990.
  * [3] Poincaré, Henri, La Mécanique nouvelle, book containing the text of a lecture presented at the congress of the “Association française pour l’avancement des sciences” (Lille, 1909), the paper dated 23 July 1905 Sur la dynamique de l’électron, Rendiconti del Circolo matematico di Palermo XXI (1906), and a “Note aux Comptes Rendus de l’Académie des Sciences” with the same title dated 15 June 1905; Gauthier-Villars, Paris, 1924; reprinted by the Éditions Jacques Gabay, Paris, 1989.