# Erasure of Unaligned Attributes from Neural Representations

Shun Shao∗ Yftah Ziser Shay B. Cohen
Institute for Language, Cognition and Computation
School of Informatics, University of Edinburgh
10 Crichton Street, Edinburgh, EH8 9AB
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

We present the Assignment-Maximization Spectral Attribute removaL (AMSAL)
algorithm, which erases information from neural representations when the
information to be erased is implicit rather than directly being aligned to
each input example. Our algorithm works by alternating between two steps. In
one, it finds an assignment of the input representations to the information to
be erased, and in the other, it creates projections of both the input
representations and the information to be erased into a joint latent space. We
test our algorithm on an extensive array of datasets, including a Twitter
dataset with multiple guarded attributes, the BiasBios dataset and the
BiasBench benchmark. The last benchmark includes four datasets with various
types of protected attributes. Our results demonstrate that bias can often be
removed in our setup. We also discuss the limitations of our approach when
there is a strong entanglement between the main task and the information to be
erased.111Our code is available at https://github.com/jasonshaoshun/AMSAL.

##  1 Introduction

Developing a methodology for adjusting neural representations to preserve user
privacy and avoid encoding bias in them has been an active area of research in
recent years. Previous work shows it is possible to erase undesired
information from representations so that downstream classifiers cannot use
that information in their decision-making process. This previous work assumes
that this sensitive information (or guarded attributes, such as gender or
race) is available for each input instance. These guarded attributes, however,
are sensitive, and obtaining them on a large scale is often challenging and,
in some cases, not feasible Han et al. (2021b). For example, Blodgett et al.
(2016) studied the characteristics of African-American English (AAE) on
Twitter, and could not couple the ethnicity attribute directly with the tweets
they collected due to the attribute’s sensitivity.

This paper introduces a novel debiasing setting in which the guarded
attributes are not paired up with each input instance and an algorithm to
remove information from representations in that setting. In our setting, we
assume that each neural input representation is coupled with a guarded
attribute value, but this assignment is unavailable. In cases where the domain
of the guarded attribute is small (for example, with binary attributes), this
means that the guarded attribute information consists of priors with respect
to the whole population and not instance-level information.

$\mathbf{x}^{(1)}$$\mathbf{x}^{(2)}$$\mathbf{x}^{(3)}$$\mathbf{x}^{(4)}$$\mathbf{x}^{(5)}$$\mathbf{z}^{(1)}$$\mathbf{z}^{(2)}$$\mathbf{z}^{(3)}$$\mathbf{z}^{(4)}$$\bm{U}$$\bm{V}$

Figure 1:  A depiction of the problem setting and solution. The inputs are
aligned to each guarded sample, based on strength using two projections
$\bm{U}$ and $\bm{V}$. We solve a bipartite matching problem to find the blue
edges, and then recalculate $\bm{U}$ and $\bm{V}$.

The intuition behind our algorithm is that if we were to find a strong
correlation between the input variable and a set of guarded grounded
attributes either in the form of an unordered list of records or as priors,
then it is unlikely to be coincidental if the sample size is sufficiently
large (§3.5). We implement this intuition by jointly finding projections of
the input samples and the guarded attributes into a joint embedding space
_and_ an alignment between the two sets in that joint space.

Our resulting algorithm (§3), the Alignment-Maximization Spectral Attribute
removaL algorithm (AMSAL), is a coordinate-ascent algorithm reminiscent of the
hard expectation-maximization algorithm (hard EM; MacKay 2003). It first loops
between two Alignment and Maximization steps, during which it finds an
alignment (A) based on existing projections and then projects the
representations and guarded attributes into a joint space based on an existing
alignment (M). After these two steps are iteratively repeated and an alignment
is identified, the algorithm takes another step to erase information from the
input representations based on the projections identified. This step closely
follows the work of Shao et al. (2023), who use Singular Value Decomposition
(SVD) to remove principal directions of the covariance matrix between the
input examples and the guarded attributes. Figure 1 depicts a sketch of our
setting and the corresponding algorithm, with $\mathbf{x}_{i}$ being the input
representations and $\mathbf{z}_{j}$ being the guarded attributes. Our
algorithm is modular: while our use of the algorithm of Shao et al. (2023) for
the removal step is natural due to the nature of the AM steps, a user can use
any such algorithm to erase the information from the input representations
(§3.4).

Our contributions are as follows: (1) We propose a new setup for removing
guarded information from neural representations where there are few or no
labeled guarded attributes; (2) We present a novel two-stage coordinate-ascent
algorithm that iteratively improves (a) an alignment between guarded
attributes and neural representations; and (b) information removal
projections.

Using an array of datasets, we perform extensive experiments to assess how
challenging our setup is and whether our algorithm is able to remove
information without having aligned guarded attributes (§4). We find in several
cases that little information is needed to align between neural
representations and their corresponding guarded attributes. The consequence is
that it is possible to erase the information such guarded attributes provide
from the neural representations while preserving the information needed for
the main task decision-making. We also study the limitations of our algorithm
by experimenting with a setup where it is hard to distinguish between the
guarded attributes and the downstream task labels when aligning the neural
representations with the guarded attributes (§4.5).

##  2 Problem Formulation and Notation

For an integer $n$ we denote by $[n]$ the set $\\{1,\ldots,n\\}$. For a vector
$\mathbf{v}$, we denote by $||\mathbf{v}||_{2}$ its $\ell_{2}$ norm. For two
vectors $\mathbf{v}$ and $\mathbf{u}$, by default in column form,
$\langle\mathbf{v},\mathbf{u}\rangle=\mathbf{v}^{\top}\mathbf{u}$ (dot
product). Matrices and vectors are in boldface font (with uppercase or
lowercase letters, respectively). Random variable vectors are also denoted by
boldface uppercase letters. For a matrix $\bm{A}$, we denote by $a_{ij}$ the
value of cell $(i,j)$. The Frobenius norm of a matrix $\bm{A}$ is
$||\bm{A}||_{F}=\sqrt{\sum_{i,j}a_{ij}^{2}}$. The spectral norm of a matrix is
$||\bm{A}||_{2}=\max_{||\mathbf{x}||_{2}=1}||\bm{A}\mathbf{x}||_{2}$. The
expectation of a random variable $\mathbf{T}$ is denoted by
$\mathbb{E}[\mathbf{T}]$.

In our problem formulation, we assume three random variables:
$\mathbf{X}\in\mathbb{R}^{d}$, $\mathbf{Y}\in\mathbb{R}$ and
$\mathbf{Z}\in\mathbb{R}^{d^{\prime}}$ such that $d^{\prime}\leq d$ and the
expectation of all three variables is $0$ (see Shao et al. 2023). Samples of
$\mathbf{X}$ are the inputs for a classifier to predict corresponding samples
of $\mathbf{Y}$. The random vector $\mathbf{Z}$ represents the guarded
attributes. We want to maintain the ability to predict $\mathbf{Y}$ from
$\mathbf{X}$, while minimizing the ability to predict $\mathbf{Z}$ from
$\mathbf{X}$.

We assume $n$ samples of $(\mathbf{X},\mathbf{Y})$ and $m$ samples of
$\mathbf{Z}$, denoted by $(\mathbf{x}^{(i)},\mathbf{y}^{(i)})$ for $i\in[n]$,
and $\mathbf{z}^{(i)}$ for $i\in[m]$ ($m\leq n$). While originally, these
samples were generated jointly from the underlying distribution
$p(\mathbf{X},\mathbf{Y},\mathbf{Z})$, we assume a shuffling of the
$\mathbf{Z}$ samples in such a way that we are only left with $m$ samples that
are unique (no repetitions) and an underlying unknown many-to-one mapping
$\pi\colon[n]\rightarrow[m]$ that maps each $\mathbf{x}^{(i)}$ to its original
$\mathbf{z}^{(j)}$.

The problem formulation is such that we need to remove the information from
the $x$s in such a way that we consider the samples of $z$s as a set. In our
case, we do so by iterating between trying to infer $\pi$, and then using
standard techniques, remove the information from $x$s based on their alignment
to the corresponding $z$s.

#### Singular Value Decomposition

Let $\bm{A}=\mathbb{E}[\mathbf{X}\mathbf{Z}^{\top}]$, the matrix of cross-
covariance between $\mathbf{X}$ and $\mathbf{Z}$. This means that
$\bm{A}_{ij}=\mathrm{Cov}(\textnormal{X}_{i},\textnormal{Z}_{j})$ for
$i\in[d]$ and $j\in[d^{\prime}]$.

For any two vectors,
$\mathbf{a}\in\mathbb{R}^{d},\mathbf{b}\in\mathbb{R}^{d^{\prime}}$, the
following holds due to the linearity of expectation:

$\displaystyle\mathbf{a}\bm{A}\mathbf{b}^{\top}=\mathrm{Cov}(\mathbf{a}^{\top}\mathbf{X},\mathbf{b}^{\top}\mathbf{Z}).$
(1)

Singular value decomposition on $\bm{A}$, in this case, finds the “principal
directions”: directions in which the projection of $\mathbf{X}$ and
$\mathbf{Z}$ maximize their covariance. The projections are represented as two
matrices $\bm{U}\in\mathbb{R}^{d\times d}$ and
$\bm{V}\in\mathbb{R}^{d^{\prime}\times d^{\prime}}$. Each column in these
matrices plays the role of the vectors $\mathbf{a}$ and $\mathbf{b}$ in Eq. 1.
SVD finds $\bm{U}$ and $\bm{V}$ such that for any $i\in[d^{\prime}]$ it holds
that:

$\displaystyle\mathrm{Cov}(\bm{U}_{i}^{\top}\mathbf{X},\bm{V}_{i}^{\top}\mathbf{Z})=\max_{(\mathbf{a},\mathbf{b})\in\mathcal{O}_{i}}\mathrm{Cov}(\mathbf{a}^{\top}\mathbf{X},\mathbf{b}^{\top}\mathbf{Z}),$
(2)

where $\mathcal{O}_{i}$ is the set of pairs of vectors
$(\mathbf{a},\mathbf{b})$ such that $||\mathbf{a}||_{2}=||\mathbf{b}||_{2}=1$,
$\mathbf{a}$ is orthogonal to $\bm{U}_{1},\ldots,\bm{U}_{i-1}$ and similarly,
$\mathbf{b}$ is orthogonal to $\bm{V}_{1},\ldots,\bm{V}_{i-1}$.

Shao et al. (2023) showed that SVD in this form can be used to debias
representations. We calculate SVD between $\mathbf{X}$ and $\mathbf{Z}$ and
then prune out the principal directions that denote the highest covariance. We
will use their method, SAL (Spectral Attribute removaL), in the rest of the
paper. See also §3.4.

Inputs: Samples $\mathbf{x}^{(1)},\ldots,\mathbf{x}^{(n)}$, and
$\mathbf{z}^{(1)},\ldots,\mathbf{z}^{(m)}$. Algorithm: (calculate projection
which removes information from the $x$s that is in the $z$s) Initialize $\pi$
randomly to a function from $[n]$ to $[m]$. Repeat the following for $T$
iterations: • (M-step) Using $\pi$, let $\bm{\Omega}_{\pi}$ as in Eq. 8.
Calculate SVD on $\bm{\Omega}_{\pi}$ to calculate
$(\bm{U},\bm{\Sigma},\bm{V})$. • (A-step) With $\bm{U}$ and $\bm{V}$ as above,
with top $k$ singular vectors, find $\pi$ by solving the problem as in Eq. 7.
Return: The singular vectors from $\bm{U}$ that have lowest singular values.

Figure 2: The main Assignment-Maximization Spectral Attribute removaL (AMSAL)
algorithm for removal of information without alignment between samples of
$\mathbf{X}$ and $\mathbf{Z}$.

##  3 Methodology

We view the problem of information removal with unaligned samples as a joint
optimization problem of: (a) finding the alignment; (b) finding the projection
that maximizes the covariance between the alignments, and using its complement
to project the inputs. Such an optimization, in principle, is intractable, so
we break it down into two coordinate-ascent style steps: A-step (in which the
alignment is identified as a bipartite graph matching problem) and M-step (in
which based on the previously identified alignment, a maximal-covariance
projection is calculated). Formally, the maximization problem we solve is:

$(\bm{U},\bm{V},\pi)=\arg\max_{\bm{U},\bm{V},\pi}\sum_{i=1}^{n}(\mathbf{x}^{(i)})^{\top}\bm{U}\bm{V}^{\top}\mathbf{z}^{(i)},$
(3)

where we constrain $\bm{U}$ and $\bm{V}$ to be matrices with orthonormal
columns in $\mathbb{R}^{n\times k}$.

Note that the sum in the above equation has a term per pair of
$(\mathbf{x}^{(i)},\mathbf{z}^{\pi(i)})$, which enables us to frame the A-step
as an integer linear programming (ILP) problem (§3.1). The full algorithm is
given in Figure 2, and we proceed in the next two steps to further explain the
A-step and the M-step.

###  3.1 A-step (Guarded Sample Assignment)

In the Assignment Step, we are required to find a many-to-one alignment
$\pi\colon[n]\rightarrow[m]$ between
$\\{\mathbf{x}^{(1)},\ldots,\mathbf{x}^{(n)}\\}$ and
$\\{\mathbf{z}^{(1)},\ldots,\mathbf{z}^{(m)}\\}$. Given $\bm{U}$ and $\bm{V}$
from the previous M-step, we can find such an assignment by solving the
following optimization problem:

$\arg\max_{\pi}\sum_{i=1}^{n}\langle\bm{U}^{\top}\mathbf{x}^{(i)},\bm{V}^{\top}\mathbf{z}^{(\pi(i))}\rangle.$
(4)

This maximization problem can be formulated as an integer linear program of
the following form:

$\displaystyle\max_{\bm{P}\in\\{0,1\\}^{n\times m}}$
$\displaystyle\sum_{j=1}^{m}\sum_{i=1}^{n}p_{ij}\langle\bm{U}^{\top}\mathbf{x}^{(i)},\bm{V}^{\top}\mathbf{z}^{(j)}\rangle$
(5) s.t. $\displaystyle\forall i.$ $\displaystyle\sum_{j=1}^{m}p_{ij}=1,$ (6)
$\displaystyle\forall j.$ $\displaystyle b_{0j}\leq\sum_{i=1}^{m}p_{ij}\leq
b_{1j}.$ (7)

This is a solution to an assignment problem Kuhn (1955); Ramshaw and Tarjan
(2012), where $p_{ij}$ denotes whether $\mathbf{x}^{(i)}$ is associated with
the (type of) guarded attribute $\mathbf{z}^{(j)}$. The values
$(b_{0j},b_{1j})$ determine lower and upper bounds on the number of $x$s a
given $\mathbf{z}^{(j)}$ can be assigned to. While a standard assignment
problem can be solved efficiently using the Hungarian method of Kuhn (1955),
we choose to use the ILP formulation, as it enables us to have more freedom in
adding constraints to the problem, such as the lower and upper bounds.

###  3.2 M-step (Covariance Maximization)

The result of an A-step is an assignment $\pi$ such that $\pi(i)=j$ implies
$\mathbf{x}^{(i)}$ was deemed as aligned to $\mathbf{z}_{j}$. With that $\pi$
in mind, we define the following empirical covariance matrix
$\bm{\Omega}_{\pi}\in\mathbb{R}^{d\times d^{\prime}}$:

$\bm{\Omega}_{\pi}=\sum_{i=1}^{n}\mathbf{x}^{(i)}(\mathbf{z}^{(\pi(i))})^{\top}.$
(8)

We then apply SVD on $\bm{\Omega}_{\pi}$ to get new $\bm{U}$ and $\bm{V}$ that
are used in the next iteration of the algorithm with the A-step, if the
algorithm continues to run. When the maximal number of iterations is reached,
we follow the work of Shao et al. (2023) in using a truncated part of $\bm{U}$
to remove the information from the $x$s. We do that by projecting
$\mathbf{x}^{(i)}$ using the singular vectors of $\bm{U}$ with the smallest
singular values. These projected vectors co-vary the least with the guarded
attributes, assuming the assignment in the last A-step was precise. This
method has been shown by Shao et al. (2023) to be highly effective and
efficient in debiasing neural representations.

###  3.3 A Matrix Formulation of the AM Steps

Let $\mathbf{e}_{1},\ldots,\mathbf{e}_{m}$ be the standard basis vectors. This
means $\mathbf{e}_{i}$ is a vector of length $m$ with $0$ in all coordinates
except for the $i$th coordinate, where it is $1$.

Let $\mathcal{E}$ be the set of all matrices $\bm{E}$ where each
$\bm{E}\in\mathcal{E}$ is such that $\bm{E}\in\mathbb{R}^{n\times m}$ and each
row is one of $\mathbf{e}_{i}$, $i\in[m]$. In that case, $\bm{E}\bm{Z}^{\top}$
is an $n\times d^{\prime}$ matrix, such that the $j$th row is a copy of the
$i$th column of $\bm{Z}\in\mathbb{R}^{d^{\prime}\times n}$. Therefore, the AM
steps can be viewed as solving the following maximization problem using
coordinate ascent:

$\underset{\bm{E}\in\mathcal{E},\bm{U},\bm{V},\bm{\Sigma}}{\operatorname{argmax}}||\bm{U}^{\top}\bm{\Sigma}\bm{V}-\bm{X}\bm{E}\bm{Z}^{\top}||_{F}^{2},$
(9)

where $\bm{U}$, $\bm{V}$ are orthonormal matrices, and $\bm{\Sigma}$ is a
diagonal matrix with non-negative elements. This corresponds to the SVD of the
matrix $\bm{X}\bm{E}\bm{Z}^{\top}$.

In that case, the matrix $\bm{E}$ can be directly mapped to an assignment in
the form of $\pi$, where $\pi(i)$ would be the $j$ such that the $j$th
coordinate in the $i$th row of $\bm{E}$ is non-zero.

###  3.4 Removal Algorithm

The AM steps are best suited for the removal of information through SVD with
an algorithm such as SAL. This is because the AM steps are optimizing an
objective of the same type of SAL – relying on the projections $\bm{U}$ and
$\bm{V}$ to project the inputs and guarded representations into a joint space.
However, a by-product of the algorithm in Figure 2 is an assignment function
$\pi$ that aligns between the inputs and the guarded representations.

With that assignment, other removal algorithms can be used, for example, the
algorithm of Ravfogel et al. (2020). We experiment with this idea in §4.

###  3.5 Justification of the AM Steps

Next, we justify our algorithm (which may be skipped on the first reading).
Our justification is based on the observation that if indeed $\mathbf{X}$ and
$\mathbf{Z}$ are linked together (this connection is formalized as a latent
variable in their joint distribution), then for a given sample that is
permuted, the singular values of $\bm{\Omega}$ will be larger the closer the
permutation is to the identity permutation. This justifies finding such a
permutation that maximizes the singular values in an SVD of $\bm{\Omega}$.

#### More Details

Let $\iota\colon[n]\rightarrow[n]$ be the identity permutation, $\iota(i)=i$.
We will assume the case in which $n=m$ (but the justification can be
generalized to the case $m<n$), and that the underlying joint distribution
$p(\mathbf{X},\mathbf{Z})$ is mediated by a latent variable $\mathbf{H}$, such
that

$p(\mathbf{X},\mathbf{Z},\mathbf{H})=p(\mathbf{H})p(\mathbf{X}\mid\mathbf{H})p(\mathbf{Z}\mid\mathbf{H}).$
(10)

This implies there is a latent variable that connects $\mathbf{X}$ and
$\mathbf{Z}$, and that the joint distribution $p(\mathbf{X},\mathbf{Z})$ is a
mixture through $\mathbf{H}$.

######  Proposition 1 (informal).

Let $\\{(\mathbf{x}^{(i)},\mathbf{z}^{(i)})\\}$ be a sample of size $n$ from
the distribution in Eq. 10. Let $\pi$ be a permutation over $[n]$ uniformly
sampled from the set of permutations. Then with high likelihood, the sum of
the singular values of $\bm{\Omega}_{\pi}$ is smaller than the sum of singular
values under $\bm{\Omega}_{\iota}$.

For full details of this claim, see Appendix A.

##  4 Experiments

In our experiments, we test several combinations of algorithms. We use the
$k$-means (KMeans) as a substitute for the AM steps as a baseline for the
assignment step of $x$s to $z$s. In addition, for the removal step (once an
assignment has been identified), we test two algorithms: SAL (Shao et al.
2023; resulting in AMSAL) and INLP Ravfogel et al. (2020). We also compare
these two algorithms in _oracle_ mode (in which the assignment of guarded
attributes to inputs is known), to see the loss in performance that happens
due to noisy assignments from the AM or $k$-means algorithm (OracleSAL and
OracleINLP).

When running the AM algorithm or $k$-means, we execute it with three random
seeds (see also §4.6) for a maximum of a hundred iterations and choose the
projection matrix with the largest objective value over all seeds and
iterations. For the slack variables ($b_{0j}$ and $b_{1j}$ variables in Eq.
7), we use 20%-30% above and below the baseline of the guarded attribute
priors according to the training set. With the SAL methods, we remove the
number of directions according to the rank of the $\bm{\Omega}$ matrix
(between $2$ to $6$ in all experiments overall).

In addition, we experiment with a partially supervised assignment process, in
which a small seed dataset of aligned $x$s and $z$s is provided to the AM
steps. We use it for model selection: rather than choosing the assignment with
the highest SVD objective value, we choose the assignment with the highest
accuracy on this seed dataset. We refer to this setting as Partial (for
“partially supervised assignment”).

Finally, in the case of a gender-protected attribute, we compare our results
against a baseline in which the input $\mathbf{x}$ is compared against a list
of words stereotypically associated with the genders of male or
female.222https://tinyurl.com/33bzddtw Based on the overlap with these two
lists, we heuristically assign the gender label to $\mathbf{x}$ and then run
SAL or INLP (rather than using the AM algorithm). While this wordlist
heuristic is plausible in the case of gender, it is not as easy to derive in
the case of other protected attributes, such as age or race. We give the
results for this baseline using the marker WL in the corresponding tables.

#### Main Findings

Our overall main finding shows that our novel setting in which guarded
information is erased from individually-unaligned representations is viable.
We discovered that AM methods perform particularly well when dealing with more
complex bias removal scenarios, such as when multiple guarded attributes are
present. We also found that having similar priors for the guarded attributes
and downstream task labels may lead to poor performance on the task at hand.
In these cases, using a small amount of supervision often effectively helps
reduce bias while maintaining the utility of the representations for the main
classification of the regression problem. Finally, our analysis of alignment
stability shows that our AM algorithm often converges to suitable solutions
that align $\mathbf{X}$ with $\mathbf{Z}$.

Due to the unsupervised nature of our problem setting, we advise validating
the utility of our method in the following way. Once we run the AM algorithm,
we check whether there is a high-accuracy alignment between $\mathbf{X}$ and
$\mathbf{Y}$ (rather than $\mathbf{Z}$, which is unavailable). If this
alignment is accurate, then we run the risk of significantly damaging task
performance. An example is given in §4.5.

###  4.1 Word Embedding Debiasing

As a preliminary assessment of our setup and algorithms, we apply our methods
to GloVe word embeddings to remove gender bias, and following the previous
experiment settings of this problem Bolukbasi et al. (2016); Ravfogel et al.
(2020); Shao et al. (2023). We considered only the 150,000 most common words
to ensure the embedding quality and omitted the rest. We sort the remaining
embeddings by their projection on the
$\overrightarrow{\text{he}}$-$\overrightarrow{\text{she}}$ direction. Then we
consider the top 7,500 word embeddings as male-associated words ($z=1$) and
the bottom 7,500 as female-associated words ($z=-1$).

Our findings are that both the $k$-means and the AM algorithms perfectly
identify the alignment between the word embeddings and their associated gender
label (100%). Indeed, the dataset construction itself follows a natural
perfect clustering that these algorithms easily discover. Since the alignments
are perfectly identified, the results of predicting the gender from the word
embeddings after removal are identical to the oracle case. These results are
quite close to the results of a random guess, and we refer the reader to Shao
et al. (2023) for details on experiments with SAL and INLP for this dataset.
Considering Figure 3, it is evident that our algorithm essentially follows a
natural clustering of the word embeddings into two clusters, female and male,
as the embeddings are highly separable in this case. This is why the alignment
score of $\mathbf{X}$ (embedding) to $\mathbf{Z}$ (gender) is perfect in this
case. This finding indicates that this standard word embedding dataset used
for debiasing is _trivial to debias_ – debiasing can be done even without
knowing the identity of the stereotypical gender associated with each word.

(a) before debiasing

(b) after debiasing

Figure 3: A t-SNE visualization of the word embeddings before and after gender
information removal. In (a) we see the embeddings naturally cluster into the
corresponding gender.

Model | Task Acc. |  | TPR-GAP |
---|---|---|---|---
BertModel | 0.79 |  | 0.20 |
\+ AMINLP |  $\downarrow$0.12 0.67 |  |  $\downarrow$0.12 0.09 |
\+ Kmeans \+ INLP |  $\downarrow$0.11 0.68 |  |  $\downarrow$0.12 0.08 |
\+ OracleINLP |  $\downarrow$0.11 0.68 |  |  $\downarrow$0.12 0.08 |
\+ PartialINLP |  $\downarrow$0.12 0.67 |  |  $\downarrow$0.13 0.08 |
\+ AMSAL | 0.79 |  |  $\downarrow$0.02 0.18 |
\+ Kmeans \+ SAL | 0.79 |  |  $\downarrow$0.02 0.18 |
\+ OracleSAL | 0.79 |  |  $\downarrow$0.02 0.18 |
\+ PartialSAL | 0.79 |  |  $\downarrow$0.02 0.18 |
\+ WL \+ SAL | 0.79 |  |  $\downarrow$0.02 0.18 |
\+ WL \+ INLP |  $\downarrow$0.12 0.68 |  |  $\downarrow$0.12 0.08 |
FastText | 0.77 |  | 0.20 |
\+ AMINLP |  $\downarrow$0.05 0.73 |  |  $\uparrow$0.01 0.21 |
\+ Kmeans \+ INLP |  $\downarrow$0.08 0.69 |  | 0.19 |
\+ OracleINLP |  $\downarrow$0.03 0.74 |  |  $\downarrow$0.10 0.09 |
\+ PartialINLP |  $\downarrow$0.04 0.74 |  |  $\downarrow$0.04 0.16 |
\+ AMSAL |  $\downarrow$0.03 0.74 |  |  $\downarrow$0.03 0.17 |
\+ Kmeans \+ SAL |  $\downarrow$0.04 0.73 |  |  $\downarrow$0.02 0.17 |
\+ OracleSAL |  $\downarrow$0.01 0.76 |  |  $\downarrow$0.08 0.12 |
\+ PartialSAL |  $\downarrow$0.01 0.76 |  |  $\downarrow$0.02 0.18 |
\+ WL \+ SAL |  $\downarrow$0.01 0.76 |  |  $\downarrow$0.08 0.12 |
\+ WL \+ INLP |  $\downarrow$0.03 0.74 |  |  $\downarrow$0.10 0.10 |

Table 1: BiasBios dataset results. The top part uses BERT embeddings to encode
the biographies, while the bottom part uses FastText embeddings.

###  4.2 BiasBios Results

De-Arteaga et al. (2019) presented the BiasBios dataset, which consists of
self-provided biographies paired with the profession and gender of their
authors. A list of pronouns and names is used to obtain the authors’ gender
automatically. They aim to expose the caveats of automated hiring systems by
showing that even the simple task of predicting a candidate’s profession can
be affected by the candidate’s gender, which is encoded in the biography
representation. For example, we want to avoid one being identified as “he” or
“she” in their biography, affecting the likelihood of them being classified as
engineers or teachers.

We follow the setup of De-Arteaga et al. (2019), predicting a candidate’s
professions ($\mathbf{y}$), based on a self-provided short biography
($\mathbf{x}$), aiming to remove any information about the candidate’s gender
($\mathbf{z}$). Due to computational constraints, we use only random 30K
examples to learn the projections with both SAL and INLP (whether in the
unaligned or aligned setting). For the classification problem, we use the full
dataset. To get vector representations for the biographies, we use two
different encoders, FastText word embeddings Joulin et al. (2016), and BERT
Devlin et al. (2019). We stack a multi-class classifier on top of these
representations, as there are 28 different professions. We use 20% of the
training examples for the Partial setting. For BERT, we followed De-Arteaga et
al. (2019) in using the last CLS token state as the representation of the
whole biography. We used the BERT model bert-base-uncased.

#### Evaluation Measures

We use an extension of the True Positive Rate (TPR) gap, the root mean square
(RMS) TPR gap of all classes, for evaluating bias in a multiclass setting.
This metric was suggested by De-Arteaga et al. (2019), who demonstrated it is
significantly correlated with gender imbalances, which often lead to unfair
classification. The higher the metric value is, the bigger the gap between the
two categories (for example, between male and female) for the specific main
task prediction. For the profession classification, we report accuracy.

#### Results

Table 1 provides the results for the biography dataset. We see that INLP
significantly reduces the TPR-GAP in all settings, but this comes at a cost:
the representations are significantly less useful for the main task of
predicting the profession. When inspecting the alignments, we observe that
their accuracy is quite high with BERT: 100% with $k$-means, 85% with the AM
algorithm and 99% with Partial AM. FastText results are lower, hovering around
55% for all three methods. The high BERT assignment performance indicates that
the BiasBios BERT representations are naturally separated by gender. We also
observe that the results of WL+SAL and WL+INLP are correspondingly identical
to Oracle+SAL and Oracle+INLP. This comes as no surprise, as the gender label
is derived from a similar word list, which enables the WL approach to get a
nearly perfect alignment (over 96% agreement with the gender label).

| Model | Stt. Score
---|---
Gender |
BERT | 57.25
\+ AM \+ INLP |  $\uparrow$0.38 57.63
\+ Kmeans \+ INLP |  $\downarrow$3.81 53.44
\+ OracleINLP |  $\downarrow$4.58 52.67
\+ PartialINLP |  $\downarrow$4.58 52.67
\+ AMSAL |  $\downarrow$3.05 54.20
\+ Kmeans \+ SAL |  $\downarrow$2.29 54.96
\+ OracleSAL |  $\downarrow$5.72 51.53
\+ PartialSAL |  $\downarrow$5.72 51.53
ALBERT | 48.09
\+ AM \+ INLP |  $\uparrow$1.14 46.95
\+ Kmeans \+ INLP |  $\uparrow$0.38 47.71
\+ OracleINLP |  $\uparrow$4.58 43.51
\+ PartialINLP |  $\uparrow$4.20 43.89
\+ AMSAL |  $\uparrow$0.76 47.33
\+ Kmeans \+ SAL |  $\downarrow$0.38 48.47
\+ OracleSAL |  $\uparrow$0.76 47.33
\+ PartialSAL |  $\uparrow$0.76 47.33
RoBERTa | 60.15
\+ AM \+ INLP |  $\downarrow$3.45 56.70
\+ Kmeans \+ INLP |  $\downarrow$7.66 52.49
\+ OracleINLP |  $\downarrow$4.98 55.17
\+ PartialINLP |  $\downarrow$4.98 55.17
\+ AMSAL |  $\downarrow$3.45 56.70
\+ Kmeans \+ SAL |  $\downarrow$3.83 56.32
\+ OracleSAL |  $\downarrow$8.81 48.66
\+ PartialSAL |  $\downarrow$8.81 48.66
GPT-2 | 56.87
\+ AM \+ INLP |  $\downarrow$6.11 50.76
\+ Kmeans \+ INLP |  $\downarrow$2.67 54.20
\+ OracleINLP |  $\downarrow$6.49 50.38
\+ PartialINLP |  $\downarrow$6.11 50.76
\+ AMSAL |  $\uparrow$1.15 58.02
\+ Kmeans \+ SAL |  $\downarrow$3.05 53.82
\+ OracleSAL | 56.87
\+ PartialSAL | 56.87

|  | Model | Stt. Score
---|---
Race |
BERT | 62.33
\+ AM \+ INLP |  $\downarrow$1.75 60.58
\+ Kmeans \+ INLP |  $\uparrow$4.85 67.18
\+ OracleINLP |  $\uparrow$5.63 67.96
\+ PartialINLP |  $\uparrow$5.63 67.96
\+ AMSAL |  $\uparrow$0.19 62.52
\+ Kmeans \+ SAL |  $\uparrow$0.19 62.52
\+ OracleSAL |  $\uparrow$0.78 63.11
\+ PartialSAL |  $\uparrow$0.78 63.11
ALBERT | 62.52
\+ AM \+ INLP |  $\uparrow$0.98 36.50
\+ Kmeans \+ INLP |  $\uparrow$3.50 33.98
\+ OracleINLP |  $\downarrow$7.18 55.34
\+ PartialINLP |  $\downarrow$7.18 55.34
\+ AMSAL |  $\downarrow$5.82 43.30
\+ Kmeans \+ SAL |  $\downarrow$6.40 43.88
\+ OracleSAL |  $\downarrow$3.69 41.17
\+ PartialSAL |  $\downarrow$3.69 41.17
RoBERTa | 63.57
\+ AM \+ INLP |  $\downarrow$9.31 45.74
\+ Kmeans \+ INLP |  $\downarrow$6.79 43.22
\+ OracleINLP |  $\downarrow$1.75 61.82
\+ PartialINLP |  $\downarrow$1.75 61.82
\+ AMSAL |  $\uparrow$1.74 65.31
\+ Kmeans \+ SAL |  $\uparrow$1.74 65.31
\+ OracleSAL |  $\uparrow$3.48 67.05
\+ PartialSAL |  $\uparrow$3.48 67.05
GPT-2 | 59.69
\+ AM \+ INLP |  $\downarrow$3.88 55.81
\+ Kmeans \+ INLP |  $\downarrow$1.16 58.53
\+ OracleINLP |  $\uparrow$0.19 59.88
\+ PartialINLP | 59.69
\+ AMSAL |  $\downarrow$4.65 55.04
\+ Kmeans \+ SAL |  $\downarrow$5.43 45.74
\+ OracleSAL |  $\downarrow$4.85 54.84
\+ PartialSAL |  $\downarrow$4.85 54.84

|  | Model | Stt. Score
---|---
Religion |
BERT | 62.86
\+ AM \+ INLP |  $\uparrow$0.95 63.81
\+ Kmeans \+ INLP |  $\uparrow$4.76 67.62
\+ OracleINLP |  $\downarrow$1.91 60.95
\+ PartialINLP |  $\downarrow$1.91 60.95
\+ AMSAL |  $\uparrow$1.90 64.76
\+ Kmeans \+ SAL |  $\uparrow$1.90 64.76
\+ OracleSAL |  $\uparrow$4.76 67.62
\+ PartialSAL |  $\uparrow$4.76 67.62
ALBERT | 60.00
\+ AM \+ INLP |  $\downarrow$0.95 59.05
\+ Kmeans \+ INLP |  $\uparrow$9.52 69.52
\+ OracleINLP |  $\downarrow$2.86 57.14
\+ PartialINLP |  $\downarrow$2.86 57.14
\+ AMSAL |  $\uparrow$10.48 70.48
\+ Kmeans \+ SAL |  $\uparrow$9.52 69.52
\+ OracleSAL |  $\uparrow$6.67 66.67
\+ PartialSAL |  $\uparrow$6.67 66.67
RoBERTa | 60.95
\+ AM \+ INLP |  $\uparrow$4.76 65.71
\+ Kmeans \+ INLP |  $\uparrow$2.86 63.81
\+ OracleINLP |  $\uparrow$1.91 62.86
\+ PartialINLP |  $\uparrow$1.91 62.86
\+ AMSAL |  $\uparrow$12.38 73.33
\+ Kmeans \+ SAL |  $\uparrow$12.38 73.33
\+ OracleSAL |  $\uparrow$10.48 71.43
\+ PartialSAL |  $\uparrow$10.48 71.43
GPT-2 | 61.90
\+ AM \+ INLP | 61.90
\+ Kmeans \+ INLP | 61.90
\+ OracleINLP | 61.90
\+ PartialINLP | 61.90
\+ AMSAL |  $\uparrow$3.81 65.71
\+ Kmeans \+ SAL |  $\downarrow$1.90 60.00
\+ OracleSAL |  $\uparrow$15.24 77.14
\+ PartialSAL |  $\uparrow$15.24 77.14

(a) |  | (b) |  | (c)
Table 2: (a) CrowS-Pairs Gender stereotype scores (Stt. score) in language
models debiased by different debiasing techniques and assignment; (b) CrowS-
Pairs Race stereotype scores; (c) CrowS-Pairs Religion stereotype scores. All
models are deemed least biased if the stereotype score is 50%. The colored
numbers are calculated as $|\,|b-50|-|s-50|\,|$ where $b$ is the top row score
and $s$ is the corresponding system score.

Model | S. Score (%) | LM Score (%)
---|---|---
BERT | 60.28 | 84.17
\+ AM \+ INLP |  $\downarrow$1.14 59.14 |  $\downarrow$0.43 83.75
\+ Kmeans \+ INLP |  $\downarrow$0.16 60.12 |  $\downarrow$0.47 83.70
\+ OracleINLP |  $\downarrow$2.93 57.35 |  $\downarrow$1.07 83.11
\+ PartialINLP |  $\downarrow$2.93 57.35 |  $\downarrow$1.07 83.10
\+ AMSAL |  $\uparrow$0.61 60.89 |  $\uparrow$0.09 84.26
\+ Kmeans \+ SAL |  $\uparrow$0.19 60.47 |  $\uparrow$0.13 84.30
\+ OracleSAL |  $\downarrow$0.83 59.44 |  $\uparrow$0.53 84.70
\+ PartialSAL |  $\downarrow$0.83 59.44 |  $\uparrow$0.53 84.70
ALBERT | 59.93 | 89.77
\+ AM \+ INLP |  $\downarrow$0.29 59.64 |  $\downarrow$1.45 88.32
\+ Kmeans \+ INLP |  $\downarrow$0.59 59.34 |  $\downarrow$0.08 89.69
\+ OracleINLP |  $\downarrow$2.73 57.20 |  $\downarrow$1.59 88.17
\+ PartialINLP |  $\downarrow$2.72 57.21 |  $\downarrow$1.62 88.15
\+ AMSAL |  $\downarrow$0.22 59.71 |  $\downarrow$0.32 89.45
\+ Kmeans \+ SAL |  $\uparrow$0.56 60.49 |  $\downarrow$0.10 89.67
\+ OracleSAL |  $\downarrow$2.18 57.75 |  $\downarrow$0.16 89.61
\+ PartialSAL |  $\downarrow$2.18 57.75 |  $\downarrow$0.16 89.61
RoBERTa | 66.32 | 88.95
\+ AM \+ INLP |  $\downarrow$4.95 61.37 |  $\uparrow$0.04 88.99
\+ Kmeans \+ INLP |  $\downarrow$2.20 64.13 |  $\downarrow$1.47 87.48
\+ OracleINLP |  $\downarrow$3.82 62.51 |  $\downarrow$0.92 88.03
\+ PartialINLP |  $\downarrow$3.82 62.51 |  $\downarrow$0.91 88.04
\+ AMSAL |  $\downarrow$0.63 65.70 |  $\uparrow$0.60 89.54
\+ Kmeans \+ SAL |  $\downarrow$0.49 65.83 |  $\uparrow$0.46 89.41
\+ OracleSAL |  $\downarrow$3.32 63.00 |  $\uparrow$0.40 89.35
\+ PartialSAL |  $\downarrow$3.32 63.00 |  $\uparrow$0.40 89.35
GPT-2 | 62.65 | 91.01
\+ AM \+ INLP |  $\downarrow$1.65 61.00 |  $\downarrow$3.77 87.24
\+ Kmeans \+ INLP |  $\downarrow$1.57 61.08 |  $\downarrow$3.09 87.93
\+ OracleINLP |  $\downarrow$1.26 61.39 | 91.01
\+ PartialINLP |  $\downarrow$1.26 61.39 | 91.01
\+ AMSAL |  $\downarrow$1.58 61.07 |  $\downarrow$0.23 90.79
\+ Kmeans \+ SAL |  $\downarrow$4.00 58.64 |  $\downarrow$0.60 90.41
\+ OracleSAL |  $\downarrow$4.55 58.09 |  $\downarrow$1.75 89.26
\+ PartialSAL |  $\downarrow$4.55 58.09 |  $\downarrow$1.75 89.26

Table 3: StereoSet stereotype scores (Stt. Score) and language modeling scores
(LM Score) for the gender category. Stereotype scores indicate the least bias
at 50%, and the LM scores indicate high usability at 100%.

###  4.3 BiasBench Results

Meade et al. (2022) followed an empirical study of an array of datasets in the
context of debiasing. They analyzed different methods and tasks, and we follow
their benchmark evaluation to assess our AMSAL algorithm and other methods in
the context of our new setting. We include a short description of the datasets
we use in this section. We include full results in Appendix B, with a
description of other datasets. We also encourage the reader to refer to Meade
et al. (2022) for details on this benchmark. We use 20% of the training
examples for the Partial setting.

#### StereoSet Nadeem et al. (2021)

This dataset presents a word completion test for a language model, where the
completion can be stereotypical or non-stereotypical. The bias is then
measured by calculating how often a model prefers the stereotypical completion
over the non-stereotypical one. Nadeem et al. (2021) introduced the language
model score to measure the language model usability, which is the percentage
of examples for which a model prefers the stereotypical or non-stereotypical
word over some unrelated word.

#### CrowS-Pairs Nangia et al. (2020)

This dataset includes pairs of sentences that are minimally different at the
token level, but these differences lead to the sentence being either
stereotypical or anti-stereotypical. The assessment measures how many times a
language model prefers the stereotypical element in a pair over the anti-
stereotypical element.

#### Results

We start with an assessment of the BERT model for the CrowS-Pairs gender, race
and religion bias evaluation (Table 2). We observe that all approaches for
gender, except AM+INLP reduce the stereotype score. Race and religion are more
difficult to debias in the case of BERT. INLP with $k$-means works best when
no seed alignment data is provided at all, but when we consider PartialSAL, in
which we use the alignment algorithm with some seed aligned data, we see that
the results are the strongest. When we consider the RoBERTa model, the results
are similar, with PartialSAL significantly reducing the bias. Our findings
from Table 2 overall indicate that the ability to debias a representation
_highly depends on the model that generates the representation_. In Table 10
we observe that the representations, on average, are not damaged for most GLUE
tasks.

As Meade et al. (2022) have noted, when changing the representations of a
language model to remove bias, we might cause such adjustments that damage the
usability of the language model. To test which methods possibly cause such an
issue, we also assess the language model score on the StereoSet dataset in
Table 3. We overall see that often SAL-based methods give lower stereotype
score, while INLP methods more significantly damage the language model score.
This implies that the _SAL-based methods remove bias effectively while less
significantly harming the usability of the language model representations_.

We also conducted comprehensive results for other datasets (SEAT and GLUE) and
categories of bias (based on race and religion). The results, especially for
GLUE, demonstrate the effectiveness of our method of unaligned information
removal. For GLUE, we consistently retain the baseline task performance almost
in full. See Appendix B.

###  4.4 Multiple-Guarded Attribute Sentiment

Figure 4: Accuracy of the AM steps with respect to age and gender separately
(on unseen data), as a function of the fraction of the labeled dataset used by
the AM algorithm.

We hypothesize that AM-based methods are better suited for setups where
multiple guarded attributes should be removed, as they allow us to target
several guarded attributes with different priors. To examine our hypothesis,
we experiment with a dataset curated from Twitter (tweets encoded using BERT,
bert-base-uncased), in which users are surveyed for their age and gender
Cachola et al. (2018). We bucket the age into three groups (0-25, 26-50 and
above 50). Tweets in this dataset are annotated with their sentiment, ranging
from one (very negative) to five (very positive). The dataset consists of more
than 6,400 tweets written by more than 1,700 users. We removed users that no
longer have public Twitter accounts and users with locations that do not exist
based on a filter,333We used a list of cities, counties and states in the
United States, taken from https://tinyurl.com/4kmc6pyn. All users were in the
United States when the data was collected by the original curators. resulting
in a dataset with over 3,000 tweets, written by 817 unique users. As tweets
are short by nature and their number is relatively small, the debiasing signal
in this dataset, the amount of information it contains about the guarded
attributes, might not be sufficient for the attribute removal. To amplify this
signal, we concatenated each tweet in the dataset to at most ten other tweets
from the same user.

We study the relationship between the main task of sentiment detection and the
two protected attributes of age and gender. As a protected attribute
$\mathbf{z}$, we use the combination of both age and gender as a binary one-
hot vector. This dataset presents a use-case for our algorithm of a composed
protected attribute. Rather than using a classifier for predicting the
sentiment, we use linear regression. Following Cachola et al. (2018), we use
Mean Absolute Error (MAE) to report the error of the sentiment predictions.
Given that the sentiment is predicted as a continuous value, we cannot use the
TPR gap as in previous sections. Rather, we use the following formula:

$\mathrm{MAEGap}=\mathrm{std}(\mathrm{MAD}_{z=j}\mid j\in[m]),$ (11)

where $\mathrm{MAD}_{z=j}=\frac{1}{\ell}\sum_{i}|\eta_{ij}-\mu_{j}|$ where $i$
ranges over the set of size $\ell$ of examples with protected attribute value
$j$, $\mu_{j}$ is the average of absolute $\mathbf{Y}$ prediction error for
that set and $\eta_{ij}$ is the absolute difference between $\mu_{j}$ and the
absolute error of example $i$.444The absolute error of prediction $a$ with
true value $b$ is $|a-b|$. The function $\mathrm{std}$ in this case indicates
the standard deviation of the $m$ values of $\mathrm{MAD}_{z=j}$, $j\in[m]$.

#### Results

Table 4 presents our results. Overall, AMSAL reduces the gender and age gap in
the predictions while not increasing by much MAE. In addition, we can see both
AM-based methods outperform their $k$-means counterparts which increase
unfairness (Kmeans \+ INLP) or significantly harm the downstream-task
performance (Kmeans \+ SAL). We also consider Figure 4, which shows the
quality of the assignments of the AM algorithm change as a function of the
labeled data used. As expected, the more labeled data we have, the more
accurate the assignments are, but the differences are not very large.

Model |  | MAE |  | Age (gap) |  | Gender (gap)
---|---|---|---|---|---|---
BertModel |  | 0.745 |  | 0.031 |  | 0.011
\+ AM + INLP |  |  $\downarrow$0.027 0.717 |  | 0.031 |  |  $\downarrow$0.008 0.003
\+ Kmeans \+ INLP |  |  $\downarrow$0.052 0.693 |  |  $\downarrow$0.001 0.030 |  |  $\uparrow$0.010 0.021
\+ OracleINLP |  |  $\downarrow$0.022 0.723 |  |  $\downarrow$0.008 0.022 |  |  $\uparrow$0.005 0.017
\+ PartialINLP |  |  $\downarrow$0.025 0.719 |  |  $\uparrow$0.007 0.038 |  | 0.011
\+ AMSAL |  |  $\uparrow$0.009 0.754 |  |  $\downarrow$0.005 0.026 |  |  $\downarrow$0.009 0.002
\+ Kmeans \+ SAL |  |  $\uparrow$0.039 0.783 |  |  $\downarrow$0.001 0.030 |  |  $\downarrow$0.007 0.004
\+ OracleSAL |  |  $\uparrow$0.012 0.757 |  |  $\downarrow$0.002 0.029 |  |  $\downarrow$0.009 0.003
\+ PartialSAL |  |  $\uparrow$0.025 0.769 |  |  $\downarrow$0.001 0.030 |  |  $\downarrow$0.005 0.006

Table 4: MAE and debiasing gap values on the Twitter dataset, when using BERT to encode the tweets. For age and gender, we give the MAE gap as in Eq. 11. Figure 5: Accuracy of the AM steps (in identifying the correct assignment of inputs to guarded information) as a function of the iteration number. Shaded gray gives upper and lower bound on the standard deviation over five runs with different seeds for the initial $\pi$. FastText refers to the BiasBios dataset, the BERT models are for the CrowS-Pairs dataset and Emb. refers to the word embeddings dataset from §4.1. Figure 6: Ratio of the objective value in iteration $t$ and iteration $0$ of the ILP for the AM steps as a function of the iteration number $t$. Shaded gray gives upper and lower bound on the standard deviation over five runs with different seeds for the initial $\pi$. See legend explanation in Table 5. | Model | Task Acc. | TPR-GAP
---|---|---
deepmoji | 0.77 | 0.14
\+ AM + INLP | 0.77 | 0.14
\+ Kmeans \+ INLP | 0.77 | 0.14
\+ OracleINLP |  $\downarrow$0.02 0.74 |  $\downarrow$0.04 0.10
\+ PartialINLP |  $\downarrow$0.01 0.75 |  $\downarrow$0.06 0.08
\+ AMSAL |  $\downarrow$0.24 0.52 |  $\uparrow$0.03 0.17
\+ Kmeans \+ SAL |  $\downarrow$0.23 0.54 |  $\uparrow$0.12 0.26
\+ OracleSAL | 0.76 |  $\downarrow$0.03 0.11
\+ PartialSAL |  $\downarrow$0.19 0.57 |  $\uparrow$0.15 0.29

|  Model $F_{1}$ (macro) TPR-GAP deepmoji 0.66 0.06 \+ AM + INLP
$\downarrow$0.0002 0.66 0.06 \+ Kmeans \+ INLP $\downarrow$0.1 0.56
$\downarrow$0.02 0.04 \+ OracleINLP $\downarrow$0.19 0.46 $\downarrow$0.06
0.00 \+ PartialINLP $\downarrow$0.14 0.52 $\downarrow$0.03 0.03 \+ AMSAL
$\downarrow$0.16 0.49 $\downarrow$0.02 0.04 \+ Kmeans \+ SAL $\downarrow$0.17
0.48 $\downarrow$0.04 0.02 \+ OracleSAL $\downarrow$0.01 0.65 $\uparrow$0.03
0.09 \+ PartialSAL $\downarrow$0.11 0.54 0.06
Table 5: The performance of removing race information from the DeepMoji
dataset is shown for two cases: with balanced ratios of race and sentiment
(left) and with ratios of 0.8 for sentiment and 0.5 for race (right). In both
cases, the total size of the dataset used is 30,000 examples. To evaluate the
performance of the unbalanced sentiment dataset, we use the $F_{1}$ macro
measure, because in an unbalanced dataset such as this one, a simple
classifier that always returns one label will achieve an accuracy of 80%. Such
a classifier would have a $F_{1}$ macro score of $0.44\dot{4}$.

###  4.5 An Example of Our Method Limitations

We now present the main limitation in our approach and setting. This
limitation arises when the random variables $\mathbf{Y}$ and $\mathbf{Z}$ are
not easily distinguishable through information about $\mathbf{X}$.

We experiment with a binary sentiment analysis ($\mathbf{y}$) task, predicted
on users’ tweets ($\mathbf{x}$), aiming to remove information regarding the
authors’ ethnic affiliations. To do so, we use a dataset collected by Blodgett
et al. (2016), which examined the differences between African-American English
(AAE) speakers and Standard American English (SAE) speakers. As information
about one’s ethnicity is hard to obtain, the user’s geolocation information
was used to create a distantly supervised mapping between authors and their
ethnic affiliations. We follow previous work Shao et al. (2023); Ravfogel et
al. (2020) and use the DeepMoji encoder Felbo et al. (2017) to obtain
representations for the tweets. The train and test sets are balanced regarding
sentiment and authors’ ethnicity. We use 20% of the examples for the Partial
setting. Table 5 gives the results for this dataset. We observe that the
removal with the assignment ($k$-means, AM or Partial) significantly harms the
performance on the main task and reduces it to a random guess.

This presents a limitation of our algorithm. A priori, there is no distinction
between $\mathbf{Y}$ and $\mathbf{Z}$, as our method is unsupervised. In
addition, the positive labels of $\mathbf{Y}$ and $\mathbf{Z}$ have the same
prior probability. Indeed, when we check the assignment accuracy in the
sentiment dataset, we observe that the $k$-means, AM and Partial AM assignment
accuracy for identifying $\mathbf{Z}$ are between 0.55 and 0.59. If we check
the assignment against $\mathbf{Y}$, we get an accuracy between 0.74 and 0.76.
This means that all assignment algorithms actually identify $\mathbf{Y}$
rather than $\mathbf{Z}$ (both $\mathbf{Y}$ and $\mathbf{Z}$ are binary
variables in this case). The conclusion from this is that our algorithm works
best when sufficient information on $\mathbf{Z}$ is presented such that it can
provide a basis for aligning samples of $\mathbf{Z}$ with samples of
$\mathbf{X}$. Suppose such information is unavailable or unidentifiable with
information regarding $\mathbf{Y}$. In that case, we may simply identify the
natural clustering of $\mathbf{X}$ according to their main task classes,
leading to low main-task performance.

In Table 5, we observe that this behavior is significantly mitigated when the
priors over the sentiment and the race are different (0.8 for sentiment and
0.5 for race). In that case, the AM algorithm is able to distinguish between
the race-protected attribute ($\mathbf{z}$) and the sentiment class
($\mathbf{y}$) quite consistently with INLP and SAL, and the gap is reduced.

We also observe that INLP changed neither the accuracy nor the TPR-GAP for the
balanced scenario (Table 5) when using a $k$-means assignment or an AM
assignment. Upon inspection, we found out that INLP returns an identity
projection in these cases, unable to amplify the relatively weak signal in the
assignment to change the representations.

###  4.6 Stability Analysis of the Alignment

In Figure 5, we plot the accuracy of the alignment algorithm (knowing the true
value of the guarded attribute per input) throughout the execution of the AM
steps for the first ten iterations. The shaded area indicates one standard
deviation. We observe that the first few iterations are the ones in which the
accuracy improves the most. For most of the datasets, the accuracy does not
decrease between iterations, though in the case of DeepMoji we do observe a
“bump.” This is indeed why the Partial setting of our algorithm, where a small
amount of guarded information is available to determine at which iteration to
stop the AM algorithm, is important. In the word embeddings case, the variance
is larger because, in certain executions, the algorithm converged quickly,
while in others, it took more iterations to converge to high accuracy.

Figure 6 plots the relative change of the objective value of the ILP from §3.1
against iteration number. The relative change is defined as the ratio between
the objective value before the algorithm begins and the same value at a given
iteration. We see that there is a relative stability of the algorithm and that
the AM steps converge quite quickly. We also observe the DeepMoji dataset has
a large increase in the objective value in the first iteration (around $\times
5$ compared to the value the algorithm starts with), after which it remains
stable.

##  5 Related Work

There has been an increasing amount of work about detecting and erasing
undesired or protected information from neural representations, with standard
software packages for this process having been developed Han et al. (2022).
For example, in their seminal work, Bolukbasi et al. (2016) showed that word
embeddings exhibit gender stereotypes. To mitigate this issue, they projected
the word embeddings to a neutral space with respect to a “he-she” direction.
Influenced by this work, Zhao et al. (2018) proposed a customized training
scheme to reduce the gender bias in word embeddings. Gonen and Goldberg (2019)
examined the effectiveness of the methods mentioned above and concluded they
remove bias in a shallow way. For example, they demonstrated that classifiers
can accurately predict the gender associated with a word when fed with the
embeddings of both debiasing methods.

Another related strand of work uses adversarial learning Ganin et al. (2016),
where an additional objective function is added for balancing undesired-
information removal and the main task Edwards and Storkey (2016); Li et al.
(2018); Coavoux et al. (2018); Wang et al. (2021). Elazar and Goldberg (2018)
have also demonstrated that an ad-hoc classifier can easily recover the
removed information from adversarially trained representations. Since then,
methods for information erasure such as INLP and its generalization Ravfogel
et al. (2020, 2022), SAL Shao et al. (2023) and methods based on similarity
measures between neural representations Colombo et al. (2022) have been
developed. With a similar motivation to ours, Han et al. (2021b) aimed to ease
the burden of obtaining guarded attributes at a large scale by decoupling the
adversarial information removal process from the main task training. They,
however, did not experiment with debiasing representations where no guarded
attribute alignments are available. Shao et al. (2023) experimented with the
removal of features in a scenario in which a low number of protected
attributes is available.

Additional previous work showed that methods based on causal inference Feder
et al. (2021), train-set balancing Han et al. (2021a), and contrastive
learning Shen et al. (2021); Chi et al. (2022) effectively reduce bias and
increase fairness. In addition, there is a large body of work for detecting
bias, its evaluation (dev-etal-2021-oscar) and its implications in specific
NLP applications. Savoldi et al. (2022) detected a gender bias in speech
translation systems for gendered languages. Gender bias is also discussed in
the context of knowledge base embeddings by Fisher et al. (2019); Du et al.
(2022), and multilingual text classification Huang (2022).

##  6 Conclusions and Future Work

We presented a new and challenging setup for removing information, with
minimal or no available sensitive information alignment. This setup is crucial
for the wide applicability of debiasing methods, as for most applications,
obtaining such sensitive labels on a large scale is challenging. To ease this
problem, we present a method to erase information from neural representations,
where the guarded attribute information does not accompany each input
instance. Our main algorithm, AMSAL, alternates between two steps (Assignment
and Maximization) to identify an assignment between the input instances and
the guarded information records. It then completes its execution by removing
the information by minimizing covariance between the input instances and the
aligned guarded attributes. Our approach is modular, and other erasure
algorithms, such as INLP, can be used with it. Experiments show that we can
reduce the unwanted bias in many cases while keeping the representations
highly useful. Future work might include extending our technique to the
kernelized case, analogously to the method of Shao et al. (2023).

## Ethical Considerations

The AM algorithm could potentially be misused by rather than using the AM
steps to erase information, using them to link records of two different types,
undermining the privacy of the record holders. Such a situation may merit
additional concern because the links returned between the guarded attributes
and the input instances will likely contain mistakes. The links are unreliable
for decision-making at the _individual level_. Instead, they should be used on
an aggregate as a statistical construct to erase information from the input
representations. Finally,555We thank the anonymous reviewer for raising this
issue. we note that the automation of the debiasing process, without properly
statistically confirming its accuracy using a correct sample may promote a
false sense of security that a given system is making fair decisions. We do
not recommend using our method for debiasing without proper statistical
control and empirical verification of correctness.

## Acknowledgments

We thank the reviewers, the action editors and Marcio Fonseca for their
thorough feedback. We also thank Daniel Preotiuc-Pietro for his help with the
Twitter data. We thank Kousha Etessami for being a sounding board for certain
parts of the paper. The experiments in this paper were supported by compute
grants from the Edinburgh Parallel Computing Center and from the Baskerville
Tier 2 HPC service (University of Birmingham).

## References

  * Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016.  Demographic dialectal variation in social media: A case study of African-American English.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1119–1130, Austin, Texas. Association for Computational Linguistics.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_ , pages 4349–4357.
  * Cachola et al. (2018) Isabel Cachola, Eric Holgate, Daniel Preoţiuc-Pietro, and Junyi Jessy Li. 2018\.  Expressively vulgar: The socio-dynamics of vulgarity and its effects on sentiment analysis in social media.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2927–2938, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Caliskan et al. (2017) Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ , 356(6334):183–186.
  * Chi et al. (2022) Jianfeng Chi, William Shand, Yaodong Yu, Kai-Wei Chang, Han Zhao, and Yuan Tian. 2022.  Conditional supervised contrastive learning for fair text classification.  _ArXiv preprint_ , abs/2205.11485.
  * Coavoux et al. (2018) Maximin Coavoux, Shashi Narayan, and Shay B. Cohen. 2018.  Privacy-preserving neural representations of text.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1–10, Brussels, Belgium. Association for Computational Linguistics.
  * Colombo et al. (2022) Pierre Colombo, Guillaume Staerman, Nathan Noiry, and Pablo Piantanida. 2022.  Learning disentangled textual representations via statistical measures of similarity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2614–2630, Dublin, Ireland. Association for Computational Linguistics.
  * De-Arteaga et al. (2019) Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019.  Bias in bios: A case study of semantic representation bias in a high-stakes setting.  In _proceedings of the Conference on Fairness, Accountability, and Transparency_ , pages 120–128.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Du et al. (2022) Yupei Du, Qi Zheng, Yuanbin Wu, Man Lan, Yan Yang, and Meirong Ma. 2022.  Understanding gender bias in knowledge base embeddings.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1381–1395, Dublin, Ireland. Association for Computational Linguistics.
  * Edwards and Storkey (2016) Harrison Edwards and Amos J. Storkey. 2016.  Censoring representations with an adversary.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Elazar and Goldberg (2018) Yanai Elazar and Yoav Goldberg. 2018.  Adversarial removal of demographic attributes from text data.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 11–21, Brussels, Belgium. Association for Computational Linguistics.
  * Feder et al. (2021) Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021.  CausaLM: Causal model explanation through counterfactual language models.  _Computational Linguistics_ , 47(2):333–386.
  * Felbo et al. (2017) Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017\.  Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1615–1625, Copenhagen, Denmark. Association for Computational Linguistics.
  * Fisher et al. (2019) Joseph Fisher, Dave Palfrey, Christos Christodoulopoulos, and Arpit Mittal. 2019\.  Measuring social bias in knowledge graph embeddings.  _ArXiv preprint_ , abs/1912.02761.
  * Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016\.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030.
  * Gonen and Goldberg (2019) Hila Gonen and Yoav Goldberg. 2019.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 609–614, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Han et al. (2021a) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021a.  Balancing out bias: Achieving fairness through training reweighting.  _ArXiv preprint_ , abs/2109.08253.
  * Han et al. (2021b) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021b.  Decoupling adversarial training for fair NLP.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 471–477, Online. Association for Computational Linguistics.
  * Han et al. (2022) Xudong Han, Aili Shen, Yitong Li, Lea Frermann, Timothy Baldwin, and Trevor Cohn. 2022.  fairlib: A unified framework for assessing and improving classification fairness.  _ArXiv preprint_ , abs/2205.01876.
  * Huang (2022) Xiaolei Huang. 2022.  Easy adaptation to mitigate gender bias in multilingual text classification.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 717–723, Seattle, United States. Association for Computational Linguistics.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016.  Fasttext. zip: Compressing text classification models.  _ArXiv preprint_ , abs/1612.03651.
  * Kuhn (1955) Harold W. Kuhn. 1955.  The hungarian method for the assignment problem.  _Naval Research Logistics Quarterly_ , 2:83–97.
  * Li et al. (2018) Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018.  Towards robust and privacy-preserving text representations.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 25–30, Melbourne, Australia. Association for Computational Linguistics.
  * MacKay (2003) David J C MacKay. 2003.  _Information theory, inference and learning algorithms_.  Cambridge university press.
  * May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019\.  On measuring social biases in sentence encoders.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 622–628, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Meade et al. (2022) Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022.  An empirical survey of the effectiveness of debiasing techniques for pre-trained language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1878–1898, Dublin, Ireland. Association for Computational Linguistics.
  * Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021.  StereoSet: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5356–5371, Online. Association for Computational Linguistics.
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020.  CrowS-pairs: A challenge dataset for measuring social biases in masked language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1953–1967, Online. Association for Computational Linguistics.
  * Ramshaw and Tarjan (2012) Lyle Ramshaw and Robert E Tarjan. 2012.  On minimum-cost assignments in unbalanced bipartite graphs.  _HP Labs, Palo Alto, CA, USA, Tech. Rep. HPL-2012-40R1_.
  * Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020\.  Null it out: Guarding protected attributes by iterative nullspace projection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7237–7256, Online. Association for Computational Linguistics.
  * Ravfogel et al. (2022) Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. 2022.  Linear adversarial concept erasure.  In _International Conference on Machine Learning_ , pages 18400–18421. PMLR.
  * Savoldi et al. (2022) Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2022.  Under the morphosyntactic lens: A multifaceted evaluation of gender bias in speech translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1807–1824, Dublin, Ireland. Association for Computational Linguistics.
  * Shao et al. (2023) Shun Shao, Yftah Ziser, and Shay B Cohen. 2023.  Gold doesn’t always glitter: Spectral removal of linear and nonlinear guarded attribute information.  In _Proceedings of the 17th Annual Meeting of the European chapter of the Association for Computational Linguistics (EACL)_ , volume abs/2203.07893.
  * Shen et al. (2021) Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, and Lea Frermann. 2021.  Contrastive learning for fair representations.  _ArXiv preprint_ , abs/2109.10645.
  * Stewart (1990) Gilbert W Stewart. 1990.  Perturbation theory for the singular value decomposition.  Technical Report UMIACS-90-120 / CS-TR 2539, University of Maryland, College Park.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Wang et al. (2021) Liwen Wang, Yuanmeng Yan, Keqing He, Yanan Wu, and Weiran Xu. 2021.  Dynamically disentangling social bias from task-oriented representations with adversarial attack.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3740–3750, Online. Association for Computational Linguistics.
  * Zhao et al. (2018) Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018.  Learning gender-neutral word embeddings.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4847–4853, Brussels, Belgium. Association for Computational Linguistics.

##  Appendix A Justification of the AM Algorithm: Further Details

We provide here the full details for the claim in §3.5. Our first observation
is that for a uniformly sampled permutation $\pi\colon[n]\rightarrow[n]$, the
probability that it has exactly $k\leq n$ elements such that $\pi(i)=i$ for
all $i$ in this set of elements is bounded from above by:666Choose $k$
elements that are fixed, and let the rest vary arbitrarily.

$\displaystyle\frac{\displaystyle{\binom{n}{k}}(n-k)!}{n!}=\displaystyle\frac{1}{k!}.$
(12)

We also assume that $\mathbb{E}[\mathbf{X}\mid\mathbf{H}]=0$ and
$\mathbb{E}[\mathbf{Z}\mid\mathbf{H}]=0$, and that the product of every pair
of coordinates of $\mathbf{X}$ and $\mathbf{Z}$ is bounded in absolute value
by a constant $B>0$. Let
$\\{(\mathbf{x}^{(i)},\mathbf{z}^{(i)},\mathbf{h}^{(i)})\\}$ be a random
sample of size $n$ from the joint distribution
$p(\mathbf{X},\mathbf{Z},\mathbf{H})$. Given a permutation
$\pi\colon[n]\rightarrow[n]$, define $I(\pi)=\\{i\mid\pi(i)=i\\}$. For a given
set $M\subseteq[n]$, define

$\bm{\Omega}_{\pi\mid M}=\sum_{i\in
M}\mathbf{x}^{(i)}(\mathbf{z}^{(\pi(i))})^{\top}.$ (13)

For a matrix $\bm{A}\in\mathbb{R}^{d\times d^{\prime}}$, let
$\sigma_{j}(\bm{A})$ be its $j$th largest singular value, and let
$\sigma^{+}(\bm{A})=\sum_{j}\sigma_{j}(\bm{A})$. Let
$\sigma^{+}=\sigma^{+}(\mathbb{E}[\bm{\Omega}_{\iota}])$.

We first note that for any permutation $\pi$, it holds that
$\mathbb{E}[\Omega_{\pi\mid K}]=0$ where we define $K=[n]\setminus I(\pi)$.

######  Lemma 1.

For any $t>0$, it holds that:

$p(||\bm{\Omega}_{\pi\mid I(\pi)}-\mathbb{E}[\bm{\Omega}_{\pi\mid
I(\pi)}]||_{2}\geq dd^{\prime}t)$ (14)

is smaller than
$2dd^{\prime}\exp\left(-\displaystyle\frac{t^{2}}{|I(\pi)|B^{2}}\right).$

###### Proof.

By Hoeffding’s inequality, for any $i\in[d]$, $j\in[d^{\prime}]$, it holds
that the probability that for $|I(\pi)|$ i.i.d. r.v. $\mathbf{X}^{k}$,
$\mathbf{Z}^{k}$ the following is true:

$\displaystyle\left|\sum_{k\in
I(\pi)}\textnormal{X}^{k}_{i}\textnormal{Z}^{k}_{j}-\sum_{k\in
I(\pi)}\mathbb{E}[\textnormal{X}^{k}_{i}\textnormal{Z}^{k}_{j}]\right|\geq t$
(15)

is smaller than $2\exp\left(-\displaystyle\frac{t^{2}}{|I(\pi)|B^{2}}\right).$
Therefore, by a union bound on each element of the matrix $\Omega_{\pi}$, we
get the upper bound on Eq. 14.

∎

######  Lemma 2.

For any $t>0$, it holds that:

$||\bm{\Omega}_{\pi\mid K}-\mathbb{E}[\bm{\Omega}_{\pi\mid K}]||_{2}$ (16)

is smaller than $2|K|dd^{\prime}B$.

###### Proof.

Since $\textnormal{X}_{i}$ and $\textnormal{Z}_{j}$ are bounded as a product
in absolute value by $B$, and the dimensions of $\bm{\Omega}_{\pi\mid K}$ is
$d\times d^{\prime}$, each cell being a sum of $|K|$ values, the bound
naturally follows. ∎

Let $n$ such that $n\sigma^{+}>2kdd^{\prime}B$ where $k=|K|$. Then from Lemma
2, $||\bm{\Omega}_{\pi\mid K}-\mathbb{E}[\bm{\Omega}_{\pi\mid
K}]||_{2}<n\sigma^{+}$. Consider the event
$\sigma^{+}(\bm{\Omega}_{\iota})<\sigma^{+}(\bm{\Omega}_{\pi})$. Its
probability is bounded from above by the probability of the event
$\sigma^{+}(\bm{\Omega}_{\iota})\leq n\sigma^{+}$ OR
$\sigma^{+}(\bm{\Omega}_{\pi})\geq n\sigma^{+}$ (for any $n$ as the above).
Due to the inequality of Weyl (Theorem 1 in Stewart 1990; see below), the fact
that $\bm{\Omega}_{\pi}=\bm{\Omega}_{\pi\mid K}+\bm{\Omega}_{\pi\mid I(\pi)}$,
Lemma 1, and the fact that $n-k\leq n$, the probability of this OR event is
bounded from above by
$4dd^{\prime}\exp\left(-\displaystyle\frac{(n-k)(\sigma^{+})^{2}}{(dd^{\prime}B)^{2}}\right)$.

The conclusion from this is that if we were to sample uniformly a permutation
$\pi$ from the set of permutations over $[n]$, then with quite high likelihood
(because the fraction of elements that are preserved under $\pi$ becomes
smaller as $n$ becomes larger), the sum of the singular values of
$\bm{\Omega}_{\pi}$ under this permutation will be smaller than the sum of the
singular values of $\bm{\Omega}_{\iota}$ – meaning, when the $x$s and the $z$s
are correctly aligned. This justifies our objective of aligning the $x$s and
the $z$s with an objective that maximizes the singular values, following
Proposition 1.

#### Inequality of Weyl (1912)

As mentioned by Stewart (1990), the following holds:

######  Lemma 3.

Let $\bm{A}$ and $\bm{E}$ be two matrices, and let
$\bm{\tilde{A}}=\bm{A}+\bm{E}$. Let $\sigma_{i}$ be the $i$th singular value
of $\bm{A}$ and $\tilde{\sigma}_{i}$ be the $i$th singular value of
$\bm{\tilde{A}}$. Then $|\sigma_{i}-\tilde{\sigma}_{i}|\leq||\bm{E}||_{2}$.

##  Appendix B Comprehensive Results on the BiasBench Datasets

We include more results for the SEAT dataset from BiasBench and for the CrowS-
Pairs dataset and StereoSet datasets for bias categories other than gender. A
description of the SEAT and GLUE datasets (with metrics used) follows.

#### SEAT May et al. (2019)

SEAT is a sentence-level extension of WEAT Caliskan et al. (2017), which is an
association test between two categories of words: attribute word sets and
target word sets. For example, attribute words for gender bias could be { _he_
, _man_ }, while a target words could be { _career_ , _office_ }. For example,
an attribute word set (in case of gender bias) could be a set of words such as
{ _he_ , _him_ , _man_ }, while a target word set might be words related to
office work. If we see a high association between an attribute word set and a
target word set, we may claim that a particular gender bias is encoded. The
final evaluation is calculated by measuring the similarity between the
different attributes and target word sets. To extend WEAT to a sentence-level
test, Caliskan et al. (2017) incorporated the WEAT attribute and target words
into synthetic sentence templates.

We use an _effect size_ metric to report our results for SEAT. This measure is
a normalized difference between cosine similarity of representations of the
attribute words and the target words. Both attribute words and target words
are split into two categories (for example, in relation to gender), so the
difference is based on four terms, between each pair of each category set of
words (target and attribute). An effect size closer to zero indicates less
bias is encoded in the representations.

#### GLUE Wang et al. (2019)

We follow Meade et al. (2022) and use the GLUE dataset to test the debiased
model on an array of downstream tasks to validate their usability. GLUE is a
highly popular benchmark for testing NLP models, containing a variety of
tasks, such as classification tasks (e.g., sentiment analysis), similarity
tasks (e.g., paraphrase identification), and inference tasks (e.g., question-
answering).

The following tables of results are included:

  * •

Table 6 presents the StereoSet results for removing the race (a) and religion
(b) guarded attributes.

  * •

Tables 7, 8, and 9 describe the SEAT effect sizes for the gender, race, and
religion cases, respectively.

  * •

Table 10 presents the scores the debiased representations achieve for the GLUE
benchmark.

Model S. Score (%) LM Score (%) Race BERT 57.03 84.17 \+ AM \+ INLP $\uparrow$1.23 58.26 $\downarrow$0.65 83.53 \+ Kmeans \+ INLP $\uparrow$0.32 57.35 $\downarrow$0.63 83.54 \+ OracleINLP $\uparrow$0.33 57.36 $\downarrow$1.05 83.12 \+ PartialINLP $\uparrow$0.33 57.36 $\downarrow$1.05 83.12 \+ AMSAL $\uparrow$1.98 59.01 $\uparrow$0.55 84.72 \+ Kmeans \+ SAL $\uparrow$2.10 59.13 $\uparrow$0.49 84.66 \+ OracleSAL $\uparrow$1.85 58.88 $\uparrow$0.76 84.93 \+ PartialSAL $\uparrow$1.85 58.88 $\uparrow$0.76 84.93 ALBERT 57.57 89.77 \+ AM \+ INLP $\downarrow$0.94 56.63 $\downarrow$2.25 87.52 \+ Kmeans \+ INLP $\uparrow$1.42 59.00 $\downarrow$2.04 87.72 \+ OracleINLP $\downarrow$2.54 55.04 $\downarrow$1.95 87.82 \+ PartialINLP $\downarrow$2.54 55.04 $\downarrow$1.97 87.80 \+ AMSAL $\downarrow$0.84 56.73 $\uparrow$0.09 89.86 \+ Kmeans \+ SAL $\downarrow$1.31 56.27 $\uparrow$0.06 89.82 \+ OracleSAL $\downarrow$0.31 57.26 $\uparrow$0.75 90.52 \+ PartialSAL $\downarrow$0.31 57.26 $\uparrow$0.75 90.52 GPT-2 58.83 91.01 \+ AM \+ INLP $\downarrow$0.90 57.93 $\downarrow$5.55 85.47 \+ Kmeans \+ INLP $\uparrow$0.10 58.93 $\downarrow$0.94 90.07 \+ OracleINLP $\uparrow$0.21 59.04 $\uparrow$0.04 91.06 \+ PartialINLP $\uparrow$0.21 59.04 $\uparrow$0.03 91.05 \+ AMSAL $\downarrow$3.15 55.69 $\downarrow$0.43 90.59 \+ Kmeans \+ SAL $\downarrow$2.65 56.18 $\uparrow$0.08 91.09 \+ OracleSAL $\downarrow$3.09 55.75 $\downarrow$2.09 88.92 \+ PartialSAL $\downarrow$3.09 55.75 $\downarrow$2.09 88.92 RoBERTa 61.67 88.95 \+ AM \+ INLP $\downarrow$7.31 54.37 $\downarrow$3.06 85.88 \+ Kmeans \+ INLP $\downarrow$5.67 56.00 $\downarrow$3.95 85.00 \+ OracleINLP $\downarrow$3.42 58.26 $\uparrow$0.01 88.96 \+ PartialINLP $\downarrow$3.42 58.26 $\uparrow$0.02 88.96 \+ AMSAL $\uparrow$0.74 62.41 $\uparrow$1.02 89.96 \+ Kmeans \+ SAL $\uparrow$0.71 62.39 $\uparrow$1.03 89.97 \+ OracleSAL $\uparrow$1.79 63.47 $\uparrow$0.49 89.44 \+ PartialSAL $\uparrow$1.79 63.47 $\uparrow$0.49 89.44 |  | Model | S. Score (%) | LM Score (%)
---|---|---
Religion |  |
BERT | 59.70 | 84.17
\+ AM \+ INLP |  $\uparrow$3.22 62.92 |  $\downarrow$0.60 83.58
\+ Kmeans \+ INLP |  $\uparrow$1.67 61.38 |  $\downarrow$0.28 83.89
\+ OracleINLP |  $\uparrow$0.61 60.31 |  $\downarrow$0.82 83.35
\+ PartialINLP |  $\uparrow$0.61 60.31 |  $\downarrow$0.82 83.35
\+ AMSAL |  $\uparrow$1.42 61.12 |  $\uparrow$0.60 84.77
\+ Kmeans \+ SAL |  $\uparrow$1.83 61.53 |  $\uparrow$0.62 84.79
\+ OracleSAL |  $\uparrow$0.09 59.79 |  $\uparrow$0.68 84.85
\+ PartialSAL |  $\uparrow$0.09 59.79 |  $\uparrow$0.68 84.85
ALBERT | 60.32 | 89.77
\+ AM \+ INLP |  $\uparrow$1.85 62.17 |  $\downarrow$1.20 88.57
\+ Kmeans \+ INLP |  $\uparrow$2.78 63.10 |  $\downarrow$1.41 88.36
\+ OracleINLP |  $\uparrow$3.45 63.77 |  $\downarrow$0.91 88.86
\+ PartialINLP |  $\uparrow$3.45 63.77 |  $\downarrow$0.91 88.86
\+ AMSAL |  $\downarrow$0.78 59.54 |  $\uparrow$0.39 90.15
\+ Kmeans \+ SAL |  $\downarrow$0.39 59.94 |  $\uparrow$0.36 90.13
\+ OracleSAL |  $\downarrow$1.18 59.14 |  $\uparrow$1.26 91.02
\+ PartialSAL |  $\downarrow$1.18 59.14 |  $\uparrow$1.26 91.02
GPT-2 | 63.26 | 91.01
\+ AM \+ INLP |  $\downarrow$1.90 61.36 |  $\downarrow$6.57 84.44
\+ Kmeans \+ INLP |  $\uparrow$1.66 64.92 |  $\downarrow$0.88 90.14
\+ OracleINLP |  $\uparrow$0.69 63.95 |  $\uparrow$0.19 91.21
\+ PartialINLP |  $\uparrow$0.69 63.95 |  $\uparrow$0.19 91.21
\+ AMSAL |  $\downarrow$4.83 58.43 |  $\downarrow$0.60 90.41
\+ Kmeans \+ SAL |  $\downarrow$2.57 60.69 |  $\downarrow$0.41 90.60
\+ OracleSAL |  $\downarrow$5.48 57.78 |  $\downarrow$3.91 87.10
\+ PartialSAL |  $\downarrow$5.48 57.78 |  $\downarrow$3.91 87.10
RoBERTa | 64.28 | 88.95
\+ AM \+ INLP |  $\downarrow$3.84 60.44 |  $\downarrow$4.09 84.86
\+ Kmeans \+ INLP |  $\downarrow$1.37 62.91 |  $\downarrow$2.82 86.13
\+ OracleINLP |  $\downarrow$3.94 60.34 |  $\downarrow$0.83 88.12
\+ PartialINLP |  $\downarrow$3.94 60.34 |  $\downarrow$0.84 88.11
\+ AMSAL |  $\downarrow$1.64 62.64 |  $\uparrow$1.00 89.95
\+ Kmeans \+ SAL |  $\downarrow$2.04 62.24 |  $\uparrow$0.99 89.93
\+ OracleSAL |  $\downarrow$1.92 62.36 |  $\uparrow$1.11 90.06
\+ PartialSAL |  $\downarrow$1.92 62.36 |  $\uparrow$1.11 90.06

(a) | (b)
Table 6: (a) StereoSet stereotype scores and language modeling scores (LM
Score) for race debiased BERT, ALBERT, RoBERTa, and GPT-2 models. Stereotype
scores are least biased at 50% and the LM Scores are best at 100%; (b)
StereoSet stereotype scores and language modeling scores (LM Score) for
religion debiased BERT, ALBERT, RoBERTa, and GPT-2 models. Stereotype scores
are least biased at 50% and the LM Scores are best at 100%.

Model | SEAT6 | SEAT6b | SEAT7 | SEAT7b | SEAT8 | SEAT8b | Avg. Effect Size
---|---|---|---|---|---|---|---
BERT |  0.931 ∗ | 0.090 | -0.124 |  0.937 ∗ |  0.783 ∗ |  0.858 ∗ | 0.620
\+ AM \+ INLP |  0.744 ∗ | -0.006 | 0.036 |  0.968 ∗ |  0.828 ∗ |  0.849 ∗ |  $\downarrow$0.049 0.572
\+ Kmeans \+ INLP |  0.809 ∗ | 0.013 | -0.084 |  0.812 ∗ |  0.756 ∗ |  0.785 ∗ |  $\downarrow$0.077 0.543
\+ OracleINLP | 0.269 | -0.339 | -0.403 |  0.437 ∗ | 0.399 | 0.289 |  $\downarrow$0.264 0.356
\+ PartialINLP | 0.269 | -0.338 | -0.404 |  0.436 ∗ | 0.399 | 0.289 |  $\downarrow$0.265 0.356
\+ AMSAL |  0.928 ∗ | 0.110 | -0.191 |  0.717 ∗ |  0.756 ∗ |  0.756 ∗ |  $\downarrow$0.044 0.576
\+ Kmeans \+ SAL |  0.925 ∗ | 0.109 | -0.190 |  0.722 ∗ |  0.752 ∗ |  0.752 ∗ |  $\downarrow$0.046 0.575
\+ OracleSAL | 0.387 | -0.301 | -0.876 | -0.192 | 0.299 | 0.309 |  $\downarrow$0.227 0.394
\+ PartialSAL | 0.387 | -0.301 | -0.876 | -0.192 | 0.299 | 0.309 |  $\downarrow$0.227 0.394
ALBERT |  0.637 ∗ | 0.151 |  0.487 ∗ |  0.956 ∗ |  0.683 ∗ |  0.823 ∗ | 0.623
\+ AM \+ INLP |  0.620 ∗ | 0.165 |  0.408 ∗ |  0.854 ∗ |  0.649 ∗ |  0.744 ∗ |  $\downarrow$0.049 0.573
\+ Kmeans \+ INLP |  0.645 ∗ | 0.147 |  0.408 ∗ |  0.822 ∗ |  0.660 ∗ |  0.829 ∗ |  $\downarrow$0.038 0.585
\+ OracleINLP |  0.464 ∗ | -0.084 | -0.222 |  0.467 ∗ | 0.215 |  0.462 ∗ |  $\downarrow$0.304 0.319
\+ PartialINLP |  0.464 ∗ | -0.084 | -0.222 |  0.467 ∗ | 0.215 |  0.462 ∗ |  $\downarrow$0.304 0.319
\+ AMSAL |  0.640 ∗ | 0.138 |  0.477 ∗ |  0.933 ∗ |  0.666 ∗ |  0.820 ∗ |  $\downarrow$0.010 0.612
\+ Kmeans \+ SAL |  0.640 ∗ | 0.136 |  0.474 ∗ |  0.933 ∗ |  0.664 ∗ |  0.818 ∗ |  $\downarrow$0.012 0.611
\+ OracleSAL |  0.468 ∗ | -0.067 | -0.230 | 0.312 | 0.305 |  0.545 ∗ |  $\downarrow$0.302 0.321
\+ PartialSAL |  0.468 ∗ | -0.067 | -0.230 | 0.312 | 0.305 |  0.545 ∗ |  $\downarrow$0.302 0.321
RoBERTa |  0.922 ∗ | 0.208 |  0.979 ∗ |  1.460 ∗ |  0.810 ∗ |  1.261 ∗ | 0.940
\+ AM \+ INLP |  0.982 ∗ | 0.262 |  0.845 ∗ |  1.575 ∗ |  0.840 ∗ |  1.395 ∗ |  $\uparrow$0.043 0.983
\+ Kmeans \+ INLP |  0.933 ∗ | 0.238 |  1.090 ∗ |  1.595 ∗ |  1.148 ∗ |  1.435 ∗ |  $\uparrow$0.133 1.073
\+ OracleINLP |  0.781 ∗ | 0.014 |  0.651 ∗ |  1.281 ∗ |  0.708 ∗ |  1.160 ∗ |  $\downarrow$0.174 0.766
\+ PartialINLP |  0.782 ∗ | 0.014 |  0.651 ∗ |  1.282 ∗ |  0.708 ∗ |  1.161 ∗ |  $\downarrow$0.174 0.766
\+ AMSAL |  0.902 ∗ | 0.187 |  1.021 ∗ |  1.549 ∗ |  0.893 ∗ |  1.386 ∗ |  $\uparrow$0.050 0.990
\+ Kmeans \+ SAL |  0.920 ∗ | 0.182 |  1.017 ∗ |  1.549 ∗ |  0.885 ∗ |  1.389 ∗ |  $\uparrow$0.051 0.990
\+ OracleSAL |  0.695 ∗ | -0.014 |  0.550 ∗ |  1.315 ∗ |  0.684 ∗ |  1.170 ∗ |  $\downarrow$0.202 0.738
\+ PartialSAL |  0.695 ∗ | -0.014 |  0.550 ∗ |  1.315 ∗ |  0.684 ∗ |  1.170 ∗ |  $\downarrow$0.202 0.738
GPT-2 | 0.138 | 0.003 | -0.023 | 0.002 | -0.224 | -0.287 | 0.113
\+ AM \+ INLP | 0.141 | 0.009 | -0.017 | 0.010 | -0.213 | -0.283 |  $\downarrow$0.001 0.112
\+ Kmeans \+ INLP | 0.138 | 0.006 | -0.024 | 0.002 | -0.223 | -0.287 | 0.113
\+ OracleINLP | 0.130 | -0.005 | -0.024 | 0.000 | -0.229 | -0.291 | 0.113
\+ PartialINLP | 0.130 | -0.005 | -0.024 | 0.000 | -0.229 | -0.291 | 0.113
\+ AMSAL | 0.280 | 0.199 |  0.900 ∗ | 0.352 | 0.408 | 0.118 |  $\uparrow$0.263 0.376
\+ Kmeans \+ SAL | 0.298 | -0.304 | 0.030 | 0.044 | 0.114 | -0.096 |  $\uparrow$0.035 0.148
\+ OracleSAL | 0.155 | -0.379 | -0.093 | -0.059 | -0.039 | -0.173 |  $\uparrow$0.037 0.150
\+ PartialSAL | 0.155 | -0.379 | -0.093 | -0.059 | -0.039 | -0.173 |  $\uparrow$0.037 0.150
Table 7: SEAT effect sizes for gender-debiased representations of BERT,
ALBERT, RoBERTa, and GPT-2 models. Effect sizes closer to 0 are indicative of
less biased model representations. Statistically significant effect sizes at
$p<0.01$ are denoted by *. The final column reports the average absolute
effect size across all six gender SEAT tests for each debiased model.

Model | ABW-1 | ABW-2 | SEAT-3 | SEAT-3b | SEAT-4 | SEAT-5 | SEAT-5b | Avg. Effect Size
---|---|---|---|---|---|---|---|---
BERT | -0.079 |  0.690 ∗ |  0.778 ∗ |  0.469 ∗ |  0.901 ∗ |  0.887 ∗ |  0.539 ∗ | 0.620
\+ AM \+ INLP | 0.155 |  0.583 ∗ |  0.769 ∗ |  0.341 ∗ |  0.889 ∗ |  0.937 ∗ |  0.403 ∗ |  $\downarrow$0.038 0.582
\+ Kmeans \+ INLP | 0.097 |  0.590 ∗ |  0.775 ∗ |  0.381 ∗ |  0.882 ∗ |  0.888 ∗ |  0.357 ∗ |  $\downarrow$0.053 0.567
\+ OracleINLP | 0.295 |  0.565 ∗ |  0.799 ∗ |  0.369 ∗ |  0.977 ∗ |  1.039 ∗ |  0.432 ∗ |  $\uparrow$0.019 0.639
\+ PartialINLP | 0.295 |  0.565 ∗ |  0.799 ∗ |  0.370 ∗ |  0.976 ∗ |  1.039 ∗ |  0.432 ∗ |  $\uparrow$0.019 0.639
\+ AMSAL | 0.138 |  0.621 ∗ |  0.797 ∗ |  0.374 ∗ |  0.911 ∗ |  1.015 ∗ |  0.435 ∗ |  $\downarrow$0.007 0.613
\+ Kmeans \+ SAL | 0.131 |  0.624 ∗ |  0.798 ∗ |  0.373 ∗ |  0.912 ∗ |  1.013 ∗ |  0.436 ∗ |  $\downarrow$0.008 0.612
\+ OracleSAL | -0.021 |  0.643 ∗ |  0.788 ∗ |  0.357 ∗ |  0.885 ∗ |  0.894 ∗ |  0.431 ∗ |  $\downarrow$0.046 0.574
\+ PartialSAL | -0.021 |  0.643 ∗ |  0.788 ∗ |  0.357 ∗ |  0.885 ∗ |  0.894 ∗ |  0.431 ∗ |  $\downarrow$0.046 0.574
ALBERT | -0.014 | 0.410 |  1.132 ∗ | -0.252 |  0.956 ∗ |  1.041 ∗ | 0.058 | 0.552
\+ AM \+ INLP | -0.150 |  0.505 ∗ |  1.149 ∗ | -0.244 |  0.982 ∗ |  1.075 ∗ | -0.036 |  $\uparrow$0.040 0.592
\+ Kmeans \+ INLP | -0.015 |  0.484 ∗ |  1.162 ∗ | -0.228 |  0.988 ∗ |  1.067 ∗ | -0.033 |  $\uparrow$0.017 0.568
\+ OracleINLP | 0.040 |  0.534 ∗ |  1.165 ∗ | -0.150 |  0.996 ∗ |  1.116 ∗ | 0.021 |  $\uparrow$0.023 0.574
\+ PartialINLP | 0.040 |  0.534 ∗ |  1.165 ∗ | -0.150 |  0.996 ∗ |  1.116 ∗ | 0.021 |  $\uparrow$0.023 0.574
\+ AMSAL | 0.283 |  0.471 ∗ |  0.985 ∗ | -0.299 |  0.802 ∗ |  0.938 ∗ | -0.063 |  $\downarrow$0.003 0.549
\+ Kmeans \+ SAL | 0.292 |  0.472 ∗ |  0.980 ∗ | -0.294 |  0.799 ∗ |  0.935 ∗ | -0.060 |  $\downarrow$0.004 0.547
\+ OracleSAL |  0.300 ∗ |  0.471 ∗ |  0.994 ∗ | -0.281 |  0.813 ∗ |  0.949 ∗ | -0.089 |  $\uparrow$0.005 0.557
\+ PartialSAL |  0.300 ∗ |  0.471 ∗ |  0.994 ∗ | -0.281 |  0.813 ∗ |  0.949 ∗ | -0.089 |  $\uparrow$0.005 0.557
RoBERTa |  0.395 ∗ | 0.159 | -0.114 | -0.003 | -0.315 |  0.780 ∗ |  0.386 ∗ | 0.307
\+ AM \+ INLP | 0.257 |  0.534 ∗ |  0.381 ∗ | 0.138 | 0.202 |  0.646 ∗ |  0.300 ∗ |  $\uparrow$0.044 0.351
\+ Kmeans \+ INLP | 0.270 |  0.466 ∗ |  0.242 ∗ | 0.116 | 0.079 |  0.627 ∗ |  0.310 ∗ |  $\downarrow$0.006 0.301
\+ OracleINLP | 0.222 | 0.445 |  0.354 ∗ | 0.130 | 0.125 |  0.636 ∗ |  0.301 ∗ |  $\uparrow$0.009 0.316
\+ PartialINLP | 0.222 | 0.445 |  0.354 ∗ | 0.130 | 0.125 |  0.636 ∗ |  0.301 ∗ |  $\uparrow$0.009 0.316
\+ AMSAL |  0.317 ∗ |  0.520 ∗ |  0.471 ∗ | 0.211 |  0.314 ∗ |  0.576 ∗ |  0.275 ∗ |  $\uparrow$0.076 0.384
\+ Kmeans \+ SAL |  0.314 ∗ |  0.522 ∗ |  0.476 ∗ | 0.212 |  0.320 ∗ |  0.579 ∗ |  0.276 ∗ |  $\uparrow$0.078 0.385
\+ OracleSAL | 0.170 | 0.220 |  0.401 ∗ | 0.227 | 0.225 |  0.471 ∗ |  0.268 ∗ |  $\downarrow$0.024 0.283
\+ PartialSAL | 0.170 | 0.220 |  0.401 ∗ | 0.227 | 0.225 |  0.471 ∗ |  0.268 ∗ |  $\downarrow$0.024 0.283
GPT-2 |  1.060 ∗ | -0.200 |  0.431 ∗ |  0.243 ∗ | 0.133 |  0.696 ∗ |  0.370 ∗ | 0.448
\+ AM \+ INLP |  1.046 ∗ | -0.169 |  0.472 ∗ |  0.257 ∗ | 0.172 |  0.686 ∗ |  0.366 ∗ |  $\uparrow$0.005 0.452
\+ Kmeans \+ INLP |  1.059 ∗ | -0.189 |  0.440 ∗ |  0.248 ∗ | 0.141 |  0.695 ∗ |  0.373 ∗ |  $\uparrow$0.002 0.449
\+ OracleINLP |  1.060 ∗ | -0.200 |  0.433 ∗ |  0.246 ∗ | 0.135 |  0.693 ∗ |  0.364 ∗ | 0.447
\+ PartialINLP |  1.060 ∗ | -0.200 |  0.433 ∗ |  0.246 ∗ | 0.135 |  0.693 ∗ |  0.364 ∗ | 0.447
\+ AMSAL | -0.525 | 0.157 |  0.654 ∗ |  0.398 ∗ |  0.327 ∗ | 0.120 | 0.117 |  $\downarrow$0.119 0.328
\+ Kmeans \+ SAL | -0.459 | 0.179 |  1.088 ∗ |  0.499 ∗ |  0.827 ∗ |  0.594 ∗ |  0.271 ∗ |  $\uparrow$0.112 0.560
\+ OracleSAL | 0.183 | 0.101 |  1.095 ∗ |  0.515 ∗ |  0.836 ∗ |  0.817 ∗ |  0.347 ∗ |  $\uparrow$0.109 0.556
\+ PartialSAL | 0.183 | 0.101 |  1.095 ∗ |  0.515 ∗ |  0.836 ∗ |  0.817 ∗ |  0.347 ∗ |  $\uparrow$0.109 0.556
Table 8: SEAT effect sizes for race debiased BERT, ALBERT, RoBERTa, and GPT-2
models. Effect sizes closer to 0 are indicative of less biased model
representations. Statistically significant effect sizes at $p<0.01$ are
denoted by *. The final column reports the average absolute effect size across
all six gender SEAT tests for each debiased model.

Model | Religion-1 | Religion-1b | Religion-2 | Religion-2b | Avg. Effect Size
---|---|---|---|---|---
BERT | 0.744 ∗ | -0.067 | 1.009 ∗ | -0.147 | 0.492
\+ AM \+ INLP | 0.530 ∗ | -0.184 | 0.847 ∗ | -0.156 |  $\downarrow$0.062 0.429
\+ Kmeans \+ INLP | 0.574 ∗ | -0.253 | 0.919 ∗ | -0.308 |  $\uparrow$0.022 0.514
\+ OracleINLP | 0.473 ∗ | -0.301 | 0.787 ∗ | -0.280 |  $\downarrow$0.031 0.460
\+ PartialINLP | 0.473 ∗ | -0.301 | 0.787 ∗ | -0.280 |  $\downarrow$0.031 0.460
\+ AMSAL | 0.683 ∗ | -0.117 | 0.941 ∗ | -0.178 |  $\downarrow$0.012 0.480
\+ Kmeans \+ SAL | 0.686 ∗ | -0.112 | 0.938 ∗ | -0.180 |  $\downarrow$0.013 0.479
\+ OracleSAL | 0.735 ∗ | -0.036 | 0.884 ∗ | -0.156 |  $\downarrow$0.039 0.453
\+ PartialSAL | 0.735 ∗ | -0.036 | 0.884 ∗ | -0.156 |  $\downarrow$0.039 0.453
ALBERT | 0.203 | -0.117 | 0.848 ∗ | 0.555 ∗ | 0.431
\+ AM \+ INLP | 0.208 | -0.065 | 0.891 ∗ | 0.557 ∗ |  $\downarrow$0.001 0.430
\+ Kmeans \+ INLP | 0.126 | -0.138 | 0.839 ∗ | 0.518 ∗ |  $\downarrow$0.025 0.405
\+ OracleINLP | 0.206 | -0.110 | 0.727 ∗ | 0.385 ∗ |  $\downarrow$0.074 0.357
\+ PartialINLP | 0.206 | -0.110 | 0.727 ∗ | 0.385 ∗ |  $\downarrow$0.074 0.357
\+ AMSAL | 0.024 | -0.256 | 0.722 ∗ | 0.418 ∗ |  $\downarrow$0.076 0.355
\+ Kmeans \+ SAL | 0.027 | -0.253 | 0.722 ∗ | 0.415 ∗ |  $\downarrow$0.077 0.354
\+ OracleSAL | 0.116 | -0.168 | 0.585 ∗ | 0.289 |  $\downarrow$0.141 0.289
\+ PartialSAL | 0.116 | -0.168 | 0.585 ∗ | 0.289 |  $\downarrow$0.141 0.289
RoBERTa | 0.132 | 0.018 | -0.191 | -0.166 | 0.127
\+ AM \+ INLP | -0.042 | -0.203 | -0.255 | -0.273 |  $\uparrow$0.067 0.193
\+ Kmeans \+ INLP | -0.014 | -0.204 | -0.187 | -0.304 |  $\uparrow$0.051 0.177
\+ OracleINLP | -0.309 | -0.347 | -0.191 | -0.135 |  $\uparrow$0.119 0.246
\+ PartialINLP | -0.309 | -0.347 | -0.191 | -0.135 |  $\uparrow$0.119 0.246
\+ AMSAL | -0.169 | -0.228 | -0.014 | 0.009 |  $\downarrow$0.022 0.105
\+ Kmeans \+ SAL | -0.172 | -0.231 | -0.011 | 0.011 |  $\downarrow$0.020 0.106
\+ OracleSAL | -0.063 | -0.208 | -0.203 | -0.109 |  $\uparrow$0.019 0.146
\+ PartialSAL | -0.063 | -0.208 | -0.203 | -0.109 |  $\uparrow$0.019 0.146
GPT-2 | -0.332 | -0.271 | 0.617 ∗ | 0.286 | 0.376
\+ AM \+ INLP | -0.326 | -0.264 | 0.671 ∗ | 0.333 |  $\uparrow$0.022 0.399
\+ Kmeans \+ INLP | -0.331 | -0.271 | 0.624 ∗ | 0.297 |  $\uparrow$0.004 0.380
\+ OracleINLP | -0.331 | -0.271 | 0.615 ∗ | 0.284 |  $\downarrow$0.001 0.375
\+ PartialINLP | -0.331 | -0.271 | 0.615 ∗ | 0.284 |  $\downarrow$0.001 0.375
\+ AMSAL | 0.087 | 0.064 | 0.767 ∗ | 0.341 |  $\downarrow$0.062 0.315
\+ Kmeans \+ SAL | 0.274 | 0.300 | 1.014 ∗ | 0.534 ∗ |  $\uparrow$0.154 0.531
\+ OracleSAL | -0.101 | -0.089 | 1.144 ∗ | 0.779 ∗ |  $\uparrow$0.152 0.528
\+ PartialSAL | -0.101 | -0.089 | 1.144 ∗ | 0.779 ∗ |  $\uparrow$0.152 0.528
Table 9: SEAT effect sizes for religion debiased BERT, ALBERT, RoBERTa, and
GPT-2 models. Effect sizes closer to 0 are indicative of less biased model
representations. Statistically significant effect sizes at $p<0.01$ are
denoted by *. The final column reports the average absolute effect size across
all six gender SEAT tests for each debiased model.

Model | cola | mnli | mrpc | qnli | qqp | rte | sst2 | stsb | wnli | Average
---|---|---|---|---|---|---|---|---|---|---
BERT | 56.50 | 84.73 | 87.67 | 91.35 | 91.00 | 64.38 | 92.55 | 88.51 | 44.60 | 77.92
\+ AM \+ INLP | 57.69 | 84.67 | 88.75 | 91.24 | 90.88 | 64.38 | 92.74 | 88.80 | 42.25 |  $\uparrow$0.01 77.93
\+ Kmeans \+ INLP | 56.84 | 84.72 | 88.23 | 91.35 | 90.94 | 64.02 | 92.62 | 88.61 | 40.38 |  $\downarrow$0.40 77.52
\+ OracleINLP | 57.44 | 84.62 | 88.32 | 91.37 | 91.02 | 63.66 | 92.74 | 88.32 | 46.48 |  $\uparrow$0.30 78.22
\+ PartialINLP | 57.36 | 84.68 | 88.09 | 91.36 | 91.05 | 65.46 | 92.47 | 88.74 | 43.19 |  $\uparrow$0.12 78.04
\+ AMSAL | 57.19 | 84.85 | 88.48 | 91.28 | 90.96 | 63.78 | 92.66 | 88.75 | 37.56 |  $\downarrow$0.64 77.28
\+ Kmeans \+ SAL | 56.94 | 84.76 | 88.68 | 91.43 | 90.91 | 64.02 | 92.66 | 88.74 | 37.56 |  $\downarrow$0.62 77.30
\+ OracleSAL | 56.68 | 84.72 | 88.18 | 91.24 | 90.93 | 64.50 | 92.74 | 88.67 | 41.78 |  $\downarrow$0.20 77.72
\+ PartialSAL | 56.16 | 84.71 | 87.82 | 91.31 | 90.92 | 64.50 | 92.58 | 88.70 | 41.78 |  $\downarrow$0.31 77.61
ALBERT | 46.55 | 85.31 | 91.17 | 91.73 | 90.82 | 70.88 | 91.55 | 90.57 | 43.66 | 78.03
\+ AM \+ INLP | 57.30 | 85.46 | 90.57 | 91.63 | 90.49 | 70.28 | 91.86 | 90.62 | 48.83 |  $\uparrow$1.65 79.67
\+ Kmeans \+ INLP | 55.91 | 85.59 | 90.74 | 91.70 | 90.62 | 68.71 | 91.86 | 90.94 | 46.48 |  $\uparrow$1.15 79.17
\+ OracleINLP | 55.43 | 85.28 | 91.27 | 91.65 | 90.80 | 72.08 | 91.86 | 90.76 | 40.38 |  $\uparrow$0.81 78.83
\+ PartialINLP | 56.27 | 85.43 | 91.39 | 91.58 | 90.68 | 71.60 | 92.51 | 90.70 | 46.01 |  $\uparrow$1.55 79.57
\+ AMSAL | 55.51 | 85.28 | 91.35 | 91.36 | 90.62 | 73.41 | 91.67 | 90.61 | 38.50 |  $\uparrow$0.67 78.70
\+ Kmeans \+ SAL | 55.45 | 85.54 | 91.33 | 91.53 | 90.66 | 73.16 | 91.97 | 90.69 | 39.91 |  $\uparrow$0.89 78.92
\+ OracleSAL | 54.50 | 85.44 | 92.09 | 91.78 | 90.70 | 72.68 | 92.05 | 90.79 | 43.19 |  $\uparrow$1.22 79.25
\+ PartialSAL | 56.68 | 85.37 | 91.31 | 91.51 | 90.80 | 69.19 | 92.35 | 90.59 | 43.19 |  $\uparrow$0.98 79.00
RoBERTa | 58.38 | 87.63 | 92.06 | 92.64 | 91.28 | 71.12 | 94.15 | 90.22 | 52.58 | 81.12
\+ AM \+ INLP | 57.91 | 87.58 | 91.66 | 92.57 | 91.31 | 71.48 | 94.15 | 90.10 | 52.11 |  $\downarrow$0.13 80.99
\+ Kmeans \+ INLP | 57.37 | 87.51 | 91.47 | 92.83 | 91.26 | 69.92 | 94.15 | 89.96 | 56.34 |  $\uparrow$0.08 81.20
\+ OracleINLP | 56.26 | 87.50 | 91.85 | 92.75 | 91.34 | 72.08 | 94.57 | 90.12 | 52.11 |  $\downarrow$0.16 80.95
\+ PartialINLP | 57.53 | 87.70 | 92.26 | 92.68 | 91.31 | 69.80 | 94.19 | 90.15 | 56.34 |  $\uparrow$0.21 81.33
\+ AMSAL | 57.47 | 87.63 | 91.50 | 92.81 | 91.31 | 69.19 | 94.30 | 90.01 | 56.34 |  $\uparrow$0.06 81.17
\+ Kmeans \+ SAL | 58.23 | 87.56 | 92.51 | 92.97 | 91.31 | 69.92 | 94.00 | 89.96 | 53.99 |  $\uparrow$0.04 81.16
\+ OracleSAL | 58.47 | 87.85 | 92.22 | 92.84 | 91.27 | 70.76 | 94.50 | 89.94 | 52.11 |  $\downarrow$0.01 81.11
\+ PartialSAL | 58.05 | 87.52 | 91.25 | 92.95 | 91.23 | 69.19 | 94.27 | 90.06 | 52.11 |  $\downarrow$0.38 80.74
GPT-2 | 32.73 | 82.69 | 84.06 | 87.80 | 89.20 | 65.46 | 92.16 | 84.50 | 40.85 | 73.27
\+ AM \+ INLP | 33.78 | 82.66 | 83.91 | 87.77 | 89.15 | 64.86 | 92.13 | 84.47 | 40.85 |  $\uparrow$0.01 73.29
\+ Kmeans \+ INLP | 32.67 | 82.68 | 84.36 | 87.83 | 89.16 | 65.46 | 92.13 | 84.51 | 40.85 |  $\uparrow$0.02 73.29
\+ OracleINLP | 33.00 | 82.67 | 84.16 | 87.82 | 89.19 | 65.46 | 92.20 | 84.51 | 40.38 |  $\downarrow$0.01 73.27
\+ PartialINLP | 34.51 | 82.56 | 84.22 | 87.88 | 89.17 | 65.34 | 91.78 | 84.10 | 40.85 |  $\uparrow$0.11 73.38
\+ AMSAL | 35.07 | 82.77 | 84.86 | 88.21 | 89.13 | 65.94 | 92.28 | 83.93 | 40.38 |  $\uparrow$0.35 73.62
\+ Kmeans \+ SAL | 35.57 | 82.78 | 84.80 | 88.27 | 89.16 | 65.34 | 92.13 | 83.93 | 38.97 |  $\uparrow$0.17 73.44
\+ OracleSAL | 37.23 | 82.77 | 84.64 | 88.39 | 89.14 | 64.50 | 92.05 | 84.02 | 40.85 |  $\uparrow$0.46 73.73
\+ PartialSAL | 37.66 | 82.71 | 85.20 | 88.28 | 89.21 | 66.55 | 92.13 | 84.15 | 39.91 |  $\uparrow$0.71 73.98

Table 10: GLUE tests for gender-debiased BERT, ALBERT, RoBERTa, and GPT-2
Models.