# Prediction Sensitivity: Continual Audit of Counterfactual Fairness in
Deployed Classifiers

Krystal Maughan∗<EMAIL_ADDRESS>University of VermontUSA ,  Ivoline
C. Ngong∗<EMAIL_ADDRESS>University of VermontUSA and  Joseph P. Near
<EMAIL_ADDRESS>University of VermontUSA

###### Abstract.

As AI-based systems increasingly impact many areas of our lives, auditing
these systems for fairness is an increasingly high-stakes problem. Traditional
group fairness metrics can miss discrimination against individuals and are
difficult to apply after deployment. Counterfactual fairness describes an
individualized notion of fairness, but is even more challenging to evaluate
after deployment. We present _prediction sensitivity_ , an approach for
continual audit of counterfactual fairness in deployed classifiers. Prediction
sensitivity helps answer the question: _would this prediction have been
different, if this individual had belonged to a different demographic
group_—for _every_ prediction made by the deployed model. Prediction
sensitivity can leverage correlations between protected status and other
features, and does not require protected status information at prediction
time. Our empirical results demonstrate that prediction sensitivity is
effective for detecting violations of counterfactual fairness.

##  1\. Introduction

***The first two authors contributed equally.

As AI-based systems take increasingly larger roles in high-stakes decisions,
ensuring fairness in these systems represents a growing concern. Previous work
has examined the questions of how to audit data and classifiers for bias
before they are deployed (buolamwini2018gender, ;
DBLP:conf/fat/MitchellWZBVHSR19, ), and how to mitigate bias either during
training or as a post-processing step (e.g. (calmon2017optimized, ;
celis2019classification, ; zemel2013learning, ; hardt2016equality, )).

Many existing approaches use _metrics_ that offer formal processes for
“measuring” fairness. _Group fairness_ metrics (feldman2015certifying, ;
hardt2016equality, ) measure disparate treatment of groups in aggregate. These
metrics are useful to demonstrate unfairness, but previous work has shown that
group-fair classifiers can still make clearly unfair predictions for
individuals (LiptonMC18, ). _Individual fairness_ metrics (dwork2012fairness,
) require that similar individuals be treated the same, but the difficulty of
formalizing similarity metrics makes the framework challenging to apply.

_Counterfactual fairness_ (kusner2017counterfactual, ) requires that a fair
classifier _would have made the same prediction_ if the individual had
belonged to a different demographic group. Demonstrating that a classifier
satisfies counterfactual fairness typically requires causal information about
the underlying data distribution (kusner2017counterfactual, ;
kaushik2020explaining, ; kaushik2019learning, ), which presents a major
challenge in deploying practical audit systems based on counterfactual
fairness. An important exception is the recent FlipTest (black2020fliptest, )
system, which uses an optimal transport mapping (rather than causal
information) to demonstrate counterfactual fairness. Like other existing
techniques, FlipTest is designed for auditing models _before_ they are
deployed. Far less work has examined methods for ensuring that classifiers
continue to perform well after deployment—even though novel examples
encountered after deployment are likely to reveal bias in trained classifiers
(buolamwini2018gender, ; googlegorillas, ).

This paper proposes a new approach called prediction sensitivity for
_continual audit_ of counterfactual fairness in _deployed classifiers_. In
contrast to FlipTest, which requires the generation and evaluation of explicit
counterfactual testing data during a distinct audit phase of model development
(i.e. before the model is deployed), prediction sensitivity helps answer the
question: _would this prediction have been different, if this individual had
belonged to a different demographic group_—for _every_ prediction made by the
deployed model.

This kind of continual audit allows deployed systems to raise an alarm before
making a potentially unfair prediction, _even for novel examples_. This
capability represents an important difference from previous work, which may
miss sources of unfairness due to missing training data. Prediction
sensitivity, by contrast, will be able to catch the resulting discrimination
when it occurs and is visible— _after_ the model is deployed.

Prediction sensitivity uses the _gradient of the model’s prediction_ with
respect to the input to approximate how the prediction would have changed if
specific parts of the input had been different. Specifically, our approach
involves training a _protected status model_ capable of weighting input
features according to their importance for determining the protected status,
and then weighting elements of the gradient using this information. Moreover,
the protected status model captures correlations between the protected
attribute and other input features, which is significant, since previous work
has shown removing the protected attribute (dressel2018accuracy, ) or just
flipping (black2020fliptest, ) its value can still cause discrimination in the
model’s predictions. We require knowledge of the protected status of
individuals at training time (to train the protected status model), but not at
prediction time.

Prediction sensitivity can help avoid making biased predictions and can notify
system maintainers of problems with their models. As a metric for
counterfactual fairness, it can be automatically and efficiently calculated
for each prediction and compared against a threshold set by the system
maintainer. The protected status of individuals (i.e. the protected attribute)
does not need to be present at prediction time in order to calculate
prediction sensitivity.

We evaluate prediction sensitivity on both synthetic and real data, using
experiments inspired by FlipTest (black2020fliptest, ). We construct _match
sets_ (similar to FlipTest’s _flipsets_)—examples for which a trained
classifier would have made the same prediction if the protected status had
been different. Non-members of the match set represent failures of
counterfactual fairness. Our results suggest that prediction sensitivity is
effective at detecting these failures.

#### Contributions.

In summary, we make the following contributions:

  * •

We propose _prediction sensitivity_ , a gradient-based method for measuring
counterfactual fairness

  * •

We show how to use prediction sensitivity to detect biased predictions at the
individual level in deployed models

  * •

We present experimental results suggesting that prediction sensitivity is
effective for detecting biased predictions

##  2\. Background & Related Work

#### Deep learning.

In this paper, we focus on machine learning models represented by _artificial
neural networks_ (goodfellow2016deep, ). A model $\mathcal{F}$ is
parameterized by a set of _weights_ which are optimized during training; we
write $\mathcal{F}(\textit{{x}})$ to represent a _prediction_ made by the
trained model on an example x. Deep learning models are typically trained by
optimizing a _loss function_ $\mathcal{L}$ by calculating the _gradient_ of
the loss with respect to the weights and updating the model accordingly.

#### Fairness in machine learning.

The bulk of previous work on fairness in machine learning attempts to improve
_group fairness metrics_ at training time, often by the introduction of new
kinds of regularization (calders2009building, ; woodworth2017learning, ;
zafar2015fairness, ; zafar2017fairness, ; agarwal2018reductions, ;
russell2017worlds, ; celis2019classification, ; beutel2017data, ;
shankar2017no, ; zhang2018mitigating, ; wadsworth2018achieving, ;
celis2019improved, ; zemel2013learning, ; louizos2015variational, ;
lum2016statistical, ; adler2018auditing, ; calmon2017optimized, ;
feldman2015computational, ; hardt2016equality, ). Many of these approaches are
suitable for deep learning, and have been empirically validated using the
metrics described above. These approaches typically apply when each example’s
features include a _protected attribute_ $z\in x$ to indicate the example’s
_protected status_ (as a member or non-member of a protected class). Existing
approaches focus on notions of group fairness, and are validated using metrics
for group fairness. As a result, they can sometimes produce models that give
blatantly _unfair_ predictions for specific individuals, even though they
score well on group fairness metrics (dwork2012fairness, ; LiptonMC18, ).

#### Counterfactual fairness.

_Counterfactual fairness_ (kusner2017counterfactual, ) requires that each
prediction a classifier makes _would have been the same_ if the protected
attribute had been different. _Counterfactual augmentation_
(kaushik2019learning, ; kaushik2020explaining, ) focuses on counterfactual
statements about individuals in the original training data. Most approaches
involve a manual process of asking a human expert to construct a new training
example that is identical to an existing one, except for the protected
attribute. For example, in the NLP setting, classifiers often associate the
profession of nursing with women (bolukbasi2016man, ), due to the large volume
of (biased) statements in the training data that involve female nurses. For
each statement about a female nurse in the training data, we might ask the
expert to write an identical statement about a male nurse—and a classifier
trained on the augmented data will be less likely to associate nursing with
women as a result.

Counterfactual augmentation has proven extremely effective in models for
language (huang2020reducing, ; garg2019counterfactual, ;
zmigrod2019counterfactual, ) and images (denton2019detecting, ), where an
explicit protected attribute is often not present. In these domains,
counterfactual augmentation is generally a manual process; recent work
provides support (madaan2020generate, ; wu2021polyjuice, ) but not complete
automation. Prediction sensitivity (defined in Section 3) can be viewed as a
way of measuring counterfactual fairness. Because our evaluation is done in
the simpler setting of a binary protected attribute, we are able to use a
relatively simple algorithm for automatic counterfactual augmentation.

#### FlipTest.

Our work is most closely related to FlipTest (black2020fliptest, ), an
approach for generating augmented datasets for auditing the counterfactual
fairness of classifiers. FlipTest determines whether a model is sensitive to
the protected status of an individual or subgroup using optimal transport
mapping. It reveals salient patterns in a model’s behavior by constructing
approximate mappings using a Generative Adversarial Network (GAN). Using these
mappings, flipsets and a transparency report are created, which are utilized
to determine which individuals or subgroups are discriminated against, as well
as identify those features which may be associated with it.

The major advantage of our approach is the ability to audit classifiers
_during deployment_. Unlike FlipTest, our approach does not require the
explicit generation of datasets to measure counterfactual fairness; instead,
it provides a measurement for each prediction the classifier makes, with no
other inputs required. Our approach also has the novel ability to detect
failures of counterfactual fairness resulting from previously-unseen data.

#### Interpretability in deep learning.

The related field of _interpretable AI_ seeks to enable auditability by
understanding _why_ a model made a specific prediction. The problem of
interpretability has been studied extensively in the setting of image
classification, where the goal is to understand which pixels of the image were
most important in deciding its class. Numerous gradient-based approaches have
been proposed (morch1995visualization, ; baehrens2010explain, ;
simonyan2013deep, ; sundararajan2017axiomatic, ; smilkov2017smoothgrad, ;
hooker2019benchmark, ); our approach to prediction sensitivity draws on these
ideas.

##  3\. Prediction Sensitivity

Figure 1. Overview of calculating prediction sensitivity. Prediction
sensitivity is based on measurements of each feature’s contribution to both
protected status and the classifier’s prediction.

This section describes prediction sensitivity, an easily calculated metric for
auditing the counterfactual fairness of deployed models. For a single
prediction, our goal is to answer the same question as FlipTest
(black2020fliptest, ): _had the individual been of a different protected
status, would the model have treated them differently?_

We consider a classifier $\mathcal{F}$ and an individual input x. Let the
_counterfactual individual_ $\textit{{x}}^{\prime}$ represent the same
individual as x, but with a different protected status. We would like to know
if $\mathcal{F}(\textit{{x}})=\mathcal{F}(\textit{{x}}^{\prime})$. Prediction
sensitivity answers this question by computing two gradients with respect to
the input x, as depicted in Figure 1: the _protected status feature weights_
(§ 3.1) and the _per-feature influence_ (§ 3.2). The final prediction
sensitivity is the dot product of the two components—a measure of how changes
in the input x would lead to changes in $\mathcal{F}$’s prediction, weighted
by each feature’s influence on the individual’s protected status. Both
components are inspired by gradient-based approaches for interpretability
(especially SmoothGrad (smilkov2017smoothgrad, )).

###  3.1. Protected Status Feature Weights

The _protected status feature weights_ measure _how the features of x
contribute to the x’s protected status_. We calculate the protected status
feature weights by training a _protected status model_ $\mathcal{A}$ to
predict x’s protected status, and computing the weights using the gradient of
$\mathcal{A}(\textit{{x}})$. If $\mathcal{A}$ captures the correlation between
input features and protected status, then these weights encode the strength of
each feature’s value on protected status.

The protected status of an individual may be encoded in an explicit protected
attribute $a\in\textit{{x}}$, and $\mathcal{A}$ will learn that the
correlation between $a$ and the protected status. When the protected attribute
is not present at prediction time, or other features are correlated with
protected status, $\mathcal{A}$ will discover the appropriate correlations.
The specifics of the protected status model $\mathcal{A}$ depend on the
problem setting. In our experiments, we use a simple neural network with a
similar architecture to the classifier we are auditing. The primary
requirement is that $\mathcal{A}$ should capture correlations between features
in x and the protected status $s$.

###### Definition 3.1 (Protected Status Feature Weights).

The _protected status feature weights_ for an example
$\textit{{x}}\in\mathbb{R}^{k}$ are a length-$k$ vector defined as follows:

$PSW(\textit{{x}})=\textit{abs}(\nabla\mathcal{A}(\textit{{x}}))$

where abs denotes element-wise absolute value.

###  3.2. Per-Feature Influence & Prediction Sensitivity

The second component of prediction sensitivity is the _per-feature influence_
, a vector that captures the influence of each input feature on the
classifier’s prediction. The per-feature influence describes _how much a
hypothetical change in each feature of x would affect $\mathcal{F}$’s
prediction_. Prediction sensitivity is defined as the dot product of this
gradient with the protected status feature weights for x.

###### Definition 3.2 (Prediction sensitivity).

The _prediction sensitivity_ $PS(\textit{{x}})\in\mathbb{R}$ for an example x
is defined as:

$\displaystyle PS(\textit{{x}})$
$\displaystyle=PSW(\textit{{x}})\cdot\textit{abs}(\nabla\mathcal{F}(\textit{{x}}))$
$\displaystyle=\textit{abs}(\nabla\mathcal{A}(\textit{{x}}))\cdot\textit{abs}(\nabla\mathcal{F}(\textit{{x}}))$

where $\nabla\mathcal{A}(\textit{{x}}))$ is the gradient of
$\mathcal{A}(\textit{{x}}))$ (with respect to x) and abs denotes element-wise
absolute value.

###  3.3. Prediction Sensitivity as a Measure of Counterfactual Fairness

Prediction sensitivity is designed to answer the question: _had the individual
been of a different protected status, would the model have treated them
differently?_ Prediction sensitivity is thus likely to be a close
approximation of a measurement of counterfactual fairness in most settings.
Formally, counterfactual fairness is defined as follows:

###### Definition 3.3 (Counterfactual fairness (kusner2017counterfactual, )).

A predictor $\hat{Y}$ of $Y$ is _counterfactually fair_ given the sensitive
attribute $A=a$ and any observed variables $X$ if:

$\Pr[\hat{Y}_{A\leftarrow a}=y\;|\;X=x,A=a]=\Pr[\hat{Y}_{A\leftarrow
a^{\prime}}=y\;|\;X=x,A=a]$

for all $y$ and $a^{\prime}\neq a$ (i.e. $a^{\prime}$ denotes a different
protected status than $a$).

This definition requires that the distribution of $\hat{Y}$ does not change
when $A$ changes, as long as things which are not causally dependent on $A$
are held constant. Demonstrating counterfactual fairness thus requires
knowledge about the causal relationships between variables (in particular,
about the causal relationships between $A$ and other attributes).

In our approach, the protected status model $\mathcal{A}$ encodes correlations
between the features of x and the protected status ($A$ in the above
definition). Our $\mathcal{A}$ model can be seen as an overapproximation of
the required causality knowledge, in the sense that it may learn correlations
that are not causal. Prediction sensitivity therefore may produce false
positives (i.e. it may be high, even for a prediction that satisfies
counterfactual fairness), but as long as the $\mathcal{A}$ model correctly
captures correlations with the protected status, it will not produce false
negatives.

A second important condition for the correspondence between prediction
sensitivity and counterfactual fairness is that the gradient should provide
good information about the influence of individual features on the prediction
(for both the $\mathcal{A}$ and $\mathcal{F}$ models). When the gradient is
smooth, then prediction sensitivity should be an effective way to measure
counterfactual fairness for individual predictions.

##  4\. Using Prediction Sensitivity to Audit Deployed Models

Figure 2. Continual audit of counterfactual fairness on a deployed classifier
using prediction sensitivity.

Prediction sensitivity can be calculated at training time, using a test set,
to measure fairness properties of the trained classifier. However, the real
strength of prediction sensitivity is that it can also be used _at prediction
time_ —calculating prediction sensitivity requires neither the true label nor
the protected attribute. Prediction sensitivity can thus be deployed
_alongside_ production classifiers, to continually measure fairness properties
of the predictions those classifiers make over time, as shown in Figure 2.

#### Training time.

At training time, we train both the classifier $\mathcal{F}$ and the protected
status model $\mathcal{A}$, as shown in Figure 2(a). Training $\mathcal{A}$
requires information about protected status to be present in the training
data. The classifier $\mathcal{F}$ can be trained to satisfy fairness
conditions using training-time techniques (e.g. (calders2009building, ;
woodworth2017learning, ; zafar2015fairness, ; zafar2017fairness, ;
agarwal2018reductions, ; hardt2016equality, )), and can be audited before it
is deployed using group fairness metrics and approaches like FlipTest
(black2020fliptest, )).

We can also use prediction sensitivity to audit the trained classifier for
counterfactual fairness at training time. We compute prediction sensitivity
for examples in the test set and ensure that (1) both the mean and variance of
the prediction sensitivity are low, and (2) there are no outliers representing
disadvantaged individuals. The mean prediction sensitivity for the classifier
on the test set can be saved as a _baseline_ for use during deployment.

#### Deployment time.

After the classifier is deployed, prediction sensitivity can be calculated for
each prediction and compared against the baseline by a monitor deployed with
the system, as shown in Figure 2(b). If a particular prediction results in
prediction sensitivity much higher than the baseline, then it is likely that
counterfactual fairness has been violated. In this case, the monitor can raise
an alarm.

#### Reacting to high prediction sensitivity.

When the classifier fails to satisfy counterfactual fairness, prediction
sensitivity can help guide action to correct the situation.

  1. (1)

Most importantly, the prediction should not be used to make decisions, since
it may cause harm. The system should fall back on manual intervention by an
expert, or a backup system (e.g. a default decision).

  2. (2)

Properties of the individual whose prediction caused the failure of
counterfactual fairness may immediately suggest an approach for improving the
classifier $\mathcal{F}$. Such failures may be due to a lack of similar
individuals in the training data; here, the classifier can be improved by re-
training on additional data.

  3. (3)

The feature-wise prediction sensitivity may indicate that the classifier has
focused on specific features in making discriminatory predictions. Particular
attention should be paid to these features when improving the training data.

#### Auditing by third parties.

Prediction sensitivity can only be calculated with whitebox access to the
classifier $\mathcal{F}$ and protected status model $\mathcal{A}$, because of
the gradient calculations involved. This limits the use of prediction
sensitivity to the party who trained and deployed the model; it is not
feasible for third parties to audit a deployed system using prediction
sensitivity without access to the classifiers involved.

This limitation means that prediction sensitivity is not a useful tool for
helping journalists or other third parties to discover discriminatory behavior
in existing systems. However, corporations face increasing public pressure to
ensure that their systems do not discriminate; this pressure provides a
significant incentive for companies to adopt approaches like prediction
sensitivity to improve their models and provide evidence of fairness to their
users.

##  5\. Evaluation on Synthetic Data via Causal Modeling

This section describes an empirical evaluation of prediction sensitivity as a
measure for counterfactual fairness using synthetic data. Using synthetic data
allows us to precisely model our desired fairness definition (including causal
information), and measure the effectiveness of our approach.

###  5.1. Evaluation Approach

Prediction sensitivity is intended to answer the question, _had the individual
been of a different protected status, would the model have treated them
differently?_ Given a dataset of otherwise identical individuals with
different protected statuses, we could ask our model to make a prediction for
each one and compare them; differing predictions for corresponding individuals
represent failures of counterfactual fairness. However, actually
_constructing_ this dataset is typically very challenging (dwork2012fairness,
; black2020fliptest, ). By employing synthetic data, we can use an alternative
approach that avoids this challenge but accomplishes the same goal.

Our approach is to train two classifiers and then compare their predictions to
detect failures of counterfactual fairness. One is the “original” classifier
$\mathcal{F}$, which may not satisfy counterfactual fairness. The second model
is the _unbiased classifier_ $\hat{\mathcal{F}}$, trained on modified data to
ensure counterfactual fairness. We evaluate the counterfactual fairness of
$\mathcal{F}$ by comparing its predictions on the test set against those of
$\hat{\mathcal{F}}$. When the two models make _different_ predictions, this
indicates a possible failure of counterfactual fairness. We call the set of
individuals in the test set for whom the two models make matching predictions
the _match set_. We evaluate prediction sensitivity as a measure of
counterfactual fairness by checking whether prediction sensitivity can
effectively distinguish members of the match set from non-members.

|  |
---|---|---
(a) |  | (b)
Figure 3. Causal graphs for synthetic data. (a) shows a causal graph for
“biased” synthetic data, in which a causal relationship exists between
protected status and outcome. (b) shows a modified causal graph that removes
this relationship. Data generated according to model (b) can be used to train
classifiers that satisfy counterfactual fairness.

#### Modeling counterfactual fairness in synthetic data.

We generate two synthetic datasets according to the causal models shown in
Figure 3. The causal model in Figure 3(a) includes a causal relationship
between protected status and outcome, and is thus likely to result in biased
classifiers. In our evaluation, we train the classifier $\mathcal{F}$ using
data generated according to this model—so we expect that $\mathcal{F}$ will
not always satisfy counterfactual fairness. The causal model in Figure 3(b)
lacks this causal relationship. We train the _unbiased classifier_
$\hat{\mathcal{F}}$ using data generated according to this model, so that
$\hat{\mathcal{F}}$ satisfies counterfactual fairness.

In our evaluation, following Lipton et al. (LiptonMC18, ), we generate a
synthetic dataset with two features and binary labels. Augmenting this dataset
with a protected attribute (gender) chosen at random will produce a dataset
consistent with the causal model in Figure 3(b). Models trained on this
dataset score well on group fairness metrics like statistical parity and
disparate impact, since gender is completely independent of the label and
other features. We use this dataset to train the model $\hat{\mathcal{F}}$. We
introduce bias in our synthetic dataset by generating the protected attribute
based on the label:

$\Pr(z=\text{woman})=\left\\{\begin{array}[]{ll}.25&\mbox{if
}y=\text{positive}\\\ .75&\mbox{if }y=\text{negative}\end{array}\right.$

This model clearly introduces bias against women—men are much more likely to
be members of the positive class, while women are much more likely to be
members of the negative class. We use this dataset to train the model
$\mathcal{F}$.

#### Enumerating match sets.

The next step is to find individuals for whom $\mathcal{F}$ and
$\hat{\mathcal{F}}$ make different predictions. Since $\hat{\mathcal{F}}$ is
defined to satisfy counterfactual fairness, such a mismatch likely indicates a
_failure_ of counterfactual fairness for $\mathcal{F}$. We call the set of
individuals for whom both models make the same predictions a _match set_. The
classifier $\mathcal{F}$ likely satisfies counterfactual fairness for members
of the match set, and likely violates it for non-members. For counterfactually
fair classifiers, the match set should include the entire test set.

#### Evaluating prediction sensitivity.

The final step of our evaluation is to demonstrate that prediction sensitivity
is effective for distinguishing members of the match set from non-members. We
calculate the prediction sensitivity for $\mathcal{F}$’s predictions on the
test set, and use it to build a classifier $\mathcal{D}$ to distinguish match
set members from non-members. $\mathcal{D}$ is defined as follows (where $p$
is the prediction sensitivity associated with an example, and $\theta$ is a
threshold):

$\mathcal{D}(p,\theta)=\left\\{\begin{array}[]{ll}\textbf{member}&\mbox{if
}p\leq\theta\\\ \textbf{non-member}&\mbox{if }p>\theta\end{array}\right.$

Here, the threshold $\theta$ allows trading off between false negatives and
false positives.

#### Comparison to FlipTest evaluation.

FlipTest (black2020fliptest, ) calculates a similar set of individuals—called
a _flipset_ —by directly generating datasets of counterfactuals. For each
individual x in the test set, FlipTest uses a generative adversarial network
(GAN) to generate an in-distribution individual $\textit{{x}}^{\prime}$ with
the opposite protected status. Then, FlipTest asks the classifier under
evaluation ($\mathcal{F}$) to make predictions on each individual and their
corresponding counterfactual. Matching predictions are considered
counterfactually fair, since the model treats both individuals the same; non-
matching predictions are considered failures of counterfactual fairness.
FlipTest calls the set of non-matching predictions a _flipset_ —the set of
individuals for whom the prediction flips when protected status changes. Our
approach has a similar goal—the flipset serves the same purpose as our match
set, and inspires its name—but our approach does not require the construction
of a GAN or the enumeration of counterfactual individuals.

###  5.2. Experiment Setup

We used Scikit-learn to generate synthetic datasets consistent with the causal
models shown in Figure 3. We trained the classifier $\mathcal{F}$ using data
consistent with the model in Figure 3(a), and the counterfactually fair
classifier $\hat{\mathcal{F}}$ using data consistent with the model in Figure
3(b). We trained $\mathcal{A}$, $\mathcal{F}$, and $\hat{\mathcal{F}}$, then
computed prediction sensitivities and constructed the distinguisher
$\mathcal{D}$ consistent with the process described in the last section. We
constructed receiver operating characteristic (ROC) curves and calculated area
under the curve (AUC) values for the distinguisher by varying $\theta$. Due to
nondeterminism in the training process, we performed 30 trials of the
experiment.

###  5.3. Results

|
---|---
(a) | (b)
Figure 4. Using prediction sensitivity to audit models trained on synthetic
data. (a) shows that prediction sensitivity is low for members of the match
set, but high for non-members (note the logarithmic scale in the vertical
axis). (b) shows that a distinguisher based on prediction sensitivity is
effective at detecting failures of counterfactual fairness.

The results of our experiment on synthetic data appear in Figure 4. Figure
4(a) shows that average prediction sensitivity is much higher for non-members
of the match set than it is for members, suggesting that prediction
sensitivity is able to distinguish individuals for whom the classifier
$\mathcal{F}$ fails to satisfy counterfactual fairness.

Figure 4(b) shows a receiver operating characteristic (ROC) curve and its area
under the curve (AUC) value for the distinguisher $\mathcal{D}$ built using
prediction sensitivity. We plot one curve per trial we performed (30 curves in
total) and give the average AUC over all trials. The results show that
prediction sensitivity is capable of distinguishing members of the match set
from non-members, even for low false positive rates.

##  6\. Evaluation on Real Data via Counterfactual Augmentation

In Section 5, we were able to generate synthetic data consistent with causal
models we designed ourselves, enabling the comparison of the two models
($\mathcal{F}$ and $\hat{\mathcal{F}}$) to build match sets. A similar process
is impossible for real data, since the underlying causal model is unknown.

For our evaluation on real datasets, we instead approximate the same procedure
using _counterfactual augmentation_. Counterfactual augmentation
(kaushik2019learning, ; kaushik2020explaining, ) attempts to ensure
counterfactual fairness in trained classifiers by adding new training
examples. For each individual x in the training data, we add a corresponding
individual $\textit{{x}}^{\prime}$ with a _different protected status_ but the
_same outcome_. Classifiers trained on the augmented data will be more likely
to satisfy counterfactual fairness, because each training example has a
corresponding counterfactual also present during training. Previous work has
shown this approach to be effective in training classifiers that also score
well on group fairness metrics (kaushik2019learning, ; kaushik2020explaining,
).

###  6.1. Evaluation Approach

Our evaluation on real data follows the same process as our synthetic data
evaluation described in Section 5, but using counterfactual augmentation to
construct training data for the counterfactually fair model
$\hat{\mathcal{F}}$.

#### Constructing augmented datasets.

In complex domains like NLP, counterfactual augmentation is a manual process.
In our simpler setting of binary classification with an explicit, binary
protected attribute, it is possible to automate this process. In particular,
for each example in the original dataset, we can construct a new example with
the opposite value for the protected attribute. For each training example x,
we construct a new one $\textit{{x}}^{\prime}$ in which the protected
attribute is replaced by its negation. The label remains the same. The
resulting dataset has exactly twice the original number of examples. This
approach is simple and effective, but it requires the protected attribute to
be discrete (ideally, binary) and be explicitly included in the data (in e.g.
NLP data, it often is not).

#### Evaluating prediction sensitivity.

Having constructed an augmented dataset, we can evaluate prediction
sensitivity in the same way as in Section 5, by constructing a match set and a
distinguisher.

#### Limitations of our implementation of counterfactual augmentation.

The goal of counterfactual augmentation is to approximate data drawn from a
similar distribution to the original data, but _without_ the causal
relationship between protected status and outcome. However, our approach
modifies only the protected attribute, and leaves other (possibly correlated)
features alone. If other features are correlated with the protected attribute,
then the correlations between those features and the individual’s _original_
protected status may still exist.

Consider the Adult dataset (adult, ) (used in our empirical evaluation), which
includes job category and gender (a protected attribute) as features, and
income as the label. It is likely that job category is correlated with gender,
but our approach for counterfactual augmentation ignores this correlation. As
a result, the classifier $\hat{\mathcal{F}}$ may _fail_ to satisfy
counterfactual fairness in some cases, even though it is trained on
counterfactually augmented data, because it may learn to discriminate based on
features correlated with the protected attribute (rather than the protected
attribute itself).

This is a limitation of our evaluation approach—not of prediction sensitivity
itself. As described in Section 3, prediction sensitivity does capture
correlations between all features and protected status, via the protected
status model.

#### Comparison to FlipTest evaluation.

As described in Section 5, the match sets we generate in our evaluation are
similar to FlipTest’s flipsets. However, FlipTest uses a generative
adversarial network (GAN) to generate the augmented data to construct
flipsets, while we use a simpler form of counterfactual augmentation. The GAN-
based approach can better capture correlations between the protected attribute
and other features, but it is more complex and may also fail to generate out-
of-distribution counterfactuals that would have exposed failures in the
classifier being tested.

###  6.2. Experiment Setup

We follow the same experimental setup as in Section 5, training $\mathcal{F}$
on the original data and $\hat{\mathcal{F}}$ on the augmented data. We
calculated prediction sensitivity for each of $\mathcal{F}$’s predictions on
the test set as described in Section 3, training the protected status model
$\mathcal{A}$ using the original training data used for $\mathcal{F}$. We
defined the distinguisher $\mathcal{D}$ based on these prediction sensitivity
values. As in Section 5, we performed 30 trials.

#### Datasets.

Our evaluation considers two commonly-used datasets in the AI fairness
literature: the Adult (adult, ) and COMPAS (compas, ) datasets. Statistics of
the two datasets are shown in Table 1. Both involve classification tasks: the
Adult dataset’s task is predicting whether an individual’s income is greater
than or less than $50,000, and the COMPAS dataset’s task is predicting
recidivism risk. Both datasets are known to contain embedded bias: models
trained on the Adult dataset tend to predict that white males have the highest
chance of having high income, and models trained on the COMPAS dataset tend to
predict that Black suspects are most likely to re-offend. We augment both
datasets with counterfactual training examples as described earlier. This
process produces datasets that are exactly twice as large as the originals.
For each male instance in the Adult dataset, for example, we add a female
instance that is identical (including the label) except for the protected
attribute.

Dataset | Dataset Size | Model Properties
---|---|---
| Features | Train | Test | Layers | Layer Width | Activation | Accuracy
Adult (adult, ) | 95 | 24752 | 6188 | 3 | 32 | ReLU | 84.6%
COMPAS (compas, ) | 399 | 4937 | 1235 | 3 | 256 | ReLU | 67.9%
Table 1. Information about the datasets and models used in our experiments.

#### Model architectures.

As shown in Table 1, we use a linear network architecture with a single hidden
layer and the ReLU activation function. The sigmoid activation function is
applied on the output layer. The loss function is the binary cross entropy
loss and Adam is chosen as the optimizer with a learning rate of 0.001. We
find that a wider model (256 neurons) works better for the COMPAS dataset,
while a narrower one (32 neurons) suffices for the Adult dataset. We train
each classifier for 40 epochs, and achieve roughly state-of-the-art accuracy
for both datasets.

#### Test sets.

When evaluating the fairness of a classifier, should we also augment the test
set via counterfactual augmentation? Wick et al. (NEURIPS2019_373e4c5d, )
argue in favor of an _unbiased_ test set (i.e. “fair” data) to evaluate
fairness. In particular, a fair classifier may produce lower test accuracy on
biased data, leading to the (false) conclusion that improvements to fairness
have “hurt” accuracy. We ran our experiments twice: once using test sets
containing original data only (20% of the original dataset), and once using
counterfactually augmented test sets. We report results for the augmented test
sets, since we found essentially no differences between the two settings (more
in Section 6.3).

|  |
---|---|---
Adult (sex) | COMPAS (sex) | COMPAS (race)
Figure 5. Distributions of group fairness metrics for real datasets, based on
30 trials for each configuration. As expected, models trained with
counterfactually augmented data score much better on group fairness metrics.

###  6.3. Results

Dataset & | Training | Classifier Accuracy | Prediction Sens. AUC
---|---|---|---
Prot. Attr. | Epochs | Original | Augmented | Original | Augmented
Adult (adult, ) (sex) | 10 | 84.0% $\pm$ 1.65 | 84.3% $\pm$ 1.53 | 57.4% $\pm$ 0.17 | 59.3% $\pm$ 0.15
40 | 84.4% $\pm$ 1.52 | 85.1% $\pm$ 0.84 | 69.8% $\pm$ 0.15 | 69.9% $\pm$ 0.17
Adult (adult, ) (race) | 10 | 83.4% $\pm$ 1.74 | 84.1% $\pm$ 1.82 | 57.5% $\pm$ 0.18 | 60.4% $\pm$ 0.17
40 | 84.6% $\pm$ 1.55 | 84.0% $\pm$ 2.13 | 66.7% $\pm$ 0.19 | 66.3% $\pm$ 0.19
COMPAS (compas, ) (sex) | 10 | 67.7% $\pm$ 0.60 | 66.8% $\pm$ 0.74 | 77.7% $\pm$ 0.07 | 79.4% $\pm$ 0.06
40 | 66.7% $\pm$ 0.54 | 66.0% $\pm$ 0.79 | 76.4% $\pm$ 0.05 | 77.5% $\pm$ 0.04
COMPAS (compas, ) (race) | 10 | 67.9% $\pm$ 0.40 | 66.9% $\pm$ 0.42 | 79.2% $\pm$ 0.07 | 78.0% $\pm$ 0.07
40 | 66.6% $\pm$ 0.47 | 65.9% $\pm$ 0.47 | 74.8% $\pm$ 0.05 | 75.1% $\pm$ 0.05
Table 2. Comparison of average area under the curve (AUC) for original and
augmented test sets. Results averaged over 30 trials.

The results of our experiments on real data appear in Figures 5, 6, and 7, and
Table 2; we summarize them below.

#### Counterfactual augmentation improves group fairness metrics (Figure 5).

Figure 5 shows that as expected, classifiers trained on counterfactually-
augmented data score much better according to common group fairness metrics
than classifiers trained on the original data. This result is expected, and
consistent with previous work (kaushik2019learning, ; kaushik2020explaining, ;
black2020fliptest, ).

#### Prediction sensitivity detects violations of counterfactual fairness
(Figures 6 and 7).

Figure 6 plots the distribution of prediction sensitivities for members and
non-members of the match sets (“matching” and “non-matching,” respectively).
In both datasets, and for both protected attributes, members of the match set
have consistently smaller prediction sensitivity; on average, non-members have
significantly higher values for prediction sensitivity.

Figure 7 plots receiver operating characteristic (ROC) curves and
corresponding average area under the curve (AUC) values for the distinguisher
$\mathcal{D}$ built using prediction sensitivity for each dataset. Each plot
contains 30 ROC curves, one per trial of the experiment. The results show that
prediction sensitivity is an effective approach for distinguishing between
members and non-members of the match set, and thus it is likely effective at
detecting failures of counterfactual fairness. The results also demonstrate
considerable variability, especially for the Adult dataset. Both figures
contain results based on a counterfactually-augmented test set. Further
discussion of these results appears in Section 6.4.

#### Results are consistent for original and augmented test sets (Table 2).

As described in Section 6.2, we performed each experiment twice—once with the
original test set, and once with a counterfactually-augmented test set. Table
2 compares the results of these two experiments. The results are nearly
identical, suggesting that prediction sensitivity is effective in both
settings. Table 2 also compares fully-trained classifiers (40 epochs) against
partially-trained classifiers (10 epochs). The impact of the fine-tuning that
occurs in later stages of training depends on the dataset; prediction
sensitivity is more effective for detecting violations in the fully-trained
classifier for the Adult dataset, but it is _less_ effective in this setting
for the COMPAS dataset.

|  |  |
---|---|---|---
Adult (sex) | Adult (race) | COMPAS (sex) | COMPAS (race)
Figure 6. Distributions of prediction sensitivities for real datasets, based on 30 trials for each configuration. Note the logarithmic scale of the vertical axis. In all cases, average prediction sensitivity is much higher for non-members of the match set than it is for members. |
---|---
Adult (sex) | Adult (race)
|
COMPAS (sex) | COMPAS (race)
Figure 7. Distributions of ROC curves for using prediction sensitivity to
detect unfair predictions on real datasets, based on 30 trials for each
configuration. In all cases, prediction sensitivity is highly predictive of
non-membership in the match set, but its effectiveness varies with the data.

###  6.4. Discussion

The results described in Section 6.3 suggest that prediction sensitivity is an
effective measure of counterfactual fairness. However, the average area under
the curve (AUC) for our experiments on real datasets is significantly lower
than the AUC for our synthetic data experiments. We suspect two possible
contributing factors. First, it could be that bias is less severe in the real
datasets, so failures of counterfactual fairness are harder to detect. This
would also explain the difference in AUC between the two real datasets.
Second, our approach to counterfactual augmentation could be missing
significant correlations between other features and protected status. This
could cause low AUC by invalidating the match set (prediction sensitivity
could be _correct_ , while the match set is _wrong_). In particular, this
factor may be responsible for the plateaus in ROC curves for some trials on
the Adult dataset (Figure 7).

Our results suggest that prediction sensitivity is effective at detecting
unfair predictions, but they also reflect the inherent challenge of this task.
Individual predictions with extremely high prediction sensitivity are likely
to be blatantly unfair, and should be easily detected using prediction
sensitivity; however, borderline cases may be more difficult to detect, as
demonstrated by the variability in the ROC curves in Figure 7.

##  7\. Conclusion

We have presented a new, end-to-end approach for continual auditing of
counterfactual fairness in deployed deep learning systems. We propose
prediction sensitivity, an efficiently computed metric for counterfactual
fairness. Prediction sensitivity can be used at prediction time, on deployed
classifiers, to raise an alarm when the classifier makes an unfair prediction.
Prediction sensitivity handles correlations between features and protected
status, and does not require access to individuals’ protected status at
prediction time. Our empirical evaluation on synthetic and real datasets
suggests that prediction sensitivity is effective for detecting failures of
counterfactual fairness.

###### Acknowledgements.

We thank David Darais and Kristin Mills for their contributions to the
development of this work. This research was supported in part by an Amazon
Research Award.

## References

  * [1] Adult dataset (UCI machine learning repository).  https://archive.ics.uci.edu/ml/datasets/adult.
  * [2] COMPAS dataset (Propublica).  https://www.propublica.org/datastore/dataset/compas-recidivism-risk-score-data-and-analysis.
  * [3] Google Photos Tags Two African-Americans As Gorillas Through Facial Recognition Software.  https://www.forbes.com/sites/mzhang/2015/07/01/google-photos-tags-two-african-americans-as-gorillas-through-facial-recognition-software/.
  * [4] Philip Adler, Casey Falk, Sorelle A Friedler, Tionney Nix, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian.  Auditing black-box models for indirect influence.  Knowledge and Information Systems, 54(1):95–122, 2018.
  * [5] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach.  A reductions approach to fair classification.  arXiv preprint arXiv:1803.02453, 2018.
  * [6] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller.  How to explain individual classification decisions.  The Journal of Machine Learning Research, 11:1803–1831, 2010.
  * [7] Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi.  Data decisions and theoretical implications when adversarially learning fair representations.  arXiv preprint arXiv:1707.00075, 2017.
  * [8] Emily Black, Samuel Yeom, and Matt Fredrikson.  Fliptest: fairness testing via optimal transport.  In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 111–121, 2020.
  * [9] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  Advances in Neural Information Processing Systems, 2016.
  * [10] Joy Buolamwini and Timnit Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In Conference on Fairness, Accountability and Transparency, pages 77–91, 2018.
  * [11] Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy.  Building classifiers with independency constraints.  In 2009 IEEE International Conference on Data Mining Workshops, pages 13–18. IEEE, 2009.
  * [12] Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney.  Optimized pre-processing for discrimination prevention.  In Advances in Neural Information Processing Systems, pages 3992–4001, 2017.
  * [13] L Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi.  Classification with fairness constraints: A meta-algorithm with provable guarantees.  In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 319–328, 2019.
  * [14] L Elisa Celis and Vijay Keswani.  Improved adversarial learning for fair classification.  arXiv preprint arXiv:1901.10443, 2019.
  * [15] Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru.  Detecting bias with generative counterfactual face attribute augmentation.  arXiv preprint arXiv:1906.06439, 2019.
  * [16] Julia Dressel and Hany Farid.  The accuracy, fairness, and limits of predicting recidivism.  Science advances, 4(1):eaao5580, 2018.
  * [17] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel.  Fairness through awareness.  In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214–226, 2012.
  * [18] Michael Feldman.  Computational fairness: Preventing machine-learned discrimination.  2015\.
  * [19] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian.  Certifying and removing disparate impact.  In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259–268, 2015.
  * [20] Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel.  Counterfactual fairness in text classification through robustness.  In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 219–226, 2019.
  * [21] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning.  MIT press, 2016.
  * [22] Moritz Hardt, Eric Price, and Nati Srebro.  Equality of opportunity in supervised learning.  In Advances in neural information processing systems, pages 3315–3323, 2016.
  * [23] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim.  A benchmark for interpretability methods in deep neural networks.  In Advances in Neural Information Processing Systems, pages 9737–9748, 2019.
  * [24] Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli.  Reducing sentiment bias in language models via counterfactual evaluation.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 65–83, 2020.
  * [25] Divyansh Kaushik, Eduard Hovy, and Zachary Lipton.  Learning the difference that makes a difference with counterfactually-augmented data.  In International Conference on Learning Representations, 2019.
  * [26] Divyansh Kaushik, Amrith Setlur, Eduard Hovy, and Zachary C Lipton.  Explaining the efficacy of counterfactually-augmented data.  arXiv preprint arXiv:2010.02114, 2020.
  * [27] MJ Kusner, J Loftus, Christopher Russell, and R Silva.  Counterfactual fairness.  Advances in Neural Information Processing Systems 30 (NIPS 2017), 30, 2017.
  * [28] Zachary C. Lipton, Julian J. McAuley, and Alexandra Chouldechova.  Does mitigating ml’s impact disparity require treatment disparity?  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pages 8136–8146, 2018.
  * [29] Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel.  The variational fair autoencoder.  arXiv preprint arXiv:1511.00830, 2015.
  * [30] Kristian Lum and James Johndrow.  A statistical framework for fair predictive algorithms.  arXiv preprint arXiv:1610.08077, 2016.
  * [31] Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Diptikalyan Saha.  Generate your counterfactuals: Towards controlled counterfactual generation for text.  arXiv preprint arXiv:2012.04698, 2020.
  * [32] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru.  Model cards for model reporting.  In danah boyd and Jamie H. Morgenstern, editors, Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, GA, USA, January 29-31, 2019, pages 220–229. ACM, 2019.
  * [33] Niels JS Morch, Ulrik Kjems, Lars Kai Hansen, Claus Svarer, Ian Law, Benny Lautrup, Steve Strother, and Kelly Rehm.  Visualization of neural networks using saliency maps.  In Proceedings of ICNN’95-International Conference on Neural Networks, volume 4, pages 2085–2090. IEEE, 1995.
  * [34] Chris Russell, Matt J Kusner, Joshua Loftus, and Ricardo Silva.  When worlds collide: integrating different counterfactual assumptions in fairness.  In Advances in Neural Information Processing Systems, pages 6414–6423, 2017.
  * [35] Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley.  No classification without representation: Assessing geodiversity issues in open data sets for the developing world.  arXiv preprint arXiv:1711.08536, 2017.
  * [36] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [37] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  arXiv preprint arXiv:1706.03825, 2017.
  * [38] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3319–3328, 2017.
  * [39] Christina Wadsworth, Francesca Vera, and Chris Piech.  Achieving fairness through adversarial learning: an application to recidivism prediction.  arXiv preprint arXiv:1807.00199, 2018.
  * [40] Michael Wick, swetasudha panda, and Jean-Baptiste Tristan.  Unlocking fairness: a trade-off revisited.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [41] Blake Woodworth, Suriya Gunasekar, Mesrob I Ohannessian, and Nathan Srebro.  Learning non-discriminatory predictors.  arXiv preprint arXiv:1702.06081, 2017.
  * [42] Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021.
  * [43] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi.  Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment.  In Proceedings of the 26th International Conference on World Wide Web, pages 1171–1180. International World Wide Web Conferences Steering Committee, 2017.
  * [44] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi.  Fairness constraints: Mechanisms for fair classification.  arXiv preprint arXiv:1507.05259, 2015.
  * [45] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork.  Learning fair representations.  In International Conference on Machine Learning, pages 325–333, 2013.
  * [46] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell.  Mitigating unwanted biases with adversarial learning.  In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335–340. ACM, 2018.
  * [47] Ran Zmigrod, Sabrina J Mielke, Hanna Wallach, and Ryan Cotterell.  Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1651–1661, 2019.