#  CertiFair: A Framework for Certified Global Fairness of Neural Networks

Haitham Khedr and Yasser Shoukry
Department of Electrical Engineering and Computer Science
University of California, Irvine
Irvine, CA 92697, USA
{hkhedr<EMAIL_ADDRESS>
###### Abstract

We consider the problem of whether a Neural Network (NN) model satisfies
global individual fairness. Individual Fairness (defined in [1]) suggests that
similar individuals with respect to a certain task are to be treated similarly
by the decision model. In this work, we have two main objectives. The first is
to construct a verifier which checks whether the fairness property holds for a
given NN in a classification task or provide a counterexample if it is
violated, i.e., the model is fair if all similar individuals are classified
the same, and unfair if a pair of similar individuals are classified
differently. To that end, We construct a sound and complete verifier that
verifies global individual fairness properties of ReLU NN classifiers using
distance-based similarity metrics. The second objective of this paper is to
provide a method for training provably fair NN classifiers from unfair
(biased) data. We propose a fairness loss that can be used during training to
enforce fair outcomes for similar individuals. We then provide provable bounds
on the fairness of the resulting NN. We run experiments on commonly used
fairness datasets that are publicly available and we show that global
individual fairness can be improved by 96 % without significant drop in test
accuracy.

##  1 Introduction

Neural Networks (NNs) have become an increasingly central component of modern
decision-making systems, including those that are used in sensitive/legal
domains such as crime prediction [2], credit assessment [3], income prediction
[3], and hiring decisions. However, studies have shown that these systems are
prone to biases [4] that deem their usage unfair to unprivileged users based
on their age, race, or gender. The bias is usually either inherent in the
training data or introduced during the training process. Mitigating
algorithmic bias has been studied in the literature [5, 6, 7] in the context
of group and individual fairness. However, the fairness of the NN is
considered only empirically on the test data with the hope that it represents
the underlying data distribution.

Unlike the _empirical_ techniques for fairness, we are interested to provide
_provable certificates_ regarding the fairness of a NN classifier. In
particular, we focus on the “global individual fairness” property which states
that a NN classification model is globally individually fair if _all_
_similar_ pairs of inputs $x$, $x^{\prime}$ are assigned the same class. We
use a feature-wise closeness metric instead of an $\ell_{p}$ norm to evaluate
similarity between individuals, i.e, a pair $x$, $x^{\prime}$ is similar if
for all features $i$, $|x_{i}-x^{\prime}_{i}|\leq\delta_{i}$. Given this
fairness notion, the objective of this paper is twofold. First, it aims to
provide a sound and complete formal verification framework that can
automatically certify whether a NN satisfy the fairness property or produce a
concrete counterexample showing two inputs that are not treated fairly by the
NN. Second, this paper provides a training procedure for certified fair
training of NNs even when the training data is biased.

#### Challenge:

Several existing techniques focus on generalizing ideas from _adversarial
robustness_ to reason about NN fairness [8, 9]. By viewing unfairness as an
adversarial noise that can flip the output of a classifier, these techniques
can certify the fairness of a NN _locally_ , i.e., in the neighborhood of a
given individual input. In contrast, this paper focuses on _global_ fairness
properties where the goal is to ensure that the NN is fair with respect to
_all_ the similar inputs in its domain. Such a fundamental difference
precludes the use of existing techniques from the literature on adversarial
robustness and calls for novel techniques that can provide provable fairness
guarantees.

#### This work:

We introduce CertiFair, a framework for certified global fairness of NNs. This
framework consists of two components. First, a verifier that can prove whether
the NN satisfies the fairness property or produce a concrete counterexample
that violates the fairness property. This verifier is motivated by the recent
results in the “relational verification” problem [10] where the goal is to
verify _hyperproperties_ that are defined over pairs of program traces. Our
approach is based on the observation that the global individual fairness
property (1) can be seen as a _hyperproperty_ and hence we can generalize the
concept of _product programs_ to _product NNs_ that accepts a pair of inputs
$(x,x^{\prime})$, instead of a single input $x$, and generates two independent
outputs for each input. A global fairness property for this product NN can
then be verified using existing NN verifiers. Moreover, and inspired by
methods in certified robustness, we also propose a training procedure for
_certified fairness_ of NNs. Thanks again to the product NN, mentioned above,
one can establish upper bounds on fairness and use it as a regularizer during
the training of NNs. Such a regularizer will promote the fairness of the
resulting model, even if the data used for training is biased and can lead to
an unfair classifier. While such _fairness regularizer_ will enhance the
fairness of the model, one needs to check if the fairness property holds
globally using the sound and complete verifier mentioned above.

#### Contributions:

Our main contributions are:

  * •

To the best of our knowledge, we present the first sound and complete NN
verifier for global individual fairness properties.

  * •

A method for training NN classifiers with a modified loss that enforces fair
outcomes for similar individuals. We provide bounds on the loss in fairness
which constructs a certificate on the fairness of the trained NN.

  * •

We applied our framework to common fairness datasets and we show that global
fairness can be achieved with a minimal loss in performance.

##  2 Preliminaries

Our framework supports regression and multi-class classification models,
however for simplicity, we only present our framework for binary
classification models $h:\mathbb{R}^{n}\rightarrow\\{0,1\\}$ of the form
$h(x)=t(f_{\theta}(x))$ where $t$ is a threshold function with threshold
equals to $0.5$. Moreover, we assume $f_{\theta}$ is an $L$-layer NN with ReLU
hidden activations and parameters
$\theta=\left((W_{1},b_{1}),\ldots,(W_{L},b_{L})\right)$ where $(W_{i},b_{i})$
denotes the weights and bias of the $i$th layer. We also assume the activation
function of the last layer of $f_{\theta}$ is a sigmoid function. The NN
accepts an input vector $x$ where the components $x_{i}\in\mathbb{R}$ (the set
of real numbers) $\text{ or }x_{i}\in\mathbb{Z}$ (the set of integer numbers).
This is suitable for most of the datasets where some features of the input are
numerical while others are categorical. In this paper, we are interested in
the following fairness property:

######  Definition 2.1 (Global Individual Fairness [1, 11]).

A model $f_{\theta}(x)$ is said to satisfy the global individual fairness
property $\phi$ if the following holds:

$\displaystyle\forall x,x^{\prime}\in D_{\phi}\quad\text{s.t.}\quad
d(x,x^{\prime})=1\Longrightarrow h(f_{\theta}(x))=h(f_{\theta}(x^{\prime})),$
(1)

where $d:\mathcal{R}^{n}\times\mathcal{R}^{n}\rightarrow\\{1,0\\}$ is a
similarity metric that evaluates to $1$ when $x$ and $x^{\prime}$ are similar
and $D_{\phi}$ is the input domain of $x$ for property $\phi$ defined as
$D_{\phi}:=D_{\phi}^{0}\times...\times D_{\phi}^{n-1}$ with
$D_{\phi}^{i}:=\\{x_{i}\;|\;l_{i}\leq x_{i}\leq u_{i}\\}$ for some bounds
$l_{i}$ and $u_{i}$.

In this paper, we utilize the feature-wise similarity metric $d$ defined as:

$\displaystyle
d(x,x^{\prime})=\begin{cases}1&\text{if}\;|x_{i}-x^{\prime}_{i}|\leq\delta_{i}\qquad\forall
i\in\\{1,\ldots n\\}\\\ 0&\text{otherwise}\end{cases}$ (2)

This feature-wise similarity metric allows the fairness property $\phi$ in (1)
to capture several other fairness properties as special cases as follows:

######  Definition 2.2 (Individual discrimination [12]).

A model $f_{\theta}(x)$ is said to be nondiscriminatory between individuals if
the following holds:

$\forall x=(x_{s},x_{ns}),x^{\prime}=(x^{\prime}_{s},x^{\prime}_{ns})\in
D_{\phi}\quad\text{s.t.}\quad x_{ns}=x^{\prime}_{ns}\;\text{and}\;x_{s}\neq
x^{\prime}_{s}\Longrightarrow h(f_{\theta}(x))=h(f_{\theta}(x^{\prime})),$

where $x_{s}$ and $x_{ns}$ denotes the sensitive attributes and non-sensitive
attributes of $x$, respectively.

Indeed, the individual discrimination corresponds to a global individual
fairness property by setting $\delta_{i}=0$ in (2) for the non-sensitive
attributes. Another definition of fairness[9] states that two individuals are
similar if their numerical features differ by no more than $\alpha$. Again,
this can be represented by the closeness metric simply by setting
$\delta_{i}=0$ for categorical attributes and $\delta_{i}=\alpha$ for
numerical attributes.

Based on Definition 1 above, we can formally verify whether the fairness
property $\phi$ holds by checking if the set of counterexamples (or
violations) $\mathcal{C}$ is empty, where $\mathcal{C}$ is defined as:

$\mathcal{C}=\left\\{(x,x^{\prime})\Big{\lvert}x,x^{\prime}\in
D_{\phi},\overset{n-1}{\underset{i=0}{\bigwedge}}|x_{i}-x_{i}^{\prime}|<\delta_{i},h(x)\neq
h(x^{\prime})\right\\}=\emptyset\text{.}$ (3)

##  3 Global Individual Fairness as a hyperproperty

In this section, we draw the connection between the verification of global
individual fairness properties (1) and hyperproperties in the context of
program verification. On the one hand, several local properties of NNs (e.g.,
adversarial robustness) are considered trace properties, i.e., properties
defined on the input-output behavior of the NN. In this case, one can search
the input space of the NN to find a _single_ input (or counterexample) that
leads to an output that violates the property. In the domain of adversarial
robustness, a counterexample corresponds to a disturbance to an input that can
change the classification output of a NN. On the other hand, other properties,
like the global fairness properties, can not be modeled as trace properties.
This stems from the fact that one can not judge the correctness of the NN by
considering individual inputs to the NN. Instead, finding a counterexample to
the fairness property will entail searching over _pairs_ of inputs and
comparing the NN outputs of these inputs. Properties that require examining
pairs or sets of traces (input-outputs of a program) are defined as
_hyperproperties_ [10].

Figure 1: Construction of the Product NN.

Modeling global individual fairness as a hyperproperty leads to a direct
certification framework. In particular, a key idea in the hyperproperty
verification literature is the notion of a product program that allows the
reduction of the hyperproperty verification problem to a standard verification
problem [10]. A product program is constructed by composing two copies of the
original program together. The main benefit is that the hyperproperties of the
original program become trace properties of the product program that can be
verified using standard techniques. Motivated by this observation, our
framework CertiFair generalizes the concept of _product programs_ into
_product NNs_ (described in detail in Section 4.2 and shown in Figure 1) that
accepts a pair of inputs and generates a pair of two independent outputs. We
then use the product network to verify fairness (hyper)properties using
standard techniques.

##  4 CertiFair: A Framework for Certified Fairness of Neural Networks

As mentioned earlier in section 3, the fairness property can be viewed as a
hyperproperty of the NN. We propose the use of a product NN that can reduce
the verification of such hyperproperty into standard trace (input/output)
property. In this section, we first explain how to construct the product NN
followed by how to use it to encode the fairness verification problem into
ones that are accepted by off-the-shelf NN verifiers. Next, we discuss how to
use this product NN to derive a fairness regualrizer that can be used during
training to obtain a certified fair NN.

###  4.1 Product Neural Network

Given a neural network $f_{\theta}$, the product network $f_{\theta_{p}}$ is
basically a side-to-side composition of $f_{\theta}$ with itself. More
formally, the parameters vector $\theta_{p}$ of the product NN is defined as:

$\displaystyle\theta_{p}=\left(\left(\begin{bmatrix}W_{1}&\mathbf{0}\\\
\mathbf{0}&W_{1}\end{bmatrix},\begin{bmatrix}b_{1}\\\
b_{1}\end{bmatrix}\right),\ldots,\left(\begin{bmatrix}W_{L}&\mathbf{0}\\\
\mathbf{0}&W_{L}\end{bmatrix},\begin{bmatrix}b_{L}\\\
b_{L}\end{bmatrix}\right)\right)$ (4)

where $(W_{i},b_{i})$ are the weights and biases of the $i$th layer of
$f_{\theta}$. The input to the product network $f_{\theta_{p}}$ is a pair of
concatenated inputs $x_{p}=(x,x^{\prime})$. Finally, we add an output layer
that results in an output $h_{p}\in\\{0,1\\}$ defined as:
$h_{p}(f_{\theta_{p}}(x_{p}))=|h(f_{\theta}(x))-h(f_{\theta}(x^{\prime}))|$
where the absolute value operator $|.|$ can be implemented using ReLU nodes by
noticing that $|a|=max(a,0)+max(-a,0)$. Figure 1 summarizes this construction.

###  4.2 Fairness Verification

Using the product network defined above, we can rewrite the set of
counterexamples $\mathcal{C}$ in (3) as:

$\displaystyle\mathcal{C}_{p}=\left\\{x_{p}\Big{\lvert}x_{p}\in D_{\phi}\times
D_{\phi},\;\overset{n-1}{\underset{i=0}{\bigwedge}}|x_{i}-x_{i}^{\prime}|<\delta_{i},\;h_{p}(x_{p})>0\right\\}$
(5)

which corresponds to the standard verification of NN input-output properties
[13], albeit being defined over the product network inputs and outputs..

To check the emptiness of the set $\mathcal{C}_{p}$ in (5) (and hence certify
the global individual fairness property), we need to search the space
$D_{\phi}\times D_{\phi}$ to find at least one counterexample that violates
the fairness property, i.e., a pair $x_{p}=(x,x^{\prime})$ that represent
similar individuals who are classified differently by the NN. Finding such a
counterexample is, in general, NP-hard due to the non-convexity of the ReLU NN
$f_{\theta_{p}}$. To that end, we use PeregriNN [14] as our NN verifier.
Briefly, PeregriNN overapproximates the highly non-convex NN with a linear
convex relaxation for each ReLU activation. This is done by introducing two
optimization variables for each ReLU, a pre-activation variable $\hat{y}$ and
a post-activation variable $y$. The non-convex ReLU function can then be
overapproximated by a triangular region of three constraints; $y\geq 0$,
$y\geq\hat{y}$, and $y\leq\frac{u}{u-l}(\hat{y}-l)$, where $l\text{,}u$ are
the lower and upper bounds of $\hat{y}$ respectively. The solver tries to
check whether the approximate problem has no solution or iteratively refine
the NN approximation until a counterexample that violates the fairness
constraint is found. PeregriNN employs other optimizations in the objective
function to guide the refinement of the NN approximation but the details of
these methods are beyond the scope of this paper. We refer the reader to the
original paper [14] for more details on the internals of the solver.

######  Proposition 4.1.

Consider a NN model $f_{\theta}$ and a fairness property $\phi$—either
representing a Global Individual Fairness property (Definition 2.1) or an
Individual Discrimination property (Definition 2.2). Consider a set of
counterexamples $\mathcal{C}_{p}$ computed using a NN verifier applied to the
product network $f_{\theta_{p}}$. The NN satisfies the property $\phi$
whenever the set $\mathcal{C}_{p}$ is empty.

###### Proof.

This result follows directly from the equivalence between the sets
$\mathcal{C}$ in (3) and $\mathcal{C}_{p}$ in (5) along with the NN verifiers
(like PeregriNN) being sound and complete and hence capable of finding any
counterexample, if one exists. ∎

###  4.3 Certified Fair Training

In this section, we formalize a _fairness regularizer_ that can be used to
train certified fair models. In particular, we propose two fairness
regularizers that correspond to _local_ and _global_ individual fairness. We
provide the formal definitions of both these regularizers below and their
characteristics.

#### Local Fairness Regularizer $\mathcal{L}_{f}^{l}$ using Robustness around
Training Data:

Our first proposed regularizer is motivated by the _robustness_ regularizers
used in the literature of certified robustness [15, 16]. The regualrizer,
denoted by $\mathcal{L}_{f}^{l}$, aims to minimize the average loss in
fairness across all training data. More formally, given a training point
$(x,y)$ and NN parameters $\theta$, let
$\mathcal{L}(f_{\theta}(x),y;\theta)=-[y\log(f_{\theta}(x))+(1-y)\log(1-f_{\theta}(x))]$
be the standard binary cross-entropy loss. The fairness regularizer
$\mathcal{L}_{f}^{l}$ can then be defined as:

$\displaystyle\mathcal{L}_{f}^{l}(\theta)$
$\displaystyle=\mathbb{E}_{(x,y)\in(X,Y)}\left[\max_{\begin{subarray}{c}x^{\prime}\in
D_{\phi}\\\
d(x,x^{\prime})=1\end{subarray}}\mathcal{L}(f_{\theta}(x^{\prime}),y;\theta)\right]$
(6)

In other words, the regularizer above aims to measure the expected value
(across the training data) for the worst-case loss of fairness due to points
$x^{\prime}$ that are assigned to different classes. Indeed, the regualrizer
(6) is not differentiable (with respect to the weights $\theta$) due to the
existence of the max operator. Nevertheless, one can compute an upper bound of
(6) and aims to minimize this upper bound instead. Such upper bound can be
derived as follows:

$\displaystyle\max_{\begin{subarray}{c}x^{\prime}\in D_{\phi}\\\
d(x,x^{\prime})=1\end{subarray}}\\!\\!\\!\mathcal{L}(f_{\theta}(x^{\prime}),y;\theta)$
$\displaystyle=\\!\\!\\!\\!\\!\max_{\begin{subarray}{c}x^{\prime}\in
D_{\phi}\\\
d(x,x^{\prime})=1\end{subarray}}\\!\\!\\!\begin{cases}-\log(1-f_{\theta}(x^{\prime}))&\text{if}\;y\\!=\\!0\\\
-\log(f_{\theta}(x^{\prime}))&\text{if}\;y\\!=\\!1\\\
\end{cases}\leq\begin{cases}-\log(1-\theta^{T}\overline{w}_{S_{\phi}(x)})&\text{if}\;y\\!=\\!0\\\
-\log(\theta^{T}\underline{w}_{S_{\phi}(x)})&\text{if}\;y\\!=\\!1\\\
\end{cases}$ (7)

where $\theta^{T}\overline{w}_{S_{\phi}(x)}$ and
$\theta^{T}\underline{w}_{S_{\phi}}$ are the linear upper/lower bound of
$f_{\theta}(x^{\prime})$ inside the set $S_{\phi}(x)=\\{x^{\prime}\in
D_{\phi}|d(x,x^{\prime})=1\\}$. Such linear upper/lower bound of
$f_{\theta}(x^{\prime})$ can be computed using off-the-shelf bounding
techniques like Symbolic Interval Analysis [17] and $\alpha$-Crown [18]. We
denote by $\overline{\mathcal{L}}(y;\theta)$ the right hand side of the
inequality in (7) which depends only on the label $y$ and the NN parameters
$\theta$. Now the fairness property can be incorporated in training by
optimizing the following problem over $\theta$ (the NN parameters):

$\min_{\theta}\mathop{\mathbb{E}}_{(x,y)\in(X,y)}\left[(1-\lambda_{f})\underbrace{\mathcal{L}(f_{\theta}(x),y;\theta)}_{\text{natural
loss}}+\lambda_{f}\underbrace{\overline{\mathcal{L}}(y;\theta)}_{\text{local
fairness loss}}\right]\text{,}$ (8)

where $\lambda_{f}$ is a regularization parameter to control the trade-off
between fairness and accuracy.

Although easy to compute and incorporate in training, the regularizer
$\mathcal{L}_{f}^{l}(\theta)$ (and its upper bound) defined above suffers from
a significant drawback. It focuses on the fairness _around_ the samples
presented in the training data. In other words, although the aim was to
promote _global_ fairness, this regularizer is effectively penalizing the
training only in the _local_ neighborhood of the training data. Therefore, its
effectiveness depends greatly on the quality of the training data and its
distribution. Poor data distribution may lead to the poor effect of this
regularizer. Next, we introduce another regularizer that avoids such problems.

#### Global Fairness Regularizer $\mathcal{L}_{f}^{g}$ using Product Network:

To avoid the dependency on data, we introduce a novel fairness regularizer
capable of capturing global fairness during the training. Such a regularizer
is made possible thanks to the product NN defined above. In particular, the
global fairness regularizer $\mathcal{L}_{f}^{g}(\theta)$ is defined as:

$\displaystyle\mathcal{L}_{f}^{g}(\theta)$
$\displaystyle=\max_{\begin{subarray}{c}(x,x^{\prime})\in D_{\phi}\times
D_{\phi}\\\
d(x,x^{\prime})=1\end{subarray}}|f_{\theta}(x)-f_{\theta}(x^{\prime})|$ (9)

In other words, the regualizer $\mathcal{L}_{f}^{g}(\theta)$ in (9) aims to
penalize the worst case loss in global fairness. Similar to (6), the
$\mathcal{L}_{f}^{g}(\theta)$ is also non-differentiable with respect to
$\theta$. Nevertheless, and thanks to the product NN, we can upper bound
$\mathcal{L}_{f}^{g}(\theta)$ as:

$\displaystyle\mathcal{L}_{f}^{g}(\theta)\leq\max_{(x,x^{\prime})\in
D_{\phi}\times
D_{\phi}}|f_{\theta}(x)-f_{\theta}(x^{\prime})|=\max_{\begin{subarray}{c}x_{p}\in
D_{\phi}\times
D_{\phi}\end{subarray}}f_{p}(x_{p})\leq\theta^{T}\overline{w}_{D_{\phi}}$ (10)

where $\theta^{T}\overline{w}_{D_{\phi}}$ is the linear upper bound of the
product network among the domain $D_{\phi}\times D_{\phi}$. Again, such bound
can be computed using Symbolic Interval Analysis and $\alpha$-Crown on the
product network after replacing the output $h_{p}$ with
$f_{p}=|f_{\theta}(x)-f_{\theta}(x^{\prime})|$. It is crucial to note that the
upper bound in (10) depends only on the domain $D_{\phi}$. Hence, the fairness
property can now be incorporated into training by minimizing this upper bound
as:

$\min_{\theta}\mathop{\mathbb{E}}_{(x,y)\in(X,y)}\left[(1-\lambda_{f})\underbrace{\mathcal{L}(f_{\theta}(x),y;\theta)}_{\text{natural
loss}}\right]+\lambda_{f}\underbrace{\theta^{T}\overline{w}_{D_{\phi}}}_{\text{global
fairness loss}}\text{,}$ (11)

where the fairness loss is now outside the $\mathbb{E}[\;.\;]$ operator.

In the next section, we show that the global fairness regularizer
$\mathcal{L}_{f}^{g}(\theta)$ empirically outperforms the local fairness
regularizer $\mathcal{L}_{f}^{l}(\theta)$. We end up our discussion in this
section with the following result:

######  Proposition 4.2.

Consider a NN model $f_{\theta}$ and a fairness property $\phi$—either
representing a Global Individual Fairness property (Definition 2.1) or an
Individual Discrimination property (Definition 2.2). Consider a NN model
$f_{\theta}$ trained using the objective function in (11). If
$\theta^{T}\overline{w}_{D_{\phi}}=0$ by the end of the training, then the
resulting $f_{\theta}$ is guaranteed to satisfy $\phi$.

###### Proof.

The result follows directly from equation (10). ∎

Indeed, the result above is just a sufficient condition. In other words, the
NN may still satisfy the fairness property $\phi$ even if
$\theta^{T}\overline{w}_{D_{\phi}}>0$. Such cases can be handled by applying
the verification procedure in Section 4.2 after training the NN.

##  5 Experimental evaluation

We present an experimental evaluation to study the effect of our proposed
fairness regularizers and hyperparameters on the global fairness. We evaluated
CertiFair on four widely investigated fairness datasets (Adult [3], German
[3], Compas [19], and Law School [20]). All datasets were pre-processed such
that any missing rows or columns were dropped, features were scaled so that
they’re between $[0,1]$, and categorical features were one-hot encoded.

Implementation: We implemented our framework in a Python tool called CertiFair
that can be used for training and verification of NNs against an individual
fairness property. CertiFair uses Pytorch for all NN training and a publicly
available implementation of PeregriNN [14] as a NN verifier. We run all our
experiments using a single GeForce RTX 2080 Ti GPU and two 24-core Intel(R)
Xeon(R) CPU E5-2650 v4 @ 2.20GHz (only 8 cores were used for these
experiments).

Table 1: Comparison between verifying local and global fairness properties on
the Adult dataset for different similarity constraint (distance $\delta_{i}$).

$\delta_{i}$ | Certified | Certified
---|---|---
| local | global
| fairness | fairness
| (%) | (%)
0.02 | 89.25 | 6.56
0.03 | 100.00 | 65.98
0.05 | 81.42 | 6.40
0.07 | 100.00 | 57.39
0.1 | 99.95 | 66.35

Measuring global fairness using verification: While the verifier (described in
Section 4.2) is capable of finding concrete counterexamples that violate the
fairness, it is also important to _quantify_ a bound on the fairness. In these
experiments, the certified global fairness is quantified as the percentage of
partitions of the input space with zero counterexamples. In particular, the
input space is partitioned using the categorical features, i.e., the number of
partitions is equal to the number of different categorical assignments and
each partition corresponds to one categorical assignment. Note that the
numerical features don’t have to be fixed inside each partition (property
dependant). To verify the property globally, we run the verifier on each
partition of the input space and verify the fairness property. Finally, we
count the number of verified fair partitions (normalized by the total number
of partitions).

Fairness properties: In the experimental evaluation, we consider two classes
of fairness properties. The first class $\mathcal{P}_{1}$ is the one in
definition 2.2 where two individuals are similar if they only differ in their
sensitive attribute. The second class of properties $\mathcal{P}_{2}$ relaxes
the first by also allowing numerical attributes to be close (not identical),
this is allowed under definition 2.1 of global individual fairness by setting
$\delta_{i}>0$ for numerical attributes.

###  5.1 Experiment 1: Global Individual Fairness vs. Local Individual
Fairness

In this experiment, we empirically show that NNs with high _local_ individual
fairness does not necessarily result into NNs with _global_ individual
fairness. In particular, we train a NN on the Adult dataset and considered
multiple fairness properties (all from class $\mathcal{P}_{2}$ defined above)
by varying $\delta_{i}$. Note that $\delta_{i}$ is equal for all features $i$
within the same property, but is different from one property to another. Next,
we use PeregeriNN verifier to find counterexamples for both the _local_
fairness (by applying the verifier to the trained NN) and the _global_
fairness (by applying the verifier to the product NN). We measure the fairness
of the NN for both cases and report the results in Table 1. The results
indicate that verifying _local_ fairness may result in incorrect conclusions
about the fairness of the model. In particular, rows1-4 in the table shows
that counterexamples were not found in the neighborhood of the training data
(reflected by the $100\%$ certified local fairness), yet verifying the product
NN was capable of finding counterexamples that are far from the training data
leading to accurate conclusions about the NN fairness.

Figure 2: Comparison between the base and CertiFair classifiers in terms of
fairness as defined in 2.2. We show the classifications for the minority group
of the adult dataset projected on two features; Age, and hours worked per
week. The figure shows that the base classifier suffers from biases against
identical individuals who are of different race (red markers). CertiFair is
able to drastically improve the individual fairness on this dataset with only
$2\%$ reduction in accuracy. Figure 3: Test accuracy (left) and certified
fairness (right) across training epochs when training a NN with the fairness
constraint ($\lambda_{f}=0.007$) and without it ($\lambda_{f}=0)$ on the Adult
dataset.

###  5.2 Experiment 2: Effects of Incorporating the Fairness Regularizer

We investigate the effect of using the global fairness regularizer (defined in
(10)) on the decisions of the NN classifier when trained on the Adult dataset.
The fairness property for this experiment is of class $\mathcal{P}_{1}$. To
investigate the predictor’s bias, we first project the data points on two
numerical features (age and hours/week). Our objective is to check whether the
points that are classified positively for the privileged group are also
classified positively for the non privileged group. Figure 2 (left) shows the
predictions for the unprivileged group when using the base classifier
($\lambda_{f}=0$). Green markers indicate points for which individuals from
both privileged and non privileged groups are treated equally while red
markers show individuals from the non privileged group that did not receive
the same NN output compared to the corresponding identical ones in the
privileged group. Figure 2 (right) shows the same predictions but using the
fair classifier ($\lambda_{f}=0.03$), the predictions from this classifier
drastically decreased the discrimination between the two groups while only
decreasing the accuracy by 2%. These results suggest that we can indeed
regularize the training to improve the satisfaction of the fairness constraint
without a drastic change in performance.

We also investigate how the certified fairness changes across epochs of
training. To that end, we train a NN for the German dataset and evaluate the
test accuracy as well as the certified global fairness after each epoch of
training for two different values of $\lambda_{f}$. Figure 3 shows the
underlying trade-off between achieving fairness versus maximizing accuracy. As
expected, lower values of $\lambda_{f}$ results in lower loss in accuracy
(compared to the base case with $\lambda_{f}=0$) while having lower effect on
the fairness. The results also show that a small sacrifice of the accuracy can
lead to significant enhancement of the fairness as shown for the
$\lambda_{f}=0.007$ case.

###  5.3 Experiment 3: Certified Fair Training

Experiment setup: The objective of this experiment is to compare the two
regularizers, the local fairness regularizer in (7) and the global fairness
regularizer in (10). To that end, we performed a grid search over the learning
rate $\alpha$, the fairness regularization parameter $\lambda_{f}$, and the NN
architecture to get the best test accuracy across all datasets. Best
performance was obtained with a NN that consists of two hidden layers of 20
neurons (except for the German dataset, where we use 30 neurons per layer),
learning rate $\alpha=0.001$, global fairness regularization parameter
$\lambda_{f}$ equal to 0.01 for Adult and Law School, 0.005 for German, and
0.1 for Compas dataset, and local fairness regularization parameter
$\lambda_{f}$ equal to 0.95 for Adult, 0.9 for Compas, 0.2 for German, and 0.5
for Law School. We trained the models for 50 epochs with a batch size of 256
(except for Law School, where the batch size is set to 1024) and used Adam
Optimizer for learning the NN parameters $\theta$. All the datasets were split
into $70\%$ and $30\%$ for training and testing, respectively.

Table 2: Comparison between a base classifier ($\lambda_{f}=0$) and CertiFair
classifier with different fairness regularizers

Constraint | Dataset | Test Accuracy (%) | Positivity Rate(%) |  | Certified
---
Global Fairness (%)
| Base |  |  $\mathcal{L}_{f}^{g}$
---
|  $\mathcal{L}_{f}^{l}$
---
Base |  |  $\mathcal{L}_{f}^{g}$
---
|  $\mathcal{L}_{f}^{l}$
---
Base |  |  $\mathcal{L}_{f}^{g}$
---
|  $\mathcal{L}_{f}^{l}$
---
$\mathcal{P}_{2}$ | Adult | 84.55 | 82.34 | 83.81 | 20.8 | 17.4 | 21.79 | 6.40 | 100.00 | 61.92
German | 75.30 | 73.00 | 70.00 | 79.00 | 72.00 | 83.00 | 8.64 | 95.06 | 86.41
Compas | 68.30 | 65.08 | 63.19 | 61.55 | 66.00 | 69.40 | 47.22 | 100.00 | 100.00
Law | 87.60 | 79.92 | 78.69 | 21.51 | 9.10 | 25.03 | 6.87 | 51.87 | 78.54
$\mathcal{P}_{1}$ | Adult | 84.55 | 82.33 | 83.25 | 20.80 | 18.13 | 20.15 | 1.77 | 97.86 | 95.31
German | 75.30 | 72.66 | 69.66 | 79.00 | 83.14 | 81.71 | 14.81 | 92.59 | 82.71
Compas | 68.30 | 65.08 | 63.82 | 61.55 | 66.00 | 71.60 | 47.22 | 100.00 | 100.00
Law | 87.60 | 84.90 | 86.69 | 21.42 | 17.50 | 21.63 | 34.16 | 70.10 | 86.45

Effect of the choice of the fairness regularizer: We investigate the certified
global fairness for the two regularizers introduced in Section 4.3. Table 2
summarizes the results for $\mathcal{P}_{1}$ and $\mathcal{P}_{2}$ fairness
properties across different datasets. For each property and dataset, we
compare the test accuracy, positivity rate (percentage of points classified as
1), and the certified global fairness of the base classifier (trained with
$\lambda_{f}=0$) and the CertiFair classifier trained twice with two different
fairness regularizers $\mathcal{L}_{f}^{l}$ and $\mathcal{L}_{f}^{g}$.
Compared to the base classifier, training the NN with the global fairness
regularizer $\mathcal{L}_{f}^{g}$ significantly increases the certified global
fairness with a small drop in the accuracy in most of the cases except for the
Law School dataset, where the test accuracy dropped by 7 % on
$\mathcal{P}_{2}$ but the global fairness increased by 55 %. Compared to the
local regularizer $\mathcal{L}_{f}^{l}$, the global regularizer achieves
higher global fairness and comparable (if not better) test accuracy on all
datasets except Law School. We think that this might be due to the network’s
limited capacity to optimize both objectives. We also report the positivity
rate (number of data points classified positively) for the classifiers. This
metric is important because most of these datasets are unbalanced, and hence
the classifiers can trivially skew all the classifications to a single label
and achieve high fairness percentage. Thus it is desired that the positivity
rate of the CertiFair classifier to be close to the one of the base classifier
to ensure that it is not trivial. Lastly, we conclude that even though the
local regularizer improves the global fairness, the global regularizer can
achieve higher degrees of certified global fairness without a significant
decrease in test accuracy, and of course, it avoids the drawbacks of the local
regularizer discussed in Section 4.3.

Effect of the fairness regularization parameter: In this experiment, we
investigate the effect of the fairness regularization parameter $\lambda_{f}$
on the classifier’s accuracy and fairness. The parameter $\lambda_{f}$
controls the trade-off between the accuracy of the classifier and its
fairness, and tuning this parameter is usually dependent on the
network/dataset. To that end, we trained a two-layer NN with 30 neurons per
layer for the German dataset using 8 different values for $\lambda_{f}$ and
summarized the results in Table 3. The fairness property verified is of class
$\mathcal{P}_{2}$. The results show that the global fairness satisfaction can
increase without a significant drop in accuracy up to a certain point, after
which the fairness loss is dominant and results in a significant decrease in
the classifier’s accuracy.

Table 3: Effect of fairness regularization parameter $\lambda_{f}$ on the test
accuracy and certified fairness.

$\lambda_{f}$ | $1\times 10^{-4}$ | $5\times 10^{-4}$ | $7\times 10^{-4}$ | $5\times 10^{-3}$ | $7\times 10^{-3}$ | $1\times 10^{-2}$ | $2\times 10^{-2}$ | $5\times 10^{-2}$
---|---|---|---|---|---|---|---|---
Test accuracy (%) | 74.33 | 74.33 | 75.00 | 72.33 | 72.66 | 73.00 | 72.6 | 66.33
Certified global fairness (%) | 8.64 | 14.81 | 13.58 | 66.66 | 82.71 | 95.06 | 98.76 | 100

##  6 Related work

Group fairness: Group fairness considers notions like demographic parity [21],
equality of odds, and equality of opportunity [22]. Tools that verify notions
of group fairness assume knowledge of a probabilistic model of the population.
FairSquare [23] relies on numerical integration to formally verify notions of
group fairness; however, it does not scale well for NNs. VeriFair [24]
considers probabilistic verification using sampling and provides soundness
guarantees using concentration inequalities. This approach is scalable to big
networks, but it does not provide worst-case proof.

Individual fairness: More related to our work is the verification of
individual fairness properties. LCIFR [9] proposes a technique to learn fair
representations that are provably certified for a given individual. An encoder
is _empirically_ trained to map similar individuals to be within the
neighborhood of the given individual and then apply NN verification techniques
to this neighborhood to certify fairness. The property verified is a _local_
property with respect to the given individual. On the contrary, our work
focuses on the global fairness properties of a NN. It also avoids the
empirical training of similarity maps to avoid affecting the soundness and
completeness of the proposed framework. In the context of individual global
fairness, a recent work [11] proposed a sound but incomplete verifier for
linear and kernelized polynomial/radial basis function classifiers. It also
proposed a meta-algorithm for the global individual fairness verification
problem; however, it is not clear how it can be used to design sound and
complete NN verifiers for the fairness properties. Another line of work [25]
focuses on proving dependency fairness properties which is a more restrictive
definition of fairness since it requires the NN outputs to avoid any
dependence on the sensitive attributes. The method employs forward and
backward static analysis and input space partitioning to verify the fairness
property. As mentioned, this definition of fairness is different from the
individual fairness we are considering in this work and is more restrictive.

NN verification: This work is algorithmically related to NN verification in
the context of adversarial robustness. However, adversarial robustness is a
local property of the network given a nominal input, and a norm bounded
perturbation. Moreover, the robustness property does not consider the notion
of sensitive attributes. The NN verification literature is extensive, and the
approaches can be grouped into three main groups: (i) SMT-based methods, which
encode the problem into a Satisfiability Modulo Theory problem [26, 27, 28];
(ii) MILP-based solvers, which solves the verification problem exactly by
encoding it as a Mixed Integer Linear Program [29, 30, 31, 32, 33, 34, 35];
(iii) Reachability based methods [36, 37, 38, 39, 40, 41, 42], which perform
layer-by-layer reachability analysis to compute a reachable set that can be
verified against the property; and (iv) convex relaxations methods [17, 43,
15, 44, 14, 45]. Generally, (i), (ii), and (iii) do not scale well to large
networks. On the other hand, convex relaxation methods use a branch and bound
approach to refine the abstraction.

##  7 Discussion:

#### On the contention between Group and Individual fairness:

Group fairness is the requirement that different groups should be treated
similarly regardless of individual merits. It is often thought of as a
contradictory requirement to individual fairness. However, this has been an
issue of debate [46]. Thus, it’s not clear how our framework for training
might affect the group fairness requirement and is left for further
investigation.

#### Can fairness be achieved by dropping sensitive attributes from data?

Fairness through unawareness is the process of learning a predictor that does
not explicitly use sensitive attributes in the prediction process. However,
dropping the sensitive attributes is not sufficient to remove discrimination
as it can be highly predictable implicitly from other features. It has been
shown [47, 48, 49, 50] that discrimination highly occurs in different systems
such as housing, criminal justice, and education that do not explicitly depend
on sensitive attributes in their predictions.

## References

  * [1] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, “Fairness through awareness,” in Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214–226, 2012.
  * [2] T. Brennan, W. Dieterich, and B. Ehret, “Evaluating the predictive validity of the compas risk and needs assessment system,” Criminal Justice and behavior, vol. 36, no. 1, pp. 21–40, 2009.
  * [3] D. Dua and C. Graff, “UCI machine learning repository,” 2017.
  * [4] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, “A survey on bias and fairness in machine learning,” ACM Computing Surveys (CSUR), vol. 54, no. 6, pp. 1–35, 2021.
  * [5] B. H. Zhang, B. Lemoine, and M. Mitchell, “Mitigating unwanted biases with adversarial learning,” in Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335–340, 2018.
  * [6] D. Xu, S. Yuan, L. Zhang, and X. Wu, “Fairgan: Fairness-aware generative adversarial networks,” in 2018 IEEE International Conference on Big Data (Big Data), pp. 570–575, IEEE, 2018.
  * [7] N. Mehrabi, U. Gupta, F. Morstatter, G. V. Steeg, and A. Galstyan, “Attributing fair decisions with attention interventions,” arXiv preprint arXiv:2109.03952, 2021.
  * [8] M. Yurochkin, A. Bower, and Y. Sun, “Training individually fair ml models with sensitive subspace robustness,” arXiv preprint arXiv:1907.00020, 2019.
  * [9] A. Ruoss, M. Balunovic, M. Fischer, and M. Vechev, “Learning certified individually fair representations,” Advances in Neural Information Processing Systems, vol. 33, pp. 7584–7596, 2020.
  * [10] G. Barthe, J. M. Crespo, and C. Kunz, “Relational verification using product programs,” in International Symposium on Formal Methods, pp. 200–214, Springer, 2011.
  * [11] P. G. John, D. Vijaykeerthy, and D. Saha, “Verifying individual fairness in machine learning models,” in Conference on Uncertainty in Artificial Intelligence, pp. 749–758, PMLR, 2020.
  * [12] A. Aggarwal, P. Lohia, S. Nagar, K. Dey, and D. Saha, “Black box fairness testing of machine learning models,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, (New York, NY, USA), p. 625–635, Association for Computing Machinery, 2019.
  * [13] C. Liu, T. Arnon, C. Lazarus, C. Barrett, and M. J. Kochenderfer, “Algorithms for Verifying Deep Neural Networks,” 2019.
  * [14] H. Khedr, J. Ferlez, and Y. Shoukry, “Peregrinn: Penalized-relaxation greedy neural network verifier,” in Computer Aided Verification (A. Silva and K. R. M. Leino, eds.), (Cham), pp. 287–300, Springer International Publishing, 2021.
  * [15] E. Wong and J. Z. Kolter, “Provable defenses against adversarial examples via the convex outer adversarial polytope,” 2017.
  * [16] H. Zhang, H. Chen, C. Xiao, S. Gowal, R. Stanforth, B. Li, D. Boning, and C.-J. Hsieh, “Towards stable and efficient training of verifiably robust neural networks,” arXiv preprint arXiv:1906.06316, 2019.
  * [17] S. Wang, K. Pei, J. Whitehouse, J. Yang, and S. Jana, “Efficient formal safety analysis of neural networks,” in Advances in Neural Information Processing Systems (S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds.), vol. 31, pp. 6367–6377, 2018.
  * [18] K. Xu, H. Zhang, S. Wang, Y. Wang, S. Jana, X. Lin, and C.-J. Hsieh, “Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers,” arXiv preprint arXiv:2011.13824, 2020.
  * [19] J. Angwin, J. Larson, S. Mattu, and L. Kirchner, “How we analyzed the compas recidivism algorithm,” 2016.
  * [20] L. F. Wightman, LSAC national longitudinal bar passage study.  Law School Admission Council, 1998.
  * [21] M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian, “Certifying and removing disparate impact,” in proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 259–268, 2015.
  * [22] M. Hardt, E. Price, and N. Srebro, “Equality of opportunity in supervised learning,” Advances in neural information processing systems, vol. 29, 2016\.
  * [23] A. Albarghouthi, L. D’Antoni, S. Drews, and A. V. Nori, “Fairsquare: Probabilistic verification of program fairness,” Proc. ACM Program. Lang., vol. 1, oct 2017.
  * [24] O. Bastani, X. Zhang, and A. Solar-Lezama, “Probabilistic verification of fairness properties via concentration,” Proceedings of the ACM on Programming Languages, vol. 3, no. OOPSLA, pp. 1–27, 2019.
  * [25] C. Urban, M. Christakis, V. Wüstholz, and F. Zhang, “Perfectly parallel fairness certification of neural networks,” Proc. ACM Program. Lang., vol. 4, nov 2020.
  * [26] G. Katz, D. A. Huang, D. Ibeling, K. Julian, C. Lazarus, R. Lim, P. Shah, S. Thakoor, H. Wu, A. Zeljić, et al., “The marabou framework for verification and analysis of deep neural networks,” in Computer Aided Verification (I. Dillig and S. Tasiran, eds.), pp. 443–452, Springer International Publishing, 2019.
  * [27] G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer, “Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks,” in Computer Aided Verification (R. Majumdar and V. Kunčak, eds.), Lecture Notes in Computer Science, pp. 97–117, Springer International Publishing, 2017.
  * [28] R. Ehlers, “Formal verification of piece-wise linear feed-forward neural networks,” in International Symposium on Automated Technology for Verification and Analysis (D. D’Souza and K. N. Kumar, eds.), pp. 269–286, Springer, 2017.
  * [29] A. Lomuscio and L. Maganti, “An approach to reachability analysis for feed-forward relu neural networks,” 2017.
  * [30] V. Tjeng, K. Xiao, and R. Tedrake, “Evaluating robustness of neural networks with mixed integer programming,” 2017.
  * [31] O. Bastani, Y. Ioannou, L. Lampropoulos, D. Vytiniotis, A. Nori, and A. Criminisi, “Measuring neural net robustness with constraints,” in Advances in Neural Information Processing Systems, vol. 29, pp. 2613–2621, 2016\.
  * [32] R. Bunel, J. Lu, I. Turkaslan, P. Kohli, P. Torr, and P. Mudigonda, “Branch and bound for piecewise linear neural network verification,” Journal of Machine Learning Research, vol. 21, no. 42, pp. 1–39, 2020.
  * [33] M. Fischetti and J. Jo, “Deep neural networks and mixed integer linear optimization,” Constraints, vol. 23, no. 3, pp. 296–309, 2018.
  * [34] R. Anderson, J. Huchette, W. Ma, C. Tjandraatmadja, and J. P. Vielma, “Strong mixed-integer programming formulations for trained neural networks,” Mathematical Programming, vol. 183, no. 1, pp. 3–39, 2020.
  * [35] C.-H. Cheng, G. Nührenberg, and H. Ruess, “Maximum resilience of artificial neural networks,” in Automated Technology for Verification and Analysis (D. D’Souza and K. Narayan Kumar, eds.), pp. 251–268, Springer, 2017.
  * [36] W. Xiang, H.-D. Tran, and T. T. Johnson, “Reachable set computation and safety verification for neural networks with relu activations,” 2017.
  * [37] W. Xiang, H.-D. Tran, and T. T. Johnson, “Output reachable set estimation and verification for multilayer neural networks,” IEEE transactions on neural networks and learning systems, vol. 29, no. 11, pp. 5777–5783, 2018.
  * [38] T. Gehr, M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. Vechev, “AI2: Safety and robustness certification of neural networks with abstract interpretation,” in 2018 IEEE Symposium on Security and Privacy (SP), pp. 3–18, IEEE, 2018.
  * [39] S. Wang, K. Pei, J. Whitehouse, J. Yang, and S. Jana, “Formal security analysis of neural networks using symbolic intervals,” in Proceedings of the 27th USENIX Conference on Security Symposium, SEC’18, pp. 1599–1614, USENIX Association, 2018.
  * [40] H.-D. Tran, X. Yang, D. Manzanas Lopez, P. Musau, L. V. Nguyen, W. Xiang, S. Bak, and T. T. Johnson, “Nnv: The neural network verification tool for deep neural networks and learning-enabled cyber-physical systems,” in Computer Aided Verification (S. K. Lahiri and C. Wang, eds.), pp. 3–17, Springer International Publishing, 2020.
  * [41] R. Ivanov, J. Weimer, R. Alur, G. J. Pappas, and I. Lee, “Verisig: verifying safety properties of hybrid systems with neural network controllers,” in Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, HSCC ’19, (New York, NY, USA), pp. 169–178, Association for Computing Machinery, 2019.
  * [42] M. Fazlyab, A. Robey, H. Hassani, M. Morari, and G. Pappas, “Efficient and accurate estimation of lipschitz constants for deep neural networks,” in Advances in Neural Information Processing Systems (H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, eds.), vol. 32, pp. 11423–11434, Curran Associates, Inc., 2019\.
  * [43] K. Dvijotham, R. Stanforth, S. Gowal, T. A. Mann, and P. Kohli, “A dual approach to scalable verification of deep networks.,” in Uncertainty in Artificial Intelligence (A. Globerson and R. Silva, eds.), vol. 1, pp. 550–559, 2018.
  * [44] P. Henriksen and A. Lomuscio, “Deepsplit: An efficient splitting method for neural network verification via indirect effect analysis,”
  * [45] S. Wang, H. Zhang, K. Xu, X. Lin, S. Jana, C.-J. Hsieh, and J. Z. Kolter, “Beta-crown: Efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification,” arXiv preprint arXiv:2103.06624, 2021.
  * [46] R. Binns, “On the apparent conflict between individual and group fairness,” in Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 514–524, 2020.
  * [47] E. Bonilla-Silva, Racism without racists: Color-blind racism and the persistence of racial inequality in the United States.  Rowman & Littlefield Publishers, 2006.
  * [48] A. E. Taslitz, “Racial blindsight: The absurdity of color-blind criminal justice,” Ohio St. J. Crim. L., vol. 5, p. 1, 2007.
  * [49] D. Pedreshi, S. Ruggieri, and F. Turini, “Discrimination-aware data mining,” in Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’08, (New York, NY, USA), p. 560–568, Association for Computing Machinery, 2008.
  * [50] Amazon, “Amazon doesn’t consider the race of its customers. should it?.”