# Bias Mitigation Methods for Binary Classification Decision-Making Systems:
Survey and Recommendations

Madeleine Waller, Odinaldo Rodrigues, Oana Cocarascu
Department of Informatics, King’s College London, UK
{madeleine.waller, odinaldo.rodrigues<EMAIL_ADDRESS>

###### Abstract

Bias mitigation methods for binary classification decision-making systems have
been widely researched due to the ever-growing importance of designing fair
machine learning processes that are impartial and do not discriminate against
individuals or groups based on protected personal characteristics. In this
paper, we present a structured overview of the research landscape for bias
mitigation methods, report on their benefits and limitations, and provide
recommendations for the development of future bias mitigation methods for
binary classification.

##  1 Introduction

Artificial Intelligence (AI), and in particular machine learning (ML)
techniques, have been used in decision-making systems which typically rely on
historical data for training. However, the data may contain biases against
groups or individuals with certain characteristics, which can lead to
discriminatory or unfair decisions. Such decision-making systems are used in
both private and public sectors. Organisations are increasingly relying on
these systems to reduce workload and free up resources (e.g. Deloitte, 2021;
Engin and Treleaven, 2019), and local governments, in particular social
services,111https://www.local.gov.uk/using-predictive-analytics-local-public-
services have deployed systems to predict a score of how at risk a child is of
neglect or abuse (Church and Fairchild, 2017). The potential harmful impact of
these systems is therefore immense.

There have been several recent examples of unfair decisions made by machine
learning models in different domains, such as criminal justice (Partnership on
AI, 2019), recruitment,222https://www.reuters.com/article/us-amazon-com-jobs-
automation-insight-idUSKCN1MK08G and social services (Gillingham, 2019).
Algorithms used to predict if criminals will re-offend are used across many US
states and yet an analysis of a popular tool, COMPAS (Northpointe Inc., 2019),
showed that black defendants were identified incorrectly as re-offending at a
higher rate than white defendants (Larson et al., 2016). A recruitment tool
used by Amazon was shown to be biased against women (Jeffrey Dastin, 2018),
reinforcing historical biases due to the small sample size of women who had
previously been hired.

Ensuring AI systems are fair to individuals and communities is an important
cross-disciplinary issue which must consider the context and application of
the systems deployed (Waller and Waller, 2020). A system is considered to be
fair if it does not discriminate based on protected personal characteristics
such as race, sex, religion, etc. Discrimination may arise from the dataset
used to train decision-making systems; specifically, unwanted bias occurs when
the system favours or gives advantage to historically favoured groups or its
outputs directly correlate with protected personal characteristics. Mehrabi et
al. (2021) identify several causes of unwanted bias including: _i) historical
bias_ where the training data includes embedded historical social biases, _ii)
representation bias_ where some groups with certain characteristics appear
more frequently in the dataset, and _iii) measurement bias_ which arises from
how certain features are measured or used.

Within the field of fair machine learning, several bias mitigation methods
have been proposed, targeting the removal of unwanted bias from training data
(e.g. Zliobaite et al., 2011; Kamiran and Calders, 2011; Feldman et al., 2015;
Iosifidis and Ntoutsi, 2018; Calders et al., 2009), the training algorithm
(e.g. Hu et al., 2020; Iosifidis and Ntoutsi, 2019; Oneto et al., 2019;
Krasanakis et al., 2018; Kamiran et al., 2010; Calders and Verwer, 2010; Grari
et al., 2019), or the decisions made by a system (e.g. Fish et al., 2016;
Kamiran et al., 2012, 2010; Lohia et al., 2019). These bias mitigation methods
use different techniques and approaches and most rely on fairness metrics
which are used to measure the amount of unwanted bias in a decision-making
system. Additionally, the aim of bias mitigation methods is to optimise for
these fairness metrics without decreasing the performance of a system (Fish et
al., 2016; Zliobaite, 2015).

In this paper, we present a structured overview of bias mitigation methods for
binary classification decision-making systems, analyse their benefits and
limitations, summarise datasets as well as fairness and performance metrics
used by these methods, and highlight directions for future work. Our
contributions are as follows:

  1. 1.

We survey existing bias mitigation methods and report their limitations and
benefits, datasets used, and the metrics used in evaluation.

  2. 2.

We review the limitations highlighted by existing methods and make
recommendations for future bias mitigation methods for binary classification
decision-making systems.

The rest of the paper is organised as follows. In Section 2 we provide the
necessary background, including types of bias mitigation methods, datasets,
and metrics used, and give an overview on existing, related surveys. In
Section 3, we survey existing bias mitigation methods and compare their
strengths and limitations. In Section 4 we provide recommendations for the
development of future bias mitigation methods for binary classification
decision-making systems based on the limitations identified, and conclude in
Section 5.

##  2 Background

In this section, we present some key concepts, the different types of bias
mitigation methods, as well as the main datasets, fairness and performance
metrics used to evaluate the methods, and report on related work.

###  2.1 Terminology

Protected personal characteristics or simply _protected attributes_ , are
features that define groups within populations which historically have
different levels of privilege, but which should receive equal treatment
(Bellamy et al., 2018; Chakraborty et al., 2020). These are often generalised
as the features outlined in the US Equal Credit Opportunity
Act333https://www.justice.gov/crt/equal-credit-opportunity-act-3 and Fair
Housing Act444https://www.justice.gov/crt/fair-housing-act-1 such as race,
sex, religion etc., but are specific to the context and application of the
decision-making model being used.

_Proxy attributes_ are non-protected attributes that correspond to a protected
attribute (Le Quy et al., 2022). These attributes may correlate with a
protected attribute and may result in reinforcing stereotypes. For example, a
system that decides the cost of an individual’s insurance should not make its
decisions based on the protected attribute of “race”; however it may make a
decision partly based on the attributes “address” and “smoker/non-smoker” (Van
Nuenen et al., 2020) which may have strong correlations with the attribute
“race”.

Bias mitigation methods aim to ensure a system’s decision does not
discriminate based on the protected or proxy attributes. In the rest of the
paper, we will refer to protected and proxy attributes as _sensitive
attributes_. An individual is said to be in the _unprivileged group_ if the
value of their sensitive attribute defines them in the historically
disadvantaged group.

###  2.2 Types of Bias Mitigation Methods

Bias mitigation methods are usually split into pre-processing, in-processing,
and post-processing. In Section 3, we will present the approaches for the
three types of bias mitigation methods

Pre-processing methods are used to mitigate bias in the training data, and are
useful when we have access to the training data, but not necessarily to the
machine learning model trained on the data (Bellamy et al., 2018).

In-processing methods are used while training ML models, and are useful when
we have access to the machine learning model to ensure that unwanted bias is
not present in the output due to the training algorithm (Zhang and Ntoutsi,
2019). These methods can be model-agnostic or model-specific. Whilst model-
agnostic methods may need to compromise on performance to be able to work for
all models, model-specific methods are designed for a particular type of
training algorithm, e.g. Naive Bayes (Calders and Verwer, 2010), decision
trees (Kamiran et al., 2010), neural networks (Hu et al., 2020). These can be
more effective as are designed specifically for the model chosen, but do not
generalise.

Post-processing methods mitigate bias in the output of trained ML models.
These methods focus on mitigating bias in the decision maker’s output and are
not concerned with the dataset or the training algorithm. Most are model-
agnostic, however Savani et al. (2020) recognises a subgroup named intra-
processing methods which relies on some knowledge of the classifier, e.g.
decision tree nodes Kamiran et al. (2010), posterior probabilities Calders and
Verwer (2010).

###  2.3 Datasets

There are several datasets used to evaluate bias mitigation methods, which
have been collected from real-world data. The datasets used in the methods
collated are shown in Table 1, ordered by the number of methods they are used
for. We find that the most commonly used datasets in the literature are the
Adult census 1994, German credit, COMPAS and Dutch census datasets.

Dataset | Abbreviation | # instances | # attributes | # methods
---|---|---|---|---
Adult Census 1994 555https://archive.ics.uci.edu/ml/datasets/Adult | Adult | 48842 | 15 | 32
German Credit 666https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+data) | German | 1000 | 21 | 15
COMPAS 777https://www.kaggle.com/danofer/compass | COMPAS | 7214 | 51 | 9
Dutch Census 888https://microdata.worldbank.org/index.php/catalog/2101 | Dutch | 189725 | 12 | 8
Communities and Crime 999https://archive.ics.uci.edu/ml/datasets/Communities+and+Crime | Communities & Crime | 1994 | 127 | 6
Bank Marketing 101010https://archive.ics.uci.edu/ml/datasets/bank+marketing | Bank | 45211 | 17 | 6
Heritage Health Prize 111111https://www.kaggle.com/c/hhp | Health | 147473 | 139 | 2
Ricci vs DeStefano 121212https://www.openml.org/d/42665 | Ricci vs DeStefano | 118 | 6 | 2
Credit Card Default 131313https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients | Default | 30000 | 24 | 1
KDD Census 141414https://archive.ics.uci.edu/ml/datasets/Census-Income+(KDD) | KDD | 299285 | 41 | 1

Table 1: Datasets used to evaluate bias mitigation methods and their number of
instances and attributes, as well as the number of methods they are used in.

###  2.4 Metrics

Bias mitigation methods are evaluated by calculating the amount of unwanted
bias before and after the methods are applied; this can be achieved using
fairness metrics. Fairness metrics can be split into two categories: _group_
and _individual_. The aim of the methods is to decrease the level of unwanted
bias according to a fairness metric without decreasing the performance of the
model (Fish et al., 2016; Zliobaite, 2015).

#### Group fairness metrics

measure unwanted bias across two groups defined by two values of a single
personal characteristic (e.g. ‘male’ and ‘female’). Table 2 shows the group
fairness metrics that are used in two or more of the collated methods
including how they are calculated. These definitions are consistent with
Pessach and Shmueli (2023) and Verma and Rubin (2018) which also provide
further intuitive descriptions of the metrics. In the table, $A$ is an
individual in the dataset, $\bar{y}$ is the classification for that
individual, $y$ is the true label of the individual in the training data, and
$s$ is the unprivileged group. Hence, $A\in{s}$ means that the individual $A$
is in the unprivileged group; $\bar{y}=1$ ($=0$) means that the classifier
predicts the outcome $1$ ($0$), respectively; and $y=1$ ($=0$) means that the
true label of the individual is $1$ ($0$), respectively. Other metrics used in
the bias mitigation methods collated that are not included in Table 2 are
discussed in Section 3.

Metric | Calculation
---|---
Equal opportunity | $|P(\bar{y}=1|A\notin{s},y=1)-P(\bar{y}=1|A\in{s},y=1)|$
Equalised odds | $|P(\bar{y}=1|A\notin{s},y)-P(\bar{y}=1|A\in{s},y)|,y\in\\{0,1\\}$
Demographic parity | $|P(\bar{y}=1|A\notin{s})-P(\bar{y}=1|A\in{s})|$
Disparate impact | $|P(\bar{y}=1|A\notin{s})/P(\bar{y}=1|A\in{s})|$
Disparate mistreatment | $|P(\bar{y}=0|A\notin{s},y=1)-P(\bar{y}=0|A\in{s},y=1)|+$
| $|P(\bar{y}=1|A\notin{s},y=0)-P(\bar{y}=1|A\in{s},y=0)|$
Table 2: Commonly used group fairness metrics.

The metrics are calculated using the values from the confusion matrix (true
positives (TP), false positives (FP), true negatives (TN), and false negatives
(FN)) across two values of the sensitive attribute. For a binary decision-
making system, TP are the correct positive classifications ($\bar{y}=1,y=1$),
FP are the incorrect positive classifications ($\bar{y}=1,y=0$), TN are the
correct negative classifications ($\bar{y}=0,y=0$) and FN are the incorrect
negative classifications ($\bar{y}=0,y=1$). The rates of these measures are
defined below.

_True positive rate (TPR)_ $\displaystyle=\frac{TP}{TP+FN}$, TP out of total
actual positive classifications

_False positive rate (FPR)_ $\displaystyle=\frac{FP}{TN+FP}$, FP out of total
actual negative classifications.

_True negative rate (TNR)_ $\displaystyle=\frac{TN}{TN+FP}$, TN out of total
actual negative classifications.

_False negative rate (FNR)_ $\displaystyle=\frac{FN}{TP+FN}$, FN out of total
actual positive classifications.

From Table 2, $P(\bar{y}=1|A\in{s},y=1)$ represents the TPR for the
unprivileged group and $P(\bar{y}=1|A\notin{s},y=1)$ represents the TPR for
the privileged group.

_Equal opportunity_ (Hardt et al., 2016) therefore aims to have equal TPRs for
the privileged and unprivileged group. For example, when predicting fraudulent
activity on a bank account, the primary aim is to correctly identify
fraudulent activity (TP). It is important that the unprivileged group does not
receive fewer TP as this would put them at a disadvantage.

_Equalised odds_ (Hardt et al., 2016) places an emphasis on minimising the
difference in TPRs and FPRs across the privileged and unprivileged groups as
represented by $P(\bar{y}=1|A\notin{s},y)-P(\bar{y}=1|A\in{s},y)$, for both
$y=1$ and $y=0$. The TPR is represented when $y=1$ and the FPR is represented
when $y=0$. Bias mitigation methods that use this metric aim to minimise these
separately and then sum or average the differences for a single metric value.
An example of where this metric may be suitable is when predicting who will be
able to pay back a loan — it is important to ensure fairness for individuals
who are correctly predicted to pay back the loan as well as incorrectly, as
this could lead to negative consequences for individuals in the future such as
damage to a credit score and difficulty getting another loan in the future.

_Demographic parity_ and _disparate impact_ are concerned with achieving the
same rates of positive classifications across the privileged and unprivileged
groups. This can be used to improve diversity with respect to an identified
group, for example improving demographic parity in a hiring system to increase
the number of women selected (Cortez, 2019).

_Disparate mistreatment_ (Zafar et al., 2017a) combines TPR, FPR, TNR, and FNR
to give an overall view of the system; this is useful when concerned with
FNRs, for example in healthcare where a false negative could be disastrous to
an individual if a condition is not predicted early.

#### Individual fairness metrics

have been created to evaluate the unwanted bias of an individual decision.
Dwork et al. (2012) created an individual fairness metric, Dwork IF, which
measures to what extent similar individuals receive the same classification,
where the definition of a similar individual is required as input and depends
on the context. The metric quantifies the number of pairs of similar
individuals receiving the same classification.

Zemel et al. (2013) introduced consistency, an individual fairness metric
which defines the similarity of individuals by considering its nearest
neighbours according to Manhattan distance and providing a score based on the
number of neighbours with different classifications. Consistency is defined
below, where $n$ is the number of individuals in the dataset, $k$ is the
number of neighbours chosen, $kNN(x_{i})$ represents the k-nearest neighbours
and $\hat{y}$ is an individual’s label.

$\displaystyle 1-\frac{1}{n}\sum_{i=1}^{n}|\hat{y}-\frac{1}{k}\sum_{j\in
kNN(x_{i})}\hat{y}|$

#### Performance metrics

are used to evaluate the performance of decision-making systems. The metric
values often decrease after the application of a bias mitigation method. The
metrics used for the evaluation of the methods collated in this survey include
Accuracy, $F_{1}$, and the Balanced Error Rate ($BER$), defined as follows.

$\displaystyle Accuracy$ $\displaystyle=$
$\displaystyle\frac{TP+TN}{TP+TN+FP+FN}$ $\displaystyle F_{1}$
$\displaystyle=$ $\displaystyle\frac{TP}{TP+\frac{1}{2}(FP+FN)}$
$\displaystyle BER$ $\displaystyle=$
$\displaystyle\frac{1}{2}\left(\frac{FN}{FN+TP}+\frac{FP}{FP+TN}\right)$

###  2.5 Related Work

In recent years, there has been a growing interest in the development of bias
mitigation methods, and as a result, there have been several surveys
conducted. In this section, we will review them and their contributions.

Pessach and Shmueli (2023) is the most similar to our survey and provides a
detailed background to bias mitigation, including metrics and datasets used
throughout the literature. They also split the methods into pre-, in- and
post-processing and include tables of methods collated, including a brief
description of each method. Our survey differs by providing a novel
contribution in finding the limitations of each surveyed method, and using
these limitations to provide evidence for recommendations for future
directions of research. We hope this will be able to guide researchers into a
new direction for creating bias mitigation methods.

Dunkelau (2020) gives an extensive overview of fairness in machine learning,
overlapping with our survey and that of Pessach and Shmueli (2023) in their
descriptions of datasets and metrics. They describe the methods, but do not
provide an analysis or critique of them as our survey does. Hort et al. (2022)
present a comprehensive survey of bias mitigation methods for machine learning
which categorises methods from 234 papers into pre-, in-, and post-processing,
and further by the approach they take, e.g. sampling, adversarial, adding
constraints. Furthermore, they provide statistics about the methods in terms
of datasets and metrics used. Some of the methods they survey however are
small extensions of other methods or not from peer-reviewed work. Our survey
differs from these works by providing a discussion of bias mitigation methods,
specifically, commenting on their limitations which should be considered for
the development of new methods.

Other related work focuses on the development of frameworks to be able to more
easily evaluate bias mitigation methods on different models and datasets, and
using different fairness metrics. These are outlined in Dunkelau (2020). Along
with frameworks, quantitative analyses of several methods are usually
provided, but the usability or limitations of the methods are not commented
on. For example, Friedler et al. (2019) developed a framework that allows the
comparison of different methods on the same dataset and model, for different
fairness metrics. In addition, Schelter et al. (2020) designed FairPrep which
allows users to tune hyper-parameters of classifiers to see the impact on
fairness and accuracy metrics across several datasets (i.e. Adult, German
Credit and Ricci vs DeStefano).

##  3 Current State of Research

In this section, we survey existing bias mitigation methods for binary
classification and compare their strengths and limitations. We split them into
pre-, in-, and post-processing and describe the method, datasets and metrics
used to evaluate them.

Other methods not designed for binary classification, which are not included
in Sections 3.1–3.3, include ones designed to improve fairness in regression
(Calders et al., 2013; Berk et al., 2017; Chzhen et al., 2020), multi-class
classification (Putzel and Lee, 2022; Alghamdi et al., 2022), clustering
(Chierichetti et al., 2017; Ziko et al., 2021; Abbasi et al., 2021; Backurs et
al., 2019; Rösner and Schmidt, 2018; Bera et al., 2019), outlier detection (P
and Abraham, 2020), clustering from demonstrations (Galhotra et al., 2021), in
online data streams (Iosifidis and Ntoutsi, 2020; Zhang and Ntoutsi, 2019) and
where there is little data available Slack et al. (2020). These methods are
evaluated on the datasets identified in the fairness domain but are not
comparable to the other methods surveyed in Tables 3–5.

The methods collated in the Appendix of Le Quy et al. (2022) are the basis for
conducting our survey of bias mitigation methods. These methods were obtained
by a systematic search process, which involved the use of Google Scholar to
find all papers that used fairness datasets for evaluation, in conjunction
with the search terms “bias”, “discrimination” and “fairness”. The resulting
papers were then filtered based on the number of citations as well as the
publication venue (i.e. conferences and journals). The filtering process
ensured that only papers of high quality and relevance were included in the
survey, providing a reliable representation of the state-of-the-art in the
field. Additionally, we filter the papers by those that propose bias
mitigation methods for binary classifiers and find 25 prominent papers from
which we extract 33 bias mitigation methods to analyse.

A limitation of the bias mitigation methods collated is that they cannot be
directly compared due to being evaluated using different fairness metrics and
different models trained on datasets of different sizes and distributions.

###  3.1 Pre-processing Methods

Pre-processing methods focus on removing unwanted bias from the training data.
By definition, this means that all these methods are model-agnostic as they
are not concerned with the machine learning model used. A limitation of pre-
processing methods is that they are considered intrusive because they change
the dataset (Kamiran and Calders, 2011). Next, we review the pre-processing
methods in the literature and summarise the datasets and metrics used by these
methods in Table 3.

Method Source Datasets Fairness Metrics Performance Metrics Suppression
Kamiran and Calders (2011) Adult, German, Dutch, Equal opportunity Accuracy
Communities & Crime Massaging Kamiran and Calders (2011) Adult, German, Dutch
Equal opportunity Accuracy Reweighing Kamiran and Calders (2011) Adult,
German, Dutch Equal opportunity Accuracy Uniform Sampling Kamiran and Calders
(2010, 2011) Adult, German, Dutch Equal opportunity Accuracy Preferential
Sampling Kamiran and Calders (2010, 2011) Adult, German, Dutch Equal
opportunity Accuracy Local Massaging Zliobaite et al. (2011) Adult, Dutch,
Illegal Accuracy Kamiran et al. (2013) Communities & Crime discrimination
Local Preferential Zliobaite et al. (2011) Adult, Dutch, Illegal Accuracy
Sampling Kamiran et al. (2013) Communities & Crime discrimination Oversampling
Iosifidis and Ntoutsi (2018) Adult, German Equalised odds $F_{1}$ SMOTE-
approach Iosifidis and Ntoutsi (2018) Adult, German Equalised odds $F_{1}$
Geometric Feldman et al. (2015) Adult, German, Disparate impact, Accuracy,
Repair Feldman (2015) Ricci vs DeStefano Demographic parity BER Combinatorial
Feldman et al. (2015) Adult, German, Disparate impact, Accuracy, Repair
Feldman (2015) Ricci vs DeStefano Demographic parity BER Learning Fair Zemel
et al. (2013) Adult, German, Health Demographic parity, Accuracy
Representations Consistency LCIFR Ruoss et al. (2020) Adult, German, Certified
individual Accuracy COMPAS, Health, fairness Communities & Crime

Table 3: Table of collated pre-processing bias mitigation methods.

#### Kamiran and Calders (2010, 2011)

proposed five pre-processing methods, namely Suppression, Massaging,
Reweighing, Uniform Sampling, and Preferential Sampling. Suppression simply
removes the protected attribute and is often used as a baseline. However, this
approach is problematic as there may still be proxy attributes which may lead
to a biased model. Massaging changes the labels of some individuals in the
training set chosen based on a learned ranker that selects individuals close
to the decision boundary to be ‘promoted’ (i.e. from $-$ to $+$) and ‘demoted’
(i.e. from $+$ to $-$). The number of individuals whose labels are changed are
selected based on the calculated level of unwanted bias according to the equal
opportunity fairness metric.

Reweighing assigns weights to individuals based on the associated
classification and whether they are in the unprivileged group. In particular,
four weights are calculated for the following combinations: positive/negative
classification (not) in the unprivileged group. For example, all individuals
with a positive classification in the unprivileged group will be assigned the
same weight which will be greater than the weight assigned to the individuals
with a negative classification in the unprivileged group. Reweighing can be
applied on its own when the training algorithm uses weights (e.g. linear-based
classifiers), otherwise we need to _sample_ individuals from the dataset
according to these weights. Uniform Sampling constructs a dataset by choosing
how many individuals to duplicate or remove according to their weight in each
of the four groups mentioned above, and selects which individuals within the
group to choose with uniform probability. Preferential Sampling similarly
constructs a dataset by choosing how many individuals in each of the four
combinations to duplicate and remove by their weight. However, it uses a
ranker to decide which individuals are close to the decision boundary between
a positive and negative classification and prioritises these individuals to
duplicate or remove.

Limitation: The methods described above only allow for a single binary
sensitive attribute in the dataset. The definitions of existing group fairness
metrics (found in Table 2) only account for a single binary sensitive
attribute meaning methods which use these metrics only allow the removal of
unwanted bias with respect to that one attribute. Group fairness metrics would
need to be adapted to account for multiple binary sensitive attributes and
sensitive attributes with multiple values.151515Unless otherwise stated, all
methods discussed in this survey that use these group fairness metrics only
consider a single binary sensitive attribute.

Datasets: All methods are evaluated on Adult, Communities & Crime, and German
datasets.

Fairness metrics: Equal opportunity.

#### Kamiran et al. (2013); Zliobaite et al. (2011)

extended the work by Kamiran and Calders (2011) and introduced two bias
mitigation methods, Local Massaging and Local Preferential Sampling. They
propose a new notion for measuring the fairness of a dataset called illegal
discrimination, where some bias can be explained and is therefore acceptable
to be included in the model, while other types of bias are unwanted (i.e.
‘illegal’).

Illegal discrimination makes use of explanatory attributes, which are
attributes that correlate to a sensitive attribute but should still be used in
making the decision. Determining explanatory attributes is domain dependent.
For example, assuming that “gender” is the sensitive attribute in a decision-
making system for monthly salaries, the correlated attribute of “relationship”
(e.g. husband or wife) would not be an explanatory attribute as there should
not be decisions based on this attribute. However, the attribute “working
hours”, which may also be correlated to gender, could be an explanatory
attribute as it may be informative in determining monthly salaries.

Illegal discrimination is measured as the difference between the level of
unwanted bias according to an existing group fairness metric and the explained
bias as shown below, where $k$ is the number of explanatory attributes,
$e_{i}$ is a value of the explanatory attribute values, and $A$ and $B$ are
the values of the sensitive attribute (e.g. male, female):

$D_{expl}=\sum_{i=1}^{k}(P(e_{i}|A)-P(e_{i}|B))P^{*}(+|e_{i})\qquad
P^{*}(+|e_{i})=\frac{P(+|e_{i},A)+P(+|e_{i},B)}{2}$

$P(e_{i}|A)$ ($P(e_{i}|B)$) is the probability that an individual has the
explanatory attribute value $e_{i}$ given they are in the unprivileged group
(in the privileged group), respectively; and $P^{*}(+|e_{i})$ represents the
probability of an individual having a positive classification given it has the
explanatory attribute value $e_{i}$.

Limitations:

  1. 1.

The choice of datasets for evaluating the two bias mitigation methods impacts
the results, making it difficult to compare and assess the usefulness of the
proposed methods. The methods are shown to be more effective when the
explanatory attribute is highly correlated with the sensitive attribute. In
particular, the explanatory attribute has a higher correlation with the
sensitive attribute in the Adult dataset than in the Communities & Crime and
Dutch datasets, resulting in a lower level of illegal discrimination on the
Adult dataset than on the other datasets.

  2. 2.

An issue with using the illegal discrimination metric is that the explanatory
attributes have to be decided by a domain expert. This raises questions about
whom the domain expert should be and whether there is potential for human bias
or error in choosing these attributes.

Datasets: The methods are evaluated on Adult, Communities & Crime, and Dutch
datasets.

Fairness metrics: Illegal discrimination.

#### Iosifidis and Ntoutsi (2018)

introduced two pre-processing methods. Oversampling is a naive method which
selects random individuals in the unprivileged group and duplicates them to
ensure a balanced training dataset. They also propose using SMOTE (Chawla et
al., 2002) to create new individuals in the training data for each individual
in the unprivileged group. The method finds the k-nearest neighbours for each
individual that are also in the unprivileged group. It chooses randomly
amongst these neighbours and generates new individuals in the unprivileged
group that lie on the line between the individual and the chosen neighbour.

Limitation: There is a trade-off between fairness and performance as the
performance was shown to decrease after applying each bias mitigation method.
There is no discussion as to whether this performance decrease is acceptable.

Datasets: The methods are evaluated on the Adult and German datasets.

Fairness metrics: Equalised odds.

#### Feldman et al. (2015); Feldman (2015)

proposed two methods, Geometric Repair and Combinatorial Repair, which ensure
that non-sensitive attributes are not used to predict stratifying columns,
where stratifying columns are sensitive attributes or combinations of
sensitive attributes (e.g. white-male). This is achieved by changing the
values of non-stratifying columns to ensure the distributions over the
different values of the stratifying columns are the same (e.g. white-males and
white-females have the same median values for every other attribute).
Geometric Repair and Combinatorial Repair include a parameter which allows the
specification of partial repair, corresponding to the desired trade-off of
performance and fairness. The methods are evaluated using disparate impact and
demographic parity which are consistent with the notion of fairness used
throughout law and regulation (Feldman, 2015). One basis for assessing whether
a decision-maker is legally fair is the 80% rule which is a minimum level of
disparate impact that should be accepted (Agarwal et al., 2018). In other
terms, the system is unfair if the disparate impact metric calculated is less
than 80% as below:

$\frac{\textrm{Rate of individuals in the unprivileged group receiving a
positive outcome}}{\textrm{Rate of individuals in the privileged group
receiving a negative outcome}}<0.8$

Benefits:

  1. 1.

The methods can be applied to datasets with multiple sensitive attributes as
the authors propose stratifying attributes which combine all values of the
sensitive attributes, e.g. white-male, black-female.

  2. 2.

Different levels of disparate impact can be removed using both Geometric
Repair and Combinatorial Repair. The partial repair of the dataset is achieved
by including a parameter $\lambda$ in the methods, which is a value between 0
and 1, that specifies to what extent the data is ‘repaired’.

Limitation: Enforcing the median values of attributes to be the same across
stratifying attributes requires the attributes to be orderable. Attributes can
be categorical only if categories have a natural order.

Datasets: The methods are evaluated on Adult, German, and Ricci vs DeStefano
datasets.

Fairness metrics: Disparate impact.

#### Zemel et al. (2013)

introduced Learning Fair Representations (LFR) which aims to solve an
optimisation problem to optimise for group fairness, individual fairness and
performance, by encoding accuracy, demographic parity and their created
individual fairness metric of _consistency_ as objective functions.
Consistency compares an individual to its k-nearest neighbours and counts how
many of the neighbours’ labels differ.

Limitation: The Learning Fair Representations method optimises for demographic
parity and consistency and there are no parameters to allow for the partial
optimisation of either metric. This may not always be desired as it could
cause positive discrimination or lead to a significant decrease in
performance.

Datasets: The method is evaluated on the Adult, German, and Health datasets.

Fairness metrics: Demographic parity, Consistency.

#### Ruoss et al. (2020)

proposed Learning Certified Individually Fair Representations (LCIFR) which
modifies the dataset using a neural network with a loss function based on a
user-specified individual similarity definition. The authors propose
_certified individual fairness_ , a metric which calculates the proportion of
individuals with the same label given all similar individuals.

Limitation: The method does not ensure that similar individuals are not
treated negatively as it only targets improving individual fairness. For
example, individuals with the attribute value female may be considered similar
and receive the same classification, therefore satisfying individual fairness.
However, if these classifications are all negative, then group fairness with
respect to gender would not be satisfied.

Datasets: The method is evaluated on the Adult, COMPAS, Communities & Crime,
German, and Health datasets.

Fairness metrics: Certified individual fairness.

###  3.2 In-processing Methods

In-processing methods can be model-agnostic or model-specific and usually
involve adjusting the training algorithm, training adversarial learners or
optimising weights for both performance and fairness. As with pre-processing
methods, in-processing methods are evaluated/optimised for a range of
different fairness and performance metrics, as well as being tested on a range
of different datasets. The methods collated are shown in Table 4, along with
the classifier, datasets, and metrics used.

Method Source Classifier Datasets Fairness Metrics Performance Metrics Fair
Learner for Choi et al. (2020) Naive Bayes Adult, Own discrimination Accuracy
Naive Bayes German, score (Equation 3.2) COMPAS Two Naive Bayes Model Calders
and Verwer (2010) Naive Bayes Adult Disparate impact Accuracy Latent Variable
Model Calders and Verwer (2010) Naive Bayes Adult Disparate impact Accuracy
Bayesian Network Mancuhan and Clifton (2014) Bayesian Adult, Count of
discriminated Accuracy Learning Network German instances Fair NN Hu et al.
(2020) Neural Adult, Equalised odds Accuracy, Network Bank BER Decision Tree
Learning Kamiran et al. (2010) Decision Tree Adult, Disparate impact Accuracy
Dutch Regularization-Inspired Bechavod and Ligett (2017) Logistic COMPAS, FPR,
FNR Accuracy Approach Regression Default Adversarial Learning Zhang et al.
(2018) Logistic Adult Equalised odds, FPR, FNR Regression Equal opportunity,
Demographic parity Fair Adversarial Grari et al. (2019) Gradient Tree Adult,
Equalised odds, Accuracy Gradient Tree Boosting Boosting Bank, Demographic
parity COMPAS Mechanisms for Fair Zafar et al. (2017b) Convex Adult, Disparate
impact, Accuracy Classification margin-based Bank Disparate mistreatment
AdaFair Iosifidis and Ntoutsi (2019) Model-agnostic Adult, Equalised odds BER
Bank, COMPAS KDD Adaptive Sensitive Krasanakis et al. (2018) Model-agnostic
Adult, Disparate impact, Accuracy Reweighting Bank, COMPAS Privileged Learning
& Quadrianto and Sharmanska (2017) Model-agnostic Adult, Equalised odds,
Accuracy Distribution Matching COMPAS Equal opportunity, Disparate
mistreatment, Demographic parity Fair Multitask Learning Oneto et al. (2019)
Model-agnostic Adult, Equalised odds, Accuracy COMPAS Equal opportunity

Table 4: Table of collated in-processing bias mitigation methods.

#### Choi et al. (2020)

introduced Fair Learner for Naive Bayes which proposes discrimination patterns
that are defined as individuals that have different classifications depending
on whether the sensitive attribute has an impact on the decision (i.e. if the
sensitive attribute is removed and the classification changes, there is a
discrimination pattern). The aim of the method is to discover these
discrimination patterns using a branch-and-bound method (Morrison et al.,
2016) and iteratively learn a Naive Bayes classifier that minimises a
discrimination score based on the probability of sensitive attributes
affecting a positive classification. The discrimination score is defined in
Equation 3.2, where $P(d|xy)$ is the probability of a positive classification
given all the attributes, calculated as the proportion of positive
classifications when the classifier is trained with all attributes. $P(d|y)$
is the probability of a positive classification given all attributes except
the sensitive attribute, calculated as the proportion of positive
classifications when the classifier is trained with all attributes except the
sensitive attribute.

Equation 1.

$\Delta=P(d|xy)-P(d|y)$

Benefit: This method can be applied to datasets with multiple sensitive
attributes.

Limitations:

  1. 1.

A new fairness metric is created, but a comparison with existing metrics is
not provided. However, the authors justify the use of the new metric by
observing that current common fairness metrics consider only one sensitive
attribute.

  2. 2.

The method can only be applied to Naive Bayes models.

Datasets: The method is applied on Adult, German, and COMPAS datasets.

Fairness metrics: Own discrimination score (Equation 3.2).

#### Calders and Verwer (2010)

introduced two in-processing methods for Naive Bayes classifiers: Latent
Variable Model and Two Naive Bayes Model. Latent Variable Model aims to
classify individuals given a discrimination-free model. This involves
modelling classifications as hidden variables and using an expectation
maximisation approach to find the individuals in the negative class who should
be in the positive class and vice-versa, and swapping their classifications.
The Two Naive Bayes method learns two different models for the data, split by
the values of the identified sensitive attribute, and uses these to classify
individuals, using the model corresponding to their value of the sensitive
attribute. This method was shown to yield the best results.

Limitations:

  1. 1.

The authors compare their Two Naive Bayes method to Fair Learner for Naive
Bayes (Choi et al., 2020) on the Adult dataset and find that it yields worse
accuracy. However, it is difficult to directly compare the impact on fairness
due to the different metrics used (i.e. own discrimination score (Equation
3.2) versus disparate impact).

  2. 2.

The methods can only be applied to Naive Bayes models.

Datasets: The methods are evaluated on the Adult dataset.

Fairness metrics: Disparate impact.

#### Mancuhan and Clifton (2014)

proposed Bayesian Network Learning which learns a Bayesian network and then
creates a new network by removing the sensitive attributes and the parents and
children of the sensitive attributes. A discriminated instance is an
individual whose classification in the new network differs from its
classification in the original network. The labels of these discriminated
instances are then swapped and the aim of the bias mitigation method is to
learn a new Bayesian network based on the updated dataset. The number of
discriminated instances is used as the metric to measure the level of unwanted
bias.

Benefit: The method does not use the sensitive attribute at test time. Whilst
it uses the sensitive attribute to identify bias and then mitigate it, the
method does not use sensitive attributes to make predictions.

Limitations:

  1. 1.

A new metric (i.e. number of discriminated instances) is created for
evaluation, which does not encapsulate the complexity of measuring fairness.

  2. 2.

This method can only be applied to Bayesian networks.

Datasets: The method is tested on the Adult and German datasets.

Fairness metrics: Number of discriminated instances.

#### Hu et al. (2020)

proposed Fair NN which improves fairness in neural networks by altering the
network’s autoencoder with the aim to suppress sensitive attributes and
constrain the loss function based on the fairness metric of equalised odds.

Limitations:

  1. 1.

As with most of the methods surveyed, this method only allows for a single
binary sensitive attribute. Hu et al. (2020) proposed a mechanism to account
for a multi-value sensitive attribute by pre-processing the dataset to only
have binary values. However, this would result in multiple sensitive
attributes, which cannot be addressed with the current method.

  2. 2.

The method can only be applied to Neural Networks.

Datasets: The method is evaluated on Adult and Bank datasets.

Fairness metrics: Equalised odds.

#### Krasanakis et al. (2018)

introduced Adaptive Sensitive Reweighting which iteratively trains a
classifier, each time adapting the training data to converge towards the best
performance-fairness trade-off according to accuracy and disparate impact. At
each iteration, weights for individuals are calculated based on the
classification error and whether they are in the unprivileged group. For
example, a misclassified individual in the unprivileged group will be given a
different weight compared to a misclassified individual in the privileged
group.

Limitation: Adaptive Sensitive Reweighting can greatly increase computational
cost as it involves training the classifier until convergence, thus
highlighting scalability issues.

Datasets: The method is evaluated on Adult, Bank, and COMPAS datasets.

Fairness metrics: Disparate impact, Disparate mistreatment.

#### Kamiran et al. (2010)

proposed two methods Discrimination-Aware Tree Construction and Relabeling
Decision Trees. Discrimination-Aware Tree Construction changes the splitting
criteria of branches by including the notion of fairness. Specifically,
instead of splitting based on the information gain with respect to the class,
the split is based on the information gain with respect to the class and the
sensitive attribute.

Benefit: There is no detriment to the performance of the system by using this
method. The results show a noticeable reduction in the disparate impact score
without a loss in accuracy on the datasets evaluated.

Limitations:

  1. 1.

The initial level of unwanted bias in the training and test datasets can
greatly impact the fairness metric values calculated and therefore impact the
evaluation of new methods. The accuracy of the mitigated classifier increases
over a standard decision tree learning classifier when there is unwanted bias
in the training data but not in the test data.

  2. 2.

This method can only be applied to Decision Tree models.

Datasets: The method is evaluated on Adult and Dutch datasets.

Fairness metrics: Disparate impact.

#### Bechavod and Ligett (2017)

proposed a Regularization-Inspired Approach that includes a fairness penalty
into the loss function of a logistic regression model which penalises for
different FPRs and FNRs across different values of the sensitive attribute.

Benefit: The method can be applied to datasets with one or two sensitive
attributes. The method can almost entirely remove unwanted bias on the COMPAS
dataset, given the difference of FPRs and FNRs with respect to two sensitive
attributes, with results showing only a small reduction in accuracy.

Limitations:

  1. 1.

The method does not provide theoretical guarantees that balancing the FPRs and
FNRs across protected groups improves fairness in all cases.

  2. 2.

The method can be applied only to logistic regression models.

Datasets: The method is evaluated on COMPAS and Default datasets.

Fairness metrics: False positive rates, False negative rates.

#### Zhang et al. (2018)

introduced an Adversarial Learning method which aims to maximise the
prediction accuracy of a classifier while minimising an adversarial
classifier’s ability to predict the sensitive attribute. Logistic regression
is used for both classifiers, with the input to the adversarial model being
the output from the classifier.

Benefit: The method allows the user to choose whether to optimise for
equalised odds, equal opportunity, or demographic parity. This means that the
method can be applied in different settings by specifying the desired notion
of fairness, depending on context.

Limitations:

  1. 1.

The method relies on hyper-parameters that impact how well the method reduces
the level of unwanted bias. In addition, more complex prediction problems
(e.g. multi-class classifiers) may require more complex adversaries to be
adapted.

  2. 2.

The method can only be applied to logistic regression models.

Datasets: The method is evaluated on the Adult dataset.

Fairness metrics: Equalised odds, Equal opportunity, Demographic parity.

#### Zafar et al. (2017b)

proposed Mechanisms For Fair Classification which defines a “novel intuitive
measure of decision boundary (un)fairness” that measures the variance of the
decision boundary and uses it to formulate an approach that either aims to
maximise fairness with accuracy constraints or aims to maximise accuracy with
fairness constraints, depending on the preference of the user of the system
towards prioritising fairness or performance.

Benefit: The method can be applied to datasets with multiple sensitive
attributes and multi-value sensitive attributes.

Limitation: The authors do not provide theoretical guarantees that the method
decreases the level of unwanted bias in any system, for any dataset. Providing
guarantees that the constraints used for optimisation map to the notion of
disparate impact is left as future work.

Datasets: The method is evaluated on Adult and Bank datasets.

Fairness metrics: Disparate impact, Disparate mistreatment.

#### Grari et al. (2019)

proposed another adversarial method, Fair Adversarial Gradient Tree Boosting,
which again aims to make an accurate prediction, this time with Gradient Tree
Boosting, whilst ensuring an adversarial cannot predict the sensitive
attribute. This is done by integrating a fairness penalty into the loss
function of the gradient tree boosting model.

Benefit: Results suggest that the method does not decrease performance
significantly. For both fairness metrics, this method shows similar reduction
in unwanted bias as Adversarial Learning (Zhang et al., 2018), Mechanisms For
Fair Classification (Zafar et al., 2017b) and Geometric Repair (Feldman et
al., 2015; Feldman, 2015), but improved accuracy over the three datasets.

Limitations:

  1. 1.

There is a risk of positive discrimination, i.e. favouring individuals from
the historically disadvantaged group over others. For example, females may be
chosen over men by the decision-maker, which may be fair statistically but not
contextually, and in some cases, illegal (Wachter et al., 2021).

  2. 2.

The method can only be applied to Gradient Tree Boosting models.

Datasets: The method is evaluated on Adult, Bank, and COMPAS datasets.

Fairness metrics: Equalised odds, Demographic parity.

#### Quadrianto and Sharmanska (2017)

proposed Privileged Learning & Distribution Matching. Privileged learning
(Vapnik and Vashist, 2009) reduces the number of attributes needed to train an
accurate classifier by using sensitive attributes to predict the slack
variables, whilst distribution matching (Sharmanska and Quadrianto, 2016)
minimises the distance between the distributions of two datasets.

Benefits:

  1. 1.

The method can be applied to datasets with multi-value and numerical sensitive
attributes (e.g. age, income).

  2. 2.

In contrast to other methods (Fish et al., 2016; Lohia et al., 2019; Kamiran
et al., 2012), the method does not use the sensitive attributes at test time,
which may be realistic and an interesting consideration in some applications.

  3. 3.

The method allows the choice to optimise for equalised odds, equal
opportunity, or demographic parity, similarly to Adversarial Learning (Zhang
et al., 2018).

Limitation: Optimising with respect to one fairness metric can be at a
detriment to a different metric. The results show that the method reduces
equal opportunity but increases the equalised odds between the privileged and
unprivileged groups.

Datasets: The method is evaluated on Adult and COMPAS datasets.

Fairness metrics: Equalised odds, Equal opportunity, Demographic parity.

#### Oneto et al. (2019)

introduced Fair Multitask Learning which splits the classification task into
smaller tasks to learn group specific classifiers that leverage information
between the privileged and unprivileged groups. If the sensitive attribute is
sex, two classifiers would be learnt, one for male individuals and one for
female individuals.

Limitation: Improving the fairness is at a detriment to the performance of a
decision-making system. The results demonstrate that removing the sensitive
attribute from training increases the fairness but decreases accuracy.

Datasets: The method is tested on Adult and COMPAS datasets.

Fairness metrics: Equalised odds, Equal opportunity.

#### Iosifidis and Ntoutsi (2019)

proposed AdaFair which is inspired by AdaBoost (Schapire, 1999) by using a
cumulative approach to fairness and creating the best classifier from an
ensemble of weighted trained classifiers based on the fairness and performance
they achieve. Cumulative fairness, which is the sum of false positive and
false negative rates over rounds of boosting, is shown to greatly outperform
non-cumulative calculations of fairness.

Limitations:

  1. 1.

The distribution of the dataset used for the testing has an impact on the
fairness value. Equalised odds is lower on datasets with higher original
imbalance with respect to the sensitive attribute (e.g, Bank and KDD), but
yields worse performance on balanced datasets (e.g. COMPAS).

  2. 2.

Convergence of the method is not guaranteed and the number of iterations it
may take to reach desired fairness cannot be calculated. These properties are
left for future work.

Datasets: The method is tested on Adult, Bank, COMPAS, and KDD datasets.

Fairness metrics: Equalised odds.

###  3.3 Post-processing Methods

Post-processing bias mitigation methods involve swapping the classifications
of individuals or adapting the output from the classifier (e.g. the posterior
probabilities or labels of decision trees) to minimise metric values. Table 5
shows the collated post-processing methods that focus on removing unwanted
bias from the classifications, along with the datasets and metrics used.

A limitation of post-processing methods is that they can be easily manipulated
to ensure some existing fairness metric is satisfied, for example by swapping
the classifications of random individuals to ensure an equal number of
positive and negative classifications across sensitive groups.

Method Source Datasets Fairness Metrics Performance Metrics Confidence Based
Approach Fish et al. (2016) Adult, German, Bank Own fairness metric: Accuracy
(RRB) Reject Option Based Kamiran et al. (2012) Adult, Disparate impact
Accuracy Classification (ROC) Communities & Crime Discrimination-Aware Kamiran
et al. (2012) Adult, Disparate impact Accuracy Ensemble (DAE) Communities &
Crime Individual and Group Debiasing Lohia et al. (2019) Adult, German, Dwork
IF, Accuracy COMPAS Disparate impact Relabelling Decision Trees Kamiran et al.
(2010) Adult, Dutch Disparate impact Accuracy Modifying Naive Bayes Calders
and Verwer (2010) Adult Disparate impact Accuracy

Table 5: Table of collated post-processing bias mitigation methods.

#### Fish et al. (2016)

proposed a Confidence Based Approach which aims to achieve fairness by moving
the decision boundary between the positive and negative classifications. They
define their own fairness metric, resilience to random bias (RRB), which is
used to evaluate the method with respect to random bias introduced in the
dataset, as opposed to the sensitive attribute. A new random attribute is
added to the individuals in the dataset that does not correlate with the
labels. The labels in the training dataset are then changed to include bias
against a particular value of the new attribute, simulating unwanted bias in
the system. Two models are trained, one with the adapted training dataset and
one with the original dataset. RRB is quantified as the proportion of the
classifications that are the same in the adapted model as in the original.

Benefit: The authors provide results with theoretical justifications, showing
that post-processing methods are useful in mitigating unwanted bias, with
minimal impact to the classifier’s performance.

Limitation: The relation between the newly created fairness metric and
existing metrics is not explained. The authors acknowledge that their metric
represents a simple interpretation of bias and optimising for this measure is
necessary but may not be sufficient to represent fairness overall.

Datasets: The method is evaluated on Adult, German, and Bank datasets.

Fairness metrics: Own fairness metric (RRB).

#### Kamiran et al. (2012)

proposed two decision theory based approaches for post-processing bias
mitigation: Reject Option Based Classification (ROC) and Discrimination-Aware
Ensemble (DAE). ROC exploits the probabilities of possible classifications of
an individual to decide whether to change the label of that individual. In
other words, depending on how confident the classifier is on its decision, the
post-processing method will decide whether to change the decision in a way
that reduces discrimination with respect to the sensitive attribute. For
example, if the classifier gives a negative classification for an individual
with the attribute value female with a confidence of 0.55, the classification
will be changed to a positive one as that probability is within the critical
region defined.

DAE uses the differences between the classifications from a classifier
ensemble to decide which individual’s classifications to change. If the
classifications differ for an individual, the classification will be made
positive if the individual is in the unprivileged group and made negative if
the individual is in the privileged group.

Benefit: The results show that both methods decrease disparate impact without
decreasing the performance significantly.

Limitation: ROC can only be applied to probabilistic classifiers and assumes
access to the posterior probabilities of the classifications.

Datasets: The methods are tested on Adult and Communities & Crime datasets.

Fairness metrics: Disparate impact.

#### Lohia et al. (2019)

proposed Individual and Group Debiasing which extends ROC (Kamiran et al.,
2012) but considers individual fairness as well as group fairness. The method
identifies individuals with low individual fairness and swaps these
individual’s classifications based on the confidence of the classification.

Limitation: Similarly to other post-processing methods, satisfying individual
fairness is more complex than group fairness and can be computationally costly
to implement, especially as the number of individuals and sensitive attributes
increases Agarwal et al. (2018).

Datasets: The method is evaluated on Adult, German and COMPAS datasets.

Fairness metrics: Dwork IF, Disparate impact.

#### Kamiran et al. (2010)

proposed Relabeling Decision Trees which swaps labels of leaves in the
decision tree if doing so decreases disparate impact on the classifications,
whilst maintaining or improving accuracy. This method is categorised as a
post-processing method by its authors as it is applied after the model has
been trained, however it involves changing the model and is specific to
decision trees so is considered an intra-processing method by Hort et al.
(2022).

Benefit: There is no detriment to the performance of the system when using
this method. The results show a noticeable reduction in the disparate impact
score without a loss in accuracy on the datasets evaluated.

Limitation: This method can only be applied to Decision Tree models.

Datasets: The method is evaluated on Adult and Dutch datasets.

Fairness metrics: Disparate impact.

#### Calders and Verwer (2010)

introduced a post-processing method for Naive Bayes classifiers. Modifying
Naive Bayes is a model-specific _post-processing_ method that changes the
probability of an individual in the unprivileged group with a positive class
by modifying the Naive Bayes model’s probabilities. It is considered an intra-
processing method Hort et al. (2022) as it post-processes the classifications
but still requires the posterior probabilities thus it is included in Table 5.

Limitation: The method can only be applied to Naive Bayes models.

Datasets: The method is evaluated on the Adult dataset.

Fairness metrics: Disparate impact.

##  4 Recommendations

In this section, we review the limitations highlighted in Section 3 and make
recommendations for the development of future bias mitigation methods for
binary classification decision-making systems.

### Generalisability

Most works on fairness in binary classification involve creating new bias
mitigation methods that improve the fairness of the system according to some
existing metric, and evaluating them using an arbitrarily chosen model and
dataset. These methods usually focus on a single identified sensitive
attribute (as the the majority of existing fairness metrics do not allow for
multiple sensitive attributes) and on this attribute being binary as opposed
to multi-value or numerical (e.g. only allows for the attribute of race having
values “white” or “non-white”). However, there can be multiple sensitive
attributes (e.g. race) and trying to remove the unwanted bias with respect to
one attribute may increase the unwanted bias towards another attribute (Yu et
al., 2021).

The specificity of these methods means they do not generalise, making it hard
to assess in what applications they would be useful. Furthermore, there is
currently no consensus of whether to choose a pre-, in-, or post-processing
method to mitigate a decision-making system. The key factor for this decision
is whether there is access to the model, in which case an in-processing method
should be chosen rather than a pre- or post-processing method.

The main disadvantage of pre-processing methods is that they can be intrusive
and involve editing the training data. Within the in-processing category,
there are model-agnostic and model-specific methods, however, the disadvantage
of these methods is that they require access to the model, which is not always
possible. The disadvantage of post-processing methods is that they can easily
manipulate the calculated fairness metrics without improving fairness. For
example, it is possible to fully optimise for a fairness metric by simply
swapping the classifications of some individuals in the unprivileged group to
positive to have an equal number of positive and negative classifications for
the privileged and unprivileged groups.

These findings contribute to the recommendation that work on bias mitigation
methods should also specify scenarios in which the methods may be applicable.

### Justification of Metrics

Currently, fairness metrics are used to justify the improved fairness of a
system, similarly to how performance metrics are used. Reducing fairness to an
optimisation problem risks removing the key contextual importance of fairness.
The methods created, often algorithmic, do not mention potential use cases or
criteria for potential uses (Weinberg, 2022). This can make it difficult to
evaluate when or where to use the different methods and whether real-life
decision-making systems meet the technical criteria of a bias mitigation
method (e.g. binary classification, access to the model).

Improving fairness can greatly impact performance. Due to the variety of
fairness metrics, some conflicting with others, the theoretical relation
between fairness and performance ought to be investigated more thoroughly.
Methods that include parameters to decide the desired performance-fairness
trade-off are useful in providing a bias mitigation method that may be
applicable to multiple scenarios.

Thus, we recommend that justifications are provided for the use of specific
metrics and how these relate to the context and application investigated.

### Explainability

Instead of simply using existing metrics to detect unwanted bias or evaluate
the impact of bias mitigation methods, we recommend developing methods that
focus on transparency and explainability, by explaining the unwanted bias
detected in decision-making systems. In particular, the fact that the fairness
of a system has improved by e.g. 20% after applying some bias mitigation
method means very little to the individuals impacted, thus explaining unwanted
bias is considered more useful compared to providing a number representing the
unwanted bias (Van Nuenen et al., 2020).

### Access to sensitive data

There is rarely a discussion around the ethics and legality of the bias
mitigation methods created. For example, data protection, specifically the use
of protected attributes for use in prediction algorithms, is not discussed and
many methods assume access to the protected attributes. However, this is not
applicable in real-life applications due to data protection and equality
treatment laws.161616https://www.legislation.gov.uk/ukpga/2010/15/contents
Haeri and Zweig (2020) recognise the crucial role of protected attributes in
being able to identify unwanted bias but that processing them is not allowed
under Article 9 of (GDPR, 2016).

### Addressing positive discrimination

Many methods inadvertently cause positive discrimination which is where
individuals in the unprivileged group are favoured over those in the
privileged group. Under UK law, positive discrimination (or “affirmative
action”) is illegal. The potential for positive discrimination is rarely
discussed, however, reducing discrimination without enforcing positive
discrimination (Mancuhan and Clifton, 2014) remains an important
consideration. Thus, considering the legality of new bias mitigation methods
should be a key aspect.

Overall, when deciding whether to apply a bias mitigation method to a
decision-making system, the recommendations highlighted should be considered
to ensure that the method has the right impact on the application and
individuals.

##  5 Conclusion

In this article, we surveyed bias mitigation methods for binary classification
decision-making systems and compared their strengths and limitations. We split
the methods into pre-, in-, and post-processing methods, and reported the
datasets and metrics used in evaluation. Our survey differs from existing work
by critically analysing existing bias mitigation methods and providing
recommendations for the development of future bias mitigation methods for
binary classification decision-making systems. The recommendations made
include ensuring generalisability of methods, justifying the use of existing
metrics for their evaluation, exploring transparency and explainability, and
considering the legality of bias mitigation methods.

## Acknowledgments

This work was supported by the UK Research and Innovation Centre for Doctoral
Training in Safe and Trusted Artificial Intelligence [grant number
EP/S023356/1].171717www.safeandtrustedai.org

## References

  * Deloitte (2021) Deloitte.  AI and Automated Decision Making, 2021.  URL https://www2.deloitte.com/uk/en/pages/deloitte-analytics/articles/ai-and-automated-decision-making.html.
  * Engin and Treleaven (2019) Zeynep Engin and Philip C. Treleaven.  Algorithmic government: Automating public services and supporting civil servants in using data science technologies.  _Computer Journal_ , 62(3):448–460, 2019.  doi: 10.1093/comjnl/bxy082.
  * Church and Fairchild (2017) Christopher E Church and Amanda J Fairchild.  In search of a silver bullet: Child welfare’s embrace of predictive analytics.  _Juvenile and Family Court Journal_ , 68(1):67–81, 2017.
  * Partnership on AI (2019) Partnership on AI.  Report on Algorithmic Risk Assessment Tools in the U.S. Criminal Justice System, 2019.  URL https://www.partnershiponai.org/report-on-machine-learning-in-risk-assessment-tools-in-the-u-s-criminal-justice-system/.
  * Gillingham (2019) Philip Gillingham.  Decision support systems, social justice and algorithmic accountability in social work: A new challenge.  _Practice_ , 31(4):277–290, 2019.
  * Northpointe Inc. (2019) Northpointe Inc.  Practitioner’s Guide to COMPAS Core, 2019.  URL https://s3.documentcloud.org/documents/2840784/Practitioner-s-Guide-to-COMPAS-Core.pdf.
  * Larson et al. (2016) Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin.  How We Analyzed the COMPAS Recidivism Algorithm, 2016.  URL https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm.
  * Jeffrey Dastin (2018) Jeffrey Dastin.  Amazon scraps secret AI recruiting tool that showed bias against women, October 2018.  URL https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G.
  * Waller and Waller (2020) Madeleine Waller and Paul Waller.  Why predictive algorithms are so risky for public sector bodies, 2020\.  URL https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3716166.
  * Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  _ACM Computing Surveys_ , 54(6):115:1–115:35, 2021.  doi: 10.1145/3457607.
  * Zliobaite et al. (2011) Indre Zliobaite, Faisal Kamiran, and Toon Calders.  Handling conditional discrimination.  In _11th IEEE International Conference on Data Mining, ICDM_ , pages 992–1001, 2011.  doi: 10.1109/ICDM.2011.72.
  * Kamiran and Calders (2011) Faisal Kamiran and Toon Calders.  Data preprocessing techniques for classification without discrimination.  _Knowledge and Information Systems_ , 33(1):1–33, 2011.  doi: 10.1007/s10115-011-0463-8.
  * Feldman et al. (2015) Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian.  Certifying and removing disparate impact.  In _Proceedings of the 21th ACM SIGKDD_ , pages 259–268, 2015\.  doi: 10.1145/2783258.2783311.
  * Iosifidis and Ntoutsi (2018) Vasileios Iosifidis and Eirini Ntoutsi.  Dealing with bias via data augmentation in supervised learning scenarios, 2018.
  * Calders et al. (2009) Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy.  Building classifiers with independency constraints.  In _ICDM Workshops 2009, IEEE International Conference on Data Mining Workshops_ , pages 13–18, 2009.  doi: 10.1109/ICDMW.2009.83.  URL https://doi.org/10.1109/ICDMW.2009.83.
  * Hu et al. (2020) Tongxin Hu, Vasileios Iosifidis, Wentong Liao, Hang Zhang, Michael Ying Yang, Eirini Ntoutsi, and Bodo Rosenhahn.  Fairnn - conjoint learning of fair representations for fair decisions.  In _Discovery Science - 23rd International Conference, DS 2020, Thessaloniki, Greece, October 19-21, 2020, Proceedings_ , pages 581–595, 2020.  doi: 10.1007/978-3-030-61527-7\\_38.
  * Iosifidis and Ntoutsi (2019) Vasileios Iosifidis and Eirini Ntoutsi.  Adafair: Cumulative fairness adaptive boosting.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM_ , pages 781–790, 2019.  doi: 10.1145/3357384.3357974.
  * Oneto et al. (2019) Luca Oneto, Michele Donini, Amon Elders, and Massimiliano Pontil.  Taking advantage of multitask learning for fair classification.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES_ , pages 227–237, 2019.  doi: 10.1145/3306618.3314255.
  * Krasanakis et al. (2018) Emmanouil Krasanakis, Eleftherios Spyromitros Xioufis, Symeon Papadopoulos, and Yiannis Kompatsiaris.  Adaptive sensitive reweighting to mitigate bias in fairness-aware classification.  In _Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW_ , pages 853–862, 2018.  doi: 10.1145/3178876.3186133.
  * Kamiran et al. (2010) Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy.  Discrimination aware decision tree learning.  In _ICDM, The 10th IEEE International Conference on Data Mining_ , pages 869–874, 2010.  doi: 10.1109/ICDM.2010.50.  URL https://doi.org/10.1109/ICDM.2010.50.
  * Calders and Verwer (2010) Toon Calders and Sicco Verwer.  Three naive bayes approaches for discrimination-free classification.  _Data Mining and Knowledge Discovery_ , 21(2):277–292, 2010.  doi: 10.1007/s10618-010-0190-x.
  * Grari et al. (2019) Vincent Grari, Boris Ruf, Sylvain Lamprier, and Marcin Detyniecki.  Fair adversarial gradient tree boosting.  In _2019 IEEE International Conference on Data Mining, ICDM_ , pages 1060–1065, 2019.  doi: 10.1109/ICDM.2019.00124.
  * Fish et al. (2016) Benjamin Fish, Jeremy Kun, and Ádám Dániel Lelkes.  A confidence-based approach for balancing fairness and accuracy.  In _Proceedings of the 2016 SIAM International Conference on Data Mining_ , pages 144–152, 2016.  doi: 10.1137/1.9781611974348.17.
  * Kamiran et al. (2012) Faisal Kamiran, Asim Karim, and Xiangliang Zhang.  Decision theory for discrimination-aware classification.  In _12th IEEE International Conference on Data Mining, ICDM_ , pages 924–929, 2012.  doi: 10.1109/ICDM.2012.45.
  * Lohia et al. (2019) Pranay Kr. Lohia, Karthikeyan Natesan Ramamurthy, Manish Bhide, Diptikalyan Saha, Kush R. Varshney, and Ruchir Puri.  Bias mitigation post-processing for individual and group fairness.  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP_ , pages 2847–2851, 2019.  doi: 10.1109/ICASSP.2019.8682620.
  * Zliobaite (2015) Indre Zliobaite.  On the relation between accuracy and fairness in binary classification.  _CoRR_ , abs/1505.05723, 2015.  URL http://arxiv.org/abs/1505.05723.
  * Bellamy et al. (2018) Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John T. Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang.  AI fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias.  _CoRR_ , abs/1810.01943, 2018.  URL http://arxiv.org/abs/1810.01943.
  * Chakraborty et al. (2020) Joymallya Chakraborty, Kewen Peng, and Tim Menzies.  Making fair ML software using trustworthy explanation.  In _35th IEEE/ACM International Conference on Automated Software Engineering, ASE_ , pages 1229–1233. IEEE, 2020.  doi: 10.1145/3324884.3418932.
  * Le Quy et al. (2022) Tai Le Quy, Arjun Roy, Vasileios Iosifidis, Wenbin Zhang, and Eirini Ntoutsi.  A survey on datasets for fairness-aware machine learning.  _WIREs Data Mining Knowl. Discov._ , 12(3), 2022.  doi: 10.1002/widm.1452.
  * Van Nuenen et al. (2020) Tom Van Nuenen, Xavier Ferrer, Jose M Such, and Mark Coté.  Transparency for whom? assessing discriminatory artificial intelligence.  _Computer_ , 53(11):36–44, 2020.
  * Zhang and Ntoutsi (2019) Wenbin Zhang and Eirini Ntoutsi.  FAHT: an adaptive fairness-aware decision tree classifier.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI_ , pages 1480–1486, 2019.  doi: 10.24963/ijcai.2019/205.
  * Savani et al. (2020) Yash Savani, Colin White, and Naveen Sundar Govindarajulu.  Intra-processing methods for debiasing neural networks.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/1d8d70dddf147d2d92a634817f01b239-Abstract.html.
  * Pessach and Shmueli (2023) Dana Pessach and Erez Shmueli.  A review on fairness in machine learning.  _ACM Comput. Surv._ , 55(3):51:1–51:44, 2023\.  doi: 10.1145/3494672.
  * Verma and Rubin (2018) Sahil Verma and Julia Rubin.  Fairness definitions explained.  In Yuriy Brun, Brittany Johnson, and Alexandra Meliou, editors, _Proceedings of the International Workshop on Software Fairness, FairWare@ICSE 2018, Gothenburg, Sweden, May 29, 2018_ , pages 1–7. ACM, 2018\.  doi: 10.1145/3194770.3194776.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nati Srebro.  Equality of opportunity in supervised learning.  In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_ , pages 3315–3323, 2016.  URL https://proceedings.neurips.cc/paper/2016/hash/9d2682367c3935defcb1f9e247a97c0d-Abstract.html.
  * Cortez (2019) Valeria Cortez.  How to define fairness to detect and prevent discriminatory outcomes in machine learning, 2019.  URL https://towardsdatascience.com/how-to-define-fairness-to-detect-and-prevent-discriminatory-outcomes-in-machine-learning-ef23fd408ef2.
  * Zafar et al. (2017a) Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi.  Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment.  In Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich, editors, _Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017_ , pages 1171–1180. ACM, 2017a.  doi: 10.1145/3038912.3052660.
  * Dwork et al. (2012) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel.  Fairness through awareness.  In _Innovations in Theoretical Computer Science_ , pages 214–226, 2012.  doi: 10.1145/2090236.2090255.
  * Zemel et al. (2013) Richard S. Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork.  Learning fair representations.  In _Proceedings of the 30th International Conference on Machine Learning, ICML_ , volume 28, pages 325–333, 2013.  URL http://proceedings.mlr.press/v28/zemel13.html.
  * Dunkelau (2020) Jannik Dunkelau.  Fairness-aware machine learning an extensive overview, 2020.  URL https://stups.hhu-hosting.de/downloads/pdf/fairness-survey.pdf.
  * Hort et al. (2022) Max Hort, Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman.  Bias mitigation for machine learning classifiers: A comprehensive survey.  _CoRR_ , abs/2207.07068, 2022.  doi: 10.48550/arXiv.2207.07068.
  * Friedler et al. (2019) Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P. Hamilton, and Derek Roth.  A comparative study of fairness-enhancing interventions in machine learning.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , pages 329–338, 2019.  doi: 10.1145/3287560.3287589.  URL https://doi.org/10.1145/3287560.3287589.
  * Schelter et al. (2020) Sebastian Schelter, Yuxuan He, Jatin Khilnani, and Julia Stoyanovich.  Fairprep: Promoting data to a first-class citizen in studies on fairness-enhancing interventions.  In _Proceedings of the 23rd International Conference on Extending Database Technology, EDBT_ , pages 395–398, 2020.  URL https://doi.org/10.5441/002/edbt.2020.41.
  * Calders et al. (2013) Toon Calders, Asim Karim, Faisal Kamiran, Wasif Ali, and Xiangliang Zhang.  Controlling attribute effect in linear regression.  In _2013 IEEE 13th International Conference on Data Mining_ , pages 71–80, 2013.  doi: 10.1109/ICDM.2013.114.
  * Berk et al. (2017) Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael J. Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth.  A convex framework for fair regression.  _CoRR_ , abs/1706.02409, 2017.  URL http://arxiv.org/abs/1706.02409.
  * Chzhen et al. (2020) Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil.  Fair regression via plug-in estimator and recalibration with statistical guarantees.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems NeurIPS_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/ddd808772c035aed516d42ad3559be5f-Abstract.html.
  * Putzel and Lee (2022) Preston Putzel and Scott Lee.  Blackbox post-processing for multiclass fairness, 2022.  URL http://ceur-ws.org/Vol-3087/paper_36.pdf.
  * Alghamdi et al. (2022) Wael Alghamdi, Hsiang Hsu, Haewon Jeong, Hao Wang, P. Winston Michalak, Shahab Asoodeh, and Flávio P. Calmon.  Beyond adult and COMPAS: fairness in multi-class prediction.  _CoRR_ , abs/2206.07801, 2022.  doi: 10.48550/arXiv.2206.07801.
  * Chierichetti et al. (2017) Flavio Chierichetti, Ravi Kumar, Silvio Lattanzi, and Sergei Vassilvitskii.  Fair clustering through fairlets.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems_ , pages 5029–5037, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/978fce5bcc4eccc88ad48ce3914124a2-Abstract.html.
  * Ziko et al. (2021) Imtiaz Masud Ziko, Jing Yuan, Eric Granger, and Ismail Ben Ayed.  Variational fair clustering.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI_ , pages 11202–11209, 2021.  URL https://ojs.aaai.org/index.php/AAAI/article/view/17336.
  * Abbasi et al. (2021) Mohsen Abbasi, Aditya Bhaskara, and Suresh Venkatasubramanian.  Fair clustering via equitable group representations.  In _FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 504–514, 2021.  doi: 10.1145/3442188.3445913.
  * Backurs et al. (2019) Arturs Backurs, Piotr Indyk, Krzysztof Onak, Baruch Schieber, Ali Vakilian, and Tal Wagner.  Scalable fair clustering.  In _Proceedings of the 36th International Conference on Machine Learning, ICML_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 405–413, 2019.  URL http://proceedings.mlr.press/v97/backurs19a.html.
  * Rösner and Schmidt (2018) Clemens Rösner and Melanie Schmidt.  Privacy preserving clustering with constraints.  In _45th International Colloquium on Automata, Languages, and Programming, ICALP_ , volume 107 of _LIPIcs_ , pages 96:1–96:14, 2018.  doi: 10.4230/LIPIcs.ICALP.2018.96.
  * Bera et al. (2019) Suman Kalyan Bera, Deeparnab Chakrabarty, Nicolas Flores, and Maryam Negahbani.  Fair algorithms for clustering.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS_ , pages 4955–4966, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/fc192b0c0d270dbf41870a63a8c76c2f-Abstract.html.
  * P and Abraham (2020) Deepak P and Savitha Sam Abraham.  Fair outlier detection.  In _Web Information Systems Engineering - WISE_ , volume 12343 of _Lecture Notes in Computer Science_ , pages 447–462, 2020.  doi: 10.1007/978-3-030-62008-0\\_31.
  * Galhotra et al. (2021) Sainyam Galhotra, Sandhya Saisubramanian, and Shlomo Zilberstein.  Learning to generate fair clusters from demonstrations.  _AIES: AAAI/ACM Conference on AI, Ethics, and Society_ , pages 491–501, 2021.  doi: 10.1145/3461702.3462558.
  * Iosifidis and Ntoutsi (2020) Vasileios Iosifidis and Eirini Ntoutsi.  FABBOO - online fairness-aware learning under class imbalance.  In _Discovery Science - 23rd International Conference, DS_ , volume 12323 of _Lecture Notes in Computer Science_ , pages 159–174, 2020\.  doi: 10.1007/978-3-030-61527-7\\_11.
  * Slack et al. (2020) Dylan Slack, Sorelle A. Friedler, and Emile Givental.  Fairness warnings and fair-maml: learning fairly with minimal data.  In _FAT* ’20: Conference on Fairness, Accountability, and Transparency_ , pages 200–209, 2020.  doi: 10.1145/3351095.3372839.  URL https://doi.org/10.1145/3351095.3372839.
  * Kamiran and Calders (2010) Faisal Kamiran and Toon Calders.  Classification with no discrimination by preferential sampling.  In _Proceedings of the 19th Machine Learning conference_ , 2010.  URL https://dtai.cs.kuleuven.be/events/Benelearn2010/submissions/benelearn2010_submission_18.pdf.
  * Kamiran et al. (2013) Faisal Kamiran, Indre Zliobaite, and Toon Calders.  Quantifying explainable discrimination and removing illegal discrimination in automated decision making.  _Knowledge and Information Systems_ , 35(3):613–644, 2013.  doi: 10.1007/s10115-012-0584-8.
  * Feldman (2015) Michael Feldman.  _Computational fairness: Preventing machine-learned discrimination_.  PhD thesis, Haverford College, 2015.
  * Ruoss et al. (2020) Anian Ruoss, Mislav Balunovic, Marc Fischer, and Martin T. Vechev.  Learning certified individually fair representations.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems NeurIPS_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/55d491cf951b1b920900684d71419282-Abstract.html.
  * Chawla et al. (2002) Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer.  SMOTE: synthetic minority over-sampling technique.  _J. Artif. Intell. Res._ , 16:321–357, 2002.  doi: 10.1613/jair.953.
  * Agarwal et al. (2018) Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna M. Wallach.  A reductions approach to fair classification.  In _Proceedings of the 35th International Conference on Machine Learning, ICML_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 60–69, 2018.  URL http://proceedings.mlr.press/v80/agarwal18a.html.
  * Choi et al. (2020) YooJung Choi, Golnoosh Farnadi, Behrouz Babaki, and Guy Van den Broeck.  Learning fair naive bayes classifiers by discovering and eliminating discrimination patterns.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI_ , pages 10077–10084. AAAI Press, 2020.  URL https://ojs.aaai.org/index.php/AAAI/article/view/6565.
  * Mancuhan and Clifton (2014) Koray Mancuhan and Chris Clifton.  Combating discrimination using bayesian networks.  _Artificial Intelligence and Law_ , 22(2):211–238, 2014.  doi: 10.1007/s10506-014-9156-4.
  * Bechavod and Ligett (2017) Yahav Bechavod and Katrina Ligett.  Learning fair classifiers: A regularization-inspired approach.  _CoRR_ , abs/1707.00044, 2017.  URL http://arxiv.org/abs/1707.00044.
  * Zhang et al. (2018) Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell.  Mitigating unwanted biases with adversarial learning.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES_ , pages 335–340. ACM, 2018.  doi: 10.1145/3278721.3278779.
  * Zafar et al. (2017b) Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi.  Fairness constraints: Mechanisms for fair classification.  In _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS_ , volume 54, pages 962–970, 2017b.  URL http://proceedings.mlr.press/v54/zafar17a.html.
  * Quadrianto and Sharmanska (2017) Novi Quadrianto and Viktoriia Sharmanska.  Recycling privileged learning and distribution matching for fairness.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems_ , pages 677–688, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html.
  * Morrison et al. (2016) David R. Morrison, Sheldon H. Jacobson, Jason J. Sauppe, and Edward C. Sewell.  Branch-and-bound algorithms: A survey of recent advances in searching, branching, and pruning.  _Discrete Optimization_ , 19:79–102, 2016.  doi: 10.1016/j.disopt.2016.01.005.
  * Wachter et al. (2021) Sandra Wachter, Brent Mittelstadt, and Chris Russell.  Bias preservation in machine learning: the legality of fairness metrics under eu non-discrimination law.  _West Virginia Law Review_ , 123, 2021.  URL https://researchrepository.wvu.edu/wvlr/vol123/iss3/4.
  * Vapnik and Vashist (2009) Vladimir Vapnik and Akshay Vashist.  A new learning paradigm: Learning using privileged information.  _Neural Networks_ , 22(5-6):544–557, 2009.  doi: 10.1016/j.neunet.2009.06.042.
  * Sharmanska and Quadrianto (2016) Viktoriia Sharmanska and Novi Quadrianto.  Learning from the mistakes of others: Matching errors in cross-dataset learning.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 3967–3975, 2016.  doi: 10.1109/CVPR.2016.430.
  * Schapire (1999) Robert E. Schapire.  A brief introduction to boosting.  In _Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, IJCAI_ , 1999.  URL http://ijcai.org/Proceedings/99-2/Papers/103.pdf.
  * Yu et al. (2021) Zhe Yu, Joymallya Chakraborty, and Tim Menzies.  Fairer machine learning software on multiple sensitive attributes with data preprocessing.  _CoRR_ , abs/2107.08310, 2021.  URL https://arxiv.org/abs/2107.08310.
  * Weinberg (2022) Lindsay Weinberg.  Rethinking fairness: An interdisciplinary survey of critiques of hegemonic ML fairness approaches.  _Journal of Artificial Intelligence Research_ , 74:75–109, 2022.  doi: 10.1613/jair.1.13196.
  * Haeri and Zweig (2020) Maryam Amir Haeri and Katharina Anna Zweig.  The crucial role of sensitive attributes in fair classification.  In _IEEE Symposium Series on Computational Intelligence, SSCI_ , pages 2993–3002, 2020.  doi: 10.1109/SSCI47803.2020.9308585.
  * GDPR (2016) GDPR.  General Data Protection Regulation (GDPR) – Official Legal Text, 2016.  URL https://gdpr-info.eu/.