11institutetext: School of Computer Science
University College Dublin
Dublin 4, Ireland

# Algorithmic Factors Influencing
Bias in Machine Learning

William Blanzeisky  Pádraig Cunningham

###### Abstract

It is fair to say that many of the prominent examples of bias in Machine
Learning (ML) arise from bias that is there in the training data. In fact,
some would argue that supervised ML algorithms _cannot_ be biased, they
reflect the data on which they are trained. In this paper we demonstrate how
ML algorithms can misrepresent the training data through _underestimation_. We
show how irreducible error, regularization and feature and class imbalance can
contribute to this underestimation. The paper concludes with a demonstration
of how the careful management of synthetic counterfactuals can ameliorate the
impact of this underestimation bias.

###### Keywords:

Bias Fairness Classification Model Capacity Regularisation

##  1 Introduction

As the applications of Machine Learning (ML) systems become ubiquitous in many
aspects of humans life, there is an undeniable appeal for discrimination-free
machine learning. An algorithm is considered to be discriminatory if it
systematically disadvantages people belonging to certain categories or groups,
instead of relying solely on individual merits [26]. Despite the increasing
number of related works over the past few years, the amount of anecdotal
evidence of algorithmic bias is still growing [2].

There are two main sources of algorithmic bias, it can be due to the data or
the algorithm. When it is due to the data, it is sometimes euphemistically
called _negative legacy_ , when it is due to the algorithm it is called
_underestimation_. [6, 17]. Negative legacy may be due to labeling errors or
poor sampling; however, it is likely to reflect discriminatory practices in
the past. Underestimation occurs when the algorithm focuses on strong signals
in the data thereby missing more subtle phenomena. Algorithmic aspects of how
ML algorithms can accentuate existing bias are not yet well-understood. Hence,
the focus of this paper is on underestimation.

Our central hypothesis is that underestimation occurs when an algorithm
_underfits_ the training data due to a combination of limitations in training
data and model capacity issues. This leads us to a series of sub-hypotheses –
that the following factors can contribute to underestimation:

  * •

Irreducible error (Bayes error)

  * •

Regularization mechanisms

  * •

Class imbalance

  * •

Under-represented categories

These hypotheses are presented in more detail in section 3. Before that the
relevant background research is reviewed in section 2. In section 4 these
hypotheses are tested on synthetic data. In section 5 we show that the
addition of synthetic counterfactuals to the training data can reduce the
impact of underestimation but there is clearly a need to manage the number of
counterfactuals. The paper concludes in section 6 with an assessment of how
this repair strategy works on real datasets.

##  2 Background

The issues around bias and fairness in ML research has received a lot of
attention in recent years. Several notions of what constitutes “fair” in ML
have been proposed [8]. Despite this, there is still no unified consensus of
what the best fairness notion shall be. In fact, recent works show that some
of these notions suffer from significant statistical limitations, or even
might perversely harm the very groups they were designed to protect [5]. In
general, an ML model is considered fair if it is not inclined to award the
desirable outcome $Y=1$ (e.g. loan approval/job offers) only to one side of
sensitive category $S\neq 1$ (e.g. gender/race).

The fairness literature has mainly focused on implementing new methods to
ensure fairness without explicitly considering the source of bias [3]. Recent
efforts in rectifying algorithmic bias include: transforming the dataset to
remove discrimination before feeding it into a ML model (pre-processing) [16],
modifying a specific algorithm’s loss function to account for fairness (in-
processing) [17], or transforming the ML model’s output to ensure fairness
(post-processing) [12] .

Many argue that algorithmic bias is purely caused by negative legacy and that
the algorithm models the data correctly (no underestimation). However, recent
work shows that the algorithm itself could amplify existing bias [6, 15]. Of
particular interest in this regard is research on model pruning in deep neural
networks [15, 14]. Since the 1990s there has been research on how a
significant proportion of the weights in a neural network can be pruned with
minimal impact on model accuracy. Recent work at Google Brain by Hooker _et
al._ [15, 14] shows that this pruning can have a significant impact on
underestimation. This phenomenon is directly related to the regularization
issue we demonstrate in section 4.3.

Although most examples of bias in ML systems occur due to negative legacy, we
argue that it is important to understand how specific algorithm mechanisms can
introduce or at least accentuate bias. Understanding how algorithms can become
discriminatory on the algorithmic level could help researchers in developing
better and more general strategies to ensure fairness in ML algorithms. We
believe that some aspects of ML training, such as choices around model
architectures, hyper-parameters and optimization criteria, could influence
algorithmic bias.

###  2.1 Quantifying Bias

Disparate Impact ($\mathrm{DI}_{S}$) is one of the accepted definitions of
unfairness [9]:

$\mathrm{DI}_{S}\leftarrow\frac{P[\hat{Y}=1|S\neq 1]}{P[\hat{Y}=1|S=1]}<\tau$
(1)

It is the ratio of desirable outcomes $\hat{Y}$ predicted for the sensitive
minority $S\neq 1$ compared with that for the majority $S=1$. $\tau=0.8$ is
the 80% rule, i.e. proportion of desirable outcomes for the minority should be
within 80% of those for the majority.

When the focus is on bias due to the algorithm only we can define an
_underestimation score_ ($\mathrm{US}_{S}$) in line with $\mathrm{DI}_{S}$:

$\mathrm{US}_{S}\leftarrow\frac{P[\hat{Y}=1|S\neq 1]}{P[Y=1|S\neq 1]}$ (2)

This is the ratio of desirable outcomes predicted by the classifier for the
sensitive minority compared with what is actually present in the data. If
$\mathrm{US}_{S}<1$ the classifier is under-predicting desirable outcomes for
the minority. It is also important to note that $\mathrm{US}_{S}=1$ does not
necessarily mean that the classifier is not biased against the minority group
(i.e. poor $\mathrm{DI}_{S}$) score. It simply means that the algorithm does
not underestimate the predictions of desirable outcome for the minority group.

##  3 Factors Contributing to Underestimation

The definition of disparate impact emphasises fairness for all subgroups – see
equation 1. This definition is independent of any data, indeed disparate
impact is likely to be caused by biased historic data. By contrast the
definition of underestimation depends on test data (equation 2) – an outcome
is under-predicted for a category. Our hypothesis is that this will occur when
a model underfits the data. We identify four factors that can contribute to
this.

####  3.0.1 Irreducible Error:

In supervised ML the assumption is that the outcome variable $Y$ is determined
by the inputs to the model. Normally the inputs do not completely determine
the outcome (there are hidden factors involved) so even the best model will
have some error. This is the irreducible error, sometimes referred to as Bayes
error [24]. If the irreducible error is low the best models will be able to
fit well to the training data and still generalise well to unseen data. If the
irreducible error is high ($\sim 25\%$) then models that can generalise well
will need to be relatively simple, i.e. they will underfit the training data.
We show in section 4.2 that underestimation directly correlates with
irreducible error.

####  3.0.2 Regularization Mechanisms:

It is standard practice in ML to use regularization methods to reduce
generalization error. These methods are used to control the model capacity to
gather relevant information from the training set. The objective of ensuring
that the model does not overfit or underfit this information is crucial in ML
because it is tied to the model’s ability to generalize well on unseen data.
There have been many proposed regularization techniques in the ML literature:
lasso, ridge, elastic-net, dropout, early-stopping, etc [23, 13]. The main
idea behind these methods is to control the model’s complexity: in some
circumstances, this is done by adding a penalty term in the loss function, or
by limiting the number of nodes in the hidden layer. In regression settings,
these methods are used to mitigate over-fitting by penalizing the effect of
each predictor in explaining the target variable (reduce variance). Thus,
excessive regularization will likely cause a model to underestimate
predictions for the minority class. Since regularization directly correlates
with bias and variance of a model, we show that it also influences
underestimation (see section 4.3).

####  3.0.3 Class Imbalance:

Class imbalance refers to a classification problem where the number of
observations in the data set differs for each class. The potential for this to
lead to bias has been known for some time. It is known that the model
predictions can accentuate the bias: if the minority class represents 30% of
the training data the model is likely to predict $<30\%$ for that class [20]
[22]. Training an ML model without addressing this issue will make it more
challenging for a model to learn the characteristics of examples from the rare
events, resulting in predictions that underestimate the overall minority class
(i.e. biased towards predicting the majority) [11]. Many remediation
strategies have been proposed to combat these issues, such as: sampling, cost-
sensitive learning, etc [25]. In section 4, we will see the varying impact of
class imbalance on underestimation.

####  3.0.4 Under-represented Categories:

One of the most common causes of bias in ML is the lack of observations for
under-represented groups/categories. For example, given a protected
attribute/feature such as race or gender, if a model is trained on 5000
observations in which 80% belongs to the majority and only 20% represents the
minority group, the model will likely struggle to learn as effectively from
the minority examples. It is important to distinguish between this and the
class imbalance issue highlighted above. This effect is aggravated when the
irreducible error is high (the model needs to be simple). Although the
question of directly using protected attributes in ML training remains open
[27], we show in section 4.4 that the level of under-representation directly
correlates with underestimation.

##  4 Underestimation on Synthetic Data

In this section, we present results of the experiments based on the hypothesis
presented in section 3. These baseline results are demonstrated on a synthetic
data model that is widely used in research on bias on ML [1].

Figure 1: Probability density functions of the synthetic data where the blue
and orange curve shows the distribution of SAT scores for admitted and
rejected applicants respectively.

###  4.1 Data Description

This dataset consists of 5000 observations, each represents an applicant’s
statistics, which include IQ score, SAT score and a sensitive attribute
(Favored, or Discriminated) [1]. The target variable, admit, indicates whether
or not a student is admitted. The synthetic data generated has the following
key properties: (i) The observations are evenly distributed (50% for both
favored and discriminated group); (ii) The IQ scores are randomly sampled
between 80 and 120; (iii) To reflect some correlation, the SAT scores are
stochastic but dependent on the sensitive attribute and IQ; (iv) The admit
outcome is also stochastic but correlated with the SAT score; and (v) since
students in the favored group on average have higher SAT score than those that
are in discriminated group, favored students have been admitted at a higher
rate than discriminated students.

70% of these observations are used for training, and the remaining 30% are
reserved for model testing. We run experiments on seven classifiers
implemented in scikit-learn111https://scikit-learn.org/. These classifiers
include tree-based classifiers (Decision Tree, Gradient Boost and Random
Forest), and four other classifiers; $k$-Nearest Neighbor, Naive Bayes,
Logistic Regression & Neural Networks. Taking into account the stochastic
process of generating the synthetic sensitive attributes, each experiments
were repeated twenty times and the median underestimation on the test set is
obtained.

(a) Decision Tree

(b) k-Nearest Neighbor

(c) Logistic Regression

(d) Naive Bayes

(e) Neural Network

(f) Random Forest

(g) Gradient Boosting

Figure 2: Impact of irreducible error on underestimation.

###  4.2 Impact of Irreducible Error

To illustrate impact of irreducible error on underestimation, we added
Gaussian noise $X\sim\mathcal{N}(\mu,\sigma^{2})$ to the numeric independent
variables. The rationale is that the more noise added, the higher the
irreducible error. In this experiment, all seven classifiers are optimized on
balanced accuracy with 10-cross validation. The plots in Figure 2 show how
underestimation varies as the noise is increased. We see that underestimation
is magnified when irreducible errors are high. In addition, we can observe
that there is a clear relationship between underestimation and balanced
accuracy. This supports our hypothesis presented in section 3 that high
irreducible error results in simpler models, resulting in bias accentuation
for the minority class (i.e. bias towards majority).

###  4.3 Impact of Regularization

Next, we evaluated the impact of regularization on underestimation. For this
we consider the four classifiers that allow for explicit control of
regularization, these are Decision Tree, Gradient Boosting, Neural Network and
Logistic Regression. For example, the Neural Network implementation provides
an $\alpha$ parameter to control over-fitting using $l2$ regularization.
Figure 3 shows how underestimation varies with these parameters. It is clear
that underestimation is exacerbated as the strength of regularization
increases (i.e. underfitting). This makes sense since higher regularization
means that the model’s ability to learn from the training data is restricted,
and thus when the observations for minority group are extremely scarce, the
model will tend to focus on the majority, resulting in underestimation for
minority group.

(a) Decision Tree

(b) Gradient Boosting

(c) Neural Network

(d) Logistic Regression

Figure 3: Impact of regularization on underestimation.

(a) k-Nearest Neighbor

(b) Logistic Regression

(c) Naive Bayes

(d) Decision Tree

(e) Gradient Boosting

(f) Random Forest

(g) Neural Network

Figure 4: Impact of class and feature imbalance on underestimation.

###  4.4 Impact of Class and Feature Imbalance

Lastly, we illustrate how sensitive underestimation is to distribution
variations in the class label and the sensitive feature. As can be seen in the
heatmaps in Figure 4, the Class Imbalance $P(Y=1)$ varies between 10% and 30%
and the Feature Incidence $P(Y=1|S=0)$ between 10% and 50%. Each model is
optimized on balanced accuracy with 10-fold cross-validation. We see that the
color of the heatmap gets darker as it gets to the most extreme case of class
and feature imbalance (top left corner). This indicates that underestimation
is accentuated when observations for minority class is scarce and categories
are extremely underrepresented.

##  5 Remediation

In this section, we discuss possible strategies to remediate underestimation.
We have shown in Section 4 that underestimation occurs when the classifier
underfits the data due to a combination of limitations in training data and
model capacity issues. Hence, the most obvious approach to fix this problem is
to get more data. Unfortunately, quantifying exactly how much data is needed
is not a trivial task in practice.

Many of the strategies for addressing Disparate Impact are also applicable for
this more specific task of addressing underestimation, these include:

  * •

Pre-processing: increasing the sample size for minority group,

  * •

In-processing: adding a constraint to a specific algorithm’s loss function to
account for underestimation, using cost-sensitive learning, or explicitly
considering underestimation in hyper-parameter tuning,

  * •

Post-processing: or selecting different optimal threshold value for the
minority group.

We evaluated multiple strategies to reduce the impact of underestimation.
First, we explicitly consider underestimation in hyper-parameter tuning.
However, preliminary experiments on this strategy did not work out, showing
only 2% improvement in underestimation. We have had more success with pre-
processing strategies and we report on these results here. We have evaluated
two strategies for generating counterfactuals (see section 5.1) and a
variation on the SMOTE algorithm for generating synthetic data samples.

SMOTE [4] is perhaps the most popular method for addressing class imbalance in
ML. SMOTE is a data augmentation strategy whereby synthetic samples of the
minority class are generated by interpolating between real samples. Our
modification on this (SMOTEF) is to only select synthetic samples
corresponding to the minority group. These are referred to as $S_{0}Y_{1}$ in
Algorithm 1, i.e. the Discriminated feature and the Positive class. In the
notation of Algorithm 1, the Favoured Positive ($S_{1}Y_{1}$) samples produced
by SMOTE are discarded.

###  5.1 Adding Counterfactuals

Counterfactual reasoning is an idea from Philosophy that first received
attention in Artificial Intelligence research in the 1980s [10]. More recently
counterfactuals have been used for explanation and data augmentation in ML
[18, 21]. Whereas our SMOTEF strategy generates synthetic examples by
interpolating between real ($Y=1|S=0$) examples we have two options for
creating counterfactuals:

  * •

CounterfactualF: creating a desirable outcome for the minority group
($Y=1|S=0$) from a undesirable outcome ($Y=0|S=0$), or

  * •

CounterfactualL: a desirable outcome for the minority group ($Y=1|S=0$) from
one of the majority group ($Y=1|S=1$).

The pseudocode for these strategies is shown in Algorithm 1. A key
consideration with the three data augmentation strategies is the number of new
samples $N$ required to mitigate underestimation ($\mathrm{US}_{S}=1$). In our
first experiments we simply double the number of samples representing the
desirable outcome for the minority group ($Y=1|S=0$). The results on varying
levels of under-representation of class labels and sensitive features are
shown in Figure 5. The classifier is the Neural Network used in earlier
experiments. Due to the stochastic nature of the counterfactuals generation
process, we repeat each experiment twenty times and the median of
underestimation is obtained.

Input : A dataset $D(X,Y,S)$, classifier $h(x)\rightarrow[0,1]$

Output : Classifier trained on repaired dataset $h(\bar{D})\rightarrow[0,1]$

  1. 1.

Divide $D$ into four groups such that:

$S_{0}Y_{1}\leftarrow\\{x\in D|x.S=0\wedge x.Y=1$}

$S_{0}Y_{0}\leftarrow\\{x\in D|x.S=0\wedge x.Y=0$}

$S_{1}Y_{1}\leftarrow\\{x\in D|x.S=1\wedge x.Y=1$}

$S_{1}Y_{0}\leftarrow\\{x\in D|x.S=1\wedge x.Y=0$}

  2. 2.

$N\leftarrow|S_{0}Y_{1}|$

  3. 3.

if CounterfactualL:

Randomly sample $N$ observations from $S_{0}Y_{0}$ with replacement

Change the class labels of the sampled data $Y$ to 1
elif CounterfactualF:

Randomly sample $N$ observations from $S_{1}Y_{1}$ with replacement

Change the sensitive attribute of the sampled data $S$ to 0

  4. 4.

Concatenate $D$ with the sampled data

  5. 5.

Train a classifier $h(x)$ on modified $D$

  6. 6.

Return Classifier trained on repaired dataset $h(\bar{D})\rightarrow[0,1]$

Algorithm 1 Adding counterfactuals

(a) No repair

(b) SmoteF

(c) CounterfactualF

(d) CounterfactualL

Figure 5: The impact of adding synthetic samples using SmoteF and the two
counterfactual strategies. In this baseline analysis we always double the size
of the $DP$ set (see Algorithm 1).

We can see that both CounterfactualF CounterfactualL performs best in almost
all cases of under-representation. In fact, both of these strategies overshoot
the underestimation target ($\mathrm{US}_{S}=1$) in extreme cases. In the next
subsection we use a cross-validation strategy on the training data to select
appropriate values for $N$.

###  5.2 Tuning

It is clear from Figure 5 that the impact of data augmentation varies
considerably depending on the dataset make-up. For instance, the
CounterfactualF strategy produces $US_{S}$ outcomes that vary between 0.21 and
3.3. However, we should be able to tune the process by estimating the best
value for $N$ using cross-validation.

Figure 6: Tuned remediation: Remediation is repeated on the dataset from
Figure 5 with class imbalance of 20% and feature imbalance of 45% but with $N$
set using cross-validation on the training data.

Given that CounterfactualL draws from $S_{0}Y_{0}$ and CounterfactualF draws
from $S_{1}Y_{1}$ we can use up to 100% of these sets in the augmentation
process. The validation process (i.e. parameter tuning) considers options from
5% to 100% in steps of 5%. In this experiment the SMOTEF strategy is also
tuned by considering options from 5% to 100% of
$N\leftarrow|S_{1}Y_{1}|-|S_{0}Y_{1}|$.

We used a cross-validation process to select the best value for $N$ for each
strategy, then the whole training dataset is augmented and the remaining 30%
that has been held back for testing is used to assess performance. The results
are shown in Figure 6. The tuned SMOTEF does little to improve underestimation
but the two counterfactual strategies do a good job, both bringing $US_{S}$
close to 1.0. It is worth noting that, in addition to fixing underestimation,
CounterfactualF also improves balanced accuracy from 60.3% to 62.8%.

##  6 Underestimation on Real Datasets

In this section, we experimentally validate our proposed remediation
strategies described in Section 5 on the Census Income dataset [19] and a
reduced version of the ProPublica Recidivism dataset [7]. These datasets have
been extensively studied in fairness research because there is clear evidence
of negative legacy. Summary statistics for these datasets are provided in
Table 1. For the Census Income dataset the prediction task is to determine
whether a person earns more or less than $50,000 per year based on their
demographic information. The reduced and anonymized version of the Recidivism
dataset includes 7 features and the target variable represents whether a
person got rearrested within two years after the first arrest. The goal of our
experiment is to learn an underestimation-free classifier while maintaining
high balanced accuracy score when Sex and Caucasian used as sensitive feature
$S$ for Income and Recidivism dataset, respectively. The classifier is the
Neural Network as used in section 5.2 and the same tuning strategy is employed
to determine the level of data augmentation.

Table 1: Summary details of the Adult and Recidivism datasets. Dataset | Samples | Features | % Minority
---|---|---|---
Census Income | 48,842 | 14 | 25%
Recidivism | 7,214 | 7 | 45%

(a) Census Income

(b) Recidivism

Figure 7: An evaluation of the remediation strategies on two real-world
datasets.

The results shown in Figure 7 demonstrate that all of the proposed methods
reduce the impact of underestimation. Looking closer at the Census Income
dataset, we see that CounterfactualF overshoots underestimation by 1̃0%. We
suspect that this impact can be removed (or at least reduced) by allowing
larger search space for $N$ and hence, we can conclude that the proposed
strategies are very sensitive to hyper-parameter $N$. Moreover, the results in
Figures 6 and 7 show that there is no one remediation strategy that performs
best in all datasets, suggesting the importance of tuning. It is worth noting
that CounterfactualF improves balanced accuracy for all three datasets. This
suggests that it is a natural policy for producing counterfactuals.

These methods could also easily be extended to account for multi-label
sensitive group. We believe that the effectiveness of these strategies is
dependent on how different the distribution of the test set to the training
set. More research need to be done to further refine these methods to account
for distribution shift.

##  7 Conclusions and Future Work

This paper started by emphasising the difference between negative legacy and
underestimation as sources of bias in ML. The first is a problem with the
data, the second is a problem with the algorithm. We have shown that
underestimation can be addressed by adding counterfactuals, suggesting that
perhaps it is really all about the data. It’s not. Negative legacy refers to
scenarios where there are undesirable patterns in historic data.
Underestimation refers to scenarios where the algorithm is not picking up the
patterns that _are_ in the data. It is worth saying then that fixing
underestimation in the sense that the algorithm reflects the data may still
leave a fairness problem due to negative legacy.

While our evaluation shows that the use of counterfactuals can ameliorate
underestimation there is still some room for improvement. Our next step will
be to use ideas from the work of Keane and Smyth [18] to select better
counterfactuals. In future we plan to move on to develop in-processing
strategies whereby underestimation is explicitly considered in the ML
algorithm optimization process.

## Acknowledgements

This work was funded by Science Foundation Ireland through the SFI Centre for
Research Training in Machine Learning (Grant No. 18/CRT/6183) with support
from Microsoft Ireland.

## References

  * [1] Adler, P., Falk, C., Friedler, S.A., Rybeck, G., Scheidegger, C., Smith, B., Venkatasubramanian, S.: Auditing black-box models for indirect influence (2016)
  * [2] Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: Can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. p. 610–623. FAccT ’21, Association for Computing Machinery, New York, NY, USA (2021)
  * [3] Caton, S., Haas, C.: Fairness in machine learning: A survey. arXiv preprint arXiv:2010.04053 (2020)
  * [4] Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: Synthetic minority over-sampling technique. J. Artif. Int. Res. 16(1), 321–357 (Jun 2002)
  * [5] Corbett-Davies, S., Goel, S.: The measure and mismeasure of fairness: A critical review of fair machine learning (2018)
  * [6] Cunningham, P., Delany, S.J.: Algorithmic bias and regularisation in machine learning. arXiv preprint arXiv:2005.09052 (2020)
  * [7] Dressel, J., Farid, H.: The accuracy, fairness, and limits of predicting recidivism. Science advances 4(1), eaao5580 (2018)
  * [8] Dunkelau, J., Leuschel, M.: Fairness-aware machine learning (2019)
  * [9] Feldman, M., Friedler, S.A., Moeller, J., Scheidegger, C., Venkatasubramanian, S.: Certifying and removing disparate impact. In: proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. pp. 259–268 (2015)
  * [10] Ginsberg, M.L.: Counterfactuals. Artificial intelligence 30(1), 35–79 (1986)
  * [11] Guo, X., Yin, Y., Dong, C., Yang, G., Zhou, G.: On the class imbalance problem. In: 2008 Fourth International Conference on Natural Computation. vol. 4, pp. 192–201 (2008)
  * [12] Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. p. 3323–3331. NIPS’16, Curran Associates Inc., Red Hook, NY, USA (2016)
  * [13] Hoerl, A.E., Kennard, R.W.: Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12(1), 55–67 (1970)
  * [14] Hooker, S., Courville, A., Clark, G., Dauphin, Y., Frome, A.: What do compressed deep neural networks forget? (2020)
  * [15] Hooker, S., Moorosi, N., Clark, G., Bengio, S., Denton, E.: Characterising bias in compressed models. arXiv e-prints pp. arXiv–2010 (2020)
  * [16] Kamiran, F., Calders, T.: Data pre-processing techniques for classification without discrimination. Knowledge and Information Systems 33 (10 2011)
  * [17] Kamishima, T., Akaho, S., Asoh, H., Sakuma, J.: Fairness-aware classifier with prejudice remover regularizer. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 35–50. Springer (2012)
  * [18] Keane, M.T., Smyth, B.: Good counterfactuals and where to find them: A case-based technique for generating counterfactuals for explainable ai (xai). In: International Conference on Case-Based Reasoning. pp. 163–178. Springer (2020)
  * [19] Kohavi, R.: Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining. vol. 96, pp. 202–207 (1996)
  * [20] Kubat, M., Holte, R.C., Matwin, S.: Machine learning for the detection of oil spills in satellite radar images. Machine learning 30(2), 195–215 (1998)
  * [21] Kusner, M.J., Loftus, J., Russell, C., Silva, R.: Counterfactual fairness. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017)
  * [22] Mac Namee, B., Cunningham, P., Byrne, S., Corrigan, O.I.: The problem of bias in training data in regression problems in medical decision support. Artificial intelligence in medicine 24(1), 51–70 (2002)
  * [23] Thodberg, H.H.: Improving generalization of neural networks through pruning. International Journal of Neural Systems 01(04), 317–326 (1991)
  * [24] Tumer, K., Ghosh, J.: Estimating the Bayes error rate through classifier combining. In: Proceedings of 13th International Conference on Pattern Recognition. vol. 2, pp. 695–699. IEEE (1996)
  * [25] Weiss, G.M.: Mining with rarity: A unifying framework. SIGKDD Explor. Newsl. 6(1), 7–19 (Jun 2004)
  * [26] Žliobaitė, I.: Measuring discrimination in algorithmic decision making. Data Mining and Knowledge Discovery 31(4), 1060–1089 (jul 2017)
  * [27] Žliobaitė, I., Custers, B.: Using sensitive personal data may be necessary for avoiding discrimination in data-driven decision models. Artificial Intelligence and Law 24(2), 183–201 (may 2016)