# Harnessing value from data science in business: ensuring explainability and
fairness of solutions

Krzysztof Chomiak London School of Economics alumnus,
https://www.linkedin.com/in/krzysztof-chomiak/ Michał Miktus Paris School of
Economics alumnus, https://www.linkedin.com/in/michalmiktus/

###### Abstract

The paper introduces concepts of fairness and explainability (XAI) in
artificial intelligence, oriented to solve a sophisticated business problems.
For fairness, the authors discuss the bias-inducing specifics, as well as
relevant mitigation methods, concluding with a set of recipes for introducing
fairness in data-driven organizations. Additionally, for XAI, the authors
audit specific algorithms paired with demonstrational business use-cases,
discuss a plethora of techniques of explanations quality quantification and
provide an overview of future research avenues.

††
The authors would like to thank Przemysław Biecek, Tomasz Trzciński and Tomasz
Bartkowiak for all their careful, constructive and insightful comments in
relation to this work.

###### Contents

  1. 1 Introduction
  2. 2 Fairness in AI
    1. 2.1 Mathematical definition of fairness
      1. 2.1.1 Demographic parity
      2. 2.1.2 Equality of odds
      3. 2.1.3 Equality of opportunity
    2. 2.2 Sources of bias
      1. 2.2.1 Input data bias
      2. 2.2.2 Algorithmic bias
      3. 2.2.3 Detecting bias
    3. 2.3 Mitigating bias
      1. 2.3.1 Ante-hoc fairness
      2. 2.3.2 In-processing fairness
      3. 2.3.3 Post-hoc fairness
    4. 2.4 Tools for fairness
    5. 2.5 Putting fairness to work
  3. 3 eXplainable Artificial Intelligence
    1. 3.1 Rationalization of XAI
    2. 3.2 Desired characteristics of XAI
    3. 3.3 XAI techniques
      1. 3.3.1 Ante-hoc explainability
      2. 3.3.2 Explainable modelling
      3. 3.3.3 Post-hoc explainability
      4. 3.3.4 Evaluating XAI Performance
    4. 3.4 Expected evolution of XAI
  4. 4 Conclusion

##  1 Introduction

Artificial Intelligence (AI) is incessantly flourishing in sophistication and
complexity. The latest developments of intelligent systems, trained to take
autonomous and consistent decisions, lead to several transformational
opportunities for business and society. Consequently, machine learning has
encroached into a plethora of business domains, many of them requiring high
level of accountability, and thus transparency and fairness.

Undoubtedly, the full understanding and fairness of model structure and its
outcomes is a prerequisite for building trust and adoption of AI systems. It
is particularly true in high-stakes disciplines requiring reliability and
safety, such as healthcare or automated transportation, as well as in
demanding industrial applications with critical economic implications, not to
mention exploration of natural resources or climate change modelling.
Nevertheless, recent mainstream machine learning research focused primarily on
improving accuracy scores on benchmark datasets, neglecting the non-trivial
implications of relying on such models.

A good illustration of the bias embedded in contemporary machine learning
algorithms is a widely-cited technique proposed by Ribeiro et al., (2016) that
could recognize wolves from husky dogs, reaching its decision by considering
solely the presence of snow in the background. Another example with serious
business implications is the controversy surrounding the black-box-based
credit scoring process behind the Apple Card with alleged discrimination based
on gender. 111Example article covering the situation:
https://edition.cnn.com/2019/11/10/business/goldman-sachs-apple-card-
discrimination/index.html Other instances include a predictive model for risk
of repeating criminal offences with a strong ethnic bias (Angwin et al.,
2016), or finally the adversarial examples, where some attackers could
selectively add distortions to images to impose on an otherwise robust
classifier to always choose a certain class (Szegedy et al.,, 2014). With an
incessant intensification of instances of artificial intelligence missteps,
the demand for ensuring fairness of models in a systematic manner is
sprouting.

The above-mentioned shortcomings and obvious failures of machine learning
methods led to the birth of research of fair and explainable AI, considered as
subfields of AI that aim to satisfy the inherent need to control and explain
machine decisions, simultaneously guaranteeing that it will continue to
perform as expected when deployed in a real-world environment.

The remainder of this paper is organized into two main sections:

  * •

Fairness in AI - the section discusses various definitions of fairness,
analyzes sources of bias, provides methods of mitigating bias at different
modelling phases, introduces state-of-the-art tools for fairness and finally
discusses a high-level guideline of implementing AI fairness in an
organization,

  * •

eXplainable AI (XAI) - the section starts with an outline of fundamental
reasons for XAI emergence. Then, basic characteristics of proper
explainability methods are summarized, leading to the synopsis of XAI
techniques applied before (pre-modelling explainability), during (explainable
modelling), and after (post-modelling explainability) the modelling stage,
with a comparison of their respective strengths and weaknesses, as well as an
illustration of business use-cases employing explainable artificial
intelligence algorithms. The final subsection provides measures for evaluating
performance of models in explainability tasks, such as Lipschitz value or
faithfulness indicator.

##  2 Fairness in AI

The fairness of AI algorithms is a burgeoning field of research that stems
from the general requisite for decisions to be unfettered from bias and
discrimination. In general, it can be delineated as the absence of any
prejudice or favouritism toward an individual or a group based on their
inherent or acquired characteristics. Nevertheless, it has to be remembered
that it is a highly nuanced concept and its definition differs across
cultures, societies, and even over time.

######  Remark.

Statisticians proposed a plethora of mathematical criteria to be satisfied for
a classifier or a model to be characterised as fair. The non-exhaustive list
of twenty-one fairness definitions can be found at the Arvind Narayanan
tutorial at ACM FAT (Fairness, Accountability and Transparency) Conference in
2018. 222https://www.youtube.com/watch?v=jIXIuYdnyyk

It has to be emphasised that eliminating AI bias is not just an ethical
obligation, yet it should be regarded as an economic and competitive
imperative. By imposing the AI systems to be fair, organizations may become
more profitable and productive. Consider the market for credit, which has a
distasteful history of redlining against racial minorities and consequently,
to a large extent, effectively restraining them out of the financial system
and limiting market size for credit providers. As part of its fairness
governance policy, the credit bureau Experian refined its Boost program to
permit clients with limited credit history to augment their scores with TV,
phone bills or even Netflix history 333https://www.experian.com/consumer-
products/score-boost.html. Since the program launch in January 2019, more than
4 million customers have participated, while more than 150 thousand people
with weak credit rating have been able to be upgraded to a mediocre one,
providing significantly more funding opportunities.

###  2.1 Mathematical definition of fairness

As mentioned in the previous section, it is relatively strenuous to align on
one clear concept of fairness in AI. Most definitions of unbiasedness are
connected to the group fairness, which handles with statistical fairness
across the whole population. Alternatively, the complementary notion of
individual fairness mandates that similar individuals should be treated
equivalently regardless of their group membership.

In order to grasp the arduousness of exact fairness definition, please find
below three main notions of group fairness. Consider input data, denoted by
$\bm{x}$, and machine learning model $f(\bm{x})$ that aims to predict a binary
outcome $\bm{y}\in\\{0,1\\}$444Proposed definitions can be easily generalised
for multilevel outcome.. Please note that the outcomes may differ
statistically between different populations. Moreover, assume that each record
$\bm{x}$ possesses a binary protected attribute $\bm{p}\in\\{0,1\\}$, such as
gender or ethnicity. In the literature observations where $\bm{p}=0$ are
referred to as the deprived population, and conversely instances where
$\bm{p}=1$ as the favoured population. Similarly, $\bm{y}=1$ denotes the
favoured outcome (representing more desirable of the two possible results) and
the goal is to estimate $\bm{\hat{y}}$

####  2.1.1 Demographic parity

Demographic parity or statistical parity claims that a predictor is unbiased
if the prediction $\bm{\hat{y}}$ is independent of the protected attribute
$\bm{p}$. In other words:

$\mathbf{P}(\bm{\hat{y}}|\bm{p})=\mathbf{P}(\bm{\hat{y}})$

Because the identical proportion of each population is classified as positive,
it may lead to different false positive and true positive rates if the true
outcome $\bm{y}$ does actually vary with the protected attribute $\bm{p}$.

Consequently, the deviations from fairness may be measured by the statistical
parity difference:

$\text{SPD}=\mathbf{P}(\bm{\hat{y}}=1|\bm{p}=1)-\mathbf{P}(\bm{\hat{y}}=1|\bm{p}=0)$

or equivalently the statistical parity ratio:

$\text{SPR}=\frac{\mathbf{P}(\bm{\hat{y}}=1|\bm{p}=1)}{\mathbf{P}(\bm{\hat{y}}=1|\bm{p}=0)}$

####  2.1.2 Equality of odds

Equality of odds requires that the prediction $\bm{\hat{y}}$ is conditionally
independent to the protected attribute $\bm{p}$, given the true value
$\bm{y}$. To put it differently:

$\mathbf{P}(\bm{\hat{y}}|\bm{y},\bm{p})=\mathbf{P}(\bm{\hat{y}}|\bm{y})$

As a consequence, the true positive rate and false positive rate are the same
for each population.

####  2.1.3 Equality of opportunity

Equality of opportunity is the equality of odds focuses solely for positive
outcome, namely:

$\mathbf{P}(\bm{\hat{y}}|\bm{y}=1,\bm{p})=\mathbf{P}(\bm{\hat{y}}|\bm{y}=1)$

Similarly to the statistical parity difference, the equal opportunity
difference is defined as:

$\text{EOD}=\mathbf{P}(\bm{\hat{y}}=1,\bm{y}=1,\bm{p}=1)-\mathbf{P}(\bm{\hat{y}}=1,\bm{y}=1,\bm{p}=0)$

while the equal opportunity ratio:

$\text{EOR}=\frac{\mathbf{P}(\bm{\hat{y}}=1,\bm{y}=1,\bm{p}=1)}{\mathbf{P}(\bm{\hat{y}}=1,\bm{y}=1,\bm{p}=0)}$

###  2.2 Sources of bias

In order to fully unleash the profitable opportunities of AI, one needs to
identify and understand the sources of discrimination. Biased human judgements
can affect AI systems in two alternative ways: input data or employed
algorithms.

####  2.2.1 Input data bias

First source of AI bias is inherently connected to the quality of input data.
More specifically, the training dataset can be non-representative or it may
possess the societal bias.

Machine learning models rely on large datasets in order to learn to identify
patterns. As a consequence, infrequent motifs may be down-weighted in the sake
of generalization abilities, leading the minority records to be unfairly
neglected. Moreover, the lack of quality data may not be driven purely by the
group’s negligible size. Namely, the data collection methodology can exclude
or disadvantage some classes, for instance if it is performed solely in one
language. In addition, missing values leading to the records removal may be
more prevalent in some factions than others.

For instance, Buolamwini and Gebru, (2018) evaluated the most commonly
employed facial analysis algorithms and recognized the considerable
disparities in their accuracy for classifying the darker-skinned females
(error rates of up to 34.7%) and the lighter-skinned males (0.8%). The
thorough investigation revealed that the training datasets did not contain
enough examples of darker-skinned females for the model to perform
satisfactorily well.

Along the same vein, even if the amount of data is sufficient to adequately
represent each class, the training dataset may reflect existing societal and
historical prejudices, such as the phenomenon that female workers are on
average paid less. Such historical unfairness in data is often referred to as
negative legacy.

As an example, consider the natural language processing algorithm to predict
the next word in the following sentence: father is to a doctor as mother is to
a. Even if it had been trained on the whole data from Wikipedia, due to the
indelible historical and cultural gender stereotypes, it is most likely to
finish with nurse.

Lastly, the usage of proxies introduces the risk of skewing the data. The
notable example of hospitals that chose healthcare spending as a proxy for
health (in lieu of difficult-to-obtain medical records) without accounting for
different healthcare spending patterns and unequal access to care means for
Black and White patients affected millions of Americans (Obermeyer et al.,,
2019). The proper input data increased the percentage of Black patients
receiving additional help from 17.7% to 46.5%.

####  2.2.2 Algorithmic bias

A second source of bias arises due to the fact that machine learning
algorithms generally reckon on correlation, rather than causal relationships.
For example, it is relatively straightforward to design a model to confirm the
positive relationship between the temperature levels and the proportion of
female at managerial positions.

Furthermore, the model architecture may be more suitable for some classes than
others. For instance, a linear model may be appropriate for one cohort but not
for another.

What is more, the bias could be embedded in the choice of cost function, for
example by restricting the set of possible decisions that the machine may
make. Instead of a yes or no resolution, it might be a one-out-of-many
verdict, introducing bias if a reasonable option to consider is eliminated.

####  2.2.3 Detecting bias

Understanding bias sources helps in their actual identification. Consequently,
the following four steps are useful in bias detection:

  1. 1.

Verify whether all data groups have an identical probability of being assigned
to the favourable outcome for protected attribute.

  2. 2.

Validate whether all factions of a protected attribute have corresponding
positive predictive value.

  3. 3.

Authenticate that all classes of a protected attribute have predictive
equality for false positive and false negative rates.

  4. 4.

Certify that statistical parity and equal opportunity ratios/differences are
below certain level.

###  2.3 Mitigating bias

Once the bias is detected, its eradication should be performed gradually at
each stage of data science life cycle.

Thus, before modelling, one can boost the fairness of the delivered solution
by ensuring equal proportional representation across all groups. During
modelling, one should focus on guaranteeing that model performance is fair for
all groups for one or more sensitive features, for example by ensuring
equivalent AUC metric for male and female instances. Finally, after modelling,
one should assure that predictions and model output sustain an equal
probability of false positive rate (or any other model performance metrics)
for all classes with all protected features.

######  Remark.

The choice of an appropriate fairness-boosting method should be grounded in an
understanding of the source of bias and which fairness definition we are
aiming for. For example, some of the more rigid methods described below aim at
achieving demographic parity, which should be used only together with a strong
conviction and evidence that indeed the protected attribute does not influence
the target variable.

####  2.3.1 Ante-hoc fairness

The ante-hoc methods for ensuring fairness focus on modifying the data used
for modelling in a way that makes the resulting model fair.

One of the simplest ways would be deleting the protected attributes. However,
it has to be emphasised that ostracising an individual’s protected attributes
is not sufficient to assure the equity of the solution, because it may be
reconstructed through subtle correlations in the input data.

######  Remark.

The degree to which there are dependencies between the data $\bm{x}$ and the
protected attribute $\bm{p}$ can be quantified using the latent prejudice
measure (Kamishima et al.,, 2011), defined as:

$\text{LP}=\sum\limits_{\bm{x},\bm{p}}\mathbf{P}(\bm{x}|\bm{p})\log\left[\frac{\mathbf{P}(\bm{x}|\bm{p})}{\mathbf{P}(\bm{x})\mathbf{P}(\bm{p})}\right]$

As the aforementioned measure rises, the protected attribute becomes more
predictable from the data. Similarly, one can introduce the indirect prejudice
measure (Kamishima et al.,, 2011) to account for the dependencies between the
labels $\bm{y}$ and the protected attribute $\bm{p}$:

$\text{IP}=\sum\limits_{\bm{y},\bm{p}}\mathbf{P}(\bm{y}|\bm{p})\log\left[\frac{\mathbf{P}(\bm{y}|\bm{p})}{\mathbf{P}(\bm{y})\mathbf{P}(\bm{p})}\right]$

Theoretically, if one cannot predict the labels from the protected attribute
and vice-versa, then there is no possibility for bias.

Ensuring fairness of to-be-delivered solution in a systematic way initiates at
the data collection step, through the identification of lack of examples or
groups and further data enhancement (Chen et al.,, 2018). It can be then
closely followed by the modification of labels and/or input data, as well as
the corresponding re-sampling and weighting through statistical calibration.

Manipulating labels: As a potential solution of fairness ensuring before
modeling phase, Kamiran and Calders, (2012), as an extension of their prior
work Kamiran and Calders, (2009), urged to modify some of the training labels.
Their so-called massaging approach initiates with the computation of the
classifier on the original dataset in order to identify observations with
predictions closest to the decision surface (in order to minimize the accuracy
loss). Then, their labels are swapped (maintaining the overall class
distribution) and the model is re-trained, ensuring that the disadvantaged
group is more likely to occur.

Manipulating observed data: Alternatively, Feldman et al., (2015) introduced
disparate impact removal technique to de-bias the dataset by mapping each of
the the cumulative distributions $F_{0}(\bm{x})=F(\bm{x}|\bm{p}=0)$ and
$F_{1}(\bm{x})=F(\bm{x}|\bm{p}=1)$ to a median distribution. In other words,
it adapts each attribute to ensure the equality of the conditional marginal
distributions based on the subsets of protected attribute with a given
sensitive value. The main flaw of the proposed approach is that it treats each
feature separately, thus ignoring the potential interactions. In addition,
since the repair process outlined above is likely to degrade model’s ability
to classify accurately, one might consider a partially repaired data set
instead, as a trade-off between the predictive power and the fairness. It can
be achieved by simply moving each distribution only part way (for instance a
quartile) towards the median distribution.

Manipulating labels and observed data: Calmon et al., (2018) recommended to
manipulate both the labels and the input data in order to probabilistically
transform data pairs ${\bm{x},\bm{y}}$ to new data values
${\bm{x^{\prime}},\bm{y^{\prime}}}$ in a way that depends explicitly on the
protected attribute $\bm{p}$. The aforementioned transformation is formulated
as a utility change minimization problem with group fairness, individual
distortion, and data fidelity constraints. Unlike previous techniques, it
takes into consideration interactions between all of the data dimensions.
Furthermore, it also enables an explicit control of individual fairness and
the feasibility of multivariate, non-binary protected features. Nevertheless,
since it is formulated as a probability table, it is mainly applicable for
datasets with limited number of discrete input and output variables.

Re-weighting data pairs: If the input dataset contains unjustified
dependencies between some data attributes and the class labels, Kamiran and
Calders, (2009) suggested to balance the dataset such that all the cases where
the protected attribute $\bm{p}$ predicts that the disadvantaged group will
get a positive outcome obtain higher weights.

Learning fair representations: Zemel et al., (2013) proposed a learning
algorithm for fair classification that aimed to achieve both group and
individual fairness. They formulated it as an optimization problem of finding
a proper representation of the data with two clashing objectives: to encode
the data as accurately as possible, while simultaneously befuddling any
information about protected attributes. Since the above-mentioned mapping is
determined during training, their approach could be classified either as a
pre-processing technique or an in-processing algorithm. Moreover, modern
research focuses on employing a variational fair autoencoder, based on deep
variational autoencoders, as in Louizos et al., (2017) or Li et al., (2014).

####  2.3.2 In-processing fairness

Regularize for fairness: Adding a discrimination-aware regularizer (a
mathematical constraint to ensure fairness in the model) to existing ML
algorithms as a part of the learning objective, for example in the form of
mutual information between the model’s prediction and the protected attribute
(Kamishima et al.,, 2011).

Constrain to be meta-fair: Regularization with standard fairness constraints
is often non-convex and hard to be satisfied directly. Consequently, one can
re-express fairness constraints and introduce some convex relaxation for the
purpose of optimization. One possibility is to define unfairness as the
covariance between the protected attribute and distances of the related
feature vectors from the decision boundary and consequently minimize loss the
function under fairness constraints or, interchangeably, to maximize fairness
under accuracy constraints (Zafar et al.,, 2017).

Adversarial de-biasing: Drawing from research in reinforcement learning,
another approach is to construct two models: one for classification task and
the other for fairness assurance. To be more specific, one needs to maximize
the classifier’s prediction accuracy, while simultaneously minimize its
adversary’s competence to ascertain the protected attribute directly from the
predictions. Beutel et al., (2017) proposed a typical adversarial
infrastructure, obliging both classifiers to use a shared representation,
while Zhang et al., (2018) proffered to urge the adversary model to predict
the protected attribute $\bm{p}$ from:

  1. 1.

the final classification to protect the demographic parity

  2. 2.

the final classification and the true class to guarantee the equality of odds

  3. 3.

the final classification and the true value for just one class to provide the
equality of opportunity

Use surrogate models: Alternatively, one can wrap a fair algorithm around
baseline ML algorithms already in use. Calders and Verwer, (2010) proposed a
Two Naive Bayes technique, which trains distinct models for each protected
attribute and iteratively modifies the observed probabilities towards the
selected fairness metrics, based on the fairness of the combined model. A more
sophisticated approach models the actual class labels by treating them as a
latent variable, trying to predict the labels that the dataset would have if
it had been free of discrimination from the very beginning (Calders and
Verwer,, 2010).

####  2.3.3 Post-hoc fairness

It is difficult to remove bias once the classifier has already been trained,
even for very simple cases (Pleiss et al.,, 2017). As a potential remedy, one
can attempt to calibrate the prediction probability threshold to maintain fair
outcomes for all groups with protected and sensitive features, yet at the cost
of decreased accuracy.

Discrimination-Aware Decision Tree Relabelling: Kamiran and Calders, (2009)
designed a strategy to modify the labels of leaves in a decision tree after
training in order to satisfy fairness constraints. In other words, their
approach aims to find the minimal subset of leaves which labels need to be
flipped in order to obtain a less discriminative prediction.

Reject-Option-Based Classification: This approach modifies the predictions of
samples close to the decision boundary, usually near 0.5 in binary
classification for balanced target variables, by incorporating a rejection
option (Kamiran et al.,, 2012). To be more specific, if the prediction is
contained in the critical region, all unprivileged individuals receive the
favorable outcome, while for the predictions outside the critical region, they
remain unchanged.

###  2.4 Tools for fairness

Since fairness in AI is incessantly gaining momentum, technology leaders
propose a variety of tools to automate the fairness-establishment process. The
list below is a compilation of most commonly used tools:

  1. 1.

What-If Tool by Google555Available at: www.pair-code.github.io/what-if-tool/

The What-if Tool is an open-source visualization tool that allows users to
analyse an ML model. Currently, it supports both binary and multi-class
classification, as well as the regression problems, permitting users to
explore the effects of alternative thresholds and fairness criteria. Moreover,
it envisions prediction counterfactual, by contrasting an observation to the
most similar point where the model predicts a different outcome.

  2. 2.

Fairlearn by Microsoft666Available at: https://fairlearn.org

Fairlearn is an open-source library to assess and ameliorate the fairness of
AI systems. It is composed of two primary components: an interactive
visualisation dashboard with a set of metric to help quantify the fairness of
a model, as well as several algorithms that can be used to mitigate models
discrimination (training or post-processing), for instance a method that
applies relevant constraints in order to reduce a fair classification problem
to a sequence of cost-sensitive classification ones.

  3. 3.

AI Fairness 360 by IBM777Available at: www.aif360.mybluemix.net

The AI Fairness 360 toolkit is an extensible open-source library, available
for both R and Python, containing a comprehensive set of metrics for datasets
and models to be validated for biases and algorithms to mitigate bias in
machine learning models throughout the AI application life-cycle. It is
tailored to translate academia research into the actual practice of domains as
wide-ranging as finance, human capital management, healthcare, and education.

  4. 4.

Responsible AI Toolkit by PwC888Available at: www.pwc.com/gx/en/issues/data-
and-analytics/artificial-intelligence/what-is-responsible-ai.html

Responsible AI Toolkit is a suite of tools designed to help to tame the power
of AI ethically and responsibly. Due to being developed with various
stakeholders such as regulators or board members, it aids to address the
regulatory and compliance facets of AI-oriented businesses.

  5. 5.

LinkedIn Fairness Toolkit (LiFT) by LinkedIn999Available at:
www.github.com/linkedin/LiFT

The LinkedIn Fairness Toolkit (LiFT) is a Scala/Spark library that empowers
the quantification of fairness in big data machine learning workflows, being
feasible to be deployed in both training and scoring phases.

###  2.5 Putting fairness to work

Fairness in AI is a rapidly evolving field and therefore it has not
established yet the standard best practices for implementation. However, it is
already possible for an organization to benefit from thinking about fairness
in AI applications. To summarize the chapter on fairness, an overview of
specific points to consider in practice is provided:

  * •

Analyse whether the context in which one aims to utilize AI involves the
potential for bias. If it does, then AI can have one of two potential
consequences: mitigating or reinforcing bias. Unfortunately, without proper
introduction of fairness methods described in the previous sections, the most
likely outcome is bias reinforcement.

  * •

Implement new ways of working, designed specifically to utilize the fairness
tools and techniques. To ensure that an organization systematically considers
issues of fairness, one needs to complete three to four initiatives:

    1. 1.

Make fairness in AI an important topic on agenda, so that people across the
the organization understand that they need to start treating it as an
important business objective. This is in contrast to simply forcing a
procedure or policy, which would likely have very limited real impact.

    2. 2.

Provide analytical teams with the tools needed to tackle fairness issues: the
required knowledge (e.g., via specific training sessions) and literally tools,
which in some cases might mean using open-source libraries, but in some more
restrictive environments it might require an investment to either buy or
develop some tooling in-house.

    3. 3.

Find opportunities for operational changes that will aid the technical work on
de-biasing. For example, it might be possible to change data collection
processes, so that future model training will require less algorithmic
tweaking for fairness.

    4. 4.

For safety-critical areas101010Areas in which utilizing biased AI could have
serious negative consequences for the organization, most commonly from a
regulatory point of view, but in some cases also from limiting the addressable
market or causing serious harm to the company’s image., create an additional
audit function, which would independently validate if the fairness objectives
have been achieved.

  * •

Extend the scope of work on fairness from AI to other business areas. For
example, one might uncover some unfair processes when models trained on
decision made in the past by humans show signs of bias.

  * •

Identify business areas which can be handled directly by AI models and ones
which require a human-in-the-loop. In some cases (e.g., previously mentioned
safety or low model quality) it makes sense to add a human as the final
decision maker, to double check for fairness. It is tightly connected with
severe negative cost implications, so should not be treated as a default
method of de-biasing,

  * •

Engage in a broader discussion, outside an organization. As fairness in AI is
the current cutting-edge research, there are a lot of opportunities to shape
the future standards of it. Undoubtedly, it will be achieved significantly
faster with support from practitioners. Consequently, some of the ways of
contributing might be providing researchers with access to datasets or sending
teams to present their work at conferences.

##  3 eXplainable Artificial Intelligence

The second chapter of this article treats the topic of explainability in AI.
XAI is presented after fairness for two reasons:

  1. 1.

the future of XAI expands into a notion of Responsible AI, which provides not
only an explainability with a satisfactory level of performance, but excavate
the concept with the apprehension of fairness, accountability, privacy and
data fusion (Arrieta et al.,, 2019),

  2. 2.

research in XAI is more advanced than in AI fairness, which translates to more
methods, which the reader might find applicable for their fairness use-case.

As this chapter puts a great weight on specific XAI techniques, we provide
business use-cases for the most relevant ones.

###  3.1 Rationalization of XAI

To start with, cooperation between agents, such as algorithms and humans, is
based intensely on trust. Improving transparency can diminish the inherent
reluctance towards the algorithmic prescriptions, since stakeholders can
comprehend all the reasoning of each decision taken. XAI provides a sense of
control and safety, as the infrastructure owner clearly understands the levers
of its AI system’s behaviour, which in turn can be subjected to a safety
guidelines and alert on their violation.

Moreover, the ability to explain a data-driven decision based on machine
learning is particularly important in banks, insurance companies, healthcare
providers and other highly regulated industries. They are prone to legal
requirements, which oblige them to explain the rationale under every single
decision. Additionally, they may be constrained by the laws against
discrimination, demanding the ethics-related justifications (so called AI
fairness), replacing black box by glass box111111A system that audits both the
inputs and the outputs of a model, with the objective of verifying its
adherence to ethical and socio-legal values, thus developing value-based
explanations. For the more detailed distinction between black box, white box
and glass box please refer to Kojarski and Lorenz, (2006). In other words,
interpretability helps ensure impartiality in decision-making, allowing to
detect, and consequently, correct from bias in the training dataset.

Other applications of XAI are knowledge extraction, permitting to learn new,
robust insights from the model and helping to pinpoint deficiencies both in
data and feature behaviours. Moreover, the interpretation does not have to be
a good explanation per-se, but only guide the user to potential flaws.
Explainable artificial intelligence can help in debugging the mis-predictions
by virtues of expert knowledge and experience, ergo boosting the model’s
performance. Furthermore, it can lead to use of a model as a starting point
for causal research, resulting in transferability of learning. The mere
understanding of the inner relations taking place within a model facilitates
the ability of a user to reuse this knowledge in another problem.

######  Remark.

There are noteworthy differences among the concepts of explainability and
interpretability. Namely, interpretability invokes a passive observation of
the decision and judging its logic with human common sense. Reciprocally,
explainability can be regarded as an active characteristic of a model,
requiring interaction in order to clarify or detail its internal functions.
For the sake of transparency, the notions of explainability, interpretability,
comprehensibility and transparency will be henceforth used interchangeably.

###  3.2 Desired characteristics of XAI

The field of data science is inherently linked to the performance versus
explainability conundrum. One approach to achieve explainability in AI systems
is to implement machine learning algorithms that are fundamentally
explainable. For instance, classical/traditional methods of machine learning,
such as decision trees or Bayesian classifiers, sometimes can relatively
cheaply provide the traceability required for decisive AI systems without
sacrificing too much performance. Conversely, more sophisticated and thus
potentially more powerful algorithms such as neural networks or ensemble
methods, like random forests, cede transparency and explainability for
accuracy. Since XAI is designed to reconcile the feuding trade-offs, it has to
possess five main characteristics:

  1. 1.

Global perspective: XAI is required to present a coherent picture,
generalizing local to global understanding and supporting a well-defined
decision-making task;

  2. 2.

Interpretability: XAI needs to provide the easily understandable results,
highlighting only a few, influential features;

  3. 3.

Local fidelity: XAI should be locally faithful, meaning reflecting the overall
model performance (also referred to as accuracy);

  4. 4.

Model-agnostic property: XAI (particularly post-hoc) should not depend on the
underlying technique used;

  5. 5.

Efficiency: XAI should achieve understandability in a reasonable amount of
time;

On the other hand, XAI techniques differ significantly among themselves as far
as the focus of the explanation is concerned. Before selecting the proper
algorithm, one needs to answer the following questions:

##### At what stage of the ecosystem is explainability needed?

There are three principal set of routines used to develop explainable systems:
ante-hoc, contemporaneous and post-hoc. Ante-hoc techniques (often referred to
as pre-modelling explainability) entail incorporating explainability into a
model from the beginning. Contemporaneous method (explainable modelling)
avoids the black-box problem from the outset by developing a model that is
explainable by design. Finally, post-hoc procedures (post-modelling
explainability) permit models to be trained unintermittedly, with
explainability being incorporated at the testing time.

##### Who is the designator of explainability?

The selected algorithm of XAI should reflect the role-specific goal of end
user. The artificial intelligence experts usually require a mechanic
explanation, for instance a sensitivity of a given layer of neural network to
input data, extremely useful in debugging and validating a model. On the other
hand, a business analyst tends to necessitate a functional explanation, for
example a main driver behind the machine prediction, advantageous in checking
the legal compliance of a model.

##### What is the scope of explainability to be achieved?

The microeconomic studies are apt to favour an instance prediction, often
referred to as local explanations, which provide a reasoning for a specific
instance of a class, such as the arguments for the unfavourable credit
decision for an agent. On the other hand, the macroeconomic research is likely
to focus on all model prediction, also named as global explanations, due to
their relative simplicity to be translated into a policy advice.

##### What level of complexity should be realized?

The choice of the explanation family should mirror the desired complexity of
an explanation. Despite the fact that the XAI algorithms should be model-
agnostic, mainly due to the computational complexity some of them perform
better in the case of a single output, like a prediction of a classifier,
while some are more suited for the sequence explanation, such as Long Short-
Term Memory (LSTM) neural networks (Sak et al.,, 2014).

###  3.3 XAI techniques

The following section depicts a non-exhaustive overview of some of the most
universal XAI methodologies and approaches, split into three main groups:
ante-hoc, contemporaneous and post-hoc techniques.

####  3.3.1 Ante-hoc explainability

The behaviour of a plethora of AI models is, to a large extent, driven by
datasets employed in their training stage. Hence, the main objective of pre-
modelling explainability is to fully understand and construe the data. Ante-
hoc methods can be partitioned into four main categories: dataset
standardization (including its description regularity), exploratory data
analysis, dataset squashing and explainable feature engineering.

##### Dataset standardization

Dataset standardization is a data processing workflow that aims to reorganize
the structure of discordant datasets into one common format. It can be
considered as a part of data preparation process that enables the final data
consumers to analyse information in a consistent manner. There are three
fundamental use case categories in dataset standardization:

  1. 1.

Elementary mapping from internal sources: manipulating internal datasets that
originate from inconsistent definitions and reorganizing them into a single
principled dataset for the entire organization;

  2. 2.

Elementary mapping from external sources: transforming data from systems
external to the organization through mapping them to keys and values
compatible with the organization schemas;

  3. 3.

Advanced reconciliation: creating sophisticated metrics that assure their own
semantics grounded in defined business logic;

In order to understand the essence of dataset standardization, one may imagine
holding companies with a plethora of independent subsidiaries, franchisees or
globally spread business units that provide inconsistent financial data, for
instance through different formats of balance sheets.

Furthermore, there is no uniformly agreed practice to present a dataset
description, often leading to the miscommunication between the data engineers
and the final data users. As a solution, several dataset description
standardizations have been proposed in order to mitigate the negative
consequences stemming from the misuse of data. Essentially, they define a
specific schema to document the data collection process and final data
composition. A non-exhaustive list includes:

  1. 1.

Datasheets for datasets (Bender and Friedman,, 2018): a framework designed
primarily for the Natural Language Processing research;

  2. 2.

Data statements (Gebru et al.,, 2020): a list of questions that each
successful data documentation should address;

  3. 3.

Dataset nutrition labels (Holland et al.,, 2018): a module resembling the
nutrition facts labels for packaged food;

  4. 4.

Global Standardization of Clinical Research Data (Pétavy et al.,, 2019): a
project to standardize data collection and sharing in clinical research;

  5. 5.

Codebook (Arslan,, 2019): package to automate the data documentation,
principally for the psychological datasets;

Consequently, dataset standardization along with its description regularities
contribute to the cleanliness of data landscape, hence improving team
efficiency and reducing long-term maintenance costs.

##### Exploratory data analysis (EDA)

Assuming a consistent dataset structure, exploratory data analysis provides a
summary of its primary characteristics, often with visual techniques. It
includes both the extraction of statistical properties, such as
dimensionality, missing observations or range, mean and variance, as well as a
graphical visualization121212The most often employed chart types can be easily
accessed on https://datavizcatalogue.com. Moreover, several operations on the
data can be performed in order to prepare it for the machine learning
algorithms, e.g.: median polishing, trimeaning or ordination.

Additionally, since real-world datasets are generally high-dimensional, one
can attempt to project them into a lower-dimensional representation while
conserving its underlying characteristics to the largest extent possible.
Widely used dimensionality reduction techniques include Principal Component
Analysis (PCA, recommended for a linear data structure), t-distributed
Stochastic Neighbour Embedding (t-SNE) 131313Implemented in Python:
https://github.com/danaugrs/go-tsne or Latent Dirichlet allocation (LDA)
141414Implemented in Python: https://github.com/RaRe-Technologies/gensim.
Moreover, it is important not to overlook Local Linear Embedding (Roweis,,
2000))151515Implemented in Python as part of scikit-learn library:
https://scikit-learn.org/stable/index.html, that consists of a nonlinear
learning approach for generating low-dimensional neighbour-preserving
representations from (unlabelled) high-dimension input and tends to be more
precise in exploiting the underlying data structure than PCA. Nevertheless,
modern methods, such as Uniform Manifold Approximation and Projection (McInnes
et al.,, 2020) 161616Implemented in Python: https://github.com/lmcinnes/umap,
are constantly gaining momentum, mainly due to their scalability and thus
computational speed.

Nonetheless, after the application of dimensionality reduction techniques, the
ability to interpret the influence of individual features usually drops
significantly. Therefore, it is essential not to overlook the rotations
methods. They be decomposed into two main families:

  1. 1.

1\. Orthogonal, in which the factor variances get changed, but factors
themselves remain uncorrelated and variable communalities are preserved:

     * •

varimax: it aims to maximize variance among the squared values of loadings of
each factor;

     * •

quartimax: its objective is to minimize the number of factors needed to
explain a variable and often produces the so-called general factor;

  2. 2.

Oblique, in which factors are allowed to lose their uncorrelatedness if that
will produce a clearer simple structure;

EDA is primarily used for two reasons – to check the validity of the extracted
data and to generate hypotheses and understanding on how to model the data at
a later stage. To provide a concrete example, suppose the task is to model
price elasticity of a selection of canned beverages. One of the ways EDA can
bring value is through revealing if the analysed beverages have different
volumes and as a consequence if one needs to control for the volume of liquid
in the modelling, as depicted in the following chart.

Figure 1: Distribution of volume of analyzed Stock Keeping Units (SKUs)
Source: Own elaboration

Furthermore, one might observe products with exceptionally low numbers, thus
indicating that volume information in a single column is stored either in
litres or millilitres and requires further investigation. In the outlined
example, correcting units is relatively straightforward, since beverage cans
have several well-known standard volumes, but typically expert domain
knowledge is substantial.

Fortunately, manually performing EDA is becoming increasingly obsolete, as
there is a plethora of available packages and toolkits for rapid and reliable
extraction of the main properties from a dataset.171717For a profound
catalogue please feel free to visit https://github.com/mstaniak/autoEDA-
resources

##### Dataset squashing

Data squashing methods attempt to provide a subset of the training set which
symbolizes the main essence of the dataset, that in turn can be granted along
with the model prediction to the final user as the explanation, disentangling
the need of storing the entire dataset. Usually, data summarization consists
of selecting a minimal subset of representative samples (so called
prototypes), for instance through the K-medoid clustering algorithm (Park and
Jun,, 2009)181818Part of PyClustering library in Python:
https://github.com/annoviko/pyclustering.

Nevertheless, some argue that in order to completely understand complex data
distributions, one need as well the parts of the input space that are not
accurately described by the chosen set of prototypes (so called criticisms).
Motivated by the Bayesian model criticism framework, Kim et al., (2016)
developed MMD-critic technique (Maximum Mean Discrepancy)191919Implemented in
Python: https://github.com/BeenKim/MMD-critic to efficiently learn both
prototypes and criticisms.

The MMD-critic algorithm should be executed in two stages. Firstly, prototypes
are elected to mimic the full dataset. Secondly, criticisms are selected from
parts of the dataset that are under-represented by the previously chosen
prototypes, with a supplementary constraint to assure the diversity of
criticisms. As an output of MMD-critic method, a set of prototypes that are
exemplary of the dataset as a whole, and a set of criticisms that diagnose
spacious parts of the dataset that alter most from the prototypes are
obtained. Titular Maximum Mean Discrepancy (MMD) refers to the specific scheme
of quantifying of the difference between the prototype distribution and the
full data distribution.

Additionally, Bayesian Case Model (BCM) proposed by Kim et al., (2015),
extends the prototype-based reasoning to the fully-fledged and operational
machine learning model. BCM is an admixture model, thus it belongs to the
family of generative models for unsupervised learning), which leverages both
prototypes and sparse features to be more interpretable (simply using examples
to intuitively explain machine learning results to humans) without any loss of
power compared to standard methods. It was further developed into an end-to-
end interactive BCM (iBCM) system for the computer science education domain
that communicates with humans through examples and accepts the feedback
through identical intuitive medium.

Finally, it is valuable to outline the prominent and relatively mature data
squashing techniques: moment-matching (DuMouchel et al.,, 1999), in which the
squashed dataset consists of a set of pseudo data points selected to replicate
the moments of the original dataset within subsets of a partition of the
mother-data and likelihood-based methods (Madigan et al.,, 2002), in which for
every mother-data point the algorithm firstly estimates the probability of
occurrence in order to generate a likelihood and then constructs one or more
pseudo data points from each likelihood profiles clusters.202020A helpful
library to build a coreset (meaning the core of the dataset), for the purposes
of Bayesian inference can be found here:
https://github.com/trevorcampbell/bayesian-coresets

Dataset summarization techniques can be employed in marketing campaign
optimization, since it involves merging analytics with a business
understanding to design the most effective communication. One of methods used
by marketing specialists is to create personas, defined as a composite sketch
of a key segment of targeted audience. Dataset summarization techniques can be
therefore incorporated to extract the mixture of any of the attributes or
activities that allow to define an accurate subset of representative personas
instances. The chart below depicts selected characteristics of personas
established in McKinsey “Digital Poles” survey (Bogdan et al.,, 2016).

Figure 2: Why do digital Poles still shop in traditional stores?
Source: Adapted from Bogdan et al., (2016)

It presents different causes for shopping in traditional stores versus their
e-commerce alternatives, constituting the basis for the growth potential of
e-commerce market in Poland discussion. Product manager could consequently use
them to tailor communication to different groups, for example: an e-commerce
player emphasising low delivery costs will be more successful when aiming such
marketing efforts at “Aspiring Enthusiasts” rather than “Wealthy Convenients”.

##### Explainable feature engineering

An explainable model should be based on the features that are easily
interpretable. Thus, explainable feature engineering assures that the
underlying bricks have been assigned an intuitive meaning. It can be tackled
in two ways: domain-specific and model-based.

Domain specific approach reckons on the expert knowledge, augmented with the
insights derived from the exploratory data analysis, to identify valuable
features. Circling back to the previous example of price elasticity modelling,
a domain specific feature defined by data scientists could be the ratio of
average prices of competing brands, which are known to be treated as
substitutes by customers.

Model-based approach, on the contrary, uncovers the underlying structure of a
given dataset through mathematical models and transformations. The incessantly
developing techniques include:

  1. 1.

Dictionary learning: it is designed to explore the input data in order to
locate a sparse representation in the form of a linear combination of basic
elements (so called dictionary). The procedure of dictionary learning can be
therefore considered as a valuable alternative to Principal Component Analysis
(PCA). Nevertheless, the atoms in the dictionary are not required to be
orthogonal and additionally they may produce an over-complete spanning set,
which allows for more flexible dictionaries and richer data representations.
Consequently, the sparse dictionary learning has been successfully applied to
various image, video and audio processing, as well as the unsupervised
clustering tasks (Sprechmann and Sapiro,, 2010)212121Part of scikit-learn
library in Python: https://github.com/scikit-learn/scikit-learn.

A generalization of k-means clustering method, K-SVD algorithm (Aharon et
al.,, 2006), focuses on creating a dictionary for sparse representations via a
singular value decomposition approach (SVD). More precisely, it iteratively
modifies the sparse coding of the input data on the basis of current
dictionary and consequently updates the atoms in the aforementioned dictionary
in order to more precisely fit the data. Currently, K-SVD is widely applicable
in biology or document analysis.

  2. 2.

Disentangled representation learning: a disentangled representation requires
single latent units to be sensitive to modifications in single generative
factors, while being invariant to changes in other. It allows the knowledge
about one factor to relatively costlessly generalize to alternative
configurations of other ones. For instance, a model trained on a dataset of 3D
objects might learn independent latent units receptive to individual data
generative factors, such as object position, scale or colour. In terms of
specific algorithms, the extension of well-known variational autoencoder (VAE)
framework (Kingma and Welling,, 2014)222222Implemented with Keras in Python:
https://keras.io/examples/variational_autoencoder, $\mathbf{\beta}$ – VAE
(Higgins et al., 2017)232323Implemented with PyTorch in Python:
https://github.com/1Konny/Beta-VAE, tends to outperform the state of the art
unsupervised (Chen et al.,, 2016)242424Implemented with Keras in Python:
https://github.com/tdeboissiere/DeepLearningImplementations and semi-
supervised (Kulkarni et al.,, 2015)252525Implemented with Tensorflow in
Python: https://github.com/yselivonchyk/TensorFlow_DCIGN approaches to
disentangled representative learning. In contrast to InfoGAN, $\mathbf{\beta}$
– VAE is stable to train, requires only few assumptions about the data and
depends on tuning of a single hyperparameter, which in turn can be optimized
through a hyperparameter search on the weakly labelled dataset or through
heuristic visual inspection for purely unsupervised data.

####  3.3.2 Explainable modelling

The following section is focused on the modelling stage of AI development,
designed to avoid the black-box problem from the very beginning. In
particular, it scrutinizes a handful of methodologies for training models that
are both explainable by design and performant. Explainable modelling can be
partitioned into two fundamental categories: implementation of inherently
explainable model, augmented with the regularization, and the family of hybrid
models, intending to merge white- and black-box approaches into one. The
section is concluded with some additional Deep Learning specific solutions for
explainable modelling.

##### Implement inherently explainable model

The classical approach to reach explainability is to implement a specific
family of models that are regarded inherently explainable, such as the linear
models, generalized additive models, Bayesian classifiers or decision trees.
In other words, it consists of building models based on features that those
who need to receive explanations inherently understand and maintain
traceability of those features through the model.

It has to be underlined that even in the family of inherently explainable
models there is a spectrum between entirely transparent models, where one
fully recognize how all the variables are jointly linked to each other, and
models that are slightly constrained in their form, such as models that are
imposed to inflate as one of the variables increases, or models that, ceteris
paribus, favour features that domain experts have classified as significant
(Rudin,, 2019). As an example of the latter, it is invaluable to mention Self-
Explaining Neural Networks (Alvarez-Melis and Jaakkola, 2018b,
)262626Implemented in Python: https://github.com/dmelis/SENN. It is a by-
design approach that intends to produce stable explanations by assuming a
specific structure for both model itself and its explainer. SENN designs self-
explaining models in stages, progressively generalizing linear classifiers to
complex yet architecturally explicit models.

One of the main use-cases for adopting explainable model families is to
satisfy regulatory requirements of model transparency in industries such as
banking or insurance. A common practice in insurance technical
pricing272727Technical pricing is modelling aimed at determining the expected
value of claims that will be incurred by the insurer if they offer protection
to a customer. is constructing a generalized linear model with categorical
variables only (continuous variables have to be binned beforehand). It can
then be used as a trivial scoring card, allowing for direct tracing of every
possible model prediction.

Figure 3: Life insurance pricing model following the tariff cell archetype by
fitting a generalized linear model on binned variables. Light blue shading
points to tariff cells of an example customer.
Source: Own elaboration

In the preceding example, one can easily trace each of the 100 possible tariff
outcomes (so called tariff cells). The highlighted bars reflect a prospective
customer who is between 45 and 65 years old, has visited 1 country in the last
10 years and has incurred a total of between 5,000 and 15,000 USD of medical
expenses in the last 5 years. Technical price prediction can be then inferred
by multiplying the relevant factors, i.e., $1.43\cdot 1.00\cdot 1.20=1.716$,
meaning that the estimated technical price is 1.716 times higher than for a
base case.

Unfortunately, explainable families generally result in relatively simple and
poorly performing models, deficient in capturing the real-world complexity.
Moreover, one needs to remember that simply using an explainable model does
not mean the results of its processing will be fully explainable. For example,
linear regression in a high-dimensional space is not transparent, meaning that
a person cannot contemplate the entire model at once, while even in the low-
dimensional, yet multicollinear space, the estimated coefficients could be
unstable. Fortunately, regularization can partially alleviate the
aforementioned concerns.

##### Regularization

Generally speaking, regularization discourages learning a more complex or
flexible model, in order to avoid the risk of overfitting. Consequently, it is
heavily employed in order to improve the predictive performance of machine
learning algorithms. However, its explainability boost characteristics are not
to be underestimated. The basic implementations consist of explicitly
restraining the model predictions to a specific value. More sophisticated ones
attempt to incorporate regularization based on the expert knowledge, while the
most advanced aim to deliver model agnostic regularizers not demanding any
expert knowledge.

For instance, Erion et al., (2020)282828Implemented in Python:
https://github.com/suinleelab/attributionpriors developed a differentiable
axiomatic feature attribution method called expected gradients that allow
humans to use the common language of attributions to sanction prior
expectations about a model’s behaviour during training. For example, expected
gradients can enforce models to treat functionally related genes similarly or
to rely on fewer features for health care datasets. By straightforward
encoding of constraints, the attribution priors tend to produce models with
more intuitive behaviour and better generalization performance.

More advanced Contextual Decomposition Explanation Penalization (Rieger et
al.,, 2020)292929Implemented in Python: https://github.com/laura-rieger/deep-
explanation-penalization technique allows a practitioner to penalize both a
model’s prediction and the corresponding explanation, thus preventing a model
from learning unwanted relationships and ultimately improving its predictive
accuracy. To put differently, CDEP permits the user to directly penalize
importance of certain features and their interactions, urging the neural
network to not only deliver the proper predictions, but also the correct
explanations for that predictions. Moreover, CDEP is a relatively general
approach, which can be applied to arbitrary neural network architectures, even
on the sizeable datasets.

Additionally, Plumb et al., (2019) proposed a strategy called Explanation-
based Optimization (EXPO)303030Implemented in Python:
https://github.com/GDPlumb/ExpO that permits to interpolate between inherently
explainable models and post-hoc approaches by adding an interpretability
regularizer to the loss function used to train the model. To be more specific,
EXPO is a differentiable and model agnostic regularizer not requiring any
domain knowledge, that approximates the fidelity metric 313131This is a
measure of explainer performance. For specific definition please visit
evaluating XAI performance section. on the training dataset in order to boost
the excellence of post-hoc explanations of the model. It was shown empirically
that EXPO can significantly improve explanation quality in terms of
explanation fidelity, without sacrificing predictive accuracy.

There are three methods conceptually similar to EXPO and worth mentioning:
Functional Transparency for Structured Data (Lee et al.,, 2019), Right for the
Right Reasons (Ross et al.,, 2017) 323232Implemented in Python:
https://github.com/dtak/rrr and Saliency Learning (Ghaeini et al.,, 2019).
Nevertheless, FTDS lacks the model agnostic feature, since it is designed to
graphs and time-series data, while RRR relies on extensive domain knowledge to
specify the set of good and bad features. To be more precise, RRR augments the
loss function with the penalty term for the input gradients, primarily based
on expert annotation, which constrains local explanations of a model to match
the domain knowledge. Expert insights are encoded as a binary annotation
matrix, indicating whether a specific feature should be taken into account
while making a prediction for each input. Furthermore, when the annotations
are not available, the validity of an explanation is assessed by the discovery
of classifiers with similar accuracies, but qualitatively divergent decision
boundaries for domain experts. Interestingly, the resemblance to the state-of-
the-art sample-based explanation as Local Interpretable Model-Agnostic
Explanations (LIME) has been confirmed empirically. Finally, recently emerged
Saliency Learning targets to teach the model to undertake the right prediction
for the relevant reason by providing explanation training and ensuring the
alignment of the model’s explanation with the ground truth explanation.

In addition, some authors aspire to approximate the results of the deep
learning model by a modest decision tree333333This is a post-modelling
technique, but it sets context for the regularization method. For more
examples of knowledge distillation please refer to post-hoc explainability
section., which in turn can be easily tractable by humans (such model property
is called human-simulatability, while the process of transferring the dark
knowledge learned by a teacher model, usually sophisticated and large, to a
student model, typically shallow and small, is named knowledge distillation).
Benefiting from the existence of multiple local optima of a deep learning
model, Wu et al., (2017)343434Implemented in Python:
https://github.com/dtak/tree-regularization-public presented that the
aforementioned exercise can be sometimes achieved by supplementary
regularization terms in the loss function. To be more precise, the novel
regularization function penalizes the complexity of a decision tree (estimated
by a meta-model as the average decision path length) approximating the initial
model.

The last described regularization method has been widely used to create models
for medical applications, namely prediction of mortality and need for
ventilation in sepsis. Tree-regularized deep learning model allowed the
researches to detect unwanted behaviour of the neural network by consulting
with clinical experts – this would not have been possible without the enforced
human-simulatable property.

As a remark, regularization can be employed additionally as a solution to
vulnerability of deep neural networks to the adversarial attacks, meaning
their susceptibility to errors after small modification of inputs. Jakubovitz
and Giryes, (2019) depicted that the regularization using the Frobenius norm
of the Jacobian of the network can lead to enhanced robustness results with a
minimal change in the original network’s accuracy.

##### Hybrid models

The purpose of hybrid approaches is to merge a model from an inherently
explainable family with black-box techniques. The use of background knowledge
in the form of logical statements or constraints in Knowledge Bases (KBs) has
shown to not only improve explainability but also performance with respect to
purely data-driven approaches (Garcez et al.,, 2019). Future data fusion
approaches may thus consider endowing Deep Learning algorithms with
explainability by externalizing other domain information sources, for instance
the interpretability inherent to probabilistic graphical models. Hitherto,
deep formulation of classical machine learning models has been employed for
example in Deep Kalman Filters353535Implemented in Python:
https://github.com/k920049/Deep-Kalman-Filter(Krishnan et al.,, 2015) or
Structural Variational Autoencoders363636Implemented in Python:
https://github.com/mattjj/svae (Johnson et al.,, 2016).

A particular example of classical machine learning model enhanced with its
deep learning counterpart was introduced by Papernot and McDaniel, (2018) as
Deep k-Nearest Neighbours (DkNN)373737Implemented in Python:
https://github.com/Eric-Wallace/deep-knn. Generally speaking, it is designed
to combine the k-nearest neighbour algorithm with the representations of the
data learned by each layer of the Deep Neural Network.

As far as image processing is concerned, Wan et al., (2021) proposed the
Neural-Backed Decision Trees (NBDT)383838Implemented in Python:
https://github.com/alvinwan/neural-backed-decision-trees technique to estimate
a decision tree augmented with the information obtained from a simultaneous
neural network. The process is as follows: firstly generate the induced
hierarchy through hierarchical agglomerative clustering (e.g. classifying a
Cat also as a Mammal and a Living Thing) and train the neural network with a
tree supervision loss (a cross entropy loss designed to separate
representatives for internal nodes). Then, run inference by featuring images
using the network backbone and run embedded decision rules.

Alternatively, one can consider using the BagNet architecture of Brendel and
Bethge, (2019)393939Implemented in Python:
https://github.com/wielandbrendel/bag-of-local-features-models, which can be
viewed as a bag-of-features (BoF) model, where the features are learned
through the deep neural networks. In other words, BoF technique consists of a
set of fundamental visual features and the number of their occurrences on each
image (bag). For the sake of illustration, consider the task of classifying
images into human and bird classes, on the basis of only two visual
characteristics: a human eye and a feather. The simplest bag-of-features model
increases the evidence for human for each eye in the image and similarly raise
the chances for the bird for each feather. The class which accumulates the
most evidence across the picture, will be the predicted one.

The idea behind BagNet architecture is then to merge the interpretability and
transparency of BoF models with the excellent performance of Deep Neural
Networks. The high-level strategy is following:

  1. 1.

Split the image into small image patches.

  2. 2.

Pass the aforementioned patches through a Deep Neural Network to obtain a
class evidences for each patch.

  3. 3.

Sum the evidence over all patches to derive an image-level decision.

Unfortunately, due to the task of image classification, the BagNet
architecture tends to perform poorly when the local image features are not
sufficient to learn the high-level characteristics.

In addition, Wang and Lin, (2019) designed a general framework for hybrid
predictive models. Their hypothesis is that there exists a subspace in the
feature space where an uncomplicated and thus more understandable model can be
almost as accurate as the black-box model. Therefore, as a substitute of
explaining the black-box part, one can just supplant it on a subset of data
where its employment is a pure overkill. Consequently, they designed a hybrid
predictive model where input is employed by the interpretable model in the
first place to verify if a prediction can be directly generated, while if not,
the black-box model gets activated.

Moreover, at least in theory, a machine learning model can be trained to
simultaneously provide a prediction and the corresponding explanation. For
instance, Hind et al., (2019) proposed a Teaching Explanations for Decision
(TED)404040Part of AI Explainability 360 library in Python:
https://github.com/IBM/AIX360 technique that, unlike previously mentioned
methods, does not pursue to probe the reasoning proceeding of a model, but
instead attempts to clone the reasoning mechanism of a human domain user.

Assuming that the training dataset composes of a set of features (connoted by
X), a decision / label / classification for each feature vector (expressed as
Y) and an explanation describing the rationale for that resolution (denoted by
E), which can take any form, not to mention a number, text string or image.
Dissimilar to the traditional approaches, E does not necessarily need to be
expressed in terms of X. On the ground of the above-mentioned augmented
training set, the TED framework encodes Y and E into a new prediction target
(YE) by combining each label from Y with a relevant explanation from E, which
together with X is provided as the final training input to any deep AI
algorithm. The last step decodes the prediction from the form of YE into its
individual components: Y and E.

For example, assume the goal is to predict whether a used car’s price will
increase in 10 years from now. Knowing the reason for a black-box model’s
prediction would certainly be helpful to a salesman trying to convince
potential buyers that a prediction of increased future value is reasonable.

Figure 4: Data used to predict change in used car’s value before and after
supplying explanations for features
Source: Own elaboration; visualization technique inspired by Hind et al.,
(2019)

Some common examples of reasons why a car’s value might fluctuate are shown in
the figure above. Using the TED framework, the model would learn to predict
jointly the direction of value change and the underlying reason. Hence, a
prediction of increase in value for a specific Porsche 911 could point the
salesman that the expected surge in price stems from it being in fact a
limited edition.

In a specific case of healthcare data, one can focus on the Reverse Time
AttentIoN (RETAIN) model414141Implemented in Python:
https://github.com/mp2893/retain, developed by Choi et al., (2017). It is
founded on the two-level neural attention model that mimics the physician
approach by digesting the data in a reverse time order, resulting in a recent
clinical visit likely to receive higher attention. The attention mechanism
helps in identifying the most meaningful visits and thus points out visit
specific features that contribute to the overall prediction. However, it has
to be emphasized that some experiments verified that the standard attention
modules do not yet provide meaningful explanations and should not be treated
as though they do (Jain and Wallace,, 2019).

##### Deep Learning specific solutions

Card et al., (2019) proposed the Deep Weight Averaging Classifiers
(DWAC)424242Implemented in Python: https://github.com/dallascard/DWAC which is
able to transform any deep learning architecture into one that is more
transparent, interpretable, and robust to out-of-domain data. It eliminates
the final SoftMax layer, responsible for the projection of a hidden
representation into a vector of probabilities, replacing it by the probability
of each label as a weighted sum of training instances, where the weights are
based on the distance between instances in the low-dimensional instance-
embedding space learned by the model.

Bayesian Deep Learning (BDL)434343An example in Python:
https://github.com/kyle-dorman/bayesian-neural-network-blogpost gauges the
uncertainty of a neural network about its predictions. It may model perplexing
assignments by leveraging the hierarchical representation capabilities of deep
learning, while additionally being able to infer sophisticated multi-modal
posterior distributions. BDL models usually construct uncertainty estimates by
either installing distributions over model weights, or by learning a direct
mapping to probabilistic outcomes. By being acquainted with the weight
distributions of sundry predictions and classes, one can tell which feature
led to a specific decision and the relative importance of it in the
uncertainty calibration.

Knowing the uncertainty of predictions is of vital importance in multiple
applications ranging from safety-critical systems to optimization of expected
profit from investments based on a model’s predictions. BDL can be employed to
model two crucial types of uncertainty: aleatoric and epistemic.

Aleatoric uncertainty is stemming from the inability of the model to explain a
particular data point. A simple example would be a performing computer vision
on input from a camera, whose lens has been covered with snow, thus
obstructing view of at least parts of the image. This type of uncertainty is
especially relevant for systems optimized for safety, such as self-driving
cars, where the correct decision could be to disengage the self-driving system
when it is not aware of its surroundings to a satisfactory degree.

Epistemic uncertainty, on the other hand, arises from a lack of sufficient
training data translating to risk of incorrect prediction on data with
features the model has not seen before. Trivially, this uncertainty can be
reduced by increasing the size of the training set. However, in many use-cases
this is not possible, e.g., in time series data epistemic uncertainty is
inevitable as naturally information about the future is not available at
prediction. Such example is provided by Zhu and Laptev, (2017), who have
employed Monte Carlo dropout444444MC dropout is sometimes considered as post-
modelling technique and thus more suitable for the next section. to estimate a
whole posterior distribution of the number of completed Uber trips, which in
turn can be employed in anomaly detection. Monte Carlo dropout works by
repeatedly and randomly deactivating neurons in the neural networks’ hidden
layers during prediction454545This should not be confused with a dropout layer
in neural network architecture. MC dropout does not affect training of the
network, it is used solely during inference.. Consequently, each input is
mapped to a vector of predictions, which can then be used to approximate model
uncertainty by empirically calculating variance. Another possible use case is
creating a framework for risk-adjusting profit expected from investments based
on the uncertainty of prediction in a manner similar to Sharpe ratio.464646For
reference see Sharpe, (1966).

####  3.3.3 Post-hoc explainability

Post-modelling explainability aims to extract explanations in order to
understand the pre-developed models. It can be partitioned into two
fundamental categories: model-level and instance-level.

##### Model-level post-modelling explainability

Model-level post-hoc explainability aspires to provide understanding of
model’s prediction overall, for the whole set of observations. It usually
starts with the assessment of statistical contribution of each feature to the
estimated model. Variable importance calculation for linear and tree-based
models is relatively straightforward, because it routinely exploits their
notable structural characteristics, for instance through the determination of
regression coefficients or computation of number of times a given feature is
employed to split tree nodes.

Nevertheless, in order to enable the comparison of different model structures,
one needs to operate with model-agnostic feature importance algorithms, such
as the permutation-based version of Altmann et al., (2010) or its modern
equivalent of Fisher et al., (2019)474747Implemented in Python as part of
scikit-learn library: https://scikit-learn.org/stable/index.html, which
measures the drop in model predictive performance when a single feature is
randomly shuffled. One has to bear in mind that the aforementioned approach
requires significant computational power for a model with a sizeable number of
features, since it demands training a new model (on the perturbed data) for
each predictor. Moreover, due to randomness in the shuffling step, it is
advised to consider replicating it for several permutations and to ensure a
lack of multicollinear or correlated features.

Another crucial subgroup of model-level techniques levies perturbations of
explanation drivers, where the statistical contribution of a given predictor
is assessed through the delta of model’s predictions after it is altered.
Partial Dependence (Friedman,, 2001)484848Implemented in Python as part of
scikit-learn library: https://scikit-learn.org/stable/index.html profiles
depicts the value of model prediction as a function of a selected explanatory
variable, typically through mean aggregation of individual Ceteris-Paribus
(CP) profiles494949CP profile illustrates the dependence of an instance-level
prediction for a given feature, for more information please refer to instance-
level post-modelling section.. They can be helpful to determine whether the
relationship between the target and a feature is linear, monotonic or more
complex, as illustrated on the example below.

Figure 5: Partial dependence plot of predicted salary versus age
Source: Own elaboration

Recently proposed SHAP Dependence plot (Lundberg and Lee,,
2017)505050Implemented in Python as part of shap library:
https://github.com/slundberg/shap provides richer information than the
traditional Partial Dependence plots, because it guarantees one additional
piece of information, namely the variance of observations. In general,
dependence plots can be easily compared across different model structures,
nevertheless they may deliver misleading summarization in the case of
correlated features. As a potential solution, one may consider Accumulated
Local Effects (ALE) profiles, introduced by Apley and Zhu,
(2019)515151Implemented in Python: https://github.com/blent-ai/ALEPython. When
handling (even strongly) correlated variables, ALE employs actual conditional
marginal distributions instead of considering each marginal distribution of
features, therefore alleviating extrapolation of the response at predictor
values that are far outside the multivariate envelope of the training data
(required by PD).

Another indispensable group of model-level ex-post explainability techniques
focuses on knowledge distillation, aiming to represent a sophisticated model
behaviour as a set of decision rules. In case of networks with less than two
layers, they can be further decomposed into three main categories. To start
with, the decomposition algorithms attempt to split the network into neuron
level and then the outcomes obtained from each neuron are aggregated to
represent a network as a whole, for instance CRED technique of Sato and
Tsukimoto, (2001). Alternatively, pedagogical rule extraction algorithms treat
a neural network as a black box, irrespective of its architecture, resulting
in much translucent approach, for example TREPAN algorithm of Craven and
Shavlik, (1994). However, since the decomposition technique works layer by
layer, it might be tedious and time-consuming. Finally, the eclectics
algorithms incorporate both the aforementioned approaches.

Unfortunately, all of the above-mentioned techniques are not especially
applicable for Deep Learning. Hence, Zilke et al., (2016) suggested a novel
rule extraction algorithm called DeepRED525252Implemented in Python:
https://github.com/camgbus/DeepRED, by extending a decompositional CRED
algorithm to arbitrary number of layers. The proposed algorithm has additional
decision trees as well as intermediate rules for every hidden layer. Rule
extraction from DNN is a divide and conquer process that characterizes each
layer in terms of the previous layer, helping to reduce memory usage and
computational time. Then, each result is merged to get the final rule that
explains the whole deep neural network. Furthermore, Liu et al., (2018)
augmented the aforementioned approach to distil Deep Neural Networks into
vanilla decision trees on multi-class datasets.

Unfortunately, although DeepRED can construct complete trees that are closely
faithful to the original network, the generated trees can be large enough to
make a model incomprehensible, and the implementation of the method still
takes substantial time and memory, being therefore limited in scalability.
Consequently, several approaches to learn the proxy models to imitate the
behaviour of a complex model only locally have been developed. As an example,
Black Box Explanations through Transparent Approximations (BETA) algorithm
(Lakkaraju et al.,, 2017) learns a compact two-level decision set in which
each rule explains part of the model behaviour unambiguously, reaching high
fidelity (agreement between the explanation and the model), low imbrication
(little overlaps between decision rules in the explanation), and high
interpretability (the explanation decision set is lightweight and small).

Similarly, the study of Henelius et al., (2017) introduced the Automatic
STRucture IDentification (ASTRID)535353Implemented in R:
https://github.com/bwrc/astrid-r technique to examine which attributes are
exploited by a classifier in generating a prediction. This approach discovers
the largest subset of features such that the accuracy of the classifier
trained with it is indistinguishable in terms from a classifier built on the
original feature set.

Some mechanisms produce the importance scores in terms of input features of
the model, starting from the last layer, computing the contributions of its
neurons to the target and then consecutively repeating the process layer by
layer until the input layer is reached. The contributions from the previous
layer can be estimated through partial gradients of a target neuron activation
with respect to previous layer neurons as in the Guided
Backprop545454Implemented in Python https://github.com/PAIR-code/saliency
(Springenberg et al.,, 2015) and SmoothGrad555555Implemented in Python:
https://github.com/PAIR-code/saliency (Smilkov et al.,, 2017) techniques or by
decomposing a target neuron activation value into its constituent values
appearing from previous layer as in the Layer-wise Relevance
Propagation565656Implemented in Python: https://github.com/sebastian-
lapuschkin/lrp_toolbox (Bach et al.,, 2015) and Deep Learning Important
FeaTures575757Implemented in Python: https://github.com/kundajelab/deeplift
(Shrikumar et al.,, 2017) approaches.

Each technique adjusts a balance between highlighting areas of high network
activation, where neurons impact strongest, and spaces of high network
sensitivity, where modifications would mostly influence the output. A
comparison of some of those can be found in Ancona et al., (2017).

More mature, but still worth-mentioning approaches focused on rule extraction
by direct mapping of inputs into outputs, rather than aiming to understand the
inner workings of neural networks. In other words, they treated the model as a
black box, but they searched for trends and functions of inputs to outputs.
For instance, Validity Interval Analysis (Thrun,, 1994) mimics neural network
behaviour in order to detect stable intervals, where the correlation between
the input and the predicted class is present. Alternatively, the MofN
algorithm (Towell and Shavlik,, 1993) locates the rules that explain single
neurons by clustering and ignoring the insignificant neurons.

##### Instance-level post-modelling explainability

The second category of ex-post explainability provoke understandability of
prediction for a single observation. The instance-level techniques usually
impose perturbations of explanation drivers and then scrutinize and summarize
their impact on a corresponding output. Therefore, the so-called perturbation
mechanisms are model-agnostic and relatively straightforward to implement.
However, the requirement of massive repetition of perturbations yields several
impediments. Firstly, the construction of a meaningful perturbation is
challenging. Ideally, the perturbations would be driven by the variation that
is observed in the dataset. Thus, widely employed rule-of-thumb modifications,
selected from a set of implausible alternative values can lead to dubious
explanations. Secondly, a plausible interference should be conducted on a
considerable amount of perturbed inputs, which results in a computational
burden, particularly for models with high-dimensional datasets. Four main
representatives of instance-level mechanisms are CP profiles, BD plots, LIME
and SHAP.

Due to the similarity to the Partial Dependence (PD) plots from previous
section, it is optimal to start with Ceteris Paribus (CP) profiles, sometimes
referred to as Individual Conditional Expectations (Goldstein et al.,,
2014)585858Implemented in Python: https://github.com/AustinRochford/PyCEbox.
An ICE plot envisions the dependence of the prediction on a regressor for each
instance separately, culminating in one line per observation, compared to one
line overall in PD plots. The values for an aforementioned line can be
calculated by creating synthetic variants of this specific instance by
replacing the examined feature’s value with values from a grid and computing
accordingly predictions with the black-box model, keeping all other features
unmodified (hence Ceteris Paribus).

For models with a vast number of explanatory variables, the visual
representations provided by ICE plots can be staggering. As a potential
solution aiming to provide a summary among the set of features, one may
consider Ceteris Paribus Oscillations (Biecek and Burzykowski,,
2021)595959Part of DALEX package in R: https://github.com/ModelOriented/DALEX
that quantify the impact of a selected variable on model’s predictions. To be
more specific, they can be viewed as the mean of the absolute deviations
between the CP profile and the instance prediction.

Break-Down (Biecek and Burzykowski,, 2021)606060Implemented in R:
https://github.com/pbiecek/breakDown plots should be considered as a valuable
alternative to ICE plots. For each instance separately, they aim to select
components of the input that cannot be modified without a significant delta in
the prediction. To be more specific, for each regressor the model prediction
is calculated with and without particular feature. Then, the feature
contribution is computed as the distance to the initial model
prediction616161This approach can be therefore regarded as an approximation of
Shapley values where each feature contribution is associated with the average
effect of a feature across all feasible relaxations.. After ordering, BS plots
can be viewed as a model agnostic tool for decomposing the predictions from
black boxes into a concise graphical way of a so-called waterfall chart.

Figure 6: Break Down plot for ice cream sales prediction in a supermarket
Source: Own elaboration

In the depicted example a BD plots of ice cream sales prediction has been
provided. A store manager can then clearly see which factors are expected to
impact demand positively and try to adjust the store accordingly.626262Note
the similarity in interpretation mechanism to the SHAP force plots introduced
later.

Local Interpretable Model-agnostic Explanations (LIME)636363Implemented in
Python: https://github.com/marcotcr/lime, proposed by Ribeiro et al., (2016),
is a flagship example of the perturbation mechanism. In order to satisfy the
model-agnostic property, LIME modifies the input to the model locally. In
other words, instead of attempting to understand the entire model at once, an
individual input instances are modified and their impact on the predictions
monitored, typically by the approximation of the underlying model by an
inherently interpretable one, such as a regularized linear model. The output
of LIME is a list of explanations, reflecting the contribution of each feature
to the prediction of a specific data observation, allowing to mitigate several
biases through a human operator overriding the decisions upon evaluating the
reasons given by the algorithm.

An invaluable alternative is the SHapley Additive exPlanations (SHAP)
method646464Implemented in Python: https://github.com/slundberg/shap, derived
by Lundberg and Lee, (2017). The aforementioned framework exploits Shapley
values, the notion from a cooperative game theory that measures the marginal
contributions of players, to derive the importance of each input feature for a
given instance prediction. Shapley values are obtained by averaging over every
feasible sequence in which the feature could have been incorporated to the
model. Alternatively, for cases where the feature is absent, it is represented
as the expected value over the whole dataset. Naturally, sampling every
possible feature subset of a long list of features is incredibly time-
consuming, especially with high feature counts. Accordingly, a number of
efficient Shapley value estimation approaches have been submitted for given
model architectures, not to mention tree ensembles and deep networks.

SHAP can be particularly useful for situations in which a human agent needs to
act upon particular prediction. A good example is uplift modelling, which is a
technique employed in optimizing the efficiency of direct marketing campaigns.
Suppose the goal is to help a bank’s call centre in targeting customers that
might be willing to buy life insurance through the bank’s bancassurance
partnership. Lead generation might be handled automatically by an uplift model
created by the data science team, but materialization of impact heavily
depends on operational execution. Conversely, the aforementioned execution is
typically based on conversation scripts aiming to master call centre agents in
maximizing conversion. Utilizing SHAP it is possible to elevate such scripts
to a much higher sophistication level.

Figure 7: SHAP force plot depicting the impact of 5 variables on predicted
uplift from marketing campaign
Source: Own elaboration

The SHAP force plot can be interpreted as follows: this particular lead is
more likely than average customer to respond to the marketing campaign
(positive model prediction). His likelihood to be persuaded to buy the life
insurance is increased mainly by his age (equal to 31 years) and the fact that
he is male with 1 child. On the contrary, his income of 3000 units and
spending on digital media of 150 units discourage him to buy more than he
would without being targeted. Consequently, SHAP force plot provides a crucial
input to the agent calling the prospective customer – since age is the main
driver of increased propensity to respond to the campaign, perhaps a script
that emphasises entering a new stage in life will be the most useful in
selling him the life insurance product.

####  3.3.4 Evaluating XAI Performance

The opponents of the XAI algorithms often highlight their limitations, such as
the lack of invariance to the simple transformations of the input (so called
robustness of explanations, Kindermans et al., (2017) or the singe point-wise
explanation. Moreover, it is not rare that two explainable artificial
intelligence techniques provide two distinct interpretations. In light of the
aforementioned obstacles, Gilpin et al., (2018) enumerates additional types of
measures quantifying the performance of XAI approaches:

  1. 1.

Completeness in comparison to the original model: an explanation model can be
assessed directly according to how precisely it approximates the model being
explained, also known as the fidelity.

  2. 2.

Completeness as quantified on a substitute task: an explanation model can be
evaluated in terms of attributes different than direct model explanation, for
instance a justification designed to reveal model sensitivity can be appraised
against a brute-force measurement of the model sensitivity.

  3. 3.

Ability to detect biased models: an explanation model can be gauged for its
ability to expose models with the presence or absence of relevant biases.

  4. 4.

Human evaluation: an explanation model can be graded in terms of
reasonableness, measuring the degree of a match between an explanation and
human expectations.

The use of diverse metrics that align with the purpose and completeness of the
targeted explanation is encouraged. In the rest of the section, the following
measures are thoroughly described: the robustness of explanations and the
fidelity of underlying classifier.

##### Robustness of explanations

Alvarez-Melis and Jaakkola, 2018a  claim that a critical property that any
interpretability method should satisfy is to be robust to local perturbations
of the input, therefore, not to provide substantially different explanations
for similar inputs. Simultaneously, it would guarantee that a simplified model
could be approximately used in lieu of the true complex model, at least in a
relatively small neighbourhood.

The first attempt to investigate the robustness is the visual inspection of
attributes, however, it may be subjective and nearly infeasible for higher-
dimensional inputs. Consequently, algebra literature proposes the notion of
Lipschitz continuity, which measures the relative change in the output with
respect to the input. To be more specific, because the typical definition of
Lipschitz continuity is global, Alvarez-Melis and Jaakkola, 2018a  suggest the
generalization in the form of a point-wise, neighbourhood-based local
Lipschitz continuity, defined as:

######  Definition 1.

Function $f:X\subseteq\mathbb{R}^{n}\to\mathbb{R}^{m}$ is locally Lipschitz if
for every $x_{0}$ there exist $\delta>0$ and $L\in\mathbb{R}$ such that
$||x-x_{0}||<\delta$ implies $||f(x)-f(x_{0})||\leq L||x-x_{0}||$

In contrast to the global Lipschitz criterion, in the above-mentioned
definition both $\delta$ and $L$ depend on the anchor point $x_{0}$. Moreover,
the notion can be easily extended to handle models with discrete inputs as
well. Unfortunately, the value of $L$ is rarely known a priori, instead it can
be estimated through an optimization problem (Alvarez-Melis and Jaakkola,
2018a, , see), for instance by the employment of Bayesian Optimization (Snoek
et al.,, 2012).

There is no universally desirable value of $L$, since its acceptable level
depends on the application and the goal of interpretability. Therefore, it is
useful to employ the robustness relatively across XAI techniques: an
explainability algorithm with lower Lipschitz constant $L$ is more robust.
Intuitively, more stable explanations (lower is better) tend to be more
trustworthy (Ghorbani et al.,, 2019).

##### Fidelity of the underlying classifier

The interpretation framework is characterized by a high fidelity if the
prediction produced by an explanation eminently agrees with the original model
(Plumb et al.,, 2019). Intuitively, an explainer with good fidelity precisely
conveys which patterns the model has used in order to deliver a certain
prediction. In other words, it depicts the degree to which each feature
influences the model’s prediction.

Moreover, satisfactory explanations should be able to illustrate a
counterfactual dependence, illustrating the level of changes in explanandum
(defined as the phenomenon to be explained) due to the modifications of
initial conditions. As a consequence, the relevance of features can be
measured as the impact of their changes to the value of explanandum.

To be more specific, Wachter et al., (2017) argue that single predictions
should be explained by w-counterfactuals that measure the minimum changes
required for an observation to flip its classification. A Counterfactual Local
Explanations via Regression (CLEAR), introduced by White and Garcez, (2019),
grasp from that analysis, since it generates w-counterfactual explanations and
then constructs local regression models. In contrast to the LIME method, which
also builds regression to produce local explanations, in the next step the
calculated w-counterfactuals are employed to improve the fidelity of its
regressions.

###  3.4 Expected evolution of XAI

In recent years, progress of XAI has been promising, with efforts in
explanation of deep learning processing yielding encouraging results. However,
there is an ample room for improvement, since diagnosing the predominant
features and simplifying the problem space does not solve all possible
obstacles associated with comprehending opaque models.

In addition, XAI techniques should not overlook the format of explanations.
Bearing in mind that understandability is often provided to not data-savvy
end-users, sophisticated statistical information provided by the vast majority
of XAI algorithms can be overwhelming and consequently redundant. Hence,
presentation formats should be further enhanced to better promote user
satisfaction. As a potential solution of more human-friendly explanations, one
can focus on contrastive explanations. Namely, instead of clarifying why a
specific prediction was made, XAI can aim to explain why this prediction was
made instead of another. For instance, consider a declined mortgage case. Then
instead of answering the “Why have I not received the mortgage?”, maybe more
plausible would be the response to “Why have I not received the mortgage,
while my cousin has?”.

##  4 Conclusion

The paper has provided an overview of fairness and XAI to help the reader to
understand the two concepts and their possible implications for business.

Since future implementations of AI in numerous fields will utilize methods
from fairness and XAI in a similar manner as predictive performance scores are
used today, businesses should start preparing immediately. Evaluating which of
current use-cases could benefit from fairness methods, adopting relevant
organizational practices to ensure interpretability and finally investing in
knowledge and tools to data practitioners can help transforming any currently
lagging firm into the data-driven market leader.

## References

  * Aharon et al., (2006) Aharon, M., Elad, M., and Bruckstein, A. (2006).  K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation.  IEEE Transactions on Signal Processing, 54(11):4311–4322.
  * Altmann et al., (2010) Altmann, A., Toloşi, L., Sander, O., and Lengauer, T. (2010).  Permutation importance: a corrected feature importance measure.  Bioinformatics, 26(10):1340–1347.
  * (3) Alvarez-Melis, D. and Jaakkola, T. S. (2018a).  On the Robustness of Interpretability Methods.  arXiv:1806.08049 [cs, stat].  arXiv: 1806.08049.
  * (4) Alvarez-Melis, D. and Jaakkola, T. S. (2018b).  Towards robust interpretability with self-explaining neural networks.
  * Ancona et al., (2017) Ancona, M., Ceolini, E., Öztireli, C., and Gross, M. (2017).  A unified view of gradient-based attribution methods for deep neural networks.
  * Angwin et al., (2016) Angwin, J., Larson, J., Mattu, S., and Kirchner, L. (2016).  Machine bias risk assessments in criminal sentencing.  https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.
  * Apley, (2018) Apley, D. (2018).  Accumulated local effects (ale) plots and partial dependence (pd) plots.
  * Apley and Zhu, (2019) Apley, D. W. and Zhu, J. (2019).  Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models.  arXiv:1612.08468 [stat].  arXiv: 1612.08468.
  * Arrieta et al., (2019) Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., Chatila, R., and Herrera, F. (2019).  Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI.  arXiv:1910.10045 [cs].  arXiv: 1910.10045.
  * Arslan, (2019) Arslan, R. C. (2019).  How to Automatically Document Data With the codebook Package to Facilitate Data Reuse.  Advances in Methods and Practices in Psychological Science, 2(2):169–187.
  * Bach et al., (2015) Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. (2015).  On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation.  PLOS ONE, 10(7):e0130140.
  * Barocas et al., (2019) Barocas, S., Hardt, M., and Narayanan, A. (2019).  Fairness and Machine Learning.  fairmlbook.org.  http://www.fairmlbook.org.
  * Bender and Friedman, (2018) Bender, E. M. and Friedman, B. (2018).  Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science.  Transactions of the Association for Computational Linguistics, 6:587–604.
  * Beutel et al., (2017) Beutel, A., Chen, J., Zhao, Z., and Chi, E. H. (2017).  Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations.  arXiv:1707.00075 [cs].  arXiv: 1707.00075.
  * Biecek and Burzykowski, (2021) Biecek, P. and Burzykowski, T. (2021).  Explanatory Model Analysis.  Chapman and Hall/CRC, New York.
  * Bogdan et al., (2016) Bogdan, W., Boniecki, D., Krok, W., Namysł, W., and Stefański, J. (2016).  Digital poles: Consumers in a time of e-revolution.  https://www.mckinsey.com/pl/~/media/McKinsey/Locations/Europe%20and%20Middle%20East/Polska/Raporty/Cyfrowi%20Polacy%20Konsumenci%20w%20czasach%20e-rewolucji/Digital%20Poles%20-%20report%20in%20English.ashx.
  * Brendel and Bethge, (2019) Brendel, W. and Bethge, M. (2019).  Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet.  arXiv:1904.00760 [cs, stat].  arXiv: 1904.00760.
  * Buolamwini and Gebru, (2018) Buolamwini, J. and Gebru, T. (2018).  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In Friedler, S. A. and Wilson, C., editors, Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pages 77–91, New York, NY, USA. PMLR.
  * Calders and Verwer, (2010) Calders, T. and Verwer, S. (2010).  Three naive Bayes approaches for discrimination-free classification.  Data Mining and Knowledge Discovery, 21(2):277–292.
  * Calmon et al., (2018) Calmon, F. d. P., Wei, D., Vinzamuri, B., Ramamurthy, K. N., and Varshney, K. R. (2018).  Data Pre-Processing for Discrimination Prevention: Information-Theoretic Optimization and Analysis.  IEEE Journal of Selected Topics in Signal Processing, 12(5):1106–1119.
  * Card et al., (2019) Card, D., Zhang, M., and Smith, N. A. (2019).  Deep Weighted Averaging Classifiers.  In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 369–378, Atlanta GA USA. ACM.
  * Chen et al., (2018) Chen, I., Johansson, F. D., and Sontag, D. (2018).  Why is my classifier discriminatory?
  * Chen et al., (2016) Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. (2016).  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.
  * Choi et al., (2017) Choi, E., Bahadori, M. T., Kulas, J. A., Schuetz, A., Stewart, W. F., and Sun, J. (2017).  Retain: An interpretable predictive model for healthcare using reverse time attention mechanism.
  * Craven and Shavlik, (1994) Craven, M. W. and Shavlik, J. W. (1994).  Using sampling and queries to extract rules from trained neural networks.  In In Proceedings of the Eleventh International Conference on Machine Learning, pages 37–45. Morgan Kaufmann.
  * Datta et al., (2016) Datta, A., Sen, S., and Zick, Y. (2016).  Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems.  In 2016 IEEE Symposium on Security and Privacy (SP), pages 598–617, San Jose, CA. IEEE.
  * DuMouchel et al., (1999) DuMouchel, W., Volinsky, C., Johnson, T., Cortes, C., and Pregibon, D. (1999).  Squashing flat files flatter.  In Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining - KDD ’99, pages 6–15, San Diego, California, United States. ACM Press.
  * Erion et al., (2020) Erion, G., Janizek, J. D., Sturmfels, P., Lundberg, S., and Lee, S.-I. (2020).  Improving performance of deep learning models with axiomatic attribution priors and expected gradients.  arXiv:1906.10670 [cs, stat].  arXiv: 1906.10670.
  * Escalante et al., (2018) Escalante, H. J., Escalera, S., Guyon, I., Baro, X., Gucluturk, Y., Guclu, U., and van Gerven, M. (2018).  Explainable and Interpretable Models in Computer Vision and Machine Learning.  Springer Publishing Company, Incorporated, 1st edition.
  * Feldman et al., (2015) Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., and Venkatasubramanian, S. (2015).  Certifying and Removing Disparate Impact.  In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 259–268, Sydney NSW Australia. ACM.
  * Fisher et al., (2019) Fisher, A., Rudin, C., and Dominici, F. (2019).  All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously.
  * Friedman, (2001) Friedman, J. H. (2001).  Greedy function approximation: A gradient boosting machine.  The Annals of Statistics, 29(5):1189 – 1232.
  * Garcez et al., (2019) Garcez, A. d., Gori, M., Lamb, L. C., Serafini, L., Spranger, M., and Tran, S. N. (2019).  Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning.  arXiv:1905.06088 [cs].  arXiv: 1905.06088.
  * Gebru et al., (2020) Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., and Crawford, K. (2020).  Datasheets for Datasets.  arXiv:1803.09010 [cs].  arXiv: 1803.09010.
  * Ghaeini et al., (2019) Ghaeini, R., Fern, X. Z., Shahbazi, H., and Tadepalli, P. (2019).  Saliency Learning: Teaching the Model Where to Pay Attention.  arXiv:1902.08649 [cs].  arXiv: 1902.08649.
  * Ghorbani et al., (2019) Ghorbani, A., Abid, A., and Zou, J. (2019).  Interpretation of Neural Networks Is Fragile.  Proceedings of the AAAI Conference on Artificial Intelligence, 33:3681–3688.
  * Gilpin et al., (2018) Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., and Kagal, L. (2018).  Explaining Explanations: An Overview of Interpretability of Machine Learning.  In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80–89, Turin, Italy. IEEE.
  * Goldstein et al., (2014) Goldstein, A., Kapelner, A., Bleich, J., and Pitkin, E. (2014).  Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation.
  * Hall and Gill, (2018) Hall, P. and Gill, N. (2018).  An Introduction to Machine Learning Interpretability: An Applied Perspective on Fairness, Accountability, Transparency, and Explainable AI.  O’Reilly Media.
  * Hardt et al., (2016) Hardt, M., Price, E., Price, E., and Srebro, N. (2016).  Equality of opportunity in supervised learning.  In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
  * Henelius et al., (2017) Henelius, A., Puolamäki, K., and Ukkonen, A. (2017).  Interpreting Classifiers through Attribute Interactions in Datasets.  arXiv:1707.07576 [cs, stat].  arXiv: 1707.07576.
  * Higgins et al., (2017) Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017).  beta-vae: Learning basic visual concepts with a constrained variational framework.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
  * Hind et al., (2019) Hind, M., Wei, D., Campbell, M., Codella, N. C. F., Dhurandhar, A., Mojsilović, A., Natesan Ramamurthy, K., and Varshney, K. R. (2019).  TED: Teaching AI to Explain its Decisions.  In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 123–129, Honolulu HI USA. ACM.
  * Holland et al., (2018) Holland, S., Hosny, A., Newman, S., Joseph, J., and Chmielinski, K. (2018).  The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards.  arXiv:1805.03677 [cs].  arXiv: 1805.03677.
  * Jain and Wallace, (2019) Jain, S. and Wallace, B. C. (2019).  Attention is not Explanation.  arXiv:1902.10186 [cs].  arXiv: 1902.10186.
  * Jakubovitz and Giryes, (2019) Jakubovitz, D. and Giryes, R. (2019).  Improving dnn robustness to adversarial attacks using jacobian regularization.
  * Johnson et al., (2016) Johnson, M. J., Duvenaud, D. K., Wiltschko, A., Adams, R. P., and Datta, S. R. (2016).  Composing graphical models with neural networks for structured representations and fast inference.  In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
  * Kamiran and Calders, (2009) Kamiran, F. and Calders, T. (2009).  Classifying without discriminating.  In 2009 2nd International Conference on Computer, Control and Communication, pages 1–6.
  * Kamiran and Calders, (2012) Kamiran, F. and Calders, T. (2012).  Data preprocessing techniques for classification without discrimination.  Knowledge and Information Systems, 33(1):1–33.
  * Kamiran et al., (2010) Kamiran, F., Calders, T., and Pechenizkiy, M. (2010).  Discrimination aware decision tree learning.  In 2010 IEEE International Conference on Data Mining, pages 869–874.
  * Kamiran et al., (2012) Kamiran, F., Karim, A. A., and Zhang, X. (2012).  Decision theory for discrimination-aware classification.
  * Kamishima et al., (2011) Kamishima, T., Akaho, S., and Sakuma, J. (2011).  Fairness-aware learning through regularization approach.  pages 643–650.
  * Kenny and M, (2019) Kenny, E. and M, K. (2019).  Twin-systems for explaining artificial neural networks using cbr: Experiments using feature-weighting schemes to retrieve explanatory cases.  In International Joint Conference on Artificial Intelligence.
  * Kim et al., (2016) Kim, B., Khanna, R., and Koyejo, O. O. (2016).  Examples are not enough, learn to criticize! criticism for interpretability.  In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
  * Kim et al., (2015) Kim, B., Rudin, C., and Shah, J. (2015).  The bayesian case model: A generative approach for case-based reasoning and prototype classification.
  * Kim et al., (2018) Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., and Sayres, R. (2018).  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).  arXiv:1711.11279 [stat].  arXiv: 1711.11279.
  * Kindermans et al., (2017) Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., Erhan, D., and Kim, B. (2017).  The (un)reliability of saliency methods.
  * Kingma and Welling, (2014) Kingma, D. P. and Welling, M. (2014).  Auto-encoding variational bayes.
  * Kojarski and Lorenz, (2006) Kojarski, S. and Lorenz, D. H. (2006).  Comparing white-box, black-box, and glass-box composition of aspect mechanisms.  In Morisio, M., editor, Reuse of Off-the-Shelf Components, pages 246–259, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Krishnan et al., (2007) Krishnan, C., George, T. I., and Arber, D. A. (2007).  Bone Marrow Metastases: A Survey of Nonhematologic Metastases With Immunohistochemical Study of Metastatic Carcinomas.  Applied Immunohistochemistry & Molecular Morphology, 15(1):1–7.
  * Krishnan et al., (2015) Krishnan, R. G., Shalit, U., and Sontag, D. (2015).  Deep kalman filters.
  * Kulkarni et al., (2015) Kulkarni, T. D., Whitney, W. F., Kohli, P., and Tenenbaum, J. (2015).  Deep convolutional inverse graphics network.  In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
  * Lakkaraju et al., (2017) Lakkaraju, H., Kamar, E., Caruana, R., and Leskovec, J. (2017).  Interpretable & Explorable Approximations of Black Box Models.  arXiv:1707.01154 [cs].  arXiv: 1707.01154.
  * Lee et al., (2019) Lee, G.-H., Jin, W., Alvarez-Melis, D., and Jaakkola, T. S. (2019).  Functional Transparency for Structured Data: a Game-Theoretic Approach.  arXiv:1902.09737 [cs, stat].  arXiv: 1902.09737.
  * Li et al., (2014) Li, Y., Swersky, K., and Zemel, R. (2014).  Learning unbiased features.  arXiv:1412.5244 [cs, stat].  arXiv: 1412.5244.
  * Liu et al., (2018) Liu, X., Wang, X., and Matwin, S. (2018).  Improving the Interpretability of Deep Neural Networks with Knowledge Distillation.  In 2018 IEEE International Conference on Data Mining Workshops (ICDMW), pages 905–912, Singapore, Singapore. IEEE.
  * Louizos et al., (2017) Louizos, C., Swersky, K., Li, Y., Welling, M., and Zemel, R. (2017).  The variational fair autoencoder.
  * Lundberg and Lee, (2017) Lundberg, S. and Lee, S.-I. (2017).  A unified approach to interpreting model predictions.
  * Madigan et al., (2002) Madigan, D., Raghavan, N., Dumouchel, W., Nason, M., Posse, C., and Ridgeway, G. (2002).  Likelihood-based data squashing: A modeling approach to instance construction.  Data Mining and Knowledge Discovery, 6(2):173–190.
  * McInnes et al., (2020) McInnes, L., Healy, J., and Melville, J. (2020).  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.  arXiv:1802.03426 [cs, stat].  arXiv: 1802.03426.
  * Molnar, (2019) Molnar, C. (2019).  Interpretable Machine Learning.  https://christophm.github.io/interpretable-ml-book/.
  * Obermeyer et al., (2019) Obermeyer, Z., Powers, B., Vogeli, C., and Mullainathan, S. (2019).  Dissecting racial bias in an algorithm used to manage the health of populations.  Science, 366(6464):447–453.
  * Papernot and McDaniel, (2018) Papernot, N. and McDaniel, P. (2018).  Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning.  arXiv:1803.04765 [cs, stat].  arXiv: 1803.04765.
  * Park and Jun, (2009) Park, H.-S. and Jun, C.-H. (2009).  A simple and fast algorithm for K-medoids clustering.  Expert Systems with Applications, 36(2):3336–3341.
  * Pleiss et al., (2017) Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J. M., and Weinberger, K. Q. (2017).  On fairness and calibration.  CoRR, abs/1709.02012.
  * Plumb et al., (2019) Plumb, G., Al-Shedivat, M., Xing, E., and Talwalkar, A. (2019).  Regularizing Black-box Models for Improved Interpretability (HILL 2019 Version).  arXiv:1906.01431 [cs, stat].  arXiv: 1906.01431.
  * Pétavy et al., (2019) Pétavy, F., Seigneuret, N., Hudson, L. D., Jauregui, B., Becnel, L. B., Galvez, J., Jackson, T., Kush, R. D., Harmon, N., Zhou, F. L., Malone, J. K., Almario, E. N., and Fitzmartin, R. (2019).  Global standardization of clinical research data.  https://www.appliedclinicaltrialsonline.com/view/global-standardization-clinical-research-data.
  * Ribeiro et al., (2016) Ribeiro, M. T., Singh, S., and Guestrin, C. (2016).  "Why Should I Trust You?": Explaining the Predictions of Any Classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144, San Francisco California USA. ACM.
  * Rieger et al., (2020) Rieger, L., Singh, C., Murdoch, W. J., and Yu, B. (2020).  Interpretations are useful: penalizing explanations to align neural networks with prior knowledge.  arXiv:1909.13584 [cs, stat].  arXiv: 1909.13584.
  * Ross et al., (2017) Ross, A. S., Hughes, M. C., and Doshi-Velez, F. (2017).  Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations.  arXiv:1703.03717 [cs, stat].  arXiv: 1703.03717.
  * Roweis, (2000) Roweis, S. T. (2000).  Nonlinear Dimensionality Reduction by Locally Linear Embedding.  Science, 290(5500):2323–2326.
  * Rudin, (2019) Rudin, C. (2019).  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature Machine Intelligence, 1(5):206–215.
  * Sak et al., (2014) Sak, H., Senior, A. W., and Beaufays, F. (2014).  Long short-term memory recurrent neural network architectures for large scale acoustic modeling.
  * Samek et al., (2019) Samek, W., Montavon, G., Vedaldi, A., Hansen, L. K., and Müller, K., editors (2019).  Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, volume 11700 of Lecture Notes in Computer Science.  Springer.
  * Sato and Tsukimoto, (2001) Sato, M. and Tsukimoto, H. (2001).  Rule extraction from neural networks via decision tree induction.  In IJCNN’01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222), volume 3, pages 1870–1875, Washington, DC, USA. IEEE.
  * Sharpe, (1966) Sharpe, W. F. (1966).  Mutual Fund Performance.  The Journal of Business, 39(S1):119.
  * Shrikumar et al., (2017) Shrikumar, A., Greenside, P., and Kundaje, A. (2017).  Learning important features through propagating activation differences.  In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3145–3153. PMLR.
  * Silberg and Manyika, (2019) Silberg, J. and Manyika, J. (2019).  Tackling bias in artificial intelligence (and in humans).  https://www.mckinsey.com/featured-insights/artificial-intelligence/tackling-bias-in-artificial-intelligence-and-in-humans.
  * Smilkov et al., (2017) Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. (2017).  SmoothGrad: removing noise by adding noise.  arXiv:1706.03825 [cs, stat].  arXiv: 1706.03825.
  * Snoek et al., (2012) Snoek, J., Larochelle, H., and Adams, R. P. (2012).  Practical bayesian optimization of machine learning algorithms.
  * Sprechmann and Sapiro, (2010) Sprechmann, P. and Sapiro, G. (2010).  Dictionary learning and sparse coding for unsupervised clustering.  In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 2042–2045.
  * Springenberg et al., (2015) Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2015).  Striving for Simplicity: The All Convolutional Net.  arXiv:1412.6806 [cs].  arXiv: 1412.6806.
  * Szegedy et al., (2014) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2014).  Intriguing properties of neural networks.
  * Thrun, (1994) Thrun, S. (1994).  Extracting rules from artificial neural networks with distributed representations.  In Proceedings of the 7th International Conference on Neural Information Processing Systems, NIPS’94, page 505–512, Cambridge, MA, USA. MIT Press.
  * Towell and Shavlik, (1993) Towell, G. G. and Shavlik, J. W. (1993).  Extracting refined rules from knowledge-based neural networks.  Machine Learning, 13(1):71–101.
  * Wachter et al., (2017) Wachter, S., Mittelstadt, B., and Russell, C. (2017).  Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.  SSRN Electronic Journal.
  * Wan et al., (2021) Wan, A., Dunlap, L., Ho, D., Yin, J., Lee, S., Jin, H., Petryk, S., Bargal, S. A., and Gonzalez, J. E. (2021).  NBDT: Neural-Backed Decision Trees.  arXiv:2004.00221 [cs].  arXiv: 2004.00221.
  * Wang and Lin, (2019) Wang, T. and Lin, Q. (2019).  Hybrid Predictive Model: When an Interpretable Model Collaborates with a Black-box Model.  arXiv:1905.04241 [cs, stat].  arXiv: 1905.04241.
  * White and Garcez, (2019) White, A. and Garcez, A. d. (2019).  Measurable Counterfactual Local Explanations for Any Classifier.  arXiv:1908.03020 [cs].  arXiv: 1908.03020.
  * Woodworth et al., (2017) Woodworth, B., Gunasekar, S., Ohannessian, M. I., and Srebro, N. (2017).  Learning Non-Discriminatory Predictors.  arXiv:1702.06081 [cs].  arXiv: 1702.06081.
  * Wu et al., (2017) Wu, M., Hughes, M. C., Parbhoo, S., Zazzi, M., Roth, V., and Doshi-Velez, F. (2017).  Beyond sparsity: Tree regularization of deep models for interpretability.
  * Zafar et al., (2017) Zafar, M. B., Valera, I., Rodriguez, M. G., and Gummadi, K. P. (2017).  Fairness constraints: Mechanisms for fair classification.
  * Zemel et al., (2013) Zemel, R., Wu, Y., Swersky, K., Pitassi, T., and Dworak, C. (2013).  Icml’13: Proceedings of the 30th international conference on international conference on machine learning - volume 28.
  * Zhang et al., (2018) Zhang, B. H., Lemoine, B., and Mitchell, M. (2018).  Mitigating Unwanted Biases with Adversarial Learning.  In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335–340, New Orleans LA USA. ACM.
  * Zhou and Chen, (2018) Zhou, J. and Chen, F., editors (2018).  Human and Machine Learning - Visible, Explainable, Trustworthy and Transparent.  Human-Computer Interaction Series. Springer.
  * Zhu and Laptev, (2017) Zhu, L. and Laptev, N. (2017).  Deep and Confident Prediction for Time Series at Uber.  2017 IEEE International Conference on Data Mining Workshops (ICDMW), pages 103–110.  arXiv: 1709.01907.
  * Zilke et al., (2016) Zilke, J. R., Loza Mencía, E., and Janssen, F. (2016).  Deepred – rule extraction from deep neural networks.  In Calders, T., Ceci, M., and Malerba, D., editors, Discovery Science, pages 457–473, Cham. Springer International Publishing.