# Evaluating explainability for machine learning predictions using model-
agnostic metrics

Cristian Munoz
Holistic AI
<EMAIL_ADDRESS>
&Kleyton da Costa
Holistic AI
<EMAIL_ADDRESS>
&Bernardo Modenesi
University of Michigan
<EMAIL_ADDRESS>
&Adriano Koshiyama
Holistic AI
<EMAIL_ADDRESS>

###### Abstract

Rapid advancements in artificial intelligence (AI) technology have brought
about a plethora of new challenges in terms of governance and regulation. AI
systems are being integrated into various industries and sectors, creating a
demand from decision-makers to possess a comprehensive and nuanced
understanding of the capabilities and limitations of these systems. One
critical aspect of this demand is the ability to explain the results of
machine learning models, which is crucial to promoting transparency and trust
in AI systems, as well as fundamental in helping machine learning models to be
trained ethically. In this paper, we present novel metrics to quantify the
degree of which AI model predictions can be easily explainable by its
features. Our metrics summarize different aspects of explainability into
scalars, providing a more comprehensive understanding of model predictions and
facilitating communication between decision-makers and stakeholders, thereby
increasing the overall transparency and accountability of AI systems.

##  1 Introduction

Despite the remarkable recent evolution in prediction performance by
artificial intelligence (AI) models, they are often deemed as “black boxes”,
i.e. models whose prediction mechanisms cannot be understood simply from their
parameters. In general terms, an explainable or interpretable algorithm is one
for which the rules for its prediction decisions can be questioned and
explained in a way that is intelligible to humans. In particular,
interpretability regards the ability to extract causal knowledge about the
world from a model, and explainability regards being able to explain precisely
how a complex model arrived at certain predictions, i.e. its mechanics.
Understanding AI models’ behavior is important to explain predictions to
support the decision-making process, debug unexpected behavior for models
(contributing to improving the model accuracy), refine modeling and data
mining processes, verify that model behavior is reasonable and fair, present
the model’s predictions to stakeholders, etc.

The objectives of explainable artificial intelligence (XAI) encompass a set of
critical goals Ali et al. (2023). Firstly, it aims to empower individuals by
providing them with the ability to make informed decisions, thereby mitigating
potential harms that fully autonomous decision-making systems may produce.
Second, it seeks to enhance the decision-making process for individuals by
offering transparent information about the outputs of AI models, facilitating
well-informed choices. Third, XAI strives to identify and address
vulnerabilities that could compromise machine learning-based systems,
ultimately bolstering their resilience. Lastly, it endeavors to boost user
confidence in AI systems by promoting transparency and fostering a clear
understanding of the decisions made by models.

Explainable metrics play a crucial role in achieving transparent and
trustworthy outcomes of machine learning models. However, in XAI, the absence
of a ground truth for explainability complicates the definition of these
metrics, requiring consideration of factors like fidelity and unambiguity. To
approach this issue, the XAI metrics can be organized into three groups with
different degrees of human intervention: subjective metrics, which gauge user
responses based on trust and satisfaction; objective metrics, focusing on
observed aspects like users interpretation on model’s output; and
computational metrics, derived mathematically without user intervention,
making them suitable for automated systems. These categories relate to diverse
evaluation approaches for explainability methods, with studies demonstrating
their application in measuring understanding and quality.

Our inquiry delves into the feasibility of quantifying elements such as the
ease of interpreting the results from explainability methods. For instance, we
leverage aspects of feature importance to construct an indicator of model
explainability. This study investigates the efficiency of automatically
comparing explainability between different machine learning models, and
examines the construction of an explainability indicator in the absence of a
ground truth. Furthermore, the paper explores methods for measuring model
interpretability that transcend the specific nature of the model in question.

This research contributes to the field of Explainable Artificial Intelligence
(XAI) by introducing a set of novel explainability computational metrics
tailored to evaluate the explanations generated for various AI models. Unlike
existing studies that primarily focus on assessing the performance of
explainability methods, our work uniquely emphasizes the evaluation of the
explanations themselves. These metrics provide distinctive features, enhancing
the toolkit for deciphering ”black box” models and empowering stakeholders in
the AI ecosystem. They systematically evaluate crucial dimensions, including
the concentration of feature importance, feasibility of surrogate
interpretation, variation in feature importance across categorical and
percentile-based regions, and the stability of feature importance across the
data or feature spectrum. The paper details as follows: Section 2 describes
the literature and common methodological approach from which we build our
methods; Section 3 presents our proposed methods for explainable AI at local
and global levels; Section 4 brings a set of applications for our new methods;
and, we conclude the paper in Section 5.

##  2 Literature and Technical Background

The literature in explainable methods for AI models has been gaining
prominence as AI models proliferate in virtually all areas of society (such as
predicting hypertension Elshawi et al. (2019), healthcare ElShawi et al.
(2020), COVID-19 diagnosis Buckmann et al. (2022), economics and finance
Thimoteo et al. (2022)). In this section, we discuss some of the XAI methods
and forms to categorize them based on their proprieties. The discussion is far
from being exhaustive, given how effervescent this field is, but the goal of
this section is to help the reader understand our proposed XAI metrics and how
they relate to the rest of the literature.

In this way, we can attribute a prominent role to metrics in achieving these
objectives. Unlike other strategies that may lead to misleading
interpretations of their outcomes, well-defined metrics tend to efficiently
elucidate issues and enhance risk management in sensitive processes. Moreover,
the use of metrics facilitates comparisons between different sets of results,
making it effective, for instance, in hypothesis validation. Several factors,
however, complicate the definition of metrics. In the case of XAI, the absence
of a ground truth for explainability adds a layer of complexity to the
comparison of different strategies, as pointed out by Zhou et al. (2021). In
this context, potential strategies involve considering aspects such as
fidelity, unambiguity, and overlap, as discussed by Lakkaraju et al. (2017).
The existing literature introduces various methods for evaluating
explainability techniques Mothilal et al. (2021). Mothilal et al. (2021)
presents a framework that unifies strategies centred around feature
attribution and counterfactual generation. In contrast, other studies propose
explainability metrics grounded in algorithmic stability Fel et al. (2022);
Khaire and Dhanalakshmi (2022); Nogueira et al. (2018). However, it is
essential to clarify that our objective is not to evaluate the explainability
methods themselves. Instead, our focus is on providing insights based on the
importance attributed to features by different models. In doing so, we aim to
facilitate comparisons of explanations across models.

We can define at least three categories of metrics with an emphasis on the
explainability of results generated for AI models: (i) subjective, (ii)
objective, and (iii) computational. The subjective metrics are employed when
evaluating aspects that elicit subjective responses from users. This type of
metric may be based on trust, understanding, and satisfaction, as proposed in
Hoffman et al. (2018). The objective metrics are those related to observed
aspects, for example, in users performing a particular task. They can be
measures of time for the execution of the task or accuracy. Schmidt and
Biessmann (2019) seeks to objectively measure the quality of explainability
methods and shows that quick and highly accurate decisions represent a good
understanding of explainability. Narayanan et al. (2018) evaluate
explainability results based on subjective (satisfaction) and objective
(response time and accuracy) metrics. Furthermore, computational metrics are
derived from mathematical indicators that assess the quality of explanations
generated by an XAI method. Since these metrics are based on specific
equations, user intervention is not necessary for obtaining them, making this
type of metric suitable for automated systems.

###  2.1 Computational Metrics for Evaluating XAI Outputs

With the growing use of explainability methods for machine learning models,
there is also an increase in studies that seek to evaluate the results based
on feature importance or feature-attribution. Works investigating
computational metrics may aim to construct metrics that assess methods based
on their desired properties, such as fidelity, stability, comprehensibility,
representativity, and consistency. Fel and Vigouroux (2020) focuses on
constructing metrics that address representativity and consistency. The work
proposed by Nguyen and Martínez (2020) suggests a set of metrics that also
rely on certain properties, and in the case of evaluating feature-attribution
methods, the metrics (monotonicity and non-sensitivity) seek to assess how
faithful the methods are. On the other hand, other strategies evaluate
feature-attribution methods based on different factors. Rong et al. (2022)
proposes an information-theoretic strategy to evaluate feature-attribution
methods.

###  2.2 Relevant approaches in XAI methods

The permutation feature importance was first introduced by Fisher et al.
(2019). The work proposed by Fisher et al. (2019) constructs a model-agnostic
method for machine learning predictions. Intuitively, if altering the values
of certain features results in a considerable change in the AI model error,
this feature is considered to be important. Alternatively, features are deemed
unimportant if the AI model error remains unchanged after altering its values.
Permutation feature importance is a powerful tool because its interpretation
is intuitive and it can be applied to any model – i.e. it is an easy-to-
understand global XAI method. In addition, it does not require retraining the
model, nor knowing its estimates and nor its modus operandi. On the other
hand, there is no consensus in the literature about whether a training or test
set should be used to compute the feature’s importance.

Another explainability method is the partial dependence curve. A machine
learning model $f$ is often a function of a multitude of features $x$, which
makes it infeasible to plot the estimated model in a high-dimensional space.
Instead, the partial dependence curve, can be used to assess how the predicted
outcome of a model $f$ behaves as a function of values of a particular
selected feature $x_{s}$, after averaging $f$ over the values of all other
features $x_{A}$. Intuitively, the partial dependence curve can be interpreted
as the expected/average model response as a function of the input feature of
interest Pedregosa et al. (2011). This helps with model explainability since
it makes it possible to assess whether the relationship between the outcome
and a feature is linear, and/or monotonic, or more complex. For example, if
applied to a linear regression, the partial dependence plot always shows a
linear relationship. The estimation assumes that features are not correlated.
The violation of this assumption indicates that the averages calculated for
the partial dependence may include data points that are implausible.

The relationship between features and the outcome of an AI model is often too
complex to be easily summarized in black-box models. On the other hand, there
are models where explainability is straightforward, such as linear regression
or relatively small decision trees. It turns out that one could use the latter
simpler models – also referred to as surrogate models – to approximate a
complex black-box model, often locally (i.e. for a subset of observations),
and reap the explainability properties of the approximating model. In other
words, for a subset of observations, it is possible that a simple and
explainable model can approximate reasonably well complex predictions made by
a complex model, and, in turn, offer a straightforward connection between
features and output in the prediction process. This surrogate model
approximation can be implemented by fitting a simple explainable model to a
dataset corresponding to the same features used to train the complex AI model,
while using the outcome generated by the AI model, as opposed to using the
observed outcome.

The additive feature attribution methods are also well-defined in literature,
with many applications Li (2022); Man and Chan (2021); Feng et al. (2021);
Meng et al. (2020); Parsa et al. (2020); Nohara et al. (2022); Rodríguez-Pérez
and Bajorath (2020); Alabi et al. (2023); Gabbay et al. (2021); Magesh et al.
(2020). As described by Lundberg and Lee (2017), this class of methods have an
explanation model there is a linear function of binary variables. Several
models follow this additive feature attribution definition. The local
interpretable model-agnostic explanations (LIME) proposed by Ribeiro et al.
(2016) focuses on providing explanations for any classifier (or regressor) at
the local level. As an additive feature attribution method, it aims to explain
why a particular prediction was made for a specific instance. The purpose of
the Shaply additive explanations (SHAP) method introduced by Lundberg and Lee
(2017), uses Shapley values Shapley et al. (1953) to compute feature
attribution. Taking as input a set function $v:2^{n}\rightarrow\mathbb{R}$, we
can define the Shapley value $\phi_{i}(v)$ for a specific variable i as your
contribution to the payoff through the weighted average of all possible
combinations.

##  3 Proposed XAI metrics

The proposed metrics are model-agnostic, applicable to all types of AI models
and they do not require knowledge of the model code nor estimates, only the
ability to run predictions with these already working models. Furthermore, our
proposed methods expand upon already existing XAI concepts such as (i) Feature
Importance, (ii) Partial Dependence curve and (iii) Surrogacy Models, as
presented below.

###  3.1 Feature Importance-based metrics

Consider a set of features importance,
$\mathcal{F}=[\mathcal{F}_{1},\mathcal{F}_{2},\ldots,\mathcal{F}_{F}]$, where
$F$ is the number of features in the input model. We start by passing this
vector through a normalizing function $\mathcal{N}$, which normalizes the
feature importance vector by the sum of their absolute values. A normalized
value for feature importance $j$ is defined as

$\displaystyle\begin{split}P(\mathcal{F}):=\\{p_{1},p_{2},...,p_{F}\\},\\\
\textrm{where}\quad p_{j}\geq 0\quad\textrm{and}\quad\sum_{i\in
F}{p_{i}}=1.\end{split}$ (1)

This implies that the vector of normalized features importance
$\mathcal{N}(\mathcal{F})$ constitutes a probability measure
$F=\mathcal{N}(\mathcal{F})=[f_{1},f_{2},...,f_{F}],~{}\textrm{where}~{}f_{j}\geq
0~{}\textrm{and}~{}\sum_{i\in F}{f_{i}}=1$. The parameter $F$ encapsulates the
feature importance at a model level, or ’Global’. When focusing on the ’Local’
aspect—about a specific data point $d$ the parameter is notated to $F_{d}$. To
evaluate the explainability of an AI model, an analysis of the distribution of
the normalized feature weights can be conducted through various metrics, as
proposed below.

####  3.1.1 Metric 1 - Feature Importance Spread ($S$)

Analyzes the feature importance distribution using the concepts of entropy and
divergence. A uniform distribution of feature importance
$U=\\{\bar{f},\bar{f},\ldots\bar{f}\\}$, where $\bar{f}=1/|F|$, indicates that
all the features have the same relevance, suggesting a more challenging
understanding of the model. Based on this, the Kullback-Leibler Divergence,
denoted as $\textrm{S}_{D}$, measures the dissimilarity between a given
feature importance distribution and a uniform distribution. As a distance
metric, the range of $\textrm{S}_{D}$ is between $0$ and $\infty$, where a
lower value indicates a similar distribution of feature importance and,
therefore, a complex model interpretation. A higher divergence value signifies
that the importance distribution diverges significantly from a uniform
distribution, concentrating importance on a few features, which makes it
easier the interpretation of the model. The entropy ratio $\textrm{S}_{R}$
exhibits similar behavior, but its score ranges between 0 and 1. Low values
indicate that few features concentrate high importance, facilitating the
interpretation of the model, while high values suggest a complex
interpretation. Both submetrics are detailed in Table 1.

Table 1: Feature Importance Spread Functions Submetric | Measure | Equation
---|---|---
Spread Ratio | Entropy Ratio | $\textrm{S}_{\textrm{R}}(F)=\frac{\sum_{j=1}^{F}f_{j}\log f_{j}}{\sum_{j=1}^{F}\bar{f}\log\bar{f}}$
Spread Divergence | Kullback-Leibler Divergence | $\textrm{S}_{\textrm{D}}(F)=\sum_{j=1}^{F}{f_{j}\log\left(\frac{f_{j}}{\bar{f}}\right)}$

####  3.1.2 Metric 2 - Feature Importance Stability ($\lambda$)

Analyzes the behaviour of the feature importance distribution in the data
space. For a given dataset $\mathcal{D}$, it is possible to compute a vector
of feature importance for each individual observation $d$. With observation-
individualized measures, we create a feature importance matrix
$\mathcal{P}\in\mathbb{R}^{|\mathcal{D}|\times F}$ where each row represents
the normalized feature importance for each instance in the data space. Similar
to the concept of global feature importance, the distribution associated with
an instance $d$ is represented, using a python notation, by
$F_{d}=\mathcal{P}[d,:]$, i.e. the $d$-th row of our matrix. For the entire
dataset $\mathcal{D}$, the matrix $\mathcal{P}$ is defined as

$\begin{array}[]{c}\mathcal{P}=[P^{1}(\mathcal{F})\quad
P^{2}(\mathcal{F})\quad\cdots\quad P^{N}(\mathcal{F})],\\\ \textrm{where}\quad
P^{d}(\mathcal{F})=[p^{d}_{1}\quad p^{d}_{2}\quad\cdots\quad
p^{d}_{F}].\end{array}$ (2)

In comparison to the previous metric presented in 3.1.1, we now face a
collection of vectors, as opposed to only one vector. However, we still aim to
collapse all these vectors into a single straightforward and intuitive measure
that reflects the complexity regarding how features contribute to the final
outcome, relative to a benchmark model in which all features contribute
equally – i.e. to a uniform feature importance distribution. The importance
values of matrix $\mathcal{P}$ by row or by column represents the importance
stability along the data and feature space respectively. The stability of the
feature importance distribution along data space $\lambda_{\mathcal{D}}$ can
be computed using the equation described as

$\begin{array}[]{c}U_{\mathcal{D}}=\bigcup\limits_{d=1}^{|\mathcal{D}|}\textrm{S}_{\textrm{R}}(\mathcal{P}[d,:])\\\
\lambda_{\mathcal{D}}:=\textrm{S}_{\textrm{R}}(P(U_{\mathcal{D}}))\end{array}$
(3)

where $U_{\mathcal{D}}$ is the list of feature importance spread values for
each sample of $\mathcal{D}$. Following the same steps, the stability of the
importance of one feature in the data space $\lambda_{\mathcal{F}}$ can be
computed using the equation described as

$\begin{array}[]{c}U_{\mathcal{F}}=\bigcup\limits_{f=1}^{|\mathcal{F}|}\textrm{S}_{\textrm{R}}(\mathcal{P}[:,f])\\\
\lambda_{\mathcal{F}}:=\textrm{S}_{\textrm{R}}(P(U_{\mathcal{F}}))\end{array}$
(4)

where $U_{\mathcal{F}}$ denotes a list that comprises the importance spread
values for each feature within $\mathcal{F}$. The entropy ratio
$\textrm{S}_{\textrm{R}}$ was the spread function selected for both metrics.
The metric values are constrained within the range [0,1], with a value closer
to 1 indicating high stability and consequently, facilitating easier
interpretation of the model. Interpreting the stability metrics
$\lambda_{\mathcal{D}}$ and $\lambda_{\mathcal{F}}$ is straightforward: a
higher stability value indicates that the feature importance behaves similarly
throughout the entire data space111The stability metric
$\lambda_{\mathcal{D}}$ quantifies variations in the concentration of
importance throughout the data space. On the other hand,
$\lambda_{\mathcal{F}}$ measures the extend of variation in the feature
importance.

####  3.1.3 Metric 3 - Predictions Groups Contrast (C)

The prediction groups contrast metric compares the top-K ranking of features
importance drawn from the entire dataset with the top-K ranking induced from
specific subgroups of predictions. It can be applied to both categorical and
regression problems, being useful for quantifying how the feature importance
priority changes between subgroups. For categorical problems, $G$ subgroups
can represent output categories, and, for regression problems, $G$ subgroups
can be represented by different portions of the output support (e.g.
quartiles). Given the top-k features importance indexed by $I$, and partial
feature importance $I^{g}$ computed using only observations from group
$g\in\mathcal{D}$, two approaches are proposed in Table 2 to analyze the
similarity between these two rankings:
$I_{k}:=\textrm{argsort}(-F)[1:k]\quad\textrm{and}I_{k}^{g}=\textrm{argsort}(-F^{g}))[1:k]$.

Table 2: Prediction Groups Contrast Functions Submetric | Equation
---|---
Position Parity | $M_{k}(F,g)=\frac{1}{k}\sum_{j\in k}\textrm{m}(I_{j},I_{j}^{g}),\quad\textrm{where}\quad\textrm{m}(a,b)=1,\text{if }a=b,0,\text{otherwise}.$
Rank Alignment | $M_{k}(F,g)=\frac{1}{k}||I_{k}^{g}\cap I_{k}||$

Then, the predictions group contrast is defined as

$\textrm{C}(\mathcal{F})=\frac{1}{GF}\sum_{g}^{G}\sum_{K=1}^{F}{M_{K}(\mathcal{F},g)}$

where $0<\textrm{C}(\mathcal{F})<1$. For Position Parity Score the value
represents the average of the percentage of features in the groups that
maintain exactly their ranking order compared with the global feature
importance order. Rank Alignment Score is more permissive and penalised only
if the feature order is outside the top-k positions.

####  3.1.4 Metric 4 - $\alpha$-Feature Importance ($\alpha_{FI}$)

The $\alpha$ Feature Importance metric quantifies the minimum proportion of
features required to represent $\alpha$ of the total importance. In other
words, this metric is focused in obtaining the minimum number of features
necessary to obtain no less than $\alpha\times 100\%$ of the total explanation
based on the normalized feature importance $f_{j}$, contained in the vector
$P(\mathcal{P})$. Define $f_{j}$ as the proportion of importance of the $j$-th
number of the vector $P$, i.e. the $j$-th order statistic. Formally, our
proposed metric can be computed as follows,

$\alpha_{FI}(\mathcal{F})=\frac{\max_{J}\left\\{F-J|\sum_{j=F}^{J}f_{(j)}<\alpha,J\leq
F\right\\}}{F}$ (5)

where $0\leq\alpha_{FI}<1$. If $\alpha$ is a relatively high number, such as
$4/5$, then a low $\alpha_{FI}$ value indicates that a considerable portion of
the model’s behavior can be explained with a limited number of representative
parameters. Conversely, a higher value indicates that a greater number of
features are needed to describe the model’s behavior. In our illustrations, we
refer to this metric as the Fourth Fifths or $4/5$ metric, pinning the value
of $\alpha$.

More broadly, an XAI user can calculate the minimal number of features for a
given $\alpha$, namely $M^{*}(\alpha):=F-J^{*}(\alpha)$, for various values of
$0\leq\alpha\leq 1$. The resulting increasingly monotonic curve
$M^{*}(\alpha)$ is also informative. It can be used to identify the most
important features to explain outputs up to a certain $\alpha$ degree of
explainability. When this curve plateaus, one can identify a set of features
that are not so relevant/dispensable to explain the AI model outcomes.

###  3.2 Partial Dependence curve-based metric

####  3.2.1 Metric 5 - Explainability Ease (EE)

The Explainability Ease metric uses the concept of the partial dependence
curve detailed in 2.2 to evaluate how simple this curve can be represented.
The partial dependence curve is used to show model predictions are affected on
average by each feature. Curves represented by linear functions are easy to
grasp and interpret, but some features may have a highly non-linear and
complex effects on the model outcomes. Our proposed metric aims to measure how
much the curve changes across the variable domain. In order to calculate the
EE metric, we suggest the following steps. First, we compute the numerical 2nd
derivative of the Partial Dependence, denoting by
$\hat{f}^{x_{s}^{\prime\prime}}(\cdot)$, which mathematically represents how
fast the 1st derivative of the function changes. If the function is a simple
line, the 2nd derivative is equal to zero. On the other hand, if the function
is highly non-linear, the 1st derivative changes often, i.e. the 2nd
derivative departs often from zero in magnitude. Given this intuition, our
proposed metric consists in the normalized norm of the vector containing the
absolute value of the numerical 2nd derivative of the Partial Dependence
curve, denoted by $\vec{v}_{x_{s}}$, computed over $q$ points of the $x_{s}$
domain.

$\begin{array}[]{c}EE_{x_{s}}:=\left|\left|\vec{v}_{x_{s}}\right|\right|,~{}\textrm{where}\\\
\vec{v}_{x_{s}}:=\left\\{|\hat{f}^{x_{s}^{\prime\prime}}(x_{s1})|,|\hat{f}^{x_{s}^{\prime\prime}}(x_{s2})|,\ldots,|\hat{f}^{x_{s^{\prime\prime}}}(x_{sq})|\right\\}.\end{array}$
(6)

The closer $EE_{x_{s}}$ is to zero, or close to zero, the more linearly the
outcome depends on feature $x_{s}$. On the other hand, when the outcome is
large, the dependence of model outcomes to the feature $x_{s}$ is not as
trivially explained as by using a line.

###  3.3 Surrogacy model-based metric

####  3.3.1 Metric 6 - Surrogacy Efficacy Score (E)

The Surrogacy Efficacy Score is a technique for gaining a better understanding
of the inner workings of complex ”black box” models, as discussed in 2.2. For
example, by using a Tree-based model, this method provides a more
interpretable representation of the model’s behavior by partitioning the input
data based on the values of certain fields and creating simple rules to
approximate the model’s predictions. The Decision Tree model is trained to
closely mimic the original model by minimizing the loss between the model’s
predictions and surrogate model predictions. Given a model $f$, a surrogate
model $DT$, the input data $x$, the true output data $y$, the model prediction
$y^{\prime}$ and the surrogate prediction $y^{\prime\prime}$, the surrogate
efficacy score are defined for classification and regression in Table 3.

Table 3: Surrogate Efficacy Score for Problem Type Problem Type | Equation
---|---
Classification | $E_{\textrm{C}}(f,DT,x)=\textrm{Accuracy}(y^{\prime},y^{\prime\prime})$
Regression | $E_{\textrm{R}}(f,DT,x)=\textrm{SMAPE}(y^{\prime},y^{\prime\prime})$

By incorporating a depth limitation of three in the decision tree ($DT$), the
implementation ensures that the surrogate model remains simple enough for ease
of visualization and interpretation while still maintaining adequate
predictive accuracy to reflect the behavior of the original model. This
approach also provides a tangible reference point for assessing the model’s
complexity.

An added benefit of the Surrogacy Efficacy Score is the ability to derive a
list of global feature importances. These importances can then be utilized to
compute the metrics described in section 3.1, providing valuable insights into
the relative importance of different features in the model’s predictions.

##  4 Experiments

In this section, we present the results of the metrics applied to regression
and classification tasks. For the regression task, a dataset was used to
predict the Unified Parkinson’s Disease Rating Scale value for an individual.
In the classification task, a dataset was employed to classify whether an
individual has an income greater or less than fifty thousand dollars annually
based on historical census data from the United States. Dataset descriptions
and results are presented in the next subsections.

Table 4: Regression metrics based on feature importance. Note: GB (Gradient
Boosting), LR (Linear Regression), RF (Random Forest)

Efficacy | LR | RF | GB | REF.
---|---|---|---|---
RMSE | 0.827 | 0.321 | 0.370 | 0
MAE | 0.668 | 0.133 | 0.226 | 0
MAPE | 1.554 | 0.225 | 0.392 | 0
Max Error | 3.357 | 1.789 | 1.515 | 0
SMAPE | 0.543 | 0.115 | 0.190 | 0
XAI-Permutation | LR | RF | GB | REF.
Explainability Ease | 1.000 | 0.632 | 0.868 | 1
Fourth Fifths | 0.053 | 0.158 | 0.053 | 0
Position Parity | 0.760 | 0.457 | 0.438 | 1
Rank Alignment | 0.954 | 0.880 | 0.917 | 1
Spread Divergence | 2.055 | 1.463 | 1.768 | -
Spread Ratio | 0.302 | 0.503 | 0.399 | 0
XAI-Surrogacy | LR | RF | GB | REF.
Explainability Ease | 0.947 | 1.000 | 0.974 | 1
Fourth Fifths | 0.053 | 0.053 | 0.053 | 0
Spread Divergence | 1.790 | 2.004 | 1.988 | -
Spread Ratio | 0.392 | 0.320 | 0.325 | 0
Surrogacy Efficacy | 0.395 | 0.337 | 0.341 | 0
XAI-Lime | LR | RF | GB | REF.
Data Stability | 0.066 | 0.031 | 0.033 | 0
Feature Stability | 0.017 | 0.010 | 0.010 | 0
XAI-Shap | LR | RF | GB | REF.
Data Stability | 0.107 | 0.103 | 0.104 | 0
Feature Stability | 0.014 | 0.015 | 0.035 | 0

Table 5: Classification metrics based on feature importance. Note: GB
(Gradient Boosting), LR (Logistic Regression), RF (Random Forest)

Efficacy | LR | RF | GB | REF.
---|---|---|---|---
Accuracy | 0.858 | 0.858 | 0.875 | 1
Balanced Accuracy | 0.773 | 0.779 | 0.788 | 1
Precision | 0.745 | 0.731 | 0.805 | 1
Recall | 0.611 | 0.630 | 0.622 | 1
F1-Score | 0.671 | 0.677 | 0.702 | 1
XAI-Permutation | LR | RF | GB | REF.
Explainability Ease | 1.000 | 0.969 | 0.974 | 1
Fourth Fifths | 0.173 | 0.173 | 0.051 | 0
Position Parity | 0.047 | 0.052 | 0.183 | 1
Rank Alignment | 0.831 | 0.868 | 0.912 | 1
Spread Divergence | 1.270 | 1.300 | 2.324 | $\infty$
Spread Ratio | 0.723 | 0.716 | 0.493 | 0
XAI-Surrogacy | LR | RF | GB | REF.
Explainability Ease | 1.000 | 1.000 | 1.000 | 1
Fourth Fifths | 0.010 | 0.020 | 0.020 | 0
Spread Divergence | 3.479 | 3.455 | 3.271 | $\infty$
Spread Ratio | 0.241 | 0.246 | 0.286 | 0
Surrogacy Efficacy | 0.931 | 0.903 | 0.947 | 1
XAI-Lime | LR | RF | GB | REF.
Data Stability | 0.013 | 0.014 | 0.018 | 0
Feature Stability | 0.056 | 0.054 | 0.054 | 0
XAI-Shap | LR | RF | GB | REF.
Data Stability | 0.055 | 0.065 | 0.108 | 0
Feature Stability | 0.282 | 0.248 | 0.150 | 0

###  4.1 Parkinson’s monitoring regression

For the regression task, we use the Parkinson’s telemonitoring dataset Tsanas
and Little (2009). This dataset, comprising 20 training features, is designed
to forecast two distinct targets - motor_UPDRS and total_UPDRS. As elucidated
in Tsanas et al. (2010), the progression of Parkinson’s disease is monitored
through the Unified Parkinson’s Disease Rating Scale (UPDRS). We use the
total_UPDRS in our regression experiments. This measure spans a scale of
0-176, where 0 signifies a healthy patient and 176 indicates a patient
profoundly impacted by the disease. Among the 20 features employed for UPDRS
estimation are age, patient ID, gender, time interval from the baseline
recruitment date, and 16 biomedical voice measures collected from patients at
home via a microphone, ensuring high-precision capture of voice signals.

Figure 1: Permutation Feature Importance for LR, RF and GB. Features over red
line represent 80% of the total importance. Fourth Fifths, Spread Divergence
or Spread Ratio metrics that quantify explainability degree based on the
importance concentration and distribution

Figure 2: Permutation feature importance plot by model. The Explainability
Ease annotates Easy, Medium or Hard for each plot based on the region
similarity described in 3.2. Figure 3: Contrast how informative is the output
region for the whole model using Position Parity and Rank Alignment to compare
the feature importance of the quantiles of the output domain and the overall
model.

Table 4 shows the metrics results applied to the four feature importance
strategies. Following we describe the metrics result. The metrics are
presented based on global feature importance such as permutation feature
importance (XAI-permutation) and surrogacy feature importance (XAI-surrogacy),
as well as local feature importance utilizing strategies like Lime (XAI-Lime)
and Shap (XAI-Shap). Metrics employing global feature importance focus on the
distribution of importance across features, the ease of interpreting partial
dependence plots, and contrasting the model’s global information dependency
for each output region. Local feature importances, such as those from Lime,
can be aggregated in a weighted manner to compute the same metrics for global
variables. However, to avoid redundancy in the results, we leverage the
granularity of feature importance to calculate Data Stability and Feature
Stability metrics. These metrics represent the variation in the dispersion of
importance for the distribution of input data and the average variation in the
importance of each feature. Small variations indicate high stability, thus
implying easier and more consistent interpretation across the dataset.
Subsequently, the most salient points of analysis for each group of XAI
metrics are presented.

##### XAI-Permutation

The metrics Fourth Fifths, Spread Divergence, and Spread Ratio, derived from
permutation feature importance, provide insights into the level of
concentration or distribution of the importance value. The principle that
’higher concentration enhances interpretability’ stems from the fact that when
a model relies heavily on a smaller subset of features to make predictions, it
simplifies the decision-making process. Figure 1 illustrates the top-k feature
importance for Logistic Regression (LR), Random Forest (RF), and Gradient
Boosting (GB) models. The RF model distributes importance more uniformly
across its features, as corroborated by lower values of Spread Divergence
(1.463) and higher values of Fourth Fifths (0.158) and Spread Ratio (0.503).
On the other hand, despite both GB and LR models having the same and lower
Fourth Fifths score of 0.053, they assign importance to different features.
The GB model prioritizes Age and DFA as the most significant features, while
the LR model emphasizes jitter:RAP and jitter:DDP. LR, named as an
interpretable model, got the higher degree of explainability in all metrics
and is a good baseline to compare other model such as RF or GB.

Explainability Ease quantifies the ease of interpreting Partial Dependence
Plot curves. Figure 2 displays these curves for features constituting 80% of
importance, as previously mentioned. Logistic Regression exhibits constant
trend lines, making it the most interpretable model. For RF and GB, the curves
demonstrate variable ascending and descending trends, increasing
interpretational difficulty. This metric allows for a classification of
‘Easy´, ‘Medium´, and ‘Difficult´ for each curve, providing a more detailed
understanding of the metric’s value. In a simplified context, a ’Easy’ curve
can be characterized by a consistent trendline, while a ’Medium’ curve implies
a change in trend at some point along its path. On the other hand, a
’Difficult’ curve exhibits multiple changes in direction, occurring two or
more times. Explainability Ease metric enables us to comprehend this
complexity solely by observing the scores.

Position Parity and Rank Alignment assess the similarity between the global
feature importance ranking and the ranking of feature importance for each
output region. In regression analysis, the feature importance of each quartile
is compared with the global feature importance. A high affinity value may
indicate that the features prioritized by a quartile are more informative for
the model. In Figure 3, the first and second rows detail Position Parity and
Rank Alignment values for each quartile, while the third row indicates the
similarity between importance values. The Position Parity and Rank Alignment
for each region are detailed in Appendix LABEL:ap:full_metric_regression. In
this scenario, a LR is attributing equal importance to features across all
regions. Additionally, the global feature importance shows a higher affinity
with the first quartile (Q0-Q1) compared to other quartiles. The quartile
Q2-Q3 is the least aligned with the global scope of feature importance. In
contrast to the previous model, a Gradient Boosting (GB) model exhibits the
opposite behavior. It shows the least affinity with the first quartile (Q0-Q1)
and gives greater importance equally to the other quartiles. This indicates
that GB finds the features in the first quartile (Q0-Q1) less influential or
representative for its decision-making process. Instead, it considers the
features in the remaining quartiles (beyond Q0-Q1) to be more significant or
aligned with its overall predictive patterns.

##### XAI-Surrogacy

The concept of surrogate efficacy is employed to assess the feasibility of
approximating a real model through a decision tree model, specifically with a
constraint of a depth of three as detaild in section 3.3. In this regard,
Table 4 shows that the Decision Tree (DT) Surrogate model derived from the
Random Forest (RF) framework demonstrates the most effective performance.
Nonetheless, additional XAI-surrogacy metrics in Table 4 reveal that the
degree of explainability associated with the surrogate model of RF is higher
when contrasted with surrogate models originating from LR and GB. Details of
that analysis can be seen in the supplementary material. The metrics suggests
that the RF surrogate model not only more accurately reflects the underlying
model performance but also offers superior interpretability relative to its LR
and GB counterparts.

##### XAI-Lime and XAI-Shap

The Data Stability metrics assess the distribution (spread ratio) of feature
importance throughout the dataset. According to our findings, for XAI-Lime,
the data stability sequence is RF (0.031) $<$ GB (0.033) $<$ LR (0.66). This
pattern is evident in Figures 4(a), where the range of spread ratio for LR is
larger than the other models. For XAI-shap, in Figure 4(b), the data stability
scores present a similar level of distribution. Detailed results show the
scores at region level. Similarly, Feature Stability evaluates the average
stability of the importance distribution for each feature. In Figure 4(a),
compared with LR, RF and GB models appear to have regions with a higher
concentration of spread ratio, yet relying solely on visual data may be
insufficient. Feature Stability aids in detecting subtle differences in the
concentration of the spread ratio. According to the XAI-Lime metrics in Table
4(a), the stability XAI-Lime follows the order of RF (0.010) $==$ GB (0.010)
$<$ LR (0.017). Meanwhile, the results for XAI-Shap metrics indicate a
stability order for Shap feature importance as LR (0.014) $<$ RF (0.015) $<$
GB (0.035). We can attribute that because the outliers that can be observed in
4(b).

(a) Data Stability (left side) and Feature Stability (right side) based Lime
feature importance for Linear Regression (LR), Random Forest Regression (RF)
and Gradient Boosting Regression (GB).

(b) Data Stability (left side) and Feature Stability (right side) based Shap
feature importance for Linear Regression (LR), Random Forest Regression (RF)
and Gradient Boosting Regression (GB).

Figure 4: LIME and SHAP results for data stability and feature stability -
classification task

###  4.2 Adult individual income classification

For the classification task, we use the Adult dataset Becker and Kohavi
(1996). This dataset aims to predict whether an individual’s income exceeds
$50K per year based on various census data features. The dataset is composed
of 14 features that include ’age’, ’workclass’, ’fnlwgt’, ’education’,
’education-num’, ’marital-status’, ’occupation’, ’relationship’, ’race’,
’sex’, ’capital-gain’, ’capital-loss’, ’hours-per-week’, and ’native-country’.
In total, the dataset has 48.842 observations for each feature. Several
variables in the dataset may lack immediate clarity. The continuous variable
”fnlwgt” signifies the final weight, denoting the number of units within the
target population that the responding unit represents. Meanwhile,
”education_num” represents the cumulative number of years of education,
offering a continuous representation of the discrete variable ”education.” The
variable ”relationship” denotes the role of the responding unit within the
family structure. Additionally, the variables ”capital_gain” and
”capital_loss” signify income derived from investment sources beyond wages or
salary. The target variable within this dataset is binary, adopting a
numerical coding system wherein it is designated as 0 if the individual’s
income falls below $50K and as 1 if it surpasses this threshold. Specifically,
the model aims to discern patterns and relationships in the input features
that contribute to predicting whether an individual’s income is within the
range of $50K or beyond.

Figure 5: Permutation Feature Importance for LR, RF and GB. Features over the
red line represent 80% of the total importance. Fourth Fifths, Spread
Divergence or Spread Ratio metrics that quantify explainability degree based
on the importance of concentration and distribution

Figure 6: Permutation feature importance plot by model. The Explainability
Ease annotates Easy, Medium or Hard each plot based on the region similarity
described in 3.2. Figure 7: Contrast Position Parity and Rank Alignment
between the feature importance of the quantiles of the output domain and the
overall model.

##### XAI-Permutation

The Gradient Boosting (GB) model demonstrates superior performance in both
efficacy and the majority of XAI-Permutation metrics. Those scores reveals
that GB concentrate crucial information for input classification and enable us
good performance utilizing fewer features in comparison to Logistic Regression
(LR) or Random Forest (RF). This characteristic is depicted in Figure 5 and
quantified using the Fourth Fifths, Spread Divergence, and Ratio metrics. The
Partial Dependence Plots, presented in Figure 6, further elucidate these
findings. For LR, all curves were categorized as ’Easy’. Regarding RF, two
curve was classified as ’Easy’, specifically for the features ’age’ and
’marital-status’, while other curves were classified as ’Medium’. In contrast,
the GB model had two features classified as ’Easy’, with the ’age’ feature
falling into the ’Medium’ category. Typically, curves classified as ’Medium’
demonstrate changing trends, thereby increasing the complexity of their
interpretability. igh values of Position Parity and Rank Alignment in the
Gradient Boosting (GB) model indicate a greater consistency in the
significance of features across different regions of model output. For
instance, as shown in Figure 7, a Position Parity of 24% for class 0 and 13%
for class 1 suggests that the rankings of feature importance shift only
slightly, remaining within a close range of positions. Conversely, low values
imply that the prioritization of features for detecting class 0 and class 1
varies significantly.

##### XAI-Surrogacy

The Surrogacy Efficacy metric indicates that the surrogate model for Gradient
Boosting (GB) achieved the best fit to the actual model, with an accuracy of
94.7%. Esto indica que el comportamiento del model GB se representa mejor que
los otros modelos por medio de reglas simples, facilitando su
interpretabilidad. Furthermore, it was observed that this model achieves the
lowest score in the Explainability Ease metric. This suggests a more uniform
distribution of feature importance in the GB surrogate model compared to the
other models. This observation can be validated by referring to surrogate
feature importance.

##### XAI-Lime and XAI-Shap

Data Stability indicates that the stability order follows the sequence of
Logistic Regression (LR) $>$ Random Forest (RF) $>$ Gradient Boosting (GB) for
both Lime and Shap strategies. This trend of stability can be meticulously
observed in Figures 8(a) and 8(b), where the distribution of the spread ratio
shows an increase in both the upper and lower limits of each box plot,
adhering to the previously mentioned order. In the case of Shap, the Data
Stability value for GB is almost double that of the LR model, which is
corroborated by observing Figure 8(b). Additionally, for Shap, there are many
more outliers compared to the Lime graph. This is quantified in the range of
values observed for Lime and Shap in Table 5. The stability indicated by the
Feature Stability metric presents the order of GB $=$ RF $>$ LR for Lime and
GB $>$ LR $>$ RF for Shap. These values can be interpreted by examining the
Feature Stability graphs (right side) in Figures 8(a) and 8(b).

(a) Data Stability (left side) and Feature Stability (right side) based Lime
feature importance for Logistic Regression (LR), Random Forest (RF) and
Gradient Boosting (GB).

(b) Data Stability (left side) and Feature Stability (right side) based Shap
feature importance for Logistic Regression (LR), Random Forest (RF) and
Gradient Boosting (GB).

Figure 8: LIME and SHAP results for data stability and feature stability -
regression task

###  4.3 Discussion of Results

The metrics demonstrated sensitivity to the robustness of each feature
importance extraction strategy. While these metrics effectively quantify a
model’s interpretability based on various criteria, they prove to be
particularly valuable when comparing different models. Quantifying the degree
of explainability facilitates a more nuanced understanding of criteria, such
as the relationship between model simplicity and improved explainability. In
our classification problem case study, we observed that Gradient Boosting
achieved better interpretability scores than Logistic Regression (a simpler
model) across several metrics based on permutation feature importance.
However, a more granular exploration using local feature importance revealed
that Logistic Regression exhibited better Data Stability, while Gradient
Boosting showed superior Feature Stability.

Although the metrics function effectively for problems with tabular data, we
believe that with minor modifications, these strategies could be extended to
deep learning models handling data types such as images and text.

##  5 Conclusion

The necessity of transparency in AI systems and the trade-off between accuracy
and explainability in AI models both fuel the recent discussion about the
interpretation of predictions generate with machine learning models. In this
paper, propose a new set of explaibility metrics concerning global and local
feature importance. Having metrics that summarize the ability of a model to be
easily explained, can aid the understanding of model complexity, compared to
e.g. the alternative of analyzing a large set of graphical observations for
feature importance. It should be taken into account that the metrics are
relative to the nature of which the importance of characteristics was
extracted.

We observe that most strategies focused on explainability leave room for
ambiguity in terms of rating or quantifying the model’s understanding and
comparability. Furthermore, many strategies are only applicable to a specific
architecture. A quantification of interpretability enables a practical
understanding of what is happening inside complex AI models. To mitigate this
gap, we propose six new explainability metrics based on three strategies -
feature importance, partial dependence, and surrogacy. The results found show
that the use of explainability metrics can improve and empower AI
stakeholders. The metrics enable an automated comparison of interpretability
across various criteria through this quantification. These metrics pave the
way for the development of new strategies in machine learning model creation,
for instance, incorporating explainability metrics during the training.

We create a reference table to interpret quickly the most important
explainability metrics. Between our proposed metrics, feature concentration,
feature priority shared between prediction groups, complexity represented by
simple rules, and local feature importance stability were revealed to be the
most important in understanding our models. The metrics enable an automated
comparison of interpretability across various criteria through this
quantification. These metrics pave the way for the development of new
strategies in machine learning model creation, for instance, incorporating
explainability metrics during the training.

As for future research, we plan on exploring the statistical properties of the
metrics proposed, potentially creating inference tools for model selection and
model auditing based on these tools. While the scope of the article focused on
constructing model-agnostic metrics to evaluate model predictions, future
studies may explore the development of metrics for methods focused on inner
interpretability.

## References

  * Alabi et al. [2023] R. O. Alabi, M. Elmusrati, I. Leivo, A. Almangush, and A. A. Mäkitie.  Machine learning explainability in nasopharyngeal cancer survival using lime and shap.  _Scientific Reports_ , 13(1):8984, 2023.
  * Ali et al. [2023] S. Ali, T. Abuhmed, S. El-Sappagh, K. Muhammad, J. M. Alonso-Moral, R. Confalonieri, R. Guidotti, J. D. Ser, N. Díaz-Rodríguez, and F. Herrera.  Explainable artificial intelligence (xai): What we know and what is left to attain trustworthy artificial intelligence.  _Information Fusion_ , 99:101805, 2023.  ISSN 1566-2535.  doi:10.1016/j.inffus.2023.101805.  URL https://www.sciencedirect.com/science/article/pii/S1566253523001148.
  * Becker and Kohavi [1996] B. Becker and R. Kohavi.  Adult.  UCI Machine Learning Repository, 1996.  DOI: https://doi.org/10.24432/C5XW20.
  * Buckmann et al. [2022] M. Buckmann, A. Joseph, and H. Robertson.  An interpretable machine learning workflow with an application to economic forecasting.  Technical report, Bank of England, 2022.  URL https://www.bankofengland.co.uk/working-paper/2022/an-interpretable-machine-learning-workflow-with-an-application-to-economic-forecasting.
  * Elshawi et al. [2019] R. Elshawi, M. H. Al-Mallah, and S. Sakr.  On the interpretability of machine learning-based model for predicting hypertension.  _BMC Medical Informatics and Decision Making_ , 19(1):146, Jul 2019.  ISSN 1472-6947.  doi:10.1186/s12911-019-0874-0.  URL https://doi.org/10.1186/s12911-019-0874-0.
  * ElShawi et al. [2020] R. ElShawi, Y. Sherif, M. Al-Mallah, and S. Sakr.  Interpretability in healthcare: A comparative study of local machine learning interpretability techniques.  _Computational Intelligence_ , 2020.  doi:10.1111/coin.12410.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/coin.12410.
  * Fel and Vigouroux [2020] T. Fel and D. Vigouroux.  Representativity and consistency measures for deep neural network explanations.  2020\.
  * Fel et al. [2022] T. Fel, D. Vigouroux, R. Cadène, and T. Serre.  How good is your explanation? algorithmic stability measures to assess the quality of explanations for deep neural networks.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 720–730, 2022.
  * Feng et al. [2021] D.-C. Feng, W.-J. Wang, S. Mangalathu, and E. Taciroglu.  Interpretable xgboost-shap machine-learning model for shear strength prediction of squat rc walls.  _Journal of Structural Engineering_ , 147(11):04021173, 2021.
  * Fisher et al. [2019] A. Fisher, C. Rudin, and F. Dominici.  All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously.  2019\.
  * Gabbay et al. [2021] F. Gabbay, S. Bar-Lev, O. Montano, and N. Hadad.  A lime-based explainable machine learning model for predicting the severity level of covid-19 diagnosed patients.  _Applied Sciences_ , 11(21):10417, 2021.
  * Hoffman et al. [2018] R. R. Hoffman, S. T. Mueller, G. Klein, and J. Litman.  Metrics for explainable ai: Challenges and prospects.  2018\.
  * Khaire and Dhanalakshmi [2022] U. M. Khaire and R. Dhanalakshmi.  Stability of feature selection algorithm: A review.  _Journal of King Saud University-Computer and Information Sciences_ , 34(4):1060–1073, 2022.
  * Lakkaraju et al. [2017] H. Lakkaraju, E. Kamar, R. Caruana, and J. Leskovec.  Interpretable and explorable approximations of black box models.  2017\.
  * Li [2022] Z. Li.  Extracting spatial effects from machine learning model using local interpretation method: An example of shap and xgboost.  _Computers, Environment and Urban Systems_ , 96:101845, 2022\.
  * Lundberg and Lee [2017] S. Lundberg and S.-I. Lee.  A unified approach to interpreting model predictions.  2017\.  URL https://arxiv.org/abs/1705.07874.
  * Magesh et al. [2020] P. R. Magesh, R. D. Myloth, and R. J. Tom.  An explainable machine learning model for early detection of parkinson’s disease using lime on datscan imagery.  _Computers in Biology and Medicine_ , 126:104041, 2020.
  * Man and Chan [2021] X. Man and E. P. Chan.  The best way to select features? comparing mda, lime, and shap.  _The Journal of Financial Data Science_ , 3(1):127–139, 2021.
  * Meng et al. [2020] Y. Meng, N. Yang, Z. Qian, and G. Zhang.  What makes an online review more helpful: an interpretation framework using xgboost and shap values.  _Journal of Theoretical and Applied Electronic Commerce Research_ , 16(3):466–490, 2020.
  * Mothilal et al. [2021] R. K. Mothilal, D. Mahajan, C. Tan, and A. Sharma.  Towards unifying feature attribution and counterfactual explanations: Different means to the same end.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 652–663, 2021.  ISBN 9781450384735.  doi:10.1145/3461702.3462597.  URL https://doi.org/10.1145/3461702.3462597.
  * Narayanan et al. [2018] M. Narayanan et al.  How do humans understand explanations from machine learning systems? an evaluation of the human-interpretability of explanation.  2018\.
  * Nguyen and Martínez [2020] A.-p. Nguyen and M. R. Martínez.  On quantitative aspects of model interpretability.  2020\.
  * Nogueira et al. [2018] S. Nogueira, K. Sechidis, and G. Brown.  On the stability of feature selection algorithms.  _Journal of Machine Learning Research_ , 18(174):1–54, 2018.
  * Nohara et al. [2022] Y. Nohara, K. Matsumoto, H. Soejima, and N. Nakashima.  Explanation of machine learning models using shapley additive explanation and application for real data in hospital.  _Computer Methods and Programs in Biomedicine_ , 214:106584, 2022.
  * Parsa et al. [2020] A. B. Parsa, A. Movahedi, H. Taghipour, S. Derrible, and A. K. Mohammadian.  Toward safer highways, application of xgboost and shap for real-time accident detection and feature analysis.  _Accident Analysis & Prevention_, 136:105405, 2020.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin.  ’why should i trust you?’: Explaining the predictions of any classifier.  2016\.  URL https://arxiv.org/abs/1602.04938.
  * Rodríguez-Pérez and Bajorath [2020] R. Rodríguez-Pérez and J. Bajorath.  Interpretation of machine learning models using shapley values: application to compound potency and multi-target activity predictions.  _Journal of computer-aided molecular design_ , 34:1013–1026, 2020.
  * Rong et al. [2022] Y. Rong et al.  A consistent and efficient evaluation strategy for attribution methods.  2022\.
  * Schmidt and Biessmann [2019] P. Schmidt and F. Biessmann.  Quantifying interpretability and trust in machine learning systems.  2019\.
  * Shapley et al. [1953] L. S. Shapley et al.  A value for n-person games.  1953\.
  * Thimoteo et al. [2022] L. M. Thimoteo, M. M. Vellasco, J. Amaral, K. Figueiredo, C. L. Yokoyama, and E. Marques.  Explainable artificial intelligence for covid-19 diagnosis through blood test variables.  _Journal of Control, Automation and Electrical Systems_ , 33(2):625–644, 2022.  URL https://link.springer.com/article/10.1007/s40313-021-00858-y.
  * Tsanas and Little [2009] A. Tsanas and M. Little.  Parkinsons Telemonitoring.  UCI Machine Learning Repository, 2009.  DOI: https://doi.org/10.24432/C5ZS3N.
  * Tsanas et al. [2010] A. Tsanas, M. A. Little, P. E. McSharry, and L. O. Ramig.  Accurate telemonitoring of parkinson’s disease progression by noninvasive speech tests.  _IEEE Transactions on Biomedical Engineering_ , 57:884–893, 2010.  URL https://ieeexplore.ieee.org/document/5339170.
  * Zhou et al. [2021] J. Zhou, A. H. Gandomi, F. Chen, and A. Holzinger.  Evaluating the quality of machine learning explanations: A survey on methods and metrics.  _Electronics_ , 10(5), 2021.  ISSN 2079-9292.  URL https://www.mdpi.com/2079-9292/10/5/593.