# Rational Shapley Values

David S. Watson  0000-0001-9632-2159 Department of Statistical Science,
University College LondonLondonUK<EMAIL_ADDRESS>
(2022)

###### Abstract.

Explaining the predictions of opaque machine learning algorithms is an
important and challenging task, especially as complex models are increasingly
used to assist in high-stakes decisions such as those arising in healthcare
and finance. Most popular tools for post-hoc explainable artificial
intelligence (XAI) are either insensitive to context (e.g., feature
attributions) or difficult to summarize (e.g., counterfactuals). In this
paper, I introduce _rational Shapley values_ , a novel XAI method that
synthesizes and extends these seemingly incompatible approaches in a rigorous,
flexible manner. I leverage tools from decision theory and causal modeling to
formalize and implement a pragmatic approach that resolves a number of known
challenges in XAI. By pairing the distribution of random variables with the
appropriate reference class for a given explanation task, I illustrate through
theory and experiments how user goals and knowledge can inform and constrain
the solution set in an iterative fashion. The method compares favorably to
state of the art XAI tools in a range of quantitative and qualitative
comparisons.

Explainable artificial intelligence, Interpretable machine learning, Shapley
values, Counterfactuals, Decision theory

††journalyear: 2022††copyright: acmlicensed††conference: 2022 ACM Conference
on Fairness, Accountability, and Transparency; June 21–24, 2022; Seoul,
Republic of Korea††booktitle: 2022 ACM Conference on Fairness, Accountability,
and Transparency (FAccT ’22), June 21–24, 2022, Seoul, Republic of
Korea††price: 15.00††doi: 10.1145/3531146.3533170††isbn:
978-1-4503-9352-2/22/06††ccs: Theory of computation Machine learning
theory††ccs: Mathematics of computing Probability and statistics

##  1\. Introduction

Machine learning algorithms are increasingly used in a variety of high-stakes
domains, from credit scoring to medical diagnosis. However, many of the most
popular and powerful models are _opaque_ , insomuch as end users cannot
understand the reasoning that goes into individual predictions. A
subdiscipline of computational statistics has rapidly emerged to address this
issue, variously referred to as _interpretable machine learning_ or
_explainable artificial intelligence_ (XAI). For good introductory surveys,
see, e.g., (Adadi and Berrada, 2018; Guidotti et al., 2018; Murdoch et al.,
2019).

The most prominent methods in this burgeoning area are post-hoc, model-
agnostic tools such as feature attributions and counterfactuals. Both
approaches have recently come under fire for failing to properly handle
dependencies between covariates. Most algorithms for computing Shapley values
– a popular feature attribution technique, formally defined in Sect. 2 –
implicitly treat predictors as mutually independent, assigning positive weight
to potentially impossible data permutations (Frye et al., 2020; Kumar et al.,
2020). Much recent work in counterfactual explanation has focused on how to
model the impact of interventions on downstream variables, a reformulation
that requires an explicitly causal approach (Mahajan et al., 2019; Karimi et
al., 2021).

Others have pushed back against these critiques. Sundararajan and Najmi (2019)
demonstrate that conditioning on covariates can misleadingly assign importance
to irrelevant features. Janzing et al. (2020) point out that standard
supervised learning algorithms do not explicitly model dependencies between
features, and so intervening to set predictors to some fixed value is not just
permissible but proper when computing Shapley values.

Often lost in these debates is the seemingly obvious realization that the
“right” explanation depends crucially on who is asking and why. Current
explainability methods tend to treat explanations as objective deliverables
that can be computed from just two ingredients: a target function and an
accompanying dataset of real or simulated samples. This ignores a basic fact
long acknowledged in philosophy and social science – that _explanations are
fundamentally pragmatic_. To succeed, they must be tailored to the inquiring
agent, who requests an explanation on the basis of certain beliefs and
interests.

If pragmatic approaches are few and far between in the XAI literature, perhaps
this is because it is not obvious how we should formalize abstract, subjective
notions like beliefs and interests. In this paper, I use tools from Bayesian
decision theory to do just that. I make three main contributions: (1) I
combine disparate work in feature attributions and counterfactuals,
demonstrating how the two can join forces to create more useful and flexible
explanations. (2) I extend the axiomatic guarantees of Shapley values,
allowing user beliefs and preferences to guide the search for a relevant
subspace of contrastive baselines. (3) I implement an expected utility
maximization procedure that optionally incorporates causal information to
compute optimal attribution sets for individual agents.

The remainder of this paper is structured as follows. In Sect. 2, I review
important background material in XAI, with a focus on Shapley values and
counterfactuals, as well as the essential formalisms of decision theory and
structural causal models. In Sect. 3, I compare several different reference
distributions, propose a new desideratum for Shapley values, and introduce an
algorithm for computing them. In Sect. 4, I put this method to work on a
number of benchmark datasets. Following a discussion in Sect. 5, Sect. 6
concludes.

##  2\. Background

In what follows, I use uppercase italics to represent variables, e.g. $X$;
lowercase italics to represent their values, e.g. $x$; uppercase boldface to
represent matrices, e.g. $\bm{X}$; lowercase boldface to represent vectors,
e.g. $\bm{x}$; and calligraphic type to represent the support of a
distribution, e.g. $\mathcal{X}$.

###  2.1. XAI Methods

Originally developed in the context of cooperative game theory (Shapley,
1953), Shapley values have been adapted for model interpretability by numerous
authors. They represent the contribution of each feature toward a particular
prediction. Let $\bm{x}_{i}\in\mathcal{X}\subseteq\mathbb{R}^{d}$ denote an
input datapoint and
$f(\bm{x}_{i})=\hat{y}_{i}\in\mathcal{Y}\subseteq\mathbb{R}$ the corresponding
output of our target function $f$ for each sample
$i\in[n]=\\{1,\dots,n\\}$.111The model $f$ may be a classifier, although in
this case we typically require class probabilities and work on the logit
scale. For ease of exposition, in this section we will assume $f$ is a
regressor. Shapley values express this value as a sum:

(1) $f(\bm{x}_{i})=\sum_{j=0}^{d}\phi_{j}(i),$

where $\phi_{0}(i)$ represents $i$’s “baseline” (i.e., the default output of
$f$; more on this shortly) and $\phi_{j}(i)$ the weight assigned to feature
$X_{j}$ at point $\bm{x}_{i}$. Feature attribution methods that satisfy Eq. 1
are said to be _efficient_. Let $v:[n]\times 2^{d}\rightarrow\mathbb{R}$ be a
value function such that $v(i,S)$ is the payoff associated with feature subset
$S\subseteq[d]$ and $v(i,\\{\emptyset\\})=0$ for all $i\in[n]$. The Shapley
value $\phi_{j}(i)$ is given by a weighted average of $j$’s marginal
contributions to all subsets that exclude it:

(2)
$\phi_{j}(i)=\frac{1}{d!}\sum_{S\subseteq[d]\backslash\\{j\\}}|S|!(d-|S|-1)!~{}v(i,S\cup\\{j\\})-v(i,S).$

Several properties of this value are worth noting.

  * •

_Linearity_. For all $j\in[d]$, all $a,b\in\mathbb{R}$, and value functions
$v_{1},v_{2}$, we have:

$\displaystyle\phi_{j}(av_{1}+bv_{2})=a\phi_{j}(v_{1})+b\phi_{j}(v_{2}),$

where $\phi_{j}(v)$ makes explicit the dependence of attributions on value
functions. (Indeed, $\phi_{j}$ only depends on $i$ through $v$.)

  * •

_Sensitivity_. If $v(i,S\cup\\{j\\})=v(i,S)$ for all
$S\subseteq[d]\backslash\\{j\\}$, then $\phi_{j}(i)=0$.

  * •

_Symmetry_. If $v(i,S\cup\\{j\\})=v(i,S\cup\\{k\\})$ for all
$S\subseteq[d]\backslash\\{j,k\\}$, then $\phi_{j}(i)=\phi_{k}(i)$.

It can be shown that Shapley values are the unique solution to the attribution
problem that satisfies efficiency, linearity, sensitivity, and symmetry
(Shapley, 1953). Computing exact Shapley values is NP-hard, although numerous
efficient approximations have been proposed (Sundararajan and Najmi, 2019).

There is some ambiguity as to how one should calculate payoffs on a proper
subset of features, since $f$ requires $d$-dimensional input. Let
$R=[d]\backslash S$, such that we can rewrite any $\bm{x}_{i}$ as a pair of
subvectors $(\bm{x}_{i}^{S},\bm{x}_{i}^{R})$. Then the payoff for feature
subset $S$ takes the form of a conditional expectation, with $\bm{x}_{i}^{S}$
held fixed while $\bm{x}^{R}$ varies. Following Merrick and Taly (2020), I
consider a general form for the value function, parametrized by a distribution
$\mathcal{D}_{R}$:

(3)
$v_{\mathcal{D}_{R}}(i,S)=\mathop{\mathbb{E}}_{\bm{x}^{R}\sim\mathcal{D}_{R}}[f(\bm{x}_{i}^{S},\bm{x}^{R})].$

Popular options for $\mathcal{D}_{R}$ include the marginal $P(\bm{X}^{R})$, as
in Lundberg and Lee (2017); the conditional $P(\bm{X}^{R}|\bm{x}_{i}^{S})$, as
in Aas et al. (2021); and the interventional
$P(\bm{X}^{R}|do(\bm{x}_{i}^{S}))$, as in Heskes et al. (2020). I revisit the
distinction between these distributions in Sect. 3.1.

Counterfactual explanations, unlike Shapley values, do not produce feature
weights. Instead, they identify nearby datapoints with alternative outcomes –
the eponymous _counterfactuals_ – intended to explain predictions by
highlighting some (preferably small) set of input perturbations. The
optimization problem can be expressed as

(4)
$\bm{x}^{*}=\operatorname*{argmin}_{\tilde{\bm{x}}\in\text{CF}(\bm{x}_{i})}~{}d(\bm{x}_{i},\tilde{\bm{x}}),$

where $\text{CF}(\bm{x}_{i})$ denotes a counterfactual space, where
$f(\bm{x}_{i})\neq f(\tilde{\bm{x}})$ (for classification) or
$f(\bm{x}_{i})=f(\tilde{\bm{x}})+\delta$ (for regression), where $\delta$ is a
hyperparameter and $d$ is some distance measure (unspecified for now). The
method was originally introduced by Wachter et al. (2018), who use generative
adversarial networks (GANs) to solve Eq. 4. Others have proposed a variety of
alternatives designed to ensure that counterfactuals are coherent and/or
actionable (Ustun et al., 2019; Poyiadzi et al., 2020; Russell, 2019; Karimi
et al., 2020).

###  2.2. Bayesian Decision Theory

Why do we use XAI tools in the first place? Watson and Floridi (2021) list
three reasons: (1) to audit for potential bias; (2) to validate performance,
guarding against unexpected errors; and (3) to discover underlying mechanisms
of the data generating process. Another use case, with elements of (1) and (3)
yet distinctly analyzed in the algorithmic recourse setting, is (4) to
recommend actions so as to alter predicted outcomes. In all four cases, the
aim is ultimately to make some sort of decision – be it about whether to sue a
firm, deploy an algorithm, or perform an experiment. These motivations may
overlap at the edges but they represent distinct tasks based on different
assumptions and requiring their own explanatory methodologies. This
heterogeneity is largely ignored by current feature attribution and
counterfactual approaches, which implicitly assume a sort of _explanatory
objectivism_. According to this view, the quality of an explanation is
independent of its context.

Decades of research in philosophy and social science has made such objectivism
untenable (Miller, 2019; Floridi, 2019; Legg and Hookway, 2020). If the view
has any adherents today, perhaps it is because the alternative is often
misconstrued as _relativism_ , an anything-goes slippery slope that inevitably
leads to what Feyerabend (1975) (approvingly) refers to as “epistemological
anarchism”. This, however, is a false dichotomy. I reject the polar extremes
of objectivism and relativism in favor of _pragmatism_ , which holds neither
that there exists some single ideal explanation nor that all explanations are
equally valid, but rather that various explanations may be more or less
appropriate depending on who requests them and why.

To operationalize this insight, I rely on the formal apparatus of decision
theory. Let $A$ and $H$ denote finite sets of actions and outcomes,
respectively. An agent’s preferences over action-outcome pairs induce a
partial ordering that we represent with a utility function $u:A\times
H\rightarrow\mathbb{R}$. Let $p(\cdot)$ denote a credence function over
outcomes such that $p(H)=1$ and $p(h|E)$ denotes the agent’s subjective degree
of belief in some $h\in H$ given a body of evidence
$E=\\{e_{1},\dots,e_{n}\\}$. The expected utility of each action $a\in A$ is
given by a weighted average over hypotheses:

(5) $\mathop{\mathbb{E}}_{h\sim p}[u(a,h)|E]=\sum_{j}p(h_{j}|E)~{}u(a,h_{j}).$

If $u$ satisfies the utility axioms (i.e., completeness, transitivity,
continuity, and independence) and $p$ satisfies the probability axioms (i.e.,
non-negativity, unit measure, and $\sigma$-additivity), then we say that the
agent is _rational_ , and it can be shown that she will tend to maximize
expected utility (von Neumann and Morgenstern, 1944). That is, she will always
choose some optimal action:

(6) $a^{*}=\operatorname*{argmax}_{a\in A}~{}\mathop{\mathbb{E}}_{h\sim
p}[u(a,h)|E]$

from a set of alternatives.

This expected utility framework is very general, and applies to explanation
instances of all four types cited above. For instance, in an auditing example
where a data subject believes her loan application was denied due to some
sensitive attribute(s), we might have $A=\\{\texttt{sue},\neg\texttt{sue}\\}$
and $H=\\{\texttt{biased},\neg\texttt{biased}\\}$, with evidence provided by
some XAI tool and fairness criteria (Barocas et al., 2019). In algorithmic
recourse, we typically consider more diverse action sets. For example, a
credit scoring case may include
$A=\\{\texttt{education},\texttt{income},\texttt{credit}\\}$, where increasing
any subset of those variables incurs some cost but may improve outcomes
$H=\\{\texttt{approve},\neg\texttt{approve}\\}$.

A key consideration, often ignored by authors in XAI, is the difference
between model- and system-level explanations (Watson, 2022). At the model
level, for instance, it is typically appropriate to treat all features as
mutually independent and observe outputs on real and/or synthetic inputs that
vary in some systematic fashion. At the system level, by contrast, we cannot
afford to ignore causal relationships between features, as an intervention on
one variable may have downstream effects on others. In these cases, nature
constrains the input space in accordance with certain structural dependencies.

This distinction is formalized using tools from the causal modeling literature
(Pearl, 2000). A structural causal model (SCM) is a tuple
$\mathcal{M}=\langle\bm{U},\bm{V},\mathcal{F}\rangle$, where $\bm{U}$ is a set
of exogenous variables, i.e. background conditions; $\bm{V}$ is a set of
endogenous variables, i.e. observed states; and $\mathcal{F}$ is a set of
structural equations, one for each $V_{j}\in\bm{V}$. These functions map
causes to direct effects in a deterministic fashion,
$f_{j}:pa(V_{j})\rightarrow V_{j}$, where $pa(\cdot)$ denotes the “parents” of
some endogenous variable(s). The union of $V_{j}$’s direct and indirect causes
(parents, grandparents, etc.) is called its “ancestors”, denoted $an(V_{j})$.
Pearl’s $do$-calculus (2000, Ch. 3.4) provides a provably complete set of
rules for identifying causal effects from observational data where possible
(Shpitser and Pearl, 2008).

A deterministic SCM can be expanded to accommodate stochastic relationships by
placing a joint probability distribution on background conditions,
$p(\bm{U})$. By equating this with the credence function introduced above, we
may encode an agent’s uncertainty with respect to a causal system, which has
implications for expected utility if $H\in\bm{V}$. This decision theoretic
approach to causality is explicitly endorsed by Dawid (2012, 2015, 2020), who
argues that treatment policies should be designed not by considering
differences between potential outcomes, but instead by minimizing expected
loss as computed over a set of observed data. A similar premise underlies
recent work in causal reinforcement learning (Lee and Bareinboim, 2018, 2019),
where the goal is to find optimal interventions by maximizing expected rewards
over trials.

##  3\. Rational Shapley Values

My basic strategy is to allow user inputs to inform the generation of
counterfactuals and estimation of Shapley values, thereby resulting in custom
feature attributions that are more useful than either method on its own.
Intuitively, this gives agents the tools to pose not just generic (i.e.,
objectivist) questions of the form “Why did model $f$ predict outcome
$\hat{y}_{i}$ for input $\bm{x}_{i}$?”, but more targeted (i.e., pragmatic)
questions, such as: (1) Why did $f$ predict $\hat{y}_{i}$ as opposed to
$\tilde{y}$ for $\bm{x}_{i}$? (2) Why did $f$ predict $\hat{y}_{i}$ as opposed
to $\tilde{y}$ for $\bm{x}_{i}$, given $\bm{x}^{S}_{i}$? (3) Why did $f$
predict $\hat{y}_{i}$ as opposed to $\tilde{y}$ for $\bm{x}_{i}$, given
$\bm{x}^{S}_{i}$ and certain constraints on $\bm{X}^{R}$? Unlike
counterfactuals, which may confuse users by generating a large number of
synthetic datapoints with no clear takeaway message, the proposed method
provides unambiguous weights for all variables of interest. Unlike feature
attributions, which ignore pragmatic information by construction, the present
approach is _rational_ in the decision theoretic sense. I will make these
claims more precise and apparent below. First, however, I introduce the notion
of a _relevant subspace_ to formalize the hierarchy of specification implied
by the questions above.

######  Definition 1 (Relevant subspace).

Let $\mathcal{Y}_{0},\mathcal{Y}_{1}$ be a partition of $\mathcal{Y}$ into
baseline and contrastive outcomes, respectively, with
$\hat{y}_{i}\in\mathcal{Y}_{0}$. Let $S\subseteq[d]$ denote a (potentially
empty) conditioning set with complement $R=[d]\backslash S$. Let
$\mathcal{D}_{S}$ be the distribution of possible values for the conditioning
set. Let $\mathcal{D}_{R}$ be a distribution encoding probable variation in
$\bm{X}^{R}$ as a function of $\bm{x}^{S}$. Define
$\mathcal{Z}=\mathcal{X}\times\mathcal{Y}$. We say that counterfactual sample
$\tilde{\bm{z}}=(\tilde{\bm{x}}^{S},\tilde{\bm{x}}^{R},\tilde{y}=f(\tilde{\bm{x}}^{S},\tilde{\bm{x}}^{R}))$
is _relevant_ for the inquiring agent if and only if: (i)
$\tilde{y}\in\mathcal{Y}_{1}$; (ii) $\tilde{\bm{x}}^{S}\sim\mathcal{D}_{S}$;
and (iii) $\tilde{\bm{x}}^{R}\sim\mathcal{D}_{R}$. Any space
$\tilde{\mathcal{Z}}\subseteq\mathcal{Z}$ that meets these criteria is a
_relevant subspace_.

The issue with popular Shapley value approximators (e.g., SHAP (Lundberg and
Lee, 2017)) is that baselines are fixed by the data centroid. Attributions are
only designed to account for the discrepancy between $f(\bm{x}_{i})$ and the
empirical mean $\phi_{0}(i)=n^{-1}\sum_{i=1}^{n}f(\bm{x}_{i})$. Moreover, it
is common to use the marginal $P(\bm{X}^{R})$ as the reference distribution
$\mathcal{D}_{R}$, thereby breaking associations between the features in $S$
and $R$, resulting in a value function that implicitly compares each $x_{ij}$
to the mean of the corresponding variable $X_{j}$. Yet these values might not
fall within the relevant subspace for a given agent, who may, for instance,
want to know why she received an average credit score instead of a better-
than-average one. Or perhaps she wants to compare her predictions only to
those of others in a similar age range and income bracket, effectively setting
attributions for those features to zero.

Counterfactuals naturally handle such cases, identifying real or synthetic
datapoints that differ from the explanans in specific ways. However, there are
several issues with the counterfactual approach. First, the cost function in
Eq. 4 is almost always equated with some distance metric in practice, despite
much discussion of more generic alternatives (Karimi et al., 2020; Ustun et
al., 2019; Watson et al., 2022). This is an unsatisfying notion of costs,
which can and should reflect the preferences of inquiring agents much more
flexibly. For instance, relevant action sets may range over interventions
exogenous to the feature space, where the notion of distance is often
inapplicable. Second, there is the question of how many counterfactuals to
provide. Eq. 4 suggests that just a single sample would suffice; however, such
a procedure overestimates the certainty with which such a datapoint is
generated. Can we really be sure that counterfactual $\bm{x}^{*}$ is to be
preferred to counterfactual $\bm{x}^{\prime}$ just because the former is
marginally closer in feature space to the input $\bm{x}_{i}$? Even if we are
willing to accept a purely distance-based notion of cost, it is not at all
obvious that very small distances should be taken seriously when there are so
many potential sources of noise – in the measurements themselves, in the
finite training data, and in whatever approximations were necessary to
efficiently generate candidate counterfactuals. That is why some authors
prefer to sample a large, diverse coalition of counterfactuals (Mothilal et
al., 2020), so that users can survey the various paths through which a
prediction may change. Yet this approach raises new problems of
intelligibility. How should an agent make sense of a relevant subspace with
dozens or hundreds of samples within a tolerable distance of $\bm{x}_{i}$? How
should she prioritize among different counterfactuals or integrate this
information into a high-level summary?

Feature attribution methods are well suited to this task. Shapley values
provide an efficient and intuitive account of the information encoded in a
large set of counterfactuals, regardless of sample size. I therefore propose a
hybrid method in which data are sampled from a relevant subspace, and Shapley
values computed only with respect to such samples. The crucial observation is
that nothing in the definition of Shapley values (Eqs. 1, 2, 3) precludes us
from shifting the reference distribution from an entire observational dataset
to a particular region with certain desirable properties – i.e., a relevant
subspace.

###  3.1. Selecting the Reference Distribution

I take it more or less for granted that agents requesting explanations for
algorithmic predictions have some contrastive outcome $\mathcal{Y}_{1}$ in
mind. They are also likely to have some intuition about the conditioning set
$S$, which will presumably include immutable features and/or any variables
that the agent does not want to enter into the explanation if it can be
avoided. The precise distribution for these features may not be available _a
priori_ , but conditioning on the observed $\bm{x}_{i}^{S}$ is probably a good
start when the data allow for it. The trickiest component to evaluate then is
the reference distribution $\mathcal{D}_{R}$, which has implications both for
how counterfactuals are computed (since this helps define the relevant
subspace) and for how Shapley values are estimated (since this helps define
the value function).

I consider three possibilities: (1) marginal: $P(\bm{X}^{R})$; (2)
conditional: $P(\bm{X}^{R}|\bm{x}^{S}_{i})$; and (3) interventional:
$P(\bm{X}^{R}|do(\bm{x}^{S}_{i}))$. The first point to observe is that the
three distributions are not equivalent. (1) comes apart from the other two
whenever statistical dependencies are present, i.e.
$\bm{X}^{R}\operatorname{\not\\!\perp\\!\\!\\!\perp}\bm{X}^{S}$. Options (2)
and (3) come apart whenever causal dependencies are present, i.e. if
$an(\bm{X}^{R})\cap\bm{X}^{S}\neq\\{\emptyset\\}$. I want to be clear that the
differences between these distributions are in no sense normative. None is
generally “better” than the others, or more useful, or more broadly
applicable. Rather, each is the right answer to a different question. What
follows is a brief discussion of the advantages and disadvantages associated
with each value of $\mathcal{D}_{R}$, as well some heuristic guidelines on
when to use which.

_Marginal_. The marginal distribution is preferable when the goal is a model-
level explanation. This may be the case in certain instances of auditing or
validation, where the analyst seeks merely to discover what the model has
learned without any further restrictions. In this case, we may not care too
much about impossible data perturbations (e.g., teenage grandmothers) since we
simply want to recreate a decision boundary in model space. The extent to
which this model space matches reality is another question altogether,
presumably one that data scientists should take seriously, both before and
throughout deployment. Once the model is in the wild, however, it may be of
independent interest to users and regulators how it performs on a wide range
of hypothetical cases, including those off the true data manifold.

_Conditional_. The conditional distribution is better suited to explanations
at the system level, where impossible data permutations could lead to issues.
This may be the case, for instance, in certain instances of validation, where
we do not wish to punish a model for failing to extrapolate to data points far
from its training set. Such methods are especially valuable when joint
distributions may be estimated with reasonable accuracy and causal
relationships are unknown or potentially inapplicable (e.g., in image
classification tasks). Estimating joint densities is a hard problem in
general, especially when data are high-dimensional and/or of mixed variable
types.

_Interventional_. The interventional distribution is optimal when seeking
explanations at the system level for causal data generating processes. With
access to the underlying SCM, there can be no more accurate estimator than
that defined by (3). These methods are required when seeking to use XAI for
discovery and/or planning, as both sorts of actions invariably rely on real-
world mechanisms that cannot be approximated by either a purely marginal
approach (which ignores all dependencies) or a conditional one (which fails to
distinguish between correlation and causation). Of course, complete causal
information is almost never available, which means that in practice analysts
must make strong assumptions and/or make do with workable approximations.

I employ marginal, conditional, and interventional value functions in all
experiments below (see Sect. 4). To compute interventional Shapley values, I
rely on partial orderings, as these are often the most readily available to
analysts at test time. A partial ordering of features implies a chain graph in
which links are composed of all variables in a single causal group. For
instance, users may not know the complete graph structure of the set
$\\{\texttt{age},\texttt{sex},\texttt{income},\texttt{savings},\texttt{credit}\\}$
but they may be more confident about the partial ordering
$\\{\texttt{age},\texttt{sex}\\}\rightarrow\\{\texttt{income},\texttt{savings}\\}\rightarrow\\{\texttt{credit}\\}$.
Lauritzen and Richardson (2002) demonstrate how to factorize the probability
distribution for a DAG of chain components. Heskes et al. (2020) combine this
procedure with a few steps of $do$-calculus to compute interventional Shapley
values from observational data.

###  3.2. A New Axiom

Say two agents are identical along all recorded variables for some credit
scoring task, and both receive lower scores than they had hoped.
Counterfactuals identify two minimal perturbations sufficient to improve
predictions: (1) increase education by one unit; or (2) increase savings by
one unit. Let us stipulate that these actions are equivalent in terms of time
and money. However, they differ in another important respect – the first agent
wants to go back to school and the second wants to save more each month. In
this case, _preferences alone_ determine which of two explanations is optimal.

We may devise a similar example to show the impact of differing credences.
Say, for instance, that two labs are planning expensive gene knockout
experiments to test the predictions of a gene regulatory network inference
algorithm. Their utilities may be identical – the goal for both groups is to
maximize true discoveries while minimizing false positives – but they are
working with different sets of evidence after preliminary, as yet unpublished
results indicated to the first lab that one set of pathways is a probable dead
end. In this case, _beliefs alone_ determine which of two explanations is
optimal.

What these examples demonstrate is that utilities and credences matter when
attempting to explain model predictions. With default software like SHAP,
feature attributions for these credit loan applicants or research labs would
not generally differ – same inputs, same outputs. However, by explicitly
incorporating pragmatic information, we can secure custom explanations at no
cost to the axiomatic guarantees of Shapley values. In fact, we can extend the
current desiderata.

Observe that for fixed value function $v$, model $f$, and input $\bm{x}_{i}$,
feature attributions $\bm{\phi}=\\{\phi_{j}\\}_{j=1}^{d}$ vary only as a
function of contrastive baseline $\tilde{\bm{z}}$. I make this dependence
explicit moving forward, writing $\bm{\phi}(\tilde{\bm{z}})$. Define a reward
function for the inquiring agent as the expected utility of the utility-
maximizing action $a^{*}$, conditional on a given set of feature attributions:

(7) $r(\tilde{\bm{z}})=\mathop{\mathbb{E}}_{h\sim
p}[u(a^{*},h)|\bm{\phi}(\tilde{\bm{z}})].$

The axiomatic constraint that characterizes pragmatic feature attributions can
now be stated as follows. For the proof of Thm. 3.1, see Appx. A.

######  Definition 2 (Rationality).

Let $r(\cdot)$ be the reward function for a rational agent, i.e. one who
behaves in accordance with the utility and probability axioms. Let
$\bm{\phi}(\tilde{\bm{z}})$ be a feature attribution vector computed with
respect to subspace $\tilde{\mathcal{Z}}$. Then $\bm{\phi}(\tilde{\bm{z}})$ is
_rational_ if and only if, for any alternative subspace
$\mathcal{Z}^{\prime}$,
$\mathbb{E}_{\tilde{\bm{z}}\sim\tilde{\bm{Z}}}[r(\tilde{\bm{z}})]\geq\mathbb{E}_{\bm{z}^{\prime}\sim\mathcal{Z}^{\prime}}[r(\bm{z}^{\prime})]$.
In other words, feature attributions are rational to the extent that they tend
to maximize expected rewards for the inquiring agent.

###### Theorem 3.1.

When the relevant subspace is nonempty, rational Shapley values are the unique
additive feature attribution method that satisfies efficiency, linearity,
sensitivity, symmetry, and rationality.

###  3.3. Rational Shapley Value Algorithm

Input: Input datapoint $\bm{x}_{i}$, set of candidate subspaces
$\\{\langle\mathcal{D}_{S},\mathcal{D}_{R},\mathcal{Y}_{1}\rangle_{k}\\}_{k=1}^{m}$,
utility function $u$, credence function $p$

Output: $\bm{\phi}(\tilde{\bm{z}}^{*})$

Compute classical Shapley values $\bm{\phi}(\bm{z})$ using mean
$\phi_{0}=n^{-1}\sum_{i=1}^{n}f(\bm{x}_{i})$ and reference $\mathcal{D}_{R}$

Initialize
$r(\bm{z})\leftarrow\sum_{j}p(h_{j}|\bm{\phi}(\bm{z}))~{}u(a^{*},h_{j})$

for  _$k=\\{1,\dots,m\\}$_ do

Draw data
$\tilde{\bm{z}}_{k}\sim\langle\mathcal{D}_{S},\mathcal{D}_{R},\mathcal{Y}_{1}\rangle_{k}$

if  _$\tilde{\bm{z}}_{k}=\\{\emptyset\\}$_ then

Fail

else

Compute $\bm{\phi}(\tilde{\bm{z}}_{k})$

Record
$r(\tilde{\bm{z}}_{k})\leftarrow\sum_{j}p(h_{j}|\bm{\phi}(\tilde{\bm{z}}))~{}u(a^{*},h_{j})$

$\tilde{\bm{z}}^{*}\leftarrow\operatorname*{argmax}_{\tilde{\bm{z}}}r(\tilde{\bm{z}})$

Algorithm 1 Rational Shapley Values

I reframe the objective function of Eq. 4 with a slight twist. Our goal is not
to minimize the cost or distance between two individual datapoints, but rather
to compute a subspace that maximizes rewards. That is:

(8)
$\mathcal{Z}^{*}=\operatorname*{argmax}_{\tilde{\mathcal{Z}}\subseteq\mathcal{Z}}~{}\mathop{\mathbb{E}}_{\tilde{\bm{z}}\sim\tilde{\mathcal{Z}}}[r(\tilde{\bm{z}})].$

This differs from (and improves upon) Eq. 4 in several respects. First, it
results not in a single point but in a region of space, which is stabler and
more informative. Second, it explicitly incorporates preferences and credences
via the functions that define $r(\cdot)$. Finally, since rewards are
conditioned upon a feature attribution vector, this target combines elements
of both counterfactuals and feature attributions, which is preferable to
either alternative alone for all the reasons argued above.

There are many ways one could go about computing this subspace in practice,
and no single method is optimal in general. See Karimi et al. (2020) for a
recent review. In the following experiments, I follow Wexler et al. (2020) in
relying on observational samples. These subspaces have the advantage of being
fast to find and guaranteed to lie on the true data manifold (up to potential
sampling or measurement errors). They avoid the assumptions and approximations
of more convoluted optimization techniques. The main disadvantage of this
method is that it may provide low coverage for regions of the feature space
where data are undersampled. However, this poses issues for other approaches
as well, as no method can confidently draw realistic points from low-density
regions of the feature space.

The basic strategy is schematized in pseudocode (see Alg. 1). The user
postulates a set of candidate subspaces, computing a sequence of rational
Shapley vectors and comparing relative rewards. Evaluating $r(\tilde{\bm{z}})$
with precision requires explicit utilities and (conditional) credences, which
may not be generally available. This can be done informally by comparing
feature attribution vectors on an ordinal basis, such that rewards are ranked
rather than directly quantified. This procedure is equivalent to the cardinal
alternative under reasonable assumptions about agentive rationality, as noted
above. The algorithm is deliberately silent on how to select candidate
subspaces, as this is irreducibly context-dependent. Credence updates may be
calculated in closed form under some parametric assumptions, or else via Monte
Carlo. The examples below illustrate how the method works in practice.

##  4\. Experiments

In this section, I describe results from a number of experiments on benchmark
datasets. I compare performance against baselines using marginal, conditional,
and interventional value functions, respectively labelled MSV, CSV, and ISV.
Code for reproducing all results and figures can be found at
https://github.com/dswatson/rational_shapley.

###  4.1. Auditing: COMPAS

The COMPAS algorithm is a statistical model used to assign risk scores to
defendants awaiting trial. High risk scores are associated with elevated
probability of recidivism, which judges in nine US states use to help decide
whether to let defendants out on bail while awaiting trial. A 2016 report by
ProPublica alleged that the COMPAS algorithm was racially biased against
African Americans (Angwin et al., 2016); subsequent analysis by independent
researchers has not always corroborated those findings (Fisher et al., 2019;
Rudin et al., 2020).

Though the creators of COMPAS have published technical reports defending their
model, they have made neither their training data nor code publicly available,
so attempts to recreate the algorithm and audit for racial bias are typically
based on data gathered by ProPublica on some 12,000 individuals arrested in
Broward County, Florida between 2013 and 2014. Using a random forest (Breiman,
2001), I regress violent risk scores on the following features: age, number of
priors, and whether the present charge is a felony (all deemed “admissible”);
as well as race and sex (deemed “inadmissible”). For the purposes of this
experiment, I take the trained model as the true target.

MSV is computed via Monte Carlo (Štrumbelj and Kononenko, 2014), with 2000
simulations per attribution. CSV is computed using the TreeSHAP method
(Lundberg et al., 2020), since random forests are composed of individual
regression trees. For ISV, I presume a simple partial ordering in which
demographic variables (age, sex, and race) are root nodes, upon which all
other predictors depend. The mean predicted response for all defendants in the
dataset is $\phi_{0}=3.69$.

I focus specifically on high-risk defendants, namely the 264 subjects with
risk scores at or above the $95^{\text{th}}$ percentile according to the
COMPAS approximator. Some 94% of these defendants are African American,
compared to just 60% in the general dataset, which immediately raises
questions about racial bias. MSV, CSV, and ISV all look fairly similar in this
case (see Fig. 1). Age and priors generally receive the highest feature
attributions in this group, consistent with the findings of other researchers.
Race also receives nonzero attribution throughout. Notably, race receives
higher attribution under the ISV reference than MSV, since MSV can only detect
direct effects, and race is presumed to have both direct and indirect effects
given the partial ordering. In this example, I presume that auditors are
working at the model level – i.e., evaluating the patterns learned by the
algorithm, rightly or wrongly – in which case the marginal distribution is
best suited to their purpose.

Figure 1. Classical and rational Shapley values for the Broward County COMPAS
dataset, computed with marginal, conditional, and interventional reference
distributions. Continuous feature values are $z$-scored for visualization.

To compute rational Shapley values, I consider a relevant subspace of
defendants in the same age range (at most 33 years old) with at least one
prior, yet who fell in the bottom half of defendants by predicted risk score.
All other features are allowed to vary unconstrained. 356 subjects in the data
meet these criteria, and therefore constitute the reference sample for this
experiment. This is too large a group of counterfactuals to “explain” anything
on their own, even if they were ranked by some measure (e.g., distance from
the target input). What the task requires is a summary of differences across
features, a job for which Shapley values are uniquely well-suited. The racial
breakdown of defendants in the relevant subspace is markedly different from
our high-risk group – just 48% are African American – further reinforcing
concerns about potential bias in the COMPAS data. The resulting Shapley values
look fairly similar to their classical counterparts, although priors and age
appear to dominate the explanations under conditional and interventional
reference distributions. Using the marginal reference, however – which, as
noted above, is most relevant for auditing scenarios – we find greater
evidence of racial bias using rational Shapley values. The racial attribution
gap between Black and white defendants is significantly greater in the
relevant subspace than it is across the complete dataset $(t=3.874,p<0.001)$.

Consider the case of a young Black defendant deciding whether to file suit
against the makers of COMPAS. Despite having just a single prior offense, this
21-year-old was placed in the highest risk group according to the algorithm. A
utility matrix for this individual is given in Table 1. Assume that the
defendant in question assigns a uniform prior over $H$ to begin with. We say
that $h_{1}$ is corroborated to the extent that race receives greater positive
attribution for African Americans than it does for defendants of other races.
Then although both classical and rational Shapley values recommend the same
action $(a_{1})$, the latter does so with higher expected reward, since
$p(h_{1}|\bm{\phi}(\tilde{\bm{z}}))>p(h_{1}|\bm{\phi}(\bm{z}))$. Rational
Shapley values are therefore preferable in this case, as theory suggests.

Table 1. Utility matrix for a 21-year-old African American defendant with a single prior, predicted to be high risk and deciding whether to sue the makers of COMPAS. | $h_{1}:\texttt{biased}$ | $h_{2}:\neg\texttt{biased}$
---|---|---
$a_{1}:\texttt{sue}$ | 5 | –1
$a_{2}:\neg\texttt{sue}$ | 0 | 0

###  4.2. Discovery: Medical Diagnosis

As an example of XAI for discovery, I consider the diabetes dataset originally
described by (Efron et al., 2004). The data consist of 442 patients and 10
predictors – age, sex, body mass index (BMI), blood pressure (MAP), and six
blood serum measurements, including three cholesterol-related variables (LDL,
HDL, and TC). The goal is to predict disease progression after one year, for
which I use an elastic net regression (Zou and Hastie, 2005). Note that exact
Shapley values can be analytically calculated for linear models under the
assumption of feature independence. However, exploratory data analysis reveals
strong correlations between some features in the data (especially blood serum
measurements), and causal dependencies between features such as BMI and MAP
are well established. In such a case, the analytic formulae for local
explanation in linear models are inapplicable. For ISV, I assume the partial
ordering
$\\{\texttt{age},\texttt{sex},\texttt{bmi}\\}\rightarrow\\{\texttt{map},\texttt{ldl},\texttt{hdl},\texttt{tc}\\}\rightarrow\\{\texttt{tch},\texttt{ltg},\texttt{glt}\\}$.
This treats age, sex, and BMI as root nodes; MAP and cholesterol as causally
intermediate; and remaining blood serum measurements are downstream. Such a
DAG admittedly oversimplifies several complex biochemical processes but is
broadly consistent with known structural relationships.

I focus on patients with especially poor prognoses, as these are typically the
subjects of greatest concern to clinicians. Specifically, I examine the top
decile of patients by disease activity $(\hat{y}\geq 265)$. Classical Shapley
values for these patients are visualized under all three value functions in
Fig. 2. CSV and ISV are fairly similar here, although both are quite different
from the marginal alternative. For instance, TC receives relatively large
attributions under the marginal reference distribution, but nearly none under
conditional or interventional distributions, owing to its strong collinearity
with other cholesterol measures.

Figure 2. Classical and rational Shapley values for the diabetes dataset,
computed with marginal, conditional, and interventional reference
distributions. Continuous feature values are $z$-scored for visualization.

For rational Shapley analysis, I consider a subspace consisting of older men
(i.e., those in the top quartile by age), who are generally at higher risk
than other groups in this dataset. Since age and sex are (more or less)
unactionable, a rational agent will likely want prognostic explanations that
place little or no weight on such features. We find in Fig. 2 that BMI and MAP
receive large rational attributions on average, reflecting their greater
variance in the relevant subspace. Conditional and interventional reference
distributions are especially valuable here, as the analytical goal almost
certainly requires a system-level approach and causal dependencies cannot
therefore be ignored. Alternative structural assumptions may lead to somewhat
different attributions, but focusing on a relevant subspace has effectively
driven the Shapley values for unactionable variables like sex and age toward
zero. The result is that explanations are provided primarily in terms of
lifestyle variables like BMI and MAP, which agents can work to minimize in
order to improve their disease prognosis.

Table 2. Complete feature vector for both Bert and Ernie in the diabetes dataset. sex | age | bmi | map | tc | ldl | hdl | tch | ltg | glu
---|---|---|---|---|---|---|---|---|---
1 | –0.064 | 0.096 | 0.105 | –0.003 | –0.005 | –0.007 | -0.003 | 0.023 | 0.073

Consider the case of two high-risk patients – call them Bert and Ernie – who
share identical feature vectors (see Table 2; note that the negative value of
age is a quirk of the scaling procedure.) Despite their striking similarities,
however, Bert and Ernie differ in their culinary tastes. Bert was raised on
carbohydrate-rich foods, and cannot imagine living without bread, pasta, and
potatoes; Ernie, by contrast, is a voracious carnivore who eats bacon for
breakfast, cold cuts for lunch, and steak for dinner. Their respective utility
matrices are given by Table 3, where we assume for the sake of this example
that only two dietary interventions are under consideration: a low-carb diet
and a low-fat diet. Evidence suggests that the latter is slightly more
effective for weight loss, while the former has the added benefit of reducing
blood pressure (Brinkworth et al., 2009). Classical Shapley values, which
place the greatest weight on BMI, would tend to favour $a_{1}$ regardless of
agentive preferences. However, rational Shapley values can distinguish between
optimal explanations based entirely on utilities, since changes to either BMI
or MAP are sufficient to bring about the desired outcome, albeit at differing
costs to Bert and Ernie. Thus we find the following two feature attribution
vectors (see Table 4), computed using the conditional reference distribution
on inputs that differ only with respect to utilities $u$.

Table 3. Utility matrices for Bert (left) and Ernie (right). Bert would like
to reduce his risk of diabetes without lowering his carb intake; Ernie wants
the same outcome without lowering his fat intake.

Bert | $h_{1}:\hat{y}<265$ | $h_{2}:\hat{y}\geq 265$
---|---|---
$a_{1}$: low-fat | 5 | –1
$a_{2}$: low-carb | 1 | –6

Ernie | $h_{1}:\hat{y}<265$ | $h_{2}:\hat{y}\geq 265$
---|---|---
$a_{1}$: low-fat | 1 | –6
$a_{2}$: low-carb | 5 | –1

Table 4. Rational Shapley vectors for Bert (top) and Ernie (bottom), computed from the same input vector (see Table 2) but with respect to different relevant subspaces. sex | age | bmi | map | tc | ldl | hdl | tch | ltg | glu
---|---|---|---|---|---|---|---|---|---
–3.789 | –4.867 | 94.922 | 29.241 | 1.763 | 0.849 | 4.053 | 6.566 | 13.019 | 4.324
–1.291 | 0.114 | 42.604 | 66.942 | 2.187 | 1.129 | –0.167 | –3.532 | 11.136 | 9.903

Observe that, among the many differences between these two vectors, they flip
the relative importance of BMI and MAP, suggesting two alternative paths
toward reducing disease risk. A counterfactual approach with a well-designed
cost function could in principle also identify different explanations for
these two agents, but it could not additionally provide complete feature
attributions summarising the relative impact of other predictors. Such a
synthesis is unique to the rational Shapley approach.

###  4.3. Recourse: Credit Scoring

As a final example, I examine the recourse setting, where the goal is to
advise an agent what interventions are necessary and/or sufficient to secure a
desired prediction. As Karimi et al. (2021) have shown, this task requires
causal information. More specifically, they prove that noncausal
recommendations, such as those provided by Ustun et al. (2019), guarantee
recourse if and only if the treatment variables have no descendants in the
underlying causal graph. For this experiment, I use the German credit dataset
from the UCI Machine Learning Repository (Dua and Graff, 2017). To simplify
the presentation, I restrict focus to just seven of the most informative
features: age $A$, gender $G$, marital status $M$, job $J$, savings $S$, loan
amount $L$, and duration $D$. The outcome $Y$ is binary (loan
approved/denied), with a base rate of 70%. I train a support vector machine
(SVM) (Cortes and Vapnik, 1995) to predict $Y$ using a radial basis kernel.

Figure 3. Classical and rational Shapley values for the German credit dataset,
computed with marginal, conditional, and interventional reference
distributions. Continuous feature values are $z$-scored for visualization.

I assume a simple partial ordering in which demographic variables are causally
antecedent to both financial predictors and loan application details:
$\\{A,G,M\\}\rightarrow\\{L,D,J,S\\}$. The UCI website notes that false
positives are more costly for banks than false negatives, and therefore
recommends penalizing the former at five times the rate of the latter. Post-
processing the SVM, an optimal decision threshold is obtained at 0.74, and I
therefore label all and only those points with predicted probabilities at or
above this value as 1. The resulting model is about 69% accurate on the true
positives and 73% accurate on the true negatives. I focus on the 189
borderline applicants with predicted success probabilities on the interval
$[0.7,0.74)$. These are subjects who could plausibly benefit from even a
slight improvement in their model predictions. Classical Shapley values are
unable to help such applicants, since the mean response in the full dataset is
$\phi_{0}=0.7$, i.e. the lower bound of the interval. (Irrational) feature
attributions for these subjects are therefore guaranteed to sum to some value
on $[0,0.04)$, explaining why they did slightly better than average instead of
why they were ultimately unsuccessful.

The most important component of the relevant subspace is the contrastive
outcome $\mathcal{Y}_{1}$, as the borderline applicants appear fairly
representative along all predictors of interest. I therefore stratify by
predicted response, zooming in on the top quartile of applicants $(\hat{y}\geq
0.76)$. Bivariate analyses suggest that such applicants are for the most part
financially secure and request small loans of relatively brief duration. The
classical and rational Shapley vectors look fairly similar in this experiment,
except for the crucial difference in offset. (See Fig. 3; note that the
response variable is transformed to the logit scale for easier visualization.)

Consider the case of a loan applicant, call her Ruth, whose predicted success
probability is 0.73 – just under the decision boundary. Her input feature
vector can be found in Table 5, where values are reported on the original
(pre-transformation) scale. She is looking to buy a new home and believes her
high-skilled job and decent savings (both well above their respective median
values in the dataset) should make her a strong applicant. She is willing to
make some changes if necessary – specifically, to increase her savings,
decrease the size of her requested loan, or pay it back sooner – but in any
case she would like to act quickly, since she fears the house she wants to buy
will not be on the market for long. Ruth’s utility matrix is given in Table 6.
We find here that she prefers changing the terms of her application to
increasing her savings in general, as this is likely the faster route to bank
approval. She is indifferent between changing the loan amount or duration. As
noted above, computing Shapley values from the entire dataset is pointless in
this case, as Ruth’s predicted outcome is already above the base rate.
Counterfactuals could potentially aid her search for an explanation, but only
by either providing an artificially narrow set of pathways to approval or
overwhelming her with an unnecessarily large reference class of successful
applicants.

Table 5. Complete feature vector for Ruth in the German credit dataset. gender | marital | age | amount | duration | savings | job
---|---|---|---|---|---|---
0 | 0 | 26 | 3181 | 26 | 290 | 2
Table 6. Utility matrix for Ruth, whose demographic and financial information place her just below the loan approval threshold for a hypothetical bank. | $h_{1}:\hat{y}\geq 0.74$ | $h_{2}:\hat{y}<0.74$
---|---|---
$a_{1}$: Increase $S$ | 1 | –2
$a_{2}$: Decrease $L$ | 2 | –1
$a_{3}$: Decrease $D$ | 2 | –1

Rational Shapley values, computed from a relevant subspace of successful loan
applicants who share Ruth’s demographic characteristics, provide a summary of
feature attributions within this reference class (see Table 7; note that
calculations are performed in logit space.) Interestingly, we find that her
requested loan amount, though above average for the relevant subspace, did not
hurt Ruth’s application. On the contrary, it appears to boost her chances
somewhat, a result we might not expect from surveying the counterfactuals
directly. The loan duration, however – about double the average for the
relevant subspace – brought her success probability down considerably. This is
a highly actionable piece of information, insomuch as it guides Ruth toward
algorithmic recourse that can push her application over the decision boundary.
Of course, if the change is too onerous for Ruth, then she is free to resample
from another subspace of successful applicants with longer loan duration on
average. The procedure may continue like this indefinitely, with agents
testing out new hypotheses in an iterative, exploratory fashion, updating
their rewards accordingly.

Table 7. Rational Shapley values for Ruth in the German credit dataset. gender | marital | age | amount | duration | savings | job
---|---|---|---|---|---|---
0 | 0 | –0.168 | 0.061 | –0.115 | 0.013 | 0.128

##  5\. Discussion

The skeptical reader may plausibly object that the examples above are fairly
neat and straightforward, with their small utility matrices of well-defined
user preferences. Reality, of course, is far messier. Complete sets of actions
and outcomes may not be known upfront, let alone utility and credence
functions defined thereon. Potential interventions may be far more numerous
than these experiments permit, and associated outcomes entirely uncertain. Can
rational Shapley values scale to larger, more complicated instances of
algorithmic explanation?

In a word, yes. These cases are merely illustrative, following on the back of
theoretical results establishing the viability of a pragmatic synthesis
between feature attributions and counterfactuals that works with various
different reference distributions. The limiting factors in these experiments
were complications around efficient Monte Carlo sampling, conditional
probability estimation, and reliable partial orderings. The first can be
entirely resolved with greater computational power. The second requires some
care, but may in principle be addressed with flexible methods such as
variational autoencoders. The third relies on background knowledge, which may
be substantial in some areas, and should in any case grow over time.
Improvements in any or all of these areas can help extend the rational Shapley
method to larger, more complex cases.

Another potential challenge to the proposed method is that it is vulnerable to
confirmation bias. If users can scan the data in search of attribution vectors
that make their preferred outcomes more likely, what’s to stop them from
finding such vectors even when do they do not exist? There are three answers
to this charge. First, it is entirely possible that the relevant subspace be
empty – that is, no observed or synthetic datapoint meets the threefold
criteria specified by
$\langle\mathcal{D}_{S},\mathcal{D}_{R},\mathcal{Y}_{1}\rangle$. Such a
failure is highly informative, as it demarcates a realm of (im)possibility for
the inquiring agent. Thus it is simply false to allege that users will always
find what they are looking for. Second, in cases where the relevant subspace
is nonempty but sparsely populated, we should expect estimates to be unstable.
Any good inference procedure should take such uncertainty into account, making
it difficult for an opportunistic user’s desired outcome to pass severe tests
on the basis of just a small handful of outlying points. I have not said much
about testing here, but it is simple in principle to extend the experiments
above with frequentist or Bayesian methods tailored to the particular task at
hand (Slack et al., 2021). Third, there is no guarantee that agents will
generally concur on proper values of $\mathcal{D}_{S},\mathcal{D}_{R}$, and
$\mathcal{Y}_{1}$. This makes it all the more important to be explicit about
which values of each input go into any given explanation. Such transparency
will make it harder for adversarial agents to game the system and easier for
those acting in good faith to come to consensus on particular cases.

##  6\. Conclusion

The two most popular XAI tools available for local, post-hoc explanations –
feature attributions and counterfactuals – each have certain advantages.
However, they tend to provide different answers in particular cases that can
confuse and/or mislead end users. In this paper, I have shown how the two can
be synthesized into a single method that preserves the best elements of each.
Rational Shapley values preserve and extend the axiomatic guarantees of their
classical forebears, giving users the flexibility to compute additive feature
attributions in a fast, flexible manner. Marginal, conditional, and
interventional value functions were all evaluated, and the applicability of
each was compared across different use cases. By formalizing the task in an
expected utility framework, I was able to demonstrate how and why individual
agents may rationally seek different explanations for the same model
predictions. The resulting explanations are concise, intuitive, and thoroughly
pragmatic.

###### Acknowledgements.

This research was supported by ONR grant N62909-19-1-2096.

## References

  * (1)
  * Aas et al. (2021) Kjersti Aas, Martin Jullum, and Anders Løland. 2021\.  Explaining individual predictions when features are dependent: More accurate approximations to Shapley values.  _Artif. Intell._ 298 (2021), 103502.
  * Adadi and Berrada (2018) Amina Adadi and Mohammed Berrada. 2018.  Peeking inside the black-box: A survey on explainable artificial intelligence (XAI).  _IEEE Access_ 6 (2018), 52138–52160.
  * Angwin et al. (2016) J Angwin, J Larson, S Mattu, and L Kirchner. 2016\.  _Machine bias_.  Technical Report. ProPublica.  https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
  * Barocas et al. (2019) Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019\.  _Fairness and Machine Learning_.  fairmlbook.org.  http://www.fairmlbook.org.
  * Breiman (2001) Leo Breiman. 2001\.  Random Forests.  _Mach. Learn._ 45, 1 (2001), 1–33.
  * Brinkworth et al. (2009) Grant D Brinkworth, Manny Noakes, Jonathan D Buckley, Jennifer B Keogh, and Peter M Clifton. 2009\.  Long-term effects of a very-low-carbohydrate weight loss diet compared with an isocaloric low-fat diet after 12 mo.  _Am. J. Clin. Neutr._ 90, 1 (2009), 23–32.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Mach. Learn._ 20, 3 (1995), 273–297.
  * Dawid (2012) Philip Dawid. 2012\.  The Decision-Theoretic Approach to Causal Inference.  In _Causality: Statistical Perspectives and Applications_ , Carlo Berzuini, Philip Dawid, and Luisa Bernardinelli (Eds.). Wiley, London, Chapter 4, 25–42.
  * Dawid (2015) Philip Dawid. 2015\.  Statistical Causality from a Decision-Theoretic Perspective.  _Annu. Rev. Stat. Appl._ 2, 1 (2015), 273–303.
  * Dawid (2020) Philip Dawid. 2020\.  Decision-theoretic foundations for statistical causality. (2020).  _arXiv_ preprint, 2004.12493.
  * Dua and Graff (2017) Dheera Dua and Casey Graff. 2017.  _UCI Machine Learning Repository_.  University of California, School of Information and Computer Science, Irvine, CA.
  * Efron et al. (2004) Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani. 2004.  Least Angle Regression.  _Ann. Stat._ 32, 2 (2004), 407–499.
  * Feyerabend (1975) Paul Feyerabend. 1975\.  _Against Method_.  New Left Books, London.
  * Fisher et al. (2019) Aaron Fisher, Cynthia Rudin, and Francesca Dominici. 2019\.  All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously.  _J. Mach. Learn. Res._ 20, 177 (2019), 1–81.
  * Floridi (2019) Luciano Floridi. 2019\.  _The Logic of Information_.  Oxford University Press, Oxford.
  * Frye et al. (2020) Christopher Frye, Ilya Feige, and Colin Rowat. 2020\.  Asymmetric Shapley values: Incorporating causal knowledge into model-agnostic explainability. In _Advances in Neural Information Processing Systems_.
  * Guidotti et al. (2018) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018.  A Survey of Methods for Explaining Black Box Models.  _Comput. Surveys_ 51, 5 (2018), 1–42.
  * Heskes et al. (2020) Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. 2020\.  Causal Shapley Values: Exploiting Causal Knowledge to Explain Individual Predictions of Complex Models. In _Advances in Neural Information Processing Systems_.
  * Janzing et al. (2020) Dominik Janzing, Lenon Minorics, and Patrick Bloebaum. 2020\.  Feature relevance quantification in explainable AI: A causal problem. In _Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics_ , Vol. 108. 2907–2916.
  * Karimi et al. (2020) Amir-Hossein Karimi, Gilles Barthe, Bernhard Schölkopf, and Isabel Valera. 2020.  A survey of algorithmic recourse: Definitions, formulations, solutions, and prospects. (2020).  _arXiv_ preprint, 2010.04050.
  * Karimi et al. (2021) Amir-Hossein Karimi, Bernhard Schölkopf, and Isabel Valera. 2021.  Algorithmic Recourse: From Counterfactual Explanations to Interventions. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 353–362.
  * Kumar et al. (2020) Indra Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, and Sorelle Friedler. 2020.  Problems with Shapley-value-based explanations as feature importance measures. In _Proceedings of the 37th International Conference on Machine Learning_ _(ICML ’20)_. 1–10.
  * Lauritzen and Richardson (2002) Steffen L Lauritzen and Thomas S Richardson. 2002.  Chain graph models and their causal interpretations.  _J. Royal Stat. Soc. Ser. B Methodol._ 64, 3 (2002), 321–348.
  * Lee and Bareinboim (2018) Sanghack Lee and Elias Bareinboim. 2018.  Structural Causal Bandits: Where to Intervene?. In _Advances in Neural Information Processing Systems_ , Vol. 31. 2568–2578.
  * Lee and Bareinboim (2019) Sanghack Lee and Elias Bareinboim. 2019.  Structural Causal Bandits with Non-Manipulable Variables.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 33, 1 (2019), 4164–4172.
  * Legg and Hookway (2020) Catherine Legg and Christopher Hookway. 2020.  Pragmatism.  In _The Stanford Encyclopedia of Philosophy_ (fall 2020 ed.), Edward N Zalta (Ed.). Metaphysics Research Lab, Stanford University.
  * Lundberg et al. (2020) Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020\.  From local explanations to global understanding with explainable AI for trees.  _Nat. Mach. Intell._ 2, 1 (2020), 56–67.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A Unified Approach to Interpreting Model Predictions.  In _Advances in Neural Information Processing Systems_. 4765–4774.
  * Mahajan et al. (2019) Divyat Mahajan, Chenhao Tan, and Amit Sharma. 2019.  Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers. In _CausalML: Machine Learning and Causal Inference for Improved Decision Making Workshop, NeurIPS 2019_.
  * Merrick and Taly (2020) Luke Merrick and Ankur Taly. 2020.  The Explanation Game: Explaining Machine Learning Models Using Shapley Values. In _Machine Learning and Knowledge Extraction - 4th International Cross-Domain Conference (CD-MAKE)_. Springer, 17–38.
  * Miller (2019) Tim Miller. 2019\.  Explanation in artificial intelligence: Insights from the social sciences.  _Artif. Intell._ 267 (2019), 1–38.
  * Mothilal et al. (2020) Ramaravind K. Mothilal, Amit Sharma, and Chenhao Tan. 2020\.  Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations. In _Proceedings of the International Conference on Fairness, Accountability, and Transparency in Machine Learning_ (Barcelona, Spain). 607–617.
  * Murdoch et al. (2019) W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. 2019.  Definitions, methods, and applications in interpretable machine learning.  _Proceedings of the National Academy of Sciences_ 116, 44 (2019), 22071 – 22080.
  * Pearl (2000) Judea Pearl. 2000\.  _Causality: Models, reasoning, and inference_.  Cambridge University Press, New York.
  * Poyiadzi et al. (2020) Rafael Poyiadzi, Kacper Sokol, Raul Santos-Rodriguez, Tijl De Bie, and Peter Flach. 2020.  FACE: Feasible and actionable counterfactual explanations. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 344–350.
  * Rudin et al. (2020) Cynthia Rudin, Caroline Wang, and Beau Coker. 2020.  The Age of Secrecy and Unfairness in Recidivism Prediction.  _Harvard Data Sci. Rev._ 2, 1 (2020).
  * Russell (2019) Chris Russell. 2019\.  Efficient Search for Diverse Coherent Explanations. In _Proceedings of the International Conference on Fairness, Accountability, and Transparency_. 20–28.
  * Shapley (1953) Lloyd Shapley. 1953\.  A Value for _n_ -Person Games.  In _Contributions to the Theory of Games_. Princeton University Press, Princeton, Chapter 17, 307–317.
  * Shpitser and Pearl (2008) Ilya Shpitser and Judea Pearl. 2008.  Complete Identification Methods for the Causal Hierarchy.  _J. Mach. Learn. Res._ 9 (2008), 1941–1979.
  * Slack et al. (2021) Dylan Slack, Anna Hilgard, Sameer Singh, and Himabindu Lakkaraju. 2021.  Reliable Post hoc Explanations: Modeling Uncertainty in Explainability. In _Advances in Neural Information Processing Systems_ , Vol. 34. 9391–9404.
  * Štrumbelj and Kononenko (2014) Erik Štrumbelj and Igor Kononenko. 2014.  Explaining prediction models and individual predictions with feature contributions.  _Knowledge and Information Systems_ 41, 3 (2014), 647–665.
  * Sundararajan and Najmi (2019) Mukund Sundararajan and Amir Najmi. 2019.  The many Shapley values for model explanation. In _Proceedings of the ACM Conference_.
  * Ustun et al. (2019) Berk Ustun, Alexander Spangher, and Yang Liu. 2019\.  Actionable Recourse in Linear Classification. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_. 10–19.
  * von Neumann and Morgenstern (1944) John von Neumann and Oskar Morgenstern. 1944.  _Theory of Games and Economic Behavior_.  Princeton University Press, Princeton, NJ.
  * Wachter et al. (2018) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018\.  Counterfactual explanations without opening the black box: Automated decisions and the GDPR.  _Harvard J. Law Technol._ 31, 2 (2018), 841–887.
  * Watson (2022) David S Watson. 2022\.  Conceptual challenges for interpretable machine learning.  _Synthese_ 200, 2 (2022), 65.
  * Watson and Floridi (2021) David S Watson and Luciano Floridi. 2021.  The explanation game: a formal framework for interpretable machine learning.  _Synthese_ 198, 10 (2021), 9211–9242.
  * Watson et al. (2022) David S Watson, Limor Gultchin, Ankur Taly, and Luciano Floridi. 2022.  Local Explanations via Necessity and Sufficiency: Unifying Theory and Practice.  _Minds Mach._ 32, 1 (2022), 185–218.
  * Wexler et al. (2020) J Wexler, M Pushkarna, T Bolukbasi, M Wattenberg, F Viégas, and J Wilson. 2020\.  The What-If Tool: Interactive Probing of Machine Learning Models.  _IEEE Transactions on Visualization and Computer Graphics_ 26, 1 (2020), 56–65.
  * Zou and Hastie (2005) Hui Zou and Trevor Hastie. 2005.  Regularization and Variable Selection via the Elastic Net.  _J. Royal Stat. Soc. Ser. B Methodol._ 67, 2 (2005), 301–320.

##  Appendix A Appendix

The following is a proof of Thm. 3.1, which states that rational Shapley
values are the unique additive feature attribution method that satisfies
efficiency, linearity, sensitivity, symmetry, and rationality.

###### Proof.

Since uniqueness is already well established for the classical axioms, all
that remains is to show that rational Shapley values (a) satisfy the
additional rationality axiom and (b) do not violate any of the classical
axioms.

Take (a) first. Assume there exists some $\mathcal{Z}^{\prime}$ that generates
greater expected rewards for an agent than the relevant subspace
$\tilde{\mathcal{Z}}$. Then conditioning on the corresponding Shapley values
$\bm{\phi}(\bm{z}^{\prime})$ must lead to an action of greater expected
utility than the action recommended by $\bm{\phi}(\tilde{\bm{z}})$. This means
that the values for some $\phi_{j}(\tilde{\bm{z}})$ are either too large or
too small. But this can only arise from a misspecification of (i)
$\mathcal{Y}_{1}$, (ii) $\mathcal{D}_{S}$, or (iii) $\mathcal{D}_{R}$, which
together permit either too much variation in $X_{j}$ (resulting in inflated
values of $|\phi_{j}(\tilde{\bm{z}})|$), or too little (resulting in deflated
values of $|\phi_{j}(\tilde{\bm{z}})|$). By Def. 1, any $\tilde{\mathcal{Z}}$
that fails to satisfy one of these criteria is not a relevant subspace. Thus
we have a contradiction.

One can verify (b) by confirming that rational Shapley values do not deviate
from the classical formulae (see Eqs. 1, 2, 3). The only difference between
the current proposal and more familiar alternatives is their respective
methods of specifying baseline expectations $\phi_{0}$ and reference
distributions $\mathcal{D}_{R}$. Shifting these two parameters changes the
interpretation of resulting Shapley vectors, but does not alter their
essential character. Indeed, much recent work on Shapley values has been
devoted to exploring alternativere reference distributions; for more on the
plurality of Shapley values, see (Sundararajan and Najmi, 2019; Merrick and
Taly, 2020). Any procedure that inputs valid values for $\phi_{0}$ and
$\mathcal{D}_{R}$ and conforms to the characteristic equations enjoys the same
properties. ∎