#  Scientific Inference With Interpretable Machine LearningAnalyzing Models to
Learn About Real-World Phenomena

Timo Freiesleben
Munich Center for Mathematical Philosophy
& Graduate School of Systemic Neurosciences
LMU Munich &Gunnar König
Department of Statistics
LMU Munich & University of Vienna Christoph Molnar
Independent Researcher
Munich & Alvaro Tejero-Cantero
Cluster of Excellence Machine Learning
New Perspectives for Science
University of Tübingen

###### Abstract

Interpretable machine learning (IML) is concerned with the behavior and the
properties of machine learning models. Scientists, however, are only
interested in models as a gateway to understanding phenomena. Our work aligns
these two perspectives and shows how to design _IML property descriptors_.
These descriptors are IML methods that provide insight not just into the
model, but also into the properties of the phenomenon the model is designed to
represent. We argue that IML is necessary for scientific inference with ML
models because their elements do not individually represent phenomenon
properties; instead, the model in its entirety does. However, current IML
research often conflates two goals of model analysis — model audit and
scientific inference – making it unclear which model interpretations can be
used to learn about phenomena. Building on statistical decision theory, we
show that IML property descriptors applied on a model provide access to
relevant aspects of the joint probability distribution of the data. We
identify what questions such descriptors can address, provide a guide to
building appropriate descriptors and quantify their epistemic uncertainty.

Keywords: Scientific Modeling, Interpretable Machine Learning, Scientific
Representation, Inference, XAI

##  1 Introduction

Scientists increasingly use machine learning (ML) in their daily work. This
development is not limited to natural sciences like the geosciences
(Reichstein et al., 2019) or material science (Schmidt et al., 2019), but also
extends to social sciences such as education science (Luan and Tsai, 2021) and
archaeology (Bickler, 2021).
When building predictive models for problems with complex data structures, ML
outcompetes classical statistical models in both performance and convenience.
Impressive recent examples of successful prediction models in science include
the automated particle tracking at CERN (Farrell et al., 2018), or DeepMind’s
AlphaFold, which has essentially solved the protein structure prediction
challenge CASP (Senior et al., 2020). In such examples, some see a paradigm
shift towards theory-free science that “lets the data speak” (Kitchin, 2014,
Anderson, 2008, Mayer-Schönberger and Cukier, 2013, Spinney, 2022). Indeed,
prediction is one of the core aims of science (Luk, 2017, Douglas, 2009), but
so are, as philosophers of science and statisticians emphasize, explanation
and knowledge generation (Salmon, 1979, Longino, 2018, Shmueli et al., 2010).
Focusing exclusively on prediction may therefore represent a historical step
back (Toulmin, 1961, Pearl, 2018).
What hinders scientists from using ML models to gain real-world insights is
model complexity and an unclear connection between model and phenomenon — the
so-called _opacity problem_ (Boge, 2022, Sullivan, 2020). Interpretable
machine learning (IML, also called XAI, for eXplainable artificial
intelligence) aims to solve the opacity problem by analyzing individual model
elements or inspecting specific model properties (Molnar, 2020). Different
stakeholders with different goals hold diverse expectations of IML (Zednik,
2021), including scientists (Roscher et al., 2020), ML engineers (Bhatt et
al., 2020), regulatory bodies (Wachter et al., 2017), and laypeople (Arrieta
et al., 2020). Due to this plurality, IML has been criticized for lacking a
proper definition (Lipton, 2018).
Nevertheless, scientists increasingly use IML for inferring which features are
predictive of e.g. crop yield (Shahhosseini et al., 2020, Zhang et al., 2019),
personality traits (Stachl et al., 2020), or seasonal precipitation (Gibson et
al., 2021). Although researchers are aware that their IML analyses remain just
model descriptions, it is often implied that the explanations, associations,
or effects found also extend to the corresponding real-world properties.
Unfortunately, drawing inferences with IML can currently be epistemically
problematic because the interpretation methods are not designed for that
purpose (Molnar et al., 2022). In particular, the difference between model-
only versus phenomenon explanations is often unclear (Chen et al., 2020,
Hooker et al., 2021), and a theory to quantify the uncertainty of
interpretations is lacking (Molnar et al., 2020a , Watson, 2022).

#### Contributions.

In this paper, we present an account of scientific inference with IML inspired
by ideas from philosophy of science and statistical inference. While we focus
on supervised learning on identically and independently distributed (i.i.d.)
data, we briefly discuss other learning scenarios in Section 5.3. Our key
contributions are: 1. We argue that ML cannot profit from the traditional
approach to scientific inference via model elements because its parameters do
not represent phenomenon properties (Section 3). While current IML methods aim
to restore representationality of the model as a whole, they conflate the
model audit and scientific inference goals of interpretation. 2. We identify
the properties that IML methods need to fulfill to provide access to aspects
of the conditional probability distribution
$\mathbb{P}(\boldsymbol{Y}\mid\boldsymbol{X})$, where $\boldsymbol{X}$
describes predictor variables and $\boldsymbol{Y}$ the target (Section 4). We
call methods that are suitable for inference _IML property descriptors_. We
provide a guide to build such descriptors starting with a phenomenon question
about $\boldsymbol{X}$ and $\boldsymbol{Y}$ and evaluating whether it can be
addressed, followed by an answer to this question with ML models and finite
data, and conclude with the quantification of epistemic uncertainty. We
illustrate our approach using conditional partial dependence plots (cPDP) as
an example IML descriptor.

#### Terminology.

For the purposes of our discussion below, a _phenomenon_ is a real-world
process whose aspects of interest can be described by random variables.
Observations of the phenomenon are drawn from the unknown joint distribution
induced by the random variables and form the dataset or just _data_. A _ML
model_ is a mathematical model optimized with the aid of a learning algorithm
applied on the collected data in order to accurately predict unknown or
withheld phenomenon observations, i.e. to generalize beyond the initial data.
Here we focus on the supervised learning setting. Finally, _scientific
inference_ is the process of rationally deriving conclusions about a
phenomenon from data (via ML, or other types of models). We employ _inference_
to imply investigating unobserved variables and parameters similar to
statistical inference, i.e. in a more general sense than is common in some of
the ML literature, where it is used exclusively as a synonym for prediction.
The knowledge gained by scientific inference can build the basis of
_scientific explanations_. These brief conceptual remarks are meant to reduce
ambiguity in our usage: we lay no claim as to their universality.

##  2 Related Work

Whether and how ML models, and specifically IML, can help obtain knowledge
about the world is a debated topic among philosophers of science,
statisticians, and also the IML community.

#### Philosophy of Science.

It has been argued that ML models are only suitable for prediction because
their parameters are instrumental and lack meaning (Bailer-Jones and Bailer-
Jones, 2002, Bokulich, 2011). On the other hand, Sullivan, (2020) argues that
nothing prevents us from gaining real-world knowledge with ML models as long
as the _link uncertainty_ — the connection between the phenomenon and the
model — can be assessed. Cichy and Kaiser, (2019) and Zednik and Boelsen,
(2022) claim that IML can help in learning about the real world, but they
remain vague about how model and phenomenon are connected. Like Watson,
(2022), we explain that IML methods relying on conditional sampling are
faithful to the phenomenon. However, while he assigns IML inferences to the
causal phenomenon level, we clarify that, without additional assumptions, such
inferences only reveal associational relationships (Räz, 2022). Our work makes
precise that ML models can be described as epistemic representations of a
certain phenomenon that allow us to perform valid inferences (Contessa, 2007)
via interpretations.

#### Statistical Modeling and Machine Learning.

Breiman et al., (2001) describes ML (algorithmic modeling) and statistics
(data modeling) as two approaches for reaching conclusions from data. On a
medical example he shows that post-hoc analysis of ML models can allow more
correct inferences about the underlying phenomenon than standard, inherently
interpretable data models. Our paper gives an epistemic foundation for such
post-hoc analyses. Shmueli et al., (2010) distinguishes statistics and ML by
their goals — prediction (ML) and explanation (statistics). Like Hooker and
Mentch, (2021), we argue against such a clear distinction and offer steps to
integrate the two fields.
This paper builds on ideas from Molnar et al., (2021), where they introduce
ground-truth and confidence intervals for partial dependence plots (PDP) and
permutation feature importance (PFI) of arbitrary ML models. Our work
generalizes these ideas to arbitrary IML methods and draws the connection to
the underlying phenomenon.

#### Interpretable Machine Learning.

IML as a field has been widely criticized for being ill-defined, mixing
different goals (e.g. transparency and causality), conflating several notions
(e.g. simulatability and decomposability), and lacking a proper measure of
success (Doshi-Velez and Kim, 2017, Lipton, 2018). Some even argued against
the central IML leitmotif of analyzing trained ML models post hoc in order to
explain them (Rudin, 2019). In this paper, we show that, if we focus on
interpretations for scientific inference, a clear foundation including a
proper theory of success can be provided and these criticisms can be partially
addressed.

##  3 Scientific Inference and Elementwise Representationality

The goal of this paper is to analyze and describe how we can conduct
scientific inference on ML models using IML methods. This section explains why
inference with ML models cannot be done as in traditional scientific models
and why current IML methods do not generally address the problem. The next
section describes our solution and illustrates it with a complete example from
question formulation to uncertainty quantification.

###  3.1 ML Models are not Elementwise Representational

In scientific modeling, there is a paradigm that many models implicitly follow
— we call it the paradigm of _elementwise representationality_.

######  Definition.

A model is _elementwise representational_ (ER) if all model elements
(variables, relations, and parameters) represent an element in the phenomenon
(components, dependencies, properties).

Figure 1 depicts the relationship between ER models and the phenomenon:111See
Appendix A for the philosophical origins of our perspective. variables
describe phenomenon components; mathematical relations between variables
describe structural, causal or associational dependencies between components;
parameters specify the mathematical relations and describe properties of the
component dependencies. The upward arrows describe _encoding_ i.e. the
translation of a phenomenon observation to a model configuration; The downward
arrows describe _decoding_ i.e. the translation of knowledge about the model
into knowledge about the phenomenon.

Figure 1: Model and phenomenon sustain an encoding-decoding relationship. The
main elements of a traditional, ER model, are shown in encoding-decoding
correspondence to the phenomenon elements they represent (Stachowiak, 1973).
Phenomenon and model elements are illustrated with a simple example of two
bodies in gravitational interaction and its classical, Newtonian mechanistic
description.

ER is obtained through model construction; ER models are usually “hand-
crafted” based on background knowledge and an underlying scientific theory.
Variables are selected carefully and sparsely during model construction, and
the relations are constrained to a relation class with few free parameters.
When ER models need to account for an additional phenomenon aspect, they are
gradually extended so that large parts of the “old” model are preserved in the
more expressive “new” model. ER even eases this model extension process
because model interventions are intelligible on the level of model elements.
Usually, ER is explicitly enforced in modeling: if there is a phenomenon
element devoid of meaning, researchers either try to interpret it or exclude
it from the model.
ER is so remarkable because it gives models capabilities that go beyond
prediction. ER simplifies the step of decoding i.e. translating model
knowledge into phenomenon knowledge. Scientists can analyze model elements and
draw immediate conclusions about the represented phenomenon element (Frigg and
Nguyen, 2021). However, only those aspects of the phenomenon that have a model
counterpart can be analyzed with this approach. Fortunately, as described
above, ER models can be extended to account for further relevant aspects
identified by the scientist.

Running example:222Since the physical model from Figure 1 is a mechanistic
causal model (Schölkopf et al., 2021), we switch henceforth to an illustrative
associational model from the social sciences that compares more fairly with
current associational ML models. We strongly simplify things in this example
and do not claim that it reflects social science methodology or that ML is
even required. Linear Model.  Suppose a researcher, we call her Laura, wants
to study what attributes influence students’ grades in mathematics.
Specifically, she wants to research how language skills and math skills are
associated. She uses data from Cortez and Silva, (2008), who collected a
dataset333see Appendix B for more details., encompassing 32 student attributes
in Portuguese schools including math/Portuguese grades, age, parents’
education, etc.
Laura starts with a classical ER model — a linear model with one predictor and
one target variable. She selects the student grade in Portuguese
$\boldsymbol{X}_{p}$ and in mathematics $\boldsymbol{Y}$ as her proxy
variables for students language and math skills respectively.444In the
Portuguese grading scheme, the range is 0-20, where 0 is the worst and 20 the
best grade. Based on her background knowledge, she assumes that the true
relationship can be described as
$\boldsymbol{Y}=\beta_{0}+\beta_{1}\boldsymbol{X}_{p}+\epsilon$ with
$\beta_{0},\beta_{1}\in\mathbb{R}$ and an error
$\epsilon\sim\mathcal{N}(\mu,\sigma^{2})$. Laura centers $\boldsymbol{X}_{p}$
by the average student grade in Portuguese and obtains the prediction model
that minimizes the mean-squared-error (MSE),

$\hat{m}_{\mathrm{LIN}}(x_{p})=10.46+0.77x_{p}.$

Laura’s model is ER: she can interpret $\hat{\beta}_{0}=10.46$ as the
predicted math grade for an average Portuguese student (if
$x_{p}=12.55$)555Centering features is common in linear regression to make the
intercept term interpretable. and $\hat{\beta}_{1}=0.77$ as the strength of
association between the Portuguese grade and the math grade.
Laura can analyze the model to draw scientific inferences about the underlying
phenomenon, for example, with 95% confidence intervals666i.e., intervals
$[a,b]$ such that, if the model assumptions hold, a ‘true’ parameter $\beta$
is found inside 95% of all observational samples,
$\mathbb{P}(a{<}\beta{<}b)=0.95$. for her estimates $\hat{\beta}_{0}$ and
$\hat{\beta}_{1}$ with $[10.05;10.88]$ and $[0.63;0.91]$ respectively. The
inference she draws is on these parameters; Laura can only draw conclusions
about the phenomenon if the model is ER and highly predictive. Laura may
conclude from $\hat{\beta}_{1}$ that language skills and math skills are
strongly and positively related. To reach a more expressive and predictively
accurate model, Laura can also extend the model to include additional
features, relations, or interaction terms. As long as she preserves ER, she
can directly draw scientific inferences from analyzing model elements. Indeed,
these inferences are only as valid as the modeling assumptions (e.g. target
normality, homoscedasticity, or linearity).

#### ML Models are generally not ER.

ER makes model elements interpretable and allows to reason about the effects
of model or even real-world interventions; as such, ER models suit our image
of science as an endeavor aimed at understanding. However, as mentioned above,
for ER we usually require background knowledge on which components are
relevant, and we need to severely restrict the class of relations that can be
considered for the given phenomenon. These difficulties might lead scientists
to either limit their investigations to phenomena that are already well
studied or, as Breiman et al., (2001) argued, to develop overly simple models
for complex phenomena and possibly draw wrong conclusions.
ML models excel for complex problems with an unbounded number of components
that display ambiguous and entangled relationships i.e. ML models are highly
expressive (Gühring et al., 2020). ML models are subject-domain independent
(Bailer-Jones and Bailer-Jones, 2002), this means that we do not necessarily
need subject-domain background knowledge in modeling. Instead, ML modeling
only requires specifying a broad model class and a set of hyperparameters. The
choice of these hyperparameters is data-domain specific i.e. they reflect
inductive biases that allow for efficient learning.
The gain in generality and convenience with ML comes at a price — ML models
are generally not ER. As also argued in Boge, (2022), Bokulich, (2011),
Bailer-Jones and Bailer-Jones, (2002), ML models (e.g. artificial neural
networks) contain model elements such as weights, activation functions, or
network structure that have no corresponding phenomenon counterpart.

Running example: Artificial Neural Network (ANN).  Suppose Laura is
dissatisfied with her linear model and fits a dense three-layer neural network
to predict math grades using all available features.777We chose a neural net
to make our argument. For training the neural network, Laura splits data into
training and test, uses ReLu activation functions and minimizes the MSE loss
via gradient descent with an adaptive learning rate. She reduces the test-set
MSE from $16.0$ in the linear-one-variable model case to $8.9$. A formal
description of the model is given by:

$\hat{m}_{\mathrm{ANN}}(\bm{x})=\sigma_{3}(W_{3}\sigma_{2}(W_{2}\sigma_{1}(W_{1}\bm{x}+b_{1})+b_{2})+b_{3})$

where model elements are the values of the weight matrices $W_{1}$, $W_{2}$,
$W_{3}$ and bias vectors $b_{1},b_{2},b_{3}$, and the activation functions
$\sigma_{1},\sigma_{2},\sigma_{3}$. Unlike in the linear model above, it is
highly unclear what these parameters correspond to in our data or phenomenon.
While the input vector $x$ is still representational, the weights, activation
functions, or three-layer architecture are very hard or even impossible to
interpret: A high value of weight $W_{1}^{(3,2)}$ might have a positive,
neutral, or negative effect on the target, dependent on all other model
elements; the activation function only reflects the currently popular
heuristics in model training; and the particular three-layer architecture is a
result of model selection based on predictive performance and rules of thumb,
but with little phenomenon-based rationale.

###  3.2 Scientific Inference in Light of Current IML

We have argued so far that:

  1. i)

If models are ER, they allow for scientific inference.

  2. ii)

ML models are generally not ER.

How can we still do scientific inference with ML models? We discuss two
strategies to enable scientific inference with ML: We argue that the first
strategy, namely restoring ER, fails because ML models are designed to
represent in a distributed manner; the second strategy, embracing holistic
representationality, is highly promising but current attempts conflate
different goals of model analysis. This discussion sets the stage for the next
section, where we show how a holistic account of representationality can
enable scientific inference.

#### Restore ER.

One strategy towards scientific inference with ML is to challenge Proposition
ii) and show that ML models are ER too. Researchers in this camp argue that
individual elements in ML have a natural phenomenon counterpart, but this
counterpart only becomes evident when these model elements are extensively
scrutinized.888The underlying epistemological reasoning is that human
representations are near-optimal and will be eventually rediscovered by ML
algorithms. This would be surprising: ER is not enforced in state-of-the-art
techniques and, even worse, some methods such as training with dropout
purposefully discourage ER in order to gain robustness (Srivastava et al.,
2014); ML models like ANNs are designed for _distributed representation_
(Buckner and Garson, 2019, McClelland et al., 1987).
It has been claimed that model elements represent high-level constructs
constituted from low-level phenomenon components that are often called
_concepts_ (Buckner, 2018, Olah et al., 2020).999The idea is that similar to
the hierarchical structure of components in nature, where lower level
components such as atoms combine to form higher level entities such as
molecules, cells, and organisms; in deep nets, hierarchies evolve from pixels
to shapes to objects. If this is the case, model elements or aggregates of
such elements can be reconnected to the phenomenon; ER would be restored by
the representations of coarse-grained phenomenon components. Research on
neural networks supports that some model elements are associated with concepts
(Mu and Andreas, 2020, Voss et al., 2021, Kim et al., 2018, Olah et al.,
2017), however, often these elements are neither the only associated elements
nor exclusively associated with one concept as shown in Figure 2 (Donnelly and
Roegiest, 2019, Bau et al., 2017, Olah et al., 2020). Problematically,
intervening on these model elements generally does not have the expected
effect on the prediction — the elements do not share the causal role of the
“represented” concepts, even in prediction (Gale et al., 2020, Donnelly and
Roegiest, 2019). It is therefore questionable in what sense they still
represent.101010Though note generative adversarial networks as an exception;
here, interventions on model elements have been linked to interventions on
concepts in the generated images (Bau et al., 2018). Moreover, this line of
research predominantly focuses on images, where nested concepts are arguably
easier to identify for humans.

Figure 2: ML models are generally not ER. Three input images that
independently trigger a single model element (unit 55 in layer mixed4e, Olah
et al., (2017; 2020)). A single unit in a neural net may respond to very
different “concepts”, e.g. heads of cats (left image), car bodies (center), or
bees (right), suggesting that units generally do not represent disentangled
concepts (Mu and Andreas, 2020, Nguyen et al., 2016).

Research on the representational correlates of model elements seems indeed
fascinating. However, current ML models that do not enforce ER will rely on
distributed representations and cannot be reduced to logical concept machines.
The associative connection between model elements and phenomenon concepts
should not be confused with their equivalence. Analyzing single model elements
will therefore be a hopeless enterprise.

#### Embrace Holistic Representationality.

An alternative route to scientific inference is to accept that ER is well-
suited for scientific inference and that ML models are not ER but reject that
ER is the only approach for scientific inference. To choose this route, one
must offer an alternative path for drawing scientific inference with ML models
that goes beyond the analysis of model elements.
Our approach is to regard the model as representational of phenomenon aspects
_only as a whole_ — we call this _holistic representationality_ (HR). HR
implicitly underlies large parts of the current research program in IML:
Model-agnostic methods, in particular, analyze the entire ML model simply as
an input-output mapping (Scholbeck et al., 2019); In the same spirit, many
model-specific IML methods like gradient or path-based feature attribution
treat ML models as mappings with additional useful properties such as
differentiability (Alqaraawi et al., 2020).
Model-agnostic and model-specific methods share the idea that relevant model
properties such as the effects or importances of variables can be derived by
analyzing the model just as a functional mapping. Initial definitions of, for
example, global feature effects (Friedman et al., 1991) and feature importance
(Breiman, 2001) or local feature contribution (Štrumbelj and Kononenko, 2014)
and model behavior (Ribeiro et al., 2016) have been presented. However, many
researchers have pointed out that these methods lead to counterintuitive
results for dependent or interacting features and offered alternative
definitions (Apley and Zhu, 2020, Strobl et al., 2008, Molnar et al., 2020b ,
Goldstein et al., 2015, König et al., 2021b , Janzing et al., 2020, Slack et
al., 2020, Alqaraawi et al., 2020).
We believe that these controversies stem from a lack of clarity about the goal
of model analysis. Are we interested in model properties to learn about the
model (model audit) or do we want to use these model properties as a gateway
to learn about the underlying phenomenon (scientific inference)? These two
goals must not be conflated.
The auditor examines model properties e.g. for debugging, to check if the
model satisfies legal or ethical norms, or to improve her understanding of the
model by intervening on it (Raji et al., 2020). Auditors even take interest in
model properties that have no corresponding phenomenon counterpart such as
single model elements or the model behavior for unrealistic feature
combinations. The scientist who wants to draw inferences, on the other hand,
wants to learn about model properties that can be interpreted in terms of the
phenomenon.
Scientific inference and model audit should be viewed as two different but
interacting goals. In each of them, we take different stances toward the ML
model: The auditor adopts a skeptical attitude of the model, she has ground-
truth information or normative standards to check the model against; the
scientist adopts a trusting attitude, she wants to learn from the model. Both
cases describe a knowledge asymmetry (Gobet, 2018, Rosser et al., 2008) but in
opposite directions. Auditing the model is an indispensable step for
scientists to gain enough trust in it. Only after several rounds of auditing
and improvement should the researcher rely on the model to draw scientific
conclusions.

##  4 Scientific Inference with IML Property Descriptors

We just argued that ML models are generally not ER and therefore do not allow
for scientific inference in the standard way. HR offers a viable alternative,
but currently different goals of model analysis are conflated. In this
section, we show that a HR perspective enables scientific inference using IML
methods. Particularly, we show that certain IML methods — we call them _IML
property descriptors_ — can represent phenomenon properties. Figure 3
describes our conceptual move: instead of matching phenomenon properties with
model parameters as in ER models, we match them with external descriptions of
the whole model.

Figure 3: IML property descriptions distill phenomenon properties from HR
models. Instead of explicitly encoding phenomenon properties as parameters
like for ER models, HR models (e.g. ML models) encode phenomenon properties in
the whole model. We propose that these encoded properties can be read out with
IML property descriptions external to the model. In this way, IML offers an
indirect route to scientific inferences through model analysis.

#### Idea.

Instead of first thinking about the model and its properties (the model audit
approach), we propose to start with the phenomenon and a scientific question
about it. IML methods for inference should answer, or at least help answer, a
scientific question concerning the phenomenon. The crucial step in our
framework is to establish a link between the phenomenon and the model; we
propose to draw this link using statistical decision theory, which shows what
optimal ML models can holistically represent. Clearly, an approximate ML model
will not provide an answer if even the optimal model cannot. However, if a
question can in principle be answered with the optimal model, an ML model
trained on data can approximate the answer in practice. The problem then
becomes to quantify the approximation error.

###  4.1 ML Representationality and Optimal Predictors

Which aspects of a phenomenon ML models can represent even under ideal
circumstances depends on the data, the learning paradigm, and the loss
function. For identically and independently distributed (i.i.d.) data used for
supervised learning, optimal predictors from statistical decision theory
provide an answer (Hastie et al., 2009, p18-22). Besides the advanced theory
available in this setting, supervised learning on i.i.d. data is the most
popular ML setup in practical applications. We briefly discuss
representationality and scientific inference in the case of unsupervised and
causal learning in Section 5.3.

#### Basic Notation.

We assume that the random variables
$\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n}$ and $\boldsymbol{Y}$ fully
characterize the phenomenon. We write the joint feature vector as
$\boldsymbol{X}\coloneqq(\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n})$ with
$\mathcal{X}\coloneqq\mathrm{Range}(\boldsymbol{X})$ and
$\mathcal{Y}\coloneqq\mathrm{Range}(\boldsymbol{Y})$. $\boldsymbol{X}$ and
$\boldsymbol{Y}$ jointly describe the phenomenon.

#### Optimal Predictors.

An optimal predictor $m$ can predict realizations of the target
$\boldsymbol{Y}$ from realizations of $\boldsymbol{X}$ with minimal expected
prediction error i.e.
$m=\underset{\hat{m}\in\mathcal{M}}{\mathop{\mathrm{arg}\,\mathrm{min}}}\,\mathrm{EPE}_{\boldsymbol{Y}|\boldsymbol{X}}(\hat{m})$,
with
$\mathrm{EPE}_{\boldsymbol{Y}|\boldsymbol{X}}(\hat{m})\coloneqq\int\limits_{\boldsymbol{Y}}\
L(\boldsymbol{Y},\hat{m}(\boldsymbol{X}))\
\mathbb{P}_{\boldsymbol{Y}|\boldsymbol{X}}(y|x)\,{\rm d}y$, where $L$
describes a loss function
$L(\boldsymbol{Y},m(\boldsymbol{X})):\mathcal{X}\times\mathcal{Y}\to\mathbb{R}^{+}$
and $\hat{m}$ a model in the set $\mathcal{M}$ of mappings from $\mathcal{X}$
to $\mathcal{Y}$. Table 1 shows the optimal predictors for standard loss
functions.

S Problem | Loss | $L(\boldsymbol{Y},\hat{m}(\boldsymbol{X}))$ | Optimal Predictor $m$
---|---|---|---
Regression |  S$\underset{y\in\mathcal{Y}}{\mathop{\mathrm{arg}\,\mathrm{max}}}$ Mean Squared Error | $(\boldsymbol{Y}-\hat{m}(\boldsymbol{X}))^{2}$ | $\mathbb{E}_{\boldsymbol{Y}\mid\boldsymbol{X}}[\boldsymbol{Y}\mid\boldsymbol{X}]$
S$\underset{y\in\mathcal{Y}}{\mathop{\mathrm{arg}\,\mathrm{max}}}$ Mean Absolute Error | $|\boldsymbol{Y}-\hat{m}(\boldsymbol{X})|$ | $\mathrm{Median}(\boldsymbol{Y}\mid\boldsymbol{X})$
Classification |  S$\underset{y\in\mathcal{Y}}{\mathop{\mathrm{arg}\,\mathrm{max}}}$ 0-1 Loss |  $0$ if $\hat{m}(\boldsymbol{X})=\boldsymbol{Y}$, else $1$ | $\underset{y\in\mathcal{Y}}{\mathop{\mathrm{arg}\,\mathrm{max}}}\;\mathbb{P}(\boldsymbol{Y}{=}y\mid\boldsymbol{X})$
S$\underset{y\in\mathcal{Y}}{\mathop{\mathrm{arg}\,\mathrm{max}}}$ KL divergence |  $\sum\limits_{r\in\mathcal{Y}}\mathbb{P}_{\boldsymbol{Y}}(r)\log\Bigl{(}\frac{\mathbb{P}_{\boldsymbol{Y}}(r)}{\mathbb{P}_{\hat{m}(\boldsymbol{X})}(r)}\Bigr{)}$111111This describes the forward KL divergence $\mathrm{KL}(\boldsymbol{Y}||{\hat{m}(\boldsymbol{X})})$ for discrete $\boldsymbol{Y}$ and $\hat{m}(\boldsymbol{X})$, which differs from the backward KL divergence $\mathrm{KL}({\hat{m}(\boldsymbol{X})}||\boldsymbol{Y})$. | $\mathbb{P}(\boldsymbol{Y}\mid\boldsymbol{X})$
Table 1:  The optimal predictors for standard loss functions can be derived
from $\mathbb{P}(\boldsymbol{Y}\mid\boldsymbol{X})$.

#### Supervised learning.

Supervised learning seeks to find an optimal predictor $m$ by using a learning
algorithm121212The domain of $I$ is only completely specified when the
parameters that define the learning procedure and the search space of the
algorithm (called hyperparameters in the context of $\hat{m}$) are fixed. For
our discussion, the reader may assume hyperparameters to have been set a
priori by a human or an automated ML algorithm (Hutter et al., 2019).
$I{:}\Delta\rightarrow\mathcal{M}$ that selects a a model $\hat{m}$ from a set
$\mathcal{M}$ with the aid of a dataset
$\mathcal{D}\coloneqq\bigl{(}(x^{(1)},y^{(1)}),\dots,(x^{(k)},y^{(k)})\bigr{)}$
with $\mathcal{D}$ in the set of datasets $\Delta$ drawn i.i.d. from the joint
distribution, i.e. $(x^{(i)},y^{(i)})\sim(\boldsymbol{X},\boldsymbol{Y}$).
Instead of the EPE itself, the learning algorithm minimizes the empirical risk
on the test data (i.e. on data not used to train $\hat{m}$), which is a
finite-data estimate of the EPE.

###  4.2 IML Property Descriptors

We have just argued that ML models, when considered as a whole, approximate
phenomenon aspects that can be derived from the conditional distribution
$\mathbb{P}(\boldsymbol{Y}\mid\boldsymbol{X})$. _IML property descriptors_ can
help to investigate these aspects by describing their relevant properties.

#### Five Steps Towards IML Methods for Inference.

Our proposal consists of the five steps in Figure 4, which we now discuss in
detail. For each of the five steps, we provide an inference example based on
the prediction of student grades in mathematics. In what follows, we assume
that we have a supervised learning ML model $\hat{m}$ that approximates a
phenomenon aspect described by the optimal predictor $m$.

Figure 4: An epistemic foundation for scientific inference with IML. Steps 1
and 2 clarify _what_ kinds of scientific inferences we can draw with IML.
Steps 3, 4, and 5 show _how_ to draw such inferences and provide estimates of
their precision.

#### Step 1) Formalize Scientific Question.

Science starts by formulating a question. To address it with ML, this question
has to be formalized. Exemplary questions that can already be addressed with
IML methods following the scheme below are discussed in Section 4.3. Note that
IML for scientific inference only helps answer questions that concern the
association between $\boldsymbol{X}$ and $\boldsymbol{Y}$.

Formally. We denote the formalized question by $Q$.

Example. Suppose Laura wants to find out how students’ language skills are
related to their math skills. She approaches this problem by asking how
students’ expected math grades are related to their Portuguese grades. Laura
formalizes this question as the conditional expectation i.e.
$Q=\mathbb{E}_{\boldsymbol{Y}\mid\boldsymbol{X}_{p}}[\boldsymbol{Y}\mid\boldsymbol{X}_{p}]$,
where $\boldsymbol{Y},\boldsymbol{X}_{p}$ respectively stand for the math and
Portuguese grade variable.131313This conditional expectation is the best
possible point estimate of the math grade under the MSE loss, given just the
Portuguese grades.

#### Step 2) Establish Question Identifiability.

Many scientific questions cannot be addressed using an ML model. ML models can
only help answer questions that could theoretically be addressed with the
optimal predictor. We call a question that the optimal predictor can help
answer together with additional probabilistic knowledge (e.g. aspects of
$\mathbb{P}(\boldsymbol{X},\boldsymbol{Y})$) _identifiable_. A constructive
strategy to establish identifiability relative to some probabilistic knowledge
is to think of the transformations, using solely the probabilistic knowledge,
that take the optimal predictor into the question $Q$. Of course, it is
desirable to keep to a minimum the amount of probabilistic knowledge required
to identify the question.

Formally. The optimal predictor is denoted by $m$. We say that a question is
_identifiable_ relative to probabilistic knowledge $K$ if we can compute $Q$
from $m$ and $K$.

Example.  Assume that Laura has trained her neural network, which, unlike the
simple linear model presented above, takes into account _all_ available
features $\boldsymbol{X}$, to minimize the MSE loss, i.e.
$m(\boldsymbol{X})=\mathbb{E}_{\boldsymbol{Y}\mid\boldsymbol{X}}[\boldsymbol{Y}\mid\boldsymbol{X}]$.
Is Laura’s question identifiable? For specific values of $\boldsymbol{X}$, the
optimal predictor allows to compute the expected value of $\boldsymbol{Y}$
i.e.
$m(x)=\mathbb{E}_{\boldsymbol{Y}\mid\boldsymbol{X}}[\boldsymbol{Y}\mid\boldsymbol{X}{=}x]$.
The only difference to $Q$ is that $m$ takes into account features besides the
Portuguese grade, that we denote $\boldsymbol{X}_{-p}$. If we have access to
the conditional distribution
$\mathbb{P}(\boldsymbol{X}_{-p}\mid\boldsymbol{X}_{p})$ (required141414Usually
we do not have access to probabilistic knowledge $K$. We discuss this in more
detail in Step 4. probabilistic knowledge $K$), we can integrate these other
features out by taking the expected value

$\displaystyle Q$
$\displaystyle\coloneqq\mathbb{E}_{\boldsymbol{Y}\mid\boldsymbol{X}_{p}}[\boldsymbol{Y}\mid\boldsymbol{X}_{p}]$
$\displaystyle=\mathbb{E}_{\boldsymbol{X}_{-p}\mid\boldsymbol{X}_{p}}[\mathbb{E}_{\boldsymbol{Y}\mid\boldsymbol{X}}[\boldsymbol{Y}\mid\boldsymbol{X}]\mid\boldsymbol{X}_{p}]\qquad\text{(by
the \emph{tower rule}, see App. \ref{app:tower}})$
$\displaystyle=\mathbb{E}_{\boldsymbol{X}_{-p}\mid\boldsymbol{X}_{p}}[m(\boldsymbol{X})\mid\boldsymbol{X}_{p}].$

Thus, $Q$ is identifiable via $m$ given
$K=\mathbb{P}(\boldsymbol{X}_{p}\mid\boldsymbol{X}_{-p})$.

#### Step 3) Design IML Property Descriptor.

It is not enough to identify a question. We need a way to estimate an answer
for ML models — we need _IML property descriptors_. An IML property descriptor
describes a continuous function that applies the transformation from the
question identification step above to a given ML model and outputs an element
of the space $\mathcal{Q}$. Thus, given the optimal predictor, an IML property
descriptor outputs an answer to $Q$. Continuity guarantees that if our ML
model is close to the optimal model, our answer is approximately correct. We
call the application of a property descriptor to a specific ML model,
$g_{K}(\hat{m})$, a _model property description_.

Formally. An _IML property descriptor_ is a continuous function $g_{K}$
(w.r.t. metrics $d_{\mathcal{M}}$ and $d_{\mathcal{Q}}$)151515The function
$d_{\mathcal{M}}$ is a metric on the function space $\mathcal{M}$,
$d_{\mathcal{M}}(m_{1},m_{2})\coloneqq\int\limits_{\boldsymbol{X}}L(m_{1}(x),m_{2}(x))\,\mathbb{P}_{\boldsymbol{X}}(x)\
{\rm d}x$ for $m_{1},m_{2}\in\mathcal{M}$, while $d_{\mathcal{Q}}$ describes a
metric appropriate for the space $\mathcal{Q}$. that identifies $Q$ using
probabilistic knowledge $K$:

$\displaystyle g_{K}:\mathcal{M}$ $\displaystyle\to\mathcal{Q}\quad\text{ with
}\quad g_{K}(m)=Q.$

The output space $\mathcal{Q}$ remains unspecified to account for the variety
of scientific questions; $\mathcal{Q}$ could denote a set of real numbers,
vectors, functions, probability distributions, etc.

Example. The property descriptor describes the transformations that identify
$Q$, i.e.

$g_{K}(\hat{m})(x_{p})\coloneqq\mathbb{E}_{\boldsymbol{X}_{-p}\mid\boldsymbol{X}_{p}}[\hat{m}(\boldsymbol{X})\mid\boldsymbol{X}_{p}{=}x_{p}].$
(4.1)

This is indeed a property descriptor because conditional expectation is
continuous on $\mathcal{M}$, and $Q$ is identifiable given
$K=\mathbb{P}(\boldsymbol{X}_{-p}\mid\boldsymbol{X}_{p})$. Note that Equation
4.1 describes the well-known conditional partial dependence plot, or cPDP,
also known as M-plot (Molnar, 2020, Apley and Zhu, 2020).

#### Step 4) Estimate IML Property Descriptor.

Often we lack access to relevant probabilistic knowledge $K$. Instead, we have
a finite amount of data on which we can evaluate our ML mapping, which we call
the _evaluation data_. It may bundle up our training and test data
$\mathcal{D}$ (see Section 4.1), as well as additionally available (unlabeled)
data, and artificially generated data. The _IML property description
estimator_ describes a way to estimate property descriptions with access only
to the ML model plus the evaluation data.

Formally. We denote the _evaluation dataset_ by $\mathcal{D}^{*}$ and the
random process that generates it by $\boldsymbol{D}^{*}$. We call
$\hat{g}_{\mathcal{D}^{*}}:\mathcal{M}\rightarrow\mathcal{Q}$ the _IML
property description estimator_ if it is an unbiased estimator of $g_{K}$ i.e.

$\mathbb{E}_{\boldsymbol{D}^{*}}[\hat{g}_{\boldsymbol{D}^{*}}(\hat{m})]=g_{K}(\hat{m})\quad\text{for
all }\hat{m}\in\mathcal{M}.$

Example. Laura’s evaluation dataset $\mathcal{D}^{*}$ is her initial training
and test dataset $\mathcal{D}$ augmented by artificial instances created by
the following manipulation: Laura makes six copies of the data, and jitters
the Portuguese grade by $1,-1,2,-2,3$ or $-3$ respectively. This augmentation
strategy reflects how Laura understands the Portuguese grade as noisy based on
her background knowledge of how much student performance varies daily and
teachers grade inconsistently. Let the students with (jittered) Portuguese
grade $i$ be $\mathcal{D}^{*}_{|x_{p}=i}\coloneqq(x\in\mathcal{D}^{*}\mid
x_{p}=i)$, then, we can define the IML property description estimator at $i$
as the conditional mean (unbiased estimator of the conditional expectation):

$\hat{g}_{\mathcal{D}^{*}}(\hat{m})(i)\coloneqq\frac{1}{|\mathcal{D}^{*}_{|x_{p}{=}i}|}\sum\limits_{x\in\mathcal{D}^{*}_{|x_{p}=i}}\hat{m}(x)$
(4.2)

The estimated answer to Laura’s question is plotted in Figure 5. The plot on
the left suggests that math grade is only strongly dependent on Portuguese
grades in the interval $8-17$. However, as we show in the next step, we must
also take into account that we have very sparse data in some regions (e.g.
very few students scored below $8$) before confirming this first impression.

Figure 5: Left Plot:  Estimate of
$\mathbb{E}_{\boldsymbol{Y}\mid\boldsymbol{X}_{p}}[\boldsymbol{Y}\mid\boldsymbol{X}_{p}]$
via Equation 4.2. Right Plot:  Histogram of grades in Portuguese.

#### Step 5) Quantify Uncertainty.

We have shown how we can estimate $Q$ using an approximate ML model paired
with a suitable evaluation dataset. But how good is our estimate? Two steps
involve approximations:

  1. 1.

Applying the IML property descriptor to the ML model $\hat{m}$ instead of the
optimal model $m$; we call the resulting error _model error_

$\mathrm{ME}[\hat{m}]=d_{\mathcal{Q}}\Bigl{(}g_{K}(m),g_{K}(\hat{m})\Bigr{)}.$

The model error depends on the ML model $\hat{m}$ we obtained from the
learning algorithm trained on the given dataset.

  2. 2.

Applying the IML property description estimator on our evaluation dataset
$\mathcal{D}^{*}$ instead of computing the true model property description
based on $K$ directly; we call this error _estimation error_

$\mathrm{EE}[\mathcal{D}^{*}]=d_{\mathcal{Q}}\Bigl{(}g_{K}(\hat{m}),\hat{g}_{\mathcal{D}^{*}}(\hat{m})\Bigr{)}.$

The estimation error depends on the evaluation dataset $\mathcal{D}^{*}$.

In theory, the model error and the estimation error can be separated. In
practice, however, they are statistically dependent because the training and
the evaluation data overlap. Generally, neither the model error nor the
estimation error can be computed perfectly; this would require access to the
optimal model $m$ and infinitely many data instances. Nevertheless, we can
quantify in expectation how large the two errors are.
An intuitive approach to quantifying the expected errors is to decompose them
into bias and variance contributions. The two decompositions below quantify
the range in which the true phenomenon property descriptions are most likely
to lie.

Formally.  For the bias-variance decomposition, we assume the metric
$d_{\mathcal{Q}}$ to be the squared error.161616A bias-variance decomposition
is also possible for other loss functions, including the 0-1 loss (Domingos,
2000). Considering the dataset that we entered into the learning algorithm as
a random variable $\boldsymbol{D}$, we can decompose the expected
$\mathrm{ME}[\hat{m}]$ error as follows

$\mathbb{E}_{\boldsymbol{D}}[\mathrm{ME}[\hat{m}]]=\underbrace{(g_{K}(m)-\mathbb{E}_{\boldsymbol{D}}[g_{K}(\hat{m})])^{2}}_{\mathrm{Bias}^{2}}\;+\;\underbrace{\mathbb{V}_{\boldsymbol{D}}[g_{K}(\hat{m})]}_{\mathrm{Variance}}$

where $\hat{m}\coloneqq I(\mathcal{D})$ is the output of a machine learning
algorithm $I$ for dataset $\mathcal{D}$ (Section 4.1). Considering the
evaluation data as a random variable $\boldsymbol{D}^{*}$, we can decompose
the expected $\mathrm{EE}_{D^{*}}$ error as follows

$\mathbb{E}_{\boldsymbol{D}^{*}}[\mathrm{EE}[D^{*}]]=\underbrace{(g_{K}(\hat{m})-\mathbb{E}_{\boldsymbol{D}^{*}}[\hat{g}_{\boldsymbol{D}^{*}}(\hat{m})])^{2}}_{\mathrm{Bias}^{2}}\;+\;\underbrace{\mathbb{V}_{\boldsymbol{D}^{*}}[\hat{g}_{\boldsymbol{D}^{*}}(\hat{m})]}_{\mathrm{Variance}}=\mathbb{V}_{\boldsymbol{D}^{*}}[\hat{g}_{\boldsymbol{D}^{*}}(\hat{m})].$

The bias term vanishes because the property description estimator is by
definition unbiased w.r.t. the IML property descriptor.

Example. Laura obtains different cPDPs (Figure 5) for different models with
similar performance, as well as for different selections of evaluation data,
how much can she then rely on these cPDPs?
The estimates of the variances of the cPDP by Molnar et al., (2021) allow to
calculate pointwise confidence intervals (Figure 6). We can define a
confidence interval that only incorporates the estimation uncertainty by

$\mathrm{CI}_{\mathrm{EE}[\boldsymbol{D}^{*}]}\coloneqq\Bigl{[}\hat{g}_{\mathcal{D}^{*}}(\hat{m})(i)\pm
t_{1-\frac{\alpha}{2}}\sqrt{\hat{\mathbb{V}}_{\boldsymbol{D}^{*}}[\hat{g}_{\boldsymbol{D}^{*}}(\hat{m})(i)]}\Bigr{]}$

and a confidence interval that incorporates both model and estimation
uncertainty by

$\mathrm{CI}_{\mathrm{ME}[\hat{m}]\land\mathrm{EE}[\boldsymbol{D}^{*}]}\coloneqq\Bigl{[}\hat{g}_{\mathcal{D}^{*}}(\hat{m})(i)\pm
t_{1-\frac{\alpha}{2}}\sqrt{\hat{\mathbb{V}}_{\boldsymbol{D},\boldsymbol{D}^{*}}[\hat{g}_{\boldsymbol{D}^{*}}(\hat{m})(i)]}\Bigr{]}.$

For the combined confidence interval we require a strong and unfortunately not
testable assumption to be satisfied — unbiasedness of the ML algorithm.
Unbiasedness implies that, in expectation over training sets, the ML algorithm
learns the optimal model, i.e.
$m=\mathbb{E}_{\boldsymbol{D}}[\hat{m}]$.171717Since unbiasedness is tied to a
specific context, there is no conflict with the no-free-lunch theorems
(Sterkenburg and Grünwald, 2021).
Figure 6 shows that for students with Portuguese grades between $8$ and $17$,
Laura can be very confident in her model and the relationship it identifies
between math and Portuguese grade.181818We used resampling techniques to
estimate the two variances. In real-data settings it is generally not possible
to always sample new data for the model training and the evaluation. Although
resampling may result in an underestimation of the variance, our goal here is
simply to illustrate the process of quantifying uncertainty for a concrete IML
method. However, both for Portuguese grades below $8$ or above $17$, the true
value might be far off from our estimated value using a given model, as we can
see from the width of the confidence intervals. For these grade ranges,
gathering more data may reduce Laura’s uncertainty.

Figure 6: Uncertainty evaluation of an IML property description. Left: cPDP
and its estimation error due to Monte-Carlo integration. Right: cPDP with
_both_ estimation and model error. Confidence bands in dashed lines cover the
true expected math grade in 95% of all cases. These plots jointly suggest that
most of the uncertainty is due to the model error.

#### Summary.

Figure 7 gives an overview of all functions and spaces involved in IML for
scientific inference. We started from a phenomenon and formalized a scientific
question $Q$ about it. Using a learning algorithm $I$ on dataset $\mathcal{D}$
from the phenomenon, we learned an ML model $\hat{m}$ that approximates the
optimal model $m$. We then set out to answer $Q$ from $\hat{m}$. We defined a
property descriptor $g_{K}$, that is, a function that allows to compute $Q$
from $m$ given $K$, respectively approximates $Q$ from $\hat{m}$ given $K$.
Because $g_{K}$ requires probabilistic knowledge about
$\mathbb{P}(\boldsymbol{X},\boldsymbol{Y})$, we introduced a property
description estimator $\hat{g}_{\mathcal{D}^{*}}$, a function estimating $Q$
solely from finite data, the evaluation set $\mathcal{D}^{*}$. Finally, we
showed how the expected error of our estimation steps can be quantified with
confidence intervals $\mathrm{CI}_{\mathrm{ME}[\hat{m}]}$ and
$\mathrm{CI}_{\mathrm{EE}[\boldsymbol{D}^{*}]}$.

Figure 7: From datasets to inferences via ML models. Mappings are represented
by arrows and sets are represented by filled circles, with confidence regions
in green shades. Practical IML descriptions
$\hat{g}_{\mathcal{D}^{\star}}(\hat{m})$ are approximate, uncertainty-aware
answers to a question $Q$ that are built from a model $\hat{m}$ fit on
$\mathcal{D}$ and an evaluation dataset $\mathcal{D}^{*}$.

###  4.3 Property Descriptors and Current IML Methods

Many questions that can be answered based on the conditional probability
distribution $\mathbb{P}(\boldsymbol{Y}\mid\boldsymbol{X})$ are widely
relevant. The goal of practical IML research for inference should be to define
relevant descriptors and provide accessible implementations of these
descriptors, including quantification of uncertainty. To find out which
specific questions are relevant to scientists, and therefore what descriptors
are necessary, IML researchers, statisticians and scientists must closely
interact.
In Table 2 we present a few examples of elementary inference questions that
can in principle be addressed by existing IML methods i.e. these methods can
operate as property descriptors already. We distinguish between global and
local phenomenon questions: global questions concern general associations,
local questions concern associations for a specific instance. The last column
highlights current IML methods that provide approximate answers, albeit often
without uncertainty quantification. Note how we ultimately require conditional
versions of existing marginal IML methods, which suggests that marginal
sampling, which generates unrealistic instances, is inadequate in scientific
inference.

Global
---
Question |  Formalization |  IML method
SEffect: What is the best estimate of $\boldsymbol{Y}$ if we only know $\boldsymbol{X}_{p}$? |  $m_{\boldsymbol{X}_{p}}(\boldsymbol{X}_{p})$ |  $\underset{\text{{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{apley2020visualizing}{\@@citephrase{ }}{})}}}}{\text{cPDP}}$
SConditional Contribution: How much worse can $\boldsymbol{Y}$ be predicted from $\boldsymbol{X}$ if we hadn’t known $\boldsymbol{X}_{p}$? |  $\mathrm{EPE}_{\boldsymbol{X},\boldsymbol{Y}}(m_{\boldsymbol{X}}(\boldsymbol{X}))-\mathrm{EPE}_{\boldsymbol{X}_{-p},\boldsymbol{Y}}(m_{\boldsymbol{X}_{-p}}(\boldsymbol{X}_{-p}))$ |  $\underset{\text{{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{fisher2019all}{\@@citephrase{ }}{})}}}}{\text{cPFI}}$
SFair Contribution: What is the fair share of feature $\boldsymbol{X}_{p}$ in the prediction of $\boldsymbol{Y}$? |  $\frac{1}{n}\sum\limits_{S\subseteq\\{1,\dots,n\\}\setminus j}\binom{n-1}{|S|}^{-1}\Bigl{(}\mathrm{EPE}_{\boldsymbol{X}_{S\cup\\{j\\}},\boldsymbol{Y}}(m_{\boldsymbol{X}_{S\cup\\{j\\}}}(\boldsymbol{X}_{S\cup\\{j\\}}))$ mmmmmmmm$-\mathrm{EPE}_{\boldsymbol{X}_{S},\boldsymbol{Y}}(m_{\boldsymbol{X}_{S}}(\boldsymbol{X}_{S}))\Bigr{)}$ |  $\underset{\text{{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{covert2020understanding}{\@@citephrase{ }}{})}}}}{\text{SAGE}}$
SRelevant Value: Under which realistic conditions can we expect to observe relevant value $y_{rel}$? |  $\underset{x\in\mathcal{X}\land\mathbb{P}(\boldsymbol{X}=x)>0}{\mathop{\mathrm{arg}\,\mathrm{min}}}d_{\mathcal{Y}}(m_{\boldsymbol{X}}(x),y_{rel})$ |  no method yet
LocalS
SEffect: How does the best estimate of $\boldsymbol{Y}$ change relative to $\boldsymbol{X}_{p}$, knowing that $\boldsymbol{X}_{-p}=x_{-p}$? |  $m_{\boldsymbol{X}}(\boldsymbol{X}_{p},x_{-p})$ |  $\underset{\text{{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{goldstein2015peeking}{\@@citephrase{ }}{})}}}}{\text{ICE-curve}}$191919Only defined on phenomenon if $\mathbb{P}(X_{p}=p,X_{-p}=x_{-p})>0$.
SConditional Contribution: How much worse can $\boldsymbol{Y}$ be predicted from $\boldsymbol{X}=x$ if we hadn’t known $\boldsymbol{X}_{p}$? |  $L(y,m_{\boldsymbol{X}}(x))-L(y,m_{\boldsymbol{X}_{-p}}(x_{-p}))$ |  no method yet
SFair Contribution: What is the fair share of feature $\boldsymbol{X}_{p}$ in the prediction of $\boldsymbol{Y}$ if $\boldsymbol{X}=x$? |  $\frac{1}{n}\sum\limits_{S\subseteq\\{1,\dots,n\\}\setminus j}\binom{n-1}{|S|}^{-1}\Bigl{(}m_{\boldsymbol{X}_{S\cup\\{j\\}}}(x_{S},x_{j})-m_{\boldsymbol{X}_{S}}(x_{S})\Bigr{)}$ |  conditional $\underset{\text{{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{aas2021explaining}{\@@citephrase{ }}{})}}}}{\text{Shapley Values}}$
SRelevant Value: Under which realistic conditions similar to $\boldsymbol{X}=x$ can we expect to observe relevant value $y_{\rm rel}$? |  $\underset{x^{\prime}\in\mathcal{X}\land\mathbb{P}(\boldsymbol{X}=x^{\prime})>0}{\mathop{\mathrm{arg}\,\mathrm{min}}}d_{\mathcal{Y}}(m_{\boldsymbol{X}}(x^{\prime}),y_{\rm rel})+\lambda\;d_{\mathcal{X}}(x,x^{\prime})$ |  $\underset{\text{{\cite[citep]{(\@@bibref{AuthorsPhrase1Year}{dandl2020multi}{\@@citephrase{ }}{})}}}}{\text{Counterfactuals}}$202020Only with the right similarity metric that accounts for the realistic constraint.
Table 2:  Global and local formalized questions and matching IML property
descriptors. Note that questions are relative to a specific loss function $L$;
for a set $S\subseteq\\{1,\dots,n\\}$, the term $m_{\boldsymbol{X}_{S}}$
describes the optimal predictor of $\boldsymbol{Y}$ w.r.t. loss function $L$
and random variable(s) $\boldsymbol{X}_{i}$ with $i\in S$. $d_{\mathcal{X}}$
and $d_{\mathcal{Y}}$ describe suitable metrics on $\mathcal{X}$ and
$\mathcal{Y}$ respectively.

##  5 Discussion

ER models enable straightforward scientific inference because their elements
represent something about the underlying phenomenon. While ML models are
generally not ER, IML can offer an indirect route to scientific inference,
provided model properties have a corresponding phenomenon counterpart. We have
shown how phenomenon representation can be achieved through optimal predictors
and described how to practically construct IML property descriptors following
five-steps: the first two steps clarify what questions we can address with
IML, step three and four show how to answer them with ML models and finite
data, and step five allows to evaluate how certain the answers are. We pointed
out that some current IML methods can already be seen as IML property
descriptors.
Is the lack of elementwise representationality specific to ML models? No, ML
shows only an extreme case. In fact, there is a continuum between fully ER
models and HR-only models: Some scientific models contain elements that are
difficult or impossible to interpret e.g. the wave function in physics
(Callender, 2015); complex classical statistical models like generalized
additive models also contain elements that are difficult to interpret. Our
main message is: the five-step approach can be used to extend inference to any
non ER model (whether ML or not).
One could argue that science should only rely on ER models (Rudin, 2019).
Indeed, it would be great if we could always build models from simple to
complex and keep ER from beginning to end. However, more and more problems
seem to be very difficult to tackle with this approach (Nearing et al., 2021);
Interpretable but inaccurate models (w.r.t. to the phenomenon) are not a
solution (Breiman et al., 2001). In situations where we cannot construct
accurate ER models because we lack background knowledge or the phenomenon is
very complex, scientific inference with ML models may thus be the only viable
alternative.

###  5.1 Implications

Adopting a phenomenon-centric perspective on IML allows us to answer a variety
of questions that were puzzling from a model-centric perspective:

Which questions can be addressed with IML property descriptors?  IML property
descriptors can help retrieve relevant phenomenon properties i.e. properties
derived from the conditional distribution
$\mathbb{P}(\boldsymbol{Y}\mid\boldsymbol{X})$. Which phenomenon properties
are relevant is context-specific and up to researchers to identify. While
formulating questions, researchers must be aware that supervised ML models are
only representational of associative structure and not the underlying causal
mechanism (see Section 5.3).

Why use (I)ML for inference?  Supervised ML can help draw scientific inference
when sampling from $\boldsymbol{X}$ is easy but sampling from $\boldsymbol{Y}$
is difficult, e.g. when $\boldsymbol{Y}$ is hard to measure or determined only
in the future. In such situations, analyzing both the model and the data with
IML methods can allow for better conclusions than analyzing just the data —
the ML model fills the gaps by interpolation. Extrapolation to out-of-
distribution data is generally not a strength of ML and can lead to incorrect
conclusions; such extrapolations should only be trusted if the learning
algorithm incorporated a powerful and suitable inductive bias.
When sampling from $\boldsymbol{X}$ is difficult or the property of interest
can be computed more reliably by other means, we advice against using IML for
inference.

How important is model performance in inference?  If the model is a poor
approximation or representation of the modeled phenomenon, the conclusions we
draw from that model are unreliable (Cox, 2006, Good and Hardin, 2012). Thus,
a good fit is vital for gaining reliable knowledge.
Note that even for the optimal model, there remains the so-called Bayes error
rate, an irreducible error arising from the fact that $\boldsymbol{X}$ does
not completely determine $\boldsymbol{Y}$ (Hastie et al., 2009). Thus, high
error does not necessarily flag a low-quality model, but rather may indicate
that $\boldsymbol{X}$ provides insufficient information about
$\boldsymbol{Y}$.

What kind of data should be used for IML?  Many IML methods (e.g. Shapley
Values, LIME, etc.) rely on probing the ML model on permuted data (Scholbeck
et al., 2019). These artificial “data” may never occur in the real world. This
may be useful to audit the model, but if we want to learn about the world,
artificial data is supposed to credibly supplement observations. Our analysis
therefore substantiates the criticism of Hooker and Mentch, (2021), Hooker et
al., (2021), Mentch and Hooker, (2016) concerning the permutation of features
irrespective of the dependency structure in the data.

###  5.2 Open Problems

There are several open issues that we have not addressed:

What about non-tabular data?  For some data types, such as images, audio, or
video data, it is extremely difficult to formulate scientific questions only
in terms of low-level features such as pixels or audio frequencies. To follow
our approach, we need a translation of high-level concepts (e.g. objects in
images or words in audio) that scientists can use to formulate their questions
into low-level features (e.g. pixels or audio frequencies) that the model
works with. Such translations are notoriously difficult to find; deep learning
may help here (Jia et al., 2013, Zaeem and Komeili, 2021, Zhou et al., 2018,
Koh et al., 2020).

How to assess if data is realistic?  In IML, we often need to augment our
data. However, using unrealistic data is highly problematic for scientific
inference, as mentioned earlier. Reasonable permutations of features such as
Laura’s grade jitter strategy (see Section 4.2), can supply realistic data.
However, this requires expert knowledge about what permutations make sense.
Conditional density estimation techniques or generative models (e.g.
generative adversarial networks, normalizing flows, variational autoencoders,
etc.) may provide additional paths to obtain realistic data. However, modeling
the conditional density can be computationally intensive and more difficult
than the original prediction problem, or may even be epistemically problematic
since it only approximates sampling real data.

To what extent does a property determine the true model?  Sometimes, we know
that the model answer to a scientific question is correct. How strongly does
this confirm the correctness of the model? Property descriptions narrow down
the potential models and sufficiently many property descriptions can even
completely determine the model, e.g. for the FANOVA decomposition (Apley and
Zhu, 2020, Hooker, 2004). Model property descriptions may eventually be used
to incorporate background knowledge in training. Both directions, extracting
knowledge from ML models, and using background knowledge to build more
adequate ML models, are elementary for scientific progress (Dwivedi et al.,
2021, Nearing et al., 2021, Razavi, 2021).

###  5.3 Other Forms of Scientific Inference With ML

In this paper, we focused exclusively on scientific inference with supervised
learning ML models on i.i.d. data. For this setting, there is sufficient
theory in both statistical decision theory and IML research to provide secure
epistemic foundations for scientific inference. We have explained what we can
learn about the conditional distribution of $\boldsymbol{Y}$ given
$\boldsymbol{X}$. We can even learn that $\boldsymbol{X}$ contains little
information about $\boldsymbol{Y}$ to predict it, which is scientifically
interesting (Taleb, 2005, Shmueli et al., 2010). However, many questions that
scientists regularly face are of a different nature and go beyond conditional
distributions.

#### Unsupervised Learning.

Unsupervised learning is concerned with estimating aspects of the joint
distribution $\mathbb{P}(\boldsymbol{X}_{1},\dots,\boldsymbol{X}_{n})$.
Unsupervised learning is hard as it typically targets a high-dimensional joint
distribution and, often, lacks a clear measure of success (Hastie et al.,
2009, p486). In principle, our five-step guide is also applicable to
unsupervised learning, however, we lack a theoretical counterpart to optimal
predictors.

#### Causal Learning.

The observational joint probability distribution is interesting, but it
remains on rung one of Judea Pearl’s ladder of causation — the associational
level (Pearl and Mackenzie, 2018). What scientists are often much more
interested in is answering causal questions such as average treatment effects
(rung 2) or counterfactual questions (rung 3) (Salmon, 1998, Woodward and
Ross, 2021). Laura may be interested not only whether students’ language and
math skills are associated (rung 1), but also in whether the provision of
tutoring in Portuguese affects students’ math skills (rung 2) or whether a
specific student (who is not a native Portuguese speaker) would have done
better in math if she had received Portuguese tutoring at a young age (rung
3).
Supervised ML models only represent aspects of the observational distribution
(rung 1) and therefore generally do not allow answering causal questions. As a
consequence, the IML descriptors of the models also generally do not allow
causal insight into the data. Many IML papers that discuss causality (Schwab
and Karlen, 2019, Janzing et al., 2020, Wang et al., 2021, Heskes et al.,
2020) are only concerned with causal effects on the model’s prediction, which
do not necessarily translate into a causal insight into the phenomenon.
In order to answer causal questions, causal models should be used
instead.212121Given a causal graph, observational data can allow to identify
average causal effects (rung 2), e.g. with the so-called backdoor criterion
(Pearl, 2009). For estimating counterfactuals (rung 3), assumptions beyond a
causal graph and observational data must be met (Holland, 1986, Peters et al.,
2017). To learn a causal model, we must gather interventional data and/or make
strong, untestable assumptions. Causal inference constitutes thus a
challenging problem and remains an active area of research (Heinze-Deml et
al., 2018, Kalisch and Bühlmann, 2014, Constantinou et al., 2021, Peters et
al., 2017).
In certain situations, ML models can nevertheless be useful for causal
inference. Firstly, if all predictor variables are causally independent and
the prediction target is caused by the features, the causal model
interpretation implies the causal data interpretation. Secondly, associative
models in combination with IML can help estimate causal effects even in the
absence of causal independence if they are in principle identifiable by
observation. For example, the partial dependence plot coincides with the so-
called adjustment formula and therefore identifies a causal effect if the
backdoor criterion is met (and the model optimally predicts the conditional
expectation) (Zhao and Hastie, 2021). Thirdly, when there is access to
observational and interventional data during training, training ML models with
invariant risk minimization yields models that predict accurately in
interventional environments (Peters et al., 2016, Pfister et al., 2021,
Arjovsky et al., 2019). For such intervention-stable models, IML methods that
provide insight into the effect of interventions on the prediction also
describe causal effects on the underlying real-world components (König et al.,
2021a ).
Another way in which ML supports causal inference is by facilitating practical
scientific inference relying on complex mechanistical models, frequently
implemented as numerical simulators. Indeed simulators can represent complex,
causal, dynamics in an ER fashion, but often at the price of an intractable
likelihood and thus expensive inference. A variety of new ML methods for
likelihood-free inference on simulators (Cranmer et al., 2020) allows to
estimate a full posterior distribution over ER parameters for increasingly
complex models.
While supervised learning learns from a fixed dataset, reinforcement learning
(RL) systems are designed to act and can therefore assess the effect of
interventions. As such, RL models can be designed to provide causal
interpretations (Bareinboim et al., 2015, Zhang and Bareinboim, 2017, Gasse et
al., 2021).

##  6 Conclusion

Traditional scientific models were designed to satisfy elementwise
representationality. This allowed scientists to directly inspect model
elements to learn about Nature. Although ML models do not satisfy elementwise
representationality, we have showed that it is still possible to learn about
the phenomenon using them. All we need to do is to interrogate the model with
suitable IML property descriptors. We have shown how such descriptors must be
designed to enable scientific inference.

## Funding

This work was supported by the Graduate School of Systemic Neuroscience (GSN)
of the LMU Munich and by the German Federal Ministry of Education and Research
(BMBF) under Grant No. 01IS18036A, and by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC
number 2064/1 – Project number 390727645.

## Acknowledgements

We are very significantly indebted to Tom Sterkenburg, Seth Axen, Thomas
Grote, Alexandra Gessner, Nacho Molina and Christian Scholbeck for their
comments on the manuscript and their hints to related literature. Moreover, we
thank the workshop participants of the Tübingen workshop “Philosophy of
Science Meets Machine Learning” for the insightful comments and discussions
after presenting some of the ideas described in the paper.

## References

  * Aas et al., (2021) Aas, K., Jullum, M., and Løland, A. (2021).  Explaining individual predictions when features are dependent: More accurate approximations to shapley values.  Artificial Intelligence, 298:103502.
  * Achinstein, (1974) Achinstein, P. (1974).  Concepts of science.  Philosophy, 49(187).
  * Alqaraawi et al., (2020) Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., and Berthouze, N. (2020).  Evaluating saliency map explanations for convolutional neural networks: a user study.  In Proceedings of the 25th International Conference on Intelligent User Interfaces, pages 275–285.
  * Anderson, (2008) Anderson, C. (2008).  The end of theory: The data deluge makes the scientific method obsolete.  Wired magazine, 16(7):16–07.
  * Apley and Zhu, (2020) Apley, D. W. and Zhu, J. (2020).  Visualizing the effects of predictor variables in black box supervised learning models.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 82(4):1059–1086.
  * Arjovsky et al., (2019) Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. (2019).  Invariant risk minimization.  arXiv preprint arXiv:1907.02893.
  * Arrieta et al., (2020) Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2020).  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  Information Fusion, 58:82–115.
  * (8) Bailer-Jones, D. M. (2003a).  Models, theories and phenomena.  Proceedings of Logic Methodology and Philosophy of Science.
  * (9) Bailer-Jones, D. M. (2003b).  When scientific models represent.  International studies in the philosophy of science, 17(1):59–74.
  * Bailer-Jones and Bailer-Jones, (2002) Bailer-Jones, D. M. and Bailer-Jones, C. A. (2002).  Modeling data: Analogies in neural networks, simulated annealing and genetic algorithms.  In Model-Based Reasoning, pages 147–165. Springer.
  * Bareinboim et al., (2015) Bareinboim, E., Forney, A., and Pearl, J. (2015).  Bandits with unobserved confounders: A causal approach.  Advances in Neural Information Processing Systems, 28.
  * Bau et al., (2017) Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. (2017).  Network dissection: Quantifying interpretability of deep visual representations.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541–6549.
  * Bau et al., (2018) Bau, D., Zhu, J.-Y., Strobelt, H., Zhou, B., Tenenbaum, J. B., Freeman, W. T., and Torralba, A. (2018).  Gan dissection: Visualizing and understanding generative adversarial networks.  arXiv preprint arXiv:1811.10597.
  * Bhatt et al., (2020) Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M., and Eckersley, P. (2020).  Explainable machine learning in deployment.  In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 648–657.
  * Bickler, (2021) Bickler, S. H. (2021).  Machine learning arrives in archaeology.  Advances in Archaeological Practice, 9(2):186–191.
  * Boge, (2022) Boge, F. J. (2022).  Two dimensions of opacity and the deep learning predicament.  Minds and Machines, 32(1):43–75.
  * Bokulich, (2011) Bokulich, A. (2011).  How scientific models can explain.  Synthese, 180(1):33–45.
  * Breiman, (2001) Breiman, L. (2001).  Random forests.  Machine learning, 45(1):5–32.
  * Breiman et al., (2001) Breiman, L. et al. (2001).  Statistical modeling: The two cultures (with comments and a rejoinder by the author).  Statistical science, 16(3):199–231.
  * Buckner, (2018) Buckner, C. (2018).  Empiricism without magic: Transformational abstraction in deep convolutional neural networks.  Synthese, 195(12):5339–5372.
  * Buckner and Garson, (2019) Buckner, C. and Garson, J. (2019).  Connectionism.  In Zalta, E. N., editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2019 edition.
  * Callender, (2015) Callender, C. (2015).  One world, one beable.  Synthese, 192(10):3153–3177.
  * Chen et al., (2020) Chen, H., Janizek, J. D., Lundberg, S., and Lee, S.-I. (2020).  True to the model or true to the data?  arXiv preprint arXiv:2006.16234.
  * Cichy and Kaiser, (2019) Cichy, R. M. and Kaiser, D. (2019).  Deep neural networks as scientific models.  Trends in cognitive sciences, 23(4):305–317.
  * Constantinou et al., (2021) Constantinou, A. C., Liu, Y., Chobtham, K., Guo, Z., and Kitson, N. K. (2021).  Large-scale empirical validation of bayesian network structure learning algorithms with noisy data.  International Journal of Approximate Reasoning, 131:151–188.
  * Contessa, (2007) Contessa, G. (2007).  Scientific representation, interpretation, and surrogative reasoning.  Philosophy of science, 74(1):48–68.
  * Cortez and Silva, (2008) Cortez, P. and Silva, A. (2008).  Using data mining to predict secondary school student performance.  EUROSIS.
  * Covert et al., (2020) Covert, I., Lundberg, S. M., and Lee, S.-I. (2020).  Understanding global feature contributions with additive importance measures.  Advances in Neural Information Processing Systems, 33:17212–17223.
  * Cox, (2006) Cox, D. R. (2006).  Principles of statistical inference.  Cambridge university press.
  * Cranmer et al., (2020) Cranmer, K., Brehmer, J., and Louppe, G. (2020).  The frontier of simulation-based inference.  Proceedings of the National Academy of Sciences, 117(48):30055–30062.
  * Dandl et al., (2020) Dandl, S., Molnar, C., Binder, M., and Bischl, B. (2020).  Multi-objective counterfactual explanations.  In International Conference on Parallel Problem Solving from Nature, pages 448–469. Springer.
  * Domingos, (2000) Domingos, P. (2000).  A unified bias-variance decomposition.  In Proceedings of 17th international conference on machine learning, pages 231–238. Morgan Kaufmann Stanford.
  * Donnelly and Roegiest, (2019) Donnelly, J. and Roegiest, A. (2019).  On interpretability and feature representations: an analysis of the sentiment neuron.  In European Conference on Information Retrieval, pages 795–802. Springer.
  * Doshi-Velez and Kim, (2017) Doshi-Velez, F. and Kim, B. (2017).  Towards a rigorous science of interpretable machine learning.  arXiv preprint arXiv:1702.08608.
  * Douglas, (2009) Douglas, H. E. (2009).  Reintroducing prediction to explanation.  Philosophy of Science, 76(4):444–463.
  * Dwivedi et al., (2021) Dwivedi, D., Nearing, G., Gupta, H., Sampson, A. K., Condon, L., Ruddell, B., Klotz, D., Ehret, U., Read, L., Kumar, P., et al. (2021).  Knowledge-guided machine learning (kgml) platform to predict integrated water cycle and associated extremes.  Technical report, Artificial Intelligence for Earth System Predictability.
  * Farrell et al., (2018) Farrell, S., Calafiura, P., Mudigonda, M., Anderson, D., Vlimant, J.-R., Zheng, S., Bendavid, J., Spiropulu, M., Cerati, G., Gray, L., et al. (2018).  Novel deep learning methods for track reconstruction.  arXiv preprint arXiv:1810.06111.
  * Fisher et al., (2019) Fisher, A., Rudin, C., and Dominici, F. (2019).  All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously.  J. Mach. Learn. Res., 20(177):1–81.
  * Friedman et al., (1991) Friedman, J. H. et al. (1991).  Multivariate adaptive regression splines.  The Annals of Statistics, 19(1):1–67.
  * Frigg and Nguyen, (2021) Frigg, R. and Nguyen, J. (2021).  Scientific Representation.  In Zalta, E. N., editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2021 edition.
  * Gale et al., (2020) Gale, E. M., Martin, N., Blything, R., Nguyen, A., and Bowers, J. S. (2020).  Are there any ‘object detectors’ in the hidden layers of cnns trained to identify objects or scenes?  Vision Research, 176:60–71.
  * Gasse et al., (2021) Gasse, M., Grasset, D., Gaudron, G., and Oudeyer, P.-Y. (2021).  Causal reinforcement learning using observational and interventional data.  arXiv preprint arXiv:2106.14421.
  * Gibson et al., (2021) Gibson, P. B., Chapman, W. E., Altinok, A., Delle Monache, L., DeFlorio, M. J., and Waliser, D. E. (2021).  Training machine learning models on climate model output yields skillful interpretable seasonal precipitation forecasts.  Communications Earth & Environment, 2(1):1–13.
  * Gobet, (2018) Gobet, F. (2018).  Three views on expertise: Philosophical implications for rationality, knowledge, intuition and education.  Education and Expertise, pages 58–74.
  * Goldstein et al., (2015) Goldstein, A., Kapelner, A., Bleich, J., and Pitkin, E. (2015).  Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation.  journal of Computational and Graphical Statistics, 24(1):44–65.
  * Good and Hardin, (2012) Good, P. I. and Hardin, J. W. (2012).  Common errors in statistics (and how to avoid them).  John Wiley & Sons.
  * Gühring et al., (2020) Gühring, I., Raslan, M., and Kutyniok, G. (2020).  Expressivity of deep neural networks.  arXiv preprint arXiv:2007.04759.
  * Hastie et al., (2009) Hastie, T., Tibshirani, R., Friedman, J. H., and Friedman, J. H. (2009).  The elements of statistical learning: data mining, inference, and prediction, volume 2.  Springer.
  * Heinze-Deml et al., (2018) Heinze-Deml, C., Maathuis, M. H., and Meinshausen, N. (2018).  Causal structure learning.  Annual Review of Statistics and Its Application, 5:371–391.
  * Heskes et al., (2020) Heskes, T., Sijben, E., Bucur, I. G., and Claassen, T. (2020).  Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models.  Advances in neural information processing systems, 33:4778–4789.
  * Holland, (1986) Holland, P. W. (1986).  Statistics and causal inference.  Journal of the American statistical Association, 81(396):945–960.
  * Hooker, (2004) Hooker, G. (2004).  Discovering additive structure in black box functions.  In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 575–580.
  * Hooker and Mentch, (2021) Hooker, G. and Mentch, L. (2021).  Bridging breiman’s brook: From algorithmic modeling to statistical learning.  Observational Studies, 7(1):107–125.
  * Hooker et al., (2021) Hooker, G., Mentch, L., and Zhou, S. (2021).  Unrestricted permutation forces extrapolation: variable importance requires at least one more model, or there is no free variable importance.  Statistics and Computing, 31(6):1–16.
  * Hutter et al., (2019) Hutter, F., Kotthoff, L., and Vanschoren, J. (2019).  Automated machine learning: methods, systems, challenges.  Springer Nature.
  * Janzing et al., (2020) Janzing, D., Minorics, L., and Blöbaum, P. (2020).  Feature relevance quantification in explainable ai: A causal problem.  In International Conference on artificial intelligence and statistics, pages 2907–2916. PMLR.
  * Jia et al., (2013) Jia, Y., Abbott, J. T., Austerweil, J. L., Griffiths, T., and Darrell, T. (2013).  Visual concept learning: Combining machine vision and bayesian generalization on concept hierarchies.  Advances in Neural Information Processing Systems, 26.
  * Kalisch and Bühlmann, (2014) Kalisch, M. and Bühlmann, P. (2014).  Causal structure learning and inference: a selective review.  Quality Technology & Quantitative Management, 11(1):3–21.
  * Kim et al., (2018) Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. (2018).  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In International conference on machine learning, pages 2668–2677. PMLR.
  * Kitchin, (2014) Kitchin, R. (2014).  Big data, new epistemologies and paradigm shifts.  Big data & society, 1(1):2053951714528481.
  * Koh et al., (2020) Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. (2020).  Concept bottleneck models.  In International Conference on Machine Learning, pages 5338–5348. PMLR.
  * (62) König, G., Freiesleben, T., and Grosse-Wentrup, M. (2021a).  A causal perspective on meaningful and robust algorithmic recourse.  arXiv preprint arXiv:2107.07853.
  * (63) König, G., Molnar, C., Bischl, B., and Grosse-Wentrup, M. (2021b).  Relative feature importance.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 9318–9325. IEEE.
  * Levy, (2012) Levy, A. (2012).  Models, fictions, and realism: Two packages.  Philosophy of Science, 79(5):738–748.
  * Lipton, (2018) Lipton, Z. C. (2018).  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  Queue, 16(3):31–57.
  * Longino, (2018) Longino, H. E. (2018).  The fate of knowledge.  Princeton University Press.
  * Luan and Tsai, (2021) Luan, H. and Tsai, C.-C. (2021).  A review of using machine learning approaches for precision education.  Educational Technology & Society, 24(1):250–266.
  * Luk, (2017) Luk, R. W. (2017).  A theory of scientific study.  Foundations of Science, 22(1):11–38.
  * Mayer-Schönberger and Cukier, (2013) Mayer-Schönberger, V. and Cukier, K. (2013).  Big data: A revolution that will transform how we live, work, and think.  Houghton Mifflin Harcourt.
  * McClelland et al., (1987) McClelland, J. L., Rumelhart, D. E., Group, P. R., et al. (1987).  Parallel Distributed Processing, Volume 2: Explorations in the Microstructure of Cognition: Psychological and Biological Models, volume 2.  MIT press.
  * Mentch and Hooker, (2016) Mentch, L. and Hooker, G. (2016).  Quantifying uncertainty in random forests via confidence intervals and hypothesis tests.  The Journal of Machine Learning Research, 17(1):841–881.
  * Molnar, (2020) Molnar, C. (2020).  Interpretable machine learning.  Lulu. com.
  * (73) Molnar, C., Casalicchio, G., and Bischl, B. (2020a).  Interpretable machine learning–a brief history, state-of-the-art and challenges.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 417–431. Springer.
  * Molnar et al., (2021) Molnar, C., Freiesleben, T., König, G., Casalicchio, G., Wright, M. N., and Bischl, B. (2021).  Relating the partial dependence plot and permutation feature importance to the data generating process.  arXiv preprint arXiv:2109.01433.
  * (75) Molnar, C., König, G., Bischl, B., and Casalicchio, G. (2020b).  Model-agnostic feature importance and effects with dependent features–a conditional subgroup approach.  arXiv preprint arXiv:2006.04628.
  * Molnar et al., (2022) Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., and Bischl, B. (2022).  General pitfalls of model-agnostic interpretation methods for machine learning models.  In Holzinger, A., Goebel, R., Fong, R., Moon, T., Müller, K.-R., and Samek, W., editors, xxAI - Beyond Explainable AI: International Workshop, Held in Conjunction with ICML 2020, July 18, 2020, Vienna, Austria, Revised and Extended Papers, pages 39–68, Cham. Springer International Publishing.
  * Mu and Andreas, (2020) Mu, J. and Andreas, J. (2020).  Compositional explanations of neurons.  Advances in Neural Information Processing Systems, 33:17153–17163.
  * Nearing et al., (2021) Nearing, G. S., Kratzert, F., Sampson, A. K., Pelissier, C. S., Klotz, D., Frame, J. M., Prieto, C., and Gupta, H. V. (2021).  What role does hydrological science play in the age of machine learning?  Water Resources Research, 57(3):e2020WR028091.
  * Nguyen et al., (2016) Nguyen, A., Yosinski, J., and Clune, J. (2016).  Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks.  arXiv preprint arXiv:1602.03616.
  * Olah et al., (2020) Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020).  Zoom in: An introduction to circuits.  Distill, 5(3):e00024–001.
  * Olah et al., (2017) Olah, C., Mordvintsev, A., and Schubert, L. (2017).  Feature visualization.  Distill, 2(11):e7.
  * Pearl, (2009) Pearl, J. (2009).  Causality.  Cambridge university press.
  * Pearl, (2018) Pearl, J. (2018).  Theoretical impediments to machine learning with seven sparks from the causal revolution.
  * Pearl and Mackenzie, (2018) Pearl, J. and Mackenzie, D. (2018).  The book of why: the new science of cause and effect.  Basic books.
  * Peters et al., (2016) Peters, J., Bühlmann, P., and Meinshausen, N. (2016).  Causal inference by using invariant prediction: identification and confidence intervals.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):947–1012.
  * Peters et al., (2017) Peters, J., Janzing, D., and Schölkopf, B. (2017).  Elements of causal inference: foundations and learning algorithms.  The MIT Press.
  * Pfister et al., (2021) Pfister, N., Williams, E. G., Peters, J., Aebersold, R., and Bühlmann, P. (2021).  Stabilizing variable selection and regression.  The Annals of Applied Statistics, 15(3):1220–1246.
  * Raji et al., (2020) Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., and Barnes, P. (2020).  Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing.  In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 33–44.
  * Räz, (2022) Räz, T. (2022).  Understanding deep learning with statistical relevance.  Philosophy of Science, 89(1):20–41.
  * Razavi, (2021) Razavi, S. (2021).  Deep learning, explained: Fundamentals, explainability, and bridgeability to process-based modelling.  Environmental Modelling & Software, 144:105159.
  * Reichstein et al., (2019) Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., et al. (2019).  Deep learning and process understanding for data-driven earth system science.  Nature, 566(7743):195–204.
  * Ribeiro et al., (2016) Ribeiro, M. T., Singh, S., and Guestrin, C. (2016).  Why should i trust you?: Explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144. ACM.
  * Ritchey, (2012) Ritchey, T. (2012).  Outline for a morphology of modelling methods.  Acta Morphologica Generalis AMG Vol, 1(1):1012.
  * Roscher et al., (2020) Roscher, R., Bohn, B., Duarte, M. F., and Garcke, J. (2020).  Explainable machine learning for scientific insights and discoveries.  IEEE Access, 8:42200–42216.
  * Rosser et al., (2008) Rosser, J. S. J. B. et al. (2008).  A nobel prize for asymmetric information: the economic contributions of george akerlof, michael spence and joseph stiglitz.  In Leading Contemporary Economists, pages 162–181. Routledge.
  * Rudin, (2019) Rudin, C. (2019).  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature Machine Intelligence, 1(5):206–215.
  * Salmon, (1979) Salmon, W. C. (1979).  Why ask,‘why?’? an inquiry concerning scientific explanation.  In Hans Reichenbach: logical empiricist, pages 403–425. Springer.
  * Salmon, (1998) Salmon, W. C. (1998).  Causality and explanation.  Oxford University Press.
  * Schmidt et al., (2019) Schmidt, J., Marques, M. R., Botti, S., and Marques, M. A. (2019).  Recent advances and applications of machine learning in solid-state materials science.  npj Computational Materials, 5(1):1–36.
  * Scholbeck et al., (2019) Scholbeck, C. A., Molnar, C., Heumann, C., Bischl, B., and Casalicchio, G. (2019).  Sampling, intervention, prediction, aggregation: a generalized framework for model-agnostic interpretations.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 205–216. Springer.
  * Schwab and Karlen, (2019) Schwab, P. and Karlen, W. (2019).  Cxplain: Causal explanations for model interpretation under uncertainty.  Advances in Neural Information Processing Systems, 32.
  * Schölkopf et al., (2021) Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. (2021).  Towards causal representation learning.
  * Senior et al., (2020) Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W., Bridgland, A., et al. (2020).  Improved protein structure prediction using potentials from deep learning.  Nature, 577(7792):706–710.
  * Shahhosseini et al., (2020) Shahhosseini, M., Hu, G., and Archontoulis, S. V. (2020).  Forecasting corn yield with machine learning ensembles.  arXiv preprint arXiv:2001.09055.
  * Shmueli et al., (2010) Shmueli, G. et al. (2010).  To explain or to predict?  Statistical science, 25(3):289–310.
  * Slack et al., (2020) Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. (2020).  Fooling lime and shap: Adversarial attacks on post hoc explanation methods.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 180–186.
  * Spinney, (2022) Spinney, L. (2022).  Are we witnessing the dawn of post-theory science?  The Guardian.
  * Srivastava et al., (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014).  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958.
  * Stachl et al., (2020) Stachl, C., Au, Q., Schoedel, R., Gosling, S. D., Harari, G. M., Buschek, D., Völkel, S. T., Schuwerk, T., Oldemeier, M., Ullmann, T., Hussmann, H., Bischl, B., and Bühner, M. (2020).  Predicting personality from patterns of behavior collected with smartphones.  Proceedings of the National Academy of Sciences, 117(30):17680–17687.
  * Stachowiak, (1973) Stachowiak, H. (1973).  Allgemeine modelltheorie.  Springer.
  * Sterkenburg and Grünwald, (2021) Sterkenburg, T. F. and Grünwald, P. D. (2021).  The no-free-lunch theorems of supervised learning.  Synthese, 199(3):9979–10015.
  * Strobl et al., (2008) Strobl, C., Boulesteix, A.-L., Kneib, T., Augustin, T., and Zeileis, A. (2008).  Conditional variable importance for random forests.  BMC bioinformatics, 9(1):1–11.
  * Štrumbelj and Kononenko, (2014) Štrumbelj, E. and Kononenko, I. (2014).  Explaining prediction models and individual predictions with feature contributions.  Knowledge and information systems, 41(3):647–665.
  * Sullivan, (2020) Sullivan, E. (2020).  Understanding from machine learning models.  The British Journal for the Philosophy of Science.
  * Suppes, (1966) Suppes, P. (1966).  Models of data.  In Studies in logic and the foundations of mathematics, volume 44, pages 252–261. Elsevier.
  * Taleb, (2005) Taleb, N. (2005).  The black swan: Why don’t we learn that we don’t learn.  NY: Random House.
  * Toulmin, (1961) Toulmin, S. E. (1961).  Foresight and understanding: An enquiry into the aims of science.  Greenwood Press.
  * Voss et al., (2021) Voss, C., Cammarata, N., Goh, G., Petrov, M., Schubert, L., Egan, B., Lim, S. K., and Olah, C. (2021).  Visualizing weights.  Distill, 6(2):e00024–007.
  * Wachter et al., (2017) Wachter, S., Mittelstadt, B., and Floridi, L. (2017).  Why a right to explanation of automated decision-making does not exist in the general data protection regulation.  International Data Privacy Law, 7(2):76–99.
  * Wang et al., (2021) Wang, J., Wiens, J., and Lundberg, S. (2021).  Shapley flow: A graph-based approach to interpreting model predictions.  In International Conference on Artificial Intelligence and Statistics, pages 721–729. PMLR.
  * Watson, (2022) Watson, D. S. (2022).  Conceptual challenges for interpretable machine learning.  Synthese, 200(1):1–33.
  * Woodward and Ross, (2021) Woodward, J. and Ross, L. (2021).  Scientific Explanation.  In Zalta, E. N., editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2021 edition.
  * Zaeem and Komeili, (2021) Zaeem, M. N. and Komeili, M. (2021).  Cause and effect: Concept-based explanation of neural networks.  In 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2730–2736. IEEE.
  * Zednik, (2021) Zednik, C. (2021).  Solving the black box problem: a normative framework for explainable artificial intelligence.  Philosophy & Technology, 34(2):265–288.
  * Zednik and Boelsen, (2022) Zednik, C. and Boelsen, H. (2022).  Scientific exploration and explainable artificial intelligence.  Minds and Machines, pages 1–21.
  * Zhang and Bareinboim, (2017) Zhang, J. and Bareinboim, E. (2017).  Transfer learning in multi-armed bandit: a causal approach.  In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 1778–1780.
  * Zhang et al., (2019) Zhang, Z., Jin, Y., Chen, B., and Brown, P. (2019).  California almond yield prediction at the orchard level with a machine learning approach.  Frontiers in Plant Science, 10:809.
  * Zhao and Hastie, (2021) Zhao, Q. and Hastie, T. (2021).  Causal interpretations of black-box models.  Journal of Business & Economic Statistics, 39(1):272–281.
  * Zhou et al., (2018) Zhou, B., Sun, Y., Bau, D., and Torralba, A. (2018).  Interpretable basis decomposition for visual explanation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 119–134.

##  Appendix A Background on Models and Phenomena

We follow Bailer-Jones, 2003b ([], p61) and others (Achinstein, 1974, Levy,
2012, Contessa, 2007) in seeing models as “an interpretative description of a
phenomenon that facilitates perceptual as well as intellectual access to that
phenomenon”, where a phenomenon describes a fact or event in nature that is
subject to be researched (Bailer-Jones, 2003a ). Phenomenon and scientific
models have been described as a continuous hierarchy with data living close to
the phenomenon and the model close to theory (Suppes, 1966). Models represent
only some phenomenon aspects but not others (Ritchey, 2012, Bailer-Jones,
2003b , Frigg and Nguyen, 2021); a good model is true to the aspects that are
relevant to the model user (Bailer-Jones, 2003b , Stachowiak, 1973).

##  Appendix B Dataset

Figure 8 gives a descriptions of the different features and is copied from
Cortez and Silva, (2008). In our trained models, we only used the final G3
student grades. The data was collected during 2005 and 2006 from two public
schools, from the Alentejo region in Portugal. The database is collected from
a variety of sources from both school reports and questionnaires. Cortez and
Silva, (2008) integrated the information into a mathematics dataset (with 395
examples) and a Portuguese language dataset (649 records).

Figure 8: Attributes in the Cortez and Silva, (2008) dataset.

##  Appendix C Tower Rule for Expectations

For arbitrary random variables $\boldsymbol{X},\boldsymbol{Y},\boldsymbol{Z}$
holds that

$\mathbb{E}_{\boldsymbol{Y}\mid\boldsymbol{X}}[\boldsymbol{Y}\mid\boldsymbol{X}]=\mathbb{E}_{\boldsymbol{Z}\mid\boldsymbol{X}}\Bigl{[}\mathbb{E}_{\boldsymbol{Y}\mid\boldsymbol{X},\boldsymbol{Z}}[\boldsymbol{Y}\mid\boldsymbol{X},\boldsymbol{Z}]\mid\boldsymbol{X}\Bigr{]}.$

This is also known as the rule of total expectation. Intuitively it says that
it doesn’t matter if we directly take the expectation of $\boldsymbol{Y}$ on
$\boldsymbol{X}$ or if we first take the expectation of $\boldsymbol{Y}$
conditioned on a set of random variables $\boldsymbol{X},\boldsymbol{Z}$ that
includes $\boldsymbol{X}$ and then, “integrate $\boldsymbol{Z}$ out”.