# A formal framework for generalized reporting methods in parametric settings

Hannah Kümpel
<EMAIL_ADDRESS>
The Institute for Medical Information Processing, Biometry, and Epidemiology,
Ludwig-Maximilians-Universität of Munich Sabine Hoffmann
<EMAIL_ADDRESS>Department of Statistics, Ludwig-
Maximilians-Universität of Munich

###### Abstract

Effect size measures and visualization techniques aimed at maximizing the
interpretability and comparability of results from statistical models have
long been of great importance and are recently again receiving increased
attention in the literature. However, since the methods proposed in this
context originate from a wide variety of disciplines and are more often than
not practically motivated, they lack a common theoretical framework and many
quantities are narrowly or heuristically defined. In this work, we put forward
a common mathematical setting for effect size measures and visualization
techniques aimed at the results of parametric regression and define a formal
framework for the consistent derivation of both existing and new variants of
such quantities. Throughout the presented theory, we utilize probability
measures to derive weighted means over areas of interest. While we take a
Bayesian approach to quantifying uncertainty in order to derive consistent
results for every defined quantity, all proposed methods apply to the results
of both frequentist and Bayesian inference. We apply selected specifications
derived from the proposed framework to data from a clinical trial and a multi-
analyst study to illustrate its versatility and relevance.

Keywords: interpretability $\cdot$ marginal effects $\cdot$ predicted change
in probability $\cdot$ partial dependence plots $\cdot$ individual expectation

##  1 Introduction

The interpretation of results of statistical models is notoriously difficult.
While the question of interpretability has recently received considerable
attention in the machine learning literature, it is important to recognize
that interpretability is also an important issue when it comes to classical
statistical quantities. Despite numerous cautions and clarifications
(Wasserstein et al., 2016, 2019), some misinterpretations of statistical
significance are so widespread that they fill entire textbooks (Cassidy et
al., 2019), making it quite difficult for researchers to recognize that these
interpretations are actually incorrect. Similarly, classical effect estimates
such as odds ratios and hazard ratios are staggeringly often misinterpreted
(Sackett et al. (1996), Greenland (1987)). And even if applied researchers are
familiar with the correct definitions, they often prefer to think of these
quantities as risk ratios or even as absolute risk differences, implicitly
choosing a wrong answer to the right question over a right answer to the wrong
question. Meanwhile, for model classes that do provide interpretable effect
size estimates, researchers still often face a problem of comparability of
these estimates when they try to compare the results of different model
classes or even the results of the same model class on different data sets.

Recent interest in these issues has led to a plethora of visualization
techniques and effect size measures that have been proposed in different
disciplines to improve the interpretability and comparability of model results
(Gelman & Pardoe, 2007; Williams, 2012; Kaufman, 1996; Apley & Zhu, 2020;
Goldstein et al., 2015; Scholbeck et al., 2022). However, so far these
approaches are lacking a common theoretical framework and many present a
number of unclarities and inconsistencies, especially regarding categorical
independent variables. In this work, we define a formal framework that not
only unifies this collection of methods in parametric settings but also allows
for the consistent derivation of definitions for special cases, thereby
avoiding discrepancies and facilitating the deduction of further
methodological variants. After outlining the contributions and setting of the
proposed framework, the following will define all therein contained quantities
and place a variety of existing methodologies from various disciplines within
them in sections 2 and 3, respectively. Subsequently, section 4 illustrates
selected aspects of the proposed methodology on data from a clinical trial
Tortora et al. (2020) and a multi-analyst study Silberzahn et al. (2018). All
proofs are given in appendix C and the code used for the analyses of section 4
may be found in appendix E.

##### Contributions

The main contribution of this work is the providing of a formal framework for
reporting methods aimed at maximizing the interpretability and comparability
of model results, in particular, one which allows researchers to customize
these methods for the specific research question at hand. In doing so, we
identify three axiomatic assumptions regarding the dependence structure of
models’ independent variables that are useful for clarifying the approach for
different specifications, including existing methods. Notably, we provide a
consistent method of deriving point estimates and well-interpretable
uncertainty regions for all proposed quantities. When defining a generalized
version of marginal effects, we include the additional option to separately
quantify main- and interaction- effects. Finally, in the interest of
increasing the communicability of results from models with parametric
distribution assumptions, we propose a method for visualizing the different
expected distributions of the target variable resulting from changes in the
values of independent variables.

###  1.1 Setting

Within this work, we restrict ourselves to cases of parametric regression.
Specifically, we will consider sequences
$\\{(y_{i},\mathbf{x}_{i})\\}_{i=1,...,n}$ of $n\in\mathbb{N}$ observations,
where the vectors $\mathbf{x}_{i}\in\mathbb{R}^{p}$, $p\in\mathbb{N}$, contain
the observed values of all regressors, dummy coded in the case of categorical
inputs, corresponding to an observation $y_{i}\in\mathbb{R}$ of the target
variable. To be precise, we consider each element of the vectors
$\mathbf{x}_{i}$ to be a realization of a variable $X_{j}$, $1\leq j\leq p$;
with $X_{j}$ representing either a metric regressor, i.e. taking values in
$\mathbb{R}$, or one category of a categorical regressor, i.e. taking values
in $\\{0,1\\}$, with the _reference category_ being equated with all other
categories taking the value zero. Generally, we denote the realization of any
such variable $X_{j}$ by $x_{j}$. Thereby, a categorical regressor with
$d\in\mathbb{N}$ categories, the first of which being represented by $X_{j}$,
is defined by the vector of binary variables $(X_{j},...,X_{j+d-2})\text{,
with }(x_{j},...,x_{j+d-2})^{\top}\in\\{\mathbf{e}_{i},1\leq i\leq
d-1\\}\cup\mathbf{0},$ where $\\{\mathbf{e}_{i},1\leq i\leq d-1\\}$ denotes
the standard basis of $\mathbb{R}^{d-1}$ and $\mathbf{0}$ the zero vector,
specifically the $(d-1)$-dimensional zero vector in this case. Since we will
use such sets repeatedly, we define for any $d\in\mathbb{N}$ the set
$\mathbf{e}^{(d)}:=\\{\mathbf{e}_{i},1\leq i\leq
d\\}\cup\mathbf{0}\subseteq\mathbb{R}^{d}$.

For the purposes of this work, we furthermore separate the regressors into
three groups:

  1. (1)

The regressor whose effect on the target variable is of interest, referred to
as _the regressor of interest_ , $X^{I}$, which can either be metric or a
categorical variable with $d_{I}+1$ categories (or $d_{I}$ non-reference
categories). For consistency of notation, we set $d_{I}=1$, if $X^{I}$ is a
metric regressor, and denote by $I$ the set that contains only $X^{I}$.

  2. (2)

The set of (remaining) metric regressors $M$. Any variable representing an
element of $M$ will be denoted by $X^{M}_{l}$, $1\leq l\leq m$, with $m$ being
the number of (remaining) metric regressors.

  3. (3)

The set of (remaining) categorical regressors $C$. We denote by $c$ the number
of (remaining) categorical regressors and by $d_{C_{k}}$, $1\leq k\leq c$, the
number of non-reference categories of the $k$th (remaining) categorical
regressor $X^{C}_{k}$.

######  Remark 1.1.

In essence, we denote the vector of models’ independent variables by
$X=(X_{1},...,X_{p})$, with each entry taking values in $\mathbb{R}$ or
$\\{0,1\\}$. We further denote any value the variable $X_{j}$, $1\leq j\leq
p$, takes by $x_{j}$ and, in a slight abuse of terminology, refer to both
$X_{j}$ and $x_{j}$ as _regressor (category)_. Moreover, instead of reordering
the vector $X$, we assign each of its elements to one of the sets $I$, $M$,
and $C$, effectively mapping subsets of the elements of $X$ to elements of the
set $I\cup M\cup C$, with $p=d_{I}+m+\sum_{k=1}^{c}d_{C_{k}}$. Lastly, in a
slight abuse of notation, we write $x_{[I]}$, $x_{[M_{l}]}$, and $x_{[C_{j}]}$
for the values that the regressors mapped to $X^{I}$, $X^{M}_{l}$,
$l\in\\{1,...,m\\}$, and $X^{C}_{k}$, $k\in\\{1,...,c\\}$, respectively, take.
For example, if the regressor of interest is a categorical regressor whose
non-reference categories are represented by the first three elements of $X$,
we write $x_{[I]}=(x_{1},x_{2},x_{3})$.

As previously mentioned, the current theory is restricted to the case of
parametric regression, by which we mean a regression setting in which only the
parameter $\theta\in\Theta$, with $\Theta\subseteq\mathbb{R}^{k}$,
$k\in\mathbb{N}_{\geq{0}}$, being a finite-dimensional vector space, is
unknown. Given a set $S$ endowed with $\sigma$-algebra $\mathcal{S}$, we
denote by $\mathcal{M}(S)$ the space of measurable functions from $S$ to
$\mathbb{R}$, $\mathcal{M}(S):=\big{\\{}f:S\longrightarrow\mathbb{R}|\text{
$f$ is
$\big{(}\mathcal{S},\mathcal{B}(\mathbb{R})\big{)}$-measurable}\big{\\}}$, and
consider a function $g_{\theta}$ to be indexed by $\theta\in\Theta$ in the
sense of the following mapping

$g:\Theta\longrightarrow\mathcal{M}(\mathbb{R}^{p}),\quad\theta\longmapsto
g_{\theta}.$

Given this, we then assume that each $y_{i}$ is a realization of a real-valued
random variable $Y$ and

$\mathbb{E}[Y|X]=g_{\theta}(X)$ (1)

where the function $g_{\theta}:\mathbb{R}^{p}\longrightarrow\mathbb{R}$ is at
least once partially differentiable w.r.t. each metric element of $x_{[I]}$
$\forall\theta\in\Theta$.

######  Note.

In the following, we will refer to both $g_{\theta}$ and
$g_{\theta}(x_{1},...,x_{p})$, i.e. the expected value of $Y$ given specific
regressor values, as expectation function or expectation, even though
particularly $g_{\theta}(x_{1},...,x_{p})$ is commonly referred to as _(model)
prediction_ in the literature.

##  2 Definition of the proposed quantities

This section will provide definitions of, among others, expectation plots and
generalized marginal effects. Throughout, the underlying idea is that one may
utilize probability measures to average over different functions of the
regressor vector $X$, with the choice of probability measures depending on the
axiomatic assumption made regarding the dependence structure of the
regressors. While those axiomatic assumptions will be introduced in section
2.1, some additional conditions need to be placed on the probability measures.
Specifically, for some function $h:\mathbb{R}^{d}\longrightarrow\mathbb{R}$,
$d\in\mathbb{N}_{>0}$, and set $D\subseteq\mathbb{R}^{\tilde{d}}$,
$d\geq\tilde{d}\in\mathbb{N}_{>0}$, we consider probability measures $\mu$
that satisfy the following requirements:

  * (M1)

$\mu$ is a probability measure on
$\big{(}\mathbb{R}^{\tilde{d}},\mathcal{B}(\,\mathbb{R}^{\tilde{d}}\,)\big{)}$.

  * (M2)

$\text{supp}(\mu)\subseteq D$, if required as a result of $\mu$ being
normalized111See definition A.1 in appendix A for a formal definition of
_normalized_ in this context. w.r.t. $D$.

  * (M3)

$\int_{\mathbb{R}^{\tilde{d}}}|h(x)|d\mu(x)<\infty,$ if $\tilde{d}=d$, or
$\forall
x_{a}\in\mathbb{R}^{d-\tilde{d}}:\int_{\mathbb{R}^{\tilde{d}}}|h(x_{a},x_{b})|d\mu(x_{b})<\infty$,
if $\tilde{d}<d$.222See remark A.4 in appendix A for details on the notation
$h(x_{a},x_{b})$.

######  Note.

While we use the term ‘averaging’ throughout, by choosing $\mu$ as the Dirac
measure $\delta_{x}$ for $x\in\mathbb{R}^{\tilde{d}}$ one of course _fixes_
regressor values at the point $x$.

In this work, we propose a shared method of deriving point estimates and
credible sets for all introduced quantities, which will be detailed in section
2.3. To this end, every quantity will be defined as a function of the
parameter vector $\theta$ in sections 2.1 and 2.2.

###  2.1 Expectation plots and individualized expectation

We begin with an accessible and already prevalent visualization technique,
versions of which are referred to under different names, such as _prediction
plots_ , _marginal plots_ , see e.g. Cook & Weisberg (1997);Apley & Zhu
(2020), and _partial dependence plots_ , see e.g. Friedman (2001);Zhao &
Hastie (2021). Given the setting of section 1.1, these terms refer to
functions which, for any given value $\theta\in\Theta$, visualize the
expectation function on the Cartesian plane, either as a line in the case of
$X^{I}$ being metric or as $d_{I}+1$ points for a categorical $X^{I}$, both
representing the expected value of $Y$ at a value or category, respectively,
of the regressor of interest, and averaged, if applicable, over the regressors
mapped to $M$ and $C$.

To incorporate any such function into our formal framework, we define the
function

$g_{avg}^{[I]}:\Theta\times\mathbb{R}^{d_{I}}\longrightarrow\mathbb{R},\quad(\theta,x_{[I]})\longmapsto\int_{\bar{\mathbb{X}}}g_{\theta}(x)d\bar{\mu}(x_{[\text{MC}]})\,,$
(2)

where $x_{[\text{MC}]}\in\mathbb{R}^{p-d_{I}}$ denotes the vector of
realizations of those regressors that are not the regressor of interest, and
$\bar{\mu}$ is a measure satisfying requirements (M1)-(M3) w.r.t. $g_{\theta}$
and some set $\bar{\mathbb{X}}\subseteq\mathbb{R}^{p-d_{I}}$
$\forall\theta\in\Theta$, and chosen according to one of the following
axiomatic assumptions, unless the regressor of interest is the only regressor
in which case $g_{avg}^{[I]}\hat{=}g_{\theta}$.

#####  Axiomatic assumption (A.I)

_All regressors are independent variables_.
In this case, $\bar{\mu}$ is a product measure, specifically

$\displaystyle\int_{\bar{\mathbb{X}}}g_{\theta}(x)$ $\displaystyle
d\bar{\mu}(x_{[\text{MC}]})=$
$\displaystyle\int_{\mathbb{X}_{\scaleto{M_{1}}{5pt}}}...\int_{\mathbb{X}_{\scaleto{M_{m}}{5pt}}}\int_{\mathbb{X}_{\scaleto{C_{1}}{5pt}}}...\int_{\mathbb{X}_{\scaleto{C_{c}}{5pt}}}g_{\theta}(x)d\mu_{\scaleto{C_{c}}{5pt}}(x_{[\scaleto{C_{c}}{5pt}]})...d\mu_{\scaleto{C_{1}}{5pt}}(x_{[\scaleto{C_{1}}{5pt}]})d\mu_{\scaleto{M_{m}}{5pt}}(x_{[\scaleto{M_{m}}{5pt}]})...d\mu_{\scaleto{M_{1}}{5pt}}(x_{[\scaleto{M_{1}}{5pt}]})\,$

where, to satisfy requirements (M1)-(M3), for each regressor in $M$, i.e.
$\forall i\in\\{1,...,m\\}$, an appropriate set
$\mathbb{X}_{M_{i}}\subseteq\mathbb{R}$ and a probability measure
$\mu_{M_{i}}$ on $\big{(}\mathbb{R},\mathcal{B}(\mathbb{R})\big{)}$ with
$\text{supp}(\mu_{M_{i}})\subseteq\mathbb{X}_{M_{i}}$ (if required as a result
of being normalized) and for each regressor in $C$, i.e. $\forall
j\in\\{1,...,c\\}$, an appropriate set
$\mathbb{X}_{C_{j}}\subseteq\mathbf{e}^{(d_{C_{j}})}$ and a probability
measure $\mu_{C_{j}}$ on
$\big{(}\mathbf{e}^{(d_{C_{j}})},\mathcal{P}\big{(}\mathbf{e}^{(d_{C_{j}})}\big{)}\big{)}$
with $\text{supp}(\mu_{C_{j}})\subseteq\mathbb{X}_{C_{j}}$ are chosen in a way
so that $\forall(\theta,x_{[I]})\in\Theta\times\mathbb{R}^{d_{I}}$ the
following holds:333In addition to reasonably limiting the choices of sets and
measures, this requirement ensures that, if one or more measures are defined
via a density, $\bar{s}$ may be computed as an iterated integral by Fubini’s
theorem, see Fubini (1907).

$\bigintss_{\mathbb{R}^{p-d_{I}}}\big{|}g_{\theta}(x)\big{|}\,d\big{(}\mu_{M_{1}}\times...\times\mu_{M_{m}}\times\mu_{C_{1}}\times...\times\mu_{C_{c}}\big{)}<\infty\,.$

#####  Axiomatic assumption (A.II)

_The regressors are not necessarily independent but follow a joint
distribution_. In this case, let $\mu_{X}$ denote the measure associated444See
remark A.2 in appendix A for a formal explanation of this terminology. with
the given or assumed joint distribution of the regressors $X$. Specifically,
one has two different options for the choice of $\bar{\mu}$, given by the
following.

  * (A.II′)

Choose $\bar{\mu}$ as the measure associated with the marginal distribution
obtained by marginalizing out the regressor of interest. The regressor of
interest is then treated as an independent variable.

  * (A.II′′)

Choose $\bar{\mu}$ as the measure associated with the conditional distribution
of all regressors but the regressor of interest given $X^{I}$, denoted by
$\bar{\mu}_{X_{[\text{MC}]}|X^{I}}$.

Under assumptions (A.II′) and (A.II′′), requirement (M1) is implicitly met.
However, one may still want to normalize $\bar{\mu}$ w.r.t. to some set
$\bar{\mathbb{X}}$, either because only the effect on this set is of interest
or because requirement (M3) does not hold for
$\bar{\mathbb{X}}=\text{supp}(\bar{\mu})$. In such a case, the sets
$\mathbb{X}_{M_{i}},\,i\in\\{1,...,m\\}$ and
$\mathbb{X}_{C_{j}},\,i\in\\{1,...,c\\}$ are chosen as in (A.I), but
$\bar{\mu}$ is not a product measure and therefore needs to be normalized
directly w.r.t.
$\bar{\mathbb{X}}=\mathbb{X}_{M_{1}}\times...\times\mathbb{X}_{M_{m}}\times\mathbb{X}_{C_{1}}\times...\times\mathbb{X}_{C_{c}}$.

Having defined a quantity that gives the average expected value of the target
variable as a function of the value of the regressor of interest, the natural
next step is additionally averaging over that function w.r.t. some probability
measure for the regressor of interest in order to quantify the _average
expected value of $Y$ given certain ranges of regressor values_. To that end,
the following defines what we refer to as ‘individualized expectation’.

######  Definition 1.

Given the setting of section 1.1, we define the _individualized expectation_
as the following function

$\displaystyle
e:\Theta\longrightarrow\mathbb{R},\quad\theta\longmapsto\begin{cases}\bigintsss_{\mathbb{X}_{I}}g_{\theta}(x)\,d\mu_{I}(x),&\text{if
$p=d_{I}$,}\\\\[10.0pt]
\bigintsss_{\mathbb{X}_{I}}g_{avg}^{[I]}(\theta,x)\,d\mu_{I}(x),&\text{otherwise,}\end{cases}$
(3)

where $\mu_{I}$ is a measure that satisfies requirements (M1)-(M3) w.r.t.
$g_{\theta}$ and $\mathbb{X}_{I}\subseteq\mathbb{R}^{d_{I}}$
$\forall\theta\in\Theta$, and $g_{avg}^{[I]}$ is given depending on the chosen
axiomatic assumption.

######  Note.

Under assumption (A.II′′), choosing $\mu_{I}$ as the measure associated with
the marginal distribution of $X^{I}$ under the joint distribution of all
regressors amounts to calculating the integral over the expectation function
w.r.t. the (normalized) joint distribution. This follows since by (Park, 2017,
theorem 4.3.6.) these choices result in the individualized expectation
$e(\theta)$ being given by
$\int_{\mathbb{X}_{I}\times\bar{\mathbb{X}}}g_{\theta}(x)d\mu_{X}(x)$, with
$\mu_{X}$ denoting the measure associated with the given or assumed joint
distribution of all regressors, normalized, if applicable, w.r.t.
$\mathbb{X}_{I}\times\bar{\mathbb{X}}$. When specifying different choices of
sets and measures to integrate over and w.r.t., we write the corresponding
individualized expectation under all assumptions as
$e\big{(}\theta\big{|}\mathbb{X}_{I},\bar{\mathbb{X}},\mu_{I},\bar{\mu}\big{)}$,
or $e\big{(}\theta\big{|}\mathbb{X}_{I},\mu_{I}\big{)}$ if $p=d_{I}$.

###  2.2 Slope of expectation and generalized marginal effects

In the previous section, two functions were defined: First, a function of the
parameter vector _and_ the regressor of interest quantifying the average
expected value of the target variable and, second, a function of _only_ the
parameter vector quantifying the value of the first function averaged over a
certain set. Next, we aim to analogously define two functions quantifying not
the average expected value of $Y,$ but the _average effect of the regressor of
interest on $Y.$_ More precisely, we are interested in quantifying the average
‘effect’ as represented by the average slope, specifically the average value
of the partial derivative for a metric regressor of interest and the average
difference between each non-reference category and the reference category for
a categorical regressor of interest. We refer to these two new functions as
_slope of expectation_ and _generalized marginal effect_ and the remainder of
this section gives their formal definition.

######  Definition 2.

Given the setting from section 1.1 and

$s(\theta,x):=\begin{cases}\dfrac{\partial g_{\theta}}{\partial
x_{[I]}}(x),&\text{if $X^{I}$ is a metric regressor,}\\\\[10.0pt]
g_{\theta}(x)-g_{\theta}(\left.x\right|_{x_{[I]}=\mathbf{0}}),&\text{otherwise,
i.e. if $X^{I}$ is a categorical regressor,}\end{cases}$ (4)

we define the _slope of expectation_ as the following function

$\displaystyle\bar{s}:\Theta\times\mathbb{R}^{d_{I}}\longrightarrow\mathbb{R},\quad(\theta,x_{[I]})\longmapsto\begin{cases}s(\theta,x),&\text{if
$p=d_{I}$,}\\\
\bigintss_{\bar{\mathbb{X}}}s(\theta,x)d\bar{\mu}(x_{[\text{MC}]}),&\text{otherwise,}\end{cases}$
(5)

where, correspondingly to eq. 2, $\bar{\mu}$ is a measure satisfying
requirements (M1)-(M3) w.r.t. $s(\theta,\boldsymbol{\cdot})$ and
$\bar{\mathbb{X}}\subseteq\mathbb{R}^{p-d_{I}}$ $\forall\theta\in\Theta$, and
chosen according to one of the axiomatic assumptions (A.I), (A.II′), or
(A.II′′).

######  Note.

Under assumption (A.II′′), eq. 5 needs to be modified as follows for cases
where the regressor is not the only regressor:

$\displaystyle(\theta,x_{[I]})$
$\displaystyle\longmapsto\begin{cases}\int_{\bar{\mathbb{X}}}s(\theta,x)\bar{\mu}_{X_{[\text{MC}]}|X^{I}}(dx_{[\text{MC}]},x_{[I]}),&\text{if
$X^{I}$ is a metric regressor,}\\\\[10.0pt]
\begin{array}[]{l}\int_{\bar{\mathbb{X}}}g_{\theta}(x)\bar{\mu}_{X_{[\text{MC}]}|X^{I}}(dx_{[\text{MC}]},x_{[I]})\,-\\\\[2.0pt]
\int_{\bar{\mathbb{X}}}g_{\theta}(\left.x\right|_{x_{[I]}=\mathbf{0}})\bar{\mu}_{X_{[\text{MC}]}|X^{I}}(dx_{[\text{MC}]},\mathbf{0}),\end{array}&\text{otherwise.}\end{cases}$

However, in the interest of readability, we use only the notation of eq. 5 in
any definition or statement which applies under all assumptions.

######  Remark 2.1.

Equivalently to $g_{avg}^{[I]}$, for any given value $\theta\in\Theta$, the
slope of expectation
$\bar{s}(\theta,\boldsymbol{\cdot}):\mathbb{R}^{d_{I}}\longrightarrow\mathbb{R},\,p\mapsto\bar{s}(\vartheta,p)$
may be nicely plotted on the Cartesian plane, either as a line in the case of
the regressor of interest being metric or as $d_{I}$ points representing the
average difference in expectation to the reference category for each other
category.

Before giving our proposed definition of generalized marginal effects, the
following briefly addresses our approach to cases where the regressor whose
effect is of interest is involved in an interaction term.

##### Interaction terms

In existing methodology, marginal effects have only been defined to quantify
the effect of a regressor of interest including all interactions it is
involved in. This makes perfect sense from an interpretability perspective, as
even in linear regression the $\beta$-coefficients for interactions between
metric regressors have no easily accessible interpretation. However, when
comparing models, the option to isolatedly quantify the main and the
interaction effect of a regressor of interest might well become relevant.
Therefore, solely in the interest of comparability, we include in our
definition of generalized marginal effects the option of deriving separate
main and interaction effects under the simplifying666See appendix B for a
detailed motivation and discussion of this assumption. assumption that
interaction terms behave independently of the regressors involved in them.
Specifically, for interactions between metric regressors, we propose to treat
the interaction term as a new regressor, with the probability measures chosen
appropriately under each assumption, see section B.1 for further details.
Meanwhile, for interactions between categorical regressors, we propose to
combine the individual regressors and interactions into one new regressor-
vector, whose dimension we again denote by $d_{I}$, and denote each (product
of) variable(s) as $X^{I}_{l}$, $1\leq l\leq d_{I}$, so the entire new
regressor is written as $X^{I}_{cat}=(X^{I}_{1},...,X^{I}_{d_{I}})^{\top}$.
Additionally, we require the following two vectors $\forall
l\in\\{1,...,d_{I}\\}$ to calculate an appropriate distance as slope:
$\mathbf{v}_{l}\in\\{0,1\\}^{d_{I}}$, which has $1$ as the $l$th entry as well
as all entries representing categories that are included in $X^{I}_{l}$; and
$\mathbf{ref}_{l}\in\\{0,1\\}^{d_{I}}$, which has $0$ as every entry but those
entries representing categories that are included in $X^{I}_{l}$. If
$X^{I}_{l}$ is not an interaction term,
$\mathbf{ref}_{l}=\mathbf{0}\in\mathbb{R}^{d_{I}}$. Example B.1 in appendix B
provides an intuitive illustration of these vectors. Please note that we would
not recommend computing interaction effects separately in settings where the
goal is to maximize interpretability.

The following now gives our definition of generalized marginal effects.

######  Definition 3.

Given the setting from section 1.1 and a _slope of expectation_ $\bar{s}$ as
defined in definition 2, the generalized marginal effect $\Delta_{s}$ is given
by the following functions:

  1. (i)

For a _metric regressor of interest_ , given a set
$\mathbb{X}\subseteq\mathbb{R}$ and measure $\mu$ that satisfies requirements
(M1)-(M3) w.r.t. $\bar{s}(\theta,\boldsymbol{\cdot})$ and $\mathbb{X}$
$\forall\theta\in\Theta$, and, if one is interested in separately calculating
the effect for interaction terms between metric regressors, chosen according
to section B.1, $\Delta_{s}$ is defined as the function

$\displaystyle\Delta_{s}:\Theta\longrightarrow\mathbb{R},\quad\theta\longmapsto\int_{\mathbb{X}}\bar{s}(\theta,x_{[I]})d\mu(x_{[I]})\,.$
(6)

######  Note.

Under axiomatic assumption (A.II′′), the most natural choice of $\mu$ is the
measure associated with the marginal distribution of $X^{I}$, since this
definition then amounts to integration over the derivative w.r.t. the
(normalized) joint distribution as was detailed in definition 1.

  2. (ii)

For a _categorical regressor of interest_ with $d_{I}+1$ categories, or
$X^{I}_{cat}$ as defined above for cases where the effect of a categorical
regressor (included in) an interaction with (an-) other categorical
regressor(s) is of interest, $\Delta_{s}$ is defined as the function

$\displaystyle\Delta_{s}:\Theta\longrightarrow\mathbb{R}^{d_{I}},\quad\theta\longmapsto\begin{pmatrix}\Delta_{s}^{[1]}(\theta)\\\
\vdots\\\
\Delta_{s}^{[d_{I}]}(\theta)\end{pmatrix}:=\begin{pmatrix}\bar{s}(\theta,\mathbf{v}_{1})&-&\bar{s}(\theta,\mathbf{ref}_{1})\\\
&\vdots&\\\
\bar{s}(\theta,\mathbf{v}_{d_{I}})&-&\bar{s}(\theta,\mathbf{ref}_{d_{I}})\end{pmatrix}\,.$
(7)

Here, we utilize the notation of $\Delta_{s}^{[j]}$, $1\leq j\leq d_{I}$, in
order to later be able to define point estimates and uncertainty regions in an
elementwise fashion.

######  Note.

When the categorical regressor of interest is not involved in any interactions
or one does not want to separately compute main and interaction effects,
$\bar{s}(\theta,\mathbf{ref}_{j})=0$ $\forall j\in\\{1,...,d_{I}\\}$, and,
therefore,
$\Delta_{s}(\theta)=\big{(}\bar{s}(\theta,\mathbf{e}_{1}),...,\bar{s}(\theta,\mathbf{e}_{d_{I}})\big{)}^{\top}$.

  3. (iii)

The definition of $\Delta_{s}$ for cases where one is interested in separately
quantifying interaction effects between metric and categorical regressors may
be found in section B.2, specifically definition B.2.1, where we have decided
to outsource it given the rather complex groundwork required to reach a full,
formal definition.

In linear regression, the generalized marginal effects under assumptions (A.I)
and (A.II’) will always be equal to the $\beta$-coefficients as the following
theorem proves.

######  Theorem 1.

Consider the case of $g_{\theta}$ being a linear polynomial in each variable
representing a regressor (category), and $\theta$ given by
$(\beta,v)^{\top}\in\Theta$, where $\beta$ denotes the vector containing the
linear polynomial’s coefficients and $v$ denotes elements of the parameter
vector that do not appear in the function term of $g_{\theta}$, such as the
error variance. Given that the regressor of interest is either not involved in
an interaction or $\Delta_{s}$ is used to separately quantify the main and
interaction effects, the following holds under assumptions (A.I) and
(A.II$\,{}^{\prime}$) as well as assumption (A.II$\,{}^{\prime\prime}$), if
the regressor of interest is a metric variable or interaction. Any
$\Delta_{s}(\theta)$ and $\Delta_{s}^{[j]}(\theta)$, $j\in\\{1,...,d_{I}\\}$,
defined in definition 3 is equal to the $\beta$-coefficient of the
corresponding regressor of interest, or element thereof, respectively.

######  Remark 2.2.

Notation-wise, we have so far not specifically considered the case of the
specification of sets and measures being more intuitive for transformed
versions of some or all regressors. Naturally, in such cases, one may treat
the transform of each concerned element of $X$ as a new regressor and
subsequently apply all methods as presented.

###  2.3 Point estimates and uncertainty regions

We adopt a Bayesian view on uncertainty as it appears to be the most natural
one in this context. Specifically, we treat the parameter $\theta\in\Theta$ as
a random variable and derive a point estimate and credible set directly for
the random variable defined by $q\circ\theta$, where $q$ denotes any of the
quantities proposed thus far.

For frequentist inference results, where no posterior distribution for
$\theta$ is available, we propose to follow the suggestion from Gelman &
Pardoe (2007) and, given a point estimate $\hat{\theta}$ and covariance matrix
$\Sigma_{\hat{\theta}}$ for $\theta$, assume that $\theta\sim
N(\hat{\theta},\Sigma_{\hat{\theta}})$. Throughout, we denote by
$\mu_{\theta}$ the probability measure that is associated either with a
posterior distribution, resulting from Bayesian inference, or the
$N(\hat{\theta},\Sigma_{\hat{\theta}})$ distribution, with frequentist point
estimate $\hat{\theta}$ and covariance matrix $\Sigma_{\hat{\theta}}$.

######  Remark 2.3.

The functions $e:\Theta\longrightarrow\mathbb{R}$,
$\Delta_{s}:\Theta\longrightarrow\mathbb{R}$, and
$\Delta_{s}^{[j]}:\Theta\longrightarrow\mathbb{R}$, $j\in\\{1,...,d_{I}\\}$,
are clearly all Borel-measurable. Therefore, given a random variable $\theta$,
$e\circ\theta$, $\Delta_{s}\circ\theta$, and $\Delta_{s}^{[j]}\circ\theta$ are
again random variables and we denote the measures associated with their
respective distributions by $\mu_{e(\theta)}$, $\mu_{\Delta_{s}(\theta)}$, and
$\mu_{\Delta_{s}^{[j]}(\theta)}$, respectively. Meanwhile, for any fixed
realization of $X^{I}$, the functions
$g_{avg}^{[I]}(\boldsymbol{\cdot},x):\Theta\longrightarrow\mathbb{R}$ and
$\bar{s}(\boldsymbol{\cdot},x):\Theta\longrightarrow\mathbb{R}$ are also
Borel-measurable $\forall x\in\mathbb{R}^{p}$, equivalently allowing for the
computation of a point estimate and credible set for any fixed
$x\in\mathbb{R}$, if the regressor of interest is metric, or
$x\in\mathbf{e}^{(d_{I})}$, if the regressor of interest is categorical.

Given an assumed (posterior) distribution for $\theta$, the choice of point
estimate and credible set is largely a matter of personal preference. We would
suggest respectively utilizing the _mean_ and _highest density region_ , or
HDR, as put forward by Hyndman (1996), for which a precise definition in the
current setting is given by definition A.2 in appendix A. However, other
choices such as median and equal-tailed interval would also be appropriate and
may be used to replace the mean and HDR, respectively. To emphasize this
point, we write _$100(1-\alpha)$ %-uncertainty region_ instead of
$100(1-\alpha)$%-HDR in the following definitions.

######  Definition 4.

Given the fully specified functions
$e\big{(}\theta\big{|}\mathbb{X}_{I},\bar{\mathbb{X}},\mu_{I},\bar{\mu}\big{)}$
and $\Delta_{s}(\theta)$, we define

  1. (i)

the point estimates $\widehat{e}$ and $\widehat{\Delta_{s}}$ as
$\widehat{e}:=\mathbb{E}[e(\theta)]=\int_{\Theta}e(x)d\mu_{\theta}(x)$ and

$\displaystyle\widehat{\Delta_{s}}:=\mathbb{E}[\Delta_{s}(\theta)]=\begin{cases}\int_{\Theta}\Delta_{s}(x)d\mu_{\theta}(x)&,\text{
if $\Delta_{s}$ has image $\mathbb{R}$}\\\\[10.0pt]
\begin{pmatrix}\int_{\Theta}\Delta_{s}^{[1]}(x)d\mu_{\theta}(x)\\\ \vdots\\\
\int_{\Theta}\Delta_{s}^{[d]}(x)d\mu_{\theta}(x)\end{pmatrix}&,\text{ if
$\Delta_{s}$ has image $\mathbb{R}^{d}$, $d\in\mathbb{N}$,}\end{cases}$

respectively.

  2. (ii)

for some $\alpha\in(0,1)$, the credible set $C_{e}(\alpha)$ as the
$100(1-\alpha)$%-uncertainty region of $\mu_{e(\theta)}$; and the credible set
$C_{\Delta_{s}}(\alpha)$ as the $100(1-\alpha)$%-uncertainty region of
$\mu_{\Delta_{s}(\theta)}$, if $\Delta_{s}$ has image $\mathbb{R}$, or the
collection of $100(1-\alpha)$%-uncertainty regions of each
$\mu_{\Delta_{s}^{[j]}(\theta)}$, $j\in\\{1,...,d\\}$, if $\Delta_{s}$ has
image $\mathbb{R}^{d}$, $d\in\mathbb{N}$. In the last case, we denote by
$C^{[j]}_{\Delta_{s}}(\alpha)$, the uncertainty region around the $j$th entry
of the point estimate $\widehat{\Delta_{s}}$.

Analogously, for any fixed $x\in\mathbb{R}$, if the regressor of interest is
metric, or $x\in\mathbf{e}^{(d_{I})}$, if the regressor of interest is
categorical, we can define (i) the point estimates for $g_{avg}^{[I]}$ and
slope of expectation $\bar{s}$ as
$\mathbb{E}\big{[}g_{avg}^{[I]}(\theta,x)\big{]}$ and
$\mathbb{E}\big{[}\bar{s}(\theta,x)\big{]}$, respectively; as well as (ii) for
some $\alpha\in(0,1)$, the credible sets $C_{g_{avg}^{[I]}|x}(\alpha)$ and
$C_{\bar{s}|x}(\alpha)$ as the $100(1-\alpha)$%-uncertainty regions of the
measures associated with the distributions of $g_{avg}^{[I]}(\theta,x)$ and
$\bar{s}(\theta,x)$, respectively.

######  Remark 2.4.

Just like $g_{avg}^{[I]}$ and $\bar{s}$ may be plotted for any given value
$\theta\in\Theta$, the corresponding point estimate and
$100(1-\alpha)$%-uncertainty regions may be plotted as functions of the
regressor of interest.

###  2.4 Visualizing sampling uncertainty

In this section, we will briefly define a method of visualizing the
distribution of the target variable $Y$ given an individualized expectation,
given that a parametric distributional assumption has been made. We include
this methodology since, firstly, it is a very natural extension of the
quantities defined thus far and, secondly, we believe a simple visual
comparison of how the assumed distribution of the target variable changes for
different individualized expectations could improve researchers’ understanding
of model results, for which we provide one possible example in section 4.2.
While we further believe that it could be worthwhile to extend the current
theory to settings without parametric distributional assumptions, the current
work is limited to the following definition.

######  Definition 5.

Given the setting of section 1.1 and a parametric distributional assumption
for $Y$, given by a conditional density or probability function
$p(y|\theta,x)$ with mean $\upmu$, we define for an individualized expectation
$e\big{(}\theta\big{|}\mathbb{X}_{I},\bar{\mathbb{X}},\mu_{I},\bar{\mu}\big{)}$

  1. (i)

the density or probability function of $Y$ under $e$ as
$p_{e}(y|\theta):=p\big{(}y;\upmu\text{$=$}e(\theta),\upsilon\big{)}\,,$ where
$p\big{(}y;\upmu,\upsilon\big{)}$ denotes the, if necessary re-parametrized,
version of $p(y|\theta,x)$ with its mean $\upmu$ one entry of $\theta$ and, if
applicable, the remaining parameter entries represented by $\upsilon$.

  2. (ii)

the density or probability function of the _individualized predictive
distribution_ of $Y$ and $\theta$ under $e$ as the function

$\Pi(y,\theta):=p_{e}(y|\theta)\pi(\theta)$ (8)

where $\pi$ denotes the density or probability function via which
$\mu_{\theta}$, the probability measure that is associated either with a
posterior distribution, resulting from Bayesian inference, or the
$N(\hat{\theta},\Sigma_{\hat{\theta}})$ distribution, with frequentist point
estimate $\hat{\theta}$ and covariance matrix $\Sigma_{\hat{\theta}}$, is
defined.

Equivalently to $g_{avg}^{[I]}$ and $\bar{s}$,
$\Pi(y,\boldsymbol{\cdot}):\Theta\longrightarrow\mathbb{R}$ is a measurable
function for every fixed $y\in\mathbb{R}$ and, therefore, the point estimate
$\mathbb{E}_{\theta}[\Pi(y,\theta)]:=\int_{\Theta}\Pi(y,\theta)d\theta$ as
well as the corresponding $100(1-\alpha)$%-uncertainty regions may be plotted
as a function of $y$, see fig. 4 for an example. Note that the quantity
$\mathbb{E}_{\theta}[\Pi(y,\theta)]$ may simply be seen as a _marginal
posterior predictive distribution_ , specifically, marginalized w.r.t. the
measure associated with the chosen distribution of the regressors $X$.

##  3 Special specifications and transforms

Having established a formal framework for a wide range of methods which may be
used to visualize and quantify the results of parametric models, this section
will now, first, detail how many established reporting and visualization
methods from different disciplines fit into this framework, and, subsequently,
make some suggestions of specifications which we believe to have beneficial
properties regarding both interpretability and comparability. Importantly,
however, we do not claim that any one specification is universally superior to
others. On the contrary, making it possible to specify any proposed quantity
according to the specific research question at hand as well as prior knowledge
about the distribution of the regressors is one very important contribution of
the developed framework. Throughout, we denote by $U(a,b)$ the continuous
uniform distribution on $[a,b]\in\mathcal{B}(\mathbb{R})$, by $\delta$ the
Dirac measure, and use the term ‘measure associated with the (joint) empirical
distribution’ in the sense of the following definition.

######  Definition 6.

Within the context of this work, given a sequence of regressor-observations
$\\{\mathbf{x}_{i}\\}_{i=1,...,n}=:\mathcal{D}_{X}$, $n\in\mathbb{N}$, as
introduced in section 1.1 and
$n_{\mathcal{D}_{X}}(x):=\sum_{s\in\mathcal{D}_{X}}\mathds{\large 1}_{x=s}$
for any $x\in\mathbb{R}^{p}$, we refer to the probability measure defined via
the probability function

$p(x)=\frac{n_{\mathcal{D}_{X}}(x)}{n}\cdot\mathds{\large
1}_{x\in\mathcal{D}_{X}}$

as the _measure associated with the joint empirical distribution_ of all
regressors. Furthermore, let $\mathcal{D}_{X}^{[j]}$, $j\in\\{1,...,1+m+c\\}$,
denote the sequence of observations of some regressor in $I\cup M\cup C$.
Correspondingly, we refer to the probability measure defined via the
probability function $p(x)=n^{-1}\cdot
n_{\mathcal{D}^{[j]}_{X}}(x)\cdot\mathds{\large
1}_{x\in\mathcal{D}_{X}^{[j]}}$ as the _measure associated with the empirical
distribution_ of that regressor. Note that the support of these measures is
given by the sets $\\{\mathbf{x}_{i}|\mathbf{x}_{i}\in\mathcal{D}_{X}\\}$ and
$\\{x_{[j]}|x_{[j]}\in\mathcal{D}^{[j]}_{X}\\}$, respectively.

###  3.1 Connection to existing methods

####  3.1.1 Marginal Effects and Adjusted Predictions

The concepts of _marginal effects_ and _adjusted predictions_ originated in
economics, but have since become popular methods in other disciplines. This
section will now lay out how exactly the individualized expectation $e$ and
generalized marginal effects $\Delta_{s}$ proposed in the current work may be
specified to equal the three commonly established versions of adjusted
predictions and marginal effects, the definitions of which may be found in
Williams (2012).

##### Average (AAPs/AMEs)

_Average Adjusted predictions_ amount to calculating $e(\hat{\theta})$ under
axiomatic assumption (A.II′′), with the joint empirical distribution of all
regressors being used as $\mu_{X}$. Equivalently, the commonly used _Average
Marginal Effects_ for a metric regressor of interest amount to calculating
$\Delta_{s}(\hat{\theta})$ under axiomatic assumption (A.II′′), with $\mu_{X}$
again the joint empirical distribution of all regressors. However, for a
categorical regressor of interest, the commonly used _Average Marginal
Effects_ amount to calculating the $\Delta_{s}(\hat{\theta})$ under axiomatic
assumption (A.II′) instead of (A.II′′), but with $\mu_{X}$ still chosen as
joint empirical distribution of all regressors.

######  Note.

Interestingly, this difference between the computation of AAPs/AMEs for a
metric regressor of interest and a categorical regressor of interest has so
far not been directly addressed, despite the fact that the marginal effects of
a categorical regressor of interest may of course be quite different when
computed under assumption (A.II′′) instead of (A.II′), see fig. 1 for one
example of this.

##### At Representative values (APRs/MERs)

_Adjusted predictions_ and _Marginal Effects at Representative values_ amount
to calculating $e(\hat{\theta})$ and $\Delta_{s}(\hat{\theta})$, respectively,
under axiomatic assumption (A.I), with the measure for each of the $p$
regressors being chosen as $\delta_{\\{x_{i}^{\text{rep}}\\}}$,
$i\in\\{1,...,p\\}$, where $x_{i}^{\text{rep}}$ denotes the $i$th entry of the
chosen "representative value" $x^{\text{rep}}\in\mathbb{R}^{p}$.

##### At the Means (APMs/MEMs)

_Adjusted predictions_ and _Marginal Effects at the Means_ simply amount to
calculating the APRs and MERs, respectively, with the representative value for
each regressor being chosen as its sample mean. Thereby, APMs and MEMs are
equivalently calculated as $e(\hat{\theta})$ and $\Delta_{s}(\hat{\theta})$,
respectively, under axiomatic assumption (A.I), with the measure for each
regressor being chosen as $\delta_{\\{\bar{x}_{i}\\}}$, $i\in\\{1,...,p\\}$,
where $\bar{x}_{i}$ denotes the $i$th entry of the observed sample mean if $C$
is an empty set.

######  Note.

Williams (2012) does not specify what constitutes the ‘mean’ for regressors
representing categories of a categorical regressor.

######  Remark 3.1.

In Stata StataCorp (2021) and other software modeled after Stata’s marginal
effects methods, standard errors and, thereby, confidence intervals for
adjusted predictions and marginal effects are derived via the delta method.
Importantly, this may lead to issues, e.g. when the adjusted prediction point
estimate in logistic regression lies close to $0$ or $1$, as the confidence
interval may then include impossible values that lie outside of $[0,1]$. In
contrast, the marginaleffects R-package Arel-Bundock (2022) calculates point
estimates and credible sets very similarly to the methods proposed in the
current work for the three commonly established versions of adjusted
predictions and marginal effects.

####  3.1.2 Medicine and Sociology

While the three commonly established versions of marginal effects detailed in
the previous section are meanwhile also being used in other disciplines, such
as medicine and sociology, other effect size measures which likewise fall into
our framework have been proposed in these disciplines. One example of this are
_average predictive comparisons_ , which give the average normalized
difference in expectation resulting from a certain change of the regressor of
interest, and whose place in the current framework is given by the following
proposition.

######  Proposition 1.

The _average predictive comparisons_ for scalar inputs as defined in Gelman &
Hill (2007) correspond to the generalized marginal effect $\Delta_{s}$ for a
metric regressor of interest under assumption (A.II$\,{}^{\prime}$), with
$\mu_{X}$ chosen as the measure associated with the joint empirical
distribution of all regressors and $\mu$ as the measure associated with the
$U\big{(}u^{(lo)},u^{(hi)}\big{)}$ distribution for some chosen
$u^{(lo)},u^{(hi)}\in\mathbb{R}$ with $u^{(lo)}<u^{(hi)}$.

######  Remark.

Gelman & Pardoe (2007) extend the predictive comparison methods of Gelman &
Hill (2007) and remark C.2 in appendix C relates their methodology to our
framework.

In addition, the concept of _predicted change in probability_ , which refers
to the average difference between probabilities for different subgroups as
predicted by a binary logistic model, may always be specified in our
framework. While several variants of this concept have been proposed, the
following proposition gives one formal representative instance of how such a
variant, specifically as defined by Kaufman (1996), may be derived from
quantities defined in this work.

######  Proposition 2.

Consider the case where $Y$ is a binary variable, the regressor of interest is
metric, and, for some linear predictor $\eta$,
$g_{\theta}(X)=\big{(}1+\exp(-\eta(X))\big{)}^{-1}$. Then the quantities
$\partial P$ and $\Delta P$ as defined by Kaufman (1996) correspond to the
quantities $s$ and $\Delta_{s}$ as defined in the current framework for a
subset of possible specifications.

######  Note.

Kaufman (1996) additionally defines standardized versions of these quantities
which may equivalently be applied to $\Delta_{s}$. However, the current work
also contains suggestions for standardizing all proposed quantities in
appendix D.

Predicted change in probability is particularly often used in medical
publications, see, e.g., Waehrer et al. (2020), Fuller & Flores (2015), and
Buchman et al. (2020), particularly as a more interpretable alternative to
odds ratios. Section 4 will provide some further illustrations of how the odds
ratio is not only difficult to interpret but can even become somewhat
misleading.

####  3.1.3 Interpretable machine learning

As mentioned in the introduction, the quantities defined in this work are not
only a generalization of marginal effects but also closely related to many
model-agnostic methods that were recently proposed in the context of
interpretable machine learning. Specifically, within the parametric setting of
section 1.1, _partial dependence-_ and _marginal- plots_ , see, e.g., Apley &
Zhu (2020), are equivalent to $g_{avg}^{[I]}(\hat{\theta},\boldsymbol{\cdot})$
under assumptions (A.II′) and (A.II′′), respectively. Furthermore, _individual
conditional expectation plots_ , see Goldstein et al. (2015), are equivalent
to the $n$ plots resulting from specifying
$g_{avg}^{[I]}(\hat{\theta},\boldsymbol{\cdot})$ under (A.II′) and normalizing
the measure associated with the joint empirical distribution w.r.t. each
observation. Lastly, the following proposition relates the concept of
_accumulated local effects plot_ as proposed by Apley & Zhu (2020) to the
generalized marginal effects defined in the current work.

######  Proposition 3.

Consider the case where, for a given $j\in\\{1,...,p\\}$, (1) $g_{\theta}$ is
not only partially but continuously partially differentiable w.r.t. $x_{j}$
and $x_{[I]}=x_{j}$, i.e. the regressor of interest is metric and the $j$th
observed regressor, and (2) the marginal distribution of $X_{j}$ induces the
measure $\mu_{j}$. Within the parametric setting of section 1.1, the following
then holds for the _uncentred ALE main effect for differentiable $f(\cdot)$_,
$g_{j,\text{ALE}}$, as defined by (Apley & Zhu, 2020, thm. 1) under assumption
(A.II$\,{}^{\prime\prime}$) with $\mu$ chosen not as $\mu_{j}$ but as the
measure associated with the $U\big{(}\min\\{\text{supp}(\mu_{j})\\},z\big{)}$
distribution $\forall z\in\text{supp}(\mu_{j})$

$g_{j,ALE}(z)=\Delta_{s}(\hat{\theta})\cdot\big{(}z-\min\\{\text{supp}(\mu_{j})\\}\big{)}\,.$
(9)

######  Remark.

Additionally, Remark C.3 in appendix C provides more information including a
formal proof regarding the connection of the current theory with the _forward
marginal effects_ as defined by Scholbeck et al. (2022).

###  3.2 Choices of probability measures

As already highlighted, we view the flexibility regarding how exactly the
proposed quantities are averaged over regressor values as a great asset of the
established framework, and, accordingly, there can be no universally
appropriate choices of probability measures. Still, this section will discuss
some properties of the two probability measures which we believe to be the
most convenient regarding interpretability and computation in many settings.

##### Empirical distribution

Most existing methodology that falls into our framework utilizes the measure
associated with the joint empirical distribution, including, incidentally,
proposed estimators for the quantities detailed in section 3.1.3. This option
is not only advantageous because of its computational simplicity, but also the
best choice for $\mu_{X}$ under assumptions (A.II′) and (A.II′′) if one does
not have any prior knowledge regarding the joint distribution of all
regressors. Additionally, one may naturally consider the (joint) empirical
distribution of all or some regressors on a fictive sequence of observations
to specify each quantity as needed.

However, we would like to alert the reader to two possible drawbacks of
utilizing the (joint) empirical distribution. Firstly, averaging the proposed
quantities over a set of observations may often be a reasonable estimate of
the ‘average quantity’, but the result may also be distorted in cases of low
observation count or notable changes in $g_{\theta}$ between observations.
Especially for $\Delta_{s}$ in the case of a metric regressor of interest,
other distribution choices will yield much more interpretable results in this
regard, e.g. the Uniform distribution, which we discuss below. Secondly, the
only possibility of extrapolation beyond the observed regressor values, if one
is interested in that, is utilizing a fictive sequence of observations that
represents the characteristics of the population to which one wishes to
extrapolate. For metric regressors, however, choosing a sequence to average
over rather than, e.g., an interval may often be restrictive in the context of
extrapolation.

##### Uniform distribution

Choosing probability measures as measures associated with the continuous
Uniform distribution will yield an equally weighted average over a clearly
defined set, which is arguably the most accessible version of an average to
interpret. Since equally averaging over a set of discrete points may already
be achieved by utilizing the (joint) empirical distribution on a fictive
sequence of observations, as above, this section will now illustrate the
advantage regarding interpretability for the case of choosing the measure
associated with the continuous Uniform distribution as $\mu$ when deriving
$\Delta_{s}$ for a metric regressor of interest and, additionally, derive a
convenient result regarding interpretability and computability in this
setting.

When the regressor of interest is categorical, $\Delta_{s}^{[j]}$,
$j\in\\{1,...,d_{I}\\}$, simply gives the difference in conditional
expectation of the outcome between each non-reference category and the
reference category, averaged, depending on the choice of assumption and
measure(s), over all other regressors and is therefore easy to interpret and
communicate. When the regressor of interest is metric, the situation becomes
slightly more complicated, however, if one chooses $\mu$ as the measure
associated with the $U(a,b)$, $[a,b]\in\mathcal{B}(\mathbb{R})$, one may
legitimately report the statement ‘on the interval $[a,b]$, the expected value
of $Y$ increases on average by $\widehat{\Delta_{s}}$ when $X^{I}$ increases
by $1$’. Moreover, if $X^{I}$ is considered independent of the remaining
regressors, $\widehat{\Delta_{s}}$ gives nothing more than the _standardized
average difference in expectation between $x_{[I]}=b$ and $x_{[I]}=a$_, i.e.
the average difference in expectation divided by the interval length. The
following proposition formalizes this statement.

######  Proposition 4.

For some $a_{\mathbb{X}},b_{\mathbb{X}}\in\mathbb{R}$ with
$a_{\mathbb{X}}<b_{\mathbb{X}}$, consider the case of the regressor of
interest $X^{I}$ being metric with
$\mathbb{X}=(a_{\mathbb{X}},b_{\mathbb{X}})$ and $\mu$ being chosen or given
as the measure associated with the $U(a_{\mathbb{X}},b_{\mathbb{X}})$
distribution

Then, the following holds under assumptions (A.$I$) and (A.$II^{\prime}$)

$\Delta_{s}(\theta)=\big{(}b_{\mathbb{X}}-a_{\mathbb{X}}\big{)}^{-1}\cdot\Bigg{(}\int_{\bar{\mathbb{X}}}g_{\theta}(\left.x\right|_{x_{[I]}=b_{\mathbb{X}}})-g_{\theta}(\left.x\right|_{x_{[I]}=a_{\mathbb{X}}})d\bar{\mu}(x_{[\text{{MC}}]})\Bigg{)}\,.$
(10)

######  Corollary 1.

Given the setting of proposition 4, the following holds for $|\mathbb{X}|=1$
under assumptions (A.$I$) and (A.$II^{\prime}$)

$\displaystyle\Delta_{s}(\theta)$
$\displaystyle=\mathbb{E}_{X_{\textup{[MC]}}}\Big{[}\mathbb{E}\big{[}Y|X^{I}=b_{\mathbb{X}},X_{\textup{[MC]}}\big{]}\Big{]}-\mathbb{E}_{X_{\textup{[MC]}}}\Big{[}\mathbb{E}\big{[}Y|X^{I}=a_{\mathbb{X}},X_{\textup{[MC]}}\big{]}\Big{]}\,.$
(11)

######  Note.

Separately from the setting of proposition 4, the RHS of above equation may be
written in terms of individualized expectation as

$e\big{(}\theta\big{|}\\{b_{\mathbb{X}}\\},\bar{\mathbb{X}},\delta_{\\{b_{\mathbb{X}}\\}},\bar{\mu}\big{)}-e\big{(}\theta\big{|}\\{a_{\mathbb{X}}\\},\bar{\mathbb{X}},\delta_{\\{a_{\mathbb{X}}\\}},\bar{\mu}\big{)}\,,$
(12)

where $\delta$ denotes the Dirac measure.

Importantly, similar results may not be obtained under assumption (A.II′′), as
the regressor of interest is not treated as independent. A more detailed
reasoning may be found in remark C.1 in appendix C.

###  3.3 Choices of sets for regressor values

Even more than in the case of probability measures, the proper choice of sets
$\mathbb{X}$, $\mathbb{X}_{I}$, and $\bar{\mathbb{X}}$ entirely depends on the
research question that one intends to answer. In this context, we consider the
option of equally averaging the effect over a chosen interval or set of points
under assumption (A.I) one valuable contribution of our work, as it
constitutes a natural and, in the latter case, more interpretable extension of
the _at representative value_ and _at means_ options from section 3.1.1.
Additionally, the following now provides guidance on how to best quantify the
generalized marginal effect in cases where the regressor of interest is metric
and the slope of expectation $\bar{s}$ changes in a relevant manner over the
course of the chosen $\mathbb{X}$.

##### Relevant slope changes for metric regressor of interest

In such cases, we suggest dividing $\mathbb{X}$ into subsets, with the
observed effect being similar within each subset, and calculating
$\Delta_{s}(\theta)$ separately for each subset of $\mathbb{X}$, using the
same measure $\mu$ _but normalized w.r.t. each subset_. Subsequently, one may
report a set of effect sizes in combination with the corresponding range of
values the regressor of interest takes. Note that, in this context, the term
‘relevant’ again very much depends on the setting and could refer to both,
cases where the slope actually changes its sign in the interval of interest
and cases with comparatively small slope changes that are, however,
detrimental to decision making.

###  3.4 Maximizing Comparability

As mentioned in section 2.2, we expect that in certain settings, e.g. multi-
analyst studies, the comparability of model results will be of greater
importance than their interpretability. By facilitating the separate
quantification of main and interaction effects, we have already introduced one
method for such situations; the following will make a further suggestion for
how to maximize the comparability of models addressing the same research
question but either fit on different data sets or with different covariates,
i.e. regressors in $M\cup C$.

When comparing two models that were fit on different data, comparability is
easily maximized by either utilizing the joint empirical distribution of the
combined data sets, if available, or by specifying the exact same probability
measures for all common regressors, regardless of the observed values. In the
case of different covariates, one could either use all available information
by still averaging over all respective regressors or choose to set the
probability measure for every covariate that does not appear in both models to
$\delta_{\\{0\\}}$.

Additionally, in the interest of maximizing comparability, we have included
suggestions for reporting standardized versions of all proposed quantities in
appendix D.

##  4 Applications

This section illustrates the relevance of the proposed framework by first,
highlighting the important role of identifying and communicating the axiomatic
assumptions underlying one’s reporting using data from the clinical trial of
Tortora et al. (2020); and, second, demonstrating how the defined quantities
may be used to solve a common comparability issue in multi-analyst studies by
exemplarily applying them to the setting of Silberzahn et al. (2018).

###  4.1 Application to a clinical trial

In their randomized controlled trial, Tortora et al. (2020) compared the
hospital length of stay (LOS) of an ethnically diverse group of patients with
severe depressive disorders that were either given standard therapy (Group S)
or genetically-guided therapy (Group G). For our first illustrative example,
we followed the sub-analysis of Crutchley & Keuler (2022) in omitting all
observations with LOS$\,\leq 3$ days; and fit a logistic regression model for
the question _how does ethnicity affect the probability of staying
hospitalized longer than the overall median LOS ( $6.625$ days)_, controlling
for the variables age, gender, and group assignment (G or S).

Here, race was taken as the categorical regressor of interest, with _White_
(58.99% of the observations) chosen as the reference category and the
categories _Latinx_ , _Black_ , and _Other_ making up 24.73%, 11.73%, and
4.56% of the observations, respectively. Given the corresponding inference
results, fig. 1 gives the expectation plots and generalized marginal effects
under axiomatic assumptions (A.II′) and (A.II′′) with $\mu_{X}$ taken as the
joint empirical distribution on the observations used to fit the model.

Figure 1: Expectation plots under assumptions (A.II′) (topleft) and (A.II′′)
(topright) and comparison of the corresponding generalized marginal effects
(bottom) with point estimates and 95%-uncertainty regions given as points and
bars, respectively. _White_ was taken as the reference category for the
categorical regressor of interest _race_.

This example highlights two advantages of the proposed framework. Firstly, the
uncertainty regions and point estimates for all quantities retain the
information about uncertainty, particularly uncertainty resulting from a low
observation count for a certain regressor value, given by the inference
regarding the regression parameters. In the current example, the uncertainty
regions for the categories _Black_ and _Other_ are wider than for _Latinx_ ,
since the latter makes up a higher percentage of all observations. Secondly,
as is especially apparent in the bottom plot of fig. 1, the numeric results
for the proposed quantities may be quite different depending on which
axiomatic assumption underlies their calculation, and, most importantly, have
different interpretations. Under assumption (A.II′′), one derives the
_collective_ average expectation, effect, etc. of all observations with the
regressor of interest taking a certain value, while under assumption (A.II′),
by treating the regressor of interest as independent, its impact is largely
detached from the impacts of the remaining regressors. In the current example,
the entry of $\widehat{\Delta_{s}}$ corresponding to the non-reference
category ‘Other’ gives _the difference between the empirical average
probability of hospital stay lasting longer than $6.625$ days if each
observation’s race was changed to ‘Other’ and the empirical average
probability if each observation’s race was changed to ‘White’_ under
assumption (A.II′). Meanwhile, under assumption (A.II′′), the entry of
$\widehat{\Delta_{s}}$ corresponding to the non-reference category ‘Other’
gives _the difference between the empirical average probability of hospital
stay lasting longer than $6.625$ days for those observations where ‘Other’ was
observed as race and the empirical average probability for those observations
where ‘White’ was observed as race._

###  4.2 Application to a multi-analyst study

In a recent comment in Nature, Wagenmakers et al. (2022) argue that the
scientific community needs to start quantifying analytical variability through
multi-analyst projects to address the problem of overconfidence in statistical
results. However, many of the multi-analyst projects that have been completed
so far have struggled with the question of how to achieve this goal if the
analysis teams are free to choose an arbitrary model class, potentially ruling
out the comparison of effect sizes on a common scale, see, e.g., Schweinsberg
et al. (2021) and Hoogeveen et al. (2022). In one of the first multi-analyst
studies, wherein twenty-nine teams of analysts were asked to answer the
question "_Are soccer referees more likely to give red cards to dark-skin-
toned players than to light-skin-toned players?_ "; Silberzahn et al. (2018)
appeared to solve this issue, at least in the setting of their study, by
asking all teams to report their findings in terms of odds ratios. This
allowed them to report the studies results as a forest plot of odds ratios in
(Silberzahn et al., 2018, fig. 3), with the teams ordered by distributional
assumption and all but two being classified as either "Linear", "Poisson", or
"Logistic".

However, closer examination of the analysis strategies reported by each of the
teams reveals that, while the incidence rate ratios resulting from Poisson
models were justifiably taken as equivalent to odds ratios under the _rare
disease assumption_ , the teams that were classified as "Linear" in the
aforementioned forest plot indeed first fit a linear model to answer the
question, but then additionally fit a logistic model in order to be able to
report an odds ratio value. To illustrate how the quantities proposed in the
current work solve the issue of effect size-comparability between different
model classes, we fit a normal, Poisson, and logistic regression model in
which we regressed the number of red cards given per game, the number of red
cards given overall with the number of games used as offset, and the number of
red cards (successes) per games (trials), respectively, on the standardized
mean skin color rating. Figure 2 gives the resulting expectation and slope
plots, with the offset fixed at ln$(1)$ for the Poisson model. Thereby, the
$y$-axes may, in a simplified way, be interpreted as the ‘expected number of
red cards given per game’ and the slope thereof, respectively.

Figure 2: The expectation- (left) and slope- (right) plots for three simple
models. The shaded areas represent the 95%-uncertainty regions as described in
remark 2.4 and were omitted for the slope plots to allow for a better visual
comparison of the line estimates.

This visualization itself illustrates that these three models can indeed be
compared despite their different distributional assumptions. Figure 3 shows
how they may, furthermore, be compared in a forest plot of generalized
marginal effects, and how the same does not hold for odds ratios or
$\beta$-coefficients.

Figure 3: A comparison of three possible reporting methods for the three model
results. For deriving $\Delta_{s}$, $\mu$ was chosen as the measure associated
with the $U(0,1)$ distribution.

Please note that, while odds ratios may obviously not be derived from the
$\beta$-coefficients of linear models, comparing the results of Poisson models
and logistic models in terms of odds ratios, as Silberzahn et al. (2018) did,
is also not generally admissible, but only in settings where the prevalence of
‘successes’ (in this case the event that a red card is given) is very low.
This was the case in the Silberzahn et al. (2018) study. However, in such
situations, the odds ratio, which, in general, is already almost impossible to
sensibly interpret (Sackett et al. (1996)) becomes somewhat misleading.
Plotting odds ratios in a forest plot with a line drawn at $1$, i.e. the odds
ratio value representing ‘no effect’, visually implies that an odds ratio of,
for example, $1.3$ must indicate some non-negligible effect - after all, the
odds for a player with $1$ as standardized average skin color rating are 30%
higher than for a player whose rating is $0$. However, when the probability of
an event, in this case a red card being given, is infinitesimally small, it is
not only approximately equal to the odds but a 30% increase is, by extension,
also infinitesimal. Figure 4 illustrates this by comparing the line estimates
and uncertainty regions of the individualized predictive distribution, as
introduced in section 2.3, under the individualized expectations
$e(\theta|\\{0\\},\delta_{\\{0\\}})$ and $e(\theta|\\{1\\},\delta_{\\{1\\}})$
for the linear and logistic models, since here $p=1$; and under
$e(\theta|\\{0\\},\\{1\\},\delta_{\\{0\\}},\delta_{\\{1\\}})$ and
$e(\theta|\\{1\\},\\{1\\},\delta_{\\{1\\}},\delta_{\\{1\\}})$ for the Poisson
model, where the number of games may be seen as the second regressor $X_{2}$,
which appears as ln$(X_{2})$ in the linear predictor.

Figure 4: The individualized predictive distribution for the linear model
(left) and Poisson & logistic models (right), with the shaded areas
representing the 95%-uncertainty regions.

To summarize, we have now shown that the framework developed in this work is
not only useful to effectively compare effect sizes derived from different
model classes on a common scale; but also to improve the interpretability of
the differences in effect size that one may observe on this common scale.

##  5 Conclusion and Future Work

In this work, we have presented a formal framework for the derivation of a
variety of visualization techniques and effect size measures that advance the
interpretability, comparability, and communicability of results from
parametric regression. We have shown how numerous existing methodologies fall
into this framework and demonstrated how methods derived from it can
concretely be used to improve and clarify the reporting of model results in
various parametric settings. Based on this, we believe the following two
extensions of the current framework to be of particular interest for future
work. Firstly, extending the setting of all methods to include semi- and non-
parametric models, especially survival models. Not only are the hazards in the
interpretation of hazard ratios well known (Hernán (2010)) and could be
avoided by deriving a different effect size measure for proportional hazard
models; but we also believe the developed framework to be particularly
relevant in the context of medical decision making, e.g. in the setting of the
PATH statement formulated by Kent et al. (2020). Secondly, building on
existing work such as Lei et al. (2018) to extend the theory of section 2.3 to
settings without parametric distributional assumptions, both for parametric
and nonparametric models. In this context, one could furthermore define
predictive credible sets around the individualized expectation that take into
account both sampling and estimation uncertainty, with the ultimate aim of
advancing how well statistical results may be communicated not only to fellow
scientists but also to the general public.

## Acknowledgments

The authors gratefully acknowledge funding by LMUexcellence within the
framework of the German Strategy of Excellence.

## References

  * (1)
  * Apley & Zhu (2020) Apley, D. W. & Zhu, J. (2020), ‘Visualizing the effects of predictor variables in black box supervised learning models’, Journal of the Royal Statistical Society Series B 82(4), 1059–1086.
  * Arel-Bundock (2022) Arel-Bundock, V. (2022), marginaleffects: Marginal Effects, Marginal Means, Predictions, and Contrasts.  R package version 0.5.0, URL: https://vincentarelbundock.github.io/marginaleffects/.
  * Buchman et al. (2020) Buchman, T. G., Simpson, S. Q., Sciarretta, K. L., Finne, K. P., Sowers, N., Collier, M., Chavan, S., Oke, I., Pennini, M. E., Santhosh, A., Wax, M., Woodbury, R., Chu, S., Merkeley, T. G., Disbrow, G. L., Bright, R. A., MaCurdy, T. E. & Kelman, J. A. (2020), ‘Sepsis among medicare beneficiaries: 3. the methods, models, and forecasts of sepsis, 2012-2018.’, Crit Care Med 48(3), 302–318.
  * Cassidy et al. (2019) Cassidy, S. A., Dimova, R., Giguère, B., Spence, J. R. & Stanley, D. J. (2019), ‘Failing grade: 89% of introduction-to-psychology textbooks that define or explain statistical significance do so incorrectly’, Advances in Methods and Practices in Psychological Science 2(3), 233–239.
  * Cook & Weisberg (1997) Cook, R. D. & Weisberg, S. (1997), ‘Graphics for assessing the adequacy of regression models’, Journal of the American Statistical Association 92(438), 490–499.
  * Crutchley & Keuler (2022) Crutchley, R. D. & Keuler, N. (2022), ‘Sub-analysis of cyp-guides data: Assessing the prevalence and impact of drug-gene interactions in an ethnically diverse cohort of depressed individuals.’, Front Pharmacol 13, 884213.
  * Friedman (2001) Friedman, J. H. (2001), ‘Greedy function approximation: A gradient boosting machine.’, The Annals of Statistics 29(5), 1189 – 1232.
  * Fubini (1907) Fubini, G. (1907), ‘Sugli integrali multipli.’, Rom. Acc. L. Rend. (5) 16(1), 608–614.
  * Fuller & Flores (2015) Fuller, J. & Flores, L. J. (2015), ‘The risk gp model: The standard model of prediction in medicine’, Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences 54, 49–61.
  * Gelman & Hill (2007) Gelman, A. & Hill, J. (2007), Data Analysis Using Regression and Multilevel/Hierarchical Models, Analytical Methods for Social Research, Cambridge University Press.
  * Gelman & Pardoe (2007) Gelman, A. & Pardoe, I. (2007), ‘2. average predictive comparisons for models with nonlinearity, interactions, and variance components’, Sociological Methodology 37(1), 23–51.
  * Goldstein et al. (2015) Goldstein, A., Kapelner, A., Bleich, J. & Pitkin, E. (2015), ‘Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation’, Journal of Computational and Graphical Statistics 24(1), 44–65.
  * Greenland (1987) Greenland, S. (1987), ‘INTERPRETATION AND CHOICE OF EFFECT MEASURES IN EPIDEMIOLOGIC ANALYSES1’, American Journal of Epidemiology 125(5), 761–768.
  * Hernán (2010) Hernán, M. A. (2010), ‘The hazards of hazard ratios’, Epidemiology 21(1), 13–15.
  * Hoogeveen et al. (2022) Hoogeveen, S., Sarafoglou, A., van Elk, M. & Wagenmakers, E.-J. (2022), ‘Many-analysts religion project: Reflection and conclusion’.
  * Hyndman (1996) Hyndman, R. J. (1996), ‘Computing and graphing highest density regions’, The American Statistician 50(2), 120–126.
  * Kaufman (1996) Kaufman, R. L. (1996), ‘Comparing effects in dichotomous logistic regression: A variety of standardized coefficients’, Social Science Quarterly 77(1), 90–109.
  * Kent et al. (2020) Kent, D. M., Paulus, J. K., van Klaveren, D., D’Agostino, R., Goodman, S., Hayward, R., Ioannidis, J. P. A., Patrick-Lake, B., Morton, S., Pencina, M., Raman, G., Ross, J. S., Selker, H. P., Varadhan, R., Vickers, A., Wong, J. B. & Steyerberg, E. W. (2020), ‘The predictive approaches to treatment effect heterogeneity (path) statement.’, Ann Intern Med 172(1), 35–45.
  * Lei et al. (2018) Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R. J. & Wasserman, L. (2018), ‘Distribution-free predictive inference for regression’, Journal of the American Statistical Association 113(523), 1094–1111.
  * Park (2017) Park, K. (2017), Fundamentals of Probability and Stochastic Processes with Applications to Communications, Springer International Publishing.
  * Sackett et al. (1996) Sackett, D. L., Deeks, J. J. & Altman, D. G. (1996), ‘Down with odds ratios!’, BMJ Evidence-Based Medicine 1(6), 164–166.
  * Scholbeck et al. (2022) Scholbeck, C. A., Casalicchio, G., Molnar, C., Bischl, B. & Heumann, C. (2022), ‘Marginal effects for non-linear prediction functions’, arXiv preprint arXiv:2201.08837 .
  * Schweinsberg et al. (2021) Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O. R., van Aert, R. C., Van Assen, M. A., Liu, Y., Althoff, T., Heer, J., Kale, A. et al. (2021), ‘Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis’, Organizational Behavior and Human Decision Processes 165, 228–249.
  * Silberzahn et al. (2018) Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E. & Nosek, B. A. (2018), ‘Many analysts, one data set: Making transparent how variations in analytic choices affect results’, Advances in Methods and Practices in Psychological Science 1, 337–356.
  * StataCorp (2021) StataCorp (2021), Stata Statistical Software: Release 17.  College Station, TX: StataCorp LLC.
  * Tortora et al. (2020) Tortora, J., Robinson, S., Baker, S. & Ruaño, G. (2020), ‘Clinical dataset of the cyp-guides trial’, Mendeley Data, V1 .  doi: 10.17632/25yjwbphn4.1.
  * Waehrer et al. (2020) Waehrer, G. M., Miller, T. R., Silverio Marques, S. C., Oh, D. L. & Burke Harris, N. (2020), ‘Disease burden of adverse childhood experiences across 14 states’, PLOS ONE 15(1), 1–18.
  * Wagenmakers et al. (2022) Wagenmakers, E.-J., Sarafoglou, A. & Aczel, B. (2022), ‘One statistical analysis must not rule them all’, Nature 605(7910), 423–425.
  * Wasserstein et al. (2016) Wasserstein, R. L., Lazar, N. A. et al. (2016), ‘The asa’s statement on p-values: context, process, and purpose’, The American Statistician 70(2), 129–133.
  * Wasserstein et al. (2019) Wasserstein, R. L., Schirm, A. L. & Lazar, N. A. (2019), ‘Moving to a world beyond “p$<$0.05”’, The American Statistician 73(sup1), 1–19.
  * Williams (2012) Williams, R. (2012), ‘Using the margins command to estimate and interpret adjusted predictions and marginal effects’, Stata Journal 12(2), 308–331(24).
  * Zhao & Hastie (2021) Zhao, Q. & Hastie, T. (2021), ‘Causal interpretations of black-box models’, Journal of Business & Economic Statistics 39(1), 272–281.

  * Appendices

##  Appendix A Relevant mathematical definitions and notation

######  Remark A.1.

Given that the usage of terminology regarding the functions that define
continuous and discrete probability distributions is not always consistent
within the literature, we would like to clarify that, within this work, we use

    1. (i)

the term _density_ to refer to any measurable function
$f:\mathbb{R}^{d}\longrightarrow\mathbb{R}$, $d\in\mathbb{N}_{>0}$, fulfilling
the following requirements

      1. 1.

$f(x)\geq 0,\quad\forall x\in\mathbb{R}^{d}$.

      2. 2.

$\int_{\mathbb{R}^{d}}f(x)d\lambda(x)=1$, where $\lambda$ denotes the Lebesgue
measure.

    2. (ii)

the term probability function to refer to any function
$p:\mathbb{R}^{d}\longrightarrow[0,1]$, $d\in\mathbb{N}_{>0}$, fulfilling the
following requirements

      1. 1.

$\text{supp}(p)=\\{x\in\mathbb{R}^{d}:p(x)\neq 0\\}$ is countable.

      2. 2.

$\underset{x\in\text{supp}(p)}{\sum}p(x)=1$.

######  Remark A.2.

For a probability measure $\mu$ on
$(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}))$, $d\in\mathbb{N}_{>0}$, we
write that

    1. 1.

_$\mu$ is defined via_ density $f$ or probability function $p$, if $\forall
A\in\mathcal{B}(\mathbb{R}^{d})$

$\mu(A)=\int_{A}f(x)d\lambda(x)\quad\quad\text{or}\quad\quad\mu(A)=\underset{x\in\text{supp}(p)\cap
A}{\sum}p(x)$

holds, respectively. Furthermore, we write $\mu(A)=\int_{A}d\mu(x)$ in both
cases.

    2. 2.

_$\mu$ is associated with_ a known distribution if $\mu$ is defined via the
distribution’s density or probability function.

######  Remark A.3.

For a probability measure $\mu$ that is defined via a conditional density
$f_{X|Y}$ or probability function $p_{X|Y}$, we use the following notation

$\displaystyle\int_{A}\mu(dx,y)=\begin{cases}\int_{A}f_{X|Y}(x,y)d\lambda(x),&\text{if
$\mu$ is defined via a density}\\\ \underset{x\in\text{supp}(p)\cap
A}{\sum}p_{X|Y}(x,y),&\text{if $\mu$ is defined via a probability
function}\,.\end{cases}$

Note that these terms are still variable in $y$.

######  Definition A.1 (Normalized Probability Measures).

Given $d\in\mathbb{N}_{>0}$ and a probability measure $\mu$ on
$\big{(}\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d})\big{)}$ defined via either
density $f$ or probability function $p$ and a set
$A\in\mathcal{B}(\mathbb{R}^{d})$, we define $\mu$ to be _normalized w.r.t.
$A$_, if $\mu$ is then defined via the density $\mu(A)^{-1}\cdot
f(x)\cdot\mathds{\large 1}_{x\in A}$ or the probability function
$\mu(A)^{-1}\cdot p(x)\cdot\mathds{\large 1}_{x\in A}\,,$ respectively.

######  Definition A.2 (Highest Density Regions as proposed by Hyndman
(1996)).

Let $\pi$ be a density or probability function via which a probability measure
$\mu_{\pi}$ on $(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}))$,
$d\in\mathbb{N}_{>0}$, is defined. For any $\alpha\in[0,1]$, we then define
the _highest density region with level $\alpha$ of $\mu_{\pi}$_, or
_$100(1-\alpha)$ %-HDR of $\mu_{\pi}$_, as the set

$R(\pi_{\alpha}):=\\{x\in\mathbb{R}^{d}|\pi(x)\geq\pi_{\alpha}\\}\,,$

where $\pi_{\alpha}$ is the largest constant that satisfies
$\mu_{\pi}\big{(}R(\pi_{\alpha})\big{)}\geq 1-\alpha$.

######  Definition A.3 (Support of measures on specific measurable spaces).

For a non-negative measure $\mu$ on
$(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}))$, $d\in\mathbb{N}_{>0}$, we
define the support of $\mu$ as

$\text{supp}(\mu):=\\{x\in\mathbb{R}^{d}\,\big{|}\,\forall\varepsilon>0:\mu(B_{\varepsilon}(x))>0\\}\,,$

where $B_{\varepsilon}(x)$ denotes the _epsilon ball_ around $x$, defined as
$B_{\varepsilon}(x):=\\{y\in\mathbb{R}^{d}\,\big{|}\,\|x-y\|<\varepsilon\\}$
for some $\varepsilon>0$

######  Remark A.4.

In this work, we often utilize specific notation for subgroups of entries of
some vector, e.g. denoting the entries of the vector of regressor-realizations
as two distinct vectors $x_{[I]}$ and $x_{[\text{MC}]}$. Consequently, when we
consider some function $h:\mathbb{R}^{d}\longrightarrow\mathbb{R}$,
$d\in\mathbb{N}_{>0}$, that maps a vector $(x_{1},...,x_{d})^{\top}$ to a
value in $\mathbb{R}$ and some notation that maps all entries of a vector in
$\mathbb{R}^{d}$ to either $x_{a}$ or $x_{b}$, we write $h(x)$ and
$h(x_{a},x_{b})$ interchangeably, depending on which version we consider
optimal for clarity.

##  Appendix B Interaction terms

This appendix contains more detailed information about the proposed method to
separately quantify the generalized marginal effect $\Delta_{s}$ for main and
interaction effects that were omitted in section 2.2. After providing
motivation for our general approach to interaction effects, sections section
B.1 and B.1 give definitions of appropriate probability measures for
separately considering interactions between metric regressors, and the
generalized marginal effect for separately quantifying main and interaction
effects for interactions between metric and categorical regressors,
respectfully.

######  Remark B.1.

Within this work, we use the expression _interaction term_ to refer to the
product of two or more entries of the regressor vector $X$ appearing in the
function term of $g_{\theta}$, if applicable after the inclusion of ‘new
regressor(s)’ in the sense of remark 2.2. We say that a regressor is _included
in an interaction term_ if it is one of the factors in said interaction term.
Furthermore, we refer to an interaction term as _being of higher order_ than
another term if the first may be written as a product in which the latter is
one factor.

As explained in section 2.2, our approach to this issue is to operate under
the simplifying assumption that the interaction terms behave independently of
the regressors that are being multiplied. Since this assumption clearly does
not correspond to reality, the resulting generalized marginal effects should
be interpreted with great caution. However, the resulting effect sizes appear
to be optimal in terms of comparability. Indeed, applying this assumption for
linear regression with interactions results simply in the reporting of
$\beta$-coefficients, as 1 proves. The following example provides a more
intuitive motivation.

######  Example.

Consider the case of linear regression that includes two _metric_ regressors,
$X_{1}$ and $X_{2}$ as well as their interaction, i.e. the expectation
function is given by

$\displaystyle g_{\theta}(X)=\beta_{0}+\beta_{1}\cdot X_{1}+\beta_{2}\cdot
X_{2}+\beta_{3}\cdot X_{1}\cdot X_{2}\,.$

In such a setting, it is common practice to report the point estimate
$(\hat{\beta}_{0},\hat{\beta}_{1},\hat{\beta}_{2},\hat{\beta}_{3})^{\top}$ as
part of the inference results. However, importantly, these
$\beta$-coefficients may not be interpreted as ‘ _keeping all other regressors
constant, the expected value of the target variable increases by
$\hat{\beta}_{i}$ when $X_{i}$ increases by $1$._’, $i\in\\{1,2,3\\}$. Neither
is the interpretation ‘ _keeping $X_{1}$ constant, the expected value of the
target variable increases by $\hat{\beta}_{2}+\hat{\beta}_{3}$ when $X_{2}$
increases by $1$._’ admissible, since $\mathbb{E}[Y|X]$ actually increases by
$\hat{\beta}_{2}+\hat{\beta}_{3}\cdot X_{1}$ when $X_{2}$ increases by $1$,
which is still a function of the metric regressor $X_{1}$. Still, the
$\beta$-coefficients of models such as the one above are being reported, as
they still contain valuable information. Making the assumption that an
interaction term behaves independently of the regressors being multiplied
simply allows us to quantify the equivalent of linear $\beta$-coefficients in
non-linear settings.

The following will now, firstly, specify how the set $\mathbb{X}$ and
probability measure $\mu$ are to be chosen if one wants to separate the main
and interaction effects of only metric variables and, secondly, give the
formal definition omitted in definition 3(iii).

###  B.1 Interactions between metric regressors

For interactions between metric regressors, we propose to treat the
interaction term as a new, independent regressor, as was also explained in
section 2.2. Specifically, this entails choosing either one single metric
regressor or one interaction term as regressor of interest $X^{I}$ and mapping
_all interaction terms of higher order_ as well as, if applicable, _all metric
regressors involved in said interaction term_ to the set of remaining metric
regressors $M$.

While choosing to assume independence of an interaction term to separately
quantify main and interaction effects is reasonable in the interest of
comparability, we do not believe it expedient to assume that interaction terms
may take values that are not the product of values in the respective support
of the measures chosen for each involved regressor. Therefore, for the purpose
of choosing an appropriate probability measure for either the new regressor of
interest or the added elements of $M$, we require the following definition.

######  Definition B.1.1.

Note that, if $A$, $B$, or both sets are chosen as $\mathbb{R}$ and neither as
$\\{0\\}$, $\boldsymbol{(}A\boldsymbol{\cdot}B\boldsymbol{)}=\mathbb{R}$.

  * 2.

For two probability measures $\mu_{1}$ and $\mu_{2}$ on
$\big{(}\mathbb{R},\mathcal{B}(\mathbb{R})\big{)}$, we denote by
$\boldsymbol{(}\mu_{1}\boldsymbol{\cdot}\mu_{2}\boldsymbol{)}$ a measure on
$\big{(}\mathbb{R},\mathcal{B}(\mathbb{R})\big{)}$ that fulfills the following
requirements w.r.t. $\mu_{1}$ and $\mu_{2}$

    * (I1)

$\boldsymbol{(}\mu_{1}\boldsymbol{\cdot}\mu_{2}\boldsymbol{)}$ is a
probability measure on $\big{(}\mathbb{R},\mathcal{B}(\mathbb{R})\big{)}$

    * (I2)

$\text{supp}\big{(}\boldsymbol{(}\mu_{1}\boldsymbol{\cdot}\mu_{2}\boldsymbol{)}\big{)}=Cl\Big{(}{\boldsymbol{(}\text{supp}(\mu_{1})\boldsymbol{\cdot}\text{supp}(\mu_{2})\boldsymbol{)}}\Big{)}$

    * (I3)

$\forall M\in\mathcal{B}(\mathbb{R})$, the following holds for
$M/z:=\\{x\in\mathbb{R}|x\cdot z\in M\\}$

$\boldsymbol{(}\mu_{1}\boldsymbol{\cdot}\mu_{2}\boldsymbol{)}(M)=\int_{\mathbb{R}}\int_{M/z}d\mu_{1}(x)d\mu_{2}(z)=\int_{\mathbb{R}}\int_{M/z}d\mu_{2}(x)d\mu_{1}(z)\,.$
(B.1)

In general, and in particular under the axiomatic assumption (A.I), we utilize
probability measures not necessarily to represent randomness, but also to
derive a non-random weighted average of a function’s value over a certain set.
However, we expect the utilized measures to often be chosen as associated with
a known distribution and, furthermore, believe that, to most readers, the
notion of a random variable following a certain distribution is more
accessible than a non-random average calculated w.r.t. a probability measure.
Therefore, the following proposition expresses the above definition in terms
of random variables.

######  Proposition B.1.

Let $X$ and $Y$ denote two independently distributed, real-valued random
variables. If the probability measures $\mu_{X}$ and $\mu_{Y}$ on
$\big{(}\mathbb{R},\mathcal{B}(\mathbb{R})\big{)}$ are associated with the
distributions of $X$ and $Y$, respectively, the measure
$\boldsymbol{(}\mu_{X}\boldsymbol{\cdot}\mu_{Y}\boldsymbol{)}$ is associated
with the distribution of the random variable $Z:=XY$.

The proof of this proposition may be found in appendix C.

##### Metric interactions under assumptions (A.I) and (A.II′)

Under assumptions (A.I) and (A.II′), where the regressor of interest is
treated as independent, the choice of sets and probability measures in the
case of metric interactions is now very straightforward. One simply makes all
specifications that would have been made when not choosing to separately
quantify the effects. However, additionally, the interaction term of interest
or interaction terms added to $M$ are treated as independent under _both_
assumptions and the corresponding sets and measures chosen as follows. For an
interaction between two metric regressors $X_{i}$ and $X_{j}$,
$i,j\in\\{1,...,p\\}$, one chooses
$\boldsymbol{(}\mathbb{X}_{i}\boldsymbol{\cdot}\mathbb{X}_{j}\boldsymbol{)}$
as the set to integrate over for the new ‘interaction-regressor’ and
$\boldsymbol{(}\mu_{i}\boldsymbol{\cdot}\mu_{j}\boldsymbol{)}$ as the measure
to integrate w.r.t., with $\mu_{i}$ and $\mu_{j}$ denoting any measure chosen
for the independent regressors $X_{i}$ and $X_{j}$ under assumption (A.I);
and, under assumption (A.II′), denoting one independently chosen measure for
one regressor and the measure associated with the _marginal distribution_ of
the other regressor.

For interactions of higher order, the set $\mathbb{X}$ and measure $\mu$ may
be derived iteratively in arbitrary order, as stated by the following
corollary.

######  Corollary B.1.

For any finite sets of probability measures on
$\big{(}\mathbb{R},\mathcal{B}(\mathbb{R})\big{)}$ $\\{\mu_{i}\\}_{i=1,...,N}$
and subsets of $\mathbb{R}$ $\\{\mathbb{X}_{i}\\}_{i=1,...,N}$,
$N\in\mathbb{N}$, the following holds

    1. 1.

$Cl\Big{(}{\boldsymbol{(}\boldsymbol{(}\boldsymbol{(}\boldsymbol{(}\mathbb{X}_{1}\boldsymbol{\cdot}\mathbb{X}_{2}\boldsymbol{)}\boldsymbol{\cdot}\mathbb{X}_{3}\boldsymbol{)}\boldsymbol{\cdot}...\boldsymbol{)}\boldsymbol{\cdot}\mathbb{X}_{N}\boldsymbol{)}}\Big{)}=Cl\Big{(}{\boldsymbol{(}\boldsymbol{(}\boldsymbol{(}\boldsymbol{(}\mathbb{X}_{\tau(1)}\boldsymbol{\cdot}\mathbb{X}_{\tau(2)}\boldsymbol{)}\boldsymbol{\cdot}\mathbb{X}_{\tau(3)}\boldsymbol{)}\boldsymbol{\cdot}...\boldsymbol{)}\boldsymbol{\cdot}\mathbb{X}_{\tau(N)}\boldsymbol{)}}\Big{)}$
for any permutation $\tau$ on the set $\\{1,...,N\\}$.

    2. 2.

$\boldsymbol{(}\boldsymbol{(}\boldsymbol{(}\boldsymbol{(}\mu_{1}\boldsymbol{\cdot}\mu_{2}\boldsymbol{)}\boldsymbol{\cdot}\mu_{3}\boldsymbol{)}\boldsymbol{\cdot}...\boldsymbol{)}\boldsymbol{\cdot}\mu_{N}\boldsymbol{)}=\boldsymbol{(}\boldsymbol{(}\boldsymbol{(}\boldsymbol{(}\mu_{\tau(1)}\boldsymbol{\cdot}\mu_{\tau(2)}\boldsymbol{)}\boldsymbol{\cdot}\mu_{\tau(3)}\boldsymbol{)}\boldsymbol{\cdot}...\boldsymbol{)}\boldsymbol{\cdot}\mu_{\tau(N)}\boldsymbol{)}$
for any permutation $\tau$ on the set $\\{1,...,N\\}$.

The proof of this corollary may also be found in appendix C.

##### Metric interactions under assumption (A.II′′)

The approach to metric interactions under assumption (A.II′′) is almost
identical to the one under assumptions (A.I) and (A.II′). However, as the
regressor of interest is not treated as independent under this assumption, we
believe that choosing the measure to integrate w.r.t. for the a new
‘interaction-regressor’ according to definition B.1.1 would again not be
reasonable. Instead, the ideal approach would be to identify the measure
associated with the distribution of a random variable that is defined as the
product of two random variables with a joint distribution equal to the
marginal distribution of the regressors involved in the interaction term. Of
course, choosing the set to integrate over according to definition B.1.1 is
still completely valid.

###  B.2 Interactions between metric and categorical regressors

When separately considering interaction effects of interactions between metric
_and_ categorical regressors, we propose to again combine all categorical
regressors, including the parts of the interaction terms that are categorical,
into one regressor and consider the interactions between metric regressors as
separate regressors, since the interaction between a categorical and metric
variable can not be reasonably combined into one separate regressor. In order
to give a coherent definition for our effect size measure, however, we need to
combine all these elements into one vector $X^{I}$. We propose to denote the
metric regressor, or interaction of several metric regressors which is viewed
as one new metric variable, by $X^{I}_{met}$; and the categorical elements by
$X^{I}_{{cat}}=(X^{I}_{1},...,X^{I}_{d_{M}},...,X^{I}_{d_{C}})$, where the
entries $X^{I}_{1}$ to $X^{I}_{d_{M}}$ represent the non-reference categories
of the categorical regressors that are included in an interaction with a
metric regressor (or interaction of metric regressors viewed as a separate
metric regressor) and entries $X^{I}_{d_{M}+1}$ to $X^{I}_{d_{C}}$ represent
these same entries interacting with $X^{I}_{met}$.

Overall, the regressor of interest is then given by
$X^{I}=\big{(}X^{I}_{met},X^{I}_{cat}\big{)}$, with $X^{I}_{met}$ taking
values in $\mathbb{R}$ and $X^{I}_{cat}$ taking values in $\\{0,1\\}^{d_{C}}$
and the respective realizations being denoted by $x_{[I_{met}]}$ and
$x_{[I_{cat}]}$. In the interest of consistent notation, we set
$d_{I}=d_{C}+1$. Furthermore, we again utilize the vectors
$\mathbf{v}_{l}\in\\{0,1\\}^{d_{C}}$ and
$\mathbf{ref}_{l}\in\\{0,1\\}^{d_{C}}$ as defined in section 2.2, for which
the following provides a concrete example.

######  Example B.1.

    1. (i)

Consider a regression with categorical regressors _‘smoking’_ , a binary
variable with reference category "non-smoking", and _‘gender’_ , with
reference category "female" and non-reference categories "male" and "non-
binary". If one were interested in separating the effect of _‘smoking’_ ,
_‘gender’_ , and their interaction, the combined regressor of interest would
be given by
$X_{cat}^{I}=(X^{I}_{1},X^{I}_{2},X^{I}_{3},X^{I}_{4},X^{I}_{5})^{\top}$, with

$X^{I}_{1}\,\hat{=}\,smoking$, $X^{I}_{2}\,\hat{=}\,male$,
$X^{I}_{3}\,\hat{=}\,non$-$binary$, $X^{I}_{4}\,\hat{=}\,smoking*male$,
$X^{I}_{5}\,\hat{=}\,smoking*non$-$binary$

and, furthermore,
$\mathbf{ref}_{1}=\mathbf{ref}_{2}=\mathbf{ref}_{3}=(0,0,0,0,0)^{\top}$,
$\mathbf{ref}_{4}=(1,1,0,0,0)^{\top}$, $\mathbf{ref}_{5}=(1,0,1,0,0)^{\top}$
and $\mathbf{v}_{1}=\mathbf{e}_{1}$, $\mathbf{v}_{2}=\mathbf{e}_{2}$,
$\mathbf{v}_{3}=\mathbf{e}_{3}$, $\mathbf{v}_{4}=(1,1,0,1,0)^{\top}$,
$\mathbf{v}_{5}=(1,0,1,0,1)^{\top}$.

    2. (ii)

Consider the setting from above. However, instead of the interaction between
_‘gender’_ and _‘smoking’_ , one is now interested in the interaction between
_‘gender’_ and _‘age’_ , a metric variable. We now propose to consider
$X_{cat}^{I}=(X^{I}_{1},X^{I}_{2},X^{I}_{3},X^{I}_{4})^{\top}\in\\{0,1\\}^{4}$
with

$X^{I}_{1}\,\hat{=}\,male$, $X^{I}_{2}\,\hat{=}\,non$-$binary$,
$X^{I}_{3}\,\hat{=}\,male\,interacting\,with\,age$,
$X^{I}_{4}\,\hat{=}\,non$-$binary\,interacting\,with\,age$.

and $\mathbf{ref}_{1}=\mathbf{ref}_{2}=(0,0,0,0)^{\top}$,
$\mathbf{ref}_{3}=(1,0,0,0)^{\top}$, $\mathbf{ref}_{4}=(0,1,0,0)^{\top}$ and
$\mathbf{v}_{1}=\mathbf{e}_{1}$, $\mathbf{v}_{2}=\mathbf{e}_{2}$,
$\mathbf{v}_{3}=(1,0,1,0)^{\top}$, $\mathbf{v}_{4}=(0,1,0,1)^{\top}$.
Furthermore, age is written as $X^{I}_{met}$ and $d_{M}=2$, since $g_{\theta}$
contains the terms $X^{I}_{met}\cdot X^{I}_{3}$ and $X^{I}_{met}\cdot
X^{I}_{4}$. The full combined regressor of interest is then given by
$\big{(}X^{I}_{met},X^{I}_{cat}\big{)}$.

For the purposes of isolating the effect size of the metric component
$X^{I}_{met}$, we need to define the following new function which ‘removes’
all other entries of $X^{I}$ from the expectation function

$\displaystyle\widetilde{g_{\theta}}(x):\mathbb{R}^{p-d_{C}}\longrightarrow\mathbb{R},\quad
x\longmapsto g_{\theta}\big{(}x_{[I_{cat}]}=\mathbf{0},x\big{)}\,.$ (B.2)

In order to formally define the proposed effect size measure, we now require
only the following definition.

######  Definition.

For $\bar{\mathbb{X}}$ and $\bar{\mu}$ chosen according to definition 2 and
$\widetilde{g_{\theta}}$ as defined in eq. B.2, we define

$\tilde{s}_{\emptyset}(\theta,x):=\bigintssss_{\bar{\mathbb{X}}}\frac{\partial\widetilde{g_{\theta}}}{\partial
x_{[I_{met}]}}(x)\,\,d\bar{\mu}(x_{[\text{MC}]})\,,$ (B.3)
$\tilde{s}(\theta,x,\mathbf{v}):=\bigintssss_{\bar{\mathbb{X}}}g_{\theta}(\left.x\right|_{x_{\scaleto{[I_{cat}]}{6pt}}=\mathbf{v}})\,\,d\bar{\mu}(x_{[\text{MC}]})\,,$
(B.4)

as well as the following $\forall\,l\in\\{d_{M}+1,...,d_{C}\\}$:

$\tilde{s}_{I_{l}}(\theta,x):=\bigintssss_{\bar{\mathbb{X}}}\dfrac{\partial
g_{\theta}}{\partial(x_{[I_{l}]}\cdot
x_{[I_{met}]})}(\left.x\right|_{x_{\scaleto{[I_{cat}]}{6pt}}=\mathbf{v}_{l}})\,\,d\bar{\mu}(x_{[\text{MC}]})\,.$
(B.5)

Note that the notation $\partial(x_{[I_{l}]}\cdot x_{[I_{met}]})$ represents
the derivation of $g_{\theta}$ w.r.t. the interaction of the category
$x_{[I_{l}]}$, i.e. the realization of the $l$th entry of $X^{I}_{cat}$, and
the metric regressor $x_{[I_{met}]}$ and not the actual product. When
$x_{[I_{l}]}=1$, we treat this interaction term as an ‘independent copy’ of
$x_{[I_{met}]}$ which is denoted by $x_{[I_{met}]}^{\bullet I_{l}}$.

Finally, the following gives the full, formal version of definition 3(iii).

######  Definition B.2.1.

In cases where metric _and_ categorical regressors are involved in an
interaction and one is interested in separately quantifying the main and
interaction effects, $\Delta_{s}$ is defined as the function

$\displaystyle\Delta_{s}:\Theta\longrightarrow\mathbb{R}^{d_{I}},\quad\theta\longmapsto\begin{pmatrix}\Delta_{s}^{[1]}(\theta)\\\
\vdots\\\
\Delta_{s}^{[d_{I}]}(\theta)\end{pmatrix}:=\begin{pmatrix}&\int_{\mathbb{X}}\tilde{s}_{\emptyset}(\theta,x)d\mu(x_{[I_{met}]})\\\\[10.0pt]
\int_{\mathbb{X}}\tilde{s}(\theta,x,\mathbf{v}_{1})d\mu(x_{[I_{met}]})&-&\int_{\mathbb{X}}\tilde{s}(\theta,x,\mathbf{ref}_{1})d\mu(x_{[I_{met}]})\\\
&\vdots&\\\
\int_{\mathbb{X}}\tilde{s}(\theta,x,\mathbf{v}_{d_{M}})d\mu(x_{[I_{met}]})&-&\int_{\mathbb{X}}\tilde{s}(\theta,x,\mathbf{ref}_{d_{M}})d\mu(x_{[I_{met}]})\\\\[10.0pt]
&\int_{\mathbb{X}}\int_{\mathbb{X}}\tilde{s}_{I_{d_{M}+1}}(\theta,x)d\mu(x_{[I_{met}]})\,d\mu(x_{[I_{met}]}^{\bullet
I_{l}})\\\ &\vdots&\\\
&\int_{\mathbb{X}}\int_{\mathbb{X}}\tilde{s}_{I_{d_{C}}}(\theta,x)d\mu(x_{[I_{met}]})\,d\mu(x_{[I_{met}]}^{\bullet
I_{l}})\end{pmatrix}\,,$

where $\mathbb{X}$ and $\mu$ are set as identical in every term, so that $\mu$
satisfies requirements (M1)-(M3) w.r.t. the function in each integral term and
$\mathbb{X}$, after, if applicable, being chosen in accordance with section
B.1.

##  Appendix C Proofs

This appendix contains proofs for all statements made in the main paper and
appendix B as well as some additional remarks throughout. The order of the
proofs generally follows the order in which the statements to be proven were
made, with the notable exception of proposition 4 and 1. Their proofs are
given immediately after the proof of 1, since the derived results may be used
to prove some statements regarding how existing methodology falls into the
proposed framework more efficiently.

###  C.1 Proof of 1

See 1

###### Proof.

We prove this statement separately for each setting of definitions
3(i)-3(iii). Throughout, w.l.o.g., we only consider cases without interactions
between elements of $M$ and $C$ unless they are of interest themselves. We are
able to do so as including such interactions does not affect any statements
made over the course of this proof. Furthermore, we consider a parameter
vector of the form $\theta=(\beta,v)^{\top}\in\Theta$ with

$\displaystyle\beta=\big{(}\beta_{0},\beta_{1},...,\beta_{\tilde{p}}\big{)}\subseteq\mathbb{R}^{\tilde{p}+1}\,,$

where, for $p\in\mathbb{N}_{>0}$ denoting the number of observed regressor
variables and $a\in\mathbb{N}_{\geq 0}$ denoting the number of interaction
terms included in $g_{\theta}$, $\tilde{p}$ is defined by $\tilde{p}:=p+a$.

    * Case 1:

The regressor of interest is a metric variable or an interaction between
metric regressors considered as a separate regressor (if applicable,
interactions of higher order are (also) considered separate regressors).

In this case, Definition 3(i) applies. Specifically, for
$t(j):=1+m+\big{(}\mathds{\large
1}_{(j-1)>0}\cdot\sum_{q=1}^{\max(1,j-1)}d_{C_{q}}\big{)},$ $j=1,...,c$,
$g_{\theta}$, if containing no interaction terms, is given by the following

$\displaystyle
g_{\theta}(x)=g_{\theta}^{(i)}(x_{[I]},x_{[\text{MC}]}):=\beta_{0}+\beta_{1}x_{[I]}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+1}x_{[M_{i}]}+\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{t(j)+1},...,\beta_{t(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}\,.$
(C.1)

If $g_{\theta}$ does contain interaction terms, we denote the vector of
realizations representing higher order interaction terms and/or the
interaction whose effect is of interest by $x_{[\text{IA}]}$ and

$\displaystyle
g_{\theta}(x)=g_{\theta}^{(i)^{\ast}}(x_{[I]},x_{[\text{MC}]},x_{[\text{IA}]}):=g_{\theta}^{(i)}(x_{[I]},x_{[\text{MC}]})+\big{(}\beta_{p+1},...,\beta_{p+|\text{IA}|}\big{)}x_{[\text{IA}]}^{\top}\,.$

Let $r$ denote the index corresponding to the $\beta$-coefficient of the
regressor of interest, i.e. $r=1$ for the case of no interaction terms and
$r\in\\{1\\}\cup\\{p+1,...,p+|\text{IA}|\\}$ when one is interested in the
effect of a metric regressor included in an interaction or the effect of the
interaction term itself. Then the following holds

$\displaystyle\frac{\partial g_{\theta}^{(i)}}{\partial
x_{[I]}}(x_{[I]},x_{[\text{MC}]})=\beta_{r}\quad\text{ and
}\quad\frac{\partial g_{\theta}^{(i)^{\ast}}}{\partial
x_{r}}(x_{[I]},x_{[\text{MC}]},x_{[\text{IA}]})=\beta_{r}\,.$ (C.2)

Finally, it follows that, if the regressor of interest is a metric variable or
an interaction between metric regressors considered as a separate regressor,

$\displaystyle\Delta_{s}(\theta)=\begin{cases}\int_{\mathbb{X}}\int_{\bar{\mathbb{X}}}\beta_{r}\,d\bar{\mu}(x_{[\text{MC}]})d\mu(x_{[I]}),&\text{under
assumptions (A.I) and (A.II${}^{\prime}$)}\\\
\int_{\mathbb{X}}\int_{\bar{\mathbb{X}}}\beta_{r}\,\bar{\mu}_{X_{[\text{MC}]}|X^{I}}(dx_{[\text{MC}]},x_{[I]})d\mu(x_{[I]}),&\text{under
assumption (A.II${}^{\prime\prime}$),}\end{cases}$ (C.3)

which is equal to $\beta_{r}$ for any probability measures $\mu$, $\bar{\mu}$,
and $\bar{\mu}_{X_{[\text{MC}]}|X^{I}}$, whereby the statement holds for
definition 3(i).

    * Case 2.1:

The regressor of interest is a categorical variable that is not included in
any interactions.

In this case, Definition 3(ii) applies and, for
$b(j):=d_{I}+m+\big{(}\mathds{\large
1}_{(j-1)>0}\cdot\sum_{q=1}^{\max(1,j-1)}d_{C_{q}}\big{)}$,
$j\in\\{1,...,c\\}$, $g_{\theta}$ is given by

$\displaystyle
g_{\theta}(x)=g_{\theta}^{(ii)}(x_{[I]},x_{[\text{MC}]}):=\beta_{0}+\big{(}\beta_{1},...,\beta_{d_{I}}\big{)}x_{[I]}^{\top}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}\,.$
(C.4)

Under assumptions (A.I) and (A.II′), the following then holds $\forall
r\in\\{1,...,d_{I}\\}$

$\displaystyle\Delta_{s}^{[r]}(\theta)$
$\displaystyle=\int_{\bar{\mathbb{X}}}g_{\theta}^{(ii)}(\left.x\right|_{x_{[I]}=\mathbf{e}_{r}})-g_{\theta}^{(ii)}(\left.x\right|_{x_{[I]}=\mathbf{0}})d\bar{\mu}(x_{[\text{MC}]})$
$\displaystyle\overset{(\star)}{=}\int_{\bar{\mathbb{X}}}g_{\theta}^{(ii)}(\left.x\right|_{x_{[I]}=\mathbf{e}_{r}})d\bar{\mu}(x_{[\text{MC}]})-\int_{\bar{\mathbb{X}}}g_{\theta}^{(ii)}(\left.x\right|_{x_{[I]}=\mathbf{0}})d\bar{\mu}(x_{[\text{MC}]})$
$\displaystyle=\Big{(}\int_{\bar{\mathbb{X}}}\beta_{r}\,d\bar{\mu}(x_{[\text{MC}]})+\int_{\bar{\mathbb{X}}}\beta_{0}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}d\bar{\mu}(x_{[\text{MC}]})\Big{)}-$
$\displaystyle\,\,\quad\Big{(}\int_{\bar{\mathbb{X}}}0\,d\bar{\mu}(x_{[\text{MC}]})+\int_{\bar{\mathbb{X}}}\beta_{0}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}d\bar{\mu}(x_{[\text{MC}]})\Big{)}$
$\displaystyle=\int_{\bar{\mathbb{X}}}\beta_{r}\,d\bar{\mu}(x_{[\text{MC}]})-\int_{\bar{\mathbb{X}}}0\,d\bar{\mu}(x_{[\text{MC}]})=\beta_{r}-0$
$\displaystyle=\beta_{r}\,,$

whereby the statement holds for definition 3(ii).

######  Note.

The equality $(\star)$ does not necessarily hold when one replaces
$d\bar{\mu}(x_{[\text{MC}]})$
with$\bar{\mu}_{X_{[\text{MC}]}|X^{I}}(dx_{[\text{MC}]},x_{[I]})$, which is
why the statement of this theorem does not apply to for a categorical
regressor of interest under (A.II′′).

    * Case 2.2:

The regressor of interest is the interaction between categorical variables or
a categorical regressor involved in (an) interaction(s) with other categorical
regressors and one is interested in separating the effects.

In this case, Definition 3(ii) again applies and $g_{\theta}$ given by
$g_{\theta}^{(ii)}(x_{[I]},x_{[\text{MC}]})$ as in eq. C.4. However, $x_{[I]}$
represents realizations of a different vector $X^{I}$ in this case.
Specifically, $X^{I}=(X^{I}_{1},...,X^{I}_{d_{I}})$ contains an entry for each
category of the categorical regressors involved in the interaction terms and
an entry for the combination of categories in each interaction term. As a
result, the term $\big{(}\beta_{1},...,\beta_{d_{I}}\big{)}x_{[I]}^{\top}$ is
no longer always equal to one single $\beta$ but may be given by a sum over
several $\beta$s. In order to formally denote such sums, we
define$L(r):=\Big{\\{}i\in\\{1,...,d_{I}\\}\backslash\\{r\\}\big{|}\text{$X^{I}_{i}$
is included in $X^{I}_{r}$}\Big{\\}}$, $r=1,...,d_{I}$.

Under assumptions (A.I) and (A.II′), the following then holds $\forall
r\in\\{1,...,d_{I}\\}$

$\displaystyle\Delta_{s}^{[r]}(\theta)$
$\displaystyle=\int_{\bar{\mathbb{X}}}g_{\theta}^{(ii)}(\left.x\right|_{x_{[I]}=\mathbf{v}_{r}})-g_{\theta}^{(ii)}(\left.x\right|_{x_{[I]}=\mathbf{0}})d\bar{\mu}(x_{[\text{MC}]})-\int_{\bar{\mathbb{X}}}g_{\theta}^{(ii)}(\left.x\right|_{x_{[I]}=\mathbf{ref}_{r}})-g_{\theta}^{(ii)}(\left.x\right|_{x_{[I]}=\mathbf{0}})d\bar{\mu}(x_{[\text{MC}]})$
$\displaystyle\overset{(\star\star)}{=}\int_{\bar{\mathbb{X}}}g_{\theta}^{(ii)}(\left.x\right|_{x_{[I]}=\mathbf{v}_{r}})d\bar{\mu}(x_{[\text{MC}]})-\int_{\bar{\mathbb{X}}}g_{\theta}^{(ii)}(\left.x\right|_{x_{[I]}=\mathbf{ref}_{r}})d\bar{\mu}(x_{[\text{MC}]})$
$\displaystyle=\Big{(}\int_{\bar{\mathbb{X}}}\sum_{l\in
L(r)\cup\\{r\\}}\beta_{l}\,d\bar{\mu}(x_{[\text{MC}]})+\int_{\bar{\mathbb{X}}}\beta_{0}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}d\bar{\mu}(x_{[\text{MC}]})\Big{)}-$
$\displaystyle\,\,\quad\Big{(}\int_{\bar{\mathbb{X}}}\sum_{l\in
L(r)}\beta_{l}\,d\bar{\mu}(x_{[\text{MC}]})+\int_{\bar{\mathbb{X}}}\beta_{0}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}d\bar{\mu}(x_{[\text{MC}]})\Big{)}$
$\displaystyle=\int_{\bar{\mathbb{X}}}\sum_{l\in
L(r)\cup\\{r\\}}\beta_{l}\,d\bar{\mu}(x_{[\text{MC}]})-\int_{\bar{\mathbb{X}}}\sum_{l\in
L(r)}\beta_{l}\,d\bar{\mu}(x_{[\text{MC}]})=\sum_{l\in
L(r)\cup\\{r\\}}\beta_{l}-\sum_{l\in L(r)}\beta_{l}$
$\displaystyle=\beta_{r}\,,$

whereby the statement holds for definition 3(iii). Please note that, just as
in case 2, this does not apply under (A.II′′) by the same reasoning as for
equality $(\star)$ applied to equality $(\star\star)$.

    * Case 3:

The regressor of interest is the interaction between categorical and metric
variables or a regressor involved in (an) interaction(s) with other
categorical and/or metric regressor(s) and one is interested in separating the
effects.

In this case, Definition 3(iii) as specified in section B.2 applies and, with
$b(j)$ defined as in case 2, $g_{\theta}$ is given by

$\displaystyle
g_{\theta}(x)=g_{\theta}^{(iii)}(x_{[I]},x_{[\text{MC}]}):=\beta_{0}+$
$\displaystyle\beta_{1}x_{[I_{met}]}+\big{(}\beta_{2},...,\beta_{d_{M}+1},...,\beta_{d_{I}}\big{)}\big{(}x_{[I_{cat}]}\odot\mathbf{m}\big{)}^{\top}+$
(C.5)
$\displaystyle\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}\,,$

with $x_{[I_{cat}]}\odot\mathbf{m}$ denoting the elementwise product of
$x_{[I_{cat}]}$ and the vector
$\mathbf{m}:=(\underset{\mathbf{m}_{1},...,\mathbf{m}_{d_{M}}}{\underbrace{1,...,1}},\underset{\mathbf{m}_{d_{M}+1},...,\mathbf{m}_{d_{I}-1}}{\underbrace{x_{[I_{met}]},...,x_{[I_{met}]}}})$.

It then immediately follows that

$\displaystyle\Delta_{s}^{[1]}(\theta)$
$\displaystyle=\bigintssss_{\mathbb{X}}\bigintssss_{\bar{\mathbb{X}}}\frac{\partial}{\partial
x_{[I_{met}]}}\bigg{(}\beta_{0}+\beta_{1}x_{[I_{met}]}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+$
(C.6) $\displaystyle\quad\hskip
113.81102pt\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}\bigg{)}d\bar{\mu}(x_{[\text{MC}]})d\mu(x_{[I_{met}]})$
$\displaystyle=\int_{\mathbb{X}}\int_{\bar{\mathbb{X}}}\beta_{1}\,d\bar{\mu}(x_{[\text{MC}]})d\mu(x_{[I_{met}]})=\beta_{1}\,.$

Next, we redefine $L(r)$ as
$L(r):=\Big{\\{}i\in\\{3,...,d_{I}\\}\backslash\\{r\\}\big{|}\text{$X^{I}_{i-1}$
is included in $X^{I}_{r-1}$}\Big{\\}}$, $r=2,...,d_{I}$, since, for any
$r\in\\{1,...,d_{C}\\}$, $X^{I}_{r}$ is the $r$th entry of $X^{I}_{cat}$, but
the $(r+1)$th entry of $X^{I}$.

Then, the following holds $\forall r\in\\{2,...,d_{M}+1\\}$

$\displaystyle\Delta_{s}^{[r]}(\theta)=\int_{\mathbb{X}}\int_{\bar{\mathbb{X}}}\beta_{0}+\beta_{1}x_{[I_{met}]}+\sum_{l\in
L(r)\cup\\{r\\}}\beta_{l}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+$
$\displaystyle\quad\hskip
56.9055pt\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}d\bar{\mu}(x_{[\text{MC}]})d\mu(x_{[I_{met}]})-\int_{\mathbb{X}}\int_{\bar{\mathbb{X}}}\beta_{0}+\beta_{1}x_{[I_{met}]}+$
$\displaystyle\quad\hskip 56.9055pt\sum_{l\in
L(r)}\beta_{l}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}d\bar{\mu}(x_{[\text{MC}]})d\mu(x_{[I_{met}]})$
$\displaystyle=\int_{\mathbb{X}}\int_{\bar{\mathbb{X}}}\beta_{0}+\beta_{1}x_{[I_{met}]}+\sum_{l\in
L(r)\cup\\{r\\}}\beta_{l}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}d\bar{\mu}(x_{[\text{MC}]})-$
$\displaystyle\,\,\quad\int_{\bar{\mathbb{X}}}\beta_{0}+\beta_{1}x_{[I_{met}]}+\sum_{l\in
L(r)}\beta_{l}+\overset{m}{\underset{i=1}{\sum}}\beta_{i+d_{I}}x_{[M_{i}]}+\overset{c}{\underset{j=1}{\sum}}\big{(}\beta_{b(j)+1},...,\beta_{b(j)+d_{C_{j}}}\big{)}x_{[C_{j}]}^{\top}d\bar{\mu}(x_{[\text{MC}]})d\mu(x_{[I_{met}]})\,.$

Combining the reasoning from case 1 and case 2.2, it follows that

$\displaystyle\Delta_{s}^{[r]}(\theta)$
$\displaystyle=\int_{\mathbb{X}}\beta_{r}\,\,d\mu(x_{[I_{met}]})=\beta_{r}\,.$
(C.7)

Lastly, $\forall r\in\\{d_{M}+2,...,d_{I}\\}$

$\displaystyle\frac{\partial g_{\theta}^{(iii)}}{\partial(x_{[I_{r-1}]}\cdot
x_{[I_{met}]})}(x_{[I]},x_{[\text{MC}]})=\beta_{r}\,,$ (C.8)

whereby $\Delta_{s}^{[r]}(\theta)=\beta_{r}$ under all assumptions by the same
reasoning as was used in case 1.

Finally, combining eq. C.6, eq. C.7, and eq. C.8 the statement also holds for
definition 3(iii). ∎

###  C.2 Proofs of Proposition 4 and 1 plus Remark C.1

See 4

###### Proof.

Under (A.I) and (A.II′), the regressor of interest is treated as an
independent variable. Therefore, given that
$\int_{\mathbb{X}\times\bar{\mathbb{X}}}|g_{\theta}(x)|d(\mu\times\bar{\mu})<\infty$
is required under all assumptions, the following holds by Fubini’s theorem
(see Fubini (1907))

$\displaystyle\Delta_{s}(\theta)$
$\displaystyle=\int_{\mathbb{X}}\bar{s}(\theta,x)d\mu(x)=\int_{\mathbb{X}}\int_{\bar{\mathbb{X}}}s(\theta,x)d\bar{\mu}(x_{[\text{MC}]})d\mu(x_{[I]})$
(C.9)
$\displaystyle=\int_{\bar{\mathbb{X}}}\int_{\mathbb{X}}s(\theta,x)d\mu(x_{[I]})d\bar{\mu}(x_{[\text{MC}]})\,.$

For $\mathbb{X}=(a_{\mathbb{X}},b_{\mathbb{X}})$ and $\mu$ being chosen or
given as the measure associated with the $U(a_{\mathbb{X}},b_{\mathbb{X}})$
distribution it immediately follows that

$\displaystyle\Delta_{s}(\theta)$
$\displaystyle=\int_{\bar{\mathbb{X}}}\big{(}b_{\mathbb{X}}-a_{\mathbb{X}}\big{)}^{-1}\cdot\Big{[}g_{\theta}(x)\Big{]}^{x_{[I]}=b_{\mathbb{X}}}_{x_{[I]}=a_{\mathbb{X}}}d\bar{\mu}(x_{[\text{{MC}}]})$
$\displaystyle=\big{(}b_{\mathbb{X}}-a_{\mathbb{X}}\big{)}^{-1}\cdot\Bigg{(}\int_{\bar{\mathbb{X}}}g_{\theta}(\left.x\right|_{x_{[I]}=b_{\mathbb{X}}})-g_{\theta}(\left.x\right|_{x_{[I]}=a_{\mathbb{X}}})d\bar{\mu}(x_{[\text{{MC}}]})\Bigg{)}\,.$

∎

######  Remark C.1.

Under assumption (A.II′′), the regressor of interest is not considered to be
independent, and therefore eq. C.9 does not hold. In fact, as mentioned in
section 3.2, this is why similar results to proposition 4 and 1 may not be
obtained under assumption (A.II′′). The following proves this statement

Let $f_{X}$ denote the joint density by which $\mu_{X}$ is defined.
Furthermore, let $f_{I}$ denote the marginal density of the regressor of
interest $X^{I}$ and $f_{X_{[\text{MC}]}|X^{I}}$ the conditional density of
all other regressors given $X^{I}$. By the definition of marginal density, the
following holds

$f_{X_{[\text{MC}]}|X^{I}}\big{(}x_{[\text{MC}]},x_{[I]}\big{)}=\frac{f_{X}\big{(}x_{[I]},x_{[\text{MC}]}\big{)}}{f_{I}(x_{[I]})}\cdot\mathds{\large
1}_{f_{I}(x_{[I]})>0}\,.$ (C.10)

Now, if $\mu$ is given as the measure associated with the
$U(a_{\mathbb{X}},b_{\mathbb{X}})$ distribution, it follows that

$f_{I}(x_{[I]})=\big{(}b_{\mathbb{X}}-a_{\mathbb{X}}\big{)}^{-1}\cdot\mathds{\large
1}_{[a_{\mathbb{X}},b_{\mathbb{X}}]}(x_{[I]})\,,$ (C.11)

and, therefore, for $\lambda$ denoting the Lebesgue measure,

$\displaystyle\Delta_{s}(\theta)$
$\displaystyle=\big{(}b_{\mathbb{X}}-a_{\mathbb{X}}\big{)}^{-1}\cdot\int_{a_{\mathbb{X}}}^{b_{\mathbb{X}}}\int_{\bar{\mathbb{X}}}\frac{\partial
g_{\theta}}{\partial
x_{[I]}}(x)\bar{\mu}_{X_{[\text{MC}]}|X^{I}}(dx_{[\text{{MC}}]},x_{[I]})d\lambda(x_{[I]})$
$\displaystyle=\big{(}b_{\mathbb{X}}-a_{\mathbb{X}}\big{)}^{-1}\cdot\int_{a_{\mathbb{X}}}^{b_{\mathbb{X}}}\int_{\bar{\mathbb{X}}}\frac{\partial
g_{\theta}}{\partial
x_{[I]}}(x)f_{X_{[\text{MC}]}|X^{I}}\big{(}x_{[\text{MC}]},x_{[I]}\big{)}d\lambda
x_{[\text{{MC}}]})d\lambda(x_{[I]})$
$\displaystyle=\big{(}b_{\mathbb{X}}-a_{\mathbb{X}}\big{)}^{-1}\cdot\int_{a_{\mathbb{X}}}^{b_{\mathbb{X}}}\int_{\bar{\mathbb{X}}}\frac{\partial
g_{\theta}}{\partial
x_{[I]}}(x)f_{X}(x)\cdot\big{(}b_{\mathbb{X}}-a_{\mathbb{X}}\big{)}\cdot\mathds{\large
1}_{[a_{\mathbb{X}},b_{\mathbb{X}}]}(x_{[I]})d\lambda(x_{[\text{{MC}}]})d\lambda(x_{[I]})\,,$

where the last equality follows from combining eq. C.10 and eq. C.11. Applying
integration by parts, it now follows that

$\displaystyle\Delta_{s}$
$\displaystyle(\theta)=\bigintssss_{\bar{\mathbb{X}}}\bigg{(}\Big{[}g_{\theta}(x)f_{X}(x)\Big{]}^{x_{[I]}=b_{\mathbb{X}}}_{x_{[I]}=a_{\mathbb{X}}}-\int_{a_{\mathbb{X}}}^{b_{\mathbb{X}}}g_{\theta}(x)\frac{\partial
f_{X}}{\partial x_{[I]}}(x)\cdot\mathds{\large
1}_{[a_{\mathbb{X}},b_{\mathbb{X}}]}(x_{[I]})d\lambda(x_{[I]})\bigg{)}d\lambda(x_{[\text{{MC}}]})$
$\displaystyle=\underset{(\ast)}{\underbrace{\big{(}b_{\mathbb{X}}-a_{\mathbb{X}}\big{)}^{-1}\Big{(}\int_{\bar{\mathbb{X}}}g_{\theta}(\left.x\right|_{x_{[I]}=b_{\mathbb{X}}})\bar{\mu}_{X_{[\text{MC}]}|X^{I}}(dx_{[\text{{MC}}]},b_{\mathbb{X}})-\int_{\bar{\mathbb{X}}}g_{\theta}(\left.x\right|_{x_{[I]}=a_{\mathbb{X}}})\bar{\mu}_{X_{[\text{MC}]}|X^{I}}(dx_{[\text{{MC}}]},a_{\mathbb{X}})\Big{)}}}-$
$\displaystyle\quad\quad\underset{(\ast\ast)}{\underbrace{\int_{\bar{\mathbb{X}}}\int_{a_{\mathbb{X}}}^{b_{\mathbb{X}}}g_{\theta}(x)\frac{\partial
f_{X}}{\partial x_{[I]}}(x)\cdot\mathds{\large
1}_{[a_{\mathbb{X}},b_{\mathbb{X}}]}(x_{[I]})d\lambda(x_{[I]})d\lambda(x_{[\text{{MC}}]})}}\,.$

The statement "$\Delta_{s}(\theta)=(\ast)$" would constitute a similar result
to proposition 4 and 1 under assumption (A.II′′), however, term $(\ast\ast)$
can not w.l.o.g. be assumed to equal zero.

See 1

###### Proof.

These results follow immediately from basic transformations.

∎

###  C.3 Proofs of Proposition 1 and Proposition 2 plus Remark C.2

See 1

###### Proof.

First, note that Gelman & Hill (2007) define the average predictive difference
_per unit_ , which is an interval denoted by
$\big{[}u^{(lo)},u^{(hi)}\big{]}\in\mathcal{B}(\mathbb{R})$. Combining
equations (21.3) and (21.4) from Gelman & Hill (2007), the average predictive
difference $B_{u}$ is given by

$B_{u}\big{(}u^{(lo)},u^{(hi)}\big{)}=\frac{1}{n}\sum_{i=1}^{n}\frac{\mathbb{E}\big{[}y|u^{(hi)}v_{i},\theta\big{]}-\mathbb{E}\big{[}y|u^{(lo)}v_{i},\theta\big{]}}{u^{(hi)}-u^{(lo)}}\,,$
(C.12)

with $v_{i}$ denoting the $i$th observation of all regressors in $M\cup C$.
Let the sequence $\mathcal{D}_{X}^{\textup{[MC]}}$ containing values in
$\mathbb{R}^{p-d_{I}}$ denote the sequence of observation-vectors contained in
$\mathcal{D}_{X}$, but with the elements mapped to $I$ removed. For the
setting in this proposition, with $\mu_{X}$ given as in definition 6 and
$\bar{\mu}$ the marginal distribution obtained by marginalizing out the
regressor of interest and $\bar{\mathbb{X}}$ given by the set
$\\{\mathbf{x}^{\scaleto{\text{[MC]}}{5pt}}_{i}|\mathbf{x}^{\scaleto{\text{[MC]}}{5pt}}_{i}\in\mathcal{D}_{X}^{\textup{[MC]}}\\}$,
it now immediately follows that

$\displaystyle B_{u}\big{(}u^{(lo)},u^{(hi)}\big{)}$
$\displaystyle=\Big{(}u^{(hi)}-u^{(lo)}\Big{)}^{-1}\cdot\frac{1}{n}\sum_{i=1}^{n}\Big{(}g_{\theta}(x_{[I]}=u^{(hi)},\mathbf{x}^{\scaleto{\text{[MC]}}{5pt}}_{i})-g_{\theta}(x_{[I]}=u^{(lo)},\mathbf{x}^{\scaleto{\text{[MC]}}{5pt}}_{i})\Big{)}$
$\displaystyle=\Big{(}u^{(hi)}-u^{(lo)}\Big{)}^{-1}\cdot\Bigg{(}\int_{\bar{\mathbb{X}}}g_{\theta}(\left.x\right|_{x_{[I]}=u^{(hi)}})-g_{\theta}(\left.x\right|_{x_{[I]}=u^{(lo)}})d\bar{\mu}(x_{[\text{{MC}}]})\Bigg{)}$
$\displaystyle\overset{\text{prop.\ref{UnifProp}}}{=}\Delta_{s}(\theta)$

under assumption (A.II′), proving the statement.

######  Note.

Beneath the definition of average predictive difference, Gelman & Hill (2007)
add "_In practice, one must also average over $\theta$ (or plug in a point
estimate)_". ‘Averaging over $\theta$’ in the sense of averaging over draws
from either the posterior distribution, resulting from Bayesian inference, or
the $N(\hat{\theta},\Sigma_{\hat{\theta}})$ distribution, with frequentist
point estimate $\hat{\theta}$ and covariance matrix $\Sigma_{\hat{\theta}}$,
would constitute the point estimate we defined in definition 4.

∎

######  Remark C.2.

The extensions of $B_{u}$ given above that are proposed in Gelman & Pardoe
(2007) may also be incorporated into the current framework, as the following
details for each identified extension. Throughout this remark, we only
consider generalized marginal effects $\Delta_{s}$ _without_ the option of
separately quantifying main and interaction effects, as this approach
corresponds to Gelman & Pardoe (2007), who propose the following for dealing
with interaction terms: "When defining the predictive comparison […], we must
alter [an] input wherever it occurs in the model".

    1. C.2.a)

For the predictive comparison $\delta_{u}\big{(}u^{(1)}\longrightarrow
u^{(2)},v,\theta\big{)}$ defined by (Gelman & Pardoe 2007, eq.(1)), the
following holds within the current framework.

      1. C.2.a.i)

For a metric regressor of interest, let $\Delta_{s}$ denote the generalized
marginal effect under assumption (A.II$\,{}^{\prime}$), with $\mu$ chosen as
the measure associated with the $U\big{(}u^{(1)},u^{(2)}\big{)}$ distribution,
for $u^{(1)},u^{(2)}\in\mathbb{R}$, and $\bar{\mu}=\delta_{\\{v\\}}$ for some
fixed $v\in\mathbb{R}^{p-d_{I}}$, then

$\delta_{u}\big{(}u^{(1)}\longrightarrow
u^{(2)},v,\theta\big{)}\,\,\hat{=}\,\,\Delta_{s}(\theta)\,.$ (C.13)

      2. C.2.a.ii)

For a categorical regressor of interest, let $u^{(1)},u^{(2)}$ denote two
different categories, including the reference-category, of $X^{I}$.
Furthermore, let $\Delta_{s}$ again denote the generalized marginal effect
under assumption (A.II$\,{}^{\prime}$), with $\bar{\mu}=\delta_{\\{v\\}}$ for
some fixed $v\in\mathbb{R}^{p-d_{I}}$, then

$\delta_{u}\big{(}u^{(1)}\longrightarrow
u^{(2)},v,\theta\big{)}\,\,\hat{=}\begin{cases}\Delta_{s}^{[j]}(\theta)\,,&\begin{array}[]{l}\text{for
$u^{(2)}$ denoting the $j$th- and}\\\ \text{$u^{(1)}$ the reference-
category}\end{array}\\\\[15.0pt]
\Delta_{s}^{[j]}(\theta)-\Delta_{s}^{[l]}(\theta)\,,&\begin{array}[]{l}\text{for
$u^{(2)}$ denoting the $j$th- and}\\\ \text{$u^{(1)}$ the $l$th-
category}\end{array}\\\\[15.0pt]
-\Delta_{s}^{[j]}(\theta)\,,&\begin{array}[]{l}\text{for $u^{(2)}$ denoting
the reference- }\\\ \text{and $u^{(1)}$ the $j$th-
category,}\end{array}\end{cases}$ (C.14)

given that $(u^{(1)}-u^{(2)})$ is set to $1$ for a categorical regressor of
interest by Gelman & Pardoe (2007).

The average predictive comparisons for _Numerical Inputs_ and _Unordered
Categorical Inputs_ , given by (Gelman & Pardoe 2007, eqs.(2),(5)) and (Gelman
& Pardoe 2007, eqs.(3),(4)), respectively, is simply translated into this
work’s framework by averaging over the quantities from equations C.13 and eq.
C.14, respectively, equivalently to how they propose averaging over
$\delta_{u}\big{(}u^{(1)}\longrightarrow u^{(2)},v,\theta\big{)}$.

Having transferred the above quantities from Gelman & Pardoe (2007) into our
framework, it immediately follows that our framework also applies to _Variance
Components Models_ , _Inputs That Are Not Always Active_ , and _Nonmonotonic
Functions_ at the very least by proceeding exactly as suggested in sections
3.4., 3.6., and 3.7., respectively, of Gelman & Pardoe (2007).

In order to prove proposition 2, we now first require the following lemma.

######  Lemma C.1.

Let $h:\mathbb{R}^{d}\longrightarrow\mathbb{R}$, $d\in\mathbb{N}_{>0}$, be a
continuous and bounded function. For any probability measure $\mu$ on
$\Big{(}\mathbb{R}^{d},\mathcal{B}\big{(}\mathbb{R}^{d}\big{)}\Big{)}$ the
following holds

$\exists\,a\in\mathbb{R}^{d}:\quad\int_{\mathbb{R}^{d}}h(x)d\mu(x)=h(a)\,.$
(C.15)

###### Proof.

Let $b_{l},b_{h}\in\mathbb{R}$ denote the boundaries of $h$, i.e. $b_{l}\leq
h(x)\leq b_{h}$ $\forall x\in\mathbb{R}^{d}$. It then follows by the
_intermediate value theorem_ , if $d=1$, and the _main theorem of
connectedness_ , if $d\geq 2$, that the image of $h$ is exactly equal to the
closed interval between its boundaries, i.e. $\mathcal{I}(h)=[b_{l},b_{h}]$.
Furthermore, since it clearly holds that

$\displaystyle
b_{l}=\int_{\mathbb{R}^{d}}b_{l}\,d\mu(x)\leq\int_{\mathbb{R}^{d}}h(x)d\mu(x)\leq\int_{\mathbb{R}^{d}}b_{h}\,d\mu(x)=b_{h}\,,$

we also have that the value of the integral $\int_{\mathbb{R}^{d}}h(x)d\mu(x)$
lies in the interval $[b_{l},b_{h}]$.

Combining these two statements, it immediately follows that there exists some
$a\in\mathbb{R}^{d}$ so that $\int_{\mathbb{R}^{d}}h(x)d\mu(x)=h(a)$. ∎

See 2

###### Proof.

In order to conform with the notation of Kaufman (1996), let the metric
regressor of interest be the $j$th regressor, i.e. $x_{[I]}=x_{j}$,
$j\in\\{1,...,p\\}$, and let $b_{j}$ denote the corresponding coefficient in
the linear predictor $\eta$. Equation (5) of Kaufman (1996) then defines the
_instantaneous slope_ as $\partial P_{j}:=\partial P/\partial X_{j}$, with $P$
denoting the equivalent of $g_{\theta}$ in this setting. This is clearly equal
to $s$ as defined in eq. 4, however, equation (5) of Kaufman (1996) also
specifies that $\partial P_{j}=b_{j}P(1-P)$, limiting the definition to cases
in which the regressor of interest is not involved in any interactions.
Furthermore, equation (6) of Kaufman (1996) defines the _predicted change in
probability_ as

$\Delta
P_{j}=\Bigg{(}1+\exp\bigg{(}-\Big{(}\log\bigg{(}\frac{P_{ref}}{1-P_{ref}}\bigg{)}+\frac{1}{2}b_{j}\Big{)}\bigg{)}\Bigg{)}^{-1}-\Bigg{(}1+\exp\bigg{(}-\Big{(}\log\bigg{(}\frac{P_{ref}}{1-P_{ref}}\bigg{)}-\frac{1}{2}b_{j}\Big{)}\bigg{)}\Bigg{)}^{-1}\,\,,$
(C.16)

where $P_{ref}$ is not definitively defined. Kaufman (1996) does suggest the
quantities
$P_{ref}=$$n^{-1}\sum_{\mathbf{x}_{i}\in\mathcal{D}_{X}}g_{\theta}(\mathbf{x}_{i})$,
in the notation of definition 6, and $P_{ref}=g_{\theta}(\bar{x})$, where
$\bar{x}$ denotes the observed sample mean, as reasonable choices, however,
just as in Williams (2012), the latter is not clearly defined for cases where
$C$ is not an empty set (see section 3.1.1).

Note that, considering that $g_{\theta}$ is continuous and bounded in this
proposition’s setting, it follows by lemma C.1 that for any $\bar{\mu}$ chosen
according to assumptions (A.I) or (A.II′)

$\displaystyle\exists\,a\in\mathbb{R}^{p}\,:\quad
g_{\theta}(a)=\int_{\bar{\mathbb{X}}\times\mathbb{X}}g_{\theta}(x)d(\mu\times\bar{\mu})$
(C.17)

as well as, for any fixed $x_{[I]},k\in\mathbb{R}$

$\displaystyle\exists\,a\in\mathbb{R}^{p-1}\,:g_{\theta}\big{(}x_{[I]}+k,a\big{)}-g_{\theta}\big{(}x_{[I]},a\big{)}=\int_{\bar{\mathbb{X}}}g_{\theta}\big{(}x_{[I]}+k,x_{[\text{MC}]}\big{)}-g_{\theta}\big{(}x_{[I]},x_{[\text{MC}]}\big{)}\,d\bar{\mu}(x_{[\text{MC}]})\,.$
(C.18)

Given eq. C.17, we will extend the ‘reasonable choices’ suggested by Kaufman
(1996) to considering, for any choice of $x^{ref}\in\mathbb{R}^{p}$,
$P_{ref}=g_{\theta}(x^{ref})$ and show that, under assumptions (A.I) and
(A.II′), $\Delta_{s}$ may be specified to correspond to $\Delta P$ for any
$x^{ref}\in\mathbb{R}^{p}$.

First, we denote by $x^{ref}_{[I]}$ and $x^{ref}_{[\text{MC}]}$, the entries
of $x^{ref}$ representing $x_{[I]}$ and $x_{[\text{MC}]}$, respectively, and
define $x_{[I]}^{(lo)}:=x_{[I]}^{ref}-0.5$ and
$x_{[I]}^{(hi)}:=x_{[I]}^{ref}+0.5$. Since $\eta$ contains no interactions
involving $x_{[I]}$, it holds that

$\displaystyle g_{\theta}(x_{[I]}^{(hi)},x_{[\text{MC}]}^{ref})$
$\displaystyle=\Bigg{(}1+\exp\bigg{(}-\Big{(}\eta(x_{[I]}^{ref},x_{[\text{MC}]}^{ref})+\frac{1}{2}b_{j}\Big{)}\bigg{)}\Bigg{)}^{-1}$
(C.19)
$\displaystyle=\Bigg{(}1+\exp\bigg{(}-\Big{(}\log\Big{(}\exp\big{(}\eta(x^{ref})\big{)}\Big{)}+\frac{1}{2}b_{j}\Big{)}\bigg{)}\Bigg{)}^{-1}$
$\displaystyle=\Bigg{(}1+\exp\bigg{(}-\Big{(}\log\bigg{(}\frac{g_{\theta}(x^{ref})}{1-g_{\theta}(x^{ref})}\bigg{)}+\frac{1}{2}b_{j}\Big{)}\bigg{)}\Bigg{)}^{-1}$

and, equivalently,

$\displaystyle
g_{\theta}(x_{[I]}^{(lo)},x_{[\text{MC}]}^{ref})=\Bigg{(}1+\exp\bigg{(}-\Big{(}\log\bigg{(}\frac{g_{\theta}(x^{ref})}{1-g_{\theta}(x^{ref})}\bigg{)}-\frac{1}{2}b_{j}\Big{)}\bigg{)}\Bigg{)}^{-1}\,.$
(C.20)

Combining eq. C.19, eq. C.20, and eq. C.18 gives the following for the proper
choice of a probability measure $\bar{\mu}$ on
$\Big{(}\mathbb{R}^{p-1},\mathcal{B}\big{(}\mathbb{R}^{p-1}\big{)}\Big{)}$
corresponding to $x^{ref}$

$\displaystyle\Delta
P_{j}=\int_{\text{supp}(\bar{\mu})}g_{\theta}(x_{[I]}^{(hi)},x_{[\text{MC}]})-g_{\theta}(x_{[I]}^{(lo)},x_{[\text{MC}]})\,d\bar{\mu}(x_{[\text{MC}]})\,.$
(C.21)

By proposition 4, since $\big{|}x_{[I]}^{(hi)}-x_{[I]}^{(lo)}\big{|}=1$, eq.
C.21 clearly corresponds to $\Delta_{s}$ under either assumption (A.I) or
assumption (A.II′) with $\mu$ chosen as the measure associated with the
$U(x_{[I]}^{ref}-1/2,x_{[I]}^{ref}+1/2)$ distribution.

Therefore, it follows that for any choice of $P_{ref}$, there exists some
$u\in\mathbb{R}$ and choice of $\bar{\mu}$ under both assumptions (A.I) and
(A.II′) so that $\Delta_{s}$ to corresponds to $\Delta P$ for $\mu$ chosen as
the measure associated with the $U(u-1/2,u+1/2)$ distribution, proving the
statement. ∎

###  C.4 Proof of Proposition 3 plus Remark C.3

See 3

###### Proof.

Firstly, note that in the notation of Apley & Zhu (2020)
$\mathcal{S}\hat{=}\text{supp}(\mu_{X})$, with $\mu_{X}$ denoting the joint
distribution of all regressors. Additionally, for the case of the regressor of
interest, denoted by $X_{j}$ in the setting of this proposition, being metric,
$f^{j}(\boldsymbol{\cdot})\hat{=}s(\hat{\theta},\boldsymbol{\cdot})$.
Therefore, it immediately follows that requirements (a) and (b) from (Apley &
Zhu 2020, theorem 1) are met in the setting of this proposition. Furthermore,
as $f^{j}(\boldsymbol{\cdot})=s(\hat{\theta},\boldsymbol{\cdot})$, it
evidently holds that $\mathbb{E}[f^{j}(X_{j},\mathbf{X}_{\backslash
j})|X_{j}]\hat{=}\bar{s}(\hat{\theta},x)$ under assumption (A.II′′) in the
current setting. Since, $\forall x_{[I]}\in\text{supp}(\mu_{j})$,
$s(\hat{\theta},x)$ is continuous in $x_{[I]}$ on $\text{supp}(\mu_{j})$ as a
finite integral over a continuous function , requirement (c) from (Apley & Zhu
2020, theorem 1) is also met. It then follows for any
$z\in\text{supp}(\mu_{j})$ that

$\displaystyle g_{j,ALE}(z)$
$\displaystyle=\int_{\min\\{\sup(\mu_{j})\\}}^{z}\bar{s}(\hat{\theta},x)\,\,dx_{j}$
$\displaystyle=\underset{=\Delta_{s}(\hat{\theta})\text{ for the
specifications made in the setting of this
proposition}}{\underbrace{\frac{1}{\text{\small$\big{(}z-\min\\{\text{supp}(\mu_{j})\\}\big{)}$}}\int_{\min\\{\sup(\mu_{j})\\}}^{z}\bar{s}(\hat{\theta},x)\,\,dx_{[I]}\,\,}}\cdot\big{(}z-\min\\{\text{supp}(\mu_{j})\\}\big{)}$
$\displaystyle=\Delta_{s}(\hat{\theta})\cdot\big{(}z-\min\\{\text{supp}(\mu_{j})\\}\big{)}\,,$

proving the statement. ∎

######  Remark C.3.