Fissler & Pesenti

Sensitivity Measures Based on Scoring Functions

Sensitivity Measures Based on Scoring Functions

Tobias Fissler Institute for Statistics and Mathematics, Vienna University of
Economics and Business (WU), Vienna, Austria††thanks<EMAIL_ADDRESS>Silvana M. Pesenti Department of Statistical Sciences, University of Toronto,
Toronto, Canada††thanks<EMAIL_ADDRESS>
We propose a holistic framework for constructing sensitivity measures for any
elicitable functional $T$ of a response variable. The sensitivity measures,
termed score-based sensitivities, are constructed via scoring functions that
are (strictly) consistent for $T$. These score-based sensitivities quantify
the relative improvement in predictive accuracy when available information,
e.g., from explanatory variables, is used ideally. We establish intuitive and
desirable properties of these sensitivities and discuss advantageous choices
of scoring functions leading to scale-invariant sensitivities.

Since elicitable functionals typically possess rich classes of (strictly)
consistent scoring functions, we demonstrate how Murphy diagrams can provide a
picture of all score-based sensitivity measures. We discuss the family of
score-based sensitivities for the mean functional (of which the Sobol indices
are a special case) and risk functionals such as Value-at-Risk, and the pair
Value-at-Risk and Expected Shortfall. The sensitivity measures are illustrated
using numerous examples, including the Ishigami–Homma test function. In a
simulation study, estimation of score-based sensitivities for a non-linear
insurance portfolio is performed using neural nets.

Consistency; Elicitability; Expected Shortfall; Information value; Value-at-
Risk

##  1 Introduction

We consider the context of quantitative risk management where ${Y}$ describes
a random variable of interest, e.g., an (insurance) portfolio. The vector
${\boldsymbol{X}}=(X_{1},\ldots,X_{n})$, $n\in{\mathbb{N}}$, describes the
risk factors or risk factor changes which determine $Y$ via a mapping or
aggregation function $\mathsf{g}\colon{\mathbb{R}}^{n}\to{\mathbb{R}}$, such
that ${Y}=\mathsf{g}({\boldsymbol{X}})$, see e.g. McNeil et al. (2015).
Typical in applications is that not all risk factors ${\boldsymbol{X}}$ are
observable, e.g., only functions of ${\boldsymbol{X}}$ are observable, or
their observation is costly. In such situations, the available information may
be expressed via a random vector ${\boldsymbol{W}}=(W_{1},\ldots,W_{d})$,
$d\in{\mathbb{N}}$, and in the absence of knowing what factors determine $Y$,
${\boldsymbol{W}}$ may contain information irrelevant for modelling $Y$.

The key question we aim to address is: How sensitive is $Y$ with respect to
${\boldsymbol{W}}$? Or more specifically: What is the information value of
${\boldsymbol{W}}$ for $Y$? This can be made more precise by assessing by how
much the uncertainty of $Y$ is reduced when knowing/learning
${\boldsymbol{W}}$. We will see that this latter question can be equivalently
rephrased as: What is the gain in predictive accuracy for $Y$ when knowing
${\boldsymbol{W}}$? To answer these questions one needs to first clarify some
imminent ones. How is uncertainty of $Y$ measured? And what is the predictive
target? As for the latter, one distinguishes between probabilistic
predictions, that is, specifying the full conditional distribution of $Y$
given ${\boldsymbol{W}}$, and point predictions. For point predictions the
conditional distribution of $Y$ given ${\boldsymbol{W}}$ is summarised by a
functional $T$ such as the mean or a law-determined risk measure, e.g., Value-
at-Risk (VaR) or Expected Shortfall (ES). Then, truthful prediction amounts to
specifying the correct conditional distribution $F_{Y|{\boldsymbol{W}}}$ or a
functional thereof, $T(F_{Y|{\boldsymbol{W}}})$.

The above raised question is typically addressed using sensitivity analysis,
and in particular, via sensitivity measures or importance measures (Saltelli
et al. 2008). Sensitivity measures associate the uncertainty in $Y$ with the
uncertainties in the risk factors in a way that allows for e.g., importance
ranking of risk factors. Such an importance ranking may inform where to direct
scarce resources to collect more data. Besides, it may lead to model
simplifications when a factor is deemed to have irrelevant explanatory power.
The literature on sensitivity measures is vast and we refer to Borgonovo and
Plischke (2016) and Razavi et al. (2021) for an extensive review. Examples of
sensitivity measures include variance-based (Saltelli and Tarantola 2002),
moment-independent (Borgonovo 2007), and quantile-based sensitivities (Browne
et al. 2017). Alternative approaches include those based on divergence
measures (Gamboa et al. 2018, Pesenti et al. 2019, Fort et al. 2021, Pesenti
2021) and differential sensitivity measures, see Tsanakas and Millossovich
(2016) and Pesenti et al. (2021) in a risk management context. However, as
argued in Borgonovo et al. (2021), the choice of a sensitivity measure should
be intimately tied to the functional of interest $T$ via the notion of
strictly consistent scoring functions and moreover reflect the information
value of risk factors.

A scoring function or scoring rule $S(\cdot,\cdot)$ maps the tuple $(z,y)$,
consisting of a prediction $z$ and an observation $y$ of $Y$, to the non-
negative real number $S(z,y)$ with the convention that smaller values of $S$
reflect more accurate predictions of $Y$. Here, the prediction $z$ may be a
point prediction, an interval, or the entire probability distribution or
density of $Y$. As forcefully argued in Murphy and Daan (1985), Engelberg et
al. (2009), and Gneiting (2011), a score $S$ should be strictly consistent for
a functional $T$. A strictly consistent score incentivises truthful
predictions in the sense that the expected score is strictly minimised by the
target at hand, which in our setup is the conditional distribution
$F_{Y|{\boldsymbol{W}}}$ or a functional thereof, $T(F_{Y|{\boldsymbol{W}}})$.
Examples for strictly consistent scoring functions are the squared loss for
the mean functional and the negative log-likelihood or the continuous ranked
probability score for the conditional distribution $F_{Y|{\boldsymbol{W}}}$
(Gneiting and Raftery 2007). If a functional $T$ admits a strictly consistent
scoring function it is called elicitable (Lambert et al. 2008). Many
functionals of interest are elicitable, e.g. the mean via the squared loss
$S(z,y)=(z-y)^{2}$ and the median via the absolute loss $S(z,y)=|z-y|$. As for
risk measures, entropic risk measures, VaR, and expectiles are elicitable,
while the variance, standard deviation, and ES fail to be elicitable (Osband
1985, Weber 2006, Gneiting 2011). However, the pair consisting of the mean and
variance (or standard deviation) and the pair VaR and ES admit strictly
consistent scoring functions (Fissler and Ziegel 2016).

If the functional of interest $T$ is elicitable, the information value of
${\boldsymbol{W}}$ for $Y$ can be measured by the potential reduction of
uncertainty when knowing ${\boldsymbol{W}}$ – expressed by a strictly
consistent scoring function $S$ – that is by
${\mathbb{E}}[S(T(F_{Y}),Y)]-{\mathbb{E}}[S(T(F_{Y|{\boldsymbol{W}}}),Y)]$; a
notion first proposed in the sensitivity literature by Borgonovo et al.
(2021). We build on their suggestion by defining for a strictly consistent
scoring function $S$ the score-based sensitivity measure of $Y$ to information
${\boldsymbol{W}}$ as

$\xi_{S}(Y;{\boldsymbol{W}})=\frac{{\mathbb{E}}[S(T(F_{Y}),Y)]-{\mathbb{E}}[S(T(F_{Y|{\boldsymbol{W}}}),Y)]}{{\mathbb{E}}[S(T(F_{Y}),Y)]}\,,$

see Definition 2.2 for a precise statement of the involved assumptions. By
construction, the score-based sensitivity measure attains values in $[0,1]$
and is unitless, which allows for comparison of sensitivities to risk factors
that live on different scales. Sensitivity measures constructed via strictly
consistent scoring functions include the Sobol indices (and extensions
thereof), corresponding to the mean functional and the squared loss (Borgonovo
et al. 2021), and the contrast index studied in Maume-Deschamps and Niang
(2018), corresponding to the pinball loss and the VaR functional. Fort et al.
(2016) consider a related concept using contrast functions in the machine
learning literature. Sensitivity concepts that reflect value of information
(which typically are intimately connected to scoring functions) have been
studied by Felli and Hazen (1998) and more recently by Borgonovo and Cillo
(2017) in the context of probability safety assessment, and by Straub et al.
(2021) in applications to reliability analysis.

In this paper, we provide a comprehensive framework for constructing score-
based sensitivity measures for any elicitable functional $T$. Moreover, we
establish universal properties of score-based sensitivity measures that are
inherited by any elicitable functional with any strictly consistent scoring
function. In particular, we argue that sensitivity measures should possess the
zero information gain property, a property significantly weaker than Borgonovo
et al. (2021)’s nullity-implies-independence property. Indeed, while the
nullity-implies-independence property means that a zero sensitivity implies
that ${\boldsymbol{W}}$ and $Y$ are independent, we only require that a
sensitivity of zero is equivalent to ${\boldsymbol{W}}$ being irrelevant for
modelling $T(F_{Y})$. Furthermore, we show that a score-based sensitivity
measure is equal to 1 if and only if ${\boldsymbol{W}}$ contains _all
relevant_ information to model $T(F_{Y})$. We additionally define an
interaction sensitivity measure – termed interaction information – that
quantifies the information value of interactions of risk factors. In the
special case of the squared loss scoring function we recover the well-known
Sobol interaction terms.

One imminent challenge of score-based sensitivity measures is the choice of
scoring function. An elicitable functionals $T$ typically admits infinitely
many strictly consistent scoring functions. Furthermore, as we illustrate in
examples, different scoring functions may lead to sensitivity measures that
rank information, and thus risk factors, differently. To overcome these
difficulties, we advocate for scoring functions that lead to scale-invariant
sensitivities. We moreover promote and illustrate the usage of Murphy
diagrams, which has been impressively demonstrated in the context of scoring
functions in Ehm et al. (2016). Another challenge is the estimation of the
score-based sensitivity, in particular the term
${\mathbb{E}}[S(T(F_{Y|{\boldsymbol{W}}}),Y)]$. One way to address this is
with neural nets from machine learning, which we do in Section 5.

This paper is organised as follows. Section 2 motivates and defines score-
based sensitivity measures. In Section 3 we discuss their universal properties
and introduce a sensitivity that quantifies the value of information of
interactions of risk factors; termed interaction information. Section 4
discusses the choice of strictly consistent scoring functions and defines
score-based Murphy diagrams for sensitivity measures. We illustrate the score-
based sensitivities on the Ishigami–Homma test function and a non-linear
insurance portfolio in Section 5.

##  2 From Scoring Functions to Sensitivity Measures

Let $(\Omega,\mathfrak{F},{\mathbb{P}})$ be a complete probability space on
which we identify random elements which almost surely coincide. Moreover, if
not stated explicitly, all events such as equalities, inequalities, etc., are
to be understood in an almost sure sense. We use the decision-theoretic setup
and notation of Gneiting (2011) and Fissler et al. (2021a). For this let
$\mathcal{M}^{0}({\mathbb{R}})$ be the class of all Borel probability measures
on ${\mathbb{R}}$ and let
$\mathcal{M}\subseteq\mathcal{M}^{\prime}\subseteq\mathcal{M}^{0}({\mathbb{R}})$
be two sub-classes. We equip $\mathcal{M}^{0}({\mathbb{R}})$ with the
$\sigma$-algebra generated by the family of evaluation maps
$\\{\pi_{B}\\}_{B\in\mathfrak{F}}$ given by
$\pi_{B}\colon\mathcal{M}^{0}({\mathbb{R}})\to[0,1]$, which map
$\mu\mapsto\pi_{B}(\mu)=\mu(B)$. We moreover identify any probability measure
$\mu\in\mathcal{M}^{0}({\mathbb{R}})$ with its cumulative distribution
function $F(x)=\mu((-\infty,x])$, $x\in{\mathbb{R}}$. Further, let
$\mathsf{A}$ be an action domain – in our context, this is typically the
interval $(0,\infty)$, ${\mathbb{R}}$, or ${\mathbb{R}}^{k}$ – equipped with
with the Borel $\sigma$-algebra. The predictive goal is then described by the
measurable functional $T\colon\mathcal{M}^{\prime}\to\mathsf{A}$ and we refer
to Fissler and Holzmann (2022) for measurability results for functionals of
interest such as the mean, expectiles, Value-at-Risk (VaR), and Expected
Shortfall (ES).

For a random variable $Y$ in some class $\mathcal{Y}\subseteq
L^{0}(\Omega,\mathfrak{F},{\mathbb{P}})$ we denote its cumulative distribution
function by $F_{Y}(\cdot)={\mathbb{P}}(Y\leq\cdot)$. For a
sub-$\sigma$-algebra $\mathfrak{A}\subseteq\mathfrak{F}$ – often referred to
as _information set_ – we denote by $F_{Y|\mathfrak{A}}$ a regular version of
the conditional distribution of $Y$ given $\mathfrak{A}$. (Recall that
$F_{Y|\mathfrak{A}}$ is a measurable map from $\Omega$ to
$\mathcal{M}^{0}({\mathbb{R}})$ (Fissler and Holzmann 2022).) If
$F_{Y}\in\mathcal{M}^{\prime}$ and $F_{Y|\mathfrak{A}}\in\mathcal{M}^{\prime}$
(almost surely, which will be suppressed in the sequel), we can consider the
(measurable) random variables $T(Y)=T(F_{Y})$ and
$T(Y|\mathfrak{A})=T(F_{Y|\mathfrak{A}})$. If $\mathfrak{A}$ is generated by
some random vector ${\boldsymbol{W}}$, i.e.
$\mathfrak{A}=\sigma({\boldsymbol{W}})$, then we simply write
$F_{Y|{\boldsymbol{W}}}=F_{Y|\mathfrak{A}}$ and use the shorthand
$T(Y|{\boldsymbol{W}})=T(Y|\sigma({\boldsymbol{W}}))$.

######  Definition 2.1 (Consistency & Elicitability)

A scoring function (or score) is a measurable map
$S\colon\mathsf{A}\times{\mathbb{R}}\to[0,\infty]$. For a functional
$T\colon\mathcal{M}^{\prime}\to\mathsf{A}$ and a sub-class
$\mathcal{M}\subseteq\mathcal{M}^{\prime}$ the scoring function $S$ may
satisfy the following properties:

  1. $(i)$

$S$ is _$\mathcal{M}$ -consistent_ for $T$, if for all $F\in\mathcal{M}$ and
for all $z\in\mathsf{A}$

$\int S(T(F),y)\,\mathrm{d}F(y)\leq\int S(z,y)\,\mathrm{d}F(y)\,.$ (1)

  2. $(ii)$

$S$ is _strictly_ $\mathcal{M}$-consistent for $T$, if it is
$\mathcal{M}$-consistent for $T$ and if the inequality in (1) is strict for
all $z\neq T(F)$.

  3. $(iii)$

$T$ is elicitable on $\mathcal{M}$, if there exists a strictly
$\mathcal{M}$-consistent scoring function for $T$.

Elicitability of a functional is equivalent to the fact that it is a Bayes act
or the minimiser of an expected score (Gneiting 2011).

We will work with the following set of assumptions throughout the rest of the
paper.

Let $S$ be an $\mathcal{M}$-consistent scoring function for
$T\colon\mathcal{M}^{\prime}\to\mathsf{A}$ and
$\mathcal{M}\subseteq\mathcal{M}^{\prime}$. Suppose that $F_{Y}\in\mathcal{M}$
for all $Y\in\mathcal{Y}$ and that $\delta_{y}\in\mathcal{M}^{\prime}$ for all
$y\in{\mathbb{R}}$.111For $y\in{\mathbb{R}}$, the point measure $\delta_{y}$
is a probability measure such that for any $B\subseteq{\mathbb{R}}$,
$\delta_{y}(B)=1$ if $y\in B$ and $\delta_{y}(B)=0$ otherwise. Note that
$F_{Y|\mathfrak{F}}=F_{Y|Y}=\delta_{Y}$ is a random probability measure on
${\mathbb{R}}$. Let the following hold:

  1. $(i)$

$S(T(\delta_{y}),y)=0$ for all $y\in{\mathbb{R}}$.

  2. $(ii)$

$S(T(\delta_{y}),y)<S(z,y)$ for all $y\in{\mathbb{R}}$, $z\neq T(\delta_{y})$.

  3. $(iii)$

For all $F\in\mathcal{M}$, $\int S(T(F),y)\,\mathrm{d}F(y)<\infty$.

  4. $(iv)$

For all $Y\in\mathcal{Y}$, $\int S(T(F_{Y}),y)\,\mathrm{d}F_{Y}(y)>0$.

Assumption 2 part (ii) amounts to strict consistency for $T$ on the class of
point measures $\\{\delta_{y}\colon y\in{\mathbb{R}}\\}$. Part (i) is a
normalisation condition and can be achieved under (ii) by considering the
normalised score $\tilde{S}(z,y)=S(z,y)-S(T(\delta_{y}),y)$. The finiteness in
(iii) is implied if $S$ is strictly $\mathcal{M}$-consistent. Indeed, if $\int
S(T(F),y)\,\mathrm{d}F(y)=\infty$, it cannot be _strictly_ smaller than $\int
S(z,y)\,\mathrm{d}F(y)$ for $z\neq T(F)$. Condition (iv) usually holds if $S$
is strictly $\mathcal{M}$-consistent and $Y$ is not constant almost surely.

It is well-known that consistent scoring functions respect increasing
information sets (Holzmann and Eulert 2014, Theorem 1). This means that if $S$
is $\mathcal{M}$-consistent for $T$ and if
$\mathfrak{A}\subseteq\mathfrak{A}^{\prime}\subseteq\mathfrak{F}$ are two
sub-$\sigma$-algebras such that
$F_{Y|\mathfrak{A}},F_{Y|\mathfrak{A}^{\prime}}\in\mathcal{M}$, then

${\mathbb{E}}[S(T(Y|\mathfrak{A}^{\prime}),Y)]\leq{\mathbb{E}}[S(T(Y|\mathfrak{A}),Y)].$
(2)

If $S$ is strictly $\mathcal{M}$-consistent for $T$, then equality in (2)
holds only if $T(Y|\mathfrak{A}^{\prime})=T(Y|\mathfrak{A})$. (The main
argument behind this result is an application of the definition of (strict)
consistency in combination with the tower property for the conditional
expectation.) This provides a motivation for considering, for
$\mathfrak{A}\subseteq\mathfrak{F}$, the term

${\mathbb{E}}[S(T(Y),Y)]-{\mathbb{E}}[S(T(Y|\mathfrak{A}),Y)]\geq 0,$ (3)

which corresponds to the _resolution term_ of the score decomposition in Pohle
(2020), the _discrimination_ in Gneiting and Resin (2021), and it is a special
instance for a _score divergence_ (Thorarinsdottir et al. 2013, Gneiting and
Raftery 2007, Dawid 2007), which is related to the _cost of uncertainty_
discussed in Frankel and Kamenica (2019). The resolution term (3) quantifies
how helpful the information $\mathfrak{A}$ is to improve the correct baseline
model $T(Y)$, when used ideally. This improvement is naturally measured in
terms of a consistent score (Gneiting 2011). The next definition quantifies
this improvement relative to the so-called oracle improvement of full
information

${\mathbb{E}}[S(T(Y),Y)]-{\mathbb{E}}[S(T(Y|\mathfrak{F}),Y)]={\mathbb{E}}[S(T(Y),Y)]>0,$

where equality follows from Assumption 2 (i) and the fact that
$T(Y|\mathfrak{F})=T(\delta_{Y})$ and strict positivity from Assumption 2
(iv).

The resolution term normalised by the oracle improvement motivates to consider
the following notion of sensitivity measure. The sensitivity measure is
inspired by Borgonovo et al. (2021) who introduced sensitivity measures based
on scoring functions to the sensitivity literature.

######  Definition 2.2 (Sensitivities based on scoring functions)

Let $S$ be an $\mathcal{M}$-consistent scoring function for
$T\colon\mathcal{M}^{\prime}\to\mathsf{A}$ with
$\mathcal{M}\subseteq\mathcal{M}^{\prime}$ satisfying Assumption 2, $Y$ a
random variable, and $\mathfrak{A}\subseteq\mathfrak{F}$ an information set
with $F_{Y},F_{Y|\mathfrak{A}}\in\mathcal{M}$. Then the _sensitivity_ of ${Y}$
to $\mathfrak{A}$ based on $S$ is given by

$\xi_{S}(Y;\mathfrak{A})=\frac{{\mathbb{E}}[S(T(Y),Y)]-{\mathbb{E}}[S(T(Y|\mathfrak{A}),Y)]}{{\mathbb{E}}[S(T(Y),Y)]}\,.$
(4)

The larger the value of $\xi_{S}(Y;\mathfrak{A})$, the larger is the
information value of $\mathfrak{A}$, measured by the scoring function $S$.
While Borgonovo et al. (2021) consider sensitivity measures based solely on
the numerator of (4), our proposed sensitivity measure is normalised to lie
between 0 and 1; see Theorem 3.1 for details. This normalisation is achieved
by the division with the oracle improvement ${\mathbb{E}}[S(T(Y),Y)]$; a
quantity called _uncertainty_ in the forecasting literature (Gneiting and
Resin 2021). ${\mathbb{E}}[S(T(Y),Y)]$ is a generalisation of the variance
which arises when $S$ is the squared loss. Our sensitivity measures closely
resemble the _universal coefficient of determination_ which was recently
introduced by Gneiting and Resin (2021) and which generalises the classical
coefficient of determination R2. The subtle difference is that in this work,
the term $T(Y|\mathfrak{A})$ in (4) is conditionally calibrated with respect
to the information $\mathfrak{A}$, whereas Gneiting and Resin (2021) require
the notion of a “conditionally $T$-calibrated forecast”, which is generally a
weaker assumption. Moreover, (4) is a special instance of a _skill score_ due
to Murphy (1973); see also Gneiting and Raftery (2007) for a discussion.

Exploiting the consistency of $S$ for $T$ in Definition 2.2, we could replace
${\mathbb{E}}[S(T(Y),Y)]$ by $\min_{z\in{\mathbb{R}}}{\mathbb{E}}[S(z,Y)]$ and
${\mathbb{E}}[S(T(Y|\mathfrak{A}),Y)]$ by
${\mathbb{E}}\big{[}\min_{Z\in\mathfrak{A}}{\mathbb{E}}[S(Z,Y)|\mathfrak{A}]\big{]}$,
where the latter minimum is taken over all $\mathfrak{A}$-measurable random
variables $Z$. Thus, the score-based sensitivity only depends on the
functional $T$ through its consistent score.

In the sequel, we shall use the shorthand
$\xi_{S}(Y;{\boldsymbol{W}})=\xi_{S}(Y;\sigma({\boldsymbol{W}}))$ to indicate
the sensitivity of $Y$ with respect to the information generated by a random
vector ${\boldsymbol{W}}$.

##  3 Properties of Score-based Sensitivities

Theorem 3.1 provides the core properties of our score-based sensitivities.

###### Theorem 3.1

Let $S$ be an $\mathcal{M}$-consistent scoring function for
$T\colon\mathcal{M}^{\prime}\to\mathsf{A}$ with
$\mathcal{M}\subseteq\mathcal{M}^{\prime}$ satisfying Assumption 2, $Y$ a
random variable, and $\mathfrak{A}$ an information set with
$F_{Y},F_{Y|\mathfrak{A}}\in\mathcal{M}$. Then, $\xi_{S}(Y;\mathfrak{A})$
satisfies the following properties.

  1. (a)

_Normalisation:_ $\xi_{S}(Y;\mathfrak{A})\in[0,1]$.

  2. (b)

_Zero information gain:_ $T(Y|\mathfrak{A})=T(Y)$ implies
$\xi_{S}(Y;\mathfrak{A})=0$. If $S$ is strictly $\mathcal{M}$-consistent then
$\xi_{S}(Y;\mathfrak{A})=0$ only if $T(Y|\mathfrak{A})=T(Y)$.

  3. (c)

_Full information gain:_ $T(Y|\mathfrak{A})=T(\delta_{Y})$ implies that
$\xi_{S}(Y;\mathfrak{A})=1$. If $S$ is strictly $\mathcal{M}$-consistent then
$\xi_{S}(Y;\mathfrak{A})=1$ only if $T(Y|\mathfrak{A})=T(\delta_{Y})$.

  4. (d)

_Monotonicity with respect to nested information:_ For any information set
$\mathfrak{A}^{\prime}$ with $\mathfrak{A}\subseteq\mathfrak{A}^{\prime}$ and
$F_{Y|\mathfrak{A}^{\prime}}\in\mathcal{M}$ it holds that
$\xi_{S}(Y;\mathfrak{A})\leq\xi_{S}(Y;\mathfrak{A}^{\prime})$. If $S$ is
strictly $\mathcal{M}$-consistent then
$\xi_{S}(Y;\mathfrak{A})=\xi_{S}(Y;\mathfrak{A}^{\prime})$ only if
$T(Y|\mathfrak{A})=T(Y|\mathfrak{A}^{\prime})$.

Proof of Theorem 3.1. The normalisation follows from the non-negativity of the
resolution term (3). The remaining assertions are a corollary of Theorem 1 in
Holzmann and Eulert (2014).

The normalisation of the proposed sensitivity has the obvious advantage of
rendering the sensitivity unitless. This facilitates comparison of
sensitivities to risk factors that live on different scales in a
straightforward manner. In the subsequent subsections we discuss the
properties of the proposed sensitivity measures, including the ones
established in Theorem 3.1, and provide illustrating examples. Interestingly,
these properties (almost) correspond to four of the five axioms stipulated in
Griessenberger et al. (2022) for a dependence measure between $Y$ and
$\mathfrak{A}$, which yields an alternative angle on our sensitivity measures.

###  3.1 Zero Information Gain

We first start with an obvious, yet important, corollary of the zero
information gain property stated in Theorem 3.1 (b).

###### Corollary 3.2

Under the assumptions of Theorem 3.1, if $Y$ and $\mathfrak{A}$ are
independent, then $\xi_{S}(Y;\mathfrak{A})=0$.

Corollary 3.2 follows from Theorem 3.1 since independence of $Y$ and
$\mathfrak{A}$ is equivalent to $F_{Y|\mathfrak{A}}=F_{Y}$, which implies that
$T(Y|\mathfrak{A})=T(Y)$.

Clearly, if $Y$ and $\mathfrak{A}$ are stochastically independent,
$\mathfrak{A}$ does not contain any information about $Y$. The perspective of
the zero information gain property, however, is more nuanced than
independence. Assume that $\mathfrak{A}$ contains information about $Y$, i.e.,
they are not independent, but the information is not relevant for modelling
$T$, i.e., $T(Y|\mathfrak{A})=T(Y)$, then the score-based sensitivity measure
$\xi_{S}(Y;\mathfrak{A})$ is equal to 0. Thus, the sensitivity measure being
equal to 0 implies that $\mathfrak{A}$ contains no relevant information for
modelling $T(Y)$. Examples 3.3 and 3.4 below illustrate such situations. For
the mean functional $T$, the identity $T(Y|\mathfrak{A})=T(Y)$ is also known
as _mean independence_ (Wooldridge 2013, p. 25) which is weaker than
independence but stronger than uncorrelatedness. Following this terminology
$T(Y|\mathfrak{A})=T(Y)$ can be called $T$-independence of $Y$ from
$\mathfrak{A}$.

Borgonovo et al. (2021) advocate that sensitivity measures should satisfy the
so-called _nullity-implies-independence_ (n.i.i.) property, see Definition 4
in Borgonovo et al. (2021); a property inspired by Rény’s postulate D for
measures of dependence (Rényi 1959). A (sensitivity) measure satisfies the
n.i.i. property – Rény’s postulate D – if the measure is zero if and only if
$\mathfrak{A}$ and $Y$ are independent. While Rényi considers measures of
dependence between any two random variables, a sensitivity measure in our
manuscript (and the extant literature) is a measure between the unconditional
distribution $F_{Y}$ and the conditional distribution $F_{Y|\mathfrak{A}}$ of
the output $Y$. Therefore, in our context, the n.i.i. property means that the
sensitivity measure is 0 if and only if $F_{Y}$ and $F_{Y|\mathfrak{A}}$
coincide (almost surely). Since the conditional distribution
$F_{Y|\mathfrak{A}}$ enters the sensitivity measure (4) only via the
functional $T$, one cannot hope for the n.i.i. property unless $T$ is the
identity functional. If a modeller is interested in the information value of
$\mathfrak{A}$ for $T(Y)$ only, the n.i.i. property needs to be replaced by an
achievable desideratum which is the zero information gain property. In this
spirit, we may call the zero information gain property the “nullity-
implies-$T$-independence”, which we illustrate in Examples 3.3 and 3.4. We
emphasise that for the special case of probabilistic predictions, i.e. when
$T$ is the identity map, the zero information gain property coincides with the
n.i.i. property; see Proposition 6 in Borgonovo et al. (2021).

###### Example 3.3

Consider the output $Y=X_{1}X_{2}+X_{3}$, where $X_{1},X_{2},X_{3}$ are
independent and non-deterministic, $X_{1}>0$, and ${\mathbb{E}}[X_{2}]=0$.
Clearly, $Y$ and $X_{1}$ are dependent. For $T$ the mean functional, we obtain
that $T(Y|X_{1})={\mathbb{E}}[Y|X_{1}]={\mathbb{E}}[X_{3}]={\mathbb{E}}[Y]$.
Thus, knowing $X_{1}$ does not help to make a mean-model more precise.
Therefore, $\xi_{S}(Y;X_{1})=0$ for any consistent scoring function $S$ for
the mean functional.

For the next example, we recall the definition of Value-at-Risk (VaR) of a
random variable $Y$ at level $\alpha\in(0,1)$ which is given by

${\mathrm{VaR}}_{\alpha}(Y)=\inf\\{t\in{\mathbb{R}}\mid
F_{Y}(t)\geq\alpha\\}\,,$

corresponding to the left-continuous $\alpha$-quantile of the distribution
$F_{Y}$ of $Y$. Since VaR is a law-determined functional, we may write
${\mathrm{VaR}}_{\alpha}(F_{Y})={\mathrm{VaR}}_{\alpha}(Y)$.

###### Example 3.4

Let $X_{1},X_{2},X_{3}$ be independent, $X_{1}$ has a Bernoulli distribution
with $p={\mathbb{P}}(X_{1}=0)=1-{\mathbb{P}}(X_{1}=1)\in(0,\alpha)$, and
$X_{2}<C<X_{3}$ almost surely for a constant $C>0$. The output
$Y={\mathds{1}}_{\\{X_{1}=0\\}}X_{2}+{\mathds{1}}_{\\{X_{1}=1\\}}X_{3}$ may be
viewed as an insurance portfolio consisting of small claims – smaller than $C$
– which occur with probability $p$, and large claims – larger than $C$ –
occurring with probability $1-p$. Then,

${\mathrm{VaR}}_{\alpha}(Y|X_{3})={\mathrm{VaR}}_{\alpha}(Y|X_{2},X_{3})=X_{3}.$

Moreover,

${\mathrm{VaR}}_{\alpha}(Y|X_{2})={\mathrm{VaR}}_{\alpha}(Y)={\mathrm{VaR}}_{(\alpha-p)/(1-p)}(X_{3}).$

Thus, knowing $X_{2}$ does not improve the modelling of the ${\mathrm{VaR}}$
functional even though $Y$ and $X_{2}$ are dependent. The reason is that
$X_{2}$ only affects the distribution of $Y$ in the left tail below the
$p$-quantile. As a consequence, for any consistent score $S$ for
${\mathrm{VaR}}_{\alpha}$ it holds $\xi_{S}(Y;X_{2})=0$.

###  3.2 Full Information Gain

The counterpart of zero information gain is full information gain. We start
with a straightforward and important observation.

###### Corollary 3.5

Under the assumptions of Theorem 3.1, if $Y$ is $\mathfrak{A}$-measurable,
then $\xi_{S}(Y;\mathfrak{A})=1$.

Again, the proof of Corollary 3.5 is simple. If $Y$ is
$\mathfrak{A}$-measurable, the conditional distribution of $Y$ given
$\mathfrak{A}$ reduces to a random point measure in $Y$, that is
$F_{Y|\mathfrak{A}}=\delta_{Y}$. This in turn implies that
$T(F_{Y|\mathfrak{A}})=T(\delta_{Y})$ and therefore
$S(T(F_{Y|\mathfrak{A}}),Y)=0$ by Assumption 2(i).

Corollary 3.5 for example applies if $\mathfrak{A}$ is generated by a real-
valued random variable $W$ such that the pair $(W,Y)$ is co- or
countermonotonic. Another example of $Y$ being $\mathfrak{A}$-measurable is if
$\mathfrak{A}$ is generated by all risk factors, i.e.,
$\mathfrak{A}=\sigma({\boldsymbol{X}})$ and $Y=\mathsf{g}({\boldsymbol{X}})$.
Similar comments about the relevance of information $\mathfrak{A}$ with
respect to only $T(Y)$ can be made corresponding to the ones in Subsection
3.1. In particular, the below example shows that we can obtain a sensitivity
of 1, i.e. we have all relevant information for $T$, even if $Y$ itself cannot
be fully explained by $\mathfrak{A}$.

######  Example 3.6 (Example 3.4 continued)

We consider $X_{1},X_{2},X_{3}$ independent, $X_{1}$ Bernoulli distributed
with $p={\mathbb{P}}(X_{1}=0)=1-{\mathbb{P}}(X_{1}=1)\in(0,\alpha)$, and
$X_{2}<C<X_{3}$ almost surely, for some constant $C>0$, where $X_{3}$ has
finite mean. Then, the output
$Y={\mathds{1}}_{\\{X_{1}=0\\}}X_{2}+{\mathds{1}}_{\\{X_{1}=1\\}}X_{3}$ is not
measurable with respect to $\sigma(X_{1},X_{3})$. However, for the functional
$T(Y)={\mathbb{E}}[Y{\mathds{1}}_{\\{Y>C\\}}]$, we have that
$T(Y|Y)=Y{\mathds{1}}_{\\{Y>C\\}}=X_{3}{\mathds{1}}_{\\{X_{1}=1\\}}=T(Y|X_{1},X_{3})$.
By part c) of Theorem 3.1, for any consistent scoring function $S$ for $T$ we
get that $\xi_{S}(Y;Y)=\xi_{S}(Y;X_{1},X_{3})=1$. Example 3.6 shows that
knowing $(X_{1},X_{3})$ fully explains
$T(Y)={\mathbb{E}}[Y{\mathds{1}}_{\\{Y>C\\}}]$, which implies that
$\xi_{S}(Y;X_{1},X_{3})=1$, however, $Y$ is not measurable with respect to
$\sigma(X_{1},X_{3})$.

###  3.3 Monotonicity with respect to Nested Information

It is difficult to compare arbitrary sets of information. Put mathematically,
there is no canonical total order on the class of all sub-$\sigma$-algebras of
$\mathfrak{F}$. However, the subset relation is a sensible partial order on
this class and Theorem 3.1 (d) asserts that our sensitivities are monotone
with respect to this partial order. Some direct consequences are immediate,
for example in regard to transformations of explanatory variables.

###### Corollary 3.7

Under the assumptions of Theorem 3.1 (d), suppose that $\mathfrak{A}^{\prime}$
is generated by a $d$-dimensional random vector ${\boldsymbol{W}}$. Then for
any measurable function $h\colon{\mathbb{R}}^{d}\to{\mathbb{R}}^{m}$, $m\leq
d$, it holds that
$\xi_{S}\big{(}Y;h({\boldsymbol{W}})\big{)}\leq\xi_{S}(Y;{\boldsymbol{W}})$.
If $h$ is injective, then
$\xi_{S}\big{(}Y;h({\boldsymbol{W}})\big{)}=\xi_{S}(Y;{\boldsymbol{W}})$.

The proof follows by invoking that
$\sigma\big{(}h({\boldsymbol{W}})\big{)}\subseteq\sigma({\boldsymbol{W}})$.
The inclusion becomes an equality if $h$ is an injection.

Corollary 3.7 implies that injective transformations of risk factors do not
affect the sensitivities. Examples include affine transformations or
(component-wise) logarithmic scaling of explanatory variables. If the
transformation is not injective, then the transformation may induce a loss of
information, e.g., a projection to components of ${\boldsymbol{W}}$ (reduction
of dimensionality) implies that
$\xi_{S}(Y;{\boldsymbol{W}}_{\mathcal{I}})\leq\xi_{S}(Y;{\boldsymbol{W}})$,
for any subvector ${\boldsymbol{W}}_{\mathcal{I}}$ of ${\boldsymbol{W}}$.

Again, we stress that monotonicity with respect to nested information sets
only takes relevant information for modelling $T(Y)$ into account. We
illustrate this with the following example.

###### Example 3.8

Let $Y=X_{1}+X_{2}$, where $X_{1},X_{2}$ are independent standard normal. The
target functional $T$ is the mean and and $S(z,y)=(z-y)^{2}$ the squared loss.
Then, we obtain that

$\xi_{S}(Y;X_{1})=\xi_{S}(Y;X_{2})=\frac{1}{2}\,.$

Next, we consider the transformation $h(x)=|x|$, then
$T\left(Y|h(X_{1})\right)={\mathbb{E}}\big{[}Y\big{|}|X_{1}|\big{]}=T\left(Y|h(X_{2})\right)={\mathbb{E}}\big{[}Y\big{|}|X_{2}|\big{]}={\mathbb{E}}[Y]=0$.
Using part (b) of Theorem 3.1, the sensitivity to $|X_{i}|$, $i=1,2$, are

$\xi_{S}(Y;|X_{1}|)=\xi_{S}(Y;|X_{2}|)=0.$

This also constitutes another example for Theorem 3.1 (b), where the
information $|X_{1}|$ is not independent of $Y$ but irrelevant for prediction
purposes.

###  3.4 Interaction Information

Important in sensitivity analysis, particularly for non-additive models, is
the calculation of interaction indices, which quantify the effects that
multiple risk factors have jointly on the functional of interest minus their
individual effects, see e.g. Chapter 4 in Saltelli et al. (2008). In an
information value setting, the relevant question that quantifies _interaction
information_ between two information sets $\mathfrak{A}_{1}$ and
$\mathfrak{A}_{2}$ is: “How much do we learn from knowing both
$\mathfrak{A}_{1},\mathfrak{A}_{2}$ jointly, once we already know
$\mathfrak{A}_{1}$ and $\mathfrak{A}_{2}$ individually”. We formalise this in
the next definition.

######  Definition 3.9 (Interaction Information)

The sensitivity of $Y$ with respect to the interaction information of
$\mathfrak{A}_{1},\mathfrak{A}_{2}$ based on $S$ is given by

$\xi_{S}(Y;\mathfrak{A}_{1}\wedge\mathfrak{A}_{2})=\max\big{\\{}\xi_{S}(Y;\mathfrak{A}_{1}\vee\mathfrak{A}_{2})-\xi_{S}(Y;\mathfrak{A}_{1})-\xi_{S}(Y;\mathfrak{A}_{2})\,,\;0\big{\\}},$

where $\mathfrak{A}_{1}\vee\mathfrak{A}_{2}$ is the smallest $\sigma$-algebra
which contains all elements of $\mathfrak{A}_{1}$ and $\mathfrak{A}_{2}$,
i.e.,
$\mathfrak{A}_{1}\vee\mathfrak{A}_{2}=\sigma(\mathfrak{A}_{1}\cup\mathfrak{A}_{2})$.

###### Remark 3.10

The sensitivity with respect to the interaction information
$\xi_{S}(Y;\mathfrak{A}_{1}\wedge\mathfrak{A}_{2})$ does generally not
coincide with the sensitivity with respect to the intersection of the two
information sets, $\xi_{S}(Y;\mathfrak{A}_{1}\cap\mathfrak{A}_{2})$.222For
example, consider the probability space
$({\mathbb{R}}^{2},\mathfrak{B}({\mathbb{R}}^{2}),\mathcal{N}(0,1)^{\otimes
2})$, where $\mathfrak{B}({\mathbb{R}}^{2})$ is the Borel $\sigma$-algebra on
${\mathbb{R}}^{2}$ and $\mathcal{N}(0,1)^{\otimes 2}$ is the measure of a
standard normal distribution on ${\mathbb{R}}^{2}$. For
$\boldsymbol{t}=(t_{1},t_{2})\in{\mathbb{R}}^{2}$ define the random variables
$X_{1}(\boldsymbol{t})=t_{1}$ and $X_{2}(\boldsymbol{t})=t_{2}$. Then
$X_{1},X_{2}$ are two independent standard normal random variables and
$\sigma(X_{1})\cap\sigma(X_{2})$ is the trivial $\sigma$-algebra
$\\{\emptyset,{\mathbb{R}}^{2}\\}$. If $X_{3}$ is a linear combination of
$X_{1}$ and $X_{2}$ such that $X_{1}$ and $X_{3}$ are not perfectly
correlated, then $\sigma(X_{1})\cap\sigma(X_{3})$ is still trivial.

######  Example 3.11 (Variance-based Interactions)

For variance-based sensitivity measures, that is, when $T$ is the mean and $S$
the squared loss, we recover the Sobol interaction term if the information
sets $\mathfrak{A}_{1}$ and $\mathfrak{A}_{2}$ are independent. Indeed, the
sensitivity of $Y$ with respect to the interaction information of
$\mathfrak{A}_{1}$ and $\mathfrak{A}_{2}$ is

$\xi_{S}(Y;\mathfrak{A}_{1}\wedge\mathfrak{A}_{2})=\frac{{\mathrm{Var}}\left({\mathbb{E}}\left[Y|\mathfrak{A}_{1}\vee\mathfrak{A}_{2}\right]\right)}{{\mathrm{Var}}(Y)}-\frac{{\mathrm{Var}}\left({\mathbb{E}}\left[Y|\mathfrak{A}_{1}\right]\right)}{{\mathrm{Var}}(Y)}-\frac{{\mathrm{Var}}\left({\mathbb{E}}\left[Y|\mathfrak{A}_{2}\right]\right)}{{\mathrm{Var}}(Y)}\;\geq\;0\,.$
(5)

The first summand in (5) is the normalised joint effect of $\mathfrak{A}_{1}$
and $\mathfrak{A}_{2}$, while the second and third term are the normalised
first-order effects to $\mathfrak{A}_{1}$ and $\mathfrak{A}_{2}$,
respectively; see e.g., Saltelli and Tarantola (2002).

While for the special case of Sobol indices the interaction terms are always
non-negative, this need not be the case for general scoring functions and
prediction functionals, as is illustrated in the next proposition and example.

######  Proposition 3.12 (Information is Not Additive)

Sensitivities are generally neither subadditive nor superadditive, in the
sense that the term
$\xi_{S}(Y;\mathfrak{A}_{1}\vee\mathfrak{A}_{2})-\xi_{S}(Y;\mathfrak{A}_{1})-\xi_{S}(Y;\mathfrak{A}_{2})$
can generally take any sign.

Proposition 3.12 follows from the following example.

###### Example 3.13

Let $Y=X_{1}+X_{2}$, where $X_{1},X_{2}$ are jointly normal with means 0,
variances 1 and correlation $\rho\in(-1,1]$. The target functional $T$ is the
mean and we consider the scoring function $S(z,y)=(z-y)^{2}$. Moreover, let
$\mathfrak{A}_{i}=\sigma(X_{i})$, $i=1,2$. In this context
$\xi_{S}(Y;\mathfrak{A}_{1}\vee\mathfrak{A}_{2})=\xi_{S}(Y;X_{1},X_{2})=\xi_{S}(Y;Y)=1$.
On the other hand,

$\xi_{S}(Y;X_{1})=\xi_{S}(Y;X_{2})=\frac{1+\rho}{2}.$

Interestingly, we obtain additivity in Example 3.13 for the case when
$\mathfrak{A}_{1}$ and $\mathfrak{A}_{2}$ are independent, i.e.,
$\xi_{S}(Y;\mathfrak{A}_{1}\vee\mathfrak{A}_{2})=\xi_{S}(Y;\mathfrak{A}_{1})+\xi_{S}(Y;\mathfrak{A}_{2})$
if $\rho=0$. The following modification of Example 3.3 shows, however, that
independence of $\mathfrak{A}_{1}$ and $\mathfrak{A}_{2}$ does not imply
additivity.

###### Example 3.14

Let $X_{1}$, $X_{2}$ be independent random variables with mean 0 and
$Y=X_{1}X_{2}$. Since
${\mathbb{E}}[Y|X_{1}]={\mathbb{E}}[Y|X_{2}]={\mathbb{E}}[Y]=0$, part (b) of
Theorem 3.1 implies that $\xi_{S}(Y;X_{1})=\xi_{S}(Y;X_{2})=0$ for any
consistent score for the mean functional. But clearly
$\xi_{S}(Y;X_{1},X_{2})=1$. The setup of Example 3.14 can be rephrased in that
$Y$ is pairwise mean-independent from $X_{1}$ and $X_{2}$, but not mean-
independent from $(X_{1},X_{2})$.

##  4 Choice of Score-based Sensitivities

So far we did not discuss the choice of (strictly) consistent scoring function
for a functional $T$ to compute score-based sensitivities. This is, however,
an important issue since almost all elicitable functionals possess rich
classes of (strictly) consistent scoring functions each of which could lead to
a different score-based sensitivity measure. In this section, we discuss
potential choices of scoring function and their implication on the
corresponding score-based sensitivities. We first discuss scale-invariant
score-based sensitivity measures which leads to the subclass of homogeneous
scores (Section 4.1). We further introduce score-based Murphy diagrams which
allow for graphical illustrations of score-based sensitivities (Sections 4.2
and 4.3).

Throughout this section, the concepts are illustrated on the mean functional
and the $\alpha$-quantile (or ${\mathrm{VaR}}_{\alpha}$). For this, we first
recall their family of consistent scoring functions in the next proposition,
referring to Gneiting (2011). For an overview of characterisation results of
other important functionals such as entropic risk measures, expectiles, the
mode, the pairs (mean, variance), (${\mathrm{VaR}}_{\alpha}$,
${\mathrm{ES}}_{\alpha}$), and Range Value-at-Risk (RVaR) together with its
VaR-components, we refer to Appendix 7.

######  Proposition 4.1 (Gneiting (2011))

  1. $i)$

Let $\mathcal{M}$ be the class of distributions with finite mean. If $\phi$ is
(strictly) convex with subgradient $\phi^{\prime}$, then

$S_{\phi}(z,y)=\phi(y)-\phi(z)+\phi^{\prime}(z)(z-y)\,,\qquad
z,y\in{\mathbb{R}}\,,$ (6)

is (strictly) $\mathcal{M}$-consistent for the mean, if
$\int|\phi(y)|\,\mathrm{d}F(y)<\infty$ for all $F\in\mathcal{M}$. Moreover, on
the class of compactly supported measures, any (strictly) consistent scoring
function for the mean which is continuously differentiable in its first
argument and which satisfies $S(y,y)=0$ is necessarily of the form (6).

  2. $ii)$

If $g$ is increasing, the score

$S_{g}(z,y)=\big{(}{\mathds{1}}_{\\{y\leq
z\\}}-\alpha\big{)}\big{(}g(z)-g(y)\big{)}\,,\qquad z,y\in{\mathbb{R}}\,,$ (7)

is $\mathcal{M}$-consistent for ${\mathrm{VaR}}_{\alpha}$ if
$\int|g(y)|\,\mathrm{d}F(y)<\infty$ for all $F\in\mathcal{M}$. If $g$ is
strictly increasing and if for all $F\in\mathcal{M}$,
$F({\mathrm{VaR}}_{\alpha}(F)+\epsilon)>\alpha$ for all $\epsilon>0$, then (7)
is strictly $\mathcal{M}$-consistent. Moreover, on the class of compactly
supported measures, any consistent scoring function for
${\mathrm{VaR}}_{\alpha}$ which is continuous in its first argument, which
admits a continuous derivative for all $z\neq y$ and which satisfies
$S(y,y)=0$ is necessarily of the form (7).

Scoring functions of the form (6) are called _Bregman scores_ and the scores
in (7) are termed _generalised piecewise linear scores_. Almost all different
choices of $\phi$ in (6) and $g$ in (7) lead to different score-based
sensitivity measures for the mean and VaR, respectively. Indeed, the next
proposition states the family of score-based sensitivity measures for the mean
and VaR arising from their scoring functions given in Proposition 4.1. To that
end, recall the definition of the Expected Shortfall (ES) of a random variable
$Y$ at level $\alpha\in(0,1)$ (Acerbi 2002)

$\displaystyle{\mathrm{ES}}_{\alpha}(Y)$
$\displaystyle=\frac{1}{1-\alpha}\int_{\alpha}^{1}{\mathrm{VaR}}_{\beta}(Y)\,\mathrm{d}\beta$
(8)
$\displaystyle=\frac{1}{1-\alpha}{\mathbb{E}}[Y{\mathds{1}}_{\\{Y>{\mathrm{VaR}}_{\alpha}(Y)\\}}]+\frac{1}{1-\alpha}{\mathrm{VaR}}_{\alpha}(Y)\big{(}1-\alpha-{\mathbb{P}}(Y>{\mathrm{VaR}}_{\alpha}(Y))\big{)}.$

Since ES is law-determined, we may write ${\mathrm{ES}}_{\alpha}(F_{Y})$
instead of ${\mathrm{ES}}_{\alpha}(Y)$.

###### Proposition 4.2

  1. $i)$

For any strictly convex function $\phi\colon{\mathbb{R}}\to{\mathbb{R}}$ and
any random variable $Y$ such that $\phi(Y)$ is integrable and such that
${\mathbb{E}}\left[\phi(Y)\right]\neq\phi\left({\mathbb{E}}[Y]\right)$, the
score-based sensitivity for $Y$ induced by the Bregman score $S_{\phi}$ (6) is

$\xi_{S_{\phi}}(Y;\mathfrak{A})=\frac{{\mathbb{E}}\left[\phi\left({\mathbb{E}}[Y|\mathfrak{A}]\right)\right]-\phi\left({\mathbb{E}}[Y]\right)}{{\mathbb{E}}\left[\phi(Y)\right]-\phi\left({\mathbb{E}}[Y]\right)}\,.$
(9)

  2. $ii)$

For any strictly increasing function $g\colon{\mathbb{R}}\to{\mathbb{R}}$ and
any random variable $Y$ such that $g(Y)$ is integrable and such that
${\mathrm{ES}}_{\alpha}(g(Y))\neq{\mathbb{E}}[g(Y)]$, the score-based
sensitivity for the $\alpha$-quantile, $\alpha\in(0,1)$, induced by the
generalised piecewise linear score $S_{g}$ (7) is

$\xi_{S_{g}}(Y;\mathfrak{A})=\frac{{\mathrm{ES}}_{\alpha}(g(Y))-{\mathbb{E}}\big{[}{\mathrm{ES}}_{\alpha}(g(Y)|\mathfrak{A})]}{{\mathrm{ES}}_{\alpha}(g(Y))-{\mathbb{E}}[g(Y)]}\,.$
(10)

Proof of Proposition 4.2. First, we prove (9). For this, let
$\mathfrak{A}\subseteq\mathfrak{F}$, then

$\displaystyle{\mathbb{E}}\big{[}S_{\phi}({\mathbb{E}}[Y|\mathfrak{A}],Y)\big{]}$
$\displaystyle={\mathbb{E}}\big{[}\phi(Y)-\phi\left({\mathbb{E}}[Y|\mathfrak{A}]\right)+\phi^{\prime}\left({\mathbb{E}}[Y|\mathfrak{A}]\right)({\mathbb{E}}[Y|\mathfrak{A}]-Y)\big{]}$
$\displaystyle={\mathbb{E}}\big{[}\phi(Y)\big{]}-{\mathbb{E}}\big{[}\phi\left({\mathbb{E}}[Y|\mathfrak{A}]\right)\big{]}+{\mathbb{E}}\big{[}{\mathbb{E}}\big{[}\phi^{\prime}\left({\mathbb{E}}[Y|\mathfrak{A}]\right)({\mathbb{E}}[Y|\mathfrak{A}]-Y)|\mathfrak{A}\big{]}\big{]}$
$\displaystyle={\mathbb{E}}\big{[}\phi(Y)\big{]}-{\mathbb{E}}\big{[}\phi\left({\mathbb{E}}[Y|\mathfrak{A}]\right)\big{]}\,,$

which shows (9). To prove (10), let $\mathfrak{A}\subseteq\mathfrak{F}$, and
note that since $g$ is strictly increasing
${\mathrm{VaR}}_{\alpha}(g(Y)|\mathfrak{A})=g\left({\mathrm{VaR}}_{\alpha}(Y|\mathfrak{A})\right)$.
Next, we use the second identity in (8) to obtain

$\displaystyle{\mathbb{E}}\big{[}S_{g}({\mathrm{VaR}}_{\alpha}(Y),Y)\big{]}$
$\displaystyle={\mathbb{E}}\big{[}\left({\mathds{1}}_{\\{Y\leq{\mathrm{VaR}}_{\alpha}(Y)\\}}-\alpha\right)\left(g\big{(}{\mathrm{VaR}}_{\alpha}(Y)\big{)}-g(Y)\right)\big{]}$
$\displaystyle={\mathbb{E}}\big{[}\left(1-\alpha-{\mathds{1}}_{\\{g(Y)>{\mathrm{VaR}}_{\alpha}(g(Y))\\}}\right)\big{(}{\mathrm{VaR}}_{\alpha}(g(Y))-g(Y)\big{)}\big{]}$
$\displaystyle=\big{(}1-\alpha-{\mathbb{P}}\big{(}g(Y)>{\mathrm{VaR}}_{\alpha}(g(Y))\big{)}\big{)}{\mathrm{VaR}}_{\alpha}(g(Y))$
$\displaystyle\qquad+{\mathbb{E}}\big{[}{\mathds{1}}_{\\{g(Y)>{\mathrm{VaR}}_{\alpha}(g(Y))\\}}g(Y)\big{]}-(1-\alpha){\mathbb{E}}[g(Y)]$
$\displaystyle=(1-\alpha){\mathrm{ES}}_{\alpha}(g(Y))-(1-\alpha){\mathbb{E}}[g(Y)]\,.$

Similarly,

$\displaystyle{\mathbb{E}}\big{[}S_{g}({\mathrm{VaR}}_{\alpha}(Y|\mathfrak{A}),Y)\big{]}$
$\displaystyle={\mathbb{E}}\big{[}{\mathbb{E}}[S_{g}({\mathrm{VaR}}_{\alpha}(Y|\mathfrak{A}),Y)|\mathfrak{A}]\big{]}$
$\displaystyle={\mathbb{E}}\big{[}(1-\alpha){\mathrm{ES}}_{\alpha}(g(Y)|\mathfrak{A})-(1-\alpha){\mathbb{E}}[g(Y)|\mathfrak{A}]\big{]}$
$\displaystyle=(1-\alpha){\mathbb{E}}\big{[}{\mathrm{ES}}_{\alpha}(g(Y)|\mathfrak{A})]-(1-\alpha){\mathbb{E}}[g(Y)]\,.$

Inserting the above into the formula for the sensitivity measures completes
the proof.

Obviously, scaling the scoring function leaves the value of the sensitivity
measure unaffected, i.e., $\xi_{cS}=\xi_{S}$, for any $c>0$. But otherwise,
different choices of scoring functions lead to different sensitivities. In
particular, these different sensitivities may induce different rankings of
information value, see Example 4.15 in Section 4.3 for an illustration.
Proposition 4.2, however, offers some further insight on the ordering of
sensitivity measures for the mean and quantile. Specifically, for the mean
functional, Proposition 4.2 (case i)) states an if and only if condition for
the ordering of sensitivities. Indeed,
$\xi_{S_{\phi}}(Y;\mathfrak{A})\geq\xi_{S_{\phi}}(Y;\mathfrak{A}^{\prime})$,
if and only if,
${\mathbb{E}}\left[\phi\left({\mathbb{E}}[Y|\mathfrak{A}]\right)\right]\geq{\mathbb{E}}\left[\phi\left({\mathbb{E}}[Y|\mathfrak{A}^{\prime}]\right)\right]$
for $\phi$ convex and two (not necessarily nested) information sets
$\mathfrak{A}$, $\mathfrak{A}^{\prime}\subseteq\mathfrak{F}$, see also Theorem
3.1 in Krüger and Ziegel (2021). This condition thus establishes monotonicity
of the sensitivity measure for the mean functional for situations beyond
nested information sets. An Example of interest is $Y=X_{1}+X_{2}$, where
$X_{1},X_{2}$ are independent and normally distributed with mean 0 and
variances $\sigma_{1}^{2},\sigma_{2}^{2}$, respectively. Then it holds for all
convex functions $\phi$ that
$\xi_{S_{\phi}}(Y;X_{1})>\xi_{S_{\phi}}(Y;X_{2})$, if and only if,
$\sigma_{1}^{2}>\sigma_{2}^{2}$, see also Example 3.2. in Krüger and Ziegel
(2021). A similar argument can be made for the quantile, i.e. part ii) of
Proposition 4.2. Indeed the sensitivities for the ${\mathrm{VaR}}_{\alpha}$
are ordered
$\xi_{S_{g}}(Y;\mathfrak{A})\geq\xi_{S_{g}}(Y;\mathfrak{A}^{\prime})$, if and
only if,
${\mathbb{E}}\big{[}{\mathrm{ES}}_{\alpha}(g(Y)|\mathfrak{A})]\leq{\mathbb{E}}\big{[}{\mathrm{ES}}_{\alpha}(g(Y)|\mathfrak{A}^{\prime})]$,
for $g$ increasing and two not necessarily nested information sets
$\mathfrak{A}$, $\mathfrak{A}^{\prime}\subseteq\mathfrak{F}$. For further
discussion, we refer the interested reader to Theorem C.1 in the online
appendix of Krüger and Ziegel (2021).

There are many ways to choose a scoring function. One can use a scoring
function motivated by tradition and interpretability. For the mean functional,
for example, the traditional choice is the squared loss which results in the
Sobol indices. We would like to promote two alternative choices. First, in
Subsection 4.1 we consider scale-invariant sensitivities, i.e., sensitivities
that are unaffected by scaling $Y$, which are induced by the important
subclass of positively homogeneous scores. Second, in Subsection 4.2, we use
Murphy diagrams to illustrate score-based sensitivities _simultaneously_ for
basically the entire class of consistent scoring functions for a functional.
Subsection 4.3 combines the ideas of its two previous subsections and promotes
a way to assess sensitivities simultaneously for all positively homogeneous
scores.

###  4.1 Scale-invariant Sensitivities

Baucells and Borgonovo (2013) introduce and advocate for the use of
sensitivities which are invariant under (strictly) monotone transformations of
the output $Y$. A sensitivity $\xi_{S}$ based on $S$ is invariant under
monotone transformations if for any random variable $Y$ and any information
set $\mathfrak{A}\subseteq\mathfrak{F}$

$\xi_{S}\big{(}u(Y);\mathfrak{A}\big{)}=\xi_{S}(Y;\mathfrak{A})$

for all strictly increasing functions $u\colon{\mathbb{R}}\to{\mathbb{R}}$.
The only score-based sensitivity reported in Table 6 of Borgonovo et al.
(2021) which is invariant under monotone transformations of the output $Y$ is
based on the log-score, $S(f,y)=-\log(f(y))$ where $f$ is a predictive
density. The log-score is a strictly proper scoring rule which is tailored to
evaluate probabilistic predictions (corresponding to the situation where $T$
is the identity map). To the best of our knowledge, we are not aware of any
other score-based sensitivity, in particular also for point predictions, which
is invariant under monotone transformations of the output $Y$. There are,
however, examples of sensitivity measures that, while not score-based
sensitivities, are transformation invariant under monotone transformation of
the output random variables. We refer the interested reader to Plischke and
Borgonovo (2019) and Baucells and Borgonovo (2013).

We suggest to consider a weaker, but still very relevant, invariance
criterion.

######  Definition 4.3 (Scale-invariance)

A sensitivity $\xi_{S}$ based on $S$ is _scale-invariant_ if for any random
variable $Y$ and any information set $\mathfrak{A}\subseteq\mathfrak{F}$

$\xi_{S}(cY;\mathfrak{A})=\xi_{S}(Y;\mathfrak{A})\,\quad\text{for all}\quad
c>0\,.$

A scale-invariant sensitivity measure is a sensitivity measure that takes on
the same value, independent of the units in which $Y$ is reported. We will see
that a score-based sensitivity measure is scale-invariant if the employed
scoring function is homogeneous, which in turn implies that the considered
functional $T$ is homogeneous of degree 1. Thus, to discuss scale-invariant
score-based sensitivity measures, we first define positive homogeneity for
scoring functions.

######  Definition 4.4 (Homogeneity)

Let $D$ be the positive half-line, the negative half-line or the whole
${\mathbb{R}}$. A scoring function $S\colon D\times D\to{\mathbb{R}}$ is
positively homogeneous of degree $b\in{\mathbb{R}}$, if $S(cz,cy)=c^{b}S(z,y)$
for all $z,y\in D$ and for all $c>0$. We call a scoring function positively
homogeneous if there exists a $b\in{\mathbb{R}}$ such that it is positively
homogeneous of degree $b$.

Next, we establish a sufficient condition for an elicitable functional to be
homogeneous of degree 1.

######  Proposition 4.5 (Homogeneous Elicitable Functionals)

Let $D$ be the positive half-line, the negative half-line, or the whole
${\mathbb{R}}$. If there is a positively homogeneous score $S\colon D\times
D\to{\mathbb{R}}$ which is strictly $\mathcal{M}$-consistent for
$T\colon\mathcal{M}\to D$, then $T$ is positively homogeneous of degree 1.

Proof of Proposition 4.5. The assertion is a special instance of Lemma 4.3 in
Fissler and Ziegel (2019). For completeness, we provide a short proof. Let
$S\colon D\times D\to{\mathbb{R}}$ be strictly $\mathcal{M}$-consistent for
$T$ and positively homogeneous of degree $b\in{\mathbb{R}}$. Then, for all $Y$
with $F_{Y}\in\mathcal{M}$ and for all $c>0$, it holds that if $z\in D$, then
setting $z^{\prime}c=z$ implies that $z^{\prime}\in D$, and we obtain that

$\displaystyle T(cY)$ $\displaystyle=\argmin_{z\in
D}{\mathbb{E}}\left[S(z,cY)\right]=c\,\argmin_{z^{\prime}\in
D}{\mathbb{E}}\left[S(cz^{\prime},cY)\right]=c\,\argmin_{z\in
D}\;c^{b}\;{\mathbb{E}}\left[S(z,Y)\right]=c\,T(Y)\,.$

Many functionals of interests such as monetary risk measures, which include
the entropic risk measures, VaR, ES, and RVaR, are by definition homogeneous
of degree 1 (scale-_equivariant_), i.e., $T(cY)=cT(Y)$ for all $c>0$.
Moreover, these elicitable functional possess homogeneous scores. Note that
positive homogeneity of $T$ implies that
$T(cY|\mathfrak{A})=cT(Y|\mathfrak{A})$ for any information set
$\mathfrak{A}$. The next proposition provides a sufficient condition for a
score-based sensitivity to be scale-invariant.

###### Proposition 4.6

Let $D$ be the positive half-line, the negative half-line, or the whole
${\mathbb{R}}$. If there is a positively homogeneous score $S\colon D\times
D\to{\mathbb{R}}$ which is strictly $\mathcal{M}$-consistent for
$T\colon\mathcal{M}\to D$, then, the score-based sensitivity $\xi_{S}$ is
scale-invariant.

The proof of Proposition 4.6 is a direct consequence of Proposition 4.5 and
that $T(cY|\mathfrak{A})=cT(Y|\mathfrak{A})$, for all $c>0$ and
$\mathfrak{A}\subseteq\mathfrak{F}$.

The above proposition implies that if scale-invariance is a desired property a
homogeneous score should be chosen for the score-based sensitivity measures.
Many elicitable and positively homogeneous functionals admit positively
homogeneous scoring functions, which enables to build scale-invariant score-
based sensitivities. We recall the family of homogeneous scoring functions for
the mean and VaR functional from the literature.

######  Proposition 4.7 (Homogeneous Scores for the Mean (Nolde and Ziegel
2017, Patton 2011))

The class of strictly consistent and $b$-homogeneous scores for the mean
satisfying $S(y,y)=0$ are given by any positive multiple of a member of the
Patton family

$S_{b}(z,y)=\begin{cases}\frac{y^{b}-z^{b}}{b(b-1)}-\frac{z^{b-1}}{b-1}(y-z),&b\in{\mathbb{R}}\setminus\\{0,1\\},\\\\[5.0pt]
\frac{y}{z}-\log\left(\frac{y}{z}\right)-1,&b=0,\\\\[5.0pt]
y\log\left(\frac{y}{z}\right)-(y-z),&b=1,\end{cases}$ (11)

where we require that $z,y>0$.

Note that homogeneous scores of a given degree are unique up to positive
scaling and that scaling a score does not affect the sensitivity. The
positively homogeneous scores in (11) arise from (6) upon choosing the
strictly convex functions $\phi_{b}(y)=y^{b}/(b(b-1))$ for
$b\in{\mathbb{R}}\setminus\\{0,1\\}$, $\phi_{0}(y)=-\log(y)$ for $b=0$, and
$\phi_{1}(y)=-y\log(y)$ for $b=1$. Note that in (11) we require $z,y$ to be
strictly positive. There are no strictly consistent and $b$-homogeneous scores
for the mean on ${\mathbb{R}}\times{\mathbb{R}}$, since there are no convex
and $b$-homogeneous functions on ${\mathbb{R}}$ for $b\leq 1$. For $b>1$,
however, one may choose

$\phi_{b}(y)=d_{1}y^{b}{\mathds{1}}_{{\\{y>0\\}}}+d_{2}|y|^{b}{\mathds{1}}_{{\\{y<0\\}}},\qquad
y\in{\mathbb{R}}$

for positive constants $d_{1},d_{2}>0$. We refer to the supplement of Nolde
and Ziegel (2017) which contains the characterisation results of positively
homogeneous and consistent scoring functions for the $\tau$-expectile; for
which the mean arises for $\tau=1/2$. Moreover, this supplement contains
similar characterisation results for VaR and the pair consisting of VaR and
ES. We state the former.

######  Proposition 4.8 (Homogeneous Scores for VaR (Nolde and Ziegel 2017))

The class of strictly consistent and $b$-homogeneous scores for
${\mathrm{VaR}}_{\alpha}$ satisfying $S(y,y)=0$ are given by any positive
multiple of

$S_{b}(z,y)=\left({\mathds{1}}_{\\{y\leq
z\\}}-\alpha\right)\big{(}g(z)-g(y)\big{)}\,,$

where

$g(y)=\begin{cases}y^{b}{\mathds{1}}_{\\{y>0\\}}-d|y|^{b}{\mathds{1}}_{\\{y<0\\}}\qquad\qquad&\text{if}\quad
b>0\text{ and }y\in{\mathbb{R}},\\\\[5.0pt] \log(y)&\text{if}\quad b=0\text{
and }y>0,\\\\[5.0pt] -y^{b}&\text{if}\quad b<0\text{ and }y>0\,,\end{cases}$

for a positive constant $d>0$.

Again, if $y>0$, homogeneous scores of a certain degree are unique up to
positive scaling. We could also consider transformations other than scaling,
such as e.g., translations. Then, if the score $S$ is invariant and the
functional $T$ is equivariant for that transformation, we directly retrieve
the corresponding invariance of the score-based sensitivity for that
transformation. If we consider transformations with respect to which the
functional $T$ of interest is not equivariant or the score $S$ is not
invariant, we do not obtain an invariance property of the resulting
sensitivity measure $\xi_{S}$. In case of the mean and the coefficient of
determination, R2, this fact is very well known and described in standard
Econometrics textbooks.333E.g. Wooldridge (2013, p. 194) writes “…it is _not_
legitimate to compare $R$-squareds from models where $y$ is the dependent
variable in one case and $\log(y)$ is the dependent variable in the other.
These measures explain variations in different variables.” (The emphasis in
the quote is original.) We further illustrate this in the following example.

###### Example 4.9

Let $Y=\exp(X_{1}+X_{2})$, where $X_{1}$ $X_{2}$ are independent and standard
normally distributed. Consider the mean functional with the squared loss
score. Then the sensitivity of $\log(Y)$ with respect to $X_{1}$ is

$\xi_{S}(\log(Y);X_{1})=\frac{{\mathrm{Var}}({\mathbb{E}}[\log(Y)|X_{1}])}{{\mathrm{Var}}(\log(Y))}=\frac{{\mathrm{Var}}(X_{1}+{\mathbb{E}}[X_{2}])}{{\mathrm{Var}}(\log(Y))}=\frac{1}{2}\,.$

In contrast, the sensitivity of $Y$ with respect to $X_{1}$ based on the
squared loss is

$\xi_{S}(Y;X_{1})=\frac{{\mathrm{Var}}\left({\mathbb{E}}\left[Y|X_{1}\right]\right)}{{\mathrm{Var}}\left(Y\right)}=\frac{{\mathrm{Var}}\left(e^{X_{1}}{\mathbb{E}}\left[e^{X_{2}}\right]\right)}{{\mathrm{Var}}\left(Y\right)}=\frac{1}{e+1}\approx
0.27\,.$

We emphasise that for a function $h$, the sensitivity of $h(Y)$ with respect
to $X_{1}$ is in general not equal to the sensitivity of $Y$ with respect to
$X_{1}$ if $h$ is not affine.

###  4.2 Murphy Diagrams Based on Elementary Scores

In contrast to Subsection 4.1, where we discuss the choice of score-based
sensitivity measure by imposing additional restrictions on the scoring
functions, here, we pursue a different strategy. Since different choices of
scoring functions may lead to different rankings of information in terms of
score-based sensitivities, we suggest to _simultaneously_ consider (ideally)
_all_ sensitivity measures that arise by (strictly) consistent scoring
functions. While there are many characterisation results for consistent
scoring functions available in the spirit of Proposition 4.1, these classes
are typically indexed by an infinite dimensional parameter space, e.g., the
space of convex functions $\phi$ for the mean or the space of increasing
functions $g$ for the quantile. This fact renders a computation and comparison
of all score-based sensitivities practically infeasible at first glance.

Ehm et al. (2016) establish so-called _mixture representations_ of classes of
all consistent scoring functions for the mean444And more generally also the
$\tau$-expectile. and the $\alpha$-quantile; subject to mild regularity
conditions. That is, they show that $S$ is a consistent scoring function for
the mean (or the $\alpha$-quantile) if and only if there is a non-negative and
$\sigma$-finite measure $H$ on ${\mathbb{R}}$ such that

$S(z,y)=\int_{{\mathbb{R}}}S_{\theta}(z,y)\,\mathrm{d}H(\theta),$ (12)

where the class $\\{S_{\theta},\,\theta\in{\mathbb{R}}\\}$ consists of co-
called _elementary scores_ , which are consistent scores for the mean (or the
$\alpha$-quantile). Moreover, the measure $H$ is uniquely determined by $S$.
We recall the formal statements.

######  Proposition 4.10 (Elementary Scores for the Mean and VaR (Ehm et al.
2016))

  1. $i)$

Let $S_{\phi}$ be a Bregman score (6) such that the subgradient
$\phi^{\prime}$ is left-continuous. Then

$S_{\phi}(z,y)=\int_{\mathbb{R}}S_{\theta}(z,y)\,\mathrm{d}\phi^{\prime}(\theta)\,,$

where for $\theta\in{\mathbb{R}}$, the _elementary scores_ for the mean
functional are given by

$S_{\theta}(z,y)=\begin{cases}|y-\theta|,&\text{if
}\theta\in\big{[}\min(z,y),\max(z,y)\big{)}\\\
0,&\text{otherwise}.\end{cases}$ (13)

  2. $ii)$

Let $S_{g}^{\alpha}$ be a generalised piecewise linear score (7) such that $g$
is left-continuous. Then

$S_{g}^{\alpha}(z,y)=\int_{\mathbb{R}}S^{\alpha}_{\theta}(z,y)\,\mathrm{d}g(\theta)\,,$

where for $\theta\in{\mathbb{R}}$, the _elementary scores_ for the
${\mathrm{VaR}}_{\alpha}$ are given by

$S^{\alpha}_{\theta}(z,y)=\begin{cases}1-\alpha,\qquad&\text{if}\quad\theta\in[y,z)\\\
\alpha,&\text{if}\quad\theta\in[z,y)\\\ 0,&\text{otherwise}.\end{cases}$

These mixture representations of Proposition 4.10 open the door to assess
prediction dominance with respect to almost any Bregman score or generalised
piecewise linear score. I.e., for any (possibly random) forecasts
$Z_{1},Z_{2}$, we obtain that
${\mathbb{E}}\big{[}S_{\phi}(Z_{1},Y)\big{]}\leq{\mathbb{E}}\big{[}S_{\phi}(Z_{2},Y)\big{]}$
for all Bregman scores (6) with left-continuous subgradient $\phi^{\prime}$,
if and only if,
${\mathbb{E}}\big{[}S_{\theta}(Z_{1},Y)\big{]}\leq{\mathbb{E}}\big{[}S_{\theta}(Z_{2},Y)\big{]}$
for all elementary scores $S_{\theta}$ in (13). Ehm et al. (2016) demonstrate
that this equivalence can be used to establish forecast dominance by
inspecting the so called Murphy diagrams.

######  Definition 4.11 (Murphy Diagrams; Ehm et al. (2016))

Let $\\{S_{\theta},\ \theta\in{\mathbb{R}}\\}$ be a class of scoring functions
indexed by $\theta\in{\mathbb{R}}$. For an observation $Y$ and two (possibly
random) forecasts $Z_{1},Z_{2}$, the Murphy diagram of the score difference
with respect to $\\{S_{\theta},\ \theta\in{\mathbb{R}}\\}$ is the map

${\mathbb{R}}\ni\theta\;\mapsto\;{\mathbb{E}}\big{[}S_{\theta}(Z_{1},Y)\big{]}-{\mathbb{E}}\big{[}S_{\theta}(Z_{2},Y)\big{]}\,.$

Clearly, in applications one can approximate the Murphy diagram with the
corresponding empirical counterparts of the expectations. The following result
shows that we can apply a similar rationale in our context of score-based
sensitivities and therefore introduce Murphy diagrams for score-based
sensitivity measures.

######  Proposition 4.12 (Ordering of Score-based Sensitivities)

Suppose $\mathcal{S}$ is a class of consistent scoring functions for a
functional $T$ and $\\{S_{\theta},\,\theta\in{\mathbb{R}}\\}$ a subclass of
$\mathcal{S}$ such that for any $S\in\mathcal{S}$ there exists a non-negative
$\sigma$-finite measure $H$ on ${\mathbb{R}}$ such that

$S(z,y)=\int_{{\mathbb{R}}}S_{\theta}(z,y)\,\mathrm{d}H(\theta).$ (14)

Then, for any response $Y$ and any information sets $\mathfrak{A}_{1}$ and
$\mathfrak{A}_{2}$, the following are equivalent:

  1. $i)$

$\xi_{S}(Y;\mathfrak{A}_{1})\leq\xi_{S}(Y;\mathfrak{A}_{2})\quad\text{for all
}S\in\mathcal{S}$;

  2. $ii)$

$\xi_{S_{\theta}}(Y;\mathfrak{A}_{1})\leq\xi_{S_{\theta}}(Y;\mathfrak{A}_{2})\quad\text{for
all }\theta\in{\mathbb{R}}$.

Proof of Proposition 4.12. The implication “$i)$ $\Rightarrow$ $ii)$” is
obvious since by assumption $S_{\theta}\in\mathcal{S}$ for all
$\theta\in{\mathbb{R}}$. For the other direction, suppose that $ii)$ holds.
Then, for all $\theta\in{\mathbb{R}}$ it holds that
$\xi_{S_{\theta}}(Y;\mathfrak{A}_{1})\leq\xi_{S_{\theta}}(Y;\mathfrak{A}_{2})$
if and only if
${\mathbb{E}}[S_{\theta}(T(Y|\mathfrak{A}_{1}),Y)]-{\mathbb{E}}[S_{\theta}(T(Y|\mathfrak{A}_{2}),Y)]\geq
0$. Hence, $ii)$ implies that for any $a,b>0$ and any
$\theta_{1},\theta_{2}\in{\mathbb{R}}$

${\mathbb{E}}[aS_{\theta_{1}}(T(Y|\mathfrak{A}_{1}),Y)+bS_{\theta_{2}}(T(Y|\mathfrak{A}_{1}),Y)]-{\mathbb{E}}[aS_{\theta_{1}}(T(Y|\mathfrak{A}_{2}),Y)+bS_{\theta_{2}}(T(Y|\mathfrak{A}_{2}),Y)]\geq
0.$ (15)

(15) in turn is equivalent to
$\xi_{aS_{\theta_{1}}+bS_{\theta_{2}}}(Y;\mathfrak{A}_{1})\leq\xi_{aS_{\theta_{1}}+bS_{\theta_{2}}}(Y;\mathfrak{A}_{2})$.
Finally, (15), the construction of the integral (14), and Fubini’s Theorem
imply $i)$.

Proposition 4.12 implies that if we want to check whether information
$\mathfrak{A}_{2}$ is more important for modelling $Y$ than $\mathfrak{A}_{1}$
with respect to all score-based sensitivities, it suffice to establish this
ranking with respect to all elementary score-based sensitivities. This
motivates to consider the following Murphy diagrams for score-based
sensitivities.

######  Definition 4.13 (Murphy Diagrams for Sensitivities)

Let $\\{S_{\theta},\ \theta\in{\mathbb{R}}\\}$ be a class of scoring functions
indexed by $\theta\in{\mathbb{R}}$. The sensitivity of $Y$ to $\mathfrak{A}$
based on $\\{S_{\theta},\ \theta\in{\mathbb{R}}\\}$ is given by the Murphy
diagram for sensitivities

${\mathbb{R}}\ni\theta\;\mapsto\;\xi_{S_{\theta}}(Y;\mathfrak{A})\,.$

######  Example 4.14 (Example 3.4 Continued)

Let $X_{1},X_{2},X_{3}$ be independent, $X_{1}$ Bernoulli distributed with
$p={\mathbb{P}}(X_{1}=0)=1-{\mathbb{P}}(X_{1}=1)\in(0,\alpha)$, and
$X_{2}<C<X_{3}$ almost surely, for $C>0$, and consider the output
$Y={\mathds{1}}_{\\{X_{1}=0\\}}X_{2}+{\mathds{1}}_{\\{X_{1}=1\\}}X_{3}$. We
further choose $p=0.8$, $C=10$, $X_{2}$ uniformly distributed on $[0,C]$, and
$X_{3}=C+Z$, where $Z$ has a Gamma distribution with mean 20 and variance 10.

$T$ | ${\mathbb{E}}$ | ${\mathrm{VaR}}_{\alpha}$
---|---|---
$T(Y)$ | $p{\mathbb{E}}[X_{2}]+(1-p){\mathbb{E}}[X_{3}]$ | ${\mathrm{VaR}}_{(\alpha-p)/(1-p)}(X_{3})$
$T(Y|X_{1})$ | ${\mathds{1}}_{\\{X_{1}=0\\}}{\mathbb{E}}[X_{2}]+{\mathds{1}}_{\\{X_{1}=1\\}}{\mathbb{E}}[X_{3}]$ | ${\mathds{1}}_{\\{X_{1}=0\\}}{\mathrm{VaR}}_{\alpha}(X_{2})+{\mathds{1}}_{\\{X_{1}=1\\}}{\mathrm{VaR}}_{\alpha}(X_{3})$
$T(Y|X_{2})$ | $pX_{2}+(1-p){\mathbb{E}}[X_{3}]$ | ${\mathrm{VaR}}_{(\alpha-p)/(1-p)}(X_{3})$
$T(Y|X_{3})$ | $p{\mathbb{E}}[X_{2}]+(1-p)X_{3}$ | $X_{3}$
$T(Y|X_{1},X_{2})$ | ${\mathds{1}}_{\\{X_{1}=0\\}}X_{2}+{\mathds{1}}_{\\{X_{1}=1\\}}{\mathbb{E}}[X_{3}]$ | ${\mathds{1}}_{\\{X_{1}=0\\}}X_{2}+{\mathds{1}}_{\\{X_{1}=1\\}}{\mathrm{VaR}}_{\alpha}(X_{3})$
$T(Y|X_{1},X_{3})$ | ${\mathds{1}}_{\\{X_{1}=0\\}}{\mathbb{E}}[X_{2}]+{\mathds{1}}_{\\{X_{1}=1\\}}X_{3}$ | ${\mathds{1}}_{\\{X_{1}=0\\}}{\mathrm{VaR}}_{\alpha}(X_{2})+{\mathds{1}}_{\\{X_{1}=1\\}}X_{3}$
$T(Y|X_{2},X_{3})$ | $pX_{2}+(1-p)X_{3}$ | $X_{3}$
Table 1: Functional and conditional functional $T$ of Example 4.14.

Table 1 contains the conditional functionals for one and two risk factors for
the mean functional and the ${\mathrm{VaR}}_{\alpha}$ used for calculating the
score-based sensitivities. Figure 1 displays the corresponding Murphy diagrams
of the elementary score-based sensitivities for the mean functional and the
${\mathrm{VaR}}_{0.9}$. All plots are based on $10^{6}$ simulations. We
observe that the sensitivities of the mean to one risk factor are ordered,
that is
$\xi_{S_{\theta}}(Y;X_{1})>\xi_{S_{\theta}}(Y;X_{2})>\xi_{S_{\theta}}(Y;X_{3})$
for all $\theta$, see top left panel of Figure 1. This implies that the
sensitivities to one risk factor are ordered for all strictly consistent
scoring functions. This is in contrast to the sensitivity for
${\mathrm{VaR}}_{0.9}$. Comparing the mean functional with the
${\mathrm{VaR}}_{0.9}$, we observe that $X_{1}$ has a large sensitivity for
both functionals, the sensitivity to $X_{2}$ is larger for the mean compared
for the ${\mathrm{VaR}}_{0.9}$, and the sensitivity to $X_{3}$ is larger for
the ${\mathrm{VaR}}_{0.9}$ compared for the mean. This reflects that $X_{2}$
influences the mean of $Y$ while $X_{3}$ influences the tail, hence the
${\mathrm{VaR}}_{0.9}$ of $Y$.

Figure 1: Elementary score-based sensitivities. Top panels display the mean
functional and bottom panels the ${\mathrm{VaR}}_{0.9}$. Elementary score-
based sensitivities to a single risk factor (left panel), to two risk factors
(middle panel), and interaction sensitivities (right panels).

###  4.3 Murphy Diagrams for Homogeneous Scores

In Subsection 4.1 we have seen that a positively 1-homogeneous functional $T$
combined with a positively homogeneous scoring function renders the
corresponding score-based sensitivity measure scale-invariant. Moreover, known
characterisation results for positively homogeneous scores (Propositions 4.7,
4.8) state that, at least when $Y$ is positive, the $b$-homogeneous and
strictly consistent scoring functions for the mean and the VaR are unique up
to scaling. Since scaling the scores leaves the sensitivities unaffected, this
means that for $Y>0$ and for the mean and the VaR, there is only a one-
dimensional family of scores – the $b$-homogeneous ones – which render the
sensitivity scale-invariant.

We propose to evaluate all of them jointly, making use of the Murphy diagram
introduced in Definition 4.13. The difference to the Murphy diagrams for
elementary scores is that here the diagram is considered with respect to the
parameter $b$, indicating the degree of homogeneity of the scoring function.
That is

$b\;\mapsto\;\xi_{S_{b}}(Y;\mathfrak{A}).$

We illustrate the homogeneous score-based Murphy diagrams for the mean and
${\mathrm{VaR}}_{\alpha}$ functional in the next example. For comparison with
the Murphy diagrams for elementary scoring function, we illustrate the
homogeneous score-based Murphy diagrams on the same Example 4.14.

######  Example 4.15 (Example 4.14 Continued)

Figure 2 displays the Murphy diagrams of the homogeneous score-based
sensitivities. Comparing with the elementary score-based Murphy diagrams in
Figure 1, we observe a similar picture in that $X_{1}$ has a large sensitivity
for both functionals, the sensitivity to $X_{2}$ is zero for the
${\mathrm{VaR}}_{0.9}$, and the sensitivity to $X_{3}$ is zero for the mean
but large for the ${\mathrm{VaR}}_{0.9}$. The interaction sensitivities, right
panels of Figure 2, show that the interaction between $X_{1}$ and $X_{3}$ are
non-negligible for both functionals. This is informative, as the sensitivity
to $X_{3}$ for the mean is equal to zero for all choices of homogeneous
scoring functions. We obtain the Socol indices for $b=2$.

The sensitivities for the mean to one risk factor (top left panel in Figure 2)
are ordered:
$\xi_{S_{b}}(Y;X_{1})\geq\xi_{S_{b}}(Y;X_{2})\geq\xi_{S_{b}}(Y;X_{3})$. This
is in contrast to the sensitivities for the ${\mathrm{VaR}}_{0.9}$ based on
the homogeneous scores to one risk factor (bottom left panel in Figure 2).
Indeed, for $b=0$ the sensitivities of ${\mathrm{VaR}}_{0.9}$ to $X_{1}$,
$X_{2}$, and $X_{3}$ are respectively $0.45$, $0$, and $0.19$. For $b=4$, the
sensitivities of ${\mathrm{VaR}}_{0.9}$ to $X_{1}$ is equal to 0.32, the
sensitivities of ${\mathrm{VaR}}_{0.9}$ to $X_{2}$ is $0$, and to $X_{3}$ is
equal to $0.50$. Thus, different choices of strictly consistent scoring
function for ${\mathrm{VaR}}$ can lead to different rankings of risk factors.

Figure 2: Homogeneous score-based sensitivities. Top panels display the mean
functional and bottom panels the ${\mathrm{VaR}}_{0.9}$. The lines correspond
to homogeneous score-based sensitivities to a single risk factor (left panel),
to two risk factors (middle panel), and to interaction sensitivities (right
panels).

Finally, we point out that the mixture representations of Proposition 4.10
hold for the corresponding elementary scores, but not for the positively
homogeneous scores of Propositions 4.7 and 4.8. Hence, while it is the case
that an ordering of the sensitivities with respect to all elementary scores
implies an ordering with respect to all positively homogeneous scores, the
reverse does not hold.

##  5 Applications

In this section we illustrate the score-based sensitivities on the well-known
Ishigami–Homma test function in sensitivity analysis and a non-linear
insurance portfolio.

The main challenge when calculating score-based sensitivities is the
estimation of the conditional functionals $T(Y|X_{i})$, for a risk factor
$X_{i}$. For the Ishigami–Homma test function the conditional mean
functionals, i.e., the conditional expectations, are available in closed form
for all risk factors of interest. Thus, estimating Murphy diagrams for
elementary and homogeneous score-based sensitivities may be conducted
straightforwardly using Monte Carlo approximations of the expectations. In the
non-linear insurance portfolio, however, closed form conditional functionals
are not available. Thus, we use neural nets to estimate the conditional VaR
and ES.

###  5.1 The Ishigami–Homma Test Function

In this section we consider the Ishigami–Homma function given by (Ishigami and
Homma 1990)

$Y=\sin(X_{1})+a_{1}\sin(X_{2})^{2}+a_{2}X_{3}^{4}\sin(X_{1})\,,$

where $X_{1},X_{2},X_{3}$ are independent uniform random variables on
$[-\pi,\pi]$. We consider the mean functional for comparison with the
literature.

Figure 3: Top panels: Elementary (top panels) and homogeneous (bottom panels)
score-based sensitivities for the mean functional and the Ishigami–Homma
function with $a_{1}=1$ and $a_{2}=2$ . Left: sensitivities to a single risk
factor. Middle: sensitivity to two risk facors. Right: interaction
sensitivities. The sensitivity to $X_{2}$, i.e., $\xi_{S_{\theta}}(Y;X_{2})$
and $\xi_{S_{b}}(Y;X_{2})$, are numerically equal to the blue line, that is 0.

Figure 3 displays the score-based Murphy diagrams for the mean functional and
for the choices $a_{1}=1$ and $a_{2}=2$. The top panels correspond to the
Murphy diagrams using the elementary scoring functions and the bottom panels
correspond to homogeneous scoring functions. Both the homogeneous and
elementary Murphy diagrams describe a similar pattern. Indeed, we observe that
risk factor $X_{1}$ is most influential and $X_{2}$ and $X_{3}$ are non-
influential with a sensitivity for all elementary and homogeneous scores close
to zero. The interaction term of $X_{1}$ and $X_{3}$, however, is equal to 1
for all scoring functions, reflecting the findings in Saltelli et al. (2008).
Note that the sensitivity for the homogeneous score with $b=2$ are equal to
the Sobol indices. We obtain that the Sobol indices of $Y$ to $X_{1}$,
$\xi_{S_{2}}(Y;X_{1})$, is equal to $0.37$. Similarly,
$\xi_{S_{2}}(Y;X_{2})=5.7\times 10^{-5}$ and $\xi_{S_{2}}(Y;X_{3})=0$. We
refer to Saltelli et al. (2008), Equation (4.34), for the analytical
derivation of the Sobol indices of the Ishigami–Homma function, and to Pianosi
and Wagener (2015) and Baroni and Francke (2020) for a recent discussion on
the Ishigami–Homma function.

Next, we consider the Ishigami–Homma function with parameters $a_{1}=7$ and
$a_{2}=0.1$ as in Marrel et al. (2009). The elementary and homogeneous score-
based sensitivities for the mean functional are displayed in Figure 4. We
observe that for these choices of $a_{1}$ and $a_{2}$, the sensitivity to
$X_{2}$ is larger than the sensitivity to $X_{1}$ for all strictly consistent
scoring functions, see left panels in Figure 4. The estimated Sobol indices
for $X_{1}$, $X_{2}$, and $X_{3}$ are $0.31$, $0.44$, and $0$, respectively.
The fact that the sensitivity to $X_{2}$ is larger than that to $X_{1}$ is in
contrast to the choice $a_{1}=1$ and $a_{2}=2$, for which the sensitivity to
$X_{2}$ is negligible. The sensitivity to $X_{3}$ is zero for both sets of
parameters $a_{1}$ and $a_{2}$. For $a_{1}=7$ and $a_{2}=0.1$, the sensitivity
to knowing two components, i.e. $(X_{1},X_{2})$, $(X_{1},X_{3})$, or
$(X_{2},X_{3})$, strongly depends on the choice of the scoring function, as
depicted in the right panels of Figure 4.

Figure 4: Top panels: Elementary (top panels) and homogeneous (bottom panels)
score-based sensitivities for the mean functional and the Ishigami–Homma
function with $a_{1}=7$ and $a_{2}=0.1$. Left: sensitivities to a single risk
factor. Middle: sensitivity to two risk factors. Right: interaction
sensitivities.

It is well-known that for the Sobol indices, the interaction of $X_{1}$ and
$X_{3}$ is non-zero, see Figures 3 and 4 bottom right panel. Interestingly,
for the choice $a_{1}=1$ and $a_{2}=2$, this is the only interaction which is
non-zero for all elementary scoring functions, see Figure 3. For homogeneous
scores, however, the interaction between $X_{1}$ and $X_{2}$ becomes positive
for large and small values of $b$. Thus, considering Murphy diagrams reveals a
more wholesome picture on the individual sensitivities and their interactions.

###  5.2 Non-linear Insurance Portfolio

In this section we consider an insurance company with three different lines of
business whose losses are $X_{1},X_{2}$, and $X_{3}$ that are subject to a
multiplicative factor $X_{4}$, e.g., inflation. The insurance company has a
reinsurance contract on the first two lines of business,
$L=X_{4}(X_{1}+X_{2})$, with deductible $d$ and limit $l$. Thus, the insurance
company’s total loss is given by

$Y=L-\min\\{(L-d)_{+},\;l\\}+X_{3}X_{4}\,,$

where $(x)_{+}=\max\\{x,0\\}$ denotes the positive part. The distributional
assumptions are presented in Table 2 and we set the deductible to $d=380$ and
the limit to $l=30$. Furthermore, we assume the factors
$(X_{1},X_{2},X_{3},X_{4})$ are dependent through a Gaussian copula with
correlation matrix

$R=\left(\quad\begin{array}[]{@{} S[table-format=1.3]S[table-
format=1.3]S[table-format=1.3]S[table-format=-1.2] @{}}$1$&$0.3$&$0$&$0.8$\\\
$0.3$&$1$&$0$&$0$\\\ $0$&$0$&$1$&$0$\\\ $0.8$&$0$&$0$&$1$\\\
\end{array}\quad\right)\,.$

Table 2: Distributional assumptions of risk factors of the non-linear insurance portfolio example. Risk factor | Distribution | Mean | Std
---|---|---|---
$X_{1}$ | Log-Normal$\,\left(4.98,~{}0.23^{2}\right)$ | 150 | 35.0
$X_{2}$ | Log-Normal$\,\left(4.98,~{}0.23^{2}\right)$ | 150 | 35.0
$X_{3}$ | Gamma$\,\left(100,~{}1\right)$ | 100 | 10.0
$X_{4}$ | Log-Normal$\,\left(-0.005,~{}0.1^{2}\right)$ | 1 | 0.1

First, we calculate the score-based sensitivity for the
${\mathrm{VaR}}_{\alpha}$ at level $\alpha\in(0,1)$. For simplicity, we choose
the pinball loss

$S_{\mathrm{VaR}}(z,y)=\big{(}{\mathds{1}}_{\\{y\leq
z\\}}-\alpha\big{)}\big{(}z-y\big{)}\,,\quad y,z\in{\mathbb{R}}\,,$ (16)

obtained with $g(x)=x$ in (7).

We calculate the score-based sensitivities to risk factors
${\boldsymbol{X}}_{\mathcal{I}}$, where $\mathcal{I}\subset\\{1,2,3,4\\}$ and
$|\mathcal{I}|\in\\{1,2\\}$ is an index set of cardinality one or two such
that ${\boldsymbol{X}}_{\mathcal{I}}$ is an at most two-dimensional subvector
of $(X_{1},\ldots,X_{4})$. For this, we first calculate the conditional
${\mathrm{VaR}}_{\alpha}$ of the aggregate portfolio loss $Y$ given
${\boldsymbol{X}}_{\mathcal{I}}$, i.e.,
$T(Y|{\boldsymbol{X}}_{\mathcal{I}})={\mathrm{VaR}}_{\alpha}(Y|{\boldsymbol{X}}_{\mathcal{I}})$.
Denote by
$q_{\mathcal{I}}({\boldsymbol{x}}_{\mathcal{I}})={\mathrm{VaR}}_{\alpha}(Y|{\boldsymbol{X}}_{\mathcal{I}}={\boldsymbol{x}}_{\mathcal{I}})$,
${\boldsymbol{x}}_{\mathcal{I}}\in{\mathbb{R}}^{|\mathcal{I}|}$, the
conditional quantile of $Y$ given
${\boldsymbol{X}}_{\mathcal{I}}={\boldsymbol{x}}_{\mathcal{I}}$ viewed, for
fixed $\alpha$, as a function of
${\boldsymbol{x}}_{\mathcal{I}}\in{\mathbb{R}}^{|\mathcal{I}|}$. For each
choice of $\mathcal{I}$, we approximate the function $q_{\mathcal{I}}$ via a
neural network (NN) denoted by
$G_{\vartheta_{\mathcal{I}}}\colon{\mathbb{R}}^{|\mathcal{I}|}\to{\mathbb{R}}$,
where $\vartheta_{\mathcal{I}}$ are the network parameters. For relevant
literature on estimating conditional VaR via NN – albeit in a different
context – we refer the reader to Fissler et al. (2021b). The network
parameters $\vartheta_{\mathcal{I}}$ are learnt by iteratively minimising the
expected score (16)

$\widehat{\vartheta}_{\mathcal{I}}=\argmin_{\vartheta}\;\frac{1}{N}\sum_{k=1}^{N}S_{\mathrm{VaR}}\left(G_{\vartheta}({\boldsymbol{x}}_{\mathcal{I}}^{(k)}),y^{(k)}\right)\,,$
(17)

over independently simulated mini-batches
$({\boldsymbol{x}}_{\mathcal{I}}^{(k)},y^{(k)})$, $k=1,\ldots,N$. That is, for
each iteration we independently simulate a mini-batch of $N$ i.i.d. samples of
$({\boldsymbol{X}}_{\mathcal{I}},Y)$. We denote the learnt NN by
$G_{\widehat{\vartheta}_{\mathcal{I}}}$. After learning the NN, we estimate
the sensitivity to ${\boldsymbol{X}}_{\mathcal{I}}$ out-of-sample, that is,
using an independent sample $({\boldsymbol{x}}_{\mathcal{I}}^{(l)},y^{(l)})$,
$l=1\ldots,M$, of $({\boldsymbol{X}}_{\mathcal{I}},Y)$, via

$\widehat{\xi}_{S_{\mathrm{VaR}}}(Y;{\boldsymbol{X}}_{\mathcal{I}})=1-\frac{\displaystyle\sum_{l=1}^{M}S_{\mathrm{VaR}}\left(G_{\widehat{\vartheta}_{\mathcal{I}}}({\boldsymbol{x}}_{\mathcal{I}}^{(l)}),y^{(l)}\right)}{\displaystyle\sum_{l=1}^{M}S_{\mathrm{VaR}}\left(\widehat{{\mathrm{VaR}}}_{\alpha}(Y),y^{(l)}\right)}\,,$
(18)

where $\widehat{{\mathrm{VaR}}}_{\alpha}(Y)$ is the sample quantile of $Y$.

Figure 5: Score-based sensitivities for ${\mathrm{VaR}}_{0.9}$. Left: scoring
function by iteration of learning the model. Middle:
${\mathrm{VaR}}_{0.9}(Y|X_{i})$ of the learnt NN. Right: score-based
sensitivities by iteration of learning the model.

For each choice of $\mathcal{I}$, the NN structure consists of 6 hidden layers
with 20 neurons per layer and SiLU activation functions apart from the last
layer. The networks are learnt via independently simulated mini-batches of
size $N=10^{3}$, i.e. every single iteration uses an independently simulated
i.i.d. samples of $({\boldsymbol{X}}_{\mathcal{I}},Y)$ of size $N=10^{3}$.
Figure 5 displays the learning of the NN and the score-based sensitivities to
single risk factors from $X_{1}$ (top row) to $X_{4}$ (bottom row), with
$\alpha=0.9$. The left panel displays the average score for each mini-batch
(Equation (17) with $N=10^{3}$ per mini-batch) against the iterations of
learning the NN. The right panel displays the out-of-sample estimates of the
score-based sensitivity (Equation (18) with $M=10^{4}$) against the iterations
of learning the model. The middle panels display the learnt NN,
$G_{\widehat{\vartheta}_{i}}$, that is an estimate of
${\mathrm{VaR}}_{0.9}(Y|X_{i})$ as a function of $X_{i}$, for $i=1,\ldots,4$
from top to bottom. The red dotted lines correspond to the mean of $X_{i}$ and
the first standard deviations from its mean. We observe that risk factor
$X_{1}$ has the largest sensitivity. The sensitivity to $X_{1}$ is
significantly larger than that to $X_{2}$, even though $X_{1}$ and $X_{2}$
have the same distribution and the aggregate portfolio loss $Y$ is symmetric
in $X_{1}$ and $X_{2}$. Thus, the difference in the sensitivities solely stems
from the dependence structure of the portfolio; recall that $X_{2}$ is
independent of $X_{4}$ while $X_{1}$ is highly correlated with $X_{4}$.
Moreover, the sensitivity to $X_{2}$ (the independent business line) is
negligible, while, the sensitivity to $X_{4}$, the multiplicative factor, is
the second largest sensitivity.

Next, we estimate the score-based sensitivities to two risk factors, i.e.,
$\xi_{S_{{\mathrm{VaR}}_{0.9}}}(Y;{\boldsymbol{X}}_{\mathcal{I}})$, for
$|\mathcal{I}|=2$. We proceed similarly to estimating the score-based
sensitivities to one risk factor in that we learn for each
$\mathcal{I}=\\{i,j\\}$ with $i\neq j$, a NN using Equation (17) and
independently simulated mini-batches. The corresponding sensitivities to
${\boldsymbol{X}}_{\mathcal{I}}$ are estimated using the learnt NN out-of-
sample via Equation (18) with $M=10^{4}$. The estimates of the sensitivities
to one and two risk factors for the ${\mathrm{VaR}}_{\alpha}$ with level
$\alpha=0.9$ are reported in Table 3. We observe that the sensitivities
$\xi_{S_{{\mathrm{VaR}}_{0.9}}}(Y;X_{1},X_{i})$ with $i\in\\{2,3,4\\}$ are
close to the sensitivity to $X_{1}$, thus knowing $X_{1}$ and then learning an
additional risk factor does not considerably increase the sensitivity. This is
in contrast to the sensitivity to the pair $(X_{2},X_{4})$ which is equal to
0.740, providing a sensitivity that is substantially larger than the sum of
the sensitivities to $X_{2}$ and to $X_{4}$.

Table 3: Score-based sensitivities for the insurance model and the ${\mathrm{VaR}}_{\alpha}$ at level $\alpha=0.9$ with respect to one and two risk factors. The 90% confidence interval (assuming the learned NNs are correct) are $(\pm 0.001)$ for all estimates. $\xi_{S_{{\mathrm{VaR}}_{0.9}}}(Y;X_{i},X_{j})$ | $X_{1}$ | $X_{2}$ | $X_{3}$ | $X_{4}$
---|---|---|---|---
$X_{1}$ | $0.601$ | $0.707$ | $0.616$ | $0.610$
$X_{2}$ |  | $0.123$ | $0.131$ | $0.740$
$X_{3}$ |  |  | $0.014$ | $0.410$
$X_{4}$ |  |  |  | $0.410$

In Figure 6 we compare the score-based sensitivities to single risk factors
for different levels of ${\mathrm{VaR}}_{\alpha}$, that is from $\alpha=0.9$
to $\alpha=0.99$. Specifically, we learn for each $\alpha$ and risk factor a
NN using the procedure prescribed above. Figure 6 displays violin plots of
$\xi_{S_{{\mathrm{VaR}}}}(Y;X_{i})$ for $i=1,\ldots,4$ and
$\alpha=0.9,\ldots,0.99$ calculated using the learnt NNs. We observe that the
sensitivities to $X_{1},X_{2}$, and $X_{4}$ are increasing in $\alpha$,
whereas the sensitivity to $X_{3}$ is decreasing. The bottom left panel,
depicting the sensitivity to $X_{3}$, contains negative values for the
sensitivities. These stem from statistical errors, as the sensitivities by
definition should be non-negative. Note, however, that zero is contained in
all 90% confidence intervals (displayed via the blue lines).

Figure 6: Score-based sensitivities for the ${\mathrm{VaR}}_{\alpha}$ with
$\alpha$ ranging from 0.9 to 0.99. The violin plot displays the average
sensitivity (red cross) and the 10% and the 90% quantile (blue lines), based
on 100 estimates from the learnt NN.

Next, we calculate the sensitivity of the insurance portfolio for the pair
$({\mathrm{VaR}}_{\alpha},{\mathrm{ES}}_{\alpha})$. Recall that
${\mathrm{ES}}_{\alpha}$ is not elicitable on its own, but is jointly
elicitable with ${\mathrm{VaR}}_{\alpha}$ (Fissler and Ziegel 2016). We
consider the 0-homogeneous strictly consistent scoring function given by

$\displaystyle
S_{{\mathrm{VaR}},{\mathrm{ES}}}(z_{1},z_{2},y)=\frac{1}{z_{2}\,(1-\alpha)}\;\left(z_{1}\left({\mathds{1}}_{\\{y\leq
z_{1}\\}}-\alpha\right)+y{\mathds{1}}_{\\{y>z_{1}\\}}\right)-1+\log\left(\frac{z_{2}}{y}\right)\,,\quad
z_{1},z_{2},y>0\,,$ (19)

which arises from the general family of scores in (22) with $g(x)=0$ and
$\phi(x)=-\log(x)$. (Note that all risk factors and also $Y$ are positive
almost surely.) We proceed similarly to estimating the score-based
sensitivities for ${\mathrm{VaR}}_{\alpha}$, in that we use NNs to estimate
the conditional functionals. Let ${\boldsymbol{X}}_{\mathcal{I}}$ be a sub-
vector of the risk factors $(X_{1},\ldots,X_{4})$ with dimension 1 or 2, and
denote by
$q_{\mathcal{I}}({\boldsymbol{x}}_{\mathcal{I}})={\mathrm{VaR}}_{\alpha}(Y|{\boldsymbol{X}}_{\mathcal{I}}={\boldsymbol{x}}_{\mathcal{I}})$
and
$e_{\mathcal{I}}({\boldsymbol{x}}_{\mathcal{I}})={\mathrm{ES}}_{\alpha}(Y|{\boldsymbol{X}}_{\mathcal{I}}={\boldsymbol{x}}_{\mathcal{I}})$
the conditional VaR and ES, viewed, for fixed $\alpha$, as a function of
${\boldsymbol{x}}_{\mathcal{I}}\in{\mathbb{R}}^{|\mathcal{I}|}$. Then, we
jointly estimate $q_{\mathcal{I}}$ and $e_{\mathcal{I}}$ via two neural nets
$G_{\vartheta_{\mathcal{I}}}$ and $H_{\varsigma_{\mathcal{I}}}$, respectively.
The network parameters are learnt by iteratively minimising the score (19)

$\left(\hat{\vartheta}_{\mathcal{I}},\;\hat{\varsigma}_{\mathcal{I}}\right)=\argmin_{\vartheta,\;\varsigma}\frac{1}{N}\sum_{k=1}^{N}S_{{\mathrm{VaR}},{\mathrm{ES}}}\left(G_{\vartheta}(x_{\mathcal{I}}^{(k)}),\;H_{\varsigma}(x_{\mathcal{I}}^{(k)}),\;y^{(k)}\right)\,,$
(20)

over independently simulated mini-batches
$({\boldsymbol{x}}_{\mathcal{I}}^{(k)},y^{(k)})$, $k=1,\ldots,N$, of
$({\boldsymbol{X}}_{\mathcal{I}},Y)$. We denote the learnt NNs by
$G_{\widehat{\vartheta}_{\mathcal{I}}}$ and
$H_{\widehat{\varsigma}_{\mathcal{I}}}$, respectively, and estimate the
sensitivity to ${\boldsymbol{X}}_{\mathcal{I}}$ using the learnt NNs out-of-
sample, that is using an independent sample
$({\boldsymbol{x}}_{\mathcal{I}}^{(l)},y^{(l)})$, $l=1\ldots,M$, of
$({\boldsymbol{X}}_{\mathcal{I}},Y)$, by

$\widehat{\xi}_{S_{{\mathrm{VaR}},{\mathrm{ES}}}}(Y;{\boldsymbol{X}}_{\mathcal{I}})=1-\frac{\displaystyle\sum_{l=1}^{M}S_{{\mathrm{VaR}},{\mathrm{ES}}}\left(G_{\widehat{\vartheta}_{\mathcal{I}}}({\boldsymbol{x}}_{\mathcal{I}}^{(l)}),H_{\widehat{\varsigma}_{\mathcal{I}}}({\boldsymbol{x}}_{\mathcal{I}}^{(l)}),y^{(l)}\right)}{\displaystyle\sum_{l=1}^{M}S_{{\mathrm{VaR}},{\mathrm{ES}}}\left(\widehat{{\mathrm{VaR}}}_{\alpha}(Y),\widehat{{\mathrm{ES}}}_{\alpha}(Y),y^{(l)}\right)}\,,$
(21)

where $\widehat{{\mathrm{VaR}}}_{\alpha}(Y)$ and
$\widehat{{\mathrm{ES}}}_{\alpha}(Y)$ are the sample quantile and sample ES of
$Y$, respectively.

Since for all ${\boldsymbol{x}}_{\mathcal{I}}$ it holds that
${\mathrm{ES}}_{\alpha}(Y|{\boldsymbol{X}}_{\mathcal{I}}={\boldsymbol{x}}_{\mathcal{I}})\geq{\mathrm{VaR}}_{\alpha}(Y|{\boldsymbol{X}}_{\mathcal{I}}={\boldsymbol{x}}_{\mathcal{I}})$,
we require that
$H_{\varsigma_{\mathcal{I}}}({\boldsymbol{x}}_{\mathcal{I}})\geq
G_{\vartheta_{\mathcal{I}}}({\boldsymbol{x}}_{\mathcal{I}})$, for all
${\boldsymbol{x}}_{\mathcal{I}}$. Thus, we define the NN for estimating the
conditional ES by
$H_{\varsigma_{\mathcal{I}}}({\boldsymbol{x}}_{\mathcal{I}})=G_{\vartheta_{\mathcal{I}}}({\boldsymbol{x}}_{\mathcal{I}})+\mathcal{H}_{\eta_{\mathcal{I}}}({\boldsymbol{x}}_{\mathcal{I}})$,
where $\mathcal{H}_{\eta_{\mathcal{I}}}({\boldsymbol{x}}_{\mathcal{I}})$ is
constrained to be non-negative for all ${\boldsymbol{x}}_{\mathcal{I}}$;
modelled using a softplus activation function on the last node. For each
choice of $\mathcal{I}$, we choose a NN structure consists of 6 hidden layers
with 20 neurons per layer. The activation functions are SiLU for each layer
apart from the last layer which uses a softplus activation function.

The estimates of the score-based sensitivities for
$({\mathrm{VaR}}_{0.9},\,{\mathrm{ES}}_{0.9})$ are reported in Table 4. We
observe that the sensitivities are similar in magnitude to the sensitivities
for the ${\mathrm{VaR}}_{0.9}$, see also Table 3. Moreover, the ranking of the
risk factors stays the same.

Table 4: Score-based sensitivities for ${\mathrm{VaR}}_{\alpha}$ and ${\mathrm{ES}}_{\alpha}$ at level $\alpha=0.9$ with respect to one and two risk factors. The 90% confidence interval (assuming the learned NNs are correct) are $(\pm 0.001)$ for all estimates. $\xi_{S_{{\mathrm{VaR}}_{0.9},\,{\mathrm{ES}}_{0.9}}}(Y;X_{i},X_{j})$ | $X_{1}$ | $X_{2}$ | $X_{3}$ | $X_{4}$
---|---|---|---|---
$X_{1}$ | $0.565$ | $0.689$ | $0.589$ | $0.586$
$X_{2}$ |  | $0.121$ | $0.122$ | $0.718$
$X_{3}$ |  |  | $0.002$ | $0.393$
$X_{4}$ |  |  |  | $0.384$

##  6 Conclusion

This paper provides a comprehensive framework for constructing sensitivity
measures induced by strictly consistent scoring functions for any elicitable
target functional $T$. These score-based sensitivities naturally quantify the
_relative_ information gain – when using available information ideally – to
model the target functional. Theorem 3.1 establishes intuitive and desirable
properties for these score-based sensitivities such as zero information gain
and full information gain. Following Griessenberger et al. (2022), these
properties suggest that the sensitivities can also be considered as a
dependence measure between an output $Y$ and an information set
$\mathfrak{A}$. We further define a sensitivity called interaction
information, which quantifies the information gain when learning the
interaction of risk factors. We show that sensitivities based on a positively
homogeneous score are scale-invariant, making them attractive in applications.
Using Murphy diagrams for score-based sensitivities, we illustrate how to
inspect entire classes of sensitivities, thus providing a holistic impression
and revealing yet unknown model characteristics.

We emphasise that our approach is general and works for sensitivities which
focus on any elicitable functional. In particular, we discuss the entire
family of score-based sensitivity measures for the mean functional (of which
the Sobol indices are one special case) and construct sensitivities for the
pair consisting of VaR and ES; both, to the authors’ best knowledge, novel
endeavours to the literature.

## Acknowledgements

The authors thank Johanna Ziegel and Timo Dimitriadis for stimulating
discussions about the topic as well as three anonymous referees whose comments
helped to substantially improve the paper. SP would like to acknowledge
support from the Natural Sciences and Engineering Research Council of Canada
(grants DGECR-2020-00333 and RGPIN-2020-04289).

##  7 Scoring Functions of Risk Functionals

This appendix provides an overview of known characterisation results of
(strictly) consistent scoring functions for popular functionals. For precise
technical conditions and statements see original references.

###  7.1 Entropic Risk Measures

Entropic risk measures, also called exponential premium principle in
insurance, (Gerber 1974) are defined for a random variable $Y$ and parameter
$\gamma>0$ by

$\rho^{\gamma}(Y)=\frac{1}{\gamma}\log\,{\mathbb{E}}\left[\exp(\gamma
Y)\right]\,.$

Invoking the revelation principle (Osband 1985, Gneiting 2011), we can exploit
the characterisation result (6) and obtain that, subject to mild regularity
conditions, any (strictly) consistent score satisfying $S(y,y)=0$ is of the
form

$S(z,y)=\phi\big{(}\exp(\gamma y)\big{)}-\phi\big{(}\exp(\gamma
z)\big{)}+\phi^{\prime}\big{(}\exp(\gamma z)\big{)}(\exp(\gamma z)-\exp(\gamma
y)),\qquad z,y\in{\mathbb{R}},$

where $\phi\colon{\mathbb{R}}\to{\mathbb{R}}$ is (strictly) convex with
subgradient $\phi^{\prime}$ such that $\int|\phi\big{(}\exp(\gamma
y)\big{)}|\,\mathrm{d}F(y)<\infty$ for all $F\in\mathcal{M}$.

###  7.2 Expectiles

For $\tau\in(0,1)$, the $\tau$-expectile of a probability distribution $F$ is
the unique solution of the equation

$\tau\int_{z}^{\infty}(y-z)\,\mathrm{d}F(Y)=(1-\tau)\int_{-\infty}^{z}(z-y)\,\mathrm{d}F(Y).$

Gneiting (2011) characterises the class of (strictly) $\mathcal{M}$-consistent
scoring functions for the $\tau$-expectile satisfying $S(y,y)=0$ as

$S(z,y)=|{\mathds{1}}_{\\{y\leq
x\\}}-\tau|\big{(}\phi(y)-\phi(z)+\phi^{\prime}(z)(z-y)\big{)},\qquad
z,y\in{\mathbb{R}},$

where $\phi\colon{\mathbb{R}}\to{\mathbb{R}}$ is (strictly) convex with
subgradient $\phi^{\prime}$ such that $\int|\phi(y)|\mathrm{d}F(y)<\infty$ for
all $F\in\mathcal{M}$.

###  7.3 Mode

The mode is generally not elicitable on classes of absolutely continuous
distributions (Heinrich 2014, Heinrich-Mertsching and Fissler 2021). However,
on distributions with countably many outcomes, the mode is elicitable.
Gneiting (2017) showed that the zero-one loss $S(z,y)={\mathds{1}}_{\\{z\neq
y\\}}$ is basically the only strictly consistent score such that $S(y,y)=0$.
This yields the score-based sensitivity

$\xi_{S}(Y;\mathfrak{A})=\frac{(1-p_{\max})-(1-{\mathbb{E}}[p_{\max|\mathfrak{A}}])}{1-p_{\max}},$

where $p_{\max}$ is the maximum of the counting density of $Y$ and
$p_{\max|\mathfrak{A}}$ is the maximum of the conditional counting density of
$Y$ given $\mathfrak{A}$.

###  7.4 Mean and Variance

We can again invoke the revelation principle to characterise the class of
(strictly) $\mathcal{M}$-consistent scoring functions for the pair (mean,
variance) satisfying Assumption 2 (i), such that $S(y,0,y)=0$. Following
Fissler and Ziegel (2019) they take the form

$S(z_{1},z_{2},y)=\phi(y,y^{2})-\phi(z_{1},z_{2}+z_{1}^{2})+\nabla\phi(z_{1},z_{2}+z_{1}^{2})\begin{pmatrix}z_{1}-y\\\
z_{2}+z_{1}^{2}-y^{2}\end{pmatrix},\qquad z_{1},y\in{\mathbb{R}},\ z_{2}\geq
0,$

where $\phi\colon{\mathbb{R}}\times[0,\infty)\to{\mathbb{R}}$ is (strictly)
convex with subgradient $\nabla\phi$, interpreted as a row vector, such that
$\int|\phi(y,y^{2})+a(y)|\,\mathrm{d}F(y)<\infty$ for all $F\in\mathcal{M}$.

###  7.5 VaR and ES

ES generally fails to be elicitable (Osband 1985, Gneiting 2011, Weber 2006).
However, the pair ${\mathrm{VaR}}_{\alpha}$ together with
${\mathrm{ES}}_{\alpha}$, at the same probability level $\alpha\in(0,1)$, is
elicitable, subject to mild conditions (Acerbi and Szekely 2014, Fissler and
Ziegel 2016). Note that different sign conventions are used in the literature.
Upon using the one introduced in 8, the (strictly) $\mathcal{M}$-consistent
scoring functions for $({\mathrm{VaR}}_{\alpha},{\mathrm{ES}}_{\alpha})$
satisfying Assumption 2 (i) such that $S(y,y,y)=0$ are of the form

$\displaystyle S(z_{1},z_{2},y)$ $\displaystyle=\big{(}{\mathds{1}}_{\\{y\leq
z_{1}\\}}-\alpha\big{)}\big{(}g(z_{1})-g(y)\big{)}$ (22)
$\displaystyle+\phi^{\prime}(z_{2})\Big{(}z_{2}-\tfrac{1}{1-\alpha}S^{+}_{\alpha}(z_{1},y)\Big{)}-\phi(z_{2})+\phi(y),\quad
z_{1},z_{2},y\in{\mathbb{R}},$

where
$S^{+}_{\alpha}(z_{1},y)=(1-\alpha-{\mathds{1}}_{\\{y>z_{1}\\}})z_{1}+{\mathds{1}}_{\\{y>z_{1}\\}}y=({\mathds{1}}_{\\{y\leq
z_{1}\\}}-\alpha)z_{1}-{\mathds{1}}_{\\{y\leq z_{1}\\}}y+y$,
$\phi\colon{\mathbb{R}}\to{\mathbb{R}}$ is (strictly) convex with subgradient
$\phi^{\prime}$ and $g\colon{\mathbb{R}}\to{\mathbb{R}}$ is such that for all
$z_{2}\in{\mathbb{R}}$

$\displaystyle z_{1}\mapsto g(z_{1})-z_{1}\phi^{\prime}(z_{2})/(1-\alpha)$

is (strictly) increasing. One convenient way to achieve this latter condition
is to choose $\phi$ such that $\phi^{\prime}<0$ and $g$ to be increasing.

To achieve strict consistency, the requirements on $\mathcal{M}$ are that the
expected scores are finite at their minimum (amounting to the fact that
$\int|\kappa(y)|\,\mathrm{d}F(y)<\infty$ for all $F\in\mathcal{M}$ and for
$\kappa$ being $g$, $\phi$, or the identity), and that for all
$F\in\mathcal{M}$, $F({\mathrm{VaR}}_{\alpha}(F)+\epsilon)>\alpha$ for all
$\epsilon>0$, see Proposition 4.2 (ii) and Fissler and Ziegel (2016) for
details.

###  7.6 RVaR together with two VaR components

The Range Value-at-Risk (RVaR) at levels $0<\alpha<\beta<1$ for a random
variable has been introduced by Cont et al. (2010) and is defined as

$\displaystyle{\mathrm{RVaR}}_{\alpha,\beta}(Y)=\frac{1}{\beta-\alpha}\int_{\alpha}^{\beta}{\mathrm{VaR}}_{u}(Y)\,\mathrm{d}u\,.$

RVaR fails to be elicitable on its own in general (Wang and Wei 2020).
However, Fissler and Ziegel (2021) show that the triplet
$({\mathrm{VaR}}_{\alpha},{\mathrm{VaR}}_{\beta},{\mathrm{RVaR}}_{\alpha,\beta})$
is elicitable subject to mild conditions. The (strictly)
$\mathcal{M}$-consistent scoring functions satisfying Assumption 2 (i) such
that $S(y,y,y,y)=0$ are of the form

$\displaystyle S(z_{1},z_{2},z_{3},y)$
$\displaystyle=\big{(}{\mathds{1}}_{\\{y\leq
z_{1}\\}}-\alpha\big{)}\big{(}g_{1}(z_{1})-g_{1}(y)\big{)}+\big{(}{\mathds{1}}_{\\{y\leq
z_{2}\\}}-\beta\big{)}\big{(}g_{2}(z_{2})-g_{2}(y)\big{)}$
$\displaystyle+\phi^{\prime}(z_{3})\Big{(}z_{3}+\frac{1}{\beta-\alpha}\big{(}S_{\beta}(z_{2},y)-S_{\alpha}(z_{1},y)\big{)}\Big{)}-\phi(z_{3})+\phi(y),\quad
z_{1},z_{2},z_{3},y\in{\mathbb{R}}\,,$

where $S_{\gamma}(z,y)=({\mathds{1}}_{\\{y\leq
z\\}}-\gamma)z-{\mathds{1}}_{\\{y\leq z\\}}y$ for
$\gamma\in\\{\alpha,\beta\\}$, $\phi\colon{\mathbb{R}}\to{\mathbb{R}}$ is
(strictly) convex with subgradient $\phi^{\prime}$ and
$g_{1},g_{2}\colon{\mathbb{R}}\to{\mathbb{R}}$ are such that for all
$z_{3}\in{\mathbb{R}}$ the functions

$\displaystyle z_{1}\mapsto
g_{1}(z_{1})-z_{1}\phi^{\prime}(z_{3})/(\beta-\alpha)\quad\text{and}$
$\displaystyle z_{2}\mapsto
g_{2}(z_{2})+z_{2}\phi^{\prime}(z_{3})/(\beta-\alpha)$

are (strictly) increasing. We refer to Fissler and Ziegel (2021) for examples
of such scores as well as to the detailed conditions on $\mathcal{M}$.

## References

  * Acerbi (2002) Acerbi C (2002) Spectral measures of risk: A coherent representation of subjective risk aversion. _Journal of Banking & Finance_ 26(7):1505–1518, URL https://doi.org/10.1016/S0378-4266(02)00281-9.
  * Acerbi and Szekely (2014) Acerbi C, Szekely B (2014) Backtesting Expected Shortfall. _Risk Magazine_ URL https://www.msci.com/documents/10199/22aa9922-f874-4060-b77a-0f0e267a489b.
  * Baroni and Francke (2020) Baroni G, Francke T (2020) An effective strategy for combining variance-and distribution-based global sensitivity analysis. _Environmental Modelling & Software_ 134:104851, URL https://doi.org/10.1016/j.envsoft.2020.104851.
  * Baucells and Borgonovo (2013) Baucells M, Borgonovo E (2013) Invariant probabilistic sensitivity analysis. _Management Science_ 59(11):2536–2549, URL http://dx.doi.org/10.1287/mnsc.2013.1719.
  * Borgonovo (2007) Borgonovo E (2007) A new uncertainty importance measure. _Reliability Engineering & System Safety_ 92(6):771–784, URL https://doi.org/10.1016/j.ress.2006.04.015.
  * Borgonovo and Cillo (2017) Borgonovo E, Cillo A (2017) Deciding with thresholds: Importance measures and value of information. _Risk Analysis_ 37(10):1828–1848, URL https://doi.org/10.1111/risa.12732.
  * Borgonovo et al. (2021) Borgonovo E, Hazen GB, Jose VRR, Plischke E (2021) Probabilistic sensitivity measures as information value. _European Journal of Operational Research_ 289(2):595–610, URL https://doi.org/10.1016/j.ejor.2020.07.010.
  * Borgonovo and Plischke (2016) Borgonovo E, Plischke E (2016) Sensitivity analysis: a review of recent advances. _European Journal of Operational Research_ 248(3):869–887, URL https://doi.org/10.1016/j.ejor.2015.06.032.
  * Browne et al. (2017) Browne T, Fort JC, Iooss B, Le Gratiet L (2017) Estimate of quantile-oriented sensitivity indices. _Preprint_ URL https://hal.archives-ouvertes.fr/hal-01450891.
  * Cont et al. (2010) Cont R, Deguest R, Scandolo G (2010) Robustness and sensitivity analysis of risk measurement procedures. _Quantitative Finance_ 10(6):593–606, URL https://doi.org/10.1080/14697681003685597.
  * Dawid (2007) Dawid AP (2007) The geometry of proper scoring rules. _Annals of the Institute of Statistical Mathematics_ 59(1):77–93, URL https://doi.org/10.1007/s10463-006-0099-8.
  * Ehm et al. (2016) Ehm W, Gneiting T, Jordan A, Krüger F (2016) Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. _Journal of the Royal Statistical Society Series B (Statistical Methodology)_ 78(3):505–562, URL https://doi.org/10.1111/rssb.12154.
  * Engelberg et al. (2009) Engelberg J, Manski CF, Williams J (2009) Comparing the point predictions and subjective probability distributions of professional forecasters. _Journal of Business & Economic Statistics_ 27(1):30–41, URL https://www.jstor.org/stable/27639017.
  * Felli and Hazen (1998) Felli JC, Hazen GB (1998) Sensitivity analysis and the expected value of perfect information. _Medical Decision Making_ 18(1):95–109, URL https://doi.org/10.1177/0272989X9801800117.
  * Fissler et al. (2021a) Fissler T, Frongillo R, Hlavinová J, Rudloff B (2021a) Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals. _Electronic Journal of Statistics_ 15(1):1034–1084, URL http://dx.doi.org/10.1214/21-EJS1808.
  * Fissler and Holzmann (2022) Fissler T, Holzmann H (2022) Measurability of functionals and of ideal point forecasts. _Preprint_ URL https://doi.org/10.48550/arXiv.2203.08635.
  * Fissler et al. (2021b) Fissler T, Merz M, Wüthrich MV (2021b) Deep quantile and deep composite model regression. _arXiv preprint arXiv:2112.03075_ .
  * Fissler and Ziegel (2016) Fissler T, Ziegel JF (2016) Higher order elicitability and Osband’s principle. _Annals of Statistics_ 44(4):1680–1707, URL https://doi.org/10.1214/16-AOS1439.
  * Fissler and Ziegel (2019) Fissler T, Ziegel JF (2019) Order-sensitivity and equivariance of scoring functions. _Electronic Journal of Statistics_ 13(1):1166–1211, URL https://doi.org/10.1214/19-EJS1552.
  * Fissler and Ziegel (2021) Fissler T, Ziegel JF (2021) On the elicitability of range value at risk. _Statistics & Risk Modeling_ 38(1-2):25–46, URL https://doi.org/10.1515/strm-2020-0037.
  * Fort et al. (2021) Fort JC, Klein T, Lagnoux A (2021) Global sensitivity analysis and wasserstein spaces. _SIAM/ASA Journal on Uncertainty Quantification_ 9(2):880–921, URL https://doi.org/10.1137/20M1354957.
  * Fort et al. (2016) Fort JC, Klein T, Rachdi N (2016) New sensitivity analysis subordinated to a contrast. _Communications in Statistics-Theory and Methods_ 45(15):4349–4364, URL https://doi.org/10.1080/03610926.2014.901369.
  * Frankel and Kamenica (2019) Frankel A, Kamenica E (2019) Quantifying information and uncertainty. _American Economic Review_ 109(10):3650–3680, URL https://doi.org/10.1257/aer.20181897.
  * Gamboa et al. (2018) Gamboa F, Klein T, Lagnoux A (2018) Sensitivity analysis based on Cramér–von Mises distance. _SIAM/ASA Journal on Uncertainty Quantification_ 6(2):522–548, URL https://doi.org/10.1137/15M1025621.
  * Gerber (1974) Gerber HU (1974) On additive premium calculation principles. _ASTIN Bulletin: The Journal of the IAA_ 7(3):215–222, URL https://doi.org/10.1017/S0515036100006061.
  * Gneiting (2011) Gneiting T (2011) Making and Evaluating Point Forecasts. _Journal of the American Statistical Association_ 106(494):746–762, URL https://www.jstor.org/stable/41416407.
  * Gneiting (2017) Gneiting T (2017) When is the mode functional the bayes classifier? _Stat_ 6(1):204–206, URL https://doi.org/10.1002/sta4.148.
  * Gneiting and Raftery (2007) Gneiting T, Raftery A (2007) Strictly Proper Scoring Rules, Prediction, and Estimation. _Journal of the American Statistical Association_ 102(477):359–378, URL https://doi.org/10.1198/016214506000001437.
  * Gneiting and Resin (2021) Gneiting T, Resin J (2021) Regression diagnostics meets forecast evaluation: Conditional calibration, reliability diagrams, and coefficient of determination. _Preprint_ URL https://arxiv.org/abs/2108.03210.
  * Griessenberger et al. (2022) Griessenberger F, Junker RR, Trutschnig W (2022) On a multivariate copula-based dependence measure and its estimation. _Electronic Journal of Statistics_ 16(1):2206–2251, URL https://doi.org/10.1214/22-EJS2005.
  * Heinrich (2014) Heinrich C (2014) The mode functional is not elicitable. _Biometrika_ 101(1):245–251, URL https://doi.org/10.1093/biomet/ast048.
  * Heinrich-Mertsching and Fissler (2021) Heinrich-Mertsching C, Fissler T (2021) Is the mode elicitable relative to unimodal distributions? _Biometrika_ URL https://doi.org/10.1093/biomet/asab065.
  * Holzmann and Eulert (2014) Holzmann H, Eulert M (2014) The role of the information set for forecasting – with applications to risk management. _Annals of Applied Statistics_ 8(1):79–83, URL https://doi.org/10.1214/13-AOAS709.
  * Ishigami and Homma (1990) Ishigami T, Homma T (1990) An importance quantification technique in uncertainty analysis for computer models. _[1990] Proceedings. First International Symposium on Uncertainty Modeling and Analysis_ , 398–403 (IEEE), URL https://doi.org/10.1109/ISUMA.1990.151285.
  * Krüger and Ziegel (2021) Krüger F, Ziegel JF (2021) Generic conditions for forecast dominance. _Journal of Business & Economic Statistics_ 39(4):972–983, URL http://dx.doi.org/10.1080/07350015.2020.1741376.
  * Lambert et al. (2008) Lambert N, Pennock DM, Shoham Y (2008) Eliciting properties of probability distributions. _EC ’08: ACM Conference on Electronic Commerce, Chicago II, USA_ , 129–138 (New York, USA: Association for Computing Machinery), URL https://doi.org/10.1145/1386790.1386813.
  * Marrel et al. (2009) Marrel A, Iooss B, Laurent B, Roustant O (2009) Calculations of sobol indices for the gaussian process metamodel. _Reliability Engineering & System Safety_ 94(3):742–751, URL https://doi.org/10.1016/j.ress.2008.07.008.
  * Maume-Deschamps and Niang (2018) Maume-Deschamps V, Niang I (2018) Estimation of quantile oriented sensitivity indices. _Statistics & Probability Letters_ 134:122–127, URL https://doi.org/10.1016/j.spl.2017.10.019.
  * McNeil et al. (2015) McNeil AJ, Frey R, Embrechts P (2015) _Quantitative Risk Management: Concepts, Techniques and Tools_. Princeton Series in Finance (Princeton University Press), revised edition edition.
  * Murphy (1973) Murphy AH (1973) Hedging and skill scores for probability forecasts. _Journal of Applied Meteorology_ 12(1):215–223, URL https://doi.org/10.1175/1520-0450(1973)012<0215:HASSFP>2.0.CO;2.
  * Murphy and Daan (1985) Murphy AH, Daan H (1985) Forecast Evaluation. Murphy AH, Katz RW, eds., _Probability, Statistics and Decision Making in the Atmospheric Sciences_ , 379–437 (Westview Press, Boulder, Colorado).
  * Nolde and Ziegel (2017) Nolde N, Ziegel JF (2017) Elicitability and backtesting: Perspectives for banking regulation. _Annals of Applied Statistics_ 11(4):1833–1874, URL https://doi.org/10.1214/17-AOAS1041.
  * Osband (1985) Osband KH (1985) _Providing Incentives for Better Cost Forecasting_. Ph.D. thesis, University of California, Berkeley, URL https://doi.org/10.5281/zenodo.4355667.
  * Patton (2011) Patton AJ (2011) Data-based ranking of realised volatility estimators. _Journal of Econometrics_ 161(2):284–303, URL https://doi.org/10.1016/j.jeconom.2010.12.010.
  * Pesenti (2021) Pesenti SM (2021) Reverse sensitivity analysis for risk modelling. _Preprint_ URL https://arxiv.org/abs/2107.01065.
  * Pesenti et al. (2019) Pesenti SM, Millossovich P, Tsanakas A (2019) Reverse sensitivity testing: What does it take to break the model? _European Journal of Operational Research_ 274(2):654–670, URL https://doi.org/10.1016/j.ejor.2018.10.003.
  * Pesenti et al. (2021) Pesenti SM, Millossovich P, Tsanakas A (2021) Cascade sensitivity measures. _Risk Analysis_ 31(12):2392–2414, URL https://doi.org/10.1111/risa.13758.
  * Pianosi and Wagener (2015) Pianosi F, Wagener T (2015) A simple and efficient method for global sensitivity analysis based on cumulative distribution functions. _Environmental Modelling & Software_ 67:1–11, URL https://doi.org/10.1016/j.envsoft.2015.01.004.
  * Plischke and Borgonovo (2019) Plischke E, Borgonovo E (2019) Copula theory and probabilistic sensitivity analysis: Is there a connection? _European Journal of Operational Research_ 277(3):1046–1059.
  * Pohle (2020) Pohle MO (2020) The Murphy decomposition and the calibration-resolution principle: A new perspective on forecast evaluation. _Preprint_ URL https://arxiv.org/abs/2005.01835.
  * Razavi et al. (2021) Razavi S, Jakeman A, Saltelli A, Prieur C, Iooss B, Borgonovo E, Plischke E, Piano SL, Iwanaga T, Becker W, et al. (2021) The future of sensitivity analysis: An essential discipline for systems modeling and policy support. _Environmental Modelling & Software_ 137:104954, URL https://doi.org/10.1016/j.envsoft.2020.104954.
  * Rényi (1959) Rényi A (1959) On measures of dependence. _Acta Mathematica Academiae Scientiarum Hungaricae_ (10):441–451.
  * Saltelli et al. (2008) Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D, Saisana M, Tarantola S (2008) _Global sensitivity analysis: the primer_ (John Wiley & Sons).
  * Saltelli and Tarantola (2002) Saltelli A, Tarantola S (2002) On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal. _Journal of the American Statistical Association_ 97(459):702–709, URL https://doi.org/10.1198/016214502388618447.
  * Straub et al. (2021) Straub D, Ehre M, Papaioannou I (2021) Decision-theoretic reliability sensitivity. _Reliability Engineering & System Safety_ 108215, URL https://doi.org/10.1016/j.ress.2021.108215.
  * Thorarinsdottir et al. (2013) Thorarinsdottir TL, Gneiting T, Gissibl N (2013) Using proper divergence functions to evaluate climate models. _SIAM/ASA Journal on Uncertainty Quantification_ 1(1):522–534, URL https://doi.org/10.1137/130907550.
  * Tsanakas and Millossovich (2016) Tsanakas A, Millossovich P (2016) Sensitivity analysis using risk measures. _Risk Analysis_ 36(1):30–48, URL https://doi.org/10.1111/risa.12434.
  * Wang and Wei (2020) Wang R, Wei Y (2020) Risk functionals with convex level sets. _Mathematical Finance_ 30(4):1337–1367, URL https://doi.org/10.1111/mafi.12270.
  * Weber (2006) Weber S (2006) Distribution-Invariant Risk Measures, Information, and Dynamic Consistency. _Mathematical Finance_ 16(2):419–441, URL https://doi.org/10.1111/j.1467-9965.2006.00277.x.
  * Wooldridge (2013) Wooldridge JM (2013) _Introductory Econometrics: A Modern Approach_ (South-Western, Cengage Learning), 5th edition.