disposition

# Backtesting Systemic Risk Forecasts using Multi-Objective Elicitability111We
are grateful to Immanuel Bomze for suggesting to consider multivariate scoring
functions equipped with general orders. Furthermore, we would like to thank
Timo Dimitriadis, Rüdiger Frey, Christoph Hanck, Jana Hlavinová, Kurt Hornik,
Marc-Oliver Pohle, Birgit Rudloff and Johanna F. Ziegel for detailed comments
and valuable discussions. Of course, all errors and and opinions expressed in
this article are solely the authors’ responsibility. Yannick Hoga gratefully
acknowledges support of the German Research Foundation (DFG) through grant HO
6305/2-1.

Tobias Fissler Vienna University of Economics and Business (WU), Department of
Finance, Accounting and Statistics, Welthandelsplatz 1, 1020 Vienna, Austria,
e-mail<EMAIL_ADDRESS>Yannick Hoga University of Duisburg-Essen,
Faculty of Economics and Business Administration, Universitätsstraße 12,
D-45117 Essen, Germany, tel. +49 201 1834365, e-mail<EMAIL_ADDRESS>due.de.

(August 27, 2024)

###### Abstract

Systemic risk measures such as CoVaR, CoES and MES are widely-used in finance,
macroeconomics and by regulatory bodies. Despite their importance, we show
that they fail to be elicitable and identifiable. This renders forecast
comparison and validation, commonly summarised as ‘backtesting’, impossible.
The novel notion of _multi-objective elicitability_ solves this problem.
Specifically, we propose Diebold–Mariano type tests utilising two-dimensional
scores equipped with the lexicographic order. We illustrate the test decisions
by an easy-to-apply traffic-light approach. We apply our traffic-light
approach to DAX 30 and S&P 500 returns, and infer some recommendations for
regulators.

Keywords: Backtest; (Conditional) Elicitability; Forecasting; Identifiability;
Lexicographic Order; Multi-objective Optimisation; Systemic Risk

JEL classification: C18 (Methodological Issues), C52 (Model Evaluation,
Validation, and Selection), C58 (Financial Econometrics)

##  1 Motivation

Regulating financial institutions in isolation is often not sufficient to
prevent financial crises due to the interdependent risks these institutions
face. In particular, their losses commonly exhibit a pronounced comonotonic
behaviour in the extreme tails: When one financial institution, or the market
as a whole, is in distress, other institutions are much more prone to being at
risk as well. The U.S. subprime mortgage crisis of 2008–2009, the European
sovereign debt crisis of 2010–2011 and the Covid-19 crash of 2020 have
forcefully demonstrated this fact and also the need to assess the _systemic_
nature of risk. As a consequence of these crises, a huge literature on
measuring systemic risk has emerged over the last decade (GK11;
ChenIyengarMoallemi2013; AB16; Aea17; BE17; FeinsteinRudloffWeber2017).

Systemic risk measures are important in various contexts. First, they are
important in banking regulation under the Basel framework of the BCBSBF19,
where they are vital in determining which banks are among the globally
systemically important banks (G-SIBs). Such G-SIBs are then subjected to
higher capital requirements. Second, in finance, systemic risk measures may be
used to study spillover effects in the financial system (AB16) or the build-up
of asset price bubbles (BRS20). Third, they may be used to study the linkage
between the financial sector and the real economy. Among others, GKP16 and
BE17 show that an increase in systemic risk is predictive of future declines
in real economic activity. All these examples underscore the importance of
accurately measuring and predicting systemic risk.

In this paper, we revisit three influential systemic risk measures. First, we
consider AB16’s (AB16) conditional value-at-risk (CoVaR) and conditional
expected shortfall (CoES) as extensions of the well-known value-at-risk (VaR)
and expected shortfall (ES) to the realm of systemic risk. If $Y$ are the
losses of interest and $X$ the losses of a reference position,
$\operatorname{CoVaR}_{\alpha|\beta}(Y|X)$
($\operatorname{CoES}_{\alpha|\beta}(Y|X)$) is the VaR (ES) of $Y$ at level
$\alpha$, given that $X$ is “in distress”. Here, we interpret the event that
$X$ is in distress as $X$ being larger or equal than its $\beta$-quantile,
i.e., $\\{X\geq\operatorname{VaR}_{\beta}(X)\\}$. Finally, we consider Aea17’s
(Aea17) marginal expected shortfall, $\operatorname{MES}_{\beta}(Y|X)$, as the
conditional mean of $Y$ given $\\{X\geq\operatorname{VaR}_{\beta}(X)\\}$.
Section 2 introduces the exact definitions.

Bea17 distinguish between the “source-specific approach” and the “global
approach” to systemic risk measurement. The source-specific approach considers
individual sources of systemic risk, such as contagion risk or liquidity
crises. In contrast, global measures of systemic risk potentially incorporate
all mechanisms studied in the source-specific approach. Bea17 categorize
CoVaR, CoES and MES under the global approach.

In practice, forecasting systemic risk measures—such as CoVaR, CoES and
MES—requires adequate models for the marginals $X$ and $Y$, and for their
dependence structure. The literature has developed numerous different
modelling approaches for this; see GT13 and BC19 for forecasting models for
CoVaR and CoES, and BE17 and Eck18 for MES models. Due to the importance of
systemic risk measures outlined above, it is vital to develop statistical
quality assessments of the various models’ predictive performances. It is the
main aim of this paper to provide such tools, which are referred to as
‘backtests’ in finance.

Backtests have two main goals. On the one hand, one may wish to assess the
_absolute quality_ of forecasting models, also called the _calibration_ , akin
to model validation in statistics. Following the terminology of FZG16, we call
such procedures “traditional backtests”. Roughly speaking, they check how well
a sequence of risk measure forecasts aligns with corresponding observations of
losses. Traditional backtests rely on the identifiability of the underlying
risk measure, which ensures the existence of a (possible multivariate)
function $\bm{V}$ that uniquely “identifies” the true report (see Definition
3.2). On the other hand, the presence of several alternative prediction models
for a risk measure necessitates “comparative backtests” (FZG16) to assess
their predictive accuracy relative to each other. This is akin to statistical
model selection procedures. Comparative backtests exploit the elicitability of
the underlying risk measure. This implies the existence of a real-valued loss
(or also: scoring) function $S$, which is minimised in expectation by the
optimal forecast (see Definition 3.1).

Our contributions in this paper are the following: First, we show that
$\operatorname{CoVaR}_{\alpha|\beta}$, $\operatorname{CoES}_{\alpha|\beta}$
and $\operatorname{MES}_{\beta}$ are not identifiable and elicitable as
standalone risk measures (Proposition 4.1). The practical implication of this
is that neither traditional nor comparative backtests can be carried out. In
particular, any regulation based solely on these systemic risk measures is
pointless, because neither the adequacy of the forecasts can be determined nor
can different systemic risk forecasts be sensibly compared (say to a
regulatory standard model).

We provide a partial remedy for this drawback by giving joint identification
functions for
$(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(Y|X))$,
$(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(Y|X),\operatorname{CoES}_{\alpha|\beta}(Y|X))$,
and $(\operatorname{VaR}_{\beta}(X),\operatorname{MES}_{\beta}(Y|X))$ (Theorem
4.2). These identification functions can be used for (conditional) calibration
tests in the spirit of NZ17. To the best of our knowledge, this entails the
first traditional backtest for these systemic risk measures apart from
Banulescu-RaduETAL2019. We contrast our approach with theirs in detail in
Remark 4.3. In particular, they use one-dimensional identification functions
for $(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(Y|X))$
and $(\operatorname{VaR}_{\beta}(X),\operatorname{MES}_{\beta}(Y|X))$, which
fail to be strict in contrast to our two-dimensional identification functions.
For the backtest of Banulescu-RaduETAL2019, this non-strictness leads to a
complete loss of power in identifying certain misspecified systemic risk
forecasts (Section G in the Supplement). Theoretically, our results are akin
to the fact that $\operatorname{ES}_{\alpha}(Y)$ is not identifiable on its
own, but the pair
$(\operatorname{VaR}_{\alpha}(Y),\operatorname{ES}_{\alpha}(Y))$ is
identifiable (FZ16a).

In stark contrast to the joint elicitability of the pair
$(\operatorname{VaR}_{\alpha},\operatorname{ES}_{\alpha})$, however, we show
that the pairs $(\operatorname{VaR}_{\beta},$
$\operatorname{CoVaR}_{\alpha|\beta})$,
$(\operatorname{VaR}_{\beta},\operatorname{MES}_{\beta})$ and the triplet
$(\operatorname{VaR}_{\beta},\operatorname{CoVaR}_{\alpha|\beta},\operatorname{CoES}_{\alpha|\beta})$
_fail_ to be elicitable (Section D). So while traditional backtests for the
above pairs and the triplet may be constructed by virtue of their
identifiability, classical comparative backtests exploiting elicitability are
not feasible.

As a remedy to this negative result, we propose the novel concept of _multi-
objective elicitability_ , which works with multivariate scores $\bm{S}$
mapping to $\mathbb{R}^{m}$ equipped with a certain (partial) order relation
$\preceq$. This contrasts sharply with classical $\mathbb{R}$-valued losses
$S$. Their prevalence to date is grounded in tradition (Gne11) and the fact
that $\mathbb{R}$ is equipped with the canonical (total) order relation
$\leq$, which allows for straightforward comparisons of losses. Subsection 3.2
introduces _multi-objective scores_ $\bm{S}$ and the corresponding concepts of
_multi-objective consistency and elicitability_. The terminology stems from
the field of _multi-objective optimisation_ : According to Ehrgott2005 it is
“a mathematical theory of optimization under multiple objectives”, and can be
encountered in various fields of science, economics, logistics and
engineering. Since this novel concept to forecast evaluation may open up the
avenue to a whole field of applications and research (which is underpinned by
further instances; see Example B.1), we give a concise general outline of the
theory, using partial orders on $\mathbb{R}^{m}$ or even infinite-dimensional
real vector spaces.

For the systemic risk forecasts we consider here, scores mapping to
$\mathbb{R}^{2}$ equipped with the lexicographic (total) order—described in
Subsection 3.3—are sufficient. In particular, the performance of different
systemic risk forecasts must be ranked with regard to the lexicographic order.
Specifically, Theorem 4.4 shows that
$(\operatorname{VaR}_{\beta},\operatorname{CoVaR}_{\alpha|\beta})$,
$(\operatorname{VaR}_{\beta},\operatorname{CoVaR}_{\alpha|\beta},\operatorname{CoES}_{\alpha|\beta})$
and $(\operatorname{VaR}_{\beta},\operatorname{MES}_{\beta})$ are multi-
objective elicitable, and it provides classes of strictly multi-objective
consistent scores. We outline in Section 5 how these scores can be used for
comparative backtests of Diebold–Mariano type. These comparative backtests are
different from—in our case infeasible—“standard” comparative backtests in that
they build on the newly introduced notion of multi-objective elicitability
(with scores mapping to $\mathbb{R}^{2}$ equipped with the lexicographic
order) instead of the “standard” notion of elicitability (with scores mapping
to $\mathbb{R}$ equipped with the canonical order $\leq$). In particular, some
systemic risk forecast is now preferable to some other forecast when the
$\mathbb{R}^{2}$-valued score of the former is smaller (with regard to the
lexicographic order) than that of the latter. Thus, financial institutions may
build on this result to improve their prediction models for
$(\operatorname{VaR}_{\beta},\operatorname{CoVaR}_{\alpha|\beta})$,
$(\operatorname{VaR}_{\beta},\operatorname{CoVaR}_{\alpha|\beta},\operatorname{CoES}_{\alpha|\beta})$
and $(\operatorname{VaR}_{\beta},\operatorname{MES}_{\beta})$, which is
crucial for an adequate assessment of the diverse risks faced by these
institutions.

Since the multi-objective scores of Theorem 4.4 take values in
$\mathbb{R}^{2}$ equipped with the lexicographic order, some particularities
arise for statistical hypothesis tests. While simple “two-sided” null
hypotheses of equal predictive performance can be tested with a classical
Wald-test, particular caution must be taken when testing for superior
predictive ability. Due to the particularities of the lexicographic order, a
straightforward “one-sided” composite null hypothesis would be insensitive to
the systemic risk measure forecast, ignoring the primary goal of the
backtesting procedure. Therefore, we suggest to use “one and a half”-sided
composite null hypotheses, testing for superior predictive ability in the
systemic risk component and equal performance in the auxiliary
$\operatorname{VaR}_{\beta}(X)$ component. Section 5 provides details,
including an adaptation of the Basel framework’s traffic-light approach to
systemic risk backtests.

An empirical application in Section 6 demonstrates the viability of the
comparative backtest. There, we consider daily log-losses of the DAX 30 with
daily log-losses of the S&P 500 as a reference quantity. We compare systemic
risk forecasts derived from a benchmark Gaussian copula model with those
produced by a $t$-copula model, where in both models the correlation parameter
of the copula is driven by generalised autoregressive score (GAS) dynamics
(CKL13). We find that the predictive performance of the $t$-copula is superior
with $p$-values close to $3\%$, which is consistent with its popularity in
empirical work. One conclusion from our empirical analysis is that fairly long
samples are required to validly distinguish between different forecasts,
because the effective sample sizes in comparing systemic risk forecasts are
(almost by definition) reduced. Thus, the one year evaluation period for
(univariate) VaR and ES forecasts in the Basel framework of the BCBSBF19 is,
in our view, insufficient for systemic risk forecasts.

The paper closes with a discussion and outlook (Section 7). Besides the parts
already mentioned above, the Supplement provides proofs for the results of
Section 3 (Section A) and further background material on multi-objective
elicitability (Section B). All other proofs are relegated to Section E.
Section F investigates the finite-sample properties of our comparative
backtests in simulations. The R code to reproduce all numerical experiments is
available online.

Throughout the paper, we indicate vectors with bold letters. We highlight the
distinction between row and column vectors only when it is essential, and use
the symbol ′ to indicate the transpose of a vector or matrix.

##  2 Formal definition of CoVaR, CoES and MES

Fix some non-atomic probability space $(\Omega,\mathfrak{A},\operatorname{P})$
where all random objects are defined. Using standard notation, let
$L^{0}(\mathbb{R}^{d})$, $d=1,2$, be the space of all $\mathbb{R}^{d}$-valued
random vectors on $(\Omega,\mathfrak{A},\operatorname{P})$. Furthermore, for
$p\in[1,\infty)$, let $L^{p}(\mathbb{R}^{d})\subseteq L^{0}(\mathbb{R}^{d})$
be the collection of random vectors whose components possess a finite $p$th
moment. For $\bm{X}\in L^{0}(\mathbb{R}^{d})$ let $F_{\bm{X}}$ be its joint
distribution. Then define for $p\in\\{0\\}\cup[1,\infty)$ the collection
$\mathcal{F}^{p}(\mathbb{R}^{d})\mathrel{\mathop{\mathchar
58\relax}}=\\{F_{\bm{X}}\colon\bm{X}\in L^{p}(\mathbb{R}^{d})\\}$. We overload
notation and identify any $F\in\mathcal{F}^{0}(\mathbb{R}^{d})$ with its
cumulative distribution function (cdf) $\mathbb{R}^{d}\to[0,1]$.

Our systemic risk measures of interest—$\operatorname{CoVaR}$,
$\operatorname{CoES}$ and $\operatorname{MES}$—are maps from
$L^{0}(\mathbb{R}^{2})$ (or $L^{1}(\mathbb{R}^{2})$ for MES) to
$\mathbb{R}^{*}\mathrel{\mathop{\mathchar 58\relax}}=(-\infty,\infty]$. They
are law-determined, meaning that their values for $(X,Y)$ and
$(\tilde{X},\tilde{Y})$ coincide if $F_{X,Y}=F_{\tilde{X},\tilde{Y}}$. Hence,
we can consider them as risk-functionals on $\mathcal{F}^{0}(\mathbb{R}^{2})$
(or $\mathcal{F}^{1}(\mathbb{R}^{2})$). Similarly, the popular univariate risk
measure $\operatorname{VaR}_{\beta}$, $\beta\in[0,1]$, is a law-determined map
$L^{0}(\mathbb{R})\to[-\infty,\infty]$. In the rest of the paper, we will
frequently overload notation and identify these law-determined risk measures
with their induced risk functionals. As such, we will use the terms ‘risk
measure’ and ‘risk functional’ interchangeably.

Let $(X,Y)\in L^{0}(\mathbb{R}^{2})$ be a two-dimensional random vector. Here,
$Y$ stands for the losses of a position of interest (with the sign convention
that positive values are losses and negative values are gains) and $X$ is a
univariate reference position or aggregate of a reference system, having the
same sign convention. Denote by $F_{X,Y}$ their joint distribution function
and by $F_{X}$ and $F_{Y}$ their marginals, respectively. Recall that for
$\beta\in[0,1]$ the $\beta$-quantile of $F_{X}$ is the closed interval
$q_{\beta}(F_{X})=\\{x\in\mathbb{R}\colon F_{X}(x-)\leq\beta\leq F_{X}(x)\\}$,
where $F_{X}(x-)\mathrel{\mathop{\mathchar 58\relax}}=\lim_{t\uparrow
x}F_{X}(x-)$. Then, $\operatorname{VaR}_{\beta}(X)$ is the lower
$\beta$-quantile of $F_{X}$, i.e.,
$\operatorname{VaR}_{\beta}(X)\mathrel{\mathop{\mathchar
58\relax}}=\operatorname{VaR}_{\beta}(F_{X})\mathrel{\mathop{\mathchar
58\relax}}=\inf q_{\beta}(F_{X})$. For $\beta\in(0,1)$,
$\operatorname{VaR}_{\beta}$ is always finite. Our sign convention is such
that the larger the risk measure of a position, the riskier it is deemed.
Hence, we typically choose a probability level of $\beta$ close to 1 for
$\operatorname{VaR}_{\beta}$, such as $\beta=0.95$ or $\beta=0.99$.

AB16 define $\operatorname{CoVaR}_{\beta}$ as the $\beta$-quantile of the
conditional distribution function $F_{Y}(\ \cdot\mid
X=\operatorname{VaR}_{\beta}(X))=\operatorname{P}\\{Y\leq\cdot\mid
X=\operatorname{VaR}_{\beta}(X)\\}$. The conditioning event in this definition
is problematic for several reasons. First, it may have probability zero (which
is the case when $F_{X}$ is continuous). Second, it does not fully capture the
tail-risk of $F_{X}$. Third, since the roles of $Y$ and $X$ are asymmetric by
construction, one may want to consider different probability levels to specify
the event of ‘being in distress’. Thus, we follow GT13 and Banulescu-
RaduETAL2019 in redefining $\operatorname{CoVaR}_{\alpha|\beta}\colon
L^{0}(\mathbb{R}^{2})\to\mathbb{R}$ for $\alpha\in(0,1)$, $\beta\in[0,1)$ as

$\operatorname{CoVaR}_{\alpha|\beta}(Y|X)\mathrel{\mathop{\mathchar
58\relax}}=\operatorname{CoVaR}_{\alpha|\beta}(F_{X,Y})\mathrel{\mathop{\mathchar
58\relax}}=\operatorname{VaR}_{\alpha}(F_{Y\mid
X\geq\operatorname{VaR}_{\beta}(X)}),$ (2.1)

where $F_{Y\mid
X\geq\operatorname{VaR}_{\beta}(X)}=\operatorname{P}\\{Y\leq\cdot\mid
X\geq\operatorname{VaR}_{\beta}(X)\\}$. For $\beta=0$, we simply have
$\operatorname{CoVaR}_{\alpha|0}(Y|X)\mathrel{\mathop{\mathchar
58\relax}}=\operatorname{VaR}_{\alpha}(Y)$, and if $\beta=\alpha$ we simply
write
$\operatorname{CoVaR}_{\alpha}(Y|X)=\operatorname{CoVaR}_{\alpha|\alpha}(Y|X)$.

Since $\operatorname{CoVaR}_{\alpha|\beta}(Y|X)$ is merely a quantile of the
distribution $F_{Y\mid X\geq\operatorname{VaR}_{\beta}(X)}$, it inherits the
same defects as $\operatorname{VaR}_{\alpha}(Y)$. That is, it ignores tail
risks beyond the quantile level $\alpha$ and it fails to be coherent,
particularly defying the rationale of advantageous diversification effects
(Aea99; MS14). The well-known Expected Shortfall at level $\alpha\in(0,1)$,
$\operatorname{ES}_{\alpha}(Y)\mathrel{\mathop{\mathchar
58\relax}}=\operatorname{ES}_{\alpha}(F_{Y})\mathrel{\mathop{\mathchar
58\relax}}=\frac{1}{1-\alpha}\int_{\alpha}^{1}\operatorname{VaR}_{\gamma}(Y)\mathrm{d}\gamma\in\mathbb{R}^{*}$,
does not suffer from these defects. This motivates AB16 to introduce the
conditional Expected Shortfall (CoES). As for
$\operatorname{CoVaR}_{\alpha|\beta}$, we modify the conditioning event and
formally introduce for $\alpha\in(0,1)$, $\beta\in[0,1)$,
$\operatorname{CoES}_{\alpha|\beta}\colon
L^{0}(\mathbb{R}^{2})\to\mathbb{R}^{*}$ via

$\displaystyle\operatorname{CoES}_{\alpha|\beta}(Y|X)\mathrel{\mathop{\mathchar
58\relax}}=\operatorname{CoES}_{\alpha|\beta}(F_{X,Y})\mathrel{\mathop{\mathchar
58\relax}}=\frac{1}{1-\alpha}\int_{\alpha}^{1}\operatorname{CoVaR}_{\gamma|\beta}(Y|X)\mathrm{d}\gamma.$
(2.2)

If $F_{X,Y}$ is continuous, then
$\operatorname{CoES}_{\alpha|\beta}(Y|X)=\operatorname{E}[Y|Y\geq\operatorname{CoVaR}_{\alpha|\beta}(Y|X),\
X\geq\operatorname{VaR}_{\beta}(X)]$. Again,
$\operatorname{CoES}_{\alpha|0}(Y|X)=\operatorname{ES}_{\alpha}(Y)$, and we
write $\operatorname{CoES}_{\alpha}(Y|X)\mathrel{\mathop{\mathchar
58\relax}}=\operatorname{CoES}_{\alpha|\alpha}(Y|X)$.

Finally, we consider the marginal Expected Shortfall (MES) of Aea17, which
measures the expectation of $Y$ when $X$ is in distress, i.e., when $X$ is in
its right tail. Specifically, we introduce for $\beta\in[0,1)$ the map
$\operatorname{MES}_{\beta}\colon L^{1}(\mathbb{R}^{2})\to\mathbb{R}$,

$\operatorname{MES}_{\beta}(Y|X)\mathrel{\mathop{\mathchar
58\relax}}=\operatorname{MES}_{\beta}(F_{X,Y})\mathrel{\mathop{\mathchar
58\relax}}=\operatorname{CoES}_{0|\beta}(Y|X)=\int_{0}^{1}\operatorname{CoVaR}_{\gamma|\beta}(Y|X)\mathrm{d}\gamma.$
(2.3)

Again, $\operatorname{MES}_{0}(Y|X)=\operatorname{E}[Y]$, and
$\operatorname{MES}_{\beta}(Y|X)=\operatorname{E}[Y|X\geq\operatorname{VaR}_{\beta}(X)]$
if $F_{X,Y}$ is continuous. If $F_{X}$ is discontinuous, Remark C.1 proposes a
novel correction term which generalises the three measures considered in this
paper.

##  3 (Conditional) identifiability and multi-objective elicitability

We present the theory in this section in all generality to serve as a basis
for future research on multi-objective scores. Therefore, we work with general
functionals which do not necessarily have the interpretation of risk
functionals. All proofs are in Section A.

###  3.1 Notation, basic definitions and results

Adopting the decision-theoretic terminology of Gne11, we denote by
$\mathsf{A}$ an action domain. This is the space of plausible forecasts, which
can be finite for categorial forecasts, $\mathbb{R}$ or $\mathbb{R}^{k}$ for
point forecasts, or a set of distributions for probabilistic forecasts.
Moreover, let $\mathsf{O}$ be an observation domain—a set where verifying
observations materialise—with $\mathsf{O}=\mathbb{R}^{d}$ as a leading
example. Denote by $\mathcal{F}^{0}(\mathsf{O})$ the set of all probability
distributions on $\mathsf{O}$. Let $\mathcal{F}\subseteq\mathcal{F}^{\prime}$
be subclasses of $\mathcal{F}^{0}(\mathsf{O})$. We consider a general,
possibly set-valued functional
$\bm{T}\colon\mathcal{F}^{\prime}\to\mathcal{P}(\mathsf{A})$ with
$\mathcal{P}(\mathsf{A})$ the power set of $\mathsf{A}$. Later on, $\bm{T}$
will have the interpretation of a risk functional. Note that
$\mathcal{F}^{\prime}$ is the class of distributions, where our functional
$\bm{T}$ is defined on, and $\mathcal{F}\subseteq\mathcal{F}^{\prime}$ is the
subclass on which $\bm{T}$ will be identifiable/elicitable (see Definitions
3.1 and 3.2). For instance, for
$\bm{T}(F_{X,Y})=(\operatorname{VaR}_{\beta}(F_{X}),$
$\operatorname{CoVaR}_{\alpha|\beta}(F_{X,Y}))$, we have
$\mathcal{F}^{\prime}=\mathcal{F}^{0}(\mathbb{R}^{2})$ and $\mathcal{F}$ is
given in Theorem 4.2 (i)/Theorem 4.4 (ii).

We adopt the _selective_ notion of forecasts discussed in FFHR2021 where one
is content with correctly specifying a single element
$\bm{t}\in\bm{T}(F)\subseteq\mathsf{A}$ as opposed to specifying the entire
set $\bm{T}(F)$. (If one is interested in _exhaustive_ forecasts—i.e., in
forecasts for the whole set $\bm{T}(F)$—one can change the action domain to
$\mathcal{P}(\mathsf{A})$.) If $\bm{T}$ attains singletons only, we identify
the value of $\bm{T}(F)$ with its unique element. This identification allows
us to treat point-valued functionals as set-valued functionals without loss of
generality. We mention that the risk functionals to be considered in Section 4
are all point-valued.

A function
$\bm{g}\colon\mathsf{A}\times\mathsf{O}\to\mathbb{R}^{\mathcal{I}}$, where
$\mathcal{I}$ is an index set, is called $\mathcal{F}$-integrable if for all
components $g_{i}$, $i\in\mathcal{I}$, it holds that
$\int|g_{i}(\bm{r},\bm{y})|\,\mathrm{d}F(\bm{y})<\infty$ for all
$\bm{r}\in\mathsf{A}$, $F\in\mathcal{F}$. If $\bm{g}$ is
$\mathcal{F}$-integrable, we define the map
$\bar{\bm{g}}\colon\mathsf{A}\times\mathcal{F}\to\mathbb{R}^{\mathcal{I}}$,
$\bar{\bm{g}}(\bm{r},F)\mathrel{\mathop{\mathchar 58\relax}}=\int
g_{i}(\bm{r},\bm{y})\,\mathrm{d}F(\bm{y})$ for $\bm{r}\in\mathsf{A}$,
$F\in\mathcal{F}$. A similar convention and notation is used for maps
$\bm{a}\colon\mathsf{O}\to\mathbb{R}^{\mathcal{I}}$. We start with the
classical definition of elicitability and consistent scoring functions,
mapping to $\mathbb{R}$.

###### Definition 3.1.

An $\mathcal{F}$-integrable map
$S\colon\mathsf{A}\times\mathsf{O}\to\mathbb{R}$ is an _$\mathcal{F}$
-consistent scoring function_ for $\bm{T}$ if
$\bar{S}(\bm{t},F)\leq\bar{S}(\bm{r},F)$ for all $\bm{t}\in\bm{T}(F)$, all
$\bm{r}\in\mathsf{A}$ and for all $F\in\mathcal{F}$. It is a _strictly_
$\mathcal{F}$-consistent scoring function for $\bm{T}$ if, additionally,
$\bar{S}(\bm{t},F)=\bar{S}(\bm{r},F)$ implies that $\bm{r}\in\bm{T}(F)$.
$\bm{T}$ is _elicitable on $\mathcal{F}$_ if there is a strictly
$\mathcal{F}$-consistent scoring function for $\bm{T}$.

###### Definition 3.2.

An $\mathcal{F}$-integrable map
$\bm{V}\colon\mathsf{A}\times\mathsf{O}\to\mathbb{R}^{m}$ is an _$\mathcal{F}$
-identification function_ for $\bm{T}$ if $\bar{\bm{V}}(\bm{t},F)=\bm{0}$ for
all $\bm{t}\in\bm{T}(F)$ and for all $F\in\mathcal{F}$. It is a _strict_
$\mathcal{F}$-identification function for $\bm{T}$ if, additionally, for all
$F\in\mathcal{F}$ and for all $\bm{r}\in\mathsf{A}$,
$\bar{\bm{V}}(\bm{r},F)=\bm{0}$ implies that $\bm{r}\in\bm{T}(F)$. $\bm{T}$ is
_identifiable on $\mathcal{F}$_ if there is a strict
$\mathcal{F}$-identification function for $\bm{T}$.

Suppose in a risk management context that $\bm{T}$ corresponds to the
$\operatorname{VaR}_{\alpha}$-functional and $y_{t}$ are the observed losses.
Subject to mild conditions on $\mathcal{F}$, $\operatorname{VaR}_{\alpha}$ is
elicitable, where the ‘pinball loss’
$S(r,y)=(\mathds{1}\\{y>r\\}-1+\alpha)(r-y)$ is strictly
$\mathcal{F}$-consistent. This allows to compare competing VaR forecasts
$(r_{t,(1)})_{t=1,\ldots,n}$ and $(r_{t,(2)})_{t=1,\ldots,n}$ via their
empirical average score differences
$\overline{d}_{n}=\overline{S}_{1n}-\overline{S}_{2n}=\frac{1}{n}\sum_{t=1}^{n}S(r_{t,(1)},y_{t})-S(r_{t,(2)},y_{t})$.
A negative (positive) sign of $\overline{d}_{n}$ indicates superiority
(inferiority) of $(r_{t,(1)})_{t=1,\ldots,n}$ over
$(r_{t,(2)})_{t=1,\ldots,n}$. Identifiability, on the other hand, opens the
way to test for calibration by checking, e.g., if the test statistic
$\frac{1}{n}\sum_{t=1}^{n}\bm{V}(r_{t,(i)},y_{t})$ is sufficiently close to
$\bm{0}$ or not. E.g., when $\bm{T}$ corresponds to
$\operatorname{VaR}_{\alpha}$ checking calibration amounts to checking if the
empirical VaR-violation rate is roughly $1-\alpha$, which can be done in terms
of $V(r,y)=\mathds{1}\\{y>r\\}-(1-\alpha)$. These examples demonstrate the
importance of elicitability and identifiability for comparing and evaluating
(risk) forecasts in practice.

Under regularity conditions, the notions of elicitability and identifiability
are equivalent for point-valued functionals mapping to $\mathbb{R}$
(SteinwartPasinETAL2014). There are also important functionals which fail to
be elicitable and identifiable, most prominently the variance and expected
shortfall (Gne11). In such situations, the notions of _conditional_
elicitability and _conditional_ identifiability can be helpful. Following the
concept presented in FZ16a, we slightly adapt EKT15’s (EKT15) original
definition of conditional elicitability. We also introduce the corresponding
counterpart of conditional identifiability.

###### Definition 3.3.

Consider two functionals $\bm{T}_{j}\mathrel{\mathop{\mathchar
58\relax}}\mathcal{F}^{\prime}\to\mathcal{P}(\mathsf{A}_{j})$, $j=1,2$, and
let $\mathcal{F}\subseteq\mathcal{F}^{\prime}$.

  1. (i)

$\bm{T}_{2}$ is conditionally elicitable with $\bm{T}_{1}$ on $\mathcal{F}$,
if $\bm{T}_{1}$ is elicitable on $\mathcal{F}$ and $\bm{T}_{2}$ is elicitable
on $\mathcal{F}_{\bm{r}_{1}}\mathrel{\mathop{\mathchar
58\relax}}=\left\\{F\in\mathcal{F}\colon\ \bm{r}_{1}\in\bm{T}_{1}(F)\right\\}$
for any $\bm{r}_{1}\in\mathsf{A}_{1}$.

  2. (ii)

$\bm{T}_{2}$ is conditionally identifiable with $\bm{T}_{1}$ on $\mathcal{F}$,
if $\bm{T}_{1}$ is identifiable on $\mathcal{F}$ and $\bm{T}_{2}$ is
identifiable on $\mathcal{F}_{\bm{r}_{1}}\mathrel{\mathop{\mathchar
58\relax}}=\left\\{F\in\mathcal{F}\colon\ \bm{r}_{1}\in\bm{T}_{1}(F)\right\\}$
for any $\bm{r}_{1}\in\mathsf{A}_{1}$.

It is easy to see that the variance is conditionally elicitable and
conditionally identifiable with the mean, and that
$\operatorname{ES}_{\alpha}$ is conditionally elicitable and conditionally
identifiable with $\operatorname{VaR}_{\alpha}$ on appropriate classes of
distributions, respectively (EKT15). The pairs (mean, variance) and
$(\operatorname{VaR}_{\alpha},\operatorname{ES}_{\alpha})$ even turn out to be
elicitable and identifiable (FZ16a). For identifiability, this is an instance
of the following proposition, which is already stated without a proof in the
discussion of FZ16a.

######  Proposition 3.4.

If $\bm{T}_{2}$ is conditionally identifiable with $\bm{T}_{1}$ on
$\mathcal{F}$, then the pair $(\bm{T}_{1},\bm{T}_{2})$ is identifiable on
$\mathcal{F}$.

Of course, an analogue to Proposition 3.4 for (conditional) elicitability
would be desirable, and it has been stated as an open problem in the
discussion of FZ16a. Unfortunately, the answer is negative: While Section C
establishes the conditional elicitability of
$\operatorname{CoVaR}_{\alpha|\beta}$,
$(\operatorname{CoVaR}_{\alpha|\beta},\operatorname{CoES}_{\alpha|\beta})$ and
$\operatorname{MES}_{\beta}$ all with $\operatorname{VaR}_{\beta}$, the
corresponding pairs and the triplet generally fail to be elicitable; see
Section D.

###  3.2 Multi-objective scores, consistency and elicitability

To overcome the structural drawback of elicitability in comparison to
identifiability, in particular the lack of an analogue to Proposition 3.4, we
introduce the novel notion of _multi-objective scoring functions_ and the
corresponding concepts of _multi-objective consistency_ and _multi-objective
elicitability_. It is inspired by the fundamental observation that
identification functions are generally multivariate.

To be more precise, the dimension $m$ of the identification function $\bm{V}$
usually coincides with the dimension $k$ of the functional. If $k=m=1$ an
identification function is often induced by the derivative of a consistent
scoring function $S$. Also, the antiderivative of an (oriented) identification
function yields a consistent score, thus roughly establishing a one-to-one
correspondence between the class of identification functions and the class of
consistent scoring functions. For a $k$-dimensional functional, the gradient
of a consistent score is $\mathbb{R}^{k}$-valued and naturally induces an
identification function, subject to smoothness conditions. However, not every
$k$-dimensional identification function possesses an antiderivative for $k\geq
2$. This is due to integrability conditions asserting that if it was
integrable, the corresponding Hessian of the stipulated antiderivative would
need to be symmetric (see also Example D.1 for an illustration). This rules
out the one-to-one relation in the multivariate setting, giving rise to a
_gap_ between the class of consistent scoring functions and the one of
identification functions. This gap and its consequences on estimation are
discussed in detail by DFZ2020.

We sidestep this integrability constraint by introducing the concept of
multivariate scoring functions. Indeed, it is this multivariate structure of
identification functions which facilitates the straightforward proof of
Proposition 3.4. Therefore, we mimic this multi-dimensionality for scores,
letting them map to $\mathbb{R}^{m}$, where usually $m=k$, or even more
generally to some real vector space $\mathbb{R}^{\mathcal{I}}$, where the
index set ${\mathcal{I}}$ may be finite, countable or even uncountable. In the
application of the general theory developed here to the systemic risk
measures, it suffices to consider $\mathbb{R}^{2}$-valued scores; see Theorem
4.4 in Section 4. The motivation for defining a classical score as a
univariate map to $\mathbb{R}$ (see Definition 3.1) is grounded in tradition
on the one hand. On the other hand, $\mathbb{R}$ is equipped with the
canonical (total) order relation $\leq$, which allows for straightforward
comparisons of forecasts by checking whether the empirical average score
differences satisfy $\overline{d}_{n}\geq 0$ or $\overline{d}_{n}\leq 0$.
Consistency in Definition 3.1 ultimately relies on the existence of an order.

Hence, we need to equip $\mathbb{R}^{\mathcal{I}}$ with a vector partial order
$\preceq$ and write $(\mathbb{R}^{\mathcal{I}},\preceq)$. A binary relation
$\preceq$ on $\mathbb{R}^{\mathcal{I}}$ is a _partial order_ if it is
reflexive ($\forall\bm{x}\in\mathbb{R}^{\mathcal{I}}$, $\bm{x}\preceq\bm{x}$),
antisymmetric ($\forall\bm{x},\bm{y}\in\mathbb{R}^{\mathcal{I}}$ if
$\bm{x}\preceq\bm{y}$ and $\bm{y}\preceq\bm{x}$, then $\bm{x}=\bm{y}$),
transitive ($\forall\bm{x},\bm{y},\bm{z}\in\mathbb{R}^{\mathcal{I}}$ if
$\bm{x}\preceq\bm{y}$ and $\bm{y}\preceq\bm{z}$, then $\bm{x}\preceq\bm{z}$).
Two elements $\bm{x},\bm{y}\in\mathbb{R}^{\mathcal{I}}$ are _comparable_ if
$\bm{x}\preceq\bm{y}$ or $\bm{y}\preceq\bm{x}$. A _total_ order is a partial
order where all elements are comparable. A partial order $\preceq$ on
$\mathbb{R}^{\mathcal{I}}$ induces a _strict_ partial order $\prec$ on
$\mathbb{R}^{\mathcal{I}}$ as follows. For all
$\bm{x},\bm{y}\in\mathbb{R}^{\mathcal{I}}$ it holds that $\bm{x}\prec\bm{y}$
if and only if $\bm{x}\preceq\bm{y}$ and $\bm{x}\neq\bm{y}$. A partial order
$\preceq$ on $\mathbb{R}^{\mathcal{I}}$ is a _vector partial order_ if it is
compatible with addition and positive scaling. That is, if for all
$\bm{x},\bm{y},\bm{z}\in\mathbb{R}^{\mathcal{I}}$ and for all
$\lambda\in(0,\infty)$ it holds that $\bm{x}\preceq\bm{y}$ implies that
$\bm{x}+\bm{z}\preceq\bm{y}+\bm{z}$, and $\bm{x}\preceq\bm{y}$ implies that
$\lambda\bm{x}\preceq\lambda\bm{y}$. (In the sequel, we always mean a vector
partial order whenever we write “partial order”.) The canonical choice is the
componentwise order defined for
$\bm{x}=(x_{i})_{i\in\mathcal{I}},\bm{y}=(y_{i})_{i\in\mathcal{I}}\in\mathbb{R}^{\mathcal{I}}$
as $\bm{x}\preceq\bm{y}$ if and only if $x_{i}\leq y_{i}$ for all
$i\in\mathcal{I}$. The use of a partial order $\preceq$ on
$\mathbb{R}^{\mathcal{I}}$ leads to two different generalisations of
Definition 3.1. These are due to the fact that in a partial order
$\bm{x}\preceq\bm{y}$ implies that $\bm{y}\nprec\bm{x}$, but the reverse
implication fails if $\bm{x}$ and $\bm{y}$ are not comparable.

###### Definition 3.5.

  1. (i)

An $\mathcal{F}$-integrable map
$\bm{S}\colon\mathsf{A}\times\mathsf{O}\to(\mathbb{R}^{\mathcal{I}},\preceq)$
is a _strongly_ multi-objective $\mathcal{F}$-consistent scoring function for
$\bm{T}\colon\mathcal{F}\to\mathcal{P}(\mathsf{A})$ if
$\bar{\bm{S}}(\bm{t},F)\preceq\bar{\bm{S}}(\bm{r},F)$ for all
$\bm{t}\in\bm{T}(F)$, all $\bm{r}\in\mathsf{A}$ and all $F\in\mathcal{F}$. It
is a strictly strongly multi-objective $\mathcal{F}$-consistent scoring
function for $\bm{T}$ if, additionally,
$\bar{\bm{S}}(\bm{t},F)=\bar{\bm{S}}(\bm{r},F)$ implies that
$\bm{r}\in\bm{T}(F)$. $\bm{T}$ is _strongly multi-objective-elicitable on
$\mathcal{F}$_ (with respect to the order $\preceq$ on
$\mathbb{R}^{\mathcal{I}}$) if there is a strictly strongly multi-objective
$\mathcal{F}$-consistent scoring function for $\bm{T}$.

  2. (ii)

An $\mathcal{F}$-integrable map
$\bm{S}\colon\mathsf{A}\times\mathsf{O}\to(\mathbb{R}^{\mathcal{I}},\preceq)$
is a _weakly_ multi-objective $\mathcal{F}$-consistent scoring function for
$\bm{T}\colon\mathcal{F}\to\mathcal{P}(\mathsf{A})$ if
$\bar{\bm{S}}(\bm{r},F)\nprec\bar{\bm{S}}(\bm{t},F)$ for all
$\bm{t}\in\bm{T}(F)$, all $\bm{r}\in\mathsf{A}$ and all $F\in\mathcal{F}$. It
is a strictly weakly multi-objective $\mathcal{F}$-consistent scoring function
for $\bm{T}$ if, additionally, for all $F\in\mathcal{F}$ and for any
$\bm{r}_{0}\in\mathsf{A}$ it holds that if
$\bar{\bm{S}}(\bm{r},F)\nprec\bar{\bm{S}}(\bm{r}_{0},F)$ for all
$\bm{r}\in\mathsf{A}$, then $\bm{r}_{0}\in\bm{T}(F)$. $\bm{T}$ is _weakly
multi-objective-elicitable on $\mathcal{F}$_ (with respect to the order
$\preceq$ on $\mathbb{R}^{\mathcal{I}}$) if there is a strictly weakly multi-
objective $\mathcal{F}$-consistent scoring function for $\bm{T}$.

As discussed, a partial order on $\mathbb{R}^{\mathcal{I}}$ is sufficient to
define multi-objective consistency. Using the notation
$\bar{\bm{S}}(B,F)\mathrel{\mathop{\mathchar
58\relax}}=\\{\bar{\bm{S}}(\bm{r},F)\colon\bm{r}\in B\\}$ for
$F\in\mathcal{F}$ and $B\subseteq\mathsf{A}$, the definition of multi-
objective consistency does not require comparability of all elements
$\bar{\bm{S}}(\mathsf{A},F)$ for some fixed $F\in\mathcal{F}$. Weak multi-
objective consistency solely implies that all elements of
$\bar{\bm{S}}(\bm{T}(F),F)$ are minimal in $\bar{\bm{S}}(\mathsf{A},F)$. The
strict version additionally ensures that all minimal elements of
$\bar{\bm{S}}(\mathsf{A},F)$ are in $\bar{\bm{S}}(\bm{T}(F),F)$. Strong multi-
objective consistency additionally means that the elements of
$\bar{\bm{S}}(\bm{T}(F),F)$ are not only minimal in
$\bar{\bm{S}}(\mathsf{A},F)$, but that they are the (unique) least element of
$\bar{\bm{S}}(\mathsf{A},F)$. Due to the uniqueness of a least element,
$\bar{\bm{S}}(\bm{T}(F),F)$ is a singleton. The strict version of strong
multi-objective consistency additionally means that
$\bar{\bm{S}}(\bm{T}(F),F)\not\subset\bar{\bm{S}}(\mathsf{A}\setminus\bm{T}(F),F)$.
See Figure 1 for an illustration of the two situations. Clearly, (strict)
strong consistency implies (strict) weak consistency. If we omit the
qualifiers “weak” or “strong”, we refer to the strong version.

Figure 1: In both panels, the shaded areas correspond to
$\bar{\bm{S}}(\mathsf{A},F)$ of a multi-objective score $\bm{S}$ mapping to
$\mathbb{R}^{2}$ equipped with the componentwise order. The red sets
correspond to $\bar{\bm{S}}(\bm{T}(F),F)$. In panel (a), this is the unique
least element, illustrating the situation of strict strong multi-objective
consistency. In panel (b), this set corresponds to all minimal elements,
depicting the situation of strict weak multi-objective elicitability.

Multi-objective consistency translates the essence of consistency as a ‘truth
serum’, or being incentive compatible, to the multivariate realm. No matter
what forecast $\bm{r}\in\mathsf{A}$ an agent issues, they would be better off
(or at least not worse off) in expectation when issuing a correctly specified
functional value $\bm{t}\in\bm{T}(F)$. (In the weak version, if an agent
issued some $\bm{t}\in\bm{T}(F)$, they would not be better off with some other
forecast.) Strictness means that any action
$\bm{r}\in\mathsf{A}\setminus\bm{T}(F)$ leads to a worse outcome in
expectation than the truth $\bm{t}\in\bm{T}(F)$. So this minimal requirement
of honouring truthful forecasts is preserved.

###### Remark 3.6.

To the best of our knowledge, the notion of multi-objective scoring functions
with the related concepts is novel to the forecast evaluation literature.
However, it has some connections to the concept of _forecast dominance_
introduced by EhmETAL2016. Let $(S_{i})_{i\in\mathcal{I}}$ be some class of
univariate (strictly) $\mathcal{F}$-consistent scoring functions
$S_{i}\colon\mathsf{A}\times\mathsf{O}\to\mathbb{R}$ for some functional
$\bm{T}\colon\mathcal{F}\to\mathcal{P}(\mathsf{A})$. Adapting their definition
slightly, we say that a forecast $\bm{r}\in\mathsf{A}$ dominates
$\tilde{\bm{r}}\in\mathsf{A}$ relative to $(S_{i})_{i\in\mathcal{I}}$ if
$\bar{S}_{i}(\bm{r},F)\leq\bar{S}_{i}(\tilde{\bm{r}},F)$ for all
$F\in\mathcal{F}$ and for all $i\in\mathcal{I}$. In our terminology, we could
phrase this forecast ranking in terms of a multivariate score
$\bm{S}\colon\mathsf{A}\times\mathsf{O}\to(\mathbb{R}^{\mathcal{I}},\preceq)$,
$\bm{S}(\bm{r},\bm{y})=\big{(}S_{i}(\bm{r},\bm{y})\big{)}_{i\in\mathcal{I}}$,
where $\preceq$ is the usual componentwise partial order. By virtue of mixture
representations, EhmETAL2016 impressively demonstrate that for quantiles and
expectiles, one can rephrase forecast dominance with respect to (nearly) all
consistent scoring functions equivalently in terms of extremal or elementary
scores $(S_{\theta})_{\theta\in\mathbb{R}}\subseteq(S_{i})_{i\in\mathcal{I}}$.
For this situation, we can easily construct a multi-objective score
$\bm{S}=(S_{\theta})_{\theta\in\mathbb{R}}$, mapping to the function space
$\mathbb{R}^{\mathbb{R}}$ equipped with the componentwise order. EhmETAL2016
provide numerous instances of comparable and incomparable forecast rankings.
This example provides an easy construction for a (strictly) multi-objective
$\mathcal{F}$-consistent score, $\bm{S}=(S_{\theta})_{\theta\in\mathbb{R}}$,
if $\bm{T}$ is elicitable.

For verifying observations $(\bm{y}_{t})_{t=1,\ldots,n}$, our Definition 3.5
suggests to compare two sequences of forecasts
$(\bm{r}_{t,(i)})_{t=1,\ldots,n}$ ($i=1,2$) via their average scores
$\overline{\bm{S}}_{1n}=\frac{1}{n}\sum_{t=1}^{n}\bm{S}(\bm{r}_{t,(1)},\bm{y}_{t})$
and
$\overline{\bm{S}}_{2n}=\frac{1}{n}\sum_{t=1}^{n}\bm{S}(\bm{r}_{t,(2)},\bm{y}_{t})$.
However, while using classical univariate scores always leads to a conclusive
forecast ranking (ignoring questions of statistical significance for a
moment), the presence of a _partial_ order may lead to inconclusive rankings,
particularly if neither of the two forecasts is correctly specified. For
instance, if $\overline{\bm{S}}_{1n}=(1,4)$ and $\overline{\bm{S}}_{2n}=(4,1)$
(e.g., for the bivariate systemic risk scores of Theorem 4.4), then neither
$\overline{\bm{S}}_{1n}\preceq\overline{\bm{S}}_{2n}$ nor
$\overline{\bm{S}}_{2n}\preceq\overline{\bm{S}}_{1n}$ in the componentwise
order on $\mathbb{R}^{2}$.

###  3.3 Multi-objective elicitability with respect to the lexicographic order
and conditional elicitability

To overcome the issue of inconclusive forecast rankings, we equip
$\mathbb{R}^{\mathcal{I}}$ with a total order. In a total order, all elements
are comparable. Hence, the notions of weak and strong multi-objective
consistency / elicitability coincide and a distinction is redundant. A
promising choice is the lexicographic order. For our purposes, and to ease the
exposition, it suffices to consider $\mathbb{R}^{2}$ equipped with the
lexicographic order $\preceq_{\mathrm{lex}}$. On $\mathbb{R}^{2}$ it holds
that $(x_{1},x_{2})\preceq_{\mathrm{lex}}(y_{1},y_{2})$ if $x_{1}<y_{1}$ or if
($x_{1}=y_{1}$ and $x_{2}\leq y_{2}$). The lexicographic order is widely used
for preference relations in microeconomics. In particular, it is well-known
for the fact that it cannot be represented by a real-valued utility function
(MWG1995, Chapter 3.C).

There are at least two reasons for our choice of the lexicographic order.
First, as a total order, it allows for conclusive forecast rankings, which we
exploit in our comparative backtests in Section 5. For instance, if as above
$\overline{\bm{S}}_{1n}=(1,4)$ and $\overline{\bm{S}}_{2n}=(4,1)$ for the
systemic risk scores, then the inconclusive ranking is resolved because
$\overline{\bm{S}}_{1n}\preceq_{\mathrm{lex}}\overline{\bm{S}}_{2n}$. Second,
the lexicographic order opens the way to the following analogue of Proposition
3.4.

######  Theorem 3.7.

If $\bm{T}_{2}$ is conditionally elicitable with $\bm{T}_{1}$ on $\mathcal{F}$
and $\bm{T}_{1}(F)$ is a singleton for all $F\in\mathcal{F}$, then the pair
$(\bm{T}_{1},\bm{T}_{2})$ is multi-objective elicitable on $\mathcal{F}$ with
respect to the lexicographic order $\preceq_{\mathrm{lex}}$ on
$\mathbb{R}^{2}$.

Theorem 3.7 is almost a direct analogue of Proposition 3.4, with the
intriguing exception that $\bm{T}_{1}$ is assumed to be a singleton on
$\mathcal{F}$ in Theorem 3.7; see Remark A.1 for further details. We
illustrate how the construction of Theorem 3.7 leads to strictly consistent
multi-objective scores mapping to $(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$
for the cases of (mean, variance) and
$(\operatorname{VaR}_{\alpha},\operatorname{ES}_{\alpha})$ in Subsection B.1.
There, we also provide further examples of conditionally elicitable
functionals failing to be elicitable in the traditional sense, but which are
multi-objective elicitable with respect to
$(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$. Section B.2 elaborates on how the
multi-objective scores can be considered a “generalised antiderivative” of a
$k$-dimensional identification function with relaxed symmetry conditions.
Section B.3 provides further details on comparing misspecified forecasts under
multi-objective elicitability, and on the sensitivity of multi-objective
scores with respect to increasing information sets. Finally, Section B.4
revisits a powerful necessary condition for identifiability and elicitability,
namely the Convex Level Sets (CxLS) property, for multi-objective scores.

Conditional IdentifiabilityConditional
ElicitabilityIdentifiabilityElicitability Multi-Objective Elicitability with
respect to $(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$ 1)Theorem 3.7Proposition
3.4 Figure 2: Illustration of the most important structural results for real-
valued (risk) functionals $\bm{T}_{1}$ and $\bm{T}_{2}$. The equivalence 1)
follows from SteinwartPasinETAL2014 under some regularity conditions.

The proof of Theorem 3.7 explicitly exploits the asymmetric structure of the
lexicographic order which fits well with the asymmetric notion of conditional
elicitability, where the roles of $\bm{T}_{1}$ and $\bm{T}_{2}$ may not be
changed. In particular, in the setup of Theorem 3.7, the pair
$(\bm{T}_{2},\bm{T}_{1})$ is generally not multi-objective elicitable on
$\mathcal{F}$ with respect to the lexicographic order $\preceq_{\mathrm{lex}}$
on $\mathbb{R}^{2}$. More generally, we suspect that under the conditions of
Theorem 3.7 the lexicographic order is the only order on $\mathbb{R}^{2}$
which renders $(\bm{T}_{2},\bm{T}_{1})$ multi-objective elicitable.

In the rest of the paper, we focus on multi-objective scores mapping to
$(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$. We use the results of Section 3,
summarised in Figure 2, extensively to prove our structural results for the
systemic risk measures in the next section.

##  4 Structural results for CoVaR, CoES and MES

###  4.1 CoVaR, CoES and MES fail to be identifiable or elicitable

The following proposition shows that the three risk measures
$\operatorname{CoVaR}_{\alpha|\beta}$, $\operatorname{CoES}_{\alpha|\beta}$
and $\operatorname{MES}_{\beta}$ generally fail to be identifiable or
elicitable on sufficiently rich classes of bivariate distributions
$\mathcal{F}\subseteq\mathcal{F}^{0}(\mathbb{R}^{2})$. It is proven by showing
that the CxLS property (which remains necessary for multi-objective scores;
see Proposition B.4) is violated in each case.

######  Proposition 4.1.

For $\alpha,\beta\in(0,1)$, $\operatorname{CoVaR}_{\alpha|\beta}$,
$\operatorname{CoES}_{\alpha|\beta}$ and $\operatorname{MES}_{\beta}$ are
neither identifiable nor elicitable on any class
$\mathcal{F}\subseteq\mathcal{F}^{0}(\mathbb{R}^{2})$ containing all bivariate
normal distributions along with their mixtures.

Proposition 4.1 casts doubt on traditional and comparative backtesting
approaches for CoVaR, CoES and MES as standalone systemic risk measures. Thus,
these measures should not be used for regulatory purposes on their own,
because forecasts for them can neither be verified for their adequacy nor can
they be sensibly compared to improve their modeling.

Proposition 4.1 can also be shown on other classes
$\mathcal{F}\subseteq\mathcal{F}^{0}(\mathbb{R}^{2})$ that are sufficiently
rich, e.g., the classes containing all measures with finite support (see
Remark E.1). Importantly, $\mathcal{F}$ must be convex and must not only
consist of distributions with independent marginals.

###  4.2 Joint identifiability results

Section C establishes the conditional identifiability and conditional
elicitability of $\operatorname{CoVaR}_{\alpha|\beta}(Y|X)$,
$(\operatorname{CoVaR}_{\alpha|\beta}(Y|X),\operatorname{CoES}_{\alpha|\beta}(Y|X))$,
and $\operatorname{MES}_{\beta}(Y|X)$ with $\operatorname{VaR}_{\beta}(X)$,
respectively. This, in combination with Proposition 3.4, immediately yields
the following joint identifiability results, where for $\alpha,\beta\in(0,1)$
and $p\in\\{0\\}\cup[1,\infty]$ we use the notation

$\displaystyle\mathcal{F}_{\alpha}^{p}(\mathbb{R})$
$\displaystyle\mathrel{\mathop{\mathchar
58\relax}}=\big{\\{}F\in\mathcal{F}^{p}(\mathbb{R})\colon
F\big{(}\operatorname{VaR}_{\alpha}(F)\big{)}=\alpha\big{\\}},$
$\displaystyle\mathcal{F}_{(\alpha)}^{p}(\mathbb{R})$
$\displaystyle\mathrel{\mathop{\mathchar
58\relax}}=\big{\\{}F\in\mathcal{F}_{\alpha}^{p}(\mathbb{R})\colon
F\big{(}\operatorname{VaR}_{\alpha}(F)+\varepsilon\big{)}>\alpha\ \text{for
all }\varepsilon>0\big{\\}},$ (4.1)
$\displaystyle\mathcal{F}_{(\beta)}^{0}(\mathbb{R}^{2})$
$\displaystyle\mathrel{\mathop{\mathchar
58\relax}}=\big{\\{}F_{X,Y}\in\mathcal{F}^{0}(\mathbb{R}^{2})\colon
F_{X}\in\mathcal{F}_{(\beta)}^{0}(\mathbb{R})\big{\\}}.$

######  Theorem 4.2.

Let $\alpha,\beta\in(0,1)$. Consider the strict
$\mathcal{F}_{(\beta)}^{0}(\mathbb{R}^{2})$-identification function for
$F_{X,Y}\mapsto\operatorname{VaR}_{\beta}(F_{X})$,
$V^{\operatorname{VaR}}\colon\mathbb{R}\times\mathbb{R}^{2}\to\mathbb{R}$,
$V^{\operatorname{VaR}}\big{(}v,(x,y)\big{)}=\mathds{1}\\{x\leq v\\}-\beta$.

  1. (i)

On $\\{F_{X,Y}\in\mathcal{F}^{0}(\mathbb{R}^{2})\colon
F_{X}\in\mathcal{F}^{0}_{(\beta)}(\mathbb{R}),\
F_{Y|X\geq\operatorname{VaR}_{\beta}(X)}\in\mathcal{F}^{0}_{(\alpha)}(\mathbb{R})\\}$,
the pair $F_{X,Y}\mapsto(\operatorname{VaR}_{\beta}(F_{X}),$
$\operatorname{CoVaR}_{\alpha|\beta}(F_{X,Y}))$ is identifiable with a strict
identification function
$\bm{V}^{(\operatorname{VaR},\operatorname{CoVaR})}\colon\mathbb{R}^{2}\times\mathbb{R}^{2}\to\mathbb{R}^{2}$,

$\displaystyle\bm{V}^{(\operatorname{VaR},\operatorname{CoVaR})}\big{(}(v,c),(x,y)\big{)}$
$\displaystyle=\begin{pmatrix}\mathds{1}\\{x\leq v\\}-\beta\\\
\mathds{1}\\{x>v\\}\big{[}\mathds{1}\\{y\leq c\\}-\alpha\big{]}\end{pmatrix}.$
(4.2)

  2. (ii)

On $\\{F_{X,Y}\in\mathcal{F}^{0}(\mathbb{R}^{2})\colon
F_{X}\in\mathcal{F}^{0}_{(\beta)}(\mathbb{R}),\
F_{Y|X\geq\operatorname{VaR}_{\beta}(X)}\in\mathcal{F}^{1}_{(\alpha)}(\mathbb{R})\\}$,
the triplet
$F_{X,Y}\mapsto(\operatorname{VaR}_{\beta}(F_{X}),\operatorname{CoVaR}_{\alpha|\beta}(F_{X,Y}),\operatorname{CoES}_{\alpha|\beta}(F_{X,Y}))$
is identifiable with a strict identification function
$\bm{V}^{(\operatorname{VaR},\operatorname{CoVaR},\operatorname{CoES})}\colon\mathbb{R}^{3}\times\mathbb{R}^{2}\to\mathbb{R}^{3}$,

$\big{(}(v,c,e),(x,y)\big{)}\mapsto\begin{pmatrix}\mathds{1}\\{x\leq
v\\}-\beta\\\ \mathds{1}\\{x>v\\}\big{[}\mathds{1}\\{y\leq
c\\}-\alpha\big{]}\\\
\mathds{1}\\{x>v\\}\Big{[}e-\frac{1}{1-\alpha}\big{(}y\mathds{1}\\{y>c\\}+c(\mathds{1}\\{y\leq
c\\}-\alpha)\big{)}\Big{]}\end{pmatrix}.$ (4.3)

  3. (iii)

On $\\{F_{X,Y}\in\mathcal{F}^{0}(\mathbb{R}^{2})\colon
F_{X}\in\mathcal{F}^{0}_{(\beta)}(\mathbb{R}),\
F_{Y|X\geq\operatorname{VaR}_{\beta}(X)}\in\mathcal{F}^{1}(\mathbb{R})\\}$,
the pair $F_{X,Y}\mapsto(\operatorname{VaR}_{\beta}(F_{X}),$
$\operatorname{MES}_{\beta}(F_{X,Y}))$ is identifiable with a strict
identification function
$\bm{V}^{(\operatorname{VaR},\operatorname{MES})}\colon\mathbb{R}^{2}\times\mathbb{R}^{2}\to\mathbb{R}^{2}$,

$\displaystyle\bm{V}^{(\operatorname{VaR},\operatorname{MES})}\big{(}(v,\mu),(x,y)\big{)}$
$\displaystyle=\begin{pmatrix}\mathds{1}\\{x\leq v\\}-\beta\\\
\mathds{1}\\{x>v\\}\big{[}\mu-y\big{]}\end{pmatrix}.$ (4.4)

Following NZ17, Theorem 4.2 can readily be used to assess the absolute
forecast quality via joint (_Wald_ -type) calibration tests. These either test
the null hypothesis of unconditional calibration,
$\operatorname{E}[\bm{V}(\bm{r}_{t},(X_{t},Y_{t}))]=\bm{0}$ for all
$t\in\mathbb{N}$, or the more informative null of _conditional_ calibration,
$\operatorname{E}[\bm{V}(\bm{r}_{t},(X_{t},Y_{t}))\,|\,\mathfrak{F}_{t-1}]=\bm{0}$
for all $t\in\mathbb{N}$. Here, the $\sigma$-algebra $\mathfrak{F}_{t-1}$
represents the information available to the forecaster at time $t-1$. Recall
that the null of conditional calibration is equivalent to
$\operatorname{E}[\varphi_{t-1}\bm{V}(\bm{r}_{t},(X_{t},Y_{t}))]=\bm{0}$ for
all $\mathfrak{F}_{t-1}$-measurable random vectors $\varphi_{t-1}$ for all
$t\in\mathbb{N}$. Unless $\mathfrak{F}_{t-1}$ is particularly simple, this
approach is statistically not feasible, because a finite selection
$\varphi_{1,t-1},\ldots,\varphi_{\ell,t-1}$ of $\mathfrak{F}_{t-1}$-measurable
random vectors is required in practice. Note that using only a finite
collection amounts to testing a broader null than the original one of
conditional calibration.

###### Remark 4.3.

It is worth pointing out the improvements upon the substantial work of
Banulescu-RaduETAL2019, who develop—inter alia—traditional backtests of
systemic risk measures. Translating their notation into ours, they propose in
their equation (4) a one-dimensional identification function for the pair
$(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(Y|X))$ of
the form $V\colon\mathbb{R}^{2}\times\mathbb{R}^{2}\to\mathbb{R}$,

$V\big{(}(v,c),(x,y)\big{)}=\mathds{1}\\{x>v\\}\mathds{1}\\{y>c\\}-(1-\alpha)(1-\beta).$
(4.5)

Banulescu-RaduETAL2019 point out that this an identification function since
$\bar{V}\big{(}(\operatorname{VaR}_{\beta}(X),$
$\operatorname{CoVaR}_{\alpha|\beta}(X|Y)),F_{X,Y}\big{)}=0.$ However, it
fails to be strict, since, so long as
$(1-\alpha^{\prime})(1-\beta^{\prime})=(1-\alpha)(1-\beta)$, we have
$\bar{V}\big{(}(\operatorname{VaR}_{\beta^{\prime}}(X),\operatorname{CoVaR}_{\alpha^{\prime}|\beta^{\prime}}(X|Y)),F_{X,Y}\big{)}=0$.
Thus, the specific null of ‘correct’ unconditional calibration
$\operatorname{E}[V(\bm{r}_{t},(X_{t},Y_{t}))]=0$ with $V$ from (4.5) is too
broad, since it is satisfied for all
$\bm{r}_{t}=(\operatorname{VaR}_{\beta^{\prime}}(X),\operatorname{CoVaR}_{\alpha^{\prime}|\beta^{\prime}}(X|Y))$
with $(1-\alpha^{\prime})(1-\beta^{\prime})=(1-\alpha)(1-\beta)$. This implies
only trivial power against such alternatives for the corresponding Wald-test
for calibration. This is in stark contrast to the Wald-test employing our two-
dimensional identification function in (4.2), where
$\operatorname{E}[\bm{V}^{(\operatorname{VaR},\operatorname{CoVaR})}(\bm{r}_{t},(X_{t},Y_{t}))]=\bm{0}$
if and only if
$\bm{r}_{t}=(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(X|Y))$.
We refer to Section G for a simulation study illustrating the possible
detrimental effects of the non-strictness of (4.5). We stress that the non-
strictness is not only problematic in the uncountably many cases where
$(1-\alpha^{\prime})(1-\beta^{\prime})=(1-\alpha)(1-\beta)$, but in all cases
where VaR is overpredicted and CoVaR underpredicted or the other way around.
This is because by overpredicting (underpredicting) VaR and underpredicting
(overpredicting) CoVaR, the different biases cancel out in the one-dimensional
identification function in (4.5), thus leading to a loss of power in
identifying misspecified forecasts; see Figure 9. Intuitively speaking, two
moment conditions via a two-dimensional identification function are necessary
to uniquely identify the two-dimensional pair
$(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(Y|X))$.
This is in line with other two-dimensional strict identification functions for
two-dimensional functionals in the literature, e.g. for Value at Risk and
Expected Shortfall, see FZ16a; NZ17.

###  4.3 Multi-objective elicitability results

Recall that $\operatorname{VaR}_{\beta}(Y)$,
$(\operatorname{VaR}_{\beta}(Y),\operatorname{ES}_{\beta}(Y))$ and
$\operatorname{E}(Y)$ are all elicitable. Surprisingly, their conditional
counterparts
$(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(Y|X))$,
$(\operatorname{VaR}_{\beta}(X),$
$\operatorname{CoVaR}_{\alpha|\beta}(Y|X),\operatorname{CoES}_{\alpha|\beta}(Y|X))$
and $(\operatorname{VaR}_{\beta}(X),\operatorname{MES}_{\beta}(Y|X))$ fail to
be elicitable despite being identifiable (Section D). This is due to
integrability conditions, causing an extreme gap between the class of strict
identification functions and the class of strictly consistent scoring
functions, which turns out to be empty. Thus, comparative backtests cannot be
implemented using a scalar strictly consistent scoring function. Furthermore,
the conditional elicitability results of Section C can hardly be used for
forecast comparisons, unless the $\operatorname{VaR}_{\beta}(X)$ forecasts are
the same and correctly specified. However, the conditional elicitability in
combination with Theorem 3.7 immediately yields the following novel joint
multi-objective elicitability results with respect to the lexicographic order
$\preceq_{\mathrm{lex}}$ on $\mathbb{R}^{2}$. These results can readily be
used for comparing systemic risk forecasts as detailed in Section 5. We again
use the notation introduced in (4.1).

######  Theorem 4.4.

Let $\alpha,\beta\in(0,1)$.

  1. (i)

On $\mathcal{F}\subseteq\mathcal{F}_{\beta}^{0}(\mathbb{R}^{2})$, the score
$S^{\operatorname{VaR}}\colon\mathbb{R}\times\mathbb{R}^{2}\to\mathbb{R}$,

$S^{\operatorname{VaR}}\big{(}v,(x,y)\big{)}=\big{(}\mathds{1}\\{x\leq
v\\}-\beta\big{)}h(v)-\mathds{1}\\{x\leq v\\}h(x)+a^{\operatorname{VaR}}(x,y)$
(4.6)

is strictly $\mathcal{F}$-consistent for $\mathcal{F}\ni
F_{X,Y}\mapsto\operatorname{VaR}_{\beta}(F_{X})$, if
$h\colon\mathbb{R}\to\mathbb{R}$ is strictly increasing and for all
$v\in\mathbb{R}$, the function $(x,y)\mapsto
a^{\operatorname{VaR}}(x,y)-\mathds{1}\\{x\leq v\\}h(x)$ is
$\mathcal{F}$-integrable.

  2. (ii)

On $\mathcal{F}\subseteq\\{F_{X,Y}\in\mathcal{F}^{0}(\mathbb{R}^{2})\colon
F_{X}\in\mathcal{F}^{0}_{(\beta)}(\mathbb{R}),\
F_{Y|X\geq\operatorname{VaR}_{\beta}(X)}\in\mathcal{F}^{0}_{\alpha}(\mathbb{R})\\}$,
the pair $\mathcal{F}\ni
F_{X,Y}\mapsto(\operatorname{VaR}_{\beta}(F_{X}),\operatorname{CoVaR}_{\alpha|\beta}(F_{X,Y}))$
is multi-objective elicitable with respect to
$(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$. A strictly $\mathcal{F}$-consistent
multi-objective scoring function
$\bm{S}^{(\operatorname{VaR},\operatorname{CoVaR})}\colon\mathbb{R}^{2}\times\mathbb{R}^{2}\to(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$
is given by

$\displaystyle\bm{S}^{(\operatorname{VaR},\operatorname{CoVaR})}\big{(}(v,c),(x,y)\big{)}$
$\displaystyle=\begin{pmatrix}S^{\operatorname{VaR}}\big{(}v,(x,y)\big{)}\\\
S_{v}^{\operatorname{CoVaR}}\big{(}c,(x,y)\big{)}\end{pmatrix},$ (4.7)
$\displaystyle S_{v}^{\operatorname{CoVaR}}\big{(}c,(x,y)\big{)}$
$\displaystyle=\mathds{1}\\{x>v\\}\Big{[}\big{(}\mathds{1}\\{y\leq
c\\}-\alpha\big{)}g(c)-\mathds{1}\\{y\leq c\\}g(y)+a(y)\Big{]}$
$\displaystyle\quad+a^{\operatorname{CoVaR}}(x,y),$

where $g\colon\mathbb{R}\to\mathbb{R}$ is strictly increasing and
$S_{v}^{\operatorname{CoVaR}}$ is $\mathcal{F}$-integrable for all
$v\in\mathbb{R}$.

  3. (iii)

On $\mathcal{F}\subseteq\\{F_{X,Y}\in\mathcal{F}^{0}(\mathbb{R}^{2})\colon
F_{X}\in\mathcal{F}^{0}_{(\beta)}(\mathbb{R}),\
F_{Y|X\geq\operatorname{VaR}_{\beta}(X)}\in\mathcal{F}^{1}_{\alpha}(\mathbb{R})\\}$,
the triplet $\mathcal{F}\ni
F_{X,Y}\mapsto(\operatorname{VaR}_{\beta}(F_{X}),\operatorname{CoVaR}_{\alpha|\beta}(F_{X,Y}),\operatorname{CoES}_{\alpha|\beta}(F_{X,Y}))$
is multi-objective elicitable with respect to
$(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$. A strictly $\mathcal{F}$-consistent
multi-objective scoring function
$\bm{S}^{(\operatorname{VaR},\operatorname{CoVaR},\operatorname{CoES})}\colon\mathbb{R}^{3}\times\mathbb{R}^{2}\to(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$
is given by

$\displaystyle\bm{S}^{(\operatorname{VaR},\operatorname{CoVaR},\operatorname{CoES})}\big{(}(v,c,e),(x,y)\big{)}=\begin{pmatrix}S^{\operatorname{VaR}}\big{(}v,(x,y)\big{)}\\\
S_{v}^{(\operatorname{CoVaR},\operatorname{CoES})}\big{(}(c,e),(x,y)\big{)}\end{pmatrix},$
(4.8) $\displaystyle
S_{v}^{(\operatorname{CoVaR},\operatorname{CoES})}\big{(}(c,e),(x,y)\big{)}=\mathds{1}\\{x>v\\}\Big{[}\big{(}\mathds{1}\\{y\leq
c\\}-\alpha\big{)}g(c)-\mathds{1}\\{y\leq c\\}g(y)$
$\displaystyle\quad+\phi^{\prime}(e)\Big{(}e-\tfrac{1}{1-\alpha}\big{(}y\mathds{1}\\{y>c\\}+c(\mathds{1}\\{y\leq
c\\}-\alpha)\big{)}\Big{)}-\phi(e)+a(y)\Big{]}+a^{\operatorname{CoES}}(x,y),$

where $g\colon\mathbb{R}\to\mathbb{R}$ is increasing,
$\phi\colon\mathbb{R}\to\mathbb{R}$ is strictly convex with subgradient
$\phi^{\prime}<0$, and $S_{v}^{(\operatorname{CoVaR},\operatorname{CoES})}$ is
$\mathcal{F}$-integrable for all $v\in\mathbb{R}$.

  4. (iv)

On $\mathcal{F}\subseteq\\{F_{X,Y}\in\mathcal{F}^{0}(\mathbb{R}^{2})\colon
F_{X}\in\mathcal{F}^{0}_{(\beta)}(\mathbb{R}),\
F_{Y|X\geq\operatorname{VaR}_{\beta}(X)}\in\mathcal{F}^{1}(\mathbb{R})\\}$,
the pair
$F_{X,Y}\mapsto(\operatorname{VaR}_{\beta}(F_{X}),\operatorname{MES}_{\beta}(F_{X,Y}))$
is multi-objective elicitable with respect to
${(\mathbb{R}^{2},\preceq_{\mathrm{lex}})}$. A strictly
$\mathcal{F}$-consistent multi-objective scoring function
$\bm{S}^{(\operatorname{VaR},\operatorname{MES})}\colon\mathbb{R}^{2}\times\mathbb{R}^{2}\to(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$
is given by

$\displaystyle\bm{S}^{(\operatorname{VaR},\operatorname{MES})}\big{(}(v,\mu),(x,y)\big{)}$
$\displaystyle=\begin{pmatrix}S^{\operatorname{VaR}}\big{(}v,(x,y)\big{)}\\\
S_{v}^{\operatorname{MES}}\big{(}\mu,(x,y)\big{)}\end{pmatrix},$ (4.9)
$\displaystyle S_{v}^{\operatorname{MES}}\big{(}\mu,(x,y)\big{)}$
$\displaystyle=\mathds{1}\\{x>v\\}\Big{[}\phi^{\prime}(\mu)(\mu-y)-\phi(\mu)+a(y)\Big{]}+a^{\operatorname{MES}}(x,y),$

where $\phi\colon\mathbb{R}\to\mathbb{R}$ is strictly convex with subgradient
$\phi^{\prime}$ and $S_{v}^{\operatorname{MES}}$ is $\mathcal{F}$-integrable
for all $v\in\mathbb{R}$.

The choice of the functions
$a,a^{\operatorname{VaR}},a^{\operatorname{CoVaR}},a^{\operatorname{CoES}},a^{\operatorname{MES}}$
in Theorem 4.4 is inessential. It may only influence the integrability of the
scoring functions on the one hand and it can control the sign of the scores on
the other hand. For the remaining functions in Theorem 4.4, one could choose
the standard choices, e.g., the identity for $h$ and $g$ in (4.6), (4.7) and
(4.8), giving rise to the common ‘pinball loss’, or $\phi(y)=y^{2}$ in (4.9)
leading to the square loss. For further possible choices, especially for
$\phi$ in (4.8), we refer to Subsection F.3.

Table 1 summarises the results of Section 4. For purposes of comparison, the
final three rows display the properties of $\operatorname{VaR}_{\beta}(Y)$ and
$\operatorname{ES}_{\beta}(Y)$. The multi-objective elicitability of the pair
$(\operatorname{VaR}_{\beta}(Y),\operatorname{ES}_{\beta}(Y))$ follows from
Example B.1. Table 1 highlights the structural difference between
$\operatorname{VaR}_{\beta}(Y)$, $\operatorname{E}(Y)$,
$(\operatorname{VaR}_{\beta}(Y),\operatorname{ES}_{\beta}(Y))$ on the one hand
and their conditional counterparts
$(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(Y|X))$,
$(\operatorname{VaR}_{\beta}(X),\operatorname{MES}_{\beta}(Y|X))$, and
$(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(Y|X),$
$\operatorname{CoES}_{\alpha|\beta}(Y|X))$ on the other hand. While
elicitability in the usual sense holds for the former, it fails for the
latter. This highlights the importance of the newly introduced concept of
multi-objective elicitability, which allows for comparative backtests. Section
5 shows how to implement comparative backtests with the multi-objective scores
of Theorem 4.4.

Table 1: Overview of properties of (systemic) risk measures. ✓(✗) indicates that property does (does not) apply. Risk Measure | Identifiability | Elicitability | Multi-objective
---|---|---|---
|  |  | Elicitability
$\operatorname{CoVaR}_{\alpha|\beta}(Y|X)$ | ✗ | ✗ | ✗
$\operatorname{CoES}_{\alpha|\beta}(Y|X)$ | ✗ | ✗ | ✗
$\operatorname{MES}_{\alpha|\beta}(Y|X)$ | ✗ | ✗ | ✗
$(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(Y|X))$ | ✓ | ✗ | ✓
$(\operatorname{VaR}_{\beta}(X),\operatorname{CoVaR}_{\alpha|\beta}(Y|X),\operatorname{CoES}_{\alpha|\beta}(Y|X))$ | ✓ | ✗ | ✓
$(\operatorname{VaR}_{\beta}(X),\operatorname{MES}_{\beta}(Y|X))$ | ✓ | ✗ | ✓
$\operatorname{VaR}_{\alpha}(Y)$ | ✓ | ✓ | ✓
$\operatorname{ES}_{\alpha}(Y)$ | ✗ | ✗ | ✗
$(\operatorname{VaR}_{\alpha}(Y),\operatorname{ES}_{\alpha}(Y))$ | ✓ | ✓ | ✓

##  5 Diebold–Mariano tests for multi-objective scores

###  5.1 Two-sided tests

DM95 propose to use formal hypothesis tests to account for sampling
uncertainty in forecast comparisons. These so-called Diebold–Mariano (DM)
tests are widely used in empirical forecast comparisons and continue to be
studied in the theoretical literature. However, consistent with the extant
notion of strict consistency, DM tests have hitherto relied on scalar scoring
functions. Thus, here we show how to use our two-dimensional multi-objective
scores from Theorem 4.4 in DM tests, with a special focus on the implications
caused by the lexicographic order.

To that end, denote by $\bm{S}=(S_{1},S_{2})^{\prime}$ one of the multi-
objective scores of Theorem 4.4. Let $\\{\bm{r}_{t,(1)}\\}_{t=1,\ldots,n}$ and
$\\{\bm{r}_{t,(2)}\\}_{t=1,\ldots,n}$ be the appertaining competing sequences
of forecasts (e.g., if
$\bm{S}=\bm{S}^{(\operatorname{VaR},\operatorname{CoVaR})}$, then
$\bm{r}_{t,(i)}=(\widehat{\operatorname{VaR}}_{t,(i)},\
\widehat{\operatorname{CoVaR}}_{t,(i)})$ for $i=1,2$). The verifying
observations are $\\{(X_{t},Y_{t})\\}_{t=1,\ldots,n}$. We compare the two
forecasts via the (bivariate) score differences
$\bm{d}_{t}\mathrel{\mathop{\mathchar
58\relax}}=(d_{1t},d_{2t})^{\prime}\mathrel{\mathop{\mathchar
58\relax}}=\bm{S}(\bm{r}_{t,(1)},(X_{t},Y_{t}))-\bm{S}(\bm{r}_{t,(2)},(X_{t},Y_{t}))$.
The two-sided null hypothesis is that both forecasts predict equally well on
average, i.e., $H_{0}^{=}\colon\operatorname{E}[\overline{\bm{d}}_{n}]=\bm{0}$
for all $n=1,2,\ldots$, where $\overline{\bm{d}}_{n}\mathrel{\mathop{\mathchar
58\relax}}=(\overline{d}_{1n},\overline{d}_{2n})^{\prime}\mathrel{\mathop{\mathchar
58\relax}}=\frac{1}{n}\sum_{t=1}^{n}\bm{d}_{t}$. (Along the lines of GW06, one
can also test the conditional null hypothesis
$H_{0}^{*=}\colon\operatorname{E}[\bm{d}_{t}\mid\mathfrak{F}_{t-1}]=\bm{0}$
for all $t=1,2,\ldots$, where the $\sigma$-algebra $\mathfrak{F}_{t-1}$
contains all information available at time $t-1$.) We test $H_{0}^{=}$ using
the Wald-type test statistic

$\mathcal{T}_{n}=n\overline{\bm{d}}_{n}^{\prime}\widehat{\bm{\varOmega}}_{n}^{-1}\overline{\bm{d}}_{n},$
(5.1)

where $\widehat{\bm{\varOmega}}_{n}$ is some consistent estimator of the
variance-covariance matrix
$\bm{\varOmega}_{n}=\operatorname{Var}(\sqrt{n}\overline{\bm{d}}_{n})$ under
the null hypothesis (i.e., in the componentwise norm,
$\|\widehat{\bm{\varOmega}}_{n}-\bm{\varOmega}_{n}\|\overset{\operatorname{P}}{\longrightarrow}0$,
as $n\to\infty$). To account for possible autocorrelation in the sequence
$\\{\bm{d}_{t}\\}_{t=1,\ldots,n}$, one can use

$\widehat{\bm{\varOmega}}_{n}=\begin{pmatrix}\widehat{\sigma}_{11,n}&\widehat{\sigma}_{12,n}\\\
\widehat{\sigma}_{12,n}&\widehat{\sigma}_{22,n}\end{pmatrix}=\frac{1}{n}\sum_{t=1}^{n}(\bm{d}_{t}-\overline{\bm{d}}_{n})(\bm{d}_{t}-\overline{\bm{d}}_{n})^{\prime}\\\
+\frac{1}{n}\sum_{h=1}^{m_{n}}w_{n,h}\sum_{t=h+1}^{n}\Big{[}(\bm{d}_{t}-\overline{\bm{d}}_{n})(\bm{d}_{t-h}-\overline{\bm{d}}_{n})^{\prime}+(\bm{d}_{t-h}-\overline{\bm{d}}_{n})(\bm{d}_{t}-\overline{\bm{d}}_{n})^{\prime}\Big{]},$
(5.2)

where $m_{n}\rightarrow\infty$ is a sequence of integers satisfying
$m_{n}=o(n^{1/4})$, and $w_{n,h}$ is a uniformly bounded scalar triangular
array with $w_{n,h}\rightarrow 1$, as $n\to\infty$, for all
$h=1,\ldots,m_{n}$; see Whi01 for detail. Under the assumption that
$\\{\bm{d}_{t}\\}_{t=1,\ldots,n}$ does not exhibit autocorrelation under the
null, $m_{n}$ can be set to 0 such that $\widehat{\bm{\varOmega}}_{n}$ is
simply the sample variance-covariance matrix.

######  Theorem 5.1.

Suppose that $\bm{\varOmega}_{n}\longrightarrow\bm{\varOmega}$, as
$n\to\infty$, where $\bm{\varOmega}$ is positive definite. Then, under
technical Assumption E.2 (see Supplement Section E), it holds under
$H_{0}^{=}$ that

$\sqrt{n}\widehat{\bm{\varOmega}}_{n}^{-1/2}\overline{\bm{d}}_{n}\overset{d}{\longrightarrow}N(\bm{0},\bm{I}_{2\times
2}),\qquad\text{as }n\to\infty,$

where $\bm{I}_{2\times 2}$ denotes the $(2\times 2)$-identity matrix. In
particular, $\mathcal{T}_{n}\overset{d}{\longrightarrow}\chi_{2}^{2}$, as
$n\to\infty$, where $\chi_{2}^{2}$ denotes a $\chi^{2}$-distribution with 2
degrees of freedom.

Thus, we reject $H_{0}^{=}$ at significance level $\nu$, if
$\mathcal{T}_{n}>\chi_{2,1-\nu}^{2}$, where $\chi_{2,1-\nu}^{2}$ is the
$(1-\nu)$-quantile of the $\chi_{2}^{2}$-distribution. A typical non-rejection
region in terms of $\overline{d}_{1n}$ and $\overline{d}_{2n}$ is sketched in
Figure 3 (a), and has the well-known ellipse shape. For brevity, we leave out
a formal investigation of our test under the alternative. We mention, however,
that consistency of our test can be established along the lines of GW06.

Figure 3: Non-rejection regions for two-sided DM test in (a), lexicographic DM
test in (b), and one and a half-sided DM test in (c).

###  5.2 One and a half-sided tests

In several contexts, it may be desirable to perform a one-sided comparative
backtest to establish the superiority of risk forecasts
$\\{\bm{r}_{t,(2)}\\}_{t=1,\ldots,n}$ over some benchmark forecasts
$\\{\bm{r}_{t,(1)}\\}_{t=1,\ldots,n}$. These different forecasts could stem
from two different internal models of a financial institution (say, a legacy
model and and extension thereof). It could also be the case that the benchmark
forecasts originate from a regulatory standard model, and—in line with the
conservative backtesting approach of FZG16—the financial institution has the
onus of proof to show the superiority of its internal model over the standard
model. In such situations, it is tempting to test the null hypothesis
$\operatorname{E}[\overline{\bm{d}}_{n}]\preceq_{\mathrm{lex}}\bm{0}$, which
is equivalent to

$\operatorname{E}[\overline{d}_{1n}]<0\qquad\text{or}\qquad\Big{(}\operatorname{E}[\overline{d}_{1n}]=0\quad\text{and}\quad\operatorname{E}[\overline{d}_{2n}]\leq
0\Big{)}.$ (5.3)

Here, the goal would be to reject the null of better or, at least, equally
good benchmark forecasts as evidence of the superiority of
$\\{\bm{r}_{t,(2)}\\}_{t=1,\ldots,n}$. However, as under standard conditions
(see Theorem 5.1), $\sqrt{n}(\overline{d}_{1n},\overline{d}_{2n})^{\prime}$
has an asymptotic bivariate normal distribution, the probability that
$\overline{d}_{1n}=0$ and $\overline{d}_{2n}\leq 0$ vanishes for large sample
sizes. Thus, testing (5.3) amounts to a test of the null
$\operatorname{E}[\overline{d}_{1n}]<0$, i.e., that the benchmark VaR
forecasts are superior. This null can be tested via
$\mathcal{T}_{1n}=\sqrt{n}\overline{d}_{1n}/\widehat{\sigma}_{11,n}^{1/2}$,
where we reject (5.3) at significance level $\nu\in(0,1)$ when
$\mathcal{T}_{1n}>\Phi^{-1}(1-\nu)$. The corresponding non-rejection region is
sketched in Figure 3 (b). In particular, we would reject
$\operatorname{E}[\overline{\bm{d}}_{n}]\preceq_{\mathrm{lex}}\bm{0}$ solely
based on the predictive performance of the $\operatorname{VaR}(X_{t})$
component. In other words, once $\operatorname{E}[\overline{d}_{1n}]<0$ is
rejected such that the internal VaR forecasts are superior, the internal
forecasts $\bm{r}_{t,(1)}$ are preferable in the lexicographic order
irrespective of the quality of the systemic risk component. Since this would
entirely ignore the motivation of the backtest, we disregard this one-sided
backtesting approach.

As a compromise, we suggest to test the following ‘one and a half-sided’ null
hypothesis. Since the two systemic risk forecasts only play a role for the
ranking in the lexicographic order when
$\operatorname{E}[\overline{d}_{1n}]=0$, our suggested null hypothesis takes
the form

$H_{0}^{\preceq_{\mathrm{lex}}}\colon\operatorname{E}[\overline{d}_{1n}]=0\quad\text{and}\quad\operatorname{E}[\overline{d}_{2n}]\leq
0\qquad\text{for all }n=1,2,\ldots.$

We demonstrate below how to interpret a rejection of this null. The region in
$\mathbb{R}^{2}$ pertaining to $H_{0}^{\preceq_{\mathrm{lex}}}$ is the lower
part of the vertical axis in Figure 3 (c).

Obviously, $H_{0}^{\preceq_{\mathrm{lex}}}$ is the union of all $H_{0}^{(c)}$
with $c\leq 0$, where

$H_{0}^{(c)}\colon\operatorname{E}[\overline{d}_{1n}]=0\quad\text{and}\quad\operatorname{E}[\overline{d}_{2n}]=c\qquad\text{for
all }n=1,2,\ldots$

For each individual $c\leq 0$, this can be tested using the Wald-type test
statistic $\mathcal{T}_{n}^{(c)}$, where $\mathcal{T}_{n}^{(c)}$ is defined
similarly as $\mathcal{T}_{n}$, only with $\bm{d}_{t}$ replaced by
$\bm{d}_{t}^{(c)}=(d_{1t},d_{2t}-c)^{\prime}$. (Note that this substitution
leaves $\widehat{\bm{\varOmega}}_{n}$ unaffected.) Thus, we reject
$H_{0}^{\preceq_{\mathrm{lex}}}$ if and only if

$\mathcal{T}_{n}^{(c)}>\chi_{2,1-\widetilde{\nu}}^{2}\qquad\text{for all
}c\leq 0,$ (5.4)

where $\widetilde{\nu}\in(0,1)$. The area associated with the appertaining
non-rejection region is shaded in pink in Figure 3 (c). The rejection
condition (5.4) is of course equivalent to
$\mathcal{T}_{n}^{\operatorname{OS}}\mathrel{\mathop{\mathchar
58\relax}}=\inf_{c\leq
0}\mathcal{T}_{n}^{(c)}=\mathcal{T}_{n}^{(c^{*})}>\chi_{2,1-\widetilde{\nu}}^{2},$
where the solution $c^{*}\mathrel{\mathop{\mathchar
58\relax}}=\min\big{\\{}0,\overline{d}_{2n}-(\widehat{\sigma}_{12,n}/\widehat{\sigma}_{11,n})\overline{d}_{1n}\big{\\}}$
follows from a simple quadratic minimisation problem. Hence,

$\mathcal{T}_{n}^{\operatorname{OS}}=n\Big{(}\overline{d}_{1n},\max\big{\\{}\overline{d}_{2n},(\widehat{\sigma}_{12,n}/\widehat{\sigma}_{11,n})\overline{d}_{1n}\big{\\}}\Big{)}\widehat{\bm{\varOmega}}_{n}^{-1}\begin{pmatrix}\overline{d}_{1n}\\\
\max\big{\\{}\overline{d}_{2n},(\widehat{\sigma}_{12,n}/\widehat{\sigma}_{11,n})\overline{d}_{1n}\big{\\}}\end{pmatrix}.$
(5.5)

To illustrate this graphically, note that the line defined by
$z_{2}=(\widehat{\sigma}_{12,n}/\widehat{\sigma}_{11,n})z_{1}$—indicated by
the dashed line in Figure 3 (c)—passes through the extremal (negative and
positive) horizontal points of the ellipse. Thus, if
$\overline{d}_{2n}>(\widehat{\sigma}_{12,n}/\widehat{\sigma}_{11,n})\overline{d}_{1n}$,
(5.5) parametrizes the upper half of the tilted ellipse, and if
$\overline{d}_{2n}\leq(\widehat{\sigma}_{12,n}/\widehat{\sigma}_{11,n})\overline{d}_{1n}$
the area below. In our numerical experiments, we use
$\mathcal{T}_{n}^{\operatorname{OS}}$ to test
$H_{0}^{\preceq_{\mathrm{lex}}}$.

The next proposition shows that rejecting $H_{0}^{\preceq_{\mathrm{lex}}}$
when $\mathcal{T}_{n}^{\operatorname{OS}}>\chi_{2,1-\widetilde{\nu}}^{2}$
leads to a test of level
$\nu=1/2\big{[}1+\widetilde{\nu}-F_{\chi_{1}^{2}}(\chi_{2,1-\widetilde{\nu}}^{2})\big{]}$,
where $F_{\chi_{1}^{2}}$ denotes the cdf of a $\chi_{1}^{2}$-distribution.

######  Proposition 5.2.

Under the conditions of Theorem 5.1, rejecting
$H_{0}^{\preceq_{\mathrm{lex}}}$ if
$\mathcal{T}_{n}^{\operatorname{OS}}>\chi_{2,1-\widetilde{\nu}}^{2}$ leads to
an asymptotic size $\nu$-test with
$\nu=1/2\big{[}1+\widetilde{\nu}-F_{\chi_{1}^{2}}(\chi_{2,1-\widetilde{\nu}}^{2})\big{]}$.
That is,

$\sup_{c\leq
0}\lim_{n\to\infty}\operatorname{P}\Big{\\{}H_{0}^{\preceq_{\mathrm{lex}}}\
\text{is rejected based on}\ \mathcal{T}_{n}^{\operatorname{OS}}\ \Big{|}\
H_{0}^{(c)}\ \text{holds}\Big{\\}}=\nu.$

###### Remark 5.3.

For a test of $H_{0}^{\preceq_{\mathrm{lex}}}$ with desired significance level
of $\nu$, one can determine the required level $\widetilde{\nu}$ from
$\nu=1/2\big{[}1+\widetilde{\nu}-F_{\chi_{1}^{2}}(\chi_{2,1-\widetilde{\nu}}^{2})\big{]}$
using standard root-finding algorithms. E.g., if $\nu=1\%$ / $\nu=5\%$ /
$\nu=10\%$, then $\widetilde{\nu}=1.60\%$ / $\widetilde{\nu}=7.66\%$ /
$\widetilde{\nu}=14.9\%$.

###### Remark 5.4.

To compare systemic risk forecasts on an equal footing, it may be desirable to
compare them based on the same marginal model being used and, hence, based on
the same VaR forecasts
$\widehat{\operatorname{VaR}}_{t}=\widehat{\operatorname{VaR}}_{t,(1)}=\widehat{\operatorname{VaR}}_{t,(2)}$.
In this case, differences in predictive ability can be attributed solely to
the different dependence models; see, e.g., NZ19+ and Hog20a+. (Note that
Theorem 5.1 no longer applies for identical VaR forecasts, since the limit of
the covariance matrix $\bm{\varOmega}$ is only positive semi-definite.) When
$\widehat{\operatorname{VaR}}_{t}=\widehat{\operatorname{VaR}}_{t,(1)}=\widehat{\operatorname{VaR}}_{t,(2)}$,
testing for $\operatorname{E}[\overline{d}_{1n}]=0$ is redundant, and
$H_{0}^{=}$ and $H_{0}^{\preceq_{\mathrm{lex}}}$ reduce to
$\operatorname{E}[\overline{d}_{2n}]=0$ and
$\operatorname{E}[\overline{d}_{2n}]\leq 0$, respectively. These hypotheses
can be tested using
$\mathcal{T}_{2n}=\sqrt{n}\overline{d}_{2n}/\widehat{\sigma}_{22,n}^{1/2}$.
Under $\operatorname{E}[\overline{d}_{2n}]=0$, $\mathcal{T}_{2n}$ converges to
an $N(0,1)$-limit when restricting the assumptions of Theorem 5.1 to the
sequence $d_{2t}$. Hence, we would reject
$\operatorname{E}[\overline{d}_{2n}]=0$ (or
$\operatorname{E}[\overline{d}_{2n}]\leq 0$) at significance level
$\nu\in(0,1)$, if $|\mathcal{T}_{2n}|>\Phi^{-1}(1-\nu/2)$ (or
$\mathcal{T}_{2n}>\Phi^{-1}(1-\nu)$).

Figure 4: Interpretation of test decisions. Yellow zone corresponds to
ellipse. Red (grey) zone corresponds to the rectangle to the left (right) of
the ellipse. Green (orange) zone corresponds to area immediately above (below)
the ellipse.

Figure 4 presents a schematic for interpreting test decisions on
$H_{0}^{\preceq_{\mathrm{lex}}}$ depending on the values of
$\overline{d}_{1n}$ and $\overline{d}_{2n}$. In a regulatory context, it
extends the three-zone traffic-light classification of the BCBSBF19:

> “The green zone corresponds to backtesting results that do not themselves
> suggest a problem with the quality or accuracy of a bank’s model. The yellow
> zone encompasses results that do raise questions in this regard, but where
> such a conclusion is not definitive. The red zone indicates a backtesting
> result that almost certainly indicates a problem with a bank’s risk model.”

The union of the yellow and the orange areas in Figure 4 corresponds to the
non-rejection region depicted in Figure 3 (c). By symmetry, the union of the
yellow and the green areas is the non-rejection region associated with the
null

$H_{0}^{\succeq_{\mathrm{lex}}}\colon\operatorname{E}[\overline{d}_{1n}]=0\quad\text{and}\quad\operatorname{E}[\overline{d}_{2n}]\geq
0\qquad\text{for all }n=1,2,\ldots.$

Thus, for $(\overline{d}_{1n},\overline{d}_{2n})^{\prime}$ in the yellow area
we can neither reject $H_{0}^{\preceq_{\mathrm{lex}}}$ nor
$H_{0}^{\succeq_{\mathrm{lex}}}$ (at significance level
$\nu=1/2\big{[}1+\widetilde{\nu}-F_{\chi_{1}^{2}}(\chi_{2,1-\widetilde{\nu}}^{2})\big{]}$,
respectively). This implies that there is no evidence of differences in
predictive ability between the internal and the benchmark model (at level
$\widetilde{\nu}$). From a regulatory perspective, the bank’s internal model
is ‘at the boundary’ of what can be deemed acceptable. Hence, we suggest
heightened attention and close monitoring by the regulator, as indicated by
the yellow colour.

When $(\overline{d}_{1n},\overline{d}_{2n})^{\prime}$ falls into the orange
area, then—while the VaR forecasts are of comparable quality—the benchmark
model provides superior systemic risk forecasts. Here, as indicated by the
orange colour, a revision of the internal systemic risk forecasts is called
for, while the internal marginal model is in order.

In contrast, the green area indicates superior systemic risk forecasts of the
internal model, with VaR forecasts being comparably accurate. In this case,
the financial institution should be allowed to use its internal model for risk
forecasting purposes. The green colour indicates a pass of the backtest.

In the red region, the VaR forecasts of the internal model are deemed inferior
to the benchmark predictions, whence there is no basis to compare the systemic
risk forecasts. (The red region corresponds to the rejection region of the
null $\operatorname{E}[\overline{d}_{1n}]\geq 0$ at level
$\nu^{\prime}=\frac{1}{2}-\frac{1}{2}F_{\chi_{1}^{2}}(\chi_{2,1-\widetilde{\nu}}^{2})$.
E.g., for $\nu=5\%$ we get $\widetilde{\nu}=7.66\%$ and
$\nu^{\prime}=1.17\%$.) Here, the bank should not be allowed to use its own
marginal model (the traffic light is red), but instead should be required to
use the benchmark model for the marginals to ensure a fair assessment of the
systemic risk forecasts. For this comparison, where the VaR forecasts are
identical, one can focus solely on the systemic risk component by using
$\mathcal{T}_{2n}$; see Remark 5.4. For the comparison via $\mathcal{T}_{2n}$,
we suggest to adopt a similar decision heuristic as in FZG16: Neither
$H_{0}^{\preceq_{\mathrm{lex}}}$ nor $H_{0}^{\succeq_{\mathrm{lex}}}$ can be
rejected when $|\mathcal{T}_{2n}|\leq\Phi^{-1}(1-\nu)$ (corresponding to our
yellow zone). When $\mathcal{T}_{2n}>\Phi^{-1}(1-\nu)$
($\mathcal{T}_{2n}<\Phi^{-1}(\nu)$) such that $H_{0}^{\preceq_{\mathrm{lex}}}$
($H_{0}^{\succeq_{\mathrm{lex}}}$) can be rejected, the internal systemic risk
forecasts are superior (inferior), corresponding to our green (red) zone.

Similarly as for the red region, there are no grounds for meaningful systemic
risk forecast comparisons in the grey area, since the internal model’s VaR
forecasts are superior. (Formally, it corresponds to the rejection region of
the null $\operatorname{E}[\overline{d}_{1n}]\leq 0$ at level
$\nu^{\prime}=\frac{1}{2}-\frac{1}{2}F_{\chi_{1}^{2}}(\chi_{2,1-\widetilde{\nu}}^{2})$.)
In this case, there is no cause for action on the end of the bank (hence the
grey colour), but the regulator should rather adopt the bank’s VaR model as a
basis for comparing the systemic risk forecasts. As before, the subsequent
comparison of systemic risk forecasts should be carried out by using
$\mathcal{T}_{2n}$, since the VaR forecasts are identical.

Section F investigates the finite-sample properties of $\mathcal{T}_{n}$,
$\mathcal{T}_{n}^{\operatorname{OS}}$, and $\mathcal{T}_{2n}$ under
$H_{0}^{=}$ and $H_{0}^{\preceq_{\mathrm{lex}}}$ in detail. Here, we only
summarize the main findings. First, size is adequate already for $n=500$,
which is encouraging since effective sample sizes in risk forecast comparisons
are small. Second, power increases markedly in $n$. Third, comparisons for
(CoVaR, CoES) are slightly more powerful than those for CoVaR alone, most
likely due to the increased informational content of the CoES component.
Fourth, as expected for one-sided tests, departures from
$H_{0}^{\preceq_{\mathrm{lex}}}$ and $H_{0}^{=}$ are easier to detect for the
former. Fifth, it is in general easier to detect differences in predictive
ability of the systemic risk component, when the VaR component is identical
(instead of only comparable) across forecasts. Intuitively, the inclusion of
comparable VaR forecasts dilutes the power of the test in the systemic risk
component.

While the schematic in Figure 4 is motivated by a regulatory framework, we
stress that it can be used in the context of any comparative backtest between
different models. Such a general example is provided in Section 6.

##  6 Empirical application

Consider daily log-losses $X_{-r+1},\ldots,X_{n}$ on the S&P 500 and log-
losses $Y_{-r+1},\ldots,Y_{n}$ on the DAX 30 from 2000–2020, where the data
are taken from www.wsj.com/market-data/quotes (ticker symbols: SPX and
DX:DAX). So if $P_{Z,t}$ denotes the stock index value at time $t$, then
$Z_{t}=-\log(P_{Z,t}/P_{Z,t-1})$ ($Z\in\\{X,Y\\}$). We only keep those
observations where data on both indexes are available, giving us $n+r=5\,193$
observations. Here, for $\alpha=\beta=0.95$, we compare rolling-window (VaR,
CoVaR, CoES) forecasts for the series $\\{(X_{t},Y_{t})\\}_{t=1,\ldots,n}$,
where $r=1\,000$ denotes the moving window length. The choice of $X$ and $Y$
amounts to considering the risk for large losses of the DAX 30, given that the
world’s leading stock index—the S&P 500—is in distress. To promote flow in
this section, we often refer to the simulation setup in Section F for details
on the time series models and the risk forecast computation.

For short-term risk management purposes, _conditional_ (systemic) risk measure
forecasts are more informative than unconditional ones. Conditional risk
measures are based on the conditional distribution of $(X_{t},Y_{t})$, that
is, $F_{(X_{t},Y_{t})\mid\mathfrak{F}_{t-1}}(x,y)=\operatorname{P}\\{X_{t}\leq
x,Y_{t}\leq y\mid\mathfrak{F}_{t-1}\\}=\mathrel{\mathop{\mathchar
58\relax}}\operatorname{P}_{t-1}\\{X_{t}\leq x,Y_{t}\leq y\\}$ for
$x,y\in\mathbb{R}$. Here, the filtration $\mathfrak{F}_{t-1}$ is generated by
the information available to a forecaster at time $t-1$. These are usually
past observations $(X_{t-1},Y_{t-1}),(X_{t-2},Y_{t-2}),\ldots$, and possibly
additional exogenous information. Here, we assume
$\mathfrak{F}_{t-1}=\sigma\big{(}(X_{t-1},Y_{t-1}),(X_{t-2},Y_{t-2}),\ldots\big{)}$,
such that we forecast the _conditional_ risk measures
$\operatorname{VaR}_{t}(X_{t})=\operatorname{VaR}_{\beta}(F_{X_{t}\mid\mathfrak{F}_{t-1}})$,
$\operatorname{CoVaR}_{t}(Y_{t}|X_{t})=\operatorname{CoVaR}_{\alpha|\beta}(F_{(X_{t},Y_{t})\mid\mathfrak{F}_{t-1}})$,
and
$\operatorname{CoES}_{t}(Y_{t}|X_{t})=\operatorname{CoES}_{\alpha|\beta}(F_{(X_{t},Y_{t})\mid\mathfrak{F}_{t-1}})$.
For notational brevity, we suppress the dependence of the risk measures on the
risk levels $\alpha$ and $\beta$, which we fix at $\alpha=\beta=0.95$.

We consider two different methods for
$(\operatorname{VaR}_{t}(X_{t}),\operatorname{CoVaR}_{t}(Y_{t}|X_{t}),\operatorname{CoES}_{t}(Y_{t}|X_{t}))$
forecasting. The first method uses a simple GARCH(1,1) model for $X_{t}$ and
$Y_{t}$ and—as a dependence model for the respective innovations
$\varepsilon_{x,t}$ and $\varepsilon_{y,t}$—a Gaussian copula driven by GAS
dynamics. That is, we assume
$(\varepsilon_{x,t},\varepsilon_{y,t})\mid\mathfrak{F}_{t-1}$ to have Gaussian
copula density $c(\,\cdot\,;\rho_{t})$ with time-varying correlation parameter
$\rho_{t}\in(-1,1)$ following GAS dynamics. Details on the precise
specification are in Subsection F.1, where the same model (with
$\vartheta=\infty$ in Equation (F.2)) is used in the simulations. Both the
marginal and the dependence model are regularly used as benchmark models in
forecast comparisons.

The second method uses the GJR–GARCH(1,1) model of GJR93. The GJR–GARCH model
possesses an additional parameter that allows positive and negative shocks of
equal magnitude to have a different effect on volatility. As a dependence
model, we now use a $t$-copula driven by GAS dynamics, similarly as in
Equation (F.2). For both models we remain agnostic regarding the specific
distribution of the $\varepsilon_{x,t}$ and $\varepsilon_{y,t}$—both in the
model estimation (by using the robust Gaussian quasi-maximum likelihood
estimator for the GARCH-type marginal models) and the risk forecasting stage
(by using their empirical cdfs $\widehat{F}_{x}$ and $\widehat{F}_{y}$ in
computing the risk measures). For details on how the risk predictions are
calculated, we refer to Subsection F.2.

We now compare two sequences of rolling-window predictions. For each rolling
window of length $r=1\,000$, we re-fit the two models on a daily basis. This
gives us $n=4\,193$ forecasts $\\{\bm{r}_{t,(1)}\\}_{t=1,\ldots,n}$ from the
GARCH model with Gaussian copula, and $\\{\bm{r}_{t,(2)}\\}_{t=1,\ldots,n}$
from the GJR–GARCH with $t$-copula. We interpret the $\bm{r}_{t,(1)}$ as
generic benchmark forecasts, which are to be improved upon by the
$\bm{r}_{t,(2)}$. In an oversight context, the $\bm{r}_{t,(1)}$ may be some
regulatory benchmark forecasts and the $\bm{r}_{t,(2)}$ forecasts from the
bank’s internal model. Or from a banking perspective, the $\bm{r}_{t,(1)}$ may
be forecasts issued from a trading desk’s legacy model and the
$\bm{r}_{t,(2)}$ are forecasts from a refined version thereof. The latter
context is more realistic here, because in the regulatory framework the focus
is more often on the relation of between an individual bank’s returns and the
market as a whole. In any case, the methodology remains the same. We compare
forecasts based on the verifying observations
$\\{(X_{t},Y_{t})\\}_{t=1,\ldots,n}$.

Figure 5: Top: DAX log-losses on days of VaR violation of S&P 500 (black).
CoVaR (CoES) forecasts are shown as the blue (red) line. Violation of CoVaR
(CoES) forecast is indicated by a blue (red) dot. All forecasts from GARCH
with Gaussian copula. Bottom: Same as top, only with forecasts from GJR–GARCH
with $t$-copula.

Figure 5 shows CoVaR and CoES forecasts from both models, where the top panel
corresponds to the GARCH model with Gaussian copula and the bottom panel to
the GJR–GARCH with $t$-copula. Specifically, the panels show the CoVaR (blue)
and CoES (red) forecasts for the DAX log-losses (black) on days where the S&P
500 exceeds its VaR forecast. Note that due to the different marginal models
(and, hence, the different VaR forecasts), the black lines differ slightly in
the upper and lower panel of Figure 5. By definition, the S&P 500 should only
exceed its VaR forecast on 5% of all trading days, i.e., on $0.05\cdot
4193=209.65$ days in our out-of-sample period. With 213 (218) VaR violations,
our marginal GARCH(1,1) model (GJR–GARCH(1,1) model) is close to the ideal
frequency. By definition, we expect our CoVaR forecasts to be not exceeded on
95% of these 213 days (218 days) with a VaR violation. With 15 and 16
exceedances (blue dots), which correspond to non-exceedance frequencies of
93.0% and 92.7%, the Gaussian copula and the $t$-copula model are reasonably
close to the 95%-benchmark. However, for the Gaussian copula, the CoVaR
exceedances seem to cluster more, such as during the beginning of the Covid-19
pandemic in early 2020 (top panel of Figure 5). Such violation clusters are
undesirable from a risk management perspective, providing some informal
evidence in favour of the $t$-copula model. We investigate this more formally
in the following. Note that both panels of Figure 5 indicate marked spikes in
systemic risk during the financial crisis of 2008–2009, during the European
sovereign debt crisis in the first half of the 2010’s, and—most markedly—in
2020 as a consequence of the Covid-19 pandemic.

As pointed out above, we regard the GARCH model with Gaussian copula as our
benchmark model. So we now want to test $H_{0}^{\preceq_{\mathrm{lex}}}$ for
(VaR, CoVaR) and (VaR, CoVaR, CoES). Due to the less clustered CoVaR
exceedances and the compelling empirical evidence in favour of GJR–GARCH
models (GJR93; BEK11) and GAS-$t$-copula models (CKL13; BC19), we expect the
GJR–GARCH model with $t$-copula to produce lower scores, i.e., better risk
forecasts, possibly leading to a rejection of
$H_{0}^{\preceq_{\mathrm{lex}}}$. We carry out the tests using
$\mathcal{T}_{n}^{\operatorname{OS}}$ with the scores of Equations (F.4) and
(F.5) having 0-homogeneous score differences, and with
$\widehat{\bm{\varOmega}}_{n}$ from (5.2) (with $m_{n}=0$); see Supplement
Subsection F.3 for details. For VaR and ES forecasts, scoring functions giving
0-homogeneous score differences are typically recommended, since they allow
for ‘unit-consistent’ and powerful comparisons (NZ17). We confirm the latter
in our simulations for systemic risk forecasts as well, thus justifying our
choice. Let
$\overline{\bm{d}}_{n}=(\overline{d}_{1n},\overline{d}_{2n})^{\prime}$ be
defined as in Subsection 5.1. Indeed, computing $\overline{\bm{d}}_{n}$, we
find that the GJR–GARCH with $t$-copula produces lower scores for both the
(VaR, CoVaR) and the (VaR, CoVaR, CoES) forecasts. The score differences are
even statistically significant at the 5%-level: The $p$-values for the
$\mathcal{T}_{n}^{\operatorname{OS}}$-based Wald test are 2.9% for (VaR,
CoVaR) and 3.0% for (VaR, CoVaR, CoES).

Figure 6: Panel (a): Values of $\overline{d}_{1n}$ and $\overline{d}_{2n}$ for
(VaR, CoVaR) forecasts, indicated by $\times$. Panel (b): Values of
$\overline{d}_{1n}$ and $\overline{d}_{2n}$ for (VaR, CoVaR, CoES) forecasts,
indicated by $\times$.

Figure 6 illustrates the test decisions. Panel (a) shows the results for the
(VaR, CoVaR) comparison. Both $\overline{d}_{1n}$ and $\overline{d}_{2n}$ are
positive, favouring the forecasts $\bm{r}_{t,(2)}$ from the GJR–GARCH model
with $t$-copula. Additionally, the pair
$\overline{\bm{d}}_{n}=(\overline{d}_{1n},\overline{d}_{2n})^{\prime}$ lies
above the yellow ellipse, which traces the contour of a bivariate normal
distribution with probability content $(100-7.66)\%=92.34\%$. This confirms
the significance of the score difference at the $5\%$-level; recall Remark
5.3. The results for the (VaR, CoVaR, CoES) forecasts in panel (b) are
qualitatively similar. Adopting our traffic-light interpretation, the
GJR–GARCH model with $t$-copula would be deemed an adequate risk forecasting
model.

Nonetheless, the borderline significance of this example shows that
discriminating between (systemic) risk forecasts requires long samples for the
given parameter choice of $\beta=0.95$. This is as expected, because by only
considering observations with one extreme component, the effective sample size
is massively reduced to roughly $n(1-\beta)$. Indeed, for both models, the
effective out-of-sample period for comparing systemic risk forecasts is
reduced from a length of 4193 to just over 200. The practical implication is
that one should allow for sufficiently large samples to prove the superiority
of the internal model over the benchmark. Alternatively, one could adapt a
slightly looser definition of distress in the reference asset $X$, e.g., by
setting $\beta=0.9$. The current evaluation period for VaR forecasts specified
in the Basel framework by the BCBSBF19 is one year, amounting to roughly 250
daily returns. Even for evaluating VaR and ES forecasts, the horizon of 250
days has been called into question for being too short (DLS20), and this is
only magnified for systemic risk forecasts. So whatever evaluation period for
systemic risk measures is eventually settled on in a regulatory context, it
likely needs to be far in excess of one year.

##  7 Discussion and outlook

To our knowledge, this is the first paper to come up with comparative
backtests for the systemic risk measures CoVaR, CoES and MES, which are
crucial inputs in financial, macroeconomic and regulatory applications. Model
selection procedures based on our results may enhance modelling attempts of
these quantities in financial institutions. Moreover, the fact that our notion
of multi-objective elicitability serves as a ‘truth serum’ implies that the
regulatory framework can be improved by enticing financial institutions to
accurately model systemic risk. In terms of calibration tests, we are only
aware of one more paper: Banulescu-RaduETAL2019 introduce such tests, which
unfortunately hinge on non-strict identification functions. This may lead to a
severe loss of power under the alternative as demonstrated in Remark 4.3 and
Section G. Due to the strictness of our identification functions, such a
phenomenon cannot happen in our context.

The novel concept of multi-objective elicitability is likely to be fruitful
also in applications beyond the proposed DM-backtests for systemic risk
measures. In Subsection B.1, we provide more examples of interesting
situations where conditional elicitability holds, yet classical joint
elicitability fails. By virtue of Theorem 3.7, one can now construct incentive
compatible elicitation mechanisms for these functionals, or can come up with
$M$-estimation procedures in a regression context.

Beyond the confines of finance, we anticipate many other interesting
applications of our backtests. For instance, in economics, ABG19 and Aea21
have recently drawn attention to tail risks and their interconnections by
popularizing the Growth-at-Risk, which is simply the VaR of GDP growth. This
literature has led to an increase in the use of risk forecast evaluation
methods in economics (e.g., BS21). Hence, the backtests developed in this
paper should be relevant for future macroeconomic applications as well.

## References

  * Acharya _et al._ (2017) Acharya VV, Pedersen LH, Philippon T, Richardson M. 2017. Measuring systemic risk. _The Review of Financial Studies_ 30: 2–47.
  * Adrian _et al._ (2019) Adrian T, Boyarchenko N, Giannone D. 2019. Vulnerable growth. _The American Economic Review_ 109: 1263–1289.
  * Adrian and Brunnermeier (2016) Adrian T, Brunnermeier MK. 2016. CoVaR. _The American Economic Review_ 106: 1705–1741.
  * Adrian _et al._ (2021) Adrian T, Grinberg F, Liang N, Malik S, Yu J. 2021. The term structure of Growth-at-Risk. _Forthcoming in American Economic Journal: Macroeconomics_ https://www.aeaweb.org/articles/pdf/doi/10.1257/mac.20180428.
  * Artzner _et al._ (1999) Artzner P, Delbaen F, Eber JM, Heath D. 1999. Coherent measures of risk. _Mathematical Finance_ 9: 203–228.
  * Bank for International Settlements (2019) Bank for International Settlements. 2019. _Basel Framework_. Basel, http://www.bis.org/basel_framework/index.htm?export=pdf.
  * Banulescu-Radu _et al._ (2021) Banulescu-Radu D, Hurlin C, Leymarie J, Scaillet O. 2021. Backtesting marginal expected shortfall and related systemic risk measures. _Management Science_ 67: 5730–5754.
  * Benoit _et al._ (2017) Benoit S, Colliard JE, Hurlin C, Pérignon C. 2017. Where the risks lie: A survey on systemic risk. _Review of Finance_ 21: 109–152.
  * Bernardi and Catania (2019) Bernardi M, Catania L. 2019. Switching generalized autoregressive score copula models with application to systemic risk. _Journal of Applied Econometrics_ 34: 43–65.
  * Brownlees _et al._ (2011) Brownlees C, Engle R, Kelly B. 2011. A practical guide to volatility forecasting through calm and storm. _Journal of Risk_ 14: 3–22.
  * Brownlees and Engle (2017) Brownlees C, Engle RF. 2017. SRISK: A conditional capital shortfall measure of systemic risk. _The Review of Financial Studies_ 30: 48–79.
  * Brownlees and Souza (2021) Brownlees C, Souza ABM. 2021. Backtesting global Growth-at-Risk. _Journal of Monetary Economics_ 118: 312–330.
  * Brunnermeier _et al._ (2020) Brunnermeier M, Rother S, Schnabel I. 2020. Asset price bubbles and systemic risk. _The Review of Financial Studies_ 33: 4272–4317.
  * Chen _et al._ (2013) Chen C, Iyengar G, Moallemi CC. 2013. An Axiomatic Approach to Systemic Risk. _Management Science_ 59: 1373–1388.
  * Creal _et al._ (2013) Creal D, Koopman SJ, Lucas A. 2013. Generalized autoregressive score models with applications. _Journal of Applied Econometrics_ 28: 777–795.
  * Diebold and Mariano (1995) Diebold FX, Mariano RS. 1995. Comparing predictive accuracy. _Journal of Business & Economic Statistics_ 13: 253–263.
  * Dimitriadis _et al._ (2020a) Dimitriadis T, Fissler T, Ziegel JF. 2020a. The Efficiency Gap. _Preprint._ https://arxiv.org/abs/2010.14146.
  * Dimitriadis _et al._ (2020b) Dimitriadis T, Liu X, Schnaitmann J. 2020b. Encompassing tests for value at risk and expected shortfall multi-step forecasts based on inference on the boundary. _Preprint._ https://arxiv.org/abs/2009.07341.
  * Eckernkemper (2018) Eckernkemper T. 2018. Modeling systemic risk: Time-varying tail dependence when forecasting marginal expected shortfall. _Journal of Financial Econometrics_ 16: 63–117.
  * Ehm _et al._ (2016) Ehm W, Gneiting T, Jordan A, Krüger F. 2016. Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ 78: 505–562.
  * Ehrgott (2005) Ehrgott M. 2005. _Multicriteria Optimization_. Berlin, Heidelberg: Springer.
  * Emmer _et al._ (2015) Emmer S, Kratz M, Tasche D. 2015. What is the best risk measure in practice? a comparison of standard measures. _Journal of Risk_ 18: 31–60.
  * Feinstein _et al._ (2017) Feinstein Z, Rudloff B, Weber S. 2017. Measures of systemic risk. _SIAM Journal on Financial Mathematics_ 8: 672–708.
  * Fissler _et al._ (2021) Fissler T, Frongillo R, Hlavinová J, Rudloff B. 2021. Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals. _Electronic Journal of Statistics_ 15: 1034–1084.
  * Fissler and Ziegel (2016) Fissler T, Ziegel JF. 2016. Higher order elicitability and Osband’s principle. _The Annals of Statistics_ 44: 1680–1707.
  * Fissler _et al._ (2016) Fissler T, Ziegel JF, Gneiting T. 2016. Expected shortfall is jointly elicitable with value-at-risk: Implications for backtesting. _Risk Magazine_ : 58–61.
  * Giacomini and White (2006) Giacomini R, White H. 2006. Tests of conditional predictive ability. _Econometrica_ 74: 1545–1578.
  * Giesecke and Kim (2011) Giesecke K, Kim B. 2011. Systemic risk: What defaults are telling us. _Management Science_ 57: 1387–1405.
  * Giglio _et al._ (2016) Giglio S, Kelly B, Pruitt S. 2016. Systemic risk and the macroeconomy: An empirical evaluation. _Journal of Financial Economics_ 119: 457–471.
  * Girardi and Tolga Ergün (2013) Girardi G, Tolga Ergün A. 2013. Systemic risk measurement: Multivariate GARCH estimation of CoVaR. _Journal of Banking & Finance_ 37: 3169–3180.
  * Glosten _et al._ (1993) Glosten LR, Jagannathan R, Runkle DE. 1993. On the relation between the expected value and the volatility of the nominal excess return on stocks. _The Journal of Finance_ 48: 1779–1801.
  * Gneiting (2011) Gneiting T. 2011. Making and evaluating point forecasts. _Journal of the American Statistical Association_ 106: 746–762.
  * Hoga (2021) Hoga Y. 2021. Modeling time-varying tail dependence, with application to systemic risk forecasting. _Forthcoming in Journal of Financial Econometrics_ : 1–31.
  * Mainik and Schaanning (2014) Mainik G, Schaanning E. 2014. On dependence consistency of CoVaR and some other systemic risk measures. _Statistics & Risk Modeling_ 31: 49–77.
  * Mas-Colell _et al._ (1995) Mas-Colell A, Whinston MD, Green JR. 1995. _Microeconomic Theory_. Oxford University Press.
  * Nolde and Zhang (2020) Nolde N, Zhang J. 2020. Conditional extremes in asymmetric financial markets. _Journal of Business & Economic Statistics_ 38: 201–213.
  * Nolde and Ziegel (2017) Nolde N, Ziegel JF. 2017. Elicitability and backtesting: Perspectives for banking regulation. _The Annals of Applied Statistics_ 11: 1833–1874.
  * Steinwart _et al._ (2014) Steinwart I, Pasin C, Williamson R, Zhang S. 2014. Elicitation and Identification of Properties. _JMLR Workshop Conf. Proc._ 35: 1–45.
  * White (2001) White H. 2001. _Asymptotic Theory for Econometricians_. San Diego: Academic Press, First edn.

## Supplement

##  Appendix A Proofs for Section 3

  * Proof of Proposition 3.4::

Let $\bm{V}_{1}\colon\mathsf{A}_{1}\times\mathsf{O}\to\mathbb{R}^{m_{1}}$ be a
strict $\mathcal{F}$-identification function for $\bm{T}_{1}$ and, for any
$\bm{r}_{1}\in\mathsf{A}_{1}$, let
$\bm{V}_{2,\bm{r}_{1}}\colon\mathsf{A}_{2}\times\mathsf{O}\to\mathbb{R}^{m_{2}}$
be a strict $\mathcal{F}_{\bm{r}_{1}}$-identification function for
$\bm{T}_{2}$. We claim that

$\bm{V}\colon(\mathsf{A}_{1}\times\mathsf{A}_{2})\times\mathsf{O}\to\mathbb{R}^{m_{1}+m_{2}},\qquad\big{(}(\bm{r}_{1},\bm{r}_{2}),\bm{y}\big{)}\mapsto\big{(}\bm{V}_{1}(\bm{r}_{1},\bm{y}),\bm{V}_{2,\bm{r}_{1}}(\bm{r}_{2},\bm{y})\big{)}$

is a strict $\mathcal{F}$-identification function for
$(\bm{T}_{1},\bm{T}_{2})$. Consider some $F\in\mathcal{F}$ and
$(\bm{r}_{1},\bm{r}_{2})\in\mathsf{A}_{1}\times\mathsf{A}_{2}$. First, assume
$(\bm{r}_{1},\bm{r}_{2})\in\bm{T}_{1}(F)\times\bm{T}_{2}(F)$. Then
$\bar{\bm{V}}_{1}(\bm{r}_{1},F)=\bm{0}$ and
$\bar{\bm{V}}_{2,\bm{r}_{1}}(\bm{r}_{2},F)=\bm{0}$. Second, assume
$\bar{\bm{V}}_{1}(\bm{r}_{1},F)=\bm{0}$ and
$\bar{\bm{V}}_{2,\bm{r}_{1}}(\bm{r}_{2},F)=\bm{0}$. Due to the former equality
and the strictness of $\bm{V}_{1}$, it holds that
$\bm{r}_{1}\in\bm{T}_{1}(F)$. Now we can invoke the strictness of
$\bm{V}_{2,\bm{r}_{1}}$ on $\mathcal{F}_{\bm{r}_{1}}$ to conclude that
$\bm{r}_{2}\in\bm{T}_{2}(F)$. ∎

  * Proof of Theorem 3.7::

Let $S_{1}\colon\mathsf{A}_{1}\times\mathsf{O}\to\mathbb{R}$ be a strictly
$\mathcal{F}$-consistent scoring function for $\bm{T}_{1}$. For any
$\bm{r}_{1}\in\mathsf{A}_{1}$ let
$S_{2,\bm{r}_{1}}\colon\mathsf{A}_{2}\times\mathsf{O}\to\mathbb{R}$ be a
strictly $\mathcal{F}_{\bm{r}_{1}}$-consistent scoring function for
$\bm{T}_{2}$. We claim that

$\bm{S}\colon(\mathsf{A}_{1}\times\mathsf{A}_{2})\times\mathbb{R}^{d}\to(\mathbb{R}^{2},\preceq_{\mathrm{lex}}),\qquad\big{(}(\bm{r}_{1},\bm{r}_{2}),\bm{y}\big{)}\mapsto\big{(}S_{1}(\bm{r}_{1},\bm{y}),S_{2,\bm{r}_{1}}(\bm{r}_{2},\bm{y})\big{)}$
(A.1)

is a strictly multi-objective $\mathcal{F}$-consistent scoring function for
$(\bm{T}_{1},\bm{T}_{2})$ with respect to $\preceq_{\mathrm{lex}}$. To see
this, consider some $F\in\mathcal{F}$ with
$(\bm{t}_{1},\bm{t}_{2})\in\bm{T}_{1}(F)\times\bm{T}_{2}(F)$ and
$(\bm{r}_{1},\bm{r}_{2})\in\mathsf{A}_{1}\times\mathsf{A}_{2}$. From the
strict $\mathcal{F}$-consistency of $S_{1}$ it follows that

$\bar{S}_{1}(\bm{t}_{1},F)\leq\bar{S}_{1}(\bm{r}_{1},F),$ (A.2)

where the inequality is strict if $\bm{r}_{1}\notin\bm{T}_{1}(F)$. If the
inequality is strict, we can already conclude that
$\bar{\bm{S}}(\bm{t}_{1},\bm{t}_{2},F)\prec_{\mathrm{lex}}\bar{\bm{S}}(\bm{r}_{1},\bm{r}_{2},F).$
Otherwise, it must be the case that $\bm{r}_{1}\in\bm{T}_{1}(F)$. Since by
assumption $\bm{T}_{1}(F)$ is a singleton, $\bm{r}_{1}=\bm{t}_{1}$. Invoking
the strict $\mathcal{F}_{\bm{t}_{1}}$-consistency of $S_{2,\bm{t}_{1}}$, we
obtain that

$\bar{S}_{2,\bm{t}_{1}}(\bm{t}_{2},F)\leq\bar{S}_{2,\bm{t}_{1}}(\bm{r}_{2},F),$
(A.3)

where the inequality is strict if $\bm{r}_{2}\notin\bm{T}_{2}(F)$. If the
inequality is strict, this implies that
$\bar{\bm{S}}((\bm{t}_{1},\bm{t}_{2}),F)\prec_{\mathrm{lex}}\bar{\bm{S}}((\bm{r}_{1},\bm{r}_{2}),F).$
Otherwise, equality implies that $\bm{r}_{2}\in\bm{T}_{2}(F)$. ∎

###### Remark A.1.

Theorem 3.7 is almost a direct analogue of Proposition 3.4, with the
intriguing exception that $\bm{T}_{1}$ is assumed to be a singleton on
$\mathcal{F}$ in Theorem 3.7. An inspection of the proof reveals that this is
needed to show the inequality in (A.3). If at this stage of the proof one has
that $\bm{t}_{1},\bm{r}_{1}\in\bm{T}_{1}(F)$ and $\bm{t}_{1}\neq\bm{r}_{1}$,
it is per se not clear (and in general also not the case) that
$S_{2,\bm{t}_{1}}=S_{2,\bm{r}_{1}}$ from which the inequality in (A.3) would
follow. What would be sufficient, however, to dispense with the assumption on
$\bm{T}_{1}$ and to still show (A.3) is the following: It is possible to
choose a family of strictly $\mathcal{F}_{\bm{r}_{1}}$-consistent scores
$S_{2,\bm{r}_{1}}$, $\bm{r}_{1}\in\mathsf{A}_{1}$, for $\bm{T}_{2}$ such that
for all $F\in\mathcal{F}$ and for all $\bm{t}_{1},\bm{r}_{1}\in\bm{T}_{1}(F)$
we have equality for the expected scores,
$\bar{S}_{2,\bm{t}_{1}}(\cdot,F)=\bar{S}_{2,\bm{r}_{1}}(\cdot,F)$. In many
practical examples, this can be achieved; see, e.g., Example B.1 (b).

##  Appendix B More background on multi-objective elicitability

###  B.1 Further examples of multi-objective elicitable functionals

We provide more examples of functionals that are conditionally elicitable, yet
fail to be elicitable in the traditional sense. Examples (c) and (d) are
concerned with $\alpha$-prediction intervals. In line with FFHR2021 for some
random variable $Y$ with distribution $F$, this is any interval
$[a,b]\subseteq\mathbb{R}$ such that
$\operatorname{P}(Y\in[a,b])=F(b)-F(a-)\geq\alpha$. Of course, each
distribution $F$ has a multitude of different $\alpha$-prediction intervals.
We consider two possible attempts of coming up with _interpretable_
$\alpha$-prediction intervals.

Two univariate scores
$S,\tilde{S}\colon\mathsf{A}\times\mathsf{O}\to\mathbb{R}$ are _equivalent_ if
there exists a positive constant $c>0$ and a function
$a\colon\mathsf{O}\to\mathbb{R}$ such that
$\tilde{S}(\bm{r},\bm{y})=cS(\bm{r},\bm{y})+a(\bm{y})$. Ignoring integrability
issues for a moment, it is clear that equivalence preserves (strict)
consistency. If $S$ and $a$ are $\mathcal{F}$-integrable, so is $\tilde{S}$.
However, by a convenient choice of $a$, the integrability conditions of
$\tilde{S}$ can be milder than those of $S$. E.g., while $S(r,y)=(r-y)^{2}$,
$r,y\in\mathbb{R}$, is a strictly $\mathcal{F}^{2}(\mathbb{R})$-consistent
scoring function for the mean, the equivalent score $\tilde{S}(r,y)=r^{2}-2ry$
is strictly $\mathcal{F}^{1}(\mathbb{R})$-consistent score for the mean. A
straightforward generalisation is obvious for multi-objective scores of the
form (A.1), i.e.,

$\bm{S}\colon(\mathsf{A}_{1}\times\mathsf{A}_{2})\times\mathbb{R}^{d}\to(\mathbb{R}^{2},\preceq_{\mathrm{lex}}),\qquad\big{(}(\bm{r}_{1},\bm{r}_{2}),\bm{y}\big{)}\mapsto\big{(}S_{1}(\bm{r}_{1},\bm{y}),S_{2,\bm{r}_{1}}(\bm{r}_{2},\bm{y})\big{)}^{\prime}.$
(B.1)

###### Example B.1.

  1. (a)

The variance functional is conditionally elicitable with the mean functional
on $\mathcal{F}^{4}(\mathbb{R})$. A strictly multi-objective
$\mathcal{F}^{4}(\mathbb{R})$-consistent score
$\bm{S}\colon\mathbb{R}^{2}\times\mathbb{R}\to(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$
is given by (B.1), where $S_{1}(r_{1},y)=(r_{1}-y)^{2}$ and
$S_{2,r_{1}}(r_{2},y)=\big{[}r_{2}-(y-r_{1})^{2}\big{]}^{2}.$ Alternatively,
to achieve a strictly multi-objective $\mathcal{F}^{2}(\mathbb{R})$-consistent
score, one can consider the equivalent version
$\tilde{S}_{2,r_{1}}(r_{2},y)=r_{2}^{2}-2r_{2}(y-r_{1})^{2}$, or more
generally
$\tilde{S}_{2,r_{1}}(r_{2},y)=-\phi(r_{2})+\phi^{\prime}(r_{2})[r_{2}-(y-r_{1})]$,
where $\phi$ is strictly convex with subgradient $\phi^{\prime}$.

  2. (b)

For $\alpha\in(0,1)$, $\operatorname{ES}_{\alpha}$ is conditionally elicitable
with the $\alpha$-quantile, $q_{\alpha}$, on $\mathcal{F}^{2}(\mathbb{R})$. A
strictly multi-objective $\mathcal{F}^{2}(\mathbb{R})$-consistent score
$\bm{S}\colon\mathbb{R}^{2}\times\mathbb{R}\to(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$
is given via (B.1), where $S_{1}(r_{1},y)=(\mathds{1}\\{y\leq
r_{1}\\}-\alpha)(r_{1}-y)$ and
$S_{2,r_{1}}(r_{2},y)=\Big{\\{}r_{2}-\frac{1}{1-\alpha}\big{[}\mathds{1}\\{y>r_{1}\\}y+r_{1}(\mathds{1}\\{y\leq
r_{1}\\}-\alpha)\big{]}\Big{\\}}^{2}.$ It is straightforward to verify that
this family of scores satisfies the condition discussed in Remark A.1. Similar
to (a), we can achieve strict multi-objective
$\mathcal{F}^{1}(\mathbb{R})$-consistency upon replacing $S_{2,r_{1}}$ by
$\tilde{S}_{2,r_{1}}(r_{2},y)=r_{2}^{2}-\frac{2r_{2}}{1-\alpha}\big{[}\mathds{1}\\{y>r_{1}\\}y+r_{1}(\mathds{1}\\{y\leq
r_{1}\\}-\alpha)\big{]}$, or more generally by
$\tilde{S}_{2,r_{1}}(r_{2},y)=-\phi(r_{2})+\phi^{\prime}(r_{2})\Big{\\{}r_{2}-\frac{1}{1-\alpha}\big{[}\mathds{1}\\{y>r_{1}\\}y+r_{1}(\mathds{1}\\{y\leq
r_{1}\\}-\alpha)\big{]}\Big{\\}}$, where $\phi$ is strictly convex with
subgradient $\phi^{\prime}$.

  3. (c)

Fix $\alpha\in(0,1)$. Motivated by prediction intervals induced by quantiles
of the form $[q_{\beta}^{-}(F),$ $q_{\alpha+\beta}^{-}(F)]$ for
$\beta\in[0,1-\alpha]$ we consider the following form of $\alpha$-prediction
intervals. Let $T_{1}\colon\mathcal{F}\to\mathbb{R}$ be elicitable on
$\mathcal{F}\subseteq\mathcal{F}_{\textrm{inc}}\subset\mathcal{F}^{0}(\mathbb{R})$,
where all $F\in\mathcal{F}_{\textrm{inc}}$ are strictly increasing cdfs, and
let $T_{2}\colon\mathcal{F}\to(-\infty,\infty]$ be the right endpoint of the
shortest $\alpha$-prediction interval $[T_{1}(F),T_{2}(F)]$ whose left
endpoint is given via $T_{1}$. FFHR2021 asserts that $(T_{1},T_{2})$ is not
elicitable, subject to smoothness conditions, but FFHR2021 yields that $T_{2}$
is conditionally elicitable with $T_{1}$ on $\mathcal{F}$. A strictly multi-
objective $\mathcal{F}$-consistent scoring function
$\bm{S}\colon\mathbb{R}^{2}\times\mathbb{R}\to(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$
is given by (B.1), where $S_{1}(r_{1},y)$ is a strictly
$\mathcal{F}$-consistent score for $T_{1}$ and

$S_{2,r_{1}}(r_{2},y)=\mathds{1}\\{y\geq
r_{1}\\}\Big{[}\big{(}\mathds{1}\\{y\leq
r_{2}\\}-\alpha\big{)}\big{(}g(x)-g(y)\big{)}\Big{]}-\mathds{1}\\{y<r_{1}\\}\alpha
g(r_{2}),$

where $g\colon(-\infty,\infty]\to\mathbb{R}$ is strictly increasing and
$\mathcal{F}$-integrable.

  4. (d)

A different approach to constructing interpretable $\alpha$-prediction
intervals is to specify the midpoint in terms of an elicitable functional,
such as the mean or the median. Again, let $\alpha\in(0,1)$ and let
$T_{1}\colon\mathcal{F}\to\mathbb{R}$ be elicitable on
$\mathcal{F}\subseteq\mathcal{F}_{\textrm{inc}}$ and let
$T_{2}\colon\mathcal{F}\to[0,\infty)$ be half of the length of the shortest
$\alpha$-prediction interval $[T_{1}(F)-T_{2}(F),T_{1}(F)+T_{2}(F)]$ with
midpoint $T_{1}$. FFHR2021 asserts that $(T_{1},T_{2})$ is not elicitable,
subject to smoothness conditions, but FFHR2021 yields that $T_{2}$ is
conditionally elicitable with $T_{1}$ on $\mathcal{F}$. A strictly multi-
objective $\mathcal{F}$-consistent scoring function
$\bm{S}\colon\mathbb{R}^{2}\times\mathbb{R}\to(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$
is given via (B.1), where $S_{1}(r_{1},y)$ is a strictly
$\mathcal{F}$-consistent score for $T_{1}$ and

$S_{2,r_{1}}(r_{2},y)=\big{(}\mathds{1}\\{|y-r_{1}|\leq
r_{2}\\}-\alpha\big{)}\big{[}g(r_{2})-g(|y-r_{1}|)\big{]},$

where $g\colon[0,\infty)\to\mathbb{R}$ is strictly increasing with
$g(|\cdot-\,r_{1}|)$ being $\mathcal{F}$-integrable for all
$r_{1}\in\mathbb{R}$.

  5. (e)

For $\alpha\in(0,1)$, let $\mathcal{F}^{0}_{(\alpha)}(\mathbb{R})$ as in
(4.1). Define the right $(1-\alpha)$-tail of a distribution
$F\in\mathcal{F}^{0}_{(\alpha)}(\mathbb{R})$ as
$T_{\alpha}(F)\in\mathcal{F}^{0}(\mathbb{R})$ where
$T_{\alpha}(F)(x)=\mathds{1}\\{x>\operatorname{VaR}_{\alpha}(F)\\}(F(x)-\alpha)/(1-\alpha)$.
Then, $T_{\alpha}$ is conditionally elicitable with
$\operatorname{VaR}_{\alpha}$ on $\mathcal{F}^{0}_{(\alpha)}(\mathbb{R})$. A
strictly multi-objective $\mathcal{F}^{0}_{(\alpha)}(\mathbb{R})$-consistent
score
$\bm{S}\colon\mathbb{R}\times\mathcal{F}^{0}_{(\alpha)}(\mathbb{R})\times\mathbb{R}\to(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$
is given via (B.1), where $S_{1}(r,y)=(\mathds{1}\\{y\leq
r\\}-\alpha)[g(r)-g(y)]$ with strictly increasing and bounded
$g\colon\mathbb{R}\to\mathbb{R}$, and the second component of $\bm{S}$ is
given by $S_{2,r}(F,y)=\mathds{1}\\{y>r\\}R(F,y).$ Here,
$R\colon\mathcal{F}^{0}(\mathbb{R})\times\mathbb{R}\to\mathbb{R}$ is a
strictly proper scoring rule. This result follows from Gne11a; see also
Theorem C.2 and HolzmannKlar2017 for related approaches.

###  B.2 Dimensionality considerations

Consider an elicitable functional
$\bm{T}\colon\mathcal{F}\to\mathcal{P}(\mathsf{A})$ with strictly
$\mathcal{F}$-consistent score
$S\colon\mathsf{A}\times\mathsf{O}\to\mathbb{R}$, where
$\mathsf{A}\subseteq\mathbb{R}^{k}$. Under sufficient smoothness conditions on
the score $S(\cdot,\bm{y})$ and the expected score $\bar{S}(\cdot,F)$, first
order conditions yield that the $k$-dimensional gradient of $S$ is an
$\mathcal{F}$-identification function for $\bm{T}$ (which is not necessarily
strict). Under the conditions of Theorem 3.7, first order conditions of the
corresponding strictly multi-objective $\mathcal{F}$-consistent score (B.1)
also induce an $\mathcal{F}$-identification function whose dimension coincides
with $\bm{T}$. To illustrate this, suppose that $\bm{T}_{1}$ and $\bm{T}_{2}$
are univariate, such that $k_{1}+k_{2}=k=2$. We have the scores
$\bm{S}((r_{1},r_{2}),\bm{y})=\big{(}S_{1}(r_{1},\bm{y}),S_{2,r_{1}}(r_{2},\bm{y})\big{)}^{\prime}\in(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$.
At the optimum $(t_{1},t_{2})\in T_{1}(F)\times T_{2}(F)$ we obtain the first
order condition $\partial\bar{S}_{1}(r_{1},F)=0$ and
$\partial\bar{S}_{2,r_{1}}(r_{2},F)=0$. Hence, if the derivatives exist and
tacitly assuming that integration and differentiation commute,
$\bm{V}((r_{1},r_{2}),\bm{y})=\big{(}\partial S_{1}(r_{1},\bm{y}),\partial
S_{2,r_{1}}(r_{2},\bm{y})\big{)}^{\prime}$ constitutes a two-dimensional
$\mathcal{F}$-identification function for $(T_{1},T_{2})$. This means we can
consider the two-dimensional score (B.1) as a generalised antiderivative of a
$k$-dimensional identification function where the symmetry conditions imposed
by the Hessian are massively relaxed, which renders the existence of such an
object possible. Thus, multi-objective scores provide a means to close the gap
between identification and scoring functions for multivariate functionals.

###  B.3 More on multi-objective elicitability with respect to the
lexicographic order

###### Remark B.2.

Resuming with the discussion right before Theorem 3.7, the use of the
lexicographic order on $\mathbb{R}^{2}$ allows to compare any forecasts, also
misspecified ones. As for the classical univariate concept of strict
consistency, strong consistency stays silent about the ranking of possibly
misspecified forecasts, which is, however, the more realistic scenario
(Patton2020). For univariate functionals, consistency implies order-
sensitivity under mild conditions (BelliniBignozzi2015; Lambert2019): If two
forecasts are both smaller or larger than the true functional value, the one
closer to the true value achieves an expected score at most as large as the
other forecast. For multivariate forecasts, there are various generalisations
of order-sensitivity (see, e.g., LambertETAL2008; FisslerZiegel2019). For
multi-objective scores similar order-sensitivity results would be desirable.
We suspect that the componentwise order-sensitivity concept would be
particularly promising in that regard.

###### Remark B.3.

On the level of the prediction space setting (GneitingRanjan2013),
HolzmannEulert2014 establish that consistent scoring functions are sensitive
with respect to increasing information sets. That is, when comparing two ideal
forecasts based on nested information sets, the more informed forecast
outperforms the less informed one on average. This is an instance of the more
general calibration–resolution principle of Pohle2020, showing that minimising
a consistent scoring function amounts to “jointly maximizing information
content and minimizing systematic mistakes”. Using the same arguments as in
HolzmannEulert2014 and Pohle2020 (basically exploiting the tower property of
conditional expectations), one can establish a similar principle for multi-
objective consistent scores mapping to
$(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$.

###  B.4 The necessity of the Convex Level Sets property

Here, we revisit a powerful necessary condition for identifiability and
elicitability, namely the Convex Level Sets (CxLS) property, for multi-
objective scores. We use the same notation as in Section 3 of the main paper,
in particular we let
$\mathcal{F}\subseteq\mathcal{F}^{\prime}\subseteq\mathcal{F}^{0}(\mathsf{O})$.
Following the terminology of FFHR2021 we say that a functional
$\bm{T}\colon\mathcal{F}^{\prime}\to\mathcal{P}(\mathsf{A})$, satisfies the
_selective CxLS property_ on $\mathcal{F}$ if for all
$F_{0},F_{1}\in\mathcal{F}$ and for all $\lambda\in(0,1)$ such that
$F_{\lambda}\mathrel{\mathop{\mathchar 58\relax}}=(1-\lambda)F_{0}+\lambda
F_{1}\in\mathcal{F}$ we have
$\bm{T}(F_{0})\cap\bm{T}(F_{1})\subseteq\bm{T}(F_{\lambda}).$ $\bm{T}$
satisfies the _selective CxLS* property_ on $\mathcal{F}$ if for all
$F_{0},F_{1}\in\mathcal{F}$ and for all $\lambda\in(0,1)$ such that
$F_{\lambda}\mathrel{\mathop{\mathchar 58\relax}}=(1-\lambda)F_{0}+\lambda
F_{1}\in\mathcal{F}$ $\bm{T}(F_{0})\cap\bm{T}(F_{1})\neq\emptyset$ implies
that $\bm{T}(F_{0})\cap\bm{T}(F_{1})=\bm{T}(F_{\lambda})$. We shall frequently
omit the term “selective” and will just speak of the CxLS and the CxLS*
property. Clearly, the CxLS* property implies the CxLS property. If $\bm{T}$
attains singletons only on $\mathcal{F}$, then the two properties coincide.
They then take the familiar form that $\bm{T}(F_{0})=\bm{T}(F_{1})$ implies
$\bm{T}(F_{0})=\bm{T}(F_{\lambda})$. It is well known that the CxLS property
is necessary for identifiability (which we state for the sake of
completeness). Proposition 3.4 in FFHR2021 establishes that the CxLS* property
is necessary for univariate elicitability. The following generalises this
result to strong multi-objective elicitability.

######  Proposition B.4.

Consider a functional
$\bm{T}\colon\mathcal{F}^{\prime}\to\mathcal{P}(\mathsf{A})$.

  1. (i)

If $\bm{T}$ is identifiable on $\mathcal{F}\subseteq\mathcal{F}^{\prime}$, it
satisfies the CxLS property on $\mathcal{F}$.

  2. (ii)

If $\bm{T}$ is strongly multi-objective elicitable on
$\mathcal{F}\subseteq\mathcal{F}^{\prime}$, it satisfies the CxLS* property on
$\mathcal{F}$.

  * Proof of Proposition B.4::

For (i), let $\bm{V}\colon\mathsf{A}\times\mathsf{O}\to\mathbb{R}^{m}$ be a
strict identification function. Let $F_{0},F_{1}\in\mathcal{F}$ such that
$F_{\lambda}\in\mathcal{F}$ for some $\lambda\in(0,1)$. For
$\bm{t}\in\bm{T}(F_{0})\cap\bm{T}(F_{1})$ it holds that
$\bm{0}=(1-\lambda)\bar{\bm{V}}(\bm{t},F_{0})+\lambda\bar{\bm{V}}(\bm{t},F_{1})=\bar{\bm{V}}(\bm{t},F_{\lambda}).$
Due to the strictness of $\bm{V}$, we have $\bm{t}\in\bm{T}(F_{\lambda})$.
For (ii), the proof follows along the lines of the proof of Proposition 3.4 in
FFHR2021. With the same set-up as above, consider some strictly strongly
multi-objective $\mathcal{F}$-consistent scoring function
$\bm{S}\colon\mathsf{A}\times\mathsf{O}\to(\mathbb{R}^{\mathcal{I}},\preceq)$.
For $i\in\\{0,1\\}$, the strict strong multi-objective
$\mathcal{F}$-consistency of $\bm{S}$ implies that for any
$\bm{r}\in\mathsf{A}$ we have
$\bar{\bm{S}}(\bm{t},F_{i})-\bar{\bm{S}}(\bm{r},F_{i})=\bm{0}$ for
$\bm{r}\in\bm{T}(F_{i})$ and for $\bm{r}\notin\bm{T}(F_{i})$ the score
difference is $\prec\bm{0}$. Hence,
$\bar{\bm{S}}(\bm{t},F_{\lambda})-\bar{\bm{S}}(\bm{r},F_{\lambda})$ equals
$(1-\lambda)\big{(}\bar{\bm{S}}(\bm{t},F_{0})-\bar{\bm{S}}(\bm{r},F_{0})\big{)}+\lambda\big{(}\bar{\bm{S}}(\bm{t},F_{1})-\bar{\bm{S}}(\bm{r},F_{1})\big{)},$
which is $\bm{0}$ for $\bm{r}\in\bm{T}(F_{0})\cap\bm{T}(F_{1})$ and otherwise
$\prec\bm{0}$. ∎

Conditional IdentifiabilityConditional ElicitabilityIdentifiability Multi-
Objective Elicitability with respect to
$(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$ ElicitabilityCxLSCxLS*1)Theorem
3.7Proposition 3.4Proposition B.4Proposition B.4 Figure 7: Illustration of the
most important structural results for functionals
$\bm{T}_{1}\mathrel{\mathop{\mathchar
58\relax}}\mathcal{F}\rightarrow\mathcal{P}(\mathsf{A}_{1})$ and
$\bm{T}_{2}\mathrel{\mathop{\mathchar
58\relax}}\mathcal{F}\rightarrow\mathcal{P}(\mathsf{A}_{2})$. The equivalence
1) holds under some regularity conditions if $\bm{T}_{1}$ and $\bm{T}_{2}$ are
point-valued mappings to $\mathbb{R}$ (SteinwartPasinETAL2014).

Proposition B.4 implies that strong multi-objective elicitability shares an
important feature with the usual univariate notion of elicitability. Showing
that the CxLS* is violated is the standard procedure to rule out
elicitability. Consequently, functionals violating the CxLS* property such as
variance or ES also fail to be strongly multi-objective elicitable. This
applies in particular to scoring functions mapping to
$(\mathbb{R}^{2},\preceq_{\mathrm{lex}})$, since $\preceq_{\mathrm{lex}}$ is a
total order.

On the other hand, weak multi-objective elicitability appears to be much more
flexible than its strong counterpart. Indeed, the following consideration
shows that the CxLS property is not necessary for weak multi-objective
elicitability: Suppose we have a strictly weakly multi-objective
$\mathcal{F}$-consistent scoring function $\bm{S}$ for $\bm{T}$ mapping to
$(\mathbb{R}^{2},\preceq)$, where $\preceq$ is the componentwise order.
Suppose further that there are $F_{0},F_{1}\in\mathcal{F}$ such that
$F_{\lambda}=(1-\lambda)F_{0}+\lambda F_{1}\in\mathcal{F}$ for all
$\lambda\in(0,1)$. Assume there is some
$\bm{t}\in\bm{T}(F_{0})\cap\bm{T}(F_{1})$ and some
$\bm{r}\notin\bm{T}(F_{0})\cup\bm{T}(F_{1})$ such that
$\bar{\bm{S}}(\bm{r},F_{0})-\bar{\bm{S}}(\bm{t},F_{0})=(-5,1)^{\prime}\nprec(0,0)^{\prime}$
and
$\bar{\bm{S}}(\bm{r},F_{1})-\bar{\bm{S}}(\bm{t},F_{1})=(1,-5)^{\prime}\nprec(0,0)^{\prime}$.
Then, invoking the linearity of the integral,
$\bar{\bm{S}}(\bm{r},F_{1/2})-\bar{\bm{S}}(\bm{t},F_{1/2})=(-2,-2)^{\prime}\prec(0,0)^{\prime}$.
Therefore, $\bm{t}\notin\bm{T}(F_{1/2})$ and the CxLS property is violated
(and thus also the CxLS* property).

A further investigation of weak multi-objective elicitability therefore seems
to be an interesting field. Such an investigation will be deferred to future
work.

As a complete summary of the results of this section and Section 3, Figure 7
complements Figure 2 in the main paper by including the implications of
Proposition B.4.

##  Appendix C Conditional identifiability and conditional elicitability
results

Here, we show that $\operatorname{CoVaR}_{\alpha|\beta}(Y|X)$,
$\operatorname{MES}_{\beta}(Y|X)$ and
$(\operatorname{CoVaR}_{\alpha|\beta}(Y|X),\operatorname{CoES}_{\alpha|\beta}(Y|X))$
are all conditionally elicitable and conditionally identifiable with
$\operatorname{VaR}_{\beta}(X)$, subject to mild assumptions on the
corresponding class of bivariate distributions.

###### Remark C.1.

Our definitions of the systemic risk measures in Section 2 rely on the
conditional cdf $F_{Y\mid X\geq\operatorname{VaR}_{\beta}(X)}$; see (2.1). If
$F_{X}$ is continuous, the event $\\{X\geq\operatorname{VaR}_{\beta}(X)\\}$
has probability $1-\beta$, such that conditioning on the right tail is akin to
the classical definition of expected shortfall at level $\beta$, as