§ INTRODUCTION: WEIGHTED SCORING RULES

When predicting future events, it is common to issue forecasts that are probabilistic. Probabilistic forecasts generally take the form of probability distributions over the range of possible outcomes, comprehensively describing the predictive uncertainty. To assess the quality of a probabilistic forecast, scoring rules are functions $S(F, y)$ that take a forecast $F$ and the corresponding outcome $y$ as inputs, and output a numerical score that quantifies the forecast's accuracy. Scoring rules therefore condense forecast performance into a single value, providing a convenient framework with which to objectively rank and compare competing forecasts. As such, scoring rules have become a key component of probabilistic forecast evaluation.

To assess probabilistic forecasts in practice, the scoringRules package [Jordan et al., 2019] in the programming language R has become a widely used resource. The package contains analytical formulae for the two most popular univariate scoring rules — the logarithmic score <cit.> and the continuous ranked probability score <cit.> — for forecast distributions belonging to a range of parametric families. These scoring rules are also available when the forecast is a sample from a predictive distribution, which is often the case in practice. The scoringRules package additionally allows samples from multivariate forecast distributions to be evaluated using popular multivariate scoring rules, including the energy score [Gneiting and Raftery, 2007], variogram score [Scheuerer and Hamill, 2015], and a kernel score based on the Gaussian kernel [Gneiting and Raftery, 2007].

The scoring rules listed above assess forecasts made for all outcomes. While this is clearly desirable when assessing overall forecast performance, it is often the case that certain outcomes are of more interest than others. For example, one could argue that it is particularly important to issue accurate forecasts for outcomes that have a high impact on the forecast users. To emphasise particular outcomes during forecast evaluation, weighted scoring rules generalise conventional scoring rules by incorporating a weight function into the score. The weight function can be chosen such that a higher weight is assigned to outcomes that are of more interest. Weighted scoring rules therefore allow competing forecast systems to be ranked and compared when predicting particular outcomes, facilitating very flexible, user-oriented forecast evaluation.

Well known examples of weighted scoring rules include the conditional and censored likelihood scores proposed by [Diks et al., 2011], and the threshold-weighted CRPS introduced by [Matheson and Winkler, 1976] and [Gneiting and Ranjan, 2011]. However, the theory underlying weighted scoring rules extends beyond these examples: [Holzmann and Klar, 2017] demonstrate that the conditional and censored likelihood scores can be generalised to construct weighted versions of any proper scoring rule, while [Allen et al., 2022] introduce a broad generalisation of the threshold-weighted CRPS that can be applied, for example, to probabilistic forecasts for multivariate outcomes.

In this paper, we describe how the scoringRules package has been extended to additionally permit the implementation of popular weighted scoring rules. While several alternative software packages exist to calculate particular scoring rules in certain situations <cit.>, the development of weighted scoring rules is more recent. Until recently, for example, efficient application of popular weighted scoring rules was limited by theoretical considerations, leading to ad hoc implementations in practice [Sharpe et al., 2018]. Hence, to our knowledge, no other packages exist that provide a comprehensive collection of weighted scoring rules. We therefore hope that this extension to scoringRules will greatly facilitate the successful implementation of weighted scoring rules in practical applications.

In the following section, we review the existing theory of weighted scoring rules, and introduce examples of weighted versions of several popular scores, such as the LogS, the CRPS, and the energy score. The remainder of the paper then illustrates how these weighted scoring rules can be implemented in practice using the scoringRules package. Section <ref> outlines the functionality of the package when calculating weighted scoring rules, and discusses implementation options. Section <ref> then presents two case studies in which these weighted scoring rules are used to target particular outcomes when evaluating probabilistic forecasts in practice. These case studies include applications in weather forecasting and economic forecasting, building on the examples presented in [Jordan et al., 2019]. A summary of the paper is presented in Section <ref>.

§ THEORETICAL BACKGROUND

§.§ Proper scoring rules

Suppose we are interested in predicting a random variable $Y$ that takes values in a set $\Omega$, and that our forecasts are in the form of probability distributions over $\Omega$. Let $\cF$ denote a set of such forecasts. A scoring rule is a function

\begin{equation*}
S: \cF \times \Omega \to \R \cup \{ -\infty, \infty \},
\end{equation*}

which takes a forecast $F \in \cF$ and an observation $y \in \Omega$ as inputs, and outputs a numerical value, or score, that quantifies the forecast accuracy. A lower score is assigned to a more accurate forecast. A scoring rule is proper with respect to $\cF$ if, when the observations are drawn from a distribution $G \in \cF$, the scoring rule is minimised in expectation by issuing $G$ as the forecast, i.e.

\begin{equation*}
\E_{Y \sim G} S(G, Y) \leq \E_{Y \sim G} S(F, Y)
\end{equation*}

for all $F, G \in \cF$. If the above inequality is strict, then $S$ is strictly proper with respect to $\cF$.

When the outcome variable is real-valued ($\Omega \subseteq \R$), probabilistic forecasts are typically evaluated using either the logarithmic score (LogS) or the continuous ranked probability score (CRPS). The LogS is defined as

\begin{equation}\label{eq:logs}
\mathrm{LogS}(F, y) = - \mathrm{log} f(y),
\end{equation}

where $f$ is the predictive density associated with the cumulative distribution function $F$ [Good, 1952]. The CRPS is defined as

\begin{equation}\label{eq:crps}
\begin{split}
\CRPS(F, y) &= \int_{\R} (F(z) - \ind\{ y \leq z\})^{2} \dd z \\
&= \E_{F}|X - y| - \frac{1}{2} \E_{F} |X - X^{\prime}|,
\end{split}
\end{equation}

where $\ind$ is the indicator function, $X, X^{\prime} \sim F$ are independent random variables, and it is assumed in the second expression that $F$ has a finite mean [Matheson and Winkler, 1976, Gneiting and Raftery, 2007].

Generalisations of the LogS and the CRPS are also commonly used to evaluate probabilistic forecasts for multivariate outcomes, i.e. $\Omega \subseteq \R^{d}$ for $d > 1$. While the LogS in Equation <ref> can readily be applied to multivariate predictive densities, it is often the case that only a sample from the multivariate forecast distribution is available, making it difficult to employ the LogS in practice. Instead, alternative scoring rules have been proposed to evaluate multivariate probabilistic forecasts that can readily be applied to samples from a forecast distribution.

Arguably the most well known multivariate scoring rule is the energy score <cit.>, which generalises the CRPS to higher dimensions:

\begin{equation}\label{eq:es}
\mathrm{ES}(F, \by) = \E_{F} \| \bX - \by \| - \frac{1}{2} \E_{F} \| \bX - \bX^{\prime} \|,
\end{equation}

where $\|\cdot\|$ is the Euclidean distance in $\R^{d}$, $\by = (y_{1}, \dots, y_{d}) \in \Omega$, and $\bX = \left( X_{1}, \dots, X_{d} \right) $, $\bX^{\prime} = \left( X_{1}^{\prime}, \dots, X_{d}^{\prime} \right) \sim F$ are independent, with $F$ a probability distribution on $\Omega$. It is assumed here and throughout that the expectations are finite where necessary.

An alternative to the energy score is the variogram score <cit.>. The variogram score aims to explicitly assess the dependence structure of the multivariate forecast distributions by measuring the distance between the variogram of the forecast and that of the observation. The variogram score of order $p > 0$ is defined as

\begin{equation}\label{eq:vs}
\mathrm{VS}^{p}(F, \by) = \sum_{i=1}^{d} \sum_{j=1}^{d} h_{i,j} \left( \E_{F} \left| X_{i} - X_{j} \right| ^{p} - \left| y_{i} - y_{j} \right| ^{p} \right)^{2},
\end{equation}

where $\bX = \left( X_{1}, \dots, X_{d} \right) \sim F$, $\by = \left( y_{1}, \dots, y_{d} \right) \in \Omega$, and $h_{i,j}$ are non-negative scaling parameters that control how much emphasis is given to a pair of dimensions. Following recommendations from [Scheuerer and Hamill, 2015], the order of the score, $p$, is often chosen to be 0.5.

Both the energy score and the variogram score belong to the very general class of kernel scores [Gneiting and Raftery, 2007]. Kernel scores are scoring rules that are constructed using conditionally negative definite kernels, and the kernel score framework has also been leveraged to introduce alternative multivariate scoring rules. [Allen et al., 2022], for example, introduced a multivariate scoring rule based on the inverse multiquadric kernel, while the so-called maximum mean discrepancy score (MMDS) is the kernel score corresponding to the Gaussian kernel:

\begin{equation}\label{eq:mmds}
\mathrm{MMDS}(F, \by) = \frac{1}{2} \E_{F} \left[\mathrm{exp}\left\{ - \frac{1}{2} \| \bX - \bX^{\prime} \|^{2} \right\} \right] - \E_{F} \left[ \mathrm{exp}\left\{ - \frac{1}{2} \| \bX - \by \| ^{2} \right\} \right],
\end{equation}

where $\bX, \bX^{\prime} \sim F$ are independent.

To facilitate the implementation of these popular scoring rules in practice, the scoringRules package provides analytical expressions of the LogS and CRPS for forecasts that correspond to several familiar parametric distributions. It is also often the case that only a sample from the forecast distribution is available; this is common, for example, when considering ensemble forecasts issued by numerical weather and climate models, or output from Markov chain Monte Carlo (MCMC) algorithms [Krüger et al., 2021]. The scoringRules package therefore additionally contains versions of the LogS, CRPS, ES, VS, and MMDS that can be used to evaluate forecasts in the form of a predictive sample. This can be achieved by replacing the expectations in Equations <ref>-<ref> with sample means (see Appendix <ref> for details). For the LogS, kernel density estimation is used to estimate the predictive density from the sample, prior to calculating the score.

§.§ Weighted scoring rules

The scoring rules introduced in the previous section evaluate the entire forecast distribution. However, one could argue that it is particularly important to issue accurate forecasts for events that have a high impact on the forecast users, and such events should therefore be given more weight during forecast evaluation. Weighted scoring rules achieve this by incorporating a non-negative weight function $w$ into conventional scoring rules, where the weight function determines how much emphasis should be placed on each possible outcome. Different approaches to weight scoring rules exist, and here we focus only on the two most popular frameworks.

§.§.§ Outcome-weighted scoring rules

[Diks et al., 2011] introduced two weighted versions of the LogS that allow particular outcomes to be emphasised when calculating forecast accuracy. The conditional likelihood score (CoLS) is defined as

\begin{equation*}
\mathrm{CoLS}(F, y) = -w(y)\mathrm{log} f(y) + w(y) \mathrm{log} \left( \int_{\R} w(z) f(z) \dd z \right),
\end{equation*}

while the censored likelihood score (CeLS) is

\begin{equation*}
\mathrm{CeLS}(F, y) = -w(y)\mathrm{log} f(y) - (1 - w(y)) \mathrm{log} \left(1 - \int_{\R} w(z) f(z) \dd z \right).
\end{equation*}

To understand how these weighted logarithmic scores behave, consider a weight function of the form $w(z) = \ind\{z \in \cA\}$, meaning only forecasts for outcomes in the set $ \cA \subseteq \R$ are of interest. In this example, if the observation $y \notin \cA$, then the CoLS is equal to zero, so that only the forecasts issued when $y \in \cA$ contribute to the score. If $y \in \cA$, then the CoLS is equivalent to the LogS applied to the conditional forecast distribution given that the observation is in $\cA$; forecast distributions are therefore assessed only via their restriction to the set $\cA$. The CeLS then extends the CoLS by additionally rewarding forecast distributions that can correctly predict when an outcome of interest will or will not occur. Note that if $\cA = \R$, then the weight function is always one, and both weighted scoring rules revert to the unweighted LogS.

[Holzmann and Klar, 2017] later generalised the CoLS and CeLS by demonstrating that this framework can readily be applied to any proper scoring rule. The resulting scoring rules, which we call outcome-weighted scoring rules, target particular outcomes by introducing a weighted version of the forecast distribution, and evaluating $F$ via its weighted representation. For the weight function $w(z) = \ind\{z \in \cA\}$, this weighted representation is simply the conditional distribution given that the outcome is in $\cA$, as discussed above for the CoLS and CeLS. Further details can be found in [Holzmann and Klar, 2017].

An outcome-weighted CRPS can be defined as

\begin{equation*}
\mathrm{owCRPS}(F, y) = \frac{1}{\bar{w}_{F}} \E_{F} \left[ |X - y|w(X)w(y) \right] - \frac{1}{2 \bar{w}_{F} ^{2}} \E_{F} \left[ |X - X^{\prime}|w(X)w(X^{\prime})w(y) \right],
\end{equation*}

where $X, X^{\prime} \sim F$ are independent and $\bar{w}_{F} = \E_{F} [w(X)]$.
Since this framework applies to any proper scoring rule, outcome-weighted versions of the ES, VS, and MMDS can similarly be introduced to target multivariate outcomes of interest during forecast evaluation:

\begin{equation*}
\begin{split}
\mathrm{owES}(F, \by) =& \frac{1}{\bar{w}_{F}} \E_{F} \left[ \| \bX - \by \| w(\bX)w(\by) \right] - \frac{1}{2 \bar{w}_{F} ^{2}} \E_{F} \left[ \| \bX - \bX^{\prime} \| w(\bX)w(\bX^{\prime})w(\by) \right]; \\
\mathrm{owVS}^{p}(F, \by) =& w(\by) \sum_{i=1}^{d} \sum_{j=1}^{d} h_{i,j} \left( \frac{1}{\bar{w}_{F}} \E_{F} \left[ \left| X_{i} - X_{j} \right| ^{p} w(\bX) \right] - \left| y_{i} - y_{j} \right| ^{p} \right)^{2}; \\
\mathrm{owMMDS}(F, \by) =& \frac{1}{2 \bar{w}_{F}^{2}} \E_{F} \left[ \mathrm{exp}\left\{ - \frac{1}{2} \| \bX - \bX^{\prime} \| ^{2} \right\} w(\bX)w(\bX^{\prime})w(\by) \right] \\
&- \frac{1}{\bar{w}_{F}} \E_{F} \left[ \mathrm{exp}\left\{ - \frac{1}{2} \| \bX - \by \| ^{2} \right\} w(\bX)w(\by) \right].
\end{split}
\end{equation*}

Note that in the multivariate case, the weight function takes a vector as an argument; $\bar{w}_{F}$ is thus defined as $\bar{w}_{F} = \E_{F} [w(\bX)]$.

The premise behind this class of weighted scoring rules is that, if attention is only on a particular set of outcomes, then the forecasts are only evaluated when these outcomes occur. When these outcomes do occur, the forecast distributions are evaluated using the conditional distribution given that the outcome of interest has occurred. In considering the conditional distribution given that an outcome of interest has occurred, the score does not consider the predicted probability that this outcome will occur. The CeLS extends the CoLS to address this, and suitable adaptations of the larger class of outcome-weighted scoring rules also exist, though these are not considered here <cit.>.

Moreover, these scores are clearly not well-defined if the conditional distribution does not exist. This is equivalent to $\bar{w}_{F}$ being equal to zero, which could occur, for example, if $w(z) = \ind\{ z \in \cA \}$ and the forecast distribution assigns zero probability to the region $\cA$. The use of these outcome-weighted scoring rules is therefore only recommended when the weight function is strictly positive, or when interest is on events that are not rare, such that $\bar{w}_{F}$ is non-zero [Allen et al., 2023].

§.§.§ Threshold-weighted scoring rules

Arguably the most well known weighted scoring rule is the threshold-weighted CRPS proposed by [Matheson and Winkler, 1976] and [Gneiting and Ranjan, 2011]. The threshold-weighted CRPS introduces a weight function into the integral defining the CRPS:

\begin{equation*}
\begin{split}
\mathrm{twCRPS}(F, y) &= \int_{\R} (F(z) - \ind\{ y \leq z \})^{2} w(z) \dd z \\
&= \E_{F}|v(X) - v(y)| - \frac{1}{2} \E_{F}|v(X) - v(X^{\prime})|,
\end{split}
\end{equation*}

where $v$ is any function such that $v(z) - v(z^\prime) = \int_{z^{\prime}}^{z} w(z) \dd z$ for all $z, z^{\prime} \in \R$ [Taillardat et al., 2022, Allen et al., 2022]. We follow [Allen et al., 2022] and refer to $v$ as a chaining function.

Just as we can generate outcome-weighted versions of any proper scoring rule, [Allen et al., 2022] demonstrate that the theory underlying the threshold-weighted CRPS can readily be extended to any kernel score. As discussed, the ES, VS, and MMDS are all kernel scores, allowing threshold-weighted versions of these scores to be introduced:

\begin{equation}
\begin{split}
\mathrm{twES}(F, \by) =& \E_{F} \| v(\bX) - v(\by) \| - \frac{1}{2} \E_{F} \| v(\bX) - v(\bX^{\prime}) \|; \\
\mathrm{twVS}^{p}(F, \by) =& \sum_{i=1}^{d} \sum_{j=1}^{d} h_{i,j} \left( \E_{F} \left| v(\bX)_{i} - v(\bX)_{j} \right| ^{p} - \left| v(\by)_{i} - v(\by)_{j} \right| ^{p} \right)^{2}; \\
\mathrm{twMMDS}(F, \by) =& \frac{1}{2} \E_{F} \left[\mathrm{exp}\left\{ - \frac{1}{2} \| v(\bX) - v(\bX^{\prime}) \| ^{2} \right\} \right] - \E_{F} \left[ \mathrm{exp}\left\{ - \frac{1}{2} \| v(\bX) - v(\by) \| ^{2} \right\} \right],
\end{split}
\end{equation}

where $\bX, \bX^{\prime} \sim F$ are independent random variables taking values on $\Omega \subseteq \R^{d}$, and $v: \R^{d} \to \R^{d}$ is a chaining function, so that $v(\by) = \left( v(\by)_{1}, \dots, v(\by)_{d} \right)$ and likewise for $v(\bX)$ and $v(\bX^{\prime})$.

In contrast to the outcome-weighted scoring rules, threshold-weighted scoring rules transform the forecasts and observations according to a chaining function $v$ prior to employing the unweighted version of the scores. The chaining function can therefore be chosen to focus the scoring rules on particular outcomes. While there exists a canonical way to obtain a chaining function from a given weight function in the univariate case, no such relationship exists when evaluating multivariate forecasts. This is discussed further in the following section.

§.§ Weight and chaining functions

These weighted scoring rules provide attractive ways to target particular outcomes of interest when evaluating forecast performance, both in the univariate and multivariate case. In this section, we discuss possible weight and chaining functions that can be used within these weighted scoring rules. Certain choices can result in weighted scoring rules that are not proper, and these weight and chaining functions must therefore be chosen with care, to ensure that forecasters are not evaluated using an improper scoring rule.

If both a weighted and unweighted version of a scoring rule are proper, then they will both be minimised on average by the same forecast distribution: the true distribution of the outcome. However, for two imperfect forecasts, the ranking of these forecasts may change depending on whether a weighted or unweighted scoring rule is employed. Weighted scoring rules may be less powerful than conventional scoring rules when discriminating between two forecast distributions, but they should be more discriminative when comparing forecasts made for particular outcomes. Put differently, if weighted scoring rules detect a difference between two forecast systems, then it is generally easier to interpret this difference than if it were detected using an unweighted scoring rule.

Readers are referred to [Gneiting and Ranjan, 2011], [Lerch et al., 2017], and [Allen et al., 2022] for further details regarding what weight and chaining functions preserve the (strict) propriety of scoring rules. The weight and chaining functions that we consider here all result in weighted scoring rules that are themselves proper (though not necessarily strictly proper).

§.§.§ Weight functions

The choice of weight and chaining function is case-specific, and should depend on what information is to be extracted from the forecasts. Most commonly, interest is on outcomes within a certain range, or above or below a predefined threshold; this range or threshold may correspond to relevant quantiles of the previously observed outcomes. A univariate weight function that restricts attention to these events is

\begin{equation}\label{eq:uv_thr_weight}
w(z) = \ind\{a < z < b\} \quad \text{for some} \quad - \infty \leq a < b \leq \infty,
\end{equation}

which is one if $z$ is between $a$ and $b$, and zero otherwise. To emphasise values above (below) some threshold $t$, we can set $a = t$ and $b = \infty$ ($a = -\infty$ and $b = t$).

Alternatively, certain events could be emphasised using a smoother weight function, which assigns a positive weight to all outcomes, but a higher weight to the events of interest. Popular weight functions to emphasise rare events include a Gaussian or logistic distribution function, e.g.

\begin{equation}\label{eq:gauss_weight}
w(z) = \Phi_{\mu, \sigma} (z),
\end{equation}

where $\Phi_{\mu, \sigma}$ is the Gaussian distribution function with mean $\mu$ and standard deviation $\sigma$, with these parameters controlling the location of the weight function and the rate at which it tends to zero and one [Gneiting and Ranjan, 2011].

Gaussian and logistic density functions could additionally be used to target outcomes that are not rare. For example, the weight function

\begin{equation*}
w(z) = \phi_{\mu, \sigma}(z),
\end{equation*}

where $\phi_{\mu, \sigma}$ is the Gaussian density function with mean $\mu$ and standard deviation $\sigma$. This weight function will emphasise values around the location parameter $\mu$, with $\sigma$ determining the concentration of the weight around $\mu$.

Similar weight functions can also be used in the multivariate case. For example, it is common to define rare multivariate events as threshold exceedances that occur simultaneously in multiple dimensions, in which case a canonical weight function is

\begin{equation}\label{eq:mv_thr_weight}
w(\mathbf{z}) = \ind\{a_{1} < z_{1} < b_{1}, \dots, a_{d} < z_{d} < b_{d} \} \quad \text{for} \quad - \infty \leq a_{i} < b_{i} \leq \infty, \quad i = 1, \dots, d.
\end{equation}

As in the univariate case, some values of the vectors $\mathbf{a} = (a_{1}, \dots, a_{d}) $ and $\mathbf{b} = (b_{1}, \dots, b_{d})$ can be set to $\pm \infty$ in order to focus on threshold exceedances. Multivariate Gaussian distribution and density functions could then again be used to target particular regions of multivariate space in a smoother way [Allen et al., 2023]. These weight functions are listed in Table <ref>.

Weight function     Chaining function

$w(z) = 1$     $v(z) = z$

$w(z) = \ind\{ a < z < b \}$     $v(z) = \min(\max(z, a), b) $

$w(z) = \Phi_{\mu, \sigma} (z)$     $v(z) = (z - \mu) \Phi_{\mu, \sigma} (z) + \sigma^{2} \phi_{\mu, \sigma} (z)$

$w(z) = \phi_{\mu, \sigma} (z)$     $v(z) = \Phi_{\mu, \sigma} (z)$

$w(\boldsymbol{z}) = \ind \left\{ a_{1} < z_{1} < b_{1}, \dots, a_{d} < z_{d} < b_{d} \right\}$     $v(\boldsymbol{z})_{i} = \min(\max(z_{i}, a_{i}), b_{i})$

$w(\boldsymbol{z}) = \boldsymbol{\Phi_{\mu, \Sigma}} (\boldsymbol{z})$     $v(\boldsymbol{z})_{i} = (z_{i} - \mu_{i}) \Phi_{\mu_{i}, \sigma_{i}} (z_{i}) + \sigma_{i} ^{2} \phi_{\mu_{i}, \sigma_{i}} (z_{i})$

$w(\boldsymbol{z}) = \boldsymbol{\phi_{\mu, \Sigma}} (\boldsymbol{z})$     $v(\boldsymbol{z})_{i} = \Phi_{\mu_{i}, \sigma_{i}}(z_{i})$

Examples of weight functions and chaining functions that could be used in weighted scoring rules. $\boldsymbol{\Phi_{\mu, \Sigma}}$ and $\boldsymbol{\phi_{\mu, \Sigma}}$ denote the multivariate Gaussian distribution and density functions with mean vector $\boldsymbol{\mu}=(\mu_{1}, \dots, \mu_{d})$ and covariance matrix $\Sigma$ with diagonal entries $\sigma_{1}, \dots, \sigma_{d}$. The multivariate chaining functions are component-wise extensions of the univariate chaining functions; hence, for concision, only the $i$-th component (for $i \in \{1, \dots, d\}$) of the multivariate chaining function is shown.

§.§.§ Chaining functions

While the outcome-weighted scoring rules depend on a weight function, the threshold-weighted scoring rules depend on a chaining function. It is arguably less intuitive to choose a chaining function to emphasise certain outcomes of interest than a weight function. In the univariate case, the chaining function can be derived easily from a given weight function: we can take any function $v$ that satisfies

\begin{equation}\label{eq:univ_v}
v(z) - v(z^\prime) = \int_{z^{\prime}}^{z} w(z) \dd z \quad \text{for all} \quad z, z^{\prime} \in \R.
\end{equation}

That is, $v$ is an anti-derivative of the chosen weight function. Table <ref> lists examples of chaining functions that correspond to the univariate weight functions given above.

In the multivariate case, however, there is no canonical approach to derive a chaining function from a given weight function. [Allen et al., 2022] discuss possible chaining functions that could be used to target certain multivariate outcomes when interest is on high-impact events. For the multivariate weight function in Equation <ref>, one possible chaining function is

\begin{equation}\label{eq:default_v}
v(z) = \left( \min(\max(z_{1}, a_{1}), b_{1}), \dots, \min(\max(z_{d}, a_{d}), b_{d}) \right),
\end{equation}

which is essentially a component-wise extension of the chaining function for the univariate weight in Equation <ref> (see Table <ref>). In this case, the weight function represents an orthant, or a box, in $\R^{d}$, and the chaining function projects points not in the orthant onto its perimeter; the points inside the orthant, i.e. for which the weight function is equal to one, remain unchanged. A two-dimensional example of this is given in Figure <ref>.

Example of the weight function in Equation <ref> and the chaining function in Equation <ref>. Ten two-dimensional observations are shown before (points) and after (crosses) applying the chaining function. The shaded region is the area specified by the weight function. Points in this region are unchanged after applying the chaining function, whereas points outside of this region are mapped onto the region's border, as indicated by the red arrows.

Similarly, for the smooth weight functions based on multivariate Gaussian distribution and density functions, a chaining function can be derived from a component-wise extension of the chaining functions corresponding to univariate Gaussian weight functions. Examples of such chaining functions are presented in Table <ref>. Note, however, that these component-wise extensions implicitly assume that the covariance matrix in the multivariate Gaussian weight function is diagonal. Readers are referred to [Allen et al., 2022] for a more detailed discussion on multivariate chaining functions.

Although the weight and chaining functions presented in this section are simple examples that are frequently used in practice, the weighted scoring rules discussed herein can be employed with arbitrary such functions, permitting very flexible, user-oriented forecast evaluation. In the remainder of this paper, we will discuss how these weighted scoring rules have been integrated into the scoringRules package, facilitating their use in practical applications.

§ PACKAGE FUNCTIONALITY

§.§ Univariate weighted scoring rules

The weighted scoring rules discussed in the previous section can all be implemented using the scoringRules package. Functionality is currently available for probabilistic forecasts that take the form of a predictive sample. In this case, it is straightforward to calculate the weighted scoring rules with arbitrary, user-specified weight functions, which is generally not the case for parametric families of distributions. Expressions for the weighted scoring rules discussed in the previous section when the forecast is a predictive sample are given in Appendix <ref>.

The scoringRules package already contains functions to calculate the LogS, CRPS, ES, VS, and MMDS for forecasts in the form of predictive samples. Suppose the sample is comprised of $m$ members. As explained in [Jordan et al., 2019], the naming convention of these functions is [score]_sample, where [score] refers to the scoring rule to be calculated. These functions take the observed value(s) and the forecast samples as inputs, and output the desired score value. For example, to calculate the CRPS corresponding to a vector of $n$ observations y and a $n \times m$ matrix dat whose rows contain the $m$ forecast samples corresponding to each observation, one could use

crps_sample(y, dat)

The output is a numeric vector containing the score for each of the $n$ forecast cases.

The same convention is adopted for the weighted scoring rules. In the univariate case, the following functions calculate the outcome-weighted and threshold-weighted CRPS, and the conditional or censored likelihood scores:

owcrps_sample(y, dat, a = -Inf, b = Inf,
weight_func = function(x) as.numeric(x > a     x < b),
w = NULL, show_messages = TRUE)
twcrps_sample(y, dat, a = -Inf, b = Inf,
chain_func = function(x) pmin(pmax(x, a), b),
w = NULL, show_messages = TRUE)
clogs_sample(y, dat, a = -Inf, b = Inf, bw = NULL,
show_messages = FALSE, cens = TRUE)

The cens argument in clogs_sample specifies whether the conditional likelihood score or the censored likelihood score should be returned; the default is cens = TRUE, in which case the CeLS is calculated.

As discussed in Section <ref>, the LogS takes a predictive density as input, and hence cannot readily be applied to predictive samples. To circumvent this, logs_sample employs kernel density estimation to estimate a predictive density from the sample, and then calculates the LogS from the estimated density function. However, [Krüger et al., 2021] demonstrate that the resulting score is sensitive to the bandwidth parameter bw of the kernel density estimation, and the authors therefore recommended using the CRPS instead of the LogS, particularly when the sample size $m$ is small. Similarly, the conditional and censored likelihood scores also require a predictive density as inputs, and kernel density estimation is used to estimate this from the predictive sample prior to calculating the weighted scores. We anticipate that these weighted scores will be yet more sensitive to the kernel density estimation parameters, especially when a weight function is used that targets more extreme outcomes. As such, when the forecast is in the form of a predictive sample, we similarly recommend employing weighted versions of the CRPS, rather than the conditional or censored likelihood score.

In addition to observations and forecast samples, the functions listed above have arguments that allow particular outcomes to be targeted when calculating the weighted scores. By default, the weighted scoring rules employ the weight function $w(z) = \ind\{a < z < b\}$, which, as discussed in the previous section, is most commonly applied in practice. The arguments a and b are single numeric values representing the lower and upper bounds in this weight function, respectively. If these arguments are not specified, then their default values are a = -Inf and b = Inf, resulting in a weight function that is always one, and thus recovering the unweighted scoring rules.

R> obs <- rnorm(5)
R> sample_m <- matrix(rnorm(5e4), nrow = 5)
R> score_df <- data.frame(crps = crps_sample(obs, sample_m),
+                         owcrps = owcrps_sample(obs, sample_m),
+                         twcrps = twcrps_sample(obs, sample_m))
R> print(score_df)

crps owcrps twcrps
1 0.275  0.275  0.275
2 1.230  1.230  1.230
3 0.246  0.246  0.246
4 0.764  0.764  0.764
5 1.355  1.355  1.355

On the other hand, if we want to emphasise outcomes above a threshold t, then we can set the lower bound in the weight function to a = t, and the upper bound to b = Inf.

R> t <- 0
R> score_df <- data.frame(crps = crps_sample(obs, sample_m),
+                         owcrps = owcrps_sample(obs, sample_m, a = t),
+                         twcrps = twcrps_sample(obs, sample_m, a = t))
R> print(score_df)

crps owcrps twcrps
1 0.275  0.000  0.120
2 1.230  0.000  0.115
3 0.246  0.000  0.119
4 0.764  0.306  0.645
5 1.355  0.809  1.235

Similarly, if we want to emphasise values below the threshold, then we can set a = -Inf and b = t. To avoid misuse, an error is returned if a is not smaller than b.

A useful diagnostic tool is to plot the average score as a function of the threshold. In this case, as the lower bound in the weight function a becomes smaller (or the upper bound b becomes larger), the weighted score tends to the unweighted score, allowing the user to simultaneously visualise overall forecast performance, as well as performance when predicting particular outcomes [Gneiting and Ranjan, 2011]. An example of this is presented in Figure <ref>, where the outcome-weighted CRPS and threshold-weighted CRPS for two forecasts distributions are displayed as a function of a in the default weight function, with b = Inf.

Average twCRPS (purple) and owCRPS (orange) as a function of the threshold $a$ in the weight function $w(z) = \ind\{z > a\}$. The observations are drawn from a standard normal distribution, and scores are shown for predictive samples from a standard normal (solid) and standard logistic (dashed) distribution. Note that for high thresholds, the owCRPS is not always well-defined.

§.§ Multivariate weighted scoring rules

Similarly to the LogS and CRPS, scoringRules contains functions to calculate the ES, VS, and MMDS for multivariate forecast distributions in the form of predictive samples.

es_sample(y, dat, w = NULL)
vs_sample(y, dat, w = NULL, w_vs = NULL, p = 0.5)
mmds_sample(y, dat, w = NULL)

These multivariate scoring rule functions can only evaluate a single multivariate forecast at a time. Hence, the observation argument y is a vector of length $d$, representing an element in $\R^{d}$, the forecast argument dat is a $d \times m$ matrix, with the columns representing the simulated samples (or ensemble members) from the multivariate forecast distribution, and the output is a single value. These functions can then be sequentially applied to multiple forecast cases using the apply functions or for loops <cit.>.

Similarly, outcome-weighted and threshold-weighted versions of these multivariate scoring rules are calculated using

owes_sample(y, dat, a = -Inf, b = Inf,
weight_func = function(x) as.numeric(all(x > a     x < b)),
w = NULL)
owvs_sample(y, dat, a = -Inf, b = Inf,
weight_func = function(x) as.numeric(all(x > a     x < b)),
w = NULL, w_vs = NULL, p = 0.5)
owmmds_sample(y, dat, a = -Inf, b = Inf,
weight_func = function(x) as.numeric(all(x > a     x < b)),
w = NULL)
twes_sample(y, dat, a = -Inf, b = Inf,
chain_func = function(x) pmin(pmax(x, a), b), w = NULL)
twvs_sample(y, dat, a = -Inf, b = Inf,
chain_func = function(x) pmin(pmax(x, a), b),
w = NULL, w_vs = NULL, p = 0.5)
twmmds_sample(y, dat, a = -Inf, b = Inf,
chain_func = function(x) pmin(pmax(x, a), b), w = NULL)

As in the univariate case, the default weight function corresponds to Equation <ref>, where interest is on a range of values in each dimension. The default chaining function used to calculate the threshold-weighted scores is Equation <ref>. Arguments a and b are again used to define the lower and upper bounds of the default weight function. In contrast to the univariate case, however, a and b are numeric vectors of length $d$, rather than single values.

If the input value of a or b is a single value, then it is automatically converted into a vector of length $d$, all containing the same element. The default values are a = -Inf and b = Inf, which again returns the unweighted scoring rule. Hence, if we want to emphasise values above the same threshold $t$ in all dimensions, then we could either use a = c(t, t, ...) and b = c(Inf, Inf, ...), or we could use a = t and b = Inf. For example, for the threshold-weighted energy score, we have

R> d <- length(obs)
R> twes_sample(obs, sample_m, a = t)

[1] 1.34

R> twes_sample(obs, sample_m, a = rep(t, d))

[1] 1.34

Finally, note that the functions to calculate the multivariate weighted scores also include optional weight arguments that cannot be used to target particular outcomes of interest. The argument w is a vector of length $m$ that allows more weight to be given to particular elements of the sample in the forecast distribution. This argument is also available when calculating the unweighted scoring rules, and the univariate weighted scores. The variogram score functions additionally have an argument w_vs, which is a $d \times d$ matrix containing the scaling parameters $h_{i,j}$ in Equation <ref>. These scaling parameters put more emphasis on combinations of dimensions of the multivariate variables, rather than targeting particular outcomes.

§.§ Custom weight and chaining functions

The functions to calculate the weighted scoring rules use a default weight function that assumes emphasis is to be placed on a particular region of the outcome space. Although this weight function is frequently applied in practice, it may be the case that another weight function is desired. As discussed, the motivation for considering only forecasts in the form of a predictive sample is that it is straightforward to calculate the resulting scores for arbitrary weight and chaining functions. The weighted scoring rule functions in scoringRules therefore additionally contain an argument that allows for a custom weight or chaining function to be used.

The weight_func argument can be used to incorporate a custom weight function into the outcome-weighted scoring rules. This argument must be a function that takes a vector as an input, and outputs either a vector of the same length as the input (if a univariate scoring rule is being used), or a single numeric value (if a multivariate scoring rule is used). An error is returned if the weight function is found to return negative weights, or if the output is not of the correct format.

For example, consider the Gaussian distribution function in Equation <ref>, with location parameter mu and scale parameter sigma. To use this as the weight function when calculating the outcome-weighted CRPS, one could use

R> mu <- 0; sigma <- 1
R> weight_func <- function(x) pnorm(x, mean = mu, sd = sigma)
R> owcrps_sample(obs, sample_m, weight_func = weight_func)

[1] 0.2002 0.0703 0.1868 0.3524 0.8788

Similarly, a multivariate Gaussian distribution could be used as a multivariate weight function. Let mu be the mean vector of this distribution, and assume the covariance matrix is diagonal with entries $\sigma_{1}, \dots, \sigma_{d}$. Then, the outcome-weighted ES with this weight function can be calculated using

R> mu <- rnorm(d, 0, 0.5); sigma <- runif(d, 0.5, 1.5)
R> weight_func <- function(x) prod(pnorm(x, mean = mu, sd = sigma))
R> owes_sample(obs, sample_m, weight_func = weight_func)

[1] 0.0418

Whereas the outcome-weighted scores depend on a weight function, the threshold-weighted scores rely on a chaining function. For the threshold-weighted CRPS, a chaining function corresponds directly to a weight function via Equation <ref>. However, computation of the threshold-weighted CRPS for a sample forecast requires the chaining function rather than a weight function, and hence functionality is not currently available to take a weight function as an argument. In this case, it is necessary to derive the chaining function corresponding to the weight. For the simple weight functions commonly used in practice, this is typically straightforward to achieve (see Table <ref> for popular choices).

The chain_func argument can be used to incorporate a custom chaining function into the threshold-weighted scoring rules. In contrast to weight_func, the chain_func argument should be a function whose inputs and outputs are the same length as the observation input y. For example, in the multivariate case, this function should both input and output a vector of length $d$.

In the univariate case, if the chaining function satisfies Equation <ref> for some non-negative weight function $w$, then it will be a non-decreasing function; that is, if $z > z^{\prime}$, then $v(z) \geq v(z^{\prime})$ for all $z, z^{\prime} \in \R$. While a decreasing chaining function could also be used within Equation <ref>, this does not correspond to the original definition of the twCRPS presented in [Gneiting and Ranjan, 2011], and is therefore not recommended: a warning message is returned if chain_func is found to be decreasing.

Table <ref> contains possible chaining functions corresponding to the Gaussian weight functions employed above. These chaining functions can be implemented within twcrps_sample and twes_sample as follows

R> chain_func <- function(x) (x - mu)*pnorm(x, mu, sigma) +
+    (sigma^2)*dnorm(x, mu, sigma)
R> mu <- 0; sigma <- 1
R> twcrps_sample(obs, sample_m, chain_func = chain_func)

[1] 0.135 0.263 0.123 0.528 1.082

R> mu <- rnorm(d, 0, 0.5); sigma <- runif(d, 0.5, 1.5)
R> twes_sample(obs, sample_m, chain_func = chain_func)

[1] 1.48

Weighted versions of the other scoring rules discussed herein can be calculated analogously.

It is challenging to construct general analytical formulae for weighted scoring rules corresponding to parametric forecast distributions and arbitrary weight functions. Similarly, since the CoLS and CeLS require kernel density estimation to estimate the predictive density given the sample, these scores cannot be readily implemented with arbitrary weight functions. Hence, clogs_sample does not take custom weight or chaining functions as arguments, so that only the default weight function is available for these scores.

§ USAGE EXAMPLES

[Jordan et al., 2019] present two practical applications in which the scoringRules functionality is used to evaluate probabilistic forecasts. In this section, we revisit these applications, and illustrate how the weighted scoring rules available in scoringRules allow particular outcomes to be targeted during forecast evaluation. In both examples, the data and probabilistic models are as described in [Jordan et al., 2019], and further details can be found therein.

§.§ Probabilistic weather forecasting via ensemble post-processing

Firstly, consider forecasts of precipitation accumulation in Innsbruck, Austria. The RainIbk data set in the crch R package [Messner et al., 2016] contains three-day precipitation accumulations recorded in Innsbruck from January 2000 to September 2013. Forecasts for these precipitation accumulations can be obtained from numerical weather prediction (NWP) models, which use physical laws to emulate the evolution of the atmosphere through time. These models are typically run several times, using different initial conditions and possibly different model configurations, yielding an ensemble of predictions that characterises the forecast uncertainty. The RainIbk data set contains 11-member ensemble forecasts corresponding to the three-day precipitation accumulations between five and eight days in advance.

However, operational ensemble forecasts tend to exhibit systematic errors when predicting surface weather variables such as precipitation. Hence, it is common for the ensemble forecasts to undergo some form of statistical post-processing. Post-processing methods try to learn the systematic errors that manifest in the NWP models, and then remove them from the forecasts. While a number of statistical post-processing methods have been proposed, the methods considered here assume that the square root of the precipitation accumulation follows a parametric distribution. The location and scale parameters of this distribution are assumed to depend linearly on the mean and the log-transformed standard deviation of the ensemble members, respectively. This general framework for post-processing ensemble weather forecasts is typically known as non-homogeneous regression or ensemble model output statistics <cit.>.

Three alternative parametric distributions are then compared within this framework: a logistic, Gaussian, and Student's $t$ distribution. These three parametric distributions are censored below at zero, resulting in a forecast distribution that assigns zero probability to negative precipitation accumulations, and a non-negligible probability to zero precipitation. Further details about the statistical post-processing methods are available in [Jordan et al., 2019] and references therein.

Data from January 2000 to November 2004 is used to train the statistical post-processing models, and the resulting forecasts are then evaluated out-of-sample using the data from January 2005 to September 2013. The models are fit to the training data using maximum likelihood estimation via the crch package. Applying these models to the ensemble forecasts in the test data set returns predictive location and scale parameters for each model and each forecast case. For concision, we only show the code used to evaluate the Gaussian forecast distributions; this can easily be extended to the other models. In this case, the vectors gauss_mu and gauss_sc contain the estimated location and scale parameters that characterise the predictive distributions in the test data, while obs represents the time series of the corresponding observed precipitation accumulations.

These three post-processing models can then be evaluated and compared using scoring rules. [Jordan et al., 2019] demonstrate how the CRPS can be used for this purpose.

R> gauss_crps <- crps_cnorm(y = obs, location = gauss_mu, scale = gauss_sc,
+                           lower = 0, upper = Inf)

The result is a vector of scores corresponding to each forecast case in the test data set, and the competing forecast strategies can then be compared using their average scores.

R> scores <- data.frame(Logistic = logis_crps, Gaussian = gauss_crps,
+                       Students_t = stud_crps, Ensemble = ens_crps)
R> sapply(scores, mean)

Logistic   Gaussian Students_t   Ensemble
0.875      0.876      0.875      1.321

The mean CRPS values indicate that all post-processing models substantially improve upon the raw ensemble forecasts, and there are only small differences between the post-processing models. Of course, in a formal study, we should accompany these scores with measures of uncertainty, or perform statistical tests that clarify whether the differences between the scores are significant.

While the CRPS assesses overall forecast performance, the three post-processing models could also be compared with respect to their predictions of particular outcomes. This can be achieved here using weighted versions of the CRPS. As discussed, weighted scoring rules are only available in scoringRules for forecasts in the form of a predictive sample. Hence, to evaluate the post-processed forecast distributions, we must first sample from the predictive distributions, and use this sample to estimate the score for the parametric forecasts. Here, we sample 1000 observations from the predictive distributions, which should provide a reasonable approximation of the score for the continuous forecast distribution <cit.>.

R> ens_size <- 1000
R> n <- length(obs)
R> gauss_sample <- replicate(ens_size, rnorm(n, gauss_mu, gauss_sc))
R> gauss_sample[gauss_sample < 0] <- 0

An obvious question concerns which weight function to employ within the weighted scores. Weight and chaining functions should be chosen such that the outcomes that are of most interest to the practitioners are emphasised. As discussed, this will likely change on a case-by-case basis, and more than one weight function could be employed to gain a more complete understanding of the forecast performance; summarising plots such as Figure <ref> are particularly useful.

In a weather forecasting context, the relationship between weather conditions and socio-economical impacts is relatively well understood. Hence, an obvious choice of the weight function is one that reflects the costs associated with each possible outcome. Outcomes that correspond to higher impacts would therefore be emphasised in the weighted scores, and forecasters would be encouraged to issue more accurate forecasts for these high-impact events. Alternatively, to reduce the impact of these events, national weather centres issue warnings to the general public. These warnings typically correspond to relevant thresholds of the outcome variable, determined from climatological records and thorough analyses of the risks of weather to infrastructure and public health. A simple alternative weight function would thus use an indicator weight function (e.g. Equation <ref>), with the parameters in this weight function defined by the warning thresholds.

For the rainfall forecasts considered here, we firstly employ this latter weight function to emphasise values above a threshold of interest $t$, i.e. $w(z) = \ind\{ z > t \}$. This weight function can be employed in scoringRules by setting the arguments a = t and b = Inf in the weighted scoring rule functions. In doing so, the weighted scoring rules will assess the forecasts in their ability to predict high precipitation accumulations, which are of particular relevance since they often lead to flooding. As a threshold, we choose $t=\sqrt{30}$mm. This choice was made since $30$mm is commonly chosen as a threshold in rainfall warning systems, and $\sqrt{30}$ is roughly equal to the 95th percentile of the square root-transformed precipitation values in the training data considered here.

R> t <- sqrt(30)
R> gauss_twcrps <- twcrps_sample(y = obs, dat = gauss_sample, a = t)

Having repeated this for the logistic and Student's $t$ forecast distributions, we can then compare the three post-processing models using their average threshold-weighted CRPS.

R> scores <- data.frame(Logistic = logis_twcrps, Gaussian = gauss_twcrps,
+                       Students_t = stud_twcrps, Ensemble = ens_twcrps)
R> sapply(scores, mean)

Logistic   Gaussian Students_t   Ensemble
0.0491     0.0490     0.0489     0.0774

These threshold-weighted CRPS values illustrate that, while the three post-processing methods are again almost indistinguishable, the relative improvement of the post-processed forecasts upon the raw ensemble forecasts has increased. This suggests that post-processing is particularly beneficial when predicting more extreme precipitation accumulations.

We could also calculate the outcome-weighted CRPS for these forecasts in a similar way. However, when interest is on extreme events, there is a chance that neither the observation nor any members of the predictive sample exceed the threshold of interest, resulting in an undefined outcome-weighted score.

Alternatively, the Gaussian distribution function could also be used to emphasise larger precipitation values without restricting attention only to values above a threshold. The threshold-weighted CRPS values corresponding to this weight function are given below.
The results largely agree with those observed for the previous weight function.

R> sigma <- 1
R> weight_func <- function(x) pnorm(x, mean = t, sd = sigma)
R> chain_func <- function(x) (x - t)*pnorm(x, mean = t, sd = sigma) +
+    (sigma^2)*dnorm(x, mean = t, sd = sigma)
R> gauss_twcrps <- twcrps_sample(obs, gauss_sample, chain_func = chain_func)

R> scores <- data.frame(Logistic = logis_twcrps, Gaussian = gauss_twcrps,
+                       Students_t = stud_twcrps, Ensemble = ens_twcrps)
R> sapply(scores, mean)

Logistic   Gaussian Students_t   Ensemble
0.0676     0.0676     0.0674     0.1079

The choice of sigma above is somewhat arbitrary, and depends on how quickly the weight function should tend to zero or one. If sigma is equal to zero, then, in theory, we should recover the indicator weight function employed above.

In general, the performance of the forecasts will change depending on the weight function. The weight functions discussed herein are only examples, and the choice of weight function will depend on the application. Other weight functions could also readily be employed.

§.§ Bayesian forecasts of US GDP growth

The second case study considers an example from economics. It is standard for national banks to issue forecasts for the country's gross domestic product (GDP) growth in the coming quarters. In this example, probabilistic forecasts of US GDP growth (in %) are obtained using a Markov switching autoregressive model, with exact details given in [Krüger et al., 2021]. This model is used to derive forecasts of quarterly US GDP growth for the following year, i.e. the next four quarters.

The data used in this example is available from the data set gdp in scoringRules, which contains observed US GDP growth for 271 quarters between 1947 and 2014. We use the data prior to 2014 as training data to estimate the Markov switching autoregressive model, which is then used to forecast the GDP growth in the four quarters of 2014.

The model implemented here is Bayesian, and is estimated using Markov chain Monte Carlo (MCMC) methods. As is common for Bayesian models that employ MCMC, the analytical form of the predictive distribution is not known. The forecast distributions considered here are therefore predictive samples obtained from the MCMC algorithm. The resulting forecast distributions are displayed in Figure 4 of [Jordan et al., 2019].

These forecast distributions for each quarter can be evaluated univariately using both the CRPS and the LogS. In the following, obs denotes a vector containing the four observed GDP growths in 2014, while X is a matrix containing the MCMC predictions.

R> scores_crps <- crps_sample(obs, X)
R> scores_logs <- logs_sample(obs, X)
R> print(cbind(scores_crps, scores_logs))

scores_crps scores_logs
2014Q1       3.454        4.05
2014Q2       1.322        2.25
2014Q3       1.711        2.51
2014Q4       0.719        1.96

As in the previous example, weighted versions of these scoring rules can again be used to target particular outcomes when quantifying forecast performance. The choice of weight function will again depend on what information is most relevant for practitioners. When forecasting GDP growth, it is particularly important to accurately predict when growth rates will be negative, since this suggests a decline in the country's economy. To emphasise negative growth rates within weighted scoring rules, a canonical weight function is $w(z) = \ind\{z < 0 \}$. Outcome-weighted and threshold-weighted CRPS values, as well as conditional and censored likelihood scores, are shown below for the Markov switching autoregressive forecasts considered here.

R> t <- 0
R> scores_owcrps <- owcrps_sample(obs, X, b = t)
R> scores_twcrps <- twcrps_sample(obs, X, b = t)
R> scores_cols <- clogs_sample(obs, X, b = t, cens = FALSE)
R> scores_cels <- clogs_sample(obs, X, b = t)
R> print(cbind(scores_owcrps, scores_twcrps, scores_cols, scores_cels))

scores_owcrps scores_twcrps scores_cols scores_cels
2014Q1         0.653        1.8714        1.97       4.049
2014Q2         0.000        0.0303        0.00       0.210
2014Q3         0.000        0.0501        0.00       0.258
2014Q4         0.000        0.0586        0.00       0.278

The outcome-weighted CRPS and the conditional likelihood scores are zero for the last three quarters, since the observed GDP growths are greater than zero in these cases. The threshold-weighted CRPS and censored likelihood score, on the other hand, additionally assess the forecast probability that is assigned to a positive GDP growth occurring.

One could argue that economists also receive considerable attention when exceptionally high growth is forecast. To address this, a weight function could be used that simultaneously emphasises low and high GDP growth: e.g. $w(z) = \ind\{ z < 0 \} + \ind\{ z > t \}$ for some reasonably high threshold $t > 0$. Although this does not align with the default weight function used within scoringRules, the weight_func and chain_func arguments allow this custom weight function to be employed within the weighted versions of the CRPS. The resulting scores are presented below. In this case, $t$ is chosen to be 9%, which again roughly corresponds to the 95th percentile of the previously observed GDP growths.

R> a <- 0
R> b <- 9
R> weight_func <- function(x) as.numeric((x < a) | (x > b))
R> chain_func <- function(x) (x < a)*(x - a) + (x > b)*(x - b) + a

R> scores_owcrps <- owcrps_sample(obs, X, weight_func = weight_func)
R> scores_twcrps <- twcrps_sample(obs, X, chain_func = chain_func)
R> print(cbind(scores_owcrps, scores_twcrps))

scores_owcrps scores_twcrps
2014Q1         0.739        1.8715
2014Q2         0.000        0.0306
2014Q3         0.000        0.0504
2014Q4         0.000        0.0592

Yet more relevant than forecasting negative GDP growth in an individual quarter is predicting a decline in GDP growth in successive quarters; this is commonly used by analysts as an indicator for a recession [U.S. Bureau of Economic Analysis (BEA), 2018]. Hence, utilising this definition, evaluating forecasts for recessions becomes a multivariate problem.

The weighted multivariate scoring rules discussed herein allow us to emphasise successive quarters with negative GDP growth when evaluating forecast accuracy. If we consider two consecutive declines in GDP growth, then a suitable weight function to employ in the weighted multivariate scores is $w(\mathbf{z}) = \ind\{ z_{1} < 0, z_{2} < 0 \}$. This weight function can readily be extended to consider further quarters of negative GDP growth.

The threshold-weighted energy score, variogram score, and maximum mean discrepancy score are all shown below for the Markov switching autoregressive forecasts when predicting negative GDP growth in the two following quarters.

R> d <- 2
R> scores_twes <- twes_sample(obs[1:d], X[1:d, ], b = 0)
R> scores_twvs <- twvs_sample(obs[1:d], X[1:d, ], b = 0)
R> scores_twmmds <- twmmds_sample(obs[1:d], X[1:d, ], b = 0)
R> print(cbind(scores_twes, scores_twvs, scores_twmmds))

scores_twes scores_twvs scores_twmmds
[1,]        1.77        2.62         0.242

Note that these weighted scoring rules consider not only the probability that a recession will occur, but also the severity of the recession.

§ SUMMARY AND DISCUSSION

Scoring rules are well-established when evaluating and comparing probabilistic forecasts, and the scoringRules package in R has become well-established when implementing popular scoring rules in practice. In this paper, we discuss how the functionality of the scoringRules package has been extended such that particular outcomes can be emphasised when using scoring rules to assess forecast performance.

Two approaches to target particular outcomes are available, which can be applied to probabilistic forecasts for both univariate and multivariate outcomes. These approaches are available for popular scoring rules including the LogS, CRPS, ES, VS, and MMDS, facilitating very flexible, user-oriented evaluation of probabilistic forecasts in a wide range of practical applications. In particular, functionality is available to calculate the conditional and censored likelihood scores proposed by [Diks et al., 2011]; outcome-weighted versions of the CRPS, ES, VS, and MMDS, which can be constructed from the general framework outlined by [Holzmann and Klar, 2017]; and threshold-weighted versions of the CRPS, ES, VS, and MMDS [Matheson and Winkler, 1976, Gneiting and Ranjan, 2011, Allen et al., 2022].

While the scoringRules package contains analytical expressions for the LogS and CRPS for several parametric distributions, the weighted scoring rules discussed herein are only available for forecast distributions in the form of a simulated sample, or an ensemble. In this case, the weighted scoring rules can readily be calculated for arbitrary weight functions, which is generally not the case for forecasts in the form of parametric distributions. While this permits very flexible forecast evaluation, the scoringRules package could be extended further by incorporating weighted scoring rules for certain families of parametric distributions.

§ ACKNOWLEDGEMENTS

This work was funded by the Swiss Federal Office for Meteorology and Climatology (MeteoSwiss) and the Oeschger Centre for Climate Change Research. I am very grateful to Fabian Krüger, Sebastian Lerch, and Alexander Jordan for their contribution to this work. Jonas Bhend and José Carlos Araujo Acuña are also thanked for their many fruitful suggestions.

[Allen et al., 2023]
Allen S, Bhend J, Martius O, Ziegel J (2023).
Weighted Verification Tools to Evaluate Univariate and
Multivariate Probabilistic Forecasts for High-impact Weather Events.
Weather and Forecasting, 38(3), 499–516.

[Allen et al., 2022]
Allen S, Ginsbourger D, Ziegel J (2022).
Evaluating forecasts for high-impact events using
transformed kernel scores.
arXiv preprint arXiv:2202.12732.

[Diks et al., 2011]
Diks C, Panchenko V, Van Dijk D (2011).
Likelihood-based scoring rules for comparing density
forecasts in tails.
Journal of Econometrics, 163(2), 215–230.

[Gneiting and Raftery, 2007]
Gneiting T, Raftery AE (2007).
Strictly Proper Scoring Rules, Prediction, and Estimation.
Journal of the American Statistical Association,
102(477), 359–378.

[Gneiting et al., 2005]
Gneiting T, Raftery AE, Westveld III AH, Goldman T (2005).
Calibrated Probabilistic Forecasting Using Ensemble Model
Output Statistics and Minimum CRPS Estimation.
Monthly Weather Review, 133(5), 1098–1118.

[Gneiting and Ranjan, 2011]
Gneiting T, Ranjan R (2011).
Comparing density forecasts using threshold-and
quantile-weighted scoring rules.
Journal of Business & Economic Statistics, 29(3),

[Good, 1952]
Good IJ (1952).
Rational Decisions.
Journal of the Royal Statistical Society B, 14(1),

[Holzmann and Klar, 2017]
Holzmann H, Klar B (2017).
Focusing on regions of interest in forecast evaluation.
The Annals of Applied Statistics, 11(4), 2404–2431.

[Jordan et al., 2019]
Jordan A, Krüger F, Lerch S (2019).
Evaluating Probabilistic Forecasts with scoringRules.
Journal of Statistical Software, 90(12), 1–37.

[Krüger et al., 2021]
Krüger F, Lerch S, Thorarinsdottir TL, Gneiting T (2021).
Predictive Inference Based on Markov Chain Monte Carlo
International Statistical Review, 89(2), 274–301.

[Lerch et al., 2017]
Lerch S, Thorarinsdottir TL, Ravazzolo F, Gneiting T (2017).
Forecaster's dilemma: Extreme events and forecast
Statistical Science, 32(1), 106–127.

[Matheson and Winkler, 1976]
Matheson JE, Winkler RL (1976).
Scoring rules for continuous probability distributions.
Management Science, 22(10), 1087–1096.

[Messner et al., 2016]
Messner JW, Mayr GJ, Zeileis A (2016).
Heteroscedastic Censored and Truncated Regression with
The R Journal, 8(1), 173–181.

[Scheuerer and Hamill, 2015]
Scheuerer M, Hamill TM (2015).
Variogram-based proper scoring rules for probabilistic
forecasts of multivariate quantities.
Monthly Weather Review, 143(4), 1321–1334.

[Sharpe et al., 2018]
Sharpe MA, Bysouth CE, Stretton RL (2018).
How well do Met Office post-processed site-specific
probabilistic forecasts predict relative-extreme events?
Meteorological Applications, 25(1), 23–32.

[Taillardat et al., 2022]
Taillardat M, Fougères AL, Naveau P, de Fondeville R (2022).
Evaluating probabilistic forecasts of extremes using
continuous ranked probability score distributions.
International Journal of Forecasting.

[U.S. Bureau of Economic Analysis (BEA), 2018]
US Bureau of Economic Analysis (BEA) (2018).
<https://www.bea.gov/help/glossary/recession>. Accessed:

§ SCORES FOR SIMULATED PREDICTIVE DISTRIBUTIONS

Consider $\Omega \subseteq \R$, and suppose the forecast distribution is only available via a simulated sample $x_{1}, \dots, x_{m} \in \Omega$. To evaluate the empirical distribution function defined by this sample,

\begin{equation*}
\hat{F}_{m}(z) = \frac{1}{m} \sum_{i=1}^{m} \ind \{x_{i} \leq z\},
\end{equation*}

the CRPS simplifies to

\begin{equation*}
\mathrm{CRPS}(\hat{F}_{m}, y) = \frac{1}{m} \sum_{i=1}^{m} |x_{i} - y| - \frac{1}{2 m^{2}} \sum_{i=1}^{m} \sum_{j=1}^{m} |x_{i} - x_{j}|.
\end{equation*}

The outcome-weighted CRPS and threshold-weighted CRPS are defined analogously: given a univariate weight function $w$, the outcome-weighted CRPS can be written as

\begin{equation*}
\mathrm{owCRPS}(\hat{F}_{m}, y) = \frac{1}{m \bar{w}} \sum_{i=1}^{m} |x_{i} - y| w(x_{i}) w(y) - \frac{1}{2 m^{2} \bar{w}^{2}} \sum_{i=1}^{m} \sum_{j=1}^{m} |x_{i} - x_{j}| w(x_{i}) w(x_{j}) w(y),
\end{equation*}

where $\bar{w} = \sum_{i=1}^{m} w(x_{i})/m$. Letting $v$ denote the corresponding chaining function, the threshold-weighted CRPS is

\begin{equation*}
\mathrm{twCRPS}(\hat{F}_{m}, y) = \frac{1}{m} \sum_{i=1}^{m} |v(x_{i}) - v(y)| - \frac{1}{2 m^{2}} \sum_{i=1}^{m} \sum_{j=1}^{m} |v(x_{i}) - v(x_{j})|.
\end{equation*}

Similarly, let $F$ be a forecast distribution on $\Omega \subseteq \R^{d}$ for $d > 1$, and suppose that only a sample $\bx_{1}, \dots, \bx_{m}$ from $F$ is available, with $\bx_{i} = \left( x_{i,1}, \dots, x_{i, d} \right) \in \Omega$ for $i = 1, \dots, m$. In this case, the energy score for the corresponding empirical multivariate distribution $\hat{F}_{m}$ can be written as

\begin{equation*}
\mathrm{ES}(\hat{F}_{m}, \by) = \frac{1}{m} \sum_{i=1}^{m} \| \bx_{i} - \by \| - \frac{1}{2 m^{2}} \sum_{i=1}^{m} \sum_{j=1}^{m} \| \bx_{i} - \bx_{j} \|,
\end{equation*}

the variogram score of order $p$ becomes

\begin{equation*}
\mathrm{VS}^{p}(\hat{F}_{m}, \by) = \sum_{i=1}^{d} \sum_{j=1}^{d} h_{i,j} \left( \frac{1}{m} \sum_{k=1}^{m} \left|x_{k, i} - x_{k, j} \right|^{p} - \left|y_{i} - y_{j} \right|^{p} \right)^2,
\end{equation*}

and the maximum mean discrepancy score is

\begin{equation*}
\mathrm{MMDS}(\hat{F}_{m}, \by) = \frac{1}{2 m^{2}} \sum_{i=1}^{m} \sum_{j=1}^{m} \mathrm{exp} \left\{ -\frac{1}{2} \| \bx_{i} - \bx_{j} \|^{2} \right\} - \frac{1}{m} \sum_{i=1}^{m} \mathrm{exp} \left\{ -\frac{1}{2} \| \bx_{i} - \by \|^{2} \right\}.
\end{equation*}

Given a multivariate weight function $w$ and chaining function $v$, the outcome-weighted and threshold-weighted versions of these scores can be calculated as follows. In this case, $\bar{w} = \sum_{i=1}^{m} w(\bx_{i})/m$, and $v(\bx_{i}) = \left( v(\bx_{i})_{1}, \dots, v(\bx_{i})_{d} \right) \in \Omega$ for $i = 1, \dots, m$.

\begin{equation*}
\mathrm{owES}(\hat{F}_{m}, \by) = \frac{1}{m \bar{w}} \sum_{i=1}^{m} \| \bx_{i} - \by \| w(\bx_{i}) w(\by) - \frac{1}{2 m^{2} \bar{w}^{2}} \sum_{i=1}^{m} \sum_{j=1}^{m} \| \bx_{i} - \bx_{j} \| w(\bx_{i}) w(\bx_{j}) w(\by);
\end{equation*}

\begin{equation*}
\mathrm{owVS}^{p}(\hat{F}_{m}, \by) = w(\by) \sum_{i=1}^{d} \sum_{j=1}^{d} h_{i,j} \left( \frac{1}{m \bar{w}} \sum_{k=1}^{m} \left|x_{k,i} - x_{k,j} \right|^{p} w(\bx_{k}) - \left|y_{i} - y_{j} \right|^{p} \right)^2;
\end{equation*}

\begin{equation*}
\begin{split}
\mathrm{owMMDS}(\hat{F}_{m}, \by) =& \frac{1}{2 m^{2} \bar{w}^{2}} \sum_{i=1}^{m} \sum_{j=1}^{m} \mathrm{exp} \left\{ -\frac{1}{2} \| \bx_{i} - \bx_{j} \|^{2} w(\bx_{i}) w(\bx_{j}) w(\by) \right\} \\
&- \frac{1}{m \bar{w}} \sum_{i=1}^{m} \mathrm{exp} \left\{ -\frac{1}{2} \| \bx_{i} - \by \|^{2} w(\bx_{i}) w(\by) \right\};
\end{split}
\end{equation*}

\begin{equation*}
\mathrm{twES}(\hat{F}_{m}, \by) = \frac{1}{m} \sum_{i=1}^{m} \| v(\bx_{i}) - v(\by) \| - \frac{1}{2 m^{2}} \sum_{i=1}^{m} \sum_{j=1}^{m} \| v(\bx_{i}) - v(\bx_{j}) \|;
\end{equation*}

\begin{equation*}
\mathrm{twVS}^{p}(\hat{F}_{m}, \by) = \sum_{i=1}^{d} \sum_{j=1}^{d} h_{i,j} \left( \frac{1}{m} \sum_{k=1}^{m} \left|v(\bx_{k})_{i} - v(\bx_{k})_{j} \right|^{p} - \left|v(\by)_{i} - v(\by)_{j} \right|^{p} \right)^2;
\end{equation*}

\begin{equation*}
\mathrm{twMMDS}(\hat{F}_{m}, \by) = \frac{1}{2 m^{2}} \sum_{i=1}^{m} \sum_{j=1}^{m} \mathrm{exp} \left\{ -\frac{1}{2} \| v(\bx_{i}) - v(\bx_{j}) \|^{2} \right\} - \frac{1}{m} \sum_{i=1}^{m} \mathrm{exp} \left\{ -\frac{1}{2} \| v(\bx_{i}) - v(\by) \|^{2} \right\}.
\end{equation*}