# Evaluating infectious disease forecasts
with allocation scoring rules

Aaron Gerding Corresponding Author<EMAIL_ADDRESS>Department of
Biostatistics and Epidemiology, School of Public Health and Health Sciences,
University of Massachusetts at Amherst  Nicholas G. Reich  Department of
Biostatistics and Epidemiology, School of Public Health and Health Sciences,
University of Massachusetts at Amherst  Benjamin Rogers  Department of
Biostatistics and Epidemiology, School of Public Health and Health Sciences,
University of Massachusetts at Amherst  Evan L. Ray  Department of
Biostatistics and Epidemiology, School of Public Health and Health Sciences,
University of Massachusetts at Amherst

###### Abstract

Recent years have seen increasing efforts to forecast infectious disease
burdens, with a primary goal being to help public health workers make informed
policy decisions. However, there has only been limited discussion of how
predominant forecast evaluation metrics might indicate the success of policies
based in part on those forecasts. We explore one possible tether between
forecasts and policy: the allocation of limited medical resources so as to
minimize unmet need. We use probabilistic forecasts of disease burden in each
of several regions to determine optimal resource allocations, and then we
score forecasts according to how much unmet need their associated allocations
would have allowed. We illustrate with forecasts of COVID-19 hospitalizations
in the US, and we find that the forecast skill ranking given by this
allocation scoring rule can vary substantially from the ranking given by the
weighted interval score. We see this as evidence that the allocation scoring
rule detects forecast value that is missed by traditional accuracy measures
and that the general strategy of designing scoring rules that are directly
linked to policy performance is a promising direction for epidemic forecast
evaluation.

##  1 Introduction

Infectious disease forecasting models have emerged as important tools in
public health. Predictions of disease dynamics have been used to inform
decision making about a wide variety of measures to reduce disease spread
and/or mitigate the severity of disease outcomes. For example, estimates of
expected onset of flu season have been used to aid national vaccination
strategies (Igboh et al., 2023), and forecasts of Ebola dynamics have been
used to allocate surveillance resources (Meltzer et al., 2014; Rainisch et
al., 2015). Bertsimas et al. (2021) developed tools to guide decision making
from infectious disease forecasts, which have been used to inform allocation
of limited medical supplies such as ventilators, ICU capacity planning, and
vaccine distribution strategy. Models developed by Fox et al. (2022) have been
used to inform resource and care site planning, as well as community
guidelines for masking, traveling, dining and shopping (University of Texas,
2022).

In decision making settings where it is possible to quantify the utility or
loss associated with a particular action, standard tools of decision theory
provide a procedure for developing forecast scoring rules that measure the
value of forecasts through the quality of the decisions that they lead to. We
give an overview of these procedures in Section 2.2.1. There is a large
history of literature applying these ideas to obtain measures of the value of
forecasts that are tied to a decision making context, primarily in fields such
as economics and finance, supply chain management, and meteorology. We review
this work only briefly here, and we refer the reader to Yardley and
Petropoulos (2021) for a general overview, and to Pesaran and Skouras (2002)
and Murphy (1993) for discussions focused on applications to economics and
meteorology, respectively. In finance, the value of forecasts can often be
measured by the profits generated by trading decisions informed by the
forecasts, perhaps adjusted for risk levels (e.g., Leitch and Tanner, 1991;
Cenesizoglu and Timmermann, 2012). In applications to supply chain management
and meteorology, the value of forecasts has typically been operationalized by
considering the costs associated with decisions regarding the amount of
inventory to hold or the level of protection against the impacts of extreme
weather events to enact (e.g., Catt et al., 2007; Petropoulos et al., 2019;
Palmer, 2002; Pappenberger et al., 2015). For example, in supply chain
management these decisions may incur costs related to holding inventory,
labor, or providing poor service, while in meteorology we may need to balance
the costs of implementing protective measures with the costs of potentially
preventable weather damages. In this framework, a forecast has value if it
leads to decisions with low total costs. In all of these fields, analyses have
consistently found that common measures of statistical forecast accuracy do
not necessarily correspond directly to measures of the value of forecasts as
an input to decision making (e.g., Leitch and Tanner, 1991; Murphy, 1993;
Cenesizoglu and Timmermann, 2012).

However, we are aware of only a limited body of work that explicitly attempts
to measure the value of infectious disease forecasts through their impact on
policy, and much of this discussion has proceeded informally. For example,
Ioannidis et al. (2022) discuss the possible negative consequences of
inaccurate forecasts of infectious disease, but do not attempt to quantify the
utility or loss incurred as a result of those forecasts. Bilinski et al.
(2023) explore ways in which policymaker preferences could inform risk
thresholds for predictive models using a framework for measuring the costs and
losses associated with taking an action that is similar to methods that have
been used in meteorology and elsewhere. Marshall et al. (2023) develop a
forecast scoring rule that is informally motivated by utility considerations,
but the score is not derived from a decision-theoretic set up. Separately,
there is a thread of literature that quantifies the link between infectious
disease modeling and policy making outside of a forecasting context. As an
example, Probert et al. (2016) develop measures of the cost of actions
designed to control a hypothetical outbreak of foot-and-mouth disease and use
this framework to explore policy recommendations from a variety of simulation-
based projection models.

In practice, probabilistic infectious disease forecasts have most often been
made for observations that emerge from public health surveillance systems and
have typically been evaluated with standard, “off-the-shelf” scoring rules.
For example, seasonal influenza forecasts in the US and dengue forecasts for
Peru and Puerto Rico targeted public health surveillance measures of incidence
over time and space, and used log-score and mean absolute errors to evaluate
forecast skill (McGowan et al., 2019; Reich et al., 2019; Johansson et al.,
2019). Pandemic COVID-19 forecasts of observed cases, hospitalizations and/or
deaths in the US and Europe, as reported by municipal, state, or federal
surveillance systems, were evaluated using the weighted interval score (WIS,
which is an approximation of the continuous ranked probability score, or
CRPS), and prediction interval coverage (Cramer et al., 2022a; Fox et al.,
2022; Sherratt et al., 2023). Similarly, CRPS was also used to assess
probabilistic forecasts of dengue incidence at the district level in Vietnam
(Colón-González et al., 2021). While some of these scores can be interpreted
through the lens of decision theory, and all of the application-specific
papers cited above had authors from public health agencies, none of them make
explicit connections between forecast evaluation and how a forecast was used
in practice.

In this work, we begin to fill this gap between the ways that infectious
disease forecasts have traditionally been evaluated and the ways that they
have been used to support public health policy. We consider a setting in which
forecasts are used to help determine the allocation of a limited quantity of
medical supplies across multiple regions. In section 2 of the paper, we define
a new forecast scoring rule — the allocation score — that evaluates forecasts
based on how beneficial resource allocations derived from them would turn out
to be.

In section 3, we present an illustrative analysis using the allocation score
to evaluate forecasts of hospital admissions in the US leading up to the
Omicron wave in winter 2022. This analysis is “synthetic” in that it does not
correspond to an actual analysis that supported decision making in real time.
However, the framework described in this paper corresponds to real-world
decisions that must be made by public health administrators around the globe,
and could be adapted in the future for such real-time situations. For example,
forecasts for districts in Sierra Leone of bed demand to care for patients
with Ebola was the subject of a real-time modeling study in late 2014 and
early 2015 (Camacho et al., 2015). And, in 2020, a model developed by an
academic research group turned predictions of COVID-19 hospitalizations into
estimates of ventilator usage and shortages. This framework was used by the
Hartford HealthCare system in Connecticut “to align ventilator supply with
projected demand at a time where the [COVID-19] pandemic was on the rise”
(Bertsimas et al., 2021). However, we emphasize again that these studies did
not take the step of evaluating forecasts based on the quality of the
allocation decisions that they supported or could have been used to support.

##  2 The Allocation Score

We begin with an informal description of the allocation score and some
examples illustrating its key characteristics in section 2.1. In section 2.2
we develop the allocation score more carefully, using a decision theoretic
procedure for deriving proper scoring rules. (See section B of the supplement
for a definition of a proper scoring rule and a more technical discussion of
the procedure). In section 2.3, we note that another group of common scores
including the quantile score, WIS, and CRPS, can also be derived from decision
theoretic foundations —starting from a different decision making context.

###  2.1 Overview of Allocation Scoring

Suppose that a decision maker is tasked with determining how to allocate $K$
available units of a resource across $N$ locations. If the decision maker is
provided with a multivariate forecast $F$ where each marginal forecast
distribution $F_{i}$ predicts resource need in a particular location, one
option is to choose the resource allocation that minimizes the expected total
unmet need according to the forecast. We will give a more precise mathematical
statement in section 2.2, but informally, the total expected unmet need
according to the forecast is

$\displaystyle\sum_{i=1}^{N}\mathbb{E}_{F_{i}}[\text{unmet need in location
$i$}],$ (1)

where the unmet need in a particular location is the difference between
resource need in that location and the number of resources that were allocated
there. This allocation problem has an intuitively appealing solution: allocate
so that the probabilities of need exceeding allocation in various locations
are as close to each other as possible. This will lead to the allocations
provided by $F$ being quantiles of the marginal distributions $F_{i}$ for some
_single_ probability level $\tau$ that is shared in common for all locations.

After time passes and the actual level of resource need has been observed, the
quality of a selected allocation can be measured by comparing the actual need
in each location to the amount of resources that were sent there.
Specifically, we compute the total unmet need that resulted from the selected
allocation:

$\displaystyle\sum_{i=1}^{N}\text{unmet need in location $i$}.$ (2)

One allocation is better than another if it results in lower total unmet need,
and one forecast is better than another if the allocation derived from it
results in lower total unmet need.

The allocation score of the forecast $F$ is the avoidable unmet need that
results from using the allocation that minimizes the expected unmet need
according to that forecast. By “avoidable unmet need”, we mean that the
allocation score does not include the amount of unmet need that was inevitable
simply because the amount of available resources $K$ was less than the need
for resources. Rather, the allocation score measures the unmet need that could
have been avoided by an oracle that knows exactly how much need will occur in
each location and divides the amount $K$ so that nothing is wasted in one
location while it could be put to use in another. An allocation score of 0 is
optimal, and indicates that no other allocation of resources could have met
need better than the allocation suggested by $F$. A larger allocation score
indicates that it would have been possible to improve upon the allocation
suggested by $F$.

##### Example 1

Suppose we have a forecast $F$ for need in two locations with
$F_{1}=\mathrm{Exp}(1/\sigma_{1})$ and $F_{2}=\mathrm{Exp}(1/\sigma_{2})$,
where $\sigma_{1}=1$ and $\sigma_{2}=4$. The means of these distributions are
given by the scale parameters $\sigma_{i}$. When the marginal forecasts are
exponential distributions, it can be shown that the optimal allocation divides
the available resources among the locations proportionally to the scale
parameters $\sigma_{i}$ (see section D of the supplemental materials). If
$K=5$ units of the resource are available, the optimal allocation according to
$F$ would be 1 unit of resources in location 1 and 4 units of resources in
location 2. If, on the other hand, $K=10$ units are available, we will
allocate 2 units of resources to location 1 and 8 units to location 2. Figure
1 illustrates the situation.

Figure 1: An illustration of the resource allocation problem in Example 1.
There are $N=2$ locations, with predictive distributions
$F_{1}=\mathrm{Exp}(1)$ and $F_{2}=\mathrm{Exp}(1/4)$. The cumulative
distribution functions of these distributions are illustrated in the panels at
bottom and right. In the center panel, the background shading corresponds to
the expected loss according to these forecasts. Diagonal black lines indicate
resource constraints at $K=5$ and $K=10$ units; any point along those lines
corresponds to an allocation that meets the resource constraint. For these
forecasts, the optimal allocations are $(1,4)$ for $K=5$ and $(2,8)$ for
$K=10$. These allocations are at the point on the constraint line where the
expected loss is smallest, which also corresponds to the point where a level
set of the expected loss surface (blue curve) is tangent to the constraint.

Next suppose that we observe resource needs of 1 and 10 in locations 1 and 2,
respectively. Based on these observed needs, we can measure the quality of the
allocation suggested by the forecast by calculating the amount of unmet need
that resulted from that allocation over and above what was unavoidable given
the resource constraint. With $K=5$ units of the resource, the allocation
based on the forecast exactly meets the observed need in location 1, but it
leaves 6 units of need unmet in location 2. However, working within the
resource constraint, no other allocation could have done better: for example,
allocating 0 units of resources to location 1 and 5 to location 2 still
results in a total unmet need of 6 across both locations. Therefore, the
forecast’s allocation score is 0 with $K=5$. On the other hand, when $K=10$,
the forecast’s allocation results in $10-8=2$ units of unmet need in location
2 despite leaving no need unmet in location 1. In this case, the oracle would
be able to prevent all but 1 of the total 11 units of need from going unmet,
for example by allocating 1 unit of resources to location 1 and the remaining
9 units of resources to location 2. The allocation score for the forecast when
$K=10$ would therefore be 1 (= 2 realized $-$ 1 unavoidable) in units of
avoidable unmet need.

These scores illustrate a general result: allocation scores for a forecast
will tend to be larger when the resource constraint is close to the observed
need, because this is when it matters most which locations are allocated more
or less resources. If the resource constraint is very small, any allocation of
those limited resources will result in a large amount of unmet need. If the
resource constraint is very large, it becomes less important which locations
receive relatively more or less resources because all locations will receive
enough resources to meet their need. In either of these extremes of resource
availability, the avoidable unmet need that arises from the allocation
suggested by a forecast (i.e., the forecast’s allocation score) will tend to
be small.

##### Example 2

Now consider a different forecast that also has exponential distributions for
resource need in each location, but that has the scale parameters
$\sigma_{1}=2$ and $\sigma_{2}=8$, twice as large as the scale parameters of
the forecast in Example 1. Because the optimal allocation is proportional to
the scale parameters, this forecast would lead to the same allocations as the
forecast in Example 1, and would therefore be assigned the same allocation
score.

Examination of these results leads to two observations. First, the reason that
these forecasts had a positive (i.e., non-optimal) allocation score at $K=10$
is that they did not get the relative magnitude of resource need across the
two locations right: the realized need was 10 times as large in location 2 as
in location 1, but the forecasts only indicated that the resource allocation
for location 2 should be 4 times the allocation for location 1. At its core,
the allocation score measures whether the forecast accurately captures the
relative magnitudes of resource need across different locations, which is
precisely the information that is needed to allocate resources to those
locations subject to a fixed resource constraint.

A second observation is that the forecasts in examples 1 and 2 predicted
different mean levels of resource needs, but had the same allocation score.
The allocation score does not directly measure whether the forecasts correctly
capture the absolute magnitude of resource need in each individual location.
This stands in contrast to other common scoring methods that aggregate scores
such as log score, CRPS, or WIS for each location, where a poor forecast for
one location is penalized regardless of alignments in other units.

###  2.2 A decision theoretic development of the allocation score

We give a high-level review of a general procedure for developing proper
scoring rules that are tailored to specific decision making tasks in section
2.2.1, and then in section 2.2.2 we apply that procedure to develop the
allocation score based on the task of deciding on how to allocate a fixed
supply of resources across multiple locations. In 2.2.3 we consider a small
extension where the resource constraint is not known, or it is desired to
consider the value of forecasts across a range of decision making scenarios.
This gives rise to the _integrated allocation score_.

####  2.2.1 The decision theoretic setup for forecast evaluation

In the framework of decision theory, a decision corresponds to the selection
of an action $x$ from some set of possible actions $\mathcal{X}$. For example,
$x$ may correspond to the level of investment in a measure designed to
mitigate severe disease outcomes such as hospital beds, ventilators,
medication, or medical staff, with $\mathcal{X}$ being the set of all possible
levels of investment that we might select. The quality of a decision to take a
particular action $x$ is measured in relation to an outcome $y$ that is
unknown at the time the decision is made. For example, $y$ may correspond to
the number of individuals who eventually become sick and would benefit from
the mitigation measure, and informally, an action $x$ is successful to the
extent that it meets the realized need. In the face of uncertainty, a decision
maker may use a forecast $F$ of the random variable $Y$ to help inform the
selection of the action to take. We measure the value of a forecast as an
input to this decision making process by the quality of the decisions that it
leads to.

We can formalize the preceding discussion with the following three-step
procedure for developing scoring rules for probabilistic forecasts:

  1. 1.

Specify a _loss function_ $s(x,y)$ that measures the loss associated with
taking action $x$ when outcome $y$ eventually occurs.

  2. 2.

Given a probabilistic forecast $F$, determine the _Bayes act_ $x^{F}$ that
minimizes the expected loss under the distribution $F$.

  3. 3.

The _scoring rule_ for $F$ calculates the score as the loss incurred when the
Bayes act was used: $S(F,y)=s(x^{F},y)$.

This is a general procedure that may be applied in settings where it is
possible to specify a quantitative loss function. We call a scoring rule
obtained from this procedure a _Bayes scoring rule_ and in section B of the
supplement we show that Bayes scoring rules are proper by construction.

####  2.2.2 The allocation score for a fixed resource constraint

In the decision making setting that we consider, an action
$x=(x_{1},\ldots,x_{N})$ is a vector specifying the amount that is allocated
to each of $N$ locations. We require that $0\leq x$, i.e., that each $x_{i}$
is non-negative, and that the total allocation across all locations equals the
amount of available resources, $K$: $\sum_{i=1}^{N}x_{i}=K$. The set
$\mathcal{X}$ consists of all possible allocations that satisfy these
constraints. The eventually realized resource need in each location is denoted
by $y=(y_{1},\ldots,y_{N})$. These levels of need are not known at the time of
decision making, so we define the random vector $Y=(Y_{1},\ldots,Y_{N})$ where
$Y_{i}$ represents the as-yet-unknown level of resource need in location $i$.
Forecasts of need in each location are collected in $F=(F_{1},\ldots,F_{N})$.
We assume that resource need is non-negative and the forecasts reflect that,
i.e. the support of each $F_{i}$ is a subset of $\mathbb{R}^{+}$. Finally, we
assume that each unit of unmet need incurs a loss denoted by $L$, so that if
the selected resource level $x_{i}$ in location $i$ is less than the realized
need $y_{i}$, a loss of $L\cdot(y_{i}-x_{i})$ results. A variety of extensions
to this setup are possible; for example, we might account for storage costs
for resources that go unused, allow for a different loss per unit of unmet
need in each location, or account for resource transportation costs. In this
work, we choose to keep the loss function relatively simple to focus on the
core ideas.

It is helpful to clearly distinguish between the time $t_{d}$ when a _d_
ecision is made about a public health resource allocation and the time $t_{r}$
when _r_ esource needs that might be addressed by that allocation occur. Our
setup assumes that $t_{d}<t_{r}$. Additionally, the structure of our loss
captures a setting where the resource in question does not impact the amount
of demand that will materialize at time $t_{r}$, but rather it is a resource
that satisfies that demand. In the context of infectious disease, this means
that we do not consider resources that are intended to reduce the number of
people who will become sick at some point in the future, such as a
preventative influenza or COVID-19 vaccine. Instead, our setup addresses
resources like hospital beds, oxygen supply, or ventilators which are intended
to meet the medical needs of patients who are already sick. Additionally, our
setup addresses decision-making that is related to resource needs only at the
time $t_{r}$; we do not explicitly consider sequences of multiple decisions
that are made over time or account for the impact of decisions on resource
needs at any time other than $t_{r}$. We outline some opportunities to extend
our work to more complex decision making settings in the discussion.

With this problem formulation in place, we can develop a proper scoring rule
following the outline in section 2.2.1.

##### Step 1: specify a loss function.

The loss associated with a particular allocation is calculated by summing
contributions from unmet need in each location:

$s_{A}(x,y)=\sum_{i=1}^{N}L\cdot\max(0,y_{i}-x_{i}).$ (3)

Here, $\max(0,y_{i}-x_{i})$ is the unmet need in location $i$, which is given
by $y_{i}-x_{i}$ if the realized need $y_{i}$ in location $i$ is greater than
the amount $x_{i}$ allocated to that location, or $0$ if the amount $x_{i}$
allocated to unit $i$ is greater than or equal to the realized need. Also, $L$
is a constant scalar value, the same across all locations, specifying the
“cost” of one unit of unmet need.

##### Step 2: Given a probabilistic forecast $F$, identify the Bayes act.

The Bayes act associated with the forecast, $x^{F,K}$, is the allocation that
minimizes the expected loss, that is, the solution of the _allocation problem_
associated with $K$:

$\displaystyle\underset{0\leq
x}{\mathrm{minimize}}\,\,\mathbb{E}_{F}[s_{A}(x,Y)]\text{ subject to
}\,\sum_{i=1}^{N}x_{i}=K,$ (4)

where
$\mathbb{E}_{F}[s_{A}(x,Y)]=\sum_{i=1}^{N}L\cdot\mathbb{E}_{F_{i}}[\max(0,Y_{i}-x_{i})]$
sums the expected loss due to unmet need across all locations.

In the supplement we show that the components of the Bayes act are quantiles
$x_{i}^{F,K}=F_{i}^{-1}(\tau^{F,K})$ at a probability level $\tau^{F,K}$ that
depends on the forecast $F$ and the resource constraint $K$, but is shared
across all locations. This probability level is the level at which the
resource constraint is satisfied: $\sum_{i=1}^{N}F_{i}^{-1}(\tau^{F,K})=K$.
This tells us that in order to allocate optimally (according to $F$), we must
divide resources among the locations so that there is an equal forecasted
probability in every location that the allocation is sufficient to meet
resource need. This solution to the allocation problem is well-known in
inventory management and is often attributed to Hadley and Whitin (1963).

##### Step 3: Define the scoring rule.

We can now use the Bayes act to define a proper scoring rule for the
probabilistic forecast $F$. Consider first the “raw” score defined as

$\displaystyle
S_{A}^{\text{raw}}(F,y;K)=s_{A}(x^{F,K},y)=\sum_{i=1}^{N}L\cdot\max(0,y_{i}-x_{i}^{F,K}).$
(5)

This measures the total unmet need across all locations that results from
using the Bayes allocation associated with the forecast $F$ when the actual
level of need in each location is observed to be $y_{i}$.

To make this a more easily interpreted measure of forecast performance, we
will adjust the raw score by subtracting the minimum loss achievable by an
_oracle_ allocator which has precise foreknowledge of the outcomes $y_{i}$.
When the oracle has sufficient resources to meet the total need, i.e., when
$\sum_{i=1}^{N}y_{i}\leq K$, the oracle’s loss is zero and allocation score
coincides with the raw score. On the other hand, when the oracle cannot cover
all need and incurs a loss of $L\cdot\left(\sum_{i=1}^{N}y_{i}-K\right)>0$, we
adjust the raw score by this loss. The oracle-adjusted score can therefore be
written as

$\displaystyle S_{A}(F,y;K)$
$\displaystyle=S_{A}^{\text{raw}}(F,y;K)-L\cdot\max\left(0,\sum_{i=1}^{N}y_{i}-K\right)$
(6)
$\displaystyle=L\left\\{\sum_{i=1}^{N}\max(0,y_{i}-x_{i}^{F,K})-\max\left(0,\sum_{i=1}^{N}y_{i}-K\right)\right\\}.$
(7)

The oracle adjustment aligns with a common theme in economic decision theory
that _opportunity loss_ (often known as _regret_ or (negative) _relative
utility_) is often a more important quantity than absolute loss (see e.g.,
Diecidue and Somasundaram (2017)).

####  2.2.3 Integrating the allocation score across resource constraint levels

The allocation score $S_{A}$ that we developed in the previous section
measures the skill of the forecast distributions $F$ based on a single
probability level $\tau^{F,K}$. This is appropriate if the resource constraint
$K$ is a known constant. However, if $K$ is not precisely known at the time of
decision making or there is interest in measuring the value of forecasts
across a range of decision making scenarios with different resource
constraints, we can use an _integrated allocation score_ (IAS) that integrates
the allocation score across values of $K$, weighting by a distribution $p$:

$S_{IAS}(F,y)=\int S_{A}(F,y;K)p(K)\,dK$

We note that the device of considering a range of hypothetical decision makers
or decision making problems with different problem parameters has been
employed in the past (e.g., Murphy, 1993).

###  2.3 Connections to Other Scores

The weighted interval score (WIS) was proposed in 2020 as a way to score
forecasts that were being made in the early stages of the COVID-19 pandemic
(Bracher et al., 2021); equivalent scores had also been used in previous
forecast evaluation efforts (e.g., Hong et al., 2016). The WIS is a proper
scoring rule for forecasts that use a set of quantiles to represent a
probabilistic forecast distribution. While pointing a reader interested in
more mathematical detail to Bracher et al. (2021), we note simply that the WIS
is a weighted sum of interval scores at different probability levels (e.g.,
50% prediction intervals, 80% PIs, 95% PIs, etc…). Larger interval scores
indicate less skillful forecasts. An interval score consists of (a) the width
of the interval, with larger intervals receiving higher scores, and (b) a
penalty if the interval does not cover the eventual observation, which
increases the further away the interval is from the observed value.
Equivalently, the WIS can also be characterized as a weighted sum of quantile
scores for each individual predictive quantile. The quantile score for a
particular quantile level assigns an asymmetric penalty to predictions that
are too high or too low, with the relative sizes of the penalties set so that
in expectation the score is minimized by the given quantile of the
distribution. The most commonly used version of WIS is one that uses an equal
weighting of all quantile levels, in which case WIS approximates the
continuous ranked probability score (CRPS), a commonly used score for
probabilistic forecasts. _It is important to note that this weighting was
proposed because the resulting score approximates the CRPS, and not because it
aligned with any particular public health decision-making rationale._

That said, the quantile score and WIS can be derived using the same decision
theoretic procedure that we outlined in section 2.2. In fields such as
meteorology and supply chain management, a great deal of attention has been
given to the problem where a decision must be made about the quantity of a
resource to purchase for a single location in the face of a fixed cost $C$ for
each unit of the resource and a loss $L$ that will be incurred for each unit
of unmet need. This leads to the quantile score for the probability level
$\tau=1-C/L$. From this point, the WIS or CRPS can be obtained by averaging
across a range of decision making settings with different cost and loss
parameters, using a similar motivation that we used to obtain the IAS from the
AS in section 2.2.3 (Gneiting and Ranjan, 2011).

##  3 Evaluating forecasts of COVID hospitalizations using the allocation
score

We illustrate with an application to hospital admissions in the U.S.,
considering a hypothetical problem of allocating a limited supply of medical
resources to states.

###  3.1 Data

The US COVID-19 Forecast Hub collected short-term forecasts of daily new
hospital admissions for individuals with COVID-19 starting in December 2020
(Cramer et al., 2022b). The target data for these forecasts were hospital
admissions as reported by the US Department of Health and Human Services
through the HealthData.gov website. Forecasts were probabilistic predictions
of the number of new hospital admissions on a particular day in the future, in
a specific jurisdiction of the US (national level, state, or territory).
Probability distributions were represented using a set of 23 quantiles for
each individual prediction, including a median and the lower and upper limits
of 11 central prediction intervals, from a 99% to a 10% prediction interval.

The analysis in this work focuses on forecasts made before and during the
first wave of the Omicron SARS-CoV-2 variant in the US. As such, we analyzed
forecasts for the 15 weeks starting with Monday November 22, 2021 through
Monday February 28, 2022.

Submission to the Forecast Hub followed a weekly cycle, and each Monday the
Hub collected the most recent forecasts submitted by all teams that met
certain inclusion criteria and created ensemble forecasts using quantile
averaging (Ray et al., 2023). Our analysis includes these ensembles (COVIDhub-
ensemble and COVIDhub-trained_ensemble) as well as one other ensemble of hub
models created by another team (JHUAPL-SLPHospEns) and several other
individual models. Models were eligible to be included in the analysis if they
were designated as a “primary” model from a team. For a model to have a
complete, eligible submission in a given week, it had to have a 14 day-ahead
forecast for all 50 states plus Washington DC. Models had to have a complete
forecast for at least 4 of the 15 weeks in the analysis to be eligible for
inclusion.

The hospitalization data used for scoring forecasts were downloaded on
December 03, 2023.

###  3.2 Evaluation metrics

We measured forecast skill using two forecast scores, the allocation score
(AS) and the weighted interval score (WIS), both defined above. We computed
these scores for the 14 day ahead forecasts made each week. For this analysis,
we fixed the resource constraint used for the AS to be $K=15,000$, based
roughly on a reported number of ventilators available for reallocation in the
US (Ajao et al., 2015). We computed the mean WIS (MWIS) across all of the
forecasted state-level locations.

For both scores, we also computed models’ standardized rank among all models
that submitted forecasts each week. The standardized rank is between 0 and 1,
where 0 corresponds to the worst rank and 1 to the best. In the case of a tie
between one or more models, all models received the better rank.

As described above, predictions were submitted to the Forecast Hub in the form
of a set of 23 quantiles of the predictive distribution. The WIS can be
directly calculated from these quantiles. However, calculation of the AS
requires that the full cumulative distribution functions of the forecast
distributions for each location are available. For the purpose of this
analysis, to calculate the allocation score we approximated the full
cumulative distribution functions based on the provided quantiles. On the
interior of the provided quantiles we used a monotonic cubic spline to
interpolate the quantiles, and in the lower and upper tails we used normal
distributions with parameters selected so as to match the two lowest and two
highest quantiles (see the supplement for further details). In the supplement,
we show that if this evaluation procedure had been specified prospectively,
the resulting score would be proper —but a post hoc application of this
procedure is improper. We use the procedure here to illustrate the properties
of the score, and note that a forecaster or collaborative forecasting exercise
interested in using the allocation score for evaluation could circumvent
issues with propriety through thoughtful elicitation of representations of
forecast distributions or by collecting the allocations at specified resource
levels as part of forecast submissions.

###  3.3 Data and code availability

All forecast data used in this evaluation are available through the COVID-19
Forecast Hub (Cramer et al., ). An R package implementing the allocation score
is available at https://github.com/aaronger/alloscore and all code for the
analyses presented in this manuscript is available at
https://github.com/aaronger/utility-eval-papers.

###  3.4 Application results

####  3.4.1 Anatomy of forecast scores for one week

To illustrate the mechanics of allocation scoring, we start by focusing on how
forecasts generated on or before December 20, 2021, with predictions for
January 03, 2022, were scored by different metrics. This week was around the
peak of the Omicon wave nationally, with individual states typically observing
a peak at or after January 3, 2022.

Of the 10 models evaluated, the CU-select model had the most accurate
forecasts according to the allocation score while the USC-SI_kJalpha model had
the most accurate forecasts based on MWIS (Table 1). The JHUAPL-Bucky model
had the second best MWIS but the third worst allocation score.

Model | AS | MWIS | IAS centered at 15k | IAS uniform
---|---|---|---|---
CU-select | 669 | 133 | 774 | 326
COVIDhub-ensemble | 873 | 159 | 1067 | 438
USC-SI_kJalpha | 995 | 91 | 1216 | 1097
JHUAPL-Gecko | 1034 | 164 | 1141 | 418
MUNI-ARIMA | 1084 | 169 | 1248 | 440
COVIDhub-trained_ensemble | 1089 | 169 | 1271 | 823
COVIDhub-baseline | 1175 | 170 | 1317 | 535
JHUAPL-Bucky | 1358 | 102 | 1566 | 1214
JHUAPL-SLPHospEns | 1540 | 129 | 1604 | 1102
UVA-Ensemble | 2469 | 213 | 2635 | 2494
Table 1: Comparison of allocation scores (AS), mean weighted interval scores
(MWIS), and two varieties of Integrated Allocation Scores (IAS). All metrics
are shown for 10 models that made forecasts of hospital admissions for
2022-01-03. Results are sorted by AS. For all metrics, lower scores indicate
better accuracy. Figure 2: Probabilistic forecasts for new hospital admissions
and the inferred resource allocations for COVID-19 on January 3, 2022 for the
states with the ten highest hospitalization counts. For each state, the dark
black line shows the data observed when the forecast was made and the grey
line shows eventually observed counts. The side-by-side shaded regions show
the median (solid horizontal line) and 50%, 80% and 95% prediction intervals
for the two selected models. The forecasts were made for new hospitalizations
on January 3, 2022 (vertical dashed line, with number of hospitalizations
indicated by red horizontal line segment at the intersection of the dashed
line and the grey line of data). The vertical bars with red and yellow shading
show the allocations: the red bar goes from zero to the level of the observed
number of hospitalizations while the yellow bar shows how many resources were
suggested by the model to be allocated to that location. The allocations
(yellow bar) may be more or less than the observed number, therefore exposed
red bar indicates unmet need, while the yellow bar extending above the
observed number of hospitalizations means that excess resources were suggested
for that given location.

A comparison of the forecasts from the JHUAPL-Bucky and CU-select models shows
that while the JHUAPL-Bucky forecast distributions were closer to the eventual
observations in many states, the allocations suggested by those forecasts
often were more inefficient than those from the CU-select model (Figure 2). In
the example of this one week, JHUAPL-Bucky had a worse allocation score than
CU-select because its forecasts led to allocations that sent excess resources
to several states, such as Ohio, Pennsylvania, and Michigan, that would have
been more effectively allocated to states that did receive enough resources to
meet their needs, such as Florida and California (Figure 3A). These allocation
errors resulted because that model’s forecasts did not consistently capture
the relative resource needs across different states. (Figure 2 & 3A). The CU-
select model made some similar errors — most prominently, over-allocating
resources to Ohio — but overall, it did a better job of forecasting the
relative resource demands across different locations.

On the other hand, CU-select had worse performance as measured by WIS. Its
forecasts were biased downwards, and it consistently incurred a large penalty
for underprediction (Figure 3B). Predictions from the JHUAPL-Bucky model were
wider, and included the observed level of daily hospital admissions more
often. Therefore, WIS did not as often assign that model severe penalties for
forecasts that underpredicted or overpredicted the actual hospitalization
count.

Figure 3: Component-wise breakdowns of the allocation score (Panel A) and
weighted interval score (Panel B), by location for forecasts of
hospitalization admissions on January 3, 2022, for two selected models
(JHUAPL-Bucky and CU-select). Panel A shows the observed resource need, in
this case the observed number of hospitalizations, for each state, along with
the hypothetical number of resources allocated to the given location based on
the forecasts from each model. The number of available resources was fixed at
15,000 and forecasts from each model were used to determine an optimal
allocation strategy before the resource need was known. For most locations the
resource need exceeded the resources allocated, indicated by the ‘observed
resource need’ bar being larger than the ‘allocation’ bar. Panel B shows the
breakdown of the weighted interval score (WIS) into components of
underprediction, overprediction and dispersion. Larger values of WIS indicate
more error, and the full WIS score for each location can be decomposed into
the three components shown here.

####  3.4.2 Forecast scores showed differences over time

Figure 4: Hospital admissions and evaluation metrics over time. Panel A shows
the number of hospital admissions in the US as a whole due to COVID-19 on a
sequence of 15 Mondays from December 2021 through March 2022. These are the
dates for which forecasts were made and evaluated. A horizontal dashed line at
15,000 shows the assumed resource constraint $K$. Panel B shows allocation
scores (AS) for each model’s 14 day-ahead forecast, across all US states. The
x-axis corresponds to the date for which the prediction was made. AS typically
are high when the observed value is near to the constraint, which occurs
during the last Monday in December (on the way up) and the last Monday in
January (on the way down). Panel C shows the MWIS metric across weeks,
averaged across all states. MWIS values tend to scale with the observed and
predicted values, and the peak MWIS values happen around and just after the
peak of the Omicron wave.

Allocation scores varied substantially by date and by model (Figure 4). For
predictions made for the first three Mondays in December 2021 and the last
three Mondays in February 2022 all models had allocation scores under 500 (and
the mean across all models was less than 100), indicating that unnecessary
unmet need was fairly low on those days. The allocation scores were on the
whole highest when the observed number of new hospital admissions was closest
to the resource threshold of 15,000, as those are the times when any mistakes
in allocation are costly in terms of wasting resources in one location that
could have been used in another.

####  3.4.3 Metrics were not consistently correlated over time

Figure 5: Association of standardized ranks for MWIS and allocation score by
model and week. Each facet of the plot corresponds to one model. Within each
facet, each point corresponds to a week. The x- and y-values correspond to the
MWIS standardized rank and the allocation score standardized rank for that
week. Points corresponding to earlier dates have darker shading. The size of
the point corresponds to the observed value on the date for which the
prediction was made. Models show different degrees of association between the
two metrics.

Models showed differing levels of correlation between their allocation scores
and MWIS values (Figure 5). Here are some examples of the different model-
specific patterns observed:

  * •

Several models showed a positive association between allocation score and MWIS
ranks (e.g., Karlen-pypm and USC-SI_kJalpha).

  * •

Several models had consistently strong MWIS ranks but also had highly variable
allocation score ranks with no clear association between the two (e.g.,
JHUAPL-SLPHospEns and CU-select)

  * •

One model performed consistently well for both metrics with no clear
association (COVIDhub-ensemble)

  * •

One model performed consistently well for allocation score but had only
middlings ranks for MWIS (CMU-TimeSeries)

####  3.4.4 Integrated allocation score across values of K

AS was computed for a range of $K$ from 200 to 60,000 for forecasts made on
December 20, 2020 predicting levels of hospitalizations on January 3, 2021
(Figure 6A). These calculations highlight that AS was highest at at values of
$K$ near the observed nationwide total hospital admissions of 19,581 that day.
Rankings of AS from all models were fairly stable across a range of values for
K, with some crossings, especially at $K$ values further away from the
observed value.

The integrated allocation score (IAS) summarizes allocation scores (AS) across
a range of possible values of the constraint ($K$), possibly taking weights
into account for different values of $K$ that might be more or less likely
(Section 2.2.3). IAS was computed for two distributions on $K$, one uniform
across the entire range and the other a symmetric distribution around 15,000
(Figure 6B). Both versions of the IAS were correlated with the original AS
conducted at $K=15,000$, with the higher correlation coming from the
distribution that was centered at $K=15,000$ (Figure 6C). Model rankings based
on the AC and the centered IAS were roughly similar, with the top and bottom
two models being the same for both scores (Table 1).

Figure 6: Allocation scores (AS) across different resource constraints (K) for
10 models that made forecasts on 2021-12-20. Panel A shows, for each model,
the AS for values of K between 200 and 60,000 at increments of 200. The AS
show a sharp peak just under 20,000, near the eventually observed number of
hospitalizations. Panel B shows two possible weighting functions for the
Integrated Allocation Score (IAS). The first (dark blue circles) computes
weights proportional to a normal distribution centered at 15,000 (solid
vertical gray line) with a standard deviation of 3,000, and truncated to be
between 5,000 and 25,000. The second (light orange triangles) uses a uniform
weight for all possible values of K. Note that the AS used in earlier sections
of the application uses the single fixed value of K=15,000. Panel C shows how
either method of computing the IAS (y-axis) is correlated with AS at K=15,000
(x-axis). Every point represents the AS and IAS for one model. The IAS
centered at 15,000 are more closely correlated with the AS values.

##  4 Discussion

In epidemiological forecasting, well-known proper scoring rules such as the
log score or variations on the continuous ranked probability score (such as
the weighted interval score, or WIS) have been frequently utilized to evaluate
probabilistic forecasts. Often models are ranked with respect to accuracy
according to a particular score, but without reference to any underlying
decision-making process for which that score was designed or from which it
might be derivable as a Bayes scoring rule. With careful thought and
collaboration between modelers and public health officials, we argue that
scores that are more aligned with public health decisions could be developed
to inform specific problems. We have demonstrated that forecast evaluation
methods that are tied to a specific decision making context can yield model
rankings that differ substantively from those based on standard measures of
forecast skill. In particular, the allocation score penalizes models that
mischaracterize the relative magnitude of resource need across different
locations, while WIS penalizes forecasts that do not capture the absolute
magnitude of the target quantity in each location.

While our example used a hypothetical scenario to illustrate a potential
application of the allocation score, resource allocation decisions are a
realistic example of the kinds of decisions that could motivate more targeted
forecasting exercises. One real-world example, which was the motivating
example presented in the work above, is allocation of ventilators during a
respiratory viral pandemic(Huang et al., 2017). Other examples include the
allocation of a limited stockpile of vaccinations(Araz et al., 2012; Persad et
al., 2023) or diagnostic tests(Du et al., 2022; Pasco et al., 2023).

In practice, there are many potential users of forecasts with many different
decision making problems. Not all can be easily modeled as an effort to
minimize the expectation of a transparent loss function. And even those that
are easily framed as expected loss minimization may differ enough that no
single score would be appropriate for all users. Ideally, targeted forecasting
tools could be developed through close collaboration between modelers and
public health officials. However, this may only be possible in settings with
sufficient staffing on both an analytics and a public health team.
Increasingly, collaborative modeling hubs are being used to generate “one-
size-fits-all” forecasts for many locations at once. In these settings, where
tailored models are not available, it still could be possible to evaluate
contributed models using a set of multiple scores to support public health end
users in understanding the value of forecasts as an input to their particular
decision making contexts.

There are several important limitations to the current work. The allocation
score we developed here does not directly account for important considerations
such as fairness or equity of allocations. Nor does the proposed framework
attempt to capture the broader context of decision making. For example, in
practice it may be possible to increase the resource constraint $K$ by
shifting funding from other disease mitigation measures. We also note that in
some settings, a “successful” epidemiological forecast may lead to policy
decisions that change the distribution of the predicted outcome $Y$. Our
framework would need considerable enhancements before being applicable to
forecast evaluation beyond horizons for which causal feedback can be
neglected.

An opportunity for further investigation is to more carefully evaluate whether
a forecast adds value to existing decision making processes. In the context of
decisions about allocations, standard procedures might involve extrapolating
need based on a current observed data (similar to the ‘baseline’ model
presented above), with or without adjustments based on other political or
real-world considerations. For example, in many settings public health
stakeholders will make decisions after synthesizing information from a variety
of quantitative and qualitative sources coupled with expert judgment. The
allocation score presented in this work does not directly measure whether a
given forecast adds useful information to such an existing decision-making
process. While the scoring procedures as presented do not directly address
this question, they could be modified (say, by comparison to a ‘baseline’
model or expert-elicited allocations in the absence of forecast data) to
quantify the benefit of using a forecast to inform a specific decision.

In conclusion, we argue that the way modelers and policymakers view and
evaluate forecasts should change depending on the specific decision-making
context. Defaulting to standard forecast evaluation metrics can mask the
utility (or disutility) of certain forecasts, or lead to forecasts being used
in decision making contexts very different from those for which they were
intended. New collaborative work between public health officials and modeling
teams is needed to assess the value and relevance of the initial findings
presented here, including real-time pilot studies or simulation exercises that
could be used to inform further development of new or alternative scoring
metrics. We see this work as an initial overture for what we hope will grow to
be a large, collaborative body of work more closely coupling applied
epidemiological forecasting with public health decision making.

## Acknowledgements

This work has been supported by the National Institutes of General Medical
Sciences (R35GM119582) and the U.S. CDC(1U01IP001122). The content is solely
the responsibility of the authors and does not necessarily represent the
official views of NIGMS, the National Institutes of Health, or CDC.

## References

  * Igboh et al. (2023) Ledor S Igboh, Katherine Roguski, Perrine Marcenac, et al.  Timing of seasonal influenza epidemics for 25 countries in africa during 2010–19: a retrospective analysis.  _The Lancet Global Health_ , 11(5):e729–e739, 2023.
  * Meltzer et al. (2014) Martin I Meltzer, Charisma Y Atkins, Scott Santibanez, et al.  Estimating the future number of cases in the Ebola epidemic–Liberia and Sierra Leone, 2014–2015.  _MMWR_ , 63:1–14, Sep 2014.
  * Rainisch et al. (2015) Gabriel Rainisch, Manjunath Shankar, Michael Wellman, et al.  Regional spread of Ebola virus, West Africa, 2014.  _Emerging Infectious Diseases_ , 21(3):444, 2015\.
  * Bertsimas et al. (2021) Dimitris Bertsimas, Leonard Boussioux, Ryan Cory-Wright, et al.  From predictions to prescriptions: A data-driven response to covid-19.  _Health Care Management Science_ , 24:253–272, 2021.
  * Fox et al. (2022) Spencer J. Fox, Michael Lachmann, Mauricio Tec, et al.  Real-time pandemic surveillance using hospital admissions and mobility data.  _Proceedings of the National Academy of Sciences_ , 119(7):e2111870119, February 2022.  doi: 10.1073/pnas.2111870119.  URL https://www.pnas.org/doi/10.1073/pnas.2111870119.  Publisher: Proceedings of the National Academy of Sciences.
  * utn (2022) COVID forecasting method using hospital and cellphone data proves it can reliably guide us cities through pandemic threats.  Available at https://news.utexas.edu/2022/02/02/covid-forecasting-method-using-hospital-and-cellphone-data-proves-it-can-reliably-guide-us-cities-through-pandemic-threats/ (2023/05/26), 2022.
  * Yardley and Petropoulos (2021) Elizabeth Yardley and Fotios Petropoulos.  Beyond error measures to the utility and cost of the forecasts.  _Foresight: The International Journal of Applied Forecasting_ , (63):36–45, 2021.
  * Pesaran and Skouras (2002) M Hashem Pesaran and Spyros Skouras.  Decision-based methods for forecast evaluation.  _A companion to economic forecasting_ , pages 241–267, 2002.
  * Murphy (1993) Allan H Murphy.  What is a good forecast? an essay on the nature of goodness in weather forecasting.  _Weather and forecasting_ , 8(2):281–293, 1993\.
  * Leitch and Tanner (1991) Gordon Leitch and J Ernest Tanner.  Economic forecast evaluation: profits versus the conventional error measures.  _The American Economic Review_ , 81(3):580–590, 1991.
  * Cenesizoglu and Timmermann (2012) Tolga Cenesizoglu and Allan Timmermann.  Do return prediction models add economic value?  _Journal of Banking & Finance_, 36(11):2974–2987, 2012.
  * Catt et al. (2007) Peter Maurice Catt et al.  Assessing the cost of forecast error: A practical example.  _Foresight: The International Journal of Applied Forecasting_ , 7:5–10, 2007.
  * Petropoulos et al. (2019) Fotios Petropoulos, Xun Wang, and Stephen M Disney.  The inventory performance of forecasting methods: Evidence from the m3 competition data.  _International Journal of Forecasting_ , 35(1):251–265, 2019.
  * Palmer (2002) Tim N Palmer.  The economic value of ensemble forecasts as a tool for risk assessment: From days to decades.  _Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography_ , 128(581):747–774, 2002.
  * Pappenberger et al. (2015) Florian Pappenberger, Hannah L Cloke, Dennis J Parker, et al.  The monetary benefit of early flood warnings in europe.  _Environmental Science & Policy_, 51:278–291, 2015.
  * Ioannidis et al. (2022) John PA Ioannidis, Sally Cripps, and Martin A Tanner.  Forecasting for covid-19 has failed.  _International journal of forecasting_ , 38(2):423–438, 2022.
  * Bilinski et al. (2023) Alyssa M. Bilinski, Joshua A. Salomon, and Laura A. Hatfield.  Adaptive metrics for an evolving pandemic: A dynamic approach to area-level COVID-19 risk designations.  _Proceedings of the National Academy of Sciences_ , 120(32):e2302528120, August 2023.  doi: 10.1073/pnas.2302528120.  URL https://www.pnas.org/doi/10.1073/pnas.2302528120.  Publisher: Proceedings of the National Academy of Sciences.
  * Marshall et al. (2023) Maximilian Marshall, Felix Parker, and Lauren Marie Gardner.  When are predictions useful? a new method for evaluating epidemic forecasts.  _medRxiv_ , pages 2023–06, 2023.
  * Probert et al. (2016) William J.M. Probert, Katriona Shea, Christopher J. Fonnesbeck, et al.  Decision-making for foot-and-mouth disease control: Objectives matter.  _Epidemics_ , 15:10–19, 2016.  ISSN 1755-4365.  doi: https://doi.org/10.1016/j.epidem.2015.11.002.  URL https://www.sciencedirect.com/science/article/pii/S175543651500095X.
  * McGowan et al. (2019) Craig J. McGowan, Matthew Biggerstaff, Michael Johansson, et al.  Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016.  _Scientific Reports_ , 9(1):683, January 2019\.  ISSN 2045-2322.  doi: 10.1038/s41598-018-36361-9.  URL https://www.nature.com/articles/s41598-018-36361-9.
  * Reich et al. (2019) Nicholas G. Reich, Logan C. Brooks, Spencer J. Fox, et al.  A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States.  _Proceedings of the National Academy of Sciences of the United States of America_ , 116(8):3146–3154, February 2019.  ISSN 1091-6490.  doi: 10.1073/pnas.1812594116.
  * Johansson et al. (2019) Michael A. Johansson, Karyn M. Apfeldorf, Scott Dobson, et al.  An open challenge to advance probabilistic forecasting for dengue epidemics.  _Proceedings of the National Academy of Sciences of the United States of America_ , 116(48):24268–24274, November 2019.  ISSN 1091-6490.  doi: 10.1073/pnas.1909865116.
  * Cramer et al. (2022a) Estee Y. Cramer, Evan L. Ray, Velma K. Lopez, et al.  Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.  _Proceedings of the National Academy of Sciences_ , 119(15):e2113561119, April 2022a.  doi: 10.1073/pnas.2113561119.  URL https://www.pnas.org/doi/full/10.1073/pnas.2113561119.  Publisher: Proceedings of the National Academy of Sciences.
  * Sherratt et al. (2023) Katharine Sherratt, Hugo Gruson, Rok Grah, et al.  Predictive performance of multi-model ensemble forecasts of covid-19 across european nations.  _Elife_ , 12:e81916, 2023.
  * Colón-González et al. (2021) Felipe J. Colón-González, Leonardo Soares Bastos, Barbara Hofmann, et al.  Probabilistic seasonal dengue forecasting in Vietnam: A modelling study using superensembles.  _PLOS Medicine_ , 18(3):e1003542, March 2021.  ISSN 1549-1676.  doi: 10.1371/journal.pmed.1003542.  URL https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1003542.  Publisher: Public Library of Science.
  * Camacho et al. (2015) Anton Camacho, Adam Kucharski, Yvonne Aki-Sawyerr, et al.  Temporal changes in ebola transmission in sierra leone and implications for control requirements: a real-time modelling study.  _PLoS currents_ , 7, 2015.
  * Hadley and Whitin (1963) G. Hadley and Thomson M. Whitin.  _Analysis of inventory systems._ Prentice-Hall international series in management. Prentice-Hall, 1963\.
  * Diecidue and Somasundaram (2017) Enrico Diecidue and Jeeva Somasundaram.  Regret theory: A new foundation.  _Journal of Economic Theory_ , 172:88–119, 2017.  ISSN 0022-0531.  doi: https://doi.org/10.1016/j.jet.2017.08.006.  URL https://www.sciencedirect.com/science/article/pii/S0022053117300844.
  * Bracher et al. (2021) Johannes Bracher, Evan L Ray, Tilmann Gneiting, and Nicholas G Reich.  Evaluating epidemic forecasts in an interval format.  _PLoS computational biology_ , 17(2):e1008618, 2021.
  * Hong et al. (2016) Tao Hong, Pierre Pinson, Shu Fan, et al.  Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond, 2016.
  * Gneiting and Ranjan (2011) Tilmann Gneiting and Roopesh Ranjan.  Comparing density forecasts using threshold- and quantile-weighted scoring rules.  _Journal of Business & Economic Statistics_, 29(3):411–422, 2011.  doi: 10.1198/jbes.2010.08110.  URL https://doi.org/10.1198/jbes.2010.08110.
  * Cramer et al. (2022b) Estee Y. Cramer, Yuxin Huang, Yijin Wang, et al.  The United States COVID-19 Forecast Hub dataset.  _Scientific Data_ , 9(1):462, August 2022b.  ISSN 2052-4463.  doi: 10.1038/s41597-022-01517-w.  URL https://www.nature.com/articles/s41597-022-01517-w.  Number: 1 Publisher: Nature Publishing Group.
  * Ray et al. (2023) Evan L. Ray, Logan C. Brooks, Jacob Bien, et al.  Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States.  _International Journal of Forecasting_ , 39(3):1366–1383, July 2023.  ISSN 0169-2070.  doi: 10.1016/j.ijforecast.2022.06.005.  URL https://www.sciencedirect.com/science/article/pii/S0169207022000966.
  * Ajao et al. (2015) Adebola Ajao, Scott V. Nystrom, Lisa M. Koonin, et al.  Assessing the Capacity of the US Health Care System to Use Additional Mechanical Ventilators During a Large-Scale Public Health Emergency.  _Disaster Medicine and Public Health Preparedness_ , 9(6):634–641, December 2015.  ISSN 1935-7893.  doi: 10.1017/dmp.2015.105.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4636910/.
  * (35) Estee Y. Cramer, Yuxin Huang, Yijin Wang, et al.  reichlab/covid19-forecast-hub: release for zenodo, 20210816.  URL https://zenodo.org/record/5208210.
  * Huang et al. (2017) Hsin-Chan Huang, Ozgur M. Araz, David P. Morton, et al.  Stockpiling Ventilators for Influenza Pandemics.  _Emerging Infectious Diseases_ , 23(6), 2017.  doi: 10.3201/eid2306.161417.  URL https://wwwnc.cdc.gov/eid/article/23/6/16-1417_article.
  * Araz et al. (2012) Ozgur M. Araz, Alison Galvani, and Lauren A. Meyers.  Geographic prioritization of distributing pandemic influenza vaccines.  _Health Care Management Science_ , 15(3):175–187, September 2012.  ISSN 1572-9389.  doi: 10.1007/s10729-012-9199-6.  URL https://doi.org/10.1007/s10729-012-9199-6.
  * Persad et al. (2023) Govind Persad, R. J. Leland, Trygve Ottersen, et al.  Fair domestic allocation of monkeypox virus countermeasures.  _The Lancet Public Health_ , 8(5):e378–e382, May 2023.  ISSN 2468-2667.  doi: 10.1016/S2468-2667(23)00061-0.  URL https://www.thelancet.com/journals/lanpub/article/PIIS2468-2667(23)00061-0/fulltext.  Publisher: Elsevier.
  * Du et al. (2022) Jiacong Du, Lauren J Beesley, Seunggeun Lee, et al.  Optimal diagnostic test allocation strategy during the COVID-19 pandemic and beyond.  _Statistics in Medicine_ , 41(2):310–327, 2022\.  ISSN 1097-0258.  doi: 10.1002/sim.9238.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.9238.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.9238.
  * Pasco et al. (2023) Remy Pasco, Kaitlyn Johnson, Spencer J. Fox, et al.  COVID-19 Test Allocation Strategy to Mitigate SARS-CoV-2 Infections across School Districts.  _Emerging Infectious Diseases_ , 29(3), 2023.  doi: 10.3201/eid2903.220761.  URL https://wwwnc.cdc.gov/eid/article/29/3/22-0761_article.

## Supplementary Materials

section.1section.2subsection.2.1section.3section.4section.5section.6section.7subsection.7.1subsection.7.2

##  Appendix A Introduction to supplement

We address some technical and methodological points from the main text. We
begin in section B by defining proper scoring rules and showing that the
allocation score is proper. In section C, we formalize the concept of a
_shortage_ of resources, give some key results about expected resource
shortages under a distribution characterizing uncertainty about (future)
levels of resource need, and illustrate how quantiles arise as the Bayes act
in a decision making problem about the quantity of a resource to purchase.
Section D gives a justification for the result that the Bayes act for the
allocation problem is given by a vector of quantiles of the forecast
distributions in each location at a shared probability level, which was stated
in section 2.2.2 of the main text. We describe the algorithm that we use to
compute allocations given a forecast distribution in each location in section
E. In section F, we describe the methods for approximating a full distribution
from a set of quantiles, implemented in the R package `distfromq`, that we
used to support computation of allocation scores from quantile forecasts that
were submitted to the US COVID-19 Forecast Hub for the application in the main
text. Finally, in section G we examine implications for propriety of an
analysis that uses summaries of forecasts (such as predictive quantiles)
rather than the full forecast distributions for computation of allocation
scores.

##  Appendix B Proper scoring rules

In decision theory, a loss function $l$ is used to formalize a decision
problem by assigning numerical value $l(x,y)$ to the _result_ of taking an
_action_ $x$ in preparation for an _outcome_ $y$. A _scoring rule_ $S$ is a
loss function for a decision problem where the action is a probabilistic
forecast $F$ of the outcome $y$ (or the statement of $F$ by a forecaster). We
refer to the realized loss $S(F,y)$ as the _score_ of $F$ at $y$.

Probabilistic forecasts can be seen as a unique kind of action in that they
can be used to generate their own (simulated) outcome data, against which they
can be scored using $S$. A probabilistic forecast $F$ is thus committed to the
“self-assessment”
$\operatorname{\mathbb{E}}_{F}[S(F,Y)]:=\operatorname{\mathbb{E}}[S(F,Y^{F})]$,
where $Y^{F}\sim F$ is the random variable defined by sampling from $F$, as
well to an assessment $\operatorname{\mathbb{E}}_{F}[S(G,Y)]$ of any
alternative forecast $G$.

A natural consistency criterion for $S$ is that, for observations assumed to
be drawn from $F$, it will not assess any other forecast $G$ as being better
than $F$ itself, that is, that

$\displaystyle\operatorname{\mathbb{E}}_{F}[S(F,Y)]\leq\operatorname{\mathbb{E}}_{F}[S(G,Y)]$
(8)

for any $F,G$. A scoring rule meeting this criterion is called _proper_. If
$S$ were improper, then from the perspective of a forecaster focussed (solely)
on expected loss minimization, the decision to state a forecast $G$ other than
the forecast $F$ which they believe describes $Y$ could be superior to the
decision to state $F$. $S$ is _strictly proper_ when (8) is a strict
inequality, in which case the _only_ optimal decision for a forecaster seeking
to minimize their expected loss is to state the forecast they believe to be
true.

###  B.1 The allocation score is proper

Our primary decision theoretical procedure, outlined in section 2.2.1 of the
main text, uses a decision problem with loss function $s(x,y)$ to define a
scoring rule

$\displaystyle S(F,y):=s(x^{F},y)$ (9)

where $x^{F}:=\operatorname*{argmin}_{x}\operatorname{\mathbb{E}}_{F}[s(x,Y)]$
is the Bayes act for $F$ with respect to $s$. Such scoring rules, which we
call _Bayes scoring rules_ , are proper by construction since

$\displaystyle\operatorname{\mathbb{E}}_{F}[S(F,Y)]$
$\displaystyle=\operatorname{\mathbb{E}}_{F}[s(x^{F},Y)]$
$\displaystyle=\mathrm{min}_{x}\operatorname{\mathbb{E}}_{F}[s(x,Y)]\quad\text{
(by definition of $x^{F}$)}$ (10)
$\displaystyle\leq\operatorname{\mathbb{E}}_{F}[s(x^{G},Y)]$ (11)
$\displaystyle=\operatorname{\mathbb{E}}_{F}[S(G,Y)].$

The allocation scoring rule is Bayes and therefore proper.

We note that in the probabilistic forecasting literature (see e.g., Gneiting
(2011a), Theorem 3) what we have termed Bayes scoring rules typically appear
via (9) where $x^{F}$ is some given functional of $F$ which can be shown to be
_elicitable_ , that is, to be the Bayes act for some loss function $s$. Such a
loss function is said to be a _consistent loss (or scoring) function_ for the
functional $F\mapsto x^{F}$, and many important recent results in the
literature (e.g., Fissler and Ziegel (2016)) address whether there _exists_
any loss function that is consistent for $x^{F}$. Our orientation is different
from this insofar as we _begin_ by specifying a decision problem and a loss
function of subject matter relevance and use the Bayes act only as a bridge to
a proper scoring rule. Consistency is never in doubt.

##  Appendix C Expected shortages

A key feature of loss functions for decision problems used to define quantiles
and related scoring rules such as the CRPS and the WIS (see e.g., Gneiting
(2011b), Jose and Winkler (2009), and Royset and Wets (2022), sections 1.C and
3.C), as well as the allocation loss function presented in this work, is the
presence of a _shortage_ : the amount $\max\\{0,y-x\\}$ by which a resource
demand $y$ exceeds a supply decision variable $x$, which, for convenience, we
write as $(y-x)_{+}$. In particular, a quantile at probability level $\alpha$
of a distribution $F$ on $\mathbb{R}^{1}$ (which we assume to have a well-
defined density $f(x)$) is a Bayes act for the loss function

$l(x,y)=Cx+L(y-x)_{+}$

where $\alpha=1-C/L$ and $C$ and $L$ can be interpreted as the cost per unit
of a resource (such as medicine) and the loss incurred when a unit of demand
(such as illness) cannot be met due to the shortage $(y-x)_{+}$. This follows
because a Bayes act, as a minimizer of
$\operatorname{\mathbb{E}}_{F}[l(x,Y)]$, must also be a vanishing point of the
derivative

$\displaystyle\frac{d{}}{d{x}}\operatorname{\mathbb{E}}_{F}\left[l(x,Y)\right]$
$\displaystyle=\operatorname{\mathbb{E}}_{F}\left[\frac{d{}}{d{x}}l(x,Y)\right]$
$\displaystyle=C+L\operatorname{\mathbb{E}}_{F}\left[\frac{d{}}{d{x}}(Y-x)_{+}\right]$
$\displaystyle=C-L\operatorname{\mathbb{E}}_{F}\left[\mathbf{1}\\{Y>x\\}\right]$
$\displaystyle=C+L(F(x)-1),$ (12)

so that $1-C/L=F(x)$. The formula
$\frac{d{}}{d{x}}\operatorname{\mathbb{E}}_{F}\left[(Y-x)_{+}\right]=F(x)-1$
for the derivative of the shortage, used above in (12), can be obtained from
an application of the “Leibniz Rule”:

$\displaystyle\frac{d}{dx}\operatorname{\mathbb{E}}_{F}[(Y-x)_{+}]$
$\displaystyle=\frac{d}{dx}\int_{x}^{\infty}(y-x)f_{Y}(y)dy$
$\displaystyle=\int_{x}^{\infty}\frac{d}{dx}(y-x)f_{Y}(y)dy-(x-x)f_{Y}(x)=-\int_{x}^{\infty}f_{Y}(y)dy=F(x)-1.$
(13)

Note that more care is required when $F$ does not have a density. We will also
use this result below in deriving the Bayes act for the allocation loss.

##  Appendix D Allocation Bayes acts as vectors of marginal quantiles.

Here we study the form of the Bayes act for the allocation problem (AP)
(equation (3) in section 2.2.2) of the text:

$\displaystyle\underset{0\leq
x}{\mathrm{minimize}}\,\,\mathbb{E}_{F}[s_{A}(x,Y)]=\sum_{i=1}^{N}L\cdot\mathbb{E}_{F_{i}}[(Y_{i}-x_{i})_{+}]\text{
subject to }\,\sum_{i=1}^{N}x_{i}=K,$ (14)

where the marginal forecasts $F_{i}$ for $i=1,\dots,N$ represent forecasts for
$N$ distinct locations. We show that the Bayes act
$x^{F,K}=(x_{1}^{F,K},\ldots,x_{N}^{F,K})$ for a forecast $F$ and resource
constraint level $K$ is a vector of quantiles of the marginal forecast
distributions $F_{i}$ at a single probability level $\tau^{F,K}$, that is,
$x_{i}^{F,K}=q_{F_{i},\tau^{F,K}}$. An immediate consequence used in the
examples in Section 2.1 in the main text is that if
$F_{i}=\mathrm{Exp}(1/\sigma_{i})$ for all $i$, then the Bayes act is
proportional to $(\sigma_{1},\ldots,\sigma_{N})$, since
$q_{\mathrm{Exp}(1/\sigma),\tau}=-\sigma\log(1-\tau)$.

In order for $x^{\star}\in\mathbb{R}^{N}_{+}$ to solve the AP it must be true
that reallocating $\delta>0$ units of the resource from location $i$ to
location $j$ will lead to a net increase in expected shortage — in other
words, the reallocation increases the expected shortage in location $i$ is at
least as much as it decreases the expected shortage in location $j$:

$\displaystyle\mathbb{E}_{F_{i}}[(Y_{i}-(x^{\star}_{i}-\delta))_{+}]-\mathbb{E}_{F_{i}}[(Y_{i}-x^{\star}_{i})_{+}]\text{
(increase in $i$) }$
$\displaystyle\qquad\geq\mathbb{E}_{F_{j}}[(Y_{j}-x^{\star}_{j})_{+}]-\mathbb{E}_{F_{j}}[(Y_{j}-(x^{\star}_{j}+\delta))_{+}]\text{
(decrease in $j$) }.$

Dividing by $\delta$ and letting $\delta\searrow 0$, this implies from (13)
that

$\displaystyle 1-F_{i}(x^{\star}_{i})$
$\displaystyle=-\frac{d}{dx_{i}}\mathbb{E}_{F_{i}}[(Y_{i}-x^{\star}_{i})_{+}]$
$\displaystyle=\lim_{\delta\searrow
0}\frac{1}{\delta}\left\\{\mathbb{E}_{F_{i}}[(Y_{i}-(x^{\star}_{i}-\delta))_{+}]-\mathbb{E}_{F_{i}}[(Y_{i}-x^{\star}_{i})_{+}]\right\\}\text{
(increase in $i$) }$ $\displaystyle\geq\lim_{\delta\searrow
0}\frac{1}{\delta}\left\\{\mathbb{E}_{F_{j}}[(Y_{j}-x^{\star}_{j})_{+}]-\mathbb{E}_{F_{j}}[(Y_{j}-(x^{\star}_{j}+\delta))_{+}]\right\\}\text{
(decrease in $j$) }$
$\displaystyle=-\frac{d{}}{d{x_{j}}}\mathbb{E}_{F_{j}}[(Y_{j}-x^{\star}_{j})_{+}]=1-F_{j}(x^{\star}_{j})$
(15)

Note that negative derivatives appear because our optimality condition
addresses how a _decrease_ in resources will _increase_ the expected shortage
in $i$ and vice versa in $j$. Since (15) also holds with $i$ and $j$ reversed,
a number $\lambda$ (a _Lagrange multiplier_) exists such that
$L(1-F_{k}(x^{\star}_{k}))=\lambda$ for all $k\in 1,\ldots,N$. (We scale by
$L$ to facilitate possible future interpretations of $\lambda$ in terms of the
partial derivatives of $\mathbb{E}_{F}[s_{A}(x,Y)]$.) That is, $x^{\star}_{k}$
is a quantile $q_{\tau,F_{k}}$ for $\tau=1-\lambda/L$. The value of $\tau$ is
then determined by the constraint equation

$\displaystyle\sum_{i=1}^{N}q_{\tau,F_{i}}=K.$ (16)

It is important to note that $\tau$ depends on $F$ and $K$ and is _not_ a
fixed parameter of the allocation scoring rule.

##  Appendix E Numerical computation of allocation Bayes acts

To compute an allocation score $S_{A}(F,y;K):=s_{A}(x^{F,K},y)$, we require
numerical values for a Bayes act solving the AP (14) — that is, we must find
the specific resource allocations for each location that are determined by the
forecast $F$ under the resource constraint $K$. Assuming we have reliable
means of calculating quantiles $q_{\alpha,F_{i}}$ of the marginal forecasts
$F_{i}$, these allocations are given by $q_{\tau^{\star},F_{i}}$ where
$\tau^{\star}$ solves the equation (16). However, this equation is not
analytically tractable and we must resort to a numerical method for finding an
approximation $\tilde{\tau}$ of $\tau^{\star}$.

We have implemented an iterative bisection method that makes use of the fact
that $\sum_{i=1}^{N}q_{\tau,F_{i}}$ is an increasing function of $\tau$. The
algorithm begins with an initial search interval $[\tau_{L,1},\tau_{U,1}]$
(such as $[0,\max_{i}F_{i}(K)]$) that clearly contains the solution
$\tau^{\star}$. At each step $j$ of the algorithm, we evaluate the total
allocation $\sum_{i=1}^{N}q_{\tau_{M,j},F_{i}}$ at the midpoint of the search
interval, $\tau_{M,j}=\frac{1}{2}(\tau_{L,j}+\tau_{U,j})$ and continue the
search on the narrowed sub-interval

$\displaystyle[\tau_{L,j+1},\tau_{U,j+1}]=\begin{cases}[\tau_{L,j},\tau_{M,j}]&\text{if
}\sum_{i=1}^{N}q_{\tau_{M,j},F_{i}}\geq K\\\ [\tau_{M,j},\tau_{U,j}]&\text{if
}\sum_{i=1}^{N}q_{\tau_{M,j},F_{i}}<K.\end{cases}$

This search continues until $\tau_{U,j+1}<(1+\varepsilon)\tau_{L,j+1}$ for a
suitably small $\varepsilon>0$. We have implemented this procedure along with
the resulting score computations in the R package `alloscore` (Gerding and
Ray, 2023) which provided all allocation score values used in the analysis of
section 3 in the main text.

Subtleties can arise when the forecast densities $f_{i}$ vanish or are very
small, in which case quantiles are non-unique or highly variable near a
probability level, leading to ambiguity or numerical instabilities in the
evaluation of $\sum_{i=1}^{N}q_{\tau,F_{i}}$. Additionally, if point masses
are present in any of the $F_{i}$, (16) will not have a unique solution for
some discrete set of constraint levels $K$. We have adopted conventions for
detecting such levels and enforcing consistency in score calculations near
them. Through extensive experimentation, we have determined that these
conditions seem to address these challenges with the forecasts we are working
with, but we leave a more rigorous approximation error analysis for later
work.

##  Appendix F Computing allocations from finite quantile forecast
representations

In section 3 of the manuscript, we used the allocation score to evaluate
forecasts of COVID-19 hospitalizations that have been submitted to the US
COVID-19 Forecast Hub. These forecasts are submitted to the Hub using a set of
23 quantiles of the forecast distribution at the 23 probability levels in the
set $\mathcal{T}=\\{0.01,0.025,0.05,0.1,0.15,\ldots,\allowbreak
0.9,0.95,0.975,0.99\\}$, which specify a predictive median and the endpoints
of central $(1-\alpha)\times 100\%$ prediction intervals at levels
$\alpha=0.02,0.05,0.1,0.2,\allowbreak 0.3,0.4,0.5,0.6,\allowbreak
0.7,0.8,0.9$. For a given week and target date, we use $q_{i,k}$ to denote the
submitted quantiles for location $i$ and probability level
$\tau_{k}\in\mathcal{T}$, $k=1,\ldots,23$.

In the event that there is some $k\in\\{1,\ldots,23\\}$ for which
$\sum_{i}q_{i,k}=K$, i.e., the provided predictive quantiles at level
$\tau_{k}$ sum across locations to the resource constraint $K$, the solution
to the allocation problem is given by those quantiles. However, generally this
will not be the case; the optimal allocation will typically be at some
probability level $\tau^{\star}\notin\mathcal{T}$.

To address this situation and support the numerical allocation algorithm
outlined in section E, we need a mechanism to approximate the full cumulative
distribution functions $F_{i}$, $i=1,\ldots,N$ based on the provided
quantiles. We have developed functionality for this purpose in the `distfromq`
package for R.(Ray and Gerding, 2023) This functionality represents a
distribution as a mixture of discrete and continuous parts, and it works in
two steps:

  1. 1.

Identify a discrete component of the distribution consisting of zero or more
point masses, and create an adjusted set of predictive quantiles for the
continuous part of the distribution by subtracting the point mass
probabilities and rescaling.

  2. 2.

For the continuous part of the distribution, different approaches are used on
the interior and exterior of the provided quantiles:

    1. (a)

On the interior, a monotonic cubic spline interpolates the adjusted quantiles
representing the continuous part of the distribution.

    2. (b)

A location-scale parametric family is used to extrapolate beyond the provided
quantiles. The location and scale parameters are estimated separately for the
lower and upper tails so as to obtain a tail distribution that matches the two
most extreme quantiles in each tail. In this work, we use normal distributions
for the tails.

The resulting distributional estimate exactly matches all of the predictive
quantiles provided by the forecaster. We use the cumulative distribution
function resulting from this procedure as an input to the allocation score
algorithm.

We refer the reader to the `distfromq` documentation for further detail.(Ray
and Gerding, 2023)

##  Appendix G Propriety of parametric approximation

In practice, open forecasting exercises are generally not able to collect a
perfect description of the forecast distribution $F$ other than in simple
settings such as for a categorical variable with a relatively small number of
categories. In settings where the outcome being forecasted is a continuous
quantity (such as the proportion of outpatient doctor visits where the patient
has influenza-like illness) or a count (such as influenza hospitalizations),
forecasting exercises have therefore resorted to collecting summaries of a
forecast distribution such as bin probabilities or predictive quantiles. In
this section, we address two practical concerns raised by this. First, we
discuss conditions under which it is possible to calculate the allocation
score when only summaries of a forecast distribution are recorded in a
submission to a forecast hub. Second, we show that a post hoc attempt to
compute the allocation score based on submitted predictive quantiles may in
fact compute an alternative score that is not proper.

###  G.1 Propriety when scoring methods are announced prospectively

We consider a setting where a forecasting exercise (such as a forecast hub)
pre-specifies that forecasts will be represented using a parametric family of
forecast distributions $G_{\theta}(y)$, and the task of the forecaster is to
select a particular parameter value $\theta$. We use $\mathcal{P}$ to denote
the collection of all distributions $G_{\theta}$ in the given parametric
family. For instance, it has recently been proposed that mixture distributions
could be used to represent forecast distributions (Wadsworth et al., 2023).
Additionally, we note that the functionality in `distfromq` can be viewed as
specifying a parametric family $\mathcal{P}_{\mathrm{dfq}}$ where the
parameters $\theta$ of $G_{\theta}$ are its quantiles at pre-specified
probability levels, and where the shape of any
$G_{\theta}\in\mathcal{P}_{\mathrm{dfq}}$ over the full range of its support
is entirely controlled by these quantiles.

We find it helpful now to formally distinguish between two decision making
problems. The first is the public health decision maker’s allocation problem
where the task is to select an allocation $x$, with the allocation loss
$s_{A}(x,y)=\sum_{i=1}^{N}L\cdot\max(0,y_{i}-x_{i})$ as described in section
2.2.2 of the main text. The second is the forecaster’s reporting problem where
the task is to select parameter values $\theta$ to report. The forecaster’s
loss is given by

$\displaystyle s_{R}(\theta,y)=s_{A}(x^{G_{\theta}},y),$ (17)

where $x^{G_{\theta}}$ is the Bayes act for the allocation problem under the
distribution $G_{\theta}$. In words, the loss associated with reporting
$\theta$ is equal to the loss associated with taking the Bayes allocation
corresponding to the distribution $G_{\theta}$.

Following our usual construction, the Bayes act for the forecast reporting
problem is the parameter set that minimizes the forecaster’s expected loss.
Breaking with our earlier notation for improved legibility, we use
$\theta^{\star}(F)$ to denote this Bayes act:

$\displaystyle\theta^{\star}(F)$
$\displaystyle=\text{argmin}_{\theta}\operatorname{\mathbb{E}}_{F}[s_{R}(\theta,Y)]$
$\displaystyle=\text{argmin}_{\theta}\operatorname{\mathbb{E}}_{F}[s_{A}(x^{G_{\theta}},Y)]$

We then arrive at the scoring rule

$S_{R}(F,y)=s_{R}(\theta^{\star}(F),y)=s_{A}(x^{G_{\theta^{\star}(F)}},y).$

It follows from the discussion in section B.1 that this is a proper scoring
rule for $F$. Although the full forecast distribution $F$ is not available in
the forecast submission, the score $S_{R}(F,y)$ can be calculated from the
reported parameter values as long as the forecaster submits the optimal
parameters $\theta^{\star}(F)$.

We emphasize that the forecaster’s true predictive distribution $F$ does not
need to be a member of the specified parametric family $\mathcal{P}$ for this
construction to yield a proper score. It is, however, necessary to specify the
parametric family to use and the foundational scoring rule $s_{A}$ (including
any relevant problem parameters such as the resource constraint $K$) in
advance, so that forecasters can identify the Bayes act parameter set
$\theta^{\star}(F)$ to report.

If the parametric family used to represent forecast distributions is flexible
enough, the reporting scoring rule $S_{R}$ and the allocation score are
equivalent in the sense that they will yield the same score for any
distribution $F$. Suppose that for a given resource constraint $K$, for any
forecast distribution $F$ it is possible to find a member $G_{\theta^{\star}}$
of the specified parametric family $\mathcal{P}$ with the same allocation as
$F$ (i.e., $x^{F}=x^{G_{\theta^{\star}}}$). Then $\theta^{\star}$ is a Bayes
act for the reporting problem since for any other parameter value $\theta$,

$\displaystyle\operatorname{\mathbb{E}}_{F}[s_{R}(\theta^{\star},Y)]$
$\displaystyle=\operatorname{\mathbb{E}}_{F}[s_{A}(x^{G_{\theta^{\star}}},Y)]$
$\displaystyle=\operatorname{\mathbb{E}}_{F}[s_{A}(x^{F},Y)]\quad\text{ (since
$x^{F}=x^{G_{\theta^{\star}}}$)}$
$\displaystyle\leq\operatorname{\mathbb{E}}_{F}[s_{A}(x^{G_{\theta}},Y)]\quad\text{
(by definition of $x^{F}$)}$
$\displaystyle=\operatorname{\mathbb{E}}_{F}[s_{R}(\theta,Y)].$

It therefore follows that

$\displaystyle S_{R}(F,y)$ $\displaystyle=s_{R}(\theta^{\star},y)$
$\displaystyle=s_{A}(x^{G_{\theta^{\star}}},y)$ $\displaystyle=s_{A}(x^{F},y)$
$\displaystyle=S_{A}(F,y).$

For the particular choice of the parametric family $\mathcal{P}_{\text{dfq}}$
(i.e., using the `distfromq` package), this flexibility requirement is
satisfied. For instance, the forecaster could pick one required quantile level
(such as 0.5, for which the corresponding predictions are predictive medians),
and set the submitted quantiles of their forecast distribution in each
location at that level to be the desired allocations, which sum to $K$ across
all locations. However, this representation of the forecast may be quite
different from the actual forecast distribution $F$. For example, for the
actual forecast distribution $F$ the allocations may occur at some quantile
level other than 0.5.

As another alternative for practical forecasting exercises, a forecast hub
could ask forecasters to directly provide the Bayes allocations associated
with their forecasts for one or more specified resource constraints $K$. At
the cost of increasing the number of quantities solicited by the forecast hub,
this would have several advantages: it would prevent any artificial distortion
of the forecast distributions, allow for direct calculation of scores, and
narrow the gap between model outputs and public health end users. For this to
be feasible, implementations of the allocation algorithm would have to be
provided to participating forecasters in the computational languages being
used for modeling.

###  G.2 Impropriety of post hoc allocation scoring with quantile forecasts

A _post hoc_ evaluation of quantile forecasts that combines the parametric
family specified by `distfromq` with the allocation score does not yield the
allocation score of the forecast distribution $F$. Instead, it computes an
alternative score that is improper. This is because the forecast distribution
$F$ and the distribution $G^{q}\in\mathcal{P}_{\text{dfq}}$ with the same
quantiles as $F$ may determine different resource allocations. In our
investigations, these discrepancies appear to be relatively minor on the
interior of the provided quantiles, but could be severe if the tail
extrapolations performed by `distfromq` do not match the tail behavior of $F$
and the allocations are in the tails of the predictive distribution.

We define

$G^{\star}(F):=\operatorname*{argmin}_{G\in\mathcal{P}_{\mathrm{dfq}}}E_{F}[S_{A}(G,Y)].$

Since $S_{R}$ is defined as the Bayes scoring rule for the forecaster’s loss
(17), $G^{\star}(F)$ coincides with $G_{\theta^{\star}(F)}$, the distribution
in $\mathcal{P}_{\mathrm{dfq}}$ given by the optimal submission parameters
$\theta^{\star}(F)$ for the forecaster with predictive distribution $F$. In
general, $G^{q}(F)$ and $G^{\star}(F)$ will be different distributions:
matching $F$ at specific quantiles does not require $G^{q}(F)$ to match $F$ at
the quantiles for $\tau^{F,K}$ (c.f. (16)), which would be necessary for it to
share $x^{F}$ as an optimal allocation.

When an analyst attempts a post hoc computation of the allocation score using
$G^{q}(F)$ (implicitly assuming that $G^{q}(F)=G^{\star}(F)$), they in fact
compute the alternative score

$\tilde{S}(F,y)=S_{A}(G^{q}(F),y)=s_{A}(x^{G^{q}(F)},y).$

This score is improper because $E_{F}[S_{A}(G,Y)]$ is minimized by
$G^{\star}(F)$, not $G^{q}(F)$. In general, we have

$\displaystyle E_{F}[\tilde{S}(G^{\star}(F),Y)]$ $\displaystyle\leq
E_{F}[S_{A}(G^{q}(F),Y)]$ (18) $\displaystyle=E_{F}[\tilde{S}(F,Y)]$

However, the inequality in (18) will typically be strict. For example, if $F$
has heavy upper tails (such as for a lognormal distribution), but normal
distributions are used for tail extrapolations in `distfromq`, then the
resource allocations based on the distribution $G^{q}(F)$ may be quite
different from the optimal allocations under the distribution $F$, leading to
a strict inequality. This demonstrates that $\tilde{S}$ is improper.

## References

  * Gneiting (2011a) Tilmann Gneiting.  Making and evaluating point forecasts.  _Journal of the American Statistical Association_ , 106(494):746–762, 2011a.
  * Fissler and Ziegel (2016) Tobias Fissler and Johanna F. Ziegel.  Higher order elicitability and Osband’s principle.  _The Annals of Statistics_ , 44(4):1680 – 1707, 2016.
  * Gneiting (2011b) Tilmann Gneiting.  Quantiles as optimal point forecasts.  _International Journal of forecasting_ , 27(2):197–207, 2011b.
  * Jose and Winkler (2009) Victor Richmond R Jose and Robert L Winkler.  Evaluating quantile assessments.  _Operations research_ , 57(5):1287–1297, 2009\.
  * Royset and Wets (2022) Johannes O Royset and Roger J-B Wets.  _An optimization primer_.  Springer, 2022.
  * Gerding and Ray (2023) Aaron Gerding and Evan Ray.  _alloscore: Tools for Implementing Allocation Scoring Rules_ , 2023\.  URL https://github.com/aaronger/alloscore.  R package version 0.0.9001.
  * Ray and Gerding (2023) Evan L Ray and Aaron Gerding.  _distfromq: Reconstruct a Distribution from a Collection of Quantiles_ , 2023.  URL https://github.com/reichlab/distfromq.  R package version 1.0.2.
  * Wadsworth et al. (2023) Spencer Wadsworth, Jarad Niemi, and Nicholas G Reich.  Mixture distributions for probabilistic forecasts of disease outbreaks.  _arXiv preprint arXiv:2310.11939_ , 2023.  URL https://doi.org/10.48550/arXiv.2310.11939.