# From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation

Eric Neyman, Tim Roughgarden

This paper forges a strong connection between two seemingly unrelated
forecasting problems: incentive-compatible forecast elicitation and forecast
aggregation. Proper scoring rules are the well-known solution to the former
problem. To each such rule $s$ we associate a corresponding method of
aggregation, mapping expert forecasts and expert weights to a “consensus
forecast,” which we call _quasi-arithmetic (QA) pooling_ with respect to $s$.
We justify this correspondence in several ways:

  * •

QA pooling with respect to the two most well-studied scoring rules (quadratic
and logarithmic) corresponds to the two most well-studied forecast aggregation
methods (linear and logarithmic).

  * •

Given a scoring rule $s$ used for payment, a forecaster agent who sub-
contracts several experts, paying them in proportion to their weights, is best
off aggregating the experts’ reports using QA pooling with respect to $s$,
meaning this strategy maximizes its worst-case profit (over the possible
outcomes).

  * •

The score of an aggregator who uses QA pooling is concave in the experts’
weights. As a consequence, online gradient descent can be used to learn
appropriate expert weights from repeated experiments with low regret.

  * •

The class of all QA pooling methods is characterized by a natural set of
axioms (generalizing classical work by Kolmogorov on quasi-arithmetic means).

##  1 Introduction and motivation

###  1.1 Probabilistic opinion pooling

You are a meteorologist tasked with advising the governor of Florida on
hurricane preparations. A hurricane is threatening to make landfall in Miami,
and the governor needs to decide whether to order a mass evacuation. The
governor asks you what the likelihood is of a direct hit, so you decide to
consult several weather models at your disposal. These models all give you
different answers: 10%, 25%, 70%. You trust the models equally, but your job
is to come up with one number for the governor — your best guess, all things
considered. What is the most sensible way for you to aggregate these numbers?

This is one of many applications of _probabilistic opinion pooling_. The
problem of probabilistic opinion pooling (or _forecast aggregation_) asks: how
should you aggregate several probabilities, or probability distributions, into
one? This question is relevant in nearly every domain involving probabilities
or risks: meteorology, national security, climate science, epidemiology, and
economic policy, to name a few.

The setting that interests us is as follows: there are $m$ experts, who report
probability distributions $\mathbf{p}_{1},\dots,\mathbf{p}_{m}$ over $n$
possible outcomes (we call these _reports_ , or _forecasts_). Additionally,
each expert $i$ has a non-negative weight $w_{i}$ (with weights adding to
$1$); this weight represents the expert’s quality, i.e. how much the
aggregator trusts the expert. A pooling method takes these distributions and
weights as input and outputs a single distribution $\mathbf{p}$. (Where do
these weights come from? How can one learn weights for experts? More on this
later.)

Linear pooling is arguably the simplest of all reasonable pooling methods: a
weighted arithmetic mean of the probability distributions.

$\mathbf{p}=\sum_{i=1}^{m}w_{i}\mathbf{p}_{i}$

Logarithmic pooling (sometimes called _log-linear_ or _geometric_ pooling)
consists of taking a weighted geometric mean of the probabilities and scaling
appropriately.

$p(j)=c\prod_{i=1}^{m}(p_{i}(j))^{w_{i}}.$

Here, $p(j)$ denotes the probability of the $j$-th outcome and $c$ is a
normalizing constant to make the probabilities add to $1$. Logarithmic pooling
can be interpreted as averaging the experts’ Bayesian evidence (see Appendix
B).

The linear and logarithmic pooling methods are by far the two most studied
ones, see e.g. [GZ86], [PR00], [KR08]. This is because they are simple and
follow certain natural rules, which we briefly discuss in Section 2.
Furthermore, they are each optimal according to some natural optimality
metrics, see e.g. [Abb09].

###  1.2 Proper scoring rules

A seemingly unrelated topic within probabilistic forecasting is the truthful
elicitation of forecasts: how can a principal structure a contract so as to
elicit an expert’s probability distribution in a way that incenvitizes
truthful reporting? This is usually done using a _proper scoring rule_.

A _scoring rule_ is a function $s$ that takes as input (1) a probability
distribution over $n$ outcomes111We specify the domain of $s$ more precisely
in Section 3. and (2) a particular outcome, and assigns a _score_ , or reward.
The interpretation is that if the expert reports a distribution $\mathbf{p}$
and event $j$ comes to pass, then the expert receives reward $s(\mathbf{p};j)$
from the principal. A scoring rule is called _proper_ if the expert’s expected
score is strictly maximized by reporting their probability distribution
truthfully. That is, $s$ is proper if

$\sum_{j=1}^{n}\mathbf{p}(j)s(\mathbf{p};j)\geq\sum_{j=1}^{n}\mathbf{p}(j)s(\mathbf{x};j)$

for all $\mathbf{x}$, with equality only for $\mathbf{x}=\mathbf{p}$. It is
worth noting that properness is preserved under positive affine
transformations. That is, if $s$ is proper, then
$s^{\prime}(\mathbf{p};j):=as(\mathbf{p};j)+b$ is proper if $a>0$.

#### Quadratic scoring rule

One example of a proper scoring rule is _Brier’s quadratic scoring rule_ ,
introduced in [Bri50]. It is given by

$s_{\text{quad}}(\mathbf{p};j):=2p(j)-\sum_{k=1}^{n}p(k)^{2}.$

The quadratic scoring rule can be interpreted as penalizing the expert by an
amount equal to the squared distance from their report $\mathbf{p}$ to the
“true answer” $\delta_{j}$ (i.e. the vector with a $1$ in the $j$-th position
and zeros elsewhere).

#### Logarithmic scoring rule

Another example of a proper scoring rule is the _logarithmic scoring rule_ ,
introduced in [Goo52]. It is given by

$s_{\log}(\mathbf{p};j):=\ln p(j).$

The logarithmic rule is the only proper scoring rule for which an expert’s
score only depends on the probability assigned to the eventual outcome and not
other outcomes [SAM66]. The quadratic and logarithmic scoring rules are by far
the most studied and most frequently used ones in practice.

#### Choice of scoring rule as a value judgment

There are infinitely many proper scoring rules. How might a principal go about
deciding which one to use? To gain some intuition, we will take a closer look
at the quadratic and logarithmic scoring rules in the case of $n=2$ outcomes.
In Figure 1, for both of these scoring rules, we show the difference between
the expert’s reward if a given outcome happens and their reward if it does not
happen, as a function of the probability that they assign to the outcome.222We
scale down the logarithmic rule by a factor of $2\ln 2$ to make the two rules
comparable. The factor $2\ln 2$ was chosen to make the range of values taken
on by the Savage representations of the two scoring rules the same (see
Section 1.3).

Figure 1: Difference between expert’s reward if an outcome happens and if it
does not happen, as a function of the expert’s report, for the quadratic and
logarithmic scoring rules. For example, if the expert reports a 70%
probability of an outcome, then under the quadratic rule they receive a score
of $2\cdot 0.7-0.7^{2}-0.3^{2}=0.82$ if the outcome happens and $2\cdot
0.3-0.7^{2}-0.3^{2}=0.02$ if it does not: a difference of $0.8$. If rewarded
with the logarithmic rule, this difference would be $0.61$.

This difference scales linearly with the expert’s report for the quadratic
rule. Meanwhile, for the logarithmic rule, the difference changes more slowly
than for the quadratic rule for probabilities in the middle, but much more
quickly at the extremes. Informally speaking, this means that the logarithmic
rule indicates a preference (of the elicitor) for high precision close to $0$
and $1$, while the quadratic rule indicates a more even preference for
precision across $[0,1]$. Put another way, an elicitor who chooses to use the
logarithmic scoring rule renders a judgment that the probabilities $0.01$ and
$0.001$ are quite different; one who uses the quadratic rule indicates that
these probabilities are very similar.

On its surface, the _elicitation_ of forecasts has seemingly little to do with
their _aggregation_. However, given that the choice of scoring rule implies a
subjective judgment about how different probabilities compare to one another,
it makes sense to apply this judgment to the aggregation of forecasts as well.
For example, if the logarithmic scoring rule accurately reflects a principal’s
preferences, how does that value judgment inform how that principal should
aggregate multiple forecasts? This brings us to the main focus of our paper:
namely, we prove a novel correspondence between proper scoring rules and
opinion pooling methods.

###  1.3 Our definitions

Before introducing the aforementioned correspondence, we need to introduce the
_Savage representation_ of a proper scoring rule.

#### Savage representation

A proper scoring rule has a unique representation in terms of its expected
reward function $G$, i.e. the expected score on an expert who believes (and
reports) a distribution $\mathbf{p}$:

$G(\mathbf{p}):=\mathbb{E}_{j\leftarrow\mathbf{p}}\left[s(\mathbf{p};j)\right]=\sum_{j=1}^{n}p(j)s(\mathbf{p};j).$

This representation of $s$, introduced in [Sav71], is known as the _Savage
representation_ , though we will usually refer to it as the _expected reward
function_. Given that $s$ is proper, $G$ is strictly convex; and conversely,
given a strictly convex function $G$, one can re-derive $s$ with the formula

$s(\mathbf{p};j)=G(\mathbf{p})+\langle\mathbf{g}(\mathbf{p}),\delta_{j}-\mathbf{p}\rangle,$
(1)

where $\mathbf{g}$ is the gradient333Or a subgradient, if $G$ is not
differentiable. of $G$ [GR07]. Pictorially, draw the tangent plane to $G$ at
$\mathbf{p}$; then the expert’s score if outcome $j$ is realized is the height
of the plane at $\delta_{j}$.

The Savage representation of the quadratic scoring rule is
$G_{\text{quad}}(\mathbf{p})=\sum_{j=1}^{n}p(j)^{2}$. The Savage
representation of the logarithmic scoring rule is
$G_{\log}(\mathbf{p})=\sum_{j=1}^{n}p(j)\ln p(j)$.

The function $\mathbf{g}$, which will be central to our paper, describes the
difference in the expert’s score depending on which outcome happens. More
precisely, the vector $(s(\mathbf{p};j_{1}),\dots,s(\mathbf{p};j_{n}))$ is
exactly the vector $\mathbf{g}(\mathbf{p})$, except possibly for a uniform
translation in all coordinates. For example,
$s(\mathbf{p};j_{1})-s(\mathbf{p};j_{2})=g_{1}(\mathbf{p})-g_{2}(\mathbf{p})$;
this is precisely the quantity plotted in Figure 1 for the quadratic and
logarithmic scoring rules. This observation about the function $\mathbf{g}$
motivates the connection that we will establish between proper scoring rules
and opinion pooling methods.

#### Quasi-arithmetic opinion pooling

We can now define our correspondence between proper scoring rules and opinion
pooling methods. Given a proper scoring rule $s$ used for elicitation, and
given $m$ probability distributions $\mathbf{p}_{1},\dots,\mathbf{p}_{m}$ and
expert weights $w_{1},\dots,w_{m}$, the aggregate distribution
$\mathbf{p}^{*}$ that we suggest is the one satisfying

$\mathbf{g}(\mathbf{p}^{*})=\sum_{i=1}^{m}w_{i}\mathbf{g}(\mathbf{p}_{i}).$
(2)

(It is natural to ask whether this $\mathbf{p}^{*}$ exists and whether it is
unique. We will discuss this shortly.)

We refer to this pooling method as _quasi-arithmetic pooling_ with respect to
$\mathbf{g}$ (or the scoring rule $s$), or _QA pooling_ for short.444This term
comes from the notion of quasi-arithmetic means: given a continuous, strictly
increasing function $f$ and values $x_{1},\dots,x_{m}$ , the _quasi-arithmetic
mean with respect to $f$_ of these values is $f^{-1}(1/m\sum_{i}f(x_{i}))$. To
get a sense of QA pooling, let us determine what this method looks like for
the quadratic and logarithmic scoring rules.

#### QA pooling with respect to the quadratic scoring rule

We have $\mathbf{g}_{\text{quad}}(\mathbf{x})=(2x_{1},\dots,2x_{n})$, so we
are looking for the $\mathbf{p}^{*}$ such that

$(2p^{*}(1),\dots,2p^{*}(n))=\sum_{i=1}^{m}w_{i}(2p_{i}(1),\dots,2p_{i}(n)).$

This is $\mathbf{p}^{*}=\sum_{i=1}^{m}w_{i}\mathbf{p}_{i}$. Therefore, _QA
pooling for the quadratic scoring rule is precisely linear pooling_.

#### QA pooling with respect to the logarithmic scoring rule

We have $\mathbf{g}_{\log}(\mathbf{x})=(\ln x_{1}+1,\dots,\ln x_{n}+1)$, so we
are looking for the $\mathbf{p}^{*}$ such that

$(\ln p^{*}(1)+1,\dots,\ln p^{*}(n)+1)=\sum_{i=1}^{m}w_{i}(\ln
p_{i}(1)+1,\dots,\ln p_{i}(n)+1).$

By exponentiating the components on both sides, we find that
$p^{*}(j)=c\prod_{i=1}^{n}(p_{i}(j))^{w_{i}}$ for all $j$, for some
proportionality constant $c$. _This is precisely the definition of the
logarithmic pooling method._ (The constant $c$ comes from the fact that values
of $\mathbf{g}(\cdot)$ should be interpreted modulo translation by the all-
ones vector; see Remark 3.9.)

The fact that this pooling scheme maps the two most well-studied scoring rules
to the two most well-studied opinion pooling methods has not been noted
previously, to our knowledge. This correspondence suggests that — beyond just
our earlier informal justification — QA pooling with respect to a given
scoring rule may be a fundamental concept. The rest of this paper argues that
this is indeed the case.

###  1.4 Our results

#### (Section 4) Max-min optimality

Suppose that a principal asks you to issue a forecast and will pay you
according to $s$. You are not knowledgeable on the subject but know some
experts whom you trust on the matter (perhaps to varying degrees). You sub-
contract the experts, promising to pay each expert $i$ according to
$w_{i}\cdot s$. By using QA pooling according to $s$ on the experts’
forecasts, you guarantee yourself a profit; in fact, this strategy maximizes
your worst-case profit, and is the unique such report. Furthermore, this
profit is the same for all outcomes. This fact can be interpreted to mean that
you have, in a sense, pooled the forecasts “correctly”: you do not care which
outcome will come to pass, which means that you have correctly factored the
expert opinions into your forecast. We give an additional interpretation of
this optimality notion as maximizing an aggregator’s guaranteed improvement
over choosing an expert at random.

In Section 4.2, we give an additional interpretation of QA pooling as an
optimal method relative to a proper scoring rule: namely, the QA pool of
expert forecasts is the forecast with respect to which the experts would be
the least wrong (as measured via the weighted average of Bregman divergences
associated with $G$).

#### (Section 5) Learning expert weights

Opinion pooling entails assigning weights to experts. Where do these weights
come from? How might one learn them from experience?

Suppose we have a fixed proper scoring rule $s$, and further consider fixing
the reports of the $m$ experts as well as the eventual outcome. One can ask:
what does the score of the aggregate distribution (per QA pooling with respect
to $s$) look like as a function of $\mathbf{w}$, the vector of expert weights?
We prove that this function is concave. This is useful because it allows for
online convex optimization over expert weights.

######  Theorem (informal).

Let $s$ be a bounded proper scoring rule.555For which QA pooling is well
defined (we discuss this below). For time steps $t=1\dots T$, $m$ experts
report forecasts to an aggregator, who combines them into a forecast
$\mathbf{p}^{t}$ using QA pooling with respect to $s$ and suffers a loss of
$-s(\mathbf{p}^{t};j^{t})$, where $j^{t}$ is the outcome at time step $t$. If
the aggregator updates the experts’ weights using online gradient descent,
then the aggregator’s regret compared to the best weights in hindsight is
$O(\sqrt{T})$.

The aforementioned concavity property is a nontrivial fact that demonstrates
an advantage of QA pooling over e.g. linear and logarithmic pooling: these
pooling methods satisfy the concavity property for some proper scoring rules
$s$ but not others.

#### (Section 6) Natural axiomatization for QA pooling methods

[Kol30] and [Nag30] independently came up with a simple axiomatization of
quasi-arithmetic means. We show how to change these axioms to allow for
weighted means; the resulting axiomatization is a natural characterization of
all quasi-arithmetic pooling methods in the case of $n=2$ outcomes.
Furthermore, although quasi-arithmetic means are typically defined for scalar-
valued functions, we demonstrate that these axioms can be extended to describe
quasi-arithmetic means with respect to vector-valued functions, as is
necessary for our purposes if $n>2$. This extension is nontrivial but natural,
and to our knowledge has not previously been described.

#### The reverse direction: pooling to scoring

Although we have mostly discussed the correspondence between proper scoring
rules and pooling methods in the context of “given a scoring rule, what is the
most natural pooling method,” the correspondence holds in reverse. That is, if
a principal has a pooling method in mind, they can choose the scoring rule
with which to reward the aggregator according to this correspondence. In
Section 7 we give an interpretation of this reverse connection in the context
of collusion between experts.

#### When is QA pooling well defined?

Is there always a $\mathbf{p}^{*}$ satisfying Equation 2, and is it guaranteed
to be unique? In Section 3 we show that the answer to the uniqueness question
is yes, and that in the $n=2$ outcomes case, the answer to the existence
question is also yes. For larger values of $n$, such a $\mathbf{p}^{*}$ may
not exist. In particular, QA pooling with respect to $s$ is well defined if
and only if the range of $\mathbf{g}(\cdot)$ is a convex set. For reasons that
we will discuss, we call this the _convex exposure property_. In Appendix F,
we will discuss when this property holds. In particular, we will show that the
it holds for the quadratic, logarithmic, and spherical scoring rules, as well
as other natural classes of proper scoring rules.

Even in the case that the convex exposure property does not hold, the
definition of QA pooling naturally extends using a property that we prove in
Section 4.2, namely that QA pooling minimizes the weighted average of the
Bregman divergences between the aggregate forecast and the experts’ forecasts.
The forecast defined via this extension is guaranteed to be unique.

##  2 Related work

#### Opinion pooling

[CW07] categorize mathematical approaches to opinion pooling as either
_Bayesian_ or _axiomatic_. A Bayesian approach to this problem is one that
entails Bayes updating on each expert’s opinion. While quite natural, Bayesian
opinion pooling is difficult to apply and, in full generality, computationally
intractable. This is because the Bayes updates must fully account for
interdependencies between expert opinions.

By contrast, axiomatic approaches do not make assumptions about the structure
of information underlying the experts’ opinions; instead, they aim to come up
with pooling methods that satisfy certain axioms or desirable properties. Such
axioms include unanimity preservation, eventwise independence, and external
Bayesianality; see e.g. [DL14] for statements of these axioms. For $n>2$
outcomes, linear pooling is the only method that is both unanimity preserving
and eventwise independent [AW80]; however, it is not externally Bayesian. On
the other hand, logarithmic pooling is both unanimity preserving and
externally Bayesian [Gen84]. It is not the only such method, but it is
arguably the most natural. Our approach has an axiomatic flavor (though it
differs substantially from previous axiomatic approaches).

The earliest comprehensive treatment of opinion pooling in a mathematical
setting was by Genest and Zidek [GZ86]. See [CW07] and [DL14] for more recent
high-level overviews of the subject, including discussion of both Bayesian and
axiomatic approaches, as well as discussion of the axioms we discussed in
Section 1.1. Recent work on Bayesian opinion pooling includes [CL13],
[Sat+14], [FCK15], and [ABS18]. Finally, see [FES20] for work on learning
weights for linear pooling under a quadratic loss function; this is an
instance of our more general theory in Section 5. For a more detailed
discussion of axiomatic approaches, see [ACR12]. See also [Abb09] for a non-
Bayesian approach that is closely related to ours; our Proposition 4.3
generalizes [Abb09, Proposition 4].

Part of our work provides an alternative interpretation of prior work on
pooling via minimizing Bregman divergence, see e.g. [Ada14] and [Pet19]. (We
define Bregman divergence in Section 3.) Concretely, [Pet19, §4] defines a
notion of pooling analogous to our Definition 4.5, though in a different
context. The main focus of their line of work is on connecting opinion pooling
to Bregman divergence; our approach connects opinion pooling to proper scoring
rules, and a connection to Bregman divergence falls naturally out of this
pursuit.

#### Scoring rules

The literature on scoring rules is quite large; we recommend [GR07] for a
thorough but technical overview, or [Car16] for a less technical overview that
focuses more on applications (while still introducing the basic theory).
Seminal work on the theory behind scoring rules includes Brier’s paper
introducing the quadratic rule [Bri50], Good’s paper introducing the
logarithmic rule [Goo52], and Savage’s work on the general theory of proper
scoring rules [Sav71]. Additionally, see [DM14] for an overview of various
families of proper scoring rules.

#### Aggregation via prediction markets

One common way to aggregate probabilistic forecasts is through prediction
markets, some of which are based on scoring rules. [Han03] introduced _market
scoring rules_ (MSRs), in which experts are sequentially presented with an
opportunity to update an aggregate forecast and are rewarded (or penalized) by
the amount that their update changed the aggregate prediction’s eventual
score. [CP07] introduced _cost-function markets_ , in which a market maker
sells $n$ types of shares — one for each outcome — where the price of a share
depends on the number of shares sold thus far according to some cost function.
They established a connection between cost-function markets and MSRs, where a
market with a given cost function will behave the same way as a certain MSR
(under certain conditions later formalized in [ACV13]). Subsequent work
explored this area further, tying cost-function market making to online
learning of probability distributions [CV10] [ACV13]. This work differs from
ours in that the goal of their online learning problem is to learn a
_probability distribution over outcomes_ , whereas our goal in Section 5 is to
learn _expert weights_.

While MSRs and cost function markets have superficial similarities to our
work, they have quite different goals and properties. For both MSRs and cost
function markets, incentives are set up so that an expert brings the market
into alignment with their own opinion, rather than an aggregate. Thus, in the
well-studied setting of experts whose beliefs do not depend on other experts’
actions, the final state of such a market reflects only the beliefs of the
most recent trader, rather than an aggregate of the experts’ beliefs.

#### Arbitrage from collusion

Part of our work can be viewed as a generalization of previous work by Chun
and Shachter done in a different context: namely, preventing colluding experts
from exploiting arbitrage opportunities [CS11]. The authors show that for the
case of $n=2$ outcomes, if experts are rewarded with the same scoring rule
$s$, preventing this is impossible: the experts can successfully collude by
all reporting what we are calling the QA pool of their reports with respect to
$s$. Our Theorem 4.1 recovers this result as a special case. See [Che+14] for
related work in the context of wagering mechanisms and [Fre+20] for follow-up
work on preventing arbitrage from colluding experts.

#### Prediction with expert advice

In Section 5 we discuss learning expert weights online. The online learning
literature is vast, but our approach fits into the framework of _prediction
with expert advice_. In this setting, at each time step each expert submits a
report (in our context a probability distribution). The agent then submits a
report based on the experts’ submissions, and suffers a loss depending on this
report and the eventual outcome. See [CBL06] for a detailed account of this
setting; the authors prove a variety of no-regret bounds in this setting,
ranging (depending form the setting) from $O(\sqrt{T})$ to $O(1)$. Our setting
is an ambitious one: while typically one desires low regret compared to the
best expert in hindsight, we desire low regret compared to the best _mixture_
of experts in hindsight. In [CBL06, §3.3], the authors discuss this more
ambitious goal, proving (in our terminology) that for a certain class of
losses — bounded exp-concave losses — it is possible to achieve $O(\log T)$
regret in comparison with the best linear pool of experts in hindsight. This
setting is different from ours in two important ways: first, the losses that
we consider are not in general exp-concave (e.g. the quadratic loss); and
second, the authors consider linear pooling for any loss, whereas we consider
QA pooling with respect to the loss function.

#### Quasi-arithmetic means

Our notion of quasi-arithmetic pooling is an adaptation (and extension to
higher dimensions) of the existing notion of quasi-arithmetic means. These
were originally defined and axiomatized independently in [Kol30] and [Nag30].
Aczél generalized this work to include weighted quasi-arithmetic means
[Acz48], though these means have weights baked in rather than taking them as
inputs, which is different from our setting. See [Gra+11, §3.1] for an
overview of this topic.

##  3 Preliminaries

Throughout this paper, we will let $m$ be the number of experts and use the
index $i$ to refer to any particular expert. We will let $n$ be the number of
outcomes and use the index $j$ to refer to any particular outcome.

Let $\Delta^{n}$ be the standard simplex in $\mathbb{R}^{n}$, i.e. the one
with vertices $\delta_{1},\dots,\delta_{n}$. (Here, $\delta_{j}$ denotes the
vector with a $1$ in the $j$-th coordinate and zeros elsewhere.) Note that
$\Delta^{n}$ is an $(n-1)$-dimensional manifold with boundary. Define an _$n$
-outcome forecast domain_ to be any convex $(n-1)$-dimensional666Formally, we
require that the forecast domain contain a subset that is homeomorphic to
$\mathbb{R}^{n-1}$. subset of $\Delta^{n}$. We will use $\mathcal{D}$ to
denote an arbitrary forecast domain.

Although our results will apply to any forecast domain, the two forecast
domains that we expect to be by far the most useful are $\Delta^{n}$ and
$\Delta^{n}$ minus its boundary: the former for bounded scoring rules (such as
the quadratic rule) and the latter for unbounded ones (such as the logarithmic
rule).

###  3.1 Proper scoring rules

Given an $n$-outcome forecast domain $\mathcal{D}$, a _proper scoring rule_ on
$\mathcal{D}$ is a function $s:\mathcal{D}\times[n]\to\mathbb{R}$ such that
for all $\mathbf{p}\in\mathcal{D}$, we have

$\mathbb{E}_{j\leftarrow\mathbf{p}}\left[s(\mathbf{p};j)\right]\geq\mathbb{E}_{j\leftarrow\mathbf{p}}\left[s(\mathbf{x};j)\right]$

for all $\mathbf{x}\in\mathcal{D}$, with equality only when
$\mathbf{x}=\mathbf{p}$. (Here, $j\leftarrow\mathbf{p}$ means that $j$ is
drawn randomly from the probability distribution $\mathbf{p}$.)777Some authors
refer to such scoring rules as _strictly_ proper while others assume that
propriety entails strictness; we choose the latter convection. Also, while
many sources define the range of $s$ to include $\pm\infty$ so as to e.g. make
the logarithmic scoring rule well defined on the boundary of $\Delta^{n}$, we
do not do this. This is an application-specific choice: we will be interested
in pooling forecasts, and pooling e.g. $(1,0)$ and $(0,1)$ under the
logarithmic scoring rule would be tantamount to adding positive and negative
infinity.

Given a proper scoring rule $s$, we define its _expected reward function_ (or
_Savage representation_) $G:\mathcal{D}\to\mathbb{R}$ by

$G(\mathbf{p}):=\mathbb{E}_{j\leftarrow\mathbf{p}}\left[s(\mathbf{p};j)\right]=\sum_{j=1}^{n}p(j)s(\mathbf{p};j).$

We will henceforth assume that $s$ is continuous; to our knowledge, this is
the case for all frequently-used proper scoring rules.888This choice is
partially for ease of exposition, though e.g. our axiomatization in Section 6
depends on this. This is equivalent to assuming that $G$ is differentiable, or
that $G$ is continuously differentiable (see Proposition C.1 for a proof of
this equivalence).

######  Proposition 3.1 ([GR07]).

Given a proper scoring rule $s$, its expected reward function $G$ is strictly
convex, and $s$ can be re-derived from $G$ via the formula

$s(\mathbf{p};j)=G(\mathbf{p})+\langle\mathbf{g}(\mathbf{p}),\delta_{j}-\mathbf{p}\rangle,$
(3)

where $\mathbf{g}=\nabla G$ (i.e. the gradient of $G$). Conversely, given a
differentiable, strictly convex function $G$, the function $s$ defined by
Equation 3 is a proper scoring rule.999The statement in [GR07] is slightly
more complicated because they consider scoring rules with $\pm\infty$ in their
range, as well as discontinuous scoring rules.

The important intuition to keep in mind for Equation 3 is that the score of an
expert who reports $\mathbf{p}$ is determined by drawing the tangent plane to
$G$ at $\mathbf{p}$; the value of this plane at $\delta_{j}$, where $j$ is the
outcome that happens, is the expert’s score.

We refer to $\mathbf{g}$ as the _exposure function_ of $s$. We borrow this
term from finance, where _exposure_ refers to how much an agent stands to gain
or lose from various possible outcomes — informally speaking, how much the
agent cares about which outcome will happen. If we view
$G(\mathbf{p})-\langle\mathbf{g}(\mathbf{p}),\mathbf{p}\rangle$ as the agent’s
“baseline profit,” then the $j$-th component of $\mathbf{g}(\mathbf{p})$ is
the amount that the agent stands to gain (or lose) on top of the baseline
profit if outcome $j$ happens.

We give a geometric intuition for Proposition 3.1 to help explain why the
_properness_ of $s$ corresponds to the _convexity_ of $G$; this intuition will
be helpful for understanding Bregman divergence (below) and the proofs in
Section 4.

Consider Figure 2, which depicts some $G$ in the $n=2$ outcome case, with the
$x$-axis corresponding to the probability of Outcome 1 (see Remark 3.5 for
formal details). Suppose that the expert _believes_ that the probability
Outcome 1 is $0.7$. If the expert _reports_ $p=0.7$, then their reward in the
cases of Outcome 1 and Outcome 2 are the $y$-values of the rightmost and
leftmost points on the red line, respectively. Thus, in expectation, the
expert’s reward is the $y$-value of the red point. If instead the expert lies
and reports $p=0.4$ (say), then the $y$-values of the rightmost and leftmost
points on the blue line represent the expert’s rewards in the cases of Outcome
1 and Outcome 2, respectively. In this case, since Outcome 1 is still 70%
likely, the expert’s expected reward is the $y$-value of the blue point.
_Because $G$ is strictly convex_, the blue point is strictly below the red
point; that is, the expert is strictly better off reporting $p=0.7$. This
argument holds in full generality: for any strictly convex function $G$ in any
number of dimensions.

Figure 2: A convex function $G$, with tangent lines drawn at $x=0.7$ and
$x=0.4$. If an expert believes that the probability of an event is $0.7$,
their expected score if they report $p=0.7$ is the $y$-value of the red point;
if they instead report $p=0.4$, their expected score is the $y$-value of the
blue point. Because $G$ is convex, the red point is guaranteed to be above the
blue point, so the expert is incentivized to be truthful.

We will find _Bregman divergence_ to be a useful concept for some of our
proofs.

######  Definition 3.2 (Bregman divergence).

Given a differentiable, strictly convex function $G:\mathcal{D}\to\mathbb{R}$
with gradient $\mathbf{g}$, where $\mathcal{D}$ is a convex subset of
$\mathbb{R}^{n}$, and given $\mathbf{p},\mathbf{q}\in\mathcal{D}$, the
_Bregman divergence_ between $\mathbf{p}$ and $\mathbf{q}$ _with respect to
$G$_ is

$D_{G}(\mathbf{p}\parallel\mathbf{q}):=G(\mathbf{p})-G(\mathbf{q})-\langle\mathbf{g}(\mathbf{q}),\mathbf{p}-\mathbf{q}\rangle.$

(Note that Bregman divergence is not symmetric.) A geometric interpretation of
$D_{G}(\mathbf{p}\parallel\mathbf{q})$ is: if you draw the tangent plane to
$G$ at $\mathbf{q}$, how far below $G(\mathbf{p})$ the value of that plane
will be at $\mathbf{p}$. For example, the distance between the red and blue
points in Figure 2 is the Bregman divergence between the expert’s belief
$(0.7,0.3)$ and their report $(0.4,0.6)$. This gives the following
interpretation of Bregman divergence in the context of proper scoring rules:

######  Remark 3.3.

If $G$ is the expected reward function of a proper scoring rule, then
$D_{G}(\mathbf{p}\parallel\mathbf{q})$ is the expected reward lost by
reporting $\mathbf{q}$ when your belief is $\mathbf{p}$. Put otherwise,
$D_{G}(\mathbf{p}\parallel\mathbf{q})$ measures the “wrongness” of the report
$\mathbf{q}$ relative to a correct answer of $\mathbf{p}$.

######  Proposition 3.4 (Well-known facts about Bregman divergence).

  * •

$D_{G}(\mathbf{p}\parallel\mathbf{q})\geq 0$, with equality only when
$\mathbf{p}=\mathbf{q}$.

  * •

For any $\mathbf{q}$, $D_{G}(\mathbf{x}\parallel\mathbf{q})$ is a strictly
convex function of $\mathbf{x}$.

Finally, we make a note about interpreting the $n=2$ outcome case in one
dimension.

######  Remark 3.5.

Because $\Delta^{n}$ is $(n-1)$-dimensional, we can think of the case of $n=2$
outcomes in one dimension. All probabilities in $\mathcal{D}$ are of the form
$(p,1-p)$; we map $\mathcal{D}$ to $[0,1]$ (or a subset) via the first
coordinate. Thus, we let $G(p):=G(p,1-p)$. We let
$g(p):=G^{\prime}(p)=\langle\mathbf{g}(p,1-p),(1,-1)\rangle$. The tangent line
to $G$ at $p$, e.g. as in Figure 2, will intersect the line $x=1$ at $s(p;1)$
(i.e. the score if Outcome 1 happens) and intersect the line $x=0$ at $s(p;2)$
(i.e. the score if Outcome 2 happens). This formulation will be helpful when
discussing the two-outcome case, e.g. in Section 6.

###  3.2 Probabilistic opinion pooling

We now introduce the central concept of this paper: quasi-arithmetic pooling.
Let $s$ be a proper scoring rule over a forecast domain $\mathcal{D}$, and let
$G$ and $\mathbf{g}$ be as previously defined. We denote the quasi-arithmetic
(QA) pooling operator with respect to $\mathbf{g}$ with $\oplus_{\mathbf{g}}$.
This operator takes as input (probability, weight) pairs with weights non-
negative and adding to $1$, and outputs a probability (all probabilities are
in $\mathcal{D}$). In particular, $\oplus_{\mathbf{g}}$ is defined by
$\sideset{}{{}_{\mathbf{g}}}{\bigoplus}\limits_{i=1}^{m}(\mathbf{p}_{i},w_{i}):=\mathbf{p}^{*}$,
where

$\mathbf{g}(\mathbf{p}^{*})=\sum_{i=1}^{m}w_{i}\mathbf{g}(\mathbf{p}_{i}).$
(4)

Is $\mathbf{p}^{*}$ in Equation 4 well defined? That is, does it exist, and if
so, is it unique? It is indeed the case that $\mathbf{p}^{*}$, if defined, is
unique. This is because $\mathbf{g}$ cannot take on the same value at two
different points, as it is the gradient of a convex function. As for
existence: $\mathbf{p}^{*}$ is guaranteed to exist if and only if $s$
satisfies the following property.

######  Definition 3.6 (convex exposure).

A proper scoring rule $s$ has _convex exposure_ if the range of its exposure
function $\mathbf{g}$ is a convex set.

######  Proposition 3.7.

  1. (a)

Given a proper scoring rule $s$, $\mathbf{p}^{*}$ as defined above is
guaranteed to exist for any $\mathbf{p}_{1},\dots,\mathbf{p}_{m}$ if and only
if $s$ has convex exposure.

  2. (b)

In the case of $n=2$, every (continuous) proper scoring rule has convex
exposure (so $\mathbf{p}^{*}$ exists).

###### Proof.

(a) is clear: the right-hand side of Equation 4 is an arbitrary convex
combination of values in the range of $\mathbf{g}$. As for (b), since
$\mathcal{D}$ is connected and $\mathbf{g}$ is continuous (see Proposition
C.1), the range of $\mathbf{g}$ is connected. In the $n=2$ outcome case, the
range of $\mathbf{g}$ lies on the line $\\{(x_{1},x_{2}):x_{1}+x_{2}=0\\}$,
and a connected subset of a line is convex. ∎

We now formally state the definition of quasi-arithmetic pooling:

######  Definition 3.8 (quasi-arithmetic pooling).

Let $s$ be a proper scoring rule with convex exposure on a forecast domain
$\mathcal{D}$. Given forecasts
$\mathbf{p}_{1},\dots,\mathbf{p}_{m}\in\mathcal{D}$ with non-negative weights
$w_{1},\dots,w_{m}$ adding to $1$, the _quasi-arithmetic (QA) pool_ of these
forecasts _with respect to $s$_ (or with respect to its exposure function
$\mathbf{g}$), denoted by
$\sideset{}{{}_{\mathbf{g}}}{\bigoplus}\limits_{i=1}^{m}(\mathbf{p}_{i},w_{i})$,
is the unique $\mathbf{p}^{*}\in\mathcal{D}$ such that

$\mathbf{g}(\mathbf{p}^{*})=\sum_{i=1}^{m}w_{i}\mathbf{g}(\mathbf{p}_{i}).$

If the forecasts and weights are clear from context, we may simply write
$\mathbf{p}^{*}$ to refer to their quasi-arithmetic pool; or, if only the
forecasts are clear, we may write $\mathbf{p}^{*}_{\mathbf{w}}$, where
$\mathbf{w}$ is the vector of weights.

In Section 4.2 we will discuss a natural generalization of QA pooling to
proper scoring rules that do not have convex exposure. In Appendix F, we will
explore which commonly used proper scoring rules the convex exposure property
holds for. (The upshot is: most of them.)

We conclude with a technical note about the range of $\mathbf{g}$ and the
correct interpretation of Equation 4.

######  Remark 3.9.

Because the domain of $G$ is a subset of $\Delta^{n}$ (and thus lies in a
plane that is orthogonal to the all-ones vector $\mathbf{1}_{n}$), its
gradient function $\mathbf{g}$ only takes on values orthogonal to
$\mathbf{1}_{n}$. When we treat $G$ as a function of $n$ variables rather than
$n-1$ variables out of convenience, thus artificially extending the domain of
$G$ outside of the plane containing $\Delta^{n}$, $\mathbf{g}$ might gain a
component parallel to $\mathbf{1}_{n}$. However, the correct way to think of
the codomain of $\mathbf{g}$ is either as $\\{\mathbf{x}:\sum_{i}x_{i}=0\\}$,
or else as $\mathbb{R}^{n}$ modulo translation by $\mathbf{1}_{n}$, which we
will denote by $\mathbb{R}^{n}/T(\mathbf{1}_{n})$. Consequently, the correct
way to think of the equality in Equation 4 is as an equality in
$\mathbb{R}^{n}/T(\mathbf{1}_{n})$, rather than an equality in
$\mathbb{R}^{n}$; this came up in Section 1.3 when we discussed the fact that
QA pooling for the logarithmic scoring rule is logarithmic opinion
pooling.101010Formally, consider the change of coordinates given by
$z_{j}=x_{n}-x_{j}$ for $j\leq n-1$ and $z_{n}=\sum_{j}x_{j}$, so that the
domain of $G$ lies in the plane $z_{n}=1$. Then for $j\leq n-1$,
$\frac{\partial G}{\partial z_{j}}$ at a given point in the domain of $G$ does
not change if $1$ is substituted for $z_{n}$; only $\frac{\partial G}{\partial
z_{n}}$ changes (to zero). Equivalently in terms of our original coordinates,
the change that $\mathbf{g}$ undergoes when we consider $G$ to be a function
only defined on $\mathcal{D}$ instead of $\mathbb{R}^{n}$ is precisely a
projection of $\mathbf{g}$ onto $H_{n}(0)$.

##  4 Optimality results

Our goal is to give a formal justification for quasi-arithmetic pooling. This
section gives two such justifications: one in terms of max-min optimality and
one (as a corollary) in terms of minimizing the weighted average of the
Bregman divergences between the pooled forecast and the experts’ forecasts. We
will give additional justifications in later sections.

###  4.1 QA pooling as a max-min optimal method

######  Theorem 4.1.

Let $s$ be a proper scoring rule with convex exposure on a forecast domain
$\mathcal{D}$. Fix any forecasts
$\mathbf{p}_{1},\dots,\mathbf{p}_{m}\in\mathcal{D}$ with non-negative weights
$w_{1},\dots,w_{m}$ adding to $1$. Define

$u(\mathbf{p};j):=s(\mathbf{p};j)-\sum_{i=1}^{m}w_{i}s(\mathbf{p}_{i};j).$

Then the quantity $\min_{j}u(\mathbf{p};j)$ is uniquely maximized by setting
$\mathbf{p}$ to
$\mathbf{p}^{*}:=\sideset{}{{}_{\mathbf{g}}}{\bigoplus}\limits_{i=1}^{m}(\mathbf{p}_{i},w_{i})$.
Furthermore, $u(\mathbf{p}^{*};j)$ is the same for all $j$. This quantity is
non-negative, and is positive unless all reports $\mathbf{p}_{i}$ with
positive weights are equal.

One interpretation for this theorem statement is as follows. Consider an agent
who is tasked with submitting a forecast, and who will be paid according to
$s$. The agent decides to sub-contract $m$ experts to get their opinions,
paying expert $i$ the amount $w_{i}s(\mathbf{p}_{i};j)$ if the expert reports
$\mathbf{p}_{i}$ and outcome $j$ happens. (Perhaps experts whom the agent
trusts more have higher $w_{i}$’s.) Finally, the agent reports some (any)
forecast $\mathbf{p}$. Then $u(\mathbf{p};j)$ is precisely the agent’s profit
(utility).

The quantity $\min_{j}u(\mathbf{p};j)$ is the agent’s minimum possible profit
over all outcomes. It is natural to ask which report $\mathbf{p}$ maximizes
this quantity. Theorem 4.1 states that this maximum is achieved by the QA pool
of the experts’ forecasts with respect to $s$, and that this is the unique
maximizer.

A possible geometric intuition to keep in mind for the proof (below): for each
expert $i$, draw the plane tangent to $G$ at $\mathbf{p}_{i}$. For any $j$,
the value of this plane at $\delta_{j}$ is $s(\mathbf{p}_{i};j)$. Now take the
weighted average of all $m$ planes; this is a new plane whose intersection
with any $\delta_{j}$ is the total reward received by the experts if $j$
happens. Since $G$ is convex, this plane lies below $G$. To figure out which
point maximizes the agent’s guaranteed profit, push the plane upward until it
hits $G$. It will hit $G$ at $\mathbf{p}^{*}$ and the agent’s profit will be
the vertical distance that the plane was pushed.

###### Proof of Theorem 4.1.

We first show that $u(\mathbf{p}^{*};j)$ is the same for all $j$. We have

$\displaystyle u(\mathbf{p}^{*};j)$
$\displaystyle=s(\mathbf{p}^{*};j)-\sum_{i}w_{i}s(\mathbf{p}_{i};j)$
$\displaystyle=G(\mathbf{p}^{*})+\langle\mathbf{g}(\mathbf{p}^{*}),\delta_{j}-\mathbf{p}^{*}\rangle-\sum_{i}w_{i}(G(\mathbf{p}_{i})+\langle\mathbf{g}(\mathbf{p}_{i}),\delta_{j}-\mathbf{p}_{i}\rangle)$
$\displaystyle=G(\mathbf{p}^{*})-\sum_{i}w_{i}G(\mathbf{p}_{i})+\langle\sum_{i}w_{i}\mathbf{g}(\mathbf{p}_{i}),\delta_{j}-\mathbf{p}^{*}\rangle-\sum_{i}w_{i}\langle\mathbf{g}(\mathbf{p}_{i}),\delta_{j}-\mathbf{p}_{i}\rangle$
$\displaystyle=G(\mathbf{p}^{*})-\sum_{i}w_{i}G(\mathbf{p}_{i})+\sum_{i}w_{i}\langle\mathbf{g}(\mathbf{p}_{i}),\mathbf{p}_{i}-\mathbf{p}^{*}\rangle,$

which indeed does not depend on $j$. We can in fact rewrite this expression as
a weighted sum of Bregman divergences:

$u(\mathbf{p};j)=\sum_{i}w_{i}D_{G}(\mathbf{p}^{*}\parallel\mathbf{p}_{i}).$

It follows that $u(\mathbf{p};j)$ is non-negative (see Proposition 3.4), and
positive except when all $\mathbf{p}_{i}$’s with positive weights are equal.

Finally we show that $\mathbf{p}=\mathbf{p}^{*}$ maximizes
$\min_{j}u(\mathbf{p};j)$. Suppose that for some report $\mathbf{q}$ we have
that $\min_{j}u(\mathbf{q};j)\geq\min_{j}u(\mathbf{p}^{*};j)$. Then
$u(\mathbf{q};j)\geq u(\mathbf{p}^{*};j)$ for _every_ $j$, since
$u(\mathbf{p}^{*};j)$ is the same for every $j$. But this means that
$s(\mathbf{q};j)\geq s(\mathbf{p}^{*};j)$ for every $j$, since the
$\sum_{i}w_{i}s(\mathbf{p}_{i};j)$ term in the definition of $u(\mathbf{p};j)$
does not depend on $\mathbf{p}$. But then $\mathbf{q}=\mathbf{p}^{*}$, since
$s$ is proper. ∎

######  Remark 4.2.

We can reformulate Theorem 4.1 as follows: suppose that an agent has access to
forecasts $\mathbf{p}_{1},\dots,\mathbf{p}_{m}$ and needs to issue a forecast,
for which the agent will be rewarded using a proper scoring rule $s$ with
convex exposure. The agent can improve upon selecting an expert at random
according to weights $w_{1},\dots,w_{m}$, no matter the outcome $j$, by
reporting $\mathbf{p}^{*}$. This improvement is the same no matter the
outcome, and is a strict improvement unless all forecasts with positive
weights are the same.

###  4.2 QA pooling as a Bregman divergence minimizing method

The quantity $\sum_{i}w_{i}D_{G}(\mathbf{p}\parallel\mathbf{p}_{i})$ that came
up in the proof of Theorem 4.1 is a natural quantity to consider, as it is a
measure of how far away overall $\mathbf{p}$ is from the expert reports
$\mathbf{p}_{i}$. In fact, it is a natural quantity to _minimize_ if we care
about aggregation. This brings us to our second formal justification of QA
pooling.

######  Proposition 4.3.

Given a proper scoring rule $s$ with convex exposure and reports
$\mathbf{p}_{1},\dots,\mathbf{p}_{m}$ with weights $w_{1},\dots,w_{m}$, the
quantity

$d(\mathbf{x}):=\sum_{i}w_{i}D_{G}(\mathbf{x}\parallel\mathbf{p}_{i})$

is uniquely minimized at $\mathbf{x}=\mathbf{p}^{*}$.

This fact makes sense in light of the geometric intuition we described for
Theorem 4.1. In these terms, Preposition 4.3 states that $\mathbf{p}^{*}$ is
the point at which $G$ is closest (in vertical distance) to the average of the
experts’ planes. Formally:

###### Proof.

Since Bregman divergence is strictly convex in its first argument (see
Proposition 3.4), $d(\mathbf{x})$ is strictly convex. This means that if there
is a point $\mathbf{p}\in\mathcal{D}$ where $\nabla d(\mathbf{p})=\mathbf{0}$,
then $\mathbf{p}$ is the unique minimizer of $d$. Now, we have

$\nabla
d(\mathbf{x})=\sum_{i}w_{i}\nabla_{\mathbf{x}}D_{G}(\mathbf{x}\parallel\mathbf{p}_{i})=\sum_{i}w_{i}(\mathbf{g}(\mathbf{x})-\mathbf{g}(\mathbf{p}_{i}))=\mathbf{g}(\mathbf{x})-\sum_{i}w_{i}\mathbf{g}(\mathbf{p}_{i}).$

But in fact,
$\mathbf{g}(\mathbf{p}^{*})=\sum_{i}w_{i}\mathbf{g}(\mathbf{p}_{i})$ by
definition, so $\nabla d(\mathbf{p}^{*})=\mathbf{0}$. This completes the
proof. ∎

Proposition 4.3 generalizes [Abb09, Proposition 4], which showed that
logarithmic pooling minimizes the weighted average of the KL divergences
between the pooled forecast and the expert forecasts. KL divergence is Bregman
divergence relative to negative entropy, which is precisely the function $G$
for the logarithmic scoring rule.

In light of the fact that $D_{G}(\mathbf{p}\parallel\mathbf{q})$ is the
expected reward lost by an expert who, believing $\mathbf{p}$, reports
$\mathbf{q}$ (see Remark 3.3), Proposition 4.3 gives another natural
interpretation of QA pooling.

######  Remark 4.4.

Consider a proper scoring rule $s$ with convex exposure and forecasts
$\mathbf{p}_{1},\dots,\mathbf{p}_{m}$ with weights $w_{1},\dots,w_{m}$. The QA
pool of these forecasts is the forecast $\mathbf{p}^{*}$ that, if it is the
correct answer (i.e. if the outcome is drawn according to $\mathbf{p}^{*}$),
would minimize the expected loss of a randomly chosen (according to
$\mathbf{w}$) expert relative to reporting $\mathbf{p}^{*}$.

In this sense, QA pooling reflects a _compromise_ between experts: it is the
probability that, if it were correct, would make the experts’ forecast least
wrong overall. This motivates a generalization of QA pooling to non-convex
exposure scoring rules that uses the formulation of $\mathbf{p}^{*}$ in terms
of minimizing Bregman divergence.

######  Definition 4.5 (generalized quasi-arithmetic pooling).

Let $s$ be a proper scoring rule on a closed forecast domain $\mathcal{D}$.
Given forecasts $\mathbf{p}_{1},\dots,\mathbf{p}_{m}\in\mathcal{D}$ with non-
negative weights $w_{1},\dots,w_{m}$ adding to $1$, the _generalized quasi-
arithmetic pool_ of these forecasts _with respect to $s$_ (or $\mathbf{g}$) is
the unique $\mathbf{p}^{*}\in\mathcal{D}$ minimizing

$d(\mathbf{x}):=\sum_{i}w_{i}D_{G}(\mathbf{x}\parallel\mathbf{p}_{i}).$

To check that Definition 4.5 is well defined, we need to check that
$\mathbf{p}^{*}$ exists and is unique. Existence follows from the fact that
$\mathcal{D}$ is (by assumption) closed and therefore compact, and that a
continuous function on a compact domain achieves its minimum. Uniqueness
follows from the fact that a strictly convex function on a convex set has at
most one minimum. (As part of proving Proposition 4.3, we showed that $d$ is
strictly convex.)

######  Remark 4.6.

Since $d$ is convex, minimizing it is a matter of convex optimization. In
particular, if $G$ is bounded,111111Note that this is a weaker condition than
$s$ being bounded; e.g. $G_{\log}$ is the negative entropy function, which is
bounded. given oracle access to $\mathbf{g}$, the ellipsoid method can be used
to efficiently find the generalized QA pool of a list of forecasts.

######  Remark 4.7.

It is natural to ask which pooling method minimizes the Bregman divergence
going the other way, i.e.
$\sum_{i}w_{i}D_{G}(\mathbf{p}_{i}\parallel\mathbf{x})$. The answer is linear
pooling [Ban+05, Proposition 1]. This makes sense, because this minimization
question asks: if an expert is selected at random to be “correct” according to
the weights, what is the overall probability of any event $j$? The answer is
achieved by linear pooling.

##  5 Convex losses and learning expert weights

Thus far when discussing QA pooling, we have regarded expert weights as given.
Where do these weights come from? As we will show in this section, these
weights can be learned from experience. The key observation is the following
theorem, which states that an agent’s score is a concave function of the
weights it uses for the experts.

######  Theorem 5.1.

Let $s$ be a proper scoring rule with convex exposure on a forecast domain
$\mathcal{D}$, and fix any
$\mathbf{p}_{1},\dots,\mathbf{p}_{m}\in\mathcal{D}$. Given a weight vector
$\mathbf{w}=(w_{1},\dots,w_{m})\in\Delta^{m}$, define the _weight-score_ of
$\mathbf{w}$ for an outcome $j$ as

$\text{WS}_{j}(\mathbf{w}):=s\left(\sideset{}{{}_{\mathbf{g}}}{\bigoplus}_{i=1}^{m}(\mathbf{p}_{i},w_{i});j\right).$

Then for every $j\in[n]$, $\text{WS}_{j}(\mathbf{w})$ a concave function of
$\mathbf{w}$.

We defer the proof to Appendix D, but the basic idea is that for any two
weight vectors $\mathbf{v}$ and $\mathbf{w}$ and any $c\in[0,1]$, the quantity

$\text{WS}_{j}(c\mathbf{v}+(1-c)\mathbf{w})-c\text{WS}_{j}(\mathbf{v})-(1-c)\text{WS}_{j}(\mathbf{w})$

can be expressed as a sum of Bregman divergences, and is therefore non-
negative.

######  Remark 5.2.

Theorem 5.1 can be stated in more generality: $s$ need not have convex
exposure; it suffices to have that for the particular
$\mathbf{p}_{1},\dots,\mathbf{p}_{m}$, the QA pool of these forecasts exists
for every weight vector.

######  Remark 5.3.

Theorem 5.1 would _not_ hold if the pooling operator in the definition of WS
were replaced by linear pooling or by logarithmic pooling.121212For a
counterexample to logarithmic pooling, consider $n=2$, let $s$ be the
quadratic scoring rule, $p_{1}=(0.1,0.9)$, $p_{2}=(0.5,0.5)$, $j=1$,
$\mathbf{v}=(1,0)$, $\mathbf{w}=(0,1)$, and $c=\frac{1}{2}$. For a
counterexample to linear pooling, consider $n=2$, let $s$ be given by
$G(p_{1},p_{2})=\sqrt{p_{1}^{2}+p_{2}^{2}}$ (this is known as the _spherical_
scoring rule), $p_{1}=(0,1)$, $p_{2}=(0.2,0.8)$, $j=1$, $\mathbf{v}=(1,0)$,
$\mathbf{w}=(0,1)$, and $c=\frac{1}{2}$. This is an advantage of QA pooling
over using the linear or logarithmic method irrespective of the scoring rule.

Beyond Theorem 5.1’s instrumental use for no-regret online learning of expert
weights (Theorem 5.5 below), the result is interesting in its own right. For
example, the following fact — loosely speaking, that QA pooling cannot benefit
from weight randomization — follows as a corollary. (Recall the definition of
$\mathbf{p}_{\mathbf{w}}^{*}$ from Definition 3.8.)

######  Corollary 5.4.

Consider a randomized algorithm $A$ with the following specifications:

  * •

Input: a proper scoring rule $s$ with convex exposure, expert forecasts
$\mathbf{p}_{1},\dots,\mathbf{p}_{m}$.

  * •

Output: a weight vector $\mathbf{w}\in\Delta^{m}$.

Consider any input $s,\mathbf{p}_{1},\dots,\mathbf{p}_{m}$, and let
$\hat{\mathbf{w}}=\mathbb{E}_{A}\left[\mathbf{w}\right]$. Then for every $j$,
we have

$s(\mathbf{p}^{*}_{\hat{\mathbf{w}}};j)\geq\mathbb{E}_{A}\left[s(\mathbf{p}^{*}_{\mathbf{w}};j)\right],$

where $\mathbf{p}^{*}_{\mathbf{x}}$ denotes the QA pool of
$\mathbf{p}_{1},\dots,\mathbf{p}_{m}$ with weight vector $\mathbf{x}$.

We now state the no-regret result that we have alluded to. The algorithm
referenced in the statement (which is in Appendix D.2) is an application of
the standard online gradient descent algorithm (see e.g. [Haz19, Theorem 3.1])
to our particular setting.

######  Theorem 5.5.

Let $s$ be a bounded proper scoring rule with convex exposure over a forecast
domain $\mathcal{D}$. For time steps $t=1\dots T$, an agent chooses a weight
vector $\mathbf{w}^{t}\in\Delta^{m}$. The agent then receives a score of

$s\left(\sideset{}{{}_{\mathbf{g}}}{\bigoplus}_{i=1}^{m}(\mathbf{p}_{i}^{t},w_{i}^{t});j^{t}\right),$

where $\mathbf{p}_{1}^{t},\dots,\mathbf{p}_{m}^{t}\in\mathcal{D}$ and
$j^{t}\in[n]$ are chosen adversarially. By choosing $\mathbf{w}^{t}$ according
to Algorithm D.3 (online gradient descent on the experts’ weights), the agent
achieves $O(\sqrt{T})$ regret in comparison with the best weight vector in
hindsight. In particular, if $M$ is an upper bound on
$\left\lVert\mathbf{g}\right\rVert_{2}$, then for every
$\mathbf{w}^{*}\in\Delta^{m}$ we have

$\sum_{t=1}^{T}s\left(\sideset{}{{}_{\mathbf{g}}}{\bigoplus}_{i=1}^{m}(\mathbf{p}_{i}^{t},w_{i}^{*});j^{t}\right)-s\left(\sideset{}{{}_{\mathbf{g}}}{\bigoplus}_{i=1}^{m}(\mathbf{p}_{i}^{t},w_{i}^{t});j^{t}\right)\leq
3\sqrt{m}M\sqrt{T}.$

Note that this bound $M$ exists because $s$ is bounded by assumption, and so
$\mathbf{g}$ is also bounded (this follows from Equation 3).

We also note that this result is quite strong in that it does not merely
achieve low regret compared to the best _expert_ , but in fact compared to the
best possible _weighted pool of experts_ in hindsight. This is a substantial
distinction, as it is possible for a mixture of experts to substantially
outperform any individual expert.

We defer the proof to Appendix D.2. The proof amounts to applying the standard
bounds for online gradient descent, though with an extra step: we use the
bound $M$ on $\left\lVert\mathbf{g}\right\rVert$ to bound the gradient of the
loss as a function of expert weights.

##  6 Axiomatization of QA pooling

In this section, we aim to show that the class of all quasi-arithmetic pooling
operators is a natural one, by showing that these operators are precisely
those which satisfy a natural set of axioms.

[Kol30] and [Nag30] independently considered the class of quasi-arithmetic
means. Given an interval $I\subseteq\mathbb{R}$ and a continuous, injective
function $f:I\to\mathbb{R}$, the _quasi-arithmetic mean with respect to $f$_,
or _$f$ -mean_, is the function $M_{f}$ that takes as input
$x_{1},\dots,x_{m}\in I$ (for any $m\geq 1$) and outputs

$M_{f}(x_{1},\dots,x_{m}):=f^{-1}\left(\frac{f(x_{1})+\dots+f(x_{m})}{m}\right).$

For example, the arithmetic mean corresponds to $f(x)=x$; the quadratic to
$f(x)=x^{2}$; the geometric to $f(x)=\log x$; and the harmonic to
$f(x)=\frac{-1}{x}$.

Kolmogorov proved that the class of quasi-arithmetic means is precisely the
class of functions $M:\bigcup_{m=1}^{\infty}I^{m}\to I$ satisfying the
following natural properties:131313Nagumo also provided a characterization,
though with slightly different properties.

  1. (1)

$M(x_{1},\dots,x_{m})$ is continuous and strictly increasing in each variable.

  2. (2)

$M$ is symmetric in its arguments.

  3. (3)

$M(x,x,\dots,x)=x$.

  4. (4)

$M(x_{1},\dots,x_{k},x_{k+1},\dots,x_{m})$ =
$M(y,\dots,y,x_{k+1},\dots,x_{m})$, where $y:=M(x_{1},\dots,x_{k})$ appears
$k$ times on the right-hand side. Informally, a subset of arguments to the
mean function can be replaced with their mean.

The four properties listed above can be viewed as an _axiomatization_ of
quasi-arithmetic means.

Our notion of quasi-arithmetic pooling is exactly that of a quasi-arithmetic
mean, except that it is more general in two ways. First, it allows for weights
to accompany the arguments to the mean. Second, we are considering quasi-
arithmetic means with respect to _vector-valued_ functions $\mathbf{g}$. In
the $n=2$ outcome case, $\mathbf{g}$ can be considered a scalar-valued
function since it is defined on a one-dimensional space (see Remark 3.5 for
details); but in general we cannot treat $\mathbf{g}$ as scalar-
valued.141414Another difference is that in the $n=2$ case, we are restricting
$\mathcal{D}\subseteq[0,1]$, though this is not a fundamental difference for
the purposes of this section.

Our goal is to extend the above axiomatization of quasi-arithmetic means in
these two ways: first (below) to include weights as arguments, and second (in
Appendix E) to general $n$ (while still allowing arbitrary weights).

#### Generalizing to include weights as arguments

The objects that we will be studying in this section are ones of the form
$(p,w)$, where $w\geq 0$ and $p\in\mathcal{D}$. In this subsection,
$\mathcal{D}$ is a two-outcome forecast domain, which we will think of as a
sub-interval of $[0,1]$ identified by the probability of the first outcome
(see Remark 3.5). We will fix the set $\mathcal{D}$ for the remainder of the
subsection. Our results generalize to any interval of $\mathbb{R}$ (as in
Kolmogorov’s work), but we focus on forecast domains since that is our
application.

######  Definition 6.1.

A _weighted forecast_ is an element of $\mathcal{D}\times\mathbb{R}_{>0}$: a
probability and a positive weight. Given a weighted forecast $\Pi=(p,w)$ we
define $\text{pr}(\Pi):=p$ and $\text{wt}(\Pi):=w$.

We will thinking of the output of pooling operators as weighted forecasts.
This is a simple extension of our earlier definition of quasi-arithmetic
pooling (Definition 3.8), which only output a probability.

######  Definition 6.2 (Quasi-arithmetic pooling with arbitrary weights
($n=2$)).

Given a continuous, strictly increasing function $g:\mathcal{D}\to\mathbb{R}$,
and weighted forecasts $\Pi_{1}=(p_{1},w_{1}),\dots,\Pi_{m}=(p_{m},w_{m})$,
define the _quasi-arithmetic pool_ of $\Pi_{1},\dots,\Pi_{m}$ with respect to
$g$ as

$\sideset{}{{}_{g}}{\bigoplus}_{i=1}^{m}(p_{i},w_{i}):=\left(g^{-1}\left(\frac{\sum_{i}w_{i}g(p_{i})}{\sum_{i}w_{i}}\right),\sum_{i}w_{i}\right).$

######  Remark 6.3.

In Definition 3.8, $g$ was the derivative of a differentiable, strictly convex
function. Here, $g$ is a continuous, strictly increasing function. These are
the same condition (see Proposition C.1).

In the case that $\sum_{i}w_{i}=1$, Definition 6.2 reduces to Definition 3.8.
In general, by linearly scaling the weights in Definition 6.2 to add to $1$,
we recover quasi-arithmetic pooling as previously defined.

The proof of the following proposition is straightforward.

######  Proposition 6.4.

Given two continuous, strictly increasing functions $g_{1}$ and $g_{2}$,
$\oplus_{g_{1}}$ and $\oplus_{g_{2}}$ are the same if and only if
$g_{2}=ag_{1}+b$ for some $a>0$ and $b\in\mathbb{R}$.

We now define properties (i.e. axioms) of a pooling operator $\oplus$, such
that these properties are satisfied if and only if $\oplus$ is $\oplus_{g}$
for some $g$. Our axiomatization will look somewhat different from
Kolmogorov’s, in part because we choose to define $\oplus$ as a binary
operator that (if it satisfies the associativity axiom) extends to the $m$-ary
case. This is a simpler domain and will simplify notation.

######  Definition 6.5 (Axioms for pooling operators ($n=2$)).

For a pooling operator $\oplus$ on $\mathcal{D}$ (i.e. a binary operator on
weighted forecasts), we define the following axioms.

  1. 1.

Weight additivity:
$\text{wt}(\Pi_{1}\oplus\Pi_{2})=\text{wt}(\Pi_{1})+\text{wt}(\Pi_{2})$ for
every $\Pi_{1},\Pi_{2}$.

  2. 2.

Commutativity: $\Pi_{1}\oplus\Pi_{2}=\Pi_{2}\oplus\Pi_{1}$ for every
$\Pi_{1},\Pi_{2}$.

  3. 3.

Associativity:
$\Pi_{1}\oplus(\Pi_{2}\oplus\Pi_{3})=(\Pi_{1}\oplus\Pi_{2})\oplus\Pi_{3}$ for
every $\Pi_{1},\Pi_{2},\Pi_{3}$.

  4. 4.

Continuity: For every $p_{1},p_{2}$, the quantity151515We allow one weight to
be $0$ by defining $(p,w)\oplus(q,0)=(q,0)\oplus(p,w)=(p,w)$.
$\text{pr}((p_{1},w_{1})\oplus(p_{2},w_{2}))$ is a continuous function of
$(w_{1},w_{2})$ on $\mathbb{R}_{\geq 0}^{2}\setminus\\{(0,0)\\}$.

  5. 5.

Idempotence: For every $\Pi_{1},\Pi_{2}$, if
$\text{pr}(\Pi_{1})=\text{pr}(\Pi_{2})$ then
$\text{pr}(\Pi_{1}\oplus\Pi_{2})=\text{pr}(\Pi_{1})$.

  6. 6.

Monotonicity: Let $w>0$ and let $p_{1}>p_{2}\in\mathcal{D}$. Then for
$x\in(0,w)$, the quantity $\text{pr}((p_{1},x)\oplus(p_{2},w-x))$ is a
strictly increasing function of $x$.

The motivation for the weight additivity axiom is that the weight of a
weighted forecast can be thought of as the _amount of evidence_ for its
prediction. When pooling weighted forecasts, the weight of an individual
forecast can be thought of as the strength of its vote in the aggregate.

The monotonicity axiom essentially states that if one pools two forecasts with
different probabilities and a fixed total weight, then the larger the share of
the weight belonging to the larger of the two probabilities, the larger the
aggregate probability.

We now state this section’s main result: these axioms describe the class of QA
pooling operators.

######  Theorem 6.6.

A pooling operator is a QA pooling operator (as in Definition 6.2) with
respect to some $g$ if and only if it satisfies the axioms in Definition
6.5.161616 As we mentioned, for an associative pooling operator $\oplus$,
$\Pi_{1}\oplus\Pi_{2}\dots\oplus\Pi_{m}$ is a well-specified quantity, even
without indicating parenthesization. This lets us use the notation
$\bigoplus_{i=1}^{m}\Pi_{i}$. This is why the statement of Theorem 6.6 makes
sense despite pooling operators not being $m$-ary by default.

We defer the proof of Theorem 6.6 to Appendix E, though we briefly summarize
it here. We first show that the axioms in Definition 6.5 hold for any QA
pooling operator. Weight additivity, commutativity, and idempotence are
trivial; associativity is a matter of simple algebra. Continuity and
monotonicity both follow from the fact that $g$ is continuous and strictly
increasing, as is $g^{-1}$.

Showing that any pooling operator $\oplus$ satisfying our axioms is a QA
pooling operator involves constructing a $g$ such that $\oplus=\oplus_{g}$.
This is simplest if $\mathcal{D}$ is a closed interval. For example, if
$\mathcal{D}=[0,1]$, we can define $g$ as follows: $g(0)=0$, $g(1)=1$, and for
$0<p<1$, $g(p)=w$ where $(1,w)\oplus(0,1-w)=(p,1)$ (such a $w$ exists by the
continuity axiom and is unique by the monotonicity axiom). The remainder
(showing that $\oplus$ pools weighted forecasts in the same way as
$\oplus_{g}$ as in Definition 6.2) follows by a sequence of applications of
the axioms.

The case where $\mathcal{D}$ is not closed is a little trickier. The idea
there is to define $g$ on two points on the interior of $\mathcal{D}$ and
build $g$ by essentially applying the same construction as described in the
previous paragraph, but with “negative weights.” We describe the details of
this argument in the proof of the analogous result for general $n$ (i.e.
Theorem E.9).

#### Generalizing to higher dimensions

In Appendix E, we discuss extending our axiomatization to arbitrary values of
$n$ in a way that, again, describes the class of QA pooling operators. An
important challenge is extending the monotonicity axiom: what is an
appropriate generalization of an increasing function in higher dimensions? We
show that the notion we need is _cyclical monotonicity_ , which we define and
discuss. We then present our axiomatization (Definition E.8) and prove that
the axioms represent precisely all QA pooling operators (Theorem E.9). On a
high level, the proof is not dissimilar to that of Theorem 6.6, though the
details are fairly different and more technical.

In conclusion, in Definition 6.5 we made a list of natural properties that a
pooling operator may satisfy. Theorem 6.6 shows that the pooling operators
satisfying these properties are exactly the QA pooling operators. In Appendix
E, we generalize this theorem to higher dimensions, thus fully axiomatizing QA
pooling. This result gives us an additional important reason to believe that
QA pooling with respect to a proper scoring rule is a fundamental notion.

##  7 Conclusions and future work

We conclude with one observation and a number of suggestions for future work.

We have established and motivated a connection between proper scoring rules
and opinion pooling methods. This connection is, in fact, a bijective
correspondence between proper scoring rules with convex exposure (up to
positive affine transformation) and pooling methods satisfying our axioms. In
most of our discussion, the focus has been on one side of this correspondence:
given a proper scoring rule $s$ (with convex exposure), we have shown that QA
pooling with respect to $s$ satisfies many desirable properties. It is natural
to consider the reverse direction as well: if a principal wishes to pool some
experts’ forecasts using a particular pooling method $\oplus$, does it make
sense for the principal to use the corresponding scoring rule for elicitation?

We argue that choosing such a scoring rule makes sense in a context where
experts may collude. If experts are rewarded with a proper scoring rule $s$
with convex exposure, they can guarantee themselves a larger total reward by
all reporting the QA pool with respect to $s$ of their true beliefs; in fact,
this is the collusion strategy that maximizes their minimum (over $j$)
possible surplus over truthful reporting. This is essentially a
restatement171717This is a stronger statement because it asserts that this is
the best of _all_ collusion strategies, not just those in which all experts
give the same report. This fact follows from the same techniques that we used
to prove Theorem 4.1. of Theorem 4.1, and was observed for the $n=2$ case in
[CS11]. This means that it makes sense for the principal to choose the scoring
rule that will incentivize the experts to collude in the manner that the
principal desires, i.e. according to $\oplus$. As such, the principal should
choose the scoring rule corresponding to $\oplus$.

Of course, the principal may wish to reward the experts using a contract
function for which collusion never generates a surplus. The question of
whether such a contract function exists is explored (but not resolved) in
[Fre+20]. Given the close ties between this line of work and ours, it will be
natural to explore whether the tools that we develop shed light on this
question.

Moving on to further intriguing research directions, it is natural to expand
our notion of forecast aggregation. All pooling methods that we considered
satisfy what we called idempotence: pooling $\mathbf{p}$ and $\mathbf{p}$
gives back $\mathbf{p}$. This is a natural assumption, but may be undesirable
in some Bayesian setups. In a situation where two experts use different
evidence to arrive at the same small probability of an outcome, it may be
sensible for the aggregate probability to be even smaller. It would be
interesting to explore whether our results and techniques are applicable to
notions of pooling that do not satisfy idempotence, or to other (perhaps
Bayesian) settings.

Finally, we present some concrete future directions for potential work:

  * •

As we discussed in Section 2, there is a fair amount of work on aggregating
forecasts with prediction markets — often ones that are based on proper
scoring rules. Is there a natural trading-based interpretation of QA pooling?

  * •

Definition 4.5 gave a natural generalization of QA pooling to proper scoring
rules that do not have convex exposure. There is another potentially natural
generalization, which is to define the QA pool as the forecast
$\mathbf{p}^{*}$ maximizing $\min_{j}u(\mathbf{p}^{*};j)$ (as defined in
Theorem 4.1). Is this generalization equivalent to Definition 4.5? If not, how
does it behave?

  * •

Although our proof of Theorem 5.1 (that the QA pool of forecasts is concave in
the experts’ weights) relies on the convex exposure property, we have not
ruled out the possibility that the result holds even without this assumption
(with QA pooling defined as in Definition 4.5). Is this the case? Even if not,
are no-regret algorithms for learning weights still possible?

  * •

Our no-regret algorithm for learning weights relies on $s$ being bounded,
because this allows us to place a concrete upper bound on $\left\lVert\nabla
L^{t}(\cdot)\right\rVert_{2}$. Intuitively it seems unlikely that no-regret
compared to the best weight vector in hindsight can be achieved if we cannot
place a bound on the loss function; is this so? Are there natural restrictions
to the model — i.e. ways to make it less than fully adversarial — under which
a no-regret algorithm would be possible?

  * •

We have presented a list of axioms characterizing QA pooling operators. Is
there an alternative axiomatization that uses equally natural but fewer
axioms?

## References

  * [Abb09] Ali E. Abbas  “A Kullback-Leibler View of Linear and Log-Linear Pools”  In _Decision Analysis_ 6.1, 2009, pp. 25–37  DOI: 10.1287/deca.1080.0133
  * [ABS18] Itai Arieli, Yakov Babichenko and Rann Smorodinsky  “Robust forecast aggregation”  In _Proceedings of the National Academy of Sciences_ 115.52, 2018, pp. E12135–E12143  DOI: 10.1073/pnas.1813934115
  * [ACR12] D Allard, A Comunian and Philippe Renard  “Probability aggregation methods in geoscience”  In _Mathematical Geosciences_ 44.5 Springer, 2012, pp. 545–581
  * [ACV13] Jacob D. Abernethy, Yiling Chen and Jennifer Wortman Vaughan  “Efficient Market Making via Convex Optimization, and a Connection to Online Learning”  In _ACM Trans. Economics and Comput._ 1.2, 2013, pp. 12:1–12:39  DOI: 10.1145/2465769.2465777
  * [Acz48] J. Aczél  “On mean values”  In _Bull. Amer. Math. Soc._ 54.4 American Mathematical Society, 1948, pp. 392–400  URL: https://projecteuclid.org:443/euclid.bams/1183511892
  * [Ada14] M. Adamčík  “Collective reasoning under uncertainty and inconsistency”, 2014
  * [AK14] Aaron Archer and Robert Kleinberg  “Truthful germs are contagious: A local-to-global characterization of truthfulness”  In _Games and Economic Behavior_ 86.C, 2014, pp. 340–366  URL: https://EconPapers.repec.org/RePEc:eee:gamebe:v:86:y:2014:i:c:p:340-366
  * [Ash+10] Itai Ashlagi, Mark Braverman, Avinatan Hassidim and Dov Monderer  “Monotonicity and Implementability”  In _Econometrica_ 78.5, 2010, pp. 1749–1772  DOI: https://doi.org/10.3982/ECTA8882
  * [AW80] J. Aczél and C. Wagner  “A Characterization of Weighted Arithmetic Means”  In _SIAM Journal on Algebraic Discrete Methods_ 1.3, 1980, pp. 259–260  DOI: 10.1137/0601029
  * [Ban+05] Arindam Banerjee, Srujana Merugu, Inderjit S. Dhillon and Joydeep Ghosh  “Clustering with Bregman Divergences”  In _J. Mach. Learn. Res._ 6, 2005, pp. 1705–1749  URL: http://jmlr.org/papers/v6/banerjee05b.html
  * [BB20] Shalev Ben-David and Eric Blais  “A New Minimax Theorem for Randomized Algorithms”  In _CoRR_ abs/2002.10802, 2020  arXiv: https://arxiv.org/abs/2002.10802
  * [BC11] Heinz H. Bauschke and Patrick L. Combettes  “Convex Analysis and Monotone Operator Theory in Hilbert Spaces”  Springer Publishing Company, Incorporated, 2011
  * [Bri50] G. W. Brier  “Verification of forecasts expressed in terms of probability”  In _Monthly Weather Review_ 78, 1950, pp. 1–3
  * [Car16] Arthur Carvalho  “An Overview of Applications of Proper Scoring Rules”  In _Decision Analysis_ 13, 2016  DOI: 10.1287/deca.2016.0337
  * [CBL06] Nicolò Cesa-Bianchi and Gábor Lugosi  “Prediction, Learning, and Games”  In _Prediction, Learning, and Games_ , 2006  DOI: 10.1017/CBO9780511546921
  * [Che+14] Yiling Chen, Nikhil R. Devanur, David M. Pennock and Jennifer Wortman Vaughan  “Removing arbitrage from wagering mechanisms”  In _ACM Conference on Economics and Computation, EC ’14, Stanford , CA, USA, June 8-12, 2014_ ACM, 2014, pp. 377–394  DOI: 10.1145/2600057.2602876
  * [CL13] Arthur Carvalho and Kate Larson  “A Consensual Linear Opinion Pool”  In _IJCAI 2013, Proceedings of the 23rd International Joint Conference on Artificial Intelligence, Beijing, China, August 3-9, 2013_ IJCAI/AAAI, 2013, pp. 2518–2524  URL: http://www.aaai.org/ocs/index.php/IJCAI/IJCAI13/paper/view/6447
  * [CP07] Yiling Chen and David M. Pennock  “A Utility Framework for Bounded-Loss Market Makers”  In _UAI 2007, Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence, Vancouver, BC, Canada, July 19-22, 2007_ AUAI Press, 2007, pp. 49–56  URL: https://dl.acm.org/doi/abs/10.5555/3020488.3020495
  * [CS11] SangIn Chun and Ross D. Shachter  “Strictly Proper Mechanisms with Cooperating Players”  In _UAI 2011, Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence, Barcelona, Spain, July 14-17, 2011_ AUAI Press, 2011, pp. 125–134  URL: https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2&article\\_id=2168&proceeding\\_id=27
  * [CV10] Yiling Chen and Jennifer Wortman Vaughan  “A new understanding of prediction markets via no-regret learning”  In _Proceedings 11th ACM Conference on Electronic Commerce (EC-2010), Cambridge, Massachusetts, USA, June 7-11, 2010_ ACM, 2010, pp. 189–198  DOI: 10.1145/1807342.1807372
  * [CW07] R.T. Clemen and R.L. Winkler  “Aggregating probability distributions”  In _Advances in Decision Analysis: From Foundations to Applications_ , 2007, pp. 154–176  DOI: 10.1017/CBO9780511611308.010
  * [Daw+95] A. Dawid, M. DeGroot, J. Mortera, R. Cooke, S. French, C. Genest, M. Schervish, D. Lindley, K. McConway and R. Winkler  “Coherent combination of experts’ opinions”  In _Test_ 4, 1995, pp. 263–313
  * [DL14] Franz Dietrich and Christian List  “Probabilistic Opinion Pooling”, 2014  URL: http://philsci-archive.pitt.edu/11349/
  * [DM14] Alexander Dawid and Monica Musio  “Theory and Applications of Proper Scoring Rules”  In _METRON_ 72, 2014  DOI: 10.1007/s40300-014-0039-y
  * [FCK15] Rafael M. Frongillo, Yiling Chen and Ian A. Kash  “Elicitation for Aggregation”  In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA_ AAAI Press, 2015, pp. 900–906  URL: http://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/9770
  * [FES20] Christian Feldbacher-Escamilla and Gerhard Schurz  “Optimal probability aggregation based on generalized brier scoring”  In _Annals of Mathematics and Artificial Intelligence_ 88, 2020  DOI: 10.1007/s10472-019-09648-4
  * [FK14] Rafael Frongillo and Ian Kash  “General Truthfulness Characterizations via Convex Analysis”  In _Web and Internet Economics_ Cham: Springer International Publishing, 2014, pp. 354–370
  * [Fre+20] Rupert Freeman, David M. Pennock, Dominik Peters and Bo Waggoner  “Preventing Arbitrage from Collusion When Eliciting Probabilities”  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ AAAI Press, 2020, pp. 1958–1965  URL: https://aaai.org/ojs/index.php/AAAI/article/view/5566
  * [Gen84] Christian Genest  “A Characterization Theorem for Externally Bayesian Groups”  In _Ann. Statist._ 12.3 The Institute of Mathematical Statistics, 1984, pp. 1100–1105  DOI: 10.1214/aos/1176346726
  * [Goo52] I. J. Good  “Rational Decisions”  In _Journal of the Royal Statistical Society. Series B (Methodological)_ 14.1 [Royal Statistical Society, Wiley], 1952, pp. 107–114  URL: http://www.jstor.org/stable/2984087
  * [GR07] Tilmann Gneiting and Adrian E Raftery  “Strictly proper scoring rules, prediction, and estimation”  In _Journal of the American Statistical Association_ 102.477 Taylor
& Francis, 2007, pp. 359–378

  * [Gra+11] Michel Grabisch, Jean-Luc Marichal, Radko Mesiar and Endre Pap  “Aggregation functions: Means”  In _Information Sciences_ 181.1, 2011, pp. 1 –22  DOI: https://doi.org/10.1016/j.ins.2010.08.043
  * [GZ86] Christian Genest and James V. Zidek  “Combining Probability Distributions: A Critique and an Annotated Bibliography”  In _Statistical Science_ 1.1 Institute of Mathematical Statistics, 1986, pp. 114–135  URL: http://www.jstor.org/stable/2245510
  * [Han03] Robin Hanson  “Combinatorial Information Market Design”  In _Information Systems Frontiers_ 5.1, 2003, pp. 107–119  DOI: 10.1023/A:1022058209073
  * [Haz19] Elad Hazan  “Introduction to Online Convex Optimization”  In _CoRR_ abs/1909.05207, 2019  arXiv: http://arxiv.org/abs/1909.05207
  * [Kol30] A.N. Kolmogorov  “Sur la notion de la moyenne”  G. Bardi, tip. della R. Accad. dei Lincei, 1930  URL: https://books.google.com/books?id=iUqLnQEACAAJ
  * [KR08] Christian Kascha and Francesco Ravazzolo  “Combining inflation density forecasts”, 2008  URL: https://ideas.repec.org/p/bno/worpap/2008_22.html
  * [LS07] Ron Lavi and Chaitanya Swamy  “Truthful Mechanism Design for Multi-Dimensional Scheduling via Cycle Monotonicity”  In _Proceedings of the 8th ACM Conference on Electronic Commerce_ , EC ’07  San Diego, California, USA: Association for Computing Machinery, 2007, pp. 252–261  DOI: 10.1145/1250910.1250947
  * [Nag30] Mitio Nagumo  “Über eine Klasse der Mittelwerte”  In _Japanese journal of mathematics :transactions and abstracts_ 7, 1930, pp. 71–79  DOI: 10.4099/jjm1924.7.0˙71
  * [Pet19] Richard Pettigrew  “Aggregating incoherent agents who disagree”  In _Synthese_ 196, 2019  DOI: 10.1007/s11229-017-1613-7
  * [PR00] David Poole and Adrian E. Raftery  “Inference for Deterministic Simulation Models: The Bayesian Melding Approach”  In _Journal of the American Statistical Association_ 95.452 Taylor & Francis, 2000, pp. 1244–1255  DOI: 10.1080/01621459.2000.10474324
  * [Roc70] R. T. Rockafellar  “On the maximal monotonicity of subdifferential mappings.”  In _Pacific J. Math._ 33.1 Pacific Journal of Mathematics, A Non-profit Corporation, 1970, pp. 209–216  URL: https://projecteuclid.org:443/euclid.pjm/1102977253
  * [Roc70a] R. Tyrrell Rockafellar  “Convex Analysis”  Princeton University Press, 1970  URL: http://www.jstor.org/stable/j.ctt14bs1ff
  * [SAM66] Emir Shuford, Arthur Albert and H. Edward Massengill  “Admissible probability measurement procedures”  In _Psychometrika_ 31.2, 1966, pp. 125–145  URL: https://EconPapers.repec.org/RePEc:spr:psycho:v:31:y:1966:i:2:p:125-145
  * [Sat+14] Ville Satopää, Jonathan Baron, Dean Foster, Barbara Mellers, Philip Tetlock and Lyle Ungar  “Combining multiple probability predictions using a simple logit model”  In _International Journal of Forecasting_ 30, 2014, pp. 344–356  DOI: 10.1016/j.ijforecast.2013.09.009
  * [Sav71] Leonard J. Savage  “Elicitation of Personal Probabilities and Expectations”  In _Journal of the American Statistical Association_ 66.336 [American Statistical Association, Taylor & Francis, Ltd.], 1971, pp. 783–801  URL: http://www.jstor.org/stable/2284229
  * [SY05] Michael Saks and Lan Yu  “Weak monotonicity suffices for truthfulness on convex domains”  In _Proceedings of the ACM Conference on Electronic Commerce_ , 2005, pp. 286–293  DOI: 10.1145/1064009.1064040
  * [Tsa88] Constantino Tsallis  “Possible generalization of Boltzmann-Gibbs statistics”  In _Journal of Statistical Physics_ 52, 1988, pp. 479–487  DOI: 10.1007/BF01016429
  * [Voh07] Rakesh V. Vohra  “Paths, Cycles and Mechanism Design”, 2007
  * [Wik18] Wikipedia contributors  “Mahler’s inequality — Wikipedia, The Free Encyclopedia” [Online; accessed 07-February-2021], 2018  URL: https://en.wikipedia.org/wiki/Mahler%27s\\_inequality

##  Appendix A Outline of appendices

  * •

In Appendix B we give an interpretation of logarithmic opinion pooling as
averaging experts’ Bayesian evidence, as we mentioned in Section 1.

  * •

In Appendix C we prove that $s$ is continuous if and only if $G$ is
differentiable if and only if $\mathbf{g}$ is continuous, as we stated in
Section 3.

  * •

In Appendix D we give the proof of Theorem 5.1 (that a QA aggregator’s score
is concave in the experts’ weights), state the no regret algorithm referenced
in Theorem 5.5, and then prove that the algorithm indeed has low regret.

  * •

In Appendix E we give the proof of Theorem 6.6, which states that the axioms
in Definition 6.5 capture the class of QA pooling methods in the case of $n=2$
outcomes. We then present and prove an analogous axiomatization in full
generality (i.e. for arbitrary values of $n$).

  * •

In Appendix F we discuss which well-known proper scoring rules satisfy the
convex exposure property.

##  Appendix B Details omitted from Section 1

#### Logarithmic pooling as averaging Bayesian evidence

We discuss for simplicity the binary outcome case, though this discussion
holds in general. Suppose that an expert assigns a probability to an event $X$
occurring by updating on some prior (50%, say, though this does not matter).
Suppose that the expert receives evidence $E$. Bayesian updating works as
such:

$\frac{\text{Pr}\left[X\mid E\right]}{\text{Pr}\left[\neg X\mid
E\right]}=\frac{\text{Pr}\left[X\right]}{\text{Pr}\left[\neg
X\right]}\cdot\frac{\text{Pr}\left[E\mid X\right]}{\text{Pr}\left[E\mid\neg
X\right]}.$

That is, the expert _multiplies their odds of $X$_ (i.e. the probability of
$X$ divided by the probability of $\neg X$) by the relative likelihood that
$E$ would be the case conditioned on $X$ versus $\neg X$. Equivalently, we can
take the log of both sides; this tells us that the posterior log odds of $X$
is equal to the prior log odds plus $\log\frac{\text{Pr}\left[E\mid
X\right]}{\text{Pr}\left[E\mid\neg X\right]}$. For every piece of evidence
$E_{k}$ that the expert receives, they make this update (assuming that the
$E_{k}$’s are mutually independent conditioned on $X$).

This means that for any $E_{k}$, we can view the quantity
$\log\frac{\text{Pr}\left[E_{k}\mid X\right]}{\text{Pr}\left[E_{k}\mid\neg
X\right]}$ as the strength of evidence that $E_{k}$ gives in favor of $X$. We
call this the “Bayesian evidence” in favor of $X$ given by $E_{k}$. The _total
Bayesian evidence_ that the expert has in favor of $X$ (i.e. the sum of these
values over all $E_{k}$) is the expert’s log odds of $X$ (i.e.
$\log\frac{\text{Pr}\left[X\right]}{\text{Pr}\left[\neg X\right]}$).

Logarithmic pooling takes the average of experts’ log odds of $X$. As we have
shown, if the experts are Bayesian, this amounts to taking the average of all
experts’ amounts of Bayesian evidence in favor of $X$.

##  Appendix C Details omitted from Section 3

######  Proposition C.1.

Given a proper scoring rule $s$, the following are equivalent:

  1. (1)

$s$ is continuous

  2. (2)

$G$ is differentiable

  3. (3)

$G$ is continuously differentiable

###### Proof.

Any differentiable convex function is continuously differentiable [Roc70a,
Theorem 25.5] so (2) implies (3).

If $G$ is continuously differentiable, then both $G$ and $\mathbf{g}$ are
continuous, so $s$ is continuous by Equation 3. Thus, (3) implies (1).

Finally, assume that $s$ is continuous. Since
$G(\mathbf{p})=\sum_{j}p(j)s(\mathbf{p};j)$, it follows that $G$ is
continuous. It follows by Equation 3 (with $\mathbf{g}$ taken to be a
subgradient of $G$, as in [GR07]) that $\mathbf{g}$ is continuous. A convex
function with a continuous subgradient is differentiable [BC11, Proposition
17.41]. This proves that (1) implies (2). ∎

##  Appendix D Details omitted from Section 5

###  D.1 Proof of Theorem 5.1

See 5.1

###### Proof of Theorem 5.1.

Let $\mathbf{v}$ and $\mathbf{w}$ be two weight vectors. We wish to show that
for any $c\in[0,1]$, we have

$\text{WS}_{j}(c\mathbf{v}+(1-c)\mathbf{w})-c\text{WS}_{j}(\mathbf{v})-(1-c)\text{WS}_{j}(\mathbf{w})\geq
0.$

Recall the notation $\mathbf{p}^{*}_{\mathbf{w}}$ from Definition 3.8. Note
that

$\mathbf{g}(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}})=\sum_{i=1}^{m}(cv_{i}+(1-c)w_{i})\mathbf{g}(\mathbf{p}_{i})=c\mathbf{g}(\mathbf{p}^{*}_{\mathbf{v}})+(1-c)\mathbf{g}(\mathbf{p}^{*}_{\mathbf{w}}).$
(5)

We have

$\displaystyle\quad\text{WS}_{j}(c\mathbf{v}+(1-c)\mathbf{w})-c\text{WS}_{j}(\mathbf{v})-(1-c)\text{WS}_{j}(\mathbf{w})$
$\displaystyle=s(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}};j)-cs(\mathbf{p}^{*}_{\mathbf{v}};j)-(1-c)s(\mathbf{p}^{*}_{\mathbf{w}};j)$
$\displaystyle=G(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}})+\langle\mathbf{g}(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}}),\delta_{j}-\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}}\rangle$
$\displaystyle\qquad-c(G(\mathbf{p}^{*}_{\mathbf{v}})+\langle\mathbf{g}(\mathbf{p}^{*}_{\mathbf{v}}),\delta_{j}-\mathbf{p}^{*}_{\mathbf{v}}\rangle)-(1-c)(G(\mathbf{p}^{*}_{\mathbf{w}})+\langle\mathbf{g}(\mathbf{p}^{*}_{\mathbf{w}}),\delta_{j}-\mathbf{p}^{*}_{\mathbf{w}}\rangle)$
$\displaystyle=G(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}})-\langle\mathbf{g}(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}}),\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}}\rangle$
$\displaystyle\qquad-c(G(\mathbf{p}^{*}_{\mathbf{v}})-\langle\mathbf{g}(\mathbf{p}^{*}_{\mathbf{v}}),\mathbf{p}^{*}_{\mathbf{v}}\rangle)-(1-c)(G(\mathbf{p}^{*}_{\mathbf{w}})-\langle\mathbf{g}(\mathbf{p}^{*}_{\mathbf{w}}),\mathbf{p}^{*}_{\mathbf{w}}\rangle)$

Step 1 follows from the definition of WS. Step 2 follows from Equation 3. Step
3 follows from Equation 5, and specifically that the inner product of each
side with $\delta_{j}$ is the same (so the $\delta_{j}$ terms cancel out,
leaving a quantity that does not depend on $j$). Continuing where we left off:

$\displaystyle\quad\text{WS}_{j}(c\mathbf{v}+(1-c)\mathbf{w})-c\text{WS}_{j}(\mathbf{v})-(1-c)\text{WS}_{j}(\mathbf{w})$
$\displaystyle=G(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}})-c\langle\mathbf{g}(\mathbf{p}^{*}_{\mathbf{v}}),\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}}\rangle-(1-c)\langle\mathbf{g}(\mathbf{p}^{*}_{\mathbf{w}}),\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}}\rangle$
$\displaystyle\qquad-c(G(\mathbf{p}^{*}_{\mathbf{v}})-\langle\mathbf{g}(\mathbf{p}^{*}_{\mathbf{v}}),\mathbf{p}^{*}_{\mathbf{v}}\rangle)-(1-c)(G(\mathbf{p}^{*}_{\mathbf{w}})-\langle\mathbf{g}(\mathbf{p}^{*}_{\mathbf{w}}),\mathbf{p}^{*}_{\mathbf{w}}\rangle)$
$\displaystyle=c(G(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}})-G(\mathbf{p}^{*}_{\mathbf{v}})-\langle\mathbf{g}(\mathbf{p}^{*}_{\mathbf{v}}),\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}}-\mathbf{p}^{*}_{\mathbf{v}}\rangle)$
$\displaystyle\qquad+(1-c)(G(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}})-G(\mathbf{p}^{*}_{\mathbf{w}})-\langle\mathbf{g}(\mathbf{p}^{*}_{\mathbf{w}}),\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}}-\mathbf{p}^{*}_{\mathbf{w}}\rangle)$
$\displaystyle=cD_{G}(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}}\parallel\mathbf{p}^{*}_{\mathbf{v}})+(1-c)D_{G}(\mathbf{p}^{*}_{c\mathbf{v}+(1-c)\mathbf{w}}\parallel\mathbf{p}^{*}_{\mathbf{w}})\geq
0.$

Step 4 again follows from Equation 5. Step 5 is a rearrangement of terms.
Finally, step 6 follows from the definition of Bregman divergence, and step 7
follows from the fact that Bregman divergence is always non-negative. This
completes the proof. ∎

###  D.2 Gradient descent algorithm for Theorem 5.5

We follow the online gradient descent algorithm as presented in [Haz19, §3.1].
Hazan’s algorithm is as follows:

######  Algorithm D.1 ([Haz19], Algorithm 6).

Input: convex set $\mathcal{K}$, convex function $f_{t}$ on $\mathcal{K}$,
$T$, $\mathbf{x}_{1}\in\mathcal{K}$, step sizes $\\{\eta_{t}\\}$.

for $t=1$ to $T$ do

Play $\mathbf{x}_{t}$ and observe cost $f_{t}(\mathbf{x}_{t})$

Update and project:

$\displaystyle\mathbf{y}_{t+1}$ $\displaystyle=\mathbf{x}_{t}-\eta_{t}\nabla
f_{t}(\mathbf{x}_{t})$ $\displaystyle\mathbf{x}_{t+1}$
$\displaystyle=\Pi_{\mathcal{K}}(\mathbf{y}_{t+1})$

end for

Algorithm D.1 achieves the following regret guarantee.

######  Theorem D.2 ([Haz19], Theorem 3.1).

Online gradient descent with step sizes
$\\{\eta_{t}=\frac{D}{G\sqrt{t}},t\in[T]$ guarantees the following for all
$T\geq 1$:

$\text{regret}_{T}=\sum_{t=1}^{T}f_{t}(\mathbf{x}_{t})-\min_{\mathbf{x}^{*}\in\mathcal{K}}\sum_{t=1}^{T}f_{t}(\mathbf{x}^{*})\leq\frac{3}{2}GD\sqrt{T}.$

Here, $D$ is an upper bound on the diameter of $\mathcal{K}$ and $G$ is an
upper bound on the Lipschitz constant of any $f_{t}$, i.e.

$\left\lvert f_{t}(\mathbf{x})-f_{t}(\mathbf{y})\right\rvert\leq
G\left\lVert\mathbf{x}-\mathbf{y}\right\rVert_{2}$

for any $t$ and any $\mathbf{x},\mathbf{y}\in\mathcal{K}$.

In our setting, $\mathcal{K}=\Delta^{m}$ and $f_{t}$ is our loss function
(i.e. negative the score) at time step $t$, as a function of $\mathbf{w}$; we
will denote this function by $L^{t}$. That is,
$L^{t}(\mathbf{w})=-\text{WS}_{j^{t}}(\mathbf{w})$, where WS is as in Theorem
5.1, relative to forecasts $\mathbf{p}_{1}^{t},\dots,\mathbf{p}_{m}^{t}$.

Our adaptation of Algorithm D.1 is as follows.

######  Algorithm D.3 (Online gradient descent algorithm for Theorem 5.5).

We proceed as follows:

  * •

For $t\geq 1$, define $\eta_{t}:=\frac{1}{M\sqrt{mt}}$.

  * •

Start with an arbitrary guess $\mathbf{w}^{1}\in\Delta^{m}$.

  * •

At each time step $t$ from $1$ to $T$:

    * –

Play $\mathbf{w}^{t}$ and observe loss $L^{t}(\mathbf{w}^{t})$.

    * –

Let $\tilde{\mathbf{w}}^{t+1}=\mathbf{w}^{t}-\eta_{t}\nabla
L^{t}(\mathbf{w}^{t})$. If $\tilde{\mathbf{w}}^{t+1}\in\Delta^{m}$, let
$\mathbf{w}^{t+1}=\tilde{\mathbf{w}}^{t+1}$. Otherwise, let $\mathbf{w}^{t+1}$
be the orthogonal projection of $\tilde{\mathbf{w}}^{t+1}$ onto $\Delta^{m}$.

We now prove that Algorithm D.3 satisfies the details of Theorem 5.5.

See 5.5

###### Proof.

In our setting, $D=\sqrt{2}$. We claim that $B\leq\sqrt{2m}M$, where $B$ is an
upper bound on $\left\lVert\nabla L^{t}(\mathbf{w})\right\rVert_{2}$. This
would make our choice of $\eta_{t}$ match that of Algorithm D.1 and guarantee
a regret of at most $\frac{3}{\sqrt{2}}B\sqrt{T}\leq 3\sqrt{m}M\sqrt{T}$. The
remainder of the proof is demonstrating that indeed $B\leq\sqrt{2m}M$.

Let $L$ be an arbitrary loss function, i.e.
$L(\mathbf{w})=-\text{WS}_{j}(\mathbf{w})$ for some
$j,\mathbf{p}_{1},\dots,\mathbf{p}_{m}$. Let
$\mathbf{p}^{*}(\mathbf{w})=\sideset{}{{}_{\mathbf{g}}}{\bigoplus}\limits_{i=1}^{m}(\mathbf{p}_{i},w_{i})$.
We claim that

$\nabla L(\mathbf{w})=\begin{pmatrix}\mathbf{g}(\mathbf{p}_{1})\\\ \vdots\\\
\mathbf{g}(\mathbf{p}_{m})\end{pmatrix}(\mathbf{p}^{*}(\mathbf{w})-\delta_{j}),$
(6)

where this $m$-dimensional vector should be interpreted modulo translation by
$\mathbf{1}_{m}$ (see Remark 3.9). To see this, observe that

$\nabla L(\mathbf{w})=-\nabla
WS_{j}(\mathbf{w})=-\nabla_{\mathbf{w}}s(\mathbf{p}^{*}(\mathbf{w});j)=-\nabla_{\mathbf{w}}(G(\mathbf{p}^{*}(\mathbf{w}))+\langle\mathbf{g}(\mathbf{p}^{*}(\mathbf{w})),\delta_{j}-\mathbf{p}^{*}(\mathbf{w})\rangle),$

where $\nabla_{\mathbf{w}}$ denotes the gradient with respect to change in the
weight vector $\mathbf{w}$ (as opposed to change in the probability vector).
Now, by the chain rule for gradients, we have

$\nabla_{\mathbf{w}}G(\mathbf{p}^{*}(\mathbf{w}))=(J_{\mathbf{p}^{*}}(\mathbf{w}))^{\top}\mathbf{g}(\mathbf{p}^{*}(\mathbf{w})),$

where $J_{\mathbf{p}^{*}}$ denotes the Jacobian matrix of the function
$\mathbf{p}^{*}(\mathbf{w})$. Also, we have

$\mathbf{g}(\mathbf{p}^{*}(\mathbf{w}))=\sum_{i=1}^{m}w_{i}\mathbf{g}(\mathbf{p}_{i}),$

so (again by the chain rule) we have

$\nabla_{\mathbf{w}}(\langle\mathbf{g}(\mathbf{p}^{*}(\mathbf{w})),\delta_{j}-\mathbf{p}^{*}(\mathbf{w})\rangle)=\begin{pmatrix}\mathbf{g}(\mathbf{p}_{1})\\\
\vdots\\\
\mathbf{g}(\mathbf{p}_{m})\end{pmatrix}(\delta_{j}-\mathbf{p}^{*}(\mathbf{w}))-(J_{\mathbf{p}^{*}}(\mathbf{w}))^{\top}\mathbf{g}(\mathbf{p}^{*}(\mathbf{w})).$

This gives us Equation 6.

Now, for any $i$, we have

$\left\lvert\langle\mathbf{g}(\mathbf{p}_{i}),\mathbf{p}^{*}(\mathbf{w})-\delta_{j}\rangle\right\rvert\leq\left\lVert\mathbf{g}(\mathbf{p}_{i})\right\rVert_{2}\left\lVert\mathbf{p}^{*}(\mathbf{w})-\delta_{j}\right\rVert_{2}\leq\sqrt{2}M.$

Therefore,

$\left\lVert\nabla
L(\mathbf{w})\right\rVert_{2}\leq\sqrt{m\cdot(\sqrt{2}M)^{2}}=\sqrt{2m}M,$

completing the proof. ∎

##  Appendix E Details omitted from Section 6

###  E.1 Proof of Theorem 6.6

See 6.6

###### Proof.

For this proof, we will use $\oplus$ (without a $g$ subscript) to denote an
arbitrary pooling operator that satisfies the axioms in Definition 6.5. We
begin by noting a few important facts about weighted forecasts and pooling
operators. First, we find it natural to define a notion of multiplying a
weighted forecast pair by a positive constant.

######  Definition E.1.

Given a weighted forecast $\Pi=(p,w)$ and $c>0$, define $c\Pi:=(p,cw)$.

Note that $m\Pi=\bigoplus_{i=1}^{m}\Pi$ for any positive integer $m$, by
idempotence; this definition is a natural extension to all $c>0$. We note the
following (quite obvious) fact.

######  Proposition E.2.

For every weighted forecast $\Pi$ and $c_{1},c_{2}>0$, we have
$c_{1}(c_{2}\Pi)=(c_{1}c_{2})\Pi$.

A natural property that is not listed in Definition 6.5 is _scale invariance_
, i.e. that
$\text{pr}((p_{1},w_{1})\oplus(p_{2},w_{2}))=\text{pr}((p_{1},cw_{1})\oplus(p_{2},cw_{2}))$
for any positive $c$; or, equivalently, that
$c(\Pi_{1}\oplus\Pi_{2})=c\Pi_{1}\oplus c\Pi_{2}$. This in fact follows from
the listed axioms.

######  Proposition E.3 (Distributive property/scale invariance).

For every $\Pi_{1},\Pi_{2}$ and any operator $\oplus$ satisfying the axioms in
Definition 6.5, we have $c(\Pi_{1}\oplus\Pi_{2})=c\Pi_{1}\oplus c\Pi_{2}$.

###### Proof.

First suppose $c$ is an integer. Then

$c\Pi_{1}\oplus
c\Pi_{2}=\bigoplus_{i=1}^{c}\Pi_{1}\oplus\bigoplus_{i=1}^{c}\Pi_{2}=\bigoplus_{i=1}^{c}(\Pi_{1}\oplus\Pi_{2})=c(\Pi_{1}\oplus\Pi_{2}).$

Here, the first and last steps follow by weight additivity and idempotence.
Now suppose that $c=\frac{k}{\ell}$ is a rational number. Let
$\Pi_{1}^{\prime}=\frac{1}{\ell}\Pi_{1}$ and
$\Pi_{2}^{\prime}=\frac{1}{\ell}\Pi_{2}$. We have

$\frac{k}{\ell}(\Pi_{1}\oplus\Pi_{2})=\frac{k}{\ell}(\ell\Pi_{1}^{\prime}\oplus\ell\Pi_{2}^{\prime})=\frac{k}{\ell}\cdot\ell(\Pi_{1}^{\prime}\oplus\Pi_{2}^{\prime})=k(\Pi_{1}^{\prime}\oplus\Pi_{2}^{\prime})=k\Pi_{1}^{\prime}\oplus
k\Pi_{2}^{\prime}=\frac{k}{\ell}\Pi_{1}\oplus\frac{k}{\ell}\Pi_{2}.$

Here, the second and second-to-last steps follow from the fact that the
distributive property holds for integers.

Finally, make use of the continuity axiom to extend our proof to all positive
real numbers $c$. In particular, it suffices to show that
$\text{pr}(\Pi_{1}\oplus\Pi_{2})=\text{pr}(c\Pi_{1}\oplus c\Pi_{2})$. Let $p$
be the former quantity; note that $\text{pr}(r\Pi_{1}\oplus r\Pi_{2})=p$ for
positive rational numbers $r$. Since the rationals are dense among the reals,
it follows that for every $\epsilon>0$, we have
$\left\lvert\text{pr}(c\Pi_{1}\oplus c\Pi_{2})-p\right\rvert\leq\epsilon$.
Therefore, $\text{pr}(c\Pi_{1}\oplus c\Pi_{2})=p$. This completes the proof. ∎

Returning to the proof of Theorem 6.6, we first prove that any QA pooling
operator $\oplus_{g}$ satisfies the axioms in Definition 6.5. Weight
additivity, commutativity, and idempotence are trivial. Associativity is also
clear: given $\Pi_{1}=(p_{1},w_{1})$ and likewise $\Pi_{2},\Pi_{3}$, we have

$\displaystyle g(\text{pr}((\Pi_{1}\oplus_{g}\Pi_{2})\oplus_{g}\Pi_{3}))$
$\displaystyle=\frac{(w_{1}+w_{2})g(\text{pr}(\Pi_{1}\oplus_{g}\Pi_{2}))+w_{3}g(p_{3})}{(w_{1}+w_{2})+w_{3}}$
$\displaystyle=\frac{(w_{1}+w_{2})\frac{w_{1}g(p_{1})+w_{2}(p_{2})}{w_{1}+w_{2}}+w_{3}g(p_{3})}{w_{1}+w_{2}+w_{3}}=\frac{w_{1}g(p_{1})+w_{2}g(p_{2})+w_{3}g(p_{3})}{w_{1}+w_{2}+w_{3}}$

and likewise for $g(\text{pr}(\Pi_{1}\oplus_{g}(\Pi_{2}\oplus_{g}\Pi_{3})))$,
so
$\text{pr}((\Pi_{1}\oplus_{g}\Pi_{2})\oplus_{g}\Pi_{3})=\text{pr}(\Pi_{1}\oplus_{g}(\Pi_{2}\oplus_{g}\Pi_{3}))$
(since $g$ is strictly increasing and therefore injective). The fact that the
weights are also the same is trivial. Continuity follows from the fact that

$\text{pr}(\Pi_{1}\oplus_{g}\Pi_{2})=g^{-1}\left(\frac{w_{1}g(p_{1})+w_{2}g(p_{2})}{w_{1}+w_{2}}\right)$

is continuous in $(w_{1},w_{2})$ (when $w_{1},w_{2}$ are not both zero). Here
we are using the fact that $g$ is _strictly_ increasing, which means that
$g^{-1}$ is continuous.

Finally, regarding the monotonicity axiom, for any fixed $w$ and $p_{1}>p_{2}$
(as in the axiom statement), we have

$g(\text{pr}((p_{1},x)\oplus_{g}(p_{2},w-x)))=\frac{xg(p_{1})+(w-x)g(p_{2})}{x+w-x}=\frac{xg(p_{1})+(w-x)g(p_{2})}{w}.$

Since $p_{1}>p_{2}$, we have $g(p_{1})>g(p_{2})$, so the right-hand side
strictly increases with $x$. Since $g^{-1}$ is also strictly increasing, it
follows that $\text{pr}((p_{1},x)\oplus_{g}(p_{2},w-x))$ strictly increases
with $x$.

The converse — that every pooling operator satisfying the axioms in Definition
6.5 is $\oplus_{g}$ for some $g$ — works by constructing $g$ by fixing it at
two points and constructing $g$ at all other points. Right now we show how to
do this when the forecast domain is $[0,1]$ (though the technique works for
any closed $\mathcal{D}$); see the proof of Theorem E.9 for the argument in
full generality.

Let $\oplus$ be a pooling operator that satisfies our axioms. Define $g$ as
follows: let $g(0)=0$ and $g(1)=1$. For $0<p<1$, define $g(p)=w$ where
$(1,w)\oplus(0,1-w)=(p,1)$. (This $w$ exists by continuity and the
intermediate value theorem; it is unique by the “strictly” increasing
stipulation of monotonicity.) Note that $g$ is continuous and increasing by
monotonicity.181818As a matter of fact, $g$ is strictly increasing because it
is impossible for $g(p_{1})$ to equal $g(p_{2})$ for $p_{1}\neq p_{2}$, as
that would mean that $(1,g(p_{1}))\oplus(0,1-g(p_{1}))=(p_{1},1)=(p_{2},1)$.
Another way to look at this is that it comes from the fact that
$(1,w)\oplus(0,1-w)$ is continuous in $w$ by the continuity axiom. In a sense,
the _continuity_ of $g$ corresponds to the _strictness_ of increase in the
monotonicity axiom and the _strictness_ of increase of $g$ corresponds to the
_continuity_ axiom.

We wish to show that for any $\Pi_{1}=(p_{1},w_{1})$ and
$\Pi_{2}=(p_{2},w_{2})$, we have that
$\Pi_{1}\oplus\Pi_{2}=\Pi_{1}\oplus_{g}\Pi_{2}$. Clearly the weight of both
sides is $w_{1}+w_{2}$, so we wish to show that the probabilities on each side
are the same. We have191919Steps 3 and 7 uses the distributive property
(Proposition E.3).

$\displaystyle\text{pr}(\Pi_{1}\oplus\Pi_{2})$
$\displaystyle=\text{pr}(w_{1}(p_{1},1)\oplus w_{2}(p_{2},1))$
$\displaystyle=\text{pr}(w_{1}((1,g(p_{1}))\oplus(0,1-g(p_{1})))\oplus
w_{2}((1,g(p_{2}))\oplus(0,1-g(p_{2}))))$
$\displaystyle=\text{pr}(w_{1}(1,g(p_{1}))\oplus w_{1}(0,1-g(p_{1}))\oplus
w_{2}(1,g(p_{2}))\oplus w_{2}(0,1-g(p_{2})))$
$\displaystyle=\text{pr}((1,w_{1}g(p_{1}))\oplus(0,w_{1}(1-g(p_{1})))\oplus(1,w_{2}g(p_{2}))\oplus(0,w_{2}(1-g(p_{2}))))$
$\displaystyle=\text{pr}((1,w_{1}g(p_{1})+w_{2}g(p_{2}))\oplus(0,w_{1}(1-g(p_{1}))+w_{2}(1-g(p_{2}))))$
$\displaystyle=\text{pr}\left(\frac{1}{w_{1}+w_{2}}((1,w_{1}g(p_{1})+w_{2}g(p_{2}))\oplus(0,w_{1}(1-g(p_{1}))+w_{2}(1-g(p_{2}))))\right)$
$\displaystyle=\text{pr}\left(\left(1,\frac{w_{1}g(p_{1})+w_{2}g(p_{2})}{w_{1}+w_{2}}\right)\oplus\left(0,\frac{w_{1}(1-g(p_{1}))+w_{2}(1-g(p_{2}))}{w_{1}+w_{2}}\right)\right),$

which by definition of $g$ is equal to the probability $p$ such that
$g(p)=\frac{g(p_{1})w_{1}+g(p_{2})w_{2}}{w_{1}+w_{2}}$. That is,
$\text{pr}(\Pi_{1}\oplus\Pi_{2})=\text{pr}(\Pi_{1}\oplus_{g}\Pi_{2})$.

Showing that $\oplus$ and $\oplus_{g}$ are equivalent for more than two
arguments is now trivial:

$\sideset{}{{}_{\mathbf{g}}}{\bigoplus}_{i=1}^{m}\Pi_{i}=\Pi_{1}\oplus_{g}\Pi_{2}\oplus_{g}\Pi_{3}\dots\oplus_{g}\Pi_{m}=\Pi_{1}\oplus\Pi_{2}\oplus_{g}\Pi_{3}\dots\oplus_{g}\Pi_{m}=\dots=\bigoplus_{i=1}^{m}\Pi_{i}.$

(Here we are implicitly using the fact that $\oplus_{g}$ is associative, as we
proved earlier.) This completes the proof. ∎

###  E.2 Generalization of our axiomatization to higher dimensions

Just as we fixed a two-outcome forecast domain $\mathcal{D}$ in Section 6, we
now fix an $n$-outcome forecast domain $\mathcal{D}$ for any $n\geq 2$. Our
definition of weighted forecasts remains the same (except that now
$\text{pr}(\Pi)$ is a vector). Our definition of quasi-arithmetic pooling,
however, needs to change to make $\mathbf{g}$ vector-valued. This raises the
question: what is the analogue of “increasing” for vector-valued functions? It
turns out that the relevant notion for us is _cyclical monotonicity_ ,
introduced by Rockafellar [Roc70] (see also [Roc70a, §27]). We will define
this notion shortly, but first we give the definition of quasi-arithmetic
pooling with arbitrary weights (analogous to Definition 6.2) for this setting.
Throughout this appendix, we will use the notation
$H_{n}(c):=\\{\mathbf{x}\in\mathbb{R}^{n}:\sum_{i}x_{i}=c\\}$. Recall from
Remark 3.9 that the range of the gradient of a function defined on
$\mathcal{D}$ is a subset of $H_{n}(0)$.

######  Definition E.4 (Quasi-arithmetic pooling with arbitrary weights).

Given a continuous, strictly cyclically monotone vector-valued function
$\mathbf{g}:\mathcal{D}\to H_{n}(0)$ whose range is a convex set, and weighted
forecasts
$\Pi_{1}=(\mathbf{p}_{1},w_{1}),\dots,\Pi_{m}=(\mathbf{p}_{m},w_{m})$, define
the _quasi-arithmetic pool_ of $\Pi_{1},\dots,\Pi_{m}$ with respect to
$\mathbf{g}$ as

$\sideset{}{{}_{\mathbf{g}}}{\bigoplus}_{i=1}^{m}(\mathbf{p}_{i},w_{i}):=\left(\mathbf{g}^{-1}\left(\frac{\sum_{i}w_{i}\mathbf{g}(\mathbf{p}_{i})}{\sum_{i}w_{i}}\right),\sum_{i}w_{i}\right).$

######  Definition E.5 (Cyclical monotonicity).

A function $\mathbf{g}:U\subseteq\mathbb{R}^{n}\to\mathbb{R}^{n}$ is
_cyclically monotone_ if for every list of points
$\mathbf{x}_{0},\mathbf{x}_{1},\dots,\mathbf{x}_{k-1},\mathbf{x}_{k}=\mathbf{x}_{0}\in
U$, we have

$\sum_{i=1}^{k}\langle\mathbf{g}(\mathbf{x}_{i}),\mathbf{x}_{i}-\mathbf{x}_{i-1}\rangle\geq
0.$

We also say that $\mathbf{g}$ is _strictly cyclically monotone_ if the
inequality is strict except when $\mathbf{x}_{0}=\dots=\mathbf{x}_{k-1}$.

To gain an intuition for this notion, consider the case of $k=2$; then this
condition says that
$\langle\mathbf{g}(\mathbf{x}_{1})-\mathbf{g}(\mathbf{x}_{0}),\mathbf{x}_{1}-\mathbf{x}_{0}\rangle\geq
0$. In other words, the change in $\mathbf{g}$ from $\mathbf{x}_{0}$ to
$\mathbf{x}_{1}$ is in the same general direction as the direction from
$\mathbf{x}_{0}$ to $\mathbf{x}_{1}$. This property is called _2-cycle_ (or
_weak_) _monotonicity_.

Cyclical monotonicity is a stronger notion, which may be familiar to the
reader for its applications in mechanism design and revealed preference
theory, see e.g. [LS07], [Ash+10], [FK14], [Voh07, §2.1]. In such settings, it
is usually the case that two-cycle and cyclical monotonicity are equivalent.
Indeed, Saks and Yu showed that these conditions are equivalent in settings
where the set of outcomes (i.e. the range of $\mathbf{g}$) is finite [SY05].
However, cyclical monotonicity is substantially stronger than two-cycle
monotonicity when the range of $\mathbf{g}$ is infinitely large, as in our
setting. In fact, the difference between these two conditions is that a two-
cycle monotone function is cyclically monotone if and only if it is also
_vortex-free_ [AK14, Theorem 3.9]. _Vortex-freeness_ means that the path
integral of $\mathbf{g}$ along any triangle vanishes. See [AK14] for a
deteailed comparison of these two notions.

The immediately relevant fact for us is that cyclically monotone functions are
gradients of convex functions (and vice versa). Speaking more precisely:

######  Theorem E.6.

A vector-valued function $\mathbf{g}$ is continuous and strictly cyclically
monotone if and only if it is the gradient of a differentiable, strictly
convex function $G$.

###### Proof of Theorem E.6.

Per a theorem of Rockafellar ([Roc70], see also Theorem 24.8 in [Roc70a]), a
function $\mathbf{g}$ is cyclically monotone if and only if it is a
subgradient of a convex function $G$. The proof of this fact shows just as
easily that a function is strictly cyclically monotone if and only if it is a
subgradient of a strictly convex function.

Consider a differentiable, strictly convex function $G$. Its gradient is
continuous (see Proposition C.1). Conversely, consider a continuous, strictly
cyclically monotone vector-valued function $\mathbf{g}$. As we just discussed,
it is a subgradient of some strictly convex function $G$. A convex function
with a continuous subgradient is differentiable [BC11, Proposition 17.41]. ∎

This means that the conditions on $\mathbf{g}$ in Definition E.4 are precisely
those necessary to let $\mathbf{g}$ be any function that it could be in our
original definition of quasi-arithmetic pooling (Definition 3.8). Our new
definition is thus equivalent to the old one (after normalizing weights to add
to $1$).

We now discuss our axioms for pooling operators that will again capture the
class of QA pooling operators. We will keep the weight additivity,
commutativity, associativity, and idempotence verbatim from our discussion of
the $n=2$ case. We will slightly strengthen the continuity argument (see
below).

We will also add a new axiom, _subtraction_ , which states that if
$\Pi_{1}\oplus\Pi_{2}=\Pi_{1}\oplus\Pi_{3}$ then $\Pi_{2}=\Pi_{3}$.
Subtraction in the $n=2$ case follows from monotonicity; in this case,
however, we the subtraction axiom will help us state the monotonicity axiom.
In particular, it allows us to make the following definition, which
essentially extends the notion of pooling to allow for negative weights.

######  Definition E.7.

Let $\oplus$ be a pooling operator satisfying weight additivity,
commutativity, associativity, and subtraction. Fix
$\mathbf{p}_{1},\dots,\mathbf{p}_{k}\in\mathcal{D}$. Define a function
$\mathbf{p}:\Delta^{k}\to\mathcal{D}$ (with
$\mathbf{p}_{1},\dots,\mathbf{p}_{k}$ serving as implicit arguments) defined
by

$\mathbf{p}(w_{1},\dots,w_{k})=\text{pr}\left(\bigoplus_{i=1}^{k}(\mathbf{p}_{i},w_{i})\right).$

We extend the definition of $\mathbf{p}$ to a partial function on $H_{k}(1)$,
as follows: given input $(w_{1},\dots,w_{k})$, let $S\subseteq[k]$ be the set
of indices $i$ such that $w_{i}<0$ and $T\subseteq[k]$ be the set of indices
$i$ such that $w_{i}>0$. We define $\mathbf{p}(w_{1},\dots,w_{k})$ to be the
$\mathbf{q}\in\mathcal{D}$ such that

$(\mathbf{q},1)\oplus\left(\bigoplus_{i\in
S}(\mathbf{p}_{i},-w_{i})\right)=\bigoplus_{i\in T}(\mathbf{p}_{i},w_{i}).$

Note that $\mathbf{q}$ is not guaranteed to exist, which is why we call
$\mathbf{p}$ a partial function. However, if $\mathbf{q}$ exists then it is
unique, by the subtraction axiom.

We can now state the full axiomatization, including the monotonicity axiom.

######  Definition E.8 (Axioms for pooling operators).

For a pooling operator $\oplus$ on $\mathcal{D}$, we define the following
axioms.

  1. 1.

Weight additivity:
$\text{wt}(\Pi_{1}\oplus\Pi_{2})=\text{wt}(\Pi_{1})+\text{wt}(\Pi_{2})$ for
every $\Pi_{1},\Pi_{2}$.

  2. 2.

Commutativity: $\Pi_{1}\oplus\Pi_{2}=\Pi_{2}\oplus\Pi_{1}$ for every
$\Pi_{1},\Pi_{2}$.

  3. 3.

Associativity:
$\Pi_{1}\oplus(\Pi_{2}\oplus\Pi_{3})=(\Pi_{1}\oplus\Pi_{2})\oplus\Pi_{3}$ for
every $\Pi_{1},\Pi_{2},\Pi_{3}$.

  4. 4.

Continuity: For every positive integer $k$ and
$\mathbf{p}_{1},\dots,\mathbf{p}_{k}$, the quantity202020The continuity axiom
is only well-defined conditioned on $\oplus$ being associative, which is fine
for our purposes. We allow a proper subset of weights to be zero by defining
the aggregate to ignore forecasts with weight zero.

$\text{pr}\left(\bigoplus_{i=1}^{k}(\mathbf{p}_{i},w_{i})\right)$

is a continuous function of $(w_{1},\dots,w_{k})$ on $\mathbb{R}_{\geq
0}^{k}\setminus\\{\mathbf{0}\\}$.

  5. 5.

Idempotence: For every $\Pi_{1}$ and $\Pi_{2}$, if
$\text{pr}(\Pi_{1})=\text{pr}(\Pi_{2})$ then
$\text{pr}(\Pi_{1}\oplus\Pi_{2})=\text{pr}(\Pi_{1})$.

  6. 6.

Subtraction: If $\Pi_{1}\oplus\Pi_{2}=\Pi_{1}\oplus\Pi_{3}$ then
$\Pi_{2}=\Pi_{3}$.

  7. 7.

Monotonicity: There exist vectors
$\mathbf{p}_{1},\dots,\mathbf{p}_{n}\in\mathcal{D}$ such that $\mathbf{p}$ (as
in Definition E.7) is a strictly cyclically monotone function from its domain
to $\mathbb{R}^{n}$.

This monotonicity axiom essentially extends our previous monotonicity axiom
(in Definition 6.5) to a multi-dimensional setting. It states that there are
$n$ “anchor points” in $\mathcal{D}$ such that the function $\mathbf{p}$ from
weight vectors to $\mathcal{D}$ that pools the anchor points with the weights
given as input obeys a notion of monotonicity (namely cyclical monotonicity).
Informally, this means that the vector of weights that one would need to give
to the anchor points in order to arrive at a forecast $\mathbf{p}$
“correlates” with the forecast $\mathbf{p}$ itself.

We now state the main theorem of our axiomatization.

######  Theorem E.9.

A pooling operator is a QA pooling operator (as in Definition E.4) with
respect to some $\mathbf{g}$ if and only if it satisfies the axioms in
Definition E.8.

###### Proof.

We begin by noting the following fact, which follows from results in [Roc70a,
§26].

######  Proposition E.10.

A strictly cyclically monotone function
$\mathbf{g}:\mathcal{D}\to\mathbb{R}^{n}$ is injective, and its inverse
$\mathbf{g}^{-1}$ is strictly cyclically monotone and continuous.212121Why
can’t we apply this result again to $\mathbf{g}^{-1}$ to conclude that
$\mathbf{g}$ is continuous, even though we did not assume it to be? The reason
is that the proof of continuity relies on the convexity of $\mathcal{D}$; if
$\mathbf{g}$ is discontinuous then the domain of $\mathbf{g}^{-1}$ may not be
convex (or even connected), so we cannot apply the result to
$\mathbf{g}^{-1}$.

We provide a partial proof below; it relies on the following observation.

######  Remark E.11.

We can instead write the condition as

$\sum_{i=1}^{k}(\mathbf{g}(\mathbf{x}_{i})-\mathbf{g}(\mathbf{x}_{i-1}))\cdot\mathbf{x}_{i}\geq
0.$

This is equivalent to the condition in Definition E.5, because it is the same
statement (with rearranged terms) when the $\mathbf{x}_{i}$’s are listed in
reverse order.

###### Proof.

First, suppose that $\mathbf{g}(\mathbf{x})=\mathbf{g}(\mathbf{y})$. Then

$\mathbf{g}(\mathbf{x})(\mathbf{x}-\mathbf{y})+\mathbf{g}(\mathbf{y})(\mathbf{y}-\mathbf{x})=0.$

Since $\mathbf{g}$ is _strictly_ cyclically monotone, this implies that
$\mathbf{x}=\mathbf{y}$. (Note that we only use two-cycle monotonicity.)

We now show that $\mathbf{g}^{-1}$ is strictly cyclically monotone. That is,
we wish to show that

$\sum_{i=1}^{k}\mathbf{x}_{i}\cdot(\mathbf{g}^{-1}(\mathbf{x}_{i})-\mathbf{g}^{-1}(\mathbf{x}_{i-1}))>0$

for any $\mathbf{x}_{1},\dots,\mathbf{x}_{k}=\mathbf{x}_{0}$ that are not all
the same. (See Remark E.11.) By the cyclical monotonicity of $\mathbf{g}$, we
have that

$\sum_{i=1}^{k}\mathbf{g}(\mathbf{p}_{i})\cdot(\mathbf{p}_{i}-\mathbf{p}_{i-1})>0$

(the strictness of the inequality follows by the injectivity of $\mathbf{g}$:
if $\mathbf{x}_{i}\neq\mathbf{x}_{j}$ then
$\mathbf{p}_{i}\neq\mathbf{p}_{j}$). This means that

$\sum_{i=1}^{k}\mathbf{x}_{i}\cdot(\mathbf{g}^{-1}(\mathbf{x}_{i})-\mathbf{g}^{-1}(\mathbf{x}_{i-1}))>0,$

as desired. As for continuity, we defer to [Roc70a, Theorem 26.5]. ∎

Back to the proof of Theorem E.9, we first prove that any such
$\oplus_{\mathbf{g}}$ satisfies the stated axioms. Weight additivity,
commutativity, associativity, and idempotence are clear. Continuity follows
from the formula

$\text{pr}((\mathbf{p}_{1},w_{1})\oplus_{\mathbf{g}}(\mathbf{p}_{2},w_{2}))=\mathbf{g}^{-1}\left(\frac{w_{1}\mathbf{g}(\mathbf{p}_{1})+w_{2}\mathbf{g}(\mathbf{p}_{2})}{w_{1}+w_{2}}\right),$

noting that $\mathbf{g}^{-1}$ is continuous by Proposition E.10. Likewise,
subtraction follows from the fact that $\mathbf{g}$ is injective (by
Proposition E.10), as is $\mathbf{g}^{-1}$ (likewise). Monotonicity remains.

The range of $\mathbf{g}$ contains an open subset222222This follows from the
invariance of domain theorem, which states that the image of an open subset of
a manifold under an injective continuous map is open. of $H_{n}(0)$, so in
particular it contains the vertices of some translated and dilated copy of the
standard simplex. That is, there are $n$ points
$\mathbf{x}_{1},\dots,\mathbf{x}_{n}$ in the range of $\mathbf{g}$ for which
there is a positive scalar $a$ and vector $\mathbf{b}$ such that
$a\delta_{i}+\mathbf{b}=\mathbf{x}_{i}$ for every $i$. (Here $\delta_{i}$ is
the $i$-th standard basis vector in $\mathbb{R}^{n}$.) We will let
$\mathbf{p}_{i}$ be the pre-image of $\mathbf{x}_{i}$ under $\mathbf{g}$, so
that $\mathbf{g}(\mathbf{p}_{i})=a\delta_{i}+\mathbf{b}$.

Observe that for any $\mathbf{w}$ in the domain of $\mathbf{p}$, we have

$\mathbf{g}(\mathbf{p}(\mathbf{w}))=\sum_{i=1}^{n}w_{i}\mathbf{g}(\mathbf{p}_{i})=\sum_{i=1}^{n}w_{i}(a\delta_{i}+\mathbf{b})=a\mathbf{w}+\mathbf{b},$

so

$\mathbf{p}(\mathbf{w})=\mathbf{g}^{-1}(a\mathbf{w}+\mathbf{b}).$

We have that $\mathbf{g}^{-1}$ is strictly cyclically monotone (by Proposition
E.10), and it is easy to verify that for any strictly cyclically monotone
function $\mathbf{f}$ and any $a>0$ and $\mathbf{b}$,
$\mathbf{f}(a\mathbf{x}+\mathbf{b})$ is a strictly cyclically monotone
function of $\mathbf{x}$. Therefore,
$\mathbf{p}(\mathbf{w})=\mathbf{g}^{-1}(a\mathbf{w}+\mathbf{b})$ is strictly
cyclically monotone, as desired.

Now we prove the converse. Assume that we have a pooling operator $\oplus$
satisfying the axioms in Definition E.8. We wish to show that $\oplus$ is
$\oplus_{\mathbf{g}}$ for some $\mathbf{g}:\mathcal{D}\to H_{n}(0)$.

For the remainder of this proof, let $\mathbf{p}_{1},\dots,\mathbf{p}_{n}$ be
vectors certifying the monotonicity of $\oplus$, and let $\mathbf{p}(\cdot)$
be as in Definition E.7.

For any $\mathbf{q}\in\mathcal{D}$, let
$\mathbf{g}(\mathbf{q}):=\mathbf{w}-\frac{1}{n}\mathbf{1}_{n}$, where
$\mathbf{w}\in H_{n}(1)$ is such that $\mathbf{p}(\mathbf{w})=\mathbf{q}$ and
$\mathbf{1}_{n}$ is the all-ones vector. This raises the question of well-
definedness: does this $\mathbf{w}$ necessarily exist, and if so, is it
unique? The following claim shows that this is indeed the case.

######  Claim E.12.

The function $\mathbf{p}$, from the subset of $H_{n}(1)$ where it is defined
to $\mathcal{D}$, is bijective.

###### Proof.

The fact that $\mathbf{p}$ is injective follows from the fact that it is
strictly cyclically monotone (see Proposition E.10). We now show that
$\mathbf{p}$ is surjective.

Let $\mathbf{q}\in\mathcal{D}$. Define the function
$\tilde{\mathbf{p}}:\Delta^{n+1}\to\mathcal{D}$ by

$\tilde{\mathbf{p}}(w_{1},\dots,w_{n+1}):=\text{pr}\left(\left(\bigoplus_{i=1}^{n}(\mathbf{p}_{i},w_{i})\right)\oplus(\mathbf{q},w_{n+1})\right).$

Since $\tilde{\mathbf{p}}$ is a continuous map232323By the continuity axiom;
here we use the more generalized form we stated earlier. from $\Delta^{n+1}$
(an $n$-dimensional manifold) to $\mathcal{D}$ (an $(n-1)$-dimensional
manifold), $\tilde{\mathbf{p}}$ is not injective.242424This follows e.g. from
the Borsuk-Ulam theorem. So in particular, let
$\mathbf{w}_{1}\neq\mathbf{w}_{2}\in\Delta^{n+1}$ be such that
$\mathbf{\tilde{p}}(\mathbf{w}_{1})=\mathbf{\tilde{p}}(\mathbf{w}_{2})$. That
is, we have

$\left(\bigoplus_{i=1}^{n}(\mathbf{p}_{i},w_{1,i})\right)\oplus(\mathbf{q},w_{1,n+1})=\left(\bigoplus_{i=1}^{n}(\mathbf{p}_{i},w_{2,i})\right)\oplus(\mathbf{q},w_{2,n+1}).$
(7)

Observe that $w_{1,n+1}\neq w_{2,n+2}$; for otherwise it would follows from
the subtraction axiom that two different combinations of the
$\mathbf{p}_{i}$’s would give the same probability, contradicting the fact
that $\mathbf{p}$ is injective. Without loss of generality, assume that
$w_{1,n+1}>w_{2,n+1}$. We can rearrange the terms in Equation 7 to look as
follows.

$(\mathbf{q},w_{1,n+1}-w_{2,n+1})\oplus\left(\bigoplus_{i\in
S}(\mathbf{p}_{i},v_{i})\right)=\bigoplus_{i\in T\subseteq[n]\setminus
S}(\mathbf{p}_{i},v_{i})$

for some positive $v_{1},\dots,v_{n}$. By the distributive property, we may
multiply all weights by $\frac{1}{w_{1,n+1}-w_{2,n+1}}$. The result will be an
equation as in Definition E.7, certifying that $\mathbf{q}$ is in the range of
the function $\mathbf{p}$, as desired. ∎

We return to our main proof, now that we have shown that our function
$\mathbf{g}(\mathbf{q}):=\mathbf{w}-\frac{1}{n}\mathbf{1}$, where
$\mathbf{w}\in H_{n}(1)$ is such that $\mathbf{p}(\mathbf{w})=\mathbf{q}$, is
well-defined. In fact, we can simply write
$\mathbf{g}(\mathbf{q})=\mathbf{p}^{-1}(\mathbf{q})-\frac{1}{n}\mathbf{1}$.
(The vector $\frac{1}{n}\mathbf{1}$ is fairly arbitrary; it only serves the
purpose of forcing the range of $\mathbf{g}$ to lie in $H_{n}(0)$ instead of
$H_{n}(1)$.)

We first show that the defining equation of $\oplus_{\mathbf{g}}$ holds — that
is, that if
$(\mathbf{q}_{1},v_{1})\oplus(\mathbf{q}_{2},v_{2})=(\mathbf{q},v_{1}+v_{2})$
(with $v_{1},v_{2}\geq 0$, not both zero), then

$\mathbf{g}(\mathbf{q})=\frac{v_{1}\mathbf{g}(\mathbf{q}_{1})+v_{2}\mathbf{g}(\mathbf{q}_{2})}{v_{1}+v_{2}}.$

Let $\mathbf{w}_{1},\mathbf{w}_{2}\in H_{n}(1)$ be such that
$\mathbf{q}_{1}=\mathbf{p}(\mathbf{w}_{1})$ and
$\mathbf{q}_{2}=\mathbf{p}(\mathbf{w}_{2})$. It is intuitive that
$\mathbf{q}=\mathbf{p}\left(\frac{v_{1}\mathbf{w_{1}}+v_{2}\mathbf{w_{2}}}{v_{1}+v_{2}}\right)$,
but we show this formally.

######  Claim E.13.

Given $\mathbf{q}_{1},\mathbf{q}_{2}\in\mathcal{D}$ with
$\mathbf{q}_{1}=\mathbf{p}(\mathbf{w}_{1}),\mathbf{q}_{2}=\mathbf{p}(\mathbf{w}_{2})$,
and $0\leq\alpha\leq 1$, we have

$\mathbf{p}(\alpha\mathbf{w}_{1}+(1-\alpha)\mathbf{w}_{2})=(\mathbf{q}_{1},\alpha)\oplus(\mathbf{q}_{2},1-\alpha).$

###### Proof.

Note that

$\displaystyle(\mathbf{q}_{1},1)\oplus\left(\bigoplus_{i:w_{1,i}<0}(\mathbf{p}_{i},w_{1,i})\right)$
$\displaystyle=\bigoplus_{i:w_{1,i}>0}(\mathbf{p}_{i},w_{1,i})$
$\displaystyle(\mathbf{q}_{2},1)\oplus\left(\bigoplus_{i:w_{2,i}<0}(\mathbf{p}_{i},w_{2,i})\right)$
$\displaystyle=\bigoplus_{i:w_{2,i}>0}(\mathbf{p}_{i},w_{2,i}).$

Applying the distributive property to the two above equations with constants
$\alpha$ and $1-\alpha$, respectively, and adding them, we get that

$\displaystyle\left(\mathbf{q}_{1},\alpha\right)\oplus\left(\mathbf{q}_{2},1-\alpha\right)\oplus\left(\bigoplus_{i:w_{1,i}<0}(\mathbf{p}_{i},\alpha
w_{1,i})\right)\oplus\left(\bigoplus_{i:w_{2,i}<0}(\mathbf{p}_{i},(1-\alpha)w_{2,i})\right)$
$\displaystyle=\left(\bigoplus_{i:w_{1,i}>0}(\mathbf{p}_{i},\alpha
w_{1,i})\right)\oplus\left(\bigoplus_{i:w_{2,i}>0}(\mathbf{p}_{i},(1-\alpha)w_{2,i})\right).$

We have that
$\left(\mathbf{q}_{1},\alpha\right)\oplus\left(\mathbf{q}_{2},1-\alpha\right)=(\mathbf{q},1)$.
It follows (after rearranging terms, from Definition E.7) that
$\mathbf{q}=\mathbf{p}\left(\frac{v_{1}\mathbf{w_{1}}+v_{2}\mathbf{w_{2}}}{v_{1}+v_{2}}\right)$.
∎

Applying Claim E.13 with $\alpha=\frac{v_{1}}{v_{1}+v_{2}}$, we find that

$\mathbf{g}(\mathbf{q})=\frac{v_{1}\mathbf{w_{1}}+v_{2}\mathbf{w_{2}}}{v_{1}+v_{2}}-\frac{1}{n}\mathbf{1}=\frac{v_{1}\left(\mathbf{g}(\mathbf{q}_{1})+\frac{1}{n}\mathbf{1}\right)+v_{2}\left(\mathbf{g}(\mathbf{q}_{2})+\frac{1}{n}\mathbf{1}\right)}{v_{1}+v_{2}}-\frac{1}{n}\mathbf{1}=\frac{v_{1}\mathbf{g}(\mathbf{q}_{1})+v_{2}\mathbf{g}(\mathbf{q}_{2})}{v_{1}+v_{2}},$

as desired.

It remains to show that $\mathbf{g}$ is continuous, strictly cyclically
monotone, and has convex range. By the monotonicity axiom, $\mathbf{p}$ is
strictly cyclically monotone. It follows by Proposition E.10 that its inverse
its continuous and strictly cyclically monotone. Therefore, $\mathbf{g}$ is
continuous and cyclically monotone (as it is simply a translation of
$\mathbf{p}^{-1}(\mathbf{q})$ by $\frac{1}{n}\mathbf{1}$).

Finally, to show that $\mathbf{g}$ has convex range, we wish to show that
$\mathbf{p}^{-1}$ has convex range; or, in other words, that the domain on
which $\mathbf{p}$ is defined is convex. And indeed, this follows
straightforwardly from Claim E.13. Let $\mathbf{w}_{1},\mathbf{w}_{2}$ be in
the domain of $\mathbf{p}$, with
$\mathbf{p}(\mathbf{w}_{1})=\mathbf{q}_{1},\mathbf{p}(\mathbf{w}_{2})=\mathbf{q}_{2}$.
Then for any $0\leq\alpha\leq 1$, we have that

$\mathbf{p}(\alpha\mathbf{w}_{1}+(1-\alpha)\mathbf{w}_{2})=(\mathbf{q}_{1},\alpha)\oplus(\mathbf{q}_{2},1-\alpha),$

so in particular $\alpha\mathbf{w}_{1}+(1-\alpha)\mathbf{w}_{2}$ is in the
domain of $\mathbf{p}$. This concludes the proof. ∎

##  Appendix F The convex exposure property

Several of our results have been contingent on the convex exposure property.
We showed in Proposition 3.7 that this property always holds in the case of
$n=2$ outcomes (assuming, as we have been, that $s$ is continuous). In this
appendix, we take this discussion further by considering when the convex
exposure property holds in higher dimensions. As we shall see, the property
holds for nearly all of the most commonly used scoring rules.

(A note on notation: in this section we use $p_{j}$ instead of $p(j)$ to refer
to the $j$-th coordinate of a probability distribution $\mathbf{p}$.)

Our first result says, roughly speaking, that scoring rules that — like the
logarithmic scoring rule — “go off to infinity” have convex exposure.

######  Proposition F.1.

Let $s$ be a proper scoring rule on a forecast domain $\mathcal{D}$ that is an
open subset of $\Delta^{n}$, such that for any point $\mathbf{x}$ on the
boundary of $\mathcal{D}$, and for any sequence
$\mathbf{x}_{1},\mathbf{x}_{2},\dots$ converging to $\mathbf{x}$,
$\lim_{k\to\infty}\left\lVert\mathbf{g}(\mathbf{x}_{k})\right\rVert_{2}=\infty$.
Then $s$ has convex exposure.

This is a statement of convex analysis — namely that if
$\left\lVert\mathbf{g}\right\rVert_{2}$ approaches $\infty$ on the boundary of
a convex set, then the range of $\mathbf{g}$ is convex (assuming $\mathbf{g}$
is the gradient of a differentiable convex function). We refer the reader to
[Roc70a, Theorem 26.5] for the proof. In non-pathological cases, the basic
intuition is that _every_ $\mathbf{v}\in\\{\mathbf{x}:\sum_{i}x_{i}=0\\}$ is
the gradient of $G$ at some point. In these cases, $\nabla
G(\mathbf{x})=\mathbf{v}$ where $\mathbf{x}$ minimizes
$G(\mathbf{v})-\mathbf{v}\cdot\mathbf{x}$; the $\lim_{k\to\infty}\left\lVert
g(\mathbf{x}_{k})\right\rVert_{2}=\infty$ condition means that this minimum
does not occur on the boundary of $\mathcal{D}$.

######  Corollary F.2.

Let $\text{int}(\Delta^{n}):=\\{\mathbf{p}\in\Delta^{n}:p_{i}>0\enskip\forall
i\\}$. The following scoring rules have convex exposure over forecast domain
$\mathcal{D}=\text{int}(\Delta^{n})$.

  * •

The logarithmic scoring rule.

  * •

The scoring rule given by $G(\mathbf{p})=-\sum_{j}p_{j}^{\gamma}$ for
$\gamma\in(0,1)$ and by $G(\mathbf{p})=\sum_{j}p_{j}^{\gamma}$ for $\gamma<0$.

  * •

The scoring rule given by $G(\mathbf{p})=-\sum_{j}\ln p_{j}$, which can be
thought of as the limit of the $G$ in the previous bullet point as $\gamma\to
0$.252525This is a natural way to think of this scoring rule because $\nabla
G(\mathbf{p})=-(p_{1}^{-1},\dots,p_{n}^{-1})$.

  * •

The scoring rule $hs$ given by $G_{hs}(\mathbf{p})=-\prod_{j}p_{j}^{1/n}$.

#### The $hs$ scoring rule

The last of these scoring rules is a generalization of the scoring rule
$hs(q)=1-\sqrt{\frac{1-q}{q}}$ used in [BB20] as part of proving their minimax
theorem for randomized algorithms.262626Here we are using the shorthand
notation for the $n=2$ outcome case discussed in Remark 3.5. The authors used
this scoring rule as a key ingredient in their minimax theorem for randomized
algorithms. The key property of the scoring rule was a result about its
_amplification_ [BB20, Lemma 3.10]. The authors define a _forecasting
algorithm_ to be a generalization of a randomized algorithm that outputs an
estimated _probability_ that an output should be accepted. Then, roughly
speaking, the authors show that given a forecasting algorithm $R$, it is
possible to create a forecasting algorithm $R^{\prime}$ that has a much larger
expected score from the scoring rule $hs$ by combining running $R$ a small
number of times and combining the outputs. This is an important new result in
theoretical computer science and suggests that $hs$ deserves more attention.

Since additive and multiplicative constants are irrelevant, we may treat
$hs(q)=-\frac{1}{2}\sqrt{\frac{1-q}{q}}$. Observe that (in the case of two
outcomes), the expected score $G_{hs}$ on a report of $q$ is

$G_{hs}(q)=q\left(-\frac{1}{2}\sqrt{\frac{1-q}{q}}\right)+(1-q)\left(-\frac{1}{2}\sqrt{\frac{1-q}{q}}\right)=-\sqrt{q(1-q)}.$

That is, $G_{hs}$ is precisely negative the geometric mean of $q$ and $1-q$.
This motivates us to generalize $hs$ to a setting with $n$ outcomes by setting

$G_{hs}(\mathbf{p}):=-\prod_{i=1}^{n}p_{i}^{1/n}.$

It should not be obvious that this function is convex, but it turns out to be;
this is the precise statement of an inequality known as Mahler’s inequality
[Wik18].

Next we note that the quadratic scoring rule has convex exposure, since its
exposure function $\mathbf{g}(\mathbf{p})=2\mathbf{p}$ (modulo
$\mathbf{1}_{n}$ as discussed in Remark 3.9) maps any convex set to a convex
set.

######  Proposition F.3.

The quadratic scoring rule has convex exposure (for _any_ convex
$\mathcal{D}$).

_Spherical_ scoring rules — the third most studied proper scoring rules, after
the quadratic and logarithmic rules — also have convex exposure.

######  Definition F.4 (Spherical scoring rules).

[GR07, Example 2] For any $\alpha>1$, define the _spherical scoring rule with
parameter $\alpha$_ to be the scoring rule given by

$G_{\text{sph},\alpha}(\mathbf{p}):=\left(\sum_{i=1}^{n}p_{i}^{\alpha}\right)^{1/\alpha}.$

If the “spherical scoring rule” is referenced with no parameter $\alpha$
given, $\alpha$ is presumed to equal $2$.

######  Proposition F.5.

For any $\alpha>1$, the spherical scoring rule with parameter $\alpha$ (over
$\mathcal{D}=\Delta^{n}$) has convex exposure.

###### Proof.

Fix $\alpha>1$. We will write $G$ in place of $G_{\text{sph},\alpha}$. We
have272727As discussed in Remark 3.9, the range of $\mathbf{g}$ should be
thought of as modulo $T(\mathbf{1}_{n})$. However, we find it convenient for
this proof to think of it as lying in $\mathbb{R}^{n}$ and project later.

$\mathbf{g}(\mathbf{p})=\left(\sum_{j=1}^{n}p_{j}^{\alpha}\right)^{(1/\alpha)-1}(p_{1}^{\alpha-1},\dots,p_{n}^{\alpha-1}).$
(8)

Now, define the _$n$ -dimensional unit $\beta$-sphere_, i.e.
$\\{\mathbf{x}:\sum_{j}x_{j}^{\beta}=1\\}$, and define the _$n$ -dimensional
unit $\beta$-ball_ correspondingly (i.e. with $\leq$ in place of $=$). The
range of $\mathbf{g}$ is precisely the part of the $n$-dimensional unit
$\frac{\alpha}{\alpha-1}$-sphere with all non-negative coordinates. Indeed, on