#  Forecasting skill of a crowd-prediction platform: A comparison of exchange
rate forecasts††thanks: I am indebted to Robert L. Czudaj and Nikos Bosse for
their valuable comments on an earlier draft of this paper. I also thank the
Metaculus team for providing relevant data and showing spirit for open science
by giving me permission to publish findings regardless of outcomes. No
Metaculus staff was involved in this analysis. None of the views expressed are
attributable to Metaculus.

Niklas V. Lehmann Technical University Bergakademie Freiberg, Researcher at
the Chair for General Economics and Macroeconomics. Contact: Niklas-
<EMAIL_ADDRESS>
(December 2023)

###### Abstract

Open online crowd-prediction platforms are increasingly used to forecast
trends and complex events. Despite the large body of research on crowd-
prediction and forecasting tournaments, online crowd-prediction platforms have
never been directly compared to other forecasting methods. In this analysis,
exchange rate crowd-predictions made on Metaculus are compared to predictions
made by the random-walk, a statistical model considered extremely hard-to-
beat. The random-walk provides less erroneous forecasts, but the crowd-
prediction does very well. By using the random-walk as a benchmark, this
analysis provides a rare glimpse into the forecasting skill displayed on open
online crowd-prediction platforms.

Keywords: Forecasting tournament, Crowd prediction, Wisdom of crowds,
Judgmental forecasting, Exchange rate depreciation

JEL: C53, C58

Data Availability & Conflicts of interest: This paper contains all relevant
information to replicate and build upon the results. Relevant data on the
Metaculus prediction can be obtained via the API that Metaculus provides. I
initially signed up for the forecasting tournament that is studied here. Only
during the tournament did I discover the potential for the research outlined
above. I did not receive any funds or benefits, monetary or otherwise, for the
research presented and therefore have no conflicts of interest.

##  1 Introduction

When obtaining forecasts, there are three essential sources of foresight:
Models, experts and crowds. Most forecasts are made by models. Models capture
our knowledge of the laws that govern nature or make use of historical data to
predict the future. Nonetheless, not all knowledge is encapsulated in models
and the value of judgmental forecasting, particular the elicitation of experts
beliefs, is undisputed ([79]). But expert resources are scarce. This makes
forecasting with experts relatively expensive.111The recent surge in
popularity of open online crowd prediction can be seen as evidence in favor of
a huge (previously) unmet demand for low-cost forecasts. As of the end of
2023, the crowd-prediction platform Metaculus features over 8600 submitted
questions (inclining currently at around 2000 questions p.a.) even though
Metaculus is not the only provider of crowd-prediction infrastructure as there
are others such as GoodJudgment Open and Manifold Markets. Whilst expert
opinion is usually the best source of judgmental forecasts, crowds of
ambitious amateurs have also come to be a respectable source of forecasts
([101], [53]). Because of the large number of amateurs available for
forecasting, crowds can be belittled with many more detailed forecasting
questions than experts. Since James Surowiecki’s book The wisdom of crowds, it
is well known that a group of forecasters often performs better than any
individual forecaster within that group ([98]). This collective approach to
forecasting, which goes back to the idea of combining multiple forecasts to
distill a better one ([57]), has become known as crowd-prediction. In recent
years, several online platforms have been developed based on this concept,
following the pioneering project of GoodJudgment Open ([102]). These platforms
allow anyone to join and make predictions for free. Users can submit
questions, which are then openly available for other users to forecast on.
Forecasts on questions are usually combined to form a ’community consensus’.
Forecasters can share and discuss evidence and their predictions via comments.
Consequently, online crowd-prediction has become a quick and cost-effective
way to obtain estimates for a wide range of timely questions, such as
e.g.:222The median predictions at Metaculus as of 10-08-2023 are: Trump 77% -
DeSantis 15% - Others 8%; 42%; Feb 23 2027 and 19% respectively.

  * •

Who will be the Republican nominee for the 2024 presidential election?

  * •

Will the People’s Republic of China annex at least half of Taiwan before 2050?

  * •

When will the first weakly general AI system be devised, tested, and publicly
announced?

  * •

Will 1 bitcoin be worth 100,000$ or more by 2026?

Obviously, asking experts is usually preferable to questioning laypeople when
one is interested in accurate forecasts. However, crowd-prediction-platforms
like the one studied in this paper are able to massively improve upon
individual estimates by e.g. weighting forecasters that tend to perform
better. The ability to generate predictions without the need to consult costly
experts is what makes crowd-prediction attractive. Compared to other methods
of gathering crowd-predictions, which typically involve manual recruitment of
forecasters, online crowd-prediction offers a simplified and versatile
approach. Any question that can be objectively resolved can be submitted
instantly. Many organizations, such as the US intelligence community, the
Virginia Department of Health, and Our World in Data have leveraged this
method to inform their research ([100], [88], [89]). Furthermore, crowd-
prediction is a dream come true for researchers as forecasts can be generated
en masse at minimal cost. For example, crowd-prediction has successfully been
used to assess whether social science studies will fail to replicate ([67],
[71]) and as a benchmark for studying the prediction capabilities of large
language models ([95]). It is widely accepted that crowd-prediction is a
valuable tool that can support policy decisions ([101]). Consequently, crowd-
prediction is increasingly seen as evidence in absence of observable truth as
forecasts from open online crowd-prediction platforms have been featured in
ECB reports ([69]) and several news sites such as The Economist, Forbes, The
Washington Post and Vox.333See e.g. ([103]) However, it is important to
determine the reliability of crowd-prediction as a forecasting method. How
much evidence do the results from open online crowd-prediction platforms
provide? To what extent should we be confident in these estimates for
decision-making or scientific inquiry? Therefore, it is crucial to assess the
forecasting skill that crowds possess on open online platforms. There is a
large literature on crowd-prediction and judgmental forecasting ([92], [99]),
which is very favorable in regards to deploying crowd-prediction in many
contexts. However, the author is not aware of any literature investigating the
performance of online crowd-prediction relative to other forecasting methods.
This paper tries to address this literature gap by providing a first glimpse
at the crowds prediction accuracy relative to a hard-to-beat statistical
model.

This paper focuses on analyzing the forecasting skill displayed on Metaculus,
a popular crowd-prediction platform. At Metaculus, the track record on past
predictions is openly accessible.444The track record can be viewed at:
www.metaculus.com/questions/track-record/ This means that the forecasting
accuracy of Metaculus on resolved questions (questions where the outcome has
been observed) can be objectively verified. Why is the track record not
informative of forecasting skill? Forecasting accuracy, as measured by the
track record, is not to be confused with forecasting skill since forecasting
accuracy depends on the difficulty of the respective question. Forecasting
skill is the ability of a forecaster to produce accurate forecasts, given
question difficulty. An illustrative example: As of August 2023, the
republican nominee for the presidential election will likely be Trump. A naive
forecast may be that Trump will be the nominee in 3 of 4 potential outcomes.
Now consider the following question: Will the People’s Republic of China annex
at least half of Taiwan before 2050? Regime changes are rare, but the time
horizon is long, so there is a lot of uncertainty, so let the forecast be 50%.
If both forecasts are, hypothetically, perfectly accurate, then the expected
squared error of the forecaster is $0,75*0,25^{2}+0,25*0,75^{2}=0,1875$ for
the first question, but $0,5*0,5^{2}+0,5*0,5^{2}=0,25$ for the second
question. It is clear that, although the forecaster has displayed perfect
skill on both questions, due to variance in question difficulty, the
forecasting accuracy of the two forecasts is wildly different. Both
forecasting skill and question difficulty affect final scores. A track record
on a set of questions with unknown difficulty is not informative beyond the
simple fact that the forecasting method may have performed better than a
random guesser. Figure 1 depicts the situation. Forecasting accuracy is
fundamentally confounded by question difficulty. It is not sufficient to only
look at the track record of a forecaster or a forecasting platform to
determine forecasting skill.

Figure 1: Forecasting accuracy is fundamentally confounded by question
difficulty

Question difficulty is very hard to define and therefore usually unobserved.
Forecasting accuracy is observed. Forecasting skill may be observed
objectively in some cases. This analysis looks at forecasting accuracy whilst
conditioning on (i.e. controlling for) question difficulty. This is achieved
simply by conditioning on questions. That is, predictive accuracy is analyzed
on the same set of questions so that comparisons between different forecasting
methods are guaranteed to be ceteris paribus comparisons. Thereby, variance in
forecasting accuracy must come from differences in forecasting skill, allowing
judgment regarding the forecasting skill displayed by Metaculus forecasters.
Put plainly, since the difficulty of forecasting questions varies, it is very
difficult to determine forecasting skill. Consequently, comparing forecasting
methods is usually only meaningful when the underlying question for which the
forecasts are generated remains the same. A forecaster may be drastically
worse than others, yet if he systematically selects questions with more
certain outcomes (i.e. easier questions), his error may be smaller and the
forecaster more accurate.555This would lead to a situation as in Simpsons
Paradox. Conversely, hosting easier questions on crowd-prediction platforms
improves the platforms average accuracy and track record, but does not change
forecasting skill.

This paper proceeds with analyzing a set of forecasting questions concerning
exchange rates. The prediction of exchange rates has been extensively studied
in the field of economic forecasting. Notably, researchers have long sought to
improve upon the performance of the random-walk model ([85]), a simple
statistical model, but have faced challenges in doing so, as highlighted by
[93] in her comprehensive review. The random-walk model thereby serves as an
important benchmark that the crowd-prediction can be compared against,
reflecting the highest level of accuracy achievable in exchange rate
forecasts. Therefore, exchange rate questions provide a valuable opportunity
to observe and evaluate the skill of the Metaculus crowd-prediction.

The paper proceeds as follows: Section 2 briefly reviews the literature on
crowd-prediction and economic forecasting. Section 3 presents the analyzed
data. Section 4 focuses on the central research question as well as the
methodology for answering it. Section 5 discusses and further analyzes the
results. Section 6 concludes.

##  2 Literature Review

###  2.1 Crowd-prediction

The idea of crowd-prediction is not new, as it dates back to [72]. Galton
studied the accuracy of crowd-predictions by conducting a series of
experiments in which individuals guessed various quantities, such as the
weight of an ox. He found that while individual guesses differed widely, the
average of the crowd’s guesses was a remarkably accurate measure of the actual
weight of the ox. There are many ways to elicit individual forecasts and
combine them to form a consensus forecast, such as simple averages. Different
ways of distilling the wisdom of crowds have been suggested and tested ([56],
[53]). Among the most well studied are prediction markets ([54], [75]),
forecasting tournaments, and prediction polls ([55]). Crowd-prediction, in the
form of prediction markets, has been successfully employed at large companies
to aid decision-making ([66]).

Open online crowd-prediction, hereinafter just ’crowd-prediction’, is a type
of forecast that results from combining predictions made by multiple
forecasters via a shared online platform. Crowd-prediction platforms are
similar to forecasting competitions in that they involve multiple
participants, and while there are often monetary rewards involved, most
forecasters are primarily driven by the desire to establish their reputation
and prestige by winning competitions and demonstrating a history of accurate
predictions ([86]). These crowd-prediction platforms have demonstrated
impressive performance in aggregating forecasts across a wide range of
questions in recent times ([100], [101]). According to [90], the online stock
prediction community has consistently performed better than professional
analysts when it comes to forecasting stock returns. [62] find that an online
community of amateur tipsters outperformed bookmakers in real-money sports
bets, when predictions of tipsters are properly combined. Additionally, [97]
reveals that crowds have been successful in accurately predicting political
and geopolitical events. This particular application of crowd-prediction is
widely utilized due to the limited alternatives available for forecasting such
events, with crowd-knowledge being one of the few reliable options. Moreover,
the paper demonstrates that crowd-prediction performs at a similar level to
expert predictions in this domain. [77] find that crowds have been the best
source of foresight regarding Supreme Court decisions in the United States. An
important area where the crowd seems to beat statistical models is
epidemiological forecasting ([70], [84]).

However, crowd-prediction continues to be underutilized. Naturally, there is a
valid concern that some participants may be unreliable or have malicious
intentions, wanting to manipulate the resulting predictions. Nevertheless,
crowd-prediction platforms have implemented two mechanisms to address these
concerns. Firstly, by incorporating predictions from a large number of
forecasters, the impact of individual predictions is reduced. Secondly, the
track record of participants is known. When combined, predictions are weighted
based on the predictors track record. This mechanism effectively excludes poor
forecasters and plays a crucial role in ensuring the accuracy and integrity of
crowd-predictions.

###  2.2 Forecasting exchange rates

Throughout history, numerous endeavors have been made to forecast exchange
rates. These endeavors have revealed the challenges associated with predicting
exchange rates ([65], [73]). It has been observed that the random-walk without
drift is not systematically outperformed by any other (more sophisticated)
statistical model ([93]).

Predicting exchange rates is a challenging task due to the potential for
profit if future exchange rate movements were known in advance. By buying
currency at a low price and selling it at a high price, easily predictable
movements should largely vanish. Consequently, information about future
exchange rates should already be factored into current rates, leaving little
room for predictable drift ([73]). However, the behavior of exchange rates
cannot be entirely explained by economic theory ([85], [93]). Despite the
extensive research on the topic, there is still limited understanding of why
exchange rates move in the manner they do and why they may not adhere to
concepts such as interest parity ([63], [78], [68]). This phenomenon is
commonly referred to as the Meese-and-Rogoff puzzle ([85]). Although there
seem to have been modest successes at predicting exchange rates beyond random
fluctuation ([81], [58]), it remains difficult to reliably outperform the
random-walk with statistical models. As a result, the prediction produced by
the random-walk serves as a benchmark, representing the best-known approach.

While statistical and econometric models dominate much of economic
forecasting, judgmental forecasts also play a significant role. One well-known
example of judgmental forecasts is the use of forecaster surveys, such as the
’Surveys of Professional Forecasters’, which are conducted by various central
banks to gather macroeconomic expectations. The ECB Surveys of Professional
Forecasters has (when combined) been predicting macroeconomic indicators quite
well ([60]). However, traditional surveys have certain limitations compared to
online crowd-prediction. Surveys can be time-consuming, with lengthy
participant recruitment processes. Additionally, regular surveys such as the
Survey of Professional Forecasters have fixed questionnaires, preventing
flexibility in modifying questions, which hinders the attainment of real-time
forecasts. A real-time forecast shall refer to one where the event or time
horizon is fixed, but forecasts can be submitted at any point in time. Real-
time forecasts as studied in this paper are the norm at crowd-prediction
platforms, but they are rare in judgmental economic forecasting. In the case
of surveys that pertain to macroeconomic variables like GDP, inflation, and
exchange rates, they typically do not change in real-time, but are conducted
on a quarterly basis, limiting the frequency of updates. In contrast, crowd-
prediction platforms provide the advantage of continuous updates. New
predictions can be submitted at any time, offering a forecast that directly
and swiftly corresponds to events and economic shocks.666This is of course
very useful when comparing the effects of shocks or sudden changes on
expectations ([69]). [82] find evidence suggesting that certain forecasters
possess valuable insights into future exchange rates, while others do not.
[91] also observe that, on average, experts outperform amateurs in short-term
exchange rate forecasts, whilst [80] find the opposite. There are also surveys
which focus on forecasting exchange rates. These surveys, typically provided
by firms such as Consensus Economics or FX4casts, collect forecasts from
economists on a quarterly basis for specific questions such as: ”What will be
the value of the Euro (measured in USD) on January 1st, 2025?” Forecasters
participating in these surveys provide a single point prediction for these
exchange rate questions. These surveys are expensive, provide minimal
information and are hard to change, providing only point forecasts for fixed-
horizons. Crowd-prediction platforms on the other hand offer inexpensive and
continuously updated forecasts on specific questions which can be tailored to
specific needs. For instance, questions like ”Will the agriculture, forestry,
and fishing sectors contribute at least 5% to global GDP by 2030?” or ”If the
number of influenza cases increases to 5 per 1000 inhabitants, will it result
in a decrease in GDP for that year?” highlight the tailored nature of the
forecasts that can be elicited through crowd-prediction. By harnessing the
collective wisdom of diverse forecasters, crowd-prediction fills an important
gap in the availability of low-cost real-time forecasts and provides a
valuable method in the forecasters toolbox.

##  3 Data

###  3.1 Metaculus forecasting platform

Metaculus is an online forecasting platform and aggregation engine working to
improve human reasoning and coordination on topics of global importance. As a
Public Benefit Corporation, Metaculus largely provides forecasts publicly.
Metaculus features questions on a wide range of topics, with a particular
focus on pressing problems such artificial intelligence, biosecurity, climate
change, and nuclear risk. As of December 2023, over 8600 questions have been
submitted (over 4000 resolved) since the platforms inception in 2015. Over 1.5
million individual forecasts have been made on the platform by thousands of
active users. Notably, crowd-prediction on Metaculus has demonstrated strong
performance when it comes to predicting complex events, especially in the
field of epidemiological forecasting related to COVID-19 ([83], [84], [87]).

Metaculus publishes two forecasts: A time-weighted median of individual
forecasts, called the ’community prediction’, and a skill-weighted forecast,
called the ’Metaculus prediction’. Numerous studies have shown that
aggregating or combining forecasts tends to improve upon individual
predictions ([97]). Even a simple mean of the individual predictions often
yields better results than any single forecaster’s prediction. The Metaculus
prediction leverages data on individual forecasters past predictions to give
skilled forecasters more weight. Unsurprisingly, the Metaculus prediction is
on average more accurate than the simple time-weighted median.777Metaculus
chooses to disclose how exactly their aggregation mechanism works. The
Metaculus prediction is usually hidden for as long as questions are still open
for predictions. The community prediction can also be hidden for some period
after the opening of a new question. This feature may limit early groupthink,
yet disadvantages predictors which predict on questions early. Figure 2 shows
how the time-weighted median on open questions is reported to forecasters. The
prediction interface is depicted in figure 3.

Figure 2: Metaculus interface to explore how predictions on a question have
evolved Figure 3: Metaculus prediction interface for the SpaceX-question from
figure 2

Forecasters submit predictions at their leisure to gain reputation and
monetary rewards, if a prize is set on a particular question. As such, this
method for crowd-prediction is a forecasting tournament ([100]) and not to be
confused with other ways of eliciting crowd opinion such as prediction
markets.

###  3.2 Forecasting question series on exchange rates

From June 2022 to December 2022 a forecasting question series concerning
foreign exchange rates was hosted on the online crowd-prediction platform
Metaculus. The question set that is of interest for this analysis comprised of
17 questions of the kind:

> Will the currency (e.g. the Euro) depreciate by 15% or more in 2022?

Furthermore, two earlier questions of similar nature asked:

> Will the British Pound reach parity with the dollar?

All exchange rates in this study are based on the US dollar as the reference
currency. Table 1 provides a comprehensive list of the relevant exchange rate
questions that were featured on Metaculus. Question 1 to 17 refer to the 2022
question series whereas questions 18 and 19 were submitted independently.
Question 18 opened on 2016-07-09 and question 19 opened on 2022-09-29.
Moreover, it is important to note that the focus of the questions 18 and 19 is
not solely on whether the currency will experience a 15% depreciation.
Specifically, the question pertains to whether the British pound will reach
parity with the dollar, which essentially encompasses the concept of a
depreciation by a certain percentage. Therefore, the question is framed in a
slightly different manner, but with the underlying objective of understanding
potential currency depreciation. The random-walk model can be utilized to
generate forecasts for the questions 1 to 12 and 18 and 19 listed in Table 1.
The random-walk can not be applied to the questions 13 through 17 because
these currencies are pegged or the exchange rates do not float freely. This
renders the random-walk model ineffective. In these instances, judgmental
methods become especially valuable as they can offer meaningful and insightful
forecasts for the currencies under consideration.

1 | Euro to depreciate by $>15\%$ in 2022? | $k=0$
---|---|---
2 | Indonesian Rupiah to depreciate by $>15\%$ in 2022? | $k=0$
3 | Thai Baht to depreciate by $>15\%$ in 2022? | $k=0$
4 | Russian Ruble to depreciate by $>15\%$ in 2022? | $k=1$
5 | Turkish Lira to depreciate by $>15\%$ in 2022? | $k=0$
6 | Polish zloty to depreciate by $>15\%$ in 2022? | $k=0$
7 | Brazilian Real to depreciate by $>15\%$ in 2022? | $k=0$
8 | Mexican Peso to depreciate by $>15\%$ in 2022? | $k=0$
9 | Indian Rupee to depreciate by $>15\%$ in 2022? | $k=0$
10 | Pakistani Rupee to depreciate by $>15\%$ in 2022? | $k=0$
11 | Chinese Yuan Renminbi to depreciate by $>15\%$ in 2022? | $k=0$
12 | Bangladeshi Taka to depreciate by $>15\%$ in 2022? | $k=0$
13 | Argentine Peso to depreciate by $>15\%$ in 2022? | $k=1$
14 | Saudi Riyal to depreciate by $>15\%$ in 2022? | $k=0$
15 | Iranian Riyal to depreciate by $>15\%$ in 2022? | $k=0$
16 | Egyptian Pound to depreciate by $>15\%$ in 2022? | $k=1$
17 | Nigerian Naira to depreciate by $>15\%$ in 2022? | $k=0$
18 | British Pound to reach market parity with the dollar by 2017? | $k=0$
19 | British Pound to reach market parity with the dollar by 2023? | $k=0$
Table 1: Studied questions from the crowd-forecasting platform

The questions in this study were classified as resolved ’Yes’ ($k=1$) if the
depreciation threshold was reached, or ’No’ ($k=0$) if the depreciation
threshold was not reached by the time the questions closed on December 31st
2022. Over a period of six months, a total of 61 amateur forecasters888Since
this study is purely observational and not conducted in a lab-setting, no
additional information about forecasters is available. participated in this
study, collectively submitting 2453 individual predictions. On average, there
were 144 total predictions per question.999The question series can be explored
online at
www.metaculus.com/questions/11505/economic-trouble-15-currency-depreciation/.
As an incentive for their participation, forecasters received a small monetary
compensation based on the accuracy of their predictions.101010The forecasters
were rewarded based on their squared error. The exact amount is hard to
determine. A 2500$ prize pool was awarded to accurate forecasts across a
larger set of 64 questions, which included the questions from 1 to 17. The
average amount of money awarded per predictor was below 10$. However, only a
top subset of forecasters could choose to claim payments. The predictions on
questions 18 and 19 were not monetarily incentivized.

##  4 Empirical Analysis

The central research question is:

> How accurate are crowd-sourced exchange rate forecasts?

Accuracy shall be measured via the squared error, often also known as the
brier score ([61]). The accuracy of the crowd-prediction shall be compared
with the prediction from a statistical model: the random-walk (see equation
1).

$\displaystyle\text{squared error}=y(t)$
$\displaystyle=(1-k)*p_{t}^{2}+k*(1-p_{t})^{2}$ with $\displaystyle
k\in\mathbf{N}\ \\{0,1\\}$

The use of the squared error as a measure serves several purposes within this
study. Firstly, the squared error is a proper scoring rule. Proper scoring
rules are a class of mechanisms in which forecasters maximize their expected
scores by providing the most accurate forecast ([74]). Therefore, there is no
incentive for forecasters to misreport their true beliefs. The squared error
is widely recognized as one of the most commonly used scoring rules. It is a
standard measure employed in various forecasting contexts, including the
scoring of forecasters’ predictions within the Metaculus question series
studied here. By using the squared error, this study enables accurate and fair
evaluation of the performance of both the crowd-prediction and random-walk
models prediction.

In order to evaluate the performance of the crowd-prediction and the random-
walk model, forecasts were compared on a question-by-question basis. For each
individual question and corresponding exchange rate time series, the squared
error of the crowd-prediction and the random-walk model’s predictions were
computed. Errors of the crowd-prediction and random-walk were separately
summed up and divided by the number of questions to arrive at the mean squared
error.111111Since some questions resolved early, the number of questions
changes across time. This gives rise to figure 4, a comparison of both
forecasting methods based on the squared error across all questions.

The ’Metaculus prediction’, which is accessible via the API that Metaculus
provides, was used to represent the crowd-prediction because it provides more
accurate forecasts than the time-weighted median. The predictions of the
random-walk were generated using a numerical approach. The rest of this
section is therefore devoted to outlining how the random-walk predictions were
generated. A pseudo-out-of-sample rolling-window forecasting approach was used
to ensure a fair comparison between the two methods.

The questions addressed in this study adhere to a mathematical framework that
involves assessing whether a time series will exceed a predetermined
threshold. This framework can be likened to the concept of a ”gamblers ruin”
([76]), where a gambler starts with a certain amount of money and the
probability of losing it all (signifying the end of the game) can be
calculated based on the number of rounds played. In this context, the aim is
to determine the probability of a exchange rate time series reaching a
specific value (15% depreciation) within a given timeframe (6 months). This is
opposed to the more common forecasting task of pinpointing the exact timing of
when such an event will occur. The random-walk is a model that simply projects
the normal distribution with the historic standard deviation into the future.
$x_{t}$ shall be the value of a currency at time $t$, as measured in US-
dollars. The next time steps value is defined as the current value plus some
random change ($\epsilon$), which is normally distributed with the historic
standard deviation of the exchange rate series. Therefore, the random-walk is
an ARIMA(0,1,0) process.

$x_{t+1}=x_{t}+\epsilon_{t},\quad\text{with}\
\epsilon_{t}\sim\mathcal{N}(0,\sigma_{h}^{2})$ (1)

Since the concept of the ”gamblers ruin” is researched, it is possible to
derive a closed-form-solution that returns the probability of a currency
depreciating below a certain threshold, depending on the time remaining, the
assumed future variance of the exchange rate and the current value. However,
deriving the random-walk predictions with numerical methods is easier to
implement, adapt and build upon. Therefore, a numerical approach was used to
forecast exchange rate deviations. To make these predictions, financial data
on exchange rates from Yahoo Finance was gathered using the quantmod-package
in `R`.121212The data used to resolve the crowd-prediction questions was
sourced from xe.com. In other words, the crowd-prediction is based on other
financial data then the random-walk-prediction. The difference between the two
is small and negligible in comparison to the exchange rate movements and
uncertainty in exchange rate expectations displayed. Exchange rate data was
collected on a daily basis from January 5, 2022, until December 31, 2022,
which encompassed the resolve time of the questions. However, there was an
exception for the questions specific to the British Pound, which opened
earlier. For these cases, exchange rate data was gathered starting from
January 1, 2015. The predictions of the random-walk-model are generated as
follows:131313Exchange rate fluctuations are not observed on weekends - a fact
that is brushed over in the Monte-Carlo-Simulation. This is insofar relevant
as currencies can not depreciate below the threshold on weekends. If a
currency is close to the depreciation barrier, but there are no more days in
which the market is open in the year, the algorithm would still assume a
potential depreciation of the currency on each day. However, this situation
has not turned up in this analysis.

Input: Historical exchange rate time series; Date from which to forecast

Output: Probability of currency depreciation

Step 1: The historical variance $\sigma_{h}^{2}$ of the exchange rate is
calculated based on the available data for that specific day. That is, if the
probability of the Euro depreciating below the specified threshold is to be
predicted for Oct. 30th 2022, then the historical standard deviation is
calculated using the exchange rate values (variance) from January 5th on up to
the 29th of October.;

Step 2: Ten thousand paths of the required length (up until December 31) are
generated using a Monte-Carlo simulation. The next days value is determined by
equation 1, whereas $\epsilon$ is a random draw from a normal distribution
with a variance equal to the historical variance $\sigma_{h}^{2}$ computed in
the previous step.;

Step 3: To derive the probability that a currency will depreciate below the
specified threshold, the number of simulated paths where this occurs (at some
point), is divided by 10000 (the total number of paths).;

Algorithm 1 The random-walks prediction

Most importantly, the approach arrives at a probability or forecast that
anyone would have been able to generate in real time. That is, only
information available at the specific day is used to make forecasts. The
forecasts reflect pseudo-out-of-sample-performance of the random-walk.
Thereby, the forecasts generated by this method provide a fair benchmark for
the crowd-prediction.

##  5 Results

The individual predictions of both the crowd-prediction as well as the random-
walk‘s prediction, are plotted in the supplementary material figures A2 to
A14. Figure 4 displays the mean squared error for questions 1 to 11, 18 and
19. The black line represents the evolving error of crowd-predictions over
time (x-axis), while the red line shows the error of the random-walk model.
Initially, the error of crowd-predictions fluctuates substantially due to the
limited number of predictions available at the start, resulting in a small
sample size for the forecast combination method. As more predictions are
received, the combined predictions and the error tend to change considerably.
It is worth noting that a mean squared error below $0.25$ indicates that the
forecasts outperform random guessing, since a maximally uncertain prediction
(predict 50%) would yield a squared error of $0.25$.

Figure 4: Accuracy of the Metaculus Prediction compared to the predictions
made by the random-walk

In general, both errors follow a similar progression over time. However, the
random-walk model is more accurate, i.e. less erroneous than the crowd-
prediction. This outcome is a null-result as the random-walk serves as the
benchmark for forecasting accuracy. Nevertheless, the crowd-prediction
performs only slightly worse. Overall, both the crowd-prediction and the
random-walk model do much better than random guessing. Both errors converges
towards zero as time progresses, since the observable outcome inches closer,
i.e. the forecast horizon becomes smaller.

One may hypothesize now that the random-walk and crowd-prediction are
relatively similar all around. However, noteworthy distinctions emerge when
examining specific questions.141414Note that the graphs A2 to A14 show the
actual prediction, not the error. Since the error directly follows from the
predictions, this representation is more user-friendly. The predictions of the
crowd and random-walk model display striking similarity in the case of the
Brazilian Real and the Pakistani Rupee, suggesting that forecasters may have
employed a random-walk approach. Similarly, predictions for the Euro also
exhibit relatively close alignment. Why didn’t the forecasters simply always
rely on the random-walk model? It appears that many forecasters may not have
been aware of the superior performance of the random-walk model, a fact that
is not common knowledge and is only discovered after some research. It is
important to note that most forecasters on Metaculus cover a wide range of
topics and may not be familiar with the academic literature in every relevant
domain where they may make predictions. Given the small amount of money at
stake, the forecasters may have not been willing to exert too much costly
effort, even if that would have gained them higher predictive accuracy.
Lastly, even if forecasters possessed information that would allow them to
improve upon the random-walk, they should take that information to foreign
exchange markets instead of crowd-prediction platforms. Some forecasters may
have used the random-walk model as an aid for their decision-making process,
but most likely relied on their subjective judgment to save time and effort.
Two broader observations hold true across multiple questions:

  1. 1.

The random-walk makes more ”extreme” predictions and the crowd more ”modest”
predictions. The crowd-prediction is usually closer to 50% than the random-
walks prediction. The random-walk prediction is often near zero for some
currencies.

  2. 2.

Random-walk-prediction and crowd-prediction mostly move in parallel. However,
the response of the crowd to exchange rate movements, as embodied by a change
in the random-walk forecast, is slightly lagged. The crowd-prediction reacts
slower to news.

These findings can be readily explained and are consistent with economic
theory. When the crowd is less accurate or confident about outcomes, it should
report probabilities that are closer to 50%, reflecting a lack of foresight.
On the other hand, the random-walk prediction relies on daily exchange rate
data and swiftly updates as the rates fluctuate. In contrast, the crowd
consists of individuals who update their forecasts every now and then, causing
their predictions to respond more slowly to changes in exchange rates.
Furthermore, there is evidence that even professional forecasters seem not to
update their predictions in a fully Bayesian manner, making forecasts of
macroeconomic indicators predictably biased ([64]). The fact that the random-
walk predicts effectively zero chance for currency depreciation for some
currencies is a bit suspicious, since that means that the currency is unlikely
to ever depreciate by 15% in thousands of counterfactual scenarios. Since the
random-walk consist of nothing but random draws with a pre-determined
variance, this result mildly implies that, if the volatility of the currency
is stable, a currency depreciation of 15% in a few months is unlikely to ever
occur in hundreds of years. This seems unrealistic. However, it is not
possible to evaluate such low probability estimates given the short time frame
of the study. The low probability estimates of currency depreciation lowers
the random-walks error in this time frame, but perhaps not in the long run.
This study is limited in the sense that it looks only at a 6 month time frame.
Arguably, results could look very different, if the time frame were much
longer. There are two exceptions to the aforementioned patterns:

  1. 1.

Forecasters seem to have overestimated the likelihood of the Turkish Lira to
depreciate. The crowd-prediction for the Turkish Lira is higher than the
random-walk-prediction.

  2. 2.

The crowd-prediction for the Polish zloty is largely lower than the random-
walk prediction, although both are below 50%. Since the currency did not
depreciate by 15%, this is the only question where the crowd-prediction
clearly beats the random-walk.

The results beg another post-hoc question. Is the crowd-prediction a good
estimator of the random-walk prediction? If the random-walk prediction acts as
a proxy for ground truth, and the crowd-prediction well approximates that,
this would mean that the crowd-prediction is a more noisy, but otherwise
approximately unbiased estimate of the actual risk of currency depreciation.
To answer this question, a simple OLS regression was performed. If the crowd-
prediction is a unbiased estimate of the random-walks prediction, than the
slope of the regression line should be 1 and the intercept 0. This shall serve
as a null hypothesis. Equation 2 shows the setup, where $x$ corresponds to the
prediction made by the random-walk. Table 2 contains results.

$x=\beta_{0}+\beta_{1}\text{crowd}+\psi$ (2)

Table 2: OLS regression results | Estimate | Std. Error |  $t$-value |  $p$-value
---|---|---|---|---
Intercept | -0.01199** | 0.00370 | -3.242 | 0.00121
crowd | 0.84076*** | 0.01248 | -12.75962 | $<10^{-6}$
$R^{2}$ | 0.6998 |  |  |

The results from table 2 show that the crowd-prediction is significantly
different from the null of being an unbiased estimator of the random-walks
prediction. The intercept is significantly negative, yet very small. This can
be explained via the fact that the random-walk is often near zero, when the
crowd-prediction is low, but not as low. As discussed above, given the short
time frame of the study, it is unclear whether this is a bias in reality. The
slope of regression line is significantly below 1. This indicates that the
crowd-prediction is on average predictably too high. This can to some degree
explained via the fact that the crowd updates slower and this seems to largely
have led to higher predictions.

In general, it appears that amateur forecasters have demonstrated considerable
insights into exchange rate movements. In absence of the random-walks
prediction (such as for pegged currencies) the crowd-prediction could arguably
have been utilized instead to inform decision-making.151515Of course, it would
have also been possible to combine the Metaculus prediction and the random-
walk prediction. If the predictions contain complementary information, this
should yield a better forecast. However, using a standard combination approach
([94]) the combined forecast did not outperform the random-walk. Figure A1
shows the mean squared error of the crowd-prediction on questions 13 to 17,
i.e. the questions to which the random-walk is not applicable. As the figure
shows, the error is substantially below $0.25$ which indicates that humans had
some insight into exchange rate movements of currencies that are pegged or
otherwise do not float freely. However, due to the limited sample size, one
should be careful with drawing conclusions from these observations. The crowd-
prediction could surely be improved by including experts, increasing payoffs
or otherwise adding more forecasters. The bottom line is that even a small
crowd that is barely incentivized and not manually recruited or selected
provides reasonably accurate exchange rate forecasts and therefore displays
considerable forecasting skill.

Although this study finds that the random-walk did better than the crowd,
judgmental forecasts should not be discarded for forecasting exchange rates
and related areas. Experts may have some knowledge regarding the current
situation that might be valuable for forecasting exchange rates. Judgmental
forecasting using experts is a common practice alongside statistical forecasts
for eliciting beliefs concerning exchange rates. Although statistical models
are the go-to-tools for economic forecasting, judgmental forecast have a
number of key advantages that are not yet fully studied or used in practice:

  1. 1.

Situational awareness

It is worth highlighting that during certain periods, such as during a crisis,
judgmental methods have the potential to produce more accurate forecasts. This
is due to their ability to leverage situational awareness and expert
knowledge, allowing them to account for critical circumstances and anomalies
that statistical models are unable to incorporate. When the recent past is a
poor guide for the future, judgmental approaches can play a crucial role in
generating insightful predictions.

  2. 2.

Conditional forecasts

Human judgments allows to condition forecasts, i.e. to envision a world in
which a certain outcome is fixed. For example:

What is the probability that the value of the US-Dollar will fall by more than
5%?

1) If a democratic candidate is elected? = $P(\text{5\%
depreciation}|\text{Democrat=1})$

2) If a republican candidate is elected? = $P(\text{5\%
depreciation}|\text{Democrat=0})$

Whereas models for economic forecasting are increasingly upgraded to condition
on variables - conditional forecasts on specific scenarios (such as
presidential elections) can most easily be gained by expert elicitation and
have been applied in forecasting economic indicators ([59]).

  3. 3.

Broad applicability

Whereas statistical models, by definition, can only be employed for a certain
set of questions, judgmental forecasts can be applied much more broadly. For
example, judgmental forecasts are available for all currencies and at all
times. This is not the case for the random-walk. Since the value of some
currencies does not float freely, but is pegged against a different currency
or otherwise manipulated, the random-walk would not provide any meaningful
prediction. An expert forecaster however, can use his knowledge to assess
whether a currency is likely to be unpegged, whether it will depreciate or
appreciate.

Whilst the crowd is worse in this example than the statistical model, the
literature review outlines other domains where the crowd outperformed the best
statistical models. This also shows that it may be difficult to generalize the
findings in this study to other areas. Nonetheless, absent other information,
the results of this study present evidence in favor of the crowd-prediction at
Metaculus possessing considerable forecasting skill.

##  6 Conclusion and further research

Crowd-prediction can be viewed as a highly versatile and accessible
forecasting tool. However, the level of confidence we can place in crowd-
prediction warrants examination. This paper aims to provide insights into this
question. It is important to note that the accuracy of crowd-prediction is
influenced by numerous factors such as the skill of individual forecasters.
The influence of such factors is difficult to determine. Nevertheless, this
analysis allows us to assess the forecasting skill of the crowd-prediction on
a set of exchange rate questions by comparing it to a hard-to-beat statistical
model.

The crowd-prediction demonstrated a noteworthy level of accuracy in
forecasting future exchange rates, despite the limited number of forecasters
and rewards. However, it exhibited a higher error than the statistical model.
In general, the findings support the use of crowd-prediction for forecasting
questions in economics. However, if established statistical models are
available, these may very well be significantly better. The results further
indicate that crowd-prediction is effective in forecasting non-floating
currencies. There are multiple avenues for future research. Some of the most
promising open research questions seem to be in comparing experts and crowd-
prediction as well as tackling the more fundamental problem of comparing
forecasting skill across different question sets. Furthermore, forecasting
competitions and crowd-prediction platforms are currently not fully incentive
compatible ([104], [96]). Future research should inform how to provide better
incentives for accurate forecasts on crowd-prediction platforms.

## References

  * [1] J Scott Armstrong  “Combining forecasts”  Springer, 2001
  * [2] Kenneth J Arrow et al.  “The Promise of Prediction Markets”  In _Science_ 320.5878 American Association for the Advancement of Science, 2008, pp. 877–878
  * [3] Pavel Atanasov et al.  “Distilling the Wisdom of Crowds: Prediction markets vs. Prediction polls”  In _Management Science_ 63.3 INFORMS, 2017, pp. 691–706
  * [4] Pavel Atanasov, Jens Witkowski, Barbara Mellers and Philip Tetlock  “Crowd Prediction Systems: Markets, Polls, and Elite Forecasters”  In _Proceedings of the 23rd ACM Conference on Economics and Computation_ , 2022, pp. 1013–1014
  * [5] John M Bates and Clive WJ Granger  “The Combination of Forecasts”  In _Journal of the Operational Research Society_ 20.4 Taylor & Francis, 1969, pp. 451–468
  * [6] Joscha Beckmann, Gary Koop, Dimitris Korobilis and Rainer Alexander Schüssler  “Exchange rate predictability and dynamic Bayesian learning”  In _Journal of Applied Econometrics_ 35.4 Wiley Online Library, 2020, pp. 410–421
  * [7] Travis J Berge, Andrew C Chang and Nitish R Sinha  “Evaluating the conditionality of judgmental forecasts”  In _International Journal of Forecasting_ 35.4 Elsevier, 2019, pp. 1627–1635
  * [8] Carlos Bowles et al.  “The ECB survey of professional forecasters (SPF)-A review after eight years experience”  In _ECB occasional paper_ , 2007
  * [9] Glenn W Brier  “Verification of forecasts expressed in terms of probability”  In _Monthly weather review_ 78.1, 1950, pp. 1–3
  * [10] Alasdair Brown and J. Reade  “The wisdom of amateur crowds: Evidence from an online community of sports tipsters”  In _European Journal of Operational Research_ 272.3, 2019, pp. 1073–1081  DOI: https://doi.org/10.1016/j.ejor.2018.07.015
  * [11] Menzie D Chinn and Guy Meredith  “Monetary policy and long-horizon uncovered interest parity”  In _IMF staff papers_ 51.3 Springer, 2004, pp. 409–430
  * [12] Olivier Coibion and Yuriy Gorodnichenko  “Information Rigidity and the Expectations Formation Process: A Simple Framework and New Facts”  In _American Economic Review_ 105.8 American Economic Association 2014 Broadway, Suite 305, Nashville, TN 37203, 2015, pp. 2644–2678
  * [13] W Bradford Cornell and J Kimball Dietrich  “The Efficiency of the Market for Foreign Exchange Under Floating Exchange Rates”  In _The Review of Economics and Statistics_ JSTOR, 1978, pp. 111–120
  * [14] Bo Cowgill and Eric Zitzewitz  “Corporate prediction markets: Evidence from Google, Ford, and Firm X”  In _The Review of Economic Studies_ 82.4 Oxford University Press, 2015, pp. 1309–1341
  * [15] Anna Dreber et al.  “Using prediction markets to estimate the reproducibility of scientific research”  In _Proceedings of the National Academy of Sciences_ 112.50 National Acad Sciences, 2015, pp. 15343–15347
  * [16] Charles Engel and Kenneth D West  “Exchange Rates and Fundamentals”  In _Journal of Political Economy_ 113.3 The University of Chicago Press, 2005, pp. 485–517
  * [17] European Central Bank  “Rotation towards normality – the impact of COVID-19 vaccine-related news on global financial markets”, ECB Economic Bulletin Issue 1, 2021  URL: https://www.ecb.europa.eu/pub/economic-bulletin/html/eb202101.en.html
  * [18] David C. Farrow et al.  “A human judgment approach to epidemiological forecasting”  In _PLoS computational biology_ 13.3, 2017  DOI: 10.1371/journal.pcbi.1005248
  * [19] Eskil Forsell et al.  “Predicting replication outcomes in the Many Labs 2 study”  In _Journal of Economic Psychology_ 75 Elsevier, 2019, pp. 102117
  * [20] Francis Galton  “Vox populi”  In _Nature_ 75.1949 Springer ScienceBusiness Media LLC, 1907, pp. 450–451
  * [21] Ian H Giddy and Gunter Dufey  “The Random Behavior of Flexible Exchange Rates: Implications for Forecasting”  In _Journal of International Business Studies_ JSTOR, 1975, pp. 1–32
  * [22] Tilmann Gneiting and Adrian E Raftery  “Strictly Proper Scoring Rules, Prediction, and Estimation”  In _Journal of the American Statistical Association_ 102.477 Taylor & Francis, 2007, pp. 359–378
  * [23] Robin Hanson  “Combinatorial information market design”  In _Information Systems Frontiers_ 5 Springer, 2003, pp. 107–119
  * [24] Guy Katriel  “Gambler’s ruin probability—A general formula”  In _Statistics & Probability Letters_ 83.10 Elsevier, 2013, pp. 2205–2210
  * [25] Daniel Martin Katz, Michael James Bommarito II and Josh Blackman  “Crowdsourcing accurately and robustly predicts Supreme Court decisions”  In _arXiv preprint arXiv:1712.03846_ , 2017
  * [26] Lutz Kilian and Mark P Taylor  “Why is it so difficult to beat the random walk forecast of exchange rates?”  In _Journal of International Economics_ 60.1 Elsevier, 2003, pp. 85–107
  * [27] Michael Lawrence, Paul Goodwin, Marcus O’Connor and Dilek Önkal  “Judgmental forecasting: A review of progress over the last 25 years”  In _International Journal of Forecasting_ 22.3 Elsevier, 2006, pp. 493–518
  * [28] Johannes Leitner and Reinhard Schmidt  “A systematic comparison of professional exchange rate forecasts with the judgemental forecasts of novices”  In _Central European Journal of Operations Research_ 14, 2006, pp. 87–102  DOI: 10.1007/s10100-006-0161-x
  * [29] Jiahan Li, Ilias Tsiakas and Wei Wang  “Predicting Exchange Rates Out of Sample: Can Economic Fundamentals Beat the Random Walk?”  In _Journal of Financial Econometrics_ 13.2 Oxford University Press, 2015, pp. 293–341
  * [30] Ronald MacDonald, Lukas Menkhoff and Rafael R Rebitzky  “Exchange Rate Forecasters’ Performance: Evidence of Skill?”  In _CESifo Working Paper Series_ , 2009
  * [31] Thomas McAndrew, Juan Cambeiro and Tamay Besiroglu  “Aggregating human judgment probabilistic predictions of the safety, efficacy, and timing of a COVID-19 vaccine”  In _Vaccine_ 40.15 Elsevier, 2022, pp. 2331–2341
  * [32] Thomas McAndrew et al.  “Chimeric forecasting: combining probabilistic predictions from computational models and human judgment”  In _BMC Infectious Diseases_ 22.1 Springer, 2022, pp. 833
  * [33] Richard A Meese and Kenneth Rogoff  “Empirical exchange rate models of the seventies: Do they fit out of sample?”  In _Journal of International Economics_ 14.1-2 Elsevier, 1983, pp. 3–24
  * [34] Barbara Mellers et al.  “Psychological Strategies for Winning a Geopolitical Forecasting Tournament”  In _Psychological Science_ 25.5 Sage Publications Sage CA: Los Angeles, CA, 2014, pp. 1106–1115
  * [35] Metaculus  “A preliminary look at metaculus and expert forecasts”, 2020  URL: https://www.metaculus.com/news/2020/06/02/LRT/
  * [36] Metaculus  “The Keep Virginia Safe Tournament 2021/22 Project Summary”, 2022  URL: https://www.metaculus.com/notebooks/11162/the-keep-virginia-safe-tournament-202122-project-summary/
  * [37] Metaculus  “Forecasting Our World in Data: The Next 100 Years”, 2023  URL: https://www.metaculus.com/notebooks/14965/forecasting-our-world-in-data-the-next-100-years/
  * [38] Michael Nofer  “Are Crowds on the Internet Wiser than Experts?–The Case of a Stock Prediction Community”  In _The Value of Social Media for Predicting Stock Returns: Preconditions, Instruments and Performance Analysis_ Springer, 2015, pp. 27–61
  * [39] Dilek Önkal, J Frank Yates, Can Simga-Mugan and Şule Öztin  “Professional vs. amateur judgment accuracy: The case of foreign exchange rates”  In _Organizational behavior and human decision processes_ 91.2 Elsevier, 2003, pp. 169–185
  * [40] Fotios Petropoulos et al.  “Forecasting: theory and practice”  In _International Journal of Forecasting_ 38.3 Elsevier, 2022, pp. 705–871
  * [41] Barbara Rossi  “Exchange Rate Predictability”  In _Journal of Economic Literature_ 51.4 American Economic Association, 2013, pp. 1063–1119
  * [42] Ville A Satopää et al.  “Combining multiple probability predictions using a simple logit model”  In _International Journal of Forecasting_ 30.2 Elsevier, 2014, pp. 344–356
  * [43] Philipp Schoenegger and Peter S Park  “Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament”  In _arXiv preprint arXiv:2310.13014_ , 2023
  * [44] Nuño Sempere and Alex Lawsen  “Alignment problems with current forecasting platforms”  In _arXiv preprint arXiv:2106.11248_ , 2021
  * [45] Lennart Sjöberg  “Are all crowds equally wise? A comparison of political election forecasts by experts and the public”  In _Journal of Forecasting_ 28.1 Wiley Online Library, 2009, pp. 1–18
  * [46] James Surowiecki  “The wisdom of crowds”  Anchor, 2005
  * [47] Philip E Tetlock and Dan Gardner  “Superforecasting: The Art and Science of Prediction”  Random House, 2016
  * [48] Philip E Tetlock, Barbara A Mellers, Nick Rohrbaugh and Eva Chen  “Forecasting Tournaments: Tools for Increasing Transparency and Improving the Quality of Debate”  In _Current Directions in Psychological Science_ 23.4 Sage Publications CA, 2014, pp. 290–295
  * [49] Philip E Tetlock, Barbara A Mellers and J Peter Scoblic  “Bringing probability judgments into policy debates via forecasting tournaments”  In _Science_ 355.6324 American Association for the Advancement of Science, 2017, pp. 481–483
  * [50] Lyle Ungar et al.  “The Good Judgment Project: A Large Scale Test of Different Methods of Combining Expert Predictions”  In _2012 AAAI Fall Symposium Series_ , 2012  Citeseer
  * [51] “What would humans do in a world of super-AI?”  In _The Economist_ , 2023  URL: www.economist.com/finance-and-economics/2023/05/23/what-would-humans-do-in-a-world-of-super-ai
  * [52] Jens Witkowski et al.  “Incentive-Compatible Forecasting Competitions”  In _Management Science_ 69.3 INFORMS, 2023, pp. 1354–1374

## References

  * [53] J Scott Armstrong  “Combining forecasts”  Springer, 2001
  * [54] Kenneth J Arrow et al.  “The Promise of Prediction Markets”  In _Science_ 320.5878 American Association for the Advancement of Science, 2008, pp. 877–878
  * [55] Pavel Atanasov et al.  “Distilling the Wisdom of Crowds: Prediction markets vs. Prediction polls”  In _Management Science_ 63.3 INFORMS, 2017, pp. 691–706
  * [56] Pavel Atanasov, Jens Witkowski, Barbara Mellers and Philip Tetlock  “Crowd Prediction Systems: Markets, Polls, and Elite Forecasters”  In _Proceedings of the 23rd ACM Conference on Economics and Computation_ , 2022, pp. 1013–1014
  * [57] John M Bates and Clive WJ Granger  “The Combination of Forecasts”  In _Journal of the Operational Research Society_ 20.4 Taylor & Francis, 1969, pp. 451–468
  * [58] Joscha Beckmann, Gary Koop, Dimitris Korobilis and Rainer Alexander Schüssler  “Exchange rate predictability and dynamic Bayesian learning”  In _Journal of Applied Econometrics_ 35.4 Wiley Online Library, 2020, pp. 410–421
  * [59] Travis J Berge, Andrew C Chang and Nitish R Sinha  “Evaluating the conditionality of judgmental forecasts”  In _International Journal of Forecasting_ 35.4 Elsevier, 2019, pp. 1627–1635
  * [60] Carlos Bowles et al.  “The ECB survey of professional forecasters (SPF)-A review after eight years experience”  In _ECB occasional paper_ , 2007
  * [61] Glenn W Brier  “Verification of forecasts expressed in terms of probability”  In _Monthly weather review_ 78.1, 1950, pp. 1–3
  * [62] Alasdair Brown and J. Reade  “The wisdom of amateur crowds: Evidence from an online community of sports tipsters”  In _European Journal of Operational Research_ 272.3, 2019, pp. 1073–1081  DOI: https://doi.org/10.1016/j.ejor.2018.07.015
  * [63] Menzie D Chinn and Guy Meredith  “Monetary policy and long-horizon uncovered interest parity”  In _IMF staff papers_ 51.3 Springer, 2004, pp. 409–430
  * [64] Olivier Coibion and Yuriy Gorodnichenko  “Information Rigidity and the Expectations Formation Process: A Simple Framework and New Facts”  In _American Economic Review_ 105.8 American Economic Association 2014 Broadway, Suite 305, Nashville, TN 37203, 2015, pp. 2644–2678
  * [65] W Bradford Cornell and J Kimball Dietrich  “The Efficiency of the Market for Foreign Exchange Under Floating Exchange Rates”  In _The Review of Economics and Statistics_ JSTOR, 1978, pp. 111–120
  * [66] Bo Cowgill and Eric Zitzewitz  “Corporate prediction markets: Evidence from Google, Ford, and Firm X”  In _The Review of Economic Studies_ 82.4 Oxford University Press, 2015, pp. 1309–1341
  * [67] Anna Dreber et al.  “Using prediction markets to estimate the reproducibility of scientific research”  In _Proceedings of the National Academy of Sciences_ 112.50 National Acad Sciences, 2015, pp. 15343–15347
  * [68] Charles Engel and Kenneth D West  “Exchange Rates and Fundamentals”  In _Journal of Political Economy_ 113.3 The University of Chicago Press, 2005, pp. 485–517
  * [69] European Central Bank  “Rotation towards normality – the impact of COVID-19 vaccine-related news on global financial markets”, ECB Economic Bulletin Issue 1, 2021  URL: https://www.ecb.europa.eu/pub/economic-bulletin/html/eb202101.en.html
  * [70] David C. Farrow et al.  “A human judgment approach to epidemiological forecasting”  In _PLoS computational biology_ 13.3, 2017  DOI: 10.1371/journal.pcbi.1005248
  * [71] Eskil Forsell et al.  “Predicting replication outcomes in the Many Labs 2 study”  In _Journal of Economic Psychology_ 75 Elsevier, 2019, pp. 102117
  * [72] Francis Galton  “Vox populi”  In _Nature_ 75.1949 Springer ScienceBusiness Media LLC, 1907, pp. 450–451
  * [73] Ian H Giddy and Gunter Dufey  “The Random Behavior of Flexible Exchange Rates: Implications for Forecasting”  In _Journal of International Business Studies_ JSTOR, 1975, pp. 1–32
  * [74] Tilmann Gneiting and Adrian E Raftery  “Strictly Proper Scoring Rules, Prediction, and Estimation”  In _Journal of the American Statistical Association_ 102.477 Taylor & Francis, 2007, pp. 359–378
  * [75] Robin Hanson  “Combinatorial information market design”  In _Information Systems Frontiers_ 5 Springer, 2003, pp. 107–119
  * [76] Guy Katriel  “Gambler’s ruin probability—A general formula”  In _Statistics & Probability Letters_ 83.10 Elsevier, 2013, pp. 2205–2210
  * [77] Daniel Martin Katz, Michael James Bommarito II and Josh Blackman  “Crowdsourcing accurately and robustly predicts Supreme Court decisions”  In _arXiv preprint arXiv:1712.03846_ , 2017
  * [78] Lutz Kilian and Mark P Taylor  “Why is it so difficult to beat the random walk forecast of exchange rates?”  In _Journal of International Economics_ 60.1 Elsevier, 2003, pp. 85–107
  * [79] Michael Lawrence, Paul Goodwin, Marcus O’Connor and Dilek Önkal  “Judgmental forecasting: A review of progress over the last 25 years”  In _International Journal of Forecasting_ 22.3 Elsevier, 2006, pp. 493–518
  * [80] Johannes Leitner and Reinhard Schmidt  “A systematic comparison of professional exchange rate forecasts with the judgemental forecasts of novices”  In _Central European Journal of Operations Research_ 14, 2006, pp. 87–102  DOI: 10.1007/s10100-006-0161-x
  * [81] Jiahan Li, Ilias Tsiakas and Wei Wang  “Predicting Exchange Rates Out of Sample: Can Economic Fundamentals Beat the Random Walk?”  In _Journal of Financial Econometrics_ 13.2 Oxford University Press, 2015, pp. 293–341
  * [82] Ronald MacDonald, Lukas Menkhoff and Rafael R Rebitzky  “Exchange Rate Forecasters’ Performance: Evidence of Skill?”  In _CESifo Working Paper Series_ , 2009
  * [83] Thomas McAndrew, Juan Cambeiro and Tamay Besiroglu  “Aggregating human judgment probabilistic predictions of the safety, efficacy, and timing of a COVID-19 vaccine”  In _Vaccine_ 40.15 Elsevier, 2022, pp. 2331–2341
  * [84] Thomas McAndrew et al.  “Chimeric forecasting: combining probabilistic predictions from computational models and human judgment”  In _BMC Infectious Diseases_ 22.1 Springer, 2022, pp. 833
  * [85] Richard A Meese and Kenneth Rogoff  “Empirical exchange rate models of the seventies: Do they fit out of sample?”  In _Journal of International Economics_ 14.1-2 Elsevier, 1983, pp. 3–24
  * [86] Barbara Mellers et al.  “Psychological Strategies for Winning a Geopolitical Forecasting Tournament”  In _Psychological Science_ 25.5 Sage Publications Sage CA: Los Angeles, CA, 2014, pp. 1106–1115
  * [87] Metaculus  “A preliminary look at metaculus and expert forecasts”, 2020  URL: https://www.metaculus.com/news/2020/06/02/LRT/
  * [88] Metaculus  “The Keep Virginia Safe Tournament 2021/22 Project Summary”, 2022  URL: https://www.metaculus.com/notebooks/11162/the-keep-virginia-safe-tournament-202122-project-summary/
  * [89] Metaculus  “Forecasting Our World in Data: The Next 100 Years”, 2023  URL: https://www.metaculus.com/notebooks/14965/forecasting-our-world-in-data-the-next-100-years/
  * [90] Michael Nofer  “Are Crowds on the Internet Wiser than Experts?–The Case of a Stock Prediction Community”  In _The Value of Social Media for Predicting Stock Returns: Preconditions, Instruments and Performance Analysis_ Springer, 2015, pp. 27–61
  * [91] Dilek Önkal, J Frank Yates, Can Simga-Mugan and Şule Öztin  “Professional vs. amateur judgment accuracy: The case of foreign exchange rates”  In _Organizational behavior and human decision processes_ 91.2 Elsevier, 2003, pp. 169–185
  * [92] Fotios Petropoulos et al.  “Forecasting: theory and practice”  In _International Journal of Forecasting_ 38.3 Elsevier, 2022, pp. 705–871
  * [93] Barbara Rossi  “Exchange Rate Predictability”  In _Journal of Economic Literature_ 51.4 American Economic Association, 2013, pp. 1063–1119
  * [94] Ville A Satopää et al.  “Combining multiple probability predictions using a simple logit model”  In _International Journal of Forecasting_ 30.2 Elsevier, 2014, pp. 344–356
  * [95] Philipp Schoenegger and Peter S Park  “Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament”  In _arXiv preprint arXiv:2310.13014_ , 2023
  * [96] Nuño Sempere and Alex Lawsen  “Alignment problems with current forecasting platforms”  In _arXiv preprint arXiv:2106.11248_ , 2021
  * [97] Lennart Sjöberg  “Are all crowds equally wise? A comparison of political election forecasts by experts and the public”  In _Journal of Forecasting_ 28.1 Wiley Online Library, 2009, pp. 1–18
  * [98] James Surowiecki  “The wisdom of crowds”  Anchor, 2005
  * [99] Philip E Tetlock and Dan Gardner  “Superforecasting: The Art and Science of Prediction”  Random House, 2016
  * [100] Philip E Tetlock, Barbara A Mellers, Nick Rohrbaugh and Eva Chen  “Forecasting Tournaments: Tools for Increasing Transparency and Improving the Quality of Debate”  In _Current Directions in Psychological Science_ 23.4 Sage Publications CA, 2014, pp. 290–295
  * [101] Philip E Tetlock, Barbara A Mellers and J Peter Scoblic  “Bringing probability judgments into policy debates via forecasting tournaments”  In _Science_ 355.6324 American Association for the Advancement of Science, 2017, pp. 481–483
  * [102] Lyle Ungar et al.  “The Good Judgment Project: A Large Scale Test of Different Methods of Combining Expert Predictions”  In _2012 AAAI Fall Symposium Series_ , 2012  Citeseer
  * [103] “What would humans do in a world of super-AI?”  In _The Economist_ , 2023  URL: www.economist.com/finance-and-economics/2023/05/23/what-would-humans-do-in-a-world-of-super-ai
  * [104] Jens Witkowski et al.  “Incentive-Compatible Forecasting Competitions”  In _Management Science_ 69.3 INFORMS, 2023, pp. 1354–1374

## Appendix

The prediction error in figure A1 drops sharply around day 120. This is due to
the fact that at this time the Egyptian Pound was devalued by more than 17%.
Since the probability of such an event was deemed to be low by the
forecasters, almost all of the forecast error in figure A1 comes from this
question.

According to figure A12 the Russian Ruble triggers the depreciation threshold
very early. There are two important remarks to be made here: First, the data
from the quantmod-package records a downward spike at around day 20 that is
not present in the data from xe.com and therefore would not have triggered the
question to resolve positively at that time. However, the data from xe.com
records the Russian Ruble crossing the depreciation threshold around day 100,
whereas the Ruble barely touches the threshold in the quantmod-data. Secondly,
both databases record the Ruble going below the depreciation threshold on day
13. However, predictions were just aggregated from day 14 onwards. Therefore,
the question did not resolve as ‘Yes‘ $k=1$ on day 13, as the tournament had
not really started at that point.

Figure A1: Accuracy of the Metaculus Prediction on questions to which the
random-walk is not applicable

Figure A2: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A3: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A4: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A5: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A6: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A7: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A8: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A9: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A10: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A11: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A12: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A13: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A14: Top picture: Metaculus Prediction compared to the predictions made
by the random-walk
Bottom picture: Exchange rate across the same time

Figure A15: Top picture: Metaculus Prediction
Bottom picture: Exchange rate across the same time

Figure A16: Top picture: Metaculus Prediction
Bottom picture: Exchange rate across the same time

Figure A17: Top picture: Metaculus Prediction
Bottom picture: Exchange rate across the same time

Figure A18: Top picture: Metaculus Prediction
Bottom picture: Exchange rate across the same time

Figure A19: Top picture: Metaculus Prediction
Bottom picture: Exchange rate across the same time

Figure A20: Top picture: Metaculus Prediction
Bottom picture: Exchange rate across the same time


across the two cases. The intuition here is that an increase in the
concentration of the von Mises-Fisher distribution leads to the concentric
contour map of the density function becoming more tightly arranged around the
mean direction. Locating the mean direction closer to the boundary between
high and moderate regret regions say, such as where the deterministic
assignment rule of Kitagawa & Tetenov (2018) is located, incurs less of a
penalty in this instance since the density function assigns less probability
mass to the high regret region than it would for a smaller value of the
concentration parameter. For the doubled sample, the mean direction is
$(0.812,0.577,0.088)$, which translates to an azimuth of $35^{\circ}$ and an
inclination of $85^{\circ}$, while for the quadrupled sample, the mean
direction is $(0.917,0.394,0.053)$, which translates to an azimuth of
$23^{\circ}$ and an inclination of $87^{\circ}$. This compares to an original
mean direction of $(0.883,0.442,0.158)$, which translates to an azimuth of
$27^{\circ}$ and an inclination of $81^{\circ}$.

###  C.2 The variance of post-programme earnings

In the following three experiments, we investigate the influence of the
variance of post-programme earnings on the results of our method.
Specifically, we investigate how altering $\sigma_{0}$ and $\sigma_{1}$ in the
specification outlined in Equation C.2 (Equation C.2) affects our results. In
particular, we are interested in whether there is a fundamental change in how
empirical welfare risk varies with the parameters of the von Mises-Fisher
distribution.

Figure 12: Variation in treatment propensity across individuals with simulated
post-programme earnings (Experiment 1) Each point represents the individual
characteristics of an individual or several individuals in the JTPA Study
sample. The size of each point is proportional to the sum of the post-
programme earnings of all individuals with those characteristics, which
corresponds to the weight given to those observations in the empirical
implementation. The colour of each point measures the propensity with which an
individual with those characteristics is assigned treatment under our estimate
of the optimal stochastic assignment rule. Under the optimal stochastic
assignment rule, more educated individuals with low pre-programme earnings are
less likely to be assigned treatment. For comparison, individuals above the
solid line are assigned treatment under the oracle assignment rule of Equation
C.9 (Equation C.9).

The JTPA Study sample consists of 9,223 observations, and we extract treatment
status and individual characteristics directly from this dataset. We then
generate potential outcomes according to Equation C.2 (Equation C.2). We
implement the affine map of Equation C.1 (Equation C.1) via the
transformations

$\begin{array}[]{lcl}X_{\textrm{earn}}&{}\mapsto{}&X_{\textrm{earn}}/\max\\{X_{\textrm{earn}}\\},\\\
X_{\textrm{educ}}&{}\mapsto{}&X_{\textrm{educ}}/\max\\{X_{\textrm{educ}}\\}.\\\
\end{array}$ (C.7)

We then regress post-programme earnings in the treatment group and the control
group on these characteristics, separately and together, so that the baseline
experiment somewhat mimics the JTPA Study sample. Using our simple regressions
as a rough guide, we let

$\begin{array}[]{lclclcl}{\alpha}&{}={}&(-1,086,+82,458,+18,804),&{}{}\hfil&\sigma_{0}&{}={}&1,5914,\\\
{\lambda}&{}={}&(+3,040,+86,446,+14,008),&{}{}\hfil&\sigma_{1}&{}={}&15,914,\\\
\end{array}\hskip 15.0ptm=5\sigma_{0}=5\sigma_{1},$ (C.8)

so that the optimal policy is

$G=\\{{x}\,:\,{x}\cdot{\beta}\geq 0,{\beta}=(+0.552,+0.533,-0.641)\\},$ (C.9)

which we plot in Figure 12 (Figure 12). We note that the coefficients of the
linear specification are so large because our simple regressions scale
individual characteristics but do not scale post-programme earnings, which
inflates the effect of pre-programme earnings and years of education. This
constitutes our first experiment.

We then propose two further experiments. Our second experiment increases the
variance of post-programme earnings by inflating $\sigma_{0}$, $\sigma_{1}$
and $m$ in Equation C.8 (Equation C.8) by a factor of five. Our third
experiment similarly reduces the variance of post-programme earnings by
deflating $\sigma_{0}$, $\sigma_{1}$ and $m$ in Equation C.8 by a factor of
five. We note that both experiments leave the oracle policy unchanged from
Equation C.9 (Equation C.9). We present the corresponding estimates of the
parameters of the posterior distribution immediately below.

Experiment | $\mu$ | $\kappa$
---|---|---
1\. Baseline | (+0.822,+0.565,+0.078) | 1.850
2\. High variance | (+0.641,+0.703,-0.309) | 0.340
3\. Low variance | (+0.831,+0.547,+0.105) | 1.970

An immediate conclusion that we can draw from these results is that the mean
direction does not align with the oracle assignment rule. That being said, it
is apparent from Figure 12 (Figure 12) that individuals who are assigned
treatment under the oracle assignment rule are more likely to be assigned
treatment under the posterior distribution that we obtain.

Figure 13: Behaviour of the objective function given variation in the
concentration $\kappa$ in Experiment 1 through Experiment 3 (influence of the
variance of post-programme earnings)

Holding fixed the mean direction, we vary the value of the concentration
parameter. We plot the USD equivalent amount of both the criterion (the
objective function) and empirical welfare risk (the first part of the
objective function) in the JTPA Study sample, which we define in the main
text, using simulated post-programme earnings.

We plot the behaviour of the objective function and of empirical welfare risk
in Figure 13 (Figure 13). We emphasise the non-smoothness of empirical welfare
risk for the second experiment, and we suggest that increasing the variance of
post-programme earnings makes the problem of finding the optimal assignment
rule more difficult.

###  C.3 The location of the oracle assignment rule

A common feature of the data generating process in Experiments 1 through 3
(Equation C.8,Equation C.8) is the oracle assignment rule, which is located in
the south-east corner of the covariate space (Figure 12, Figure 12). Two
things can be inferred from this. Firstly, that only a small minority of
individuals, with what amounts to post-graduate educational qualifications,
are assigned to the control group under the oracle assignment rule. In other
words, a small subset of the data with extreme characteristics. Secondly,
that, all else being equal, average outcomes in the treatment group are higher
than average outcomes in the control group, and that this baseline difference
is relatively important as compared to pre-programme earnings and years of
education. These features, we suggest, are indicative of a relatively easy
problem (of determining the optimal assignment rule).

Figure 14: Variation in treatment propensity across individuals in the JTPA
Study sample with simulated post-programme earnings (Experiment 4) Each point
represents the individual characteristics of an individual or several
individuals in the JTPA Study sample. The size of each point is proportional
to the sum of the post-programme earnings of all individuals with those
characteristics, which corresponds to the weight given to those observations
in the empirical implementation. The colour of each point measures the
propensity with which an individual with those characteristics is assigned
treatment under our estimate of the optimal stochastic assignment rule. Under
the optimal stochastic assignment rule, more educated individuals with low
pre-programme earnings are less likely to be assigned treatment. For
comparison, individuals above the solid line are assigned treatment under the
oracle assignment rule of Equation C.11 (Equation C.11).

The following three experiments largely replicate Experiments 1 through 3 but
shift the oracle assignment rule to the left, narrowing the difference between
the baseline average outcomes in the experimental and control groups. In
Experiment 4, we assume that

$\begin{array}[]{lclclcl}{\alpha}&{}={}&(-489,+82,458,+18,804),&{}{}\hfil&\sigma_{0}&{}={}&15,914,\\\
{\lambda}&{}={}&(+2,442,+86,446,+14,008),&{}{}\hfil&\sigma_{1}&{}={}&15,914,\\\
\end{array}\hskip 15.0ptm=5\sigma_{0}=5\sigma_{1},$ (C.10)

so that the optimal policy is

$G=\\{{x}\,:\,{x}\cdot{\beta}\geq 0,{\beta}=(+0.425,+0.579,-0.696)\\},$ (C.11)

which we plot in Figure 14 (Figure 14). Experiments 5 and 6 then mirror
Experiments 2 and 3 in that they inflate and deflate, respectively, the
variance of post-programme earnings. We suggest that Equation C.10 is a more
difficult problem that Equation C.8 (Equation C.8).

We specifically design these experiments so that the oracle assignment rule is
such that as close to 50% of the sample is assigned to the treatment group as
is possible without altering the importance of pre-programme earnings relative
to years of education in the earnings process. We present the corresponding
estimates of the parameters of the posterior distribution immediately below.

Experiment | $\mu$ | $\kappa$
---|---|---
4\. Baseline | (+0.932,+0.362,-0.035) | 0.00
5\. High variance | (-0.810,-0.526,-0.259) | 0.390
6\. Low variance | (+0.899,+0.425,+0.105) | 0.00

Figure 15: Behaviour of the objective function given variation in the
concentration $\kappa$ in Experiment 4 through Experiment 6 (influence of the
variance of post-programme earnings)

Holding fixed the mean direction, we vary the value of the concentration
parameter. We plot the USD equivalent amount of both the criterion (the
objective function) and empirical welfare risk (the first part of the
objective function) in the JTPA Study sample, which we define in the main
text, using simulated post-programme earnings.

A curious feature of these results is that, for Experiments 4 and 6, the
posterior distribution is uniform. We note that the mean direction is
irrelevant in this case, but we include it anyway because it is informative
about the mean direction in Figure 15 (Figure 15). This uniformity of the
posterior distribution is apparently driven by how empirical welfare risk
increases alongside $\kappa$ in Figure 15. A possible explanation for this
feature is that this data generating process is indeed hard. We suggest that
the narrow gap between the baseline average outcomes of the experimental and
control groups makes treating everyone versus treating no-one equally
appealing (or unappealing). For instance, contrasting Figure 7 (Figure 7) with
its analogue in the main text (for the raw data), we see that the difference
in empirical welfare risk between the treat everyone rules and treat no-one
rules that occupy the southern and northern regions of the heatmaps,
respectively, narrows. A similar effect happens here. Given the fact that the
density contours of the von Mises-Fisher distribution are concentric, there is
then no natural region of the sphere to locate in order to minimise empirical
welfare risk unless the concentration is particularly high and the posterior
distribution allocates substantial probability mass to an extremely localised
area on the frontier between the collections of treat everyone rules and treat
no-one rules. Such a high value of the concentration parameter is penalised by
the Kullback-Leibler divergence, and so uniformity cannot be improved upon.
The fact that a different pattern is observed for Experiment 5 is compatible
with this argument, and could be achieved if the high variance of post-
programme earnings makes a few observations in the sample pivotal (recall that
the weight attached to each observation is proportional to post-programme
earnings; inflating the variance of post-programme earnings can lead to
greater concentration of weight on a few observations, since extreme outliers
are more likely).

###  C.4 Individual characteristics and their distribution

Figure 16: Variation in treatment propensity across individuals for simulated
characteristics (Experiments 7 and 8)

Each point represents the individual characteristics of an individual or
several individuals. The size of each point is proportional to the sum of the
post-programme earnings of all individuals with those characteristics, which
corresponds to the weight given to those observations in the empirical
implementation. The colour of each point measures the propensity with which an
individual with those characteristics is assigned treatment under our estimate
of the optimal stochastic assignment rule. The oracle assignment rule does not
intercept the studied region, and assigns every individual to treatment.

The final experiments that we undertake vary the distribution of individual
characteristics. Our intention is to understand whether the JTPA Study sample
is, in some sense, special and whether altering individual characteristics
substantially alters our results.

Experiments 7 and 8 mirror Experiment 1 in how the outcome is generated: both
experiments rely on Equation C.2 and the parameter values that we outline in
Equation C.8 for Experiment 1. The distinction between Experiments 7 and 8 and
Experiment 1 is how individual characteristics are generated. Whereas
Experiment 1 uses data taken from the JTPA Study sample, Experiments 7 and 8
generate individual characteristics according to a bivariate normal
distribution with the mean vector equal to half of the maximum of pre-
programme earnings and the average number of years of education, respectively.
We estimate the covariance matrix of pre-programme earnings and years of
education in the JTPA Study sample, and set the covariance of the bivariate
normal distribution equal to this (Pearson correlation coefficient of 0.126).
In the case of Experiment 7, we discretise years of education by assigning
each observation to one of 12 equal-sized bins. We then map each
characteristic to the unit interval by means of the aforementioned linear
transformation.

We plot the individual characteristics that we use in Experiments 7 and 8 in
Figure 16 (Figure 16). Absent from either plot is the oracle assignment rule.
This is not an oversight. Rather, the oracle assignment rule is such that it
recommends that all individuals be assigned to the experimental group (i.e.,
it to the south-east of the plotting area). We plot the criterion (left-hand
panel) and empirical welfare risk (right-hand panel) for Experiments 7 and 8
in Figure 17 (Figure 17). Due to the similarity of Figure 17 to the other
figures that we have presented, we do not provide any further discussion of
this figure. What is somewhat interesting though is how the propensity of
treatment is highest for those individuals with individual characteristics
located around their mean values.

We present the corresponding estimates of the parameters of the posterior
distribution immediately below.

Experiment | $\mu$ | $\kappa$
---|---|---
7\. Discrete | (+0.794,+0.405,+0.454) | 3.090
8\. Continuous | (+0.801,+0.408,+0.438) | 3.120

The mean direction of the posterior distribution does not align with the
oracle assignment rule, but does dictate that all individuals are assigned to
the experimental group (as they are under the oracle assignment rule). The
value of the concentration parameter is also relatively large, as compared to
its value in the previous experiments.

So as to make the assignment problem harder – at least, what we understand to
be harder – we propose Experiments 9 and 10. These experiments follow
Experiments 7 and 8 in how individual characteristics are generated, but
differ slightly in how they generate the outcome. While Experiments 9 and 10
broadly follow Experiments 7 and 8 with respect to how the outcome is
generated, they shift the process for the potential outcomes and, thereby, the
oracle assignment rule. Specifically, in Experiments 9 and 10, we assume that

$\begin{array}[]{lclclcl}{\alpha}&{}={}&(-668,+82,458,+18,804),&{}{}\hfil&\sigma_{0}&{}={}&15,914,\\\
{\lambda}&{}={}&(+1,286,+86,446,+14,008),&{}{}\hfil&\sigma_{1}&{}={}&15,914,\\\
\end{array}\hskip 15.0ptm=5\sigma_{0}=5\sigma_{1},$ (C.12)

such that the oracle policy is

$G=\\{{x}\,:\,{x}\cdot{\beta}\geq 0,{\beta}=(+0.098,+0.636,-0.765)\\}.$ (C.13)

The effect of this change is to maintain the slope of the oracle assignment
rule (i.e., the contribution of pre-programme earnings relative to years of
education) but to alter the intercept. In other words, to narrow the
difference in baseline average outcomes between the experimental and control
groups. In this regard, Experiments 9 and 10 are similar in intent to
Experiment 4.

Figure 17: Behaviour of the objective function given variation in the
concentration $\kappa$ in Experiments 7 and 8 (influence of the distribution
of individual characteristics)

Holding fixed the mean direction, we vary the value of the concentration
parameter. We plot the USD equivalent amount of both the criterion (the
objective function) and empirical welfare risk (the first part of the
objective function), which we define in the main text, using simulated post-
programme earnings. Figure 18: Variation in treatment propensity across
individuals for simulated characteristics (Experiments 9 and 10)

Each point represents the individual characteristics of an individual or
several individuals. The size of each point is proportional to the sum of the
post-programme earnings of all individuals with those characteristics, which
corresponds to the weight given to those observations in the empirical
implementation. The colour of each point measures the propensity with which an
individual with those characteristics is assigned treatment under our estimate
of the optimal stochastic assignment rule. The oracle assignment rule
partitions the sample into two approximately equal-sized groups.

We plot the individual characteristics that we use in Experiments 9 and 10 in
Figure 18 (Figure 18). The oracle assignment rule is designed to partition the
sample into approximately two groups of equal size. We plot the criterion
(left-hand panel) and empirical welfare risk (right-hand panel) for
Experiments 9 and 10 in Figure 19 (Figure 19). We draw attention to several
differences between the results that we obtain for Experiments 9 and 10 versus
Experiments 7 and 8. Firstly, we highlight the fall in the average propensity
of treatment, as is apparent in Figure 18. Whereas in Experiments 7 and 8 (and
the preceding experiments too), the propensity of treatment is close to one
(i.e., everyone is assigned to the experimental group), here the propensity is
closer to one third. There is more randomisation. Secondly, individuals that
have a relatively high propensity of treatment as compared to their peers in
Experiments 7 and 8 have a relatively low propensity of treatment in
Experiments 9 and 10 (visually, the colours are inverted), as is apparent in
Figure 18. That is, individuals whose characteristics are close to the mean
values have a low propensity of treatment. Thirdly, the value of the
concentration parameter decreases from above three to below one (i.e., the
posterior distribution is less concentrated and more uniform), as is apparent
in Figure 19.

We present the corresponding estimates of the parameters of the posterior
distribution immediately below.

Experiment | $\mu$ | $\kappa$
---|---|---
9\. Discrete | (-0.766,-0.470,-0.438) | 0.770
10\. Continuous | (-0.766,-0.470,-0.438) | 0.770

Interestingly, the mean direction is, in both cases, almost the exact opposite
of the corresponding mean direction in Experiments 7 and 8 (i.e., the mean
direction is approximately the negative of the mean direction in Experiments 9
and 10.

Figure 19: Behaviour of the objective function given variation in the
concentration $\kappa$ in Experiments 9 and 10 (influence of the distribution
of individual characteristics)

Holding fixed the mean direction, we vary the value of the concentration
parameter. We plot the USD equivalent amount of both the criterion (the
objective function) and empirical welfare risk (the first part of the
objective function), which we define in the main text, using simulated post-
programme earnings.