# Assessing Human Judgment Forecasts in the Rapid Spread of the Mpox Outbreak:
Insights and Challenges for Pandemic Preparedness

Thomas McAndrew  Department of Community and Population Health, College of
Health, Lehigh University, Bethlehem, Pennsylvania, United States of America
<EMAIL_ADDRESS>Maimuna S. Majumder  Boston Children’s Hospital, Boston,
MA, United States of America  Harvard Medical School, Boston, MA, United
States of America  Andrew A. Lover  Dept. of Biostatistics and Epidemiology,
School of Public Health and Health Sciences, University of Massachusetts
Amherst, Amherst, Massachusetts, United States of America  Srini Venkatramanan
Biocomplexity Institute and Initiative, University of Virginia,
Charlottesville, Virginia, United States of America  Paolo Bocchini
Department of Civil and Environmental Engineering, P.C. Rossin College of
Engineering and Applied Science, Lehigh University, Bethlehem, Pennsylvania,
United States of America  Tamay Besiroglu  MIT Computer Science and Artificial
Intelligence Laboratory, Cambridge, Massachusetts, United States of America
Metaculus, Santa Cruz, California, United States of America  Allison Codi
Department of Biostatistics, Rollins School of Public Health, Emory
University, Atlanta, Georgia, United States of America  Gaia Dempsey
Metaculus, Santa Cruz, California, United States of America  Sam Abbott
Centre for Mathematical Modelling of Infectious Diseases, London School of
Hygiene & Tropical Medicine, Keppel St, London, WC1E 7HT, UK  Sylvain
Chevalier  Metaculus, Santa Cruz, California, United States of America  Nikos
I. Bosse  Centre for Mathematical Modelling of Infectious Diseases, London
School of Hygiene & Tropical Medicine, Keppel St, London, WC1E 7HT, UK
Metaculus, Santa Cruz, California, United States of America  Juan Cambeiro
Department of Epidemiology, Mailman School of Public Health, Columbia
University, New York, United States.  Metaculus, Santa Cruz, California,
United States of America  David Braun  Department of Psychology, Lehigh
University, Bethlehem, Pennsylvania, United States of America

###### Abstract

In May 2022, mpox (formerly monkeypox) spread to non-endemic countries
rapidly. Human judgment is a forecasting approach that has been sparsely
evaluated during the beginning of an outbreak. We collected—between May 19,
2022 and July 31, 2022—1275 forecasts from 442 individuals of six questions
about the mpox outbreak where ground truth data are now available. Individual
human judgment forecasts and an equally weighted ensemble were evaluated, as
well as compared to a random walk, autoregressive, and doubling time model. We
found (1) individual human judgment forecasts underestimated outbreak size,
(2) the ensemble forecast median moved closer to the ground truth over time
but uncertainty around the median did not appreciably decrease, and (3)
compared to computational models, for 2-8 week ahead forecasts, the human
judgment ensemble outperformed all three models when using median absolute
error and weighted interval score; for one week ahead forecasts a random walk
outperformed human judgment. We propose two possible explanations: at the time
a forecast was submitted, the mode was correlated with the most recent (and
smaller) observation that would eventually determine ground truth. Several
forecasts were solicited on a logarithmic scale which may have caused humans
to generate forecasts with unintended, large uncertainty intervals. To aide in
outbreak preparedness, platforms that solicit human judgment forecasts may
wish to assess whether specifying a forecast on logarithmic scale matches an
individual’s intended forecast, support human judgment by finding cues that
are typically used to build forecasts, and, to improve performance, tailor
their platform to allow forecasters to assign zero probability to events.

##  I Introduction

An outbreak of mpox (formerly monkeypox) was detected in early May 2022, and
on July 23, 2022, the World Health Organization declared this outbreak a
Public Health Emergency of International Concern (PHEIC) [1, 2, 3, 4, 5, 6, 7,
8, 2]. To date, there have been 87k cases of human mpox reported by at least
110 countries in association with this outbreak [9, 10]. Transmission has
occurred most frequently between men who have sex with men (MSM); however,
cases are not restricted to this population [11, 12, 13].

Historically, forecasts of public health metrics of interest—including
reported incident cases, deaths, and hospitalizations—have been used to
support decision making during outbreaks [14, 15, 16, 17, 18]. Agencies such
as the Centers for Disease Control and Prevention (CDC) have routinely hosted
forecasting challenges, where they elicit forecasts of public health metrics
from research groups during outbreaks of communicable diseases [19, 20, 14,
21]. These agencies have invested considerable time and resources to host
platforms such as the Epidemic Prediction Initiative [22] that are designed to
collect and communicate probabilistic forecasts of infectious diseases such as
seasonal influenza, dengue, neuroinvasive West-Nile Virus disease, Zika virus
disease, and COVID-19 [19, 20, 23, 24].

Computational models typically train on existing surveillance data and other
external data sources that are hypothesized to be correlated with the
forecasting target of interest [25, 26, 27, 28, 29, 30, 31, 32, 33, 33, 34,
35, 36]. Models may also leverage interventions or behavioral adaptations that
could influence future disease dynamics [37, 38, 39]. Computational models may
be either statistical (i.e., leveraging associations between a set of
variables and epidemiological outcomes), mechanistic (i.e., encoding domain
knowledge on the known causal linkages between population and disease spread
dynamics), or a hybrid of both approaches [40, 41, 42, 43, 44]. Past work has
shown that statistical and mechanistic models are able to incorporate a
diverse range of data sources external to standard health surveillance systems
to produce more accurate forecasts. For example, these models have included
Google searches for symptoms associated with disease or social media posts
related to a disease, genomics data, human mobility data, and meteorological
data [25, 26, 27, 28, 45, 46, 47]. Furthermore, mechanistic models in
particular have had past success in describing how the trajectory of an
infectious agent may change after choosing one of many potential interventions
[29, 30, 31, 32, 33]. Because mechanistic models assume a set of rules for how
an infection spreads throughout a population, these models can also be useful
for interpreting surveillance data and testing potential public health
interventions [33, 34, 35, 36].

In contrast to statistical and mechanistic forecasts, human judgment forecasts
collect and combine predictive densities from a set of individuals [48, 49,
50, 51, 52, 53, 54]. Humans may decide to submit a prediction that was
produced by a computational model, from their intuition, or both. Past work
using human judgment to forecast infectious diseases has shown that human
judgment forecasts can perform similarly or at times better compared to
computational models [48, 49, 50, 51], can often predict a broader class of
targets than computational models [55, 56], can add data that complements
traditional surveillance data (i.e. case counts) [54, 57], and may perform
better than computational models when data on a pathogen is sparse [53, 58,
59, 52].

Within two weeks of the first reported case of mpox in London, six
questions—created by the authors of this study—were posted to a community of
more than 15,000 registered forecasters on a public prediction platform,
Metaculus [60]. During the study period, from May 19, 2022 to July, 31, 2022,
the goal of these six questions was to understand the potential intensity and
burden of mpox. These questions were: (i) what will be the total number of
confirmed and suspected monkeypox cases in Europe as of July 1, 2022?, (ii)
what will be the total number of confirmed and suspected monkeypox cases in
the United States as of July 1, 2022?, (iii) what will be the total number of
confirmed and suspected monkeypox cases in Canada as of July 1, 2022? (iv) how
many countries will confirm at least one case of monkeypox by July 31, 2022?,
(v) how many states in the United States will report having a case of
monkeypox as of July 1, 2022?, and (vi) will the World Health Organization
declare the spread of monkeypox a Public Health Emergency of International
Concern before 2023? We received a total of 1275 probabilistic predictions
(including initial forecasts and subsequent updates) from 442 individual
forecasters.

Here, we evaluate the performance of a human judgement ensemble (HJE) of the
2022 mpox outbreak, as well as the individual human judgment forecasts that
comprised the ensemble. We show that the majority of individuals
underestimated the size of the outbreak and that, though the median prediction
was accurate, the level of uncertainty remained the same or did not
appreciably decrease as the time between when the forecast was generated and
the ground truth (for example the total number of cases for a country)
decreased. We propose two potential mechanisms for these observations: (1) for
each question, the most recent observation from the data source that would
determine the final ground truth was associated with a given individual’s mode
prediction and (2) forecasts were solicited on a logarithmic scale, which may
have led to unexpectedly large prediction intervals.

##  II Methods

###  II.1 Collecting crowdsourced predictions

From May 19, 2022 to July 31, 2022, we collected 1275 individual estimates of
six questions posed on the Metaculus platform from 442 users. Users of
Metaculus were permitted to submit a prediction to one or more questions
related to mpox. Forecasters were not explicitly recruited for the study. They
were either existing or new users on the Metaculus platform, all of whom have
agreed to the terms and conditions outlined by Metaculus
(https://www.metaculus.com/terms-of-use/). Members of the Metaculus team
advertised mpox questions on their mailing list, which is sent to all
Metaculus users, and on Twitter. We also sent personal emails to those who
were considered experts in the modeling of infectious diseases, asking if they
would volunteer their time and submit predictions. However, only one such
individual identified themselves as submitting predictions [56].

The format for how a human submits a forecast on Metaculus can be found in the
supplement under the header ‘Forecast submission’. Humans were able to submit
a forecast at any time before the date on which the ground truth would be
resolved, and they were allowed to revise their original forecast as many
times as they wished. Notably, some questions solicited forecasts on a
logarithmic scale while other questions were solicited on natural scale (see
Table 4).

###  II.2 Surveillance data

Surveillance data was used to train computational models that served as
control against human judgment forecasts. Surveillance data is defined as any
data source that (1) updates over time and (2) contains the ground truth that
answers a forecasting task. Surveillance data was used as ground truth for all
the questions posed on Metaculus. For this work, we found surveillance data
that reported over time the number of total cases of mpox; and number
countries/states with at least one positive mpox case. Data sources used to
define the ground truth were at a weekly temporal scale and were collected
from the Centers for Disease Control and Prevention, Global Health [61] and
the European Centre for Disease Control and Prevention [62]. Supplemental
Table 4 provides information about surveillance data sources used to train
models.

###  II.3 Computational models

We compared our human judgment ensemble forecasts to (i) a random walk model
(constant trend), (ii) an autoretrogressive model (linear trend), and (iii) a
doubling-time model (simple exponential growth). These three simple
computational models were chosen as a baseline to compare against the ensemble
of human judgments. These models are not an attempt to rigorously predict the
mpox outcomes that were posed to human forecasters. Instead, these three
models represent three potential schema for how humans may produce
predictions: (1) the random walk supposes a constant trend from the last
observed data point and the uncertainty grows with the time between this last
observation and the target, thus representing a scenario where a human cannot
justify an increasing or decreasing trend; (2) the autoretrogressive model
allows for an increasing/decreasing trend that is linear, acknowledging that
linear predictions are often used to characterize human predictions [63]; and
(3) the doubling-time allows for exponential increases/decreases in trend.
Notably, past work has shown that humans poorly understand exponential growth
and thus is the least likely model employed by humans in this experiment [64,
65, 66].

Computational models were trained retrospectively, and did not train on data
that may have been revised as the mpox outbreak was ongoing. Models were
trained every week from the start of data collection up until 1-8 weeks before
the date of ground truth. For example, if the ground truth was evaluated at
week $T$ then we cut training data for all computational models at $T-1$,
$T-2$ weeks etc.

All three models were fit with a No-U-Turn Sampler (i.e. Hamiltonian MCMC)
using Numpyro [67]. For each model, we requested 4,000 warm-up samples and
10,000 draws from a single chain.

For this analysis, note that the question that asked whether the WHO would
declare a Public Health Emergency of International Concern (PHEIC) is not
included. This is because there is no clear mapping from the three
computational models to a declaration of PHEIC.

Our random walk model assumes that our time series of interest follows a
Gaussian random walk

$\displaystyle\sigma^{2}\sim\text{Gamma}(1,1);\;Y_{t}=\sum_{t=0}^{t}\mathcal{N}(0,\sigma^{2})$

where $\mathcal{N}(0,\sigma^{2})$ is a normally distributed random variable
with expected value $0$, variance $\sigma^{2}$. Given training data up until
time $T-X$, a forecast at time $T$ was generated as

$\displaystyle Y_{T}=y_{T-X}+\sum_{t=1}^{X}\mathcal{N}(0,\sigma^{2})$

were $y_{T-X}$ is the last observed value in the training data.

Our autoregressive model (AR) assumes that our outcome of interest $y_{t}$ is
drawn from a sequence of random variables $Y_{t}$ that follow

$\displaystyle\beta\sim\text{Gamma}(1,1);\;\sigma^{2}\sim\text{Gamma}(1,1);\;Y_{t}|y_{t-1}\sim\mathcal{N}(\beta
y_{t-1},\sigma^{2})$

Our doubling-time model assumes that our target $y_{1},y_{2},\cdots,y_{T}$
follows

$\displaystyle\alpha$
$\displaystyle\sim\text{Gamma}(1,1);\;\beta\sim\text{Gamma}(1,1)$
$\displaystyle t_{0}$
$\displaystyle\sim\text{Uniform}(T_{0},T_{1});\;\sigma^{2}\sim\text{Gamma}(1,1);\;Y_{t}\sim\mathcal{N}(\alpha
2^{\beta(t-t_{0})},\sigma^{2})$

where $\alpha$, $\beta$, $t_{0}$, and $\sigma^{2}$ are parameters. This model
assumes that the expected number of cases doubles every $\frac{1}{\beta}$ time
steps and never decreases. Though the choice of a Normal distribution assigns
probability to values outside the total number of countries and/or cases, this
non-zero probability is small.

###  II.4 Forecast evaluation

####  II.4.1 Timeline

We chose to evaluate 1-8 week ahead forecasts generated by humans and by
computational models. The number of weeks ahead that we could evaluate
depended on which date individuals were asked to generate a forecast. All
questions asked for ground truth values on either July 1 or July 31. If a
question was associated with ground truth at a later date (July 31) then we
could evaluate longer forecast horizons (Supp. Table 4). In addition, we
collected data on individual forecasters and studied these individual
forecasts for the same time horizons. For each question, computational models
were given observed data from the data source used to determine the ground
truth. The ground truth was a single value on a given date, called the
resolution date.

####  II.4.2 Trend and Performance

Trend and performance of forecasts were assessed using the median absolute
error (MAE) and weighted interval score (WIS [68]). The MAE is presented to
emphasize the trend in forecasts while WIS, which includes MAE, is a more
formal evaluation metric. Given a forecast ($F$) as a probability density
function, the median absolute error is the median of $F$ minus ground truth.
An individual forecaster’s bias ($b$) is defined as their median prediction
($m$) minus the ground truth ($g$), or $b=m-g.$ By computing the bias for all
individual forecasters, we can then compute sample statistics from this
population of biases. Because the ground truth for all six targets is
positive, a negative $b$ indicates a prediction that is smaller than the
ground truth and vice-versa.

The WIS takes as input a select number of quantiles from a forecast $(F)$ and
the truth $y$ and then returns a number from zero to infinity. We chose to
evaluate quantiles from the 1st percentile up to the 99th percentile in
increments of 1 percent. Smaller values of WIS indicate better performance,
whereas larger WIS values indicate weaker performance. The WIS is an absolute
score that depends on the magnitude of the target and so WIS scores cannot be
compared across targets. We used the scoringUtils package in R to compute WIS
[69] To aid in the interpretation of WIS values, and to scale weighted
interval scores for different targets in a unified way, they have been
rescaled as relative WIS. We define the relative WIS as the weighted interval
score for one forecast divided by the weighted interval score for the random
walk presented above—a model that assumes no trend—for the same target at the
same time horizon.

###  II.5 Forecaster rationales and interpretation

Forecasters on the Metaculus platform had the option to attach to their
forecasts a text-based rationale or comment. All text data is available online
(See Table 4 for links to questions). Text data was evaluated and interpreted
based on the author’s content-area expertise. The goal was to identify a small
set of themes—concepts that were frequently discussed in the comments—from
forecasters. There was no formal qualitative analysis conducted.

###  II.6 Data Availability

Forecasting data, code to process forecasts, and code to generate figures in
this manuscript are available at
https://github.com/computationalUncertaintyLab/mpox_eval. Real time forecasts
about the mpox outbreak are available at
https://www.metaculus.com/project/monkeypox/.

##  III Results

Our results may be categorized into seven broad findings, each of which are
further delineated below. We find that (1) as forecast horizon shrinks, human
judgment predictions intervals do not show evidence of shrinking and that the
human judgment ensemble (HJE) often assigns positive probability to
implausible values; (2) individual forecasters tended to predict values that
were smaller than the ground truth; the HJE had (3a) an MAE and WIS that was
better than a random walk (RW), autoregressive (AR), and doubling time model
(DT) at forecast horizons of two weeks or greater, and 3b) WIS that was worse
than a RW model at a one week ahead horizon; (4) well-performing forecasts may
be in part due to the skew and large uncertainty introduced when soliciting
forecasts on a logarithmic scale; (5) the mode of individual human judgment
forecasts was associated with the last observed value of the time series to be
predicted; (6) six major themes were present in text comments submitted by
forecasters with the most frequent theme being ‘Taking cues from epidemiology
and surveillance’; and (7) forecasters tended to make forecasts within two
weeks of when a question was posed and continually revised their forecasts
over time.

###  III.1 Temporal trends in human judgement predictions

For continuous targets, the median HJE prediction moved closer to the truth as
the resolution date was approached, but the 80% prediction interval for HJE
did not show evidence of decreasing at the same rate as the resolution date
was approached. (Fig. 1).

For the binary question regarding the PHEIC, the median and the 80% prediction
interval were not appreciably changed as the resolution date was approached.
The median [80% prediction interval] at three, two, and one weeks before the
PHEIC declaration equalled 0.39 [0.15, 0.70], 0.39 [0.15, 0.67], and 0.40
[0.15, 0.70], respectively (Fig. 1 bottom right).

The observed time series for continuous targets followed a non-negative trend
over time. This is because the questions posed asked for predictions of
cumulative targets, i.e., cumulative number of mpox cases, countries with an
mpox case, etc. The HJE assigned probability to values well below the most
recently reported true value at three, two, and one week ahead (see Supp. Fig.
9). For example, at one week before the ground truth, the HJE 10th percentile
(lower bound for a 90% prediction interval) for the question ‘What will be the
total number of confirmed and suspected monkeypox cases in Canada as of July
1, 2022?’ was equal to 38 despite the number of true cases at that time point
equaling 260. The assignment of positive probably far below the reported true
number of cases at the time of forecasting likely rules out issues with lag or
revisions to the observed time series of cases.

###  III.2 Under prediction of outbreak

The bias (defined above in Methods) is negative for all targets at all
forecast horizons before the ground truth is determined and moves closer to
zero as the forecast horizon shrinks (Fig. 2). We observed a positive bias for
only one target for two-day ahead forecasts. For predictions of the total
number of cases that would be reported in Canada by July 01, 2022, the bias
was negative until June 24th, close to zero between June 24th and June 29th,
and positive from June 29th until the ground truth was observed.

An alternative way to characterize this bias is to compute the proportion of
individual forecasts with a median prediction above the observed ground truth
(See Supplemental Fig. 10).

###  III.3 Comparison of trend and performance between human judgment and
computational models

HJE had improved (i.e. smaller) median absolute error and improved (i.e.
smaller) WIS compared to a random walk (RW), auto-retrogressive (AR), and
doubling-time (DT) model for forecast horizons of 2-8 weeks (Fig. 3). For one
week ahead forecasts, the median absolute error and WIS was better than AR and
DT models but worse than the RW model.

Forecasts solicited on a logarithmic scale tended to perform better than
forecasts solicited on natural scale (Fig. 4 A. and Table 1). The difference
in performance between forecasts solicited on logarithmic vs natural scale was
most pronounced at 2 and 3 week ahead horizons.

When forecasting on a natural scale, human judgment forecasts naturally
sharpen though not necessarily over the ground truth (Fig. 4 B. and C.). When
solicited on logarithmic scale, forecasts do not sharpen over the ground truth
but the long tail includes ground truth (Fig. 4 D. and E.). The (possibly
unintended) skew introduced by soliciting forecasts on a logarithmic scale may
have improved WIS scores.

###  III.4 Forecasts on logarithmic scale create immoderate skew

Soliciting forecasts on a logarithmic scale tended to right skew individual
human judgment predictive densities (Fig. 5). Forecasts generated on the
logarithmic scale likely appear reasonable to the human eye (Fig. 5 A.);
however, transforming this density back to the natural scale highlights the
right-skew introduced (Fig. 5 B. and C.). The skew—presented as the median
divided by the mode for an individual human judgment forecast—on a logarithmic
scale increased as forecast horizon increased. This is because humans tended
to submit forecasts that are more uncertain at larger forecast horizons. There
is minimal skew for forecasts solicited on a natural scale, and this skew does
not tend to change with increased forecast horizon.

The mode (value with largest density value) of the predictive density for
individuals was associated with the most recently observed value of the target
at the time an individual produced a forecast (Fig. 6). This association was
present for 1-3 week ahead forecasts, 4-6 week ahead forecasts, and 7-8 week
ahead forecasts (see Table 2).

###  III.5 Themes present in the community that forecasted mpox

We identified six potential themes from the comments associated with mpox
forecasting targets: (1) Taking cues from external data (excluding
surveillance data of the target itself) and news; (2) introducing code or
algorithms for how to produce a forecast; (3) signaling data updates to the
crowd; (4) confusion about the target or ground truth; (5) introspection about
past forecasts; and (6) taking cues from epidemiology or surveillance.

The most frequent (N=6 comments) theme present in comments was discussion that
linked to mpox epidemiology, such as symptoms exhibited by patients; the
process by which the WHO declares a public health emergency of international
concern; and how testing practices may impact how cases are reported (Table 3
and see questions QID=10977, QID=10979 with urls in Supp. 4).

Similarly, humans used cues from external datasets (not the surveillance
reports that would eventually contain the ground truth) and the news cycle to
generate forecasts. Humans submitted URLs to news articles that they felt were
related to the target of interest and included python code to aide others in
keeping track of targets over time (N=5; see QID=10975, 11039, 10977, 10978).

It is important to note a frequent comment was related to confusion about how
to define the ground truth and the final value of the ground truth (N=4;
QIDs=10975, 10978, 10979,10981).

###  III.6 Frequency and timing of forecasts

Forecasters most often submitted their first forecast two weeks after the six
considered questions were posted on Metaculus, submitted most revisions within
one week of their first forecast, and submitted their last forecast one week
after their first forecast (Fig. 7 A.-C.). An exception is the question that
asked individuals to forecast the number of European countries that would
report one or more cases of mpox. Similar to other questions, forecasters
submitted their first forecast within two weeks; however, the median time at
which forecasters submitted their last forecast for this question was eight
weeks after the question was first posted, or approximately six weeks after
their first forecast (Fig. 7 C.). The mean number of revisions across all
questions was $3.58~{}(95\text{CI}:[0,16.45])$

##  IV Discussion

Crowdsourced human judgments can produce rapid, moderately accurate forecasts
up to eight weeks ahead of an emerging outbreak. The mechanics of how humans
build forecasts varies. We suppose that it is likely humans generated their
forecasts by extracting cues: from the surveillance system used to evaluate
the ground truth, news cycle, and from one another. In particular, this crowd
was likely anchored to recent surveillance observations. We hypothesize that
the way in which forecasts were solicited has a large impact on the forecasts
that were submitted [70, 71, 72], and, though soliciting forecasts on a
logarithmic scale resulted in better WIS than the three computational models,
this solicitation method may not be representative of an individual’s intended
forecast.

In particular, we would like to highlight several expected and unexpected
properties of the human judgment ensemble forecasts presented for the six
considered questions.

We expected, and observed, that as the resolution date was approached the
difference between the median prediction and ground truth would decrease (Fig.
1). This is in line with the intuition that more data (via a longer time
series) and less randomness (shorter time interval for forecast) will produce
a more accurate prediction. Comments submitted by individuals support the
claim that they continued to monitor surveillance reports about the time
series that they were forecasting and collecting data about this time series
as time passed (Table. 3). For example, forecasters commented “The total
number of confirmed + suspected monkeypox cases in Europe on July 1st 2022 at
12pm EST as per this spreadsheet is 5762 cases. I have the HTML and a csv of
the cases if anyone comes to a different number.” and “As of June 28, 27
states (not counting DC) have reported at least one case of monkeypox
according to the U.S. CDC”. In addition, the association between the most
recently observed value and the mode for individual predictive densities is
also supportive of a more accurate forecast as the resolution date was
approached (Fig. 6).

We also expected individuals to, on average, underestimate the size of the
outbreak (Fig. 2). This outbreak was largely driven by novel transmission
pathways, and is to date several orders of magnitude larger than the largest
prior outbreak of mpox [73]. Similarly, in past work, experts routinely
underestimated the number of cases reported during the COVID-19 pandemic in
2020 [55].

We expected the individual human judgment forecasts and the human judgment
ensemble to assign probabilities to near-impossible events (See lower bound of
prediction interval in Fig. 1). In past work, subject matter experts, those
with experience forecasting, and laymen have often assigned positive
probabilities to near-impossible events [58, 55, 53]. Comments from one
individual forecaster of mpox highlighted this issue, commenting “If my count
is right, it’s looking like 70 at the moment. About 35% of the community
weight is on impossible numbers.”. We note that when submitting a forecast to
Metaculus, individuals are not able to define the support (i.e. the interval
where there exists a non-zero probability) for their forecast, nor are they
able to assign probability zero to any intervals. This means that even if a
forecaster knows that a given interval has a probability of zero the platform
doesn’t allow them to assign a zero. To characterize the mechanism by which
humans assign positive probabilities to impossible events, we suggest a study
that conducts key informant interviews with forecasters who do or do not
assign probability to near-impossible events. For example, one mechanism could
be that individuals may have expected the most recent value for an mpox target
could be revised to be smaller than the currently reported value. Even more,
we recommend studying the difference in forecast performance between two
platforms: one that does not allow the assignment of zero probability and one
that does allow this type of assignment.

We did not expect that the uncertainty around these forecasts would be similar
or not appreciably decrease as the resolution date was approached. We posit
that the most likely mechanism by which these intervals remained large was
soliciting forecasts on a logarithmic scale (Fig. 5). Forecasts solicited on a
natural scale tended to become more focused around a single value, but
forecasts solicited on a logarithmic scale tended to look exponential as the
resolution date was approached (Fig. 4). We cannot determine whether the
humans for this forecasting challenge were susceptible to misunderstanding the
difference between their forecast on a logarithmic vs. natural scale. A
rigorous, randomized design that posed the same questions on both logarithmic
and natural scales could provide evidence for this line of inquiry. However,
past work indicates that humans have a poor understanding of exponentials and
logarithms [64, 65, 66]. That said, the large uncertainty produced by
logarithmic solicitation ‘saved’ forecasts by shifting probability density
from small values (where the ground truth was not) to much larger values
(where the ground truth was). This is illustrated in Figure 4 panels D and E,
comparing the random walk to human judgment ensemble forecast. However, this
positive gain in performance may have come at the cost of misrepresenting the
true forecast that individual’s intended to produce.

A major limitation to this work was the small number of targets that we could
evaluate. We expect future work to evaluate human judgment forecasts of the
trajectory of an infectious agent on a larger set of targets. We are also
limited by not surveying forecasters about the process that they used to
generate their forecasts. Because we did not survey participants, we can only
speculate about the cues that they relied on from submitted comments and try
to infer mechanisms from collected forecasts.

Nevertheless, the forecasts we analyze here involve predictions for a novel
outbreak, and a number of the issues raised in this work are related to past
issues raised in human judgment forecasting of other novel outbreaks [55].
Beyond these well-accepted issues, we also bring attention to general issues
with humans’ misperceptions of logarithms, which require further investigation
within the setting of human judgment modeling [64, 65, 66]. Moreover, we
highlight technical challenges that human judgment platforms should aim to
overcome. One concrete recommendation is to allow individuals to first propose
the interval where they will assign a positive probability before specifying a
functional form over this interval. This technical change, which would need to
be developed for the current platform, could guard against humans assigning
probability to unlikely values and improve forecast performance.

We feel human judgment has the potential to play a role in supporting public
health efforts during an outbreak such as mpox, but only after intense efforts
to address issues like the ones raised in this work.

## References

  * [1] WHO.  Who director-general declares the ongoing monkeypox outbreak a public health emergency of international concern.
  * [2] Sophia Toya Kröger, Max Christian Lehmann, Melanie Treutlein, Achim Fiethe, Annelene Kossow, Annika Küfer-Weiß, Johannes Nießen, and Barbara Grüne.  Mpox outbreak 2022: an overview of all cases reported to the cologne health department.  Infection, 51(5):1369–1381, 2023.
  * [3] Henry Laurenson-Schafer, Nikola Sklenovská, Ana Hoxha, Steven M Kerr, Patricia Ndumbi, Julia Fitzner, Maria Almiron, Luis Alves de Sousa, Sylvie Briand, Orlando Cenciarelli, et al.  Description of the first global outbreak of mpox: an analysis of global surveillance data.  The Lancet Global Health, 11(7):e1012–e1023, 2023.
  * [4] Catharina E van Ewijk, Fuminari Miura, Gini van Rijckevorsel, Henry JC de Vries, Matthijs RA Welkers, Oda E van den Berg, Ingrid HM Friesema, Patrick R van den Berg, Thomas Dalhuisen, Jacco Wallinga, et al.  Mpox outbreak in the netherlands, 2022: public health response, characteristics of the first 1,000 cases and protection of the first-generation smallpox vaccine.  Eurosurveillance, 28(12):2200772, 2023.
  * [5] Pengfei Li, Jiajing Li, Ibrahim Ayada, Amine Avan, Qinyue Zheng, Maikel P Peppelenbosch, Annemarie C de Vries, and Qiuwei Pan.  Clinical features, antiviral treatment, and patient outcomes: a systematic review and comparative analysis of the previous and the 2022 mpox outbreaks.  The Journal of Infectious Diseases, 228(4):391–401, 2023.
  * [6] Kassiani Mellou, Kyriaki Tryfinopoulou, Styliani Pappa, Kassiani Gkolfinopoulou, Sofia Papanikou, Georgia Papadopoulou, Evangelia Vassou, Evangelia-Georgia Kostaki, Kalliopi Papadima, Elissavet Mouratidou, et al.  Overview of mpox outbreak in greece in 2022–2023: Is it over?  Viruses, 15(6):1384, 2023.
  * [7] C Besombes, F Mbrenga, C Malaka, E Gonofio, Laura Schaeffer, X Konamna, B Selekon, J Namsenei-Dankpea, C Gildas Lemon, Jordi Landier, et al.  Investigation of a mpox outbreak in central african republic, 2021-2022.  One Health, 16:100523, 2023.
  * [8] Marlene K Wolfe, Alexander T Yu, Dorothea Duong, Madhura S Rane, Bridgette Hughes, Vikram Chan-Herur, Marisa Donnelly, Shua Chai, Bradley J White, Duc J Vugia, et al.  Use of wastewater for mpox outbreak surveillance in california.  New England Journal of Medicine, 388(6):570–572, 2023.
  * [9] CDC.  2022 mpox outbreak global map.
  * [10] Durre Aden, Sufian Zaheer, Rohit Kumar, and Sunil Ranga.  Monkeypox (mpox) outbreak during covid-19 pandemic—past and the future.  Journal of medical virology, 95(4):e28701, 2023.
  * [11] Akira Endo, Hiroaki Murayama, Sam Abbott, Ruwan Ratnayake, Carl AB Pearson, W John Edmunds, Elizabeth Fearon, and Sebastian Funk.  Heavy-tailed sexual contact networks and monkeypox epidemiology in the global outbreak, 2022.  Science, 378(6615):90–94, 2022.
  * [12] Kai Kupferschmidt.  Why monkeypox is mostly hitting men who have sex with men.  Science, 376(6600):1364–1365, 2022.
  * [13] Lao-Tzu Allan-Blitz, Monica Gandhi, Paul Adamson, Ina Park, Gail Bolan, and Jeffrey D Klausner.  A position statement on mpox as a sexually transmitted disease.  Clinical Infectious Diseases, 76(8):1508–1512, 2023.
  * [14] Chelsea S Lutz, Mimi P Huynh, Monica Schroeder, Sophia Anyatonwu, F Scott Dahlgren, Gregory Danyluk, Danielle Fernandez, Sharon K Greene, Nodar Kipshidze, Leann Liu, et al.  Applying infectious disease forecasting to public health: a path forward using influenza forecasting examples.  BMC Public Health, 19(1):1–12, 2019.
  * [15] Dylan B George, Wendy Taylor, Jeffrey Shaman, Caitlin Rivers, Brooke Paul, Tara O’Toole, Michael A Johansson, Lynette Hirschman, Matthew Biggerstaff, Jason Asher, et al.  Technology to advance infectious disease forecasting for outbreak management.  Nature communications, 10(1):1–4, 2019.
  * [16] Pei-Ying Kobres, Jean-Paul Chretien, Michael A Johansson, Jeffrey J Morgan, Pai-Yei Whung, Harshini Mukundan, Sara Y Del Valle, Brett M Forshey, Talia M Quandelacy, Matthew Biggerstaff, et al.  A systematic review and evaluation of zika virus forecasting and prediction research during a public health emergency of international concern.  PLoS neglected tropical diseases, 13(10):e0007451, 2019.
  * [17] Evan L Ray, Nutcha Wattanachit, Jarad Niemi, Abdul Hannan Kanji, Katie House, Estee Y Cramer, Johannes Bracher, Andrew Zheng, Teresa K Yamana, Xinyue Xiong, et al.  Ensemble forecasts of coronavirus disease 2019 (covid-19) in the us.  MedRXiv, 2020.
  * [18] Cécile Viboud and Alessandro Vespignani.  The future of influenza forecasts.  Proceedings of the National Academy of Sciences, 116(8):2802–2804, 2019.
  * [19] Matthew Biggerstaff, Michael Johansson, David Alper, Logan C Brooks, Prithwish Chakraborty, David C Farrow, Sangwon Hyun, Sasikiran Kandula, Craig McGowan, Naren Ramakrishnan, et al.  Results from the second year of a collaborative effort to forecast influenza seasons in the united states.  Epidemics, 24:26–33, 2018.
  * [20] Craig J McGowan, Matthew Biggerstaff, Michael Johansson, Karyn M Apfeldorf, Michal Ben-Nun, Logan Brooks, Matteo Convertino, Madhav Erraguntla, David C Farrow, John Freeze, et al.  Collaborative efforts to forecast seasonal influenza in the united states, 2015–2016.  Scientific reports, 9(1):1–13, 2019.
  * [21] CDC.  Mpox technical reports.
  * [22] Epidemic prediction initiative.  https://predict.cdc.gov/.  Accessed: 2023-04-08.
  * [23] Sara Y Del Valle, Benjamin H McMahon, Jason Asher, Richard Hatchett, Joceline C Lega, Heidi E Brown, Mark E Leany, Yannis Pantazis, David J Roberts, Sean Moore, et al.  Summary results of the 2014-2015 darpa chikungunya challenge.  BMC infectious diseases, 18(1):1–14, 2018.
  * [24] Karen M Holcomb, Sarabeth Mathis, J Erin Staples, Marc Fischer, Christopher M Barker, Charles B Beard, Randall J Nett, Alexander C Keyel, Matteo Marcantonio, Marissa L Childs, et al.  Evaluation of an open forecasting challenge to assess skill of west nile virus neuroinvasive disease prediction.  Parasites & Vectors, 16(1):1–13, 2023.
  * [25] Yue Teng, Dehua Bi, Guigang Xie, Yuan Jin, Yong Huang, Baihan Lin, Xiaoping An, Dan Feng, and Yigang Tong.  Dynamic forecasting of zika epidemics using google trends.  PloS one, 12(1):e0165085, 2017.
  * [26] Sarah F McGough, John S Brownstein, Jared B Hawkins, and Mauricio Santillana.  Forecasting zika incidence in the 2016 latin america outbreak combining traditional disease surveillance with search, social media, and news report data.  PLoS neglected tropical diseases, 11(1):e0005295, 2017.
  * [27] Yien Ling Hii, Huaiping Zhu, Nawi Ng, Lee Ching Ng, and Joacim Rocklöv.  Forecast of dengue incidence using temperature and rainfall.  PLoS neglected tropical diseases, 6(11):e1908, 2012.
  * [28] Mathew V Kiang, Mauricio Santillana, Jarvis T Chen, Jukka-Pekka Onnela, Nancy Krieger, Kenth Engø-Monsen, Nattwut Ekapirat, Darin Areechokchai, Preecha Prempree, Richard J Maude, et al.  Incorporating human mobility data improves forecasts of dengue fever in thailand.  Scientific reports, 11(1):1–12, 2021.
  * [29] Dave Osthus.  Fast and accurate influenza forecasting in the united states with inferno.  PLoS computational biology, 18(1):e1008651, 2022.
  * [30] Graham C Gibson, Nicholas G Reich, and Daniel Sheldon.  Real-time mechanistic bayesian forecasts of covid-19 mortality.  MedRxiv, 2020.
  * [31] Michal Ben-Nun, Pete Riley, James Turtle, David P Bacon, and Steven Riley.  Forecasting national and regional influenza-like illness for the usa.  PLoS computational biology, 15(5):e1007013, 2019.
  * [32] James Turtle, Pete Riley, Michal Ben-Nun, and Steven Riley.  Accurate influenza forecasts using type-specific incidence data for small geographic units.  PLoS Computational Biology, 17(7):e1009230, 2021.
  * [33] Rebecca K Borchering, Cécile Viboud, Emily Howerton, Claire P Smith, Shaun Truelove, Michael C Runge, Nicholas G Reich, Lucie Contamin, John Levander, Jessica Salerno, et al.  Modeling of future covid-19 cases, hospitalizations, and deaths, by vaccination rates and nonpharmaceutical intervention scenarios—united states, april–september 2021.  Morbidity and Mortality Weekly Report, 70(19):719, 2021.
  * [34] Mariusz Maziarz and Martin Zach.  Agent-based modelling for sars-cov-2 epidemic prediction and intervention assessment: A methodological appraisal.  Journal of Evaluation in Clinical Practice, 26(5):1352–1360, 2020\.
  * [35] Alexander D Becker, Kyra H Grantz, Sonia T Hegde, Sophie Bérubé, Derek AT Cummings, and Amy Wesolowski.  Development and dissemination of infectious disease dynamic transmission models during the covid-19 pandemic: what can we learn from other pathogens and how can we move forward?  The Lancet Digital Health, 3(1):e41–e50, 2021.
  * [36] Emily Howerton, Matthew J Ferrari, Ottar N Bjørnstad, Tiffany L Bogich, Rebecca K Borchering, Chris P Jewell, James D Nichols, William JM Probert, Michael C Runge, Michael J Tildesley, et al.  Synergistic interventions to control covid-19: Mass testing and isolation mitigates reliance on distancing.  PLoS computational biology, 17(10):e1009518, 2021.
  * [37] Joseph C Lemaitre, Kyra H Grantz, Joshua Kaminsky, Hannah R Meredith, Shaun A Truelove, Stephen A Lauer, Lindsay T Keegan, Sam Shah, Josh Wills, Kathryn Kaminsky, et al.  A scenario modeling pipeline for covid-19 emergency planning.  Scientific reports, 11(1):7534, 2021.
  * [38] Michael C Runge, Katriona Shea, Emily Howerton, Katie Yan, Harry Hochheiser, Erik Rosenstrom, William JM Probert, Rebecca Borchering, Madhav V Marathe, Bryan Lewis, et al.  Scenario design for infectious disease projections: Integrating concepts from decision analysis and experimental design.  medRxiv, 2023.
  * [39] Hans Heesterbeek, Roy M Anderson, Viggo Andreasen, Shweta Bansal, Daniela De Angelis, Chris Dye, Ken TD Eames, W John Edmunds, Simon DW Frost, Sebastian Funk, et al.  Modeling infectious disease dynamics in the complex landscape of global health.  Science, 347(6227):aaa4339, 2015.
  * [40] Michael Haber, Ira M Longini Jr, and George A Cotsonis.  Models for the statistical analysis of infectious disease data.  Biometrics, pages 163–173, 1988.
  * [41] Niels G Becker and Tom Britton.  Statistical studies of infectious disease incidence.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(2):287–307, 1999.
  * [42] Lu Tang, Yiwang Zhou, Lili Wang, Soumik Purkayastha, Leyao Zhang, Jie He, Fei Wang, and Peter X-K Song.  A review of multi-compartment infectious disease models.  International Statistical Review, 88(2):462–513, 2020.
  * [43] Constantinos I Siettos and Lucia Russo.  Mathematical modeling of infectious disease dynamics.  Virulence, 4(4):295–306, 2013.
  * [44] C Jessica E Metcalf and Justin Lessler.  Opportunities and challenges in modeling emerging infectious diseases.  Science, 357(6347):149–152, 2017.
  * [45] Maimuna S Majumder, Mauricio Santillana, Sumiko R Mekaru, Denise P McGinnis, Kamran Khan, and John S Brownstein.  Utilizing nontraditional data sources for near real-time estimation of transmission dynamics during the 2015-2016 colombian zika virus disease outbreak.  JMIR public health and surveillance, 2(1):e5814, 2016.
  * [46] Emily L Aiken, Sarah F McGough, Maimuna S Majumder, Gal Wachtel, Andre T Nguyen, Cecile Viboud, and Mauricio Santillana.  Real-time estimation of disease activity in emerging outbreaks using internet search information.  PLoS computational biology, 16(8):e1008117, 2020.
  * [47] Canelle Poirier, Wei Luo, Maimuna S Majumder, Dianbo Liu, Kenneth D Mandl, Todd A Mooring, and Mauricio Santillana.  The role of environmental factors on transmission rates of the covid-19 outbreak: an initial assessment in two spatial scales.  Scientific reports, 10(1):17002, 2020.
  * [48] Thomas McAndrew, Allison Codi, Juan Cambeiro, Tamay Besiroglu, David Braun, Eva Chen, Luis Enrique Urtubey De Cèsaris, and Damon Luk.  Chimeric forecasting: combining probabilistic predictions from computational models and human judgment.  BMC Infectious Diseases, 22(1):1–17, 2022.
  * [49] Allison Codi, Damon Luk, David Braun, Juan Cambeiro, Tamay Besiroglu, Eva Chen, Luis Enrique Urtubey de Cesaris, Paolo Bocchini, and Thomas McAndrew.  Aggregating Human Judgment Probabilistic Predictions of Coronavirus Disease 2019 Transmission, Burden, and Preventive Measures.  Open Forum Infectious Diseases, 9(8), 07 2022.  ofac354.
  * [50] Nikos I Bosse, Sam Abbott, Johannes Bracher, Habakuk Hain, Billy J Quilty, Mark Jit, Centre for the Mathematical Modelling of Infectious Diseases COVID-19 Working Group, Edwin van Leeuwen, Anne Cori, and Sebastian Funk.  Comparing human and model-based forecasts of covid-19 in germany and poland.  PLoS computational biology, 18(9):e1010405, 2022.
  * [51] David C Farrow, Logan C Brooks, Sangwon Hyun, Ryan J Tibshirani, Donald S Burke, and Roni Rosenfeld.  A human judgment approach to epidemiological forecasting.  PLoS computational biology, 13(3):e1005248, 2017.
  * [52] Srinivasan Venkatramanan, Juan Cambeiro, Tom Liptay, Bryan Lewis, Mark Orr, Gaia Dempsey, Alex Telionis, Justin Crow, Chris Barrett, and Madhav Marathe.  Utility of human judgment ensembles during times of pandemic uncertainty: A case study during the covid-19 omicron ba. 1 wave in the usa.  medRxiv, 2022.
  * [53] Gabriel Recchia, Alexandra LJ Freeman, and David Spiegelhalter.  How well did experts and laypeople forecast the size of the covid-19 pandemic?  PloS one, 16(5):e0250935, 2021.
  * [54] David Braun, Daniel Ingram, B Kahn, J Marsh, and T McAndrew.  Crowdsourced perceptions and covid-19: Improving computational forecasts of us national incident cases of covid-19 with crowdsourced perceptions of human behavior.  JMIR Public Health and Surveillance, 2022.
  * [55] Thomas McAndrew and Nicholas G Reich.  An expert judgment model to predict early stages of the covid-19 pandemic in the united states.  PLoS Computational Biology, 18(9):e1010485, 2022.
  * [56] Thomas McAndrew, Maimuna S Majumder, Andrew A Lover, Srini Venkatramanan, Paolo Bocchini, Tamay Besiroglu, Allison Codi, David Braun, Gaia Dempsey, Sam Abbott, et al.  Early human judgment forecasts of human monkeypox, may 2022.  The Lancet Digital Health, 4(8):e569–e571, 2022.
  * [57] Thomas McAndrew, Casey Gibson, David Braun, Abhishek Srivastava, and Kate Brown.  Chimeric forecasting: An experiment to leverage human judgment to improve forecasts of infectious disease using simulated surveillance data.  Epidemics, page 100756, 2024.
  * [58] Thomas McAndrew, Juan Cambeiro, and Tamay Besiroglu.  Aggregating human judgment probabilistic predictions of the safety, efficacy, and timing of a covid-19 vaccine.  Vaccine, 40(15):2331–2341, 2022.
  * [59] Thomas McAndrew, Nutcha Wattanachit, Graham C Gibson, and Nicholas G Reich.  Aggregating predictions from experts: A review of statistical methods, experiments, and applications.  Wiley Interdisciplinary Reviews: Computational Statistics, 13(2):e1514, 2021.
  * [60] Metaculus.  https://www.metaculus.com/questions/.  Accessed: 2023-04-08.
  * [61] Global health a data science initiative.  https://global.health/.  Accessed: 2023-04-08.
  * [62] European entre for disease prevention and control.  https://www.ecdc.europa.eu/en.  Accessed: 2023-04-08.
  * [63] J Scott Armstrong.  Judgmental bootstrapping: Inferring experts’ rules for forecasting.  Principles of forecasting: A handbook for researchers and practitioners, pages 171–192, 2001.
  * [64] Gerda Ana Melnik-Leroy, Linas Aidokas, Gintautas Dzemyda, Giedrė Dzemydaitė, Virginijus Marcinkevičius, Vytautas Tiešis, and Ana Usovaitė.  Is my visualization better than yours? analyzing factors modulating exponential growth bias in graphs.  Frontiers in Psychology, 14:1125810, 2023.
  * [65] William A Wagenaar and Sabato D Sagaria.  Misperception of exponential growth.  Perception & Psychophysics, 18:416–422, 1975.
  * [66] Andrew F Heckler, Brendon Mikula, and Rebecca Rosenblatt.  Student accuracy in reading logarithmic plots: The problem and how to fix it.  In 2013 IEEE frontiers in education conference (FIE), pages 1066–1071. IEEE, 2013.
  * [67] Du Phan, Neeraj Pradhan, and Martin Jankowiak.  Composable effects for flexible and accelerated probabilistic programming in numpyro.  arXiv preprint arXiv:1912.11554, 2019.
  * [68] Johannes Bracher, Evan L. Ray, Tilmann Gneiting, and Nicholas G. Reich.  Evaluating epidemic forecasts in an interval format.  PLoS computational biology, 17(2):e1008618, February 2021.
  * [69] Nikos I Bosse, Hugo Gruson, Anne Cori, Edwin van Leeuwen, Sebastian Funk, and Sam Abbott.  Evaluating forecasts with scoringutils in r.  arXiv preprint arXiv:2205.07090, 2022.
  * [70] Fabiana Scapolo and Ian Miles.  Eliciting experts’ knowledge: A comparison of two methods.  Technological forecasting and social change, 73(6):679–704, 2006\.
  * [71] Robert R Hoffman, John W Coffey, Kenneth M Ford, and Joseph D Novak.  A method for eliciting, preserving, and sharing the knowledge of forecasters.  Weather and Forecasting, 21(3):416–428, 2006.
  * [72] Anthony O’Hagan.  Expert knowledge elicitation: subjective but scientific.  The American Statistician, 73(sup1):69–81, 2019.
  * [73] Centers for Disease Control, Prevention (CDC, et al.  Update: multistate outbreak of monkeypox–illinois, indiana, kansas, missouri, ohio, and wisconsin, 2003.  MMWR. Morbidity and mortality weekly report, 52(24):561–564, 2003\.

##  V Figures

Figure 1:  Human judgment ensemble forecasts for one week ahead and up to
eight weeks ahead for six questions where the truth was collected. The dashed,
black vertical is when the ground truth was determined. The one week ahead
forecast is closest the dashed line, the two week ahead is second closest,
etc. Black circles mark the observed values at one through eight weeks ahead.
The 10th predictive quantile for human judgment is the bottom of the line, the
90th quantile is the top of the line, and the median (50th quantile) is
presented as a circle.  Figure 2: Individual median predictions (gray lines)
of continuous targets from three weeks to one hour before the ground truth in
intervals of one hour, and the average (opaque blue line) plus 95% confidence
interval (lighter blue shading) for the median prediction. The ground truth is
a solid black line.  Figure 3: The (A.) median absolute error (MAE) and (B.)
weighted interval score (WIS) for the auto-regressive model (AR), doubling
time, and human judgment ensemble (HJE) relative to the random walk. MAE/WIS
are plotted for each individual question and summarized with a boxplot. A
horizontal line in black is plotted at the value 1 (i.e. a forecast that
performs equally to a forecast generated from the random walk model).  Figure
4: (A.) Relative WIS between the human judgment ensemble (HJE) and a random
walk (RW) over forecast horizons of 1-8 weeks and stratified by forecasts
solicited on a logarithmic (green) or natural (blue) scale. Values less than 0
indicate human judgment ensemble forecasts performed better and values greater
than 0 indicate the random walk model performed better. (B.)-(D.) Predictive
densities for the human judgment ensemble (orange) and random walk (blue) at 5
week (B. and D.) and 2 week (C. and E.) forecast horizons for the number of
countries to report at least one mpox case (natural scale) and number of
reported cases in Canada (logarithmic scale)  Figure 5:  A forecast for the
number of mpox cases in Canada generated by a randomly selected single human 5
weeks before the ground truth: (A.) on the logorithmic scale which was used by
the Metaculus platform to capture forecasts, (B.) on the transformed natural
scale for values smaller than 1000 cases, and (C.) the forecast on the natural
scale over all possible values solicited on the platform. (D.) The mode/median
(a measure of skew) for individual human judgment forecasts at forecast
horizons 1-8, stratified by questions that solicited forecasts on a
logarithmic scale (green) and on natural scale (blue).  Figure 6: The observed
value of the target quantity at the time of prediction (horizonal axis) vs the
mode of individual human judgment forecasts (vertical axis) stratified by
target and by forecast horizon. Red line denotes the final ground truth. Black
line is the identity line.  Figure 7:  Empirical cumulative density function
(ecdf) of (A.) the number of days from when a question was posed to the crowd
to when an individual submitted their first forecast, (B.) when an individual
submitted any forecast, including revisions, and (C.) when an individual
submitted their last forecast. Below the ecdfs are corresponding medians
(colored circles), 25th percentiles (left side of bars) and 75th percentiles
(right side of bars). Black circles represent the number of days from when the
question was posed to when the question was closed for submission.

##  VI Tables

Parameter | $\hat{\beta}$ | 95CI | p
---|---|---|---
Intercept | 0.42 | [-0.07,0.77] | 0.02
Log scale (vs Nat scale) | -0.21 | [ -0.48,0.08] | 0.15
Forecast horizon | -0.16 | [-0.24,-0.08] | 0.01
Table 1: Linear regression fit to WIS (dependent variable), whether the forecast was solicited on a logarithmic (vs natural) scale, and forecast horizon. We present estimated coefficients ($\hat{\beta}$), 95% confidence intervals (95CI), and associated pvalues (p)  Question/Target | Spearmens rho | p
---|---|---
Number of Cases Canada |  |
1-3 weeks ahead | 0.72 | $<0.01$
4-6 weeks ahead | 0.43 | $<0.01$
Number of US states |  |
1-3 weeks ahead | 0.53 | $<0.01$
4-6 weeks ahead | -0.23 | 0.20
Number of US cases |  |
1-3 weeks ahead | 0.73 | $<0.01$
4-6 weeks ahead | 0.40 | $<0.01$
Number of Europe cases |  |
1-3 weeks ahead | 0.76 | $<0.01$
4-6 weeks ahead | 0.34 | $<0.01$
Number of European countires |  |
1-3 weeks ahead | 0.41 | $<0.01$
4-6 weeks ahead | 0.29 | $<0.01$
7-8 weeks ahead | 0.27 | $<0.01$
Table 2: Spearman’s correlation and associated pvalue to assess the association between the mode of individual forecasts and the most recently observed value at the time of the forecast. Frequency |  Theme |  Example
---|---|---
5 |  Taking cues from external data and news |  BNO: NEW: Washington state reports 1st case of monkeypox
2 |  Introducing code or algorithms for how to produce a forecast |  Since this value [total number of reported suspected+confirmed cases on July 1, 2022 at 12pm EST according to this spreadsheet that is being actively updated by Global.health] changes throughout the day, I am including some code now, so that come 12 pm EST, another person can verify the # of confirmed + suspected cases in Europe.
5 |  Signaling to the crowd data updates |  Based on the spreadsheet, 45 countries with suspected+confirmed cases so far.
4 |  Confusion about the target or ground truth |  Good Morning. Is 82 the correct resolution? Thanks
2 |  Introspection about past forecast |  Just as post-mortem now that the question is resolved, I want to say that I still stand behind [sic] my reasoning[…]
6 |  Taking cues from epidemiology or surveillance |  Interesting context on the vote taken by the WHO committee in which they did not recommend declaration of a PHEIC:
Table 3: The most common themes among comments submitted by forecasters.

##  Appendix A Supplemental

###  A.1 Forecast submission

A forecaster on the Metaculus platform can submit a predictive density $f$ as
a convex combination of up to five logistic distributions

$\displaystyle f(x)$ $\displaystyle=\sum_{i=1}^{5}\pi_{k}\,g(x|\mu_{k},s_{k})$
(1) $\displaystyle g(x)$
$\displaystyle=\frac{\exp\left(-(x-\mu)/s\right)}{s(1+\exp\left(-(x-\mu)/s\right))}$
(2)

where $\pi_{k}$ is a non-negative weight associated with the $k$th logistic
distribution $g(x|\mu_{k},s_{k})$ and all weights are constrained to sum to
unity. By default, a forecaster is presented with a single logistic density
with a slider bar underneath this density that contains a square and two
circles (see supplemental figure 8). A forecaster can change the value of
$\mu$ by shifting the square to the left for lower values and to the right for
higher values. The parameter $s$ is adjusted by sliding the circles left and
right independently. If they wish, a forecaster may press “add a component” to
add additional logistic distributions (up to 5 component distributions in
total). A second slider will appear under the first and two additional sliders
that control the weights ($\pi$) associated with the first and second logistic
distribution. The predictive density a user creates is continually updated in
the browser to facilitate iteratively building their forecast for
submission.They are also shown the Community Forecast [needs to be defined]
along with a Current Points calculation reflecting the points they may
received based on the resolution (see
https://www.metaculus.com/help/faq/#howscore for more details). When a
forecaster is satisfied with their prediction, they press the submit button.
After the first submission, a forecaster may revise their original prediction
as many times as they choose until the question is closed on the platform or
the ground truth is reported. The ensemble was hidden from individuals for X
days or until there were X forecasts, whichever came first.

Forecasters construct their predictive density over a bounded interval,
presented either on a linear scale or on a log scale (base 10). The choice of
intervals, and linear vs. log scale, was made by forecast question developers
at Metaculus. After 10 predictions from 10 unique forecasters were submitted
to the Metaculus platform, an ensemble of these predictions was revealed to
the community. In addition to a formatted method for building a forecast,
users were also able to post comments that could be viewed by all other users
of the Metaculus system.

###  A.2 Ensemble building

An ensemble predictive density $f$ in response to a question posed on the
Metaculus platform was created as a weighted combination of all $M$ individual
forecaster densities

$\displaystyle f(x)=\sum_{m=1}^{M}\pi_{m}f_{m}(x)$ (3)

where $f_{m}$ is a predictive density submitted by user $m$ and the weights
assigned to forecaster $m$ are annotated as $\pi_{m}$. Weights could be a
function of the forecaster’s accuracy on past questions and of how recently
they submitted a prediction, but in this study we chose to weight all
forecasts equally (i.e., we assigned $\pi_{m}=1/M\;\forall m$). The above
ensemble strategy is used for questions where a forecaster submits a
predictive density over a closed interval (a continuum). For questions
entailing answers in the form of a scalar number, such as when forecasters are
asked to submit a single probability of a particular event (see question (vi)
above), the ensemble strategy build an empirical cumulative density function
from predicted probabilities (See Supp. Fig. 9 bottom right panel for an
example).

Question |  Link to Metaculus question | Data source | Scale
---|---|---|---
Total cases in Europe |  https://www.metaculus.com/questions/10978/total-monkeypox-cases-in-europe-july-1-2022/ | Global health | Logarithmic
Total cases in US |  https://www.metaculus.com/questions/10979/total-monkeypox-cases-in-usa-on-july-1-2022/ | Global health | Logarithmic
Total cases in Canada |  https://www.metaculus.com/questions/11039/monkeypox-cases-in-canada-on-july-1-2022/ | Global health | Logarithmic
Countries with at least one case |  https://www.metaculus.com/questions/10975/countries-with-monkeypox-by-july-31-2022/ | Global health | Natural
Number of US states |  https://www.metaculus.com/questions/10981/-us-states-with-monkeypox-cases-on-july-1/ | CDC | Natural
WHO declaration of PHEIC |  https://www.metaculus.com/questions/10977/who-declares-that-monkeypox-is-a-pheic/ | WHO | Natural
Table 4:  Listing of data sources, forecast horizon cut points, and the data
sources that were used as resolution criteria for questions posed to users on
Metaculus. All datasets started

These questions were: (i) what will be the total number of confirmed and
suspected monkeypox cases in Europe as of July 1, 2022?, (ii) what will be the
total number of confirmed and suspected monkeypox cases in the United States
as of July 1, 2022?, (iii) what will be the total number of confirmed and
suspected monkeypox cases in Canada as of July 1, 2022? (iv) how many
countries will confirm at least one case of monkeypox by July 31, 2022?, (v)
how many states in the United States will report having a case of monkeypox as
of July 1, 2022?, and (vi) will the World Health Organization declare the
spread of monkeypox a Public Health Emergency of International Concern before
2023? We received a total of 1275 probabilistic predictions (initial forecasts
or subsequent updates) from 442 individual forecasters.

Figure 8: Example of the Metaculus platform used by forecasters to generate
probabilistic predictions of questions posed about monkeypox.

##  Appendix B Figures

Figure 9: Human judgment ensemble predictive densities one (blue), two (red),
and three (purple) weeks ahead of the date of the true value (black vertical
line). Below each plot is presented a 25th quantile (leftmost point of the
line), 50th quantile (dot), and 75th quantile (rightmost point of the line).
Figure 10: The proportion plus 95% confidence intervals of individual median
predictions that were greater than the ground truth for five continuous
targets from three weeks before the ground truth to one week before the ground
truth in intervals of one hour. The majority of human judgment forecasters
submitted densities with a median below the ground truth. Only when asked to
predict the total number of cases in Canada did human forecasters over
estimate the severity of the mpox outbreak.


capbtabboxtable[][]

# Duluth at SemEval-2020 Task 7:
Using Surprise as a Key to Unlock Humorous Headlines

Shuning Jin1 Yue Yin2 XianE Tang2 Ted Pedersen2
1Rutgers University 2University of Minnesota Duluth
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We use pretrained transformer-based language models in SemEval-2020 Task 7:
Assessing the Funniness of Edited News Headlines. Inspired by the incongruity
theory of humor, we use a contrastive approach to capture the surprise in the
edited headlines. In the official evaluation, our system gets 0.531 RMSE in
Subtask 1, 11th among 49 submissions. In Subtask 2, our system gets 0.632
accuracy, 9th among 32 submissions.

##  1 Introduction

00footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International License. License details:
http://creativecommons.org/licenses/by/4.0/.

Humor detection is a challenging problem in natural language processing.
SemEval-2020 Task 7 [Hossain et al., 2020a]111Competition page:
https://competitions.codalab.org/competitions/20970 focuses on detecting humor
in English news headlines with micro-edits. Specifically, the edited headlines
have one selected word or entity that is replaced by editors, which are then
graded by the degree of funniness. Accurate scoring of the funniness from
micro-edits can serve as a footstone of humorous text generation [Hossain et
al., 2020a].

Inspired by the incongruity theory [Veale, 2004, Morreall, 2016], we believe
that contrast and surprise is a key ingredient of humor. We instantiate this
intuition with a contrastive framework. We then systematically compare three
widely used models: CBOW, BERT [Devlin et al., 2019], and RoBERTa [Liu et al.,
2019], providing a benchmark for this task. Our best system, based on RoBERTa,
achieves compelling performance for both subtasks. Our code is available on
GitHub.222Code: https://github.com/dora-tang/SemEval-2020-Task-7

##  2 Related Work

Early humor recognition systems are mostly based on traditional machine
learning methods, such as support vector machine, decision tree, Naive Bayes,
and $k$-nearest neighbors [Castro et al., 2016]. Besides, an $n$-gram language
model shows good performance [Yan and Pedersen, 2017] in learning a sense of
humor from tweets. Yet $n$-gram models are limited to a small number of
context words.

Pretrained language models based on Transformer [Vaswani et al., 2017] can
obtain contextual information of a whole sentence. Among this family, BERT has
been used to assess the humor in tweets and jokes [Mao and Liu, 2019, Weller
and Seppi, 2019]. Enlightened by these recent advances, we use BERT to judge
the funniness of edited news headlines. We additionally experiment with
RoBERTa, a robustly optimized variant of BERT.

Lastly, several works also attempt to explicitly model incongruity and
surprise of humourous text, focusing on homophonic puns. ?) formalizes
incongruity as a mixed effect of ambiguity and distinctiveness, quantified by
entropy and Kullback-Leibler divergence. ?) proposes a local-global surprisal
measure based on the log-likelihood ratio, to assess whether a sentence is a
pun. However, we focus on a broader definition of humor, and formulate
incongruity as an input pair to a dual encoder framework.

##  3 Task Data

The Humicroedit dataset [Hossain et al., 2019] provides the training,
development, and test data for this task. We also use additional training data
from the FunLines dataset [Hossain et al., 2020b]. The dataset statistics are
summarized in Appendix A. In Subtask 1, the goal is predicting the funniness
score of an edited headline. The score $z$ ranges from 0 to 3, where 0 means
not funny and 3 means very funny. In Subtask 2, the goal is to predict the
funnier between an edited sentence pair. For labels
$y\in\left\\{0,1,2\right\\}$, 0 implies two headlines are equally funny, 1
implies the first is the funnier, and 2 implies the second is the funnier.
Examples of the two subtasks are in Table 1.

Task | ID | Original Headline | Edit | Score | Label
---|---|---|---|---|---
Subtask 1 | 33210 | California and President Trump are going to war with each other | monkeys | 1.8 | /
1664 | What if sociologist had as much influence as economists? | donkeys | 2.8 | /
Subtask 2 | 9934 | Chibok girls reunited with families | smartphones | 1.8 | 1
14279 | Chibok girls reunited with families | cats | 0.0
10920 | Gene Cernan, last astronaut on the Moon, dies at 82 | dancer | 1.2 | 1
9866 | Gene Cernan, last astronaut on the Moon, dies at 82 | impregnated | 0.8
Table 1: Examples in train data of both subtasks. Underlined words in original
headlines are to be substituted by the edit words.

##  4 Methods

What are the important characteristics of humor? The incongruity theory, a
dominant theory of humor, states that “it is the perception of something
incongruous—something that violates our mental patterns and expectations”
[Morreall, 2016]. Therefore, we hypothesize an edited headline is funny if the
edit words are semantically distant from the context words or the original
words. This can be exemplified by the first two examples in Table 1. We start
by looking at Headline 33210 (the Monkey Example), also shown below. The
context sentence is extracted from the edit sentence by replacing the edit
words with a single [MASK] token, motivated by masked language models. Humans
are likely to predict a place or a character for the masked token, while the
edit token is “monkeys”.

Original |  $\underset{1}{\textsc{California}}$ $\underset{2}{\textsc{And}}$ $\underset{i=3}{\textsc{\text@underline{President}}}$ $\underset{j=4}{\textsc{\text@underline{Trump}}}$ $\underset{5}{\textsc{Are}}$ $\underset{6}{\textsc{Going}}$ $\underset{7}{\textsc{To}}$ $\underset{8}{\textsc{War}}$ | $T_{o}=8$
---|---|---
Edit |  $\underset{1}{\textsc{California}}$ $\underset{2}{\textsc{And}}$ $\underset{i=3,\;k=3}{\textsc{\text@underline{Monkeys}}}$ $\underset{4}{\textsc{Are}}$ $\underset{5}{\textsc{Going}}$ $\underset{6}{\textsc{To}}$ $\underset{7}{\textsc{War}}$ | $T_{e}=7$
Context |  $\underset{1}{\textsc{California}}$ $\underset{2}{\textsc{And}}$ $\underset{i=3}{\textsc{\text@underline{[Mask]}}}$ $\underset{4}{\textsc{Are}}$ $\underset{5}{\textsc{Going}}$ $\underset{6}{\textsc{To}}$ $\underset{7}{\textsc{War}}$ | $T_{c}=7$

Similarly for Headline 1664, given the context What If [MASK] Had As Much
Influence As Economists, humans might fill in occupation-related words like
“scientist” or “sociologist” (as in the original headline). However, the edit
word is “donkeys”, which is a surprising prediction and is considered very
funny (scored 2.8 out of 3). In this section, we describe a concrete
architecture that models the strength of contrast and surprise, which
translates into the funniness score.

###  4.1 Span Representation

Figure 1: Transformer architecture to predict the funniness score of an edited
headline using edit-context sentence pair. The full edited sentence is “What
if donkeys had as much influence as economists”. “donkeys” is the edited word
and is tokenized into two subwords in this example.

Let $\mathbf{x}=(x_{1}\ldots x_{T_{o}})$,
$\tilde{\mathbf{x}}=(\tilde{x}_{1}\ldots\tilde{x}_{T_{e}})$,
$\mathbf{x}^{\star}=(x^{\star}_{1}\ldots x^{\star}_{T_{c}})$ denote the
original, edit, context token sequences. A pretrained word embedding or
pretrained encoder maps the tokens into vector sequences
$\mathbf{e}_{1:T_{o}}$, $\tilde{\mathbf{e}}_{1:T_{e}}$,
$\mathbf{e}^{\star}_{1:T_{c}}$. The goal is to encode edit sentence, original
sentence, context sentence into fixed-length vector representations
$\mathbf{u}$, $\mathbf{v}^{\prime}$, $\mathbf{v}$ $\in\mathbb{R}^{d}$.
Importantly, we use span (a.k.a. sub-sentence) representation rather than
whole sentences, which corresponds to the underlined ranges in the above
Monkey Example. Denote a span as a tuple of start and end position of
contiguous tokens: $s=[i,j]$ for edit, $\tilde{s}=[i,k]$ for original, and
$s^{\star}=[i,i]$ for the [MASK] token.

#### CBOW

We first explore context-independent word representations. We use pretrained
GloVe [Pennington et al., 2014] vectors with $d=300$ and a vocabulary of 2.2
million words. We use word averaging to get edit word vector
$\mathbf{u}=\operatorname*{MeanPool}(\tilde{\mathbf{e}}_{i}\ldots\tilde{\mathbf{e}}_{k})$
and original word vector
$\mathbf{v}^{\prime}=\operatorname*{MeanPool}(\mathbf{e}_{i}\ldots\mathbf{e}_{j})$.
We max pool all context words to extract the most salient features. The
context vector is
$\mathbf{v}=\operatorname*{MaxPool}(\mathbf{e}^{\star}_{1}\ldots\mathbf{e}^{\star}_{i-1},\mathbf{e}^{\star}_{i+1}\ldots\mathbf{e}^{\star}_{T_{c}})$.

#### Transformer

We use pretrained transformer-based language models to obtain contextual word
representations. The architecture is shown in Figure 1, a Siamese network
[Bromley et al., 1994] where the two encoders have identical structures and
shared parameters.333An alternative architecture is to use a single
transformer to encode the concatenation of a sentence pair, which enforces
cross-sentence attention. However, since the paired sentences here are almost
identical, cross-sentence attention seems unnecessary. With the self-attention
mechanism, each word attends to all other words in the sentence and aggregates
contextual information. The edit and original vectors are obtained by
averaging:
$\mathbf{u}=\operatorname*{MeanPool}(\tilde{\mathbf{e}}_{i}\ldots\tilde{\mathbf{e}}_{k})$,
$\mathbf{v}^{\prime}=\operatorname*{MeanPool}(\mathbf{e}_{i}\ldots\mathbf{e}_{j})$.
The context vector is simply the contextual embedding of the masked token:
$\mathbf{v}=\mathbf{e}^{\star}_{i}$. We experiment with BERT and RoBERTa,
using the PyTorch [Paszke et al., 2019] implementation from HuggingFace
Transformers library [Wolf et al., 2019].444HuggingFace Transformers library:
https://github.com/huggingface/transformers We use bert-base-uncased ($L=12$,
$d=768$, lower-cased) and roberta-base ($L=12$, $d=768$).

#### Transfer Paradigm

When using those pretrained word representations, we consider two transfer
paradigms: finetuning (Finetune) and not finetuning (Freeze). In the case of
Freeze, we use fixed word embedding directly as the feature for CBOW. For
transformers, we use a weighted average of hidden layers from the frozen
encoder, with trainable mixing scalars. This approach follows ELMo [Peters et
al., 2018] and the edge probing model [Tenney et al., 2019]. Specifically, the
final aggregated embedding for $i^{\text{th}}$ position is
$\mathbf{e}_{i}=\gamma\sum_{l=0}^{L}\alpha_{l}\;\mathbf{e}_{l,i}$, where
$\gamma$ is a scaling factor, $\alpha_{l}$ is the weight of the
$l^{\text{th}}$ layer, $L$ is the total number of layers, and $l=0$
corresponds to the embedding layer.

###  4.2 Task Specific

#### Regression

As mentioned at the beginning of the section, contrast and surprise is the key
to humor. To represent the pairwise relationship between two vectors, we
derive feature from
$\mathbf{h}=f(\mathbf{x},\mathbf{y})=[\mathbf{x};\,\mathbf{y};\,\left|{\mathbf{x}-\mathbf{y}}\right|;\,\mathbf{x}*\mathbf{y}]\in\mathbb{R}^{4d}$,
where $;\,$ denotes concatenation and $*$ denotes element-wise multiplication.
This feature has been used as the input to the classifier in the sentence pair
tasks of SentEval [Conneau and Kiela, 2018]. To formulate the contrast pair,
we either use edit sentence and its context $f(\mathbf{u},\mathbf{v})$, or
edit sentence and original sentence $f(\mathbf{u},\mathbf{v}^{\prime})$. We
denote the two scenarios as Context and Original respectively. Finally, we use
a classifier to predict the funniness score of the edited headline.
$\hat{z}=\operatorname*{Classifier}(\mathbf{h})\in\mathbb{R}$. The classifier
is a two-layer MLP with 256 hidden dimensions. When finetuning transformers we
use single-layer linear projection instead, since its large number of
parameters have already given us sufficient flexibility. The optimization
objective is mean squared error
$\mathcal{L}=\|\mathbf{z}-\hat{\mathbf{z}}\|^{2}$.

#### Classification

In Subtask 2, we use the same method to predict the scores of two edited
versions $\hat{z}^{(1)}$ and $\hat{z}^{(2)}$. By comparing the scores, the
funnier version is found during evaluation and testing time:
$\hat{y}=\operatorname*{arg\;max}_{i\in\\{1,2\\}}\hat{z}^{(i)}$. The loss
function is
$\mathcal{L}=\|\mathbf{z}^{(1)}-\hat{\mathbf{z}}^{(1)}\|^{2}+\|\mathbf{z}^{(2)}-\hat{\mathbf{z}}^{(2)}\|^{2}$.

| Subtask 1 | Subtask 2
---|---|---
Model | RMSE† | Gain | Spearman | Accuracy† | Gain | Reward | RMSE
Baseline | 0.575 | -0.033 | / | 0.490 | -0.109 | -0.020 | /
CBOW |  |  |  |  |  |  |
with Context+Freeze | 0.542 | 0.000 | 0.319 | 0.599 | 0.000 | 0.184 | 0.546
+Original | 0.559 | -0.017 | 0.243 | 0.599 | 0.000 | 0.169 | 0.760
+Extra | 0.544 | -0.002 | 0.308 | 0.605 | 0.006 | 0.191 | 0.545
+Original+Extra | 0.558 | -0.016 | 0.250 | 0.601 | 0.002 | 0.173 | 0.574
+FT | 0.544 | -0.002 | 0.311 | 0.604 | 0.005 | 0.178 | 0.545
+FT+Original | 0.561 | -0.019 | 0.247 | 0.592 | -0.007 | 0.165 | 0.604
+FT+Extra | 0.548 | -0.006 | 0.313 | 0.606 | 0.007 | 0.188 | 0.547
+FT+Original+Extra | 0.563 | -0.021 | 0.261 | 0.589 | -0.010 | 0.161 | 0.581
BERT |  |  |  |  |  |  |
with Context+Freeze | 0.531 | 0.011 | 0.384 | 0.616 | 0.017 | 0.207 | 0.546
+Original | 0.534 | 0.008 | 0.366 | 0.603 | 0.004 | 0.186 | 0.547
+Extra | 0.530 | 0.012 | 0.382 | 0.615 | 0.016 | 0.207 | 0.530
+Original+Extra | 0.541 | 0.001 | 0.346 | 0.615 | 0.016 | 0.204 | 0.542
+FT | 0.536 | 0.006 | 0.366 | 0.635 | 0.036 | 0.234 | 0.552
+FT+Original | 0.536 | 0.006 | 0.365 | 0.628 | 0.029 | 0.231 | 0.555
+FT+Extra | 0.541 | 0.001 | 0.373 | 0.630 | 0.031 | 0.232 | 0.554
+FT+Original+Extra | 0.533 | 0.009 | 0.387 | 0.629 | 0.030 | 0.236 | 0.550
RoBERTa |  |  |  |  |  |  |
with Context+Freeze | 0.528 | 0.014 | 0.388 | 0.635 | 0.036 | 0.246 | 0.529
+Original | 0.536 | 0.006 | 0.366 | 0.625 | 0.026 | 0.224 | 0.540
+Extra | 0.528 | 0.014 | 0.390 | 0.640 | 0.041 | 0.252 | 0.536
+Original+Extra | 0.533 | 0.009 | 0.368 | 0.618 | 0.019 | 0.207 | 0.543
+FT | 0.534 | 0.008 | 0.382 | 0.649 | 0.050 | 0.254 | 0.535
+FT+Original | 0.527 | 0.015 | 0.425 | 0.650 | 0.051 | 0.254 | 0.538
+FT+Extra | 0.526 | 0.016 | 0.407 | 0.638 | 0.039 | 0.233 | 0.594
+FT+Original+Extra | 0.522 | 0.020 | 0.410 | 0.626 | 0.027 | 0.216 | 0.625
Table 2: Post-evaluation test results. Freeze: no finetuning; FT: finetuning;
Context: use context of edit headline; Original: use original headline; Extra:
use additional training data from the FunLines dataset. RMSE† and Accuracy†
are primary metrics. The best within each model type is bolded, and the
overall best is underlined. Gain measures the improvement over CBOW-Context-
Freeze. In Subtask 1 this is w.r.t. to RMSE and in Subtask 2 w.r.t. accuracy.
Baseline 1 uses the average score and Baseline 2 uses the majority label.

##  5 Experiments and Results

###  5.1 Metrics

For Subtask 1, the primary metric for official ranking is Root Mean Squared
Error (RMSE). In addition, we calculate Spearman’s rank correlation
coefficient which measures the monotonic relationship between predicted scores
and true scores. In the evaluation of Subtask 2, instances with label 0 are
ignored. The primary metric for official ranking is accuracy. As an auxiliary
metric, reward takes pairwise score differences into account:
$r=\frac{1}{N}\sum_{i=1}^{N}(\mathbbm{1}_{\hat{y}_{i}=y_{i}}-\mathbbm{1}_{\hat{y}_{i}\neq
y_{i}})|z_{i}^{(1)}-z_{i}^{(2)}|$, where $y_{i}$ and $\hat{y}_{i}$ are true
labels and predicted labels respectively, and $z_{i}^{(1)}$ and $z_{i}^{(2)}$
are true scores.

###  5.2 Official Evaluation

For the official evaluation, our submitted system is RoBERTa-Freeze-Context.
We use Adam [Kingma and Ba, 2015] optimizer with a learning rate of 1e-3 and
use the 10th epoch. Our system gets 0.531 RMSE for Subtask 1
(11${}^{\text{th}}$ among 49 submissions) and 0.632 accuracy for Subtask 2
(9${}^{\text{th}}$ among 32 submissions) on the test set.555Task leaderboard:
https://competitions.codalab.org/competitions/20970#results. “Evaluation-
Task-1” is for Subtask 1 and “Evaluation-Task-2” is for Subtask 2.

###  5.3 Post-Evaluation

In the post-evaluation phase, we conduct a more extensive search on
hyperparameters and select the best models based on validation performance.
Experiment details and hyperparameters are in Appendix B. We systematically
compare CBOW, BERT, and RoBERTa, and perform an ablation study to understand
the effects of various factors: extra training data, finetuning or freezing
the pretrained embeddings, and using Context or Original feature. The post-
evaluation results on the test set are in Table 2.

#### Contextual Representation

Despite its simplicity, CBOW is surprisingly effective. Its best result is
significantly better than the baseline and is comparable to Subtask 1 #19
(0.547) and Subtask 2 #17 (0.605) on the leaderboard. By comparing the three
models, we see that pretrained language models have better performance than
context-independent word embedding. While results for BERT and RoBERTa are
similar, both of them outperform CBOW, evidencing that contextual information
is essential for humor detection.

####  Context vs. Original

In the ablation study, we first notice that neither finetuning nor using extra
data from the FunLines dataset make much difference for all models.
Interestingly, using different contrast pairs as the feature has different
effects on models. Context is better than Original for CBOW, yet they are
similar for pretrained language models. Why does this happen? CBOW with
Original only uses the information of edited word and original word while
completely neglecting the contextual relation. Pairing CBOW with Context can
alleviate this limitation. On the other hand, pretrained language models
exploit the contextual relation between edit words and context words in both
cases.

##  6 Analysis and Discussion

###  6.1 Non-contrastive Approach

In our main experiment, we focus on the contrastive approach using a sentence
pair (i.e., Context and Original) and show its effectiveness. The remaining
question is, can we predict humor using the edited sentence as the only input?
Thus, we investigate a non-contrastive approach, with a single encoder to
obtain the span representation of the edit sentence. We refer to this as Edit.
This is equivalent to only using the left part in Figure 1. We conduct an
experiment with RoBERTa on both subtasks. The results are in Table 3. We see
that Edit has similar performance as contrastive approaches. We conjecture
that Edit captures contrast implicitly, while Context and Original capture
contrast explicitly by design.

Feature | Subtask 1 RMSE | Subtask 2 Accuracy
---|---|---
Context | 0.526 | 0.649
Original | 0.522 | 0.650
Edit | 0.526 | 0.651
Table 3:  Comparison between non-contrastive and contrastive approaches, based on RoBERTa. Context and Original are contrastive, using the edit-context pair and the edit-original pair as input respectively. Edit is non-contrastive, using the edit sentence only. The results are from test sets of the two subtasks. We tune other hyperparameters on the validation sets and select the best model for each cell.  | Human | Original | Context | Edit
---|---|---|---|---
Human | / | 0.41 | 0.41 | 0.42
Original | 0.42 | / | 0.84 | 0.77
Context | 0.41 | 0.84 | / | 0.80
Edit | 0.43 | 0.79 | 0.81 | /
Table 4: The correlation matrix of human scores and predicted scores from
RoBERTa with different features: Edit, Context, and Original. Lower triangle:
Pearson correlation coefficients. Upper triangle: Spearman correlation
coefficients. Results are from the test set of Subtask 1.

###  6.2 Error Analysis

To understand the relationship between human judgment and model predictions,
we calculate the correlation matrix between true funniness scores and
predicted scores from RoBERTa with different features (Context, Original, and
Edit) for Subtask 1. From Table 4, we see that the models correlate poorly
with human judgment (correlations $\approx$ 0.4), while correlating well with
each other (correlation $\approx$ 0.8).

To further learn when the models make an erroneous judgment, we look at model
predictions on the test set of Subtask 1. We see the models generally capture
the incongruity phenomenon. While being key to many examples, incongruity does
not account for others. We summarize some typical examples in Table 5:

  * •

For Headline 9100, the edit word “children” is incongruous with the context of
national security. While humans consider it not funny at all, models assign a
high funniness score. The fallacy is that incongruity is not a sufficient
condition for humor.

  * •

In other cases, the edit words are congruous with the context. While humans
consider them very funny, models predict the opposite. That is, incongruity is
not a necessary condition for humor. Humor has diverse underlying causes. For
instance, Headline 12685 shows sarcasm, taunting Trump’s lack of geography
knowledge and common sense. Headline 12271 uses pun based on polysemy:
“turkey” can either mean a country (when capitalized) or a bird. Also, humor
can require an understanding of cultural commentary, exemplified by Headline
9406. Since the Cheesecake Factory is a large chain of restaurants that some
may look down upon, they are happy to see it blown up with “no complaints”.

ID |  Original Headline |  Edit |  Human | Prediction
---|---|---|---|---
|  |  |  |  Original |  Context |  Edit
9100 |  WSJ: Trump’s top national security adviser is being investigated for his communications with Russia |  children |  0.0 |  1.35 |  1.50 |  0.94
9406 |  Man Sets Off Explosive Device at L.A.-Area Cheesecake Factory, No Injuries |  complaints |  2.4 |  0.76 |  0.52 |  0.72
12271 |  Turkey tells citizens to reconsider travelling to US |  poultry |  2.4 |  0.85 |  0.83 |  0.65
12685 |  CBS Poll: Americans lack confidence in Trump’s ability to handle North Korea |  locate |  2.4 |  0.97 |  0.94 |  0.96
Table 5: Error analysis on the test set of Subtask 1. The examples show
disagreements between human ratings and model predictions.

##  7 Conclusions

We use incongruity as the key to assessing funniness in edited news headlines.
Specifically, we use pretrained transformer-based language models to encode
contrastive pairs. Our best performing model is RoBERTa, which is submitted
for the official evaluation and achieves competitive performance in both
subtasks. The additional experiment shows that a non-contrastive approach may
also encode incongruity implicitly. While incongruity is a common ingredient
of humor, error analysis indicates it is neither sufficient nor necessary.
This invites future research to take other factors (e.g., sarcasm, pun, or
world knowledge) into account to better tackle humor, an intricate phenomenon
rooted in human creativity.

## Acknowledgements

The authors would like to thank Karl Stratos for his insightful feedback.

## References

  * [Bromley et al., 1994]  Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah.  1994\.  Signature verification using a “Siamese” time delay neural network.  In Advances in Neural Information Processing Systems, pages 737–744.
  * [Castro et al., 2016]  Santiago Castro, Matías Cubero, Diego Garat, and Guillermo Moncecchi.  2016\.  Is this a joke? Detecting humor in Spanish tweets.  In Ibero-American Conference on Artificial Intelligence (IBERAMIA 2016), pages 139–150. Springer.
  * [Conneau and Kiela, 2018]  Alexis Conneau and Douwe Kiela.  2018\.  SentEval: An evaluation toolkit for universal sentence representations.  In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018).
  * [Devlin et al., 2019]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2019\.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
  * [He et al., 2019]  He He, Nanyun Peng, and Percy Liang.  2019\.  Pun generation with surprise.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1734–1744.
  * [Hossain et al., 2019]  Nabil Hossain, John Krumm, and Michael Gamon.  2019\.  “president vows to cut <taxes> hair”: Dataset and analysis of creative text editing for humorous headlines.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 133–142.
  * [Hossain et al., 2020a]  Nabil Hossain, John Krumm, Michael Gamon, and Henry Kautz.  2020a.  Semeval-2020 Task 7: Assessing humor in edited news headlines.  In Proceedings of International Workshop on Semantic Evaluation (SemEval-2020).
  * [Hossain et al., 2020b]  Nabil Hossain, John Krumm, Tanvir Sajed, and Henry Kautz.  2020b.  Stimulating creativity with FunLines: A case study of humor generation in headlines.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 256–262.
  * [Kao et al., 2016]  Justine T. Kao, Roger Levy, and Noah D. Goodman.  2016\.  A computational model of linguistic humor in puns.  Cognitive Science, 40(5):1270–1285.
  * [Kingma and Ba, 2015]  Diederik P. Kingma and Jimmy Ba.  2015\.  Adam: A method for stochastic optimization.  In Proceedings of International Conference on Learning Representations (ICLR).
  * [Liu et al., 2019]  Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  2019\.  RoBERTa: A robustly optimized BERT pretraining approach.  arXiv preprint arXiv:1907.11692.
  * [Mao and Liu, 2019]  Jihang Mao and Wanli Liu.  2019\.  A BERT-based approach for automatic humor detection and scoring.  In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019).
  * [Morreall, 2016]  John Morreall.  2016\.  Philosophy of humor.  In The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, winter 2016 edition.
  * [Paszke et al., 2019]  Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  2019\.  PyTorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems, pages 8026–8037.
  * [Pennington et al., 2014]  Jeffrey Pennington, Richard Socher, and Christopher Manning.  2014\.  GloVe: Global vectors for word representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
  * [Peters et al., 2018]  Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  2018\.  Deep contextualized word representations.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237.
  * [Sennrich et al., 2016]  Rico Sennrich, Barry Haddow, and Alexandra Birch.  2016\.  Neural machine translation of rare words with subword units.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725.
  * [Tenney et al., 2019]  Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick.  2019\.  What do you learn from context? Probing for sentence structure in contextualized word representations.  In Proceedings of International Conference on Learning Representations (ICLR).
  * [Vaswani et al., 2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In Advances in Neural Information Processing Systems, pages 5998–6008.
  * [Veale, 2004]  Tony Veale.  2004\.  Incongruity in humor: Root cause or epiphenomenon?  Humor, 17(4):419–428.
  * [Weller and Seppi, 2019]  Orion Weller and Kevin Seppi.  2019\.  Humor detection: A transformer gets the last laugh.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3621–3625.
  * [Wolf et al., 2019]  Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew.  2019\.  HuggingFace’s Transformers: State-of-the-art natural language processing.  arXiv preprint arXiv:1910.03771.
  * [Wu et al., 2016]  Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  arXiv preprint arXiv:1609.08144.
  * [Yan and Pedersen, 2017]  Xinru Yan and Ted Pedersen.  2017\.  Duluth at SemEval-2017 task 6: Language models in humor detection.  In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 385–389.

##  Appendix A Task Summary

Task | Type | Metric | Train | Train Extra | Dev | Test
---|---|---|---|---|---|---
Subtask 1 | Regression | RMSE | 9653 | 8248 | 2420 | 3025
Subtask 2 | Classification | Accuracy | 9382 | 1959 | 2356 | 2961
Table 6: Summary of subtasks. Train, Dev, Test are from the Humicroedit
dataset. Train Extra is from the FunLines dataset.

##  Appendix B Experiment Details

#### Preprocessing

We use spaCy word tokenizer for CBOW. The pretrained transformers use byte-
pair encoding [Sennrich et al., 2016, BPE] to convert text into subword units.
BERT uses WordPiece [Wu et al., 2016] tokenization, a character-level BPE,
with a vocabulary size of 30K. RoBERTa preserves cases and uses a byte-level
BPE with a vocabulary size of 50K.

#### Training

For training, we use a batch size of 32 in Subtask 1 and 16 in Subtask 2. We
use Adam optimizer and perform gradient clipping with a max $\mathcal{L}_{2}$
norm of 5. For most experiments, we train for 10 epochs with a learning rate
in {1e-3, 3e-4}. However, when finetuning transformers, we choose max epochs
in {3, 10}, and use either a constant learning rate or a linear decreasing
schedule with an initial learning rate in {2e-5, 5e-5}. We perform validation
on the development set every 1/3 epoch and save the best checkpoint.