# Simple models predict behavior at least as well as behavioral scientists

Dillon Bowen Wharton School of Business<EMAIL_ADDRESS>
How accurately can behavioral scientists predict behavior? To answer this
question, we analyzed data from five studies in which 640 professional
behavioral scientists predicted the results of one or more behavioral science
experiments. We compared the behavioral scientists’ predictions to random
chance, linear models, and simple heuristics like “behavioral interventions
have no effect” and “all published psychology research is false.” We find that
behavioral scientists are consistently no better than - and often worse than -
these simple heuristics and models. Behavioral scientists’ predictions are not
only noisy but also biased. They systematically overestimate how well
behavioral science “works”: overestimating the effectiveness of behavioral
interventions, the impact of psychological phenomena like time discounting,
and the replicability of published psychology research 111Code available here:
https://gitlab.com/dsbowen/publication-bias

Keywords: Forecasting, Behavioral science

##  1 Introduction

Behavioral scientists’ predictions regularly inform academia, public policy,
and business decisions. Academic researchers decide which projects to pursue
based on which hypotheses seem most plausible. Nudge units test and implement
interventions they expect will be most effective. Businesses regularly consult
marketing and management experts for advice.

The underlying assumption in these instances is that behavioral scientists’
predictions are accurate. In general, we expect experts in any domain to be
able to make accurate predictions about their domain of expertise (Tetlock,
2009). We should have especially high expectations of scientific experts,
given that prediction is a fundamental function of science (Kuhn, 2011). We
might even argue that accurate predictions are necessary for the credibility
of a scientific discipline (Tetlock, 2009; Tetlock and Gardner, 2016).

These considerations raise a fundamental question: How accurately can
behavioral scientists predict behavior?

To answer this question, we examined five studies in which behavioral
scientists predicted the results of one or more behavioral science experiments
(Milkman et al., 2021, 2022; DellaVigna and Pope, 2018a; DellaVigna and Linos,
2022; Dreber et al., 2015). The goals of these studies included testing
methods for aggregating predictions (Dreber et al., 2015), comparing subgroups
of behavioral scientists (DellaVigna and Pope, 2018a; DellaVigna and Linos,
2022), and comparing behavioral scientists to non-scientists and scientists in
other domains (Milkman et al., 2021). However, they did not compare the
behavioral scientists’ predictions to simple benchmarks (e.g., random chance)
or heuristics (e.g., behavioral interventions have no effect). Such
comparisons are critical considering research demonstrating that simple models
often perform at least as well as experts in political science (Mellers et
al., 2014; Tetlock, 2009; Tetlock and Gardner, 2016), clinical diagnosis
(Dawes and Corrigan, 1974; Dawes et al., 1989; Garb, 1989), and other domains
(Dawes, 1979; Dawes and Corrigan, 1974).

Exercise study. The first experiment used 53 behavioral nudges to encourage
24-Hour Fitness customers to exercise more (Milkman et al., 2021). The
researchers measured the nudges’ effectiveness as the increase in average
weekly gym visits during a four-week intervention compared to a control
condition. They then asked 90 practitioners from behavioral science companies
to predict how effective each nudge would be.

We compared the behavioral scientists’ predictions to a null model that
predicted that none of the behavioral nudges would increase exercise.
Specifically, the null model predicted that all nudges would increase exercise
by zero gym visits per week compared to the control condition.

Flu study. The second experiment used 22 text-message treatments to encourage
Walmart customers to get a flu vaccine (Milkman et al., 2022). The treatments
varied the text messages’ phrasing and the number of messages sent. The
researchers measured a treatment’s effectiveness as the increase in
vaccination rates (number of people per hundred) compared to a control
condition in which they did not text customers. They then asked 24 professors
and graduate students, most affiliated with top-10 business schools, to
predict how effective each treatment would be.

As in the exercise study, we compared the behavioral scientists to a null
model that predicted that none of the text-message treatments would increase
vaccination rates. Specifically, the null model predicted that all text-
message treatments would increase vaccination rates by zero people per hundred
compared to the control condition.

RCT study. The third study assembled data from 126 randomized control trials
(RCTs) from two of the largest Nudge Units in the United States (DellaVigna
and Linos, 2022). The researchers measured the RCTs’ effectiveness as the
percentage point increase in adopting a target behavior compared to a control
condition. They then asked 237 behavioral scientists from academia, non-
profits, government agencies, and nudge units to estimate the effectiveness of
14 randomly selected RCTs.

We compared the behavioral scientists’ predictions to a null model that
predicted that none of the nudges would increase adoption. Specifically, the
null model predicted that all nudges would increase the adoption of the target
behavior by zero percentage points compared to the control condition.

Effort study. The fourth experiment measured how much effort participants
exerted in a key-pressing task in 18 experimental conditions (DellaVigna and
Pope, 2018b). Participants scored points for alternating between pressing “a”
and “b” for 10 minutes (participants earned one point each time they pressed
“a” then “b”). The experimental conditions were monetary and non-monetary
incentives to score points, such as piece-rate payments, time-delayed
payments, and peer comparisons. The researchers measured the effort
participants exerted in each condition as the number of points they scored.
They then recruited 213 academic economists to predict their results
(DellaVigna and Pope, 2018a). The researchers showed the economists the
results of three experimental conditions: how much effort participants exerted
when the researchers did not pay them, when they paid them 1 cent per 100
points, and when they paid them 10 cents per 100 points. The economists then
predicted the results of the other 15 conditions.

We compared the economists’ predictions to a simple linear interpolation
between the three conditions the economists saw (see Figure 1). This model
makes the simplifying assumption that participants only care about their
expected piece-rate payment. Importantly, it assumes that participants do not
exhibit motivational crowding out, time discounting, risk aversion, framing
effects, or other psychological tendencies commonly studied by behavioral
economists.

This linear interpolation model can be “selfish” or “altruistic.” In two
experimental conditions, participants earned a piece-rate payment on behalf of
the Red Cross (e.g., the researchers donated 1 cent to the Red Cross for every
100 points the participant scored). The selfish version of the linear
interpolation model assumes that participants do not care about the Red Cross.
The altruistic version assumes that participants are just as motivated to earn
money for the Red Cross as they are to earn money for themselves.

Figure 1: Linear interpolation model for the effort study. The three orange
x’s are the three conditions for which the economists saw the results (no
payment, 1-cent piece-rate, and 10-cent piece-rate).

Reproducibility study. The fifth study attempted to reproduce 100 psychology
studies published in top academic journals (Collaboration, 2015). The
researchers defined a replication as successful if it obtained a p-value of
less than .05 and the estimated effect’s direction matched the original
experiment. They then asked 76 psychology professors and graduate students to
predict how likely 44 of the studies were to replicate successfully (Dreber et
al., 2015). We compared the psychologists’ predictions to three benchmarks:

  1. 1.

Null model. All published psychology research is false (i.e., the null
hypothesis is always true). Therefore, the probability of a successful
replication is 2.5% (5% chance of $p<.05$, half the time the direction of the
estimated effects in the original and replication experiments will match).

  2. 2.

Random chance. There is a 50% chance each study will replicate.

  3. 3.

Linear regression model. This model predicts the estimated effect in the
replication study using the estimated effect in the original study. It then
uses this prediction to estimate a study’s probability of replicating.

###  1.1 Dependent measures

We scored predictions using squared error for the exercise, flu, RCT, and
effort studies. Because the reproducibility study used a binary outcome
variable (an indicator that the study successfully replicated), we scored
predictions using Brier scores. We define risk as the expected score. Our
primary dependent measure was comparative risk, the expected difference
between the behavioral scientists’ and models’ scores. We also analyze bias,
the expected difference between the behavioral scientists’ predictions and the
treatment effects (or, for the reproducibility study, the probability of
replicating).

To measure bias in the effort study, we first grouped the experimental
conditions into eight categories described in the effort study paper. The
categories group conditions that measure the same psychological phenomenon
(e.g., social preferences). Then, we reverse coded some of the categories. For
example, economic theory suggests that participants should exert less effort
when payments are time-delayed. Therefore, a behavioral scientist
overestimates the effect of time discounting when she underestimates the
number of points scored in the time-delayed category. Finally, we measured
bias as the expected difference between the behavioral scientists’ predictions
and the number of points scored in each category222The exception to this is
the framing effects category. We measure framing effects as the difference
between the loss-framed and gain-framed conditions. Behavioral scientists are
biased if they overestimate the difference between these conditions. See
Section 4 for a formal description of how we computed bias for the framing
effects category..

Throughout our analysis, we compare the behavioral scientists and models to an
“oracle” to estimate the irreducible uncertainty. The oracle makes the best
possible prediction given the experiment results. The oracle’s risk is
positive because the oracle only knows the experimental results, not the true
effects. For example, because the oracle does not know the true effects of the
exercise study, its expected squared error will be positive.

##  2 Results

Table 1: Risk for behavioral scientists and models. Study | Predictor | Mean | 95% CI
---|---|---|---
Exercise | Scientist | 6.02 | (5.19, 6.93)
| Null | 0.03 | (0.01, 0.05)
| Oracle | 0.00 | (0.00, 0.01)
Flu | Scientist | 64.24 | (14.80, 184.52)
| Null | 4.80 | (2.76, 7.28)
| Oracle | 0.11 | (0.03, 0.38)
RCT | Scientist | 54.59 | (27.97, 101.94)
| Null | 7.48 | (2.24, 17.04)
| Oracle | 1.13 | (0.18, 4.70)
Effort | Scientist | 49884 | (38858, 64352)
| Selfish | 49865 | (26356, 80066)
| Altruistic | 34639 | (18785, 55836)
| Oracle | 727 | (232, 1636)
Reproducibility | Scientist | 0.31 | (0.28, 0.35)
| Null | 0.31 | (0.20, 0.43)
| Random | 0.25 | (0.25, 0.25)
| Linear regression | 0.23 | (0.18, 0.28)
| Oracle | 0.16 | (0.12, 0.21)
Table 2: Bias. Study | Units | Mean | 95% CI
---|---|---|---
Exercise | Weekly gym visits | 2.32 | (2.15, 2.50)
Flu | People per 100 | 2.41 | (0.33, 5.90)
RCT | Percentage points | 3.37 | (1.75, 5.31)
Effort | Points | 66.90 | (23.35, 126.63)
Reproducibility | Pr. replication | 0.22 | (0.08, 0.33)

Exercise study. The behavioral scientists’ exercise study predictions were
significantly worse than the null model (see Table 1). That is, the
comparative risk was significantly positive ($M=6.00,P<.001$). Behavioral
scientists estimated that the average behavioral nudge would increase exercise
by 2.5 gym visits every week. The study results suggest that the average nudge
increased exercise by only one gym visit every six weeks. This bias was
statistically significant ($P<.001$, see Table 2). Even after applying a
correction for multiple hypothesis testing (Romano and Wolf, 2005; Kitagawa
and Tetenov, 2018; Mogstad et al., 2020), behavioral scientists significantly
overestimated the effectiveness of every treatment in the study.

Flu study. The behavioral scientists’ flu study predictions were also
significantly worse than the null model ($M=59.44,P<.001$). Behavioral
scientists estimated that the average text-message treatment would increase
vaccination rates by 4.5 people per hundred. The study results suggest that
the average treatment increased vaccination rates by only 2.1 people per
hundred. This bias was statistically significant ($P=.015$).

RCT study. The null model also outperformed behavioral scientists in the RCT
study ($M=47.11,P<.001$). Behavioral scientists estimated that the average
nudge would increase adoption of the target behavior by 4.98 percentage
points. The study results indicate that the average nudge increased adoption
by only 1.61 percentage points. This bias was statistically significant
($P=.001$).

The RCT prediction study dataset also included information about the
behavioral scientists’ experience with RCTs (i.e., the number of nudge RCTs
they had run). The authors grouped the behavioral scientists into three
categories: those with no experience running RCTs (novices), those who had run
1-5 RCTs (moderately experienced), and those who had run more than five RCTs
(most experienced). They observed that more experienced behavioral scientists
were less biased based on summary statistics.

We formally analyzed the relationship between experience and bias in the RCT
study (see Figure 2). Consistent with the RCT study’s conclusion, we estimate
that the most experienced behavioral scientists are 0.65 percentage points
less biased than the moderately experienced behavioral scientists and 1.44
percentage points less biased than the novices. However, pairwise hypothesis
tests suggest that none of the experience groups significantly differ from any
other. Additionally, even the most experienced behavioral scientists
overestimate nudges’ effectiveness by 2.20 percentage points ($P=.005$ after
correcting for multiple hypothesis testing).

Figure 2: Bias for the novice, moderaly experienced, and most experienced
behavioral scienitists. Error bars show simultaneous 95% confidence intervals.

Effort study. The economists’ effort study predictions were worse than the
selfish and altruistic linear interpolation models. However, the differences
are not statistically significant ($M=19.38,P=1.000$ for the selfish model,
$M=15245.30,P=.108$ for the altruistic model, p-values adjusted for multiple
hypothesis testing).

Economists overestimated the impact of psychological phenomena by 67 points on
average ($P=.003$). For perspective, consider that participants scored 2,029
points on average in the one-cent piece-rate condition (i.e., when researchers
paid them one cent per 100 points within a day of taking the study). When
researchers delayed the payment by two weeks (i.e., researchers paid them one
cent per 100 points two weeks after taking the study), participants scored
2,004 points. Therefore, the two-week delay decreased effort by 2,029-2,004=25
points. Economists predicted that the two-week delay would decrease effort by
71 points. Although the economists overestimated the impact of every
psychological phenomenon, only three were statistically significant after
correcting for multiple hypothesis testing: motivational crowding out, time
discounting, and social preferences (see Figure 3).

Figure 3: Bias for the effort study. Error bars show simultaneous 95%
confidence intervals.

Reproducibility study. Figure 4 shows that the psychologists’ reproducibility
study predictions were worse than all three models we considered: the null
model, random chance, and linear regression. Even after correcting for
multiple hypothesis testing, this difference was statistically significant for
random chance ($M=.06,P<.001$) and linear regression ($M=.09,P=.014$).
Psychologists estimated that the average published psychology study had a 54%
chance of replicating. The reproducibility study results suggest that the
average study had only a 32% chance of replicating. This bias was
statistically significant ($P=.002$).

Figure 4: Risk for the reproducibility study.

##  3 Discussion

Critical public policy decisions depend on predictions from behavioral
scientists. In this paper, we asked how accurate those predictions are. To
answer this question, we compared the predictions of 640 behavioral scientists
to those of simple mathematical models on five prediction tasks. Our sample
included a variety of behavioral scientists: economists, psychologists, and
business professionals from academia, industry, and government. The prediction
tasks also covered various domains, including text-message interventions to
increase vaccination rates, behavioral nudges to increase exercise, randomized
control trials, incentives to encourage effort, and attempts to reproduce
published psychology studies. The models to which we compared the behavioral
scientists were deliberately simple, such as random chance, linear
interpolation, and heuristics like “behavioral interventions have no effect”
and “all published psychology research is false.”

We consistently found that behavioral scientists are no better than - and
often worse than - these simple heuristics and models. In the exercise, flu,
and RCT studies, null models significantly outperformed behavioral scientists.
These null models assume that behavioral treatments have no effect; behavioral
interventions will not increase weekly gym visits, text messages will not
increase vaccination rates, and nudges will not change behavior. As we can see
in Table 1, compared to behavioral scientists, null models are nearly
indistinguishable from the oracle.

In the effort study, linear interpolations performed at least as well as
professional economists. These interpolations assumed that all psychological
phenomena are inert; people do not exhibit risk aversion, time discounting, or
biases like framing effects.

In the reproducibility study, professional psychologists’ Brier scores were
virtually identical to those of a null model, which assumed that all published
psychology research is false. Professional psychologists were significantly
worse than both linear regression and random chance.

Notably, the linear regression model used data from the reproducibility study,
which were not accessible to psychologists during their participation. While
this is not a fair comparison, we believe it is a useful comparison, as the
linear regression model can serve as a benchmark for future attempts to
predict reproducibility.

Why is it so hard for behavioral scientists to outperform simple models? One
possible answer is that human predictions are noisy while model predictions
are not (Kahneman et al., 2021). Indeed, there is likely a selection bias in
the prediction tasks we analyzed. Recall that most of the prediction tasks
asked behavioral scientists to predict the results of ongoing or recently
completed studies. Behavioral scientists presumably spend time researching
questions that have not been studied exhaustively and do not have obvious
answers. In this case, the prediction tasks were likely exceptionally
challenging, and behavioral scientists’ expertise would be of little use.

However, behavioral scientists’ predictions are not only noisy but also
biased. Previous research noted that behavioral scientists overestimate the
effectiveness of nudges (DellaVigna and Linos, 2022; Milkman et al., 2021).
Our research extends these findings, suggesting that behavioral scientists
believe behavioral science generally “works” better than it does. Behavioral
scientists overestimated the effectiveness of behavioral interventions in the
exercise, flu, and RCT studies. In the exercise study, behavioral scientists
significantly overestimated the effectiveness of all 53 treatments, even after
correcting for multiple testing. Economists overestimated the impact of
psychological phenomena in the effort study, especially for motivational
crowding out, time discounting, and social preferences. Finally, psychologists
significantly overestimated the replicability of published psychology research
in the reproducibility study. In general, behavioral scientists overestimate
not only the effect of nudges, but also the impact of psychological phenomena
and the replicability of published behavioral science research.

Behavioral scientists’ bias can have serious consequences. A recent study
found that policymakers were less supportive of an effective climate change
policy (carbon taxes) when a nudge solution was also available (Hagmann et
al., 2019). However, accurately disclosing the nudge’s impact shifted support
back towards carbon taxes and away from the nudge solution. In general, when
behavioral scientists exaggerate the effectiveness of their work, they may
drain support and resources from potentially more impactful solutions.

Our results raise many additional questions. For example, is it only
behavioral scientists who are biased, or do people, in general, overestimate
how well behavioral science works? The general public likely has little
exposure to RCTs, social science experiments, and academic psychology
publications, so there is no reason to expect that they are biased in either
direction. Then again, the little exposure they have had likely gives an
inflated impression of behavioral science’s effectiveness. For example, a TED
talk with 64 million as of May 2022 touted the benefits of power posing,
whereby one can reap the benefits of improved self-confidence and become more
likely to succeed in life by adopting a powerful pose for one minute (Carney
et al., 2010; Cuddy, 2012). However, the power posing literature was based on
p-hacked results (Simmons and Simonsohn, 2017), and researchers have since
found that power posing yields no tangible benefits (Jonas et al., 2017).

Additionally, people may generally overestimate effects due to the “What you
see is all there is” (WYSIATI) bias (Kahneman, 2011). For example, the
exercise study asked behavioral scientists to consider, among other
treatments, how much more people would exercise if researchers told them they
were “gritty.” After the initial “gritty diagnosis,” dozens of other factors
determined how often participants in that condition went to the gym during the
following four-week intervention period. Work schedule, personal
circumstances, diet, mood changes, weather, and many other factors also played
key roles. These other factors may not have even crossed the behavioral
scientists’ minds. The WYSIATI bias may have caused them to focus on the
treatment and ignore the noise of life that tempers the treatment’s signal. Of
course, this bias is likely to cause everyone, not only behavioral scientists,
to overestimate the effectiveness of behavioral interventions and the impact
of psychological phenomena.

If people generally overestimate how well behavioral science works, are they
more or less biased than behavioral scientists? Experimental economics might
suggest that behavioral scientists are less biased because people with
experience tend to be less biased in their domain of expertise. For example,
experienced sports card traders are less susceptible to the endowment effect
(List, 2004), professional traders exhibit less ambiguity aversion than
novices (List and Haigh, 2010), experienced bidders are immune to the winner’s
curse (Harrison and List, 2008), and CEOs who regularly make high-stakes
decisions are less susceptible to possibility and certainty effects (List and
Mason, 2011). Given that most people have zero experience with behavioral
science, they should be more biased than behavioral scientists.

Then again, there are at least three reasons to believe that behavioral
scientists should be more biased than the general population: selection bias,
selective exposure, and motivated reasoning. First, behavioral science might
select people who believe in its effectiveness. On the supply side, students
who apply to study psychology for five years on a measly PhD stipend are
unlikely to believe that most psychology publications fail to replicate. On
the demand side, marketing departments and nudge units may be disinclined to
hire applicants who believe their work is ineffective. Indeed, part of the
experimental economics argument is that markets filter out people who make
poor decisions (List and Millimet, 2008). The opposite may be true of
behavioral science: the profession might filter out people with an accurate
assessment of how well behavioral science works.

Second, behavioral scientists are selectively exposed to research that finds
large and statistically significant effects. Behavioral science journals and
conferences are more likely to accept papers with significant results.
Therefore, most of the literature behavioral scientists read promotes the idea
that behavioral interventions are effective and psychological phenomena
substantially influence behavior. However, published behavioral science
research often fails to replicate. Lack of reproducibility plagues not only
behavioral science (Collaboration, 2012, 2015; Camerer et al., 2016; Mac
Giolla et al., 2022) but also medicine (Freedman et al., 2015; Prinz et al.,
2011), neuroscience (Button et al., 2013), and genetics (Hewitt, 2012;
Lawrence et al., 2013). Scientific results fail to reproduce for many reasons,
including publication bias, p-hacking, and fraud (Simmons et al., 2011; Nelson
et al., 2018). Indeed, most evidence that behavioral scientists overestimate
how well behavioral science works involves asking them to predict the results
of nudge studies. However, there is little to no evidence that nudges work
after correcting for publication bias (Maier et al., 2022). Even when a study
successfully replicates, the effect size in the replication study is often
much smaller than that reported in the original publication (Camerer et al.,
2016; Collaboration, 2015). For example, the RCT study paper estimates that
the academic literature overstates nudges’ effectiveness by a factor of six
(DellaVigna and Linos, 2022).

Finally, behavioral scientists might be susceptible to motivated reasoning
(Kunda, 1990; Epley and Gilovich, 2016). As behavioral scientists, we want to
believe that our work is meaningful, effective, and true. Motivated reasoning
may also drive selective exposure (Bénabou and Tirole, 2002). We want to
believe our work is effective, so we disproportionately read about behavioral
science experiments that worked.

Our analysis finds mixed evidence of the relationship between experience and
bias in behavioral science. The RCT study informally examined the relationship
between experience and bias for behavioral scientists predicting nudge effects
and concluded that more experienced scientists were less biased. While we also
estimate that more experienced scientists are less biased, we do not find
statistically significant pairwise differences between the novice, moderately
experienced, and most experienced scientists.

Even if the experimental economics argument is correct that behavioral
scientists are less biased than the general population, why are behavioral
scientists biased at all? The experimental economics literature identifies two
mechanisms to explain why more experienced people are less biased (List, 2003;
List and Millimet, 2008). First, markets filter out people who make poor
decisions. Second, experience teaches people to think and act more rationally.
We have already discussed that the first mechanism might not apply to
behavioral science. And, while our results are consistent with the hypothesis
that behavioral scientists learn from experience, they still suggest that even
the most experienced behavioral scientists overestimate the effectiveness of
nudges. The remaining bias for the most experienced scientists is larger than
the gap between the most experienced scientists and novices. Why has
experience not eliminated this bias entirely? Perhaps the effect of experience
competes with the forces of “What you see is all there is,” selection bias,
selective exposure, and motivated reasoning such that experience mitigates but
does not eliminate bias in behavioral science.

Finally, how can behavioral scientists better forecast behavior? One promising
avenue is to use techniques that help forecasters predict political events
(Chang et al., 2016; Mellers et al., 2014). For example, the best political
forecasters begin with base rates and then adjust their predictions based on
information specific to the event they are forecasting (Tetlock and Gardner,
2016). Behavioral scientists’ predictions would likely improve by starting
with the default assumptions that behavioral interventions have no effect,
psychological phenomena do not influence behavior, and published psychology
research has a one in three chance of replicating (Collaboration, 2012). Even
though these assumptions are wrong, they are much less wrong than what
behavioral scientists currently believe.

##  4 Methods

###  4.1 Setup

All of the studies we considered have a similar data generating process.
First, the researchers selected a set of $K$ treatments to test. Denote the
true effects of these treatments as $\mu\coloneqq(\mu_{1},...,\mu_{K})^{T}$.

Then, the researchers asked $F$ behavioral scientists to predict how effective
the selected treatments would be. Let $X$ be the $K\times F$ matrix of the
behavioral scientists’ predictions, where $X_{k,f}$ is behavioral scientist
$f$’s prediction about the effect of treatment $k$. We also predict the
treatment effects using a model and call these predictions
$\mu^{m}\coloneqq(\mu^{m}_{1},...,\mu^{m}_{K})^{T}$.

Finally, the researchers ran the experiment, which gave them noisy estimates
$Y\coloneqq(Y_{1},...,Y_{K})^{T}$ of the treatment effects. By the central
limit theorem, these estimates are approximately normally distributed
$Y\sim\mathcal{N}(\mu,\Sigma)$, where $\Sigma$ is the $K\times K$ covariance
with which the treatment effects were estimated.

This setup above applies to the exercise, flu, RCT, and effort studies. Much
of the same setup applies to the replication study as well. Instead of $K$
treatments to test, the researchers selected $K$ studies to replicate.
$\mu_{k}$ is the true probability that study $k$ will replicate. The
probability that study $k$ will replicate depends on the sample size of the
replication study $n_{k}$ and the effect of the study’s treatment
$\mu^{*}_{k}$. Specifically,

$\mu_{k}=\Phi(\sqrt{n_{k}}\mu^{*}_{k}-c_{\alpha})$

where $c_{\alpha}$ is the critical value for significance level $\alpha$
(around 1.96 for a two-tailed test with $\alpha=.05$). Note that we normalize
$\mu^{*}_{k}$ by study $k$’s sample standard deviation for comparability
across studies.

When the researchers ran the replication studies, they observed noisy
estimates $Y^{*}$ of the treatment effects. By the central limit theorem,
these estimates are approximately normally distributed
$Y^{*}\sim\mathcal{N}(\mu^{*},\Sigma^{*})$, where $\Sigma^{*}$ is a $K\times
K$ diagonal covariance matrix where the $k$th diagonal element is $1/n_{k}$.

We can back out $Y^{*}_{k}$ from the replication study data set using the
sample size $n_{k}$, p-value $p_{k}$, and the direction of the estimated
effect.

$\sqrt{n_{k}}Y^{*}_{k}=\begin{cases}\Phi^{-1}(1-p_{k}/2)&\text{if
$Y^{*}_{k}>0$}\\\ \Phi^{-1}(p_{k}/2)&\text{else}\end{cases}$

###  4.2 Bias

Bias is how much behavioral scientists overestimate the effects of treatments
in general,

$B(\mu,X)\coloneqq\operatorname{\mathbb{E}}_{\mu}\big{[}\operatorname{\mathbb{E}}[X_{k,f}-\mu_{k}]\big{]}.$

If the bias is positive, the behavioral scientists overestimated the treatment
effects.

Similarly, the bias for a given treatment $k$ is

$B_{k}(\mu,X)\coloneqq\operatorname{\mathbb{E}}_{\mu}\big{[}\operatorname{\mathbb{E}}[X_{k,f}-\mu_{k}|k]\big{]}.$

This is how we measured the bias for seven of the eight categories of
psychological phenomena tested in the effort study.

The bias for the difference between two treatments $k$ and $l$ is

$B_{k,l}(\mu,X)\coloneqq\operatorname{\mathbb{E}}_{\mu}\big{[}\operatorname{\mathbb{E}}[(X_{k,f}-X_{l,f})-(\mu_{k}-\mu_{l})|k,l]\big{]}.$

If the bias is positive, the behavioral scientists overestimated the
difference between the effectiveness of treatments $k$ and $l$. This is how we
measured bias for the framing effects category in the effort study.
Specifically, $k$ was a loss-framed condition (the researchers gave
participants $0.40, which they would lose if they scored fewer than 2,000
points) and $l$ was a gain-framed condition (the researchers promised
participants $0.40 if they scored at least 2,000 points).

###  4.3 Risk

Loss functions measure how accurately forecasters and models predicted the
results of the experiment. Let $l:\mathbb{R}\times\mathbb{R}\to\mathbb{R}$ be
a generic loss function that takes the true and predicted treatment effect and
returns a real value (the loss). For example, $l$ could be squared error. So,
the squared error for behavioral scientist $f$’s prediction about the effect
of treatment $k$ is $l(\mu_{k},X_{f,k})=(\mu_{k}-X_{f,k})^{2}$.

Behavioral scientists’ corresponding risk function is the expected loss

$R(\mu,X)\coloneqq\operatorname{\mathbb{E}}_{\mu}\big{[}\operatorname{\mathbb{E}}[l(\mu_{k},X_{k,f})]\big{]}$

If the risk is low, the behavioral scientists’ predictions were accurate. If
the risk is high, the predictions were inaccurate.

Our main outcome variable is the comparative risk; the expected difference
between the behavioral scientists’ loss and a model’s loss

$CR(\mu,X,\mu^{m})\coloneqq\operatorname{\mathbb{E}}_{\mu}\big{[}\operatorname{\mathbb{E}}[l(\mu_{k},X_{k,f})-l(\mu_{k},\mu^{m}_{k})]\big{]}.$

If the comparative risk is negative, the behavioral scientists’ predictions
were better than those of the model. If the comparative risk is positive, the
model’s predictions were better than those of the behavioral scientists.

###  4.4 Estimating bias and risk

We now turn to our procedure for estimating bias and risk. It is tempting to
simply replace the true effects $\mu$ with their noisy estimates $Y$ and then
use standard statistical methods. For example, it is tempting to test whether
behavioral scientists are biased for treatment $k$ by taking
$X_{k,1}-Y_{k},...,X_{k,F}-Y_{k}$ as our sample and testing whether the mean
is different from 0 using a $t$-test.

However, this “plug-in procedure” ignores the variability in $Y_{k}$ as an
estimate of $\mu_{k}$. This could lead us to conclude that behavioral
scientists are biased when in fact they are not. To see this, suppose that
behavioral scientists are unbiased, $\operatorname{\mathbb{E}}[X_{f}]=\mu$. By
chance, $Y_{k}$ underestimates $\mu_{k}$. Then, as the number of behavioral
scientists in our sample increases, the sample mean of
$X_{1.k}-Y_{k},...,X_{k,F}-Y_{k}$ will converge to $\mu_{k}-Y_{k}>0$, making
it appear as if behavioral scientists are biased. A similar problem applies to
plugging in $Y_{k}$ for $\mu_{k}$ when estimating risk.

In sum, there are three sources of uncertainty:

  1. 1.

Researchers sampled treatments from a population of treatments they could have
tested. For example, in the exercise study, this population is “the sort of
behavioral interventions researchers would use to try to increase exercise in
an study like this.”

  2. 2.

Researchers sampled behavioral scientists from a population of scientists they
could have asked to predict the experimental results.

  3. 3.

Researchers obtained a noisy estimate of the true treatment effects.

To account for our uncertainty about the true treatment effects, we use
empirical Bayes estimators from the `multiple-inference` statistics package
(Bowen, 2022) to estimate the posterior distribution of $\mu|Y$. When
possible, we used nonparametric empirical Bayes estimators to avoid making
parametric assumptions about the prior distribution of treatment effects.
Nonparametric empirical Bayes estimators perform well with many data points
and usually assume that the treatment effects are estimated independently (Cai
et al., 2021; Brown and Greenshtein, 2009). Therefore, we used nonparametric
empirical Bayes for the RCT and reproducibility studies. We used parametric
(normal-prior, normal-likelihood) empirical Bayes for the exercise and flu
studies because the treatment effects were estimated with correlated errors.
We also used parametric empirical Bayes for the effort study because it
included only 18 treatments.

To account for uncertainty about the populations of treatments and behavioral
scientists, we use a Bayesian bootstrap with a uniform Dirichlet prior and
sample weights at the treatment level and the at behavioral scientist level
(Rubin, 1981). Sampling weights at the treatment level accounts for the fact
that some treatment effects are harder to predict than others. Sampling
weights at the behavioral scientist level accounts for the fact that some
behavioral scientists are better forecasters than others. This is the
bootstrap equivalent of clustering standard errors by treatment and behavioral
scientist.

For concreteness, Algorithm 1 describes how to use our sampling procedure to
estimate the probability that behavioral scientists’ predictions are more
accurate than those of a mathematical model. That is, we estimate the
probability that the comparative risk is negative. We use similar sampling
procedures to estimate bias.

Algorithm 1 Estimating comparative risk

$K\times 1$ vector of estimated effects $Y$ and $K\times K$ covariance matrix
$\Sigma$

$K\times 1$ vector of model predictions $\mu^{m}$

$K\times F$ matrix of behavioral scientist predictions $X$

Number of bootstrap samples to draw $N_{S}$

$N_{T}\leftarrow 0$

$P\leftarrow$ posterior distribution of $\mu|Y$ using empirical Bayes

for $s=\\{1,...,N_{S}\\}$ do

Sample $\mu^{s}$ from $P$

Sample $w$ from $\text{Dir}(\mathbf{1}_{K})$ where $\mathbf{1}_{K}$ is a
$K\times 1$ vector of 1’s

Sample $m$ from $\text{Dir}(\mathbf{1}_{F})$

Compute $L$ such that
$L_{k,f}=l(\mu^{s}_{k},X_{k,f})-l(\mu^{s}_{k},\mu^{m}_{k})$

if $w^{T}Lm<0$ then

$N_{T}\leftarrow N_{T}+1$

end if

end for

$\hat{Pr}_{\mu}\big{\\{}CR(\mu,X,\mu^{m})<0\big{\\}}=N_{T}/N_{S}$

###  4.5 Linear regression model

Here, we describe how we use linear regression to predict a study’s
probability of replicating using data from the Reproducibility Project
(Collaboration, 2012). The linear regression model itself is standard. Our
feature matrix $Z$ is a $K\times 2$ matrix containing a constant regressor and
the estimated effect in the original study (normalized by the sample standard
deviation of the original study). We can compute point estimates simply by
regressing $Y^{*}$ on $Z$ and predicting the effect of replication study $k$
as $\hat{Y}^{*}_{k}=\hat{\beta}^{T}Z_{k}$, where $\hat{\beta}$ is a vector of
OLS coefficients. Then, we can estimate the replication probability as,

$\hat{Pr}\\{\text{replicate}_{k}\\}=\Phi(\sqrt{n_{k}}\hat{Y}^{*}_{k}-c_{\alpha})$

However, this procedure does not account for the uncertainty in
$\hat{Y}^{*}_{k}$ as an estimate of $Y^{*}_{k}$. Algorithm 2 describes a Gibbs
sampling procedure to estimate the probability that study $k$ will replicate
using a linear regression model. Additionally, this algorithm assumes that the
variance of the error term is a linear function of $Z$, $\sigma^{2}=Z\gamma$.

Algorithm 2 Estimating replication probability using linear regression

$K\times 2$ feature matrix with a constant regressor and the estimated effect
size from the original study

$K\times 1$ vector of estimated effects $Y^{*}$ from the replication study

Number of Gibbs samples to draw $N$

$P\leftarrow\emptyset$

for $s=\\{1,...,N\\}$ do

Regress $Y^{*}$ on $Z$ to obtain a joint distribution of OLS coefficients
$\hat{\beta}$

Sample $\beta^{S}$ from the joint distribution of $\hat{\beta}$

Regress $(Z\beta^{s}-Y^{*})^{2}$ on $Z$ to obtain a joint distribution of OLS
coefficients $\hat{\gamma}$

Sample $\gamma^{s}$ from the joint distribution of $\hat{\gamma}$

Sample $Y^{s}_{k}$ from
$\mathcal{N}({\beta^{s}}^{T}Z_{k},{\gamma^{s}}^{T}Z_{k})$

$P\leftarrow P\cup\big{\\{}\Phi(\sqrt{n_{k}}Y^{s}_{k}-c_{\alpha})\big{\\}}$

end for

$\hat{Pr}\\{\text{replicate}_{k}\\}=\frac{1}{N}\sum_{s}P^{s}$

## References

  * Tetlock [2009] Philip E Tetlock.  Expert political judgment.  In _Expert Political Judgment_. Princeton University Press, 2009\.
  * Kuhn [2011] Thomas S Kuhn.  9\. the essential tension: Tradition and innovation in scientific research.  In _The essential tension_ , pages 225–239. University of Chicago Press, 2011.
  * Tetlock and Gardner [2016] Philip E Tetlock and Dan Gardner.  _Superforecasting: The art and science of prediction_.  Random House, 2016.
  * Milkman et al. [2021] Katherine L Milkman, Dena Gromet, Hung Ho, Joseph S Kay, Timothy W Lee, Pepi Pandiloski, Yeji Park, Aneesh Rai, Max Bazerman, John Beshears, et al.  Megastudies improve the impact of applied behavioural science.  _Nature_ , 600(7889):478–483, 2021.
  * Milkman et al. [2022] Katherine L Milkman, Linnea Gandhi, Mitesh S Patel, Heather N Graci, Dena M Gromet, Hung Ho, Joseph S Kay, Timothy W Lee, Jake Rothschild, Jonathan E Bogard, et al.  A 680,000-person megastudy of nudges to encourage vaccination in pharmacies.  _Proceedings of the National Academy of Sciences_ , 119(6), 2022.
  * DellaVigna and Pope [2018a] Stefano DellaVigna and Devin Pope.  Predicting experimental results: who knows what?  _Journal of Political Economy_ , 126(6):2410–2456, 2018a.
  * DellaVigna and Linos [2022] Stefano DellaVigna and Elizabeth Linos.  Rcts to scale: Comprehensive evidence from two nudge units.  _Econometrica_ , 90(1):81–116, 2022.
  * Dreber et al. [2015] Anna Dreber, Thomas Pfeiffer, Johan Almenberg, Siri Isaksson, Brad Wilson, Yiling Chen, Brian A Nosek, and Magnus Johannesson.  Using prediction markets to estimate the reproducibility of scientific research.  _Proceedings of the National Academy of Sciences_ , 112(50):15343–15347, 2015.
  * Mellers et al. [2014] Barbara Mellers, Lyle Ungar, Jonathan Baron, Jaime Ramos, Burcu Gurcay, Katrina Fincher, Sydney E Scott, Don Moore, Pavel Atanasov, Samuel A Swift, et al.  Psychological strategies for winning a geopolitical forecasting tournament.  _Psychological science_ , 25(5):1106–1115, 2014\.
  * Dawes and Corrigan [1974] Robyn M Dawes and Bernard Corrigan.  Linear models in decision making.  _Psychological bulletin_ , 81(2):95, 1974.
  * Dawes et al. [1989] Robyn M Dawes, David Faust, and Paul E Meehl.  Clinical versus actuarial judgment.  _Science_ , 243(4899):1668–1674, 1989.
  * Garb [1989] Howard N Garb.  Clinical judgment, clinical training, and professional experience.  _Psychological bulletin_ , 105(3):387, 1989.
  * Dawes [1979] Robyn M Dawes.  The robust beauty of improper linear models in decision making.  _American psychologist_ , 34(7):571, 1979.
  * DellaVigna and Pope [2018b] Stefano DellaVigna and Devin Pope.  What motivates effort? evidence and expert forecasts.  _The Review of Economic Studies_ , 85(2):1029–1069, 2018b.
  * Collaboration [2015] Open Science Collaboration.  Estimating the reproducibility of psychological science.  _Science_ , 349(6251):aac4716, 2015.
  * Romano and Wolf [2005] Joseph P Romano and Michael Wolf.  Stepwise multiple testing as formalized data snooping.  _Econometrica_ , 73(4):1237–1282, 2005.
  * Kitagawa and Tetenov [2018] Toru Kitagawa and Aleksey Tetenov.  Who should be treated? empirical welfare maximization methods for treatment choice.  _Econometrica_ , 86(2):591–616, 2018.
  * Mogstad et al. [2020] Magne Mogstad, Joseph P Romano, Azeem Shaikh, and Daniel Wilhelm.  Inference for ranks with applications to mobility across neighborhoods and academic achievement across countries.  Technical report, National Bureau of Economic Research, 2020.
  * Kahneman et al. [2021] Daniel Kahneman, Olivier Sibony, and Cass R Sunstein.  _Noise: A flaw in human judgment_.  Little, Brown, 2021.
  * Hagmann et al. [2019] David Hagmann, Emily H Ho, and George Loewenstein.  Nudging out support for a carbon tax.  _Nature Climate Change_ , 9(6):484–489, 2019\.
  * Carney et al. [2010] Dana R Carney, Amy JC Cuddy, and Andy J Yap.  Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance.  _Psychological science_ , 21(10):1363–1368, 2010\.
  * Cuddy [2012] Amy Cuddy.  Your body language may shape who you are, 06 2012.  URL https://www.ted.com/talks/amy_cuddy_your_body_language_may_shape_who_you_are.
  * Simmons and Simonsohn [2017] Joseph P Simmons and Uri Simonsohn.  Power posing: P-curving the evidence.  _Psychological science_ , 2017.
  * Jonas et al. [2017] Kai J Jonas, Joseph Cesario, Madeliene Alger, April H Bailey, Dario Bombari, Dana Carney, John F Dovidio, Sean Duffy, Jenna A Harder, Dian van Huistee, et al.  Power poses–where do we stand?  _Comprehensive Results in Social Psychology_ , 2(1):139–141, 2017.
  * Kahneman [2011] Daniel Kahneman.  _Thinking, fast and slow_.  Macmillan, 2011.
  * List [2004] John A List.  Testing neoclassical competitive theory in multilateral decentralized markets.  _Journal of Political Economy_ , 112(5):1131–1156, 2004.
  * List and Haigh [2010] John A List and Michael S Haigh.  Investment under uncertainty: Testing the options model with professional traders.  _The Review of Economics and Statistics_ , 92(4):974–984, 2010.
  * Harrison and List [2008] Glenn W Harrison and John A List.  Naturally occurring markets and exogenous laboratory experiments: A case study of the winner’s curse.  _The Economic Journal_ , 118(528):822–843, 2008\.
  * List and Mason [2011] John A List and Charles F Mason.  Are ceos expected utility maximizers?  _Journal of Econometrics_ , 162(1):114–123, 2011\.
  * List and Millimet [2008] John A List and Daniel L Millimet.  The market: Catalyst for rationality and filter of irrationality.  _The BE Journal of Economic Analysis & Policy_, 8(1), 2008\.
  * Collaboration [2012] Open Science Collaboration.  An open, large-scale, collaborative effort to estimate the reproducibility of psychological science.  _Perspectives on Psychological Science_ , 7(6):657–660, 2012.
  * Camerer et al. [2016] Colin F Camerer, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan, et al.  Evaluating replicability of laboratory experiments in economics.  _Science_ , 351(6280):1433–1436, 2016.
  * Mac Giolla et al. [2022] Erik Mac Giolla, Simon Karlsson, David A Neequaye, and Magnus Bergquist.  Evaluating the replicability of social priming studies.  2022\.
  * Freedman et al. [2015] Leonard P Freedman, Iain M Cockburn, and Timothy S Simcoe.  The economics of reproducibility in preclinical research.  _PLoS biology_ , 13(6):e1002165, 2015.
  * Prinz et al. [2011] Florian Prinz, Thomas Schlange, and Khusru Asadullah.  Believe it or not: how much can we rely on published data on potential drug targets?  _Nature reviews Drug discovery_ , 10(9):712–712, 2011.
  * Button et al. [2013] Katherine S Button, John Ioannidis, Claire Mokrysz, Brian A Nosek, Jonathan Flint, Emma SJ Robinson, and Marcus R Munafò.  Power failure: why small sample size undermines the reliability of neuroscience.  _Nature reviews neuroscience_ , 14(5):365–376, 2013.
  * Hewitt [2012] John K Hewitt.  Editorial policy on candidate gene association and candidate gene-by-environment interaction studies of complex traits.  _Behavior genetics_ , 42(1):1, 2012.
  * Lawrence et al. [2013] Michael S Lawrence, Petar Stojanov, Paz Polak, Gregory V Kryukov, Kristian Cibulskis, Andrey Sivachenko, Scott L Carter, Chip Stewart, Craig H Mermel, Steven A Roberts, et al.  Mutational heterogeneity in cancer and the search for new cancer-associated genes.  _Nature_ , 499(7457):214–218, 2013.
  * Simmons et al. [2011] Joseph P Simmons, Leif D Nelson, and Uri Simonsohn.  False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.  _Psychological science_ , 22(11):1359–1366, 2011\.
  * Nelson et al. [2018] Leif D Nelson, Joseph Simmons, and Uri Simonsohn.  Psychology’s renaissance.  _Annual review of psychology_ , 69:511–534, 2018.
  * Maier et al. [2022] Maximilian Maier, František Bartoš, TD Stanley, David R Shanks, Adam JL Harris, and Eric-Jan Wagenmakers.  No evidence for nudging after adjusting for publication bias.  _Proceedings of the National Academy of Sciences_ , 119(31):e2200300119, 2022.
  * Kunda [1990] Ziva Kunda.  The case for motivated reasoning.  _Psychological bulletin_ , 108(3):480, 1990.
  * Epley and Gilovich [2016] Nicholas Epley and Thomas Gilovich.  The mechanics of motivated reasoning.  _Journal of Economic perspectives_ , 30(3):133–40, 2016.
  * Bénabou and Tirole [2002] Roland Bénabou and Jean Tirole.  Self-confidence and personal motivation.  _The quarterly journal of economics_ , 117(3):871–915, 2002.
  * List [2003] John A List.  Does market experience eliminate market anomalies?  _The Quarterly Journal of Economics_ , 118(1):41–71, 2003.
  * Chang et al. [2016] Welton Chang, Eva Chen, Barbara Mellers, and Philip Tetlock.  Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments.  _Judgment & Decision Making_, 11(5), 2016.
  * Bowen [2022] Dillon Bowen.  Multiple inference: A python package for comparing multiple parameters.  _Journal of Open Source Software_ , 7(75):4492, 2022.  doi: 10.21105/joss.04492.  URL https://doi.org/10.21105/joss.04492.
  * Cai et al. [2021] Junhui Cai, Xu Han, Ya’acov Ritov, and Linda Zhao.  Nonparametric empirical bayes estimation and testing for sparse and heteroscedastic signals.  _arXiv preprint arXiv:2106.08881_ , 2021.
  * Brown and Greenshtein [2009] Lawrence D Brown and Eitan Greenshtein.  Nonparametric empirical bayes and compound decision approaches to estimation of a high-dimensional vector of normal means.  _The Annals of Statistics_ , pages 1685–1704, 2009.
  * Rubin [1981] Donald B Rubin.  The bayesian bootstrap.  _The annals of statistics_ , pages 130–134, 1981.