# How Much Should We Trust Instrumental Variable Estimates in Political
Science? Practical Advice Based on Over 60 Replicated Studies††thanks: Apoorva
Lal, PhD Candidate, Stanford University; Email<EMAIL_ADDRESS>Mac
Lockhart, PhD Candidate, University of California, San Diego; Email:
<EMAIL_ADDRESS>Yiqing Xu (corresponding author), Assistant Professor,
Stanford University; Email<EMAIL_ADDRESS>Ziwen Zu, PhD Student,
University of California, San Diego; Email<EMAIL_ADDRESS>We thank Te Bao,
Daniel Chen, Gary Cox, Charles Crabtree, Ted Enamorado, Hanming Fang, Avi
Feller, Don Green, Justin Grimmer, Anna Grzymala-Busse, Jens Hainmueller,
David Laitin, Justin McCrary, Jacob Montgomery, Doug Rivers, Henrik Sigstad,
Brandon Stewart, and seminar participants at Stanford University, Washington
University in St. Louis, APSA 2021, and Polmeth 2021 for extremely valuable
comments.

Apoorva Lal (Stanford)  Mac Lockhart (UCSD)  Yiqing Xu (Stanford)  Ziwen Zu
(UCSD)

(
First Version: July 10, 2021
This Version:
)

###### Abstract

Instrumental variable (IV) strategies are widely used in political science to
establish causal relationships. However, the identifying assumptions required
by an IV design are demanding, and it remains challenging for researchers to
assess their validity. In this paper, we replicate 67 papers published in
three top journals in political science during 2010-2022 and identify several
troubling patterns. First, researchers often overestimate the strength of
their IVs due to non-i.i.d. errors, such as a clustering structure. Second,
the most commonly used $t$-test for the two-stage-least-squares (2SLS)
estimates often severely underestimates uncertainty. Using more robust
inferential methods, we find that around 19-30% of the 2SLS estimates in our
sample are underpowered. Third, in the majority of the replicated studies, the
2SLS estimates are much larger than the ordinary-least-squares estimates, and
their ratio is negatively correlated with the strength of the IVs in studies
where the IVs are not experimentally generated, suggesting potential
violations of unconfoundedness or the exclusion restriction. To help
researchers avoid these pitfalls, we provide a checklist for better practice.

Keywords: instrumental variables, two-stage-least-squared, replications, weak
instrument, exclusion restriction, replication

## 1\. Introduction

The instrumental variable (IV) approach is a commonly used empirical method in
the social sciences, including political science, for establishing causal
relationships. It is often used when selection on observables is implausible,
experimentation is infeasible or unethical, and rule-based assignments that
allow for sharp regression-discontinuity (RD) designs are not available. In
recent years, there has been a growing number of papers published in top
political science journals, such as the _American Political Science Review_
(APSR), _American Journal of Political Science_ (AJPS), and _Journal of
Politics_ (JOP), that use IV as a primary causal identification strategy. This
trend can be traced back to the publication of the textbook _Mostly Harmless
Econometrics_ (Angrist and Pischke, 2008), which popularized the modern
interpretation of IV designs, and Sovey and Green (2011), which clarifies the
assumptions required by an IV approach and provides a useful checklist for
political scientists.

Figure 1: IV Papers Published in the _APSR_ , _AJPS_ , and _JOP_

Note: Our criteria rule out IV models appearing in the online appendix only,
in dynamic panel settings, with multiple endogenous variables, and with
nonlinear link functions. Non-replicability is primarily due to a lack of data
and/or coding errors.

Despite its popularity, some researchers have questioned the validity of the
IV approach based on the observation that two-stage least-squares (2SLS)
estimates are often much larger in magnitude than “naïve” ordinary-least-
squares (OLS) estimates, even when the main concern with the latter is upward
omitted-variables bias.111For example, in the 2016 National Bureau of Economic
Research–Political Economy Meeting, following a presentation of a study using
an IV approach, the late political economist Alberto Alesina asked the
audience: “How come 2SLS estimates are always five times bigger than OLS
estimates in political economy?” Others have raised concerns about the
validity of inferential methods used for 2SLS estimation (e.g. Lee et al.,
2022; Young, 2022).

These observations motivate our systematic examination of the use of IVs in
the empirical political science literature. We set out to replicate all papers
published in the APSR, AJPS, and JOP during the past thirteen years
(2010-2022) that use an IV design with a single endogenous variable as one of
the main identification strategies.222Focusing on design with a single
endogenous variable allows us to calculate the correlation coefficient between
the treatment and the predicted treatment and apply powerful tools such as the
Anderson-Rubin (AR) test and the $tF$-$cF$ test (when there is only one IV).
Moreover, we find it difficult to justify the exclusion restriction in a
multiple-treatment-multiple-instrument setting in the first place. We start
with the universe of 114 papers that meet this criterion, of which 76 have
complete replication materials online, a finding that is a troubling pattern
in its own right. Among the 71 papers, we successfully replicate at least one
of the main IV results for 67 papers. Among those, three papers each have two
separate IV designs, producing separate 2SLS results.

Using data from these 70 IV designs, we conduct a programmatic replication
exercise and find three troubling patterns. First, we find that a significant
number of IV designs in political science either do not report the first-stage
partial $F$ statistic or underestimate it by failing to adjust standard errors
(SEs) for factors such as heteroskedasticity, serial correlation, or
clustering structure. Using the effective $F$-statistic (Olea and Pflueger,
2013), we find that at least 11% of the published IV studies rely on what
econometricians call “weak instruments,” the consequences of which have been
well-documented in the literature under the assumption that the instrument is
valid.333See Andrews, Stock and Sun (2019) for a comprehensive review.

A second related issue we examine is statistical inference. We find that a
considerable number of the IV designs we replicate are underpowered. Most
studies rely on $t$-tests based on analytic standard errors and traditional
critical values (such as 1.96 for statistical significance at the 5% level) to
make inferences about the 2SLS coefficients. However, recent research (Young,
2022) demonstrates that in the presence of outliers, such tests severely
underestimate the uncertainties of the 2SLS estimates and may lead to false
discoveries in economic research. Our findings are consistent with this
pattern. When we use bootstrapping procedures, the AR test, or the $tF$-$cF$
procedure, an $F$-statistic-dependent $t$-test (Lee et al., 2022), to perform
hypothesis testing, we find that, depending on the method employed, 19-30% of
designs are statistically underpowered at the 5% level, whereas the number
based on the SEs or $p$-values reported in the original papers is only 10%.
This suggests that inferences based on traditional $t$-tests may not
accurately reflect the uncertainties in 2SLS estimates in a significant
portion of cases.

Last but not least, our replications corroborate evidence from economics and
finance that the 2SLS estimates are often much bigger in magnitude than the
OLS estimates obtained from regressing the outcome on the potentially
endogenous treatment variables and covariates (Jiang, 2017). In 68 out of the
70 designs (97%), the 2SLS estimates are bigger than the OLS estimates in
magnitude; among them, 24 (34%) are at least five times bigger. This is
alarming because, in an IV design with observational data, researchers often
say that they are most concerned about the upward bias of the treatment effect
estimates produced by naïve OLS. Even after we exclude 15 papers that
explicitly claim to expect downward biases in OLS estimates, the percentages
remain high (96% and 35%, respectively).

The first two patterns may be due to researchers’ unfamiliarity with recent
development in the IV literature, such as the effective $F$ statistic and the
$tF$-$cF$ test, or under-utilization of inferential procedures robust to weak
instruments, such as the AR test. Therefore, researchers can avoid these
problems by adopting better practices. The third finding, however, is the most
concerning. We cannot explain it with weak instruments alone because, at least
in the case of i.i.d. errors, when the other identifying assumptions are
satisfied. weak instruments bias 2SLS estimates toward OLS estimates in finite
samples (Bound, Jaeger and Baker, 1995), but what we observe is the opposite.
The ratio between the magnitudes of the 2SLS and OLS estimates is strongly
negatively correlated with the replicated partial correlation coefficient
between the instrument and the treatment among studies that use non-
experimental instruments; the relationship is much weaker among studies with
experimental instruments.

We suspect several possibilities are primarily responsible for this pattern.
First, it can be caused by a combination of weak first stages and the failure
of the (conditional) exogeneity assumption, either because the instruments are
confounded or because they violate the exclusion restriction. Intuitively,
because the 2SLS estimator is a ratio, when the magnitude of the numerator is
inflated due to endogenous IVs or the failure of the exclusion restriction
while the magnitude of the denominator is a small number because of weak
instruments, the 2SLS estimate explodes. Another possibility is the
publication bias: a weak first stage leads to highly variant 2SLS estimates
and those with large magnitudes are more likely to be reported and published
(Gelman and Carlin, 2014), hence the negative correlation. Other explanations,
such as heterogeneous treatment effects (HTE) and measurement errors in
treatment variables, may also contribute to this phenomenon, though we doubt
they are the main driving forces given the different patterns between
observational and experimental studies.

What do these findings mean for the empirical IV literature in political
science? First, traditional $t$ tests for the 2SLS estimates (especially those
based on classic analytic SEs) mask the fact that most IV results are highly
uncertain, which creates additional opportunities for selective reporting,
leading to publication bias. Second, and more importantly, many of the 2SLS
estimates likely suffer from large biases due to failures of the identifying
assumptions and hence are not credible. As Sekhon and Titiunik (2012) point
out, many “natural experiments” in observational studies on which many
instruments are generated are fundamentally different from real experiments
and seldom straightforward to analyze without detailed knowledge of the
assignment mechanism. Although we cannot definitively say which estimates are
problematic, the underlying issue seems to prevail in the IV literature.

The goal of this paper is not to discredit existing IV studies or dissuade
researchers from ever using the IV approach. On the contrary, our objectives
are twofold: We want to caution researchers against the danger of justifying
their instruments in an ad-hoc fashion, especially in observational studies;
and we want to provide researchers with practical advice that we hope can
improve future practice, including correctly quantifying the strength of
instruments and uncertainties of 2SLS estimates, as well as conducting
additional analysis such as a placebo test to corroborate the identifying
assumptions.

We contribute to a growing literature that evaluates the use of IV strategies
in empirical work across the social sciences and provides methods to improve
empirical practice. Young (2022) replicates IV designs from 32 articles
published in economics journals and finds that IV estimates are more sensitive
to outliers than their OLS counterparts and that conventional $t$ tests
systematically understate uncertainties and therefore have high false
discovery rates. Jiang (2017) surveys over 250 articles published in finance
journals and finds that the vast majority of them report IV estimates that are
larger than the corresponding OLS estimates regardless of the sign of the
potential omitted variables bias, and postulates that this bias can be
attributed to exclusion restriction violations exacerbated by weak
instruments, or local effects that are far from representative of the
population treatment effect; however, the author does not conduct
replications. Mellon (2020) proposes to use sensitivity analysis to quantify
the vulnerability of using weather as instruments to exclusion restriction
violations. Dieterle and Snell (2016) develop a quadratic over-identification
test for the first stage 2SLS, apply it to 15 published papers that use linear
first stage, and find significant non-linearities in 10 of them, and suggest
that evaluating the implied patterns of heterogeneity from their test relative
to theoretical predictions can be used to gauge the validity of an instrument.
Felton and Stewart (2022) survey 15 IV papers published in top sociology
journals and find that the assumptions required by an IV design are often
unstated and none of the studies use weak-instrument robust tests or
bootstrapped CIs. To the best of our knowledge, we are the first to link the
discrepancy between 2SLS and OLS estimates with the problem of weak IVs using
large-scale replication data from the social sciences.

The organization of the paper is as follows. In Section 2, we briefly review
the IV method under the traditional parametric framework, including the
required identifying assumptions, estimation strategies, potential pitfalls,
and newly developed methods in the literature. Section 3 details our case
selection criteria and the resulting replication sample. Section 4 describes
our replication procedure and presents the main results of our replication
exercise. The last section concludes the paper with a set of practical advice
for researchers.

## 2\. Theoretical Refresher

In this section, we offer a brief overview of the IV approach, including the
setup, the identifying assumptions, as well as the 2SLS estimators. We then
discuss potential pitfalls and survey several inferential methods. To cover
the vast majority of IV studies in political science, we adopt a traditional
constant treatment effect approach to IV designs, which imposes a set of
parametric assumptions. For example, 51 (73%) designs in our replicated sample
employ continuous treatment variables and make no reference to treatment
effect heterogeneity, hence, they are ill-suited for the local average
treatment effect (LATE) approach (Angrist, Imbens and Rubin, 1996).

For simplicity, we do not include additional exogenous controls in the
discussion without loss of generality. This is because, by the Frisch-Waugh-
Lovell theorem, we can remove them by regressing the outcome, treatment, and
IVs on the controls and using the residuals for all subsequent analyses.

Figure 2: A Directed Acyclic Graph of an IV Design

Apart from the “canonical” use of IVs in addressing non-compliance in
experimental encouragement designs, we observe that in the majority of the
papers we review, researchers use IVs in observational settings to establish
causality between a single treatment variable $d$ and an outcome variable $y$.
The basic idea of this approach is to use an instrument $z$ to isolate
”exogenous” variation in $d$ (i.e., the variation in $d$ that is not related
to potential confounders) and estimate its causal effect on $y$. Figure 1
illustrates the directed acyclic graph for an IV design, where $\varepsilon$
denotes the error term that captures all unexplained variations in $y$. The
figure depicts that because $d$ and $\varepsilon$ are correlated, an observed
correlation between $d$ and $y$ does not identify the causal effect of $d$ on
$y$. It also shows that an IV approach relies on three crucial assumptions:
(1) the _relevance_ of the instrument, which is directly testable, meaning
that $z$ is correlated with $d$; (2) the _unconfoundedness_ assumption, which
states that $z$ is quasi-randomly assigned, and (3) the _exclusion
restriction_ , which posits that $z$ does not have a direct effect on $y$
beyond the channel through $d$.

###  2.1 Estimation Strategies

Imposing a set of parametric assumptions, we define a system of simultaneous
equations:

$\displaystyle\text{Structural equation:}\quad y=$ $\displaystyle\ \alpha+\tau
d+\varepsilon$ (2.1) $\displaystyle\text{First-stage equation:}\quad d=$
$\displaystyle\ \pi_{0}+\pi_{1}z+\nu$ (2.2)

in which $y$ is the outcome variable, $d$ is a scalar treatment variable; $z$
is a vector of instruments for $d$; $\tau$ captures the (constant) treatment
effect and is the key quantity of interest. Equations (2.1) and (2.2) are
referred to as the structural equation and the first-stage equation,
respectively. The error terms $\varepsilon$ and $\nu$ in the two equations may
be correlated.

The endogeneity problem for $\tau$ in Equation (2.1) arises when $d$ and
$\varepsilon$ are correlated, which renders $\hat{\tau}_{OLS}$ from a naïve
OLS regression of $y$ on $d$ inconsistent. The endogeneity problem may be due
to one of the following reasons: (1) unmeasured omitted variables that are
correlated with both $y$ and $d$; (2) measurement error in $d$, or (3)
simultaneity or reverse causality, which means $y$ may also affect $d$. The IV
approach addresses this problem by taking advantage of the exogenous variation
in $d$ brought by $z$. Substituting $d$ in Equation (2.1) using Equation
(2.2), we have the reduced form equation:

$\text{Reduced form:}\quad
y=\underbrace{(\alpha+\tau\pi_{0})}_{\gamma_{0}}+\underbrace{(\tau\pi_{1})}_{\gamma_{1}}z+(\tau\nu+\varepsilon).$
(2.3)

Substitution establishes that $\gamma_{1}=\tau\pi_{1}$, rearranging yields
$\tau=\frac{\gamma_{1}}{\pi_{1}}$ (assuming that we only use one instrument,
but the intuition carries over to cases with multiple instruments). The IV
estimate, therefore, is the ratio of the reduced-form and first-stage
coefficients. To identify $\tau$, we make the following assumptions (Greene,
2003, Chapter 12).

###### Assumption 1 (Relevance).

$\pi_{1}\neq 0$. This assumption requires that the IVs can predict the
treatment variable, and is therefore equivalently stated as
$d\not\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern
5.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 5.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
5.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
5.0mu{\scriptscriptstyle\perp}}}z$.

###### Assumption 2 (Exogeneity).

$\text{Cov}(z,\varepsilon)=0$ and $\mathbb{E}[\varepsilon]=0$. Assumption 2 is
satisfied when unconfoundedness and the exclusion restriction are satisfied.
However, without additional structural assumptions. failures of
unconfoundedness and the exclusion restriction are observationally equivalent,
therefore, we do not distinguish them in the analysis and diagnostics.

Under Assumptions 1 and 2, the 2SLS estimator is shown to be consistent for
the structural parameter $\tau$.

##### The 2SLS estimator.

Consider a sample of $N$ observations. We can write
$\mathbf{d}=(d_{1},d_{2},\cdots,d_{N})^{\prime}$ and
$\mathbf{y}=(y_{1},y_{2},\cdots,y_{N})^{\prime}$ as $(N\times 1)$ vectors of
the treatment and outcome data, and
$\mathbf{z}=(z_{1},z_{2},\cdots,z_{N})^{\prime}$ as $(N\times p_{z})$ matrix
of instruments in which $p_{z}$ is the number of instruments. To simplify
mathematics, we residualize original $\mathbf{d}$, $\mathbf{y}$, and each
column of $\mathbf{z}$ against the exogenous covariates, obtaining
$\mathbf{y}$, $\mathbf{d}$, and $\mathbf{z}$, respectively. The 2SLS estimator
is written as follows:

$\hat{\tau}_{2SLS}=\left({\mathbf{d}^{\prime}\mathbf{P}_{z}\mathbf{d}}\right)^{-1}\mathbf{d}^{\prime}\mathbf{P}_{z}\bm{y}\vspace{-0.5em}$
(2.4)

in which
$\mathbf{P}_{z}=\mathbf{z}\left({\mathbf{z}^{\prime}\mathbf{z}}\right)^{-1}\mathbf{z}^{\prime}$
is the hat-maker matrix from the first stage which projects the endogenous
treatment variable $\mathbf{d}$ into the column space of $\mathbf{z}$, thereby
preserving only the exogenous variation in $\mathbf{d}$ that is uncorrelated
with $\varepsilon$. This formula permits the use of more than one instrument,
in which case the model is said to be “overidentified.” The 2SLS estimator
belongs to a class of generalized method of moments (GMM) estimators taking
advantage of the moment condition $\mathbb{E}[z\varepsilon]=0$, including the
two-step GMM (Hansen, 1982) and limited information maximum likelihood (LIML)
estimators (Anderson, Kunitomo and Sawa, 1982). We use the 2SLS estimator
throughout the replication exercise because of its simplicity and because
every single paper in our replication sample uses it in at least one
specification.

When the model is exactly identified, i.e., the number of treatment variables
equals the number of instruments, the 2SLS estimator can be simplified as the
IV estimator:
$\hat{\tau}_{2SLS}=\hat{\tau}_{IV}=\left({\mathbf{z}^{\prime}\mathbf{d}}\right)^{-1}\mathbf{z}^{\prime}\mathbf{y}$.
In the case of one instrument and one treatment, the 2SLS estimator can also
be written as a ratio of two sample covariances:
$\hat{\tau}_{2SLS}=\hat{\tau}_{IV}=\frac{\hat{\gamma}_{1}}{\hat{\pi}_{1}}=\frac{\widehat{\rm
Cov}(\mathbf{y},\mathbf{z})}{\widehat{\rm Cov}(\mathbf{d},\mathbf{z})}$, which
illustrates that the 2SLS estimator is a ratio between reduced-form and first-
stage coefficients in this special case. This further simplifies to a ratio of
the difference in means when $z$ is binary, which is called a Wald estimator.

###  2.2 Potential Pitfalls in Implementing an IV Strategy

The challenges with 2SLS estimation and inference are often due to the
violation of the two identifying assumptions. These difficulties can result in
(1) significant uncertainty around 2SLS estimates and size distortion for $t$
tests due to weak instruments even when Assumption 2 is valid; and (2)
potentially larger biases in 2SLS estimates compared to OLS estimates when
both assumptions are violated.

##### Inferential problem due to weak instruments.

Since the IV coefficient is a ratio, the weak instrument problem is a “divide-
by-zero” problem, which arises when $\text{Cov}(z,x)\approx 0$ (i.e., when
Assumption 1 is violated). The instability of ratio estimators like
$\widehat{\tau}_{2SLS}$ when the denominator is approximately zero has been
extensively studied going back to Fieller (1954). The conventional wisdom in
the past two decades has been that the first-stage partial $F$ statistic needs
to be bigger than 10, and it should be clearly reported (Staiger and Stock,
1997). As a rule of thumb, the original cutoff is chosen based on simulation
results to meet two criteria under i.i.d. errors: (1) in the worst case, the
bias of the 2SLS estimator does not exceed 10% of the bias of the OLS
estimator, and (2) a $t$-test based on the 2SLS estimator with a size of 5%
does not lead to size over 15%.

The literature has discussed at least three issues caused by weak instruments
when Assumption 2 is valid. First, under i.i.d. errors, a weak first stage
exacerbates the finite bias of the 2SLS estimator toward the inconsistent OLS
estimator, thereby reproducing the endogeneity problem that an IV design was
meant to solve (Staiger and Stock, 1997). Second, the 2SLS estimates become
very imprecise.444To illustrate, a commonly used variance estimator for
$\hat{\tau}_{IV}$ can be written as:
$\hat{\mathbb{V}}(\hat{\tau}_{IV})\approx\frac{\hat{\Sigma}^{2}}{\sum_{i=1}^{N}(x_{i}-\overline{x})^{2}}\frac{1}{R_{xz}^{2}}=\hat{\mathbb{V}}(\hat{\tau}_{OLS})\frac{1}{R_{xz}^{2}}$
in which $\hat{\sigma}^{2}$ is a variance estimator for the error term and
$R^{2}_{xz}$ is the R-squared from the first stage. The estimated variance is
mechanically larger than the estimated variance of the OLS estimator as long
as $R^{2}_{xz}<1$. It is decreasing in $R^{2}_{xz}$, i.e. stronger instruments
produce more precise IV estimates. A third and related issue is that the tests
are of the wrong size and the $t$-statistics don’t follow a $t$-distribution
(Nelson and Starz, 1990). Issues relating to imprecision and test-statistic
size arise from the fact that the distribution of $\hat{\tau}$ is derived from
its linear approximation of $\hat{\tau}$ in ($\hat{\gamma},\hat{\pi}$),
wherein normality of the two OLS coefficients implies the normality of their
ratio. However, this normal approximation breaks down when $\hat{\pi}\approx
0$. Moreover, this approximation failure cannot generally be rectified by
bootstrapping (Andrews and Guggenberger, 2009), although Young (2022) argues
that it nevertheless allows for improved inference when outliers are present.
Overall, valid IV inference relies crucially on correctly identifying strong
IVs.

In general, there are two approaches for conducting inference in an IV design:
the pretesting approach and the direct testing approach. The pretesting
approach involves testing the strength of the first stage using some form of
$F$ statistic, and if the $F$ statistic exceeds a certain threshold (e.g.
$F>10$), proceeding with a test on the null hypothesis about the treatment
effect, such as $\tau=0$, while the direct testing approach does not rely on
passing a pretest. Almost all of the studies we review employ the pretesting
approach. In this paper, we review four methods for statistical inference for
IV designs. The first three methods are related to pretesting, while the last
one is a direct test.

First, Olea and Pflueger (2013) propose the effective $F$ statistic for both
just-identified and over-identified settings and accommodates robust or
cluster-robust SEs. The effective $F$ is a scaled version of the first-stage
$F$ statistic and is computed as
$F_{\text{Eff}}=\hat{\pi}^{\prime}\hat{Q}_{ZZ}\hat{\pi}/\text{tr}(\hat{\Sigma}_{\pi\pi}\hat{Q}_{ZZ})$,
where $\hat{\Sigma}_{\pi\pi}$ is the variance-covariance matrix of the first
stage regression, and
$\hat{Q}_{ZZ}=\frac{1}{N}\sum_{i=1}^{N}z_{i}z_{i}^{\prime}$. In just-
identified cases, $F_{\text{Eff}}$ is the same as $F$ statistics based on
robust or cluster-robust SEs. Olea and Pflueger (2013) derive the critical
values for $F_{\text{Eff}}$ and notes that the statistic and corresponding
critical values are identical to the better-known robust $F$ statistic
$\hat{\pi}\hat{\Sigma}_{\pi\pi}^{-1}\hat{\pi}$ and corresponding Stock and
Yogo (2005) critical values. Using Monte Carlo exercises, researchers have
shown that $F_{\text{Eff}}>10$ is a reasonable rule of thumb under
heteroskedasticity or group structure (Olea and Pflueger, 2013; Andrews, Stock
and Sun, 2019).

Second, Young (2022) recommend researchers report two types of bootstrap
confidence intervals (CIs), _bootstrap-c_ and _bootstrap-t_ , for
$\hat{\tau}_{2SLS}$ under non-i.i.d. errors with outliers, which is common in
social science settings. This involves $B$ replications of the following
procedure: (1) sample $n$ triplets $(y_{i}^{*},d_{i}^{*},\mathbf{z}_{i}^{*})$
independently and without replacement from the original sample (with
appropriate modifications for clustered dependence) and (2) compute the
$\hat{\tau}_{\text{2SLS}}$ coefficient and SE, as well as the corresponding
test statistic
$t^{*}=\hat{\tau}^{*}_{\text{2SLS}}/\hat{\text{SE}}(\hat{\tau}^{*}_{\text{2SLS}})$
on each replication. The _bootstrap-c_ method calculates the CIs by taking the
$\alpha/2$ and $(1-\alpha/2)$ percentiles of the bootstrapped 2SLS coefficient
$\hat{\tau}_{\text{2SLS}}^{*}$, while the _bootstrap-t_ method calculates the
percentile-$t$ refined CIs by plugging in the $\alpha/2$ and $(1-\alpha/2)$
percentile of bootstrapped $t$ statistics $t^{*}_{\alpha/2}$ and
$t^{*}_{1-\alpha/2}$ into the expression $\hat{\tau}_{\text{2SLS}}\pm
t^{*}_{\alpha\mid 1-\alpha}\hat{\text{SE}}(\hat{\tau}^{*}_{\text{2SLS}})$.
Hall and Horowitz (1996) show that _bootstrap-t_ achieves an asymptotic
refinement over _bootstrap-c_.555We use the percentile method instead of
bootstrapped SEs because the $t$-test for the 2SLS estimates based on the
latter may be overly conservative (Hahn and Liao, 2021).

Third, in just-identified single treatment settings, Lee et al. (2022) propose
the $tF$-$cF$ procedure that smoothly adjusts the $t-$ratio inference based on
the first-stage $F$ statistic, which improves upon the ad-hoc screening rule
of $F>10$. The adjustment factor applied to 2SLS SEs is based on the first
stage $t-$ratio
$\hat{f}\vcentcolon=\hat{\pi}/\sqrt{\hat{\mathbb{V}}(\hat{\pi})}$, with the
first stage $\hat{F}=\hat{f}^{2}$, and relies on the fact that the distortion
from employing the standard 2SLS $t$-ratio
$\hat{t}\vcentcolon=\hat{\tau}/\sqrt{\hat{\mathbb{V}}(\hat{\tau})}$ can be
quantified in terms of $\hat{F}$ and $AR-$statistic, which gives rise to a set
of critical values for a given pair of $\hat{t}$ and $\hat{F}$. The authors
also show that, if no adjustment is made to the $t$-test’s critical value
(e.g., using 1.96 as the threshold for 5% statistical significance), a first
stage $\hat{F}$ of 104.7 is required to guarantee a correct size of $5\%$ for
a two-sided $t$-test for the 2SLS coefficient.

Finally, in the single treatment case, the AR procedure, which is essentially
an $F$ test on the reduced form, is a direct inferential method robust to weak
instruments (Anderson and Rubin, 1949; Chernozhukov and Hansen, 2008). Without
loss of generality, assume that we are interested in testing the null
hypothesis that $\tau=0$, which then implies that the reduced form coefficient
from regressing $y$ on $\mathbf{z}$, $\gamma_{1}=0$. This motivates the
following procedure: given a set $\mathcal{T}$ of potential values for
$\widetilde{\tau}$, for each value $\widetilde{\tau}$, construct
$\widetilde{y}=y-d\widetilde{\tau}$, and regress $\widetilde{y}$ on
$\mathbf{z}$ to obtain a point estimate $\widetilde{\gamma}$ and (robust, or
cluster robust) covariance matrix
$\widetilde{\mathbb{V}}(\widetilde{\gamma})$, and construct a Wald statistic
$\widetilde{W}_{s}(\widetilde{\gamma})\vcentcolon=\widetilde{\gamma}^{\prime}\widetilde{\mathbb{V}}(\widetilde{\gamma})^{-1}\widetilde{\gamma}$.
Then, the AR confidence region (CR) is the set of $\widetilde{\gamma}$ such
that $\widetilde{W}_{s}(\widetilde{\gamma})\leq c(1-p)$ where $c(1-p)$ is the
$(1-p)^{\text{th}}$ percentile of the $\chi^{2}_{1}$ distribution. The AR test
not only requires no pretesting but is also shown to be the most efficient in
the just-identified case. However, it is not as commonly used as procedures
that involve pretesting, possibly because researchers are more accustomed to
using $t$-tests than $F$/Wald tests and reporting SEs rather than CRs.

##### Bias amplification and the failure of Assumption 2.

When the number of instruments is bigger than the number of endogenous
treatments, researchers can use an over-identification test to gauge the
plausibility of Assumption 2 (Arellano, 2002). However, such a test is often
underpowered and has bad finite sample properties (Davidson and MacKinnon,
2015). In just-identified cases, Assumption 2 is not directly testable. When
combined with weak instruments, even small violations of Assumption 2 can
produce inconsistency. This is because
$\text{plim}\;\hat{\tau}_{IV}=\tau+\frac{\text{Cov}(z,\varepsilon)}{\text{Cov}(z,d)}$.
When $\text{Cov}(z,d)\approx 0$, even small violations of exogeneity, i.e.,
$\text{Cov}(z,\varepsilon)\neq 0$, will enlarge the second term, resulting in
large biases. Thus, the two identifying assumption failures exacerbate each
other: having weak instruments compounds problems from confounding or
exclusion restriction violations, and vice versa. With invalid instruments, it
is possible that the asymptotic bias of the 2SLS estimator is greater than
that of the OLS estimator, i.e.,
$\left|\frac{\text{Cov}(z,\varepsilon)}{\text{Cov}(z,d)}\right|\gg|\text{Cov}(d,\varepsilon)|$
in the single instrument case.

While the inference problem can be alleviated by employing alternative
inferential methods as described above, the failure of Assumption 2 is much
harder to diagnose and address. This is because it is inherently a research
design issue and should ultimately be tackled at the research design stage.
Indeed, researchers typically spend considerable effort arguing for both
unconfoundedness and the exclusion restrictions in their particular setting.
In the SI, we provide an explanation of the “zero-first-stage” (ZFS) test
(Bound and Jaeger, 2000), which is essentially a placebo test on a subsample
where the instrument is expected to be uncorrelated with the treatment, to
help researchers gauge the validity of their instruments. These estimates can
then be used to debias the 2SLS estimate using the methods proposed in Conley,
Hansen and Rossi (2012). We discuss these methods in the Supporting Materials
(SM).

## 3\. Data and Types of Instruments

In this section, we first discuss our case selection criteria and the
replication sample, which is the focus of our subsequent analysis. We then
describe the types of instruments in the replicable studies.

##### Data.

We examine all empirical papers published in the APSR, AJPS, and JOP (printed
versions) from 2010 to 2022 and identify studies that use an IV strategy as
one of the main identification strategies, including papers that use binary or
continuous treatments and that use a single or multiple instruments.
Specifically, we use the following criteria: (1) the discussion of the IV
result needs to appear in the main text and support a main argument in the
paper; (2) we consider linear models only; in other words, papers that use
discrete outcome models are excluded from our sample; (3) we exclude papers
that include multiple endogenous variables in a single specification because
estimated treatment effects are correlated with one another (multiple
endogenous variables in separate specifications are included); (4) we exclude
papers that use IV or GMM estimators in a dynamic panel setting because they
are subject to a separate set of empirical issues and their poor performance
has been thoroughly discussed in the literature (Bun and Windmeijer, 2010).
These criteria result in 30 papers in the APSR, 33 papers in the AJPS, and 51
papers in the JOP. We then strive to find replication materials for these
papers from public data-sharing platforms, such as the Harvard Dataverse, and
the authors’ websites. We are able to locate replication materials for 76
(62%) papers.666Among them, two papers have incomplete data, which makes it
impossible for us to replicate the results. However, code completeness and
quality of documentation vary a great deal. Data availability has
significantly improved since 2016-2017 following new editorial policies
requiring authors to make replication materials publicly available, though
none of the journals requires full replicability administrated by a third
party as a condition for publication (Key, 2016), which would constitute a
major improvement in our view.

Table 1: Data Availability and Replicability of Papers Using IVs |  |  Incomplete |  Incomplete |  Replication |
---|---|---|---|---|---
| #All Papers | Data | Code | Error | Replicable
APSR | 30 | 16 | 0 | 3 | 14 (42%)
AJPS | 33 | 3 | 1 | 1 | 25 (83%)
JOP | 51 | 19 | 3 | 1 | 28 (55%)
Total | 114 | 38 | 4 | 5 | 67 (59%)

Using data and code from the replication materials, we set out to replicate
the main IV results in the 67 papers. Our replicability criterion is simple:
As long as we can exactly replicate _one_ 2SLS point estimate that appears in
the paper, we deem the paper replicable. We do not aim at exactly replicating
SEs, $z$-scores, or level of statistical significance for the 2SLS estimates
because they involve the choice of the inferential method, which we will
discuss in the next section.

After much effort and hundreds of hours of work, we are able to replicate the
main results of 61 papers.777For three papers, we are able to produce the 2SLS
estimates with perfectly executable code; however, our replicated estimates
are inconsistent with what was reported in the original studies. We suspect
the inconsistencies are caused by data rescaling or misreporting; hence, we
keep them in the sample. The low replication rate is consistent with what is
reported in Hainmueller, Mummolo and Xu (2019). The main reasons for failures
of replication are incomplete data (38 papers), incomplete code or poor
documentation (4 papers), and replication errors (5 papers). Table 1 presents
summary statistics on data availability and replicability of IV papers for
each of the three journals. The rest of this paper focuses on results based on
these 61 replicable papers (and 64 IV designs).

##### Types of IVs.

Inspired by Sovey and Green (2011), in Table 2, we summarize the types of IVs
in the replicable designs, although our categories differ from theirs to
reflect changes in the types of instruments used in the discipline. As in
Sovey and Green (2011), the biggest category is “Theory,” in which the authors
justify Assumption 2, including IVs’ quasi-randomness and the exclusion
restriction, using social science theories or substantive knowledge. We
further divide theory-based IVs into four subcategories:
geography/climate/weather, history, treatment diffusion, and others.

Many studies in the theory category justify the choices of their instruments
based on geography, climate, or weather conditions. For example, Zhu (2017)
uses weighted geographic closeness as an instrument for the activities of
multinational corporations; Hager, Krakowski and Schaub (2019) use mean
elevation and distance to rivers to instrument equitable inheritance customs;
and Grossman, Pierskalla and Boswell Dean (2017a) use the number of distinct
landmasses as an instrument for government fragmentation. Henderson and Brooks
(2016) use rainfall around Election Day as an IV for democratic vote margins.
The popularity of weather instruments for a whole host of outcomes necessarily
implies that the exclusion restriction is especially tenuous in such cases
(Mellon, 2020).

Table 2: Types of Instruments Type | #Papers | Percentage %
---|---|---
Theory | 42 | 60.0
Geography/climate/weather |  13 |  18.6
History |  11 |  15.7
Treatment diffusion |  2 |  2.9
Others |  16 |  22.9
Experiment | 12 | 17.1
Econometrics | 9 | 12.9
Interactions/“Bartik” |  7 |  10.0
Lagged treatment |  1 |  1.4
Empirical test |  1 |  1.4
Rules & policy changes | 7 | 10.0
Change in exposure |  3 |  4.3
Fuzzy RD |  4 |  5.7
Total | 70 | 100.0

Historical IVs are based on historical differences between units that cannot
be explained by current levels of the treatment. For example, Vernby (2013)
uses historical immigration levels as an instrument for the current number of
non-citizen residents. Similarly, Spenkuch and Tillmann (2018a) use historical
decisions by rulers in Europe over the religion of their region to instrument
for the current religion of survey respondents. These studies use historical
variation as instruments for current or modern variables.

Several studies base their choices on regional diffusion of treatment. For
example, Dube and Naidu (2015) use US military aid to countries outside Latin
America as an instrument for US military aid to Colombia. Grossman, Pierskalla
and Boswell Dean (2017b) use over-time variation in the number of regional
governments to instrument government fragmentation in sub-Saharan Africa.
Dorsch and Maarek (2019a) use regional share of democracies as an IV for
democratization in a country-year panel.

Finally, several papers rely on a unique instrument based on theories that we
could not place in a category. For example, Carnegie and Marinov (2017a) use
the rotating presidency of the Council of the European Union as an instrument
for official development aid. They argue that countries that were colonized by
the country that holds the presidency receive exogenously more aid than other
countries. In another example, Dower et al. (2018b) use religious polarization
as an instrument for the frequency of unrest and argue that religious
polarization could only impact collective action through its impact on
representation in local institutions. These papers employed IVs that were
usually unique to the paper.

The second-biggest category is randomized experiments. Articles in this
category employ randomization, designed and conducted by researchers or a
third party, to make causal inference and use 2SLS estimation to address non-
compliance issues in an encouragement design—the IV normally is being
encouraged to take the treatment. With random assignment, we have more
confidence in Assumption 2 because $z\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
5.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 5.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
5.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
5.0mu{\scriptscriptstyle\perp}}}v$ by design, and the direct effect of
encouragement on the outcome is easier to rule out than without random
assignment.

Another category of IVs are based on explicit rules, which generate quasi-
random variation in the treatment. Sovey and Green (2011) refer to this
category as “Natural Experiment.” We avoid this terminology because it is
widely misused. We limit this category to two circumstances: fuzzy regression
discontinuity (RD) designs and variation in exposure to policies due to time
of birth or eligibility.888The difference between the two is subtle: For the
latter, the gap in the forcing variable, such as birth cohort, is fixed and
cannot be arbitrarily small. For example, Kim (2019) leverages a reform in
Sweden that requires municipalities above a population threshold to adopt
direct democratic institutions. Dinas (2014) uses eligibility to vote based on
age at the time of an election as an instrument for whether respondents did
vote. Respondents who were 18 as of election day could vote while those who
were 17 could not, making this a problem of one-sided non-compliance, as some
respondents who were over 18 did not vote.

The last category of IVs are based on econometric assumptions. This category
includes what Sovey and Green (2011) call lags. These are econometric
transformations of variables the author argues constitute an instrument. For
example, Lorentzen, Landry and Yasuda (2014a) use a measure of the independent
variable from 8 years earlier to mitigate endogeneity concerns. Another
example of this are instruments that rely on a transformation of variables to
satisfy the assumptions. For example, Dorsch and Maarek (2019b) use the sum of
neighboring countries with similar institutions that are democracies as an
instrument for democratization within a country. Shift-share “Bartik”
instruments that are based on interactions between multiple variables are also
included in this category (Goldsmith-Pinkham, Sorkin and Swift, 2020). For
example, Baccini and Weymouth (2021) use the share of jobs in a given industry
within a county interacted with national-level changes in employment in that
industry to study the effect of manufacturing layoffs on voting.

Compared with IV papers published before 2010, there is a significant increase
in the proportion of papers using experiment-generated IVs (from 2.9% to
18.8%) thanks to an increased popularity of survey and field experiments. In
stark contrast, the number of papers relying on econometric techniques or
flawed empirical tests (such as regression $y$ on $d$ and $z$ in one
regression and check whether the coefficient of $z$ is significant) to justify
potential IVs has decreased thanks to improving empirical practice in the
discipline. The percentage of papers using theory-justified IVs remains almost
the same at around 60%.

## 4\. Replication Procedure and Results

In this section, we describe our replication procedure and report the main
findings.

##### Procedure.

For each paper, we select the main IV specification that plays a central role
in supporting a main claim in the paper; it is either referred to as the
baseline specification or appears in one of the main tables or figures.
Focusing on this specification, our replication procedure involves the
following steps. First, we compute the first-stage partial $F$ statistics
based on (1) classic analytic SEs, (2) Huber White heteroskedastic-robust SEs,
(3) cluster-robust SEs, if there is a clustering structure according to the
authors, and (4) bootstrapped SEs.999They are calculated by
$F_{boot}=\hat{\tau}_{2SLS}^{\prime}{\hat{\rm
Var}_{boot}(\hat{\tau}_{2SLS})}^{-1}\hat{\tau}_{2SLS}/p_{z}$, where $p_{z}$ is
the number of IVs and $\hat{\rm Var}_{boot}(\hat{\tau}_{2SLS})$ is the
estimated variance-covariance matrix based on a nonparametric bootstrap
procedure, in which we repeatedly sample the rows of the data matrix with
replacement. If the data have a clustered structure, we use cluster-
bootstrapping instead by sampling with replacement each cluster of data (Colin
Cameron and Miller, 2015; Esarey and Menger, 2019). We include $F_{boot}$ as a
reference to the classic $F$ and effective $F$. In Section A.2 of the SM, we
compare the five types of $F$ statistics and show that the effective $F$ and
$F$ based on bootstrapping are usually more conservative than other $F$
statistics. We also calculate the effective $F$.

Second, we replicate the original IV result using the 2SLS estimator and apply
four different procedures for inference. First, we make inferences about the
2SLS results based on analytic SEs, including robust SEs or cluster-robust SEs
(if applicable). Additionally, we use two nonparametric bootstrap procedures,
as described in Section 2, _bootstrap-c_ and _bootstrap-t_. For specifications
with only a single instrument, we also employ the $tF$ procedure proposed by
Lee et al. (2022), using 2SLS $t$-statistics and first-stage $F$-statistics
based on analytic SEs accounting for the originally specified clustering
structure. Finally, we conduct an AR procedure and record the $p$-values and
CRs.

We record the point estimates, SEs (if applicable), 95% CIs or CRs, and
$p$-values for each procedure (the point estimates fully replicate the
reported estimates in the original papers and are the same across all
procedures). In addition, we estimate a naïve OLS model by regressing the
outcome variable on the treatment and control variables, leaving out the
instrument. We calculate the ratio between the magnitudes of the 2SLS and OLS
estimates. We also record other useful information, such as the number of
observations, the number of clusters, the types of instruments, the methods
used to calculate SEs or CIs, and the rationale for each paper’s IV strategy.
Our replication yields the following three main findings.

##### Finding 1. First-stage partial $F$ statistics.

Our first finding regards the strengths of the instruments. To our surprise,
among the 70 IV designs, 12 (17%) do not report this crucial statistic despite
its key role in justifying the validity of an IV design. Among the remaining
58 studies that report $F$ statistics, 9 (16%) use classic analytic SEs, thus
not adjusting for potential heteroskedasticity or clustering structure. In
Figure 3, we plot the replicated first-stage partial $F$ statistics based on
the authors’ original model specifications and choices of variance estimators
on the x-axis against effective $F$ statistics (a) or bootstrapped $F$
statistics (b) on the y-axis. Both axes are on a logarithmic scale.101010We
use the replicated $F$ statistics instead of the reported ones because some
authors either report do not or misreport their $F$ statistics (see SM for a
comparison between the reported and replicated $F$ statistics).

Figure 3: Original vs. Effective and Bootstrapped $F$

(a) Original $F$ vs. Effective $F$ (b) Original $F$ vs. Bootstrapped $F$

Note: Circles and triangles represent applications with and without a
clustering structure, respectively. Studies that do not report $F$ statistics
are painted in red. The original $F$ statistics are the replicated $F$
statistics based on the authors’ original model specifications and choices of
variance estimators. They may differ from those reported in the papers because
of misreporting.

In the original studies, the authors used various SE estimators, such as
classic SEs, robust SEs, or cluster-robust SEs. As a result,
$F_{\texttt{Eff}}$ may be larger or smaller than the replicated ones. However,
a notable feature of Figure 3 is that when a clustering structure exists, the
replicated $F$ tend to be larger than the effective $F$ (15 studies, 21%) or
bootstrapped $F$ (53 studies, 75%). When using the effective $F$ as the
benchmark, 8 studies (11%) had $F_{\texttt{Eff}}<10$. This number increases to
12 (17%) when the bootstrapped $F$ statistics are used as the benchmark. The
median first-stage $F_{\texttt{Eff}}$ statistic is higher in experimental
studies compared to non-experimental ones (67.7 versus 53.5). It is well known
that failing to cluster the SEs at the appropriate level or using the analytic
cluster-robust SE with too few clusters can lead to a severe overstatement of
statistical significance (Cameron, Gelbach and Miller, 2008). However, this
problem has received less attention when evaluating the strength of
instruments using $F$ statistics.

##### Finding 2. Inference.

Next, we compare the reported and replicated $p$-values for the null
hypothesis of no effect. For studies that do not report a $p$-value, we
calculate it based on a standard normal distribution. The replicated
$p$-values are based on (1) the _bootstrap-c_ method, (2) the _bootstrap-t_
method, and (3) the AR procedure. Since we can exactly replicate the point
estimates for the papers in the replication sample, the differences in
$p$-values are the result of the inferential methods used. Figure 4 plots
reported and replicated $p$-values, from which we observed two patterns.
First, most of the reported $p$-values are smaller than 0.05 or 0.10, the
conventional thresholds for statistical significance. Second, consistent with
Young (2022)’s finding, our replicated $p$-values based on the AR procedure or
bootstrap methods are usually bigger than the reported $p$-value, which are
primarily based on $t$ statistics calculated using analytic SEs.

Figure 4: $P$-Values for 2SLS Estimates: Original vs. Replicated

(a) Bootstrap-c method (b) Bootstrap-t method

(c) Anderson-Rubin

Note: All tests are against the null that $\tau=0$. Both axes are on a
squared-root scale. Original $p$-values are either adapted from the original
papers when they are reported or calculated based on a standard-normal
approximation of the sampling distribution of the $z$-scores when they are
not. The solid circles represent Arias and Stasavage (2019), in which the
authors use the IV strategy to argue for a null effect. _Bootstrap-c_ and _-t_
represent percentile methods based on the distributions of 2SLS estimates and
$t$-statistics, respectively, using the authors’ original model
specifications. The hollow triangles in panel (c) indicate unbounded 95%
confidence intervals from the Anderson-Rubin test using the inversion method.
The look of these plots is inspired by Figure 3 in Young (2022).

Using the AR test, we cannot reject the null hypothesis of no effect at the 5%
level in 13 studies (19%), compared with 7 (10%) in the original studies. The
number increases to 20 (29%) and 15 (21%) when we use $p$-values from the
_bootstrap-c_ and _-t_ methods. Note that very few papers we review utilize
inferential procedures specifically designed for weak instruments, such as the
AR test (2 papers), the conditional likelihood-ratio test (Moreira, 2003) (1
paper), and confident sets (Mikusheva and Poi, 2006) (none).

We also apply the $tF$-$cF$ procedure to 54 studies that use single IVs using
$F_{\texttt{Eff}}$ statistics and $t$ statistics based on robust or cluster-
robust SEs. Figures 5 shows that 16 studies (30%) are not statistically
significant at the 5% level, and 5 studies deemed statistically significant
when using the conventional fixed critical values for the $t$-test become
statistically insignificant using the $tF$-$cF$ procedure, indicating that
overly optimistic critical values due to weak instruments also contribute to
overestimation of statistical power, but not as the primary factor. These
results suggest that both weak instruments and non-i.i.d. errors have
contributed to severe overstatements of power in IV studies in political
science.

Figure 5: Statistical Significance for 54 Single Instrument Studies: $c(F)$
Threshold

Note: The green and red dots represent the studies that remain statistically
significant at the 5% level using the $tF$ procedure and those that do not,
respectively. We use the effective $F$ statistics and $t$ statistics based on
robust or cluster-robust SEs in the procedure. This plot is inspired by Figure
3 in Lee et al. (2022).

##### Finding 3. 2SLS-OLS discrepancy.

Finally, we investigate the relationship between the 2SLS estimates and naïve
OLS estimates. In Figure 6(a), we plot the 2SLS coefficients against the OLS
coefficients, both normalized using reported OLS SEs. The shaded area
indicates the range beyond which the OLS estimates are statistically
significant at the 5% level. It shows that for most studies in our sample, the
2SLS estimates and OLS estimates share the same direction and that the
magnitudes of the former are often much larger than those of the latter.
Figure 6(b) plots the distribution of the ratio between the 2SLS and OLS
estimates (in absolute terms). The mean and median of the absolute ratios are
12.4 and 3.4, respectively. In fact, in all but two designs (97%), the 2SLS
estimates are bigger than the OLS estimates, consistent with Jiang (2017)’s
finding based on finance research. While it is theoretically possible for most
OLS estimates in our sample to be biased towards zero, only 21% of the studies
have researchers expressing their belief in downward biases of the OLS
estimates. Meanwhile, 40% of the studies consider the OLS results to be their
main findings. The fact that researchers use IV designs as robustness checks
for OLS estimates due to concerns of upward biases seems at odds with the
significantly larger magnitudes of the 2SLS estimates.

Figure 6: Relationship between OLS and 2SLS Estimates

(a) Scatterplot (b) Ratio Histogram

Note: Reported 2SLS and OLS coefficient estimates are used in both plots. The
left panel uses reported OLS SE estimates for normalization; the gray band
indicates the $[-1.96,1.96]$ interval.

We further explore whether the 2SLS-OLS discrepancy is related to IV strength,
measured by $\hat{\rho}(d,\hat{d})$, the estimated correlation coefficient
between the treatment and predicted treatment. We find a strong negative
correlation between $\left|\frac{\hat{\tau}_{2SLS}}{\hat{\tau}_{OLS}}\right|$
and $|\hat{\rho}(d,\hat{d})|$ among studies using non-experimental instruments
(grey dots in Figure 7(a). The adjusted $R^{2}$ is $0.268$, with $pm=0.000$.
However, the relationship is much weaker among studies using experiment-
generated instruments (red dots). The adjusted $R^{2}$ is $-0.014$ with
$p=0.378$. In Figure 7(b), we limit our focus to the subsample in which the
OLS estimates are statistically significant at the 5% level and researchers
accept them as (part of) the main findings, and the strong negative
correlation remains.

Figure 7: IV Strength and the 2SLS-OLS Discrepancy

(a) Full Sample (b) Subsample with Statistically Significant OLS Results

Note: In the left panel, gray and red circles represent observational and
experimental studies, respectively. In the right panel, we highlight studies
in which the OLS results are claimed to be (part of) the main findings.

This indicates that even when researchers do not show explicit concerns about
their OLS results (and often use IV strategies as robustness checks), the
2SLS-OLS discrepancy is still related to IV strength. At first glance, this
result may seem mechanical: as the correlation between $d$ and $\hat{d}$
increases, the 2SLS estimates naturally converge to the OLS estimates.
However, the properties of the 2SLS estimator under the identifying
assumptions do not predict the negative relationship (we confirm it in
simulations in the SM), and such a relationship is not found in experimental
studies.

We believe a multitude of factors contribute to this pattern, including (1)
the failure of the exogeneity assumption (Assumption 2), (2) publication bias
(3) treatment effect heterogeneity (HTE), and (4) measurement error in $d$. We
suspect the first two factors are the main driving forces. As previously
mentioned, when Assumption 2 is violated, weak instruments amplify the biases
from endogenous IVs or exclusion restriction failures, i.e.,
$\frac{\text{Bias}_{IV}}{\text{Bias}_{OLS}}=\frac{\text{Cov}(z,\varepsilon)}{\text{Cov}(z,d)\text{Cov}(d,\varepsilon)}\gg
1$. Second, publication bias may also play a role. When the first stage is
weak, IV estimates have a larger variance and can be very large or very small
in magnitude compared to OLS estimates. If researchers selectively report
statistically significant results or journals tend to publish papers with
statistically significant findings, we may observe a negative relationship as
in Figure 7. This phenomenon is also referred to as Type-M bias in the
psychology and sociology literature (Gelman and Carlin, 2014; Felton and
Stewart, 2022).

Third, 30% of the replicated studies in our sample mention HTE as a possible
explanation for this discrepancy. OLS and 2SLS place different weights on
covariate strata in the sample, and therefore if compliers, those whose
treatment status is affected by the instrument, are more responsive to the
treatment than the rest of the units in the sample, we might see diverging OLS
and 2SLS estimates. Under the assumption that the exclusion restriction holds,
this gap can be decomposed into covariate weight difference, treatment-level
weight difference, and endogeneity bias components using the procedure
developed in (Ishimaru, 2021). In the SM, we investigate this possibility and
find that it is highly unlikely that HTE _alone_ can explain the difference in
magnitudes between 2SLS and OLS estimates we observe in the replication data,
i.e., the variance in treatment effects needed for this gap is implausibly
large.

Finally, an IV design can correct for the downward bias of the measurement
error in $d$, resulting in $|\hat{\tau}_{2SLS}/\hat{\tau}_{OLS}|>1$. If the
measurement error is large, this can weaken the relationship between $z$ and
$d$, producing a negative correlation. However, it is worth noting that almost
no papers in our sample attribute the IV strategy to measurement error; the
negative correlation remains even when the OLS estimates are the main findings
(indicating measurement error may not be as concerning for researchers).

Table 3: Summary of Replication Results (%) | APSR (15) | AJPS (25) | JOP (30) | All (70)
---|---|---|---|---
_Panel A: First-Stage $F$ Statistic_ |  |  |  |
Unreported | 0.0 | 20.0 | 23.3 | 17.1
Large discrepancy | 20.0 | 25.0 | 30.4 | 25.9
Effective $F<10$ | 13.3 | 12.0 | 10.0 | 11.4
Bootstrapped $F<10$ | 13.3 | 20.0 | 16.7 | 17.1
_Panel B: Inference for IV Designs_ |  |  |  |
Original $p>0.05$ | 20.0 | 8.0 | 6.7 | 10.0
AR $p>0.05$ | 13.3 | 24.0 | 16.7 | 18.6
Bootstrap-c $p>0.05$ | 20.0 | 32.0 | 30.0 | 28.6
Bootstrap-t $p>0.05$ | 26.7 | 24.0 | 16.7 | 21.4
$tF$-$cF$ procedure $p>0.05$ | 38.5 | 23.6 | 29.2 | 29.6
_Panel C: 2SLS-OLS Relationship_ |  |  |  |
$sign(\hat{\tau}_{2SLS})=sign(\hat{\tau}_{OLS})$ | 93.3 | 100.0 | 86.7 | 92.9
$\quad\ |\hat{\tau}_{2SLS}/\hat{\tau}_{OLS}|>1$ | 93.3 | 100.0 | 96.7 | 97.1
$\quad\ |\hat{\tau}_{2SLS}/\hat{\tau}_{OLS}|>3$ | 53.3 | 44.0 | 60.0 | 52.9
$\quad\ |\hat{\tau}_{2SLS}/\hat{\tau}_{OLS}|>5$ | 40.0 | 32.0 | 33.3 | 34.3
$\quad\ |\hat{\tau}_{2SLS}/\hat{\tau}_{OLS}|>10$ | 13.3 | 16.0 | 20.0 | 17.1
Note: “Large Discrepancy” in Panel A is defined as $F_{\texttt{Eff}}$ being at
least 30% larger than the reported first-stage $F$ statistics.

We summarize the main findings from our replication exercise in Table 3. The
three issues we have identified are observed in all three journals included in
the study. Based on these results, we believe that a significant portion of
the IV results either lack credibility or do not provide new information
beyond what is already provided by OLS regressions.

## 5\. Recommendations

In this paper, we replicate 70 IV designs in 67 studies published in three top
journals in political science that employ IVs as one of the main
identification strategies. We make two main contributions to the literature.
First, we show that many political science studies overestimate the strength
of the IVs and underestimate the uncertainty of the 2SLS estimates.
Specifically, we find that close to 19-30% of designs are underpowered
according to current best practice, and underestimation of uncertainties seem
to be related to non-i.i.d. error processes in data, such as
heteroscedasticity and clustering structures. Second, we observe significant
disparities between 2SLS and OLS estimates, sometimes reaching magnitudes of
tens or hundreds. We believe that this is primarily driven by a combination of
weak instruments and failure of exogeneity, as well as publication bias,
although other mechanisms such as HTE and measurement error may also
contribute. Our findings suggest that 2SLS estimates in many existing
political science studies, especially observational ones, likely have larger
biases than naïve OLS estimates.

It is important to emphasize that IVs in experimental and observational
studies are fundamentally different. When researchers use IVs to account for
non-compliance in randomized experiments, their unconfoundedness is guaranteed
by design. While the exclusion restriction may fail, researchers can guard
against this at the design stage, for example, by testing potential design
effects through randomization (Gerber and Green, 2012, pp. 140-141). Moreover,
as transparency-enhancing practices, such as power analysis, placebo tests,
and preregistration, become more common in experimental studies (see, for
example, McDermott 2014), they will likely reduce the risk of improper use of
IV designs.

Our findings suggest that using an IV strategy in an observational setting is
much more challenging. Since unconfoundedness is not guaranteed by design,
researchers have a greater burden of proof for the validity of IVs. On the one
hand, truly random (and strong) instruments are rare; on the other hand, it is
difficult to conduct placebo tests, such as the ZFS test, for the exclusion
restriction after data collection. Additionally, researchers often cannot
easily increase the sample size to obtain sufficient statistical power. To
prevent misusing IVs in observational studies, we provide a checklist for
researchers to consider when applying or considering applying an IV strategy
with observational data (in the case of one endogenous treatment variable):

  * Design

  * •

Prior to using an IV strategy, consider how selection bias may be affecting
treatment effect estimates obtained through OLS. If the main concern is
underestimating an already statistically significant treatment effect, an IV
strategy may not be necessary. Very few studies currently do so.

  * •

During the research design phase, consider whether the chosen instrument can
realistically create random or quasi-random variations in treatment assignment
while remaining excluded from the outcome equation.

  * Characterizing the first-stage

  * •

Calculate and report the effective $F$ statistic for the first stage, taking
into account heteroscedasticity and clustering structure as needed.

  * •

If $d$ and $z$ are continuous, plot $d$ against its predicted values $\hat{d}$
(with covariates and fixed effects already partialled out from both) and
visually verify whether their relationship aligns with theoretical
expectations.

  * Hypothesis testing and inference

  * •

Option 1. $t$-test after pretesting based on $F$. Use the effective $F$ for
pretesting; consider more conservative inferential methods such as _bootstrap-
t_ and _bootstrap-c_ procedures.

  * •

Option 2. $tF$-$cF$. For cases with one treatment and one instrument, consider
adjusting the critical values for the $t$-test based on the instrument’s
strength.

  * •

Option 3. direct testing. Use direct, weak-instrument-robust procedures, such
as the AR test.

  * •

Make sure to report estimation results from the first stage and the reduced
form, including 95% CIs of the coefficients, as they are informative of both
the strength of the instruments and statistical power.

  * Additional diagnostics

  * •

If you expect the OLS results to be upward biased, be concerned if the 2SLS
estimator yields much larger estimates.

  * •

If there is good reason to believe that treatment effects on compliers are
significantly larger in magnitude than those on non-compliers, explain this
through profiling of these principal strata (Abadie, 2003; Marbach and
Hangartner, 2020).

  * •

If it is possible to identify an observational analogue of “never takers” or a
subset of them, conduct a placebo test by estimating the effect of the
instrument on the outcome of interest in this ZFS sample. Using results from
the ZFS test, obtain LTZ IV estimates and CIs and compare them to the original
estimates and CIs. See the SM for a detailed example.

We develop an R package, ivDiag, to systematically implement the recommended
procedures. We hope that our recommendations, as well as the software routine,
will address concerns about using IVs in social science research and improve
the quality of estimation and inference, particularly when the IVs are not
generated through experiments.

## References

  * (1)
  * Abadie (2003) Abadie, Alberto. 2003.  “Semiparametric Instrumental Variable Estimation of Treatment Response Models.” Journal of Econometrics 113(2):231–263.
  * Acharya, Blackwell and Sen (2016) Acharya, Avidit, Matthew Blackwell and Maya Sen. 2016.  “The political Legacy Of American Slavery.” The Journal of Politics 78(3):621–641.
  * Alt, Marshall and Lassen (2016) Alt, James E., John Marshall and David D. Lassen. 2016.  “Credible Sources And Sophisticated Voters: When Does New Information Induce Economic Voting?” The Journal of Politics 78(2):327–342.  Publisher: University of Chicago Press Chicago, IL.
  * Anderson and Rubin (1949) Anderson, Theodore W and Herman Rubin. 1949.  “Estimation of the parameters of a single equation in a complete system of stochastic equations.” The Annals of mathematical statistics 20(1):46–63.
  * Anderson, Kunitomo and Sawa (1982) Anderson, Theodore W, Naoto Kunitomo and Takamitsu Sawa. 1982.  “Evaluation Of The Distribution Function Of The Limited Information Maximum Likelihood Estimator.” Econometrica: Journal of the Econometric Society pp. 1009–1027.
  * Andrews and Guggenberger (2009) Andrews, Donald WK and Patrik Guggenberger. 2009.  “Validity Of Subsampling And” Plug-in Asymptotic” Inference For Parameters Defined By Moment Inequalities.” Econometric Theory pp. 669–709.
  * Andrews, Stock and Sun (2019) Andrews, Isaiah, James Stock and Liyang Sun. 2019.  “Weak Instruments In Instrumental Variables Regression: Theory And Practice.” Annual Review of Economics 11:727–753.
  * Angrist, Imbens and Rubin (1996) Angrist, Joshua D, Guido W Imbens and Donald B Rubin. 1996.  “Identification Of Causal Effects Using Instrumental Variables.” Journal of the American statistical Association 91(434):444–455.
  * Angrist and Pischke (2008) Angrist, Joshua D and Jörn-Steffen Pischke. 2008.  Mostly Harmless Econometrics.  Princeton university press.
  * Arellano (2002) Arellano, Manuel. 2002.  “Sargan’s Intrumental Variables Estimation And The Generalized Method Of Moments.” Journal of Business & Economic Statistics 20(4):450–459.
  * Arias and Stasavage (2019) Arias, Eric and David Stasavage. 2019.  “How Large Are The Political Costs Of Fiscal Austerity?” The Journal of Politics 81(4):1517–1522.
  * Baccini and Weymouth (2021) Baccini, Leonardo and Stephen Weymouth. 2021.  “Gone for good: Deindustrialization, white voter backlash, and US presidential voting.” American Political Science Review 115(2):550–567.
  * Barth, Finseraas and Moene (2015) Barth, Erling, Henning Finseraas and Karl O. Moene. 2015.  “Political Reinforcement: How Rising Inequality Curbs Manifested Welfare Generosity.” American Journal of Political Science 59(3):565–577.  Publisher: Wiley Online Library.
  * Bhavnani and Lee (2018) Bhavnani, Rikhil R. and Alexander Lee. 2018.  “Local Embeddedness And Bureaucratic Performance: Evidence From India.” The Journal of Politics 80(1):71–87.  Publisher: University of Chicago Press Chicago, IL.
  * Blair, Di Salvatore and Smidt (2022) Blair, Robert A, Jessica Di Salvatore and Hannah M Smidt. 2022.  “When do UN peacekeeping operations implement their mandates?” American Journal of Political Science 66(3):664–680.
  * Blattman, Hartman and Blair (2014) Blattman, Christopher, Alexandra C. Hartman and Robert A. Blair. 2014.  “How To Promote Order And Property Rights Unde rWeak Rule Of Law? An Experiment In Changing Dispute Resolution Behavior Through Community Education.” American Political Science Review p. 100–120.  Publisher: JSTOR.
  * Bound and Jaeger (2000) Bound, John and David A Jaeger. 2000.  “Do Compulsory School Attendance Laws Alone Explain The Association Between Quarter Of Birth And Earnings?” Research in labor economics 19(4):83–108.
  * Bound, Jaeger and Baker (1995) Bound, John, David A Jaeger and Regina M Baker. 1995.  “Problems With Instrumental Variables Estimation When The Correlation Between The Instruments And The Endogenous Explanatory VariableIs Weak.” Journal of the American statistical association 90(430):443–450.
  * Bun and Windmeijer (2010) Bun, Maurice JG and Frank Windmeijer. 2010.  “The Weak Instrument Problem Of The System GMM Estimator In Dynamic Panel Data Models.” The Econometrics Journal 13(1):95–126.
  * Cameron, Gelbach and Miller (2008) Cameron, A Colin, Jonah B Gelbach and Douglas L Miller. 2008.  “Bootstrap-based Improvements For Inference With Clustered Errors.” The Review of Economics and Statistics 90(3):414–427.
  * Carnegie and Marinov (2017a) Carnegie, Allison and Nikolay Marinov. 2017a.  “Foreign Aid, Human Rights, And Democracy Promotion: Evidence From A Natural Experiment.” American Journal of Political Science 61(3):671–683.
  * Carnegie and Marinov (2017b) Carnegie, Allison and Nikolay Marinov. 2017b.  “Foreign Aid, Human Rights, And Democracy Promotion: Evidence From A Natural Experiment.” American Journal of Political Science 61(3):671–683.  Publisher: Wiley Online Library.
  * Charron et al. (2017) Charron, Nicholas, Carl Dahlström, Mihaly Fazekas and Victor Lapuente. 2017.  “Careers, Connections, And Corruption Risks: Investigating The Impact Of Bureaucratic Meritocracy On Public Procurement Processes.” The Journal of Politics 79(1):89–104.  Publisher: University of Chicago Press Chicago, IL.
  * Charron and Lapuente (2013) Charron, Nicholas and Victor Lapuente. 2013.  “Why Do Some Regions In Europe Have A Higher Quality Of Government?” The Journal of Politics 75(3):567–582.  Publisher: Cambridge University Press New York, USA.
  * Chernozhukov and Hansen (2008) Chernozhukov, Victor and Christian Hansen. 2008.  “The reduced form: A simple approach to inference with weak instruments.” Economics Letters 100(1):68–71.
  * Chong et al. (2019) Chong, Alberto, Gianmarco León-Ciliotta, Vivian Roza, Martín Valdivia and Gabriela Vega. 2019.  “Urbanization Patterns, Information Diffusion, And Female Voting In Rural Paraguay.” American Journal of Political Science 63(2):323–341.  Publisher: Wiley Online Library.
  * Cirone and Van Coppenolle (2018) Cirone, Alexandra and Brenda Van Coppenolle. 2018.  “Cabinets, Committees, And Careers: The Causal Effect Of Committee Service.” The Journal of Politics 80(3):948–963.  Publisher: University of Chicago Press Chicago, IL.
  * Clarke (2014) Clarke, Damian. 2014.  “PLAUSEXOG: Stata Module To Implement Conley Et Al’s Plausibly Exogenous bounds.” Statistical Software Components, Boston College Department of Economics.
https://ideas.repec.org/c/boc/bocode/s457832.html

  * Colantone and Stanig (2018a) Colantone, Italo and Piero Stanig. 2018a.  “Global Competition And Brexit.” American political science review 112(2):201–218.
  * Colantone and Stanig (2018b) Colantone, Italo and Piero Stanig. 2018b.  “The Trade Origins Of Economic Nationalism: Import Competition And Voting Behavior In Western Europe.” American Journal of Political Science 62(4):936–953.  Publisher: Wiley Online Library.
  * Colin Cameron and Miller (2015) Colin Cameron, A and Douglas L Miller. 2015.  “A Practitioner’s Guide To Cluster-robust Inference.” The Journal of Human Resources 50(2):317–372.
  * Conley, Hansen and Rossi (2012) Conley, Timothy G, Christian B Hansen and Peter E Rossi. 2012.  “Plausibly Exogenous.” The review of economics and statistics 94(1):260–272.
  * Coppock and Green (2016) Coppock, Alexander and Donald P. Green. 2016.  “Is Voting Habit Forming? New Evidence From Experiments And Regression Discontinuities.” American Journal of Political Science 60(4):1044–1062.  Publisher: Wiley Online Library.
  * Croke et al. (2016) Croke, Kevin, Guy Grossman, Horacio A. Larreguy and John Marshall. 2016.  “Deliberate Disengagement: How Education Can Decrease Political Participation In Electoral Authoritarian Regimes.” American Political Science Review 110(3):579–600.  Publisher: Cambridge University Press.
  * Davidson and MacKinnon (2015) Davidson, Russell and James G MacKinnon. 2015.  “Bootstrap Tests For Overidentification In Linear Regression Models.” Econometrics 3(4):825–863.
  * De La O (2013) De La O, Ana L. 2013.  “Do Conditional Cash Transfers Affect Electoral Behavior? Evidence From A Randomized Experiment In Mexico.” American Journal of Political Science 57(1):1–14.  Publisher: Wiley Online Library.
  * Dieterle and Snell (2016) Dieterle, Steven G and Andy Snell. 2016.  “A Simple Diagnostic To Investigate Instrument Validity And Heterogeneous Effects When Using A Single Instrument.” Labour Economics 42:76–86.
  * Dietrich and Wright (2015) Dietrich, Simone and Joseph Wright. 2015.  “Foreign Aid Allocation Tactics And Democratic Change In Africa.” The Journal of Politics 77(1):216–234.  Publisher: University of Chicago Press Chicago, IL.
  * DiGiuseppe and Shea (2022) DiGiuseppe, Matthew and Patrick E Shea. 2022.  “Us patronage, state capacity, and civil conflict.” The Journal of Politics 84(2):767–782.
  * Dinas (2014) Dinas, Elias. 2014.  “Does Choice Bring Loyalty? Electora lParticipation And The Development Of Party Identification.” American Journal of Political Science 58(2):449–465.
  * Dorsch and Maarek (2019a) Dorsch, Michael T. and Paul Maarek. 2019a.  “Democratization And The Conditional Dynamics Of Income Distribution.” American Political Science Review 113(2):385–404.  Publisher: Cambridge University Press.
  * Dorsch and Maarek (2019b) Dorsch, Michael T. and Paul Maarek. 2019b.  “Democratization And The Conditional Dynamics Of Income Distribution.” American Political Science Review 113(2):385–404.
  * Dower et al. (2018a) Dower, Paul Castaneda, Evgeny Finkel, Scott Gehlbach and Steven Nafziger. 2018a.  “Collective Action And Representation In Autocracies: Evidence From RussiaExtquoterights Great Reforms.” American Political Science Review 112(1):125–147.  Publisher: Cambridge University Press.
  * Dower et al. (2018b) Dower, Paul Castañeda, Evgeny Finkel, Scott Gehlbach and Steven Nafziger. 2018b.  “Collective Action And Representation In Autocracies: Evidence From Russia’s Great Reforms.” American Political Science Review 112(1):125–147.
  * Dube and Naidu (2015) Dube, Oeindrila and Suresh Naidu. 2015.  “Bases, Bullets, And Ballots: The Effect Of Us Military Aid On Political Conflict In Colombia.” The Journal of Politics 77(1):249–267.  Publisher: University of Chicago Press Chicago, IL.
  * Eggers, Tuñón and Dafoe (2021) Eggers, Andrew C, Guadalupe Tuñón and Allan Dafoe. 2021.  “Placebo Tests For Causal Inference.” Mimeo, University of Chicago.
  * Esarey and Menger (2019) Esarey, J and A Menger. 2019.  “Practical And Effective Approaches To Dealing With Clustered Data.” Political Science Research and Methods 7(3):541–559.
  * Feigenbaum and Hall (2015) Feigenbaum, James J. and Andrew B. Hall. 2015.  “How Legislators Respond To Localized Economic Shocks: Evidence From Chinese Import Competition.” The Journal of Politics 77(4):1012–1030.  Publisher: University of Chicago Press Chicago, IL.
  * Felton and Stewart (2022) Felton, Chris and Brandon M. Stewart. 2022.  “Handle With Care: A Sociologist’s Guide To Causal Inference With Instrumental Variables.” Mimeo, Princeton University.
  * Fieller (1954) Fieller, Edgar C. 1954.  “Some Problems In Interval Estimation.” Journal of the Royal Statistical Society: Series B (Methodological) 16(2):175–185.
  * Flores-Macias and Kreps (2013) Flores-Macias, Gustavo A. and Sarah E. Kreps. 2013.  “The Foreign Policy Consequences Of Trade: ChinaExtquoterights Commercial Relations With Africa And Latin America,1992Extendash2006.” The Journal of Politics 75(2):357–371.  Publisher: Cambridge University Press New York, USA.
  * Gehlbach and Keefer (2012) Gehlbach, Scott and Philip Keefer. 2012.  “Private Investment And The Institutionalization Of Collective Action In Autocracies: Ruling Parties And Legislatures.” The Journal of Politics 74(2):621–635.  Publisher: Cambridge University Press New York, USA.
  * Gelman and Carlin (2014) Gelman, Andrew and John Carlin. 2014.  “Beyond Power Calculations: Assessing Type S (sign) And Type M (magnitude) Errors.” Perspectives on Psychological Science 9(6):641–651.
  * Gerber and Green (2012) Gerber, Alan S and Donald P Green. 2012.  Field Experiments: Design, Analysis And Interpretation.  New York: W. W. Northon.
  * Gerber, Huber and Washington (2010) Gerber, Alan S., Gregory A. Huber and Ebonya Washington. 2010\.  “Party Affiliation, Partisanship, And Political Beliefs: A Field Experiment.” American Political Science Review 104(4):720–744.  Publisher: Cambridge University Press.
  * Goldsmith-Pinkham, Sorkin and Swift (2020) Goldsmith-Pinkham, Paul, Isaac Sorkin and Henry Swift. 2020.  “Bartik Instruments: What, When, Why, and How.” American Economic Review 110(8):2586–2624.
  * Goldstein and You (2017) Goldstein, Rebecca and Hye Young You. 2017.  “Cities As Lobbyists.” American Journal of Political Science 61(4):864–876.  Publisher: Wiley Online Library.
  * Greene (2003) Greene, William H. 2003.  Econometric Analysis.  Pearson Education India.
  * Grossman, Pierskalla and Boswell Dean (2017a) Grossman, Guy, Jan H. Pierskalla and Emma Boswell Dean. 2017a.  “Government Fragmentation And Public Goods Provision.” The Journal of Politics 79(3):823–840.
  * Grossman, Pierskalla and Boswell Dean (2017b) Grossman, Guy, Jan H. Pierskalla and Emma Boswell Dean. 2017b.  “Government Fragmentation And Public Goods Provision.” The Journal of Politics 79(3):823–840.  Publisher: University of Chicago Press Chicago, IL.
  * Guiso, Sapienza and Zingales (2016) Guiso, Luigi, Paola Sapienza and Luigi Zingales. 2016.  “Long-term Persistence.” Journal of the European Economic Association 14(6):1401–1436.
  * Hager and Hilbig (2019) Hager, Anselm and Hanno Hilbig. 2019.  “Do Inheritance Customs Affect Political And Social Inequality?” American Journal of Political Science 63(4):758–773.  Publisher: Wiley Online Library.
  * Hager and Krakowski (2022) Hager, Anselm and Krzysztof Krakowski. 2022.  “Does state repression spark protests? evidence from secret police surveillance in communist poland.” American Political Science Review 116(2):564–579.
  * Hager, Krakowski and Schaub (2019) Hager, Anselm, Krzysztof Krakowski and Max Schaub. 2019.  “Ethnic Riots And Prosocial Behavior: Evidence From Kyrgyzstan.” American Political Science Review 113(4):1029–1044.
  * Hahn and Liao (2021) Hahn, Jinyong and Zhipeng Liao. 2021.  “Bootstrap Standard Error Estimates And Inference.” Econometrica: journal of the Econometric Society 89(4):1963–1977.
  * Hainmueller, Mummolo and Xu (2019) Hainmueller, Jens, Jonathan Mummolo and Yiqing Xu. 2019.  “How Much Should We Trust Estimates From Multiplicative Interaction Models? Simple Tools To Improve Empirical Practice.” Political Analysis .
  * Hall and Horowitz (1996) Hall, Peter and Joel L Horowitz. 1996.  “Bootstrap critical values for tests based on generalized-method-of-moments estimators.” Econometrica: Journal of the Econometric Society pp. 891–916.
  * Hansen (1982) Hansen, Lars Peter. 1982.  “Large Sample Properties Of Generalized Method Of Moments Estimators.” Econometrica: Journal of the Econometric Society pp. 1029–1054.
  * Healy and Malhotra (2013) Healy, Andrew and Neil Malhotra. 2013.  “Childhood Socialization And Political Attitudes: Evidence From A Natural Experiment.” The Journal of Politics 75(4):1023–1037.  Publisher: Cambridge University Press New York, USA.
  * Henderson and Brooks (2016) Henderson, John and John Brooks. 2016.  “Mediating The Electoral Connection: The Information Effects Of Voter Signals On Legislative Behavior.” The Journal of Politics 78(3):653–669.
  * Hong, Park and Yang (2022) Hong, Ji Yeon, Sunkyoung Park and Hyunjoo Yang. 2022.  “In Strongman We Trust: The Political Legacy of the New Village Movement in South Korea.” American Journal of Political Science .
  * Ishimaru (2021) Ishimaru, Shoya. 2021.  “Empirical Decomposition Of The IV-OLS Gap With Heterogeneous And Nonlinear Effects.”.
http://arxiv.org/abs/2101.04346

  * Jiang (2017) Jiang, Wei. 2017.  “Have Instrumental Variables Brought Us Closer To The Truth.” The Review of Corporate Finance Studies 6(2):127–140.
  * Johns and Pelc (2016) Johns, Leslie and Krzysztof J. Pelc. 2016.  “Fear Of Crowds In World Trade Organization Disputes: Why DonExtquoterightt More Countries Participate?” The Journal of Politics 78(1):88–104.  Publisher: University of Chicago Press Chicago, IL.
  * Kang et al. (2020) Kang, Hyunseung, Yang Jiang, Qingyuan Zhao and Dylan S Small. 2020.  “IV Model: An R Package For Inference And Sensitivity Analysis Of Instrumental Variables Models With One Endogenous Variable.” arXiv preprint arXiv:2002.08457 .
  * Kapoor and Magesan (2018) Kapoor, Sacha and Arvind Magesan. 2018.  “Independent Candidates And Political Representation In India.” American Political Science Review 112(3):678–697.  Publisher: Cambridge University Press.
  * Key (2016) Key, Ellen M. 2016.  “How Are We Doing? Data Access And Replication In Political Science.” PS: Political Science & Politics 49(2):268–272.
  * Kim (2019) Kim, Jeong Hyun. 2019.  “Direct Democracy And Women’s Political Engagement.” American Journal of Political Science 63(3):594–610.  Publisher: Wiley Online Library.
  * Kocher, Pepinsky and Kalyvas (2011) Kocher, Matthew Adam, Thomas B. Pepinsky and Stathis N. Kalyvas. 2011.  “Aerial Bombing Snd Counterinsurgency In The Vietnam War.” American Journal of Political Science 55(2):201–218.  Publisher: Wiley Online Library.
  * Kriner and Schickler (2014) Kriner, Douglas L. and Eric Schickler. 2014.  “Investigating The President: Committee Probes And Presidential Approval,1953Extendash2006.” The Journal of Politics 76(2):521–534.  Publisher: Cambridge University Press New York, USA.
  * Kuipers and Sahn (2022) Kuipers, Nicholas and Alexander Sahn. 2022.  “The Representational Consequences of Municipal Civil Service Reform.” American Political Science Review pp. 1–17.
  * Laitin and Ramachandran (2016) Laitin, David D. and Rajesh Ramachandran. 2016.  “Language Policy And Human Development.” American Political Science Review 110(3):457–480.  Publisher: Cambridge University Press.
  * Lee et al. (2022) Lee, David S, Justin McCrary, Marcelo J Moreira and Jack Porter. 2022.  “Valid t-ratio Inference for IV.” American Economic Review 112(10):3260–90.
  * Lei and Zhou (2022) Lei, Zhenhuan and Junlong Aaron Zhou. 2022.  “Private returns to public investment: Political career incentives and infrastructure investment in China.” The Journal of Politics 84(1):455–469.
  * Lelkes, Sood and Iyengar (2017) Lelkes, Yphtach, Gaurav Sood and Shanto Iyengar. 2017.  “The Hostile Audience: The Effect Of Access To Broadband Internet On Partisan Affect.” American Journal of Political Science 61(1):5–20.  Publisher: Wiley Online Library.
  * Leonardi, Nanetti and Putnam (2001) Leonardi, Robert, Raffaella Y Nanetti and Robert D Putnam. 2001.  Making Democracy Work: Civic Traditions In Modern Italy.  Princeton university press Princeton, NJ.
  * Lerman, Sadin and Trachtman (2017) Lerman, Amy E., Meredith L. Sadin and Samuel Trachtman. 2017\.  “Policy Uptake As Political Behavior: Evidence From The Affordable Care Act.” The American Political Science Review 111(4):755.  Publisher: Cambridge University Press.
  * López-Moctezuma et al. (2022) López-Moctezuma, Gabriel, Leonard Wantchekon, Daniel Rubenson, Thomas Fujiwara and Cecilia Pe Lero. 2022.  “Policy Deliberation and Voter Persuasion: Experimental Evidence from an Election in the Philippines.” American Journal of Political Science 66(1):59–74.
  * Lorentzen, Landry and Yasuda (2014a) Lorentzen, Peter, Pierre Landry and John Yasuda. 2014a.  “Undermining Authoritarian Innovation: The Power Of China’s Industrial Giants.” The Journal of Politics 76(1):182–194.
  * Lorentzen, Landry and Yasuda (2014b) Lorentzen, Peter, Pierre Landry and John Yasuda. 2014b.  “Undermining Authoritarian Innovation: The Power Of China’s Industrial Giants.” The Journal of Politics 76(1):182–194.  Publisher: Cambridge University Press New York, USA.
  * Marbach and Hangartner (2020) Marbach, Moritz and Dominik Hangartner. 2020.  “Profiling Compliers And Noncompliers For Instrumental-Variable Analysis.” Political analysis: an annual publication of the Methodology Section of the American Political Science Association 28(3):435–444.
  * McClendon (2014) McClendon, Gwyneth H. 2014.  “Social Esteem And Participation In Contentious Politics: A Field Experiment At An LGBT Pride Rally.” American Journal of Political Science 58(2):279–290.  Publisher: Wiley Online Library.
  * McDermott (2014) McDermott, Rose. 2014.  “Research Transparency And Data Archiving For Experiments.” PS: Political Science & Politics 47(1):67–71.
  * Mellon (2020) Mellon, Jonathan. 2020.  “Rain, Rain, GoAway: 137 Potential Exclusion-restriction Violations For Studies Using Weather As An Instrumental Variable.” Available at SSRN .
  * Meredith (2013) Meredith, Marc. 2013.  “Exploiting Friends-and-neighbors To Estimate Coattail Effects.” American Political Science Review p. 742–765.  Publisher: JSTOR.
  * Mikusheva and Poi (2006) Mikusheva, Anna and Brian P Poi. 2006.  “Tests And Confidence Sets With Correct Size When Instruments Are Potentially Weak.” The Stata journal 6(3):335–347.
  * Moreira (2003) Moreira, Marcelo J. 2003.  “A Conditional Likelihood Ratio Test For Structural Models.” Econometrica: journal of the Econometric Society 71(4):1027–1048.
  * Nellis and Siddiqui (2018) Nellis, Gareth and Niloufer Siddiqui. 2018.  “Secular Party Rule And Religious Violence In Pakistan.” The American Political Science Review 112(1):49.  Publisher: Cambridge University Press.
  * Nelson and Starz (1990) Nelson, Charles and Richard Starz. 1990.  “Some Further Results On The Exact Small Sample Properties Of The Instrumental Variables Estimator.” Econometrica 58(41):967–976.
  * Nunn (2008) Nunn, Nathan. 2008.  “The Long-term Effects Of Africa’s Slave Trades.” The Quarterly Journal of Economics 123(1):139–176.
  * Nunn and Wantchekon (2011) Nunn, Nathan and Leonard Wantchekon. 2011.  “The Slave Trade And The Origins Of Mistrust In Africa.” American Economic Review 101(7):3221–52.
  * Olea and Pflueger (2013) Olea, José Luis Montiel and Carolin Pflueger. 2013.  “A Robust Test For Weak Instruments.” Journal of business & economic statistics: a publication of the American Statistical Association 31(3):358–369.
  * Pianzola et al. (2019) Pianzola, Joëlle, Alexander H. Trechsel, Kristjan Vassil, Guido Schwerdt and R. Michael Alvarez. 2019.  “The Impact Of Personalized Information On Vote Intention: Evidence From A Randomized Field Experiment.” The Journal of Politics 81(3):833–847.  Publisher: The University of Chicago Press Chicago, IL.
  * Ritter and Conrad (2016) Ritter, Emily Hencken and Courtenay R. Conrad. 2016.  “Preventing And Responding To Dissent: The Observational Challenges Of Explaining Strategic Repression.” American Political Science Review 110(1):85–99.  Publisher: Cambridge University Press.
  * Rueda (2017) Rueda, Miguel R. 2017.  “Small Aggregates, Big Manipulation: Vote Buying Enforcement And Collective Monitoring.” American Journal of Political Science 61(1):163–177.  Publisher: Wiley Online Library.
  * Schleiter and Tavits (2016) Schleiter, Petra and Margit Tavits. 2016.  “The Electoral Benefits Of Opportunistic Election Timing.” The Journal of Politics 78(3):836–850.  Publisher: University of Chicago Press Chicago, IL.
  * Schubiger (2021) Schubiger, Livia Isabella. 2021.  “State violence and wartime civilian agency: Evidence from Peru.” The Journal of Politics 83(4):1383–1398.
  * Sekhon and Titiunik (2012) Sekhon, Jasjeet S and Rocio Titiunik. 2012.  “When Natural Experiments Are Neither Natural Nor Experiments.” American Political Science Review pp. 35–57.
  * Sexton, Wellhausen and Findley (2019) Sexton, Renard, Rachel L. Wellhausen and Michael G. Findley. 2019.  “How Government Reactions To Violence Worsen Social Welfare: Evidence From Peru.” American Journal of Political Science 63(2):353–367.  Publisher: Wiley Online Library.
  * Sovey and Green (2011) Sovey, Allison J and Donald P Green. 2011.  “Instrumental Variables Estimation In Political Science :A Readers’ Guide.” American Journal of Political Science 55(1):188–200.
  * Spenkuch and Tillmann (2018a) Spenkuch, Jorg L. and Philipp Tillmann. 2018a.  “Elite Influence? Religion And The Electoral Success Of The Nazis.” American Journal of Political Science 62(1):19–36.
  * Spenkuch and Tillmann (2018b) Spenkuch, Jörg L. and Philipp Tillmann. 2018b.  “Elite Influence? Religion And The Electoral Success Of The Nazis.” American Journal of Political Science 62(1):19–36.  Publisher: Wiley Online Library.
  * Staiger and Stock (1997) Staiger, Douglas and James H Stock. 1997.  “Instrumental Variables Regression With Weak Instruments.” Econometrica: journal of the Econometric Society 65(3):557–586.
  * Stewart and Liou (2017) Stewart, Megan A. and Yu-Ming Liou. 2017.  “Do Good Borders Make Good Rebels? Territorial Control And Civilian Casualties.” The Journal of Politics 79(1):284–301.  Publisher: University of Chicago Press Chicago, IL.
  * Stock and Yogo (2005) Stock, James and Motohiro Yogo. 2005.  “Asymptotic Distributions Of Instrumental Variables Statistics With Many Instruments.” Identification and inference for econometric models: Essays in honor of Thomas Rothenberg pp. 109–120.
  * Stokes (2016) Stokes, Leah C. 2016.  “Electoral Backlash Against Climate Policy: A Natural Experiment On Retrospective Voting And Local Resistance To Public Policy.” American Journal of Political Science 60(4):958–974.  Publisher: Wiley Online Library.
  * Tajima (2013) Tajima, Yuhki. 2013.  “The Institutional Basis Of Intercommunal Order: Evidence From Indonesia’s Democratic Transition.” American Journal of Political Science 57(1):104–119.  Publisher: Wiley Online Library.
  * Trounstine (2016) Trounstine, Jessica. 2016.  “Segregation And Inequality In Public Goods.” American Journal of Political Science 60(3):709–725.  Publisher: Wiley Online Library.
  * Urpelainen and Zhang (2022) Urpelainen, Johannes and Alice Tianbo Zhang. 2022.  “Electoral Backlash or Positive Reinforcement? Wind Power and Congressional Elections in the United States.” The Journal of Politics 84(3):1306–1321.
  * Van Kippersluis and Rietveld (2018) Van Kippersluis, Hans and Cornelius A Rietveld. 2018.  “Beyond Plausibly Exogenous.” The econometrics journal 21(3):316–331.
https://academic.oup.com/ectj/article/21/3/316/5145983

  * Vernby (2013) Vernby, Kare. 2013.  “Inclusion And Public Policy: Evidence From Sweden’s Introduction Of Noncitizen Suffrage.” American Journal of Political Science 57(1):15–29.
  * Webster, Connors and Sinclair (2022) Webster, Steven W, Elizabeth C Connors and Betsy Sinclair. 2022.  “The social consequences of political anger.” The Journal of Politics 84(3):1292–1305.
  * West (2017) West, Emily A. 2017.  “Descriptive Representation And Political Efficacy: Evidence From Obama And Clinton.” The Journal of Politics 79(1):351–355.  Publisher: University of Chicago Press Chicago, IL.
  * Wood and Grose (2022) Wood, Abby K and Christian R Grose. 2022.  “Campaign finance transparency affects legislators’ election outcomes and behavior.” American Journal of Political Science 66(2):516–534.
  * Young (2022) Young, Alwyn. 2022.  “Consistency Without Inference: Instrumental Variables In Practical Application.” European Economic Review 147\.
  * Zhu (2017) Zhu, Boliang. 2017.  “MNCs, Rents, And Corruption: Evidence From China.” American Journal of Political Science 61(1):84–99.
  * Ziaja (2020) Ziaja, Sebastian. 2020.  “More Donors, More Democracy.” The Journal of Politics 82(2):433–447.  Publisher: The University of Chicago Press Chicago, IL.

##  Appendix A Supplementary Materials – Appendix A

Table of Contents

  1. A.1.

Additional Replication Results

    1. A.1.1.

Replication Sample

    2. A.1.2.

Comparison of Multiple $F$ Statistics

  2. A.2.

Monte Carlo Evidence

    1. A.2.1.

Comparing $F$ Tests for Detecting Weak Instruments

    2. A.2.2.

Explaining the 2SLS-OLS Discrepancy

  3. A.3.

Evaluating IV Validity

    1. A.3.1

The ZFS Test and Modified Inference

    2. A.3.2.

Simulation Evidence

    3. A.3.3.

A Case Study

  4. A.4.

Summary of Replicated Papers

###  A.1 Additional Information on the Replication Sample

####  A.1.1 Replication Sample

Figure A1 plots the histograms of effective $F$ statistics using experiment-
generated IVs (dark gray) and non-experimental IVs (light gray). The median
effective $F$ for experimental and observational designs are 67.7 and 53.5,
respectively.

Figure A1: Histogram of Effective $F$ Statistic

Figure A2 shows the overtime change in the percentage of experimental studies
(a), the median effective $F$ statistics (b), and the median ratio between
2SLS and OLS coefficients (c) in the replication sample.

Figure A2: Additional Information on the Sample

(a) Percentage of experimental studies

(b) Effective $F$ statistics (median)

(c) Ratio between 2SLS and OLS coefficients

####  A.1.2 Comparison of Multiple $F$ Statistics

Figure A3 compares the reported and replicated first-stage partial $F$
statistics (for studies that have reported the $F$ statistics). The replicated
$F$ statistics are based on the authors’ chosen model specifications and
variance estimators in 2SLS estimation. The discrepancy arises from the fact
that some authors report the first-stage $F$ statistic based on a different
variance estimator than the one used in the 2SLS estimation. In the paper, we
use the replicated ones to maintain consistency.

Figure A3: Reported vs. Replicated $F$ Statistics

Note: Circles and triangles represent applications with and without a
clustering structure, respectively. Studies that do not report $F$ statistics
are not shown.

In Figure A4, we compare the traditional $F$ statistics (based on classic
analytic SEs), the Huber White robust $F$ statistics, the effective $F$
statistics (robust or cluster-bootstrap SEs) and (cluster-)bootstrapped $F$
statistics. It shows that (cluster-)bootstrapped $F$ statistics are usually
the most conservative (smallest). Circles and triangles represent applications
with and without a clustering structure, respectively.

Figure A4: Comparison of Different $F$ Statistics

(a) Traditional $F$ vs. H.W. Robust $F$ (b) H.W. Robust $F$ vs. Effective $F$
(c) Effective $F$ vs. Bootstrapped $F$ (d) Traditional $F$ vs. Bootstrapped
$F$

Note: Circles and triangles represent applications with and without a
clustering structure, respectively.

###  A.2 Monte Carlo Evidence

####  A.2.1 Comparing $F$ Tests for Detecting Weak Instruments

We conduct a simulation study with a clustered DGP in order to evaluate the
relative performance of analytic and bootstrap $F$ tests to detect weak
instruments. We simulate data from the following DGP

$\displaystyle\text{clustered instrument and error components
}\;\nu_{j},\eta_{j}$ $\displaystyle\sim\mathcal{N}\left(0,0.5\right)$
$\displaystyle\text{instrument }\;z_{i}$
$\displaystyle\sim\mathcal{N}\left(0,1\right)+\nu_{j}$
$\displaystyle\text{error }\;\varepsilon_{i}$
$\displaystyle\sim\mathcal{N}\left(0,1\right)+\eta_{j}$
$\displaystyle\text{endogenous variable }\;x_{i}$ $\displaystyle=\pi
z_{i}+\varepsilon_{i}$

with errors and instrument components drawn from $J$ clusters. This DGP
ensures that the data has dependent structure within each cluster $j$. We then
evaluate the strength of the instrument analytically by computing the
t-statistic for $H_{0}:\pi=0$, or by using the corresponding bootstrap
statistic $\frac{\pi^{2}}{\widehat{\sigma}^{2}}$ where $\widehat{\sigma}^{2}$
is the bootstrap estimate of the variance of $\pi$. We evaluate the analytic
and bootstrap $F$ statistics for various values of $\pi$ and $J$ for 100
replications of the above DGP in Figure (A5).

Figure A5: Comparisons of $F$ Statistics

(a) Cluster-bootstrap $F$ statistic vs. Huber-White (non-clustered) $F$
statistic (b) Cluster-bootstrap $F$ statistic vs. cluster-robust analytic $F$
statistic ($F_{\texttt{Eff}}$)

As seen in panel A, when robust analytic standard errors ignore the clustered
structure, they vastly over-estimate the strength of the instrument relative
to the block-bootstrap, with both “few” (10) and “many” (50) clusters and with
“strong” ($\pi=0.5$) and “weak” ($\pi=0.001$) instruments. With appropriate
clustered analytic SEs, however, the $F$ statistic is typically comparable to
the bootstrap based equivalent (panel B), although the bootstrap F is
marginally more conservative with a small number of clusters and weak
instrument.

In summary, we find that cluster-bootstrap $F$ statistic and the cluster-
robust F statistic, which is equivalent to the “effective” $F$ (Olea and
Pflueger, 2013) in just-identified settings such as this one, are comparable
in detecting weak instruments, and recommend reporting these statistics in
applied settings. We also recommend reporting Anderson-Rubin confidence
intervals for the IV coefficient, as it is robust to arbitrarily weak
instruments (Andrews, Stock and Sun, 2019; Kang et al., 2020).

####  A.2.2 Explaining the 2SLS-OLS Discrepancy

In this section, we conduct Monte Carlo exercises to explore potential causes
of the discrepancy between 2SLS and OLS estimates observed in the replication
data. We consider three causes: (1) violations of the exclusion restriction
(A2), (2) publication bias, and (3) heterogeneous treatment effects (HTE).
Below is our data-generating process (DGP):

$\displaystyle y_{i}$ $\displaystyle=5+\beta_{i}x_{i}+\mu z_{i}+u_{i}+b_{i}$
$\displaystyle x_{i}^{*}$
$\displaystyle=\delta_{i}z_{i}+(1-\delta_{i})a_{i}+0.2v_{i}\quad\text{and}\quad\delta_{i}=\max(\min(\kappa_{i}\pi_{i},1),0)$
$\displaystyle x_{i}$ $\displaystyle=x_{i}^{*},\
z_{i}\overset{i.i.d.}{\sim}N(0,2)\qquad\text{(continuous-continuous case)}$
$\displaystyle\text{or}\qquad x_{i}$ $\displaystyle=1\\{x_{i}^{*}>0\\},\
z_{i}\overset{i.i.d.}{\sim}\text{Bern}(0.5)\qquad\text{(binary-binary case)}$

in which $z$ is the instrument, $x$ is the treatment, and $y$ is the outcome.
We consider two scenarios: (1) both $x$ and $z$ are continuous, and (2) both
are binary. Correlated errors $\begin{bmatrix}u_{i}\\\
v_{i}\end{bmatrix}\overset{i.i.d.}{\sim}N\left(\begin{bmatrix}0\\\
0\end{bmatrix},\begin{bmatrix}1&0.5\\\ 0.5&1\end{bmatrix}\right)$;
$a_{i}\overset{i.i.d.}{\sim}N(0,1)$, $b_{i}\overset{i.i.d.}{\sim}N(0,1)$ are
i.i.d. errors. We use $\kappa$ to control the strength of the instrument. HTE
can be generated by $\begin{bmatrix}\beta_{i}\\\
\pi_{i}\end{bmatrix}\overset{i.i.d.}{\sim}N\left(\begin{bmatrix}2\\\
1\end{bmatrix},\sigma_{h}^{2}\begin{bmatrix}1&\lambda\\\
\lambda&0.5\end{bmatrix}\right)$, in which $\sigma_{h}$ controls the amount of
heterogeneity in $\beta_{i}$ and $\pi_{i}$ while $\lambda$ controls the
correlation between the first stage and reduced form coefficients.
$\delta_{i}$ is limited to be in $[0,1]$. When $\lambda>0$, it means that a
unit’s treatment effect is positively correlated with its responsiveness to
the IV.A1A1A1For example, under selection-on-gains type settings, which are
typically considered in generalized Roy models underlying MTE approaches to
IV. The sample size is fixed at 200.

Under constant treatment effect ($\sigma_{h}=0$) and with a valid instrument
($\mu=0$), the expected value of $\hat{\beta}_{2SLS}/\hat{\beta}_{OLS}$ is
$0.74$ for the continuous-continuous case and $0.57$ for the binary-binary
case. We consider four scenarios sequentially:

  1. 1.

Violations of Assumption 2 are captured by $\mu\neq 0$ (failure of the
exclusion restriction).

  2. 2.

Publication bias can be simulated by dropping the cases in which the 2SLS
estimates are statistically insignificant at the 5% using a conventional $t$
test.

  3. 3.

HTE is generated by setting $\sigma_{h}=0.05$ and $\lambda=0.7$, i.e.,
$\beta_{i}$ and $\pi_{i}$ are highly correlated.

  4. 4.

The combination of HTE and publication bias.

Figure A6: Consequences of Exclusion Restriction Failure
under Constant Effect

(a) Continuous-Continuous Case: w/o and w/ exclusion restriction failure

(b) Binary-Binary Case: w/o and w/ exclusion restriction failure

##### Violating Assumption 2.

The results for Scenario 1 are shown in Figure A6. Each dot represents one
simulated sample. Figure A6 shows that, in both continuous-continuous and
binary-binary setups, when the treatment effect is constant
($\beta_{i}=\beta,\pi_{i}=\pi$), in expectation, there is no mechanical
negative relationship between the correlation coefficient between $d$ and
$\hat{d}$ and the 2SLS-OLS discrepancy (left panels in both subfigures).
However, when the exclusion restriction fails, e.g., $\mu=1$(right panels in
both subfigures), a strong negative correlation appears. These results support
our argument in the paper that a weak first stage amplifies the bias from the
failure of Assumption 2.

Figure A7: Consequences of Publication Bias
under Constant Treatment Effect

(a) Continuous-Continuous Case: w/o and w/ publication bias

(b) Binary-Binary Case: w/o and w/ publication bias

##### Publication bias.

Figure A7 illustrates the consequences of publication bias (Scenario 2), where
statistically insignificant results are omitted (right panels). The left
panels are identical to the left panels in Figure A6. In the binary-binary
case, we observe a moderate negative correlation; however, this correlation is
much weaker than those caused by exclusion restriction failures.

##### HTE and publication bias.

Finally, we investigate the consequences of HTE (Scenario 3) and its
interaction with publication bias (Scenario 4). Figures A8 shows results under
HTE, i.e., $\sigma_{h}>0$ and $\lambda=0.7$ ($\beta_{i}$ and $\pi_{i}$ are
highly positively correlated). On the logarithmic scale, the correlation is
almost nonexistent (left panels in Figure A8).

Figure A8: Consequences of Publication Bias under HTE

(a) Continuous-Continuous Case: w/o and w/ publication bias

(b) Binary-Binary Case: w/o and w/ publication bias

When we revert to the original scale, we do observe a small to moderate
negative correlation in both continuous-continuous and binary-binary cases
(figures not shown). When we further introduce publication bias, we begin to
see weak negative correlations between the first stage $\rho$ and the 2SLS-OLS
discrepancy on the logarithmic scale, especially in the binary-binary case.
However, their magnitudes are much smaller than what we observed in Figure A6
under the exclusion restriction failure. This suggests that the observed
strong negative relationship in the paper is unlikely to be solely explained
by HTE and different levels of responsiveness to the IV.

In summary, the Monte Carlo exercises demonstrated that the strong negative
correlations between the first stage $\rho$ and the 2SLS-OLS discrepancy are
most likely caused by violations of Assumption 2. Other factors, such as
publication bias and HTE, may also play a role.

###  A.3 Evaluating the Exogeneity Assumption

Assumption 2 is a strong and generally untestable assumption that underlies
the validity of the instrument; indeed, researchers typically spend
considerable effort arguing for both unconfoundedness and the exclusion
restrictions in their particular setting. However, some placebo tests have
recently become popular as a way to argue for the validity of identification
assumptions in causal designs (Eggers, Tuñón and Dafoe, 2021), especially in
observational settings where the choice of IV is guided by detailed domain
knowledge. Bound and Jaeger (2000) suggest first using an auxiliary regression
on a subsample where the IV is not expected to influence treatment assignment,
known as “zero-first-stage” (ZFS) tests. The primary intuition is that in a
subsample that one has a strong prior that the first stage is zero—hence, they
are “never takers,” to use the language of the LATE framework—the reduced form
effect should also be zero if Assumption 2 is satisfied. In other words,
motivated by a substantive prior that the first-stage effect of the IV is
likely zero for a subsample of the population (henceforth, the “ZFS
subsample”), the researcher then proceeds to show that the reduced-form
coefficient for the IV (by regression $Y$ on $Z$) is approximately zero _in
the ZFS subsample_ , which is suggestive evidence in favor of IV validity.
Most observational instruments ought to yield some ZFS subsample based on
substantive knowledge of the assignment mechanism.

This style of placebo is particularly popular in studies of historical
political economy, where particular historical or geographic features are
argued to be valid instruments for treatment assignment, and thus they are
unlikely to be driving treatment assignment outside a specific context. For
example, Nunn (2008) studies the effects of the slave trade on modern-day
development in Africa using sailing distance from each country to the nearest
locations of demand for slave labor as an IV for the normalized number of
slaves taken. The author then argues that distance to demand locations in the
New World are likely to be a valid IV by using a placebo test that the first-
stage effect (the IV regressed on the outcome, modern-day GDP) is
approximately zero for countries outside Africa, where the posited mechanism
(that places close to demand locations exported more slaves only in the
transatlantic slave trade) has no traction, thereby providing a candidate ZFS
sample. In a related paper, Nunn and Wantchekon (2011) use the same strategy
to show that distance to slave-trade ports does not predict modern-day trust
attitudes in the Asiabarometer, while they do in the Afrobarometer (which is
the primary study population). Acharya, Blackwell and Sen (2016) perform a
similar exercise where they believe that their instrument (cotton suitability)
predicts the treatment (slaves per capita) in the Southern States but not the
Northern states, and therefore find that the reduced form effect of cotton
suitability on modern day racial attitudes is approximately zero in the
Northern states.

####  A.3.1 The ZFS Test and Modified Inference

While this is a useful heuristic check that we advise most observational IV
papers adopt, it is an informal test and provides no debiasing procedure to
correct potentially biased IV estimates. Van Kippersluis and Rietveld (2018)
suggest that the ZFS test can be fruitfully combined with the “plausibly
exogenous” method suggested by Conley, Hansen and Rossi (2012) (henceforth,
CHR 2012). To illustrate the method, we first rewrite the IV simultaneous
equations in CHR (2012)’s notation:

$Y=X\beta+Z\gamma+\varepsilon;\quad X=Z\Pi+\nu,$ (A1)

where $Z$ also enters the structural equation, and the exclusion restriction
amounts to a dogmatic prior that $\gamma=0$. CHR (2012) suggest that this
assumption can be relaxed, and replaced with a user-specified assumption on a
plausible value, range, or distribution for $\gamma$ depending on the
researcher’s beliefs regarding the degree of exclusion restriction violation.
They propose three different approaches for inference that involve specifying
the range of values for $\gamma$, a prior distributional assumption for
$\gamma$, and a fully Bayesian analysis that requires priors over all model
parameters and corresponding parametric distributions. We focus on the second
method, which CHR (2012) call the “local to zero” (LTZ) approximation because
of its simplicity and transparency. The LTZ approximation considers “local”
violations of the exclusion restrictionA2A2A2LTZ asymptotics consider a
sequence of constants $\gamma=C/\sqrt{N}$ for some constant $C$ and sample
size $N$ and requires a prior over $\gamma$ alone. CHR (2012) show that
replacing the standard assumption that $\gamma=0$ with the weaker assumption
that $\gamma\sim\mathbb{F}$, a prior distribution, implies distribution for
$\widehat{\beta}$ in Equation (A2).

$\displaystyle\widehat{\bm{\beta}}$
$\displaystyle\sim^{a}\mathcal{N}\left(\bm{\beta},\mathbb{V}_{2SLS}\right)+\mathbf{A}\gamma\;\;\text{where}\;\mathbf{A}\equiv(\mathbf{X}^{\prime}\mathbf{Z}(\mathbf{Z}^{\prime}\mathbf{Z})^{-1}\mathbf{Z}^{\prime}\mathbf{X})^{-1}\mathbf{X}^{\prime}\mathbf{Z}$
(A2) $\displaystyle\widehat{\bm{\beta}}$
$\displaystyle\sim^{a}\mathcal{N}\left(\bm{\beta}+\mathbf{A}\mu_{\gamma},\mathbb{V}_{2SLS}+\mathbf{A}\mathbf{\Omega}\mathbf{A}^{\prime}\right)$
(A3)

where the original 2SLS asymptotic distribution is inflated by the additional
term. While a simulation-based approach can be used to implement Equation (A2)
for an arbitrary distribution for $\gamma$, the distribution takes its most
convenient form when one uses a Gaussian prior over
$\gamma\sim\mathcal{N}\left(\mu_{\gamma},\Omega_{\gamma}\right)$, which
simplifies Equation (A2) to Equation (A3), with a posterior being a Gaussian
centered at $\bm{\beta}+\mathbf{A}\mu_{\gamma}$.

CHR (2012) suggest that researchers use domain knowledge to choose
$\mu_{\gamma},\Omega_{\gamma}$, since they often hold strong priors about
instruments anyway (which presumably motivates the choice of the instrument).
Van Kippersluis and Rietveld (2018) suggest that a principled method to choose
$\mu_{\gamma}$ is to estimate Equation (A1) on the ZFS population (wherein
$\Pi$ is assumed to be zero), and use this estimate $\widehat{\gamma}_{ZFS}$
as $\mu_{\gamma}$. This approach combines the informal ZFS test with the
plausibly exogenous method in a straightforward manner, and software to
implement it is available in both R (accompanying this paper) and STATA
(Clarke, 2014). We begin with a simulation-based illustration and illustrate
the application of this method to a published empirical paper next.

####  A.3.2 Simulation Evidence

In this subsection, we demonstrate the LTZ method when the exclusion
restriction is not satisfied. Consider the following DGP,

$\displaystyle Y_{i}=$ $\displaystyle\beta_{i}D_{i}+\gamma
Z_{i}+\varepsilon_{i}$ $\displaystyle D_{i}=$
$\displaystyle\mathbf{1}\\{D_{i}^{*}>0\\}$ $\displaystyle D_{i}^{*}=$
$\displaystyle\alpha_{i}+\pi_{i}Z_{i}+\varepsilon_{i}$

in which $Z_{i}\sim\text{Bernoulli}(0.5)$ is a binary instrument,
$\pi_{i}\sim\mathsf{U}\left[1.5,2.5\right]$,
$\alpha_{i}\sim\mathcal{N}\left(-1,1\right)$,
$\varepsilon_{i}\sim\mathcal{N}\left(0,1\right)$,
$\beta_{i}\sim\mathcal{N}\left(1,0.25\right)$. We generate $Y_{i}$ with
$Z_{i}$ directly entering the structural equation, which allows us to vary the
magnitude of the exclusion restriction violation. We then estimate
$\widehat{\beta}_{2SLS}$ using conventional two-staged-least-squares on this
data. As we vary $\gamma$, $\hat{\beta}_{2SLS}$ is inconsistent for all values
except when $\gamma=0$. We set $\pi=0$ for the last 20% observations of the
simulated data (the ZFS subsample). We then estimate the reduced-form
regression on this (known) subsample and use the coefficient as a prior for
$\mu_{\gamma}$, and compute the LTZ IV estimate.

Figure A9: IV and LTZ Estimates for Varying $\gamma$

Figure A9 shows, unlike the 2SLS estimator (blue), the LTZ estimator (orange)
uncovers the true value of $\beta=2$ even for large degrees of exclusion
restriction violations (large $|\gamma|$).

####  A.3.3 A Case Study

We illustrate the diagnostics described above by applying it to the IV
analysis in Guiso, Sapienza and Zingales (2016) (henceforth GSZ 2016), who
revisit Leonardi, Nanetti and Putnam (2001)’s conjecture that Italian cities
that achieved self-government in the Middle Ages have higher modern-day levels
of social capital. More specifically, they study the effects of free city-
state status on social capital as measured by the number of non-profits and
organ donations per capita, and a measure of whether students cheat in
mathematics.

Figure A10: Table 6 in Guiso, Sapienza and Zingales (2016)

Note: “Ease of coordination” is the IV “Bishop in city.” We replicated columns
(I), (II), (IV), and (V).

Table A1:  Replication of GSZ (2016) Table 6
Reduced Form Regressions | North |  | South (ZFS)
---|---|---|---
_Outcome Variables_ | Nonprofit | Organ Donation |  | Nonprofit | Organ Donation
| (1) | (2) |  | (3) | (4)
Bishop (IV) | 1.612 | 0.472 |  | 0.178 | 0.189
| (0.219) | (0.047) |  | (0.137) | (0.065)
Observations | 5,357 | 5,535 |  | 2,175 | 2,178
Note: Bootstrapped SEs are in the parentheses. See Figure A4 in the SM for the
original table.

GSZ (2016) use a dummy for whether the city was the seat of a bishop in the
Middle Ages, based on historical accounts of coordination preceding commune
formation in the Middle Ages as an IV for the “free-city experience” (Section
5). They argue that conditional on a host of geographic covariates, this IV, a
bishop seat, influences contemporary social capital solely through its
increasing the likelihood of commune formation. As suggestive evidence for the
validity of their instrument, they estimate the reduced-form effect of
medieval bishop presence of contemporary social capital measures separately in
the north (where the IV is conjectured to have an effect) and the south (where
it is conjectured to be irrelevant). They fail to reject the null of no
effects in the south, conclude that the IV appears to have face validity, and
proceed to use bishop presence as an IV for their IV estimates.

Figure A11: IV Coefficients for Non-profits and Organ Donation

We begin by calculating the first-stage partial $F$ statistic based on
bootstrapped SEs for the north sample, which is 67.3. Because there were no
“free cities” in the south, the $F$ statistic for the south is zero by
definition. We then replicate their reduced-form estimates in Table A1. The
separate north and south reduced-form estimates in GSZ (2016) can be readily
used for the LTZ test described above. The authors substantively believe that
the south is a ZFS sample where bishop presence is irrelevant for treatment
assignment,A3A3A3The authors claim this indirectly by reporting the reduced
form effects separately for the north and south subsamples in Table 6, and
state that since the reduced form is attenuated in the south, this justifies
the use of bishop presence as an IV (p. 1427). we can use the reduced-form
estimates of 0.178 and 0.189 in the south for non-profits per capita and organ
donation (columns 3-4 in Table A1) as the prior $\mu_{\gamma}$ for the direct
effect of the IV on the outcome. Finally, we report the analytic, bootstrap,
and LTZ IV results in Figure A11. We find that conventional robust SEs
understate the uncertainty of the estimates relative to the bootstrap and that
accounting for direct effect using LTZ attenuates GSZ (2016)’s estimates
somewhat and substantially increases the SE of the estimate for the non-profit
outcome. For organ donation, however, where we suspect a violation of
Assumption 2 because the reduced form effect is statistically distinguishable
from zero, the use of the LTZ method to account for this exclusion restriction
violation yields a smaller and substantially more uncertain estimate whose CI
contains 0. This example shows how researchers may take advantage of the ZFS
test and the LTZ technique to gauge the robustness of their findings based on
an IV strategy.

###  A.4 Summary of Replicated Papers

Table A2: Summary of Replicated Papers |  |  |  |  |
---|---|---|---|---|---
Paper | Instrument | Treatment | Outcome | IV Type | Justification for IV Validity
APSR
Gerber, Huber and Washington (2010) |  Being sent mail |  Aligning party identification with latent partisanship |  Voting and party alignment scale |  Experiment |  NA
Meredith (2013) |  Governor’s home county |  Democratic governor |  Down-ballot Democratic candidates’ vote share |  Theory (Other) |  “The validity of the instruments hinges on the assumption that, conditional on the control variables, coattail effects are the only channel through which the place of birth or residence of a party’s gubernatorial candidate affects the vote shares received by its down-ballot candidates.”(p.745)
Blattman, Hartman and Blair (2014) |  Assignment to treatment blocks |  Mass education campaign for dispute resolution |  Serious land dispute |  Experiment |  NA
Laitin and Ramachandran (2016) |  Geographic distance from the origins of writing |  Language choice |  Human development index |  Theory (Geography) |  “[T]he distance from these sites of invention should have no independent impact on socioeconomic development today, except through the channel of affecting the probability of possessing a writing tradition.” (p. 470)
Ritter and Conrad (2016) |  Rainfall |  Mobilized dissent |  Repression |  Theory (Weather) |  “[R]ainfall is an exogenous predictor of dissent onset, meeting the key criteria for the instrumental analysis to allow for causal inference.”(p.89)
Croke et al. (2016) |  Access to the secondary education |  Education attainment |  Political participation |  Rules & policy changes (Change in exposure) |  “There are, however, good reasons to believe that the secondary education reform only affects participation through its effect on educational attainment.”(p.592)
Dower et al. (2018a) |  Level of serfdom |  Frequency of unrest |  Peasant representation and unrest |  Theory (History) |  “After conditioning on these covariates, we are left with that portion of serfdom largely determined by idiosyncratic variation in land grants to the nobility decades or centuries before the zemstvo reform of 1864.” (p. 133)
Dower et al. (2018a) |  Religious polarization |  Frequency of unrest |  Peasant representation and unrest |  Theory (History) |  “After conditioning on these covariates, we are left with that portion of serfdom largely determined by idiosyncratic variation in land grants to the nobility decades or centuries before the zemstvo reform of 1864.” (p. 133)
Nellis and Siddiqui (2018) |  Narrow victory by secular parties in a district |  The proportion of MNA seats in a district won by secularist candidates |  Religious violence |  Theory (Election) |  “Our identifying assumption is that the outcomes of such close elections are as good as randomly decided.” (p. 50)
Kapoor and Magesan (2018) |  Changes in entry costs. |  Number of independent candidates |  Voter turnout |  Rules & policy changes (Change in exposure) |  “It is worth reiterating that the deposit increases had nothing to do with historical differences in voter and candidate participation across reserved and open constituencies.” (p. 681)
Colantone and Stanig (2018a) |  Imports from China to the United States $\times$ local industrial structure |  Regional-level import shock from China |  Leave support in Brexit |  Econometrics (Interaction) |  “[The ]instrument is meant to capture the variation in Chinese imports, which is due to the exogenous changes in supply conditions in China, rather than to domestic factors in the United Kingdom that could be correlated with electoral outcomes.” (p. 206)
Hager, Krakowski and Schaub (2019) |  Distance to the nearest location where armored military vehicles were stolen |  Ethnic riots (destruction) |  Prosocial behavior |  Theory (Other) |  “[W]e present a falsification test which corroborates that the instrument is unrelated to prosocial behavior in a sample of 136 nearby villages, thus underlining the exclusion restriction.” (p. 1037)
Baccini and Weymouth (2021) |  Bartik instrument |  Manufacturing Layoffs |  Change of Democratic Vote Share |  Econometrics (Interaction) |  “Since layoffs are not randomly assigned, we develop an instrumental variables strategy using shift-share methodology (Bartik 1991) derived from national layoff shocks, weighted by initial county-level employment.”(p.550)
Hager and Krakowski (2022) |  Number of corrupted Catholic priests |  Number of secret police officers |  Resistance |  Theory (History) |  “In the early days of the regime, the secret police’s ability to servile citizens depended critically on the cooperation of the Catholic Church…Importantly, the corruptibility of priests was plausibly exogenous: priests were sent to municipalities by the Catholic Church, often when another priest had retired.” (p.565)
Kuipers and Sahn (2022) |  Statewide assignment mandate |  Civil service reform |  Descriptive representation on an unrestricted sample |  Rules & policy changes (Assignment) |  “First, we assume that state-level mandates are a strong instrument for city adoption; we verify the strength of the instrument in the main presentation of the results. The exclusion restriction, which is untestable, seems a reasonable assumption in our case.”(p.9)
|  |  |  |  |
AJPS
Kocher, Pepinsky and Kalyvas (2011) |  Past insurgent control |  Aerial bombing |  Changes in local control |  Theory (Other) |  “Because instrumental variables require only conditional independence between instruments and the error term, we need only assume that there are no unobserved hamlet-specific variables that affected insurgent control in July, August, and December 1969, but not in September of that year as well.” (p. 212)
Vernby (2013) |  Immigration Inflow 1940–1950; immigration Inflow 1960–1967 |  Share of noncitizens in the electorate |  Municipal education and social spending |  Theory (History) |  “Furthermore, it is unlikely that the initial locations of these refugees were affected by the level of local public services, suggesting that the instrument is also valid.” (p. 25)
Tajima (2013) |  Distance to health station |  Distance to police posts (as a proxy for exposure to military intervention) |  Incidence of communal violence |  Theory (Geography) |  “According to a Health Department official, primary health stations must be located in every subdistrict at their population centers, regardless of the propensity for violence of those locations” (p. 112)
De La O (2013) |  Random assignment to early coverage |  Early coverage of Conditional Cash Transfer |  Incumbent party’s vote share |  Experiment |  NA
McClendon (2014) |  Assignment to treatment |  Reading social esteem promising email |  Participation in LGBTQ events |  Experiment |  NA
Barth, Finseraas and Moene (2015) |  Adjusted bargaining coverage and effective number of union confederations |  Wage inequality |  Welfare support |  Theory (Other) |  “Yet conditional on union density and country fixed effects, we argue that certain properties of the bargaining system are likely to affect wages, but not union involvement in politics.” (p. 574)
Stokes (2016) |  Wind speed |  Turbine location |  Vote turnout |  Theory (Climate) |  “Wind speed is theoretically orthogonal to precinct boundaries but predicts the placement of wind turbine locations.” (p. 965)
Coppock and Green (2016) |  Mailing showing 2005 Vote |  Voting in November 2007 municipal elections |  Voting in the 2008 presidential primary |  Experiment |  NA
Trounstine (2016) |  The number of waterways in a city combined with logged population |  Racial segregation |  Direct general expenditures |  Theory (Geography) |  “I focus on waterways (including large streams and rivers), which vary in number across cities and are arguably exogenous to segregation and spending.” (p. 717)
Carnegie and Marinov (2017b) |  Being a former colony of one of the Council members |  Foreign aid |  CIRI Human Empowerment index |  Theory (History) |  “In 1965, the EU stipulated that countries would hold the presidency for 6 months at a time […] and would rotate alphabetically according to each member state’s name as spelled in its own language. ” (p. 676)
Zhu (2017) |  Weighted geographic distance from economic centers |  MNC activity |  Corruption |  Theory (Geography) |  “This instrumental variable (IV) is rooted in the gravity models of international trade and FDI flows.” (p. 90)
Rueda (2017) |  The size of the polling station |  Actual polling place size |  Citizens’ reports of electoral manipulation |  Rules & policy changes (Fuzzy RD) |  “The institutional rule predicts sharp reductions in the size of the average polling station of a municipality every time the number of registered voters reaches a multiple of the maximum number of voters allowed to vote in a polling station.” (p. 173)
Lelkes, Sood and Iyengar (2017) |  State-level ROW index |  Number of providers |  Affective polarization (partisan hostility) |  Theory (Other) |  “[A]n index of state regulation of right-of-way laws strongly predicts the number of providers in a county, which, as we discuss later, is a good proxy for broadband uptake.” (p. 4).
Goldstein and You (2017) |  Direct flight from city to Washington DC |  Lobbying spending |  Total earmarks or grants awarded |  Theory (Other) |  “The existence of a direct flight captures the convenience of travel to Washington, DC, from each city.” (p. 865)