LaLonde (1986) after Nearly Four Decades:
Lessons Learned
Guido Imbens Yiqing Xu
Guido Imbens, Stanford University. Email<EMAIL_ADDRESS>Yiqing Xu, Stanford University. Email<EMAIL_ADDRESS>
We thank Susan Athey, Alexis Diamond, Dean Eckles, and Scott Cunningham for comments,

the Office of Naval Research for support under grant numbers N00014-17-1-2131 and N00014-19-1-2468 and Amazon for a gift. Data and code required to replicate the analyses in this paper, as well as an online tutorial, can be found at <https://github.com/xuyiqing/lalonde>. We thank Zihan Xie and Jinwen Wu for their excellent research assistance, which makes the tutorial possible.

In 1986, Robert LaLonde published an article that compared nonexperimental estimates to experimental benchmarks <cit.>. He concluded that the nonexperimental methods at the time could not systematically replicate experimental benchmarks, casting doubt on the credibility of these methods. Following LaLonde's critical assessment, there have been significant methodological advances and practical changes, including ($i$) an emphasis on estimators based on unconfoundedness, ($ii$) a focus on the importance of overlap in covariate distributions, ($iii$) the introduction of propensity score-based methods leading to doubly robust estimators, ($iv$) a greater emphasis on validation exercises to bolster research credibility, and ($v$) methods for estimating and exploiting treatment effect heterogeneity.  To demonstrate the practical lessons from these advances, we reexamine the LaLonde data and the Imbens-Rubin-Sacerdote lottery data. We show that modern methods, when applied in contexts with sufficient covariate overlap, yield robust estimates for the adjusted differences between the treatment and control groups. However, this does not mean that these estimates are valid. To assess their credibility, validation exercises (such as placebo tests) are essential, whereas goodness of fit tests alone are inadequate. Our findings highlight the importance of closely examining the assignment process, carefully inspecting overlap, and conducting validation exercises when analyzing causal effects with nonexperimental data.

§ INTRODUCTION

In 1986, Robert LaLonde published a paper <cit.> based on part of his PhD thesis, which has had a profound impact on both methodological and empirical literature on estimating causal effects.[As of May 2024, it has been cited 2900 times, but this number does not do justice to its influence on the causal inference literature and the credibility revolution.]
In his paper, he assessed whether the then state-of-the-art nonexperimental evaluation methods could match experimental benchmarks. LaLonde's conclusion was quite negative regarding the credibility of the nonexperimental methods he examined. He wrote:

“This comparison shows that many of the econometric procedures do not replicate the experimentally determined results, and it suggests that researchers should be aware of the potential for specification errors in other nonexperimental evaluations.” <cit.>,

and eventually concluding that

“policymakers should be aware that the available nonexperimental evaluations of employment and training programs may contain large and unknown biases resulting from specification errors.” <cit.>,

A year earlier LaLonde's thesis advisors Orley Ashenfelter and David Card had raised similar concerns:

“we conclude that randomized clinical trials are
necessary to reliably determine program effects.”   <cit.>

Quickly after the publication of LaLonde's paper, several other studies echoed LaLonde's concern about the credibility of nonexperimental methods in the context of the evaluation of labor market programs <cit.>, leading to further skepticism about their value. More generally, <cit.> had earlier raised concerns about the credibility of applied econometric methods in nonexperimental evaluations, and asked “Is Randomization Essential?” <cit.>.

One important development that greatly increased the influence of the original LaLonde study was the accessibility of the data in the early 2000s, thanks to the work by Rajeev Dehejia and Sadek Wahba
<cit.>.[As part of a research project starting in a graduate class taught by Imbens and Rubin at Harvard in 1996, Dehejia and Wahba obtained the original data from LaLonde, stored on tapes. They successfully located an old tape reader capable of retrieving the data. The data are now publicly available on Dehejia's website (<https://users.nber.org/ rdehejia/data/.nswdata2.html>) and have been extensively used in the causal inference literature on causal inference.] More recently, <cit.> reconstructed the female samples used in LaLonde study. Throughout the remainder of the paper, we refer to the data compiled by Dehejia and Wahba as the LaLonde-Dehejia-Wahba (LDW) data and the reconstructed female samples as the Lalonde-Calónico-Smith (LCS) data.

In this paper, we summarize our perspective on the lessons from the methodological literature that followed LaLonde's seminal paper. We argue that while some of LaLonde's original conclusions have stood the test of time, in our view substantial progress has been made in the methodological literature both in terms of improved estimators and in the form of suggested additional analyses. At this point, almost four decades after LaLonde's paper, the answer to his original question—whether nonexperimental methods can successfully replicate experimental benchmarks—is more nuanced than his conclusions: sometimes we can, and we now have better methods both for achieving this when we can, and for telling whether we can. We stress that these views are our own, based on our reading of, and contributions to, this literature. Many issues in this literature remain subjects of ongoing debate, and  other researchers hold different opinions on these developments.

What specifically has the methodological literature since <cit.> taught us? What particular analyses would we recommend a researcher analyzing data of this type do in the light of the subsequent theoretical research? In this discussion, we focus on five issues related to the particular setting studied in <cit.>, the evaluation of an intervention at the individual level, based on detailed background information on those individuals. There is a literature on causal inference more broadly which has grown substantially over the nearly four decades as well, as documented in <cit.>. We do not cover this broader literature here. There are more comprehensive surveys of the general causal literature,  <cit.>, as well as numerous textbooks <cit.>. Here, we limit the discussion to five lessons learned from the subsequent literature for settings similar to those in <cit.>.

First, for the type of data used in LaLonde's paper, the literature has predominantly focused on methods based on the unconfoundedness (or, using other terms, exogeneity, ignorability, conditional independence, or selection-on-observables) assumption. Although <cit.>, as well as <cit.> and others, explored alternative identification strategies in their analyses, including difference-in-differences and selection models, these methods are rarely applied to the LaLonde data in the subsequent literature. No compelling case has been made that other identification strategies are credible in that context.
The lack of alternatives, of course, does not make the unconfoundedness assumption itself credible. The positive case for an analysis based on unconfoundedness is as follows: with the LaLonde data, or similar datasets, we argue that comparing treated and control untis identical or similar in terms of the full set of available pretreatment variables makes a causal interpretation more plausible than any other comparison between treated and control units. Any alternative strategy that would lead to point-identification would involve comparing treated and control units with different values for the pretreatment variables, which, in our view, makes a causal interpretation less credible. For that reason, we focus in this discussion on methods relying on unconfoundedness assumptions. For recent reviews of panel methods, see <cit.> and <cit.>.

Second, and this is perhaps the most important insight, the literature has recognized the crucial role of assessing overlap in covariate distributions and dealing directly with the lack thereof.  The various comparison groups <cit.> used to evaluate nonexperimental methods all differ substantially from the experimental sample in terms of the distributions of the covariates. This creates challenges for conventional statistical adjustment methods such as regression and matching. LaLonde attempted to address these by simply discarding individuals in the comparison groups who do not meet certain specific eligibility criteria based on age, employment status and earnings. The subsequent literature has emphasized that, in practice, lack of overlap is a key issue in such analyses. Effective and systematic, data-driven, ways of diagnosing, and addressing this lack of overlap have been developed subsequently.

Third, and somewhat related to the overlap issue, the role of the propensity score in estimation has been stressed. There are two components to this role. First, the propensity score plays an important role in uncovering and addressing the lack of overlap in covariate distributions. Second, it is important in the estimation of treatment effects, either directly through inverse propensity score weighting (IPW) or, more importantly in the current state-of-the-art approaches, as part of doubly robust methods that incorporate both models for the conditional outcome distributions and the models for the propensity score. Propensity scores played no role in LaLonde's analyses. In fact, the term “propensity score” does not appear. The paper that introduced the propensity score, <cit.>, and which by now has over 37,000 Google Scholar cites, had only recently been published at that time and had not yet influenced the econometrics literature.[The first mention of the propensity score in the econometrics literature appears to be <cit.>. Interestingly, they cite <cit.>
rather than the original propensity score paper <cit.>.] Subsequently, recognizing the importance of modeling both the assignment mechanism and the conditional outcome distribution has spurred the development of various doubly robust methods. First introduced by <cit.>, these methods are now generally viewed as the most attractive methods based on unconfoundedness in practice. The incorporation of machine learning techniques into causal inference has further enabled this by reducing the need for ad hoc  specification searches.

Fourth, scholars have come to realize the importance of establishing the credibility of estimates through validation exercises, particularly placebo analyses. <cit.> did some placebo analyses looking at the estimated effect for lagged earnings, but primarily focused on the comparisons between nonexperimental and experimental estimates of average treatment effects. The recent empirical literature has placed significant emphasis on supporting main estimates with supplementary analyses, which often take the form of placebo analyses that present estimates of causal effects known to be zero.

Fifth, effective methods have emerged for other estimands, such as conditional average treatment effects for the treated (CATT)—conditional on (possibly high-dimensional) covariates—and quantile treatment effects. In many cases, decision-makers seek to understand not just the average effects for the entire population but also the extent and nature of effect heterogeneity, or even to estimate personalized assignment rules. Do some subpopulations benefit more from the treatment than others? Do some experience negative effects? The availability of large datasets has led to the development of effective ways of estimating heterogeneous treatment effects  <cit.>.

To demonstrate the insights from these five lessons, we reexamine the LaLonde data, including the LDW data, the male samples from the original LaLonde study, and the LCS female samples, using modern methods. We show that once we ensure sufficient overlap, multiple methods based on unconfoundedness yield quite similar estimates for the statistical estimand, that is, the average adjusted difference between treatment and control groups. However, these estimates often do not match experimental benchmarks for the average treatment effect on the treated (ATT). Similarly, CATT estimates and quantile treatment effect estimates do not consistently align with experimental results. Moreover, standard validation exercises based on placebo estimates do not support the unconfoundedness assumption using the LDW or LCS data. Despite these challenges, there is room for optimism. Analyzing the data from <cit.> (hereafter, the IRS data), which benefits from a clearer assignment mechanism (lottery) and more detailed pretreatment information, we find that the placebo analyses support the unconfoundedness assumption.

The rest of this paper is organized as follows. In Section <ref>, we discuss LaLonde's original findings. In Section <ref> we discuss the methodological advances since <cit.>. Next, in Section <ref>, we revisit the LaLonde data and reanalyze the IRS data. Finally, Section <ref> concludes with recommendations to applied researchers.

§ LALONDE'S FINDINGS

In this section, we first describe the data Lalonde used and explain the main econometric specifications of the models in <cit.>. We then examine the regression estimates in his paper, followed by a discussion of the selection model estimates. After summarizing Lalonde's findings, we proceed to describe the Lalonde-Dehejia-Wahba data and explore some of the immediate responses to LaLonde's findings.

§.§ LaLonde's Data

<cit.> analyzed the training effects of the National Supported Work Demonstration (NSW) program for female and male participants separately. The female participants were drawn from the Aid to Families with Dependent Children (AFDC) program. The male participants came from three other target groups, all with relatively poor labor market prospects: ex-drug addicts, ex-criminal offenders, and high-school dropouts. For both female and male participants, LaLonde used two main data sources to construct comparison groups: CPS-SSA-1, drawn from Westat's Matched Current Population Survey–Social Security Administration File, includes all females or males under 55 meeting Westat's criteria; PSID-1, from the Panel Study of Income Dynamics, includes all female or male household heads under 55 from 1975 to 1978 for males and 1979 for females who did not identify as retired in 1975. the age cutoff was chosen to make the comparison group more comparable to the experimental sample. LaLonde further refined these datasets based on criteria like employment status, time of survey, and poverty status, creating four additional comparison groups: CPS-SSA-2, CPS-SSA-3, PSID-2, and PSID-3. Our subsequent reanalysis will primarily focus on the male samples because they are commonly used in methodological literature and because the LDW data include two pretreatment outcomes. Results based on the female samples are provided in the Supplementary Materials (SM).

Descriptive Statistics: LaLonde and LDW Male Samples

Note: Standard deviations are in the parentheses. Tables 3, 5, and 6 in <cit.> use data described in columns 1–4. <cit.> primarily use data described in columns 3–6.

Table <ref> columns 1–4 provide summary statistics for the male samples in the experimental data and comparison groups used in <cit.>. LaLonde showed that respondents in the comparison groups were, on average, older, more educated, substantially less likely to be high school dropouts, more likely to be married, and less likely to be Black or Hispanic, than participants of the NSW experiment; they also had substantially higher earnings in years before the program took place. The large difference in average earnings between the comparison groups and the experimental sample motivated LaLonde to construct the other comparison groups, using earnings-based criteria.
Note that the male CPS-SSA-1 sample, with 15,922 observations, is significantly larger than the male PSID-1 sample, which has 2,490 observations.

The experimental treated units in the LDW data are a subset of the treated units in the original LaLonde data, specifically chosen to include information on 1974 earnings. Using this subset turns out to be important as it allows analysts to adjust longer earnings histories and conduct additional placebo analyses.

§.§ Econometric Approach

To estimate the causal effect of the NSW program on 1978 earnings using experimental and nonexperimental data, <cit.> employed a variety of models, which can be broadly divided into two categories: regression methods that rely solely on an outcome model, referred to as the “earnings equation,” and selection models that include an additional “participation equation.” The error terms in both equations are assumed to have zero means, but the selection model approach allows these error terms to be correlated.

Regression estimates In Tables 4 and 5, <cit.> presented results for the training effects on female and male participants, respectively, using seven different estimators based on six comparison groups, with minor specification variations in the regression function for  earnings.
All regression models are linear and implicitly assume a constant treatment effect.  The seven models include (i) a simple regression estimator both without or with controls for age, education, and race (but, notably, not including earnings in 1975 in the set of controls); (ii) a difference-in-differences (DID) estimator using 1975 earnings as the pretreatment outcome—hence, the original outcome is replaced with a first-differenced outcome—both without or with controls for age; (iii) a quasi-DID estimator that places 1975 earnings on the right-hand side of the regression to account for transitory shocks, also known as the Ashenfelter dip <cit.>, both without or with controls for age, education, and race; and (iv) a specification that controls for all pretreatment covariates, including pre-training earnings and unemployment status in 1975, as well as marital status. The comparison groups, as described above, include CPS-SSA-1 and PSID-1, as well as subsets with different selection criteria to make them more similar to the NSW sample. Interestingly, and somewhat prescient of the emphasis these analyses would receive in contemporary literature, LaLonde also reported results from placebo tests using 1975 earnings as the placebo outcome.

LaLonde's findings based on the linear regression model estimates are several-fold. First, using the experimental NSW  data, all seven estimators produce similar training effect estimates around $851 for female participants and $886 for male participants, and the estimated placebo effects are close to zero. Second, when using nonexperimental comparison groups, the estimates diverge significantly from the experimental benchmarks, often yielding large and negative values in both female and male samples. Third, these estimates vary widely, and specification tests focusing on goodness-of-fit are unlikely to guide an analyst to the experimental benchmarks. Collectively, these findings led LaLonde to conclude that none of the regression adjustment methods popular at the time of his writing, when applied to nonexperimental data, were credible.

Selection model estimates In addition to these estimates based on exogeneity of the treatment,  <cit.> also presented results in Tables 6 that account for endogeneity of the treatment indicator. This approach uses the two-step estimator proposed by <cit.>, which allows the error terms in the earnings equation and participation equation to be correlated. Identification relies on the presence of covariates included (with non-zero coefficients) in the participation equation but excluded from the earnings equation, or on an assumption of a joint normal distribution for the error terms. For both female and male samples, LaLonde employed three comparison groups: the experimental controls, CPS-SSA-1, and PSID-1, and tested four different specifications for females and three for males. Each specification uses a distinct set of variables in the participation equation that are excluded from the earnings equation. A priori, none of the  specifications appears more justified by economic or econometric theories than any other.

LaLonde reported that all selection model estimates using the experimental data remain close to $851 for females and $886 for males. However, those based on nonexperimental data vary substantially across specifications and, as in the exogenous regression case, deviate substantially from the experimental benchmarks. He concluded that while the two-step procedure brings estimates closer to the experimental benchmarks, it still results in a “considerable range of imprecise estimates,” <cit.>.

§.§ The LaLonde-Dehejia-Wahba (LDW) Data

<cit.> focused solely on male participants, stating that “estimates for this group were the most sensitive to functional-form specification” <cit.>. They constructed a subsample from LaLonde's original data that includes participants with available 1974 earnings and unemployment status. <cit.> argued that this subsample remains a valid experimental sample because its construction relies on pretreatment information only, such as month of assignment and employment history, ensuring that treatment assignment remains orthogonal to all pretreatment variables. Notably, this subsample contains only 62% of the original treated group used by LaLonde. They also use the subsets of the same six datasets as LaLonde for nonexperimental controls, which likewise contain 1974 earnings and unemployment information. This collection of datasets, referred to as the LDW data, is now widely used in causal inference literature.[In fact, most methodological research on causal inference uses a specific sample of the LDW data, with controls from CPA-SSA-1.]

Columns 5 and 6 in Table <ref> show summary statistics for the treated observations in the LDW data. The table shows that the NSW participants in the LDW treated sample had a higher unemployment rate in 1975 and lower average earnings in 1975 compared to LaLonde's original male samples. The 1974 earnings data, available only in the LDW sample, suggest that many LDW participants faced long-term unemployment. These factors may explain why the estimated training effect in the LDW sample, $1,794, is more than double that in LaLonde's male sample, which is $886.

§.§ Subsequent Literature

The publication of <cit.> ignited a vigorous debate in the applied econometrics literature. <cit.> responded to Lalonde's criticism of the nonexperimental evaluation literature by suggesting the use of specification tests to eliminate particularly poor estimators. However, this approach falls short in differentiating among numerous estimators that adequately fit the data but are based on varying identifying assumptions. As a result, this approach has not found many followers in the subsequent literature.

<cit.> introduced propensity score-based methods (including stratification and matching using propensity scores) to address LaLonde's challenge, obtaining estimates close to the experimental benchmark. They concluded, in sharp contrast to LaLonde's findings, that

“the estimates of the training effect for LaLonde's ... dataset are close to the benchmark experimental estimates and are robust to the specification of the comparison group and to the functional form used to estimate the propensity score. ... our methods succeed for a transparent reason: They use only the subset of the comparison group that is comparable to the treatment group, and discard the complement.” <cit.>.

The contrast between the conclusions in <cit.> and those in <cit.> started an explosion of methodological work probing these conclusions. This led to the development of further nonlinear estimators, methods for accounting for overlap, and placebo methods. We discuss these developments in the next section.

§ METHODOLOGICAL IMPROVEMENTS IN ECONOMETRIC EVALUATION METHODS SINCE LALONDE

This section begins by introducing the potential outcome framework and fixing notation. We then examine two key assumptions: unconfoundedness and overlap, followed by a brief discussion of various estimation strategies applicable under these assumptions. Next, we discuss alternative estimands, such as CATT and quantile treatment effects, and the methods for estimating them. Finally, we highlight the significance of supplementary analyses, including placebo tests and sensitivity analyses, which are key to validating these key assumptions and improving research credibility.

§.§ Potential Outcome Framework

To facilitate the discussion of the modern causal inference literature, we now adopt the potential outcome model originally used by Jerzey Neyman in the context of randomized experiments <cit.>, and extended to nonexperimental studies by Donald Rubin <cit.>.  For each individual $i$, for $i=1,\ldots,N$, two potential outcomes exist: $Y_i(0)$ represents the outcome (earnings in 1978) had individual $i$ not participated in the NSW program, and $Y_i(1)$ represents the outcome for the same individual had they participated in the program. The difference between those two potential outcomes, $\tau_{i} \equiv Y_i(1)-Y_i(0)$ is the causal effect of the program for that individual. The binary treatment for individual $i$, participation in the job training program, is denoted by $W_i\in\{0,1\}$. The realized outcome is $Y_i\equiv Y_i(W_i)=(1-W_i)Y_i(0)+W_iY_i(1)$. In addition, individual $i$'s pretreatment characteristics are denoted by $X_i$.  In the LaLonde case, the basic vector of covariates includes age, years of schooling, high school dropout status, marital status, and indicators for African-American and Hispanic backgrounds. Following <cit.>'s analysis, we also consider settings where the covariate vector is augmented to include two lagged earnings variables—earnings in 1974 and 1975—and binary indicators for these lagged earnings being zero (indicating unemployment). Hence, we use $X_i$ to represent all ten pretreatment covariates.

Our primary estimand is the average treatment effect for the treated (ATT),

$$ATT \equiv  \frac{1}{N_{tr}}\sum_{i:W_i=1}\left\{Y_i(1)-Y_i(0)\right\} = \overline{Y}(1) - \overline{Y}(0),$$

where $N_{tr}$ is the number of treated units.
In other settings researchers may also be interested in the average treatment effect $ATE \equiv \frac{1}{N}\sum_{i=1}^N \left\{Y_i(1)-Y_i(0)\right\}.$ Most analyses of the LaLonde data that explicitly allow for treatment effect heterogeneity focus on the ATT, as it makes little sense to contemplate the effect of the program for individuals in the control group with long term jobs and high earnings. <cit.> does not draw a distinction between these two estimands as he generally de-emphasized effect heterogeneity.

Of course, we cannot directly estimate the ATT because we do not observe the control outcomes for the treated units, what <cit.> called the “fundamental problem of causal inference.” To make progress,  let us define the statistical estimand, the covariate adjusted difference between treated and controls,

\[
\mathbb{E}\left[\mathbb{E}[Y_i|W_i=1,X_i]
-\mathbb{E}[Y_i|W_i=0,X_i]\ |\ W_i=1
\right].
\]

This is an object we can estimate consistently given a random sample. However, it is only under two critical assumptions: unconfoundedness and overlap that this statistical estimand is equal to the causal estimand, the ATT, which is the object of interest. Part of the subsequent literature has focused on better statistical methods for estimating the covariate adjusted difference. Formal results typically require additional regularity conditions, such as the smoothness of conditional means and propensity scores, along with moment conditions. For a more formal treatment of this topic, we refer readers to the original papers, as referenced in reviews such as <cit.> and <cit.>. A separate question concerns the plausibility of the assumptions, and here the literature has made progress through the development of placebo and sensitivity analyses.

§.§ Unconfoundedness

The unconfoundedness assumption, first introduced by <cit.> as part of the “ignorable treatment assignment”  concept that also included overlap, has played a crucial role in identifying average treatment effects using nonexperimental data. It states that, conditional on the covariates, the treatment assignment is independent of the pair of potential outcomes:

\[W_i\
\indep\
\left\{(Y_i(0),Y_i(1)\right\} |\ X_i.\]

Identifying the ATT in fact only requires $W_i\ \indep\ Y_i(0)\ |\ X_i\ $, a weaker version of unconfoundedness. The unconfoundedness assumption is also referred to as “conditional independence” <cit.> or “selection on observables” <cit.>.
This assumption stands in contrast to traditional econometric  assumptions of exogeneity that were articulated in terms of residuals, themselves defined in terms of functional forms. Unconfoundedness elegantly separates the functional form part of the assumptions from their essence. Essentially, it is sufficient that researchers understand (a crucial aspect of) the “design,” or the treatment assignment mechanism, without full knowledge of the data-generating process of the potential outcomes. A key result in
<cit.>  shows that Assumption <ref> implies

\[W_i\  \indep\ \left\{(Y_i(0),Y_i(1)\right\} |\ e(X_i),\]

in which $e(X_i) \equiv \Pr(W_i = 1\ |\ X_i)$ is the propensity score for unit $i$. This result is important in guiding many estimation strategies because it reduces the dimension of the conditioning set from the dimension of $X_i$ to one, the dimension of the propensity score.

When the parametric outcome model (e.g., the earnings equation) is correctly specified, unconfoundedness implies a zero conditional mean for the error term. Thus, except for those using a DID approach, the estimates in Tables 4 and 5 of <cit.> can be interpreted as based on a combination of unconfoundedness and functional form assumptions. At the time of LaLonde's study, however, the design-based perspective had not yet gained popularity, and these specifications were motivated almost entirely from an outcome modeling perspective.

In practice, unconfoundedness is a very strong assumption. For a general discussion on this topic, see <cit.> and <cit.>. While we acknowledge concerns about its validity in the absence of a clear understanding of the treatment assignment mechanism, we believe that supplementary analyses using ancillary data and domain knowledge, such as placebo tests and sensitivity analyses, can help assess the plausibility of this assumption and, by doing so, improve the credibility of analyses based on it. We will illustrate these methods in the next section.
In the context of the LaLonde data, it is evident that all ten covariates are appropriate pretreatment variables that should be controlled for. In other cases, whether one should adjust for differences between treated and control units based on specific covariates is less clear. <cit.> cautions against adjusting for variables that are affected by the treatment. <cit.> further discuss the selection of variables to adjust for in causal analyses.

§.§ Overlap and Balance

To identify the average causal effect under unconfoundedness, we need to ensure that we can estimate the average effect at every value for the covariates, requiring overlap, or that the propensity score is between zero and one:

$$0 < \Pr(W_i = 1\ |\ X_i) < 1.$$

If the ATT is of interest, in fact only a weaker overlap assumption, $\Pr(W_i = 1\ |\ X_i) < 1$, is required. Overlap is crucial in identifying the ATT when researchers are unwilling to make functional form assumptions about the conditional means of the potential outcomes. When $X_{i}$ includes fewer than a handful of covariates, inspecting pairs of the covariates' marginal or joint distributions by treatment status may be sufficient for assessing overlap. However, this approach becomes impractical in high-dimensional settings. In such cases, a more attractive method is to inspect the distribution of the propensity scores, estimated by a flexible method, by treatment status. The lack of overlap in covariate distributions implies, and is implied by, a lack of overlap in the propensity score distributions.

<cit.> did not explicitly discuss overlap, nor did he inspect it beyond report sample averages for covariates by treatment status. Both the regression and selection model approaches he used assume correct functional forms, which allow for interpolation or extrapolation of treatment effects across all covariate levels and their combinations, thereby formally eliminating the need for overlap.
However, LaLonde was clearly concerned about the possible implications of the lack of overlap between the experimental treated group and the comparison group based on CPS and PSID data. As mentioned earlier, to improve comparability, he trimmed the original comparison groups based on “characteristics [that] are consistent with some of the eligibility criteria used to admit applicants into the NSW program” <cit.>, aligned with the goal of improving balance on covariates that determine selection. However, by modern standards, his methods, such as removing all male participants working in March 1976 (CPS1-SSA-2) or further removing unemployed respondents with 1975 incomes above the poverty line (CPS1-SSA-3), are ad hoc and do not necessarily achieve overlap in all relevant covariates.

Since <cit.>, many scholars have proposed more principled methods to improve overlap, often relying on propensity scores.
These methods take different forms, partly depending on whether simply overlap in the covariate distributions is sought, or whether, more aggressively, balance in the covariate distributions is pursued. Balance refers to settings where the covariate distributions are identical, or close to identical, in treatment and control groups. In expectation, balance is achieved by design in a completely randomized experiment, and can be further improved upon through stratification prior to the randomization.

Ensuring overlap or improving balance typically involves dropping some units from the full sample. Although in principle this leads to some loss of information, the improvement in robustness and reduction of bias may outweigh the loss in precision. In fact, the potential increase in variance from a subsantial amout of  trimming of the sample is typically modest. Suppose one has a sample with $N_{tr}$ treated units, and $N_{co}$ control units. Under homoskedasticity and random assignment, the variance of the difference in mean estimator is $\sigma^2(1/(N_{tr}+1/N_{co})$. For instance, if we start with $N_{tr}=185$ treated units and $N_{co}=15,922$ control units as in the LDW-CPS sample, dropping 15,737 control individuals to leave just 185 control individuals (a 99% reduction in the control sample) increases the standard error only by 30% in the “best-case scenario,” which assumes no bias from including the additional control individuals. In practice, concerns about bias suggest that aggressive trimming may lead to more robust and credible estimates.

Focusing on overlap alone, and with the estimand the ATT,
<cit.> drop all control individuals with a propensity score less than the smallest propensity score among the treated individuals. <cit.> develop a more aggressive approach to address the lack of overlap. They characterize subsamples optimized for precise average treatment effect estimation, with a rule of thumb suggesting trimming data with estimated propensity scores outside $[0.1, 0.9]$. <cit.> and <cit.> propose balancing covariates through propensity score weighting, introducing the “overlap weights” that are proportional to the product of the propensity score and one minus the propensity score.
A third approach, particularly well suited to settings where the focus is on the ATT, is to created a matched sample in which all treated units are matched to a distinct control unit in terms of the estimated propensity score. Beyond ensuring overlap, this method creates a sample that is much better balanced in the covariate distributions.

In practice, overlap, like unconfoundedness, is critical for obtaining credible estimates. This is particularly true in cases with poor overlap in the raw data, such as the LaLonde samples, where trimming to ensure overlap is more important than the choice of specific estimation strategies.

§.§ Estimation Given Unconfoundedness and Overlap

All estimators in <cit.> are linear in the covariates. Since then, a variety of methods have been proposed to estimate average causal effects in more flexible ways under both unconfoundedness and overlap assumptions, a combination also referred to as ignorable treatment assignment <cit.>. We divide these methods into three groups: first, outcome modeling, including linear regressions; second, methods that directly adjust for covariate imbalance, including those based on propensity scores; and third, doubly robust methods.

Outcome modeling Denote the conditional means of the two potential outcomes as
$\mu_{w}(x) \equiv\E[Y_{i}(w)|X_i = x],$ for $ w \in \{0, 1\}$. The simplest and still the most commonly used method by applied researchers is a simple linear regression using the treatment indicator and covariates (the level terms) as regressors, which resembles the earnings equation in <cit.>. The regression method models the conditional means of potential outcomes parametrically, that is, $\mu_{0}(x) = x^{\top} \beta$ is a linear function of $x$, and the treatment effect is constant: $\mu_{1}(x) = \mu_{0}(x) + \tau$.

Relaxing the functional form assumptions slightly, one can use two separate linear regressions to model $\mu_{0}(x)$ and $\mu_{1}(x)$. This estimator is sometimes referred to as the Oaxaca-Blinder estimators <cit.>. More generally, researchers can model the two conditional means using semiparametric or nonparametric approaches <cit.>.

Adjusting covariate imbalance The second group of methods focuses on directly adjusting covariate imbalance between the treatment and control groups. This includes blocking on covariates, covariate matching <cit.>, and weighting methods to achieve covariate balance <cit.>.

When the number of covariates is large, particularly with many continuous variables, simple covariate matching methods suffer from the curse of dimensionality, rendering them either infeasible or prone to large biases <cit.>. Under such circumstances, adjusting for differences in the propensity score, rather than attempting to adjust for all covariates, can be beneficial. This can be implemented through various methods, such as blocking/matching <cit.> and reweighting <cit.>. For example, one of the IPW estimators for the ATT is
$\hat\tau_{IPW} = \ \sum_{i: W_{i} = 1}Y_{i}/N_{tr} - \sum_{i: W_{i} =  0}\hat\omega_i Y_{i}$, with weights
$\hat\omega_i=\hat{e}(X_{i})/(1 - \hat{e}(X_{i}))/\sum_{j: W_{j}=0}\hat{e}(X_{j})/(1 - \hat{e}(X_{j}))$. Here, $\hat{e}(x)$ is the estimated propensity score. <cit.> show that this Hájek estimator can achieve the semiparametric efficiency bound with a nonparametric estimator for the propensity score.

There are also attempts to improve covariate balance while estimating the propensity score. For example, <cit.> propose covariate balancing propensity score estimated via the generalized method of moments using covariate balance as moment conditions. Moreover, one can interpret covariate balancing methods as IPW estimators. For example, entropy balancing proposed by <cit.> can be seen as an IPW estimator with a linear propensity score model and a logistic link <cit.>. Balancing estimators sometimes have more desirable statistical properties compared to generic IPW because they address covariate imbalance more directly.

Doubly robust methods
Neither outcome modeling nor the balancing methods are currently the most recommended methods in the methodological literature. Instead, scholars have developed various mixed methods that combine outcome modeling, such as regression, with methods addressing covariate imbalance to achieve the benefits from both. These methods include regression within propensity score blocks <cit.>, matching combined with regression <cit.>, and methods integrating weighting with regression <cit.>. The rationale for the mixed methods is that, although covariate-balancing or propensity score methods by themselves may be consistent, or even fully semiparametrically efficient, incorporating outcome models can improve small sample performance by eliminating remaining biases or improving precision by leveraging the correlation between the covariates and the outcome. For instance, while the bias of a simple matching estimator might dominate variance in high-dimensional cases, adding regression to account for the remaining imbalance can substantially reduce such biases <cit.>.

<cit.> introduce the term “double robustness,” an important concept for these mixed methods. They show that if either the propensity score or regression model is correctly specified parametrically, the augmented inverse propensity weighting (AIPW) estimator that combines weighting and regression is consistent. The AIPW estimators for the ATT can be written as:

\[
\hat{\tau}_{AIPW}  =\  \frac{1}{N_{tr}} \sum_{i: W_{i} = 1} \left( Y_i- \hat{\mu}_0(X_{i})\right) - \frac{1}{N_{tr}}\sum_{i: W_{i} = 0} \hat{\omega}_{i}\left(Y_i- \hat{\mu}_0(X_{i})\right)\ ,\]

in which $\hat{\omega}_{i}$ is the IPW (or balancing) weights. They can be viewed as combining an outcome model with an adjustment term, which consists of an IPW estimator applied to the residuals from the outcome model.

In the past few years, machine learning methods have rapidly entered the toolkit of applied researchers for estimating causal effects due to advancement in the methodological literature <cit.>. Many of these estimators adopt the form of an AIPW estimator and satisfy the “Neyman orthogonality” condition <cit.>, which ensures the stability of the moment conditions used to identify the causal parameter against small perturbations in nuisance functions, including the conditional mean $\mu_{w}(x)$ and the propensity score $e(x)$.  <cit.> show a particularly attractive feature of the double/debiased machine learning estimators based on estimating the influence function for the semiparametrically efficient estimator: they accommodate slower convergence rates for the estimators of the nuisance functions.

§.§ Alternative Estimands and Heterogeneous Treatment Effects

Much of the methodological and applied research has focused on estimating average treatment effects, such as the ATT. However, there are other quantities of interest to researchers. For example, researchers are often interested in understanding variations in the treatment effects. Understanding effect heterogeneity is crucial for discerning the mechanisms and impacts of a policy, for a more precise evaluation of policy effectiveness, and for guiding personalized policy assignments. Econometrically, researchers can study heterogeneous treatment effects by estimating the CATT, i.e., $\tau(x)\equiv \frac{1}{N_{x}}\sum_{i: X_i= x, W_i = 1} \tau_{i},$ in which $N_{x}$ is the number of treated units whose covariate values equal to $x$. Researchers have proposed to use machine learning methods to estimate CATT nonparametrically or using low-dimensional representations, such as causal forests, and obtain valid inference or error bounds <cit.>.

Another important but less commonly used group of estimands by empirical researchers are the quantile treatment effects. They are defined as the difference between the quantiles of the treated and untreated potential outcome distributions for the population or the treated group. Because Assumptions <ref> and  <ref> allow for the identification of the full marginal distribution of $Y_{i}(0)$ and $Y_{i}(1)$, quantile treatment effects are identified under those assumptions. <cit.> proposes a semiparametrically efficient estimator for these quantities, combining conditional quantile estimations with IPW. See <cit.> for an application to the LaLonde data. One potentially underexplored area in the literature is that estimates of CATT or quantile treatment effects can inform the plausibility of the unconfoundedness assumption, given that researchers often possess insights into the range of these effects.

§.§ Validation through Placebo and Sensitivity Analyses

While researchers can assess overlap using observed data, the unconfoundedness assumption is not directly testable. To evaluate the credibility of treatment effect estimates, the literature has developed two main approaches: placebo analyses and sensitivity analyses.

Placebo Analyses

Placebo analyses indirectly assess unconfoundedness by formally testing a conditional independence restriction that is testable. This testable assumption differs from unconfoundedness in two aspects. First, it conditions on a subset of the full set of covariates that appear in the unconfoundedness assumption. Second, it uses one of the remaining covariates as a pseudo-outcome that serves as a proxy for the target outcome. A common placebo test estimates the treatment effect on a pretreatment variable, known to be unaffected by the treatment. A lagged outcome is an appealing choice as it is typically a good proxy for the target outcome.[Other forms of placebo tests include estimating the effect of a pseudo-treatment on the outcome, often using multiple control groups. For further details, see <cit.>.]

<cit.> regressed 1975 earnings, which predated the program, on the treatment indicator and covariates, and reported findings in columns 2 and 3 of Tables 4 and 5. He found that most nonexperimental estimates are negative, large, and often statistically significant, indicating a potential violation of unconfoundedness. Although LaLonde did not explicitly use the term, this approach is a placebo test. One limitation of the LaLonde data is the availability of only one pretreatment outcome. With the LDW data, we can test whether 1975 earnings are correlated with participation in the job training program, conditional on 1974 earnings and other covariates. With additional pretreatment periods, as in the IRS data (where $T_{0} = 6$), researchers can construct placebo tests that are statistically more powerful and substantively more credible. A limitation of LaLonde's analyses is that he only tested one aspect of the full conditional independence assumption, i.e., whether the two conditional means, averaged over the conditioning variables, are the same. <cit.> discusses testing additional implications of the conditional independence relationship.

Sensitivity Analyses
Sensitivity analyses adopt a different approach. Rather than validating unconfoundedness with auxiliary data, they assume unconfoundedness only holds conditional on observed covariates $X_{i}$ and an unobserved confounder $U_{i}$. <cit.> suggest modeling the conditional distribution of potential outcomes and the treatment assignment probability (propensity score) given $X_{i}$ and $U_{i}$. While relationships with $X_{i}$ are estimated, the dependence on $U_{i}$ is supplied by the researcher to estimate a treatment effect. A causal relationship is considered insensitive to unobserved confounding if the estimated effect remains robust against strong dependence with $U_{i}$. <cit.> improves this approach by benchmarking the association between the unobserved  $U_{i}$ and the potential outcomes and treatment  with those estimated on observed covariates and introducing contour plots for interpretation. <cit.> further refine this method by relaxing treatment assignment functional forms and incorporating multiple confounders. Alternatively, in a series of papers (see <cit.> for references), Rosenbaum proposes using the relative odds ratio of propensity scores between treated and control units, examining the range of odds necessary to significantly alter the $p$-value in a test for a null effect.

§ REANALYZING THE LALONDE AND IRS DATA

To demonstrate the methodological advances since <cit.> in practice, we revisit the LaLonde data, including both the LDW data, the original Lalonde male samples, and the LCS female samples. Overlap is a major concern for all of these datasets. We also analyze the IRS lottery data from <cit.>, where extensive pretreatment information is available and overlap is less concerning. Throughout, we emphasize the importance of overlap and assessing unconfoundedness. We focus on the ATT for both the LaLonde and IRS datasets.

Assessing the Overlap in LDW Data

Trimmed LDW-CPS

Trimmed LDW-PSID

Note: Histograms depict the log odds ratios, i.e., $\log\frac{\hat{e}}{1-\hat{e}}$, using propensity score estimated through GRF. Each subfigure represents a different sample. Subfigure A: LDW-Experimental. Subfigure B: LDW-CPS. Subfigure C: LDW-PSID. Subfigure D: Trimmed LDW-CPS. Subfigure E: Trimmed LDW-PSID. For C and D, the propensity scores are re-estimated after trimming.

§.§ The LaLonde Data

We primarily focus on the LDW data, give their inclusion of 1974 earnings and employment status. We use three LDW datasets: (1) LDW-Experimental, which consists of 185 treated and 280 control individuals from the experimental data; (2) LDW-CPS, including the same treated individuals and 15,992 controls from CPS-SSA-1; and (3) LDW-PSID, comprising the same treated individuals and 2,490 controls from PSID-1. We do not use CPS-SSA-2 and CPS-SSA-3 controls as they are subsets of CPS-SSA-1, and similarly, we do not use PSID-2 and PSID-3 as they are part of PSID-1. LaLonde's construction of these subsamples was a relatively ad hoc approach to improving overlap. We use more modern, fully data-driven ways to detect and address issues related to overlap.

Figure <ref> (A)-(C) display the overlaps in propensity score estimated from generalized random forest <cit.> between the treated and control units for all three samples, using histograms of the log-odds of propensity scores, i.e., $\log\left(\hat{e}/(1-\hat{e})\right)$. As expected, LDW-Experimental shows almost perfect overlap. However, both nonexperimental samples show very poor overlaps. Most notably, the propensity scores of many treated units do not lie within the support of the controls' propensity scores, and a substantial proportion of the control units possess extremely low log odds. Similar patterns are observed with the original LaLonde male samples.

Trimming to improve overlap Based on LDW-CPS and LDW-PSID, we further construct two trimmed samples to improve overlap. Trimming involves two steps. First, we merge the experimental controls from LDW-Experimental into LDW-CPS (or LDW-PSID) and estimate each unit's propensity of being included in the experiment using GRF.[In other words, both experimental treated units and controls are labeled as 1 and the nonexperimental units are labeled as 0. This approach is clearly not feasible when experimental controls are unavailable; our objective is to establish experimental benchmarks for the trimmed samples. We include the full set of covariates, including real earnings and unemployment status for the years 1974 and 1975.]

Trimming the LDW Data

Note: A demonstration of the trimming procedure.

We set a threshold to trim data with estimated propensity scores exceeding this value. The thresholds for LDW-CPS and LDW-PSID, chosen based on the numbers of available controls units, are 0.9 and 0.8, respectively.[Using CPS-SSA-1 as controls, only four control units have $\hat{e} > 0.9$; using PSID-1 as controls, only nine control units have $\hat{e} > 0.8$.] The transition from Step (A) to (B) in Figure <ref> illustrates this step. We conduct this procedure to trim all three samples simultaneously to improve overlap in the final samples. This is because merely trimming the nonexperimental controls is inadequate, as the nonexperimental datasets lack particular profiles of participants of the experiment. For LDW-CPS, 22 treated units (12%) are excluded, whereas for LDW-PSID, 105 treated units (57%) are dropped. This underscores the large differences between the nonexperimental controls and the experiment participants based on covariate distributions.

Second, using the trimmed data and the same set of covariates, we re-estimate propensity scores using GRF, this time excluding the experimental controls. We then employ a 1:1 matching based on the re-estimated propensity scores to further trim the nonexperimental controls. This step is illustrated by the progression from Step (B) to (C) in Figure <ref>. This procedure yields two samples: trimmed LDW-CPS (or trimmed LDW-PSID), composed of the trimmed experimental treated units and trimmed nonexperimental controls, and another trimmed experimental sample, consisting of trimmed experimental treated units and controls. The latter serves as an experimental benchmark for the former. As shown in Figure <ref>(D)-(E), overlap improves significantly in both samples post-trimming, though this comes with the cost of reduced sample sizes. We conduct similar procedures to the LaLonde male samples.

Estimating the ATT Next, we estimate the ATT using both the original LDW nonexperimental samples and the newly constructed trimmed samples. We apply a variety of estimators, including simple difference-in-means, regression, the Oaxaca Blinder estimator, GRF as an outcome model, nearest neighbor matching with bias correction, propensity score matching, IPW with propensity scores estimated by GRF, covariate balancing propensity score (CBPS), entropy balancing, double/debiased matching learning using elastic net, and AIPW implemented via GRF. All estimators use the same set of ten covariates as before.

ATT Estimates Given Unconfoundedness: LDW Samples

Note: The above figures show the ATT estimates and their 95% confidence intervals using four different samples: LDW-CPS, LDW-PSID, Trimmed LDW-CPS, and Trimmed LDW-PSID. Eleven estimators are employed, including difference-in-means, linear regression, Oaxaca Blinder, GRF as an outcome model, 1: 5 nearest neighbor matching with bias correction, propensity score matching, IPW with propensity scores estimated by GRF, CBPS, entropy balancing, double/debiased matching learning with elastic net (implemented using ), and AIPW with GRF  (implemented using ). The red dashed line and pink band represent the experiment benchmark and its 95% confidence intervals, respectively.

We present the findings in Figure <ref>, as well as Table <ref> in the SM. The first two panels present the ATT estimates and their 95% confidence intervals from LDW-CPS and LDW-PSID, while the third and fourth panels show the results from the trimmed samples with improved overlap. In each figure, the ATT estimates using experimental data and their 95% confidence intervals are highlighted with a red dashed line and pink band, respectively.

As shown in Figure <ref>(A), when using LDW-CPS, all estimators, except difference-in-means, produce positive estimates, although there are noticeable variations among them. Nearest neighbor matching outperforms other estimators, aligning closely with the experimental benchmark of $1,794. Notably, CBPS, entropy balancing, and  AIPW-GRF also produce results close to the benchmark. Despite numerical differences, these estimates, except for difference-in-means, cannot be statistically distinguished from one another. Figure <ref>(B) shows that estimates based on LDW-PSID exhibit greater variations. Setting aside the difference-in-means, the estimates span from $4 to $2,420. Among them, the AIPW-GRF estimator produces an estimate closest to the experimental benchmark.

Figure <ref>(C) and (D) show that by using trimmed data with improved overlap, estimates produced by various estimators are substantially more stable. For trimmed LDW-CPS, all estimates hover around the experimental benchmark of $1,911. This includes the simple difference-in-means estimator, which, due to trimming, is equivalent to propensity score matching without replacement. In the case of trimmed LDW-PSID, the experimental benchmark stands at $306, and it is not statistically significantly different from zero at the 5% level. While each estimator yields results that are closely aligned, they all have a negative sign. They are statistically indistinguishable from the experimental benchmark due to the large uncertainties associated with these estimates. The stability of the range of estimators after trimming is also found in the simulations in <cit.>.

These findings suggest that while improved overlap based on observed covariates can reduce model dependency and estimate variability across different estimators, it does not guarantee consistency without validating unconfoundedness. The fact that many methods produce estimates that match the experimental benchmark for the ATT using LDW-CPS might have instilled unwarranted confidence in researchers that modern estimators can help achieve causal identification even when there are no compelling reasons to believe unconfoundedness. In other words, while modern methods may be effective in estimating the statistical estimand, the  covariate-adjusted difference between the treated and control outcomes, that does not mean that this adjusted difference is close to the ATT. For that, we need some version of the unconfoundedness assumption, which is untestable.

Alternative estimands: CATT and quantile treatment effects Exploring causal estimates for alternative estimands, such as heterogeneous treatment effects and quantile treatment effects, can shed light on the plausibility of unconfoundedness. Using both the original LDW data and the trimmed versions, we estimate the CATT using a causal forest through AIPW-GRF. In Figure <ref>, we plot the estimated CATT from nonexperimental data at the covariate values of each treated unit against their corresponding experimental benchmarks. Each of the four panels uses a different dataset. Every gray dot represents a pair of CATT estimates, while the red cross depicts the pair of estimated ATTs, also estimated via AIPW-GRF. This exercise is, in essence, similar to the test proposed by <cit.> to explore a finding's robustness to model specifications by splitting samples based on covariate values.

CATT Estimates using LDW Data: Experimental vs. Nonexperimental

LDW-CPS Trimmed

LDW-PSID Trimmed

Note: Scatterplots show the CATT using both experimental data (x-axis) and nonexperimental data (y-axis). Each dot corresponds to a CATT estimate based on the covariate values of a treated unit, while each red cross symbolizes the ATT estimates. For every estimate, the AIPW estimator is employed, with the GRF approach for estimating nuisance parameters. Different subfigures indicate various data comparisons: Subfigure A: Compares LDW-Experimental with LDW-CPS. Subfigure B: Compares LDW-Experimental with LDW-PSID. Subfigure C: Compares trimmed LDW-Experimental (removing 22 treated units) against trimmed LDW-CPS. Subfigure D: Compares trimmed LDW-Experimental (removing 70 treated units) to trimmed LDW-PSID.

Figure <ref> shows that although the AIPW estimator can produce ATT estimates closely aligned with the experimental benchmark using LDW data, its performance for revealing the true CATT is considerably worse. Specifically, with LDW-CPS, CATT estimates span from $-5,693 to $7,364, contrasting with the CATT estimated from experimental data which ranges from $-329 to $3,946. It overestimates CATT that exceed the ATT and underestimates CATT that fall below the ATT. Employing LDW-PSID generates CATT estimates ranging from $-8874 to $4701. With trimmed LDW-CPS, the CATT estimates align more closely with those from the experimental data. However, using trimmed LDW-PSID, the majority of CATT estimates are negative, suggesting significant biases.

Quantile Treatment Effects: Experimental vs. Nonexperimental

Note: Figures show the quantile treatment effects on the treated (QTET) using both experimental data (in blue) and nonexperimental data (in red for raw estimates and black for covariate-adjusted estimates). Each dot corresponds to a QTET estimate at a particular quantile, while shaded areas represent bootstrapped 95% confidence intervals. Unadjusted models do not incorporate covariates while adjustment models use the full set of covariates to estimate the propensity scores with a logit. Subfigure A: Compares LDW experimental data with LDW-CPS. Subfigure B: Compares LDW experimental data with LDW-PSID.

We estimate the quantile treatment effects on the treated (QTET) using an IPW approach proposed by <cit.>. Figure <ref> plots the QTET estimates using both the LDW experimental data and nonexperimental data.  The QTET estimates from either the original or trimmed LDW-CPS data align reasonably well with the true QTET, although they are often underpowered. On the other hand, using LDW-PSID data, be it original or trimmed, the estimated QTET display notable biases when compared to the experimental benchmark, which is close to zero.

These exercises suggest that, when considering alternative estimands such as CATT and QTET, among the four nonexperimental samples, overall, only trimmed LDW-CPS produces results consistently aligned closely with the experimental benchmarks.

Validation through placebo analyses We conduct placebo analyses to further assess how plausible unconfoundedness is. To do so, we select earnings in 1975 () as the placebo outcome and remove both  and  from the set of conditioning variables. Two new trimmed samples are created without using  and . We then estimate the ATT for the placebo outcome, conditional on the remaining covariates, using a variety of estimators. Figure <ref> presents the findings. Not surprisingly, the experimental benchmarks are near zero and statistically insignificant. However, all estimators using nonexperimental data generate large, negative estimates. Again, with trimmed data, the estimates are stable but remain statistically different from zero. Moreover, Figure <ref> in the SM further shows that while the CATT estimates from the experimental data hover close to zero, their nonexperimental counterparts are all negative and substantial in magnitude, indicating large biases.

Placebo Tests: '75 Earnings as the Outcome

Note: The above figures show the placebo estimates and their 95% confidence intervals using four different samples: LDW-CPS, Trimmed LDW-CPS, LDW-PSID, and Trimmed LDW-PSID. We use the same eleven estimators as before. The red dashed line and pink band represent the experiment benchmark and its 95% confidence intervals, respectively. The difference-in-means estimate for LDW-PSID is -17,531, which is not shown because it exceeds the y-axis range.

Sensitivity analyses We also conduct sensitivity analyses using the LDW data, with results depicted in contour plots in Figure <ref> in the SM. The analyses suggest that the estimated training effect based on trimmed LDW-CPS is less sensitive to potential confounders compared to trimmed LDW-PSID. For instance, with trimmed LDW-CPS, the estimate remains positive and large even when a confounder's correlations with treatment and outcome are triple those of , whereas the presence of a confounder similar to  would lead to a negative estimated effect using trimmed LDW-PSID.

The original LaLonde male samples For comparison, we also revisit the original male samples used in <cit.>. Table <ref> in the SM reports the ATT estimates using the original LaLonde male samples, of which the LDW sample is a subset. Two covariates,  and , are absent from this sample. It shows that, with sufficient overlap, most modern estimators yield estimates within relatively narrow ranges when using either CPS-SSA-1 or PSID-1 as control groups. However, these estimates do not align with the experimental benchmarks, with most estimates being negative. <cit.> report similar, negative findings.

The reconstructed AFDC female samples Using the LCS female samples, we find that many modern methods yield estimates close to the experimental benchmarks, though standard errors are often quite large. While selection appears to be less severe for AFDC women compared to the male NSW participants, as suggested by <cit.>, overlap remains a significant challenge. Additionally, we fail to substantiate the unconfoundedness assumption with a placebo test using the number of children in 1975, a variable absent in <cit.>, as the placebo outcome.

Summary After reexamining the LaLonde data, we offer some new insights into the challenge posed by <cit.>. First, we agree with existing literature that ensuring overlap and using comparable control units are essential for credible causal estimates. Second, while the choice of method is less critical with overlap, as most methods yield similar results, the propensity score remains a vital tool for assessing overlap and is integral to many estimators. Third, we stress the importance of understanding the treatment assignment mechanism and the need for additional tests to validate unconfoundedness. With the LDW and LCS data, many methods approximate the experimental benchmark for the average effects under overlap, a success not mirrored with the original LaLonde male samples. However, even with LDW or LCS data, placebo tests fail to substantiate unconfoundedness, and sensitivity analysis reveals the fragility of the regression estimate using either LDW-PSID or LCS-PSID.

§.§ Lottery Prizes on Labor Earnings

We now reanalyze the lottery data from <cit.>, who carried out an original survey to investigate the impact of the size of lottery prizes in Massachusetts during the mid-1980s on the economic behavior of lottery players. The primary outcome is post-winning labor earnings. This empirical example is appealing for two reasons: (1) we have a much better understanding of the treatment assignment process (lottery), and (2) six periods of lagged outcomes are available to validate the unconfoundedness assumption.

There are three treatment and control groups. The control group, termed “non-winners,” consists of 259 season ticket holders who have won a small, one-time prize, ranging from $100 to $5,000 (in essence, they are one-time, minor winners). The treatment groups, labeled “big winners” (43 individuals) and “small winners” (194 individuals), are those who clinched a major prize. They might be season ticket holders or one-time buyers. The annual installments for these prizes ranged from $1,139 to $99,888 and exceeded $100,000, respectively. These prizes were disbursed in yearly installments for over 20 years.

While randomization should ideally ensure that the treatment and control groups are comparable at the time of the lottery entry, the authors highlight three potential reasons this might not be the case. First, individuals can purchase multiple tickets, increasing their odds of winning. Second, those who hold season tickets might differ from those who buy single tickets. Lastly, there were discrepancies in the response rates between winners and non-winners (49% and 42%, respectively), and these response rates could be influenced by a range of factors. However, the authors expect that the unconfoundedness assumption will hold once they condition on a set of observable covariates, including the year of winning and the number of tickets bought. Importantly, they also gathered data on past labor earnings for up to six years before the individuals won a prize. These past outcomes can be utilized either as conditioning variables or as placebo outcomes.

Assessing Overlap in IRS Data

Big Winners vs Non-Winners

Small Winners vs Non-Winners

Note: Histograms depict the log odds ratios, i.e., $\log\frac{\hat{e}}{1-\hat{e}}$, using propensity scores estimated through GRF.

In the subsequent analysis, we will consider labor earnings from seven post-lottery-winning periods as the outcomes. These are denoted as $Y_{i, 0}, ..., Y_{i, 6}$, where $t=0$ represents the year of winning a lottery—recall that individuals in the control group also received a modest, one-time prize that year. We will treat the labor earnings from the three years immediately preceding the lottery win, i.e., $Y_{i,-3}, Y_{i,-2}, Y_{i,-1}$, as well as their average, as placebo outcomes. The labor earnings from the three years before those, i.e., $Y_{i,-6}, Y_{i,-5}, Y_{i,-4}$, will be used as covariates for adjustment, alongside a set of time-invariant pre-lottery-winning variables. These include the number of tickets purchased (), gender (), employment status at the time of winning (), age when the lottery was won (), total years of education (), and the presence of a college degree (). Figure <ref> assesses the overlap between the two treatment groups and the control group using the mentioned covariates. The figure indicates that while the propensity score distribution of individuals in the treatment groups differ from that of the control group, the propensity scores of the treatment groups still fall within the support of the control group.[To improve overlap, we further trim the control group for each of the two treatment groups by implementing 1:1 matching based on propensity scores, resulting in two trimmed samples. As shown in the SM, the results based on the trimmed samples are very similar to those based on the original data.]

ATT and Placebo Estimates: IRS Data

Note: Subfigure (A) and (C) show the ATT estimates of big and small winners, respectively, versus non-winners on the average annual labor earnings from Year 0 to Year 6, along with their 95% confidence intervals. Subfigure (B) and (D) show the ATT estimates of big and smaller winners, respectively, versus non-winners on the placebo outcome, the average annual labor earnings from Year –3 to Year –1, along with their 95% confidence intervals. We use the same eleven estimators as before.

Figure <ref> plots the ATT estimates, using various estimators, of the effect of lottery prizes on the real outcome (the average annual labor earnings from Year $0$ to Year $6$) and the placebo outcome (the average annual labor earnings from Year $-3$ to Year $-1$). The first two panels compare “big winners” with “non-winners,” while the third and fourth panels compare “small winners” with “non-winners.” Figure <ref> shows that various covariate adjustment methods produce consistent results, which align with the findings reported in the original paper: winning a large prize leads to a significant decrease in labor income in the following years, averaging as much as $8,000 annually. In contrast, winning a smaller prize results in a more modest decline, averaging approximately $3,000 per year. When doubly robust estimators (such as AIPW-GRF) are employed, the estimates for the placebo effects are very close to zero, lending support to the unconfoundedness assumption.

ATT and CATT Estimates: IRS Data

ATT: Big Winners vs Non-Winners

ATT: Small Winners vs Non-Winners

CATT: Big Winners vs Non-Winners

CATT: Small Winners vs Non-Winners

Note: Figures show the ATT and CATT estimates using the IRS data. The outcome variables include earnings from 3 years before winning to 6 years after winning.  The estimates for pre-winning outcomes serve as placebo tests. Adjusted covariates include: time of playing, #tickets bought, gender, work then, age at winning, years of education, college degree, and earnings 6 to 4 years before winning. Subfigures A-B: we use the difference-in-means estimator (gray diamonds) and the AIPW-GRF estimator (black solid circles for the original data and red triangles for the trimmed data).   Subfigures C-D: the distributions of CATT estimates using AIPW-GRF based on the full sample (with the black dots representing the corresponding ATT estimates).

We then estimate the ATT and CATT for labor income from Year $-3$ to Year $6$ separately using both difference-in-means and AIPW-GRF. The results are shown in Figure <ref> (A) and (B). These representations closely resemble an event study plot used in panel data analysis although our main identification assumption is unconfoundedness. They show that in estimating the effect of big prizes, AIPW-GRF using the original or trimmed data produces estimates very similar to a simple difference-in-means estimator, suggesting minimal selection between the two groups. On the other hand, when estimating the effect of small prizes, the estimates from AIPW-GRF and difference-in-means diverge. However, findings from the former are much more credible than those from the latter because difference-in-means does not fare well in the placebo tests, whereas the former yields placebo estimates that are nearly zero. Figure <ref>(C) and (D) display the CATT estimates (calculated at the covariate values of the treated units) for each of the ten outcomes in both comparisons. Beyond providing corroborative evidence for the placebo tests (i.e., CATT estimates align closely with 0 in the years leading up to the win), they also reveal substantial treatment effect heterogeneity among lottery winners. Notably, the distributions of the CATT for many post-winning years appear bimodal.

Overall, we find that in the lottery study, placebo tests provide strong evidence for the unconfoundedness assumption, bolstering the credibility of the causal estimates. Importantly, unconfoundedness is much more believable in this study than in the LaLonde case because the inherent randomization of lotteries played a key role in treatment assignment, while supplementary covariates help account for discrepancies between treatment and control groups stemming from challenges like differential responses to the survey. The inclusion of six preceding outcomes also proves invaluable, as they likely explain both selection and the outcome variables; moreover, they also serve as good candidates for placebo outcomes, given their comparability to these outcomes.

§ CONCLUSION

In his influential 1986 paper, Robert LaLonde critically evaluated nonexperimental methods, such as those based on linear regression methods and two-step selection models, by comparing their estimates to experimental benchmarks. He discovered that nonexperimental approaches produce highly variable estimates and fail to match experimental findings. Since then, there has been considerable progress in nonexperimental evaluation methodology. To illustrate these advances, we revisit the LaLonde and IRS datasets. Our analysis demonstrates that modern methods are effective in estimating adjusted group differences. A key caveat is that this relies on sufficient overlap. The estimates are less sensitive to the choice of the specific estimator. However, these estimates should be cautiously interpreted regarding causality, especially when there is a limited understanding of the assignment mechanism. In such cases, substantiating unconfoundedness is challenging and often requires supplementary analyses such as placebo tests. We show that unconfoundedness cannot be credibly validated using the LaLonde nonexperimental data. In contrast, the IRS data, with its clear lottery-based assignment and extensive pretreatment information, makes unconfoundedness a more credible assumption.

Based on the lessons learned since <cit.> in nearly four decades, we offer the following recommendations to practitioners:

* Begin analyses of causal effects with an effort to understand the assignment mechanism. A clear grasp of the “design” is crucial for the credibility of the unconfoundedness assumption.

* Estimate the propensity score using a flexible method. Assess overlap by plotting the distributions of propensity scores for treated and control units. Trim the data based on the propensity score to make the groups more comparable.

* Apply modern methods, such as doubly-robust estimators, to estimate the average causal effects. Explore alternative estimands, such as the conditional average treatment effects and quantile treatment effects.

* Perform placebo tests, such as those using pretreatment outcomes, to validate unconfoundedness. Conduct sensitivity analyses to gauge the robustness of the findings.

We provide an online tutorial with  code to assist researchers in implementing these methods.

§ TABLES 4-6 IN LALONDE (1986)

The following tables are adapted from Tables 4, 5, and 6 in <cit.>. We thank Robert LaLonde's estate for allowing us to include these tables in the Appendix.

§ ONLINE SUPPLEMENTARY MATERIALS

Table of Contents

B.1. Additional Results Using the LDW Data

* Assessing overlap

* ATT and placebo estimates

* CATT plots for placebo tests

* Sensitivity analyses

* ATT estimates without using '74 information

* CATT plots without using '74 information

B.2. Results based on the LaLonde Male Samples

* Assessing overlap

* ATT estimates

* CATT estimates

* Quantile treatment effects

B.2. Results based on the Reconstructed Female Samples

* Descriptive statistics

* Assessing overlap

* ATT estimates

* CATT estimates

* Quantile treatment effects

* Placebo test

B.4. Additional Results Using the IRS Data

* Assessing overlap on the trimmed samples

* ATT and placebo estimates

* Results using the trimmed samples

* Quantile treatment effects

* Sensitivity analyses

§.§ Additional Results Using the LDW Data

Assessing overlap Figure <ref> demonstrates overlap in the LDW data using the propensity scores estimated via GRF (instead of log odds).

Assessing Overlaps in LDW Data: Propensity Scores

LDW-CPS Trimmed

LDW-PSID Trimmed

Note: Histograms of the propensity scores estimated through GRF are depicted. Each subfigure represents a different sample. Subfigure A: LDW-Experimental. Subfigure B: LDW-CPS. Subfigure C: LDW-PSID. Subfigure D: Trimmed LDW-CPS. Subfigure E: Trimmed LDW-PSID. For C and D, the propensity scores are reestimated after trimming.

ATT and placebo estimates Table <ref> shows the ATT estimates using four different samples: LDW-CPS, LDW-PSID, trimmed LDW-CPS, and trimmed LDW-PSID. The ATT estimates based on the experimental benchmarks are highlighted in bold font in the first row. These estimates are visualized in Figure <ref> in the main text.

ATT Estimates under Unconfoundedness: LDW Samples

Note: ATT estimates using the LDW data. The outcome variable is . We adjust for the following covariates: , , , , , , , , , and . The trimmed samples are based on 1:1 matching on propensity scores estimated via GRF. Robust standard errors are in the parentheses.

Table <ref> shows the results of the placebo analyses using earnings in 1975 () as the placebo outcome and remove both  and  from the set of conditioning variables. These estimates are visualized in Figure <ref> in the main text.

Placebo Test: '75 Earnings as the Outcome

Note: ATT estimates using the LDW data. The outcome variable is . We adjust for the following covariates: , , , , , , , and .  The trimmed samples are based on 1:1 matching on propensity score estimated via GRF. Robust standard errors are in the parentheses.

CATT plots for placebo tests Figure <ref> shows that the CATT estimates using 1975 earnings as the placebo outcome.

Placebo Tests using LDW Data: Experimental vs. Nonexperimental

LDW-CPS Trimmed

LDW-PSID Trimmed

Note: Scatterplots show the CATT on a placebo outcome, real earning in 1975 (), using both experimental data (x-axis) and nonexperimental data (y-axis). Each dot corresponds to a CATT estimate based on the covariate values of a treated unit, while each red cross symbolizes the ATT estimates. For every estimate, the AIPW estimator is employed, with the GRF approach for estimating nuisance parameters. Different subfigures indicate various data comparisons: Subfigure A: Compares LDW-Experimental with LDW-CPS. Subfigure B: Compares LDW-Experimental with LDW-PSID. Subfigure C: Compares trimmed LDW-Experimental (removing 22 treated units) against trimmed LDW-CPS. Subfigure D: Compares trimmed LDW-Experimental (removing 70 treated units) to trimmed LDW-PSID.

Sensitivity analyses Figure <ref> shows the results of the sensitivity analyses using the trimmed LDW data, including trimmed LDW-CPS and trimmed LDW-PSID. We used 1975 earnings () as the benchmark covariate. The analysis suggests that the estimated training effect based on LDW-CPS is less sensitive to potential confounders compared to trimmed LDW-PSID.

Sensitivity Analyses for Trimmed LDW-CPS and LDW-PSID

Note: Contour plot for the treatment effect coefficient $\hat{\tau}_{OLS}$ based on sensitivity analysis first proposed by <cit.> and then modified by <cit.>. The red dashed line indicates $\hat\tau_{OLS} = 0$. The benchmark covariate is . The model is a linear regression with all available long-term covariates included. Both datasets are preprocessed by trimming observations whose estimated propensity scores are smaller than 0.1 or bigger than 0.9.

ATT Estimates without using '74 information

Table <ref> presents ATT estimates based on the LDW samples without using earnings  and employment status in 1974.

ATT Estimates without Using  and : LDW Samples

Note: ATT estimates using the LDW data. The outcome variable is . We adjust for the following covariates: , , , , , , , and . The trimmed samples are based on 1:1 matching on propensity scores estimated via GRF. Robust standard errors are in the parentheses.

CATT plots without using '74 information Figure <ref> shows that the CATT estimates without using 1974 earnings and employment status.

Placebo Tests using LDW Data: Experimental vs. Nonexperimental

LDW-CPS Trimmed

LDW-PSID Trimmed

Note: Scatterplots show the CATT on real earning in 1978 (), using both experimental data (x-axis) and nonexperimental data (y-axis), but without using 1974 earnings and employment status. . Each dot corresponds to a CATT estimate based on the covariate values of a treated unit, while each red cross symbolizes the ATT estimates. For every estimate, the AIPW estimator is employed, with the GRF approach for estimating nuisance parameters. Different subfigures indicate various data comparisons: Subfigure A: Compares LDW-Experimental with LDW-CPS. Subfigure B: Compares LDW-Experimental with LDW-PSID. Subfigure C: Compares trimmed LDW-Experimental (removing 10 treated units) against trimmed LDW-CPS. Subfigure D: Compares trimmed LDW-Experimental (removing 106 treated units) to trimmed LDW-PSID.

§.§ Results based on the LaLonde Male Samples

Two pretreatment variables, earnings in 1974 and employment status in 1974 are absent from this sample.

Assessing overlap Figure <ref> demonstrates overlap in the LDW using the propensity score estimated via GRF (log odds ratio).

Assessing the Overlap in the LaLonde Data (Male Sample)

Trimmed LaLonde-CPS1

Trimmed LaLonde-PSID1

Note: Histograms depict the log odds ratios, i.e., $\log\frac{\hat{e}}{1-\hat{e}}$, using propensity score estimated through GRF. Each subfigure represents a different sample. Subfigure A: Experimental (male sample). Subfigure B: LaLonde-CPS1. Subfigure C: LaLonde-PSID1. Subfigure D: Trimmed LaLonde-CPS1. Subfigure E: Trimmed LaLonde-PSID1. For C and D, the propensity scores are reestimated after trimming. Covariates do not include  and .

ATT estimates Table <ref> shows the ATT estimates using the original LaLonde male samples, of which the LDW sample is a subset. Two covariates,  and , are absent from this sample. CPS-SSA-1 and PSID-1 are used as nonexperimental control groups. We also trim the data to improve overlap, using the same procedure applied to the LDW samples. Figure <ref> visualizes the ATT estimates.

ATT Estimates: LaLonde Male Samples

Note: ATT estimates using the orignal LaLonde data (male sample). The control groups are CPS-SSA-1 (CPS1) and PSID-1 (PSID1). The outcome variable is . We adjust for the following covariates: , , , , , , , and . The trimmed samples are based on 1:1 matching on propensity score estimated via GRF. Robust standard errors are in the parentheses.

ATT Estimates Given Unconfoundedness: LaLonde Male Samples

Note: The above figures show the ATT estimates and their 95% confidence intervals using four different samples: NSW-CPS, NSW-PSID, Trimmed NSW-CPS, and Trimmed NSW-PSID. Eleven estimators are employed, including difference-in-means, linear regression, Oaxaca Blinder, GRF as an outcome model, 1: 5 nearest neighbor matching with bias correction, propensity score matching, IPW with propensity scores estimated by GRF, CBPS, entropy balancing, double/debiased matching learning with elastic net (implemented using ), and AIPW with GRF  (implemented using ). The red dashed line and pink band represent the experiment benchmark and its 95% confidence intervals, respectively.

CATT estimates  Figure <ref> shows the CATT estimates using the original LaLonde data (male sample). Two covariates,  and , are not included in this sample.

CATT Estimates for the LaLonde Data (Male Sample)

NSW-CPS Trimmed

NSW-PSID Trimmed

Note: Scatterplots show the CATT using both experimental data (x-axis) and nonexperimental data (y-axis) from <cit.> (male sample). Each dot corresponds to a CATT estimate based on the covariate values of a treated unit, while each red cross symbolizes the ATT estimates. For every estimate, the AIPW estimator is employed, with the GRF approach for estimating nuisance parameters. Different subfigures indicate various data comparisons: Subfigure A: Compares Experimental with LaLonde-CPS1. Subfigure B: Compares Experimental with LaLonde-PSID1. Subfigure C: Compares trimmed Experimental (removing 30 treated units) against trimmed NSW-CPS. Subfigure D: Compares trimmed Experimental (removing 150 treated units) to trimmed NSW-PSID.

Quantile treatment effects Figures <ref> shows the quantile treatment effects on the treated using the original LaLonde (NSW) male samples.

Quantile Treatment Effects: Experimental vs. Nonexperimental

Note: Figures show the quantile treatment effects on the treated (QTET) using both experimental data (in blue) and nonexperimental data (in red for raw estimates and black for covariate-adjusted estimates). Each dot corresponds to a QTET estimate at a particular quantile, while shaded areas represent bootstrapped 95% confidence intervals. Unadjusted models do not incorporate covariates while adjustment models use the full set of covariates to estimate the propensity scores with a logit. Subfigure A: Compares NSW experimental data with NSW-CPS. Subfigure B: Compares NSW experimental data with NSW-PSID.

Sensitivity analyses Figure <ref> shows the results of the sensitivity analyses using the trimmed LaLonde male samples, including trimmed NSW-CPS and trimmed NSW-PSID. We used 1975 earnings () as the benchmark covariate. The analysis suggests that the estimated training effects are robust to potential confounders that behavior like . For instance, with trimmed NSW-CPS or NSW-PSID, the estimate remains positive and substantial even when a confounder's correlations with treatment and outcome are triple those of .

Sensitivity Analyses for Trimmed NSW-CPS and NSW-PSID

Note: Contour plot for the treatment effect coefficient $\hat{\tau}_{OLS}$ based on sensitivity analysis first proposed by <cit.> and then modified by <cit.>. The red dashed line indicates $\hat\tau_{OLS} = 0$. The benchmark covariate is . The model is a linear regression with all available long-term covariates included. Both datasets are preprocessed by trimming observations whose estimated propensity scores are smaller than 0.1 or bigger than 0.9.

§.§ Results based on the Reconstructed LaLonde Female Samples

We report findings using the LaLonde female samples reconstructed by <cit.>, referred to as the LaLonde-Calónico-Smith (LCS) sample. Consistent with LaLonde's original analysis, the outcome variable is earnings in 1979 (). We use the same set of covariates as in <cit.>. Notably, this set does not include two pretreatment variables: earnings in 1974 and employment status in 1974. We also exclude the number of children in 1975 (), which is available in the LCS dataset, from the covariates so that it can serve as a placebo outcome. We also trim the data to improve overlap, using the same procedure applied to the LDW samples. The threshold for the propensity score to trim the sample in Step (A) is set at 0.9.

Descriptive statistics Table <ref> shows the descriptive statistics of the reconstructed LaLonde female sample.

Descriptive Statistics: LCS Female Samples

Note: Standard deviations are in the parentheses.

Assessing overlap Figure <ref> demonstrates overlap in the LDW using the propensity score estimated via GRF (log odds ratio).

Assessing the Overlap in the Reconstructed LaLonde Female Samples

Trimmed LCS-PSID

Note: Histograms depict the log odds ratios, i.e., $\log\frac{\hat{e}}{1-\hat{e}}$, using propensity score estimated through GRF. Each subfigure represents a different sample. Subfigure A: LCS-Experimental. Subfigure B: LCS-PSID. Subfigure C: Trimmed LCS-PSID. For C, the propensity score is reestimated after trimming. Covariates do not include , , and .

ATT estimates Table <ref> shows the ATT estimates using the reconstructed LaLonde female samples. Reconstructed PSID-1 is used as the nonexperimental control group.  Figure <ref> visualizes the ATT estimates.

ATT Estimates: Reconstructed LaLonde Female Samples

Note: ATT estimates the reconstructed LaLonde female samples. The control group is PSID-1 (PSID1). The outcome variable is . The trimmed samples are based on 1:1 matching on propensity score estimated via GRF. Robust standard errors are in the parentheses.

ATT Estimates: Reconstructed Female Samples

Note: The above figures show the ATT estimates and their 95% confidence intervals using two different samples: LCS-PSID and Trimmed LCS-PSID. The same eleven estimators are employed. The red dashed line and pink band represent the experiment benchmark and its 95% confidence intervals, respectively.

CATT estimates  Figure <ref> shows the CATT estimates using the reconstructed LaLonde female samples.

CATT Estimates: Reconstructed Female Samples

LCS-PSID Trimmed

Note: Scatterplots show the CATT using both experimental data (x-axis) and nonexperimental data (y-axis) from the reconstructed LaLonde female samples. Each dot corresponds to a CATT estimate based on the covariate values of a treated unit, while each red cross symbolizes the ATT estimates. For every estimate, the AIPW estimator is employed, with the GRF approach for estimating nuisance parameters. Different subfigures indicate various data comparisons: Subfigure A: Compares LCS-Experimental with LaLonde-PSID1. Subfigure B: Compares Trimmed LCS-Experimental to Trimmed LCS-PSID.

Quantile treatment effects Figures <ref> shows the quantile treatment effects on the treated using the reconstructed LaLonde female samples.

Quantile Treatment Effects: Reconstructed Female Samples

Note: Figures show the quantile treatment effects on the treated (QTET) using the reconstructed LaLonde female samples. Results from the experimental data are shown in blue and results from the  nonexperimental data are shown in red for raw estimates and black for covariate-adjusted estimates. Each dot corresponds to a QTET estimate at a particular quantile, while shaded areas represent bootstrapped 95% confidence intervals. Unadjusted models do not incorporate covariates while adjustment models use the full set of covariates to estimate the propensity scores with a logit.

Placebo Tests Table <ref> shows the results of the placebo analyses using the number of children in 1975 () as the placebo outcome. These estimates are visualized in Figure <ref>.

Placebo Test: Number of Children in 1975 as the Outcome

Note: ATT estimates using the LDW data. The outcome variable is . The trimmed samples are based on 1:1 matching on propensity score estimated via GRF. Robust standard errors are in the parentheses.

Placebo Test: Number of Children in 1975 as the Outcome

Note: The above figures show the placebo estimates and their 95% confidence intervals using two samples: LDW-PSID, and Trimmed LDW-PSID. We use the same eleven estimators as before. The red dashed line and pink band represent the experiment benchmark and its 95% confidence intervals, respectively.

Sensitivity analyses Figure <ref> shows the results of the sensitivity analyses using the trimming LCS data. We used 1975 earnings () as the benchmark covariate. The analysis shows that the estimated training effect based on LCS-PSID is sensitive to potential confounders that behave like .

Sensitivity Analyses for Trimmed LDW-CPS and LDW-PSID

Note: Contour plot for the treatment effect coefficient $\hat{\tau}_{OLS}$ based on sensitivity analysis first proposed by <cit.> and then modified by <cit.>. The red dashed line indicates $\hat\tau_{OLS} = 0$. The benchmark covariate is . The model is a linear regression with all available long-term covariates included. Both datasets are preprocessed by trimming observations whose estimated propensity scores are smaller than 0.1 or bigger than 0.9.

§.§ Additional Results Using the IRS Data

Assessing overlap on the trimmed samples Figure <ref> shows the overlap in the propensity score (log odds ratio) using the trimmed IRS data.

Assessing Overlaps on Trimmed IRS Data

Big Winners vs Control

Small Winners vs Control

Note: Histograms depict the log odds ratios, i.e., $\log\frac{\hat{e}}{1-\hat{e}}$, using propensity scores estimated through GRF based on the trimmed samples. Subfigure A: “Bigger winners” versus controls (with “small winners” removed from the sample). Subfigure B: Small winners versus controls (with “big winners” removed from the sample).

ATT and placebo estimates Table <ref> shows the ATT estimates for the real outcome, average annual labor earnings from Year 0 to Year 6, and placebo outcome, average annual labor earnings from Year –3 to Year –1, using the IRS data. These estimates are visualized in Figure <ref> in the main text.

ATT and Placebo Estimates: IRS Data

Note: ATT estimates using the IRS data. Robust standard errors are in the parentheses. The outcome is measured in thousand USD.

Table <ref> shows the ATT estimates and placebo estimates using the trimmed samples.

ATT and Placebo Estimates: IRS Trimmed Data

Note: ATT estimates using the IRS trimmed data (based on 1:1 matching on the propensity score). Robust standard errors are in the parentheses. The outcome is measured in thousand USD.

Results using the trimmed samples Figure <ref> and <ref> show the ATT and CATT estimates based on the trimmed samples of the IRS data, respectively.

ATT Estimates: IRS Data, Trimmed Sample

Big Winners vs Control

Small Winners vs Control

Note: Figure shows the ATT estimates based on the trimmed samples of the Imbens-Rubin-Sacerdote (IRS) data.  We use the difference-in-means estimator (gray diamonds) and the AIPW estimator (black solid circles). For the latter, we adjust for the following covariates: #tickets bought, gender, work then, age at winning, years of education, college degree, earnings 6 to 4 years before winning. The outcome variables include earnings from 3 years before winning to 6 years after winning.  The estimates for pre-winning outcomes serve as placebo tests. Subfigure A: “Bigger winners” versus controls (with “small winners” removed from the sample). Subfigure B: Small winners versus controls (with “big winners” removed from the sample).

CATT Estimates: IRS Data, Trimmed Sample

Big Winners vs Controls

Small Winners vs Controls

Note: Figures show the distributions of CATT estimates using the Imbens-Rubin-Sacerdote (IRS) data (with the black dots representing the corresponding ATT estimates) based on the trimmed samples.  We adjust for the following covariates: #tickets bought, gender, work then, age at winning, years of education, college degree, earnings 6 years before winning. The outcome variables include earnings from 5 years before winning to 6 years after winning. For each estimate, the Augmented Inverse Probability Weighting (AIPW) estimator is employed, with the Generalized Random Forest (GRF) approach for estimating nuisance parameters. The estimates for pre-winning outcomes serve as placebo tests. Subfigure A: “Bigger winners” versus controls (with “small winners” removed from the sample). Subfigure B: Small winners versus controls (with “big winners” removed from the sample).

Quantile treatment effects Figures <ref> and<ref> show the quantile treatment effects on the treated using the IRS data.

Quantile Treatment Effects: IRS Data

Big Winners vs Controls (Trimmed Sample)

Small Winners vs Controls (Trimmed Sample)

Note: Figures show the quantile treat effects on the treated (QTET) using with or without adjusting for the covariates (in grey and pink, respectively) based on the full samples. Each dot corresponds to a QTET estimate at a particular quantile, while gray/pink areas represent bootstrapped 95% confidence intervals. Unadjusted models do not incorporate covariates while adjustment models use the full set of covariates (including #tickets bought, gender, work then, age at winning, years of education, college degree, and earnings 6 to 4 years before winning.) to estimate the propensity scores with a logit. Different subfigures use different samples for the nonexperimental data: Subfigure A: “Bigger winners” versus controls (with “small winners” removed from the sample). Subfigure B: Small winners versus controls (with “big winners” removed from the sample).

Quantile Treatment Effects: IRS Data  (Trimmed Sample)

Big Winners vs Controls

Small Winners vs Controls

Note: Figures show the quantile treat effects on the treated (QTET) using with or without adjusting for the covariates (in grey and pink, respectively) based on the trimmed samples. Each dot corresponds to a QTET estimate at a particular quantile, while gray/pink areas represent bootstrapped 95% confidence intervals. Unadjusted models do not incorporate covariates while adjustment models use the full set of covariates (including #tickets bought, gender, work then, age at winning, years of education, college degree, and earnings 6 to 4 years before winning.) to estimate the propensity scores with a logit. Different subfigures use different samples for the nonexperimental data: Subfigure A: “Bigger winners” versus controls (with “small winners” removed from the sample). Subfigure B: Small winners versus controls (with “big winners” removed from the sample).

Sensitivity analyses Figure <ref> shows results from a sensitivity analysis using the lottery data. The estimated causal effect of big prizes is less sensitive to potential confounders than that of small prizes.

Sensitivity Analyses for the Lottery Example

Big Winners

Small Winners

Note: Contour plot for the treatment effect coefficient $\hat{\tau}_{OLS}$ based on sensitivity analysis first proposed by <cit.> and then modified by <cit.>. The red dashed line indicates $\hat\tau_{OLS} = 0$. The benchmark covariate is earnings one year before winning the lottery. The model is a linear regression with all available long-term covariates included.