# Model Averaging and Double Machine Learning††thanks: Many thanks to Elliott
Ash, Daniel Björkegren, David Cai, Ben Jann, Michael Knaus, Rafael Lalive,
Moritz Marbach, Martin Huber, Blaise Melly, Gabriel Okasa, and Martin Spindler
for helpful discussions and comments. We are also thankful for the helpful
feedback we have received at the AI+Economics Workshop at the ETH Zürich in
2022, the Italian and Swiss Stata meetings in 2022, the 2022 Machine Learning
in Economics Summer Institute in Chicago, the LISER workshop “Machine Learning
in Program Evaluation, High-dimensionality and Visualization Techniques,” the
IAAE Annual Conference in 2023, the London 2023 Stata meeting, the 2023 Stata
Economics Virtual Symposium and the European Summer Meetings of the
Econometric Society in 2023. All remaining errors are our own. An earlier
version of the paper was presented under the title “A Practitioners’ Guide to
Double Machine Learning.”

Achim Ahrens ETH Zürich, Switzerland<EMAIL_ADDRESS>Christian B. Hansen University of Chicago, United States.
<EMAIL_ADDRESS>(Hansen<EMAIL_ADDRESS>(Wiemann). Mark E. Schaffer Heriot-Watt University, Edinburgh, United Kingdom
and IZA Institute of Labor Economics<EMAIL_ADDRESS>Thomas
Wiemann33footnotemark: 3

###### Abstract

This paper discusses pairing double/debiased machine learning (DDML) with
_stacking_ , a model averaging method for combining multiple candidate
learners, to estimate structural parameters. We introduce two new stacking
approaches for DDML: _short-stacking_ exploits the cross-fitting step of DDML
to substantially reduce the computational burden and _pooled stacking_
enforces common stacking weights over cross-fitting folds. Using calibrated
simulation studies and two applications estimating gender gaps in citations
and wages, we show that DDML with stacking is more robust to partially unknown
functional forms than common alternative approaches based on single pre-
selected learners. We provide Stata and R software implementing our proposals.

Keywords: causal inference, partially linear model, high-dimensional models,
super learners, nonparametric estimation
JEL: C21, C26, C52, C55, J01, J08

##  1 Introduction

Motivated by their robustness to partially unknown functional forms,
supervised machine learning estimators are increasingly leveraged for causal
inference. For example, lasso-based approaches such as the post-double-
selection lasso (PDS lasso) of [14] have become popular estimators of causal
effects under conditional unconfoundedness in applied economics [41, 32,
e.g.]. Yet, a recent literature also raises practical concerns about the use
of machine learning for causal inference. [70] find that lasso often fails to
select relevant confounds in small samples while inference based on linear
regression performs relatively well. [40] and [49] argue that the sparsity
assumption, on which the lasso fundamentally relies, is frequently not
plausible in economic data sets. [6] show that conditioning on confounders
using random forests may yield spurious results in IV regressions.111See also
[5] for additional discussion. In an application to the evaluation of active
labor market programs, [42] find that random forests are not suitable for the
estimation of propensity scores. A key characteristic shared by many of these
studies using machine learning for causal inference is the focus on a single
pre-selected machine learner.

This paper revisits the application of machine learning for causal inference
in light of this recent literature. In particular, we highlight the benefits
of pairing double/debiased machine learning (DDML) estimators of [30] with
stacking [69, 21, 51]. DDML can leverage generic machine learners meeting mild
convergence rate requirements for the estimation of common (causal)
parameters. Stacking improves the robustness to the underlying structure of
the data by allowing the researcher to combine multiple candidate estimators
with differing strengths rather than requiring an ad-hoc choice between them.
Based on a diverse set of applications and calibrated simulation studies, we
illustrate the finite sample performance of stacking-based DDML estimators.
The results suggest that stacking with a rich set of candidate estimators can
address some of the shortcomings highlighted in the recent literature on
causal inference with single pre-selected machine learners.

We further introduce two novel ways of combining stacking and DDML aimed at
improving practical feasibility and stability in finite samples: Short-
stacking leverages the cross-fitting step of DDML to reduce the computational
burden of stacking substantially. Pooled stacking decreases the variance of
stacking-based learners across the DDML cross-fitting folds. Both approaches
facilitate interpretability compared to conventional stacking by enforcing
common stacking weights. We complement the paper with software packages for
Stata and R that implement the proposed approaches [4, 67].

The benefits of combining multiple estimators into a ‘super learner’ via
stacking to improve robustness to the structure of the underlying data-
generating process are well-known in the statistics literature. Loss-
minimizing combinations of a pre-specified set of estimators were introduced
by [68] and [21] and generalized by [51]. Under appropriate restrictions on
the data generating process and loss-function, [52] show asymptotic
equivalence between stacking and the best-performing candidate learner.222See
also [45] for discussion of jackknife (leave-one-out) stacking. [46] and [53]
provide textbook treatments of stacking and super learning.

Despite its theoretical appeal, stacking has hitherto been rarely used for the
estimation of causal effects in economics or other social sciences. Instead,
estimators are often based on parametric (frequently linear) specifications or
single pre-selected machine learners. This can have severe consequences for
the properties of causal effect estimators if the given choice is ill-suited
for the application at hand. A simple example is shown in Figure 1 which
compares the performance of DDML using either cross-validated (CV) lasso or a
feed-forward neural network to estimate a partially linear model across two
different data-generating processes. The results show that the bias associated
with each learner strongly depends on the structure of the data. Since true
functional forms are often unknown in the social sciences, indiscriminate
choices of machine learners in practice can thus result in poor estimates.
DDML with stacking is a practical solution to this problem. As the example
showcases, DDML using stacking is associated with low bias when considering a
rich set of candidate learners that are individually most suitable to
different structures of the data.

Figure 1: Estimation bias of DDML with cross-validated lasso, feed-forward
neural net and stacking

(a) Non-linear DGP

(b) Approximately sparse linear DGP

_Notes:_ The figures compare the bias of DDML paired with either cross-
validated lasso, a feed-forward neural net (with two hidden layers of size 20)
or a stacking learner combining 13 candidate learners (including cross-
validated lasso and ridge, random forests, gradient-boosted trees and feed-
forward neural nets). See [4], where this example is taken from, for details
on the specification of each learner. With respect to the data-generating
processes, we generate 1000 samples of size $n=1000$ using the PLM
$Y_{i}=\theta_{0}D_{i}+c_{Y}g(X_{i})+\varepsilon_{i}$,
$D_{i}=c_{D}g(X_{i})+u_{i}$ where $X_{i}$ are drawn from
$\mathcal{N}(0,\Sigma)$ with $\Sigma_{i,k}=0.5^{|j-k|}$. $\varepsilon_{i}$ and
$u_{i}$ are drawn from standard normal distributions. In Figure (a), the
nuisance function is
$g(X_{i})=X_{i,1}X_{i,2}+X_{i,3}^{2}+X_{i,5}X_{i,5}+X_{i,6}X_{i,7}+X_{i,8}X_{i,9}+X_{i,10}+xX_{i,11}^{2}+X_{i,12}X_{i,13}$.
In Figure (b), the nuisance function is $g(X_{i})=\sum_{j}0.9^{j}X_{ij}$.
$c_{Y}$ and $c_{D}$ are two constants chosen to ensure that the $R^{2}$ of the
regression of $Y$ onto $X$ is approximately 0.5.

We conduct simulation studies calibrated to real economic datasets to
demonstrate that stacking approaches can safeguard against ill-chosen or
poorly tuned estimators in practical settings. Throughout, stacking estimators
are associated with relatively low bias regardless of the simulated data-
generating process, strongly contrasting the data-dependent performance of the
causal effect estimators based on single pre-selected learners. The proposed
stacking approaches thus appear relevant in the ubiquitous scenario where
there is uncertainty about the set of control variables, correct functional
form or the appropriate regularization assumption.

By revisiting the simulation design of [70], we further show that stacking can
outperform linear regression for even small sample sizes. We argue that the
poor small sample performance of lasso-based approaches is partially driven by
the choice of covariate transformations and illustrate how stacking can
accommodate a richer set of specifications, including competing parametric
models. We also find that short-stacking and pooled stacking outperform DDML
paired with conventional stacking in small to moderate sample sizes. Paired
with its lower computational cost, this finding suggests that short-stacking
may be an attractive baseline approach to select and combine competing reduced
form specifications.

Finally, we demonstrate the value of pairing of DDML with stacking with two
applications. First, we examine gender gaps in citations of articles published
in top-30 economic journals from 1983 to 2020, and assess how the difference
in citations change when conditioning on content and quality proxied by the
abstract text. Estimating these conditional differences is a challenging
statistical problem due to the non-standard nature of text data, which is
increasingly encountered in economic applications [8, 28, 33, see also e.g.,].
Second, we revisit a UK sample of the OECD Skill Survey to estimate
semiparametric Kitagawa-Oxaca-Binder estimates of the unexplained gender wage
gap. Both applications highlight that estimators of structural parameters
based on single learners can be highly sensitive to the underlying structure
of the data and/or poor tuning. The applications further demonstrate that DDML
with stacking is a simple and practical solution to resolve the difficult
problem of choosing a particular candidate learner in practice. Further, we
observe that the optimal stacking weights often vary across reduced-form
equations – meaning that different conditional expectation functions in the
same data set are best estimated using different learners. This behavior
sharply contrasts with common estimation approaches, such as OLS and PDS
lasso, that impose the same form for each conditional expectation function.

The remainder of the paper is organized as follows: Section 2 provides a brief
review of DDML. Section 3 discusses DDML with stacking, short-stacking, and
pooled stacking. Section 4 presents our calibrated simulation studies. Section
5 discusses the applications, and Section 6 concludes.

##  2 Double/Debiased Machine Learning

This section outlines double/debiased machine learning as discussed in [30].
Throughout, we focus on the partially linear model as a natural extension of
commonly applied linear regression methods. Despite its simplicity, the
partially linear model illustrates practical challenges in the application of
DDML that can be addressed by stacking. We highlight, however, that our
discussion also applies to the wide range of models outlined in [30] and more
generally to estimation of low-dimensional structural parameters in the
presence of high-dimensional nuisance functions.333A key example not
explicitly discussed in [30] is doubly-robust estimation of difference-in-
difference parameters with staggered treatment assignment as in [23] and [27].
In settings with conditional parallel trends assumptions, high-dimensional
nuisance functions arise in the estimation of group-time specific average
treatment effect on the treated. The pairing of DDML and stacking, as proposed
in this paper, also directly applies to the estimator of [23] under a
conditional unconfoundedness assumption.

The partially linear model is defined by a random vector $(Y,D,X^{\top},U)$
with joint distribution characterized by

$\displaystyle Y=\theta_{0}D+g_{0}(X)+U,$ (1)

where $Y$ is the outcome, $D$ is the scalar variable of interest, and $X$ is a
vector of control variables. The parameter of interest $\theta_{0}$ and the
unknown nuisance function $g_{0}$ are such that the corresponding residual $U$
satisfies the conditional orthogonality property $E[Cov(U,D|X)]=0$. These
properties are analogous to the orthogonality properties of residuals in
multiple linear regression with the key difference here being that $g_{0}$
need not be linear in the controls.

Albeit a seemingly small change in specification, the partially linear model
has several important advantages over linear regression. For discrete $D$, for
example, results in [7] imply that $\theta_{0}$ can be interpreted as a
positively weighted average of incremental changes in the conditional
expectation function $E[Y|D=d,X]$. Under appropriate conditional
unconfoundedness assumptions, $\theta_{0}$ thus corresponds to a convex
combination of conditional average treatment effects.444Similarly, for
continuous $D$, $\theta_{0}$ corresponds to a positively weighted average of
derivatives of the conditional expectation function $E[Y|D=d,X]$ with respect
to $d$. Under a conditional unconfoundedness assumption, $\theta_{0}$ is thus
a convex combination of derivatives of the causal response function.
Importantly, these interpretations remain valid even if the additive
separability assumption of the partially linear model fails. Linear regression
coefficients, in contrast, do not correspond to positively weighted averages
of causal effects without imposing strong linearity assumptions that are
questionable in real applications.555In the context of IV estimation where
instrument validity relies on observed confounders, [16] emphasize that, in
the absence of strong functional form assumptions, two stage least squares
does not generally correspond to a convex combination of local average
treatment effects (LATE). However, the IV analogue to the partially linear
model discussed here does admit a causal interpretation under the LATE
assumptions.

The advantages of the partially linear model in the interpretation of its
parameter of interest come at the cost of a more challenging estimation
problem relative to estimating a model that is linear in a pre-specified set
of variables. Estimators for $\theta_{0}$ are based on the solution to the
moment equation

$\displaystyle
E\left[\left(Y-\ell_{0}(X)-\theta_{0}(D-m_{0}(X))\right)\left(D-m_{0}(X)\right)\right]=0,$

given by

$\displaystyle\theta_{0}=\frac{E\left[\big{(}Y-\ell_{0}(X)\big{)}\big{(}D-m_{0}(X)\big{)}\right]}{E\left[(D-m_{0}(X))^{2}\right]},$

where $\ell_{0}(X)\equiv E[Y|X]$ and $m_{0}(X)\equiv E[D|X]$ are the
conditional expectations of the outcome and variable of interest given the
controls, respectively. Since conditional expectation functions are high-
dimensional in the absence of strong functional form assumptions, a sample
analogue estimator for $\theta_{0}$ requires nonparametric first-step
estimators for the nuisance parameters $\ell_{0}$ and $m_{0}$. While
nonparametric estimation generally reduces bias compared to linear regression
alternatives, the increased variance associated with more flexible functional
form estimation introduces additional statistical challenges: To allow for
statistical inference on $\theta_{0}$, the nonparametric estimators need to
converge sufficiently quickly to the true conditional expectation functions as
the sample size increases.

DDML defines a class of estimators that allows for statistical inference on
the parameter of interest $\theta_{0}$ while only imposing relatively mild
convergence requirements on the nonparametric estimators. These mild
requirements are central to the wide applicability of DDML as they permit the
use of a large variety of machine learners.666The exact convergence rate
requirement for nonparametric estimators depends on the parameter of interest.
[30] name the crude rate requirement of $o(n^{-1/4})$, but provide examples
where the rate requirement is considerably weaker. Recent contributions show
that these requirements are satisfied by specific instances of machine
learners; see, e.g., results for lasso [15, 13], random forests [65, 64, 9],
neural networks [60, 36], and boosting [55]. The exact asymptotic properties
of many other machine learners remain an active research area.

Two key devices permit the mild convergence requirements of DDML:
Identification of the parameter of interest based on Neyman-orthogonal moment
conditions and estimation using cross-fitting. Neyman-orthogonal moment
conditions are insensitive to local perturbations around the true nuisance
parameter.777In the context of the partially linear model, the formal Neyman-
orthogonality requirement is  $\displaystyle 0=$
$\displaystyle\frac{\partial}{\partial\lambda}E\Big{[}\big{(}Y-\\{\ell_{0}(X)+\lambda(\ell(X)-\ell_{0}(X))\\}-\tau_{0}(D-\\{m_{0}(X)+\lambda(m(X)-m_{0}(X))\\})\big{)}$
$\displaystyle\qquad\qquad\times\big{(}D-\\{m_{0}(X)+\lambda(m(X)-m_{0}(X))\\}\big{)}\Big{]}\bigg{|}_{\lambda=0}$
for arbitrary measurable functions $\ell$ and $m$, which can easily be
verified using properties of the residuals. Cross-fitting is a sample-
splitting approach that addresses the own-observation bias that arises when
the nuisance parameter estimation and the estimation of $\theta_{0}$ are
applied to the same observation. In practice, cross-fitting is implemented by
randomly splitting a sample $\\{(Y_{i},D_{i},X^{\top}_{i})\\}_{i\in I}$
indexed by $I=\\{1,\ldots,n\\}$ into $K$ evenly-sized folds, denoted as
$I_{1},\ldots,I_{K}$. For each fold $k$, the conditional expectations
$\ell_{0}$ and $m_{0}$ are estimated using only observations not in the $k$th
fold — i.e., in $I^{c}_{k}\equiv I\setminus I_{k}$ — resulting in
$\hat{\ell}_{I^{c}_{k}}$ and $\hat{m}_{I^{c}_{k}}$, respectively, where the
subscript ${I^{c}_{k}}$ indicates the subsample used for estimation. The out-
of-sample predictions for an observation $i$ in the $k$th fold are then
computed via $\hat{\ell}_{I^{c}_{k}}(X_{i})$ and $\hat{m}_{I^{c}_{k}}(X_{i})$.
Repeating this procedure for all $K$ folds then allows for computation of the
DDML estimator for $\theta_{0}$:

$\displaystyle\hat{\theta}_{n}=\frac{\frac{1}{n}\sum_{i=1}^{n}\big{(}Y_{i}-\hat{\ell}_{I^{c}_{k_{i}}}(X_{i})\big{)}\big{(}D_{i}-\hat{m}_{I^{c}_{k_{i}}}(X_{i})\big{)}}{\frac{1}{n}\sum_{i=i}^{n}\big{(}D_{i}-\hat{m}_{I^{c}_{k_{i}}}(X_{i})\big{)}^{2}},$

where $k_{i}$ denotes the fold of the $i$th observation.

Since the cross-fitting algorithm depends on the randomized fold split, and
since some machine learners rely on randomization too, DDML estimates vary
with the underlying random-number generator and seed. To reduce dependence on
randomization, it is thus worthwhile to repeat the cross-fitting procedure and
apply mean or median aggregation over DDML estimates [4, see Remark 2 in]. We
show in Section 5 that repeating the cross-fitting procedure is a useful
diagnostic tool, allowing to gauge the stability of DDML estimators.

Under the conditions of [30] — including, in particular, the convergence
requirements on the nonparametric estimators — $\hat{\theta}_{n}$ is root-$n$
asymptotically normal around $\theta_{0}$. As already highlighted by the
example in Figure 1, however, a poorly chosen or poorly tuned machine learner
for the estimation of nuisance parameters $\hat{\ell}$ and $\hat{m}$ can have
detrimental effects on the properties of $\hat{\theta}_{n}$. Since no machine
learner can be best across all settings, this raises the difficult question of
which learner to apply in a particular setting. In the next section, we
discuss how DDML can be paired with stacking to provide a practical solution
to the choice of learner. We also illustrate how the cross-fitting structure
naturally arising in DDML estimators can be leveraged to substantially reduce
the computational burden otherwise associated with stacking.

##  3 Pairing DDML with Stacking Approaches

This section discusses the estimation of structural parameters by pairing DDML
with stacking approaches. After the discussion of DDML with conventional
stacking, we introduce two stacking variants that leverage the cross-fitting
structure of DDML estimators: short-stacking and pooled stacking. To fix
ideas, we focus on the nuisance parameter $\ell_{0}(X)=E[Y|X]$ arising in the
partially linear model where we consider an i.i.d. sample
$\\{(Y_{i},X_{i})\\}_{i\in I}$. Further, we consider a rich set of $J$ pre-
selected base or candidate learners. The set of learners could include
distinct parametric and nonparametric estimators — e.g., linear or logistic
regression, regularized regression such as the lasso, or tree-based methods
such as random forests — as well as the same algorithm with varying
(hyper-)tuning parameters or different (basis) expansions of the control
variables. It is important to note that the set of candidate learners for
stacking can readily incorporate commonly used unregularized learners such as
linear or logistic regression; in practice, sometimes the best-performing
candidate learner may be one such learner.

Figure 2: Cross-fitting with conventional stacking

_Notes:_ The diagram illustrates cross-fitting with $K=5$ cross-fitting folds
combined with stacking using $V=3$ cross-validation folds and three candidate
learners. The illustration uses the estimation of $\ell_{0}=E[Y|X]$ as an
example.

#### DDML with conventional stacking.

Combining DDML with conventional stacking involves two layers of re-sampling,
as we illustrate in Figure 2. The cross-fitting layer divides the sample into
$K$ cross-fitting folds, denoted by $I_{1},\ldots,I_{K}$. In each cross-
fitting step $k\in\\{1,\ldots,K\\}$, the stacking learner is trained on the
training sample which excludes fold $I_{k}$ and which we label $T_{k}\equiv
I\setminus I_{k}$. Fitting the stacking learner, in turn, requires sub-
dividing the training sample $T_{k}$ further into $V$ cross-validation folds.
This second sample split constitutes the cross-validation layer. We denote the
cross-validation folds in cross-fitting step $k$ by $T_{k,1},\ldots,T_{k,V}$.
Each candidate learner $j\in\\{1,\ldots,J\\}$ is cross-validated on these
folds, yielding cross-validated predicted values for each learner.

The final learner fits the outcome $Y_{i}$ against the cross-validated
predicted values of each candidate learner. The most common choice is to
construct a convex combination via constrained least squares (CLS), with
weights restricted to be non-negative and summing to one. Specifically, for
each $k$, candidate learners are combined to solve

$\underset{w_{k,1},\ldots,w_{k,J}}{\min}\sum_{i\in
T_{k}}\left(Y_{i}-\sum_{j=1}^{J}w_{k,j}\hat{\ell}^{(j)}_{T_{k,v(i)}^{c}}(X_{i})\right)^{2}\qquad\textrm{s.t.}\
w_{k,j}\geq 0,\ \sum_{j=1}^{J}|w_{k,j}|=1.$

Here, $\hat{\ell}^{(j)}_{T_{k,v(i)}^{c}}(X_{i})$ denotes the out-of-sample
predicted value for observation $i$, which is calculated from training
candidate learner $j$ on sub-sample $T_{k,v(i)}^{c}\equiv T_{k}\setminus
T_{k,v(i)}$, i.e., all step-$k$ cross-validation folds but fold $(k,v(i))$
which is the fold of the $i$th observation. We call the resulting
$\hat{w}_{k,j}$ the stacking weights. The stacking predictions are obtained as
$\sum_{j}\hat{w}_{k,j}\hat{\ell}_{T_{k}}^{(j)}(X_{i})$ where each learner $j$
is re-fit on $T_{k}$.

Although various options for combining candidate learners are available, CLS
facilitates the interpretation of stacking as a weighted average of candidate
learners [46]. Due to this constraint, CLS tends to set some stacking weights
to exactly zero. The constraint also regularizes the final estimator, which is
important to mitigate issues arising from potential multicollinearity of the
candidate learners. An alternative to CLS, which we refer to as single-best
learner, is to impose the constraint that $w_{k,j}\in\\{0,1\\}$ and
$\sum_{j}w_{k,j}=1$, implying that only the candidate learner with lowest
cross-validated loss is used as the final estimator. Under appropriate
restrictions on the data-generating process and loss function, [52] show
asymptotic equivalence between stacking and the best-performing candidate
learner.888The scikit-learn [22] routines StackingRegressor and
StackingClassifier implement stacking for Python. In Stata, stacking
regression and classification are available via pystacked, which is a Stata
front-end for these Python routines [3].

A drawback of DDML with stacking is its computational complexity. Considering
the estimation of a single candidate learner as the unit of complexity (and
ignoring the cost of fitting the final learner), DDML with stacking
heuristically has a computational cost proportional to $K\times V\times J$.
For example, when considering DDML with $K=5$ cross-fitting folds and $J=10$
candidate learners that are combined based on $V=5$ fold cross-validation,
more than 250 candidate learners need to be individually estimated. Although
DDML with stacking is “embarrassingly parallel” and can thus be expected to
decrease in computational time nearly linearly in the number of available
computing processes, the increased complexity limits its application to
moderately complex applications. Another potential concern (which we
investigate in Section 4.2) is that DDML with stacking might not perform well
in small samples, given that candidate learners are effectively trained on
approximately $\frac{(K-1)(V-1)}{KV}$% of the full sample (see Figure 2).
These two concerns motivate _short-stacking_.

#### DDML with short-stacking.

In the context of DDML, we propose to take a short-cut: Instead of fitting the
final learner on the cross-validated fitted values in each step $k$ of the
cross-fitting process, we can directly train the final learner on the cross-
fitted values using the full sample; see Figure 3. Formally, candidate
learners are then combined to solve

$\underset{w_{1},\ldots,w_{J}}{\min}\sum_{i=1}^{n}\left(Y_{i}-\sum_{j=1}^{J}w_{j}\hat{\ell}^{(j)}_{I_{k(i)}^{c}}(X_{i})\right)^{2}\qquad\textrm{s.t.}\
w_{j}\geq 0,\ \sum_{j}|w_{j}|=1$

where $w_{j}$ are the short-stacking weights. Cross-fitting thus serves a
double purpose: First, it avoids the own-observation bias by avoiding overlap
between the samples used for estimating high-dimensional nuisance functions
and the samples used for estimating structural parameters. Second, it yields
out-of-sample predicted values which we leverage for constructing the final
stacking learner. As a consequence, the computational cost of DDML with short
stacking is heuristically only proportional to $K\times J$ in units of
estimated candidate learners. In the example from the previous paragraph,
short-stacking thus requires estimating about 200 fewer candidate learners.

We recommend DDML with short-stacking in settings where the number of
candidate learners is small relative to the sample size, i.e., $J\ll n$. If
instead the number of considered learners is very large relative to the sample
size — i.e., settings in which inference for standard linear regression on $J$
variables is invalid — pairing DDML with short-stacking may introduce
bias.999Suppose, for simplicity, we consider ordinary (unconstrained) least
squares as the final learner. Heuristically, the regression of $Y_{i}$ against
$J$ sets of cross-fitted predicted values is akin to a conventional least
squares regression of $Y_{i}$ against $J$ observed regressors where good
performance would require $J/n\rightarrow 0$, ignoring that the cross-fitted
predicted values are estimated. The additional regularization by constrained
least squares should further weaken this rate requirement. Since current
applications of machine learning in economics and other social sciences rarely
consider more than a few candidate learners, however, this is unlikely to be a
strong concern in practice.

Figure 3: Cross-fitting with _short-_ stacking

_Notes:_ The diagram illustrates cross-fitting combined with short-stacking
with $K=5$ cross-fitting folds. The illustration uses the estimation of
$\ell_{0}=E[Y|X]$ as an example.

#### DDML with pooled stacking.

While DDML with conventional stacking has one vector of weights per cross-
fitting fold, short-stacking yields a single weight for each learner. A single
weight for each learner decreases the variance of the final estimator and
facilitates the interpretation of the stacking weights. Another way of
achieving common stacking weights is DDML with pooled stacking. Pooled
stacking relies on the two-layer re-sampling strategy outlined above, but
combines candidate learners to solve

$\underset{w_{1},\ldots,w_{J}}{\min}\sum_{i\in I}\sum_{k\neq
k(i)}\left(Y_{i}-\sum_{j=1}^{J}w_{j}\hat{\ell}^{(j)}_{T_{k,v(i)}^{c}}(X_{i})\right)^{2}\qquad\textrm{s.t.}\
w_{j}\geq 0,\ \sum_{j=1}^{J}|w_{j}|=1.$

That is, pooled stacking collects the cross-validated predicted values that
are calculated in each step $k$ of the cross-fitting process for each learner
$j$ and estimates the stacking weights based on the pooled data set. We note
that the computational costs are approximately the same as for DDML with
conventional stacking.

##  4 The Practical Benefits of DDML with Stacking: Two Simulation Studies

In this section, we discuss two simulation studies illustrating the advantages
of pairing DDML with stacking over alternative approaches based on single pre-
selected learners. We begin with a simulation calibrated to household data on
wealth and 401k eligibility from the 1991 wave of the Survey of Income and
Program Participation (SIPP) in Section 4.1. In Section 4.2, we revisit the
simulation of [70] to assess the robustness of DDML with stacking approaches
in very small samples.

###  4.1 Simulation calibrated to the SIPP 1991 household data

To assess the performance of DDML with conventional stacking, short-stacking
and pooled stacking in a realistic setting, we consider the analysis of 401(k)
eligibility and total financial assets in [58] as the basis for an empirically
calibrated Monte Carlo simulation. The application has recently been revisited
by [12], [30], and [70] to approximate high-dimensional confounding factors
using machine learning. We focus on estimating the partially linear model
discussed in the previous section. The outcome is measured as net financial
assets, the treatment variable is an indicator for eligibility to the 401(k)
pension scheme, and the set of controls includes age, income, education in
years, family size, as well as indicators for two-earner status, home
ownership, and participation in two alternative pension schemes.

Algorithm 1:  Algorithm for the calibrated Monte Carlo simulation

Let $\\{(y_{i},d_{i},x_{i})\\}_{i=1,\ldots,n}$ denote the observed sample,
where $i$ is a household in the 1991 SIPP and $y_{i}$, $d_{i}$, and $x_{i}$
respectively denote net financial assets, an indicator for 401(k) eligibility,
and the vector of control variables. 1. Using the full sample, obtain the
slope coefficient $\hat{\theta}_{OLS}\approx 5\,896$ from linear regression of
$d_{i}$ against $d_{i}$, and $x_{i}$ in the original data. Construct the
partial residuals $y_{i}^{(r)}=y_{i}-\hat{\theta}_{OLS}d_{i},\,\forall i$. 2.
Fit a supervised learning estimator (either linear regression or gradient
boosting) to predict $y_{i}^{(r)}$ with the controls $x_{i}$. Denote the
fitted estimator by $\tilde{g}$. Similarly, fit a supervised learning
estimator to predict $d_{i}$ with $x_{i}$ and denote the fitted estimator by
$\tilde{h}$. 3. Repeat to generate simulated samples of size $n_{b}$: (a)
Sample from the empirical distribution of $x_{i}$ by bootstrapping $n_{b}$
observations from the original data. Denote the bootstrapped sample by
$\mathcal{D}_{b}$. (b) Draw
$\nu_{i}\overset{iid}{\sim}\mathcal{N}(0,\kappa_{1})$ and
$\varepsilon_{i}\overset{iid}{\sim}\mathcal{N}(0,\kappa_{2})$, where
$\kappa_{1}$ and $\kappa_{2}$ are simulation hyperparameters. Define
$\displaystyle\tilde{d}^{(b)}_{i}$
$\displaystyle=\mathbbm{1}\\{\tilde{h}(x_{i})+\nu_{i}\geq 0.5\\}$
$\displaystyle\tilde{y}^{(b)}_{i}$
$\displaystyle=\theta_{0}\tilde{d}^{(b)}_{i}+\tilde{g}(x_{i})+\varepsilon_{i}\qquad\forall
i\in\mathcal{D}_{b}$ where we set $\theta_{0}=6\,000$ to roughly resemble the
magnitude of the regression coefficient of 401(k) eligibility in the full
data.

Notes: We set the hyper-parameter $\kappa_{1}$ and $\kappa_{2}$ to
approximately match variance of 401(k) eligibility and log-wealth in the data.
The values of the simulation hyperparameters $(\kappa_{1},\kappa_{2})$ differ
slightly depending on the supervised learning estimator used to fit the
reduced form equations in the data. We take $\kappa_{1}=0.35$ in both
scenarios but take $\kappa_{2}=55\,500$ when using linear regression and
$\kappa_{2}=54\,000$ when using gradient boosting. Differences arise because
gradient boosting reduces residual variance in the true data.

The simulation involves three steps. In the calibration step, we fit two
generative models to the $n=9\,915$ households from the 1991 wave of the
Survey of Income and Program Participation. The first generative model is
fully linear while the second is partially linear, allowing controls to enter
non-linearly through gradient-boosted trees fitted to the real data. This
approach is aimed at extracting and magnifying the linear or non-linear
structures in the empirical conditional distributions, respectively, enabling
us to compare the performance of estimators across favorable and unfavorable
structures of the data. The generative step then simulates datasets of size
$n_{b}=\\{9\,915,\,99\,150\\}$ from the respective fully linear model and the
partially linear model. Throughout, we set the effect of 401(k) eligibility on
total financial wealth to $\theta_{0}=6\,000$. Finally, in the estimation
step, we fit various estimators to bootstrapped samples of the generated
datasets and assess their statistical properties. We outline the steps used
for constructing the two generative models in more detail in Algorithm 1.

For each bootstrap sample, we calculate estimates of the effect of 401(k)
eligibility on simulated net financial assets. The estimators we consider are
linear regression, the post-double selection (PDS) lasso estimator proposed by
[14], as well as DDML estimators with and without stacking. The candidate
learners of the DDML estimators are linear regression, cross-validated lasso
and ridge regression with interactions and second-order polynomial expansions
of the controls, cross-validated lasso and ridge with no interactions but
10th-order polynomial expansions of the controls, two versions of random
forests, two versions of gradient-boosted trees, and feed-forward neural nets
with three hidden layers of size five (see Table 4.1 notes for details). We
estimate DDML paired with conventional stacking, short-stacking and pooled
stacking, and consider different methods to construct the final conditional
expectation function estimator: CLS, unconstrained linear regression (OLS),
selecting the single best estimator, and an unweighted average.

Table 4.1 presents the mean bias, median absolute bias (MAB) and coverage
rates of a 95% confidence interval associated with estimates of the effect of
401(k) eligibility on net financial assets. The left and right panels
correspond to results based on data simulated from the linear (Panel A) and
non-linear (Panel B) generative models, respectively. The CLS weights
associated with each candidate learner are shown in Table 4.1.101010Further
results are provided in the Appendix. Tables A and A show the stacking weights
when using single-best and OLS as the final learner. Table A in the Appendix
gives the mean-squared prediction errors (MSPE) for each candidate learner for
comparison.

Table 1: Bias and Coverage Rates in the Linear and Non-Linear DGP

|  | Panel (A): Linear DGP |  | Panel (B): Non-linear DGP
---|---|---|---|---
|  | $n_{b}=9{\small,}915$ |  | $n_{b}=99{\small,}150$ |  | $n_{b}=9{\small,}915$ |  | $n_{b}=99{\small,}150$
|  | Bias | MAB | Rate |  | Bias | MAB | Rate |  | Bias | MAB | Rate |  | Bias | MAB | Rate
Full sample: |  |  |  |  |  |  |  |  |  |  |  |  |
| OLS | $-24.8$ | $820.2$ | 0.95 |  | $-1.9$ | $269.1$ | 0.95 |  | $-2587.3$ | $2642.4$ | 0.95 |  | $-2644.9$ | $2640.5$ |  0.00
| PDS-Lasso | $-25.4$ | $821.2$ | 0.95 |  | $0.6$ | $269.4$ | 0.95 |  | $-2598.4$ | $2661.3$ | 0.95 |  | $-2644.1$ | $2638.4$ |  0.00

DDML methods:  Candidate learners
OLS-30.5826.80.95-2.3270.50.95-2617.92622.60.95-2647.22645.80.00 Lasso with CV
(2nd order poly)-28.5830.20.96-1.5272.60.95746.21105.50.96703.5714.60.61 Ridge
with CV (2nd order
poly)-25.5821.10.95-1.9275.90.95801.91143.60.95714.3725.30.60 Lasso with CV
(10th order poly)187.41047.60.9561.4281.70.94-4684.72111.10.95-2.1284.50.94
Ridge with CV (10th order
poly)1069.31221.00.9438.1273.40.94-3070.82499.60.94-1.9287.50.95 Random forest
(low regularization)-196.5982.20.91-33.3356.70.87-64.01065.00.91-43.4331.80.87
Random forest (high
regularization)-28.1853.10.95-22.5288.70.94-133.0932.30.95-18.5272.80.94
Gradient boosting (low
regularization)-82.2825.00.95-19.4270.90.9552.4924.30.9513.6267.70.95 Gradient
boosting (high
regularization)28.6819.60.9671.4279.80.94199.8895.40.96182.3319.30.93 Stacking
approaches Stacking: CLS8.8829.30.95-1.5274.40.95226.81129.30.9524.9262.90.95
Stacking: Average-0.1812.90.941.4273.70.95-101.71102.70.9460.0271.90.95
Stacking: OLS-33.7874.10.942.6272.00.95871.41264.70.9426.6265.70.94 Stacking:
Single-best-28.6823.30.96-3.5272.10.95179.21016.20.9615.0266.40.95 Short-
stacking: CLS-23.8817.60.96-1.5274.30.95221.9897.30.9621.9261.70.95 Short-
stacking: Average-0.1812.90.941.4273.70.95-101.71102.70.9460.0271.90.95 Short-
stacking: OLS-27.9818.10.96-2.4274.40.95169.0888.20.9615.8262.50.94 Pooled
stacking: CLS-24.8817.80.96-1.5273.40.95251.5960.10.9624.8264.80.95 Pooled
stacking: Average-0.1812.90.941.4273.70.95-101.71102.70.9460.0271.90.95 Pooled
stacking: OLS-48.1834.60.95-2.2272.10.95376.81068.30.9516.7266.50.94

Notes: The table reports mean bias, median absolute bias (MAB) and coverage
rate of a 95% confidence interval for the listed estimators. We consider DDML
with $K=2$ cross-fit folds and the following individual learners: OLS with
elementary covariates, CV lasso and CV ridge with second-order polynomials and
interactions, CV lasso and CV ridge with 10th-order polynomials but no
interactions, random forest with low regularization (8 predictors considered
at each leaf split, no limit on the number of observations per node, bootstrap
sample size of 70%), highly regularized random forest (5 predictors considered
at each leaf split, at least 10 observation per node, bootstrap sample size of
70%), gradient-boosted trees with low regularization (500 trees and a learning
rate of 0.01), gradient-boosted trees with high regularization (250 trees and
a learning rate of 0.01), feed-forward neural nets with three hidden layers of
size five. For reference, we report two estimators using the full sample: OLS
and PDS lasso. Finally, we report results for DDML paired with conventional
stacking, short-stacking and pooled stacking where the final estimator is
either CLS, OLS, the unweighted average of candidate learners or the single-
best candidate learner. Results are based on 1,000 replications.

Given the construction of the generative models, we would expect that linear
regression performs best in the fully linear setting and that DDML with
gradient boosting performs best in the nonlinear setting where the nuisance
function is generated by gradient boosting. The simulation results confirm
this intuition, showing that the two procedures achieve among the lowest bias
and median absolute bias in the data-generating processes that are based on
them. Researchers are rarely certain of the functional structure in economic
applications, however, so that it is more interesting to consider their
respective performance in the non-favorable setting. In the non-linear data-
generating process, linear regression is among the estimators with the worst
performance across all three measures. Similarly, gradient boosting-based DDML
is non-optimal in the linear data-generating process. It is outperformed by
linear regression and CV lasso, both of which enforce a linear functional form
on the control variables, in terms of MAB.

The simulation results are consequences of the “no free lunch” theorem in
machine learning [69]. Informally, the theorem states that there exists no
estimator that performs best across all empirical settings. Researchers must,
therefore, carefully match estimators to their application. However, with
limited knowledge about underlying data-generating processes and few
functional form restrictions implied by economic theory, the number of
plausibly suitable estimators is typically large.

The bottom section of Table 4.1 reports results for DDML combined with the
three stacking approaches outlined in Section 3. For each stacking approach,
we consider stacking weights estimated by (CLS) as outlined in Section 3, set
equal to $1/J$ (Average), estimated without constraint by OLS (OLS), and by
selecting only the single best candidate learner (Single-best). We find that
short-stacking performs similarly to, and sometimes better than, conventional
and pooled stacking, while being computationally much cheaper (as shown in
Table A). For example, at $K=10$ and $V=5$, DDML combined with short-stacking
ran around 4.4 times faster on the full sample than DDML with conventional or
pooled stacking, which is roughly in line with a speed improvement by a factor
of $1/V$.111111The computations were performed on the high-performance cluster
of the ETH Zurich. Each instance used a single core of an AMD EPYC processor
with 2.25-2.6GHz (nominal)/3.3-3.5 GHz (peak) and 4GB RAM. The run time of
DDML with conventional stacking was $2\,393$s on the full sample, while short-
stacking ran in only 540s.

Table 2: Average stacking weights with CLS

& Panel (B): Non-Linear DGP $E[Y|X]$ $E[D|X]$ $E[Y|X]$ $E[D|X]$ $E[Y|X]$
$E[D|X]$ OLS0.0000.0000.0000.0000.0000.000 Lasso with CV (2nd order
poly)0.0000.0000.0000.0000.0000.000 Ridge with CV (2nd order
poly)0.0000.0340.0000.0350.0000.026 Lasso with CV (10th order
poly)0.0000.0010.0000.0010.0000.000 Ridge with CV (10th order
poly)0.0000.0380.0000.0370.0000.027 Random forest (low
regularization)0.1530.0030.1540.0010.1800.001 Random forest (high
regularization)0.0000.0610.0000.0650.0000.070 Gradient boosting (low
regularization)0.8450.8520.8460.8560.8200.870 Gradient boosting (high
regularization)0.0000.0000.0000.0000.0000.000 Neural
net0.0020.0110.0000.0060.0000.006

Notes: The table shows the average stacking weights associated with the
candidate learner for DDML with conventional stacking, pooled stacking and
short-stacking. The final learner is CLS. The bootstrap sample size is
$n_{b}=9,915$ and the number of cross-fitting folds is $K=2$. Results are
based on 1,000 replications. See Table 4.1 for more information.

Even though the simulation set-up should favor single-best as the final
learner, since there is one ‘true’ candidate learner, single-best does not
clearly outperform CLS. The bias of the OLS final learner is overall similar
to CLS, except when employing conventional stacking under the non-linear DGP
for $n_{b}=9\,915$ where the average bias is almost four times as large. The
unweighted average appears sub-optimal for $n_{b}=99\,150$ under the non-
linear DGP, and its performance likely deteriorates further if many poorly
chosen candidate learners are included.

The CLS weights in Table 4.1 indicate that stacking approaches successfully
assign the highest weights to the estimators aligning with the data-generating
process (i.e., either OLS or gradient boosting) among the ten included
candidate learners, illustrating the ability to adapt to different data
structures. Specifically, the stacking methods applied to the linear data-
generating process assign the largest weight to linear models while they
assign the largest weights to the gradient-boosting estimators and the lowest
weights to estimators that impose a linear functional form on the control
variables in the non-linear data-generating process.121212The rates at which
each candidate learner is selected by the single-best final learner are shown
in Table A in the appendix and provide similar insights. We conclude that DDML
paired with stacking approaches reduces the burden of choice researchers face
when selecting between candidate learners and specifications by allowing for
the simultaneous consideration of multiple options, thus implying attractive
robustness properties across a variety of data-generating processes.

###  4.2 DDML and Stacking in Very Small Samples

A possible concern for estimators relying on machine learning is that they
might not perform well for very small samples, given that their flexibility
comes at the cost of increased variance compared to parametric estimators.
[70] use two simulations to demonstrate that PDS lasso tends to underselect
controls, which may result in a substantial small-sample bias. They also show
that the bias heavily depends on the exact lasso penalty chosen (i.e., whether
the plugin penalty of [14] is scaled by 0.5 or 1.5), and argue in favor of OLS
with appropriately chosen standard errors over PDS lasso in high-dimensional
settings.

We revisit the 401(k) simulation set-up in WZ to assess if DDML with stacking
suffers from similar issues in small samples and to compare the performance of
DDML paired with stacking with PDS lasso and OLS. Following WZ, we run
simulations on bootstrap samples of the data for $n_{b}=\\{200,400,$
$800,1600\\}$ and approximate the bias as the mean difference relative to the
full-sample estimates ($n=9\,915$).131313The full-sample estimates are
reported in Table B. WZ consider two sets of controls: two-way interactions
(TWI), and quadratic splines with interactions (QSI) [12, as in]. The number
of predictors is 167 and 272, respectively. Figure 4 replicates the main
results of WZ (Figure 8 in their paper). Panels (a) and (b) show the bias
relative to the full sample estimate for the TWI and QSI specification based
on OLS and PDS lasso with tuning parameter equal to the plugin penalty of [14]
scaled by $c$ for $c\in\\{0.5,1,1.5\\}$. It is noteworthy that the speed at
which the bootstrapped estimates converge to the full-sample estimate depends
on the set of controls for the PDS lasso, but less so for OLS. While PDS lasso
with $c=\\{0.5,1\\}$ and OLS perform similarly if QSI controls are used, PDS
lasso converges much more slowly to the full-sample estimate with TWI
controls.

Figure 4: Replication of Figure 8 in [70].

(a) Bias (TWI)

(b) Bias (QSI)

_Notes:_ The figures report the mean bias calculated as the mean difference to
the full sample estimates. Full sample estimates reported in Table B.
Following WZ, we draw $1\,000$ bootstrap samples of size
$n_{b}=\\{200,400,600,800,1200,1600\\}$. ‘TWI’ indicates that the predictors
have been expanded by two-way interactions. ‘QSI’ refers to the quadratic
spline & interactions specification of [12].

The DDML-stacking framework allows us to choose between, and combine, OLS and
lasso with both the TWI and QSI set of controls. Another advantage of DDML
over PDS lasso is that we can leverage lasso with cross-validated penalization
for a fully data-driven penalization approach. Table 4.2 compares the
performance of the full-sample estimators OLS and PDS lasso (shown in Panel A)
to DDML-stacking estimators only relying on OLS and CV lasso with TWI and QSI
controls as candidate learners (Panel B). We again consider conventional
stacking, short-stacking and pooled stacking together with either CLS or
single-best as the final learner. We set the number of cross-fitting folds to
$K=10$ (but also consider $K=2$ below for comparison in Panel E).

Table 3: Mean bias relative to full-sample estimates

Panel B. DDML-stacking with only OLS and CV lasso ($K=10$) Short-stacking:
CLS1020-113.8-181.1-538.2-575.6-292.4 Short-stacking: Single-
best1002.3-122.2-270.1-499.7-550.3-197.7 Pooled stacking:
CLS925.7-237.3-319.1-628.1-711-370.5 Pooled stacking: Single-
best782.3-200.5-358.9-541.2-580.2-237.5 Stacking:
CLS1155.8-254.7-266.9-645-633-315.1 Stacking: Single-
best999.5-23.6-184.9-503.9-571.1-248.2 Panel C. DDML-stacking will all
candidate learners ($K=10$) Short-stacking: CLS1355.1342.2403.334.2-103.943.8
Short-stacking: Single-best669.2113.5144.6-182.3-272.648.9 Pooled stacking:
CLS2849.31345.71197383.8-102.3-10.6 Pooled stacking: Single-
best724.1-69.445-250.7-309-19.4 Stacking: CLS1394.1296.9344.52.8-168.556.9
Stacking: Single-best718.4-47104.3-141.5-318.642.5 Panel D. DDML with
candidate learners ($K=10$) OLS963-150.8210-161.7-235.531.8 Lasso with CV
(TWI)5948.63223.12589.11706.2872.2734.1 Ridge with CV
(TWI)4137.31853.81617.5951.8657.5879.2 Lasso with CV
(QSI)297.5-343.9-311.9-551.8-597.1-239.8 Ridge with CV
(QSI)426.1-11185.3-240.8-294.4-7.8 Random forest (low
regularization)1852.8618.3709.6259.77.795.5 Random forest (high
regularization)9987.44270.12940.21919.51037.8925 Gradient boosting (low
regularization)772.3-25306.370.7-127.2113 Gradient boosting (high
regularization)1060.894.3564.6292.544.2228.6 Neural
net8892.37481.26915.45653.23716.52224.2 Panel E. DDML-stacking will all
candidate learners ($K=2$) Short-stacking: CLS1842.31078.3-144.461.2446.7282.9
Short-stacking: Single-best1303.5582.3-436.4-248.8194111.1 Pooled stacking:
CLS27991471.3159.5209.8572.7508.8 Pooled stacking: Single-
best1791.9622.9-542.3-296.3144.784.8 Stacking:
CLS1924.61196.1-191.259.4390.3310.9 Stacking: Single-
best1173.4549.6-604.2-285181.8138.3

Notes: The table reports the mean bias calculated as the mean difference to
the full sample estimates. Following WZ, we draw $1\,000$ bootstrap samples of
size $n_{b}$. In Panel A, we show results for the full-sample estimators OLS
and PDS lasso using either two-way interactions as controls (denoted TWI) or
the quadratic spline & interactions specification of [12]. We scale the PDS
lasso penalty by $c=0.5$, $1$ or $1.5$. In Panel B, we report results for DDML
with stacking approaches and only relying on OLS and CV lasso. In Panel C, we
consider a larger set of candidate learners. These are: OLS, CV lasso and CV
ridge with either TWI or QSI controls, random forest with low regularization
(8 predictors considered at each leaf split, no limit on the number of
observations per node, bootstrap sample size of 70%) or high regularization (5
splitting predictors, at least 10 observation per node, bootstrap sample size
of 70%), gradient-boosted tree with either low (500 trees, learnings rate of
0.01) or high (250 trees, learning rate of 0.01) regularization, and a neural
net with three hidden layers of size 5. Panel D shows results for these
individual candidate learners. In Panels B–D, we use $K=10$ cross-fitting
folds and $R=5$ cross-fitting repetitions. Panel D uses the same
specifications as Panel C, but uses $K=2$.

Across all sample sizes, the DDML-stacking estimators strictly outperform both
OLS specifications, as well as PDS lasso with TWI, and exhibit overall similar
performance to PDS lasso utilizing OLS controls and $c=\\{0.5,1\\}$. The
differences across DDML-stacking estimators are relatively minor. The CLS
short-stacking weights reported in Table 4.2, Panel A-B, reveal that CV-lasso
with QSI controls receives the largest weights, while both OLS specifications
contribute jointly between nearly zero (at $n_{b}=200$) and only up to 15%
(for $n_{b}=1600$ and the estimation of $E[D|X]$). When selecting only a
single candidate learner, CV-lasso with QSI is chosen in more than 90% of
bootstrap iterations for the estimation of $E[Y|X]$ and $E[D|X]$ (Panel C-D in
Table 4.2), suggesting that CV-lasso with QSI controls is strictly preferable
over OLS and lasso with TWI controls in this application. This simulation
exercise again highlights that relying on poorly chosen specifications that
are not validated against other choices might be sub-optimal. In practice, the
researcher does not know whether TWI or QSI controls perform better and
whether to use OLS or lasso. Crucially, DDML paired with stacking allows for
simultaneous consideration of OLS and lasso with both TWI and QSI controls and
thus resolves the choice between learners and control specifications in a
data-driven manner.

Table 4: Short-stacking weights

Panel B. Constrained least squares. $E[D|X]$, $K=10$ OLS
(TWI).005.037.055.074.1.127.13 OLS (QSI)00.001.003.011.022.134 Lasso with CV
(TWI).264.163.137.119.114.111.232 Lasso with CV (QSI).731.8.807.803.775.74.504
Panel C. Single-best. $E[Y|X]$, $K=10$ OLS (TWI)0000.00100 OLS (QSI)0000000
Lasso with CV (TWI).186.141.128.112.09.0810 Lasso with CV
(QSI).814.859.872.888.909.9191 Panel D. Single-best. $E[D|X]$, $K=10$ OLS
(TWI)0000000 OLS (QSI)0000000 Lasso with CV (TWI).239.126.098.079.06.068.003
Lasso with CV (QSI).761.874.902.921.94.932.997

Notes: The table reports the stacking weights corresponding to the DDML short-
stacking estimators in Figure 4.2. Panel A-B use constrained least squares.
Panel C-D rely on the single-best final learner. Panel A and C refer to the
estimation of $E[Y|X]$; Panel C and D to the estimation of $E[D|X]$. See notes
below Table 4.2 for more information.

In the next step, we expand the set of candidate learners by two types of
random forests, two types of gradient-boosted trees and a feed-forward neural
net. In principle, widening the set of candidate learners increases robustness
to a larger class of unknown confounding structures. We show the results in
Panel C, Table 4.2. When measuring performance based on the difference to the
full-sample estimates, we find there are benefits of extending the set of
candidate learners for bootstrap sample sizes of $n_{b}=800$ or larger. The
results are generally comparable across conventional, short and pooled
stacking. However, single-best exhibits a lower bias for small bootstrap
sample sizes vs. CLS, while pooled stacking with CLS appears to perform worse.
The CLS weights reported in Appendix Table B illustrate how DDML-stacking
estimators adapt to the sample size. For example, for smaller sample sizes, a
larger weight is put on OLS in the estimation of $E[Y|X]$. In Panel D, we
report results for each candidate learner individually. DDML-stacking
approaches perform better than most individual candidate learners and similar
to the best-performing individual learner, which is DDML with CV lasso and QSI
controls. In Panel E, we also show results if we reduce the number of folds to
$K=2$. The performance deteriorates drastically for smaller sample sizes,
indicating that—while DDML stacking appears competitive for small sample
sizes—it is important to increase the number of folds to ensure larger
training samples for the CEF estimators.

A drawback of measuring the bias as the difference to the full-sample estimate
is that we do not gain insights about convergence to the true parameter. We
thus revisit the calibrated simulation exercise from Section 4.1, which allows
us to measure the bias as the difference to the true parameter. When the DGP
is linear (see Figure 5(a)), DDML with short-stacking or pooled stacking
perform overall similarly to OLS. DDML with conventional stacking exhibits
relatively large bias with $n_{b}=400$. If the true DGP is non-linear, see
Figure 5(b), OLS and PDS-Lasso are unable to recover the true effect, while
DDML with short and pooled stacking yield reasonably close approximations of
the true parameter even for small sample sizes. DDML with conventional
stacking is competitive only for larger samples. We provide extensive results
for mean bias and coverage rates in Tables B and B in the Appendix.

Figure 5: Mean bias for very small sample sizes

(a) Linear DGP

(b) Non-linear DGP

_Notes:_ The figure shows results from the calibrated simulation in Table 4.1,
but with smaller bootstrap sample sizes. See table notes in Table 4.1 for more
information. Full results for bias and coverage in small samples can be found
in Table B and B.

To conclude, the results highlight the risks of relying on inappropriate
functional form assumptions. DDML paired with stacking approaches—when
combined with a diverse set of candidate learners—imposes weaker conditions on
the underlying data-generating process compared to relying on a single pre-
selected learner. Short-stacking and pooled stacking outperform conventional
stacking in small samples. We conjecture the improvement is due to short and
pooled stacking imposing common weights across cross-fitting folds.

##  5 Applications

In this section, we use two applications to illustrate how pairing DDML and
stacking can increase the robustness of structural parameter estimates to the
underlying structure of the data. In the first application, we estimate gaps
in citations of articles in top economics journals across different gender
compositions among the authors. We condition on the abstract to proxy for the
content and quality of the paper and demonstrate that stacking-based DDML is a
practical solution to challenging estimation problems using text data. In the
second application, we revisit the UK sample of the OECD Skills Survey for
Kitagawa-Oaxaca-Binder estimates of the unexplained gender wage gap where we
condition on a large set of individual characteristics. Both applications
pertain to the literature on gender gaps in various domains, e.g., entry to
STEM programs [25], ICT literacy [62] or wages [63, 20], and are
methodologically also closely related to the broader literature on
discriminatory attitudes towards minority groups [44, e.g.,].

###  5.1 Gender gap in citations

This section uses DDML with stacking to estimate a partially linear model
applied to average differences in citations of articles published in top-30
economic journals from 1983 to 2020 by the gender composition of the authors.
Following [24], we distinguish between papers with (imputed) all-male, all-
female, and mixed-gender authorship.141414As we explain below, we impute the
gender mix of authors from the authors’ names. Instead of conditioning on
hand-coded characteristics such as JEL codes, we leverage the abstract text as
a proxy for the topic and quality of the article. Estimating these conditional
differences is a challenging statistical problem due to the non-standard
nature of text data, and researchers are faced with two key decisions when
operationalizing an estimator using text data: how to encode the text data
into numerical features, and how to select a suitable learner given the
encoded data. Both decisions are ex-ante challenging, but also practically
highly relevant as text data is becoming increasingly encountered in economic
applications [39, 28, 66, e.g.,]. We show that these decisions can be
consequential and that by simultaneously considering different encoding
procedures and multiple learners, DDML with stacking provides a simple
practical solution to both problems.

In documenting average differences in citations, the analysis presented also
contributes to the broader literature on gender biases in academia [54, 24,
47, e.g.,]. It is well-documented that women are under-represented in
academia, especially in senior positions [26, 54]. A possible reason for the
persistent gap in representation include is that scholarly work produced by
women faces more sceptical scrutiny compared to work produced by their male
counterparts [47, 50]. Higher scrutiny could be, for example, reflected at the
refereeing stage when a publication decision is made and, as we examine here,
after publication when scholarly work is attributed by other scholars through
citations [24, 59, 43].

Throughout our analysis, we focus on a descriptive characterization of the
average gaps in citations across different gender compositions of the authors
as given by $\theta_{0}$ in the partially linear model of Equation (1) where
$Y$ denotes log-citations, $D$ is a two-dimensional vector whose first
component is an indicator for all-female authorship and whose second component
is an indicator for mixed-gender authorship. The vector $X$ collects the
content of the abstract and a set of year-of-publication indicators. The two
components of $\theta_{0}$ may thus be interpreted as summarizing the average
relative difference in total citations between all-male and all-female
authorship, and all-male and mixed-gender authorship, respectively,
conditional on the article’s year of publication and abstract. Throughout, we
make no conditional unconfoundedness assumptions that would be necessary for
causal interpretations.

We consider a sample of $29\,185$ articles that have been published between
1983–2020. The data was sourced from Scopus and is a sub-sample of the data
analyzed in [1], who kindly shared their data with us. For each article, we
have a record of the citation count and the authors’ names, which we use to
infer the authors’ gender.151515We use the software _Namsor_ which ranks among
the best-performing algorithms for gender classification using names [61].
Articles of authors whose gender could not be classified with a probability of
less than 70% were excluded. Our sample includes 620 articles for which no
citation is recorded. These were excluded from the analysis. We also provide
results using the number of citations in Appendix Table C. In the sample, 6.2%
of articles are authored by only female authors and 23.5% have authors from
both genders.

Before turning to estimation, the text of the abstract needs to be transformed
into a numerical vector. To admit estimation conditional on the content of the
abstract, it is necessary to find a representation (referred to as embedding)
of the text that is lower-dimensional but captures its core meaning. An active
literature in statistics and computer science provides solutions to this
problem, suggesting a large variety of algorithms to construct text embeddings
[8, see the overview in]. Thus, in addition to the choice of candidate
learner, researchers intent on using text data for their analysis are faced
with the additional choice of embedding algorithm. To illustrate how stacking-
based DDML can help support this choice, we consider two procedures for
encoding the text of the abstract into numerical features: First, we consider
a bag-of-word model summarizing the text as (stemmed) word counts [34, 35, as
used in, e.g.,]. In our data, this results in a 211-dimensional vector of word
counts for each abstract. Second, since the bag-of-word approach disregards
the word order and context, we construct word embeddings generated by a pre-
trained BERT model, a transformer-based large-language model [31]. In
particular, for each abstract, we extract the 768-dimensional vector of
weights from the last hidden layer of the BERT model that was pre-trained on a
large corpus of (uncased) English text data.161616The model bert-base-uncased
is freely available from, among others, the Python library huggingface.
Instead of embedding individual words, BERT attempts to reconstruct both whole
sentences and the context of these sentences, making it particularly suitable
to characterize the content of the abstracts. Recently, [11] use BERT to
construct embeddings of product descriptions on Amazon.com.

The numerical abstract embeddings are then used in several base learners. We
consider OLS, PDS lasso and DDML with CV lasso, CV ridge, XGBoost [29], random
forests and a feed-forward neural net (see table notes for details).171717To
reduce the run time, we use regression approaches both for the estimation of
$E[Y|D,X]$ and $E[D|X]$). The base learners are aggregated by pairing DDML
with either conventional stacking or short-stacking, and with either CLS or
single-best.181818We omit pooled stacking from this application since the R
package ddml, which was used for this application, does currently not support
pooled stacking. The final estimator thus simultaneously aggregates across
both the text embedding algorithm and the base learner.

Figure 6: The citation gap by authors’ gender composition

$\underbracket{\hskip 79.6678pt}_{\textsf{Full-sample estimators}}$
$\underbracket{\hskip 241.84842pt}_{\textsf{DDML with candidate learners}}$
$\underbracket{\hskip 73.97733pt}_{\textsf{DDML and stacking}}$

Notes: The figure shows estimates of $\theta_{0}$ summarizing average relative
difference in total citations between all-male and all-female authorship, and
all-male and mixed-gender authorship, respectively, conditional on the
article’s year of publication and abstract. Error-bars show
heteroskedasticity-robust 95% confidence intervals. We consider the following
estimators: OLS, PDS lasso and DDML with the following candidate learners:
OLS, CV ridge, CV lasso, XGBoost (using 500 trees, learning rate of 0.3),
random forest (using 500 trees) and feed-forward neural net (early stopping
with 15 rounds, 0.5 dropout, 0.1 learning rate, 0.1 validation split, 50
epochs, 500 batch size and 3 hidden layers of size 10). Finally, we pair DDML
with either conventional stacking or short-stacking, and with either CLS or
single-best as the final learner based on the above candidate learners.
Throughout, we use five cross-fitting repetitions, five cross-validation folds
and five cross-fitting folds. Results from each cross-fitting replication are
illustrated in green, and median aggregates across the cross-fitting
replications are shown in orange. The sample includes $29\,185$ articles
published between 1983–2020 in top-30 economics journals. A tabular version is
provided in Table C.

Figure 5.1 shows estimates of the average relative difference in total
citations between all-male and all-female authorship (top-panel) and all-male
and mixed-gender authorship (bottom-panel), respectively, for different
control specifications and estimators. When we only condition on the
publication year, the citation penalty for all-female authorship is close to
zero, while there is a large positive effect of $+21.2$% ($s.e.=1.7$) for
mixed-gender authorship. We next employ PDS lasso to add the abstract text
either in the form of word counts or as BERT features. Using the latter, the
citation gap increases to $-8.1$% ($2.8$) for articles with all-female
authorship, while the average relative difference of articles with mixed-
gender authorship reduces to $+8.6$% ($1.6$). The estimates are qualitatively
similar when using word counts instead of BERT features.

In the figure, we also show five cross-fitting repetitions of pairing DDML
with each candidate learner (in green) and the median aggregates over these
repetitions (in orange). There are considerable differences across DDML
estimators, with the median estimates of the citation gap ranging between
$-4.2$% ($4.3$) and $-13.4$ ($3.2$) for articles with all-female authorship
and between $-4.4$ (14.3) and $+9.3$ ($1.7$) for articles with mixed-gender
authorship, highlighting that different candidate learner specifications can
yield vastly different effect sizes. These stark differences emphasize the
need to choose and tune CEF estimators carefully. Without thoroughly
validating each candidate learner, judging which results are more credible is
difficult. Furthermore, it is noteworthy that some candidate learners exhibit
substantial instability across cross-fitting repetitions, especially the
neural net learners, which is also reflected in the large median-aggregate
standard errors.

We show results from pairing DDML and stacking approaches on the right-hand
side of the same figure. Relative to the DDML estimates based on the
individual candidate learners, the stacking approaches yield lower variability
over cross-fitting repetitions, suggesting higher stability. All four
stacking-based approaches agree on an average relative difference in citations
of $-7.7$% ($2.8$) for articles with all-female authorship and suggest a
citation advantage of between $+5.4$ ($1.6$) and $+6.4$ ($1.7$) for articles
with mixed-gender authorship.

Table 5: Stacking weights in the gender citation gap application.

& Citations All female Mixed gender Panel B. Mean-squared prediction error OLS
& Unigrams 1.335 0.058 0.169 OLS & BERT 1.287 0.058 0.168 CV-lasso & Unigrams
1.334 0.057 0.168 CV-lasso & BERT 1.268 0.056 0.164 CV-ridge & Unigrams 1.335
0.057 0.168 CV-ridge & BERT 1.276 0.056 0.164 XGBoost & Unigrams 1.428 0.074
0.198 XGBoost & BERT 1.517 0.068 0.196 Random forest & Unigrams 1.347 0.059
0.170 Random forest & BERT 1.629 0.059 0.172 Neural net & Unigrams 4.269 0.149
1.424 Neural net & BERT 5.576 0.058 0.184

_Notes:_ Panel A shows stacking weights for conventional stacking (labelled
‘Conv.’) and short-stacking (labelled ‘Short’) by candidate learners and by
variable. Panel B reports the mean-squared prediction error. The final learner
is constrained least squares. The stacking weights are averaged over cross-
fitting repetitions. Treatment variables are an indicator for all-female
authors and mixed-gender authors.

Table 5.1 shows that stacking weights of conventional and short-stacking with
constrained least squares as the final learner. The stacking estimators assign
small weights to learners exhibiting a relatively large MSPE and large
variability over cross-fitting repetitions. For example, in the CEF estimation
of log citations, the neural nets have an MSPE that is over twice as large as
that of other learners. Stacking assigns, as desired, zero weights to the
neural nets, whereas OLS leveraging BERT as one of the best-performing
learners receives the largest weights. It is noteworthy that the stacking
weights often vary markedly across CEFs, highlighting that there is no reason
to assume that the same learner is best suited for estimating both $E[Y|X]$
and $E[D|X]$. This insight is especially important since most estimation
approaches (including OLS and PDS lasso) impose the same structure for each
CEF.

The results on the citation gaps in top economic journals conditional on the
content of the abstract are consistent with a citation penalty for all-female
authored articles, possibly due to a higher degree of skepticism towards all-
female author teams compared to all-male author teams. However, similar to
[24] and [56], the estimates also suggest a conditional citation advantage of
articles with mixed-gender authorship.

###  5.2 Gender gap in wages

The gap in wages between men and women is a central measure of economic gender
equality and has been the focus of an extensive empirical literature [17, see,
e.g., the review in]. The classic approach to estimating the unexplained
gender wage gap relies on a linear version of the Kitagawa-Oaxaca-Binder
decomposition ([48, 57, 18]; for an overview, see [38]). Several recent
articles by [20], [63], [19] and [10], among others, focus instead on semi-
parametric decompositions of the wage gap leveraging more flexible machine
learning algorithms. Much of this literature focuses, however, on lasso-based
approaches, even though there is no apparent reason to favor sparsity-based
approaches over learners relying on other regularization assumptions. In
contrast to the recent literature that primarily focuses on lasso-based
approaches to estimate the high-dimensional nuisance functions, we consider a
diverse set of candidate learners and aggregate them via stacking.

The parameter of interest in this application is the unexplained gender wage
gap, which is the expected difference in wages after conditioning on observed
characteristics. Formally,

$\displaystyle\theta_{0}\equiv
E\left[E\left[Y|D=1,X\right]-E\left[Y|D=0,X\right]|D=1\right],$

where $Y$ denotes the logarithm of wages, $D$ is an indicator equal to one for
women, and $X$ is a vector of potentially many individual characteristics. The
parameter is well-defined if $P(D=1|X)>0$ with probability 1.191919As in the
previous section, we focus our analysis on a descriptive parameter of interest
and do not make conditional unconfoundedness assumptions that would be
necessary for causal interpretations.

In the absence of functional form assumptions, estimation of $\theta_{0}$ is a
challenging statistical problem due to its dependence on unknown conditional
expectation functions that need to be nonparametrically estimated. Analogous
to the DDML estimator for the partially linear model outlined in Section 2, we
consider estimation of $\theta_{0}$ via the split-sample analogue of the
efficient score function for $\theta_{0}$ – i.e.,

$\displaystyle\hat{\theta}_{n}$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\left(\frac{D_{i}(Y_{i}-\hat{g}_{I^{c}_{k_{i}}}(0,X_{i}))}{\hat{p}_{I^{c}_{k_{i}}}}-\frac{\hat{m}_{I^{c}_{k_{i}}}(X_{i})(1-D_{i})(Y_{i}-\hat{g}_{I^{c}_{k_{i}}}(0,X_{i}))}{\hat{p}_{I^{c}_{k_{i}}}(1-\hat{m}_{I^{c}_{k_{i}}}(X_{i}))}\right),$

where $\hat{g}_{I^{c}_{k}}$ and $\hat{m}_{I^{c}_{k}}$ are cross-fitted
estimators for $g_{0}(D,X)\equiv E[Y|D,X]$ and $m_{0}(X)\equiv E[D|X]$, and
$\hat{p}_{I^{c}_{k}}$ is a cross-fitted estimator of $P(D=1)$.

Following [37], we take the data for this application from the UK sample of
the OECD Skills Survey, which was collected in 2011-12 and comes with a rich
set of covariates, including age, experience, education, occupation, and
industry. The final data includes $4\,836$ British respondents, allowing us to
test the performance of DDML with stacking on a relatively small sample. We
specify three sets of control variables: The reduced set of controls only
includes a selection of essential covariates: age (in levels and squared),
years of education, a literacy and numeracy test score, years of tenure in the
current job (in levels and squared), education level, hours worked per week,
and number of children. The base set of controls adds, among others,
management level, age of children, and parents’ education level. Furthermore,
we interact age and tenure with all categorical covariates. The expanded set
comprises all variables and interacts each continuous covariate with each
categorical covariate.202020The base set adds the following variables to the
reduced set: area of study, part of larger organization, management position,
type of contract, job satisfaction, health status, living with a partner, age
of youngest child, immigration age, mother’s and father’s highest level of
education, immigration status of parents, informal job-related education in
last 12 months, informal non-job-related education in last 12 months.

We include a diverse set of candidate learners to allow for a high level of
flexibility. We employ regression approaches for the estimation of the CEF of
log wages (i.e., $E[Y|D,X]$) and classification approaches for the CEF
estimation of gender (i.e., $E[D|X]$). Our candidate learners are linear (or
logistic) regression with the reduced and base set of controls; linear (or
logistic) CV-lasso and CV-ridge with the base and extended set of controls;
three random forests with 500 regression (or classification) trees and minimum
leaf sizes of 1, 50 and 100; two types of gradient-boosted regression (or
classification) trees with and without early stopping; two feed-forward neural
nets with hidden layer sizes of $(40,20,1,20,50)$ and $(30,30,30)$, and early
stopping. Finally, we aggregate the candidate learners via conventional, short
and pooled stacking, and using either CLS or single-best as the final learner.

Figure 7: Unexplained gender wage gap

Notes: The figure reports DDML estimates of the unexplained gender wage gap
based. 95% heteroskedasticity-robust confidence intervals are shown. The
candidate learners (shown on the left-hand side) are as follows: OLS (for the
outcome equation) and logit (for the propensity scores) with the reduced and
base set of controls; CV-lasso and CV-ridge with the base and extended set of
controls; three random forests with 500 trees and minimum leaf sizes of 1, 50
and 100; two gradient-boosted trees with and without early stopping; two feed-
forward neural nets with hidden layer sizes of $(40,20,1,20,50)$ and
$(30,30,30)$, and early stopping. On the right-hand side, we show DDML paired
with conventional, short and pooled stacking based on the above candidate
learners, and with either CLS or single-best as the final learner. We use 10
cross-fitting folds and 10 cross-fitting repetitions. Results from each cross-
fitting replication are illustrated in green, and median aggregates across the
cross-fitting replications are shown in orange. A tabular version is provided
in Table LABEL:tab:gwg_results_1-D.

Figure 7 reports results for individual candidate learners (on the top) and
stacking approaches (on the bottom). We show results from 10 cross-fitting
repetitions (in green) and the median aggregates (in orange). We again find
that some candidate learners exhibit substantial variability over cross-
fitting repetitions, which is also reflected in the large median-aggregate
standard errors. The variability is especially large for CV-ridge with the
extended set of controls and the neural nets, which are the candidate learners
exhibiting the largest MSE (see Appendix Table D). The stacking results are,
in contrast, relatively stable over cross-fit repetitions when using CLS as
the final learner and a little less stable when relying on single-best as the
final learner, indicating that a combination of candidate learners seems to
better fit the data than a single learner. The stacking weights and MSE in
Appendix Table D confirm that there is no single candidate learner dominating
the others. The instability of the single-best final learner is reflected in
the stacking standard errors. Given this potential for instability of choosing
a single candidate learner, we recommend favoring constrained least squares
over single-best if one is not confident that one of the chosen learners will
be significantly better than the rest, and thus stably selected, which seems
likely to be the most common setting in practice.

##  6 Conclusion

This article assesses the performance of DDML estimators in realistic settings
using applications and simulation studies calibrated to real economic data. We
highlight that estimators of structural parameters based on single pre-
selected (machine) learners can be highly sensitive to the underlying
structure of the data and/or poor tuning, and we show that pairing DDML with
stacking can help alleviate these concerns, provided that a sufficiently
diverse set of candidate learners is considered.

We discuss pairing DDML with conventional stacking but also suggest two novel
stacking approaches: Short-stacking, which substantially reduces the
computational burden by leveraging the cross-fitting naturally arising in the
computation of DDML estimates, and pooled stacking, which decreases the
variance of the stacking estimator by imposing common stacking weights over
cross-fitting folds. In our simulations, both strategies are competitive with
conventional stacking in settings with large and moderate sample sizes and are
better in small samples. The advantages of short-stacking are particularly
worth highlighting, given its substantially lower computational cost.

A key advantage of the DDML-stacking approach is that it accommodates both
traditional parametric and nonparametric specifications by allowing
simultaneous consideration of, for example, OLS with several sets of controls,
sparsity-based learners, tree-based ensembles and neural networks. In this
sense, researchers are not forcibly deviating from standard (often linear)
specifications unless the data suggests there is reason to. While machine-
learning-based causal methods may yield fundamentally different results from
linear regression only in specific examples, the additional robustness to
unexpected structures in the data thus seems to come at relatively little
cost.

## References

  * [1] Arun Advani, Elliott Ash, David Cai and Imran Rasul “Race-related research in economics and other social sciences”, CEPR Discussion Paper 16115, 2021
  * [2] Achim Ahrens, Christian B Hansen and Mark E Schaffer “PDSLASSO: Stata module for post-selection and post-regularization OLS or IV estimation and inference” Medium: Statistical Software Components, Boston College Department of Economics, 2018 URL: https://ideas.repec.org/c/boc/bocode/s458459.html
  * [3] Achim Ahrens, Christian B. Hansen and Mark E. Schaffer “pystacked: Stacking generalization and machine learning in Stata” Forthcoming in The Stata Journal arXiv, 2022 DOI: 10.48550/ARXIV.2208.10896
  * [4] Achim Ahrens, Christian B. Hansen, Mark E. Schaffer and Thomas Wiemann “ddml: Double/debiased machine learning in Stata” Forthcoming in The Stata Journal arXiv, 2023 DOI: 10.48550/ARXIV.2301.09397
  * [5] Joshua D. Angrist “Empirical Strategies in Economics: Illuminating the Path From Cause to Effect” In _Econometrica_ 90.6, 2022, pp. 2509–2539 DOI: https://doi.org/10.3982/ECTA20640
  * [6] Joshua D Angrist and Brigham Frandsen “Machine labor” In _Journal of Labor Economics_ 40.S1 The University of Chicago Press Chicago, IL, 2022, pp. S97–S140
  * [7] Joshua D Angrist and Alan B Krueger “Empirical strategies in labor economics” In _Handbook of Labor Economics_ 3 Elsevier, 1999, pp. 1277–1366
  * [8] Elliott Ash and Stephen Hansen “Text Algorithms in Economics” In _Annual Review of Economics_ 15.1, 2023, pp. annurev–economics–082222–074352 DOI: 10.1146/annurev-economics-082222-074352
  * [9] Susan Athey, Julie Tibshirani and Stefan Wager “Generalized random forests” In _Annals of Statistics_ 47.2, 2019, pp. 1148–1178 DOI: 10.1214/18-AOS1709
  * [10] Philipp Bach, Victor Chernozhukov and Martin Spindler “Heterogeneity in the US gender wage gap” In _Journal of the Royal Statistical Society Series A: Statistics in Society_ , 2023, pp. qnad091 DOI: 10.1093/jrsssa/qnad091
  * [11] Patrick Bajari et al. “Hedonic prices and quality adjusted price indices powered by AI” In _arXiv preprint arXiv:2305.00044_ , 2023
  * [12] A Belloni, V Chernozhukov, I Fernández-Val and C Hansen “Program Evaluation and Causal Inference With High-Dimensional Data” In _Econometrica_ 85.1, 2017, pp. 233–298 DOI: 10.3982/ECTA12723
  * [13] Alexandre Belloni, D Chen, Victor Chernozhukov and Christian Hansen “Sparse Models and Methods for Optimal Instruments With an Application to Eminent Domain” Publisher: Blackwell Publishing Ltd In _Econometrica_ 80.6, 2012, pp. 2369–2429 DOI: 10.3982/ECTA9626
  * [14] Alexandre Belloni, Victor Chernozhukov and Christian Hansen “Inference on treatment effects after selection among high-dimensional controls” In _Review of Economic Studies_ 81, 2014, pp. 608–650 URL: https://doi.org/10.1093/restud/rdt044
  * [15] Peter J Bickel, Ya’acov Ritov and Alexandre B Tsybakov “Simultaneous analysis of Lasso and Dantzig selector” In _Annals of statistics_ 37.4 Institute of Mathematical Statistics, 2009, pp. 1705–1732
  * [16] Christine Blandhol, John Bonney, Magne Mogstad and Alexander Torgovitsky “When is TSLS Actually LATE?” In _BFI Working Paper_ , 2022
  * [17] Francine D. Blau and Lawrence M. Kahn “The Gender Wage Gap: Extent, Trends, and Explanations” In _Journal of Economic Literature_ 55.3, 2017, pp. 789–865 DOI: 10.1257/jel.20160995
  * [18] Alan Blinder “Wage Discrimination: Reduced Form and Structural Estimates” In _Journal of Human Resources_ 8, 1973, pp. 436–455
  * [19] René Böheim and Philipp Stöllinger “Decomposition of the gender wage gap using the LASSO estimator” In _Applied Economics Letters_ 28.10 Taylor & Francis, 2021, pp. 817–828
  * [20] Marina Bonaccolto-Töpfer and Stephanie Briel “The gender pay gap revisited: Does machine learning offer new insights?” In _Labour Economics_ 78, 2022, pp. 102223 DOI: 10.1016/j.labeco.2022.102223
  * [21] Leo Breiman “Stacked regressions” In _Machine Learning_ 24.1, 1996, pp. 49–64 DOI: 10.1007/BF00117832
  * [22] Lars Buitinck et al. “API design for machine learning software: experiences from the scikit-learn project” In _ECML PKDD Workshop: Languages for Data Mining and Machine Learning_ , 2013, pp. 108–122
  * [23] Brantly Callaway and Pedro HC Sant’Anna “Difference-in-differences with multiple time periods” In _Journal of Econometrics_ 225.2 Elsevier, 2021, pp. 200–230
  * [24] David Card, Stefano DellaVigna, Patricia Funk and Nagore Iriberri “Are Referees and Editors in Economics Gender Neutral?*” In _The Quarterly Journal of Economics_ 135.1, 2020, pp. 269–327 DOI: 10.1093/qje/qjz035
  * [25] David Card and A. Abigail Payne “High School Choices and the Gender Gap in Stem” _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/ecin.12934 In _Economic Inquiry_ 59.1, 2021, pp. 9–28 DOI: 10.1111/ecin.12934
  * [26] Stephen J. Ceci, Donna K. Ginther, Shulamit Kahn and Wendy M. Williams “Women in Academic Science: A Changing Landscape” In _Psychological Science in the Public Interest: A Journal of the American Psychological Society_ 15.3, 2014, pp. 75–141 DOI: 10.1177/1529100614541236
  * [27] Neng-Chieh Chang “Double/debiased machine learning for difference-in-differences models” In _The Econometrics Journal_ 23.2 Oxford University Press, 2020, pp. 177–191
  * [28] Daniel L Chen and Arianna Ornaghi “Gender Attitudes in the Judiciary: Evidence from US Circuit Courts” In _American Economic Journal: Applied Economics_ , 2023
  * [29] Tianqi Chen and Carlos Guestrin “Xgboost: A scalable tree boosting system” In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , 2016, pp. 785–794
  * [30] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey and James Robins “Double/debiased machine learning for treatment and structural parameters” tex.ids= Chernozhukov2018a publisher: John Wiley & Sons, Ltd (10.1111) In _The Econometrics Journal_ 21.1, 2018, pp. C1–C68 DOI: 10.1111/ectj.12097
  * [31] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova “Bert: Pre-training of deep bidirectional transformers for language understanding” In _arXiv preprint arXiv:1810.04805_ , 2018
  * [32] Diva Dhar, Tarun Jain and Seema Jayachandran “Reshaping adolescents’ gender attitudes: Evidence from a school-based experiment in India” In _American Economic Review_ 112.3, 2022, pp. 899–927
  * [33] Markus Eberhardt, Giovanni Facchini and Valeria Rueda “Gender Differences in Reference Letters: Evidence from the Economics Job Market” In _SSRN Electronic Journal_ , 2022 DOI: 10.2139/ssrn.4114599
  * [34] Benjamin Enke “Moral Values and Voting” Publisher: The University of Chicago Press In _Journal of Political Economy_ 128.10, 2020, pp. 3679–3729 DOI: 10.1086/708857
  * [35] Elena Esposito, Tiziano Rotesi, Alessandro Saia and Mathias Thoenig “Reconciliation narratives: The birth of a nation after the US civil war” In _American Economic Review_ 113.6, 2023, pp. 1461–1504 DOI: 10.1257/aer.20210413
  * [36] Max H Farrell, Tengyuan Liang and Sanjog Misra “Deep neural networks for estimation and inference” In _Econometrica_ 89.1 Wiley Online Library, 2021, pp. 181–213
  * [37] Rachel Forshaw, Vsevolod Iakovlev, Mark E. Schaffer and Cristina Tealdi “Using machine learning methods to estimate the gender wage gap” Paper presented at the 16th International Conference of the Thailand Econometric Society, Chiang Mai University, Thailand, 4-6 January 2023., 2023
  * [38] Nicole Fortin, Thomas Lemieux and Sergio Firpo “Chapter 1 - Decomposition Methods in Economics” In _Handbook of Labor Economics_ 4 Elsevier, 2011, pp. 1–102 DOI: 10.1016/S0169-7218(11)00407-2
  * [39] Matthew Gentzkow and Jesse M. Shapiro “What drives media slant? Evidence from U.S. daily newspapers” Publisher: [Wiley, Econometric Society] In _Econometrica : journal of the Econometric Society_ 78.1, 2010, pp. 35–71 URL: http://www.jstor.org/stable/25621396
  * [40] Domenico Giannone, Michele Lenza and Giorgio E Primiceri “Economic predictions with big data: The illusion of sparsity” In _Econometrica_ 89.5, 2021, pp. 2409–2437
  * [41] Duncan Sheppard Gilchrist and Emily Glassberg Sands “Something to talk about: Social spillovers in movie consumption” In _Journal of Political Economy_ 124.5 University of Chicago Press Chicago, IL, 2016, pp. 1339–1382
  * [42] Daniel Goller, Michael Lechner, Andreas Moczall and Joachim Wolff “Does the estimation of the propensity score by machine learning improve matching estimation? The case of Germany’s programmes for long term unemployed” In _Labour Economics_ 65, 2020, pp. 101855 DOI: https://doi.org/10.1016/j.labeco.2020.101855
  * [43] Shoshana Grossbard, Tansel Yilmazer and Lingrui Zhang “The gender gap in citations of articles published in two demographic economics journals” In _Review of Economics of the Household_ 19.3, 2021, pp. 677–697 DOI: 10.1007/s11150-020-09508-9
  * [44] Dominik Hangartner, Daniel Kopp and Michael Siegenthaler “Monitoring hiring discrimination through online recruitment platforms” In _Nature_ 589.7843 Nature Publishing Group, 2021, pp. 572–576
  * [45] Bruce E. Hansen and Jeffrey S. Racine “Jackknife model averaging” In _Journal of Econometrics_ 167.1, 2012, pp. 38–46 DOI: https://doi.org/10.1016/j.jeconom.2011.06.019
  * [46] Trevor Hastie, Robert Tibshirani and Jerome Friedman “The Elements of Statistical Learning” New York: Springer-Verlag, 2009
  * [47] Erin Hengel “Publishing While Female: are Women Held to Higher Standards? Evidence from Peer Review” In _The Economic Journal_ 132.648, 2022, pp. 2951–2991 DOI: 10.1093/ej/ueac032
  * [48] Evelyn M. Kitagawa “Components of a Difference Between Two Rates” In _Journal of the American Statistical Association_ 50.272 Taylor & Francis, 1955, pp. 1168–1194 DOI: 10.1080/01621459.1955.10501299
  * [49] Michal Kolesár, Ulrich K. Müller and Sebastian T. Roelsgaard “The Fragility of Sparsity” arXiv:2311.02299 [econ, stat] arXiv, 2023 DOI: 10.48550/arXiv.2311.02299
  * [50] Michał Krawczyk and Magdalena Smyk “Author’s gender affects rating of academic articles: Evidence from an incentivized, deception-free laboratory experiment” In _European Economic Review_ 90, Social identity and discrimination, 2016, pp. 326–335 DOI: 10.1016/j.euroecorev.2016.02.017
  * [51] Mark J. Van Laan, Eric C Polley and Alan E. Hubbard “Super Learner” In _Statistical Applications in Genetics and Molecular Biology_ 6.1, 2007 DOI: doi:10.2202/1544-6115.1309
  * [52] Mark J. van der Laan and Sandrine Dudoit “Unified cross-validation methodology for selection among estimators and a general cross-validated adaptive epsilon-net estimator: Finite sample oracle inequalities and examples” 130, U.C. Berkeley Division of Biostatistics Working Paper Series., 2003
  * [53] Mark J Laan and Sherri Rose “Targeted learning: causal inference for observational and experimental data” New York: Springer, 2011
  * [54] Shelly Lundberg and Jenna Stearns “Women in Economics: Stalled Progress” In _Journal of Economic Perspectives_ 33.1, 2019, pp. 3–22 DOI: 10.1257/jep.33.1.3
  * [55] Ye Luo, Martin Spindler and Jannis Kück “High-Dimensional $L\\_2$ Boosting: Rate of Convergence” In _arXiv preprint arXiv:1602.08927_ , 2022
  * [56] Abdelghani Maddi and Yves Gingras “Gender diversity in research teams and citation impact in economics and management” tex.eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/joes.12420 In _Journal of Economic Surveys_ 35.5, 2021, pp. 1381–1404 DOI: https://doi.org/10.1111/joes.12420
  * [57] Ronald Oaxaca “Male-Female Wage Differentials in Urban Labor Markets"” In _International Economic Review_ 14, 1973, pp. 693–709
  * [58] James M Poterba, Steven F Venti and David A Wise “Do 401 (k) contributions crowd out other personal saving?” In _Journal of Public Economics_ 58.1 Elsevier, 1995, pp. 1–32
  * [59] Margaret E. Roberts, Brandon M. Stewart and Richard A. Nielsen “Adjusting for Confounding with Text Matching” In _American Journal of Political Science_ 64.4, 2020, pp. 887–903 DOI: 10.1111/ajps.12526
  * [60] Johannes Schmidt-Hieber “Nonparametric regression using deep neural networks with ReLU activation function” In _Annals of Statistics_ 48.4 Institute of Mathematical Statistics, 2020, pp. 1875–1897
  * [61] Paul Sebo “Performance of gender detection tools: a comparative study of name-to-gender inference services” In _Journal of the Medical Library Association : JMLA_ 109.3, 2021, pp. 414–421 DOI: 10.5195/jmla.2021.1185
  * [62] Fazilat Siddiq and Ronny Scherer “Is there a gender gap? A meta-analysis of the gender differences in students’ ICT literacy” In _Educational Research Review_ 27, 2019, pp. 205–217 DOI: 10.1016/j.edurev.2019.03.007
  * [63] Anthony Strittmatter and Conny Wunsch “The Gender Pay Gap Revisited with Big Data: Do Methodological Choices Matter?” arXiv, 2021 DOI: 10.48550/ARXIV.2102.09207
  * [64] Stefan Wager and Susan Athey “Estimation and Inference of Heterogeneous Treatment Effects using Random Forests” Publisher: Taylor & Francis In _Journal of the American Statistical Association_ 113.523, 2018, pp. 1228–1242 DOI: 10.1080/01621459.2017.1319839
  * [65] Stefan Wager and Guenther Walther “Adaptive concentration of regression trees, with application to random forests” In _arXiv preprint arXiv:1503.06388_ , 2016
  * [66] Philine Widmer, Sergio Galletta and Elliott Ash “Media Slant is Contagious”, 2023 arXiv:2202.07269 [econ.GN]
  * [67] Thomas Wiemann, Achim Ahrens, Christian B Hansen and Mark E Schaffer “ddml: Double/Debiased Machine Learning in R” https://github.com/thomaswiemann/ddml, 2023
  * [68] David H. Wolpert “Stacked generalization” In _Neural Networks_ 5.2, 1992, pp. 241–259 DOI: https://doi.org/10.1016/S0893-6080(05)80023-1
  * [69] David H Wolpert “The lack of a priori distinctions between learning algorithms” In _Neural computation_ 8.7, 1996, pp. 1341–1390
  * [70] Kaspar Wüthrich and Ying Zhu “Omitted variable bias of Lasso-based inference methods: A finite sample analysis” In _Review of Economics and Statistics_ 0.(0), 2021, pp. 1–47

Supplementary material

##  Appendix A The benefits of pairing DDML and stacking

Table A.1: Mean-squared prediction error

& Panel (B): Non-Linear DGP $E[Y|X]$ $E[D|X]$ $E[Y|X]$ $E[D|X]$ Candidate
learners OLS3.6850.2033.6730.203 Lasso with CV (2nd order
poly)3.4820.2013.4510.200 Ridge with CV (2nd order poly)3.4800.2013.4500.200
Lasso with CV (10th order poly)5.9550.2253.4230.200 Ridge with CV (10th order
poly)7.0620.2353.4250.200 Random forest (low
regularization)3.7950.2313.5150.236 Random forest (high
regularization)3.5910.2043.2520.205 Gradient boosting (low
regularization)3.3530.2003.0950.198 Gradient boosting (high
regularization)3.4050.2003.2160.199 Neural net1.4330.1741.6180.175

Notes: The table shows the mean-squared prediction error of each candidate
learner. The bootstrap sample size is $n_{b}=9\,915$ or $99\,150$. Results are
based on $1\,000$ replications. See Table 4.1 for more information.

Table A.2: Average stacking weights using OLS as the final learner

& Panel (B): Non-Linear DGP $E[Y|X]$ $E[D|X]$ $E[Y|X]$ $E[D|X]$ $E[Y|X]$
$E[D|X]$ OLS0.0120.0490.0000.040–0.0440.028 Lasso with CV (2nd order
poly)–0.125–0.319–0.136–0.2020.016–0.375 Ridge with CV (2nd order
poly)0.3690.5340.3950.4530.1660.564 Lasso with CV (10th order
poly)–0.0080.1220.0190.1060.0950.109 Ridge with CV (10th order
poly)0.0660.0270.0470.001–0.0170.034 Random forest (low
regularization)0.048–0.0150.052–0.0160.052–0.017 Random forest (high
regularization)–0.0970.071–0.1020.077–0.0960.065 Gradient boosting (low
regularization)1.1140.0241.1670.0541.3450.164 Gradient boosting (high
regularization)–0.5510.525–0.6180.506–0.7150.429 Neural
net–11.4320.0280.1560.0400.1730.052

Notes: The table shows the (average) stacking weights of each candidate
learner for conventional stacking, pooled stacking and short-stacking using
OLS as the final learner. The bootstrap sample size is $n_{b}=9\,915$ or
$99\,150$. Results are based on $1\,000$ replications. See Table 4.1 for more
information.

Table A.3: Average stacking weights using single-best

& Panel (B): Non-Linear DGP $E[Y|X]$ $E[D|X]$ $E[Y|X]$ $E[D|X]$ $E[Y|X]$
$E[D|X]$ OLS0.0000.0000.0000.0000.0000.000 Lasso with CV (2nd order
poly)0.0970.1520.0720.1420.0510.079 Ridge with CV (2nd order
poly)0.1040.1160.1000.0950.0590.080 Lasso with CV (10th order
poly)0.0890.0550.0520.0310.0440.033 Ridge with CV (10th order
poly)0.0220.0450.0070.0120.0040.035 Random forest (low
regularization)0.0000.0000.0000.0000.0000.000 Random forest (high
regularization)0.0020.0010.0010.0000.0010.000 Gradient boosting (low
regularization)0.6720.3640.7660.4090.8310.637 Gradient boosting (high
regularization)0.0140.2680.0020.3110.0080.136 Neural
net0.0010.0010.0000.0000.0020.000

Notes: The table shows the (average) rates at which each candidate learner is
selected by the single-best final learner when using conventional stacking,
pooled stacking and short-stacking. The bootstrap sample size is
$n_{b}=9\,915$ or $99\,150$. Results are based on $1\,000$ replications. See
Table 4.1 for more information.

Table A.4: Computational time of DDML with conventional and short-stacking

Notes: The table reports the computational time in seconds of DDML paired with
conventional stacking (‘Conv.’) or short-stacking (‘Short’) as implemented in
[4], OLS as implemented in Stata’s regress, post-double-selection lasso as
implemented in pdslasso [2]. DDML uses $V=5$ cross-validation folds and $K$
cross-fitting folds as indicated. Times reported are in seconds (average over
1 000 replications). The computations were performed on the high-performance
cluster of the ETH Zurich. Each instance used a single core of an AMD EPYC
processor with 2.25-2.6GHz (nominal)/3.3-3.5 GHz (peak) and 4GB RAM.

##  Appendix B DDML and stacking in very small samples

Table B.1: Estimates based on the full sample ($N=9{\small,}915$). Estimator | Estimate
---|---
Panel A. No sample splitting
| OLS TWI | 6751.907
| OLS QSI | 5988.413
| Post double Lasso TWI c=0.5 | 6562.923
| Post double Lasso QSI c=0.5 | 5648.14
| Post double Lasso TWI c=1 | 6630.751
| Post double Lasso QSI c=1 | 4646.575
| Post double Lasso TWI c=1.5 | 7474.508
| Post double Lasso QSI c=1.5 | 4472.324

Panel B. DDML with candidate learners Neural net6433.092 OLS6463.73 Lasso with
CV (TWI)6780.161 Ridge with CV (TWI)6760.134 Lasso with CV (QSI)5722.624 Ridge
with CV (QSI)5995.346 Random forest (low regularization)6089.389 Random forest
(high regularization)6552.221 Gradient boosting (low regularization)7003.373
Gradient boosting (high regularization)7992.538 Panel C. DDML with stacking
approaches Neural net6433.092 OLS6463.73 Lasso with CV (TWI)6780.161 Ridge
with CV (TWI)6760.134 Lasso with CV (QSI)5722.624 Ridge with CV (QSI)5995.346
Random forest (low regularization)6089.389 Random forest (high
regularization)6552.221 Gradient boosting (low regularization)7003.373
Gradient boosting (high regularization)7992.538

Notes: In the case of DDML estimators, the average estimates and standard
errors are based on 50 replications. Panel A is reproduced from Table 1 in WZ.

Table B.2: Short-stacking weights using CLS

Panel B. $E[D|X]$, $K=10$ OLS.132.196.234.252.245.257.163 Neural
net.04.041.038.036.031.029.038 Lasso with CV (TWI).053.031.025.02.016.012.106
Ridge with CV (TWI).038.018.013.015.008.005.029 Lasso with CV
(QSI).173.225.25.248.25.228.413 Ridge with CV (QSI).202.124.072.06.068.0640
Random forest (low regularization).103.123.144.187.249.307.006 Random forest
(high regularization).159.129.107.09.051.031.102 Gradient boosting (low
regularization).043.046.054.047.045.041.144 Gradient boosting (high
regularization).059.065.064.046.038.0250 Panel C. $E[Y|X]$, $K=10$
OLS.122.098.066.026.003.0010 Neural net0000000 Lasso with CV
(TWI).03.022.01.013.014.0230 Ridge with CV (TWI).074.077.079.052.03.0130 Lasso
with CV (QSI).323.376.361.381.393.405.995 Ridge with CV
(QSI).239.314.379.428.478.479.005 Random forest (low
regularization).129.058.05.049.049.0440 Random forest (high
regularization).022.005.001.0010.0010 Gradient boosting (low
regularization).025.033.046.046.032.0340 Gradient boosting (high
regularization).035.016.009.004000 Panel D. $E[D|X]$, $K=10$
OLS.038.108.17.189.173.132.005 Neural net0000000 Lasso with CV
(TWI).058.032.017.011.005.003.002 Ridge with CV (TWI).06.013.009.01.002.0010
Lasso with CV (QSI).232.309.313.287.261.168.754 Ridge with CV
(QSI).242.105.032.034.05.0320 Random forest (low
regularization).079.028.011.008.004.0030 Random forest (high
regularization).185.249.256.304.344.5070 Gradient boosting (low
regularization).009.022.048.064.115.141.24 Gradient boosting (high
regularization).098.135.143.092.046.0130

Notes: The table reports the stacking weights corresponding to the DDML
stacking estimator in Figure 4.2. The stacking weights are averaged over
folds, based on 10-fold cross-fitting and shows for the estimation of $E[Y|X]$
and $E[D|X]$ in Panel A and B, respectively. See notes below Table 4.2 for
more information.

Table B.3: Mean bias in small samples based on the calibrated Monte Carlo in
Section 4.1

Candidate learners ($K=10$)
OLS-282.8-308.6255.4164.3-19.9-2833.5-2364.7-2861.6-2952.6-2600.8 Lasso with
CV (2nd order poly)-236.3-206.1277.8174.3-19.6-912.8-163.8-82.8264.4754.2
Ridge with CV (2nd order
poly)-144.7-187.2303.5151.2-22.0-1803.8-728.0122.1697.7778.0 Lasso with CV
(10th order poly)6339.9180.01258.7565.92.4-7471.8-3768.5182.84054.5-951.6
Ridge with CV (10th order
poly)5321.14716.56706.989.8290.3691.73137.6-6506.9-8867.9515.8 Random forest
(low regularization)-149.1-337.2162.378.1-110.7-274.3233.1-448.8-293.4-15.0
Random forest (high
regularization)633.6132.6454.7288.4-15.3-381.2185.4-361.3-313.4-74.3 Gradient
boosting (low
regularization)-489.1-498.4125.6113.6-53.5-539.0182.4-348.5-209.559.7 Gradient
boosting (high
regularization)-240.3-284.2291.9257.540.8-316.7329.6-159.4-32.6213.5 Meta
learners ($K=10$) Stacking: Single-
best-366.2-1414.4185.2124.6-25.2-1774.8-313.1-622.0-708.9112.6 Meta learners
($K=2$) Stacking: Single-
best1496.3-348.9-111.6-206.4-28.6-2024.5-746.4-38.9-95.9179.2

Notes: The table reports mean bias, median absolute bias (MAB) and coverage
rate of a 95% confidence interval for the listed estimators. We consider DDML
with the following individual learners: OLS with elementary covariates, CV
lasso and CV ridge with second-order polynomials and interactions, CV lasso
and CV ridge with 10th-order polynomials but no interactions, random forest
with low regularization (8 predictors considered at each leaf split, no limit
on the number of observations per node, bootstrap sample size of 70%), highly
regularized random forest (5 predictors considered at each leaf split, at
least 10 observation per node, bootstrap sample size of 70%), gradient-boosted
trees with low regularization (500 trees and a learnings rate of 0.01),
gradient-boosted trees with high regularization: 250 trees and a learnings
rate of 0.01, feed-forward neural nets with three hidden layers of size five.
For reference, we report two estimators using the full sample: OLS and PDS
lasso. We report results for four meta learners: Stacking with CLS, short-
stacking with CLS, single best overall and single best by fold. Results are
based on 1,000 replications.

Table B.4: Coverage in small samples based on the calibrated Monte Carlo in
Section 4.1

Candidate learners ($K=10$) OLS0.940.950.950.950.950.930.940.930.910.59 Lasso
with CV (2nd order poly)0.950.950.950.950.950.950.950.950.950.90 Ridge with CV
(2nd order poly)0.950.950.950.950.950.950.950.950.940.89 Lasso with CV (10th
order poly)0.900.920.930.930.950.870.850.850.880.95 Ridge with CV (10th order
poly)0.810.890.870.890.950.820.820.830.870.94 Random forest (low
regularization)0.920.920.930.920.910.930.920.930.930.91 Random forest (high
regularization)0.950.950.950.950.940.950.950.950.950.93 Gradient boosting (low
regularization)0.920.930.940.940.950.920.930.950.960.94 Gradient boosting
(high regularization)0.930.940.950.950.950.940.940.960.960.94 Meta learners
($K=10$) Stacking: Single-best0.930.940.950.950.950.940.950.950.940.94 Meta
learners ($K=2$) Stacking: Single-best0.920.950.910.940.960.920.940.940.950.92

Notes: The table reports mean bias, median absolute bias (MAB) and coverage
rate of a 95% confidence interval for the listed estimators. We consider DDML
with the following individual learners: OLS with elementary covariates, CV
lasso and CV ridge with second-order polynomials and interactions, CV lasso
and CV ridge with 10th-order polynomials but no interactions, random forest
with low regularization (8 predictors considered at each leaf split, no limit
on the number of observations per node, bootstrap sample size of 70%), highly
regularized random forest (5 predictors considered at each leaf split, at
least 10 observation per node, bootstrap sample size of 70%), gradient-boosted
trees with low regularization (500 trees and a learnings rate of 0.01),
gradient-boosted trees with high regularization: 250 trees and a learnings
rate of 0.01, feed-forward neural nets with three hidden layers of size five.
For reference, we report two estimators using the full sample: OLS and PDS
lasso. We report results for four meta learners: Stacking with CLS, short-
stacking with CLS, single best overall and single best by fold. Results are
based on 1,000 replications.

##  Appendix C Gender citation gap

Table C.1: Estimates for the citation penalty of all-female and mixed-gender
authored articles

_Notes:_ The table shows median-aggregated estimates of the gender citation
gap for all–female and mixed-gender authored articles. We show results using
both log citations and citation counts as the outcome variable. Standard
errors are robust to heteroskedasticity. See Table C for information on the
candidate learners and stacking approaches.

##  Appendix D Gender wage gap

Table D.1: Stacking weights in the gender wage gap application.

| Conventional stacking |  | Short-stacking |  | Mean-squared error
---|---|---|---|---|---
| $g_{0}(0,X)$ | $g_{0}(1,X)$ | $m_{0}(X)$ |  | $g_{0}(0,X)$ | $g_{0}(1,X)$ | $m_{0}(X)$ |  | $g_{0}(0,X)$ | $g_{0}(1,X)$ | $m_{0}(X)$
OLS/logit | 0.023 | 0.012 | 0.242 |  | 0.027 | 0.013 | 0.211 |  | 0.369 | 0.347 | 0.161
OLS/logit (simple) | 0.004 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000 |  | 0.267 | 0.204 | 0.223
CV-lasso | 0.103 | 0.136 | 0.109 |  |  0.030 | 0.076 | 0.047 |  | 0.236 | 0.178 |  0.160
CV-ridge | 0.189 |  0.040 | 0.064 |  | 0.225 | 0.024 | 0.108 |  | 0.237 |  0.180 | 0.161
CV-lasso (extended) | 0.041 | 0.157 | 0.016 |  | 0.035 | 0.266 | 0.002 |  | 0.238 |  0.180 | 0.161
CV-ridge (extended) | 0.011 |  0.040 | 0.011 |  | 0.003 | 0.024 | 0.022 |  | 0.336 | 0.194 | 0.161
Random forest 1 | 0.435 | 0.506 | 0.275 |  | 0.483 | 0.507 |  0.280 |  |  0.230 | 0.176 | 0.161
Random forest 2 |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000 |  | 0.258 |  0.190 | 0.171
Random forest 3 |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000 |  | 0.274 | 0.199 | 0.179
Gradient boosting 1 | 0.025 | 0.008 | 0.039 |  | 0.011 | 0.003 | 0.022 |  | 0.239 | 0.183 |  0.160
Gradient boosting 2 |  0.150 | 0.059 | 0.216 |  | 0.175 | 0.063 | 0.285 |  | 0.254 | 0.196 | 0.161
Neural net 1 | 0.013 | 0.022 |  0.000 |  |  0.000 |  0.000 |  0.000 |  | 0.349 | 0.263 | 0.241
Neural net 2 | 0.008 |  0.020 | 0.027 |  |  0.010 | 0.023 | 0.023 |  | 0.643 | 0.357 | 0.176

_Notes:_ The table shows weights of conventional and short-stacking along with
the mean-squared prediction error by candidate learners and by variable. The
final learner is constrained least squares. The stacking weights are averaged
over cross-fitting repetitions. Pooled stacking weights are shown in Appendix
Table LABEL:tab:gwg_weights_pooled.

Notes: The table shows pooled stacking weights for each of the considered
candidate learners. The final learner is constrained least squares. The
stacking weights are averaged over cross-fitting repetitions.

Table D.3: Stacking weights using single-best final learner.

| Conventional stacking |  | Short-stacking |  | Pooled stacking
---|---|---|---|---|---
| $g_{0}(0,X)$ | $g_{0}(1,X)$ | $m_{0}(X)$ |  | $g_{0}(0,X)$ | $g_{0}(1,X)$ | $m_{0}(X)$ |  | $g_{0}(0,X)$ | $g_{0}(1,X)$ | $m_{0}(X)$
OLS/logit |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000
OLS/logit (simple) |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000
CV-lasso |  0.060 |  0.030 |  0.790 |  |  0.000 |  0.000 |  1.000 |  |  0.000 |  0.000 |  0.400
CV-ridge |  0.070 |  0.000 |  0.040 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.100
CV-lasso (extended) |  0.020 |  0.060 |  0.030 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.100
CV-ridge (extended) |  0.000 |  0.000 |  0.010 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000
Random forest 1 |  0.850 |  0.910 |  0.020 |  |  1.000 |  1.000 |  0.000 |  |  1.000 |  1.000 |  0.000
Random forest 2 |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000
Random forest 3 |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000
Gradient boosting 1 |  0.000 |  0.000 |  0.110 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.400
Gradient boosting 2 |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000
Neural net 1 |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000
Neural net 2 |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000 |  |  0.000 |  0.000 |  0.000

_Notes:_ The table shows weights of conventional stacking, short-stacking and
pooled stacking by candidate learners and by conditional expectation function.
The stacking weights are averaged over cross-fitting repetitions.

Notes: The table reports median aggregate estimates by stacking type and final
learner. See Figure 7 for more information.

Table D.5: Median aggregate estimates for each candidate learner

Observations 4836

Notes: The table reports median aggregate estimates by candidate learner. See
Figure 7 for more information.