Testing the identification of causal effects in data

Martin Huber

University of Fribourg, Dept. of Economics

###### Abstract

This study demonstrates the existence of a testable condition for the
identification of the causal effect of a treatment on an outcome in
observational data, which relies on two sets of variables, namely observed
covariates to be controlled for and a suspected instrument. Under a causal
structure commonly found in empirical applications, the testable conditional
independence of the suspected instrument and the outcome given the treatment
and the covariates has two implications. First, the instrument is valid, i.e.
it does not directly affect the outcome (other than through the treatment) and
is unconfounded conditional on the covariates. Second, the treatment is
unconfounded conditional on the covariates such that the treatment effect is
identified. We suggest tests of this conditional independence based on doubly
robust estimators and investigate their finite sample performance in a
simulation study. We also apply our testing approach to the evaluation of the
impact of fertility on female labor supply when using the sibling sex ratio of
the first two children as supposed instrument, which by and large points to a
violation of our testable implication, at least for the moderate set of socio-
economic covariates considered.
JEL Classification: C12, C21, C26
Keywords: treatment effects, causality, conditional independence, instrument,
covariates, hypothesis test.

Address for correspondence: Martin Huber, University of Fribourg, Bd. de
Pérolles 90, 1700 Fribourg, Switzerland<EMAIL_ADDRESS>
##  1 Introduction

Methods of causal inference for assessing the effect of a treatment or policy
intervention (e.g. training participation or a marketing campaign) on an
outcome of interest (e.g. earnings or sales) typically make use of identifying
assumptions that are deemed to be untestable. For instance, the popular
selection-on-observables, unconfoundedness, conditional independence, or
ignorability assumption imposes that the treatment is as good as randomly
assigned after controlling for observed covariates, see e.g. Imbens and
Wooldridge (2009) for a review of evaluation approaches in this context.
Whether the set of covariates is sufficient for this assumption to hold is
conventionally motivated based on theory, intuition, domain knowledge, or
previous empirical findings.

In this paper, we demonstrate that there exists a testable condition for the
satisfaction of the selection-on-observables assumption in observational data
in empirically relevant causal models, which implies that identification can
be tested in the data. This condition relies on two types of observed
characteristics, namely the covariates to be controlled for and a suspected
instrument. The condition arises if the following assumptions hold: First,
there is no reverse causality from the outcome to the treatment, covariates,
or the suspected instrument and from the treatment to the covariates or the
suspected instrument. Second, the suspected instrument is statistically
associated with the treatment conditional on the covariates, e.g. through a
first stage effect of the suspected instrument on the treatment. If the
suspected instrument is conditionally independent of the outcome given the
treatment and the covariates in this context, then the instrument and the
treatment satisfy the following two conditions: (A) the instrument is valid,
i.e. does not directly affect the outcome (other than through the treatment)
and is not associated with unobservables affecting the outcome conditional on
the covariates and (B) the treatment is not associated with unobservables
affecting the outcome conditional on the covariates. The latter means that the
selection-on-observables assumption holds. Therefore, the conditional
independence of the suspected instrument, which is testable in the data,
points to the identification of treatment effects.

We construct tests for the conditional independence of the instrument by
making use of estimators conventionally applied to treatment or policy
evaluation, namely so-called doubly robust (DR) methods, see Robins et al.
(1994) and Robins and Rotnitzky (1995), which are based on statistical models
for both the instrument and the outcome. More concisely, we apply the double
machine learning (DML) approach of Chernozhukov et al. (2018), in which the
models for the instrument and the outcome are learned in a data-driven way
based on machine learning. This appears particularly attractive in high
dimensional settings where the number of available covariates is relatively
large. Under specific regularity conditions, such an estimation and testing
for violations of the conditional independence is root-n-consistent despite
the use of flexible machine learning methods. In a simulation study with 50
covariates, we find that the test performs decently in terms of empirical size
and power even under moderate sample sizes of 1000 or 4000 observations.
Following Athey and Imbens (2016), Wager and Athey (2018), Athey et al.
(2019), and Lee et al. (2020), we also discuss a machine learning-based
algorithm for detecting heterogeneity in the violations of the conditional
independence as a function of the covariates. This permits detecting subgroups
in which the violations are particularly large, in order to increase
(asymptotic) testing power.

As an empirical illustration, we apply our testing approach to US census data
previously considered in Angrist and Evans (1998) for assessing the impact of
fertility, which is the treatment, on female labor supply when using the sex
ratio of the first two children as instrument. The intuition for this
instrumental variable (IV) strategy is that if parents tend to have a
preference for mixed sex children, then having two children of the same sex,
which is arguably randomly assigned by nature, increases the chances of
getting a third child. Based on findings in Rosenzweig and Wolpin (2000) and
Lee (2008), one might, however, challenge whether all identifying assumptions
required for the IV-based identification of causal effects are satisfied.
Here, we do not impose such IV assumptions a priori, but instead use the
sibling sex ratio instrument to test the joint satisfaction of IV validity and
the selection-on-observables assumption. Our results by and large point to a
violation of the conditional independence of the instrument, at least under
our moderate set of covariates, which consists of several socio-economic
characteristics like e.g. mother’s age and father’s income.

This paper connects to several strands of the causal inference literature.
Most closely related are studies assuming an instrument that is conditionally
valid given covariates, in order to test the selection-on-observables
assumption on the treatment based on the very same condition as in this paper,
see de Luna and Johansson (2014) and Black et al. (2015). Angrist (2004),
Brinch et al. (2017), and Huber (2013) consider related tests when instrument
validity is assumed to hold unconditionally, i.e. without controlling for
covariates, in order to test the unconditional independence of the treatment
and potential outcomes. Bertanha and Imbens (2015) focus on a fuzzy regression
discontinuity design (RDD), where instrument validity is assumed to hold at a
specific threshold of a running variable which discontinuously affects
treatment assignment. All these contributions have in common that they a
priori assume some form of IV validity like in condition (A) to test treatment
exogeneity, e.g. the selection-on-observables condition (B). Here, we
highlight that (A) and (B) can be tested jointly, such that one need not
impose the existence of a valid instrument prior to testing. This is
conceptually distinct from the previous approaches, as it implies that under
some causal structure, we can test for identification in the data by checking
a condition that implies the satisfaction of both (A) and (B).

Furthermore, our approach relates to Angrist and Rokkanen (2015), who in some
sense consider exactly the opposite scenario of a treatment satisfying a
specific selection-on-observables assumption (B) in order to test whether a
further variable is a valid instrument (A). More concisely, they consider the
sharp RDD, in which the treatment is a deterministic function of a specific
threshold of a running variable and therefore by design not associated with
unobservables affecting the outcome. For instance, the admission to some
college, which serves as treatment, might be conditional on reaching a
specific threshold in the score of an entrance exam, which serves as running
variable. Because the admission treatment is exclusively by driven by staying
above or below the score’s threshold, it is unconfounded (i.e. as good as
random) conditional on the score. For this reason, effect identification in
the RDD relies on comparing the outcomes of treated and non-treated subjects
close to the threshold, i.e. with comparable values in the running variable.
However, for subjects with scores further away from the threshold, the
treatment effect is generally confounded if the running variable directly
affects the outcome or is associated with unobservables affecting the outcome
(e.g. the score is associated with ability, which affects the wage). For this
reason, Angrist and Rokkanen (2015) test whether the score is independent of
the outcome given the treatment and observed covariates. If this is the case,
the running variable is (conditional on the covariates) a valid instrument and
not a confounder. This implies that treatment effects are identified away from
the threshold, as one need not control for the running variable.

Our study is also related to Caetano et al. (2021), who consider a treatment
with multiple values and bunching at a specific value. For instance, schooling
laws may impose a minimum number of years of eduction, such that all subjects
who would otherwise have acquired a lower level of eduction bunch at this
minimum. For this reason, also the subjects’ unobserved characteristics
affecting the treatment bunch at this point, which might for instance imply a
comparably low level of ability at the minimum education requirement relative
to other educational levels. One may then verify the selection-on-observables
assumption by testing whether a dummy variable for the bunching point, which
is a function of the unobserved characteristics, is conditionally independent
of the outcome given the treatment and the covariates. Therefore, the dummy
variable at the bunching point of the unobservables has the same role as the
supposed instrument in our context, under the condition that the outcome model
as a function of the treatment and covariates is correctly specified. Indeed,
under the selection-on-observables assumption, observed instruments and
unobservables affecting the treatment share the property that they neither
directly affect the outcome, nor are associated with unobservables affecting
the outcome conditional on the treatment and the covariates.

Our method of testing identification addresses in some sense a statistical
problem ‘in between’ classical treatment evaluation, where both the treatment
of interest and the identifying assumptions are predetermined, and so-called
causal structure learning or causal discovery, see e.g. Kalisch and Bühlmann
(2014), Peters et al. (2017), Glymour et al. (2019), or Breunig and Burauel
(2021). Causal discovery does typically not predefine the treatment and
outcome variables, but aims at learning the causal relations between two or
more variables in a data-driven way, possibly under parametric assumptions or
the assumption that all relevant variables in the causal system (apart from
random error terms) are observed. Here, we do not rely on such assumptions,
but instead impose more causal structure, which permits us to distinguish the
treatment, outcome, covariates, and the supposed instrument. Such a causal
structure appears realistic in many empirical contexts with information about
the timing of measurement of the various variables. For instance, it is
obvious that a treatment taking place in an earlier period can affect and
outcome measured in a later period, but not vice versa. In contrast to
classical treatment evaluation, we do, however, not pre-impose specific
identifying assumptions, but test them in the data.

One subfield of causal discovery most closely related to our study is so-
called Y-learning, see e.g. Mani et al. (2012) and Sevilla and Mayn (2021).
Conditional on covariates, Y-learning implies that if two variables are (1)
independent of each other when not controlling for the treatment, (2)
statistically associated with each other when controlling for the treatment,
and (3) both independent of the outcome when controlling for the treatment,
then these two variables are valid instruments. A further implication is that
the selection-on-observables assumption holds, in analogy to the satisfaction
of our testable implication under a single instrument. The difference to
Y-learning is that here, we impose more causal structure by predefining a
supposed instrument that must not be causally affected by the treatment, which
holds if the instrument is measured prior to treatment assignment. In
contrast, testing identification by Y-learning hinges on detecting (at least)
two instruments in a data-driven way, rather than predefining (at least) one
instrument and ruling out reverse causality from the treatment to the
instrument.

The remainder of this study is organized as follows. Section 2 discusses a set
of identifying assumptions. The latter are used in Section 3 to derive a
testable implication of the joint satisfaction of IV validity and selection-
on-observables assumptions and thus, for the identifiability of treatment
effects. Section 4 provides a modified testable implication, i.e. conditional
mean rather than full independence of the instrument, when assuming that the
IV validity and selection-on-observables assumptions hold with respect to the
mean potential outcomes, rather than the entire distributions of potential
outcomes. Section 5 proposes a testing approach based on treatment effect
estimators, in particular doubly robust approaches that may be combined with
machine learning to control for (possibly high dimensional) covariates in a
data-driven way. Section 6 provides a simulation study analyzing the finite
sample performance of our testing approach. Section 7 presents an application
to the evaluation of the impact of fertility on female labor supply when using
the sibling sex ratio of the first two children as supposed instrument.
Section 8 concludes.

##  2 Assumptions

We are interested in the causal effect of a treatment variable $D$ on an
outcome variable $Y$, and both variables might be discretely or continuously
distributed. In our discussion of causality, we will make use of the potential
outcome framework as for instance advocated in Neyman (1923) and Rubin (1974),
and denote by $Y(d)$ the potential outcome when exogenously setting the
treatment $D$ of a subject to some value $d$ in the support of the treatment.
More generally, we will use capital and lower case letters for referring to
random variables and specific values thereof, respectively. Importantly,
denoting the potential outcome $Y(d)$ as a function of a subject’s own
treatment status $D=d$ alone implicitly imposes the assumption that someone’s
potential outcomes are not affected by the treatment status of others. This is
known as the ‘stable unit treatment value assumption’ (SUTVA), see for
instance the discussion in Rubin (1980) and Cox (1958), and is imposed
throughout this paper. Furthermore, we denote by $X$ a vector of observed
covariates to be used as control variables and by $Z$ one or several observed
instrumental variables, whose properties are yet to be defined. We will
subsequently discuss the assumptions which permit testing identification.

Our first assumption imposes some structure concerning which variables may
causally affect other variables in our framework.

######  Assumption 1 (Causal structure).

$\displaystyle D(y)=D,\quad X(d,y)=X,\textrm{ and }Z(d,y)=Z\quad\forall
d\in\mathcal{D}\textrm{ and }y\in\mathcal{Y},$

where $\mathcal{D}$ and $\mathcal{Y}$ denote the support of $D$ and $Y$,
respectively. Assumption 1 rules out a reverse causal effect of outcome $Y$ on
$D$, $X$, or $Z$. It also states that the treatment $D$ must not causally
affect $X$ or $Z$, while both $X$ and $Z$ might affect $D$, $Y$, or also each
other. Assumption 1 is satisfied in the causal framework given in Figure 1.
The latter represents the causal associations between the quasi-instrument
$Z$, treatment $D$, and outcome $Y$ by means of arrows in a directed acyclic
graph (DAG), as e.g. also considered in Pearl (2000) or Cunningham (2021). In
the DAG, $Z$ and $X$ affect $D$ and $D$ and $X$ affect $Y$. Furthermore, $X$
might affect $Z$ or vice versa, as indicated by the bidirectional causal
arrow. This scenario is in line with the practice of measuring covariates and
instruments prior to treatment assignment, which rules out reverse causality
of $D$ and $Y$ on the pre-treatment $X$ and $Z$. The DAG also includes the
unobserved terms $U$ and $V$ which affect $Y$ and $D$, respectively, with the
dashed arrows indicating that these effect cannot be observed.

Figure 1:  Causal graph satisfying Assumption 1

Our second assumption requires that any values $D$ or $Z$ might take are
observed across any values $X$ might take, which is known as common support.

######  Assumption 2 (Common support).

$\displaystyle\Pr(D=d|X)>0,\quad\Pr(Z=z|X)>0\quad\forall
d\in\mathcal{D}\textrm{ and }z\in\mathcal{Z},$

where $\mathcal{D}$ and $\mathcal{Z}$ denote the support of $D$ and $Z$,
respectively. Assuming discretely distributed treatments and instruments,
Assumption 2 implies that the conditional probabilities for any $D=d$ and
$Z=z$ given $X$, the so-called treatment and instrument propensity scores, are
larger than zero. In the case of continuously distributed treatments and/or
instruments, the propensity scores are to be replaced by conditional density
functions.

Our third assumption requires the treatment and the instrument to be
statistically associated conditional on the covariates.

######  Assumption 3 (Conditional dependence between the treatment and
instrument).

$\displaystyle D\not\\!\perp\\!\\!\\!\perp Z|X,$

where $\not\\!\perp\\!\\!\\!\perp$ denotes statistical dependence. Together
with Assumption 1, which rules out effects of $D$ on $Z$, Assumption 3 either
implies that $Z$ causally affects $D$, which is known as first stage effect in
the IV literature, or that some (unobserved) characteristics jointly affect
$Z$ and $D$ given $X$. This assumption is satisfied in Figure 1, where the
instrument has an impact on the treatment.

Our fourth assumption invokes the conditional independence of the treatment
and the potential outcomes given the covariates. This popular assumption in
the policy or treatment evaluation literature is also known as selection on
observables, exogeneity, or unconfoundedness, see e.g. Imbens (2004) and
Imbens and Wooldridge (2009) for review articles.

######  Assumption 4 (Conditional independence of the treatment).

$\displaystyle Y(d){\perp\\!\\!\\!\perp}D|X\quad\forall d\in\mathcal{D},$

where ${\perp\\!\\!\\!\perp}$ denotes statistical independence. Assumption 4
implies that conditional on covariates $X$, there exist no unobserved
confounders jointly affecting outcome $Y$ and treatment $D$.

Our fifth assumption invokes IV validity, requiring that the instrument is
conditionally independent of the outcome give the covariates $X$.

######  Assumption 5 (Conditional independence of the instrument).

$\displaystyle Y(d){\perp\\!\\!\\!\perp}Z|X\quad\forall d\in\mathcal{D}.$

Assumption 5 has two implications. First, $Z$ does not directly affect the
outcome other than through the treatment conditional on $X$, such that
potential outcome is a function of $d$ alone, rather than the instrument, too.
For this reason, an IV exclusion restriction holds such that conditional on
$X$, $Y(d,z)=Y(d,z^{\prime})=Y(d)$ for any instrument values $z$ and
$z^{\prime}$, otherwise the conditional independence would be violated.
Second, there exist no unobserved confounders jointly affecting $Y$ and $Z$
when controlling for $X$, which is analogous to Assumption 4, but now
concerning the instrument rather than the treatment. It is worth noting that
Assumption 5 alone is not sufficient for identifying causal effects based on
the instrument, like the local average treatment effect (LATE) on the
subpopulation whose treatment reacts to (or complies with) the instrument, see
Imbens and Angrist (1994) and Angrist, Imbens and Rubin (1996). In fact, the
IV-based assessment of the LATE hinges on further assumptions, like e.g. the
(conditional) monotonicity of $D$ in $Z$, which we do not consider here, as it
is not required for the proposed identification test.

To provide an example where Assumption 5 is violated, let us consider the
following nonparametric structural model for the outcome: $Y=\psi(D,X,Z,U)$,
where $\psi$ is an unknown function and $U$ denotes unobserved variables
affecting the outcome. This model implies that $Z$ directly affects the
outcome, as considered in Figure 2. In this case, the potential outcome
$Y(d)=\psi(d,X,Z,U)$ is not independent of $Z$ conditional on $X$. Let us now
alternatively assume that the outcome model corresponds to $Y=\psi(D,X,U)$,
such that $Y(d)=\psi(d,X,U)$. In this case, $Z$ does not directly affect $Y$
and Assumption (5) is satisfied if $U\bot Z|X$ holds such that the
unobservables and the instrument are conditionally independent. This is the
case in Figure 1, where $Z$ is a conditionally valid instrument for $D$, as it
neither directly shifts the outcome nor is associated with unobservables
affecting the outcome.

Figure 2:  Causal graph violating Assumption 5

##  3 Testable implication for identification

In the subsequent discussion, we use the assumptions introduced in Section 2
to derive a testable implication for the identification of causal effects.
Assuming the causal structure imposed by Assumption 1, we note that if
Assumption 4 holds, then Assumption 5 implies that

$\displaystyle Y(d){\perp\\!\\!\\!\perp}Z|D=d,X.$ (3.1)

This follows from the independence of the treatment and the potential outcomes
given $X$, see for instance Dawid (1979) for a discussion of properties of
conditional independence relations. Furthermore, as the potential outcome
$Y(d)$ corresponds to the observed outcome $Y$ conditional on $D=d$, (3.1) is
equivalent to

$\displaystyle Y{\perp\\!\\!\\!\perp}Z|D=d,X.$ (3.2)

As the conditional independence (3.2) is only based on observed (rather than
potential) variables, it generally provides a testable implication of the
joint satisfaction of Assumptions 4 and 5. ‘Generally’ means that only in very
special parametric models, (3.2) can be violated even though Assumptions 4 and
5 hold (namely when a direct effect of $Z$ on $Y$ and an association of $Z$
with unobservables affecting $Y$ exactly offset each other), while this is not
the case in general models, see the discussion in de Luna and Johansson
(2014).

Figure 3:  A violation of Assumption 4 under a satisfaction of condition (3.2)

However, Assumption 4 is not necessary for the satisfaction of (3.2) in the
special case that $Z$ and $D$ are conditionally independent given $X$, i.e.
$D{\perp\\!\\!\\!\perp}Z|X$. Also in the latter scenario, it follows that
Assumption 5 is equivalent to expression (3.1) and thus, (3.2). More formally,
$D{\perp\\!\\!\\!\perp}Z|X$ and $Y(d){\perp\\!\\!\\!\perp}Z|X$ $\Rightarrow$
$(Y(d),D){\perp\\!\\!\\!\perp}Z|X$ $\Rightarrow$
$Y(d){\perp\\!\\!\\!\perp}Z|D,X$ $\Rightarrow$
$Y{\perp\\!\\!\\!\perp}Z|D=d,X$. Figure 3 provides causal model satisfying
this statistical framework. $Z$ is not associated with $D$ given $X$, as
neither $Z$ affects $D$, nor $D$ affects $Z$ and no variables jointly affect
$D$ and $Z$ conditional on $X$. Therefore, (3.2) holds even if (4) is
violated, due to confounding by $U$ which jointly affects $D$ and $Y$.

For this reason, we additionally impose Assumption 3, i.e. conditional
dependence between $Z$ and $D$, to guarantee that (3.2) can be considered as a
condition for the joint satisfaction of Assumptions 4 and 5. Under both
Assumptions 1 and 3, (3.2) no longer holds if Assumption 4 is violated. Figure
4 provides several examples for such a causal framework, in which $Z$ affects
$D$ such that Assumption 3 holds. However, the treatment-outcome association
is confounded by unobservables $U$ given $X$. For this is reason, controlling
for $D$ introduces a spurious association between $Z$ and $U$ even conditional
on $X$. This in turn introduces a statistical dependence between $Z$ and $Y$,
because $U$ affects $Y$, which is known as collider bias, see e.g. the
discussion in Pearl (2000). This issue arises in both graphs (a) and (b),
implying that the causal effect of $D$ on $Y$ is not identified conditional on
$X$. It is not identified conditional on both $Z$ and $X$ either, as
additionally controlling for $Z$ does not tackle the confounding of the
treatment-outcome relation due to $U$. As already previously mentioned, $Z$
might be used as an instrument to identify the LATE, but this requires
additional IV assumptions like monotonicity of the treatment in the
instrument, see Imbens and Angrist (1994) and Angrist, Imbens and Rubin
(1996).

Figure 4:  No identification given $X$ or $(X,Z)$ due to $D-Y$ confounding

The previous discussion has shown that under the satisfaction of Assumptions
1, 5, and 3, a violation of Assumption 4, implying that the treatment effect
is not identified conditional on $X$, entails a violation of (3.2). Such a
violation also occurs under the satisfaction of Assumptions 1, 4, and 3, but a
violation of Assumption 5: $Y(d)\not\\!\perp\\!\\!\\!\perp Z|X$ generally
entails $Y(d)\not\\!\perp\\!\\!\\!\perp Z|D=d,X$ and thus
$Y\not\\!\perp\\!\\!\\!\perp Z|D=d,X$. Figure 5 provides two examples for such
a causal framework, in which the unobserved characteristics $V$ that affect
$D$ are not associated with $U$. While $Z$ and $Y$ is confounded by $U$ in
graphs (a) and (b) such that Assumption 5 is violated, controlling for $X$
permits identifying the causal effect of $D$ on $Y$ such that Assumption 4
holds. In this case, identification is obtained conditional on $X$ but cannot
be revealed through a satisfaction of (3.2) due to the invalidity of the
instrument.

Figure 5:  Identification given $X$ under a violation of condition (3.2)

To gain further intuition about the conditional independence (3.2), Figure 6
provides two causal scenarios in which both Assumptions 4 and 5 are violated.
In graphs (a) and (b), conditioning on $X$ alone introduces so-called M-bias,
a form of collider bias. Because $X$ is influenced by both $Z$ and $U$,
controlling for $X$ introduces a spurious association between $Z$ (which
affects $D$) and $U$ (which affects $Y$), such that the treatment-outcome
association is confounded. Nevertheless, the causal effect of $D$ is
identified based on observed variables, namely when controlling for both $Z$
and $X$. In graph (a), this approach simultaneously avoids M-bias by
conditioning on the instrument and confounding of the treatment-outcome
relation by conditioning on the covariates. In graph (b), the same strategy
additionally blocks the direct impact of $U$ on $Z$. For this reason, a
violation of (3.2) does not necessarily imply non-identification, as treatment
effects might be just identified when controlling for both $X$ and $Z$.
However, if (3.2) is violated, we cannot distinguish non-identification from
identification in a data-driven way but need to rely on untestable
assumptions.

Figure 6:  Identification given $Z$ and $X$ under a violation of condition
(3.2)

Finally, Figure 7 provides several causal graphs in which both Assumptions 4
and 5 are satisfied (in addition to Assumptions 1 and 6), such that (3.2)
holds. In all cases considered, controlling for $X$ is sufficient for the
conditional independence of both the treatment and the instrument on the one
hand and the potential outcomes on the other hand. Very much in contrast to
Figure 6, conditioning on the covariates does not introduce M-bias. Therefore,
$Z$ is conditionally independent of $Y$ given both $D$ and $X$, which
illustrates that (3.2) provides a testable condition for the identification of
the treatment effect of $D$ on $Y$ given $X$. Our discussion so far has
apparently not made use of Assumption 2, but we point out that common support
is required for verifying (3.2) and identifying causal effects across the
entire support of covariates $X$.

Figure 7:  Identification conditional on $X$ under a satisfaction of condition
(3.2)

In Theorem 1, we formalize the findings of our previous discussion, namely
that under certain conditions, the joint satisfaction of the selection-on-
observables and IV validity assumptions implies the conditional independence
of the instrument and the outcome given the treatment and the covariates.

######  Theorem 1.

Conditional on Assumptions 1, 3, and 2, the joint satisfaction of Assumptions
4 and 5 is generally necessary and sufficient for the conditional independence

$\displaystyle Y{\perp\\!\\!\\!\perp}Z|D,X.$ (3.3)

Theorem 1 shows that the conditional independence of the outcome and the
instrument is a testable condition for the causal effect of $D$ on $Y$ being
unconfounded conditional on $X$, as postulated in Assumption 4. The proof of
the theorem is provided in Appendix A.1 and is very closely related to that in
de Luna and Johansson (2014) who also consider the same testable implication,
even though their framework differs conceptually from ours. In fact, de Luna
and Johansson (2014) a priori impose Assumption 5 to test Assumption 4, i.e.
selection on observables. That is, the same testable implication as considered
in Theorem 1 also arises when imposing conditional IV validity and testing the
selection-on-observables assumption w.r.t. treatment, see also Black et al.
(2015) for a related study. Taking in some sense the opposite approach, one
may alternatively assume a treatment that is unconfounded to test IV validity,
which again entails the same testable implication. Angrist and Rokkanen (2015)
consider this approach in an RDD context to test whether the running variable
is a conditionally valid instrument. The latter implies the identification of
causal effects even away from the running variable’s threshold determining
treatment assignment, at which the treatment is locally (approximately)
unconfounded by design.

In contrast to these contributions, our Theorem 1 demonstrates that we can
jointly test Assumptions 4 and 5, i.e. the conditional independence of both
the treatment and the instrument, given that the two variables are
conditionally dependent. This implies that we need not be convinced a priori
that conditional instrument validity holds in order to test the selection-on-
observables assumption, or vice versa. By jointly testing both assumptions
together, we conceptually move away from pre-imposing one specific set of
identifying assumptions and towards verifying identification in the data. This
idea is related to the approach of Caetano et al. (2021), who test whether
conditional on the treatment and the covariates, the outcome is statistically
independent of a function of unobservables. The latter function corresponds to
a dummy variable for a specific treatment value under the conditions that
bunching (i.e. a mass point) in the treatment occurs at that value and that
the outcome model is correctly specified. In other words, the function of the
unobservables corresponds to the instrument $Z$ in Theorem 1. Also in Caetano
et al. (2021), it is a priori not clear whether the unobservables are valid
instruments (i.e. not confounders) or whether the selection-on-observables
assumption holds, but both assumptions are verified jointly by verifying the
statistical significance of the dummy variable at the bunching point.

Our approach of jointly testing several assumptions might also be applied in
the context of classical specification tests, like the Hausman (1978) test for
comparing the results from an IV regression with those of a method relying on
selection on observables, e.g. OLS. Also in this case, one may assume
conditional IV validity to test the selection-on-observables assumption or
vice versa (which is rare in practice). But in fact, both sets of assumptions
could actually be tested jointly. However, as the Hausman test relies on
comparing the causal effects obtained under IV and selection-on-observables
methods, one disadvantage is that its behavior is generally affected by a
misspecification of the models assumed for estimating the effects. Even when
using nonparametric causal models with possibly heterogeneous effects across
different subpopulations as in Donald et al. (2014), one issue is that IV-
based identification, e.g. of the LATE, relies on further identifying
assumptions like monotonicity of $D$ in $Z$. Because such functional form
restrictions compromise on the generality of the model and the robustness of
the tests, directly testing the conditional independence in Theorem 1 seems
preferable.

##  4 Conditional mean independence

The conditional independence assumptions 4 and 5 refer to the entire
distribution of potential outcomes as e.g. required for the evaluation of the
quantile treatment effect (QTE), see for instance Firpo (2007). When, however,
assessing the average treatment effect (ATE) as it is frequently the case in
empirical applications, we may consider somewhat weaker conditional
independence assumptions w.r.t. to the means (rather than the full
distributions) of potential outcomes instead of Assumptions 4 and 5 and the
testable implication (3.3). More formally, we may replace Assumption 4 by the
following condition.

######  Assumption 6 (Conditional mean independence of the treatment).

$\displaystyle E[Y(d)|D,X]=E[Y(d)|X]\quad\forall d\in\mathcal{D}.$

Assumption 6 is weaker than Assumption 4 as it only imposes the conditional
mean independence of $Y(d)$ and $D$ given $X$, but not that of other moments.
Analogously, we may also weaken Assumption 5 to a conditional mean
independence of the instrument as invoked in Assumption 7.

######  Assumption 7 (Conditional mean independence of the instrument).

$\displaystyle E[Y(d)|Z,X]=E[Y(d)|X]\quad\forall d\in\mathcal{D}.$

When considering Assumptions 6 and 7 rather than 4 and 5, then also Assumption
3 about the conditional dependence of $Z$ and $D$ needs to be replaced, namely
by a somewhat more stringent conditional mean dependence.

######  Assumption 8 (Conditional mean dependence between the treatment and
instrument).

$\displaystyle E[D|Z,X]\neq E[D|X].$

Assumption 8 imposes a nonzero first-stage effect of $Z$ in a regression of
$D$ on a constant, $Z$, and $X$. It is stronger than the previously imposed
Assumption 3, because it requires that the conditional dependence of $D$ and
$Z$ necessarily affects the mean of these variables. Any conditional
dependence w.r.t. to other moments (like the variance) is therefore irrelevant
for this assumption.

Theorem 2 formally states the conditional mean independence of outcome $Y$ and
instrument $Z$ given treatment $D$ and the covariates $X$ under the
satisfaction of Assumptions 1, 8, and 2. The proof is provided in Appendix
A.2.

######  Theorem 2.

Conditional on Assumptions 1, 2, and 8, the joint satisfaction of Assumptions
6 and 7 is generally necessary and sufficient for the conditional mean
independence of $Y$ and $Z$, such that

$\displaystyle E[Y|Z,D,X]=E[Y|D,X].$ (4.1)

##  5 Testing

We subsequently discuss possible approaches to testing the conditional mean
independence condition (4.1), which implies the identification of average
effects, under the assumption that the supposed instrument $Z$ is a scalar,
i.e. a single variable, while the covariate vector $X$ may contain several or
even many observed characteristics. Such a scenario frequently occurs in
empirical problems, where (at most) one variable seems to be a plausible
instrument, while the remaining observables might be used as control
variables. As an example, the application in Section 7 follows Angrist and
Evans (1998) and considers the sibling sex ratio, i.e. the gender composition
of the first two children, as instrument to assess the effect of fertility
(defined as having at least three children) on mother’s labor supply. A set of
observed socio-economic characteristics (like mother’s age and spouse’s
income) serves as covariates when applying the test.

To ease notation, we henceforth denote the conditional mean outcome as
$\mu(z,d,x)=E[Y|Z=z,D=d,X=x]$. Considering a discretely distributed instrument
$Z$, the conditional mean independence (4.1) is equivalent to the following
null hypothesis $H_{0}$:

$\displaystyle H_{0}:\mu(z,d,x)-\mu(z^{\prime},d,x)=0\textrm{ for all values
$z\neq z^{\prime}$, $d$, and $x$ in the support of $Z$, $D$, and $X$}.$ (5.1)

That is, the mean conditional outcome is constant across $Z$ given any value
of $D$ and $X$. For a binary instrument, (5.1) for instance corresponds to
$\mu(1,d,x)-\mu(0,d,x)=0$. For a continuous instrument $Z$, (4.1) implies the
following null hypothesis:

$\displaystyle H_{0}:\frac{\partial\mu(z,d,x)}{\partial z}=0\textrm{ for all
values $z$, $d$, and $x$ in the support of $Z$, $D$, and $X$}.$ (5.2)

That is, the first derivative of the conditional mean outcome w.r.t. to the
instrument must be zero given any value of $Z$, $D$, and $X$.

Under a linear regression model for the outcome with homogeneous effects, we
can straightforwardly test the null hypotheses (5.1) or (5.2) based on
regressing $Y$ on a constant, $Z$, $D$, and $X$ and verifying whether the
coefficient on $Z$ is statistically significantly different to zero. However,
when assuming a more flexible nonparametric model, the conditional statistical
association of $Z$ and $Y$ is allowed to be heterogeneous across different
values of $Z$, $D$, and $X$. In theory, one then might want to verify the
respective null hypothesis at all values the instrument, the treatment, and
the covariates might take, to make sure to check for any possible violation.
However, if some or all of these variables are continuously distributed, this
implies an infinite amount of hypotheses that could be tested. But even under
(mostly) discretely distributed variables, statistical power in finite samples
quickly decreases in the conditioning set as a function of the dimension and
support of $X$, i.e. the number of covariates to be controlled for and the
number of values they might take. Furthermore, testing many hypotheses
introduces a multiple hypothesis testing issue. The more hypothesis tests one
conducts, the higher is the risk that one or several of them spuriously reject
the null hypothesis in the sample, even when the null hypothesis is satisfied
for every set of conditioning variables in the population.

One way to circumvent such issues of limited finite sample power and multiple
hypothesis testing is to verify whether the null hypothesis holds on average.
For a binary instrument as the sibling sex ratio considered in our
application, this amounts to testing the following condition:

$\displaystyle H_{0}:E[\mu(1,D,X)-\mu(0,D,X)]=0.$ (5.3)

We henceforth denote this average difference in conditional means by
$\Delta=E[\mu(1,d,x)-\mu(0,d,x)]$. The latter parameter may be estimated by
methods commonly used for assessing the ATE in the treatment or policy
evaluation literature, many of which have been shown to be root-$n$-consistent
under specific regularity conditions. Such estimation approaches include
covariate or propensity score matching, see e.g. Rosenbaum and Rubin (1983)
and Rosenbaum and Rubin (1985), inverse probability weighting (IPW) by the
propensity score, see Horvitz and Thompson (1952) and Hirano et al. (2003), or
doubly robust (DR) methods, see Robins et al. (1994) and Robins and Rotnitzky
(1995), including targeted maximum likelihood estimation (TMLE) as suggested
in van der Laan and Rubin (2006). In the case of direct matching, observations
with $Z=1$ and $Z=0$ are to be matched on $D,X$. When applying propensity
score methods, one makes use of an estimate of the conditional instrument
probability or instrument propensity score, henceforth denoted by
$p(D,X)=\Pr(Z=1|D,X)$, in the matching or weighting process.

The DR approach exploits both propensity scores and conditional means to
estimate the following, alternative expression for $\Delta$, which can easily
be shown to be equivalent to $E[\mu(1,d,x)-\mu(0,d,x)]$ (by acknowledging that
$Y-\mu(Z,D,X)$ is zero in expectation):

$\displaystyle\Delta$ $\displaystyle=$ $\displaystyle
E\left[\phi(D,X)\right],$ (5.4) $\displaystyle\textrm{ with }\phi(D,X)$
$\displaystyle=$ $\displaystyle\mu(1,D,X)-\mu(0,D,X)+\frac{(Y-\mu(1,D,X))\cdot
D}{p(D,X)}-\frac{(Y-\mu(0,D,X))\cdot(1-D)}{1-p(D,X)}.$

$\phi(D,X)$ is the so-called efficient score function, into which $\mu(Z,D,X)$
and $p(D,X)$ enter as first-step or nuisance parameters. Based on the sample
analog of (5.4), one may consistently estimate $\Delta$ if either the model
for $\mu(Z,D,X)$ or for $p(D,X)$ is correctly specified, which is known as DR
property.

In particular when the set of covariates $X$ is large (i.e. high dimensional),
one might estimate $\mu(Z,D,X)$ and $p(D,X)$ by machine learning (ML)
algorithms, in order to control for important confounders in a data-driven way
(rather than controlling for them by some ad-hoc rule which may introduce pre-
testing issues). This approach is known as double machine learning (DML) and
is typically combined with so-called cross-fitting. The latter consists of
estimating the models for the nuisance parameters on the one hand and the
score function on the other hand in non-overlapping subsets of the data, with
the roles of the subsets for the estimation steps being sequentially swapped.
As no observation enters both estimation steps at the same time, cross-fitting
avoids correlations between the estimation of the models of $\mu(Z,D,X)$ and
$p(D,X)$ on the one hand and of the score function $\phi(D,X)$ on the other
hand and thus, overfitting bias. Finally, taking the sample average of the
estimated score function yields an estimate of $\Delta$, in analogy to the
population average in (5.4). Under specific regularity conditions, e.g. the
ML-based estimators of $\mu(Z,D,X)$ and $p(D,X)$ converge to the respective
true models with a rate of at least $n^{1/4}$, DML is root-$n$-consistent, see
the discussion in Chernozhukov et al. (2018).

However, verifying the condition (5.6) rather than (5.1) has its caveats. By
averaging over values of $D$ and $X$ (and $Z$, if it is non-binary) when
testing the null hypothesis, there is a risk of averaging out violations of
the conditional mean independence (4.1) such that they cannot be detected
statistically. In other words, by aiming at increasing finite sample power
through averaging, we sacrifice asymptotic power as we do not test the
hypotheses separately for distinct values of our conditioning set. To more
optimally trade off asymptotic and finite sample power, we briefly outline a
further testing approach, which borrows from the literature on estimating
conditional average treatment effects (CATE) and investigating effect
heterogeneity across observed characteristics based on ML, see e.g. Wager and
Athey (2018). Denoting the conditional mean difference $\mu(1,d,x)-\mu(0,d,x)$
by $\Delta(d,x)$, one can make use of the score function $\phi(D,X)$ for
assessing whether $\Delta(d,x)$ is heterogeneous across values of $(D,X)$ and
thus necessarily non-zero for some values of the conditioning set. This is in
analogy to the regression approaches in Semenova and Chernozhukov (2021),
Zimmert and Lechner (2019) and Fan et al. (2020).

A practical issue is that we would ideally like to focus on those variables in
the set $(D,X)$ which importantly predict the effect heterogeneity of
$\Delta(D,X)$ and thus, drive the statistical power of our test. This suggests
the use of ML for determining the crucial predictors of the estimate of
$\phi(D,X)$. However, if the ML-based detection of important predictors of
$\Delta(d,x)$ and hypothesis testing (e.g. of (5.1)) based on those predictors
proceeds in the very same data, this may entail overfitting bias. The latter
may entail spurious rejections of the null hypothesis due to a correlation of
the estimation steps of predictor selection and testing, in analogy to the
discussion in Athey and Imbens (2016). For this reason, we suggest applying a
sample splitting approach related to that in Lee et al. (2020), which avoids
such correlations.

To this end, the sample is randomly split into two non-overlapping subsamples.
In the first subsample, one applies the previously discussed DML procedure to
estimate the efficient score function, henceforth denoted by
$\hat{\phi}(D,X)$, for all observations. Still in the first subsample,
$\hat{\phi}(D,X)$ is predicted as a function of the predictors $(D,X)$ using
ML, for instance based on a so-called decision tree, as discussed in Morgan
and Sonquist (1963) and Breiman et al. (1984). A decision tree consists of
splitting the covariate space into several subsets, also known as leaves, such
that predictive power w.r.t. $\hat{\phi}(D,X)$, and thus heterogeneity in the
conditional association between $Z$ and $Y$ across subsets, is maximized. In
the second subsample, one first estimates $\hat{\phi}(D,X)$ based on DML and
then conducts the hypotheses tests conditional on the (functions of)
covariates that have been found to importantly predict the score in the first
subsample. In the case of the regression tree, this implies running the
hypothesis tests in the second subsample separately within all leaves (or
subsets of the covariate space) resulting from the tree in the first
subsample.

More formally, let $n$ denote the total sample size (i.e. the sum of
observations in the first and second subsample) and $i$ be the index of
observations in the sample, i.e. $i\in\\{1,...,n\\}$. Furthermore, denote by
$L_{m}$ a specific leaf and by $M$ the total number of leaves defined by the
regression tree in the first subsample, such that $m\in\\{1,...,M\\}$.
Finally, let $S$ be a dummy variable for belonging to the second (rather than
the first) subsample and $I\\{A\\}$ be the indicator function, which is one if
event $A$ holds and zero otherwise. This permits defining the leaf-specific
average of the estimated score functions in the second subsample, denoted by
$\hat{\Delta}(L_{m},S=1)$:

$\displaystyle\frac{\sum_{i=1}^{n}\hat{\phi}(D_{i},X_{i})\cdot S\cdot
I\\{D_{i},X_{i}\in L_{m}\\}}{\sum_{i=1}^{n}S\cdot I\\{D_{i},X_{i}\in
L_{m}\\}}$ (5.5)

We note that this average is an estimate of the average of $\Delta(D,X)$
within the respective leaf $L_{m}$, denoted by
$\Delta(L_{m})=E[\Delta(D,X)|D,X\in L_{m}]$. Therefore, hypothesis testing
amounts to verifying whether the averages (5.5) in the various leaves are
statistically significantly different to zero. Formally, the leaf-specific
null hypothesis as defined as follows:

$\displaystyle H_{0}:E[\mu(1,D,X)-\mu(0,D,X)|(D,X)\in L_{m}]=0.$ (5.6)

In practice, such hypothesis testing within leaves can easily be implemented
by running a linear regression of $\hat{\phi}(D_{i},X_{i})$ without intercept
on all indicator functions $I\\{D_{i},X_{i}\in L_{m}\\}$ for
$m\in\\{1,...,M\\}$ in the second subsample ($S=1$). The estimated
coefficients correspond to (5.5) and are under specific regularity conditions
root-n-consistent and asymptotically normally distributed with a variance
which is not affected by the fact that $\hat{\phi}(D_{i},X_{i})$ is a ML-based
estimate of the true (but unknown) score function $\phi(D_{i},X_{i})$. As
discussed in Semenova and Chernozhukov (2021), such a favorable behavior can
be attained if the ML-based estimators of the nuisance parameters converge to
the respective true models with a rate of at least $n^{1/4}$ and if the number
of leaves $M$ is small relative to the sample size $n$. In this case, we can
directly consider the t-statistics of the linear regression for testing the
null hypothesis in specific leaves. To account for multiple hypothesis testing
issues due to running the test in several leaves, we may apply a statistical
correction which controls for the expected proportion of spurious rejections
among the rejected hypotheses, the so-called false discovery rate, as e.g.
suggested Benjamini and Hochberg (1995). Alternatively, one may run an F-test
for the joint satisfaction of the null hypothesis (5.6) across all leaves
$L_{m}$.

In our application of Section 7, we use the random forest as suggested by
Breiman (1996), which averages over the predictions of many regression trees
estimated in a range of subsamples, instead of a single regression tree to
determine the most important predictors of heterogeneity in $\Delta(D,X)$. Our
motivation for this approach is that the variance of a single tree is
comparably high, implying that the tree structure and definition of leaves can
be rather unstable across samples. In contrast, averaging over many trees as
in the random forest reduces the variance. However, a disadvantage is that the
random forest does not yield any leaves that could be directly used for
constructing indicator functions $I\\{D_{i},X_{i}\in L_{m}\\}$ to be applied
as regressors in the linear regression for testing. Instead, one may consider
a measure for variable importance, which ranks the predictive power of each
element in $(D,X)$, e.g. based on how much the mean squared error in
predicting $\hat{\phi}(D_{i},X_{i})$ increases when discarding the respective
variable. Using this latter criterion, one may e.g. construct indicator
functions for the regression and hypothesis tests to follow based on the
quantiles of the most predictive variables, as we do in our application
presented in Section 7.

##  6 Simulation study

This section provides a simulation study to investigate the finite sample
behavior of our testing approach based on the following data generating
process:

$\displaystyle Y$ $\displaystyle=$ $\displaystyle D+X^{\prime}\beta+\gamma
Z+W+U,$ $\displaystyle D$ $\displaystyle=$ $\displaystyle
I\\{X^{\prime}\beta+\delta W+V>0\\},$ $\displaystyle X$ $\displaystyle\sim$
$\displaystyle N(0,\sigma^{2}_{X}),Z\sim binomial(0,5),$ $\displaystyle W$
$\displaystyle\sim$ $\displaystyle N(0,1),U\sim N(0,1),V\sim N(0,1),\textrm{
with }X,Z,W,U,V\textrm{ being independent of each other}.$

Outcome $Y$ is a linear function of $D$ (whose treatment effect is one),
covariates $X$ (for $\beta\neq 0$), the unobservables $W$ and $U$, and the
supposed instrument $Z$ if the coefficient $\gamma\neq 0$. The binary
treatment $D$ is a function of $X$ and the unobservable $V$, as well as $W$ if
coefficient $\delta\neq 0$. While the supposed instrument $Z$ is binary, the
unobserved terms $U,V,W$ are random, standard normally distributed variables
that are independent of each other, of $Z$, and of $X$. $X$ is a vector of
covariates which follow a normal distribution with a zero mean and a
covariance matrix $\sigma^{2}_{X}$ that is obtained by setting the covariance
of the $i$th and $j$th covariate in $X$ to $0.5^{|i-j|}$. $\beta$ gauges the
effects of the covariates on $Y$ and $D$, respectively, and thus, the
magnitude of confounding due to observables. The $i$th element in the
coefficient vector $\beta$ is set to $0.7/i$ for $i=1,...,p$, implying a
linear decay of covariate importance in terms of confounding.

We analyse the performance of our testing approach in $1000$ simulations under
two sample sizes of $n=1000$ and $4000$ when setting the number of covariates
to $50$. We estimate $\Delta$ based on DML with cross-fitting using the
default options of the ‘treatDML’ command in the ‘causalweight’ package by
Bodory and Huber (2018) for the statistical software R, see R Core Team
(2022). The command uses lasso regression, see Tibshirani (1996), as ML method
for estimating the nuisance parameters, i.e. linear and logit specifications
of the outcome and treatment equations (and more generally makes use of the
‘SuperLearner’ package by van der Laan et al. (2007) for selecting ML
algorithms). We note that observations whose instrument propensity scores are
close to zero, namely smaller than a threshold of $0.01$ (or 1%), are dropped
from the estimation in order to avoid an explosion of the propensity score-
based weights, which might importantly increase the variance of estimating
$\Delta$.

Table 1: Simulations

sample size | est | std | mean se | rejection rate
---|---|---|---|---
| Assumptions 4 and 5 hold ($\delta=0$, $\gamma=0$)
1000 | -0.020 | 0.097 | 0.092 | 0.071
4000 | -0.006 | 0.046 | 0.046 | 0.057
| Ass. 4 violated, Ass. 5 holds ($\delta=0.5$, $\gamma=0$)
1000 | -0.181 | 0.096 | 0.090 | 0.498
4000 | -0.172 | 0.046 | 0.045 | 0.968
| Ass. 4 holds, Ass. 5 violated ($\delta=0$, $\gamma=0.2$)
1000 | 0.180 | 0.097 | 0.092 | 0.513
4000 | 0.194 | 0.046 | 0.046 | 0.986

Notes: columns ‘est’, ‘std’, and ‘mean se’ provide the average estimate of
$\Delta$, its standard deviation, and the average of the estimated standard
error across all samples, respectively. ‘rejection rate’ gives the empirical
rejection rate when setting the level of statistical significance to 0.05 (or
5%).

Table 1 reports the simulation results. The top panel focusses on the case
that $\delta=\gamma=0$, such that both Assumptions 4 and 5 are satisfied.
Already under the smaller sample size of $n=1000$, the average estimate of
$\Delta$ (‘est’) across all simulations is close to zero, and quickly
approaches this true value as the sample size increases. Accordingly, the
empirical rejection rate amounts to 0.071 or 7.1% under the smaller sample
size, which is only somewhat higher than the nominal rate of 5% when setting
the level of statistical significance to $0.05$. Under the larger sample size
of $n=4000$, the rejection rate corresponds to 5.7% and is thus quite close to
the nominal level. Furthermore, the average standard error across all
simulations (‘mean se’) is generally quite close to the actual standard
deviation (‘std’) of DML. We also see a root-n consistent behavior in the
sense that the standard deviation of the estimator is cut by half when the
sample size is quadrupled, while the bias is close to zero.

The intermediate panel presents the results when Assumption 4 does not hold
($\delta=0.5$, $\gamma=0$). The DML estimates are substantially different to
zero, amounting to $-0.181$ and $-0.172$ for $n=1000$ and $n=4000$,
respectively. Furthermore, the test’s statistical power to detect the
violation quickly increases in the sample size, with a rejection rate
amounting to 49.8% under the lower and 96.8% under the higher sample size.
Similar conclusions apply to the violation of Assumption 5 ($\delta=0$,
$\gamma=0.2$) investigated in the lower panel of Table 1, where the rejection
rates correspond to 51.3% and 98.6% for $n=1000$ and $n=4000$, respectively.
We therefore find the empirical size and power of our test to be very decent
in the simulation scenarios considered.

##  7 Empirical application

This section provides an empirical application to US census data from Angrist
and Evans (1998), who used the sex ratio of a mother’s first two children as
instrument $Z$ in order to estimate the effect of (higher) fertility, defined
as a treatment dummy $D$ for having at least three children, on mothers’ labor
supply outcomes $Y$. The intuition for this IV strategy is that if parents
tend to have a preference for mixed sex children, then having two children of
the same sex, which is arguably randomly assigned by nature, increases the
chances of getting a third child. This implies a positive first stage effect
of $Z$, defined as a dummy for having two kids of the same sex, on $D$.
However, Rosenzweig and Wolpin (2000) argue that having mixed sex siblings may
violate IV validity by directly affecting both the marginal utility of leisure
and child rearing costs and, thus, labor supply. Furthermore, for a data set
from rural India, they also provide empirical evidence that expenses for
clothing of the third born are significantly lower if the older siblings are
of the same sex, which may also affect labor supply decisions and thus,
violate the exclusion restriction. Even though it is not clear to which extent
this issue carries over to the US data of Angrist and Evans (1998), concerns
about IV validity remain.

In addition, a further IV assumption required for the identification of the
LATE might be challenged, namely monotonicity, which states that no mother (or
family) in the population prefers two children of the same sex over mixed sex
children. Lee (2008) for instance finds that South Korean parents with one son
and one daughter are more likely to continue childbearing than parents with
two sons and it might be doubted that such cases can be fully excluded in the
US data of Angrist and Evans (1998). Here, we do not a priori impose the
satisfaction of all IV assumptions required for LATE identification, but
instead use the sibling sex ratio instrument to test the joint satisfaction of
IV validity and the selection-on-observables assumption. To this end, we
consider a subsample of the 1980 wave in Angrist and Evans (1998) from the US
Census Public Use Micro Samples, namely married white couples with mother’s
education amounting to 12 years, all in all 143,410 observations. While this
sample restriction already controls for ethnicity and mother’s education, we
in addition consider mother’s age, mother’s age at first birth, father’s age,
and father’s income as covariates $X$.

Table 2: Empirical application to sibling sex instruments

| total sample | $D=1$ | $D=0$
---|---|---|---
| est | se | pval | est | se | pval | adj | est | se | pval | adj
lasso-based DML | 0.22 | 0.11 | 0.05 | 0.08 | 0.17 | 0.66 | 0.66 | 0.31 | 0.14 | 0.03 | 0.06
causal forest | 0.25 | 0.12 | 0.03 | 0.13 | 0.18 | 0.49 | 0.49 | 0.30 | 0.15 | 0.04 | 0.09

Notes: columns ‘est’, ‘se’, ‘pval’, and ‘adj’ provide the estimates for
$\Delta$, the standard errors, the p-values, and the adjusted p-values (which
account for multiple hypothesis testing based on Benjamini and Hochberg (1995)
to control the false discovery rate), respectively. ‘given $X,D$’ provides the
results when conditioning on $X$ and $D$, while ‘given $X,D=1$’ and ‘$X,D=0$’
implies conditioning on $X$ in the treated and non-treated subsamples,
respectively.

As for the simulations in Section 6, we estimate $\Delta$ based on DML with
lasso regression as ML, using the ‘causalweight’ package by Bodory and Huber
(2018). To make the lasso regressions more flexible, we also include
interaction terms between the variables in the conditioning set as well as
squared and cubic terms of the age and income variables. As a second method
for estimating $\Delta$, we also consider the causal forest of Athey et al.
(2019), using the ‘grf’ package for R by Tibshirani et al. (2020). In the
latter case, we do not include interaction of higher order terms, as the
causal forest is inherently nonparametric, in some regard similar to kernel
regression. Furthermore, we set the number of trees in the causal forest (and
any other application of random forests) to 200. Other than this, we use the
default options of the respective R commands.

Table 2 presents the estimates of $\Delta$ in the total sample (‘est’), as
well as separately among the treated ($D=1$) and the non-treated ($D=0$),
along with the standard errors (‘se’) and p-values (‘pval’). For both lasso-
based DML and the causal forest, the estimated conditional association between
$Z$ and $Y$ of 0.22 and 0.25, respectively, are statistically significantly
different to zero at the 5% level. This suggests that at least one out of the
two assumptions of selection on observables and IV validity is violated. This
overall result is driven by the subsample of non-treated observations, in
which the estimates of $\Delta$ (amounting to 0.31 and 0.30) are again
statistically significant at the 5% level, while they are insignificant at any
conventional levels among the treated. To account for testing multiple, i.e.
two hypotheses across the treated and non-treated subsamples, we also consider
adjusted p-values (‘adj’) based on the method by Benjamini and Hochberg (1995)
to control the false discovery rate, i.e. the expected proportion of false
positives among rejected hypotheses. In this case, the estimates of $\Delta$
among the non-treated remain statistically significant at the 10% level.

Following the testing approach outlined at the end of Section 5, we also aim
at estimating $\Delta$ in subsets of the data defined by elements in $(D,X)$
that importantly predict the score function $\phi(D,X)$, and thus,
heterogeneity in $\Delta(D,X)$. To this end, we randomly split our total
sample into two subsamples and estimate the score functions based on cross-
fitting in the first subsample, using the random forest implemented in the
‘grf’ package for estimating the nuisance parameters. We then apply yet
another random forest in the first subsample for determining the importance of
the variables in $(D,X)$ for predicting the estimated scores based on the mean
squared error-criterion discussed in Section 5, using the ‘randomForest’
package by Liaw and Wiener (2002). The results suggest that father’s income is
by far the most important predictor of heterogeneity in $\Delta(D,X)$ and
thus, violations of the conditional independence in (4.1). We therefore
proceed by estimating the score functions in the second subsample and
splitting the latter into two further subsets based on the median of father’s
income. That is, the subsets consist of observations in either the lower or
the upper halves of father’s income distribution. We then run a linear
regression of the estimated score function on indicators for being in one or
the other subset without constant, with the coefficients corresponding to the
subset-specific estimates of $\Delta$. We also repeat the entire testing
approach when splitting the second subsample into four (rather than two)
subsets according to father’s income, i.e. into quarters.

Table 3: Empirical application to sibling sex instruments

father’s income | est | se | pval | F-test pval
---|---|---|---|---
in lower half | 0.05 | 0.22 | 0.81 |
in upper half | 0.46 | 0.21 | 0.03 | 0.08
in 1st (lowest) quarter | 0.02 | 0.31 | 0.94 |
in 2nd quarter | 0.09 | 0.32 | 0.78 |
in 3rd quarter | 0.60 | 0.31 | 0.05 |
in 4th quarter | 0.33 | 0.29 | 0.25 | 0.268

Notes: columns ‘est’, ‘se’, ‘pval’, and ‘adj’ provide the estimates for
$\Delta$ within subsets defined upon the quantiles of father’s income, the
respective standard errors, and the p-values. ‘F-test pval’ yields the p-value
for the joint significance across all subsets.

Table 3 reports the results of our hypothesis tests in the second subsample of
the data within the subsets defined upon father’s income. The upper panel
provides the estimates of $\Delta$ (‘est’) separately for observations in the
lower and upper halves of the income distribution. The estimate in the upper
half (0.46) is statistically significantly differen to zero at the 5% level
(see ‘pval’), while that in the lower one is close to zero and insignificant.
When running an F-test for testing the null hypothesis in both subsets
jointly, the p-values amounts to 8% (‘F-test pval’). The lower panel of Table
3 provides the estimates when splitting the second subsample into four
quarters as a function of father’s income distribution. The standard errors
(‘se’) are now considerably larger than in the upper panel and the null
hypothesis is only rejected in the third quarter at the 5% level of
statistical significance. Furthermore, the F-test is not significant at any
conventional level. A comparison of the findings in the first and second
panels illustrates the trade-off of augmenting the number of subsets, which
increases asymptotic power, i.e. the chance of detecting violations in
specific subsets, but decreases finite sample power due to fewer observations
per subset and larger standard errors.

By and large, the results of our statistical tests presented in Tables 2 and 3
suggest that (4.1) is violated, such that identification might fail. Given the
limited set of covariates in our empirical illustration, our findings are most
likely due to a violation of the selection-on-observables assumption, even
though IV validity cannot be taken for granted for the reasons discussed at
thebeginning of this section.

##  8 Conclusion

In this paper, we demonstrated the existence of a testable condition for the
identification of treatment effects, given that a set of covariates to be
controlled for and a suspected instrument are available in the data and
specific assumptions about the causal relations of the observed variables
hold. The testable condition corresponds to the conditional independence of
the suspected instrument and the outcome given the treatment and the
covariates. It at the same time implies that the suspected instrument is valid
(i.e. satisfies the exclusion restriction and is as good as random conditional
on the covariates) and that the treatment is as good as random conditional on
the covariates such that the treatment is identified given the covariates. We
proposed tests of this conditional independence based on doubly robust
treatment effect estimators and analyzed their finite sample behavior in a
simulation study, which pointed to a decent statistical power in the scenarios
considered even under moderate sample sizes. Finally, we applied our testing
approach to the evaluation of the impact of fertility on female labor supply
when using the sibling sex ratio of the first two children as supposed
instrument. We found by and large compelling statistical evidence for a
violation of our testable implication given the moderate set of socio-economic
covariates considered.

##  Appendix A Appendix

###  A.1 Proof of Theorem 1

We subsequently provide a proof for Theorem 1, which states that conditional
on Assumptions 1, 3, and 2, the joint satisfaction of Assumptions 4 and 5 is
generally necessary and sufficient for $Y{\perp\\!\\!\\!\perp}Z|D,X$.

###### Proof.

Under Assumptions 4 and 5,

$\displaystyle Y(d){\perp\\!\\!\\!\perp}D|X\textrm{ and
}Y(d){\perp\\!\\!\\!\perp}Z|X$ $\displaystyle\Rightarrow$ $\displaystyle
Y(d){\perp\\!\\!\\!\perp}(D,Z)|X$ $\displaystyle\Rightarrow$ $\displaystyle
Y(d){\perp\\!\\!\\!\perp}Z|D,X,$

where the second implication follows from the properties of conditional
independence relations as for instance discussed in Dawid (1979).
Acknowledging that the observed outcome satisfies $Y=Y(D)$ yields the
conditional independence postulated in Theorem 1. Under the common support
conditions postulated in Assumption 2, this results holds for all values of
$X$ in the population. Under the causal structure imposed in Assumption 1, the
satisfaction of Assumption 4 implies that the causal effect of $D$ on $Y$ is
unconfounded conditional $X$. Therefore, the satisfaction of Assumptions 4 and
5 is sufficient for the conditional independence (3.3) in Theorem 1,
conditional on Assumptions 1 and 2.

When additionally imposing Assumption 3, the satisfaction of Assumptions 4 and
5 is also necessary for (3.3). To see this, consider a violation of Assumption
4, implying that

$\displaystyle Y(d)\not\perp\\!\\!\\!\perp D|X$ $\displaystyle\Rightarrow$
$\displaystyle Y(d)\not\perp\\!\\!\\!\perp(D,Z)|X$ $\displaystyle\Rightarrow$
$\displaystyle Y(d)\not\perp\\!\\!\\!\perp Z|D,X,$

where the second implication holds due to the conditional dependence of $Z$
and $D$ given $X$ and even if Assumption 5 is satisfied. Likewise, under a
violation of Assumption 5,

$\displaystyle Y(d)\not\perp\\!\\!\\!\perp Z|X$ $\displaystyle\Rightarrow$
$\displaystyle Y(d)\not\perp\\!\\!\\!\perp(D,Z)|X$ $\displaystyle\Rightarrow$
$\displaystyle Y(d)\not\perp\\!\\!\\!\perp Z|D,X,$

which follows even without imposing Assumption 3 and independent of the
satisfaction of Assumption 4. For this reason, the joint satisfaction of
Assumptions 4 and 5 is both necessary and sufficient for the conditional
independence (3.3) under Assumptions 1, 3, and 2. ∎

###  A.2 Proof of Theorem 2

We subsequently provide a proof for Theorem 2, which states that conditional
on Assumptions 1, 2, and 8, the joint satisfaction of Assumptions 6 and 7 is
generally necessary and sufficient for $E[Y|Z,D,X]=E[Y|D,X]$.

###### Proof.

Under Assumptions 6 and 7,

$\displaystyle E[Y(d)|D,X]=E[Y(d)|X]\textrm{ and }E[Y(d)|Z,X]=E[Y(d)|X]$ (A.1)
$\displaystyle\Rightarrow$ $\displaystyle E[Y(d)|Z,D,X]=E[Y(d)|D,X],$

Acknowledging that the observed outcome satisfies $Y=Y(D)$ yields the
conditional independence postulated in Theorem 2. Under the common support
conditions postulated in Assumption 2, this results holds for all values of
$X$ in the population. Under the causal structure imposed in Assumption 1, the
satisfaction of Assumption 6 implies that the average effect of $D$ on $Y$ is
unconfounded conditional $X$. Therefore, the satisfaction of Assumptions 6 and
7 is sufficient for the conditional independence (4.1) in Theorem 2,
conditional on Assumptions 1 and 2.

When additionally imposing Assumption 8, the satisfaction of Assumptions 6 and
7 is also necessary for (4.1). To see this, consider a violation of Assumption
6, implying that

$\displaystyle E[Y(d)|D,X]\neq E[Y(d)|X]$ $\displaystyle\Rightarrow$
$\displaystyle E[Y(d)|Z,D,X]\neq E[Y(d)|X]$ $\displaystyle\Rightarrow$
$\displaystyle E[Y(d)|Z,D,X]\neq E[Y(d)|D,X],$

where the second implication holds due to the conditional mean dependence of
$Z$ and $D$ given $X$ and even if Assumption 7 is satisfied. Likewise, under a
violation of Assumption 7,

$\displaystyle E[Y(d)|Z,X]\neq E[Y(d)|X]$ $\displaystyle\Rightarrow$
$\displaystyle E[Y(d)|Z,D,X]\neq E[Y(d)|X]$ $\displaystyle\Rightarrow$
$\displaystyle E[Y(d)|Z,D,X]\neq E[Y(d)|D,X],$

which follows even without imposing Assumption 8 and independent of the
satisfaction of Assumption 6. For this reason, the joint satisfaction of
Assumptions 6 and 7 is both necessary and sufficient for the conditional
independence (4.1) under Assumptions 1, 2, and 8. ∎

## References

  * Angrist and Evans (1998) Angrist, J., Evans, W., 1998\.  Children and their parents labor supply: Evidence from exogeneous variation in family size.  American Economic Review 88, 450–477.
  * Angrist et al. (1996) Angrist, J., Imbens, G.W., Rubin, D., 1996.  Identification of causal effects using instrumental variables.  Journal of American Statistical Association 91, 444–472 (with discussion).
  * Angrist (2004) Angrist, J.D., 2004.  Treatment effect heterogeneity in theory and practice.  The Economic Journal 114, C52–C83.
  * Angrist and Rokkanen (2015) Angrist, J.D., Rokkanen, M., 2015\.  Wanna get away? regression discontinuity estimation of exam school effects away from the cutoff.  Journal of the American Statistical Association 110, 1331–1344.
  * Athey and Imbens (2016) Athey, S., Imbens, G., 2016\.  Recursive partitioning for heterogeneous causal effects.  Proceedings of the National Academy of Sciences 113, 7353–7360.
  * Athey et al. (2019) Athey, S., Tibshirani, J., Wager, S., 2019.  Generalized random forests.  The Annals of Statistics 47, 1148–1178.
  * Benjamini and Hochberg (1995) Benjamini, Y., Hochberg, Y., 1995\.  Controlling the false discovery rate: A practical and powerful approach to multiple testing.  Journal of the Royal Statistical Society Series B 57, 289–300.
  * Bertanha and Imbens (2015) Bertanha, M., Imbens, G., 2015\.  External validity in fuzzy regression discontinuity designs.  NBER working paper 20773 .
  * Black et al. (2015) Black, D.A., Joo, J., LaLonde, R.J., Smith, J.A., Taylor, E.J., 2015.  Simple tests for selection bias: Learning more from instrumental variables.  IZA Discussion Paper No 9346 .
  * Bodory and Huber (2018) Bodory, H., Huber, M., 2018\.  The causalweight package for causal inference in r.  SES Working Paper 493, University of Fribourg .
  * Breiman (1996) Breiman, L., 1996.  Bagging predictors.  Machine Learning 24, 123–140.
  * Breiman et al. (1984) Breiman, L., Friedman, J., Olshen, R., Stone, C., 1984\.  Classification and Regression Trees.  Wadsworth, Belmont, California.
  * Breunig and Burauel (2021) Breunig, C., Burauel, P., 2021\.  Testability of reverse causality without exogeneous variation.  arXiv preprint 2107.05936 .
  * Brinch et al. (2017) Brinch, C.N., Mogstad, M., Wiswall, M., 2017.  Beyond late with a discrete instrument.  Journal of Political Economy 125, 985 – 1039.
  * Caetano et al. (2021) Caetano, C., Caetano, G., Fe, H., Nielsen, E., 2021\.  A dummy test of identi?cation in models with bunching.  Finance and Economics Discussion Series 2021-068, Washington: Board of Governors of the Federal Reserve System .
  * Chernozhukov et al. (2018) Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., Robins, J., 2018.  Double/debiased machine learning for treatment and structural parameters.  The Econometrics Journal 21, C1–C68.
  * Cox (1958) Cox, D., 1958.  Planning of Experiments.  Wiley, New York.
  * Cunningham (2021) Cunningham, S., 2021.  Causal Inference: The Mixtape.  Yale University Press, New Haven.
  * Dawid (1979) Dawid, A., 1979.  Conditional independence in statistical theory.  Journal of the Royal Statistical Society, Series B 41, 1–31.
  * Donald et al. (2014) Donald, S.G., Hsu, Y.C., Lieli, R.P., 2014.  Testing the unconfoundedness assumption via inverse probability weighted estimators of (L)ATT.  Journal of Business & Economic Statistics 32, 395–415.
  * Fan et al. (2020) Fan, Q., Hsu, Y.C., Lieli, R.P., Zhang, Y., 2020.  Estimation of conditional average treatment effects with high-dimensional data.  forthcoming in the Journal of Business & Economic Statistics , 1–15.
  * Firpo (2007) Firpo, S., 2007.  Efficient Semiparametric Estimation of Quantile Treatment Effects.  Econometrica 75, 259–276.
  * Glymour et al. (2019) Glymour, C., Zhang, K., Spirtes, P., 2019.  Review of causal discovery methods based on graphical models.  Frontiers in Genetics 10, 1–15.
  * Hausman (1978) Hausman, J.A., 1978.  Specification tests in econometrics.  Econometrica 46, 1251–71.
  * Hirano et al. (2003) Hirano, K., Imbens, G.W., Ridder, G., 2003.  Efficient estimation of average treatment effects using the estimated propensity score.  Econometrica 71, 1161–1189.
  * Horvitz and Thompson (1952) Horvitz, D., Thompson, D., 1952\.  A generalization of sampling without replacement from a finite population.  Journal of American Statistical Association 47, 663–685.
  * Huber (2013) Huber, M., 2013.  A simple test for the ignorability of non-compliance in experiments.  Economics Letters 120, 389–391.
  * Imbens (2004) Imbens, G.W., 2004.  Nonparametric estimation of average treatment effects under exogeneity: a review.  The Review of Economics and Statistics 86, 4–29.
  * Imbens and Angrist (1994) Imbens, G.W., Angrist, J., 1994\.  Identification and estimation of local average treatment effects.  Econometrica 62, 467–475.
  * Imbens and Wooldridge (2009) Imbens, G.W., Wooldridge, J.M., 2009\.  Recent developments in the econometrics of program evaluation.  Journal of Economic Literature 47, 5–86.
  * Kalisch and Bühlmann (2014) Kalisch, M., Bühlmann, P., 2014\.  Causal structure learning and inference: A selective review.  Quality Technology & Quantitative Management 11, 3–21.
  * van der Laan and Rubin (2006) van der Laan, M., Rubin, D., 2006\.  Targeted maximum likelihood learning.  The International Journal of Biostatistics 2, 1–38.
  * van der Laan et al. (2007) van der Laan, M.J., Polley, E.C., Hubbard, A.E., 2007.  Super learner.  Statistical Applications in Genetics and Molecular Biology 6.
  * Lee (2008) Lee, J., 2008.  Sibling size and investment in childrenï¿½s education: an asian instrument.  Journal of Population Economics 21, 855–875.
  * Lee et al. (2020) Lee, K., Bargagli-Stoffi, F.J., Dominici, F., 2020.  Causal rule ensemble: Interpretable inference of heterogeneous treatment effects.  arXiv preprint 2009.09036 .
  * Liaw and Wiener (2002) Liaw, A., Wiener, M., 2002.  Classification and regression by randomforest.  R News 2, 18–22.
  * de Luna and Johansson (2014) de Luna, X., Johansson, P., 2014\.  Testing for the unconfoundedness assumption using an instrumental assumption.  Journal of Causal Inference 2, 187–199.
  * Mani et al. (2012) Mani, S., Spirtes, P.L., Cooper, G.F., 2012.  A theoretical study of y structures for causal discovery.  arXiv preprint 1206.6853 .
  * Morgan and Sonquist (1963) Morgan, J.N., Sonquist, J.A., 1963\.  Problems in the analysis of survey data, and a proposal.  Journal of the American Statistical Association 58, 415–434.
  * Neyman (1923) Neyman, J., 1923.  On the application of probability theory to agricultural experiments. essay on principles.  Statistical Science Reprint, 5, 463–480.
  * Pearl (2000) Pearl, J., 2000.  Causality: Models, Reasoning, and Inference.  Cambridge University Press, Cambridge.
  * Peters et al. (2017) Peters, J., Janzing, D., Schölkopf, B., 2017.  Elements of causal inference: foundations and learning algorithms.  The MIT Press.
  * R Core Team (2022) R Core Team, 2022.  R: A Language and Environment for Statistical Computing.  R Foundation for Statistical Computing. Vienna, Austria.  URL: https://www.R-project.org/.
  * Robins and Rotnitzky (1995) Robins, J.M., Rotnitzky, A., 1995\.  Semiparametric efficiency in multivariate regression models with missing data.  Journal of the American Statistical Association 90, 122–129.
  * Robins et al. (1994) Robins, J.M., Rotnitzky, A., Zhao, L., 1994.  Estimation of regression coefficients when some regressors are not always observed.  Journal of the American Statistical Association 90, 846–866.
  * Rosenbaum and Rubin (1983) Rosenbaum, P.R., Rubin, D.B., 1983\.  The central role of the propensity score in observational studies for causal effects.  Biometrika 70, 41–55.
  * Rosenbaum and Rubin (1985) Rosenbaum, P.R., Rubin, D.B., 1985\.  Constructing a control group using multivariate matched sampling methods that incorporate the propensity score.  The American Statistician 39, 33–38.
  * Rosenzweig and Wolpin (2000) Rosenzweig, M.R., Wolpin, K.I., 2000\.  Natural ”natural experiments” in economics.  Journal of Economic Literature 38, 827–874.
  * Rubin (1980) Rubin, D., 1980.  Comment on ’randomization analysis of experimental data: The fisher randomization test’ by d. basu.  Journal of American Statistical Association 75, 591–593.
  * Rubin (1974) Rubin, D.B., 1974.  Estimating causal effects of treatments in randomized and nonrandomized studies.  Journal of Educational Psychology 66, 688–701.
  * Semenova and Chernozhukov (2021) Semenova, V., Chernozhukov, V., 2021\.  Debiased machine learning of conditional average treatment effects and other causal functions.  The Econometrics Journal 24, 264–289.
  * Sevilla and Mayn (2021) Sevilla, J., Mayn, A., 2021\.  A conditional independence test for causality in econometrics.  arXiv preprint 2107.09765 .
  * Tibshirani et al. (2020) Tibshirani, J., Athey, S., Wager, S., 2020.  grf: Generalized random forests.  R package .
  * Tibshirani (1996) Tibshirani, R., 1996.  Regresson shrinkage and selection via the lasso.  Journal of the Royal Statistical Society 58, 267–288.
  * Wager and Athey (2018) Wager, S., Athey, S., 2018.  Estimation and inference of heterogeneous treatment effects using random forests.  Journal of the American Statistical Association 113, 1228–1242.
  * Zimmert and Lechner (2019) Zimmert, M., Lechner, M., 2019\.  Nonparametric estimation of causal heterogeneity under high-dimensional confounding.  arXiv preprint 1908.08779 .