# A Causal Inference Framework for Leveraging External Controls in Hybrid
Trials

Michael Valancius1,∗<EMAIL_ADDRESS>Herbert Pang2 Jiawen Zhu2
Stephen R Cole3 Michele Jonsson Funk3 and Michael R Kosorok1
1Department of Biostatistics  University of North Carolina at Chapel Hill
Chapel Hill  NC  U.S.A.
2Product Development Data Sciences  Genentech  South San Francisco  CA  U.S.A.
3Department of Epidemiology  University of North Carolina at Chapel Hill
Chapel Hill  NC  U.S.A

###### Abstract

We consider the challenges associated with causal inference in settings where
data from a randomized trial is augmented with control data from an external
source to improve efficiency in estimating the average treatment effect (ATE).
Through the development of a formal causal inference framework, we outline
sufficient causal assumptions about the exchangeability between the internal
and external controls to identify the ATE and establish the connection to a
novel graphical criteria. We propose estimators, review efficiency bounds,
develop an approach for efficient doubly-robust estimation even when unknown
nuisance models are estimated with flexible machine learning methods, and
demonstrate finite-sample performance through a simulation study. To
illustrate the ideas and methods, we apply the framework to a trial
investigating the effect of risdisplam on motor function in patients with
spinal muscular atrophy for which there exists an external set of control
patients from a previous trial.

###### keywords:

Double-robustness, external-control, target-trial, machine-learning

##  1 Introduction

Establishing the causal effect of a novel intervention is imperative for
making informed decisions about its adoption or approval. Randomized clinical
trials (RCTs) generate robust causal evidence by ensuring treatment assignment
is independent of baseline factors. However, some situations warrant
alternative study designs. Investigators might have access to additional
patient records under the control intervention (Yue, 2018), while rare-disease
settings face ethical or feasibility concerns with enrolling patients into
standard RCTs (Jansen-van der Weide et al., 2018; Massicotte et al., 2003). As
a motivating example, we consider a study of risdiplam’s effect on motor
functioning for patients with spinal muscular atrophy (SMA) (Mercuri et al.,
2022). While the trial’s control arm is small, patient data exists from a
historical control arm. More generally, we consider hybrid trials where study
participants are randomized to treatment via a known mechanism and externally-
collected control patient records (external controls) are available for
analysis (Baumfeld Andre et al., 2020). Amidst heightening emphasis on
utilizing real-world data (Corrigan-Curay et al., 2018) and increasing
regulatory approval of hybrid trials (Hatswell et al., 2016), we investigate
the implications of adopting these trials on our ability to estimate causal
effects. The analysis draws focus to two primary themes. (1) Randomization to
treatment is no longer (fully) under investigator control. (2) Adjustment for
imbalances due to a lack of randomization can increase dependence on
statistical models.

Our major contributions are formally characterizing the setting’s structure
and providing an analysis framework for estimating average treatment effects.
To address (1), we present a causal inference framework for rigorously
defining the target parameter, detailing assumptions, and exploring potential
sources of bias. Leveraging advances in statistical causal inference theory
(Kennedy, 2023; Chernozhukov et al., 2018), we alleviate (2) by extending
efficient estimators of the average treatment effect (ATE) to allow for
estimating nuisance functions with machine learning methods and establishing
theory for valid statistical inference.

###  1.1 Related Work

The challenges associated with using external controls in favor of a
traditional RCT have been studied for at least half of a century (Pocock,
1976; Sacks et al., 1983; Fleming, 1982), with Pocock’s 1976 criteria becoming
a frequently-referenced standard for evaluating the suitability of external
controls. More recently, others have outlined practical considerations and
potential sources of bias in hybrid trials with external controls (Ghadessi et
al., 2020; Schmidli et al., 2020; Hall et al., 2021).

Our work expands upon and provides a justification for causal inference
methods for trials with external controls. The closest work, which forms an
important basis for our discussion on estimation, uses semiparametric
efficiency theory to derive the efficient influence curve of the ATE, leading
to a doubly-robust estimator (Li et al., 2021). Propensity score methods
estimate the conditional probability of an individual receiving the treatment
and then use this estimated probability to either match external controls to
study participants (Lin et al., 2018) or inversely weight external controls
(Magaret et al., 2022), Others have focused on estimating the conditional
average treatment effect (CATE) (Zhou and Ji, 2021). A related topic is the
more general problem of causal inference when combining data from multiple
sources (Dunipace, 2022; Shi et al., 2023). An alternative approach to
incorporating external controls forgoes an explicit causal model, borrowing
information from external controls through an adaptive Bayesian prior. The
amount of borrowing adapts to heterogeneity between the internal and external
controls, with external controls whose outcome conflicts with internal
controls contributing less to the prior (Schmidli et al., 2014; Ibrahim et
al., 2015; Liu et al., 2021)

###  1.2 Outline

In this work, we develop a framework for the causal analysis of a randomized
trial incorporating external controls. We define the target parameter, outline
sufficient causal assumptions to link the parameter to the external control
data, and propose a graphical method for to help evaluate the credibility of
these assumptions. We outline three general strategies for estimating the
target parameter and demonstrate their dependence upon unknown nuisance
functions. Based upon efficiency theory, we build upon the previously proposed
doubly-robust estimator (Li et al., 2021) and prove conditions under which the
estimator is efficient and asymptotically normal even when machine learning
methods are used for the nuisance functions, an important advance that allows
for valid inference under broader data generating mechanisms. The variance
reduction and double-robustness is explored through a simulation study, and
the entire causal framework is demonstrated through a study of patients with
spinal muscular atrophy.

##  2 Setting

###  2.1 The Target Trial

In a hybrid trial design with external controls, data from two sources are
used to estimate a treatment effect (Baumfeld Andre et al., 2020). Precisely
defining this objective requires carefully answering two questions. (1) What
two populations generated the data? Differences between the populations
generating the data governs both the capacity to combine the two data sources
and the methods to do this accurately and efficiently. (2) What population’s
treatment effect? Unless the treatment effect is constant, the ATE varies
across populations.

Because randomization no longer fully describes treatment assignment, these
studies share challenges similar to those of observational studies. For this
reason, we recommend evaluating the study through the lens of a target trial
(Hernan and Robins, 2023). Briefly, a target trial is the hypothetical (fully)
randomized trial that a non-randomized study would like to emulate. Our target
trial is a fully randomized study sampling $n$ participants from the study
population, where $n$ is the number of internal and external records in the
hybrid trial (Figure 1). Adopting this perspective is useful for at least two
reasons. First, it orients the goal of the analysis as having, at a minimum,
high internal validity. Second, it forces researchers to critically assess the
suitability of a proposed external control arm in the design stages. For
example, can the same inclusion/exclusion criteria be applied to the external
control arm? Can the treatment strategy of the external control arm reasonably
resemble the same treatment strategy of the internal control arm, including
the degree of monitoring and enforcement of adherence? If the answer to these
questions is ”no,” then it might be sensible to preemptively modify the
randomized portion of the trial (and thus the target trial) or find a more
compatible external control cohort.

Figure 1: A comparison of the target trial (left) and the hybrid trial with
external controls (right). Elements from the study population and external
control population are white and gray, respectively.

###  2.2 Data Structure

Definitions of populations relevant to inference vary across contexts and
discipline; we adopt definitions commonly used within the causal inference
literature (Laan and Rose, 2011). The study sample consists of (internal)
enrolled study participants. The study population is the (possibly
hypothetical) population from which the study sample is a simple random
sample. The target population is the population of investigator interest, but
is rarely the same as the study population due to logistical barriers,
scientific considerations, or ethical concerns (Rothwell, 2006; Bell et al.,
2016). Differences in treatment effects between the study and target
populations leads to a lack of external validity; combating this challenge is
the central focus of the research disciplines of generalizability and
transportability (Cole and Stuart, 2010; Bareinboim and Pearl, 2013; Lesko et
al., 2017; Westreich et al., 2017; Dahabreh et al., 2019). Investigators also
have access to data from a second sample of individuals, the external
controls. These external controls might have come from one of several sources,
including previous clinical trials (Darras et al., 2021) and electronic health
record (EHR) systems (Carrigan et al., 2020), and it is implicitly understood
that they have been selected to satisfy the inclusion/exclusion criteria of
the trial and were administered the same control treatment as the internal
controls. These individuals are viewed as a simple random sample from another
hypothetical population, which we title the external control population. No
assumptions are made about the relationship between the external control and
study populations. Collectively, the observed data are a simple random sample
from a mixture population of the the study population and the external control
population.

With these populations defined, it is now possible to describe the structure
of the observed data. Let the outcome of interest be denoted by $Y$, the
baseline covariates shared by the external control and sample populations by
$Z$, and the binary treatment by $A$ ($a=1$ is the experimental treatment and
$a=0$ is the control treatment, possibly a placebo, active control, or other
intervention). For ease of exposition, we will refer to these dichotomous
treatments as treatment and control, respectively. $D$ is an indicator for the
data source ($d=1$ for RCT and $d=0$ for external control). The observed data
consists of $n$ independent and identically distributed samples of
$O=(Y,Z,A,D)$ from the mixture population, where the $n_{rct}$ RCT and
$n_{ec}$ external controls samples ($n_{rct}+n_{ec}=n$) have data structured
as $(Y,Z,A,D=1)$ and $(Y,Z,A=0,D=0)$, respectively. For $a\in\\{0,1\\}$, we
posit the existence of potential outcomes $Y^{a}$ that represent the outcome
that would have been observed under the the intervention to receive treatment
$A=a$ (Rubin, 2005). We make three assumptions about the RCT-design. (1)
Consistency of potential outcomes: for each individual with $D=1$ and for
$a\in\\{0,1\\}$, $Y^{a}=Y$ if $A=a$. (2) Exchangeability of potential
outcomes: for $a\in\\{0,1\\}$, $Y^{a}\perp A|D=1$. (3) Positivity of treatment
assignment: $0<\mathrm{Pr}(A=1|D=1)<1$. Furthermore, it is assumed that there
is no interference between the potential outcomes of different units. All
three of these assumptions are expected to hold for well-defined treatments in
a RCT.

###  2.3 Target Parameter

A number of different parameters might be of scientific interest, including
the ATE in the target population or a CATE. In this paper, we focus on a more
easily interpreted causal target parameter: the ATE in the study population.
Because this is frequently the treatment effect estimate of interest in a
clinical trial, this is consistent with the target trial framework previously
outlined. This treatment effect, $\tau$, is defined as the expected difference
between the outcome if the study population were assigned the treatment versus
if the study population were assigned the control: $\tau\equiv
E[Y^{1}-Y^{0}|D=1]$. When treatment effects are heterogeneous and the external
control population differs from the study population, $\tau$ may differ from
$E[Y^{1}-Y^{0}|D=0]$ or $E[Y^{1}-Y^{0}]$. Identification of these two
parameters also requires assumptions beyond those for $\tau$. Because $D=0$
implies $A=0$ and internal study participants are randomized, we can
equivalently define $\tau$ as $E[Y^{1}-Y^{0}|A=1]$, which is the average
treatment effect among the treated (ATT).

##  3 Causal Identification

It is simple to verify that, under Assumptions 1-3,
$\tau=E[Y|A=1,D=1]-E[Y|A=0,D=1]$, giving rise to the simple nonparametric
estimator of the difference of mean outcomes between the treatment and
internal control arms. Such identification is possible because, within the RCT
($D=1$), patients are randomized to treatment. However, this identification
result does not incorporate external controls, and thus no efficiency has been
gained. The incorporation of external controls comes at the expense of
randomization. Indeed, while $\mathrm{Pr}(A=1|D=1)$ is under investigator
control, $\mathrm{Pr}(A=1)=\mathrm{Pr}(A=1|D=1)\mathrm{Pr}(D=1)$, and the
assignment to $D$ is not randomized. To express $\tau$ in terms of external
control data, we make assumptions about the similarity of $Y^{0}$ between the
study and external control populations that aim to recover randomization.

Assumption 4. Mean-exchangeability of potential outcomes across populations:
There exists some $X\subset Z$ such that $E[Y^{0}|D=1,X=x]=E[Y^{0}|D=0,X=x]$
for all $x$ such that $p(x|D=1)>0$.

Assumption 5. Mean-consistency of potential outcomes among external controls:
$E[Y^{0}|D=0,X,A=0]=E[Y|D=0,X,A=0]$

Assumption 4 states that while $Y^{0}$ might systematically differ between the
study and external control population due to differences in their distribution
of $X$, once these differences are accounted for, the expectation of $Y^{0}$
is the same across populations. Investigators do not randomize individuals to
their population $D$ (and thus their treatment $A$), but once $X$ is
conditioned upon, the mean of $Y^{0}$ is independent of $D$, mimicking
randomization. Assumption 5 is related but weaker than Assumption 2 in that it
allows for inconsistencies in treatment delivery between internal and external
controls as long as these inconsistencies are not systematic.

Equipped with these causal assumptions, the target parameter $\tau$ can be
identified with the observed data. By conditioning, $\tau$ can be written as
$E[Y^{1}|D=1]-E_{X|D=1}[E[Y^{0}|X,D=1]]$. The first term, as noted earlier, is
equivalent to $E[Y|A=1]$ under Assumptions 1-3. The second term can be shown,
using Assumptions 1-5, to be equal to $E_{X|D=1}[E[Y|A=0,X]]$.

Therefore, we have that

$\tau=E[Y|A=1]-E_{X|D=1}[E[Y|A=0,X]],$ (1)

where $E[Y|A=0,X]$ is over the mixture distribution of internal and external
controls.

Alternatively, $E[Y^{0}|D=1]$ can be expressed in terms of the probability of
belonging to the study population. Letting $\pi_{d}(x)\equiv$ Pr$(D=1|X=x)$
and $\pi_{a}(x)\equiv$ Pr$(A=1|X=x,D=1)$, we have that

$\begin{split}&E_{X|D=1}[E[Y^{0}|X,D=1]]=\frac{1}{\mathrm{Pr}(D=1)}\\\ &\times
E\left[\frac{(1-A)}{(1-\pi_{a}(X))\pi_{d}(X)+(1-\pi_{d}(X))}\pi_{d}(X)Y\right].\end{split}$
(2)

The proposed identifiability criteria make no assumptions about the functional
relationship of the expected potential outcomes under experimental treatment
between the two data sources. Intuitively, no information from the external
controls is informative about the outcome under the experimental treatment
since no external controls receive the treatment.

###  3.1 Graphical Criteria

Assumptions 1-5 provide sufficient conditions under which the estimand
$E[Y|A=1]-E_{X|D=1}[E[Y|A=0,X]]$ is (causally) identified. However, the
assumptions are not constructive; a sufficient conditioning set of variables
$X$ can only be inferred from the underlying causal model. Causal directed
acyclic graphs (DAGS) present a powerful tool for allowing researchers to
transparently communicate their qualitative knowledge and assumptions about
the causal relationships among variables (Judea Pearl, 2009). More recently,
Single World Intervention Graphs (SWIGs) have unified causal DAGS and
independence statements about potential outcomes (Richardson and Robins,
2013), such as Assumption 4. One of our objectives is to use causal graphical
tools to characterize Assumption 4.

Previous work has outlined graphical conditions necessary to identify average
treatment effects in the cases of generalizability (Dahabreh et al., 2019) and
transportability (Bareinboim and Pearl, 2013). Constructing causal graphs for
this external control setting poses unique challenges. First, whereas a causal
graph is typically defined with respect to a specific population, in the
external control setting data is sampled from two, possibly distinct,
populations. Furthermore, while the typical goal of causal graphs is to verify
if $\\{Y^{0},Y^{1}\\}\perp\\!\\!\\!\perp A|X$, in the external control setting
the objective is instead to verify if $Y^{0}\perp\\!\\!\\!\perp D|X$
(Assumption 4).

Motivated by these unique characteristics, we propose a variant of SWIGs based
upon selection diagrams, which are modified causal graphs that determine the
transportability of causal relations (Bareinboim and Pearl, 2013). Let
$\mathbf{G}$ denote a causal graph shared by the study and external control
populations. A selection diagram $\mathbf{D}$ is created as follows:

  * •

Every edge in $\mathbf{G}$ is also an edge in $\mathbf{D}$;

  * •

$\mathbf{D}$ contains an extra edge, $S_{Z_{i}}\longrightarrow Z_{i}$,
whenever $p(z_{i}|v,D=1)\neq p(z_{i}|v,D=0)$ for some collection of variables
$V\subset Z\backslash Z_{i}$;

  * •

$\mathbf{D}$ contains an extra edge, $S_{Y}\longrightarrow Y$, whenever
$p(y|z,D=1)\neq p(y|z,D=0)$.

These additional $S$ variables pinpoint differences across populations in the
distribution of the $Z$ variables or in the distribution of $Y^{0}|Z$.
Importantly, $S$ variables identify differences in joint distributions, not
just marginal ones. All variables with a causal effect on $Y^{0}$ should be
included in the graph $\mathbf{G}$. See the Supporting Information for an
extended discussion and the subsequent role of effect modification. From this
selection diagram, we can create a SWIG by performing a node-split on $A$.
Then, Assumption 4 is satisfied if there exists some set of variables
$X\subset Z$ such that $Y^{0}\perp\\!\\!\\!\perp S|X$ using the rules of
d-separation. While this result is informative about which variables $X$
satisfy Assumption 4, it does not aid efficiency if $X$ completely separates
the two populations such as when $X$ contains a binary feature that only takes
on one of its values in the study population and its other value in the
external control population.

Theorem 1 Let the selection SWIG  $\mathbf{D}$ be constructed as above. Then
the following hold:

  1. 1.

If  $Y^{0}\perp S|X$,  where  $S$ is the set of selection variables, then
Assumption 4 holds, i.e.  $E[Y^{0}|X,D=1]=E[Y^{0}|X,D=1]$

  2. 2.

If  $Y^{0}\not\perp S|X$,  then there exists some two distributions compatible
with  $D$ such that Assumption 4 does not hold, even if
$p(y^{0}|z,D=1)=p(y^{0}|z,D=0)$.

As a concrete example, consider the setting in which it is hypothesized that
the variables $\\{Z_{1},Z_{2},Z_{3}\\}$ are causes of $Y^{0}$. Furthermore,
$\mathrm{Pr}(Z_{1}|D=1)\neq\mathrm{Pr}(Z_{1}|D=0)$,
$\mathrm{Pr}(z_{2}|D=1)=p(z_{2}|D=0)$,
$\mathrm{Pr}(z_{3}|D=1)=\mathrm{Pr}(z_{3}|D=0)$, and $\\{Z_{1},Z_{2},Z_{3}\\}$
are mutually independent in both populations. It follows that $Y^{0}\perp
D|Z_{1}$. If the same marginal distributions hold but $Z_{1}$ and $Z_{2}$ are
not independent and
$\mathrm{Pr}(Z_{1},Z_{2}|D=1)\neq\mathrm{Pr}(Z_{1},Z_{2}|D=0)$, we have that
$Y^{0}\not\perp D|Z_{1}$ but $Y^{0}\perp D|Z_{1},Z_{2}$. The selection SWIGs
are depicted in Figure 2.

When the causal assumptions are violated, the causal parameter $\tau$ cannot
be computed even with infinite data, a bias sometimes referred to as
systematic bias (Hernan and Robins, 2023). While sources of systematic bias in
this setting, as well as heuristics to guard against it, have been previously
proposed, the lack of a causal centering has led to criteria that can be vague
or not directly target the underlying reason for the bias (Pocock, 1976). We
encourage the viewpoint that discussions of bias should focus on the extent to
which the outlined causal assumptions are violated. Toward this goal, we
highlight how common concerns with using external controls can threaten sound
causal inference.

Geographic or temporal discrepancies between external controls patients and
the study population can result in a cause of the outcome being imbalanced
(hereafter meaning differing in distribution between the study population and
the external control population). In the selection SWIGs, this is demarcated
by the inclusion of a selection variable. It follows that the imbalance only
threatens causal inference when the variable cannot be adjusted for, such as
when it is unmeasured. Another concern, especially when the external control
sample comes from a less quality-controlled source such as an EHR system, is
measurement error. Because the external controls are all assigned the control,
any form of mis-measurement will be differential with respect to the
treatment.

Figure 2: Examples of Selection SWIGs. (a) By Theorem 1, $Y^{0}\perp D|W_{1}$.
(b) By Theorem 1, $Y^{0}\perp D|W_{1},W_{2}$. (c) $W_{1}$ and $W_{2}$ are
causes of $Y^{0}$, but only $W_{1}$ is imbalanced. Once $W_{1}$ is conditioned
on, $S$ is independent of $Y^{0}$, and therefore $W_{1}$ is sufficient for
adjustment. (d) When $W_{1}$ is unmeasured (dotted lines), causal bias is
incurred. (e) $W_{1}$ should be adjusted for, but $W^{*}_{1}$ is the mis-
measured variable available to investigators. The node $S_{W^{*}_{1}}$
pointing into $W_{1}^{*}$ represents the belief that the quality of
$W^{*}_{1}$ (how well it measures $W_{1}$) differs between external controls
and the study sample.

##  4 Estimation

In what follows, let $\hat{m}_{a}(x)$ denote an estimator of $m_{a}(x)\equiv
E[Y|X=x,A=a]$, let $\hat{\pi}_{a}(x)$ be an estimator of the treatment
propensity score $\pi_{a}(x)\equiv\mathrm{Pr}(A=1|X=x,D=1)$, and let
$\hat{\pi}_{d}(x)$ be an estimator of the study propensity score
$\pi_{d}(x)\equiv\mathrm{Pr}(D=1|X=x)$.

Under the proposed causal assumptions, the target parameter $\tau$ is
identified as the difference between two functionals: $\mu_{1}=E[Y|A=1]$ and
$\mu_{0}=E_{X|D=1}[E[Y|X,A=0]]$. Because external controls are not
administered the experimental treatment, they do not contribute to the
estimation of this quantity. $\mu_{0}$ can be more efficiently estimated using
both internal and external controls, through $\hat{m}_{0}(x)$, $\hat{\pi}(x)$,
or both. Because these functions are not the target parameter but are required
for its estimation, we refer to them as nuisance functions. When $X$ is high
dimensional or contains continuous covariates, simple nonparametric estimates
are no longer feasible, forcing the analyst to resort to modeling. Thus,
differences in the distribution between internal and external controls, from a
statistical perspective, heightens model dependency. In this section, we
outline three classes of estimators: outcome model-based estimators that model
$m_{0}(x)$, study-propensity score estimators that model $\pi_{d}(x)$, and
efficient doubly-robust estimators that model both $m_{0}(x)$ and
$\pi_{d}(x)$.

###  4.1 Outcome Based Models

The quantity $\mu_{0}$ averages the expected outcome of controls given
baseline covariates $X$ over the distribution of $X$ in the study. Because it
depends on the data only through $m_{0}(X)$ and the marginal distribution of
$X$ in the study population, we can consider a simple plug-in estimator, where
a regression model $\hat{m}_{0}(x)$ using the internal and external control
data is fit and then averaged over the empirical distribution of $X$ from the
study sample:

$\displaystyle\hat{\mu}_{0,om}=\int\hat{m}_{0}(x)dP_{n,X|D=1}=(\sum_{i=1}^{n}D_{i})^{-1}\sum_{i=1}^{n}D_{i}\hat{m}_{0}(X_{i}).$

In the causal inference literature, this estimator is frequently referred to
as standardization or g-computation (Hernan and Robins, 2023; Chatton et al.,
2020). The practical benefit of using external controls for this estimator is
a reduction in variance of the estimate of $m_{0}(x)$. To estimate $m_{0}(x)$,
one might assume a parametric model $m_{0}(x,\beta)$, where $\beta$ is a
finite dimensional parameter, and use the estimator $m_{0}(x,\hat{\beta})$.
However, this can introduce bias through over-smoothing, and therefore more
flexible (nonparametric or machine learning) methods might be desirable. The
consequences of using such estimators will be explored shortly.

###  4.2 Study Propensity Score-Based Models (IPDW)

As an alternative to modeling the outcome, $\pi_{d}(x)$ can be used to re-
weight the outcomes of the controls so that their distribution of $X$ in this
re-weighted population is the same as the treated. This idea is analogous to
propensity score weighting methods in causal inference. The usual propensity
score, $\pi(x)\equiv\mathrm{Pr}(A=1|x)$ is equal to $\pi_{a}(x)\pi_{d}(x)$.
While $\pi_{a}(x)$ is known by trial-design, $\pi_{d}(x)$ is unknown, and so
the treatment assignment mechanism across the entire sample is unknown, even
though the hybrid trial incorporates data from an RCT.

The weighting method we consider is based upon the propensity formulation for
$\mu_{0}$ (2). After estimating $\hat{\pi}_{d}(x)$ and $\hat{\pi}_{a}(x)$, we
consider the plug-in estimator
$\hat{\mu}_{0,ipdw}=n^{-1}\sum_{i=1}^{n}\hat{W}_{i}Y_{i}$, where

$\displaystyle\hat{W}_{i}=\frac{n}{n_{rct}}\frac{1-A_{i}}{(1-\hat{\pi}_{a}(X_{i}))\hat{\pi}_{d}(X_{i})+(1-\hat{\pi}_{d}(X_{i})}\hat{\pi}_{d}(X_{i}).$

While $\pi_{a}(x)$ is known through the trial design, it has been shown that
estimating it (with the known model form) can lead to improved performance in
finite-sample settings (Hahn, 1998). As before, one might assume a parametric
model indexed by $\beta$ for $\pi_{d}(x)$. However, the conditional
probability of belonging to the study population is likely a complicated
function, and therefore more flexible models might be considered to limit the
bias. Methods that handle extreme weights in $\hat{\mu}_{0,ipdw}$ or leverage
$\hat{\pi}_{d}$ for matching instead of weighting are discussed in the
Supporting Information.

###  4.3 Efficient Estimators

In the previous section, the proposed estimators incorporated a model for
either $m_{0}(x)$ or $\pi_{d}(x)$. Heuristically, more precise estimators for
$m_{0}(x)$ or $\pi_{d}(x)$ can produce a less variable estimator of $\tau$.
However, this precision through incorporating external controls comes at the
expense of additional causal assumptions (Assumptions 4-5) and an increased
dependency on modeling. Since the goal of incorporating external controls is
to increase efficiency in estimating $\tau$, a relevant aim is to understand
how much variance reduction they can provide.

Under results from the semiparametric efficiency theory literature, any
regular and asymptotically linear (RAL) estimator for $\tau$ has an asymptotic
variance of at least $\mathrm{B}_{\tau}\equiv E[\phi(O;\mathbb{P})^{2}]$,
where $\phi(O;\mathbb{P})$ is the so-called efficient influence curve for
$\tau$ (Kosorok, 2008). This provides a bound on the efficiency of any RAL
estimator. The efficient influence curve in this setting has been shown (Li et
al., 2021) to be

$\begin{split}\phi(O;\mathbb{P})&=\frac{1}{q}\bigg{[}D(m_{1}(X)-m_{0}(X)-\tau)+\frac{DA}{\pi_{a}(X)}(Y-m_{1}(X))\\\
&-W(A,D,X)(Y-m_{0}(X))\bigg{]},\end{split}$ (3)

where $q\equiv\int\pi_{d}(x)d\mathbb{P}$,
$r(X)\equiv\mathrm{Var}(Y^{0}|X,D=1)/\mathrm{Var}(Y^{0}|X,D=0)$ and where

$\displaystyle
W(A,D,X)\equiv\frac{D(1-A)\pi_{d}(X)+(1-D)\pi_{d}(X)r(X)}{\pi_{d}(X)(1-\pi_{a}(X))+(1-\pi_{d}(X))r(X)}.$

The authors showed that $\tilde{\mathrm{B}}_{\tau}-\mathrm{B}_{\tau}$, where
$\tilde{\mathrm{B}}_{\tau}$ is the efficiency bound in the model where no
external controls are incorporated (Hahn, 1998), is strictly positive as long
as there exists a nonzero measure set $A$ for which
$\mathrm{Pr}(A|D=1)\mathrm{Pr}(A|D=0)>0$. Furthermore, the difference is
greatest when there is large overlap in the distribution of $X$ between the
study and external control populations and when the noise in the distribution
of $Y^{0}|D=0,X$ is small.

We consider two types of estimators based upon the form of $\phi(O;P)$, where
we write $\phi$ as a function of a distribution $P$ to highlight that it
depends on the nuisance functions $m_{1}(x)$, $m_{0}(x)$, $\pi_{d}(x)$, and
$r(x)$ as well as the target parameter $\tau$. If these functions were known,
then the estimator $n^{-1}\sum_{i=1}^{n}\tilde{\phi}(O_{i},\mathbb{P})$ would
achieve the efficiency bound. Therefore, the first influence-curve based
estimator, as proposed by (Li et al., 2021) and based upon an estimating
equation, plugs in fitted models for the nuisance functions:

$\begin{split}\hat{\tau}_{aipw}&=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{\hat{q}}\bigg{[}D_{i}(\hat{m}_{1}(X_{i})-\hat{m}_{0}(X_{i}))\\\
&+\frac{D_{i}A_{i}}{\hat{\pi}_{a}(X_{i})}(Y_{i}-\hat{m}_{1}(X_{i}))-\hat{W}(A_{i},D_{i},X_{i})(Y_{i}-\hat{m}_{0}(X_{i}))\bigg{]},\end{split}$

where $\hat{q}=\frac{n_{rct}}{n}$ and

$\displaystyle\hat{W}(A,D,X)=\frac{D(1-A)\hat{\pi}_{d}(X)+(1-D)\hat{\pi}_{d}(X)\hat{r}(X)}{\hat{\pi}_{d}(X)(1-\hat{\pi}_{a}(X))+(1-\hat{\pi}_{d}(X))\hat{r}(X)}.$

An alternative strategy is through targeted maximum likelihood estimation
(TMLE) (Laan and Rose, 2011). While the two approaches here produce
asymptotically equivalent estimators, TMLE is a plug-in estimator and thus
respects the boundaries of the parameter space for all sample sizes,
potentially leading to improved performance in small sample sizes. First,
initial models $\hat{m}_{1}(x)$ and $\hat{m}_{0}(x)$ are fit to the data.
Instead of using these models to form a simple plug-in estimator (in a fashion
similar to $\hat{\mu}_{0,om}$), the models are fluctuated so that
$n^{-1}\sum_{i=1}^{n}\phi(O_{i},\hat{\mathbb{P}}^{*})=0$, implying that it,
just like $\hat{\tau}_{aipw}$, solves the efficient influence curve estimating
equation. Then, the TMLE estimator, $\hat{\tau}_{tmle}$ is just the plug-in
estimator using the updated models $\hat{m}_{1}^{*}(x)$ and
$\hat{m}_{0}^{*}(x)$:
$\hat{\tau}_{tmle}=n^{-1}\sum_{i=1}^{n}\big{(}\hat{m}_{1}^{*}(X_{i})-\hat{m}_{0}^{*}(X_{i})\big{)}$.

###  4.4 Inference

When $\hat{m}_{0}$ is consistent for $m_{0}$, then $\hat{\mu}_{0,om}$ is
consistent for $\mu_{0}$. Furthermore, when $\hat{\pi}_{d}(x)$ is consistent
for $\pi_{d}(x)$, then $\hat{\mu}_{0,ipdw}$ is consistent for $\mu_{0}$. While
this would seemingly encourage using flexible, data-adaptive methods to avoid
misspecifying $m_{0}(x)$ and $\pi_{d}(x)$, doing so generally leads to slow
rates of convergence and a poorly understood limiting distribution for
$\hat{\tau}$. Under correctly specified parametric models (or more generally,
models that belongs to a Donsker class (Kosorok, 2008)), the nonparametric
bootstrap can be used to construct confidence intervals for $\tau$ for either
the outcome-modeling or IPDW approaches.

Proposition 2 Let $\hat{m}_{0}(x)$ and $\hat{\pi}_{d}(x)$ belong to a Donsker
class $\mathcal{F}$ and assume that
$\int(\hat{m}_{0}(x)-m_{0}(x))^{2}dP_{x}=o_{p}(1)$ and
$\int(\hat{\pi}_{d}(x)-\pi_{d}(x))^{2}dP_{x}=o_{p}(1)$. Then the nonparametric
bootstrap using outcome modeling or the IPDW approach is valid for $\mu_{0}$.

The benefit of the outcome modeling and IPDW approaches is their relative
simplicity and agnosticism about how $E[Y^{1}|D-1]$ is modeled. However, these
methods are not efficient and face a trade-off between reducing bias through
using flexible models and maintaining valid $\sqrt{n}$ inference. This
motivates the usage of $\hat{\tau}_{aipw}$ and $\hat{\tau}_{tmle}$, which
possess several desirable statistical properties. First, the models are
doubly-robust in the sense that $\hat{\tau}$ converges in probability to
$\tau$ if either $\hat{m}_{0}(x)$ converges in probability to $m_{0}(x)$ and
$\hat{m}_{1}(x)$ converges in probability to $m_{1}(x)$ or if
$\hat{\pi}_{d}(x)$ converges in probability to $\pi_{d}(x)$ (Li et al., 2021).
While this double-robustness is an attractive property, neither nuisance
function is known in advance, leading to the plausible scenario in which
parametric assumptions are too strong for both models. A more attractive
property that we demonstrate is that, when cross-fitting is used,
$\hat{\tau}_{aipw}$ and $\hat{\tau}_{tmle}$ are both efficient and
asymptotically normally distributed even when the nuisance functions are
estimated at slower rates, allowing for flexible estimation of these functions
using popular machine learning methods that can limit bias.

Theorem 2 Suppose that the regularity conditions described in the Appendix
hold. Then for $\hat{\tau}_{dr}$, meaning either $\hat{\tau}_{aipw}$ or
$\hat{\tau}_{tmle}$, we have that
$\sqrt{n}(\hat{\tau}_{dr}-\tau)\longrightarrow N(0,E[\phi(O,\mathbb{P})])$
Therefore, $\hat{\tau}_{dr}$ is root-n consistent, semiparametric efficient,
and asymptotically normal with asymptotically-valid confidence intervals given
by $\hat{\tau}_{dr}\pm
1.96\sqrt{\hat{\mathrm{var}}(\phi(O,\hat{\mathbb{P}}))/n}$

##  5 Analysis Pipeline

We envision an analysis pipeline for hybrid trials utilizing external controls
based upon this causal inference framework. First, the target trial is
defined, with precise definitions for the study population, inclusion
criteria, and intervention of interest. Then, investigators encode their
assumptions about the causal model and distributional differences between the
two populations through a selection SWIG. Using the rules of d-separation on
the selection SWIG, a set of variables $X$ can be determined that are
sufficient to satisfy assumptions 4-5. An efficient estimator of the ATE can
be constructed using either $\hat{\tau}_{aipw}$ or $\hat{\tau}_{tmle}$, with
confidence intervals constructed as per Theorem 2.

While the selection SWIG provides a theoretical justification for Assumptions
4-5 based upon background knowledge of the causal process, empirical evidence
can help weigh the validity of the assumptions. Assumption 4, when combined
with the consistency assumptions, implies that
$E[Y|A=0,D=1,X]=E[Y|A=0,D=0,X]$, a so-called ”testable implication” of the
causal assumptions. This is an assumption that two (nonparametric) regression
functions, $m_{0}(x)$ and $m_{1}(x)$, are equal to each other (or,
equivalently, that the regression function does not depend upon $D$). There is
an expansive literature on testing this hypothesis, although many focus on the
settings in which the dimension of $X$ is one, $X\in[0,1]$, or design points
for $X$ are chosen in advance. We refer readers to (Racine et al., 2006),
(Lavergne et al., 2015), or (Luedtke et al., 2019) for examples of applicable
tests.

Alternatively, the study propensity score can be used to diagnose violations
of the causal assumptions and to assess the degree to which the external
controls differ from the study sample (and thus how dependent the results are
upon modeling). When Assumption 4 holds, we expect that
$E[Y|D,\pi_{d}(X),A=0]=E[Y|\pi_{d}(X),A=0]$. A simple diagnostic is therefore
to compare the mean outcomes between internal and external controls with
similar estimated propensity scores. Dissimilar mean outcomes between the two
groups would suggest a violation of the causal assumptions.

Greater distributional difference in $X$ between the internal and external
controls results in heightened reliance upon the models for $m_{0}(x)$ and
$\pi_{d}(x)$. A comparison of study propensity scores can be used to determine
if the external controls are too different from the internal controls to
reliably increase efficiency in estimation. In a parallel to methods used for
generalizability, we define the propensity score difference as
$\Delta_{\pi_{d}}=n_{rct}^{-1}\sum_{i=1}^{n}D_{i}\hat{\pi}_{d}(X_{i})-n_{ec}^{-1}\sum_{i=1}^{n}(1-D_{i})\hat{\pi}_{d}(X_{i})$.
When the external control population is the same as the study population, we
expect this difference to be 0. Larger values indicate greater differences
between the two populations. Related research in the context of using
propensity scores in observational studies suggested that
$\Delta_{\pi_{d}}>0.25$ is suggestive of an over-dependence on modeling
(Stuart, 2010).

##  6 Simulation Study

Table 1: Results (bias, mean-squared error, and 95% confidence interval
coverage) of the simulation. In Setting 1, all nuisance functions are
estimated with correctly specified linear models. In Setting 2, an incorrectly
specified linear model is used for $\pi_{d}(x)$ while Random Forests are used
for $m_{a}(x)$. In Setting 3, incorrectly specified linear models are used for
$m_{a}(x)$ while a Random Forest is used for $\pi_{d}(x)$. In Setting 4,
incorrectly specified linear models are used for both $m_{a}(x)$ and
$\pi_{d}(x)$. Both $m_{a}(x)$ and $\pi_{d}(x)$ are modeled with Random Forests
in Setting 5

| $\hat{\tau}_{rct}$ | $\hat{\tau}_{om}$ | $\hat{\tau}_{ipdw}$ | $\hat{\tau}_{aipw}$ | $\hat{\tau}_{tmle}$
---|---|---|---|---|---
| Bias | MSE | Cov. | Bias | MSE | Cov. | Bias | MSE | Cov. | Bias | MSE | Cov. | Bias | MSE | Cov.
Setting 1 | -1.8e-03 | 0.31 | 0.96 | 2.5e-04 | 0.22 | 0.93 | -2.9e-03 | 0.24 | 0.93 | -4.2e-04 | 0.23 | 0.95 | 6.6e-06 | 0.22 | 0.96
Setting 2 | 3.4e-02 | 0.31 | 0.95 |  | — | — | 4.0e-01 | 0.38 | 0.86 | 2.0e-01 | 0.25 | 0.94 | 2.2e-01 | 0.25 | 0.94
Setting 3 | 3.6e-02 | 0.34 | 0.95 | 4.1e-01 | 0.38 | 0.85 | — | — | — | 1.1e-01 | 0.24 | 0.96 | 6.8e-02 | 0.24 | 0.96
Setting 4 | 2.5e-02 | 0.34 | 0.96 | 4.0e-01 | 0.39 | 0.85 | 4.0e-01 | 0.39 | 0.86 | 4.2e-01 | 0.42 | 0.86 | 4.1e-01 | 0.40 | 0.88
Setting 5 | -4.7e-03 | 0.32 | 0.95 | 2.2e-01 | 0.24 | 0.93 | 4.8e-02 | 0.22 | 0.94 | 5.8e-02 | 0.21 | 0.96 | 6.3e-02 | 0.21 | 0.96

We perform a simulation study examining sensitivity to model mis-
specification, confidence interval coverage, and power. Consistent with our
SMA application, we simulate a trial with a continuous endpoint, 150 RCT
patients randomized 2:1 to treatment, and 50 external controls. Full details
of the data generation and models specifications are available in the
Supporting Information.

To estimate $\tau$, we compare an RCT-only covariate-adjusted AIPW estimator
($\hat{\tau}_{rct}$) with the four proposed estimators
($\hat{\tau}_{om}\equiv\hat{\mu}_{1}-\hat{\mu}_{0,om}$,
$\hat{\tau}_{ipdw}\equiv\hat{\mu}_{1}-\hat{\mu}_{0,ipdw}$,
$\hat{\tau}_{aipw}$, and $\hat{\tau}_{tmle}$). Both doubly-robust approaches
correspond to their cross-fit variants. During the simulations, 95% confidence
intervals were constructed for each method. For $\hat{\tau}_{om}$ and
$\hat{\tau}_{ipdw}$, confidence intervals were constructed via the non-
parametric bootstrap. Confidence intervals were generated for
$\hat{\tau}_{aipw}$ and $\hat{\tau}_{tmle}$ using the closed-form formula from
Theorem 1. All simulations are run for 1000 replicates.

First, we explore the role of model misspecification on bias, mean squared
error, and confidence interval coverage via different combinations of
correctly and incorrectly specified models for the nuisance functions (Table
1). With correctly specified parametric models (Setting 1), all external
control approaches are unbiased and exhibit less variability. However,
performance gains are not guaranteed under departures from correctly-specified
parametric models. In Settings 2 and 3 where one nuisance function is
incorrectly specified, the singly-robust approaches ($\hat{\tau}_{om}$ and
$\hat{\tau}_{ipdw}$) are biased. Conversely, the doubly-robust approaches
(with one nuisance function estimated flexibly and the other missecified) have
small degrees of bias but lower MSEs than $\hat{\tau}_{rct}$. Furthermore,
while double robustness offers no protection against bias when both nuisance
functions are incorrectly specified (Setting 4), the data-adaptive estimation
of both functions (Setting 5) leads to estimates with minimal bias and
variability, highlighting the inferential gains that are possible through
flexibly modeling the nuisance functions.

Next, to highlight applicability to clinical trials, we simulate power under
varying degrees of treatment effect sizes and external control sample sizes
(Figure 3). We replicate Setting 5, modeling the (unknown) nuisance functions
with Random Forests. Results are only depicted for $\hat{\tau}_{aipw}$ and
$\hat{\tau}_{tmle}$ since valid confidence intervals are not generally
attainable in this setting for $\hat{\tau}_{om}$ and $\hat{\tau}_{ipdw}$. When
$\tau=0$, both external control methods achieve satisfactory type-1 error
control ($\leq\alpha=5\%$). Across treatment effect sizes, power gains
compared to $\hat{\tau}_{rct}$ ranged from 13% to 65%. Furthermore, while our
SMA application only has a pool of $50$ external controls, more power gains
are possible with larger sample sizes.

Figure 3: (A) Power across different treatment effect sizes (points for
$\hat{\tau}_{aipw}$ and $\hat{\tau}_{tmle}$ are jittered due to overlapping
values). The dashed red line corresponds to the significance level of $0.05$
to demonstrate Type-1 error control under the null hypothesis. (B) Power
across different external control sample sizes (with RCT sample sizes and
treatment effect size held fixed). While the SMA application has an external
control sample size around 50, other studies might have access to larger
external control groups.

##  7 SMA Example

We demonstrate an implementation of the outlined causal inference framework
for external controls in an application to a study of risdiplam for patients
with spinal muscular atrophy (SMA). SUNFISH (NCT02908685) (Mercuri et al.,
2022) was a two-part multi-site randomized placebo-controlled trial designed
to investigate the efficacy of risdiplam. In Part 2 of the study, 180 patients
with Type 2 and non-ambulant Type 3 SMA were randomized 2:1 to receive
risdiplam or control; the primary endpoint of interest was the change in Motor
Function Measure (MFM) at 12 months ($\Delta_{MFM}$). Further details of the
trial are provided in the Supporting Information. We explore a hybrid trial
design that would incorporate external controls from the placebo arm of a
Phase 2 trial of olesoxime (NCT01302600). After restricting to complete cases
with Type 2 or non-ambulant Type 3 SMA between the ages of 2-25, the RCT and
external control samples contain 159 and 48 observations, respectively. The
target parameter of interest is defined as
$\tau=E[\Delta_{MFM}^{1}|D=1]-E[\Delta_{MFM}^{0}|D=1]$. We are interested in
testing $H_{0}:\tau=0$ versus $H_{1}:\tau\neq 0$. We hypothesize the causal
model depicted in Figure 4 where no assumptions are made about differences in
distribution of the covariates between the RCT and the external control
populations. Based upon this selection diagram,
$E[\Delta_{MFM}^{0}|D,X]=E[\Delta_{MFM}^{0}|X]$, where $X=($Age, SMA Type,
Scoliosis, MFM${}_{0})$.

Figure 4: (a) Selection Diagram for the SMA hybrid trial. MFM0 and MFM12
correspond to the MFM scores at baseline and 12 months, respectively. Type
refers to the SMA Type and A is the treatment (risdiplam). (b) Diagnostic of
Assumptions 4-5. Buckets were created based on $\hat{\pi}_{d}(x)$ in
increments of 0.05. Error bars correspond to 95% confidence intervals

We estimate $\tau$ with just the RCT data ($\hat{\tau}_{rct}$) using AIPW and
compare results to the four proposed estimators: $\hat{\tau}_{om}$,
$\hat{\tau}_{ipdw}$, $\hat{\tau}_{aipw}$, and $\hat{\tau}_{tmle}$. For the
doubly-robust methods, the nuisance functions are fit using Random Forests and
cross-fitting while linear models are used for $\hat{\tau}_{om}$ and
$\hat{\tau}_{ipdw}$. Confidence intervals are based upon the nonparametric
bootstrap for $\hat{\tau}_{om}$ and $\hat{\tau}_{ipdw}$ and from the closed-
form formula for $\hat{\tau}_{aipw}$ and $\hat{\tau}_{tmle}$. All five methods
provide similar estimates for $\tau$ and reject $H_{0}$. Overall, the
confidence intervals for the four methods incorporating external controls have
widths between 80% and 93% the width of the confidence interval using just the
RCT data.

Table 2: Estimates of the causal effect for the SMA application. Method | $\hat{\tau}$ | 95% CI | p-value
---|---|---|---
$\hat{\tau}_{rct}$ | 2.33 | (0.82, 3.85) | 0.002
$\hat{\tau}_{om}$ | 1.93 | (0.71, 3.13) |  $<$ 0.001
$\hat{\tau}_{ipdw}$ | 1.79 | (0.54, 3.00) | 0.002
$\hat{\tau}_{aipw}$ | 1.92 | (0.66, 3.18) | 0.003
$\hat{\tau}_{tmle}$ | 1.80 | (0.54, 3.06) | 0.005

To explore possible violations of Assumption 5, we plot (Figure 4) the means
of $\Delta_{MFM}$ for controls grouped by $D$ and binning by
$\hat{\pi}_{d}(x)$. Under Assumptions 5-6, we would expect that
$E[\Delta_{MFM}|D=1,A=0,\pi_{d}(x)]=E[\Delta_{MFM}|D=0,A=0,\pi_{d}(x)]$.
Overall, the results appear consistent with the assumptions.

##  8 Discussion

While it is well-understood that deviations in characteristics and protocols
between the RCT and external controls pose significant challenges to
synthesizing information to estimate causal effects, the absence of a causal
inference framework has hindered the clear communication of relevant concepts.
The (causal) target parameter is rarely defined ((Li et al., 2021; Zhou and
Ji, 2021) are notable exceptions) and discussions of bias center on practical
heuristics (Pocock, 1976; Viele et al., 2014). Our proposed causal framework
gives rise to an easy-to-interpret target parameter, encourages the embedding
of assumptions and investigator knowledge into a causal graph, and allows bias
to be explicitly defined as a property of the causal model or violation of the
assumptions.

A limitation is the dependency upon Assumptions 4-5 to identify the causal
effect, which can only be assessed after data collection and for which there
might be little power to test empirically due to the small sample sizes
commonplace in this setting. Furthermore, a null hypothesis of no difference
might be of less interest than developing bounds based on the severity of the
violations. While the methods discussed are applicable to continuous and
binary outcomes, the extension to survival data would be an important
contribution.

## Acknowledgements

This work is partially supported by a grant from the FDA (NIH U01 FD007206,
Kosorok, Pang, Valancius, Zhu) and a grant from the NIH (R01AI157758, Cole and
Kosorok). We would like to thank our colleagues Tammy McIver and Wai Yin
(Winnie) Yeung for their great comments and feedback.

## References

  * Bareinboim and Pearl (2013) Bareinboim, E. and Pearl, J. (2013).  A General Algorithm for Deciding Transportability of Experimental Results1, 107–134.
  * Baumfeld Andre et al. (2020) Baumfeld Andre, E., Reynolds, R., Caubel, P., Azoulay, L., and Dreyer, N. A. (2020).  Trial designs using real-world data: The changing landscape of the regulatory approval process.  Pharmacoepidemiology and Drug Safety 29, 1201–1212.
  * Bell et al. (2016) Bell, S. H., Olsen, R. B., Orr, L. L., and Stuart, E. A. (2016).  Estimates of External Validity Bias When Impact Evaluations Select Sites Nonrandomly.  Educational Evaluation and Policy Analysis 38, 318–335.  Publisher: American Educational Research Association.
  * Carrigan et al. (2020) Carrigan, G., Whipple, S., Capra, W. B., Taylor, M. D., Brown, J. S., Lu, M., Arnieri, B., Copping, R., and Rothman, K. J. (2020).  Using Electronic Health Records to Derive Control Arms for Early Phase Single-Arm Lung Cancer Trials: Proof-of-Concept in Randomized Controlled Trials.  Clinical Pharmacology and Therapeutics 107, 369–377.
  * Chatton et al. (2020) Chatton, A., Le Borgne, F., Leyrat, C., Gillaizeau, F., Rousseau, C., Barbin, L., Laplaud, D., Léger, M., Giraudeau, B., and Foucher, Y. (2020).  G-computation, propensity score-based methods, and targeted maximum likelihood estimator for causal inference with different covariates sets: a comparative simulation study.  Scientific Reports 10, 9219\.
  * Chernozhukov et al. (2018) Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018).  Double/debiased machine learning for treatment and structural parameters.  The Econometrics Journal 21, C1–C68.
  * Cole and Stuart (2010) Cole, S. R. and Stuart, E. A. (2010).  Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial.  American Journal of Epidemiology 172, 107–115.
  * Corrigan-Curay et al. (2018) Corrigan-Curay, J., Sacks, L., and Woodcock, J. (2018).  Real-World Evidence and Real-World Data for Evaluating Drug Safety and Effectiveness.  JAMA 320, 867–868.
  * Dahabreh et al. (2019) Dahabreh, I. J., Robins, J. M., Haneuse, S. J.-P. A., and Hernán, M. A. (2019).  Generalizing causal inferences from randomized trials: counterfactual and graphical identification.  arXiv:1906.10792 [stat].
  * Darras et al. (2021) Darras, B. T., Masson, R., Mazurkiewicz-Bełdzińska, M., Rose, K., Xiong, H., Zanoteli, E., Baranello, G., Bruno, C., Vlodavets, D., Wang, Y., El-Khairi, M., Gerber, M., Gorni, K., Khwaja, O., Kletzl, H., Scalco, R. S., Fontoura, P., Servais, L., and FIREFISH Working Group (2021).  Risdiplam-Treated Infants with Type 1 Spinal Muscular Atrophy versus Historical Controls.  The New England Journal of Medicine 385, 427–435.
  * Dunipace (2022) Dunipace, E. (2022).  Optimal transport weights for causal inference.  arXiv:2109.01991 [cs, econ, stat].
  * Fleming (1982) Fleming, T. R. (1982).  Historical controls, data banks, and randomized trials in clinical research: a review.  Cancer Treatment Reports 66, 1101–1105.
  * Ghadessi et al. (2020) Ghadessi, M., Tang, R., Zhou, J., Liu, R., Wang, C., Toyoizumi, K., Mei, C., Zhang, L., Deng, C. Q., and Beckman, R. A. (2020).  A roadmap to using historical controls in clinical trials - by Drug Information Association Adaptive Design Scientific Working Group (DIA-ADSWG).  Orphanet Journal of Rare Diseases 15, 69\.
  * Hahn (1998) Hahn, J. (1998).  On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects.  Econometrica 66, 315–331.  Publisher: [Wiley, Econometric Society].
  * Hall et al. (2021) Hall, K. T., Vase, L., Tobias, D. K., Dashti, H. T., Vollert, J., Kaptchuk, T. J., and Cook, N. R. (2021).  Historical Controls in Randomized Clinical Trials: Opportunities and Challenges.  Clinical Pharmacology and Therapeutics 109, 343–351.
  * Hatswell et al. (2016) Hatswell, A. J., Baio, G., Berlin, J. A., Irs, A., and Freemantle, N. (2016).  Regulatory approval of pharmaceuticals without a randomised controlled study: analysis of EMA and FDA approvals 1999–2014.  BMJ Open 6, e011666.  Publisher: British Medical Journal Publishing Group Section: Pharmacology and therapeutics.
  * Hernan and Robins (2023) Hernan, M. and Robins, J. M. (2023).  Causal Inference: What If.  Chapman & Hall/CRC, 1 edition.
  * Ibrahim et al. (2015) Ibrahim, J. G., Chen, M.-H., Gwon, Y., and Chen, F. (2015).  The power prior: theory and applications.  Statistics in Medicine 34, 3724–3749.
  * Jansen-van der Weide et al. (2018) Jansen-van der Weide, M. C., Gaasterland, C. M. W., Roes, K. C. B., Pontes, C., Vives, R., Sancho, A., Nikolakopoulos, S., Vermeulen, E., and van der Lee, J. H. (2018).  Rare disease registries: potential applications towards impact on development of new drug treatments.  Orphanet Journal of Rare Diseases 13, 154\.
  * Judea Pearl (2009) Judea Pearl (2009).  Causal inference in statistics: An overview.  Statistics Surveys 3, 96–146.
  * Kennedy (2023) Kennedy, E. H. (2023).  Semiparametric doubly robust targeted double machine learning: a review.  arXiv:2203.06469 [stat].
  * Kosorok (2008) Kosorok, M. R. (2008).  Introduction to Empirical Processes and Semiparametric Inference.  Springer, New York, 2008th edition edition.
  * Laan and Rose (2011) Laan, M. J. v. d. and Rose, S. (2011).  Targeted Learning: Causal Inference for Observational and Experimental Data.  Springer, New York, 2011th edition edition.
  * Lavergne et al. (2015) Lavergne, P., Maistre, S., and Patilea, V. (2015).  A significance test for covariates in nonparametric regression.  Electronic Journal of Statistics 9, 643–678.  Publisher: Institute of Mathematical Statistics and Bernoulli Society.
  * Lesko et al. (2017) Lesko, C. R., Buchanan, A. L., Westreich, D., Edwards, J. K., Hudgens, M. G., and Cole, S. R. (2017).  Generalizing study results: a potential outcomes perspective.  Epidemiology (Cambridge, Mass.) 28, 553–561.
  * Li et al. (2021) Li, X., Miao, W., Lu, F., and Zhou, X.-H. (2021).  Improving efficiency of inference in clinical trials with external control data.  Biometrics n/a,.  Publisher: John Wiley & Sons, Ltd.
  * Lin et al. (2018) Lin, J., Gamalo-Siebers, M., and Tiwari, R. (2018).  Propensity score matched augmented controls in randomized clinical trials: A case study.  Pharmaceutical Statistics 17, 629–647.
  * Liu et al. (2021) Liu, M., Bunn, V., Hupf, B., Lin, J., and Lin, J. (2021).  Propensity-score-based meta-analytic predictive prior for incorporating real-world and historical data.  Statistics in Medicine 40, 4794–4808.
  * Luedtke et al. (2019) Luedtke, A. R., Carone, M., and van der Laan, M. J. (2019).  An omnibus non-parametric test of equality in distribution for unknown functions.  Journal of the Royal Statistical Society. Series B, Statistical Methodology 81, 75–99.
  * Magaret et al. (2022) Magaret, A. S., Warden, M., Simon, N., Heltshe, S., Retsch-Bogart, G. Z., Ramsey, B. W., and Mayer-Hamblett, N. (2022).  A new path for CF clinical trials through the use of historical controls.  Journal of Cystic Fibrosis: Official Journal of the European Cystic Fibrosis Society 21, 293–299.
  * Massicotte et al. (2003) Massicotte, P., Julian, J. A., Gent, M., Shields, K., Marzinotto, V., Szechtman, B., Andrew, M., and REVIVE Study Group (2003).  An open-label randomized controlled trial of low molecular weight heparin compared to heparin and coumadin for the treatment of venous thromboembolic events in children: the REVIVE trial.  Thrombosis Research 109, 85–92.
  * Mercuri et al. (2022) Mercuri, E., Deconinck, N., Mazzone, E. S., Nascimento, A., Oskoui, M., Saito, K., Vuillerot, C., Baranello, G., Boespflug-Tanguy, O., Goemans, N., Kirschner, J., Kostera-Pruszczyk, A., Servais, L., Gerber, M., Gorni, K., Khwaja, O., Kletzl, H., Scalco, R. S., Staunton, H., Yeung, W. Y., Martin, C., Fontoura, P., Day, J. W., and SUNFISH Study Group (2022).  Safety and efficacy of once-daily risdiplam in type 2 and non-ambulant type 3 spinal muscular atrophy (SUNFISH part 2): a phase 3, double-blind, randomised, placebo-controlled trial.  The Lancet. Neurology 21, 42–52.
  * Pocock (1976) Pocock, S. J. (1976).  The combination of randomized and historical controls in clinical trials.  Journal of Chronic Diseases 29, 175–188.
  * Racine et al. (2006) Racine, J. S., Hart, J., and Li, Q. (2006).  Testing the Significance of Categorical Predictor Variables in Nonparametric Regression Models.  Econometric Reviews 25, 523–544.  Publisher: Taylor & Francis.
  * Richardson and Robins (2013) Richardson, T. S. and Robins, J. M. (2013).  Single world intervention graphs (SWIGs): A unification of the counterfactual and graphical approaches to causality.  Center for the Statistics and the Social Sciences, University of Washington Series. Working Paper 128, 2013\.  Publisher: Citeseer.
  * Rothwell (2006) Rothwell, P. M. (2006).  Factors That Can Affect the External Validity of Randomised Controlled Trials.  PLoS Clinical Trials 1, e9.
  * Rubin (2005) Rubin, D. B. (2005).  Causal Inference Using Potential Outcomes: Design, Modeling, Decisions.  Journal of the American Statistical Association 100, 322–331.  Publisher: [American Statistical Association, Taylor & Francis, Ltd.].
  * Sacks et al. (1983) Sacks, H. S., Chalmers, T. C., and Smith, H. (1983).  Sensitivity and specificity of clinical trials. Randomized v historical controls.  Archives of Internal Medicine 143, 753–755.
  * Schmidli et al. (2014) Schmidli, H., Gsteiger, S., Roychoudhury, S., O’Hagan, A., Spiegelhalter, D., and Neuenschwander, B. (2014).  Robust meta-analytic-predictive priors in clinical trials with historical control information.  Biometrics 70, 1023–1032.
  * Schmidli et al. (2020) Schmidli, H., Häring, D. A., Thomas, M., Cassidy, A., Weber, S., and Bretz, F. (2020).  Beyond Randomized Clinical Trials: Use of External Controls.  Clinical Pharmacology and Therapeutics 107, 806–816.
  * Shi et al. (2023) Shi, X., Pan, Z., and Miao, W. (2023).  Data integration in causal inference.  WIREs Computational Statistics 15, e1581.
  * Stuart (2010) Stuart, E. A. (2010).  Matching methods for causal inference: A review and a look forward.  Statistical science : a review journal of the Institute of Mathematical Statistics 25, 1–21.
  * Viele et al. (2014) Viele, K., Berry, S., Neuenschwander, B., Amzal, B., Chen, F., Enas, N., Hobbs, B., Ibrahim, J. G., Kinnersley, N., Lindborg, S., Micallef, S., Roychoudhury, S., and Thompson, L. (2014).  Use of historical control data for assessing treatment effects in clinical trials.  Pharmaceutical statistics 13, 41–54.
  * Westreich et al. (2017) Westreich, D., Edwards, J. K., Lesko, C. R., Stuart, E., and Cole, S. R. (2017).  Transportability of Trial Results Using Inverse Odds of Sampling Weights.  American Journal of Epidemiology 186, 1010–1014.
  * Yue (2018) Yue, L. Q. (2018).  Leveraging Real-World Evidence Derived from Patient Registries for Premarket Medical Device Regulatory Decision-Making.  Statistics in Biopharmaceutical Research 10, 98–103.  Publisher: Taylor & Francis.
  * Zhou and Ji (2021) Zhou, T. and Ji, Y. (2021).  Incorporating external data into the analysis of clinical trials via Bayesian additive regression trees.  Statistics in Medicine 40, 6421–6442.  Publisher: John Wiley & Sons, Ltd.

h

See pages 1- of supplement.pdf