latex‘!h’ float specifier changed to ‘!ht’

# Causally-Interpretable Random-Effects Meta-Analysis

Justin M. Clark∗ Kollin W. Rott∗ James S. Hodges∗ Jared D. Huling111University
of Minnesota School of Public Health, Division of Biostatistics

###### Abstract

Recent work has made important contributions in the development of causally-
interpretable meta-analysis. These methods transport treatment effects
estimated in a collection of randomized trials to a target population of
interest. Ideally, estimates targeted toward a specific population are more
interpretable and relevant to policy-makers and clinicians. However, between-
study heterogeneity not arising from differences in the distribution of
treatment effect modifiers can raise difficulties in synthesizing estimates
across trials. The existence of such heterogeneity, including variations in
treatment modality, also complicates the interpretation of transported
estimates as a generic effect in the target population. We propose a
conceptual framework and estimation procedures that attempt to account for
such heterogeneity, and develop inferential techniques that aim to capture the
accompanying excess variability in causal estimates. This framework also seeks
to clarify the kind of treatment effects that are amenable to the techniques
of generalizability and transportability.

Keywords: causal inference, generalizability, transportability, meta-analysis,
evidence synthesis, clinical trials

##  1 Introduction

Given data from a collection of randomized controlled trials (RCTs), an
important question faced by clinicians and policy-makers alike is whether such
results apply to target populations of interest. Recent work has made
important advances in tackling this question by developing causal inference
methods designed for meta-analysis (Dahabreh et al. 2022). These techniques
are designed to account for differences between the trial and target
populations, resulting in effect estimates that have a causal interpretation
for the target population.

The impact of such population differences is one example of a challenge in
research synthesis long recognized by practitioners of meta-analysis: between-
study heterogeneity of many kinds should be taken into account when evaluating
results from one trial to the next. Differences in the characteristics of
populations represents just one form of between-study heterogeneity. In this
paper, we develop causal quantities and estimators that build on the existing
causally-interpretable meta-analysis framework to additionally take into
account unexplained between-study heterogeneity beyond that induced by
covariate differences between the collection of trials and the target
population.

Our approach produces effect estimates that both are applicable to a target
population of scientific interest and remain interpretable even when between-
study heterogeneity prevents data pooling across trials. We also see our work
as a conceptual bridge between the developing field of causally-interpretable
meta-analysis and random-effects approaches that are well-established in
evidence synthesis. This structural resemblance to traditional random-effects
meta-analysis allow us to adopt some of the analytical frameworks important to
that approach while retaining causal interpretability.

In principle, systematic reviews of randomized trials should have important
implications for health policy and clinical practice (Berlin and Golub 2014).
For example, the Strength of Recommendation Taxonomy (SORT) assigns meta-
analysis to the highest level of study quality (Ebell et al. 2004). However,
recent work has suggested that meta-analyses–particularly meta-analyses
involving individual patient data (IPD)–have had a relatively small impact on
organizations developing and publishing clinical guidelines (Vale et al.
2015), even among guidelines for which relevant IPD meta-analyses were readily
available.

Analogous work published in 2021 also found limited use of systematic reviews
to inform clinical guidelines: in that study, only 34% of analyzed guidelines
conducted a systematic review of available evidence to inform clinical
practice (Lunny et al. 2021). These authors give many possible explanations
for their findings, including ignorance on the part of guideline development
groups, a related over-reliance on expert opinion, and the amount of labor and
time required to complete a high-quality systematic review. However, it is
worth interrogating whether the low uptake of systematic reviews to inform
clinical guidelines might arise from the usefulness of the meta-analyses
themselves. That is, a review of clinical evidence may be systematic,
rigorous, and follow international standards while still being of limited
applicability to decision-making in a particular population or setting.

Much of the relevance of a given trial to clinical practice stems from how
well the patient population targeted by the practitioner aligns with trial
participants. The complex, idiosyncratic process of recruitment can result in
studies that, while exhibiting a high level of internal validity, are unlikely
to apply to individuals distinct from trial participants, thereby reducing
external validity (Degtiar and Rose 2021). Concerns over external validity are
especially relevant to health system decision making, where beneficiaries may
differ in key ways from participants in the RCTs that are used to justify
coverage decisions. For example, a 2008 study suggested that participants in
the RCTs underlying a meta-analysis informing CMS policy differed
substantially from the Medicare population (Dhruva 2008). This internal-
external validity gap may help to explain growing interest in the use of large
electronic health records (EHR) databases to inform clinical research (Galson
and Simon 2016).

Issues of relevance and external validity apply to an even greater degree when
synthesizing evidence across several studies. Many of these challenges relate
to possible sources of heterogeneity between the RCTs included in a given
meta-analysis. Considering only one such possible difference, that of
heterogeneity in the participant population underlying each RCT, it is
difficult to imagine how an estimated treatment effect averaged over such
populations—as is done in standard meta-analyses—would apply to a given
clinician’s or health system’s population of interest. Moreover, other
relevant sources of heterogeneity remain, including treatment modalities and
methods of evaluating primary outcomes (Berlin and Golub 2014).

The standard approach to modeling such between-study heterogeneity is random-
effects meta-analysis, in which the effects or outcomes of each study are
conceived as random draws from some distribution, typically a normal
distribution (Higgins, Thompson, and Spiegelhalter 2009). Heterogeneity
between trials is therefore viewed as part of the total variance in trial
outcomes: one component arising from sampling variation within a trial and
another reflecting systematic (though still i.i.d.) variation between the
trials.

However, estimates from random-effects meta-analyses still fail to explicitly
account for differences between trial participants and a target population of
interest to clinicians and policy makers. Recent work in causally
interpretable meta-analysis has yielded methods for making population-specific
inferences using data from multiple RCTs (Dahabreh et al. (2022), (2020)).
Roughly, this work uses a representative sample from the target population of
interest to transport treatment effects estimated in the RCT populations to
the target population.

A key assumption in this approach is that two individuals from the same target
population would have identical average responses to treatment regardless of
which clinical trial in the meta-analysis they may have participated in.
Moreoever, the treatment response observed for an individual in the target
population had they participated in any one of the trials is assumed analogous
to the treatment response expected in the population more generally.
Systematic heterogeneity in trial conduct and the possibility of trial
participation effects threaten both of these assumptions (Dahabreh, Robertson,
and Hernán 2022). While some of this heterogeneity might be captured by
covariate differences, other site-specific mechanisms may be unrelated to
participant characteristics.

###  1.1 Examples of Between-Study Heterogeneity

Defining such heterogeneity more concretely, consider a scenario where each
study in the meta-analysis applied a somewhat different version of treatment.
In the context of studies designed to ameliorate lower back pain, for example,
providers in one study may apply a slightly different form of spinal
manipulation than providers in another study. Such differences could be pre-
defined in each study’s protocol, but may also arise simply because different
chiropractors implement the manipulation in slightly different ways. Moreover,
differences in how a pain outcome is measured (e.g., the exact time at which
the pain scale is assessed relative to treatment) could also require an
expansion of potential outcome notation.

These differences in, say, treatment version, would persist even after
accounting for variation in the distribution of treatment effect modifiers
both between studies and between the trial and target populations. In the back
pain example, if two individuals with identical covariate data enrolled in
separate studies, differences in the spinal manipulation type would persist
when attempting to compare and define their respective potential outcomes.
This suggests indexing such heterogeneity in the potential outcomes
themselves, e.g., an individual’s potential outcome had they been assigned
treatment $a$ under one provider versus another.

Broader differences in trial conduct also have the potential to induce this
heterogeneity. For example, after disruptions to biomedical research caused by
the COVID-19 pandemic, some clinical trials opted to deliver self-administered
medications to trial participants by mail, rather than distribute them in-
person (U.S. Food and Drug Administration 2021). Delivery timings under these
conditions may vary significantly between different trials, e.g., trials whose
participants reside in more remote, rural locations may have slower delivery
times than trials in urban locations. For diseases like COVID-19, where timing
of treatment affects subsequent outcomes, these differences in delivery time
could induce heterogeneity in the potential outcomes from each trial.

Differences in the timing of treatments administered in-person may also have
important implications for research synthesis. A recent preprint examined
whether passively administered antibodies altered health outcomes associated
with SARS-CoV-2 infection (Stadler et al. 2022). They found that earlier
administration of such treatments may improve their efficacy. Imagine two
patients with identical baseline covariates enrolling in two studies
investigating the same monoclonal antibody treatment. If the two studies
differed in the timing of treatment relative to infection, this study suggests
that those individuals may have distinct, study-specific expected potential
outcomes.

Another example where differences in trial conduct may induce heterogeneity is
when provider-assessed ratings are an outcome. For example, The Positive and
Negative Syndrome Scale (PANSS) is routinely used to assess symptom severity
in patients with schizophrenia (Kay, Fiszbein, and Opler 1987). For each
symptom evaluated in the PANSS, raters assign a score of 1 (absent) to 7
(extreme). Differences between treatment groups might then be assessed by
differences in their aggregated PANSS scores. However, individual raters may
perform these qualitative evaluations in different ways. Analyses synthesizing
outcomes from many sites employing different raters may need to take these
differences into account.

Animated by such concerns, our work aims to lay the groundwork for a causally-
interpretable meta-analysis that accounts for heterogeneity between studies
beyond that induced by differences in the distribution of treatment effect
modifiers. Our approach attempts to combine the techniques and intuitions of
traditional random-effects meta-analysis with the causal inference-based meta-
analytic framework introduced by Dahabreh et al. (2022). Concretely, this
involves posing two causal questions. First, what outcomes would we observe,
on average, if a member of the target population had participated in a
particular study included in the collection of relevant RCTs? Moreover, what
would we observe if we conducted a new RCT whose participants are drawn
directly from the target population? This latter question is of primary
scientific interest in this work, and follows directly from our novel
framework. By answering such questions, we hope to expand the toolkit
available to health policy makers and clinicians when evaluating treatment
efficacy through meta-analysis.

##  2 Estimands for Capturing Between-Study Heterogeneity

Causally-interpretable meta-analysis analyzes outcomes and treatment effects
observed in a collection of studies through the lens of transportability and
generalizability. We assume the existence of some target population of
interest and consider what this collection of studies can tell us about the
effect of an intervention in that target population. Suppose we have data from
a collection of clinical trials indexed by a set $\mathcal{S}=\\{1,...,m\\}$.
One causal question of interest is: what treatment effect can we expect if a
member of the target population had participated in study $s\in\mathcal{S}$?
As we will describe more precisely later, we can roughly describe our approach
as decomposing such an average treatment effect $\tau_{s}$ into the sum of an
overall grand mean $\tau$ and a deviation $\delta_{s}$ specific to study $s$:

$\tau_{s}=\tau+\delta_{s}.$ (1)

In such a formulation, the grand mean $\tau$ gives the treatment effect in the
target population averaged over the collection of studies while the
$\delta_{s}$ reflects between-study heterogeneity in the underlying treatment
effect.

Our ultimate goal is to precisely characterize, estimate, and perform
inference on $\tau_{0}$, which is the expected treatment effect if a new trial
were conducted in the target population. Such a treatment effect comes about
via a new draw $\delta_{0}$, a new shift about $\tau$.

The decomposition in (1) is structurally similar to parameters studied in
random-effects meta-analysis, wherein study-specific treatment effects are
assumed to be i.i.d. draws from a distribution, typically normal. In this
work, we define and estimate parameters that maintain the familiar form of (1)
while admitting a precise causal interpretation. This is accomplished by
incorporating the study-specific heterogeneity reflected in $\delta_{s}$
within the potential outcomes framework. Specifically, we incorporate the
variation represented by $\delta_{s}$ as an additional argument to the
standard potential outcomes notation.

###  2.1 The Data

Adding notation to the example outlined above, suppose we have IPD from a
collection of $m$ trials, all of which examined the same set of treatments.
Again, we index these trials with the set $\mathcal{S}=\\{1,...,m\\}$ and
treatments with the set $\mathcal{A}$. For each participant $i$ in a given
trial $s\in\mathcal{S}$ with $n_{s}$ participants total, we have outcome data
$Y_{i}$, baseline covariate information $X_{i}$ and treatment assignment
$A_{i}$, where $A_{i}\in\mathcal{A}$. We also assume to have baseline
covariate information from a random sample of individuals in a target
population of interest; we do not require treatment or outcome information in
this sample. As in Dahabreh et al. (2022), we let $S=0$ for individuals in the
target population and introduce another variable $R$ which takes value 1 for
participants in the collection of trials and 0 for members of the target
population. Thus, the observed data for each individual in the full
dataset—that is, combined data from both the collection of trials and the
target population—is of the form

$(R_{i}Y_{i},R_{i}A_{i},X_{i},S_{i}).$

$R_{i}Y_{i}$ and $R_{i}A_{i}$ evaluate to zero for those in the target
population, indicating that such data are unavailable. The total sample size
including both the sample from the target population and the collection of
trials is $n=n_{0}+n_{1}+\cdots+n_{m}$.

###  2.2 Rationale for New Causal Quantities

In introducing causally-interpretable meta-analysis, Dahabreh et al. (2022)
define potential outcomes $Y(a)$ that depend on the assigned treatment
$a\in\mathcal{A}$. Different causal estimands describe the distribution of
these potential outcomes within various populations of interest. For instance,
$E[Y(a)|R=0]$ gives the expected potential outcome under treatment $a$ in the
target population. To identify quantities like $E[Y(a)|R=0]$, Dahabreh et al.
assume exchangeability in potential outcomes across values of $S$, conditional
on baseline covariates. They also show that $E[Y(a)|R=0]$ is identifiable
under the weaker assumption of mean exchangeability, roughly defined as the
assumption that:

$E[Y(a)|X=x,S=0]=E[Y(a)|X=x,S=s]$ (2)

for $s=1,...,m$, along with additional conditions ensuring that the assumption
is only made across values of $S$ where the covariate pattern $X=x$ has
positive probability of occurring.

Intuitively, the primary risk in assuming (2) when it does not, in fact, hold
is that the particular idiosyncrasies of each trial have the potential to
muddle a transported treatment effect estimated from pooled data. This is
especially problematic if the trials in the meta-analytic collection differ a
great deal in sample size. In that setting, violations of (2) may result in
causal estimates that are heavily weighted to the particular conditions of the
largest trial. Such a weighting scheme has no clinical meaning and is merely
an artifact of the mean exchangeability assumption.

The motivation for referencing such between-study heterogeneity stems directly
from the logic underlying standard random-effects meta-analysis (Higgins,
Thompson, and Spiegelhalter 2009). In that paradigm, a meta-analysis of a
collection of trials proceeds under the assumption that the underlying
parameter of interest—typically an average treatment effect—differs between
trials. More precisely, in the traditional random-effects model, each study’s
latent true treatment effect is assumed to be randomly sampled from a
distribution of effects. The variance of this underlying distribution is often
of interest, and serves to quantify the degree of between-study heterogeneity.

Our proposed method models the average effects arising from studies
$s\in\mathcal{S}$ transported to the target population as varying about a
grand mean according to draws of a latent random variable. Draws of this
latent random variable fix these underlying average effects at different
values, where differences between these values reflect between-study
heterogeneity beyond that induced by covariate differences. The key
distinction between these methods and traditional random-effects models is
that heterogeneity in underlying parameter values (e.g., heterogeneity between
studies in treatment effects) stems from variation in the potential outcomes
of study participants, thereby retaining causal interpretations absent from
standard meta-analysis.

Recall that a key causal question motivating this study is: what outcome would
we expect if an individual were assigned treatment $a\in\mathcal{A}$ in study
$s\in\mathcal{S}$? This question suggests a need to distinguish potential
outcomes arising from one trial setting versus another. Using the notation of
VanderWeele and Hernán (2013), we write the potential outcome of a subject
receiving treatment $a\in\mathcal{A}$ under the conditions present in study
$s\in\mathcal{S}$ as having two arguments: one specifying the treatment
assignment and another fixing the conditions present in a given setting, e.g.,
treatment group $a$ in study $s$:

$Y(a,k^{a}_{s}).$

Here, $k^{a}_{s}$ constitutes a realization of a random variable
$K^{a}_{s}(a)$. Because we conceptualize values for $K^{a}_{s}(a)$ even for
individuals who did not receive treatment $a$ in study $s$, its notation
mimics that of a standard potential outcome, as one’s experience in a
particular study could be a function of the treatment received. We suppose
that each combination of a study and treatment arm is associated with one such
random variable, the collection of which, across studies for a given treatment
$a$, is an i.i.d. sample from some common distribution:

$K^{a}_{1}(a),...,K^{a}_{m}(a)\overset{i.i.d.}{\sim}F_{K^{a}(a)}.$

We do not impose a particular interpretation on $K^{a}_{s}(a)$ except that it
reflects the conditions an individual would have experienced had they been in
treatment group $a$ of study $s$. That is, two individuals with common values
of this random variable can be understood to have been assigned treatment $a$
under equivalent conditions. Again, we treat $K^{a}_{s}(a)$ as a potentially
counterfactual variable which fixes the heterogeneity in setting/treatment
group combinations even for individuals who were not assigned treatment $a$ or
did not participate in study $s$.

We suppose these random variables are an i.i.d. sample from some common
distribution not because we think this gives the most accurate approximation
to the data generating process that induces between-study heterogeneity.
Rather, such an assumption constitutes our best guess as to the nature of this
heterogeneity in the absence of additional information. Equipped with only
outcome, baseline covariate, and treatment arm information, residual between-
study heterogeneity is as good as i.i.d. noise from the perspective of the
(meta) analyst, playing a role similar to residual errors in a simple linear
regression model. If more information were available about the nature of
between-trial heterogeneity, this assumption could be relaxed or refined.

The impact of this heterogeneity is related to but distinct from the effects
of trial participation per se, as studied in Dahabreh et al. (2019). Here, we
develop a conceptual framework for understanding the effects of participating
in one trial versus another rather than participation in and of itself. In
part, this effort can be interpreted as clarifying the interpretation of
causal estimands defined in earlier work on causally-interpretable meta-
analysis. Recognizing the impact of between-trial heterogeneity, we study the
effects of participation in one of the trials under study, or trials similar
to those under study, rather than a more generic effect of treatment
assignment in the target population.

###  2.3 Defining Causal Estimands

As in Dahabreh et al. (2022), our interest is in transporting inferences from
the collection of trials to a target population. Since the realized draw of
$K^{a}_{s}(a)=k^{a}_{s}$ indexes the heterogeneity in potential outcomes
arising from trial $s$, a causal quantity relevant to evaluating the kind of
outcomes we would observe had members of the target population participated in
trial $s$ is

$\mu_{a,0}(k^{a}_{s})=E[Y(a,k^{a}_{s})|R=0].$ (3)

This is a fixed quantity which can be interpreted as the average potential
outcome in the target population where the heterogeneity in application of
treatment $a$ is fixed at the value associated with trial $s$.

We recognize that policy makers and clinicians are often most interested in
treatment _effects_ within their target population, rather than mean potential
outcomes alone. In this paper, we focus our attention on mean potential
outcomes for simplicity of presentation. However, contrasts of causal
quantities like (3), e.g.

$\tau^{s}_{a,a^{\prime}}=\mu_{a,0}(k^{a}_{s})-\mu_{a^{\prime},0}(k^{a^{\prime}}_{s})=E[Y(a,k^{a}_{s})|R=0]-E[Y(a^{\prime},k^{a^{\prime}}_{s})|R=0]$

can be defined corresponding to average treatment effects in the target
population under the conditions of trial $s$. Focus on such effects also
allows relaxation of mean exchangeability assumptions on mean potential
outcomes to be replaced by mean effect exchangeability.

Returning our focus to (3), note that the random variables whose $m$ realized
values give these transported potential outcomes for each study are a function
of the latent random sample $K^{a}_{1}(a),...,K^{a}_{m}(a)$:

$\mu_{a,0}(a,K_{1}^{a}(a)),...,\mu_{a,0}(a,K_{m}^{a}(a))$ (4)

As such, they also constitute a simple random sample from some distribution.

We assume this random sample
$\mu_{a,0}\left(K^{a}_{1}(a)\right),...,\mu_{a,0}\left(K^{a}_{m}(a)\right)$
arises from a distribution with mean $\mu_{a,0}$ and finite variance. Without
any further modeling assumptions, we can decompose each such random variable
in the sample as:

$\mu_{a,0}\left(K^{a}_{s}(a)\right)=\mu_{a,0}+\Delta^{a}_{0,s},$ (5)

where $\Delta^{a}_{0,s}$ is a random variable with mean zero and finite
variance. Note that (5) does not impose any particular modeling assumption on
$\mu_{a,0}\left(K^{a}_{s}(a)\right)$; it simply labels random variation about
its expectation over $K^{a}_{s}(a)$ as $\Delta^{a}_{0,s}$. Rearranging (5), we
have

$\displaystyle\Delta^{a}_{0,s}$
$\displaystyle=\mu_{a,0}\left(K^{a}_{s}(a)\right)-\mu_{a,0}$
$\displaystyle=\mu_{a,0}\left(K^{a}_{s}(a)\right)-E_{K^{a}(a)}\left[\mu_{a,0}\left(K^{a}_{s}(a)\right)\right].$

The decomposition in (5) allows us to better understand the role that the
latent random variables $K^{a}_{s}(a)$ play in driving systematic differences
between studies in causal quantities. Namely, a draw of
$K^{a}_{s}(a)=k^{a}_{s}$ corresponds to a realization of
$\Delta^{a}_{0,s}=\delta^{a}_{0,s}$ which in turn shifts the average potential
outcome under $a$ transported from study $s$ to the target population.

Recalling the stylized version of our approach in Equation (1), our model for
the treatment effect we would observe if a member of the target population had
been assigned to treatment $a$ vs. $a^{\prime}$ in study $s$ is therefore

$\displaystyle\mu_{a,0}(k^{a}_{s})-\mu_{a^{\prime},0}(k^{a^{\prime}}_{s})$
$\displaystyle=\mu_{a,0}+\delta^{a}_{0,s}-\left(\mu_{a^{\prime},0}-\delta^{a^{\prime}}_{0,s}\right)$
$\displaystyle=\left(\mu_{a,0}-\mu_{a^{\prime},0}\right)+\left(\delta^{a}_{0,s}-\delta^{a^{\prime}}_{0,s}\right).$
(6)

We can conceptualize $\tau_{a,a^{\prime},0}=\mu_{a,0}-\mu_{a^{\prime},0}$ as
the “grand mean effect” of treatment $a$ vs. $a^{\prime}$ in the target
population and $\delta^{a}_{0,s}-\delta^{a^{\prime}}_{0,s}$ as the
heterogeneity of that overall effect when transporting from the setting of
study $s$. These two quantities stand in for $\tau$ and $\delta_{s}$,
respectively in Equation (1).

Besides considering the expected potential outcome if an individual from the
target population participated in one of the trials in our collection, we
might also ask: what potential outcomes would we expect if a new trial were
conducted that recruited a simple random sample from the target population? We
make an important assumption here that the same random process which induces
heterogeneity in expected potential outcomes within the collection of trials
would apply to a new trial of the same collection of treatments.

We can represent this assumption by adding $K^{a}_{0}(a)$ to our original
random sample:

$K^{a}_{1}(a),...,K^{a}_{m}(a),K^{a}_{0}(a)\overset{i.i.d.}{\sim}F_{K^{a}(a)}$

and defining $\mu_{a,0}\left(k^{a}_{0}\right)$ and
$\mu_{a,0}\left(K^{a}_{0}(a)\right)$ as previously.

This framing is analogous to the idea in random-effects meta-analysis that we
can make inferences for the treatment effects of studies that are not included
in the meta-analysis (Higgins, Thompson, and Spiegelhalter 2009). Again, our
i.i.d. assumption regarding $K^{a}_{0}(a)$ reflects our best guess as to the
random process governing between-study heterogeneity. In the absence of
additional information, we posit both that the same sources of heterogeneity
that induce systematic differences between RCTs in the trial sample would
apply to a hypothetical RCT in the target population and that this additional
variation stems from i.i.d. draws from a common distribution. As explained
below, we use this assumption when producing prediction intervals that contain
$\mu_{a,0}\left(k^{a}_{0}\right)$ with some specified probability.

There are several possible motivations for estimating the outcomes in an
unobserved trial. One is as an input to planning and preparation for new
clinical studies. Clinicians may desire to estimate treatment effects they
might expect in a planned study with participants similar to members of the
target population. Alternatively, transported estimates from a clinical trial
might serve as a kind of bound for effects we are likely to observe in
practice. Estimating outcomes in an unobserved trial also produces such a
bound, albeit one not overly influenced by the idiosyncrasies present in any
particular study in the collection of trials.

##  3 Identification and Estimation of Causal Quantities

###  3.1 Identification

We first consider identification of $\mu_{a,0}(k^{a}_{s})$, which gives the
expected outcome that would have been observed if a member of the target
population had been assigned treatment $a$ in study $s$. To express this as a
functional of the observed data, we make the following identifying
assumptions, many of which are similar to those in Dahabreh et al. (2022) and
VanderWeele and Hernán (2013):

  1. 1.

Exchangeability in mean between trials:

$E[Y(a,k^{a}_{s})|X=x,S=s_{1}]=E[Y(a,k^{a}_{s})|X=x,S=s_{2}]$

for all $s_{1},s_{2}\in\\{0,1,...,m\\}$ and $x\in\mathcal{X}$ such that
$f(x,S=s_{1})\neq 0$ and $f(x,S=s_{2})\neq 0.$ Note here that the
heterogeneity in trial $s_{1}$ and $s_{2}$ is fixed at $k^{a}_{s}$ in both
settings.

  2. 2.

Exchangeability over treatment groups within a trial:

$Y(a,k^{a}_{s})\perp\\!\\!\\!\perp A|(X,S=s)$

for all $a\in\mathcal{A}$, $k^{a}_{s}\in\mathcal{K}^{a}$, and
$s\in\mathcal{S}$.

  3. 3.

Consistency: If $A_{i}=a$ and $K^{a}_{i}(a)=k^{a}$ for individual $i$, then
$Y_{i}(a,k^{a})=Y_{i}$.

  4. 4.

Positivity: For all $s\in\mathcal{S}$, if $f(x,R=0)\neq 0$, then
$P(S=s|X=x)>0$.

  5. 5.

Distribution of $K^{a}(a)$: For $s=1,...,m$ and $a\in\mathcal{A}$ each random
variable $K^{a}_{s}(a)$ consitutes an i.i.d. draw from a distribution
$F_{K^{a}(a)}$.

  6. 6.

Constancy of $K^{a}(a)$ by treatment group/study: Within each treatment group
$a$ and study $s$, the value of $K^{a}(a)$ for each participant is fixed at
the realized value of $K^{a}_{s}(a)=k^{a}_{s}$. That is, if $A=a$ and $S=s$,
then

$K^{a}_{i}(a)=K^{a}_{s}(a)=k^{a}_{s}$

for $i\in 1,...,n_{s}$.

Assumptions 3 and 5 involve a slight abuse of notation, wherein we define a
random variable $K^{a}_{i}(a)$ giving the version of treatment for individual
$i$. The key point here is that such a random variable takes on the same
realized value of $K^{a}_{s}(a)$ for all participants in trial $s$.
Alternatively, one might assume a hierarchical model where each individual in
treatment arm $a$ and study $s$ has an associated random variable
$\left(K^{a}_{s}(a)\right)_{i}$, $i=1,...,n_{s}$, centered at a the realized
value of $K^{a}_{s}(a)=k^{a}_{s}$. In this paper, we focus on trial/treatment
group-wide heterogeneity to greatly simplify the mathematical presentation.
However, a hierarchical model of the kind proposed above can–under certain
assumptions–lead to the same identification results presented here.

Under these assumptions, we identify $\mu_{a,0}(k^{a}_{s})$ as

$\psi_{s,0}(a)=E[E[Y|X,S=s,A=a]|R=0].$ (7)

That is, we average a regression function relating outcomes under treatment
$a$ in study $s$ to covariates $X$ over the distribution of such covariates in
the observed target population. (A full proof of this result is given in the
Appendix.) This observed data functional is analogous to that identified in
Equation (6) of Theorem 1 in Dahabreh et al. (2022), with $R=1$ in their case
replaced by $S=s$ in ours.

###  3.2 Estimation

Letting $g^{s}_{a}(X)=E[Y|X,S=s,A=a]$, we could apply an outcome
model/standardization approach that averages an estimate $\hat{g}^{s}_{a}(X)$
of $E[Y|X,S=s,A=a]$ over the distribution of covariates in the target
population:

$\hat{\psi}_{s,0}(a)=\left\\{\sum_{i=1}^{n}I(S_{i}=0)\right\\}^{-1}\sum_{i=1}^{n}I(S_{i}=0)\left\\{\hat{g}^{s}_{a}(X_{i})\right\\}$

Another option involves inverse probability weighting, in which the outcomes
observed among participants in study $s$ are weighted by the similarity of
each participant to members of the target population. The estimator applying
this approach is given by

$\hat{\psi}^{ipw}_{s,0}(a)=\left\\{\sum_{i=1}^{n}I(S_{i}=0)\right\\}^{-1}\sum_{i=1}^{n}\left(\frac{I(A_{i}=a)}{\hat{e}^{s}_{a}(X_{i})}\right)I(S_{i}=s)\frac{\hat{p}_{0}(X_{i})}{\hat{p}_{s}(X_{i})}Y_{i}$

where $\hat{p}_{0}(X_{i})$ estimates $P(R=0|X_{i})$, $\hat{e}_{a}(X_{i})$
estimates $P(A_{i}=a|S=s,X_{i})$, and $\hat{p}_{s}(X_{i})$ estimates
$P(S=s|X_{i})$. Identification of this result, proceeding from (7) is given in
the appendix.

A final option is the so-called augmented inverse probability weighting (AIPW)
estimator, which combines the two approaches introduced above:

$\hat{\psi}^{aipw}_{s,0}(a)=\left\\{\sum_{i=1}^{n}I(S_{i}=0)\right\\}^{-1}\sum_{i=1}^{n}\left[I(S_{i}=0)\left\\{\hat{g}^{s}_{a}(X_{i})\right\\}+\left(\frac{I(A_{i}=a)}{\hat{e}^{s}_{a}(X_{i})}\right)I(S_{i}=s)\frac{\hat{p}(X_{i})}{1-\hat{p}(X_{i})}(\hat{g}^{s}_{a}(X_{i})-Y_{i})\right]$

This approach adds a correction to the outcome model approach via inverse
probability weighting over the residuals of the outcome model for arm $a$ of
study $s$.

Our second causal quantity of interest is $E[Y(a,k^{a}_{0})|R=0]$, where
$k^{a}_{0}$ is the realized value of the random variable $K^{a}_{0}(a)$
described in the previous section. In the absence of additional information
concerning the distribution $F_{K^{a}(a)}$, our estimation strategy for
$E[Y(a,k^{a}_{0})|R=0]$ relies on the following approximation, the full
details of which are included in the Appendix:

$\displaystyle E[Y(a,K^{a}_{0}(a))|R=0]$
$\displaystyle=\sum_{k^{a}\in\mathcal{K}^{a}}E\left\\{E[Y(a,K^{a}_{0}(a))|R=0,K^{a}_{0}(a)=k^{a}]|R=0\right\\}P(K^{a}_{0}(a)=k^{a})$
$\displaystyle\approx\sum_{k^{a}\in\mathcal{K}^{a}}E\left\\{E[Y(a,K^{a}_{0}(a))|R=0,K^{a}_{0}(a)=k^{a}]|R=0\right\\}\left\\{\frac{1}{m}\sum_{s=1}^{m}\mathbbm{1}(k^{a}=k^{a}_{s})\right\\}$
$\displaystyle=\frac{1}{m}\sum_{s=1}^{m}E[Y(a,k^{a}_{s})|R=0]$
$\displaystyle=\frac{1}{m}\sum_{s=1}^{m}E[E[Y|X,S=s,A=a]|R=0]\hskip
96.73918pt\text{by \eqref{eq: id_one_trial}}$
$\displaystyle=\frac{1}{m}\sum_{s=1}^{m}\psi_{s,0}(a).$

Again, taking contrasts of the above quantity applied to different treatments
implies identification of treatment effects of interest, e.g., causal
quantities of the form given in (6). We can then estimate
$\frac{1}{m}\sum_{s=1}^{m}\psi_{s,0}(a)$ using any of the methods discussed
above for estimating each $\psi_{s,0}(a)$ individually. For instance, we might
estimate this quantity using

$\frac{1}{m}\sum_{s=1}^{m}\hat{\psi}_{s,0}(a)=\frac{1}{m}\sum_{s=1}^{m}\left(\left\\{\sum_{i=1}^{n}I(S_{i}=0)\right\\}^{-1}\sum_{i=1}^{n}\left\\{\hat{g}^{s}_{a}(X_{i})\right\\}\right).$

###  3.3 Estimation of Between-Study Variability

A central parameter of interest in traditional random-effects meta-analysis is
the between-study variance, which describes the variability of the effects
underlying each study. Recalling the decomposition given in (5), the analogous
parameter in our work is the variance of $\Delta^{a}_{0,s}$, which specifies
the variability between studies in the transported mean potential outcomes. We
propose an estimate of this variance inspired by the method of moments
approach derived by Rao et al. (1981) and subsequently applied by DerSimonian
and Laird (1986) in their seminal work on random-effects meta-analysis. The
key distinction between our estimate and the traditional random-effects
estimate is that, in practice, the estimates for
$\mu_{a,0}(k^{a}_{1}),...,\mu_{a,0}(k^{a}_{m})$ are correlated due to their
dependence on the same sample from the target population. This correlation
complicates the derivation and form of the resulting estimator.

In the derivation below, we consider estimating between-study variability in a
highly general setting; the only relationship to our causal framework is that
of correlation between study-specific estimates. Operating in a simplified
setting, suppose we have $m$ study-specific means $\mu_{1},...,\mu_{m}$ drawn
from a distribution $F_{\mu}$ with mean $\mu$ and variance $\gamma^{2}$. The
analogy to our setting is obtained by letting $\mu_{s}=\mu_{a,0}(k^{a}_{s})$.
We estimate these means with $\hat{\mu}_{1},...,\hat{\mu}_{m}$ each of which
is individually unbiased for its respective study-specific mean. The analogous
quantities for us are $\hat{\mu}_{s}=\hat{\psi}_{s,0}(a)$. We estimate the
grand mean $\mu$ using a weighted sum of the study-specific estimates
$\hat{\mu}=\sum_{s=1}^{m}w_{s}\hat{\mu}_{s}$, where $\sum_{s=1}^{m}w_{s}=1$.
(For instance, we might have $w_{s}=\frac{1}{m}$ when taking a simple
average.) Also, let $\text{Var}\left(\hat{\mu}_{s}\right|S=s)=\sigma^{2}_{s}$
denote the sampling variance of each study-specific estimator and
$\text{Var}\left(\hat{\mu}_{s}\right)=\sigma^{2}_{s}+\gamma^{2}$, which
reflects both within- and between-study variance. Letting
$Q=\sum_{s=1}^{m}\left(\hat{\mu}_{s}-\hat{\mu}\right)^{2}$ and
$C_{s}=-2\sum_{i\neq s}w_{i}(1-w_{s})\sigma_{is}+\sum_{i\neq
s}\left[\sum_{j\neq i,j\neq s}w_{i}w_{j}\sigma_{ij}\right]$ we can show that

$E[Q]=m\sum_{s=1}^{m}w_{s}^{2}(\sigma^{2}_{s}+\gamma^{2})+\sum_{s=1}^{m}\left[(1-2w_{s})(\sigma^{2}_{s}+\gamma^{2})+C_{s}\right].$

A moment-based estimator for $\gamma^{2}$ is then given by the value
$\tilde{\gamma}^{2}$ that satisfies

$\sum_{s=1}^{m}\left(\hat{\mu}_{s}-\hat{\mu}\right)^{2}=m\sum_{s=1}^{m}w_{s}^{2}(\sigma^{2}_{s}+\tilde{\gamma}^{2})+\sum_{s=1}^{m}\left[(1-2w_{s})(\sigma^{2}_{s}+\tilde{\gamma}^{2})+C_{s}\right].$

Solving for such a $\tilde{\gamma}^{2}$, we obtain

$\tilde{\gamma}^{2}=\frac{\sum_{s=1}^{m}\left(\hat{\mu}_{s}-\hat{\mu}\right)^{2}-\sum_{s=1}^{m}\sigma^{2}_{s}\left\\{mw_{s}^{2}+(1-2w_{s})\right\\}-\sum_{s=1}^{m}C_{s}}{\sum_{s=1}^{m}\left\\{mw_{s}^{2}+(1-2w_{s})\right\\}}.$

Our final estimate for the between-study heterogeneity would then be

$\hat{\gamma}^{2}=\max(0,\tilde{\gamma}^{2}).$ (8)

A full derivation of the above estimator is given in the Appendix. The key
distinction between our estimator and the traditional random-effects estimator
introduced by Rao et al. (1981) is the inclusion of $-\sum_{s=1}^{m}C_{s}$ in
the numerator of $\hat{\gamma}^{2}$. This has the effect of “correcting” the
traditional estimator to account for the nonzero correlation between study-
specific estimates. Beyond our setting, the estimator can be generally applied
to any analysis which takes a weighted average of correlated quantities to
estimate some underlying grand mean; note, though, that it is subject to bias
when individual estimates are themselves biased.

###  3.4 Inference for Outcomes in a New Trial

Inference for the outcomes we would expect if members of the trial population
participated in an _observed_ trial, e.g., $\mu_{a,0}(k^{a}_{s})$,
$s\in\mathcal{S}$ can proceed in a variety of ways, including simple bootstrap
resampling on data from the target population and study $s$ or via asymptotic
approximations to the sampling distribution of $\hat{\psi}_{s,0}(a)$. Here, we
focus on inference for $\mu_{a,0}(K^{a}_{0}(a))$. We keep $K^{a}_{0}(a)$ as a
random variable in this estimand since it refers to outcomes in a trial we
have yet to observe. As such, a simple bootstrap applied to all observed
trials and the target population would fail to capture the excess variability
induced by conducting a new trial. That is, our prediction intervals aim to
capture both the sampling variability of our estimator
$\frac{1}{m}\sum_{s=1}^{m}\hat{\psi}_{s,0}(a)$ as reflected in the observed
data and additional uncertainty about the unobserved trial. We can make
progress under the assumption that the unobserved trial is subject to similar
heterogeneity as that between the observed trials. Here, we propose three
methods for constructing these prediction intervals, and later evaluate them
in a simulation experiment.

####  3.4.1 Inference Based on $\hat{\gamma}^{2}$

Above, we derived an estimator $\hat{\gamma}^{2}$, which we generically
interpret as an estimate of between-study heterogeneity beyond that induced by
measured covariates. When we replace $\hat{\mu}_{s}$ with
$\hat{\psi}_{s,0}(a)$ and $\hat{\mu}$ with
$\frac{1}{m}\sum_{s=1}^{m}\hat{\psi}_{s,0}(a)$, $\hat{\gamma}^{2}$ then
estimates $\text{Var}\left(\mu_{a,0}\left(K^{a}_{s}(a)\right)\right)$. That
is, within our framework, $\hat{\gamma}^{2}$ quantifies the variability in
expected transported potential outcomes from one trial to another, where the
variability is driven by setting-specific heterogeneity. To construct a
prediction interval centered at
$\frac{1}{m}\sum_{s=1}^{m}\hat{\psi}_{s,0}(a)$, we employ the same form of the
interval considered in Higgins et al. (2009) for inference on “the effect in
an unspecified study”:

$\frac{1}{m}\sum_{s=1}^{m}\hat{\psi}_{s,0}(a)\pm
t^{\alpha}_{m-2}\sqrt{\hat{\gamma}^{2}+\widehat{\text{Var}}\left(\frac{1}{m}\sum_{s=1}^{m}\hat{\psi}_{s,0}(a)\right)},$
(9)

where
$\widehat{\text{Var}}\left(\frac{1}{m}\sum_{s=1}^{m}\hat{\psi}_{s,0}(a)\right)$
is obtained using a simple bootstrap on all of the observed studies and the
target population, and $t^{\alpha}_{m-2}$ is the $(1-\alpha)$th quantile of
the $t$ distribution with $m-2$ degrees of freedom.

####  3.4.2 Inference Based on the Bootstrap

One way to approximate the difference between our observed estimates and that
of an unobserved trial is to compute
$\frac{1}{m-1}\sum_{s=1}^{m-1}\hat{\psi}_{s,0}(a)$ using data from a subset of
$m-1$ trials and compare this result with that of the transported potential
outcome from the $m$th, left-out trial (as repeated for all trials). This
procedure aims to capture variability related to between-trial heterogeneity.
To additionally capture sampling variability in the observed data, we couple
the leave-one-out approach with application of the simple or wild bootstrap.
Both such bootstraps are described below. The result of both procedures is to
construct an estimate for the distribution of
$\mu_{a,0}\left(K^{a}_{s}(a)\right)$. For ease of notation in the following,
we let $\hat{\mu}_{0}=\frac{1}{m}\sum_{s=1}^{m}\hat{\psi}_{s,0}(a)$ and
$\hat{\mu}_{s}=\hat{\psi}_{s,0}(a)$.

Let $D_{s}$ denote all of the data available in our original sample from trial
$s=1,...,m$. Let $X$ denote the covariate data available from the target
population. For $b=1,...,B$,

  1. 1.

Randomly choose one of $m$ studies to treat as the “unobserved” trial on which
we’re trying to make a prediction. Denote this choice $s_{b}\in\\{1,...,m\\}$.

  2. 2.

Draw a simple bootstrap sample $D^{*}_{s_{b}}$ from $D_{s_{b}}$ and
$X^{*}_{1}$ from $X$ and estimate $\hat{\mu}^{*}_{s_{b}}$ using these two
datasets.

  3. 3.

Draw simple bootstrap samples $\\{D^{*}_{s}:s\neq s_{b}\\}$ from the other
trial data. Draw another bootstrap sample $X^{*}_{2}$ from $X$. With these
samples, compute $\mu^{*}=\frac{1}{m-1}\sum_{s\neq s_{b}}\hat{\mu}^{*}_{s}$.

  4. 4.

Compute an estimated residual
$\hat{\delta}^{*}_{b}=\mu^{*}-\hat{\mu}^{*}_{s_{b}}$ that quantifies our
prediction error

  5. 5.

Finally, construct another estimate $\hat{\mu}^{*}$ using a new set of
bootstrap samples for all of the original data, including the trial that we
left out when computing the prediction error. Let
$\hat{\mu}^{pred}_{b}=\hat{\mu}^{*}-\hat{\delta}^{*}_{b}$.

We also consider constructing prediction intervals using the wild bootstrap
based on the influence function of the estimator $\hat{\mu}_{s}$. This
procedure follows an outline similar to the simple bootstrap above, except
that the approach of Matsouaka et al. (2022) replaces sampling the data with
replacement. Note that the exact form of the influence function depends on the
estimator. Because we use the outcome model in the simulation study described
below, we use its influence function in implementing the wild bootstrap. See,
e.g., Tsiatis (2006) for additional detail regarding the definition and
derivation of influence functions.

##  4 Simulation Study

To evaluate the performance of the three approaches outlined above, we conduct
a simulation study as follows.

###  4.1 Data Generating Procedure

Covariates: We assume we have three covariates $X_{1}$, $X_{2}$, $X_{3}$ for
each of the $m$ trials and the sample from the target population. Each trial
has 100 participants total, split into two treatment groups, and the target
population sample is of size 1000. The covariates in each setting $s=0,1...,m$
are generated as follows:

  1. 1.

Determine the mean covariate vector $\boldsymbol{\mu}_{s}\in\mathbb{R}^{3}$ in
study $s$ by generating $m$ equally spaced values in the interval $[0,1.5]$
and setting $\boldsymbol{\mu}_{s}$ equal to the $s$th such value (repeating
the value three times for each entry in $\boldsymbol{\mu}_{s}$). Thus, if
$m=3$, then $\boldsymbol{\mu}_{1}=(0,0,0)^{T}$,
$\boldsymbol{\mu}_{2}=(0.75,0.75,0.75)^{T}$, and
$\boldsymbol{\mu}_{3}=(1.5,1.5,1.5)^{T}$. In every scenario, the mean
covariate vector in the target population is
$\boldsymbol{\mu}_{0}=(1,1,1)^{T}$.

  2. 2.

For each of $n_{s}$ participants in study $s$, draw a vector
$(\mathbf{X}_{s})_{i}\sim\mathcal{N}(\boldsymbol{\mu}_{s},\Sigma)$, where

$\Sigma=\begin{pmatrix}1&0.5&0.5\\\ 0.5&1&0.5\\\ 0.5&0.5&1\end{pmatrix}.$

Note that this matrix is identical across studies and the target population.

Potential Outcomes: Equipped with these covariate values, we generate each
participant’s potential outcome under treatment $a$ in study $s$ as a linear
combination of $(X_{s,1},X_{s,2},X_{s,3})_{i}^{T}$. That is,

$\left(Y(a,k^{a}_{s})|S=s,\mathbf{X}=\mathbf{x}\right)_{i}=\beta_{0}+\beta_{1}x_{s,1}+\beta_{2}x_{s,2}+\beta_{3}x_{s,3}+\delta^{a}_{s}+\epsilon,$
(10)

where $\beta_{0}=\beta_{1}=\beta_{2}=\beta_{3}=0.5$ and $\epsilon\sim N(0,1)$.
The value $\delta^{a}_{s}$ is the realized value of random variation in
potential outcomes that results from applying treatment $a$ in study $s$; it
is fixed across all participants in study $s$. The distribution of this
variation is distinct in different simulation scenarios, as detailed below.
Under the specification in (10), the true value for the estimand of interest
is

$\displaystyle E[Y(a,k^{a}_{0})|R=0]=E[E[Y(a,k^{a}_{0})|S=0,\mathbf{X}]]$
$\displaystyle=0.5+0.5E[X_{0,1}]+0.5E[X_{0,2}]+0.5E[X_{0,3}]+\delta^{a}_{0}+E[\epsilon]$
$\displaystyle=0.5+0.5(1)+0.5(1)+0.5(1)+\delta^{a}_{0}+0$
$\displaystyle=2+\delta^{a}_{0}.$ (11)

This is the true expected value under treatment $a$ if members of the target
population participated in an “unobserved” trial. Treatment assignment in each
observed trial to $a$ or placebo proceeds under 1:1 randomization.

Estimation: Using notation defined earlier, we estimate the average observed
outcome under treatment $a$ in study $s$ (denoted
$g^{s}_{a}(x)=E[Y|X=x,S=s,A=a]$) using a correctly specified outcome model

$\hat{g}^{s}_{a}(x)=\mathbf{x}^{T}\hat{\boldsymbol{\beta}}_{s},$

where $\mathbf{x}^{T}=(1,x_{1},x_{2},x_{3})^{T}$ and
$\boldsymbol{\beta}_{s}=(\beta_{s,0},\beta_{s,1},\beta_{s,2},\beta_{s,3})^{T}$.
The expected value of this outcome averaged over the distribution of
covariates in the target population (denoted
$\psi_{s,0}(a)=E[E[Y|X,S=s,A=a]|R=0]$) is estimated as

$\hat{\psi}_{s,0}(a)=\frac{1}{n_{0}}\sum_{i:S_{i}=0}\hat{g}^{s}_{a}(X_{i}).$

As described above, the average of the $\psi_{s,0}(a)$’s over the $m$ studies
is our best guess of the target estimand $\mu_{a,0}(k^{a}_{0})$:

$\hat{\mu}_{a,0}(k^{a}_{0})=\frac{1}{m}\sum_{s=1}^{m}\hat{\psi}_{s,0}(a).$

We then construct a prediction interval centered at $\hat{\mu}_{a,0}$ which
aims to contain $E[Y(a,k^{a}_{0})|R=0]=2+\delta^{a}_{0}$ with some pre-
specified probability, using the three methods described above.

###  4.2 Simulation Scenarios

The above data generating process is highly simplified. It assumes the true
outcome model is identical across scenarios and that we specify this model
correctly. Future work will complicate this setup to investigate other issues,
e.g, model misspecification. For now, this simple scenario focuses our
attention on two main parameters, which we vary across simulations:

  1. 1.

The number of trials in the meta-analysis study dataset. This takes values
across $m=$5, 10, 15, 30, and 50 studies. While we rarely expect to have 50
studies in a meta-analysis, the goal of that scenario is to investigate
whether a given method over/under covers as we collect data from more and more
studies.

  2. 2.

The distribution of the setting-specific variation. Settings include:,
$\text{Unif}[-2,2]$, $N(0,1)$, $\text{Exponential}(1)-1$, and
$\text{Pareto}(1,3)$.

The settings above define $5\times 4=20$ distinct simulation scenarios. We
evaluate each scenario with 1000 artificial datasets simulated from the data
generating process described above. Where applicable, any application of the
bootstrap in our estimation procedure includes 1000 replications. For a given
method of constructing the prediction interval of interest, we estimate the
coverage probability as the proportion of prediction intervals across those
1000 iterations that contain the true parameter value.

###  4.3 Results

Figure 1 displays estimates of coverage for prediction intervals constructed
according to the quantiles of the bootstrap estimates. The second set of plots
does so using using an approximation to the normal distribution,, i.e., the
mean of the bootstrapped estimates $\pm 1.96$ times the standard deviation of
the bootstrapped estimates. In the figure, Method of Moments refers to
intervals constructed using our estimate of $\hat{\gamma}^{2}$, as in (9).
(Note that the Method of Moments results are the same in both figures because
they do not depend on such choices.)

In general, the method of moments intervals based on the corrected estimator
of between-study variance perform better than bootstrap-based alternatives
across different distributions of residual heterogeneity. The normal-based
approximations for the bootstrap approaches also outperform prediction
intervals based on empirical quantiles, though the normal-based approximations
are still outperformed by the method of moments estimator. Although
simulations assuming a very large number of MA studies are helpful for
understanding aspects of our approaches’ asymptotic behavior, we recognize
that 15 or more studies is larger than the vast majority of meta-analysis
datasets, especially given the need for IPD. Thus, these simulation results
suggest applying the method of moments intervals based on the corrected
estimator of between-study variance in most circumstances when researchers are
interested in estimating effects measured in a new, unobserved trial recruited
from the target population.

Although the performance of the two bootstrap alternatives is similar, we note
that the wild bootstrap has non-negligible computational advantages. On a
standard Windows machine with an Intel Core i7-9700 Processor, the wild
bootstrap approach generated prediction intervals more than twice as fast as
the simple bootstrap in a scenario with 5 studies and the sample sizes given
above. The absolute times in such cases were negligible (16 seconds vs. 6
seconds); however, with more studies and a much larger target population
sample, the wild bootstrap becomes far more computationally attractive.

Figure 1: Coverage for prediction intervals constructed according to the
quantiles of the bootstrap estimates. Each plot corresponds to a separate
distribution for setting-specific variation.

Figure 2: Coverage for prediction intervals constructed according to the
quantiles of the normal distribution. Each plot corresponds to a separate
distribution for setting-specific variation.

##  5 Discussion

In this work, we extend recent developments in causally-interpretable meta-
analysis to account for between-study heterogeneity beyond covariate
differences between trials and the target population. Our causal framework
attempts to bridge the structure of traditional random-effects meta-analysis
with causal inference. To do so, we introduce novel estimands, estimates, and
inferential procedures that all explicitly reference the role that a trial’s
setting can play in driving systematic differences between study-specific
parameters.

The impact of between-study heterogeneity can severely limit the
interpretability of treatment effect estimates transported from a collection
of trials to the target population. Under such heterogeneity, estimates
derived from data pooled across distinct trials may reflect a combination of
the idiosyncrasies present in each trial setting. In many contexts, such
heterogeneity persists despite standardizing the trial covariate distributions
to that of a target population. For instance, variation in treatment version
could cause differences in the potential outcomes we would expect to observe
across studies, even after conditioning on all effect-modifiers, as well as
covariates relevant to treatment selection or trial participation.

To address these issues, we begin by defining potential outcomes that
explicitly reference setting-specific heterogeneity and, subsequently,
construct estimators and inferential procedures that take this additional
variability into account. By allowing potential outcomes to depend on both
assigned treatment and the setting of that assignment, we can disaggregate the
effects of a trial from the trial’s population.

Our work is a first step toward clarifying the dual influence of trial setting
and participants in the context of causally-interpretable meta-analysis. With
additional information about protocol-level differences between trials, our
framework might be extended to reference specific trial characteristics of
interest. Future work might consider transporting or up-weighting a specific
source of heterogeneity (e.g. differing levels of adherence) to produce
transported effects most relevant to the clinical setting of interest.
Relatedly, it is also important to clarify the distinction between intention-
to-treat and per-protocol effects when attempting to account for setting-
specific heterogeneity. That is, investigators should distinguish between
studying what would happen if a member of the target population had
participated in a given trial and was compliant or was simply assigned
treatment. The estimands introduced in this work should be interpreted as
defining intention-to-treat transported effects, as our observed data contains
information on treatment assignment alone. However, these methods may be
extended to describe per-protocol effects when such data are available.

In part, our work follows from a simple acknowledgment that the particular
context of an RCT has relevance for causal interpretability. In other words,
estimates transported from RCTs do not necessarily have a generic
interpretation uncoupled from the effects of trial setting. We view our
framework as one step toward developing a causal structure that can
accommodate between-study heterogeneity. In the future, we aim to expand on
this structure to incorporate trial-specific information relevant to a variety
of clinical questions in populations of interest.

## References

Berlin, Jesse A., and Robert M. Golub. 2014. “Meta-Analysis as Evidence:
Building a Better Pyramid.” _JAMA_ 312 (6): 603\.
https://doi.org/10.1001/jama.2014.8167.

Dahabreh, Issa J., Lucia C. Petito, Sarah E. Robertson, Miguel A. Hernán, and
Jon A. Steingrimsson. 2020. “Toward Causally Interpretable Meta-Analysis:
Transporting Inferences from Multiple Randomized Trials to a New Target
Population.” _Epidemiology_ 31 (3): 334–44.
https://doi.org/10.1097/EDE.0000000000001177.

Dahabreh, Issa J., Sarah E. Robertson, and Miguel A. Hernán. 2022.
“Generalizing and Transporting Inferences about the Effects of Treatment
Assignment Subject to Non-Adherence.” https://arxiv.org/abs/2211.04876.

Dahabreh, Issa J., Sarah E. Robertson, Lucia C. Petito, Miguel A. Hernán, and
Jon A. Steingrimsson. 2022. “Efficient and Robust Methods for Causally
Interpretable Meta-Analysis: Transporting Inferences from Multiple Randomized
Trials to a Target Population.” _Biometrics_.
https://doi.org/https://doi.org/10.1111/biom.13716.

Dahabreh, Issa J., James M. Robins, Sebastien J.-P. A. Haneuse, and Miguel A.
Hernán. 2019. “Generalizing Causal Inferences from Randomized Trials:
Counterfactual and Graphical Identification.” arXiv.
http://arxiv.org/abs/1906.10792.

Degtiar, Irina, and Sherri Rose. 2021. “A Review of Generalizability and
Transportability.” https://doi.org/10.48550/ARXIV.2102.11904.

DerSimonian, Rebecca, and Nan Laird. 1986. “Meta-Analysis in Clinical Trials.”
_Controlled Clinical Trials_ 7 (3): 177–88.

Dhruva, Sanket S. 2008. “Variations Between Clinical Trial Participants and
Medicare Beneficiaries in Evidence Used for Medicare National Coverage
Decisions.” _Archives of Internal Medicine_ 168 (2): 136.
https://doi.org/10.1001/archinternmed.2007.56.

Ebell, Mark H, Jay Siwek, Barry D Weiss, Steven H Woolf, Jeffrey Susman,
Bernard Ewigman, and Marjorie Bowman. 2004. “Strength of Recommendation
Taxonomy (SORT): A Patient-Centered Approach to Grading Evidence in the
Medical Literature.” _The Journal of the American Board of Family Practice_ 17
(1): 59–67.

Galson, Steven, and Gregory Simon. 2016. “Real-World Evidence to Guide the
Approval and Use of New Treatments.” _NAM Perspectives_ 6 (10).
https://doi.org/10.31478/201610b.

Higgins, Julian P. T., Simon G. Thompson, and David J. Spiegelhalter. 2009\.
“A Re-Evaluation of Random-Effects Meta-Analysis.” _Journal of the Royal
Statistical Society. Series A (Statistics in Society)_ 172 (1): 137–59.
http://www.jstor.org/stable/30136745.

Kay, Stanley R., Abraham Fiszbein, and Lewis A. Opler. 1987. “The Positive and
Negative Syndrome Scale (PANSS) for Schizophrenia.” _Schizophrenia Bulletin_
13 (2): 261–76. https://doi.org/10.1093/schbul/13.2.261.

Lunny, Carole, Cynthia Ramasubbu, Lorri Puil, Tracy Liu, Savannah Gerrish,
Douglas M. Salzwedel, Barbara Mintzes, and James M. Wright. 2021\. “Over Half
of Clinical Practice Guidelines Use Non-Systematic Methods to Inform
Recommendations: A Methods Study.” Edited by Tim Mathes. _PLOS ONE_ 16 (4):
e0250356. https://doi.org/10.1371/journal.pone.0250356.

Matsouaka, Roland A., Yi Liu, and Yunji Zhou. 2022. “Variance Estimation for
the Average Treatment Effects on the Treated and on the Controls.” arXiv.
https://doi.org/10.48550/ARXIV.2209.10742.

Rao, Poduri S. R. S., Jack Kaplan, and William G. Cochran. 1981. “Estimators
for the One-Way Random Effects Model with Unequal Error Variances.” _Journal
of the American Statistical Association_ 76 (373): 89–97.
http://www.jstor.org/stable/2287050.

Stadler, Eva, Khai Li Chai, Timothy E Schlub, Deborah Cromer, Mark N
Polizzotto, Stephen J Kent, Nicole Skoetz, et al. 2022. “Determinants of
Passive Antibody Effectiveness in SARS-CoV-2 Infection.” _medRxiv_.
https://doi.org/10.1101/2022.03.21.22272672.

Tsiatis, Anastasios A. 2006. “Semiparametric Theory and Missing Data.”

U.S. Food and Drug Administration. 2021. “FDA Guidance on Conduct of Clinical
Trials of Medical Products During the COVID-19 Public Health Emergency.”
Guidance for Industry, Investigators, and Institutional Review Boards
FDA-2020-D-1106-0002.

Vale, C. L., L. H. M. Rydzewska, M. M. Rovers, J. R. Emberson, F. Gueyffier,
L. A. Stewart, and on behalf of the Cochrane IPD Meta-analysis Methods Group.
2015. “Uptake of Systematic Reviews and Meta-Analyses Based on Individual
Participant Data in Clinical Practice Guidelines: Descriptive Study.” _BMJ_
350 (mar06 6): h1088–88. https://doi.org/10.1136/bmj.h1088.

VanderWeele, Tyler J., and Miguel A. Hernán. 2013. “Causal Inference Under
Multiple Versions of Treatment.” _Journal of Causal Inference_ 1 (1): 1–20.
https://doi.org/doi:10.1515/jci-2012-0002.

## Acknowledgements

The authors acknowledge the Minnesota Supercomputing Institute (MSI) at the
University of Minnesota for providing resources that contributed to the
research results reported within this paper. URL: http://www.msi.umn.edu

##  6 Appendix

###  6.1 Full proof of identification formula result for
$\mu_{a,0}(k^{a}_{s})$

$\displaystyle E[Y(a,k^{a}_{s})|R=0]$
$\displaystyle=E[E[Y(a,k^{a}_{s})|X,R=0]|R=0]$ Law of total expectation
$\displaystyle=E[E[Y(a,k^{a}_{s})|X,S=s]|R=0]$ By Assumption 1
$\displaystyle=E[E[Y(a,k^{a}_{s})|X,S=s,A=a]|R=0]$ By Assumption 2
$\displaystyle=E[E[Y(a,k^{a}_{s})|X,S=s,A=a,K^{a}(a)=k^{a}_{s}]|R=0]$ By
Assumption 5 $\displaystyle=E[E[Y|X,S=s,A=a]|R=0]$ By Assumption 3

###  6.2 Full derivation of the approximation of $E[Y(a,K^{a}_{0}(a))|R=0]$

$\displaystyle E[Y(a,K^{a}_{0}(a))|R=0]$
$\displaystyle=\sum_{k^{a}\in\mathcal{K}^{a}}E\left\\{E[Y(a,K^{a}_{0}(a))|R=0,K^{a}_{0}(a)=k^{a}]|R=0\right\\}P(K^{a}_{0}(a)=k^{a})$
$\displaystyle\approx\sum_{k^{a}\in\mathcal{K}^{a}}E\left\\{E[Y(a,K^{a}_{0}(a))|R=0,K^{a}_{0}(a)=k^{a}]|R=0\right\\}\hat{P}(K^{a}_{0}(a)=k^{a})$
$\displaystyle=\sum_{k^{a}\in\mathcal{K}^{a}}E\left\\{E[Y(a,K^{a}_{0}(a))|R=0,K^{a}_{0}(a)=k^{a}]|R=0\right\\}\left\\{\frac{1}{m}\sum_{s=1}^{m}\mathbbm{1}(k^{a}=k^{a}_{s})\right\\}$
$\displaystyle=\sum_{k^{a}\in\mathcal{K}^{a}}\frac{1}{m}\sum_{s=1}^{m}\mathbbm{1}(k^{a}=k^{a}_{s})E\left\\{E[Y(a,K^{a}_{s}(a))|R=0,K^{a}_{s}(a)=k^{a}]|R=0\right\\}$
$\displaystyle=\frac{1}{m}\sum_{s=1}^{m}\sum_{k^{a}\in\mathcal{K}^{a}}\mathbbm{1}(k^{a}=k^{a}_{s})E\left\\{E[Y(a,K^{a}_{s}(a))|R=0,K^{a}_{s}(a)=k^{a}]|R=0\right\\}$
$\displaystyle=\frac{1}{m}\sum_{s=1}^{m}E\left\\{E[Y(a,K^{a}_{s}(a))|R=0,K^{a}_{s}(a)=k^{a}_{s}]|R=0\right\\}$
$\displaystyle=\frac{1}{m}\sum_{s=1}^{m}E[Y(a,k^{a}_{s})|R=0]$
$\displaystyle=\frac{1}{m}\sum_{s=1}^{m}E[E[Y|X,S=s,A=a]|R=0]\hskip
96.73918pt\text{By \eqref{eq: id_one_trial}}$

###  6.3 Full derivation of our estimate of between-study variance in the
presence of correlated estimates

Applying the notation defined in the main manuscript, write the overall sum of
squares of our estimators as

$Q=\sum_{s=1}^{m}\left(\hat{\mu}_{s}-\hat{\mu}\right)^{2}=\sum_{s=1}^{m}\left(\hat{\mu}_{s}-\sum_{l=1}^{m}w_{l}\hat{\mu}_{l}\right)^{2}$

and

$Q_{s}=\left(\hat{\mu}_{s}-\hat{\mu}\right)^{2}=\left(\hat{\mu}_{s}-\sum_{l=1}^{m}w_{l}\hat{\mu}_{l}\right)^{2}.$

Thus, $Q=\sum_{s=1}^{m}Q_{s}$. As in Rao, Kaplan, and Cochran (1981), we
derive an expression for the expectation of $Q$ and an estimator for
$\gamma^{2}$. To that end, let

$\mathbf{a}_{s}=\begin{pmatrix}-w_{1}\\\ \vdots\\\ 1-w_{s}\\\ \vdots\\\
-w_{m}\end{pmatrix}$

and

$\Sigma=\begin{pmatrix}\sigma^{2}_{1}+\gamma^{2}&\sigma_{12}&\cdots&\sigma_{1m}\\\
\sigma_{12}&\sigma^{2}_{2}+\gamma^{2}&\cdots&\sigma_{2m}\\\
\vdots&\vdots&\vdots&\vdots\\\
\sigma_{1m}&\sigma_{2m}&\cdots&\sigma^{2}_{m}+\gamma^{2}\end{pmatrix}$

be the variance-covariance matrix for $\hat{\boldsymbol{\mu}}$ with
$\sigma^{2}_{i}+\gamma^{2}$ as the $i$th diagonal entry and $\sigma_{ij}$ as
the $(i,j)$ off diagonal entry (i.e. $\sigma_{ij}$ is
$\text{Cov}(\hat{\mu}_{i},\hat{\mu}_{j})$). Then,

$\displaystyle\mathbf{a}_{s}\mathbf{a}_{s}^{T}\Sigma$
$\displaystyle=\begin{pmatrix}-w_{1}\\\ \vdots\\\ 1-w_{s}\\\ \vdots\\\
-w_{m}\end{pmatrix}\begin{pmatrix}-w_{1}&\cdots&1-w_{s}&\cdots-
w_{m}\end{pmatrix}\begin{pmatrix}\sigma^{2}_{1}+\gamma^{2}&\sigma_{12}&\cdots&\sigma_{1m}\\\
\sigma_{12}&\sigma^{2}_{2}+\gamma^{2}&\cdots&\sigma_{2m}\\\
\vdots&\vdots&\vdots&\vdots\\\
\sigma_{1m}&\sigma_{2m}&\cdots&\sigma^{2}_{m}+\gamma^{2}\end{pmatrix}$
$\displaystyle=\begin{pmatrix}w_{1}^{2}&\cdots&-w_{1}(1-w_{s})&\cdots&w_{1}w_{m}\\\
\vdots&\vdots&\vdots&\vdots&\vdots\\\
-w_{1}(1-w_{s})&\cdots&(1-w_{s})^{2}&\cdots&-w_{m}(1-w_{s})\\\
\vdots&\vdots&\vdots&\vdots&\vdots\\\
w_{1}w_{m}&\cdots&-w_{m}(1-w_{s})&\cdots&w_{m}^{2}\end{pmatrix}\begin{pmatrix}\sigma^{2}_{1}+\gamma^{2}&\sigma_{12}&\cdots&\sigma_{1m}\\\
\sigma_{12}&\sigma^{2}_{2}+\gamma^{2}&\cdots&\sigma_{2m}\\\
\vdots&\vdots&\vdots&\vdots\\\
\sigma_{1m}&\sigma_{2m}&\cdots&\sigma^{2}_{m}+\gamma^{2}\end{pmatrix}.$

That is, after computing $\mathbf{a}_{s}\mathbf{a}_{s}^{T}$, we obtain a
matrix where the $i$th diagonal entry is $w_{i}^{2}$ except for the $s$th such
entry which is $(1-w_{s})^{2}$. In each row $i$, $i\neq s$, entries $(i,j)$,
$j\neq s$ and $j\neq i$ are given by $w_{i}w_{j}$ while entry $(i,s)$ is
$-w_{i}(1-w_{s})$. In the $s$th row, each entry $(s,j)$, $j\neq s$ is
$-w_{j}(1-w_{s})$. To compute $E[Q_{s}]$, we need to compute
$\text{tr}(\mathbf{a}_{s}\mathbf{a}_{s}^{T}\Sigma)$. Looking at the matrices
in the second equality above, taking the dot product of the first row in the
matrix of weights with the first column of $\Sigma$ we can see that the $i$th
such diagonal entry ($i\neq s$)

$w_{i}^{2}(\sigma^{2}_{i}+\gamma^{2})-w_{i}(1-w_{s})(\sigma_{is})+\sum_{j\neq
i,j\neq s}w_{i}w_{j}\sigma_{ij}.$

The $s$th diagonal entry is then

$(1-w_{s})^{2}(\sigma^{2}_{s}+\gamma^{2})+\sum_{i\neq
s}(-w_{i})(1-w_{s})\sigma_{is}.$

Summing all the diagonal entries,

$\displaystyle\text{tr}(\mathbf{a}_{s}\mathbf{a}_{s}^{T}\Sigma)$
$\displaystyle=(1-w_{s})^{2}(\sigma^{2}_{s}+\gamma^{2})+\sum_{i\neq
s}(-w_{i})(1-w_{s})\sigma_{is}+\sum_{i\neq
s}\left[w_{i}^{2}(\sigma^{2}_{i}+\gamma^{2})-w_{i}(1-w_{s})(\sigma_{is})+\sum_{j\neq
i,j\neq s}w_{i}w_{j}\sigma_{ij}\right]$
$\displaystyle=(1-2w_{s})(\sigma^{2}_{s}+\gamma^{2})+\sum_{i=1}^{m}w_{i}^{2}(\sigma^{2}_{i}+\gamma^{2})-2\sum_{i\neq
s}w_{i}(1-w_{s})\sigma_{is}+\sum_{i\neq s}\left[\sum_{j\neq i,j\neq
s}w_{i}w_{j}\sigma_{ij}\right].$

Now the question is: how can we compute $\gamma^{2}$ using the moment
equations? To begin, let

$C_{s}=-2\sum_{i\neq s}w_{i}(1-w_{s})\sigma_{is}+\sum_{i\neq
s}\left[\sum_{j\neq i,j\neq s}w_{i}w_{j}\sigma_{ij}\right]$

denote all the components of
$\text{tr}(\mathbf{a}_{s}\mathbf{a}_{s}^{T}\Sigma)$ that are constant as a
function of $\gamma^{2}$. Then, recognizing that
$\boldsymbol{\mu}^{T}\mathbf{a}_{i}\mathbf{a}_{i}^{T}\boldsymbol{\mu}=0$, we
have

$E[Q_{s}]=(1-2w_{s})(\sigma^{2}_{s}+\gamma^{2})+\sum_{i=1}^{m}w_{i}^{2}(\sigma^{2}_{i}+\gamma^{2})+C_{s}$

and, moreover,

$\displaystyle E[Q]$ $\displaystyle=\sum_{s=1}^{m}E[Q_{s}]$
$\displaystyle=\sum_{s=1}^{m}\left[(1-2w_{s})(\sigma^{2}_{s}+\gamma^{2})+\sum_{i=1}^{m}w_{i}^{2}(\sigma^{2}_{i}+\gamma^{2})+C_{s}\right]$
$\displaystyle=m\sum_{s=1}^{m}w_{s}^{2}(\sigma^{2}_{s}+\gamma^{2})+\sum_{s=1}^{m}\left[(1-2w_{s})(\sigma^{2}_{s}+\gamma^{2})+C_{s}\right].$

We apply this expression for $E[Q]$ to derive the estimate for between-study
heterogeneity given in the main part of the paper.

###  6.4 Identification for IPW estimator

This identification result proceeds similarly to the analogous IPW
identification result in Dahabreh et al. (2022). The important distinction is
that the weights are distinct for each trial, rather than applying to all
members of the trial population. The result below follows from application of
iterated expectation and our positivity assumption (Assumption 4).

$\displaystyle\psi_{s,0}(a)$ $\displaystyle=E[E[Y|X,S=s,A=a]|R=0]$
$\displaystyle=E\left.\left[E\left[\left.\frac{I(S=s,A=a)Y}{P(S=s|X)P(A=a|X,S=s)}\right|X\right]\right|R=0\right]$
$\displaystyle=\frac{1}{P(R=0)}E\left[I(R=0)E\left.\left[\frac{I(S=s,A=a)Y}{P(S=s|X)P(A=a|X,S=s)}\right|X\right]\right]$
$\displaystyle=\frac{1}{P(R=0)}E\left[E\left.\left[\frac{I(S=s,A=a)YP(R=0|X)}{P(S=s|X)P(A=a|X,S=s)}\right|X\right]\right]$
$\displaystyle=\frac{1}{P(R=0)}E\left[E\frac{I(S=s,A=a)YP(R=0|X)}{P(S=s|X)P(A=a|X,S=s)}\right]$

###  6.5 Full Simulation Results: Quantile-Based Prediction Intervals

###  6.6 Full Simulation Results: Normal-Based Prediction Intervals