# Transportability without positivity: a synthesis of statistical and
simulation modeling

Paul N Zivich1,2, Jessie K Edwards2, Eric T Lofgren3, Stephen R Cole2,
Bonnie E Shook-Sa4, Justin Lessler2,5,6

(1Institute of Global Health and Infectious Diseases, University of North
Carolina at Chapel Hill, Chapel Hill, NC
2Department of Epidemiology, Gillings School of Global Public Health,
University of North Carolina at Chapel Hill, Chapel Hill, NC
2Paul G. Allen School for Global Health, Washington State University, Pullman,
WA
4Department of Biostatistics, Gillings School of Global Public Health,
University of North Carolina at Chapel Hill, Chapel Hill, NC
5Carolina Population Center, University of North Carolina at Chapel Hill,
Chapel Hill, NC
6Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health,
Baltimore, MD
)

###### Abstract

When estimating an effect of an action with a randomized or observational
study, that study is often not a random sample of the desired target
population. Instead, estimates from that study can be transported to the
target population. However, transportability methods generally rely on a
positivity assumption, such that all relevant covariate patterns in the target
population are also observed in the study sample. Strict eligibility criteria,
particularly in the context of randomized trials, may lead to violations of
this assumption. Two common approaches to address positivity violations are
restricting the target population and restricting the relevant covariate set.
As neither of these restrictions are ideal, we instead propose a synthesis of
statistical and simulation models to address positivity violations. We propose
corresponding g-computation and inverse probability weighting estimators. The
restriction and synthesis approaches to addressing positivity violations are
contrasted with a simulation experiment and an illustrative example in the
context of sexually transmitted infection testing. In both cases, the proposed
model synthesis approach accurately addressed the original research question
when paired with a thoughtfully selected simulation model. Neither of the
restriction approaches were able to accurately address the motivating
question. As public health decisions must often be made with imperfect target
population information, model synthesis is a viable approach given a
combination of empirical data and external information based on the best
available knowledge.

## Introduction

To aid in public health decision making, epidemiologists produce quantitative
estimates of the effect of taking different actions (e.g., exposure,
treatments, interventions) for specific populations, commonly referred to as
target populations. However, the available data are often not a random sample
from the desired target population, a problem shared by both randomized trials
and observational studies. Instead, one can attempt to ‘transport’ estimates
from a separate study sample to the desired target population [1, 2, 3, 4, 5,
6, 7]. Methods for transportability have generally proceeded under the
assumption that population membership and potential outcomes are independent
conditional on a set of observed covariates (i.e., conditional
exchangeability) [1, 2, 3, 5]. This assumption requires that all relevant
covariate patterns in the target population are observed in the study sample
(i.e., deterministic positivity [8]), so that individuals in the study sample
(properly reweighted) can ‘stand-in’ for the target population. However,
strict eligibility criteria or practical constraints may lead to violations of
this assumption.

Here, we consider transportability when positivity does not hold. To address
the positivity violation, we review two common approaches: restricting the
target population to regions where the positivity assumption holds, and
restricting the adjustment set of covariates to a subset where the positivity
assumption holds. Both approaches have problems: restricting the target
population no longer addresses the motivating question, and restricting the
adjustment set requires that the excluded covariates induce little-to-no bias.
To avoid these shortcomings, we propose a third approach that uses a
combination or synthesis of statistical and simulation modeling. We apply each
of these approaches to an illustrative example on an intervention to improve
sexually transmitted infection (STI) testing coverage (i.e., proportion
tested).

## Motivating Problem

Consider the following scenario: a clinical colleague approaches us with the
goal of increasing STI testing coverage among their patients. They wish to
offer patients either electronically ordered self-sampling STI test kits
(e-STI) or standard walk-in STI testing via text message. While the clinic is
unable to conduct its own randomized trial comparing these approaches, our
colleague has access to data from a randomized trial comparing these
strategies at a different clinic. They also have a data set containing key
demographic variables of patients from their clinic.

Let $A_{i}$ indicate the STI testing information presented via text message to
patient $i$, with $A_{i}=1$ being e-STI and $A_{i}=0$ being walk-in STI
testing. Let $Y_{i}^{a}$ indicate the (binary) potential outcome of STI
testing uptake under the STI testing option $a\in\\{0,1\\}$, and $Y_{i}$ be
the observed outcome. Additionally, $V_{i}$ indicate age (in years) and
$W_{i}$ indicates gender (male, female) of patient $i$. Say these covariates
were determined to be important factors related to uptake of testing based on
substantive knowledge and differed between the clinic and trial populations.
Finally, $R_{i}$ indicates whether patient $i$ was in the clinic data
($R_{i}=1$) or the randomized trial data ($R_{i}=2$). For inference, we
proceed under a population model, where data are considered to be a random
sample of a much larger population [9].

Our colleague’s question is: what is the difference in the probability of
having an STI test by 6 weeks for e-STI versus walk-in STI testing being
offered among their clinic patients? This parameter can be written as

$\psi=E[Y^{1}|R=1]-E[Y^{0}|R=1]$

with the observed clinic and trial data consisting of $(V_{i},W_{i},R_{i}=1)$
and $(Y_{i},A_{i},V_{i},W_{i},R_{i}=2)$, respectively.

To mimic the described scenario, we use data from the GetTested trial
($n_{2}=2063$) [10], a randomized trial comparing e-STI versus walk-in STI
testing. One of the primary outcomes of the trial was any STI test within 6
weeks of randomization. To focus our illustration, observations were treated
as marginally randomized (GetTested used stratified randomization with
different allocation probabilities, so an actual analysis must incorporate
this to be unbiased) and observations with missing outcome data (324, 16%)
were ignored. This data is referred to as the ‘full data’, and consisted of
the assigned testing option, outcome, age, and gender. To induce a
deterministic positivity violation, we restricted the GetTested trial data to
only men (the ‘restricted data’, $n_{2}=1016$, 49%). Because the GetTested
trial included women, we can benchmark the different approaches based on the
restricted data against the full data. As a stand-in for the clinical target
population, we simulated a data set ($n_{1}=1000$) where age and gender
distributions differed from the GetTested trial (Table 1).

Table 1: Descriptive statistics for the clinic and restricted trial data

| Trial ($n_{2}=1016$)* |  | Clinic ($n_{1}=1000$)
---|---|---|---
| e-STI | Walk-in |  |
Age† | 23 [20, 26] | 22 [22,26] |  | 21 [19,24]
Female | 0 (0%) | 0 (0%) |  | 574 (57%)
Any STI test at 6 weeks‡ | 289 (53%) | 99 (21%) |  | -

STI: sexually transmitted infection.
* The GetTested trial restricted to only males.
† The first number is the median, with the bracketed numbers being the 25th
and 75th percentiles, respectively.
‡ Any STI test completed at 6 weeks post randomization.

### Nonparametric Point Identification

To proceed, $\psi$ is written in terms of the observable data without placing
parametric constraints on the relationships or functional forms between
variables, referred to as nonparametric point identification. For the
randomized trial, we rely on the following assumptions

$Y_{i}=Y_{i}^{a}\text{ if }a=A_{i}$ $E[Y^{a}|R=2]=E[Y^{a}|A=a,R=2]\text{ for
}a\in\\{0,1\\}$ $\Pr(A=a|R=2)>0\text{ for }a\in\\{0,1\\}$

which correspond to causal consistency, marginal exchangeability for
treatment, and positivity for treatment, respectively [8, 11, 12]. These
assumptions are given by design for a marginally randomized trial. To
transport the trial results to the clinic population [1], the following
additional assumptions may be considered

$\displaystyle E[Y^{a}|R=1,V=v,W=w]=E[Y^{a}|R=2,V=v,W=w]$ (1)
$\displaystyle\text{ for all }v,w\text{ where }\Pr(V=v,W=w|R=1)>0$
$\Pr(R=2|V=v,W=w)>0\text{ for all }v,w\text{ where }\Pr(V=v,W=w|R=1)>0$ (2)

where 1 is exchangeability between the trial and target populations and 2 is
the corresponding positivity assumption. As the target population consists of
both men and women but the trial was limited to only men, assumtion 2 is
violated and $\psi$ is not nonparametrically point identified under these
assumptions.

## Potential Solutions

While one always has the option to state that the question cannot be
satisfactorily addressed given the available data, we instead consider three
possible solutions to addressing the motivating research question: restricting
the target population to only men, restricting the covariate set for
exchangeability between the trial and clinic such that we no longer need to
account for gender, and synthesizing statistical and simulation models. To aid
with intuition, the parameter of interest is re-expressed using the law of
total expectation

$E[Y^{a}|R=1]=E[Y^{a}|R=1,W=0]\Pr(W=0|R=1)+E[Y^{a}|R=1,W=1]\Pr(W=1|R=1)$ (3)

As $\Pr(W=w|R=1)$ is identified given the clinic data and $E[Y^{a}|R=1,W=0]$
is identified with the trial data, the challenge is how to identify
$E[Y^{a}|R=1,W=1]$.

### Solution 1: Restrict the Target Population

As $E[Y^{a}|R=1,W=1]$ is not identifiable from the trial, we could revise the
parameter of interest to be the average causal effect among only men at the
clinic, $\psi^{\prime}=E[Y^{1}-Y^{0}|R=1,W=0]$. Effectively, the original
research question is modified by restricting the target population to regions
where positivity is met. Identification of this revised parameter replaces 1
and 2 with

$\displaystyle E[Y^{a}|R=1,V=v,W=0]=E[Y^{a}|R=2,V=v,W=0]$ (4)
$\displaystyle\text{ for all }v\text{ where }\Pr(V=v|R=1,W=0)>0$
$\Pr(R=2|V=v,W=0)>0\text{ for all }v\text{ where }\Pr(V=v|R=1,W=0)>0$ (5)

Here, 5 is not violated, so $\psi^{\prime}$ is nonparametrically point
identified. For consistent estimation, either g-computation or inverse
probability weighting (IPW) estimators for transportability can be applied
(see Appendix A) [1, 13].

While $\psi^{\prime}$ is nonparametrically identified, it is distinct from
$\psi$. Specifically, this revised analysis no longer addresses the original
research question as only a portion of the original target population is
considered. Regardless, decisions regarding STI testing at the clinic must be
made for women (i.e., avoidance of a decision is still a decision). The
unwillingness to address the original question may lead the clinic to restrict
e-STI testing offers to men. If e-STI testing is as or more effective in women
than men, our analytical decision would then lead to a potential increase or
perpetuation of inequalities in STI testing uptake by gender. Alternatively,
if e-STI dissuades women but not men from completing STI tests, then adoption
e-STI testing for all clinic patients could similarly result in inequalities.
By restricting the target population, this analysis provides limited guidance
on the policy the clinic should set.

### Solution 2: Restricted Covariates for Exchangeability

A second option is to modify the conditional exchangeability assumption in 1
by replacing the adjustment set $\\{V,W\\}$ with $\\{V\\}$. Therefore, 1 and 2
are replaced by

$E[Y^{a}|R=1,V=v]=E[Y^{a}|R=2,V=v]\text{ for all }v\text{ where
}\Pr(V=v|R=1)>0$ (6) $\Pr(R=2|V=v)>0\text{ for all }v\text{ where
}\Pr(V=v|R=1)>0$ (7)

This update to exchangeability implies that the effect of STI testing options
is homogenous on the difference scale by gender [14]. Therefore, $\psi$ for
the target population is nonparametrically point identified under these
revised assumptions. Again, either g-computation or IPW estimators could be
used [1, 13].

Restricting the covariate set to age seems unjustified, especially if
discussion with clinical experts led us to originally include gender in the
set of covariates for exchangeability between the trial and clinic. Further,
this analysis has men directly ‘stand-in’ for women, which leads to similar
concerns discussed above if effectiveness differs by gender.

### Solution 3: Model Synthesis

As a third option, we propose a synthesis of statistical and simulation (e.g.,
mechanistic, agent-based, microsimulation) models, where the simulation model
is denoted by $m(a,W_{i}=1;\beta)$. Importantly, this simulation model is
based on knowledge external from the available data. Often, specific values
for the simulation model parameters, $\beta$, may not be known but a range,
set, or distribution of plausible values (denoted as $\tilde{\beta}$) are
feasible to specify. To maintain the generality of our discussion for the
proposed approach, how a simulation model is built is left unspecified, with
further discussion on this process provided at the end of this section. For
estimation, we propose extensions of g-computation and IPW estimators.

#### G-computation

First, we propose an outcome modeling approach akin to standard g-computation
[15]. To generate predicted values of $Y$ under $a$, a combination of
statistical and simulation modeling is used

$f_{a}(V_{i},W_{i};\alpha,\beta)=t\\{s(a,V_{i},W_{i}=0;\alpha)+m(a,W_{i}=1;\beta)\\}$

where $s(a,V_{i},W_{i}=0;\alpha)$ is a statistical model for
$E[Y|A_{i}=a,V_{i},W_{i}=0]$ and $t()$ is a generic transformation (e.g.,
identity for linear models, inverse logit for logistic models). Here, the
simulation model, $m(a,W_{i}=1;\beta)$, shifts the fitted statistical model to
generate predicted values of $E[Y^{a}|V,W=1]$. The statistical and simulation
models can then be used to generate predicted values of $Y$ under
$a\in\\{0,1\\}$ for both $W=0$ and $W=1$. The synthesis g-computation
estimator for $\psi$ is

$\hat{\psi}_{g}=\frac{1}{n_{1}}\sum_{i=1}^{n}\\{f_{1}(V_{i},W_{i};\hat{\alpha},\tilde{\beta})I(R_{i}=1)\\}-\frac{1}{n_{1}}\sum_{i=1}^{n}\\{f_{0}(V_{i},W_{i};\hat{\alpha},\tilde{\beta})I(R_{i}=1)\\}$

where $n$ is the number of observations in the combined clinic and trial data,
and $n_{1}=\sum_{i=1}^{n}I(R_{i}=1)$. To apply this estimator, first the
nuisance parameters for the statistical model, $\alpha$, are estimated using
the trial data. Keeping with the motivating example, a logistic regression
model could be fit predicting STI test completion among men. Next, a
simulation model is developed to update the statistical model values for
women. Then $\hat{\psi}_{g}$ is computed based on
$f_{a}(V_{i},W_{i};\hat{\alpha},\tilde{\beta})$. To incorporate the
uncertainty in $\hat{\alpha}$ and the distribution of $\tilde{\beta}$, we
propose the following semiparametric bootstrap or Monte Carlo procedure.
First, randomly draw $\tilde{\beta}^{*}$ from the specified distribution for
$\tilde{\beta}$, and $\hat{\alpha}^{*}$ from a multivariate normal
distribution based on $\hat{\alpha}$ and the estimated covariance matrix for
$\hat{\alpha}$. Next, resample with replacement the clinic data. Using
$\hat{\alpha}^{*}$, $\tilde{\beta}^{*}$, and the resampled clinic data,
compute $\hat{\psi}_{g}$ as described above. This process is then repeated
many times (e.g., $10^{5}$ times). Results from all the repetitions can then
be jointly summarized by the set of results (e.g., mean and standard
deviation, percentiles) or presented visually (e.g., histogram, violin plot,
boxplot).

A difficulty in application of this g-computation estimator is the specified
simulation model parameters must be conditional on covariates. For example,
consider the following synthesis model

$\displaystyle f_{a}(V_{i},W_{i};\alpha,\beta)=$
$\displaystyle\text{expit}\\{s(a,V_{i},W_{i}=0;\alpha)+m(a,W_{i}=1;\beta)\\}$
$\displaystyle=$
$\displaystyle\text{expit}\\{\alpha_{0}+\alpha_{1}a+\alpha_{2}V_{i}+\beta_{0}W_{i}+\beta_{1}aW_{i}\\}$

Here, $\beta$ is conditional on $V_{i}$. As suggested by the Table 2 Fallacy
and shown in application [16, 17, 18, 19, 20], the direction of conditional
coefficients can be counterintuitive. Additionally, the best available
external information for simulation models are often marginal parameters [21,
22]. Therefore, properly specifying $\beta$ for the synthesis g-computation
estimator may be challenging. To avoid simulation model parameters being
conditional on $V$, we propose an IPW estimator.

#### Inverse Probability Weighting

The synthesis IPW estimator is

$\hat{\psi}_{w}=\frac{1}{n_{1}}\sum_{i=1}^{n}\\{h_{1}(V_{i},W_{i};\hat{\gamma},\tilde{\delta},\hat{\eta})I(R_{i}=1)\\}-\frac{1}{n_{1}}\sum_{i=1}^{n}\\{h_{0}(V_{i},W_{i};\hat{\gamma},\tilde{\delta},\hat{\eta})I(R_{i}=1)\\}$

where

$h_{a}(V_{i},W_{i};\gamma,\delta,\eta)=t\\{s(a,V_{i},W_{i}=0;\gamma,\eta)+m(a,W_{i}=1;\delta)\\}$

a key distinction from g-computation is the chosen statistical model, with the
IPW estimator consisting of a marginal structural model. For example, the
following logistic marginal structural model could be specified

$t\\{s(a,V_{i},W_{i}=0;\gamma,\eta)\\}=\text{expit}\\{\gamma_{0}+\gamma_{1}A_{i}\\}$

where $\eta$ are nuisance parameters for the inverse probability weights and
only apply to the statistical model (i.e., not used in the simulation model).
Therefore,

$\displaystyle h_{a}(V_{i},W_{i};\gamma,\delta,\eta)=$
$\displaystyle\text{expit}\\{s(a,V_{i},W_{i}=0;\gamma,\eta)+m(a,W_{i}=1;\delta)\\}$
$\displaystyle=$
$\displaystyle\text{expit}\\{\gamma_{0}+\gamma_{1}a+\delta_{0}W_{i}+\delta_{1}aW_{i}\\}$

with $\delta$ not conditional on $V_{i}$.

To implement this estimator, the overall inverse probability weights are
estimated. The weights for the illustrative example are

$\frac{1}{\Pr(A_{i}=a|R_{i}=2)}\times\frac{\Pr(R_{i}=2|V_{i},W_{i}=0)}{\Pr(R_{i}=2|V_{i},W_{i}=0)}$

where a logistic model can be used to estimate the probability of membership
in the clinic data restricted to men, $\Pr(R_{i}=1|V_{i},W_{i}=0)$ [1]. The
parameters of a marginal structural model can then be estimated using weighted
maximum likelihood estimation with the clinic data. For point and variance
estimation, an analogous Monte Carlo procedure with $\hat{\gamma}$ and
$\tilde{\delta}$ can be used.

While the synthesis IPW estimator avoids the need for $\delta$ to be
conditional on $V$, $\delta$ are assumed to originate from a population with a
similar age distribution to the clinic. If the chosen distribution for
$\delta$ was based on a population with a different age distribution, then
biased estimates of $\psi$ can result. While differences in the age
distribution could be accounted for through a more richly specified simulation
model, selection of the simulation model parameters for a richer model may be
difficult without equally rick background information.

#### Building a Simulation Model

A primary challenge for application of the proposed estimators is the
development of a simulation model. As the simulation model and corresponding
parameters are inestimable given the available data, external information must
be relied upon. Simulation models can be developed using a variety of
strategies, including building a model based on published tables [23, 24, 25],
or modeling intermediate mechanisms (i.e., a mechanistic model) [26, 27]. We
provide the following advice, but note more detailed discussions on the design
of simulation models exist as part of a rich literature in both epidemiology
and disease ecology [23, 28, 29, 30]. First, the range or distribution of
simulation model parameters should aim to capture both uncertainty in the
chosen model and variability of the information sources. Second, a diverse set
of external information sources may be used to inform selection of plausible
parameter distributions (e.g., trials, observational studies, studies on other
treatments with similar mechanisms of action, pharmacokinetic studies, animal
models, etc.). Third, involvement of a variety of subject-matter experts,
using best practices in eliciting expert views [31, 32], may better capture
the state of the evidence base. For building complex models, seeking out
modeling-specialist collaborators is highly advisable.

Building a simulation model may appear to entail stronger assumptions relative
to restricting the target population or set of covariates. However, ease of
implementation is often confused with the simplicity of assumptions. Both
restriction approaches can be framed as special cases of the synthesis
approach. Restricting the covariate set is equivalent to a simulation model
that always contributes zero to $f_{a}$ or $h_{a}$ (i.e.,
$m(a,W_{i}=1;\beta)=0$ with complete certainty). If one (intentionally or
mistakenly) uses results from the restricted population to draw inference for
the original target population (i.e., assumes $\psi=\psi^{\prime}$), the same
assumptions as the restricted covariate set are implicitly imposed, in
addition to assuming the distribution of $V$ does not differ by $W$ (i.e.,
$V_{i},W_{i}=0)$ stands in for $(V_{i},W_{i}=1)$). When viewed in this manner,
approaches based on restricting the population or covariate set are shown to
require more restrictive assumptions relative to the synthesis approach. So,
even if one is skeptical that such external knowledge could exist, model
synthesis constitutes an alternative to addressing the original question that
clarifies the underlying assumptions and allows for their exploration.

## Application

As stated previously, the restricted trial and clinic data are used for the
illustrative application (Table 1). We estimated the difference in probability
of 6-week STI test uptake between text messages offering e-STI versus walk-in
STI testing in the clinic population. Analyses used Python 3.9.5 (Beaverton,
OR) with the following packages: NumPy [33], SciPy [34], pandas [35], and
delicatessen [36]. The corresponding code is provided at
github.com/pzivich/publications-code. Appendix B includes a brief simulation
study demonstrating the performance of the various estimators.

Age was included in all nuisance models with both a linear and quadratic term,
and no interaction terms. Details on the standard g-computation and IPW
estimators are provided in Appendix A. As a benchmark, we present results that
used the full data. The remainder of analyses used the restricted data. For
the restricted target population approach, estimators were applied to the
clinic data include both men and women. For point and variance estimation, we
used M-estimation [36, 37].

For the synthesis approach, the described g-computation and IPW estimators
were applied with 10,000 Monte Carlo iterations. The synthesis models were

$f_{A_{i}}(V_{i},W_{i};\alpha,\beta)=\text{expit}(\alpha_{0}+\alpha_{1}A_{i}+\alpha_{2}V_{i}+\alpha_{3}V_{i}^{2}+\beta_{0}W_{i}+\beta_{1}A_{i}W_{i})$

for g-computation and

$h_{A_{i}}(V_{i},W_{i};\alpha,\beta)=\text{expit}(\gamma_{0}+\gamma_{1}A_{i}+\delta_{0}W_{i}+\delta_{1}A_{i}W_{i})$

for the IPW estimator. Several different specifications for $\beta$ and
$\delta$ were compared (Table 2). The first assumed a strict null (i.e., no
effect with certainty), where results from the restricted covariate set and
model synthesis were expected to match (up to Monte Carlo error). The second
set proposed a null but uncertain distribution using a trapezoidal
distribution ranging from -2 to 2 (corresponding to odds ratios from 0.14 to
7.4) and was uniform between -1 to 1 [38]. This selection corresponded to a
case where uncertainty was allowed with linearly decreasing belief for values
beyond -1,1 and anything beyond -2,2 is thought to be impossible. The third
set corresponded to a case where accurate (valid and precise) outside
information was available. As a stand-in, we estimated the coefficients of the
synthesis model above using the full GetTested data. The corresponding $\beta$
or $\delta$ estimates were then used as inputs for the simulation model.
Notice that $\beta$ and $\delta$ differ in Table 2 due to the distinction
between marginal and conditional covariate effects. This choice of simulation
model parameters is expected to provide a similar point estimate to the full
data, but variance estimates are expected to differ due to the covariance
between simulation model parameters being ignored and additional uncertainty
in the estimated statistical model parameters in the restricted data. Finally,
the fourth set multiplied the previous by negative one to correspond to
inaccurate external information.

Table 2: Simulation model parameter distributions

|  |  $\beta_{0}$ or $\delta_{0}$ |  $\beta_{1}$ or $\delta_{1}$
---|---|---|---
G-computation |  |
| Strict null | 0 | 0
| Uncertain null | $\text{Trapezoid}(-2,-1,1,2)$ | $\text{Trapezoid}(-2,-1,1,2)$
| Accurate* | $\text{Normal}(\mu=-0.0160,\sigma=0.1761)$ | $\text{Normal}(\mu=-0.6270,\sigma=0.2227)$
| Inaccurate† | $\text{Normal}(\mu=0.0160,\sigma=0.1761)$ | $\text{Normal}(\mu=0.6270,\sigma=0.2227)$
IPW |  |
| Strict null | 0 | 0
| Uncertain null | $\text{Trapezoid}(-2,-1,1,2)$ | $\text{Trapezoid}(-2,-1,1,2)$
| Accurate* | $\text{Normal}(\mu=0.1380,\sigma=0.1931)$ | $\text{Normal}(\mu=-0.6914,\sigma=0.2460)$
| Inaccurate† | $\text{Normal}(\mu=-0.1380,\sigma=0.1931)$ | $\text{Normal}(\mu=0.6914,\sigma=0.2460)$

IPW: inverse probability weighting.
* The accurate simulation model inputs were based on fitting corresponding statistical models using the full GetTested data set.
† The inaccurate simulation model inputs were based on the accurate parameters
but flipping the sign for the center of the distribution.

### Results

Results for g-computation and IPW estimators are presented in Figures 1 and 2,
respectively. Here, the restricted target population and restricted covariate
set approaches similarly overestimated the effectiveness of offering e-STI
testing for the clinic population, as offering e-STI testing was more
effective among men. The similar results were attributable to men and women
having similar age distributions in the clinic population. For the model
synthesis, results were dependent on the choice of simulation model
parameters, as expected. When the simulation model assumed a strict null, the
results were nearly identical to the restriction approaches. When the
uncertain null distributions were used, results were less precise. However,
e-STI testing remained beneficial. In the setting with accurate external
knowledge, the synthesis approach gave a nearly identical point estimate to
the full data approach, but with a larger estimated variance. In the setting
with inaccurate knowledge, the synthesis approach over-estimated the
effectiveness of e-STI testing.

Figure 1: G-computation results for transportation of the average causal
effect to the clinic.

RD: risk difference, 95% CI: confidence interval. Dashed vertical line
indicates the full data point estimate.

Figure 2: Inverse probability weighting estimator results for transportation
of the average causal effect to the clinic.

RD: risk difference, 95% CI: confidence interval. Dashed vertical line
indicates the full data point estimate.

## Discussion

Here, we examined three solutions to dealing with positivity violations in
transportability problems: restrict the target population, restrict the
covariate set for exchangeability, and a synthesis of statistical and
simulation modeling. Model synthesis allows epidemiologists to address
questions where positivity is violated (unlike restricting the target
population) but also avoids modifying the conditional exchangeability
assumption (unlike restricting the covariate set). Even if one is willing to
assume null or homogeneous effects across non-positive regions, model
synthesis allows one to incorporate uncertainty into this assumption. For
estimation with the synthesis approach, we proposed two estimators and
demonstrated their application. Unsurprisingly, the synthesis approach was
sensitive to the quality of the external information.

Previous epidemiologic research has largely focused on comparing statistical
and simulation modeling to each other, or triangulation of statistical and
simulation model results [21, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
51, 52, 53]; but less epidemiologic research has explicitly described how
these methods can be integrated to improve our understanding [7]. Regardless,
a variety of approaches can be framed as syntheses of statistical and
simulation modeling, with notable examples including quantitative bias
analysis, sensitivity analysis, Bayesian methods, and measurement error
correction [54, 55, 56, 57]. Outside of epidemiology, research in physics,
engineering, and climate science have used hybrids of statistical and
simulation modeling (often referred to as ‘phenomenological’ and
‘mathematical’ models, respectively) [58, 59, 60, 61].

Our discussion focused on point identification, but one could instead opt for
partial identification via nonparametric bounds [62, 63], which avoids
exchangeability and positivity assumptions. Here, we could have calculated the
lower and upper bounds for pieces of $\psi$ that violated the positivity
assumption by predicting either 0 or 1 for all women, respectively. These can
then be combined to get lower and upper bounds for $\psi$. Importantly, the
proposed synthesis g-computation and IPW estimators correspond to these bounds
when the simulation model always generates predictions of 0 (or 1) for women.
Therefore, the synthesis approach can also be viewed as a way to narrow the
nonparametric bounds via external information.

Outside information was incorporated through a simulation model but there are
alternative frameworks for integrating external information. For example, if
another trial on e-STI testing among only women was available, the parameter
of interest could have been estimated using a fusion design [64], whereby the
two trials and clinic data are combined. In this case, the fusion design is
likely to be preferred since it means the parameter could be nonparametrically
identified. However, access to individual-level data of both trials may not be
possible. Simulation models offer a viable alternative that can be
accomplished with only published summary results [65].

Future areas ripe for extending the proposed synthesis framework include
addressing other sources of systematic errors, alternative estimators, and
more precisely stating the inferential model. Here, we focused on
transportability under and assumption of marginal randomization of treatment.
However, conditional exchangeability and positivity assumptions are commonly
used to address other systematic errors, including confounding, missing data,
and measurement error [8]. Returning to the example, observations in the
GetTested trial with missing outcomes were dropped, but this analysis strategy
would not be recommended in practice [66]. To account for missing outcomes by
observed covariates, one could consider multiple imputation or inverse
probability of missingness weights [67]. If missingness depended on unobserved
covariates or the missing values themselves; then sensitivity analyses [56,
68], or extensions of the proposed synthesis could be applied. While some
extensions of model synthesis may be trivial, there are important details to
consider. For example, conditional exchangeability for treatment and the
average causal effect requires as symmetric positivity assumption, not present
in the positivity assumption for transportability [8, 69]. Additionally,
positivity for some systematic errors may remain uncorrectable (e.g., the
average causal effect of pregnancy on blood pressure in a population that
includes persons unable to become pregnant). Second, alternative estimators
might be of interest. Here, we proposed extensions of g-computation and IPW.
Further variations on the statistical models, simulation models, or Monte
Carlo procedure are also possible. The use of alternative statistical models,
machine learning, and potential for multiply robust estimation are of interest
[70, 71]. Further, testable implications for simulation models, similar to the
natural course for the g-formula [72] or balance with IPW [73], are also of
interest. Finally, the inferential specifics for model synthesis were left
largely unspecified. While we refer to the intervals as ‘confidence
intervals’, there is no reason to believe that these intervals will always
operate the same as standard frequentist confidence intervals, an issue shared
with semi-Bayes and other interval estimation methods [55, 74].

Decisions must be made, with or without data on the entire target population.
The logical positivist dream of the data alone guiding all actions is long
dead [75, 76, 77]. To make any reasonable progress in learning about causal
effects, we depend on relevant knowledge external to the data [78]. Within
transportability and causal inference more generally, external information has
often been limited to the choice of a statistical model and the relations
between variables (i.e., arrows in a causal diagram). Here, we extended this
idea to include substantive knowledge outside of the data to address
positivity violations through joint use of statistical and simulation
modeling.

## Acknowledgments

This work was supported in part by T32-AI007001 (PNZ), R01-AI157758 (JKE, BES,
SRC), R01- GM140564 (JKE, JL), and R35-GM147013 (ETL). Corresponding code is
available at https://github.com/pzivich/publications-code

## References

  * [1] D. Westreich, J. K. Edwards, C. R. Lesko, E. Stuart, and S. R. Cole, “Transportability of Trial Results Using Inverse Odds of Sampling Weights,” American Journal of Epidemiology, vol. 186, pp. 1010–1014, Oct. 2017.
  * [2] I. J. Dahabreh, S. E. Robertson, E. J. Tchetgen Tchetgen, E. A. Stuart, and M. A. Hernán, “Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals,” Biometrics, vol. 75, no. 2, pp. 685–694, 2019.
  * [3] I. J. Dahabreh, L. C. Petito, S. E. Robertson, M. A. Hernán, and J. A. Steingrimsson, “Toward Causally Interpretable Meta-analysis: Transporting Inferences from Multiple Randomized Trials to a New Target Population,” Epidemiology, vol. 31, pp. 334–344, May 2020.
  * [4] E. Bareinboim and J. Pearl, “Causal inference and the data-fusion problem,” Proceedings of the National Academy of Sciences, vol. 113, no. 27, pp. 7345–7352, 2016.
  * [5] I. Degtiar and S. Rose, “A Review of Generalizability and Transportability,” Annual Review of Statistics and Its Application, vol. 10, no. 1, p. null, 2023.
  * [6] N. Keiding and T. A. Louis, “Perils and potentials of self-selected entry to epidemiological studies and surveys,” Journal of the Royal Statistical Society. Series A (Statistics in Society), vol. 179, no. 2, pp. 319–376, 2016.
  * [7] M. Cerdá and K. M. Keyes, “Systems Modeling to Advance the Promise of Data Science in Epidemiology,” American Journal of Epidemiology, vol. 188, pp. 862–865, May 2019.
  * [8] P. N. Zivich, S. R. Cole, and D. Westreich, “Positivity: Identifiability and Estimability,” arXiv:2207.05010 [stat], July 2022.
  * [9] E. L. Lehmann, Elements of large-sample theory.  Springer Science & Business Media, 2004.
  * [10] E. Wilson, C. Free, T. P. Morris, J. Syred, I. Ahamed, A. S. Menon-Johansson, M. J. Palmer, S. Barnard, E. Rezel, and P. Baraitser, “Internet-accessed sexually transmitted infection (e-STI) testing and results service: A randomised, single-blind, controlled trial,” PLOS Medicine, vol. 14, p. e1002479, Dec. 2017.
  * [11] M. A. Hernán and J. M. Robins, “Estimating causal effects from epidemiological data,” Journal of Epidemiology & Community Health, vol. 60, pp. 578–586, July 2006.
  * [12] S. R. Cole and C. E. Frangakis, “The consistency statement in causal inference: a definition or an assumption?,” Epidemiology, vol. 20, no. 1, pp. 3–5, 2009.
  * [13] I. J. Dahabreh, S. E. Robertson, J. A. Steingrimsson, E. A. Stuart, and M. A. Hernán, “Extending inferences from a randomized trial to a new target population,” Statistics in Medicine, vol. 39, no. 14, pp. 1999–2014, 2020.
  * [14] M. Webster-Clark and A. Breskin, “Directed Acyclic Graphs, Effect Measure Modification, and Generalizability,” American Journal of Epidemiology, vol. 190, pp. 322–327, Feb. 2021.
  * [15] J. M. Snowden, S. Rose, and K. M. Mortimer, “Implementation of G-computation on a simulated data set: demonstration of a causal inference technique,” American Journal of Epidemiology, vol. 173, no. 7, pp. 731–738, 2011\.
  * [16] D. Westreich and S. Greenland, “The table 2 fallacy: presenting and interpreting confounder and modifier coefficients,” American Journal of Epidemiology, vol. 177, pp. 292–298, Feb. 2013.
  * [17] G. Bandoli, K. Palmsten, C. D. Chambers, L. L. Jelliffe-Pawlowski, R. J. Baer, and C. A. Thompson, “Revisiting the Table 2 Fallacy: A Motivating Example Examining Preeclampsia and Preterm Birth,” Paediatric and Perinatal Epidemiology, vol. 32, pp. 390–397, July 2018.
  * [18] E. J. Williamson, A. J. Walker, K. Bhaskaran, S. Bacon, C. Bates, C. E. Morton, H. J. Curtis, A. Mehrkar, D. Evans, P. Inglesby, J. Cockburn, H. I. McDonald, B. MacKenna, L. Tomlinson, I. J. Douglas, C. T. Rentsch, R. Mathur, A. Y. S. Wong, R. Grieve, D. Harrison, H. Forbes, A. Schultze, R. Croker, J. Parry, F. Hester, S. Harper, R. Perera, S. J. W. Evans, L. Smeeth, and B. Goldacre, “Factors associated with COVID-19-related death using OpenSAFELY,” Nature, vol. 584, pp. 430–436, Aug. 2020.
  * [19] D. Westreich, J. K. Edwards, and M. van Smeden, “Comment on Williamson et al. (OpenSAFELY): The Table 2 Fallacy in a Study of COVID-19 Mortality Risk Factors,” Epidemiology, vol. 32, pp. e1–e2, Jan. 2021\.
  * [20] S. C. Keller, D. Williams, M. Gavgani, D. Hirsch, J. Adamovich, D. Hohl, A. P. Gurses, and S. E. Cosgrove, “Rates of and Risk Factors for Adverse Drug Events in Outpatient Parenteral Antimicrobial Therapy,” Clinical Infectious Diseases, vol. 66, pp. 11–19, Jan. 2018.
  * [21] E. J. Murray, J. M. Robins, G. R. Seage, K. A. Freedberg, and M. A. Hernan, “A Comparison of Agent-Based Models and the Parametric G-Formula for Causal Inference,” American Journal of Epidemiology, vol. 186, pp. 131–142, July 2017.
  * [22] E. J. Murray, J. M. Robins, G. R. Seage, K. A. Freedberg, and M. A. Hernán, “The Challenges of Parameterizing Direct Effects in Individual-Level Simulation Models,” Medical Decision Making, vol. 40, pp. 106–111, Jan. 2020.
  * [23] E. M. Krijkamp, F. Alarid-Escudero, E. A. Enns, H. J. Jalal, M. M. Hunink, and P. Pechlivanoglou, “Microsimulation modeling for health decision sciences using R: a tutorial,” Medical Decision Making, vol. 38, pp. 400–422, Apr. 2018.
  * [24] a. Çağlayan, H. Terawaki, Q. Chen, A. Rai, T. Ayer, and C. R. Flowers, “Microsimulation Modeling in Oncology,” JCO Clinical Cancer Informatics, vol. 2, p. CCI.17.00029, Mar. 2018.
  * [25] A. H. Grummon, N. R. Smith, S. D. Golden, L. Frerichs, L. S. Taillie, and N. T. Brewer, “Health warnings on sugar-sweetened beverages: simulation of impacts on diet and obesity among us adults,” American Journal of Preventive Medicine, vol. 57, no. 6, pp. 765–774, 2019.
  * [26] J. Lessler and D. A. T. Cummings, “Mechanistic Models of Infectious Disease and Their Impact on Public Health,” American Journal of Epidemiology, vol. 183, pp. 415–422, Mar. 2016.
  * [27] C. Kirkeby, V. J. Brookes, M. P. Ward, S. Dürr, and T. Halasa, “A Practical Introduction to Mechanistic Modeling of Disease Transmission in Veterinary Science,” Frontiers in Veterinary Science, vol. 7, p. 546651, Jan. 2021.
  * [28] M. Roberts, L. B. Russell, A. D. Paltiel, M. Chambers, P. McEwan, and M. Krahn, “Conceptualizing a Model: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force–2,” Medical Decision Making, vol. 32, pp. 678–689, Sept. 2012.
  * [29] S. F. Railsback and V. Grimm, Agent-Based and Individual-Based Modeling: A Practical Introduction.  Princeton University Press, 2011.
  * [30] R. B. Slayton, J. J. O’Hagan, S. Barnes, S. Rhea, R. Hilscher, M. Rubin, E. Lofgren, B. Singh, A. Segre, P. Paul, and for the Centers for Disease Control and Prevention MInD-Healthcare Program, “Modeling Infectious Diseases in Healthcare Network (MInD-Healthcare) Framework for Describing and Reporting Multidrug-resistant Organism and Healthcare-Associated Infections Agent-based Modeling Methods,” Clinical Infectious Diseases, vol. 71, pp. 2527–2532, Nov. 2020.
  * [31] L. Bojke, M. Soares, K. Claxton, A. Colson, A. Fox, C. Jackson, D. Jankovic, A. Morton, L. Sharples, and A. Taylor, Good practice in structured expert elicitation: learning from the available guidance.  NIHR Journals Library, June 2021.
  * [32] A. O’Hagan, “Expert Knowledge Elicitation: Subjective but Scientific,” The American Statistician, vol. 73, pp. 69–81, Mar. 2019.
  * [33] C. R. Harris, K. J. Millman, S. J. Van Der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, et al., “Array programming with NumPy,” Nature, vol. 585, no. 7825, pp. 357–362, 2020.
  * [34] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, and J. Bright, “SciPy 1.0: fundamental algorithms for scientific computing in Python,” Nature Methods, vol. 17, no. 3, pp. 261–272, 2020.
  * [35] W. McKinney, “Data Structures for Statistical Computing in Python,” Proceedings of the 9th Python in Science Conference, pp. 56–61, 2010\.
  * [36] P. N. Zivich, M. Klose, S. R. Cole, J. K. Edwards, and B. E. Shook-Sa, “Delicatessen: M-Estimation in Python,” arXiv:2203.11300 [stat], Mar. 2022.
  * [37] L. A. Stefanski and D. D. Boos, “The Calculus of M-Estimation,” The American Statistician, vol. 56, pp. 29–38, Feb. 2002.
  * [38] M. P. Fox, T. L. Lash, and S. Greenland, “A method to automate probabilistic sensitivity analyses of misclassified binary variables,” International Journal of Epidemiology, vol. 34, pp. 1370–1376, Dec. 2005.
  * [39] S. J. Mooney, A. B. Shev, K. M. Keyes, M. Tracy, and M. Cerdá, “G-computation and agent-based modeling for social epidemiology: Can population interventions prevent post-traumatic stress disorder?,” American Journal of Epidemiology, Aug. 2021.
  * [40] E. J. Murray, B. D. L. Marshall, and A. L. Buchanan, “Emulating Target Trials to Improve Causal Inference From Agent-Based Models,” American Journal of Epidemiology, vol. 190, pp. 1652–1658, Aug. 2021\.
  * [41] E. H. Ip, H. Rahmandad, D. A. Shoham, R. Hammond, T. T.-K. Huang, Y. Wang, and P. L. Mabry, “Reconciling Statistical and Systems Science Approaches to Public Health,” Health Education & Behavior, vol. 40, pp. 123S–131S, Oct. 2013.
  * [42] M. A. Hernán, “Invited Commentary: Agent-Based Models for Causal Inference—Reweighting Data and Theory in Epidemiology,” American Journal of Epidemiology, vol. 181, pp. 103–105, Jan. 2015.
  * [43] A. L. Buchanan, S. Bessey, W. C. Goedel, M. King, E. J. Murray, S. R. Friedman, M. E. Halloran, and B. D. L. Marshall, “Disseminated Effects in Agent-Based Models: A Potential Outcomes Framework and Application to Inform Preexposure Prophylaxis Coverage Levels for HIV Prevention,” American Journal of Epidemiology, vol. 190, pp. 939–948, May 2021.
  * [44] A. M. El-Sayed, P. Scarborough, L. Seemann, and S. Galea, “Social network analysis and agent-based modeling in social epidemiology,” Epidemiologic Perspectives & Innovation, vol. 9, p. 1, Feb. 2012.
  * [45] E. T. Lofgren, B. D. Marshall, and S. Galea, “Re: Integrating Complex Systems Thinking into Epidemiologic Research,” Epidemiology, vol. 28, no. 5, p. e50, 2017.
  * [46] S. F. Ackley, J. Lessler, and M. M. Glymour, “Dynamical Modeling as a Tool for Inferring Causation,” American Journal of Epidemiology, vol. 191, pp. 1–6, Jan. 2022.
  * [47] M. E. Halloran, K. Auranen, S. Baird, N. E. Basta, S. E. Bellan, R. Brookmeyer, B. S. Cooper, V. DeGruttola, J. P. Hughes, J. Lessler, E. T. Lofgren, I. M. Longini, J.-P. Onnela, B. Özler, G. R. Seage, T. A. Smith, A. Vespignani, E. Vynnycky, and M. Lipsitch, “Simulations for designing and interpreting intervention trials in infectious diseases,” BMC Medicine, vol. 15, p. 223, Dec. 2017.
  * [48] K. M. Keyes, M. Tracy, S. J. Mooney, A. Shev, and M. Cerdá, “Invited Commentary: Agent-Based Models—Bias in the Face of Discovery,” American Journal of Epidemiology, vol. 186, pp. 146–148, July 2017.
  * [49] J. K. Edwards, C. R. Lesko, and A. P. Keil, “Invited Commentary: Causal Inference Across Space and Time—Quixotic Quest, Worthy Goal, or Both?,” American Journal of Epidemiology, vol. 186, pp. 143–145, July 2017.
  * [50] E. T. Lofgren, M. E. Halloran, C. M. Rivers, J. M. Drake, T. C. Porco, B. Lewis, W. Yang, A. Vespignani, J. Shaman, J. N. S. Eisenberg, M. C. Eisenberg, M. Marathe, S. V. Scarpino, K. A. Alexander, R. Meza, M. J. Ferrari, J. M. Hyman, L. A. Meyers, and S. Eubank, “Mathematical models: A key tool for outbreak response,” Proceedings of the National Academy of Sciences, vol. 111, pp. 18095–18096, Dec. 2014.
  * [51] K. F. Arnold, W. J. Harrison, A. J. Heppenstall, and M. S. Gilthorpe, “DAG-informed regression modelling, agent-based modelling and microsimulation modelling: a critical comparison of methods for causal inference,” International Journal of Epidemiology, vol. 48, no. 1, pp. 243–253, 2019.
  * [52] S. F. Ackley, E. R. Mayeda, L. Worden, W. T. A. Enanoria, M. M. Glymour, and T. C. Porco, “Compartmental Model Diagrams as Causal Representations in Relation to DAGs,” Epidemiologic Methods, vol. 6, Dec. 2017.
  * [53] S. J. Mooney, A. B. Shev, K. M. Keyes, M. Tracy, and M. Cerdá, “G-Computation and Agent-Based Modeling for Social Epidemiology: Can Population Interventions Prevent Posttraumatic Stress Disorder?,” American Journal of Epidemiology, vol. 191, pp. 188–197, Jan. 2022.
  * [54] S. Greenland, “Bayesian perspectives for epidemiologic research: III. Bias analysis via missing-data methods,” International Journal of Epidemiology, vol. 38, no. 6, pp. 1662–1673, 2009.
  * [55] S. Greenland, “Interval estimation by simulation as an alternative to and extension of confidence intervals,” International Journal of Epidemiology, vol. 33, pp. 1389–1397, Dec. 2004.
  * [56] J. M. Robins, A. Rotnitzky, and D. O. Scharfstein, “Sensitivity Analysis for Selection bias and unmeasured Confounding in missing Data and Causal inference models,” in Statistical Models in Epidemiology, the Environment, and Clinical Trials (M. E. Halloran and D. Berry, eds.), The IMA Volumes in Mathematics and its Applications, (New York, NY), pp. 1–94, Springer, 2000.
  * [57] B. E. Shepherd, M. W. Redman, and D. P. Ankerst, “Does Finasteride Affect the Severity of Prostate Cancer? A Causal Sensitivity Analysis,” Journal of the American Statistical Association, vol. 103, pp. 1392–1404, Dec. 2008.
  * [58] S. Rahmstorf, “A Semi-Empirical Approach to Projecting Future Sea-Level Rise,” Science, vol. 315, pp. 368–370, Jan. 2007.
  * [59] W. Wright and M. Potapczuk, “Semi-Empirical Modelling of SLD Physics,” in 42nd AIAA Aerospace Sciences Meeting and Exhibit, American Institute of Aeronautics and Astronautics, 2004.
  * [60] M. G. Sausen, F. B. Scheufele, H. J. Alves, M. G. A. Vieira, M. G. C. da Silva, F. H. Borba, and C. E. Borba, “Efficiency maximization of fixed-bed adsorption by applying hybrid statistical-phenomenological modeling,” Separation and Purification Technology, vol. 207, pp. 477–488, Dec. 2018\.
  * [61] R. Rezaei, C. Hayduk, E. Alkan, T. Kemski, T. Delebinski, and C. Bertram, “Hybrid Phenomenological and Mathematical-Based Modeling Approach for Diesel Emission Prediction,” in SAE Technical Paper, Apr. 2020.
  * [62] C. F. Manski, “Nonparametric Bounds on Treatment Effects,” The American Economic Review, vol. 80, no. 2, pp. 319–323, 1990.
  * [63] S. R. Cole, M. G. Hudgens, J. K. Edwards, M. A. Brookhart, D. B. Richardson, D. Westreich, and A. A. Adimora, “Nonparametric Bounds for the Risk Function,” American Journal of Epidemiology, vol. 188, no. 4, pp. 632–636, 2019.
  * [64] S. R. Cole, J. K. Edwards, A. Breskin, S. Rosin, P. N. Zivich, B. E. Shook-Sa, and M. G. Hudgens, “Illustration of Two Fusion Designs and Estimators,” American Journal of Epidemiology, p. kwac067, Apr. 2022\.
  * [65] E. T. Lofgren, M. Mietchen, K. V. Dicks, R. Moehring, D. Anderson, and Centers for Disease Control Modeling Infectious Diseases in Healthcare Program, “Estimated Methicillin-Resistant Staphylococcus aureus Decolonization in Intensive Care Units Associated With Single-Application Chlorhexidine Gluconate or Mupirocin,” JAMA Network Open, vol. 4, p. e210652, Mar. 2021.
  * [66] J. H. Ware, D. Harrington, D. J. Hunter, and R. B. D’Agostino, “Missing Data,” New England Journal of Medicine, vol. 367, pp. 1353–1354, Oct. 2012.
  * [67] N. J. Perkins, S. R. Cole, O. Harel, E. J. Tchetgen Tchetgen, B. Sun, E. M. Mitchell, and E. F. Schisterman, “Principled Approaches to Missing Data in Epidemiologic Studies,” American Journal of Epidemiology, vol. 187, p. 568, Mar. 2018.
  * [68] S. Greenland, “Basic Methods for Sensitivity Analysis of Biases,” International Journal of Epidemiology, vol. 25, pp. 1107–1116, Dec. 1996.
  * [69] P. N. Zivich, B. E. Shook-Sa, J. K. Edwards, D. Westreich, and S. R. Cole, “On the Use of Covariate Supersets for Identification Conditions,” Epidemiology, vol. 33, pp. 559–562, Mar. 2022.
  * [70] H. Bang and J. M. Robins, “Doubly robust estimation in missing data and causal inference models,” Biometrics, vol. 61, no. 4, pp. 962–973, 2005.
  * [71] P. N. Zivich and A. Breskin, “Machine Learning for Causal Inference: On the Use of Cross-fit Estimators,” Epidemiology, vol. 32, pp. 393–401, May 2021.
  * [72] A. P. Keil, J. K. Edwards, D. R. Richardson, A. I. Naimi, and S. R. Cole, “The parametric G-formula for time-to-event data: towards intuition with a worked example,” Epidemiology, vol. 25, no. 6, pp. 889–897, 2014.
  * [73] P. C. Austin and E. A. Stuart, “Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies,” Statistics in Medicine, vol. 34, no. 28, pp. 3661–3679, 2015.
  * [74] S. Greenland, “Bayesian perspectives for epidemiological research: I. Foundations and basic methods,” International Journal of Epidemiology, vol. 35, pp. 765–775, June 2006.
  * [75] W. V. Quine, “Main Trends in Recent Philosophy: Two Dogmas of Empiricism,” The Philosophical Review, vol. 60, no. 1, pp. 20–43, 1951.
  * [76] J. Robins and L. Wasserman, “On the impossibility of inferring causation from association without background knowledge,” in Computation, Causation, and Discovery (P. Glymour and C. G., eds.), pp. 305–231, The MIT Press, 1999.
  * [77] O. Fajardo-Fontiveros, I. Reichardt, H. R. D. L. Rios, J. Duch, M. Sales-Pardo, and R. Guimera, “Fundamental limits to learning closed-form mathematical models from data,” arXiv:2204.02704 [cond-mat, physics:physics], Apr. 2022.
  * [78] J. M. Robins, “Data, Design, and Background Knowledge in Etiologic Inference,” Epidemiology, vol. 12, no. 3, pp. 313–320, 2001.
  * [79] T. P. Morris, I. R. White, and M. J. Crowther, “Using simulation studies to evaluate statistical methods,” Statistics in Medicine, vol. 38, no. 11, pp. 2074–2102, 2019.

## Appendix

### Appendix A: Estimators

In the applied example and simulations, M-estimation is used for the
restricted population and covariate set estimators. Briefly, an M-estimator,
$\hat{\theta}$, is the solution for $\theta$ in the estimating equation
$\sum_{i=1}^{n}\varphi(O_{i};\theta)=0$; where $O_{i}$ is the observed data
for $n$ independent and identically distributed units, $\theta$ is a $k$
dimensional vector of parameters, and $\varphi$ is a known $k$-dimensional
vector of estimating functions that do not depend on $i$ [36, 37]. The
variances of the parameter estimates can be estimated using the empirical
sandwich estimator

$V(O_{i};\hat{\theta})=\frac{1}{n}\left[B(O_{i};\hat{\theta})^{-1}F(O_{i};\hat{\theta})\\{B(O_{i};\hat{\theta})^{-1}\\}^{T}\right]$

where the ‘bread’ is

$B(O_{i};\hat{\theta})=\frac{1}{n}\sum_{i=1}^{n}\\{-\varphi(O_{i};\hat{\theta})\\}$

with $\varphi^{\prime}(O_{i};\hat{\theta})$ indicating the matrix of partial
derivatives, and the ‘filling’ is

$F(O_{i};\hat{\theta})=\frac{1}{n}\sum_{i=1}^{n}\\{\varphi(O_{i};\hat{\theta})\varphi(O_{i};\hat{\theta})^{T}\\}$

Importantly, the sandwich estimator allows for the propagation of uncertainty
between parameters that depend on each other. In short, a consistent variance
can be easily estimated while avoiding more computationally demanding
procedures, like the nonparametric bootstrap.

#### Restrict the Target Population

For the restricted target population g-computation estimator, the stacked
estimating functions were

$\varphi(O_{i};\theta)=\begin{bmatrix}I(R_{i}=2)\left(\left[Y_{i}-\text{expit}\left\\{g(A_{i},V_{i})^{T}\alpha\right\\}\right]g(A_{i},V_{i})\right)\\\
I(R_{i}=1,W_{i}=0)\left[\text{expit}\left\\{g(0,V_{i})^{T}\alpha\right\\}-\theta_{0}\right]\\\
I(R_{i}=1,W_{i}=0)\left[\text{expit}\left\\{g(1,V_{i})^{T}\alpha\right\\}-\theta_{1}\right]\\\
(\theta_{1}-\theta_{0})-\theta_{2}\end{bmatrix}$

where $\theta=(\alpha,\theta_{0},\theta_{1},\theta_{2})$,
$g(A_{i},V_{i})=(1,A_{i},V_{i},V_{i}^{2})$ is the design matrix and
$\text{expit}(b)=\frac{\exp(b)}{1+\exp(b)}$. The first estimating function is
a logistic regression model for $Y_{i}$ among the trial data. The second and
third estimating functions are the predicted mean under $A_{i}=0$ and
$A_{i}=1$ among men in the clinic data, respectively. The last estimating
function is the risk difference.

The inverse probability weighting (IPW) estimator consisted of the following
stacked estimating functions

$\varphi(O_{i};\theta)=\begin{bmatrix}I(R_{i}=2)\left\\{A_{i}-\text{expit}(\mu)\right\\}\\\
I(W_{i}=0)\left(\left[I(R_{i}=1)-\text{expit}\left\\{l(V_{i})^{T}\sigma\right\\}\right]l(V_{i})\right)\\\
(Y_{i}-\theta_{0})\frac{I(R_{i}=2)\text{expit}\left\\{l(V_{i})^{T}\sigma\right\\}}{1-\text{expit}\left\\{l(V_{i})^{T}\sigma\right\\}}\frac{I(A_{i}=0)}{1-\text{expit}(\mu)}\\\
(Y_{i}-\theta_{1})\frac{I(R_{i}=2)\text{expit}\left\\{l(V_{i})^{T}\sigma\right\\}}{1-\text{expit}\left\\{l(V_{i})^{T}\sigma\right\\}}\frac{I(A_{i}=1)}{\text{expit}(\mu)}\\\
(\theta_{1}-\theta_{0})-\theta_{2}\end{bmatrix}$

where $\theta=(\mu,\sigma,\theta_{0},\theta_{1},\theta_{2})$ and
$l(V_{i})=(1,V_{i},V_{i}^{2})$ is the design matrix. The first estimating
function is an intercept-only regression model for assigned STI testing in the
trial data. The second estimating function is a logistic model for the
conditional probability of being in the clinic, fit using only men. The third
and fourth estimating functions are the Hajek estimators for $A_{i}=0$ and
$A_{i}=1$, respectively. The last estimating function is the risk difference.

#### Restricting the Covariate Set

The g-computation estimator for the restricted covariate set was implemented
via the following stacked estimating functions

$\varphi(O_{i};\theta)=\begin{bmatrix}I(R_{i}=2)\left(\left[Y_{i}-\text{expit}\left\\{g(A_{i},V_{i})^{T}\alpha\right\\}\right]g(A_{i},V_{i})\right)\\\
I(R_{i}=1)\left[\text{expit}\left\\{g(0,V_{i})^{T}\alpha\right\\}-\theta_{0}\right]\\\
I(R_{i}=1)\left[\text{expit}\left\\{g(1,V_{i})^{T}\alpha\right\\}-\theta_{1}\right]\\\
(\theta_{1}-\theta_{0})-\theta_{2}\end{bmatrix}$

The key difference from the previous g-computation estimating functions is
predictions are generated in the second and third equations for both men and
women.

The stacked estimating functions for the IPW estimator

$\varphi(O_{i};\theta)=\begin{bmatrix}I(R_{i}=2)\left\\{A_{i}-\text{expit}(\mu)\right\\}\\\
\left[I(R_{i}=1)-\text{expit}\left\\{l(V_{i})^{T}\sigma\right\\}\right]l(V_{i})\\\
(Y_{i}-\theta_{0})\frac{I(R_{i}=2)\text{expit}\left\\{l(V_{i})^{T}\sigma\right\\}}{1-\text{expit}\left\\{l(V_{i})^{T}\sigma\right\\}}\frac{I(A_{i}=0)}{1-\text{expit}(\mu)}\\\
(Y_{i}-\theta_{1})\frac{I(R_{i}=2)\text{expit}\left\\{l(V_{i})^{T}\sigma\right\\}}{1-\text{expit}\left\\{l(V_{i})^{T}\sigma\right\\}}\frac{I(A_{i}=1)}{\text{expit}(\mu)}\\\
(\theta_{1}-\theta_{0})-\theta_{2}\end{bmatrix}$

Again, the key difference from the IPW estimator under the restricted target
population was the second estimating function not being restricted to men
(i.e., $W_{i}=0$).

### Appendix B: Simulations

#### Data Generation

Covariates for the clinic population ($R_{i}=1$) were generated according to
the following distributions

$W_{i}\sim\text{Bernoulli}(0.667)$
$V_{i}\sim\text{Trapezoid}(\text{min}=18,\text{mode}_{1}=18,\text{mode}_{2}=25,\text{max}=30)$

$V_{i}$ was additional rounded to be an integer. Covariates for the trial
population ($R_{i}=2$) were generated from

$W_{i}\sim\text{Bernoulli}(0.0)$
$V_{i}\sim\text{Trapezoid}(\text{min}=18,\text{mode}_{1}=25,\text{mode}_{2}=30,\text{max}=30)$

with $V_{i}$ being rounded again. To summarize, the clinic population included
men and women, and consisted of a younger population than the trial. Outcomes
were generated from the following model

$Y_{i}\sim\text{Bernoulli}(p_{Y_{i}})$
$p_{Y_{i}}=\text{expit}(-3.25+1.50A_{i}+0.08-0.02W_{i}-0.65A_{i}W_{i})$

The true value for $\psi$ ($0.216697$) was approximated by simulating the
potential outcomes for 10 million observations with the clinic covariate
pattern.

#### Methods

The previously described estimating functions for the restricted g-computation
and IPW estimators were applied in the simulations, with the following
modifications to the design matrices for the corresponding nuisance models:
$g(A_{i},V_{i})=(1,A_{i},V_{i})$ and $l(V_{i})=(1,V_{i},V_{i}I(V_{i}>25))$.
For model synthesis, the model for g-computation was

$f_{a}(V_{i},W_{i};\alpha,\beta)=\text{expit}(\alpha_{0}+\alpha_{1}a+\alpha_{2}V_{i}+\beta_{0}W_{i}+\beta_{1}A_{i}W_{i})$

and the model for the IPW estimator was

$h_{a}(V_{i},W_{i};\gamma,\delta,\eta)=\text{expit}(\gamma_{0}+\gamma_{1}a+\delta_{0}W_{i}+\delta_{1}A_{i}W_{i})$

Point, lower, and upper confidence intervals were estimated by the median,
2.5th, and 97.5th percentiles, respectively, from 5000 repetitions of the
semiparametric bootstrap. Five sets of $\beta$ and $\delta$ specifications for
each estimator were compared: strict null, uncertain null, accurate,
inaccurate, and accurate with covariance. The strict null assumed all
$\beta=\delta=0$. Results for this specification are expected to match the
restricted target population and covariate set. The uncertain null used a
trapeziod distribution that is uniform from -1 to 1 and ranges from -2 to 2.
For the accurate parameter specification, we used a ‘secret’ trial to reflect
variable external knowledge in the simulations. In tandem with the clinic and
trial data sets, a secret second trial was conducted in each iteration of the
simulation. The secret trial consisted of 2000 observations generated
following the covariate pattern in the clinic population with $A_{i}$ randomly
assigned and outcomes generated from the preceding model for $p_{Y_{i}}$. The
secret trial was used to estimate $\hat{\beta}$, $\hat{\delta}$,
$\widehat{Var}(\hat{\beta})$, and $\widehat{Var}(\hat{\delta})$; which were
then provided to the model synthesis estimators. Importantly, no individual
level data from the secret trial was used by the model synthesis estimators
(i.e., the synthesis estimators only saw the estimated parameters from the
secret trial). By taking this approach, the simulation model parameters were
accurate in expectation but vary between iterations. The accurate
specification is expected to be unbiased, but confidence interval coverage is
expected to be conservative since the covariance between the parameters in
$\hat{\beta}$ or $\hat{\delta}$ was ignored. The inaccurate parameter
specification used the accurate specification parameters multiplied by
negative one. This specification was expected to be biased due to the reversal
of the direction of the relationships. The accurate with covariance parameters
used the secret trial, but the estimated covariance matrices for $\hat{\beta}$
and $\hat{\delta}$ were used to draw parameters from a multivariate normal
distribution (as opposed to independent normal distributions). As the
covariance between parameters is not ignored, confidence intervals are
expected to have nominal coverage.

Simulations consisted of 2000 iterations with $n_{1}=1000$ for the clinic data
and $n_{2}=1000$ for the the trial data. Evaluation metrics were bias,
confidence limit difference (CLD), and 95% confidence interval coverage [79].
As the variance estimates for the model synthesis approach were not assumed to
be normally distributed in all cases, we do not report the average standard
error or the standard error ratio. Bias was defined as the mean of the
difference between the estimate and true value. CLD was defined as the mean of
the difference between the upper and lower confidence intervals, and is an
indicator of precision (e.g., smaller CLD indicates greater precision). 95%
confidence interval coverage was defined as the proportion of intervals that
contained the truth. Simulations were conducted using Python 3.9.5 (Beaverton,
OR) with the following packages: Numpy [33], SciPy [34], pandas [35], and
delicatessen [36].

#### Results

The restricted target population and restricted covariate set were biased and
had poor confidence interval coverage (Appendix Table 3). Results were
approximately equal across these approaches since the distribution of $V_{i}$
did not differ by $W_{i}$. As expected, the performance of the model synthesis
approach depended on the selection of parameters. For the strict null, results
for g-computation and IPW were nearly identical to the restriction approaches.
For the uncertain null, there was some bias, but 95% confidence interval
coverage was 100%. This over-coverage results from the wide confidence
interval, as indicated by the CLD. For the accurate specification, bias was
near zero and confidence interval coverage was slightly above 95% for both
estimators. G-computation was more precise than IPW as seen in the CLD. For
the inaccurate specification, bias was the greatest relative to all other
approaches. Finally, the accurate specification with the covariance matrix had
near zero bias and 95% confidence interval coverage, which is in line with the
expected frequentist performance of the synthesis model for this mechanistic
model specification.

Table 3: Simulation results

|  |  | Bias | CLD | 95% CI Coverage
---|---|---|---|---|---
Restrict target population |  |  |
| G-computation | 0.105 | 0.112 | 4%
| IPW | 0.106 | 0.159 | 27%
Restrict covariate set |  |  |
| G-computation | 0.105 | 0.112 | 4%
| IPW | 0.106 | 0.158 | 26%
Model synthesis |  |  |
| G-computation |  |  |
|  | Strict Null | 0.104 | 0.111 | 4%
|  | Uncertain Null | 0.077 | 0.430 | 100%
|  | Accurate | -0.003 | 0.168 | 98%
|  | Inaccurate | 0.204 | 0.163 | 0%
|  | Accurate with covariance | -0.003 | 0.151 | 95%
| IPW |  |  |
|  | Strict Null | 0.105 | 0.157 | 27%
|  | Uncertain Null | 0.077 | 0.448 | 100%
|  | Accurate | -0.001 | 0.201 | 96%
|  | Inaccurate | 0.204 | 0.192 | 2%
|  | Accurate with covariance | -0.001 | 0.186 | 94%

CLD: confidence limit difference, CI: confidence interval, IPW: inverse
probability weighting. Results are for 2000 iterations.
Bias was defined as mean of the difference between the estimate and true
value, where the true value was based on the potential outcomes of 10 million
simulated observations.
CLD was defined as the mean of the difference between the upper and lower CI.
95% CI coverage was defined as the proportion of intervals that contained the
truth.