Smith et al.

*Matthew J. Smith.

London School of Hygiene and Tropical Medicine, Keppel Street, London,
England, United Kingdom, WC1E 7HT

± Both authors contributed equally to this work.

# Application of targeted maximum likelihood estimation in public health and
epidemiological studies: a systematic review

Matthew J. Smith  Rachael V. Phillips  Miguel Angel Luque-Fernandez± Camille
Maringe± Inequalities in Cancer Outcomes Network, London School of Hygiene and
Tropical Medicine, London, England, United Kingdom  Division of Biostatistics,
School of Public Health, University of California at Berkeley, California,
United States of America  Department of Statistics and Operations Research,
University of Granada, Granada, Spain<EMAIL_ADDRESS>
(xx xxxx xxxx; xx xxxx xxxx; xx xxxx xxxx)

###### Abstract

[Abstract] Background The Targeted Maximum Likelihood Estimation (TMLE)
statistical data analysis framework integrates machine learning, statistical
theory, and statistical inference to provide a least biased, efficient and
robust strategy for estimation and inference of a variety of statistical and
causal parameters. We describe and evaluate the epidemiological applications
that have benefited from recent methodological developments.
Methods We conducted a systematic literature review in PubMed for articles
that applied any form of TMLE in observational studies. We summarised the
epidemiological discipline, geographical location, expertise of the authors,
and TMLE methods over time. We used the Roadmap of Targeted Learning and
Causal Inference to extract key methodological aspects of the publications. We
showcase the contributions to the literature of these TMLE results.
Results Of the 81 publications included, 25% originated from the University of
California at Berkeley, where the framework was first developed by Professor
Mark van der Laan. By the first half of 2022, 70% of the publications
originated from outside the United States and explored up to 7 different
epidemiological disciplines in 2021-22. Double-robustness, bias reduction and
model misspecification were the main motivations that drew researchers towards
the TMLE framework. Through time, a wide variety of methodological, tutorial
and software-specific articles were cited, owing to the constant growth of
methodological developments around TMLE.
Conclusions There is a clear dissemination trend of the TMLE framework to
various epidemiological disciplines and to increasing numbers of geographical
areas. The availability of R packages, publication of tutorial papers, and
involvement of methodological experts in applied publications have contributed
to an exponential increase in the number of studies that understood the
benefits, and adoption, of TMLE.

###### keywords:

Targeted Maximum Likelihood Estimation (TMLE), Epidemiology, Observational
Studies, Causal Inference, Systematic Review

††articletype: Article Type

##  1 Background

Public health decisions across many clinical specialities are often informed
by research exploring the relationship between exposures and patient health
outcomes. To ascertain a causal relationship, randomised control trials (RCT)
are considered the gold standard because, through randomisation of subjects to
a treatment, they reduce the possibility of bias. Observational data offer
invaluable opportunities to study causal relationships in contexts where
clinical trials might prove infeasible or unethical, as well as for studying
groups of the population typically excluded from trials or beyond the initial
target population. Under correct adjustment for selection bias, missingness,
interference, and confounding, observational data complements the evidence
coming from RCTs.

In both RCT and observational studies, a causal exposure-outcome relationship
is presented with a causal estimand, such as the average treatment effect
(ATE). Methodological statistical developments for causal inference attempt to
produce the least biased estimate of the estimand and accurate inference.
G-computation, Propensity Score (PS), and Inverse Probability of Treatment
Weighting (IPTW) estimators rely on parametric modelling assumptions, which
are susceptible to model misspecification (i.e., exclusion of the underlying
and unknown data-generating distribution from the model, which can lead to
biased estimates with small standard errors and thus misleading results).
Double-robust methods, like Augmented Inverse Probability of Treatment
Weighting (AIPTW) and Targeted Maximum Likelihood Estimation (TMLE), aim to
minimise model misspecification by requiring estimation of both the outcome
and exposure mechanisms. They provide a consistent estimator as long as either
the outcome or exposure model is correctly specified. Double-robust methods
often outperform single-robust methods in point and interval estimation.1, 2

TMLE, also known as targeted minimum loss-based estimation, was introduced by
van der Laan and Rubin in 2006.3 In general, TMLE is a two-step process that
involves (1) initial estimation of the outcome and intervention models, and
then (2) in a “targeting” step, uses information from them to optimise the
bias-variance trade-off for the target estimand (e.g., ATE), rather than the
whole outcome probability distribution. Furthermore, to avoid model
misspecification, ensemble machine learning algorithms are used to estimate
the initial models. In particular, the Super Learner (SL) algorithm for
stacked ensemble machine learning is most commonly used as it is theoretically
grounded and proven to perform optimally in large samples.4

We lightly detail the technical steps involved in the TMLE of the ATE, i.e.,
the effect of a binary exposure $A$ on a post-exposure outcome $Y$, adjusted
by baseline covariates $W$.5 The prediction function for the mean outcome $Y$,
given exposure $A$ and covariates $W$ is estimated, most commonly, using SL.
We could use this estimated prediction function, $\hat{E}[Y|A,W]$, to arrive
at an estimate of the ATE. Specifically, we would obtain predicted outcomes
under a counterfactual scenario where all subjects receive the
exposure/treatment versus another scenario where no one receives it. The
average difference between these predicted counterfactual outcomes is an
estimate of the ATE. However, formal statistical inference (i.e., confidence
intervals and p-values) cannot be obtained for this estimate and it is
susceptible to residual confounding; the latter can be reduced by using the
information on how each individual was assigned or allocated to each level of
the exposure. We, therefore, estimate the function for predicting the
probability of being exposed, given the covariates $W$, using SL (exposure
model, i.e. propensity score). These first steps are common to other double-
robust estimators of the ATE, such as AIPTW. We then calculate the so-called
“clever covariate” for the ATE, which is the individual values of the binary
exposure weighted by the predicted probabilities of the exposure, given $W$.
This is similar to IPTW, except here we weight the predicted probability of
each exposure level instead of the outcome. The fluctuation parameter
($\epsilon$) describes the difference between the observed outcome $Y$ and the
initial predictions of the outcome from the outcome model. It is calculated
through maximum likelihood estimation (MLE) by regressing the clever covariate
on the observed outcome and incorporating as offset the predicted outcome
values. When the fluctuation parameter is estimated as close to 0 there is
little difference between the observed and predicted outcomes; thus, the
propensity score does not provide additional information for the initial
estimate of the outcome model because it was correctly specified. If the
fluctuation parameter is not close to 0, then this indicates the presence of
residual confounding in the initial estimate. The initial outcome model’s
predictions for each level of the binary exposure are updated using the
fluctuation parameter $\epsilon$ as a weight, and the final ATE estimate is
calculated from these updated estimates. The functional delta method based on
the influence function can be used to derive the standard error of the ATE and
construct Wald-type confidence intervals.

Since 2006, the TMLE framework has experienced a growing number of theoretical
and applied developments, and it has expanded further after a book that shared
the TMLE framework to the international community of applied researchers was
published in 2011.2 Targeting specifically applied researchers, efforts were
made to provide lay-language descriptions of the framework and exemplify its
applications.6, 7, 5 Furthermore, in 2018, a second book was published
disseminating more advanced applications of the TMLE framework to data
scientists with a particular focus on longitudinal settings.8 TMLE is a robust
framework for statistical analysis in clinical, observational and randomised
studies. Since 2016, the American Causal Inference Conference has hosted a
data challenge in which teams compete to estimate a causal effect in simulated
data sets based on real-world data, such as from healthcare or education.9 The
competition is a proving ground for cutting-edge causal inference methods that
have the potential to transform program evaluation. TMLE has consistently been
a top-performing method.10

The use of robust statistical methods is key to obtaining reliable results for
public health and epidemiological research and maximising their benefit to
society. Evidence shows that TMLE, by blending flexible machine learning
methods and causal inference principles, is one such step towards robust
causal claims that bear significant and practical effects. We reviewed the
literature around public health and epidemiological applications of TMLE to
date, alongside key TMLE developments over the last 20 years. We highlight the
speed at which the field developed and spread through the scientific
community, and identify areas for further development to increase the utility
of the TMLE framework in epidemiological and applied research.

##  2 Methods

#### Protocol registration and reporting standards

This study is reported using the Preferred Reporting Items for Systematic
Reviews and Meta-Analyses (PRISMA) guideline. We registered this systematic
review with PROSPERO (ID: CRD42022328482).

#### Information sources

We searched the PubMed medical literature database for published
epidemiological studies using TMLE in any epidemiological field (i.e.,
observational settings in biomedical sciences, including clinical epidemiology
and public health). We searched for publications from any time up to 20th May
2022, the date the search was executed. The search strategy comprised of two
specific groups of search terms focusing on TMLE and epidemiology. Relevant
Mesh headings were included as were free-text terms, searched in the title,
abstract or keyword fields. We used general and specific TMLE search terms,
such as “targeted maximum likelihood estimation”, “targeted minimum loss-based
estimation”, and “targeted machine learning”. Epidemiological search terms
included “epidemiology”, “public health”, “population”, or “treatment”. The
two specific groups of terms were combined with ‘AND’ to retrieve the final
set of results. Search strategies were developed with an information
specialist (MALF). The full search strategy is shown in Table 1.

Table 1: Boolean search queries Query | Boolean terms | Results
---|---|---
#1 | (epidemiology OR (public AND health) OR population OR treat*) | 11,004,506
#2 |  | ("targeted maximum likelihood estimation") OR ("targeted minimum loss based estimation") OR
---
("targeted minimum loss-based estimation") OR ("TMLE") OR ("targeted machine
learning") OR
("targeted learning") OR ("targeted machine-learning")
311
#3 | #1 AND #2 | 226

#### Eligibility criteria

We excluded studies that did not report the use of TMLE as a tool to explore
their (possibly causal) estimand of interest. We also excluded experimental
studies, such as RCTs (n=15). We did not consider manuscripts that compared
the performance of TMLE to other estimators, when there was no new development
proposed and even if there was an applied question of interest, such as in
Luque-Fernandez et al.11 Studies were restricted to the English language and
primary research studies. Secondary research studies, such as reviews and
comments of TMLE, conference abstracts and brief reports, and preprints were
not searched. We classified the retained manuscripts into observational,
methodological and tutorial articles. TMLE methodological development articles
and tutorials were considered separately, even if they contained a
methodological development specifically designed to investigate an
epidemiological question within the same article. We make reference to these
methodological articles throughout this review, as they underpin the applied
publications.

#### Study selection, data extraction and management

All retrieved publications were imported into the Endnote reference software
where they were checked for duplication. Two of the three lead researchers
(authors MJS, MALF and CM) were randomly allocated two-thirds of the 226
articles, to screen titles and abstracts of each publication independently and
classify these into (1) observational, (2) methodological developments, (3)
tutorial, (4) systematic review, (5) RCT, or (6) not relevant (Figure 1).
Disagreements were discussed and adjudicated by the third independent
reviewer, where necessary. Two researchers (authors MJS and CM) independently
reviewed the full text of all eligible observational publications for data
extraction.

Figure 1: Flow diagram of studies included in the systematic review

##  3 Results

We found 226 unique publications published prior to 20th May 2022 in PubMed
(Figure 1). Of these, 95 articles were a methodological development (including
theoretical - or software-based), eight were tutorials, five were systematic
reviews, and fifteen were RCTs. Of the 22 articles that were not relevant,
three mentioned “TMLE” only in the author fields, one was a discussion of
currently existing methods, twelve were assessments of learning (educational)
programs that are targeted towards clinical environments, and six were
comparisons of machine learning algorithms to other prediction models (without
mention of TMLE). Overall, we focused on 81 observational studies in this
systematic review for which full texts were available; six publications were
not open-access and full texts were obtained from the corresponding authors.

###  3.1 Dissemination and uptake of the TMLE framework

There has been a growing uptake of the TMLE framework over time, with five or
fewer applied publications per year until 2017, and up to 21 in 2021. The
majority (58, 71.6%) of publications using TMLE were published in the last
three and a half years, starting in 2019. Most studies (77, 95.1%) cited the
authors of particular TMLE methods they apply, whereas four (4.9%) did not
cite any TMLE references. The large majority of these first epidemiological
studies benefit from the expert knowledge of an author who is (or was) part of
Professor Mark van der Laan’s lab. (Table 3.2)

Of the 81 studies included, two-thirds were conducted in the United States of
America (US) (53, 65.4%, Figure 2),12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,
42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60,
61, 62, 63, 64 with 100% of articles before 2017 being published in the US and
down to 50% of all 2021 articles. Publications from Europe or Scandinavia (10,
12.3%),65, 66, 67, 68, 69, 70, 71, 72, 73, 74 Africa (4, 4.9%),75, 76, 77, 78
Middle East (5, 6.2%),79, 80, 81, 82, 83 and Oceania or Asia (8, 9.9%)84, 85,
86, 87, 88, 89, 90, 91 represent between 20% (in 2017) and 70% (in 2022, up to
May 2022) of all applied studies published in the last 6 years. (Figure 2,
Table 3.2) In the US, the majority of publications (28) were from California,
including 20 from the University of California at Berkeley, where TMLE was
first described.

In the early years, the first authors tended to be qualitative academic
experts, but we saw more variety in expertise and a larger number of
practising clinicians leading observational studies in many epidemiological
and public health fields in recent publications. The most common
epidemiological sub-discipline was non-communicable disease (23, 28.4%),65,
16, 27, 31, 32, 34, 36, 38, 79, 70, 47, 71, 80, 52, 55, 81, 82, 78, 73, 60,
62, 87, 63 followed by behavioural epidemiology (17, 21.0%),14, 21, 28, 67,
40, 43, 44, 45, 48, 49, 84, 53, 54, 58, 59, 89, 90 and then infectious disease
epidemiology (12, 14.8%).17, 24, 69, 92, 46, 50, 51, 76, 77, 83, 88, 74
Through time we see an uptake of TMLE in many more disciplines, such as
pharmaco-epidemiology,26, 75, 72, 61 policy,23, 25, 30, 41, 56, 64, 91
biomarker epidemiology,12, 13, 19, 20, 22, 42, 85, 86 environmental
epidemiology,15, 66, 29, 35, 39, 68, 57 occupational epidemiology,18, 33 and
health economy.37

We also studied the evolution of citations. When only methodological overviews
of the TMLE framework were available, these were cited despite their heavy
statistical requisite. Since 2016, tutorials were published and started to be
cited alongside references for statistical packages. 6, 93, 94, 7, 95, 5, 96,
1(Table 3.2)

Cohort study design13, 14, 17, 18, 19, 25, 26, 27, 30, 32, 33, 36, 38, 39, 40,
69, 41, 70, 43, 46, 49, 50, 84, 75, 52, 85, 55, 77, 58, 59, 78, 73, 86, 60,
61, 87, 88, 63, 74, 89, 90, 91 was the most prevalent design (42, 51.9%),
followed by cross-sectional (32, 39.5%) (Appendix Table
LABEL:table:discipline).12, 15, 65, 16, 21, 22, 23, 24, 66, 28, 29, 31, 34,
35, 37, 67, 68, 42, 79, 44, 45, 47, 48, 71, 51, 76, 53, 54, 81, 56, 72, 62
Other types of epidemiological design included case-control20, 92, 80, 82, 83
and ecological.57, 64

Many articles reported results from other statistical methods, in addition to
reporting those obtained from TMLE. Over one-quarter of the studies used
adjusted parametric regression (24, 29.6%),15, 16, 17, 23, 26, 31, 33, 36, 67,
38, 68, 42, 79, 44, 92, 50, 71, 55, 77, 72, 78, 73, 87, 83 one sixth (12,
14.8%) used IPTW,14, 17, 29, 34, 43, 75, 76, 53, 54, 57, 59, 61 one (1.2%)
used AIPTW,56 three (3.7%) used non-parametric methods (e.g. Kaplan Meier),17,
33, 55 and seven (8.6%) used unadjusted regression.43, 48, 52, 63, 64 Some
studies included more than one comparative method.

SuperLearner (SL) provides a flexible machine learning approach to the
estimation of the initial outcome and intervention models. Of the 81 articles,
more than half (47, 58.0%) used the SL algorithm, 23, 66, 25, 27, 29, 30, 31,
32, 34, 67, 38, 40, 69, 41, 42, 43, 92, 45, 46, 47, 49, 50, 71, 84, 51, 76,
52, 85, 54, 55, 81, 56, 82, 77, 57, 72, 59, 86, 61, 62, 87, 83, 88, 63, 64,
74, 89, 90, 91 18 (22.2%) used logistic regression,12, 13, 14, 15, 16, 17, 18,
21, 26, 28, 33, 35, 36, 44, 75, 80, 53 and 16 (19.8%) did not specify the
approach for the estimation of either the outcome or intervention model.65,
19, 20, 22, 24, 37, 39, 68, 79, 70, 48, 58, 78, 73, 60 The average number of
machine-learning algorithms considered by the SL was 6.3 (range 1 - 16), 19
different machine-learning algorithms were used across the articles (a
machine-learning algorithm is a wrapper included within the SuperLearner2
library in R software).

Variance of point estimates obtained from TMLE were estimated using the
influence function (n=19, 23.5%),15, 65, 16, 17, 18, 20, 23, 28, 36, 79, 71,
51, 80, 81, 82, 72, 87, 83, 64 bootstrap (n=6, 7.4%),13, 14, 26, 29, 34, 53
and Wald tests (n=2, 2.5%),21, 25 while 54 (66.7%) studies did not specify how
standard errors were obtained.12, 19, 22, 24, 66, 97, 31, 32, 27, 33, 35, 37,
67, 38, 39, 68, 40, 69, 41, 92, 42, 70, 43, 44, 45, 46, 47, 48, 49, 50, 84,
75, 76, 52, 85, 54, 55, 56, 77, 57, 58, 59, 78, 73, 86, 60, 61, 62, 88, 63,
74, 89, 90, 91

The Causal Inference Roadmap98 contains seven recommended criteria to define a
causal effect: (i) specify the scientific question, (ii) specify the causal
model, (iii) define the target causal quantity, (iv) link the observed data to
the causal model, (v) assess identifiability assumptions, (vi) estimate the
target statistical parameters, and (vii) interpretation of the results. On
average, 5.3 (SD 1.0) criteria were complete per article. We considered a
version of the Targeted Learning Roadmap99 that contains five criteria: (i)
specify the observed data and describe the data-generating experiment, (ii)
specify a statistical model representing a set of realistic assumptions about
the underlying true probability distribution of the data, (iii) define a
target estimand of the data distribution that “best" approximates the answer
to the scientific question of interest, (iv) given statistical model and
target estimand, construct an optimal plug-in estimator of the target estimand
of the observed data distribution, while respecting the model, and (v)
construct a confidence interval by estimating the sampling distribution of the
estimator. When the scientific question of interest is causal, step (iii) of
the Targeted Learning Roadmap incorporates steps (ii)–(v) of the Causal
Inference Roadmap.99 On average, 3.3 (SD 1.0) criteria were complete per
article. Most studies have room to state the necessary content for at least
one more criteria.

Most publications (77, 95.1%) used R software to perform TMLE,14, 15, 65, 16,
17, 18, 19, 20, 21, 23, 24, 66, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 36,
37, 67, 38, 39, 40, 69, 41, 42, 79, 70, 43, 44, 45, 46, 47, 48, 49, 71, 84,
51, 75, 92, 76, 80, 52, 85, 54, 55, 81, 56, 82, 77, 57, 58, 72, 73, 86, 83,
63, 64, 74, 90, 91 except four that used STATA.68, 78, 62, 87 Nonetheless,
nine articles reported using another software tool (i.e., Stata/SAS/SPSS)
alongside R for TMLE.14, 31, 37, 45, 75, 82, 77, 72, 63 The most commonly used
R software packages were tmle100 (40, 49.4%) and ltmle101 (16, 19.8%).

###  3.2 Showcase of the TMLE framework in different disciplines

In all disciplines and applications, applying the TMLE framework to their
specific research question encouraged authors to review the strengths and
limitations of their data and carefully consider how their data and setting
might violate identifiability assumptions, which are assumptions necessary for
causal inference but not TMLE. If, and only if, the identifiability
assumptions are assumed to hold, the estimated effect is a causal effect.
However, for observational studies, it cannot be known whether identifiability
assumptions hold. Therefore, if an estimate is interpreted as a causal effect,
then this interpretation should be accompanied by a discussion of the
plausibility of identifiability assumptions. All disciplines and disease areas
highlight issues with missing data and measurement errors and incorporate
subject-matter knowledge. (Appendix Table LABEL:table:discipline) Model
misspecification, which might result from imposing constraints that are
unrealistic or not informed by subject-matter knowledge, was the first and
foremost driver for using the TMLE framework. Three-quarters of the studies
(n=68, 84.0%) provided at least one justification for using TMLE compared to
another method. (Table 3.2) Standard regression techniques in settings with
low incidence,24 rare outcomes75 or low sample size67, 53 may over-fit the
data or not converge: careful SL specifications overcome these limitations.24,
37 Least biased and double-robust estimation were also widely cited advantages
of the framework. These mean (i) the estimated parameter will be closest to
the true value of our quantity of interest, even when our sample size is low,
and (ii) only one of our initial models needs to be correctly specified.

There was a range of disease areas covered in the 23 noncommunicable disease
epidemiology studies. The appealing property of TMLE was that it is a
semiparametric estimator, allowing the use of machine learning algorithms to
minimise model misspecification.62, 34, 60, 47, 81, 78, 87, 73 Additionally,
extensions of TMLE have developed ways to appropriately handle the dual nature
of time-varying confounding, which have been utilised in longitudinal studies
analysing data on depression,32 survival from acute respiratory distress
syndrome,55 caries arising from intake of soda,36 effects of smoking on
rheumatoid arthritis,38 and effects of asthma medication on asthma symptoms.70
Improved predictive performance65 and adjusting for informative censoring55
were additional reasons for using TMLE. Furthermore, the extension of TMLE to
case-control studies, in which sampling is biased with respect to the disease
status, provided a platform for analysing the causal effect of reproductive
factors on breast cancer by using case-control weighted TMLE.82

In infectious disease epidemiology (IDE) articles, most were concerned with
having a flexible modelling approach that does not impose assumptions on the
functional form of the exposure-outcome relationship.51, 76, 88, 69, 92, 46 A
key feature of the IDE subdiscipline is that baseline confounders and
exposures may change over time and can obscure the causal effect of
interest.17 Standard survival modelling assumes that censoring and survival
processes are independent, which is likely violated in this setting, and it
assumes there is no time-dependent confounding.17 TMLEs with a working
marginal structural model and for time-to-event outcomes permit evaluation of
the effect of an exposure at multiple time points, which is beneficial when
the interpretation of causal effects from hazard models is often difficult.102
Other studies have overcome this issue by using TMLE in a target trial
framework or case-cohort studies.50, 83

In behavioural epidemiology manuscripts, the behavioural nature of the topics
covered implied that RCTs are mostly unethical, bear prohibitive costs or have
very small sample sizes. There are several key challenges for using
observational data to study the causal effects of childhood adversities,21, 28
physical activity,14, 43, 90 alcohol consumption45 or supply 84 on various
outcomes, including fractures,14 mental health,21, 58, 59 asthma,67 and
pregnancy outcomes.43, 44 They include a risk for reverse causation;55, 58, 59
high dimensional data and in particular, multidimensional exposures;21, 28 and
measurement error resulting from self-reported exposures or outcomes.44, 59,
89, 90 Longitudinal relationships and time-varying confounding, where
confounders of the effect of an exposure on an outcome can themselves be
affected by prior exposures, as well as sample attrition,40, 84, 89, 90 are
particular challenges faced by survey data that are collected in consecutive
waves.40, 84, 54, 59, 89, 90 TMLE adjusts for time-varying confounders
affected by prior exposure and employs a doubly robust estimation approach
that allows for flexible model fitting. Additionally, as pointed out in 2016
by Ahern et al.,21 “TMLE with machine learning addresses the challenge of a
multidimensional exposure because it facilitates ‘learning’ from the data the
strength of the relations between each adversity [dimensions of the exposure]
and outcome, incorporating any interactions or nonlinearity, specific to each
[sub-group]".

The field of biomarker epidemiology is driven by the search for sets of
candidate biomarkers that are important in determining given outcomes. Ranking
the contributions of these candidate biomarkers is also of interest. Some
studies used TMLE to measure variable importance in biomarker research12, 20,
86 and in other fields.65 Dimension reduction for the estimation of causal
effects is an aim in some biomarker examples.22, 42, 85 In the models
presented in the publications, there are complex joint effects to consider in
large correlated data, as well as longitudinal patterns and time-dependent
confounding.22, 42, 85 Furthermore, two manuscripts present covariate
selection algorithms ahead of causal effect estimation.85, 86

Research published in environmental epidemiology highlights challenges around
the clear definitions of exposure and outcomes,15 as there are likely many
proxy and surrogate measures of exposure,66 coupled with potential exposure
misclassification and measurement errors.15, 35 Nonetheless, TMLE was
successfully applied to determine either causal attributable risk,15, 66 or
risk differences.29

The only observational study of TMLE in health economics explored the
relationship between financial resources leading to food insecurity and
healthcare expenditure in a pay-to-access healthcare system. It uses
ecological measures of exposure and outcome and leads to evidence for
policy.37

Two publications focused on occupational epidemiology.18, 33 A key aspect of
occupational epidemiology is accounting for the healthy worker survivor
effect: a bias arising due to healthier workers accruing more exposure over
time. These studies looked at exposure to particulate matter from aluminium or
metalworking fluids in metal factory workers, which varied depending on the
length of employment. Both studies benefited from TMLE’s flexibility to allow
for time-varying confounding of the exposure.

The field of pharmacoepidemiology is concerned with assessing treatment’s
efficacy in real-world settings and monitoring long-term side effects of
treatments. Both objectives would be either impractical or too costly to study
in RCTs, given the limited follow-up time available in clinical trials. TMLE
has been used in this setting, as it provides a robust procedure for
estimation.26, 75, 72, 61 In particular, the flexibility of TMLE, provided
through the specification of a diverse and rich set of machine learning
algorithms in the SL, is crucial for appropriately adjusting for confounding
in observational studies.103

Policy epidemiology assesses the effects of population programs or mandates.
Lack of randomisation, such as in studies examining the association between
speciality probation and public safety outcomes,25, 30 leads to an imbalance
in the covariate distribution by exposure levels. Studies of cost-
effectiveness may involve dealing with outliers which can be addressed with
TMLE.30, 56 Other challenges include zero-inflation, such as the assessment of
the effect of primary care physician density on arthroplasty outcomes, in
which some areas had zero density.56 This is dealt with by using a mixture of
models to assess the probability of non-exposure (i.e., very low density).56
Other policy studies presented challenges around missing data,56 reliance on
epidemic modelling assumptions,64 target trial emulation,91 or infeasible
randomisation process.41

Figure 2: World map of publications using targeted maximum likelihood
estimation by the geographical location of the first author (2006 to
mid-2022). Colours represent the number of observational studies and the
crosshatch pattern identifies where at least one methodological publication
stem from.

[ht] Distribution of observational papers by year of publication and selected
characteristics Year of publication 2009 2010 2011 2012 2013 2014 2015 2016
2017 2018 2019 2020 2021 2022 ⋆ Total N (%) N (%) N (%) N (%) N (%) N (%) N
(%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N Publications 2 1 1 1 2 1 5 4 6
12 16 21 9 81 TMLE expert (author)± 2 (100) 1 (100) 1 (100) 1 (100) 1 (50) 1
(100) 3 (60) 1 (25) 4 (67) 6 (50) 6 (38) 5 (24) 1 (11) 33 USA-based
publication 2 (100) 1 (100) 1 (100) 2 (100) 1 (100) 5 (100) 3 (75) 6 (100) 9
(75) 10 (63) 11 (52) 2 (22) 52 Discipline Behavioural epi 1 (100) 1 (20) 1
(17) 2 (17) 6 (38) 4 (19) 2 (22) 17 Biomarker 2 (100) 3 (60) 1 (6) 2 (10) 8
Environmental epi 1 (100) 1 (25) 1 (17) 3 (25) 1 (5) 7 Health economy 1 (8) 1
Infectious disease 1 (50) 1 (25) 1 (8) 4 (25) 2 (10) 3 (33) 12 Non-
Communicable Disease 1 (100) 1 (50) 3 (50) 3 (25) 4 (25) 9 (43) 2 (22) 23
Occupational epi 1 (100) 1 (8) 2 Pharmaco-epi 1 (25) 1 (6) 2 (10) 4 Policy 1
(20) 1 (25) 1 (17) 1 (8) 1 (5) 2 (22) 7 Motivations † Bias 1 (50) 2 (100) 1
(100) 1 (20) 1 (25) 2 (33) 5 (42) 5 (31) 12 (57) 6 (67) 36 Double-robust 1
(50) 1 (20) 1 (25) 2 (33) 2 (17) 5 (31) 6 (29) 4 (44) 22 Efficient 2 (17) 5
(31) 2 (10) 3 (33) 12 Finite sample 1 (100) 2 (10) 3 Model misspecification 1
(50) 1 (100) 1 (100) 3 (60) 1 (25) 2 (33) 1 (8) 3 (17) 1 (57) 2 (22) 16
Positivity assumption 1 (17) 1 (8) 1 (6) 3 Power 1 (6) 1 None specified 1
(100) 1 (20) 2 (50) 5 (42) 4 (25) 2 (10) 1 (11) 16 Expertise first author
Biostatistician 1 (50) 1 (50) 1 (100) 1 (25) 2 (33) 2 (17) 2 (13) 4 (19) 2
(22) 16 Epidemiologist 1 (100) 1 (100) 1 (50) 3 (60) 1 (17) 4 (33) 6 (38) 7
(33) 1 (11) 25 MD 1 (20) 2 (50) 1 (8) 1 (6) 5 (24) 2 (22) 12 MD, MPH 1 (8) 1
(6) 2 MD, PhD 1 (100) 3 (25) 2 (13) 3 (14) 3 (33) 12 Other 2 (13) 2 (10) 4 PhD
1 (50) 1 (25) 2 (33) 1 (6) 5 Not known 1 (20) 1 (17) 1 (8) 1 (6) 1 (11) 5
Citations ${}^{\boxdot}$ Overall TMLE method. 2 (100) 1 (50) 1 (100) 5 (56) 1
(33) 4 (33) 4 (50) 3 (33) 6 (19) 13 (42) 20 (38) 4 (16) 64 Specific TMLE
method. 1 (50) 1 (100) 4 (44) 1 (33) 6 (50) 1 (13) 3 (33) 15 (47) 7 (23) 13
(25) 8 (32) 60 Tutorial 1 (11) 7 (22) 7 (23) 14 (27) 7 (28) 36 R software 1
(33) 2 (17) 3 (38) 2 (22) 4 (13) 4 (13) 5 (10) 6 (24) 27 ⋆ Up to 20th May 2022
± TMLE expert is a current or past member of M.J. van der Laan’s Lab. †
Proportions calculated over the number of publications within that year.
${}^{\boxdot}$ Proportions calculated over the total number of citations
within that year.

###  3.3 Methodological developments and their implementation

Over the years since the TMLE framework was first laid out,3 many
contributions have been made to expand the settings in which TMLE is used,
provide tools for implementation in standard software, and describe the TMLE
framework and application in lay language. Thanks to this, the community of
public health researchers and epidemiologists have started implementing the
TMLE framework and its latest developments to obtain double robust, least
biased and efficient estimates and statistical inference from studies. The
properties of TMLE, in contrast to other estimators commonly used for causal
inference, include that it is loss-based, well-defined, unbiased, efficient
and can be used as a substitution estimator.

Figure 3 shows schematically when and why extensions of TMLE have happened in
the last 15 years, as well as extensions and uptake. The 81 applied
epidemiological studies are classified by methodological development used
during the study. Appendix Table 2 provides further details regarding the
methodological development references between 2006 and mid-2022 that are
highlighted in Figure 3.

TMLE’s superior efficiency and power are evidenced in small sample size
settings where marginal effects from logistic regression models adjusted for
(possibly many) covariates would not be recommended.104 The implementation of
TMLE in complex causal effect estimation problems is discussed in many
publications, such as in settings with multiple time point interventions,105,
106 longitudinal data,107, 108 post-intervention effect modifiers,109
dependence of the treatment assignment between units110 or censoring,111
causally connected units,112, 113 hierarchical data structures,114
randomisation at the cluster level,115 large electronic health record data,
116 and in meta-analyses.117, 118

The TMLE framework is extended and discussed in the setting of case-control
studies. One study matched cases to controls,119 another used two-stage
sampling and nested case-control design.120 Other studies required the design
to be adaptive to possibly invalid assumptions of independent units121 or if
the sample population differs from the (possibly ill-defined) target
population.122

The collaborative TMLE (C-TMLE), introduced in 2010,123 is an extension of
TMLE, in which information on the causal parameter of interest is used when
estimating and selecting initial model(s). C-TMLE aims to improve the
robustness and efficiency of the TMLE. Schnitzer et al.111 highlight the
pitfalls and their consequences of automated variable selection in causal
inference, such as in the propensity score model, and how C-TMLE corrects for
this. C-TMLE was later extended to measure variable importance124 and to
longitudinal data settings.125 Proposals to enhance the C-TMLE algorithm
include ordering covariates to decrease C-TMLE time complexity,126 using LASSO
with C-TMLE for the estimation of the propensity scores,127 and adaptive
truncation of the propensity scores with C-TMLE to ensure positivity.128

The pooled TMLE129 was developed for the context of longitudinal data
structures with baseline covariates, time-dependent intervention nodes,
intermediate time-dependent covariates, and a possibly time-dependent outcome.
Extensions include advice for the optimal discretisation of time130 and to the
hazard function.131

The one-step TMLE aims to preserve the performance of the original two-step
TMLE, and achieves bias reduction in one step (i.e., without additional
iterations of the TMLE update step and possible over-fitting in finite
samples).132 This one-step TMLE was later extended to counterfactual average
survival curves133 and heterogeneous treatment effects.134

The robust TMLE was proposed in 2017 for transporting intervention effects
from one population to another.135

The cross-validated TMLE (CV-TMLE) provides asymptotic inference under minimal
conditions (i.e., non-parametric smoothness136) keeping the bounds of the
parameter estimates. It is also used in the estimation of data-adaptive target
parameters, like optimal treatment regimes. Recently, TMLE was shown to be
useful in defining thresholds and marking specified levels of risks.137

The set of observational articles that use TMLE in their main or sensitivity
analyses shows that TMLE has successfully been used to examine
associations,12, 14, 65, 20, 21, 22, 24, 26, 28, 29, 31, 32, 34, 35, 37, 39,
40, 42, 79, 43, 44, 46, 47, 48, 51, 75, 76, 52, 53, 85, 54, 81, 56, 57, 58,
72, 59, 73, 60, 61, 62, 88, 63, 74, 89, 90 causation,13, 15, 16, 17, 18, 19,
23, 66, 25, 97, 33, 36, 67, 38, 68, 70, 45, 49, 50, 71, 84, 80, 55, 82, 77,
78, 86, 87, 83, 64, 91 and variable importance. 12, 65, 20, 86 It has been
used to analyse data with varying numbers of observations, from less than 100
to over hundreds of thousands, from clinical trials, cohort studies,13, 14,
17, 18, 19, 25, 26, 32, 33, 36, 38, 39, 40, 70, 43, 46, 49, 50, 84, 75, 52,
85, 55, 77, 58, 59, 78, 73, 86, 60, 61, 87, 88, 63, 74, 89, 90, 91 and
observational studies.

Figure 3: Applied clinical and epidemiological research by year of publication
and TMLE method implemented

##  4 Discussion

We aimed to investigate the use of the TMLE framework in epidemiology and
public health research and to describe the uptake of its methodological
developments since its inception in 2006. We focused on TMLEs for point
treatment, time-to-event/survival, and longitudinal exposure-outcome
relationships. We did not discuss TMLE methodological developments and
software for mediation analyses.138, 139, 140 We found that the TMLE framework
and its different estimators were implemented in at least 81 epidemiological
observational studies. The majority of these studies have come from the US,
many of which are from the University of California, Berkeley. Recently, the
use of TMLE has spread across the world. Until 2016, TMLE in observational
studies was used by select groups of expertise and disciplines, such as
biostatisticians or epidemiologists in academia exploring noncommunicable and
infectious diseases, or behavioural epidemiology. From 2016 onward, there was
a faster uptake amongst a wider range of expertise and disciplines. There is
potential for even wider dissemination and acceptance, both geographically and
in some specific disease areas or epidemiological disciplines. We hope this
review of explicit and applied examples will contribute to enhancing the
relevance of the TMLE framework and increasing its uptake and acceptance in
settings where challenges with regard to data, unrealistic assumptions, or
subject-matter knowledge lend themselves to the framework.

Initially, causal inference methods and estimators relied on parametric
modelling assumptions but, to quote Box (1976), “all models are wrong but some
are useful".141 It highlights that model misspecification was and remains a
challenge, even with ever-growing datasets and computing power: exclusion of
the unknown data-generating distribution leads to biased results. Semi-
parametric and non-parametric causal inference estimators, such as AIPTW,
double-debiased,142 and TMLE3 aim to provide the least biased estimate of the
effect of an exposure on an outcome.143, 1 Maximum Likelihood Estimation (MLE)
based methods (stratification, propensity score and parametric regression
adjustment) and other estimating equations (AIPTW) do not have all of the
properties of TMLE, and evidence shows that they under-perform in comparison
to TMLE in specific settings.3, 120, 5, 1 For more detailed comparisons
between the different methods, the interested reader is referred to Chapter 6
of van der Laan and Rose’s TMLE textbook.2 It is important to highlight that
in contrast to the AIPTW estimator, TMLE respects the global constraints of
the statistical model (i.e., $P(0<Y<1)=1$) and solves the score estimation
equations being equal to zero using substitution and fluctuation
approaches.144 TMLE augments the initial estimates to obtain an optimal bias-
variance trade-off for the target estimand of interest and produces a well-
defined, unbiased, efficient substitution estimator. Furthermore, the
targeting step (i.e., update of the initial estimate) may remove finite sample
bias. Lastly, the TMLE framework can be tailored to specific research
questions that are difficult to answer using other causal inference methods,
such as rare diseases,145, 146 ordinal147 or continuous exposures,148 dynamic
treatment regimes,129 and missing outcome data.149

We argue that dissemination of any new statistical methodology relies on five
key factors: (i) software availability, (ii) accessibility of available
material (e.g., quality of software help files, language used in publications,
etc.), (iii) number of experts in the area, (iv) teaching, and (v)
collaborations. In the following, we discuss the dissemination of TMLE with
regard to each of them.

#### (i) Software availability:

Various TMLEs have been developed for complex study designs, such as those
with time-to-event outcomes, case-control studies, hierarchical data
structures (including cluster randomised trials), longitudinal data, and time-
dependent confounding. These methodological developments were accompanied by
the release of R software packages, increasing the usability of TMLE. Such
software developments include the SuperLearner4 R package in 2007 and the tmle
R package in 2012.150 TMLE software for survival analysis (survtmle),151, 152
longitudinal data (ltmle),101 doubly-robust confidence intervals (drtmle),153
and estimation of the survival curve under static, dynamic and stochastic
interventions (stremr)154 were implemented in 2017. To match the expanding
framework, further software developments occurred in the following years, such
as the tlverse suite of software packages for Targeted Learning
(https://tlverse.org/tlverse-handbook/), which includes R packages for cross-
validation (origami),155 highly adaptive lasso (HAL, hal9001),156, 157 super
learning (sl3),158 and TMLEs for a range of target estimands, such as effects
under static interventions on an exposure (tmle3),159 optimal dynamic
treatment regimes for binary and categorical exposures (tmle3mopttx),160 and
stochastic treatment regimes that shift the treatment mechanism of a
continuous exposure (tmle3shift).161 Additional recently developed packages in
R include ctmle for collaborative TMLE,162 haldensify for conditional density
estimation with HAL, 163, 164 txshift for estimating causal effects of
stochastic interventions,165, 166, 167 and lmtp for longitudinal modified
treatment policies.148

Although the TMLE framework is well developed in the R software, applied
epidemiological research is performed in several other software languages,
such as Stata, Python, and SAS. TMLE implementations for binary point exposure
and outcome studies are available in all of these languages. A SAS macro for
the general implementation of TMLE was programmed in 2016.93 TMLE has been
developed for the Python software language in the library zEpid.168 The number
of applied researchers in epidemiological studies using Python is relatively
low but is increasing; thus, this tool is not currently widely used among
applied health sciences researchers. Further development could improve on
software packages in the widely used statistical software in health sciences
and econometrics, such as Stata.169 Nonetheless, the development version of
the user-written Stata command eltmle is currently available to Stata
users.169 Not all features of TMLE are available in this Stata command, such
as longitudinal analysis and cross-validated TMLE. Additionally, eltmle
provides ensemble learning capabilities by accessing the SuperLearner R
package. Lastly, any new software development needs to have a friendly user
interface, together with standard programming features to be easily
disseminated and quickly adopted.

#### (ii) Accessibility of available material:

The TMLE framework is a series of potentially statistically-complex modelling
approaches and computational algorithms, grounded in statistical theory that
requires a solid understanding of highly advanced statistics (i.e., theory for
semi-parametric estimation, asymptotics, efficiency, empirical processes,
functional analyses, and statistical inference). Tutorials in a more lay
language targeting applied researchers and epidemiologists have become more
common over the past five years and the uptake of TMLE is expected to increase
in the future because of them.6, 93, 94, 7, 95, 5, 96, 1, 120, 103 Their
beneficial impact is evident from this review, as these articles are highly
referenced in applied work, from the year of their publication, alongside more
methodologically heavy contributions to start with, and as sole references in
later years. This shows evidence of the importance of speaking the language of
the target audience and disseminating advanced mathematical statistics and
algorithms from an applied perspective.

Additionally, the gradual dissemination of the TMLE framework was evident from
our systematic review of the methods sections of the 81 selected manuscripts.
We observed that papers published in the early years lay out their TMLE
strategy and carefully describe each step in the methods section; whereas,
more recently, publications of applied research have placed details of the
methods in appendices (or supplementary material) and only cite tutorials and
software packages. This shows that the community (e.g., authors, editors,
reviewers, readers, etc.) is now aware of the TMLE framework, its utility, and
its advantages. A wide range of journals have published the applied research
articles studied here, from non-specific public health journals to statistical
or disease-specific journals.

#### (iii) Experts:

Dissemination outside the US needs further work, as evidenced in our
systematic review. We have shown that the TMLE framework appears to be well
consolidated in the US, and adoption from Europe and other regions are lower
in comparison. This may be related to the delayed introduction of causal
inference education outside the US. Fostering targeted local seminars and
dedicated short courses for the interested applied audience could be a useful
strategy to disseminate the framework. Disease- or discipline-specific experts
would be useful for the wider distribution of the methods in specific areas
that would benefit from improved methodology.

TMLE remains dominant in non-communicable or infectious disease epidemiology
compared to other disciplines, but it has high applicability in many
disciplines and its use has increased in several of them. The slower uptake of
the TMLE framework in other disciplines might be due to a lack of empirical
examples of how one performed and interpreted statistical analyses using TMLE
in a specific disease area. We aimed to provide such a showcase of the
application of the methods in specific settings, based on the available
literature, and we demonstrated how the framework was successfully used to
advance research by providing robust results. We believe interested readers
will find it useful to refer to the studies that faced similar challenges, or
were based in settings comparable to theirs.

#### (iv) Teaching:

There have been tremendous efforts of dissemination of causal inference
methods across disciplines, with a particular emphasis on epidemiology and
econometrics sciences in the US during the last 20 years. Most graduate
programs in epidemiology have included the teaching of causal inference as a
leading topic in the field. In Europe, the trends have not been as fast-paced
and for a long time, introductions to causal inference methods have mainly
been provided through week-long intensive short courses and at international
conferences. These different approaches have major impacts on how quickly the
methods are adopted by the community of researchers, journal editors, public
health groups, and regulatory agencies. In recent years, there has been a
development and acceptance of real-world evidence in various public-health
fields, such as the Food and Drug Administration’s 21st Century Cures Act of
2016 in the US, which specifically promotes the use of causal inference
methodology and designs, such as the emulated trial and TMLE frameworks.170,
171, 172

#### (v) Collaborations:

The Center for Targeted Machine Learning and Causal Inference (CTML) is an
interdisciplinary research centre at the University of California at Berkeley
that is focused on applications of causal inference and targeted learning. The
CTML mission is to advance, implement and disseminate methodology to address
problems arising in public health and clinical medicine
(https://ctml.berkeley.edu/home). CTML provides a great resource for courses,
ongoing research, partners, collaborators, and Berkeley faculty members
involved in TMLE. CTML sponsors include the Danish multinational
pharmaceutical company, Novo Nordisk A/S, the Patient-Centered Outcomes
Research Institute (pcori), Kaiser Permanente, the US National Institutes of
Health, and the Bill and Melinda Gates Foundation. Academic partners include
the University of Washington, University of Copenhagen, UCLA David Geffen
School of Medicine, University of California at San Francisco, and Monash
University.

#### Conclusions

Evidence shows that cross-validated, double-robust, efficient and unbiased
estimators are at the forefront of causal inference and statistics, as they
aim to avoid model misspecification, bias and invalid inference. The TMLE
framework for causal and statistical inference was first developed in 2006 and
its expansion in applied studies arose in 2018 via applied epidemiological
work, tutorials and user-friendly software. The theoretical properties and
practical benefits of the TMLE framework have been highlighted across
different fields of applied research (such as various epidemiological, public
health and clinical disciplines). More can be done to reach a wider audience
across varied disease areas and scientific fields (e.g., genomics,
econometrics, political and sociological sciences), including the development
of software packages outside the R software, tutorial articles as well as
seminars and courses targeted to audiences in specific disciplines, lay-
language demonstration, such as by example, of the benefits of TMLE in
improving epidemiological output, to name only a few ideas. Many recent TMLE
developments answer a variety of methodological problems that expand across
scientific disciplines and further efforts can be made to disseminate the
framework. This would facilitate the conscientious application of TMLE for
causal inference and statistical data analyses, so more researchers could use
it in their applied work to minimise the risk of reporting misleading results
that are biased due to misspecification.

## Funding

This work was supported by the Medical Research Council [grant number
MR/W021021/1]. A CC BY or equivalent licence is applied to the Author Accepted
Manuscript (AAM) arising from this submission, in accordance with the grant’s
open access conditions. Camille Maringe is supported by a Cancer Research UK
Population Research Committee Programme Award (C7923/A29018).

## Authors contributions

The article arose from the motivation to disseminate the principles of modern
epidemiology among clinicians and applied researchers. All authors developed
the concept and wrote the first draft of the article. MJS and CM reviewed the
literature. MJS, RVP, MALF and CM drafted and revised the manuscript. RVP, SG
and MJL provided comments on the draft manuscript. RVP contributed to drafting
some sections. All authors read and approved the final version of the
manuscript. CM is the guarantor of the article.

## Acknowledgments

The motivation and some parts of the manuscript come from MALF’s work in a
visiting academic position in the Division of Biostatistics at the Berkeley
School of Public Health in 2019. We acknowledge Dr Susan Gruber for her
careful read over one draft manuscript and Professor Mark J. van der Laan for
his additional guidance.

For the purpose of open access, the authors have applied a Creative Commons
Attribution (CC BY) licence to any Author Accepted Manuscript version arising.

## References

  * 1 Smith MJ, Mansournia MA, Maringe C, et al. Introduction to computational causal inference using reproducible Stata, R, and Python code: A tutorial. Stat Med 2022; 41(2): 407-432.  doi: 10.1002/sim.9234
  * 2 van der Laan MJ, Rose S. Targeted Learning. 2011.  doi: 10.1007/978-1-4419-9782-1
  * 3 van der Laan MJ, Rubin D. Targeted maximum likelihood learning. The International Journal of Biostatistics 2006; 2(1).  doi: https://doi.org/10.2202/1557-4679.1043
  * 4 van der Laan MJ, Polley EC, Hubbard AE. Super learner. Statistical applications in genetics and molecular biology 2007; 6(1).  doi: 10.2202/1544-6115.1309
  * 5 Luque-Fernandez MA, Schomaker M, Rachet B, Schnitzer ME. Targeted maximum likelihood estimation for a binary treatment: A tutorial. Stat Med 2018; 37(16): 2530-2546.  doi: 10.1002/sim.7628
  * 6 Gruber S. Targeted Learning in Healthcare Research. Big Data 2015; 3(4): 211-8.  doi: 10.1089/big.2015.0025
  * 7 Schuler MS, Rose S. Targeted Maximum Likelihood Estimation for Causal Inference in Observational Studies. Am J Epidemiol 2017; 185(1): 65-73.  doi: 10.1093/aje/kww165
  * 8 van der Laan MJ, Rose S. Targeted learning in data science: causal inference for complex longitudinal studies. 2011.  doi: 10.1007/978-3-319-65304-4
  * 9 UC Berkeley . American Causal Inference Conference (2022). https://ctml.berkeley.edu/american-causal-inference-conference-2022; 2022
  * 10 UC Berkeley . American Causal Inference Conference (2022): Results. https://acic2022.mathematica.org/results; 2022
  * 11 Luque-Fernandez MA, Belot A, Valeri L, Cerulli G, Maringe C, Rachet B. Data-Adaptive Estimation for Double-Robust Methods in Population-Based Cancer Epidemiology: Risk Differences for Lung Cancer Mortality by Emergency Presentation. American Journal of Epidemiology 2018; 187(4): 871–878.  doi: 10.1093/AJE/KWX317
  * 12 Bembom O, Petersen ML, Rhee SY, et al. Biomarker discovery using targeted maximum-likelihood estimation: application to the treatment of antiretroviral-resistant HIV infection. Stat Med 2009; 28(1): 152-72.  doi: 10.1002/sim.3414
  * 13 Rosenblum M, Deeks SG, van der Laan MJ, Bangsberg DR. The risk of virologic failure decreases with duration of HIV suppression, at greater than 50% adherence to antiretroviral therapy. PLoS One 2009; 4(9): e7196.  doi: 10.1371/journal.pone.0007196
  * 14 Mackey DC, Hubbard AE, Cawthon PM, Cauley JA, Cummings SR, Tager IB. Usual physical activity and hip fracture in older men: an application of semiparametric methods to observational data. Am J Epidemiol 2011; 173(5): 578-86.  doi: 10.1093/aje/kwq405
  * 15 Padula AM, Mortimer K, Hubbard A, Lurmann F, Jerrett M, Tager IB. Exposure to traffic-related air pollution during pregnancy and term low birth weight: estimation of causal associations in a semiparametric model. Am J Epidemiol 2012; 176(9): 815-24.  doi: 10.1093/aje/kws148
  * 16 Leslie HH, Karasek DA, Harris LF, et al. Cervical cancer precursors and hormonal contraceptive use in HIV-positive women: application of a causal model and semi-parametric estimation methods. PLoS One 2014; 9(6): e101090.  doi: 10.1371/journal.pone.0101090
  * 17 Schnitzer ME, Moodie EE, van der Laan MJ, Platt RW, Klein MB. Modeling the impact of hepatitis C viral clearance on end-stage liver disease in an HIV co-infected cohort with targeted maximum likelihood estimation. Biometrics 2014; 70(1): 144-52.  doi: 10.1111/biom.12105
  * 18 Brown DM, Petersen M, Costello S, et al. Occupational Exposure to PM2.5 and Incidence of Ischemic Heart Disease: Longitudinal Targeted Minimum Loss-based Estimation. Epidemiology 2015; 26(6): 806-14.  doi: 10.1097/ede.0000000000000329
  * 19 Gianfrancesco MA, Balzer L, Taylor KE, et al. Genetic risk and longitudinal disease activity in systemic lupus erythematosus using targeted maximum likelihood estimation. Genes Immun 2016; 17(6): 358-62.  doi: 10.1038/gene.2016.33
  * 20 Hsu LI, Briggs F, Shao X, et al. Pathway Analysis of Genome-wide Association Study in Childhood Leukemia among Hispanics. Cancer Epidemiol Biomarkers Prev 2016; 25(5): 815-22.  doi: 10.1158/1055-9965.Epi-15-0528
  * 21 Ahern J, Karasek D, Luedtke AR, Bruckner TA, van der Laan MJ. Racial/Ethnic Differences in the Role of Childhood Adversities for Mental Disorders Among a Nationally Representative Sample of Adolescents. Epidemiology 2016; 27(5): 697-704.  doi: 10.1097/ede.0000000000000507
  * 22 Salihu HM, Das R, Morton L, et al. Racial Differences in DNA-Methylation of CpG Sites Within Preterm-Promoting Genes and Gene Variants. Matern Child Health J 2016; 20(8): 1680-7.  doi: 10.1007/s10995-016-1967-3
  * 23 Tran L, Yiannoutsos CT, Musick BS, et al. Evaluating the Impact of a HIV Low-Risk Express Care Task-Shifting Program: A Case Study of the Targeted Learning Roadmap. Epidemiol Methods 2016; 5(1): 69-91.  doi: 10.1515/em-2016-0004
  * 24 Davis FM, Sutzko DC, Grey SF, et al. Predictors of surgical site infection after open lower extremity revascularization. J Vasc Surg 2017; 65(6): 1769-1778.e3.  doi: 10.1016/j.jvs.2016.11.053
  * 25 Skeem JL, Manchak S, Montoya L. Comparing Public Safety Outcomes for Traditional Probation vs Specialty Mental Health Probation. JAMA Psychiatry 2017; 74(9): 942-948.  doi: 10.1001/jamapsychiatry.2017.1384
  * 26 Sukul D, Seth M, Dixon SR, Khandelwal A, LaLonde TA, Gurm HS. Contemporary Trends and Outcomes Associated With the Preprocedural Use of Oral P2Y12 Inhibitors in Patients Undergoing Percutaneous Coronary Intervention: Insights From the Blue Cross Blue Shield of Michigan Cardiovascular Consortium (BMC2). J Invasive Cardiol 2017; 29(10): 340-351.
  * 27 Akosile M, Zhu H, Zhang S, Johnson NP, Lai D, Zhu H. Reassessing the Effectiveness of Right Heart Catheterization (RHC) in the Initial Care of Critically Ill Patients using Targeted Maximum Likelihood Estimation. International Journal of Clinical Biostatistics and Biometrics 2018; 4(1).  doi: 10.23937/2469-5831/1510018
  * 28 Platt JM, McLaughlin KA, Luedtke AR, Ahern J, Kaufman AS, Keyes KM. Targeted Estimation of the Relationship Between Childhood Adversity and Fluid Intelligence in a US Population Sample of Adolescents. Am J Epidemiol 2018; 187(7): 1456-1466.  doi: 10.1093/aje/kwy006
  * 29 Pearl M, Ahern J, Hubbard A, et al. Life-course neighbourhood opportunity and racial-ethnic disparities in risk of preterm birth. Paediatr Perinat Epidemiol 2018; 32(5): 412-419.  doi: 10.1111/ppe.12482
  * 30 Skeem JL, Montoya L, Manchak SM. Comparing Costs of Traditional and Specialty Probation for People With Serious Mental Illness. Psychiatr Serv 2018; 69(8): 896-902.  doi: 10.1176/appi.ps.201700498
  * 31 Mosconi L, Rahman A, Diaz I, et al. Increased Alzheimer’s risk during the menopause transition: A 3-year longitudinal brain imaging study. PLoS One 2018; 13(12): e0207885.  doi: 10.1371/journal.pone.0207885
  * 32 Torres JM, Rudolph KE, Sofrygin O, Glymour MM, Wong R. Longitudinal associations between having an adult child migrant and depressive symptoms among older adults in the Mexican Health and Aging Study. Int J Epidemiol 2018; 47(5): 1432-1442.  doi: 10.1093/ije/dyy112
  * 33 Izano MA, Sofrygin OA, Picciotto S, Bradshaw PT, Eisen EA. Metalworking Fluids and Colon Cancer Risk: Longitudinal Targeted Minimum Loss-based Estimation. Environ Epidemiol 2019; 3(1): e035.  doi: 10.1097/ee9.0000000000000035
  * 34 Yu YH, Bodnar LM, Brooks MM, Himes KP, Naimi AI. Comparison of Parametric and Nonparametric Estimators for the Association Between Incident Prepregnancy Obesity and Stillbirth in a Population-Based Cohort Study. Am J Epidemiol 2019; 188(7): 1328-1336.  doi: 10.1093/aje/kwz081
  * 35 Rudolph KE, Shev A, Paksarian D, et al. Environmental noise and sleep and mental health outcomes in a nationally representative sample of urban US adolescents. Environ Epidemiol 2019; 3(4): e056.  doi: 10.1097/ee9.0000000000000056
  * 36 Lim S, Tellez M, Ismail AI. Estimating a Dynamic Effect of Soda Intake on Pediatric Dental Caries Using Targeted Maximum Likelihood Estimation Method. Caries Res 2019; 53(5): 532-540.  doi: 10.1159/000497359
  * 37 Berkowitz SA, Basu S, Gundersen C, Seligman HK. State-Level and County-Level Estimates of Health Care Costs Associated with Food Insecurity. Prev Chronic Dis 2019; 16: E90.  doi: 10.5888/pcd16.180549
  * 38 Gianfrancesco MA, Trupin L, Shiboski S, et al. Smoking Is Associated with Higher Disease Activity in Rheumatoid Arthritis: A Longitudinal Study Controlling for Time-varying Covariates. J Rheumatol 2019; 46(4): 370-375.  doi: 10.3899/jrheum.180262
  * 39 Casey JA, Goin DE, Rudolph KE, et al. Unconventional natural gas development and adverse birth outcomes in Pennsylvania: The potential mediating role of antenatal anxiety and depression. Environ Res 2019; 177: 108598\.  doi: 10.1016/j.envres.2019.108598
  * 40 Torres JM, Rudolph KE, Sofrygin O, Wong R, Walter LC, Glymour MM. Having an Adult Child in the United States, Physical Functioning, and Unmet Needs for Care Among Older Mexican Adults. Epidemiology 2019; 30(4): 553-560.  doi: 10.1097/ede.0000000000001016
  * 41 You Y, Doubova SV, Pinto-Masis D, Pérez-Cuevas R, Borja-Aburto VH, Hubbard A. Application of machine learning methodology to assess the performance of DIABETIMSS program for patients with type 2 diabetes in family medicine clinics in Mexico. BMC Medical Informatics and Decision Making 2019; 19(1).  doi: 10.1186/S12911-019-0950-5
  * 42 Izano MA, Cushing LJ, Lin J, et al. The association of maternal psychosocial stress with newborn telomere length. PLoS One 2020; 15(12): e0242064.  doi: 10.1371/journal.pone.0242064
  * 43 Ehrlich SF, Neugebauer RS, Feng J, Hedderson MM, Ferrara A. Exercise During the First Trimester and Infant Size at Birth: Targeted Maximum Likelihood Estimation of the Causal Risk Difference. Am J Epidemiol 2020; 189(2): 133-145.  doi: 10.1093/aje/kwz213
  * 44 Bodnar LM, Cartus AR, Kirkpatrick SI, et al. Machine learning as a strategy to account for dietary synergy: an illustration based on dietary intake and adverse pregnancy outcomes. Am J Clin Nutr 2020; 111(6): 1235-1243.  doi: 10.1093/ajcn/nqaa027
  * 45 Kagawa RMC, Pear VA, Rudolph KE, Keyes KM, Cerdá M, Wintemute GJ. Distress level and daily functioning problems attributed to firearm victimization: sociodemographic-specific responses. Ann Epidemiol 2020; 41: 35-42.e3.  doi: 10.1016/j.annepidem.2019.12.002
  * 46 Kempker RR, Mikiashvili L, Zhao Y, et al. Clinical Outcomes Among Patients With Drug-resistant Tuberculosis Receiving Bedaquiline- or Delamanid-Containing Regimens. Clin Infect Dis 2020; 71(9): 2336-2344.  doi: 10.1093/cid/ciz1107
  * 47 Yu YH, Bodnar LM, Himes KP, Brooks MM, Naimi AI. Association of Overweight and Obesity Development Between Pregnancies With Stillbirth and Infant Mortality in a Cohort of Multiparous Women. Obstet Gynecol 2020; 135(3): 634-643.  doi: 10.1097/aog.0000000000003677
  * 48 Puryear SB, Balzer LB, Ayieko J, et al. Associations between alcohol use and HIV care cascade outcomes among adults undergoing population-based HIV testing in East Africa. Aids 2020; 34(3): 405-413.  doi: 10.1097/qad.0000000000002427
  * 49 Torres JM, Sofrygin O, Rudolph KE, Haan MN, Wong R, Glymour MM. US Migration Status of Adult Children and Cognitive Decline Among Older Parents Who Remain in Mexico. Am J Epidemiol 2020; 189(8): 761-769.  doi: 10.1093/aje/kwz277
  * 50 Westling T, Cowden C, Mwananyanda L, et al. Impact of chlorhexidine baths on suspected sepsis and bloodstream infections in hospitalized neonates in Zambia. Int J Infect Dis 2020; 96: 54-60.  doi: 10.1016/j.ijid.2020.03.043
  * 51 Aiemjoy K, Aragie S, Wittberg DM, et al. Seroprevalence of antibodies against Chlamydia trachomatis and enteropathogens and distance to the nearest water source among young children in the Amhara Region of Ethiopia. PLoS Negl Trop Dis 2020; 14(9): e0008647.  doi: 10.1371/journal.pntd.0008647
  * 52 Reilly ME, Conti MS, Day J, et al. Modified Lapidus vs Scarf Osteotomy Outcomes for Treatment of Hallux Valgus Deformity. Foot Ankle Int 2021; 42(11): 1454-1462.  doi: 10.1177/10711007211013776
  * 53 Kang L, Vij A, Hubbard A, Shaw D. The unintended impact of helmet use on bicyclists’ risk-taking behaviors. J Safety Res 2021; 79: 135-147.  doi: 10.1016/j.jsr.2021.08.014
  * 54 Torres JM, Mitchell UA, Sofrygin O, et al. Associations between spousal caregiving and health among older adults in Mexico: A targeted estimation approach. Int J Geriatr Psychiatry 2021; 36(5): 775-783.  doi: 10.1002/gps.5477
  * 55 Torres LK, Hoffman KL, Oromendia C, et al. Attributable mortality of acute respiratory distress syndrome: a systematic review, meta-analysis and survival analysis using targeted minimum loss-based estimation. Thorax 2021; 76(12): 1176-1185.  doi: 10.1136/thoraxjnl-2020-215950
  * 56 Mehta B, Brantner C, Williams N, et al. Primary Care Provider Density and Elective Total Joint Replacement Outcomes. Arthroplast Today 2021; 10: 73-78.  doi: 10.1016/j.artd.2021.05.010
  * 57 Nardone A, Rudolph KE, Morello-Frosch R, Casey JA. Redlines and Greenspace: The Relationship between Historical Redlining and 2010 Greenspace across the United States. Environ Health Perspect 2021; 129(1): 17006\.  doi: 10.1289/ehp7495
  * 58 Lee JO, Kapteyn A, Clomax A, Jin H. Estimating influences of unemployment and underemployment on mental health during the COVID-19 pandemic: who suffers the most?. Public Health 2021; 201: 48-54.  doi: 10.1016/j.puhe.2021.09.038
  * 59 Shiba K, Torres JM, Daoud A, et al. Estimating the Impact of Sustained Social Participation on Depressive Symptoms in Older Adults. Epidemiology 2021; 32(6): 886-895.  doi: 10.1097/ede.0000000000001395
  * 60 Goel AR, Bruce HA, Williams N, Alexiades G. Long-Term Effects of Hearing Aids on Hearing Ability in Patients with Sensorineural Hearing Loss. J Am Acad Audiol 2021; 32(6): 374-378.  doi: 10.1055/s-0041-1731592
  * 61 Kahkoska AR, Abrahamsen TJ, Alexander GC, et al. Association Between Glucagon-Like Peptide 1 Receptor Agonist and Sodium-Glucose Cotransporter 2 Inhibitor Use and COVID-19 Outcomes. Diabetes Care 2021; 44(7): 1564-1572.  doi: 10.2337/dc21-0065
  * 62 Beydoun HA, Beydoun MA, Huang S, Eid SM, Zonderman AB. Hospitalization outcomes among brain metastasis patients receiving radiation therapy with or without stereotactic radiosurgery from the 2005-2014 Nationwide Inpatient Sample. Sci Rep 2021; 11(1): 19209.  doi: 10.1038/s41598-021-98563-y
  * 63 Crowner JR, Marston WA, Freeman NLB, et al. The Society for Vascular Surgery Objective Performance Goals for Critical Limb Ischemia are attainable in select patients with ischemic wounds managed with wound care alone. Ann Vasc Surg 2022; 78: 28-35.  doi: 10.1016/j.avsg.2021.06.034
  * 64 Wong AK, Balzer LB. State-Level Masking Mandates and COVID-19 Outcomes in the United States: A Demonstration of the Causal Roadmap. Epidemiology 2022; 33(2): 228-236.  doi: 10.1097/ede.0000000000001453
  * 65 Legrand M, Pirracchio R, Rosa A, et al. Incidence, risk factors and prediction of post-operative acute kidney injury following cardiac surgery for active infective endocarditis: an observational study. Crit Care 2013; 17(5): R220.  doi: 10.1186/cc13041
  * 66 Herrera R, Berger U, Ehrenstein vOS, et al. Estimating the Causal Impact of Proximity to Gold and Copper Mines on Respiratory Diseases in Chilean Children: An Application of Targeted Maximum Likelihood Estimation. Int J Environ Res Public Health 2017; 15(1).  doi: 10.3390/ijerph15010039
  * 67 Rodríguez-Molina D, Barth S, Herrera R, Rossmann C, Radon K, Karnowski V. An educational intervention to improve knowledge about prevention against occupational asthma and allergies using targeted maximum likelihood estimation. Int Arch Occup Environ Health 2019; 92(5): 629-638.  doi: 10.1007/s00420-018-1397-1
  * 68 Papadopoulou E, Haug LS, Sakhi AK, et al. Diet as a Source of Exposure to Environmental Contaminants for Pregnant Women and Children from Six European Countries. Environ Health Perspect 2019; 127(10): 107005.  doi: 10.1289/ehp5324
  * 69 Vauchel T, Pirracchio R, Chaussard M, et al. Impact of an Acinetobacter baumannii outbreak on kidney events in a burn unit: A targeted machine learning analysis. American Journal of Infection Control 2019; 47(4): 435–438.  doi: 10.1016/j.ajic.2018.09.010
  * 70 Veit C, Herrera R, Weinmayr G, et al. Long-term effects of asthma medication on asthma symptoms: an application of the targeted maximum likelihood estimation. BMC Med Res Methodol 2020; 20(1): 307.  doi: 10.1186/s12874-020-01175-9
  * 71 Decruyenaere A, Steen J, Colpaert K, Benoit DD, Decruyenaere J, Vansteelandt S. The obesity paradox in critically ill patients: a causal learning approach to a casual finding. Crit Care 2020; 24(1): 485.  doi: 10.1186/s13054-020-03199-5
  * 72 Rossides M, Kullberg S, Di Giuseppe D, et al. Infection risk in sarcoidosis patients treated with methotrexate compared to azathioprine: A retrospective ’target trial’ emulated with Swedish real-world data. Respirology 2021; 26(5): 452-460.  doi: 10.1111/resp.14001
  * 73 Zou R, El Marroun H, Cecil C, et al. Maternal folate levels during pregnancy and offspring brain development in late childhood. Clin Nutr 2021; 40(5): 3391-3400.  doi: 10.1016/j.clnu.2020.11.025
  * 74 Isfordink CJ, Smit C, Boyd A, et al. Low hepatitis C virus-viremia prevalence yet continued barriers to direct-acting antiviral treatment in people living with HIV in the Netherlands. Aids 2022; 36(6): 773-783.  doi: 10.1097/qad.0000000000003159
  * 75 Bell-Gorrod H, Fox MP, Boulle A, et al. The Impact of Delayed Switch to Second-Line Antiretroviral Therapy on Mortality, Depending on Definition of Failure Time and CD4 Count at Failure. Am J Epidemiol 2020; 189(8): 811-819.  doi: 10.1093/aje/kwaa049
  * 76 Amusa L, Zewotir T, North D, Kharsany ABM, Lewis L. Association of medical male circumcision and sexually transmitted infections in a population-based study using targeted maximum likelihood estimation. BMC Public Health 2021; 21(1): 1642.  doi: 10.1186/s12889-021-11705-9
  * 77 Kerschberger B, Boulle A, Kuwengwa R, Ciglenecki I, Schomaker M. The Impact of Same-Day Antiretroviral Therapy Initiation Under the World Health Organization Treat-All Policy. Am J Epidemiol 2021; 190(8): 1519-1532.  doi: 10.1093/aje/kwab032
  * 78 Dadi AF, Miller ER, Woodman RJ, Azale T, Mwanri L. Effect of perinatal depression on risk of adverse infant health outcomes in mother-infant dyads in Gondar town: a causal analysis. BMC Pregnancy Childbirth 2021; 21(1): 255.  doi: 10.1186/s12884-021-03733-5
  * 79 Mozafar Saadati H, Mehrabi Y, Sabour S, Mansournia MA, Hashemi Nazari SS. Estimating the effects of body mass index and central obesity on stroke in diabetics and non-diabetics using targeted maximum likelihood estimation: Atherosclerosis Risk in Communities study. Obes Sci Pract 2020; 6(6): 628-637.  doi: 10.1002/osp4.447
  * 80 Abdollahpour I, Nedjat S, Almasi-Hashiani A, Nazemipour M, Mansournia MA, Luque-Fernandez MA. Estimating the Marginal Causal Effect and Potential Impact of Waterpipe Smoking on Risk of Multiple Sclerosis Using the Targeted Maximum Likelihood Estimation Method: A Large, Population-Based Incident Case-Control Study. Am J Epidemiol 2021; 190(7): 1332-1340.  doi: 10.1093/aje/kwab036
  * 81 Mozafar Saadati H, Sabour S, Mansournia MA, Mehrabi Y, Hashemi Nazari SS. Effect modification of general and central obesity by sex and age on cardiovascular outcomes: Targeted maximum likelihood estimation in the atherosclerosis risk in communities study. Diabetes Metab Syndr 2021; 15(2): 479-485.  doi: 10.1016/j.dsx.2021.02.024
  * 82 Almasi-Hashiani A, Nedjat S, Ghiasvand R, et al. The causal effect and impact of reproductive factors on breast cancer using super learner and targeted maximum likelihood estimation: a case-control study in Fars Province, Iran. BMC Public Health 2021; 21(1): 1219.  doi: 10.1186/s12889-021-11307-5
  * 83 Akhtar S, El-Muzaini H, Alroughani R. Recombinant hepatitis B vaccine uptake and multiple sclerosis risk: A marginal structural modeling approach. Mult Scler Relat Disord 2022; 58: 103487.  doi: 10.1016/j.msard.2022.103487
  * 84 Clare PJ, Dobbins T, Bruno R, et al. The overall effect of parental supply of alcohol across adolescence on alcohol-related harms in early adulthood-a prospective cohort study. Addiction 2020; 115(10): 1833-1843.  doi: 10.1111/add.15005
  * 85 Wang L, Sun X, Jin C, Fan Y, Xue F. Identification of Tumor Microenvironment-Related Prognostic Biomarkers for Ovarian Serous Cancer 3-Year Mortality Using Targeted Maximum Likelihood Estimation: A TCGA Data Mining Study. Front Genet 2021; 12: 625145.  doi: 10.3389/fgene.2021.625145
  * 86 Sun X, Wang L, Li H, et al. Identification of microenvironment related potential biomarkers of biochemical recurrence at 3 years after prostatectomy in prostate adenocarcinoma. Aging (Albany NY) 2021; 13(12): 16024-16042.  doi: 10.18632/aging.203121
  * 87 Chavda MP, Bihari S, Woodman RJ, Secombe P, Pilcher D. The impact of obesity on outcomes of patients admitted to intensive care after cardiac arrest. J Crit Care 2022; 69: 154025.  doi: 10.1016/j.jcrc.2022.154025
  * 88 Chen C, Chen X, Chen J, et al. Association between Preoperative hs-crp/Albumin Ratio and Postoperative sirs in Elderly Patients: A Retrospective Observational Cohort Study. J Nutr Health Aging 2022; 26(4): 352-359.  doi: 10.1007/s12603-022-1761-4
  * 89 Ikeda T, Cooray U, Murakami M, Osaka K. Maintaining Moderate or Vigorous Exercise Reduces the Risk of Low Back Pain at 4 Years of Follow-Up: Evidence From the English Longitudinal Study of Ageing. J Pain 2022; 23(3): 390-397.  doi: 10.1016/j.jpain.2021.08.008
  * 90 Ikeda T, Tsuboya T. Effects of changes in depressive symptoms on handgrip strength in later life: A four-year longitudinal study in England. J Affect Disord 2022; 299: 67-72.  doi: 10.1016/j.jad.2021.11.057
  * 91 Moreno-Betancur M, Lynch JW, Pilkington RM, et al. Emulating a target trial of intensive nurse home visiting in the policy-relevant population using linked administrative data. Int J Epidemiol 2022\.  doi: 10.1093/ije/dyac092
  * 92 Figueroa SC, Kennedy CJ, Wesseling C, Wiemels JM, Morimoto L, Mora AM. Early immune stimulation and childhood acute lymphoblastic leukemia in Costa Rica: A comparison of statistical approaches. Environmental research 2020; 182: 109023.  doi: 10.1016/J.ENVRES.2019.109023
  * 93 Pang M, Schuster T, Filion KB, Eberg M, Platt RW. Targeted Maximum Likelihood Estimation for Pharmacoepidemiologic Research. Epidemiology 2016; 27(4): 570-7.  doi: 10.1097/ede.0000000000000487
  * 94 Pearl M, Balzer L, Ahern J. Targeted Estimation of Marginal Absolute and Relative Associations in Case-Control Data: An Application in Social Epidemiology. Epidemiology 2016; 27(4): 512-7.  doi: 10.1097/ede.0000000000000476
  * 95 Kreif N, Tran L, Grieve R, De Stavola B, Tasker RC, Petersen M. Estimating the Comparative Effectiveness of Feeding Interventions in the Pediatric Intensive Care Unit: A Demonstration of Longitudinal Targeted Maximum Likelihood Estimation. Am J Epidemiol 2017; 186(12): 1370-1379.  doi: 10.1093/aje/kwx213
  * 96 Almasi-Hashiani A, Nedjat S, Mansournia MA. Causal Methods for Observational Research: A Primer. Arch Iran Med 2018; 21(4): 164-169.
  * 97 Skeem JL, Montoya L, Manchak SM. Comparing Costs of Traditional and Specialty Probation for People With Serious Mental Illness. Psychiatric services (Washington, D.C.) 2018; 69(8): 896–902.  doi: 10.1176/APPI.PS.201700498
  * 98 Saddiki H, Balzer LB. A Primer on Causality in Data Science. 2018.  doi: 10.48550/arxiv.1809.02408
  * 99 Coyle JR, Hejazi NS, Malenica I, et al. Targeting Learning: Robust Statistics for Reproducible Research. 2020.  doi: 10.48550/arXiv.2006.07333
  * 100 Gruber S, van der Laan MJ. tmle: An R Package for Targeted Maximum Likelihood Estimation. Journal of Statistical Software 2012; 51(13): 1–35.  doi: https://doi.org/10.18637/jss.v051.i13
  * 101 Lendle SD, Schwab J, Petersen ML, van der Laan MJ. ltmle: An R Package Implementing Targeted Minimum Loss-Based Estimation for Longitudinal Data. Journal of Statistical Software 2017; 81: 1–21.  doi: 10.18637/JSS.V081.I01
  * 102 Hernán MA. The Hazards of Hazard Ratios. Epidemiology (Cambridge, Mass.) 2010; 21(1): 13.  doi: 10.1097/EDE.0B013E3181C1EA43
  * 103 Phillips RV, Laan v. dMJ, Lee H, Gruber S. Practical considerations for specifying a super learner. arXiv 2022\.  doi: doi.org/10.48550/arXiv.2204.06139
  * 104 Moore KL, van der Laan MJ. Covariate adjustment in randomized trials with binary outcomes: targeted maximum likelihood estimation. Stat Med 2009; 28(1): 39-64.  doi: 10.1002/sim.3445
  * 105 van der Laan MJ. Targeted maximum likelihood based causal inference: Part I. Int J Biostat 2010; 6(2): Article 2.  doi: 10.2202/1557-4679.1211
  * 106 van der Laan MJ. Targeted maximum likelihood based causal inference: Part II. Int J Biostat 2010; 6(2): Article 3.  doi: 10.2202/1557-4679.1241
  * 107 van der Laan MJ, Balzer LB, Petersen ML. Adaptive matching in randomized trials and observational studies. J Stat Res 2012; 46(2): 113-156.
  * 108 Schomaker M, Luque-Fernandez MA, Leroy V, Davies MA. Using longitudinal targeted maximum likelihood estimation in complex settings with dynamic interventions. Stat Med 2019; 38(24): 4888-4911.  doi: 10.1002/sim.8340
  * 109 Zheng W, Luo Z, van der Laan MJ. Marginal Structural Models with Counterfactual Effect Modifiers. Int J Biostat 2018; 14(1).  doi: 10.1515/ijb-2018-0039
  * 110 van der Laan MJ, Gruber S. Targeted minimum loss based estimation of causal effects of multiple time point interventions. Int J Biostat 2012; 8(1).  doi: 10.1515/1557-4679.1370
  * 111 Schnitzer ME, Lok JJ, Bosch RJ. Double robust and efficient estimation of a prognostic model for events in the presence of dependent censoring. Biostatistics 2016; 17(1): 165-77.  doi: 10.1093/biostatistics/kxv028
  * 112 van der Laan MJ. Causal Inference for a Population of Causally Connected Units. J Causal Inference 2014; 2(1): 13-74.  doi: 10.1515/jci-2013-0002
  * 113 Sofrygin O, van der Laan MJ. Semi-Parametric Estimation and Inference for the Mean Outcome of the Single Time-Point Intervention in a Causally Connected Population. J Causal Inference 2017; 5(1).  doi: 10.1515/jci-2016-0003
  * 114 Balzer LB, Zheng W, van der Laan MJ, Petersen ML. A new approach to hierarchical data analysis: Targeted maximum likelihood estimation for the causal effect of a cluster-level exposure. Stat Methods Med Res 2019; 28(6): 1761-1780.  doi: 10.1177/0962280218774936
  * 115 Balzer LB, van der Laan M, Ayieko J, et al. Two-Stage TMLE to reduce bias and improve efficiency in cluster randomized trials. Biostatistics 2021\.  doi: 10.1093/biostatistics/kxab043
  * 116 Sofrygin O, Zhu Z, Schmittdiel JA, et al. Targeted learning with daily EHR data. Stat Med 2019; 38(16): 3073-3090.  doi: 10.1002/sim.8164
  * 117 Gruber S, van der Laan MJ. An application of targeted maximum likelihood estimation to the meta-analysis of safety data. Biometrics 2013; 69(1): 254-62.  doi: 10.1111/j.1541-0420.2012.01829.x
  * 118 Liu Y, Schnitzer ME, Wang G, et al. Modeling treatment effect modification in multidrug-resistant tuberculosis in an individual patientdata meta-analysis. Stat Methods Med Res 2022; 31(4): 689-705.  doi: 10.1177/09622802211046383
  * 119 Rose S, Laan MJ. Why match? Investigating matched case-control study designs with causal effect estimation. Int J Biostat 2009; 5(1): Article 1.  doi: 10.2202/1557-4679.1127
  * 120 Rose S, van der Laan MJ. A targeted maximum likelihood estimator for two-stage designs. Int J Biostat 2011; 7(1): 17.  doi: 10.2202/1557-4679.1217
  * 121 Balzer LB, Petersen ML, van der Laan MJ. Adaptive pair-matching in randomized trials with unbiased and efficient effect estimation. Stat Med 2015; 34(6): 999-1011.  doi: 10.1002/sim.6380
  * 122 Balzer LB, Petersen ML, van der Laan MJ. Targeted estimation and inference for the sample average treatment effect in trials with and without pair-matching. Stat Med 2016; 35(21): 3717-32.  doi: 10.1002/sim.6965
  * 123 van der Laan MJ, Gruber S. Collaborative double robust targeted maximum likelihood estimation. Int J Biostat 2010; 6(1): Article 17\.  doi: 10.2202/1557-4679.1181
  * 124 Pirracchio R, Yue JK, Manley GT, van der Laan MJ, Hubbard AE. Collaborative targeted maximum likelihood estimation for variable importance measure: Illustration for functional outcome prediction in mild traumatic brain injuries. Stat Methods Med Res 2018; 27(1): 286-297.  doi: 10.1177/0962280215627335
  * 125 Schnitzer ME, Sango J, Ferreira Guerra S, van der Laan MJ. Data-adaptive longitudinal model selection in causal inference with collaborative targeted minimum loss-based estimation. Biometrics 2020; 76(1): 145-157.  doi: 10.1111/biom.13135
  * 126 Ju C, Gruber S, Lendle SD, et al. Scalable collaborative targeted learning for high-dimensional data. Stat Methods Med Res 2019; 28(2): 532-554.  doi: 10.1177/0962280217729845
  * 127 Ju C, Schwab J, van der Laan MJ. On adaptive propensity score truncation in causal inference. Stat Methods Med Res 2019; 28(6): 1741-1760.  doi: 10.1177/0962280218774817
  * 128 Ju C, Wyss R, Franklin JM, Schneeweiss S, Häggström J, van der Laan MJ. Collaborative-controlled LASSO for constructing propensity score-based estimators in high-dimensional data. Stat Methods Med Res 2019; 28(4): 1044-1063.  doi: 10.1177/0962280217744588
  * 129 Petersen M, Schwab J, Gruber S, Blaser N, Schomaker M, van der Laan MJ. Targeted Maximum Likelihood Estimation for Dynamic and Static Longitudinal Marginal Structural Working Models. J Causal Inference 2014; 2(2): 147-185.  doi: 10.1515/jci-2013-0007
  * 130 Ferreira Guerra S, Schnitzer ME, Forget A, Blais L. Impact of discretization of the timeline for longitudinal causal inference methods. Stat Med 2020; 39(27): 4069-4085.  doi: 10.1002/sim.8710
  * 131 Zheng W, Petersen M, van der Laan MJ. Doubly Robust and Efficient Estimation of Marginal Structural Models for the Hazard Function. Int J Biostat 2016; 12(1): 233-52.  doi: 10.1515/ijb-2015-0036
  * 132 van der Laan M, Gruber S. One-Step Targeted Minimum Loss-based Estimation Based on Universal Least Favorable One-Dimensional Submodels. International Journal of Biostatistics 2016; 12(1): 351–378.  doi: 10.1515/IJB-2015-0054
  * 133 Cai W, van der Laan MJ. One-step targeted maximum likelihood estimation for time-to-event outcomes. Biometrics 2020; 76(3): 722-733.  doi: 10.1111/biom.13172
  * 134 Zhu J, Gallego B. Targeted estimation of heterogeneous treatment effect in observational survival analysis. J Biomed Inform 2020; 107: 103474.  doi: 10.1016/j.jbi.2020.103474
  * 135 Rudolph KE, van der Laan MJ. Robust estimation of encouragement-design intervention effects transported across sites. J R Stat Soc Series B Stat Methodol 2017; 79(5): 1509-1525.  doi: 10.1111/rssb.12213
  * 136 van der Laan MJ, Luedtke AR. Targeted Learning of the Mean Outcome under an Optimal Dynamic Treatment Rule. J Causal Inference 2015; 3(1): 61-95.  doi: 10.1515/jci-2013-0022
  * 137 van der Laan L, Zhang W, Gilbert PB. Nonparametric estimation of the causal effect of a stochastic threshold-based intervention. Biometrics 2022\.  doi: 10.1111/biom.13690
  * 138 Zheng W, van der Laan MJ. Targeted maximum likelihood estimation of natural direct effects. Int J Biostat 2012; 8(1).  doi: 10.2202/1557-4679.1361
  * 139 Lendle SD, Subbaraman MS, van der Laan MJ. Identification and efficient estimation of the natural direct effect among the untreated. Biometrics 2013; 69(2): 310-7.  doi: 10.1111/biom.12022
  * 140 Rudolph KE, Sofrygin O, Zheng W, van der Laan MJ. Robust and Flexible Estimation of Stochastic Mediation Effects: A Proposed Method and Example in a Randomized Trial Setting. Epidemiol Methods 2018; 7(1).  doi: 10.1515/em-2017-0007
  * 141 Box GE. Science and statistics. Journal of the American Statistical Association 1976; 71(356): 791–799.  doi: 10.1080/01621459.1976.10480949
  * 142 Chernozhukov V, Chetverikov D, Demirer M, et al. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal 2018; 21(1): C1-C68.  doi: 10.1111/ECTJ.12097
  * 143 Zivich PN, Breskin A. Machine Learning for Causal Inference: On the Use of Cross-fit Estimators. Epidemiology 2021; 32(3): 393-401.  doi: 10.1097/ede.0000000000001332
  * 144 Gruber S, van der Laan MJ. A targeted maximum likelihood estimator of a causal effect on a bounded continuous outcome. Int J Biostat 2010; 6(1): Article 26.  doi: 10.2202/1557-4679.1260
  * 145 Balzer L, Ahern J, Galea S, van der Laan M. Estimating Effects with Rare Outcomes and High Dimensional Covariates: Knowledge is Power. Epidemiol Methods 2016; 5(1): 1-18.  doi: 10.1515/em-2014-0020
  * 146 Benkeser D, Carone M, Gilbert PB. Improved estimation of the cumulative incidence of rare outcomes. Stat Med 2018; 37(2): 280-293.  doi: 10.1002/sim.7337
  * 147 Díaz I, Colantuoni E, Rosenblum M. Enhanced precision in the analysis of randomized trials with ordinal outcomes. Biometrics 2016; 72(2): 422-31.  doi: 10.1111/biom.12450
  * 148 Díaz I, Williams N, Hoffman KL, Schenck EJ. Nonparametric Causal Effects Based on Longitudinal Modified Treatment Policies. Journal of the American Statistical Association 2021\.  doi: 10.1080/01621459.2021.1955691
  * 149 Díaz I, van der Laan MJ. Doubly robust inference for targeted minimum loss-based estimation in randomized trials with missing outcome data. Stat Med 2017; 36(24): 3807-3819.  doi: 10.1002/sim.7389
  * 150 Gruber S, van der Laan MJ. tmle: An R Package for Targeted Maximum Likelihood Estimation. Journal of Statistical Software 2012; 51(13): 1–35.
  * 151 Benkeser DC, Carone M, Gilbert PB. Improved estimation of the cumulative incidence of rare outcomes. Statistics in Medicine 2017\.  doi: 10.1002/sim.7337
  * 152 Benkeser DC, Hejazi NS. survtmle: Targeted Minimum Loss-Based Estimation for Survival Analysis in R. 2017.  doi: 10.5281/zenodo.835868
  * 153 Benkeser D. drtmle: Doubly-Robust Nonparametric Estimation and Inference. 2017\.  doi: 10.5281/zenodo.844836
  * 154 Sofrygin O, van der Laan MJ, Neugebaueer R. stremr: Streamlined Estimation for Static, Dynamic and Stochastic Treatment Regimes in Longitudinal Data. R package 2017\.  doi: https://github.com/osofr/stremr
  * 155 Coyle JR, Hejazi NS. origami: A Generalized Framework for Cross-Validation in R. The Journal of Open Source Software 2018; 3(21).  doi: 10.21105/joss.00512
  * 156 Coyle JR, Hejazi NS, Phillips RV, van der Laan L, van der Laan MJ. hal9001: The scalable highly adaptive lasso. https://doi.org/10.5281/zenodo.3558313; 2022.  R package version 0.4.2
  * 157 Hejazi NS, Coyle JR, van der Laan MJ. hal9001: Scalable highly adaptive lasso regression in R. Journal of Open Source Software 2020\.  doi: 10.21105/joss.02526
  * 158 Coyle JR, Hejazi NS, Malenica I, Phillips RV, Sofrygin O. sl3: Modern Pipelines for Machine Learning and Super Learning. https://github.com/tlverse/sl3; 2021.  R package version 1.4.2
  * 159 Coyle JR. tmle3: The Extensible TMLE Framework. 2021.  doi: 10.5281/zenodo.4603358
  * 160 Malenica I, Coyle J, van der Laan MJ. tmle3mopttx: Targeted Learning and Variable Importance with Optimal Individualized Categorical Treatment. https://github.com/tlverse/tmle3mopttx; 2022.  R package version 1.0.0
  * 161 Hejazi NS, Coyle JR, van der Laan MJ. tmle3shift: Targeted Learning of the Causal Effects of Stochastic Interventions. https://github.com/tlverse/tmle3shift; 2021.  R package version 0.2.0
  * 162 Ju C, Gruber S, Laan v. dM. ctmle: Collaborative Targeted Maximum Likelihood Estimation. 2017.  doi: https://CRAN.R-project.org/package=ctmle
  * 163 Hejazi NS, van der Laan MJ, Benkeser D. haldensify: Highly adaptive lasso conditional density estimation in R. Journal of Open Source Software 2022; 7(77): 4522.  doi: 10.21105/JOSS.04522
  * 164 Ertefaie A, Hejazi NS, van der Laan MJ. Nonparametric inverse probability weighted estimators based on the highly adaptive lasso. Biometrics 2022\.  doi: 10.1111/biom.13719
  * 165 Hejazi NS, van der Laan MJ, Janes HE, Gilbert PB, Benkeser DC. Efficient nonparametric inference on the effects of stochastic interventions under two-phase sampling, with applications to vaccine efficacy trials. Biometrics 2020\.  doi: 10.1111/biom.13375
  * 166 Hejazi NS, Benkeser DC. txshift: Efficient estimation of the causal effects of stochastic interventions in R. Journal of Open Source Software 2020\.  doi: 10.21105/joss.02447
  * 167 Hejazi NS, Benkeser DC. txshift: Efficient Estimation of the Causal Effects of Stochastic Interventions. Journal of Open Source Software 2022\.  doi: 10.5281/zenodo.4070042
  * 168 Zivich P, Davidson-Pilon C, Diong J, Reger D, Badger TG. zEpid. https://github.com/pzivich/zepid; 2022.  Python package version 0.9.1
  * 169 Luque-Fernandez MA. ELTMLE: Stata module to provide Ensemble Learning Targeted Maximum Likelihood Estimation. Statistical Software Components 2021\.  doi: https://ideas.repec.org/c/boc/bocode/s458337.html
  * 170 Gruber S, Phillips RA, Lee H, Ho M, Concato J, van der Laan MJ. Targeted learning: Towards a future informed by real-world evidence. Arxiv 2022\.  doi: arXiv:2205.08643
  * 171 Sentinel Innovation Center . Sentinel Innovation Center Master Plan. 2021.  doi: https://www.sentinelinitiative.org/news-events/publications-presentations/innovation-center-ic-master-plan
  * 172 Gruber S, Lee H, Phillips R, Ho M, Laan v. dM. Developing a Targeted Learning-Based Statistical Analysis Plan. Statistics in Biopharmaceutical Research 2022: 1–8.  doi: https://doi.org/10.1080/19466315.2022.2116104

Table 2: TMLE methodological developments and their associated extensions Year | Development | First author | Related developments
---|---|---|---
2006 | Seminal paper (TMLE) | van der Laan MJ |
2009 | Small sample size | Moore KL | Gruber S, 2010
| Case-control studies | Rose S | Rose S, 2011
|  |  | Balzer L, 2015
|  |  | Balzer L, 2016
2010 | Collaborative TMLE | van der Laan MJ | Pirracchio R, 2018
| (c-TMLE) |  | Ju C, 2019
| Time to event data |  | Ju C, 2019
|  |  | Ju C, 2019
|  |  | Schnitzer ME, 2020
| Longitudinal data (LTMLE) | van der Laan MJ | van der Laan, 2012
|  |  | Schomaker M, 2019
2012 | Sequential randomised trials | Chaffee PH |
| Natural direct effect | Zheng W | Lendle SD, 2013
|  |  | Rudolph KE, 2018
| Non-independence | van der Laan MJ | van der Laan MJ, 2014
|  |  | Schnitzer M, 2016
|  |  | Sofrygin O, 2017
|  |  | Balzer L, 2019
|  |  | Balzer L, 2021
2013 | Meta-analysis / Safety outcomes | Gruber S | Liu Y, 2022
2014 | Pooled TMLE | Petersen M | Zheng W, 2016
|  |  | Ferreira Guerra S, 2020
| Interval-censored TMLE | Sapp S |
| Genetics | Wang H | Benkeser D, 2019
|  |  | Yang G, 2022
2015 | PS | Lendle SD |
| Cross-validated TMLE | van der Laan MJ |
| cv-TMLE |  |
2016 | One-step TMLE | van der Laan MJ | Cai W, 2020
|  |  | Zhu J, 2020
| TMLE for rare outcomes | Balzer L | Benkeser D, 2018
| TMLE for ordinal outcomes | Díaz I |
2017 | TMLE with missing outcome data | Díaz I |
| Robust TMLE | Rudolph KE |
| Targeted sequential inference of an optimal treatment rule | Chambaz A |
2018 | Projected TMLE | Zheng W |
2019 | TMLE for cluster-level exposure | Balzer LB |
| Long-format TMLE | Sofrygin O |
2020 |  | Highly-Adaptive least absolute shrinkage and selection
---
operator (LASSO) Targeted Minimum Loss Estimator
(HAL-TMLE)
Cai W |
2022 | Threshold response function | van der Laan MJ |

See pages 1 of Table_of_disciplines.pdf

See pages 2- of Table_of_disciplines.pdf