tcb@breakable

# End-To-End Causal Effect Estimation
from Unstructured Natural Language Data

Nikita Dhawan
University of Toronto, Vector Institute
<EMAIL_ADDRESS>
&Leonardo Cotta
Vector Institute
<EMAIL_ADDRESS>
Karen Ullrich
Meta AI
<EMAIL_ADDRESS>
&Rahul G. Krishnan
University of Toronto, Vector Institute
<EMAIL_ADDRESS>
&Chris J. Maddison
University of Toronto, Vector Institute
<EMAIL_ADDRESS>
Work done during internship at Meta AI.

###### Abstract

Knowing the effect of an intervention is critical for human decision-making,
but current approaches for causal effect estimation rely on manual data
collection and structuring, regardless of the causal assumptions. This
increases both the cost and time-to-completion for studies. We show how large,
diverse observational text data can be mined with large language models (LLMs)
to produce inexpensive causal effect estimates under appropriate causal
assumptions. We introduce _NATURAL_ , a novel family of causal effect
estimators built with LLMs that operate over datasets of unstructured text.
Our estimators use LLM conditional distributions (over variables of interest,
given the text data) to assist in the computation of classical estimators of
causal effect. We overcome a number of technical challenges to realize this
idea, such as automating data curation and using LLMs to impute missing
information. We prepare six (two synthetic and four real) observational
datasets, paired with corresponding ground truth in the form of randomized
trials, which we used to systematically evaluate each step of our pipeline.
NATURAL estimators demonstrate remarkable performance, yielding causal effect
estimates that fall within 3 percentage points of their ground truth
counterparts, including on real-world Phase 3/4 clinical trials. Our results
suggest that unstructured text data is a rich source of causal effect
information, and NATURAL is a first step towards an automated pipeline to tap
this resource.

##  1 Introduction

Estimating the causal effects of interventions is time consuming and costly,
but the resulting outcomes are precious. Health agencies around the world
often require randomized controlled trial (RCT) data to approve medical
interventions. Clinical trials are key contributors to large R&D costs for
drug developers [27]. Natural experiments are another source of rich
interventional data, but they may not always exist or have enough data
relevant to a given causal hypothesis [11].

When treatment randomization is infeasible, observational data can be used to
identify average treatment effects (ATEs) [45], under common assumptions,
e.g., no unobserved confounding. Such data is abundant but even when the
necessary assumptions are satisfied, it must be _structured_ (_i.e._ , the
outcomes, treatments, and relevant covariates must be defined, recorded, and
tabulated) before it becomes amenable to computational analyses.

Yet, unstructured observational data presents unique opportunities for
cheaper, more accessible, and potentially even better [29] effect estimation.
For example, thousands of people living with diabetes choose to share their
experiences with _treatments_ on online patient forums. Some of their posts
contain rich descriptions of daily lives, the drugs they have been prescribed,
the treatment responses and side effects, as well as pre-treatment information
like age and sex. Their posts contain their lived experiences including
evidence of an _outcome_ in an observational experiment, albeit in an
unstructured form. Other potential sources of rich unstructured, observational
data include newspaper classifieds, police reports, social media, and clinical
reports. Despite being collected for a myriad of purposes, researchers have
often turned to such data to test hypotheses since:  (i) unstructured data
does not require restrictive data collection designs, _e.g._ , measurement
choice, and can admit many different post-hoc analyses;  (ii) the reported
outcomes may reflect what matters to subjects better than standard outcome
measures;  (iii) value may be recouped from outcomes that would otherwise be
lost;  (iv) there may be _more_ unstructured data available on underserved or
marginalized populations.  Figure 1 contrasts our setting with previous works
using randomized or structured observational data.

This work asks a simple question: _How can we use large language models to
automate treatment effect estimation using freely available text data?_ We
introduce _NATURAL_ , a family of text-conditioned estimators that addresses
this by performing _NATural language analysis to Understand ReAL effects_.

At a high level, the steps required to compute NATURAL estimators are as
follows. Given an observational study design and a dataset of natural language
reports, filter for reports that are likely to conform to the experimental
design. Then, using a large language model (LLM), extract the conditional
distribution of structured variables of interest (outcome, treatment,
covariates) given the report. Finally, use the conditionals to compute
estimators of the ATE, using classical strategies such as inverse propensity
score weighting and outcome imputation.

NATURAL is a data-driven pipeline. It leverages and relies on the LLM in a
manner that mimics the learning task it was trained for: providing parametric
approximations to conditional distributions. As in all observational studies,
the validity of NATURAL also depends on prior causal knowledge about the task.
Expert knowledge is required to define appropriate covariates and confirm that
they satisfy the necessary assumptions for effect estimation. However, we
anticipate that NATURAL estimators could be developed under other structural
assumptions (e.g. instrumental variables) as well.

Figure 1: When compared to experimental and other observational studies,
NATURAL has lower costs and provides greater diversity in cohort selection,
for causal effect estimation.

The core contributions of our work are:

  * •

We derived NATURAL ATE estimators based on classical estimators of the ATE,
like inverse propensity score weighting and outcome imputation. NATURAL
estimators operate on entirely unstructured data under two novel data-access
assumptions.

  * •

We implemented NATURAL estimators using an LLM-based pipeline.

  * •

We developed six observational datasets to systematically evaluate parts of
this pipeline: two synthetic datasets constructed using marketing data, and
four clinical datasets curated from public (pre-December 2022) migraine and
diabetes subreddits from the Pushshift collection [6].

  * •

For each dataset, we treated the ATE from a corresponding real-world
completely randomized experiment (CRE) as ground truth. Remarkably, our
predicted ATEs all fell within 3 percentage points of the ground truth ATEs, a
potential cost savings of many millions of dollars.

###  1.1 Related Work

Leveraging natural language data [39] to support causal claims is pervasive in
applied research [40, 12]. Our work falls under the broad umbrella of
accelerating the identification of real-world evidence (RWE) [37]. For
instance, in the context of healthcare, RWE supports not only drug
repurposing, but also post-market safety evaluations — its most common
application. NATURAL expands the boundaries of how quickly one can obtain and
validate such real-world evidence from observational data [38].

The use of natural language data in causal inference comes in different
flavors: i) using text to measure confounders [21], ii) using text to measure
causal effect outcomes [13], or iii) producing interpretable causal features
from text [13, 5], e.g., what words are more likely to explain the cause of an
event. NATURAL distinguishes itself from these lines of research in two ways:
i) NATURAL does not require any curated task-specific training data (it is
zero-shot), and ii) NATURAL is not interested in how the text itself, i.e.,
its words, relate to the causal problem —that is, we are only leveraging the
model’s ability to predict the distribution of a specified variable
conditional on the input text. We highlight that our work lies distinct from
research at the intersection of text and causality that has studied the
ability of language models to infer _latent_ variables (that are implied but
not explicitly identified in text data) [33, 12]. Rather, we require the
precise specification of covariates to condition on – we view this as being
crucial to creating a more direct way for an end user to verify the validity
of information extracted with our approach.

Prior works have also leveraged LLMs in a black-box fashion for causal tasks
by querying the model for causal statements. In the context of causal
discovery, users directly ask for the existence of cause-and-effect
relationships, e.g.,“Does changing the age of an abalone causes a change in
its length?” [23, 30, 3, 4, 41, 19, 5]. Due to the large amount of training
data, it is possible that the model learns to apply a causal model described
in the training data and answer causal questions with it [32, 44]. The issue
with this approach is i) the user is limited to the causal models observed in
training, ii) the user is not aware of _which_ causal model they are using,
and iii) the queries tend to present high prompt sensitivity [26]. Finally, we
note that a recent work created a benchmark and showed how LLMs struggle to
distinguish pairwise correlation from causation [18], while another shows that
checking causal relationships in a pairwise manner can lead to invalid causal
graphs [42].

##  2 Preliminaries

We are interested in estimating the causal effect of a treatment relative to
either another treatment or no treatment in a population of interest. More
precisely, we consider treatments $t\in\\{0,1\\}$ and the corresponding
potential outcomes $Y(1)$ and $Y(0)$ under each treatment. We wish to compute
the quantity $\tau:=\mathbb{E}[Y(1)-Y(0)]$, often referred to as Average
Treatment Effect (ATE). Sometimes, $Y(0)$ may correspond to no treatment
(control). Throughout this work, we assume binary treatments and outcomes in
the Neyman-Rubin causal model. We provide a full list of notation in appendix
A.

A Completely Randomized Experiment (CRE) with $n$ participants requires no
prior causal knowledge. In a CRE, the treatment assignment vector
$(\tilde{T}_{i})_{i=1}^{n}$ is a random permutation of $n_{1}$ ones and
$n-n_{1}$ zeros sampled independently of the outcomes. In this case, the
difference-in-means
$\tfrac{1}{n_{1}}\sum_{i=1}^{n}\tilde{T}_{i}Y_{i}(1)-\tfrac{1}{n-n_{1}}\sum_{i=1}^{n}(1-\tilde{T}_{i})Y_{i}(0)$
provides us with an unbiased estimate of $\tau$.

Despite the indisputable necessity of CREs in high-stakes settings, it is
often expensive and/or infeasible to have complete control over the treatment
assignment. Instead, _observational_ data is more readily available.
Observational data often contains spurious correlations between the observed
treatment $T$ and the observed outcome $Y=TY(1)+(1-T)Y(0)$ through a common
cause (confounder). Typically, this confounding is formalized as a variable
$X$, which we assume to be discrete throughout this work, representing
covariates associated with each individual. Given i.i.d. samples
$\\{(X_{i},T_{i},Y_{i})\\}_{i=1}^{n}$ from the target population, standard
causal inference techniques can correct for confounding bias and provide
consistent estimates of $\tau$ under Assumptions 1 and 2:

######  Assumption 1 (Strong Ignorability.)

The potential outcomes are independent of treatment assignments conditional on
covariates, i.e., $(Y(0),Y(1))\perp\\!\\!\\!\perp T|X.$

######  Assumption 2 (Positivity.)

For every treatment $t$ and covariate set $x$, $0<P(T=t\mid X=x)<1.$

Following are two classical estimators of the ATE $\tau$ from observational
data, each of which rely on $X$ satisfying Assumptions 1 and 2. We refer the
reader to Ding, [10] for further details.

#### Inverse Propensity Score Weighting (IPW).

The propensity score is the conditional probability of receiving a treatment
given the observed features, i.e., $e(x)=P(T=1|X=x)$. The IPW estimator is
given by

$\hat{\tau}_{\text{IPW}}=\frac{1}{n}\sum_{i=1}^{n}\frac{T_{i}Y_{i}}{\hat{e}(X_{i})}-\frac{(1-T_{i})Y_{i}}{1-\hat{e}(X_{i})},$
(1)

where $\hat{e}(x)$ is an approximation of $P(T=1\mid X=x)$. When $\hat{e}(x)$
is the true propensity score, $\hat{\tau}_{\text{IPW}}$ is an unbiased
estimator of $\tau$. When $\hat{e}(x)$ is estimated as empirical probability,
$\hat{\tau}_{\text{IPW}}$ is consistent.

#### Outcome Imputation (OI).

Outcome Imputation learns a model to impute outcomes from features and
treatment and then marginalizes away the features to estimate $\tau$ with

$\hat{\tau}_{\text{OI}}=\frac{1}{n}\sum_{i=1}^{n}\hat{\tau}(X_{i},1)-\hat{\tau}(X_{i},0),$
(2)

where $\hat{\tau}(x,t)$ approximates $P(Y=1\mid X=x,T=t)$. Note that if
$\hat{\tau}(x,t)$ is an unbiased estimation of this quantity,
$\hat{\tau}_{\text{OI}}$ is an unbiased estimator of $\tau$.

##  3 NATURAL estimators of the ATE

Both CRE and observational studies require direct access to tabulated data
$(X_{i}$, $T_{i}$, $Y_{i})$ for every individual $i$. Our NATURAL estimators
on the other hand estimate the ATE from observational, unstructured natural
language data in the form of language reports $R_{i}$. In addition to
Assumptions 1 and 2, NATURAL estimators require the following assumptions to
guarantee their consistency.

######  Assumption 3 (Natural language report data.)

The target population is described by an observational data-generating process
$P(X,T,Y,R)$ of data $(X,T,Y)$, which satisfies Assumptions 1 and 2 and is
jointly distributed with a random natural language string $R$, called a
_report_. We assume access to an i.i.d. sample of reports
$\\{R_{i}\\}_{i=1}^{n}$ from the marginal of this process.

######  Assumption 4 (Access to the true observational conditional over
$(X,T,Y)$.)

We can either  (i) compute the conditional $P(X=x,T=t,Y=y|R=r)$ of the true
data-generating process, or  (ii) we can sample from $P(X=x|R=r)$ and compute
$P(T=t,Y=y|R=r,X=x)$.

Intuitively, these assumptions give NATURAL indirect access to $(X,T,Y)$
through $R$. They can be weak or strong, depending on the definition of the
reports $R$. On the one hand, if reports are copies of the observational data,
_i.e._ , $R=(X,T,Y)$, then Assumption 4 is trivial to satisfy. On the other
hand, if reports are all the constant, empty string, $R=\epsilon$, then
Assumption 4 guarantees that we have full access to the _true_ observational
joint density function over $(X,T,Y)$, which is a strong assumption. We
consider how we might satisfy these assumptions in practice in the next
section. Here, we assume that they hold and develop a series of consistent
estimators of the ATE.

NATURAL Full. Given $\\{R_{i}\\}_{i=1}^{n}$ and $P(X=x,T=t,Y=y|R=r)$, we can
construct an idealized version of NATURAL. Let us start by noting that the law
of total expectation gives us

$\tau=\mathbb{E}_{X,T,Y}\left[\frac{TY}{e(X)}-\frac{(1-T)Y}{1-e(X)}\right]=\mathbb{E}_{R}\left[\mathbb{E}_{X,T,Y|R}\left[\frac{TY}{e(X)}-\frac{(1-T)Y}{1-e(X)}\right]\right].$
(3)

A Monte Carlo estimate over reports is given by

$\hat{\tau}_{\text{N-Full}}=\frac{1}{n}\sum_{i=1}^{n}\sum_{x,t,y}P(X=x,T=t,Y=y|R_{i})\left[\frac{ty}{\hat{e}_{\text{N-Full}}(x)}-\frac{(1-t)y}{1-\hat{e}_{\text{N-Full}}(x)}\right],$
(4)

which further approximates $\hat{e}_{\text{N-Full}}(x)$ from the given
conditional. We used eq. 7 below. We note that $\hat{\tau}_{\text{N-Full}}$
above is derived using the form of IPW, but the same estimator can be derived
from OI, as shown in appendix B.

The estimator $\hat{\tau}_{\text{N-Full}}$ above relies on enumerating all
possible values of $(X,T,Y)$, making it computationally expensive for high-
dimensional $X$. Below, we present two hybrid versions of our method which
combine sampling of some variables and computation of conditional
probabilities of others.

NATURAL IPW. To construct our hybrid estimator, we augment the data
$\\{R_{i}\\}_{i=1}^{n}$ by sampling from $P(X|R_{i})$ independently for each
report $R_{i}$. This gives us a dataset $\\{(R_{i},X_{i})\\}_{i=1}^{n}$ drawn
i.i.d. from $P(X,R)$ by Assumption 4. Then, our hybrid estimator is derived
from the form of IPW as follows:

$\displaystyle\tau$
$\displaystyle=\mathbb{E}_{R,X}\left[\mathbb{E}_{T,Y|R,X}\left[\frac{TY}{e(X)}-\frac{(1-T)Y}{1-e(X)}\right]\right],$
(5) $\displaystyle\hat{\tau}_{\text{N-IPW}}$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\sum_{(t,y)\in\mathcal{T}\times\mathcal{Y}}P(T=t,Y=y|R_{i},X_{i})\left[\frac{ty}{\hat{e}_{\text{N-IPW}}(X_{i})}-\frac{(1-t)y}{1-\hat{e}_{\text{N-IPW}}(X_{i})}\right],$
(6)

where $\hat{e}_{\text{N-IPW}}(x)$ is consistently estimated in the following
manner:

$\hat{e}_{\text{N-IPW}}(x)=\frac{\sum_{i=1}^{n}P(T=1|R_{i},X_{i})\mathbb{I}\\!\left(X_{i}=x\right)}{\sum_{i=1}^{n}\mathbb{I}\\!\left(X_{i}=x\right)}\overset{\text{a.s.}}{\to}\frac{\mathbb{E}_{R,X}\big{[}P(T=1|R,X)\mathbb{I}\\!\left(X=x\right)\big{]}}{\mathbb{E}_{R,X}\big{[}\mathbb{I}\\!\left(X=x\right)\big{]}}=e(x).$
(7)

NATURAL OI. Similarly inspired by the OI estimator in equation 2, we have for
$t\in\\{0,1\\}$,

$P(Y=1\mid
T=t,X=x)=\frac{\mathbb{E}_{R,X,T}\big{[}P(Y=1|R,X,T)\mathbb{I}\\!\left(X=x,T=t\right)\big{]}}{\mathbb{E}_{R,X,T}\big{[}\mathbb{I}\\!\left(X=x,T=t\right)\big{]}}.$
(8)

Thus, for our hybrid OI estimator, we augment the data $\\{R_{i}\\}_{i=1}^{n}$
by sampling from $P(X,T|R_{i})$ independently for each report $R_{i}$. This
gives us a dataset $\\{(R_{i},X_{i},T_{i})\\}_{i=1}^{n}$ drawn i.i.d. from
$P(R,X,T)$ by Assumption 4. Then, our consistent outcome predictor is given by

$\hat{\tau}_{\text{N-OI}}(x,t)=\frac{\sum_{i=1}^{n}P(Y=1|R_{i},X_{i},T_{i})\mathbb{I}\\!\left(X_{i}=x,T_{i}=t\right)}{\sum_{i=1}^{n}\mathbb{I}\\!\left(X_{i}=x,T_{i}=t\right)},$
(9)

and the final estimator is given by:

$\hat{\tau}_{\text{N-OI}}=\frac{1}{n}\sum_{i=1}^{n}\hat{\tau}_{\text{N-OI}}(X_{i},1)-\hat{\tau}_{\text{N-OI}}(X_{i},0).$
(10)

NATURAL Monte Carlo. Further in the direction of sampling more variables, we
can obtain samples $(X_{i},T_{i},Y_{i})$ from $P(X,T,Y|R_{i})$ and compute a
Monte Carlo estimate, $\hat{\tau}_{\text{N-MC}}$. The set of samples
$\\{(X_{i},T_{i},Y_{i})\\}_{i=1}^{n}$ constitute a tabular dataset which can
be plugged into a standard ATE estimator like IPW or OI, as described in
section 2. We refer to these sample-only estimators as N-MC IPW and N-MC OI,
respectively.

##  4 Implementing NATURAL estimators with Large Language Models

LLMs are trained on vast datasets of real-world data, _e.g._ , [2], which
likely contain records of data generated by processes that are consistent with
Assumption 3. Because LLMs can learn well-calibrated conditionals [20], our
hypothesis is that LLMs can be prompted to approximate the conditionals
required by Assumption 4 for real-world causal effect questions of interest.
Our LLM implementation of NATURAL estimators is built on this hypothesis to
try to satisfy Assumptions 3 and 4 (Assumptions 1 and 2 must be guaranteed by
a domain expert). We defer exact prompts to appendix D, a full worked example
to appendix C, and a discussion of the limitations of our approach to the next
section. Figure 2 summarizes our pipeline.

Figure 2: Our pipeline leverages LLMs to curate data that can be plugged into
natural language conditioned estimators for average treatment effects.

Filtering to match Assumption 3. For our real-data clinical settings, our
first goal is to produce a dataset of i.i.d. reports $R_{i}$ that are very
likely to be jointly distributed with the random variables
$(X_{i},T_{i},Y_{i})$ of a specific observational study of interest. Given a
study of interest and a dataset of real-world reports that are potentially
relevant to the study, we pass it through a sequence of filters with
increasing detail and strictness:

  1. (i)

Initial filter. Inspired by other work with social media data [1, 35], we
first use deterministic rules to filter out uninformative reports: posts that
were removed, are too short, have "bot" in the author’s name, have no mention
of any keyword related to the study, etc.

  2. (ii)

Filter by relevance. We prompt an LLM to determine whether each report
contains information that would make it relevant to the study. We remove
reports that are deemed irrelevant.

  3. (iii)

Filter by treatment-outcome. We ensure that each report pertains specifically
to the treatments and outcomes of interest. We do so by prompting an LLM to
extract only treatment and outcome information, and retaining only the posts
that are deemed to both mention one of the treatments in question and also
contain outcome information.

  4. (iv)

Filter known covariates by inclusion criteria. We are sometimes interested in
ATEs over populations defined by constraints on pre-treatment covariates
$X_{i}$ known as _inclusion criteria_. In such cases, we included a filter to
enforce inclusion criteria. Managing inclusion criteria is complicated by the
fact that many reports $R_{i}$ contain no or partial information about
covariates that are required to verify inclusion. So, in this filtering step,
our goal was to ensure that the final set of reports have non-zero probability
of matching the inclusion criteria. We begin by prompting an LLM to extract
the full set of covariates $X_{i}$, following constraints on the possible
values each attribute can take, but we allow the LLM to extract Unknown if it
is impossible for the LLM to determine the value of a covariate. We then
remove reports, if any of the non-Unknown covariates are determined to fail
their inclusion criteria. We found the JSON-mode made available for generation
by certain LLM APIs, like GPT-4, to suffice for this task; however more
involved strategies for constrained generation are also possible [43, 47].

Sampling from and computing conditional probabilities to match Assumption 4.
Given a set of reports $\\{R_{i}\\}_{i=1}^{n}$ that pass the filtering stage
above, our next steps use LLMs to extract the samples and conditionals
$P_{\text{LLM}}(X,T,Y\mid R)$, required to compute NATURAL estimators. For
each $R_{i}$, we:

  1. (v)

Extract covariates, both known and unknown. We run a final covariate
extraction by prompting an LLM to determine the full set of covariates $X_{i}$
from the report $R_{i}$, subject to the constraint that $X_{i}$ satisfies the
inclusion criteria. In contrast to (iv), we ask the LLM to guess the values of
Unknown covariates. We verified that this second extraction agreed exactly
with the first extraction (iv) on the known covariates (_i.e._ , the ones that
were not extracted as Unknown in the first extraction). We contrast the
empirical distributions of these known and unknown/guessed covariates for our
experiments in section F.1.

  2. (vi)

Infer conditionals. Given $\\{R_{i},X_{i}\\}_{i=1}^{n}$ from the previous
steps, we compute the probabilities $P_{\text{LLM}}(T=t,Y=y|R_{i},X_{i})$ by
prompting an LLM that makes log-probabilities accessible, like LLAMA2.
Specifically, we ask an LLM to answer questions about $T,Y$ given access to
$R_{i},X_{i}$, and we score every possible answer $T=t,Y=y$ using the LLM log-
probabilities. We exponentiate and renormalize these scores across the space
of possible realizations to obtain a valid probability distribution.

To manage inclusion criteria, our LLM pipeline requires three additional
technical conditions, which we discuss in more detail in appendix G:  (i)
inclusion criteria define a box, i.e., a separate criterion is specified for
each covariate dimension,  (ii) the event that the set of Unknown in (iv)
satisfy their inclusion criteria is conditionally independent of the report
and the known covariates given the event that the knowns satisfy their
criteria, and  (iii) the set of known and unknown covariates is conditionally
independent of the treatment and outcome given the report and the covariate
values.  We used these to avoid an expensive estimation or a data-wasteful
rejection sampling step, but these assumptions are not required in principle
by NATURAL estimators.

Nevertheless, while our empirical results are remarkably consistent with the
correctness of our pipeline, we cannot formally guarantee that it satisfies
Assumptions 3 and 4. The final outcome of this pipeline is a dataset
$\\{R_{i},X_{i}\\}_{i=1}^{n}$ and a set of conditionals
$P(T=t,Y=y|R_{i},X_{i})$ that can be plugged into the hybrid NATURAL
estimators in section 3 to predict ATEs. Therefore, we see this as a first
implementation of NATURAL estimators, which we anticipate can be improved.

##  5 Limitations and Broader Impact

In addition to the limitations that NATURAL shares with every observational
study, i.e., the validity of the practitioner’s causal assumptions, it comes
with an _extra dependence on how well one can approximate the desired
conditional distributions_. While more and more capable LLMs are being
continually developed, the extent to which they satisfy NATURAL’s assumptions
is nearly impossible to formally test. Indeed, while pretraining tends to
produce calibrated LLM predictions [20], post-training techniques can
compromise calibration [31]. Therefore, we emphasize that NATURAL was _not_
developed to recommend therapeutics directly to end-users or to directly
inform high-stakes public policies. Instead, we envision NATURAL as a powerful
tool to help us approximate ATEs at scale and prioritize confirmatory CREs. We
strongly recommend that all predictions made by NATURAL estimators be
validated experimentally before being used to inform high-stakes decision-
making. Apart from its dependence on LLM capabilities, NATURAL is also limited
by the nature of observational, unstructured natural language data:

  * •

_Network Interference._ In practice, acquiring i.i.d. reports can be
challenging. For instance, social network users might talk to each other and
influence their treatment choices. This is a well-known issue in causal
inference and statistical sciences in general. Existing solutions rely on a
known network structure to sample individuals or correct for their neighbors’
treatments [9, 24, 15].

  * •

_Outcome Measurement._ Since NATURAL deals with self-reports, subjects need to
be able to report the outcomes of interest. For example, this cannot be
applied if the outcome is measured with an expensive, inaccessible test.
Therefore, the study design implemented with NATURAL must account for the
accessibility of endpoints to users.

  * •

_Selection Bias._ Results might be biased towards individuals’ choice of
reporting an outcome given their experience with the treatment. Luckily,
outcome missingness is a widely studied problem in causality research, see
e.g., how to test [8] or how to mitigate [28] it. Note, however, that
solutions will often accumulate assumptions on top of NATURAL and should
always be critically evaluated by practitioners. Finally, apart from
individuals’ choice of reporting, selection bias might also arise from which
individuals participate in online forums, i.e., our framework is only capable
of estimating _local_ ATEs —external validity is not guaranteed a priori.

##  6 Empirical Evaluation

Figure 3: For Hillstrom (left) and Retail Hero (right), the KL divergence
between estimated joint and propensity distributions and their true
counterparts reduces with increasing number of posts (top), as does the RMSE
between the NATURAL Full estimate and true ATE (bottom).

Evaluating an end-to-end pipeline for causal inference from unstructured real-
world text data to ATEs presents challenges regarding access to data, ground
truth ATEs and insightful intermediate metrics. We used two synthetic datasets
where we augmented randomized data to mimic real-world observations, while
continuing to have access to ground truth evaluation. In addition, we study
four real datasets, curated from publicly available Reddit posts from the
PushShift dataset, as described in section 4. These six datasets allowed us to
systematically evaluate NATURAL.

Table 1: The NATURAL IPW ATE outperforms other versions of the method as well
as trained baselines on synthetic datasets, as measured by RMSE.

| Hillstrom | Retail Hero
---|---|---
|  ATE $(\%)$ | RMSE |  ATE $(\%)$ | RMSE
Uncorrected | $1.86\pm 0.67$ | $4.28$ | $0.26\pm 0.30$ | $3.08$
N-Full | $4.26\pm 0.86$ | $2.02$ | $1.86\pm 1.38$ | $2.08$
N-MC OI | $6.17\pm 1.61$ | $1.61$ | $4.94\pm 2.17$ | $2.70$
N-MC IPW | $4.81\pm 0.80$ | $1.51$ | $1.85\pm 2.01$ | $2.49$
N-OI | $4.58\pm 0.61$ | $1.62$ | $2.99\pm 1.43$ | $1.72$
N-IPW | $\mathbf{5.23\pm 1.00}$ | $\mathbf{1.32}$ | $\mathbf{3.83\pm 1.29}$ | $\mathbf{1.39}$
Bag-of-Words | $7.57\pm 1.37$ | $2.23$ | $2.61\pm 2.08$ | $2.42$
Sentence Encoder | $0.00\pm 0.00$ | $6.09$ | $1.97\pm 1.62$ | $2.10$
IPW (Structured) | $6.38\pm 0.26$ | $0.39$ | $3.09\pm 0.19$ | $0.30$
Ground Truth |  $\mathbf{6.09}$ [17] | - |  $\mathbf{3.32}$ [46] | -

#### Synthetic Datasets.

Causal effect estimation is typically evaluated using synthetic datasets with
one or more relationships between the observed covariates, treatment and
outcome being contrived. We instead synthesized unstructured observational
text data from real randomized tabular datasets, using an LLM. Specifically,
we (i) introduced confounding bias by sampling datapoints according to an
artificial propensity score, (ii) randomly dropped covariates, (iii) described
covariates, treatment and outcome in shuffled orderings, (iv) simulated
realism by sampling a persona from the the Big Five personality traits [25]
for each datapoint and finally, (v) prompted the LLM to generate a realistic
report describing the provided information in the style of someone with the
given traits (see appendix D for the full prompt). We used two standard,
publicly available randomized datasets: Hillstrom [17] and Retail Hero [46],
and plan to open-source scripts to generate our data. Step (i) above is in a
similar vein as Keith et al., [22], in that our subsampling strategy does not
modify the marginal distribution over covariates and the ATE remains
identifiable from observational data.

#### Real-world Datasets.

To study how our framework may be deployed to test hypotheses using real data
from online forums; we considered two medical conditions for which there exist
abundant Reddit posts in the PushShift collection [6], with individuals’
personal experiences: the effect of diabetes medications (e.g. Semaglutide) on
weight loss and the tolerability of migraine treatments. For each condition,
we picked two clinical trials which performed head-to-head comparisons of two
treatments that we expected to find references to in relevant subreddits.
Moreover, to mitigate selection bias we selected pairs of similar treatments,
e.g., comparable availability, where we believe the probability of a user
reporting their experience is approximately equal in both. As we will discuss,
our results suggest external validity as well, meaning that the probability of
a user reporting their experience with the treatments seems to be
(approximately) equal to the prior probability of a user reporting any
experience. We limited our data collection to posts that were written before
December 2022 and made publicly available in the PushShift archives. We
curated four datasets for comparison between different treatments, each of
which has a ground truth RCT: Semaglutide vs. Tirzepatide [16] and Semaglutide
vs. Liraglutide [7] for their effect on weight loss and Erenumab vs.
Topiramate [34] and OnabotulinumtoxinA vs. Topiramate [36] for their
tolerability. We used the first of these to validate implementation choices
NATURAL (like filtering, imputations, prompt specifications) and the other
three as held-out test settings, see appendix C. We include further details
for all our datasets in appendix E, including the definitions of covariates
and outcomes.

Next, we investigate several questions about the performance of NATURAL
empirically. We used GPT-4 Turbo for sampling and LLAMA2-70B for computing
conditional probabilities.

#### How well does NATURAL estimate observational distributions from self-
reported data?

Our synthetic datasets give us access to the true joint distributions
$P(X,T,Y)$ and true propensity scores $P(T=1|X)$. The top row of fig. 3 shows
the KL divergence between these distributions and those estimated by NATURAL
Full, for Hillstrom (left) and Retail Hero (right). We find that these KL
divergences decrease steadily as the number of reports used in the estimation
increases. The bottom row shows corresponding root-mean-squared error (RMSE)
between NATURAL and the true ATE. This corroborates the insight that as the
joint distribution and propensity scores are estimated more accurately, the
predicted ATE gets closer to its true value. In particular, we observe a clear
correlation between the quality of estimated propensity scores and estimated
ATEs.

| Tuned | Held-out
---|---|---
|  Semaglutide vs. Tirzepatide (% weight loss $\geq$ 5%) |  Semaglutide vs. Liraglutide (% weight loss $\geq$ 10%) |  Erenumab vs. Topiramate (% discontinued due to AE) |  OnabotulinumtoxinA vs. Topiramate (% discontinued due to AE)
|  ATE $(\%)$ | RMSE |  ATE $(\%)$ | RMSE |  ATE $(\%)$ | RMSE |  ATE $(\%)$ | RMSE
Uncorrected | $-33.56\pm 0.77$ | $43.67$ | $-83.57\pm 0.43$ | $68.87$ | $29.07\pm 0.48$ | $2.87$ | $21.55\pm 1.22$ | $19.49$
N-MC OI | $5.43\pm 1.01$ | $4.79$ | $-7.71\pm 0.91$ | $7.05$ | $23.91\pm 1.63$ | $4.68$ | $46.21\pm 1.94$ | $5.55$
N-MC IPW | $5.23\pm 0.93$ | $4.97$ | $-7.43\pm 0.93$ | $7.33$ | $25.29\pm 1.72$ | $3.47$ | $46.23\pm 1.93$ | $5.57$
N-OI | $4.36\pm 2.05$ | $6.09$ | $\mathbf{-15.90\pm 1.14}$ | $\mathbf{1.65}$ | $31.21\pm 1.68$ | $3.36$ | $44.91\pm 1.46$ | $4.17$
N-IPW | $\mathbf{8.83\pm 0.36}$ | $\mathbf{1.33}$ | $-12.21\pm 1.09$ | $2.72$ | $\mathbf{27.90\pm 0.99}$ | $\mathbf{1.06}$ | $\mathbf{42.60\pm 2.02}$ | $\mathbf{2.58}$
Ground Truth |  $\mathbf{10.11}$ [NCT03987919, 16] |  $\mathbf{-14.7}$ [NCT03191396, 7] |  $\mathbf{28.3}$ [NCT03828539, 34] |  $\mathbf{41.00}$ [NCT02191579, 36]

Table 2: Using real data, best performing NATURAL estimators fall within 3
percentage points of their corresponding ground truth clinical trial ATEs.
Possible ATE values lie between $-100$ and $100$.

#### How do NATURAL methods compare to one another and to trained baselines?

We present our estimated ATE and its RMSE on the synthetic datasets in table
1. Further, we evaluate two trained baselines, which use a Bag-of-Words model
and a sentence encoder respectively, to train representations of text data
with their labels. Here, for each attribute in the set of covariates,
treatments, and outcomes, we train a MLP model with 5-fold cross validation to
predict that attribute. We then use these predicted attributes as a tabular
dataset of samples that can be plugged into any causal inference estimator. We
find that our methods are competitive with or outperform these baselines,
despite not being trained with any labels. In particular, the sentence encoder
baseline collapsed to an ATE of zero for Hillstrom, having learned the
constant predictor for outcomes.

Table 2 compares NATURAL methods to estimate the ATEs in real-world clinical
settings using self-reported data from the PushShift collection of Reddit
posts. Remarkably, our predicted ATEs (a) depict the same _direction of
effect_ , and (b) fall _within 3 percentage points_ of their corresponding
ground truth clinical trial ATEs. For both synthetic and real data
experiments, NATURAL IPW outperforms other versions across datasets, except
for the Semaglutide vs. Liraglutide setting, where NATURAL OI performed the
best. Both N-MC versions perform similarly on all datasets.

This result is significant. Clinical trials can take on the order of years and
costs in the tens to hundreds of millions of dollars. Going from the raw
language observational data to ATE in our framework takes on the order of days
and costs at most a few hundred dollars of compute. For problems in medicine,
economics, sociology, and political science where randomization is infeasible
or expensive, NATURAL provides a tractable way to leverage observational data
to rank potential experiments prior to conducting them.

Figure 4: Ablation study on Semaglutide vs. Tirzepatide, to tease apart the
effect of data filtering and imputation (left) as well as LLM scale for
conditionals (right) on NATURAL performance.

#### How do different choices in the NATURAL pipeline effect ATE prediction?

We assess the impact of key choices in our pipeline described in section 4, by
ablating them one-by-one. We investigated and selected these choices on the
Semaglutide vs. Tirzepatide experiment. Figure 4 (left) compares the RMSE of
predicted ATEs when data is not filtered according to inclusion criteria and
LLM imputations are replaced with samples from an uniform distribution. It
shows that both inclusion-based filtering and imputations from a pretrained
LLM are crucial for the performance of NATURAL. We also compared performance
of our method when the conditional probabilities in eq. 6 are evaluated using
models of different scales in fig. 4 (right), and found that performance
improves at larger scales and with greater quantity of data.

Figure 5: NATURAL propensity scores balance the Semaglutide vs. Tirzepatide
covariates better than uniform scores.

#### How well do different estimates of propensity score balance covariates?

A property of accurate propensity score estimates is that they balance
covariates across treatment cohorts (see Ding, [10] for details and proofs),
_i.e._ the average treatment effect on each covariate, corrected using
propensity scores, is close to zero. Figure 5 visualizes this quantity for
different covariates of the Semaglutide vs. Tirzepatide experiment and shows
that propensity scores estimated using LLAMA2 conditional distributions
balance the covariates far better than a uniform distribution does, with the
70B model consistently estimating the treatment effect on each covariate as
close to zero. Similar visualizations for the test settings are shown in fig.
10 of section F.2.

##  7 Conclusion

In this work, we introduced _NATURAL_ , a family of text-conditioned
estimators, to automate treatment effect estimation using free-form text data.
We demonstrated NATURAL’s efficacy with six synthetic and real datasets for
systematic evaluation of its pipeline. We exposed the ability of LLMs to
extract meaningful conditional distributions over structured variables and how
their combination with classical causal estimators can predict real-world
causal effects with remarkable accuracy. Given this promising performance,
exciting directions for future work include  (i) incorporating automatic
prompt tuning methods into the pipeline,  (ii) extending our methods to real-
valued treatments, outcomes and covariates,  (iii) exploring whether our
assumptions can be weakened,  (iv) exploring other domains in applied
research, e.g., social sciences,  (v) performing a more extensive evaluation
of NATURAL on different study designs to better understand what type of
treatments, outcomes, and reports show better or worse practical performance
with NATURAL or  (vi) deploying the pipeline to test hypotheses at even larger
scales.

NATURAL estimators have numerous use cases with potentially far-reaching
impact. As long as patients have access to treatments and report their
experiences, NATURAL can be used to compare two treatments in new indications
or new populations. Therefore, our pipeline can in principle support efforts
to prioritize trials for repurposed drugs or supplements in under-served
diseases or populations. Further, a crucial step after drug approval is post-
marketing surveillance for side effects (positive or negative) that may not
have been measured or may have been too rare to identify in a smaller trial.
NATURAL can leverage the diversity of available language data to detect these
effects. While our motivations largely stem from the challenges of drug
development, our NATURAL estimators are applicable to any effect estimation
setting for which there exists relevant natural language data.

## Acknowledgements

Our work was directly inspired by work done by Noah MacCallum, George Hosu,
Sina Hartung, and Zain Memon at Eureka Health. Their project, Social Treatment
Insights, explored the use of LLMs with social media data to draw medical
insights. We thank them for the spark that led to this project and for the
suggestion to study weight loss treatments. We would also like to thank Dexter
Ju for useful practical suggestions on filtering social medial posts, Amol
Verma and Fahad Razak for pointers on migraine-related clinical keywords, and
David Lopez-Paz, Patrick Forr$\acute{\text{e}}$, Roger Grosse and Sheldon
Huang for feedback on an initial draft of the paper. Resources used in
preparing this research were provided in part by the Province of Ontario, the
Government of Canada through CIFAR, and companies sponsoring the Vector
Institute. We acknowledge the support of the Natural Sciences and Engineering
Research Council of Canada (NSERC), RGPIN-2021-03445.

## References

  * Adiwardana et al., [2020] Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020).  Towards a human-like open-domain chatbot.  arXiv preprint arXiv:2001.09977.
  * AI@Meta, [2024] AI@Meta (2024).  Llama 3 model card.
  * Antonucci et al., [2023] Antonucci, A., Piqué, G., and Zaffalon, M. (2023).  Zero-shot causal graph extrapolation from text via llms.  arXiv preprint arXiv:2312.14670.
  * Arsenyan and Shahnazaryan, [2023] Arsenyan, V. and Shahnazaryan, D. (2023).  Large language models for biomedical causal graph construction.  arXiv preprint arXiv:2301.12473.
  * Ban et al., [2023] Ban, T., Chen, L., Wang, X., and Chen, H. (2023).  From query tools to causal architects: Harnessing large language models for advanced causal discovery from data.  arXiv preprint arXiv:2306.16902.
  * Baumgartner et al., [2020] Baumgartner, J., Zannettou, S., Keegan, B., Squire, M., and Blackburn, J. (2020).  The pushshift reddit dataset.  In Proceedings of the international AAAI conference on web and social media, volume 14, pages 830–839.
  * Capehorn et al., [2020] Capehorn, M., Catarig, A.-M., Furberg, J., Janez, A., Price, H., Tadayon, S., Vergès, B., and Marre, M. (2020).  Efficacy and safety of once-weekly semaglutide 1.0 mg vs once-daily liraglutide 1.2 mg as add-on to 1–3 oral antidiabetic drugs in subjects with type 2 diabetes (sustain 10).  Diabetes & metabolism, 46(2):100–109.
  * Chen et al., [2023] Chen, J. M., Malinsky, D., and Bhattacharya, R. (2023).  Causal inference with outcome-dependent missingness and self-censoring.  In Evans, R. J. and Shpitser, I., editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 358–368. PMLR.
  * Cotta et al., [2023] Cotta, L., Bevilacqua, B., Ahmed, N., and Ribeiro, B. (2023).  Causal lifting and link prediction.  Proceedings of the Royal Society A, 479(2276):20230121.
  * Ding, [2023] Ding, P. (2023).  A first course in causal inference.  arXiv preprint arXiv:2305.18793.
  * Dunning, [2012] Dunning, T. (2012).  Natural Experiments in the Social Sciences: A Design-Based Approach.  Strategies for Social Inquiry. Cambridge University Press.
  * Egami et al., [2022] Egami, N., Fong, C. J., Grimmer, J., Roberts, M. E., and Stewart, B. M. (2022).  How to make causal inferences using texts.  Science Advances, 8(42):eabg2652.
  * Feder et al., [2022] Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022).  Causal inference in natural language processing: Estimation, prediction, interpretation and beyond.  Transactions of the Association for Computational Linguistics, 10:1138–1158.
  * Flury and Riedwyl, [1986] Flury, B. K. and Riedwyl, H. (1986).  Standard distance in univariate and multivariate analysis.  The American Statistician, 40(3):249–251.
  * Forastiere et al., [2021] Forastiere, L., Airoldi, E. M., and Mealli, F. (2021).  Identification and estimation of treatment and interference effects in observational studies on networks.  Journal of the American Statistical Association, 116(534):901–918.
  * Frías et al., [2021] Frías, J. P., Davies, M. J., Rosenstock, J., Pérez Manghi, F. C., Fernández Landó, L., Bergman, B. K., Liu, B., Cui, X., and Brown, K. (2021).  Tirzepatide versus semaglutide once weekly in patients with type 2 diabetes.  New England Journal of Medicine, 385(6):503–515.
  * Hillstrom, [2008] Hillstrom, K. (2008).  The MineThatData E-Mail Analytics And Data Mining Challenge.  Challenge Dataset.
  * Jin et al., [2023] Jin, Z., Liu, J., Lyu, Z., Poff, S., Sachan, M., Mihalcea, R., Diab, M., and Schölkopf, B. (2023).  Can large language models infer causation from correlation?  arXiv preprint arXiv:2306.05836.
  * Jiralerspong et al., [2024] Jiralerspong, T., Chen, X., More, Y., Shah, V., and Bengio, Y. (2024).  Efficient causal graph discovery using large language models.  arXiv preprint arXiv:2402.01207.
  * Kadavath et al., [2022] Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. (2022).  Language models (mostly) know what they know.
  * Keith et al., [2020] Keith, K., Jensen, D., and O’Connor, B. (2020).  Text and causal inference: A review of using text to remove confounding from causal estimates.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics.
  * Keith et al., [2023] Keith, K. A., Feldman, S., Jurgens, D., Bragg, J., and Bhattacharya, R. (2023).  Rct rejection sampling for causal estimation evaluation.  arXiv preprint arXiv:2307.15176.
  * Kıcıman et al., [2023] Kıcıman, E., Ness, R., Sharma, A., and Tan, C. (2023).  Causal reasoning and large language models: Opening a new frontier for causality.  arXiv preprint arXiv:2305.00050.
  * Leung, [2022] Leung, M. P. (2022).  Causal inference under approximate neighborhood interference.  Econometrica, 90(1):267–293.
  * Lim, [2023] Lim, A. (2023).  Big five personality traits: The 5-factor model of personality. simply psychology.
  * Long et al., [2023] Long, S., Schuster, T., Piché, A., de Montreal, U., Research, S., et al. (2023).  Can large language models build causal graphs?  arXiv preprint arXiv:2303.05279.
  * Mestre-Ferrandiz et al., [2012] Mestre-Ferrandiz, J., Sussex, J., and Towse, A. (2012).  The R&D cost of a new medicine.
  * Miao et al., [2015] Miao, W., Liu, L., Tchetgen, E. T., and Geng, Z. (2015).  Identification, doubly robust estimation, and semiparametric efficiency theory of nonignorable missing data with a shadow variable.  arXiv preprint arXiv:1509.02556.
  * Mueller and Pearl, [2023] Mueller, S. and Pearl, J. (2023).  Personalized decision making–a conceptual introduction.  Journal of Causal Inference, 11(1):20220050.
  * Naik et al., [2023] Naik, N., Khandelwal, A., Joshi, M., Atre, M., Wright, H., Kannan, K., Hill, S., Mamidipudi, G., Srinivasa, G., Bifulco, C., et al. (2023).  Applying large language models for causal structure learning in non small cell lung cancer.  arXiv preprint arXiv:2311.07191.
  * OpenAI et al., [2024] OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Łukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W., Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., Łukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O’Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M., Pong, V. H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B. (2024).  Gpt-4 technical report.
  * Pearl, [2023] Pearl, J. (2023).  Judea pearl, ai, and causality: What role do statisticians play?  [Online; accessed May-2024].
  * Pryzant et al., [2020] Pryzant, R., Card, D., Jurafsky, D., Veitch, V., and Sridhar, D. (2020).  Causal effects of linguistic properties.  arXiv preprint arXiv:2010.12919.
  * Reuter et al., [2022] Reuter, U., Ehrlich, M., Gendolla, A., Heinze, A., Klatt, J., Wen, S., Hours-Zesiger, P., Nickisch, J., Sieder, C., Hentschke, C., et al. (2022).  Erenumab versus topiramate for the prevention of migraine–a randomised, double-blind, active-controlled phase 4 trial.  Cephalalgia, 42(2):108–118.
  * Roller et al., [2020] Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Shuster, K., Smith, E. M., et al. (2020).  Recipes for building an open-domain chatbot.  arXiv preprint arXiv:2004.13637.
  * Rothrock et al., [2019] Rothrock, J. F., Adams, A. M., Lipton, R. B., Silberstein, S. D., Jo, E., Zhao, X., Blumenfeld, A. M., and Group, F. S. I. (2019).  Forward study: evaluating the comparative effectiveness of onabotulinumtoxina and topiramate for headache prevention in adults with chronic migraine.  Headache: The Journal of Head and Face Pain, 59(10):1700–1713.
  * Schurman, [2019] Schurman, B. (2019).  The framework for fda’s real-world evidence program.
  * Sheldrick, [2023] Sheldrick, R. C. (2023).  Randomized trials vs real-world evidence: how can both inform decision-making?  Jama, 329(16):1352–1353.
  * Sridhar and Blei, [2022] Sridhar, D. and Blei, D. (2022).  Causal inference from text: A commentary.  Science advances.
  * Sridhar and Getoor, [2019] Sridhar, D. and Getoor, L. (2019).  Estimating causal effects of tone in online debates.  arXiv preprint arXiv:1906.04177.
  * Tu et al., [2023] Tu, R., Ma, C., and Zhang, C. (2023).  Causal-discovery performance of chatgpt in the context of neuropathic pain diagnosis.  arXiv preprint arXiv:2301.13819.
  * Vashishtha et al., [2023] Vashishtha, A., Reddy, A. G., Kumar, A., Bachu, S., Balasubramanian, V. N., and Sharma, A. (2023).  Causal inference using llm-guided discovery.  arXiv preprint arXiv:2310.15117.
  * Willard and Louf, [2023] Willard, B. T. and Louf, R. (2023).  Efficient guided generation for llms.  arXiv preprint arXiv:2307.09702.
  * Willig et al., [2023] Willig, M., Zecevic, M., Dhami, D. S., and Kersting, K. (2023).  Causal parrots: Large language models may talk causality but are not causal.  preprint, 8.
  * Winship and Morgan, [1999] Winship, C. and Morgan, S. L. (1999).  The estimation of causal effects from observational data.  Annual review of sociology, 25(1):659–706.
  * X5, [2019] X5 (2019).  X5 Retail Hero: Uplift Modeling for Promotional Campaign.  Challenge Dataset.
  * Zheng et al., [2023] Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. (2023).  Efficiently programming large language models using sglang.

## Appendix

##  Appendix A Notation

$\displaystyle R$ |  Random variable corresponding to unstructured natural language text from a social media post (or report).
---|---
$\displaystyle X$ |  Random variable corresponding to features of an individual in a causal inference dataset.
$\displaystyle T$ |  Random variable corresponding to treatment or intervention assigned to an individual in a causal inference dataset.
$\displaystyle Y$ |  Random variable corresponding to outcome observed for an individual in a causal inference dataset.
$\displaystyle x$ |  Possible instance of $X$ from its support $\mathcal{X}$.
$\displaystyle t$ |  Possible instance of $T$ from its support $\mathcal{T}=\\{0,1\\}$ (binary treatments).
$\displaystyle y$ |  Possible instance of $Y$ from its support $\mathcal{Y}=\\{0,1\\}$ (binary outcomes).
$\displaystyle r$ |  Possible instance of $R$ from its support $\mathcal{R}$.
$\displaystyle Y(t)$ |  Random variable corresponding to potential outcome observed for an individual after receiving treatment $t$.
$\displaystyle e(X)$ |  Propensity score function for binary treatments, equal to $P(T=1|X)$.
$\displaystyle X_{i}$ |  Sampled value of $X$ for individual $i$.
$\displaystyle T_{i}$ |  Sampled value of $T$ for individual $i$.
$\displaystyle Y_{i}$ |  Sampled value of $Y$ for individual $i$.
$\displaystyle R_{i}$ |  Sampled report $R$ for individual $i$.
$\displaystyle\tau$ |  Average treatment effect (ATE) given by $\mathbb{E}[Y(1)-Y(0)]$, where the expectation is over some defined population of individuals.
$\displaystyle n$ |  Total number of individuals.
$\displaystyle n_{1}$ |  Total number of individuals that are assigned treatment $T=1$.
$\displaystyle n_{0}$ |  Total number of individuals that are assigned treatment $T=0$.

##  Appendix B Deriving NATURAL Full from Outcome Imputation estimator

In section 3, we derived the idealized version of our method, NATURAL Full
using the form of the IPW estimator. In fact, any unbiased estimator would
lead to same form as eq. 3, which can then be used to derive NATURAL Full
using reports $R$ and teh law of iterated expectations. Here, we show this
using the Outcome Impuation estimator. For discrete $X$ and binary $T$ and
$Y$, we have

$\displaystyle\tau$ $\displaystyle=\mathbb{E}_{X}[P(Y=1\mid T=1,X)-P(Y=1\mid
T=0,X)]$ (outcome imputation) (11)
$\displaystyle=\sum_{x\in\mathcal{X}}P(X=x)[P(Y=1\mid T=1,X=x)-P(Y=1\mid
T=0,X=x)]$ (expectation of discrete $X$) (12)
$\displaystyle=\sum_{x\in\mathcal{X}}P(X=x)\left[\frac{P(Y=1,T=1,X=x)}{P(X=x)P(T=1|X=x)}-\frac{P(Y=1,T=0,X=x)}{P(X=x)P(T=0|X=x)}\right]$
(expanding conditionals) (13)
$\displaystyle=\sum_{x\in\mathcal{X}}\left[\frac{P(Y=1,T=1,X=x)}{e(x)}-\frac{P(Y=1,T=0,X=x)}{1-e(x)}\right]$
(definition of $e(x)$) (14)
$\displaystyle=\sum_{x\in\mathcal{X}}\sum_{y\in\mathcal{Y}}y\left[\frac{P(Y=y,T=1,X=x)}{e(x)}-\frac{P(Y=y,T=0,X=x)}{1-e(x)}\right]$
(since $\mathcal{Y}=\\{0,1\\}$) (15)
$\displaystyle=\sum_{x\in\mathcal{X}}\sum_{y\in\mathcal{Y}}y\sum_{t\in\mathcal{T}}\left[t\frac{P(Y=y,T=t,X=x)}{e(x)}-(1-t)\frac{P(Y=y,T=t,X=x)}{1-e(x)}\right]$
(since $\mathcal{T}=\\{0,1\\}$) (16)
$\displaystyle=\sum_{(x,t,y)}P(Y=y,T=t,X=x)\left[\frac{ty}{e(x)}-\frac{(1-t)y}{1-e(x)}\right]$
(rearranging terms) (17)
$\displaystyle=\mathbb{E}_{(X,T,Y)}\left[\frac{TY}{e(X)}-\frac{(1-T)Y}{1-e(X)}\right],$
(18)

which is equivalent to eq. 3.

##  Appendix C Worked Example: Semaglutide vs. Tirzepatide

To make the NATURAL pipeline and its implementation more concrete, we now work
through an end-to-end example using the Semaglutide vs. Tirzepatide dataset.
We used this setting to develop our evaluation setup and constructed the
pipeline as a function of the experiment design. While it is infeasible to
exhaustively describe the entire decision space, major decisions that were
found to impact ATE prediction are  (i) filtering strategy,  (ii) prompt
tuning for extraction, and  (iii) covariates’ discretization.  The strategies
we tried are included in the pipeline description below, with the specific
decisions made for each highlighted in this color.

#### Pipeline for Semaglutide vs. Tirzepatide.

Given the experimental design of the clinical trial in Frías et al., [16]
(NCT03987919), we defined our experiment as follows:

  1. 1.

Treatments: Semaglutide and Tirzepatide

  2. 2.

Outcome: Percentage of participants who lost 5% or more of their initial
weight

  3. 3.

Covariates: Age, Sex, BMI, Start weight, Start HbA1c, Duration

  4. 4.

Inclusion criteria:

    1. (a)

The user must be diagnosed with Type 2 Diabetes with starting HbA1c between 7%
and 10.5%.

    2. (b)

They must already be on a regime of the treatment called Metformin.

    3. (c)

They must have a BMI of 25 kg/m2 or more.

    4. (d)

Since different treatment dosages can have varying effects, we also included
dosage as an inclusion criterion, _i.e._ we aimed to include only posts that
reported taking 1mg for Semaglutide and 5mg for Tirzepatide, as in the
clinical trial.

Decisions in the pipeline that are a direct function of an experimental design
like the one above are highlighted in this color in the following description.

Next, we implemented the entire pipeline as follows:

  1. 1.

Initial filter. We identified nine subreddits relevant to this problem
setting: r/Mounjaro, r/Ozempic, r/fasting, r/intermittentfasting, r/keto,
r/loseit, r/Semaglutide, r/SuperMorbidlyObese, r/PlusSize. From each
subreddit, we downloaded all submissions and comments posted upto December
2022 from the PushShift collection, so as to only use publicly available data.
This resulted in a dataset of 577,733 submissions and comments. An initial
deterministic, task-agnostic and rule-based filter removed any submission or
comment if its content was not a string, if it had no score, if the content
was "[deleted]" or "[removed]", if it was a comment with fewer than ten space-
separated strings (presumably, words), if the author’s name contained the
string "bot", if there were no spaces in the first 2048 characters, and if
less than 50% of all characters were alphabetic. This reduced the dataset size
to 380,276. We then formatted this data into dictionary-like datapoints with
fields: subreddit, title, date created, post/comment, author replies. We
indcluded the last field because comments written by the author as replies to
their own post may contain additional relevant information when combined with
with original post and other replies. We then passed these through a task-
dependent string-matching filters. For this dataset in particular, we listed
strings used commonly to refer to the treatments, ["ozempic", "mounjaro",
"semaglutide", "tirzepatide", "wegovy", "rybelsus", "zepbound"], included
common misspellings generated with GPT-4 and Perplexity, and filtered out
datapoints that did not contain any of these strings. Similarly, we listed
keywords relevant to the outcome of interest, ["kg", "kilo", "lb", "pound",
"weigh", "drop", "loss", "lost", "gain", "hb", "a1c", "hemoglobin",
"haemoglobin", "glucose", "sugar"] and filtered out datapoints that did not
contain any of these strings. This filtered dataset now contained 50,654
datapoints.

  2. 2.

Filter by relevance. Next, we wrote a problem setting description and prompted
GPT-3.5 Turbo to determine whether the posts, along with auxiliary information
from the formatted dictionaries described above, were relevant to the
described setting. The description and instructions for this particular
dataset are shown in appendix D. We manually labeled a handful of datapoints
as Yes or No and included these as incontext examples to improve the LLM’s
generations. We removed datapoints that were deemed irrelevant, resulting in a
"relevant" dataset of 21,229 datapoints.

  3. 3.

Filter by treatment-outcome. To further filter the data to points that refer
specifically to the treatments and outcome of interest, we prompted
GPT-3.5-Turbo to extract only information required to ascertain the treatment
and outcome, as shown in appendix D. Since the outcome for this dataset,
achievement of a target weight loss of 5% or more, may be reported in several
ways, we attempted to cover all those possibilities. Specifically, we prompted
the LLM to extract the user’s starting weight, end weight, change in weight
and percentage of change in weight. Several combinations of these attributes
allow us to programmatically infer the final outcome. We also extracted the
units in which weight was reported, converting all extractions to be in lbs.
We filtered out any datapoint for which the extracted treatment was not one of
the treatments considered for this task or for which it was not possible to
infer the outcome using the above-mentioned extracted information. This
finally gave us a natural language dataset of 4619 relevant reports, each of
which contained treatment and outcome information pertaining to the defined
problem setting.

  4. 4.

Filter known covariates by inclusion criteria. To evaluate against the
clinical trial, we further filtered the dataset according to its inclusion
criteria. At a high level, different strategies for this filtering trade-off
how strictly we match the criteria with how many datapoints remain in the
filtered set. Strict filtering to match every criteria exactly resulted in
very few reports. Less strict filtering to match a subset of the criteria
resulted in predicted ATEs that varied depending on which criteria we chose to
satisfy, with no apparent task-agnostic principle to determine crucial
criteria. Finally, we aimed for a set of reports with non-zero probability of
satisfying these criteria. As described in item (iv) of section 4 and
motivated in appendix G, we extracted all covariates including ones related to
the inclusion criteria and removed datapoints whose extractions were not
Unknown and failed to satisfy the criteria above. This resulted in a dataset
of 1265 reports.

  5. 5.

Extract known and unknown covariates. Treating these 1265 reports as the final
dataset from which to estimate an ATE, we extracted the set of covariates
given in our experiment definition. We also included the duration of treatment
as a covariate since this information is often reported and is likely to
influence the outcome. This extraction step was conditioned on inclusion
criteria being satisfied, a description of which was included in the
extraction prompt, as in appendix D. We tuned the prompts for extracting
attributes, which include general instructions for the task and specific
questions for each attribute. This was done by inspecting a handful of reports
and corresponding extractions and then modifying prompts to correct any
observed errors.

  6. 6.

Infer conditionals. We inferred conditional distributions from LLAMA2-70B for
different versions of NATURAL, with the strategy described in item (vi) of
section 4 and LLM inputs of the form shown in appendix D. Here, "conditioning
on covariates" was implemented by adding questions about the covariates and
their sampled answers to the input. For instance, for sex, the question "What
is the reported sex of the user?" was followed by its previously extracted
answer (Male or Female). The scoring strategy required enumerating possible
options for treatments and outcomes for each input, which were ["Semaglutide
like Ozempic or Wegovy or Rybelsus", "Tirzepatide like Mounjaro or Zepbound"]
and ["No", "Yes"], respectively.

  7. 7.

Finally, given all the required extractions and conditional probabilities, we
required discrete covariates to plug them into our NATURAL estimators. Hence,
we converted any continuous covariates into discrete categories. These
categories for each dataset are shown in table 4 for all our datasets.
Different choices of discretization led to slightly different ATE predictions.
We found it most helpful to discretize continuous numerical covariates into
intervals such that the number of datapoints were roughly balanced across
intervals. This avoided covariate strata with too many or too few datapoints
and resulted in ATE predictions from all NATURAL estimators that were
sufficiently close to the ground truth.

#### Adapting the pipeline to test trials.

The decisions above in this color directly or indirectly influenced the ATE
and we made our choices with access to the ground truth for Semaglutide vs.
Tirzepatide. Hence, we call this a "tuned" setting. We fixed these decisions
for the three test settings. Note that no other aspect of the pipeline depends
on the ATE. All the choices in this color are a function of the experimental
design. Hence, the pipeline can be easily adapted to any new setting, given
its experimental design and without knowledge of the true ATE.

##  Appendix D LLM Prompts

` Prompt 1: Synthetic report generation (Hillstrom) ` Prompt 2: Relevance
filtering (Weight Loss) ` Prompt 3: Treatment-outcome filtering (Weight Loss)
` Prompt 4: Covariate extraction (Weight Loss) ` Prompt 5: Covariate
imputation (Weight Loss) ` Prompt 6: Conditional distribution inference
(Weight Loss)

##  Appendix E Dataset Details

We provide further details about the treatments, outcomes and covariates,
along with inclusion criteria and discrete categories used in our experiments,
for each dataset in tables 3 and 4.

Dataset | Treatment | Outcome | Synthetic confounder
---|---|---|---
Hillstrom | email communication | website visit | newbie
Retail Hero | SMS communication | purchase | age
Semaglutide vs. Tirzepatide | corresponding drug | weight loss of 5% or more | NA
Semaglutide vs. Liraglutide | corresponding drug | weight loss of 10% or more | NA
Erenumab vs. Topiramate | corresponding drug | discontinuation due to adverse effects | NA
OnabotulinumtoxinA vs. Topiramate | corresponding drug | discontinuation due to adverse effects | NA

Table 3: Treatments, outcomes and synthetic confounders (where applicable) for
each dataset.

Covariate | Description | Discrete categories | Inclusion criteria
---|---|---|---
Hillstrom |  |  |
recency | number of months since last purchase |  [$1-4,5-8,9-12$] | NA
history | dollar value of previous purchase |  [$0-100,100-200,...,>1000$]
mens | purchase of men’s merchandise | [True,False]
womens | purchase of women’s merchandise | [True,False]
zip_code | type of area of residence | [Suburban area,Rural area,Urban area]
newbie | new customer | [True,False]
channel | channel used for purchases | [Phone,Web,Multichannel]
Retail Hero |  |  |
avg. purchase | avg. purchase value per transaction |  [$1-263,264-396,397-611,>612$] | NA
avg. product quantity | avg. number of products bought |  [$\leq 7,>7$]
avg. points received | avg. number of points received |  [$\leq 5,>5$]
num transactions | total number of transactions so far |  [$\leq 8,9-15,16-27,>28$]
age | age of user |  [$\leq 45,>45$]
Semaglutide vs. Tirzepatide |  |  |
age | age of user |  [$\leq 45,>45$] |  (t2 diabetes==True) & (7 $\leq$ start HbA1c $\leq$ 10.5)) & (metformin==True) & (bmi $\geq$ 25)
sex | sex of user | [Male,Female]
bmi | body mass index of user |  [$\leq 28.5,>28.5$]
start HbA1c | initial glycated haemoglobin value |  [$\leq 7.5,>7.5$]
start weight | initial weight in lbs |  [$\leq 220,>220$]
duration (days) | number of days treatment was taken for |  [$\leq 90,>90$]
Semaglutide vs. Liraglutide |  |  |
age | age of user |  [$\leq 45,>45$] |  (t2 diabetes==True) & (7 $\leq$ start HbA1c $\leq$ 11)) & (metformin/other==True)
sex | sex of user | [Male,Female]
bmi | body mass index of user |  [$\leq 28.5,>28.5$]
start HbA1c | initial glycated haemoglobin value |  [$\leq 7.5,>7$]
start weight | initial weight in lbs |  [$\leq 220,>220$]
duration (days) | number of days treatment was taken for |  [$\leq 120,>120$]
Erenumab vs. Topiramate |  |  |
age | age of user |  [$\leq 32,>32$] |  (18 $\leq$ age $\leq$ 65) & (pregnant==False) & (baseline MMD $\geq$ 4)
sex | sex of user | [Male,Female]
country | country of residence | [United States,Canada,…]
baseline MMD | initial number of monthly migraine days |  [$\leq 6,>6$]
duration (days) | number of days treatment was taken for |  [$\leq 30,>30$]
OnabotulinumtoxinA vs. Topiramate |  |  |
age | age of user |  [$\leq 25,>25$] |  (18 $\leq$ age $\leq$ 65) & (baseline MMD $\geq$ 15)
sex | sex of user | [Male,Female]
country | country of residence | [United States,Canada,…]
baseline MMD | initial number of monthly migraine days |  [$\leq 15,>15$]
duration (days) | number of days treatment was taken for |  [$\leq 30,>30$]

Table 4: Covariate descriptions, corresponding discrete categories and
inclusion criteria enforced for each dataset. Intervals for continuous
numerical variables were determined from the extracted values such that each
discrete category is roughly balanced in terms of its number of datapoints.

##  Appendix F Further Experimental Results

###  F.1 Known and Unknown/Imputed covariates for real data experiments

We refer the reader to figs. 6, 7, 8 and 9 for empirical distributions of
covariates extracted by an LLM in its first extraction as well as those
imputed in its second imputataion conditioned on inclusion criteria.

Figure 6: Distributions of "known" (top) vs "unknown" and imputed (bottom)
covariates for Semaglutide vs. Tirzepatide. Figure 7: Distributions of "known"
(top) vs "unknown" and imputed (bottom) covariates for Semaglutide vs.
Liraglutide. Figure 8: Distributions of "known" (top) vs "unknown" and imputed
(bottom) covariates for Erenumab vs. Topiramate. Figure 9: Distributions of
"known" (top) vs "unknown" and imputed (bottom) covariates for
OnabotulinumtoxinA vs. Topiramate.

###  F.2 Balancing property of propensity scores

We refer the reader to Figure 10 for visualizations of the propensity score
corrected average treatment effect on covariates for all test clinical
settings. For each setting, our estimated propensity score balances each
covariate, far better than a uniform propensity distribution would.

Since covariates may take values at different scales, we computed the standard
mean difference (SMD) across cohorts for each covariate $X^{(i)}$ [14], given
by:

$SMD=\frac{X^{(i)}(1)-X^{(i)}(0)}{\sqrt{0.5*(\texttt{var}(X^{(i)}(1))+\texttt{var}(X^{(i)}(0)))}},$
(19)

where $X^{(i)}(1)-X^{(i)}(0)$ estimates the average treatment effect on
$X^{(i)}$, using propensity score weighting, and $\texttt{var}(\cdot)$ denotes
sample variance.

(a) Semaglutide vs. Liraglutide

(b) Erenumab vs. Topiramate

(c) OnabotulinumtoxinA vs. Topiramate

Figure 10: Propensity scores estimated with LLAMA2-70B balance covariates of
the real clinical datasets, far better than uniform scores do.

##  Appendix G Inclusion Criteria conditioned Estimator

We are interested in an ATE conditioned on inclusion criteria denoted $I$,

$\displaystyle\tau(I)=\mathbb{E}[Y(1)-Y(0)\mid X\in I].$ (20)

Let $\tau(X,T,Y)$ be a function such that

$\displaystyle\tau(I)=\mathbb{E}_{X,T,Y}[\tau(X,T,Y)\mid X\in I].$ (21)

For example, if $\tau(I)$ can estimated by the IPW estimator,

$\displaystyle\tau(X,T,Y)=\frac{TY}{e(X)}-\frac{(1-T)Y}{1-e(X)},$

because the $P(T=1|X=x,X\in I)=P(T=1|X=x)$. Throughout this section, we
operate under Assumptions 3 and 4 and assume that the LLM gives us access to
the true data-generating conditionals.

The law of total expectation gives us an estimator that can operate on samples
of reports $R$:

$\displaystyle\tau(I)$ $\displaystyle=\mathbb{E}_{X,T,Y}[\tau(X,T,Y)\mid X\in
I]$ $\displaystyle=\mathbb{E}_{R|X\in I}[\mathbb{E}_{X,T,Y}[\tau(X,T,Y)\mid
X\in I,R]]$ $\displaystyle=\sum_{r}P(R=r|X\in
I)\mathbb{E}_{X,T,Y}[\tau(X,T,Y)\mid X\in I,R]$
$\displaystyle=\sum_{r}P(R=r)\frac{P(X\in I|R=r)}{P(X\in
I)}\mathbb{E}_{X,T,Y}[\tau(X,T,Y)\mid X\in I,R]$
$\displaystyle=\mathbb{E}_{R}\left[\frac{P(X\in I|R)}{P(X\in
I)}\mathbb{E}_{X,T,Y}[\tau(X,T,Y)\mid X\in I,R]\right].$

To summarize, we have the identities:

$\displaystyle\tau(I)$ $\displaystyle=\mathbb{E}_{R|X\in
I}[\mathbb{E}_{X,T,Y}[\tau(X,T,Y)\mid X\in I,R]]$ (22)
$\displaystyle=\mathbb{E}_{R}\left[\frac{P(X\in I|R)}{P(X\in
I)}\mathbb{E}_{X,T,Y}[\tau(X,T,Y)\mid X\in I,R]\right].$ (23)

It is possible to directly estimate eq. 23 with Monte Carlo samples from
$P(R)$ and an approximation of $P(X\in I|R)/P(X\in I)$ using an LLM, but the
latter is computationally very expensive. Alternatively, one can use rejection
sampling to simulate $R|X\in I$ by sampling $X_{i}\sim P(X|R_{i})$ and
rejecting $R_{i}$ if $X_{i}\notin I$. This step could be very wasteful for
precious data. In fact, in our experiments with the Semaglutide vs.
Tirzepatide dataset, we found that using GPT-4 Turbo for sampling $X_{i}\sim
P(X|R_{i})$ resulted in a very peaky distribution with little diversity in the
extractions, even after increasing its temperature argument as much as
possible without sacrificing generation quality. As a result, after rejecting
$R_{i}$ if $X_{i}\notin I$, we were left with a very small number of reports,
that would be infeasible to plug into any estimator. Therefore, we devise a
few assumptions that allow us to estimate eq. 21 without these approaches.

Let $X\in\mathbb{R}^{D}$ and let us make the following assumption on the
inclusion criteria:

######  Assumption 5 (Inclusion criteria specification)

The inclusion criterion $I$ defines a box, i.e., it is specified separately
for each covariate dimension $I^{d},d\in\\{1,\dots,D\\}$ and the set of
covariates satisfying every inclusion criteria is given by the product of
individual criteria over the covariate dimensions, i.e., $\\{X\in
I\\}=\prod_{d=1}^{D}\\{X^{d}\in I^{d}\\}$ where $X^{d}$ is the $d$-th
dimension of $X=(X^{d})_{d=1}^{D}$.

Recall from section 4 that inclusion-based filtering leaves us with reports
whose covariates are either “known” and satisfy their criteria or Unknown. We
also have the value of the known covariates. Let $K\in\\{0,1\\}^{D}$ be the
binary vector of variables $K^{d}$ that indicate whether the covariate $X^{d}$
is found to be “known” for a random report $R$. Let $X^{K}=(X^{d}:K^{d}=1)$ be
the vector of length $\sum_{d}K^{d}$ holding the values of the known
covariates. For ease of notation, define the event that the known covariates
satisfy their criteria and the event that the unknown covariates satisfy their
criteria:

$\displaystyle\\{X^{K}\in I^{K}\\}$ $\displaystyle=\\{X^{d}\in I^{d},\forall
d\colon K^{d}=1\\}$ (24) $\displaystyle\\{X^{1-K}\in I^{1-K}\\}$
$\displaystyle=\\{X^{d}\in I^{d},\forall d\colon K^{d}=0\\}$ (25)

Notice that $\\{X^{K}\in I^{K}\\}\cap\\{X^{1-K}\in I^{1-K}\\}=\\{X\in I\\}$.
Thus, after the filtering steps we have

$\displaystyle\\{R_{i},K_{i},X_{i}^{K_{i}}\\}_{i=1}^{n}$ (26)

with the guarantee that the knowns satisfy their inclusion criteria,
$\\{X^{K_{i}}_{i}\in I^{K_{i}}\\}$. Formally, assuming that the LLM computes
the true conditional distribution of the data-generating process (Assumption
4), this gives us data sampled i.i.d. from $P(R=r,K=k,X^{k}=x^{k}|X^{K}\in
I^{K})$. Note, that we are assuming the existence of an additional ground-
truth random variable $K$ in the data-generating process that describes
whether a covariate is knowable from a report. Here, we show how to estimate
$\tau(I)$ from this dataset of filtered reports using importance sampling,
under the following assumption:

######  Assumption 6 (Satisfaction of $I$ by Unknown covariates)

Satisfaction of inclusion criteria by unknown covariates is conditionally
independent of the report and the known covariates given satisfaction of
inclusion criteria by known covariates, i.e., for all $r,k,x^{k}$:

$\displaystyle P(R=r,K=k,X^{k}=x^{k}|X^{K}\in I^{K},X^{1-K}\in
I^{1-K})=P(R=r,K=k,X^{k}=x^{k}|X^{K}\in I^{K})$ (27)

One can derive the following identity in a similar fashion as eq. 23

$\displaystyle\tau(I)=\mathbb{E}_{R,K,X^{K}|X^{K}\in
I^{K}}\left[\frac{P(R,K,X^{K}|X\in I)}{P(R,K,X^{K}|X^{K}\in
I^{K})}\mathbb{E}_{X,T,Y}[\tau(X,T,Y)\mid X\in I,R,K,X^{K}]\right].$ (28)

From assumption 6, the fraction above simplifies to $1$, leaving us with the
following estimator

$\tau(I)=\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}_{X_{i},T_{i},Y_{i}}[\tau(X_{i},T_{i},Y_{i})\mid
X_{i}\in I,R_{i},K_{i},X_{i}^{K_{i}}],$ (29)

which can be computed from the information available at the end of filtering.
In practice, we do not condition the LLM on $K_{i}$ in the final inference
step (vi), which amounts to an additional conditional independence assumption:

######  Assumption 7 (Conditional independence of knowable covariates)

$K\perp\\!\\!\\!\perp(T,Y)\mid(X,R)$.

Equation 29 above can now be more efficiently estimated by prompting the LLM
to extract covariates under the constraints of the inclusion criteria for each
report in our filtered dataset, and then following the remaining steps in the
pipeline to an ATE estimate.

``````