:
2024

# Zero-Shot Clinical Trial Patient Matching with LLMs

Michael Wornow*<EMAIL_ADDRESS>
Alejandro Lozano*<EMAIL_ADDRESS>
Dev Dash<EMAIL_ADDRESS>
Jenelle Jindal<EMAIL_ADDRESS>
Kenneth W. Mahaffey<EMAIL_ADDRESS>
Nigam H. Shah<EMAIL_ADDRESS>
Stanford University  United States

###### Abstract

Matching patients to clinical trials is a key unsolved challenge in bringing
new drugs to market. Today, identifying patients who meet a trial’s
eligibility criteria is highly manual, taking up to 1 hour per patient.
Automated screening is challenging, however, as it requires understanding
unstructured clinical text. Large language models (LLMs) offer a promising
solution. In this work, we explore their application to trial matching. First,
we design an LLM-based system which, given a patient’s medical history as
unstructured clinical text, evaluates whether that patient meets a set of
inclusion criteria (also specified as free text). Our zero-shot system
achieves state-of-the-art scores on the n2c2 2018 cohort selection benchmark.
Second, we improve the data and cost efficiency of our method by identifying a
prompting strategy which matches patients an order of magnitude faster and
more cheaply than the status quo, and develop a two-stage retrieval pipeline
that reduces the number of tokens processed by up to a third while retaining
high performance. Third, we evaluate the interpretability of our system by
having clinicians evaluate the natural language justifications generated by
the LLM for each eligibility decision, and show that it can output coherent
explanations for 97% of its correct decisions and 75% of its incorrect ones.
Our results establish the feasibility of using LLMs to accelerate clinical
trial operations.

#### Data and Code Availability

This paper uses the 2018 n2c2 Cohort Selection Challenge dataset (Stubbs et
al., 2019). Our code is available here: https://github.com/som-
shahlab/clinical_trial_patient_matching.

#### Institutional Review Board (IRB)

This was not determined to be human subject research, and thus did not require
IRB approval.

##  1 Introduction

Figure 1: We explore zero-shot clinical trial patient matching with LLMs under
two system designs: (a) We inject the patient’s entire set of notes into a
prompt input into an Assessment LLM (e.g. GPT-4) for evaluation. (b) In our
two-stage retrieval pipeline, we first query the top-$k$ most relevant chunks
from the patient’s notes, then inject only those top-$k$ chunks into the
prompt input into an Assessment LLM. Both paradigms are compared using the
same prompting strategies, with the only distinction being the amount of
patient information included in the prompt.

Identifying eligible patients for clinical trials is a key bottleneck in
advancing new drugs to market (Woo, 2019). Roughly one third of clinical
trials fails because it cannot enroll enough eligible patients, and
recruitment costs eat up 32% of a trial’s budget (Deloitte, 2020).

For patients, enrolling in a trial can bring several benefits, such as access
to novel therapies, increased monitoring from expert care teams, and better
overall health outcomes (Locock and Smith, 2011; Bouzalmate et al., 2022).
Unfortunately, 94% of patients are never informed by their doctors about
trials for which they might qualify (Nuttall, 2012).

That is because identifying patients who are eligible for a trial is often
highly manual and time-consuming (Brøgger-Mikkelsen et al., 2020). Each trial
has a long list of eligibility criteria which must be exactly met by every
patient who enrolls (FDA, 2020). To identify these patients, a trained
clinical research coordinator manually reviews hundreds of patients’
electronic health records (EHRs) (Penberthy et al., 2012; Calaprice-Whitty et
al., 2020). Over 80% of the relevant information is stored as unstructured
text – e.g. progress notes, emails, radiology reports, and genetic tests
(Kong, 2019) – making it difficult to automatically process (Leaman et al.,
2015). As a result, eligibility screening for a Phase III cancer trial can
take almost an hour per patient (Penberthy et al., 2012).

Traditional natural language processing (NLP) approaches have seen limited
success with clinical text due to its highly idiosyncratic grammar and
terminology (Leaman et al., 2015). Large language models (LLMs) represent a
key inflection point in the capabilities of NLP systems (Brown et al., 2020),
and thus offer a promising approach for accelerating patient recruitment.

In this paper, we aim to build upon recent work exploring the application of
LLMs to patient matching (Jin et al., 2023; Wong et al., 2023; Nievas et al.,
2023). In contrast to this prior work, our primary focus is on the efficiency
of LLM-based approaches – namely, their cost in terms of dollars, money, and
time. These considerations are essential if such systems are proven effective
and aim to be deployed at scale.

Our contributions are as follows.

  1. 1.

We investigate the zero-shot (i.e. out-of-the-box) performance of LLMs on
evaluating patient eligibility. Unlike prior work, this requires no training
data or in-context examples. Our GPT-4-based system achieves state-of-the
performance on Track 1 of the 2018 n2c2 cohort selection benchmark (Stubbs et
al., 2019).

  2. 2.

We show how to improve the data and cost efficiency of LLM-based systems. We
identify prompting strategies that best balance performance and cost. We also
design a two-stage retrieval-based system, as shown in Figure 1, which pre-
filters relevant clinical notes and can reduce token usage by over a third
while still beating the state-of-the-art on certain metrics.

  3. 3.

We demonstrate the potential for human-in-the-loop deployment by having
clinicians evaluate the natural language justifications generated by the LLM
for each assessment it makes. We find that GPT-4 is able to output a coherent
rationale for its correct decisions 97% of the time, and even when it makes an
incorrect decision, our annotators agreed with 75% of its justifications.

The rest of the paper is structured as follows. First, we provide an overview
of related work. Second, we detail the models that we test, our different
prompting strategies, and our retrieval system. Third, we describe our dataset
and evaluation set-up. Fourth, we report our system’s results. Fifth, we
discuss limitations and propose areas for further research.

##  2 Related Work

###  2.1 Large Language Models (LLMs)

Natural language processing (NLP) was revolutionized by the advent of deep
learning models which could learn from huge amounts of unlabeled text, e.g.
all of Wikipedia (Howard and Ruder, 2018). As these models increased in size
and their datasets became even larger (e.g. all of the Internet), their
capabilities continued to improve (Brown et al., 2020; Kaplan et al., 2020).
Known as “large language models” (LLMs), they have surpassed humans on a
number of benchmarks (Anil et al., 2023) and can now conduct diagnostic
conversations with patients (Tu et al., 2024), translate clinical jargon into
layman’s terms (Mirza et al., 2024), and perform chart review (Goel et al.,
2023). Most impressively, LLMs have demonstrated the ability to tackle many of
these tasks without ever explicitly being trained to solve them (Brown et al.,
2020).

Asking an LLM to solve a task without providing any training examples is
referred to as “zero-shot” learning (Xian et al., 2018). This stands in sharp
contrast to traditional machine learning models which rely on labeled training
datasets, as well as conventional rule-based systems which require domain
experts to hand-craft programs. In contrast, LLMs such as GPT-4 (Achiam et
al., 2023) can be adapted for a wide range of tasks expressed via text. This
makes them promising for the task of patient matching.

###  2.2 Clinical Trial Patient Matching

Initial work in automated trial matching focused on automatically transforming
free text eligibility criteria into structured SQL queries, which could then
be run over the structured data within patient EHRs. Systems such as EliXR
(Weng et al., 2011) accomplished this via rule-based algorithms, while EliIE
(Kang et al., 2017) and Criteria2Query (Yuan et al., 2019) used more advanced
NLP methods such as named entity recognition, relation extraction, and
negation detection. Most relevant to this paper, the RBC model from Oleynik et
al. (2019) achieved the highest score on the 2018 n2c2 cohort selection
benchmark. While well-suited for the specific trials for which they are
developed, these systems are difficult to maintain and tend not to generalize
(Gehrmann et al., 2017).

More recently, focus has shifted to developing end-to-end systems that can
directly match patients to free text criteria. Works such as COMPOSE (Gao et
al., 2020) and DeepEnroll (Zhang et al., 2020) used neural networks to jointly
encode enrollment criteria and patient EHRs. Both of these models, however,
only incorporated structured EHR data and ignored clinical text.

TrialGPT was the first end-to-end LLM-based system for processing both
unstructured clinical text and eligibility criteria (Jin et al., 2023), and
achieved high accuracy. However, they did not conduct zero-shot evaluation and
only tested GPT-3.5, which as we later demonstrate missed out on large
performance gains. Scaling with LLMs (Wong et al., 2023) used an GPT-4 to
convert free text criteria into structured queries, but did not directly use
the LLM to evaluate patients. More broadly, TRIALSCOPE offered an exciting
vision of simulating entire trials with real world evidence, but did not
develop their own patient matching solution (González et al., 2023).
TrialLlama (Nievas et al., 2023) fine-tuned the open source Llama2 model
(Touvron et al., 2023) for trial matching, but this required extensive
finetuning using 16 A100s and did not consider zero-shot settings.
Additionally, none of these works considered the use of a preliminary
retrieval step to first curate the notes fed into the LLM.

##  3 Methods

In the following, assume we are given a set of patients $\mathcal{P}$ and
notes $\mathcal{N}$, where each patient $p\in\mathcal{P}$ has a set of notes
$\mathcal{N}_{p}\in N$. We have a set $\mathcal{C}$ of trial criteria. There
is some unknown function $E:\mathcal{C}\times\mathcal{P}\rightarrow\\{0,1\\}$
that maps each criterion and patient to a binary indicator of whether that
patient meets (1) or does not meet (0) that criterion. Our goal is to
calculate $E(c,p)$ for all patients $p$ and criteria $c$.

Assume we have a small “embedding” model
$\mathbf{M}:\mathcal{T}\rightarrow\mathcal{V}$ which maps some text
$t\in\mathcal{T}$ to a vector $v\in V$ and can be run quickly at virtually no
cost. We also have a larger downstream “assessment” model
$\mathbf{A}:\mathcal{T}\rightarrow\mathcal{O}$ which maps a prompt
$t\in\mathcal{T}$ to some output $o\in\mathcal{O}$ and is expensive and slow.
Finally, we have some function $\phi$ which takes a variable number of
criteria and patient notes (depending on the prompting strategy used), and
outputs a set of prompts. These prompts are then fed into the assessment model
$\mathbf{A}$ to generate an output $o\in\mathcal{O}$ (typically a JSON
string), which is then parsed into an estimate for $E(c,p)$.

###  3.1 Models

For our assessment model $\mathbf{A}$, we consider the proprietary models
GPT-3.5 Brown et al. (2020) and GPT-4 (Achiam et al., 2023), as well as the
popular open source models Llama-2-70b-32k (Peng et al., 2023) and
Mixtral-8x7B (Jiang et al., 2024). We run GPT-3.5 and GPT-4 via a secure Azure
PHI-compliant instance. We run the open source models on a local PHI-compliant
server with 4 Nvidia A100 GPUs using the vLLM (Kwon et al., 2023) and Outlines
(Willard and Louf, 2023) packages for high-speed inference and enforcing valid
JSON output.

Table 1: Zero-shot results on the 2018 n2c2 cohort selection challenge using the ACIN prompt strategy. We use the 32k context length version of all models, with the exception of GPT-3.5 (which is limited to 16k tokens). Model | Prec. | Rec. |  Overall Macro-F1 |  Overall Micro-F1
---|---|---|---|---
Baseline | 0.69 | 0.78 | 0.43 | 0.76
Prior SOTA | 0.88 | 0.91 | 0.75 | 0.91
Llama-2-70b | 0.82 | 0.41 | 0.46 | 0.67
Mixtral-8x7B | 0.72 | 0.83 | 0.64 | 0.79
GPT-3.5 | 0.76 | 0.82 | 0.63 | 0.82
GPT-4 | 0.91 | 0.92 | 0.81 | 0.93
Table 2: Increasing the specificity of criteria definitions improves model accuracy. Model | Prompt | Prec. | Rec. |  Overall Macro-F1 |  Overall Micro-F1
---|---|---|---|---|---
GPT-3.5 | Original | 0.76 | 0.48 | 0.57 | 0.69
| Improved | 0.76 | 0.82 | 0.63 | 0.82
GPT-4 | Original | 0.89 | 0.74 | 0.75 | 0.85
| Improved | 0.91 | 0.92 | 0.81 | 0.93

###  3.2 Zero-Shot Criteria Assessment

We begin by evaluating the zero-shot capabilities of our assessment LLMs. We
choose the zero-shot setting as it offers the simplest path towards real-world
deployment – it requires minimal engineering, no data labeling, and can
instantly be adapted to any trial.

Given a set of notes $\mathcal{N}_{p}\subseteq\mathcal{N}$ for a patient $p$,
we investigate four strategies for generating the prompt(s) that we feed into
the assessment LLM: (1) All Criteria, All Notes (ACAN) in which all notes in
$\mathcal{N}_{p}$ are merged into a single prompt and the model generates all
$C=|\mathcal{C}|$ eligibility assessments at once. (2) All Criteria,
Individual Notes (ACIN) in which the model is given $N=|\mathcal{N}_{p}|$
separate prompts (each containing one note), and for each note generates all
$C$ eligibility assessments. (3) Individual Criteria, All Notes (ICAN) in
which all $N$ notes are merged into a single prompt, but the model only
assesses one criterion at a time, and thus must be re-prompted with the same
merged set of notes $C$ times. (4) Individual Criteria, Individual Notes
(ICIN) in which the model is given $N*C$ separate prompts each containing one
note and one criterion. More precisely, we thus have:

$\displaystyle ACAN:~{}$
$\displaystyle\phi(\mathcal{N}_{p}^{(1)}\mathbin{\|}...\mathbin{\|}\mathcal{N}_{p}^{(N)},c_{1}\mathbin{\|}...\mathbin{\|}c_{C})=t$
$\displaystyle ACIN:~{}$
$\displaystyle\phi(\mathcal{N}_{p}^{(1)},...,\mathcal{N}_{p}^{(N)},c_{1}\mathbin{\|}...\mathbin{\|}c_{C})=\\{t_{1},...,t_{N}\\}$
$\displaystyle ICAN:~{}$
$\displaystyle\phi(\mathcal{N}_{p}^{(1)}\mathbin{\|}...\mathbin{\|}\mathcal{N}_{p}^{(N)},c_{1},...,c_{C})=\\{t_{1},...,t_{C}\\}$
$\displaystyle ICIN:~{}$
$\displaystyle\phi(\mathcal{N}_{p}^{(1)},...,\mathcal{N}_{p}^{(N)},c_{1},...,c_{C})=\\{t_{1},...,t_{NC}\\}$

Where $\mathbin{\|}$ represents concatenation. Please see Appendix Figures 5
and 6 for the exact prompt templates used.

The output $o\in\mathcal{O}$ that we expect from the Assessment model
$\mathbf{A}$ is a JSON string containing four elements:

  1. 1.

Criterion: Specific criterion the model is assessing.

  2. 2.

Medications: List of medications extracted from the note.

  3. 3.

Rationale: Step-by-step reasoning as to how the model claims to have come to
its decision.

  4. 4.

Decision: Binary “MET” or “NOT MET” decision.

  5. 5.

Confidence: ”Low”, ”Medium”, or ”High” confidence in prediction

We include the Medications element to helped the model recall specific drugs
when assessing certain criteria, as prior literature demonstrated the ability
of LLMs to perform this extraction (Agrawal et al., 2022). Rationale was
included to both improve model performance via chain-of-thought-like reasoning
(Wei et al., 2022) and provide interpretability for a human reviewer. For the
Llama-2 model only, we removed Medications and Confidence from the prompt as
we were unable to get it to generate valid completions.

###  3.3 Retrieval Pipeline

We design a two-stage pipeline that leverages a small embedding model
$\mathbf{M}$ to pre-filter the set of notes $n\in\mathcal{N}$ that get fed
into the more expensive assessment model $\mathbf{A}$. This pipeline is
depicted in Figure 1.

We consider two models for $\mathbf{M}$: BGE for its high performance on
retrieval benchmarks (Xiao et al., 2023), and MiniLM for its speed (Reimers
and Gurevych, 2019). These models are small and can be run locally and
cheaply. In a deployment scenario, these models would be run once over all
clinical text within an EHR.

To initialize our pipeline, we loop through every patient and embed all of
their notes using our embedding model $\mathbf{M}$. We also we embed each
criterion.. Thus, we get a database of embeddings $\mathcal{D}$ such that
$\mathcal{D}=\\{\mathbf{M}(n)~{}\forall
n\in\mathcal{N}\\}\cup\\{\mathbf{M}(c)~{}\forall c\in\mathcal{C}\\}$

During inference we have two stages – Pre-Filtering and Assessment, which
slightly differ depending on the prompting strategy used. The ICAN and ICIN
strategies work as follows. In Pre-Filtering, we are given a patient $p$ and
individual criterion $c$. We calculate the cosine similarity between
$\mathbf{M}(n)$ and $\mathbf{M}(c)$ for all $n\in N_{p}$. We keep the top-$k$
ranking notes and discard the rest. We repeat this step for each criterion
$c\in\mathcal{C}$. In Assessment, we construct our prompts with our subset of
top-$k$ notes using our chosen prompting strategy
$\phi(n_{1},...,n_{k},c_{1},...,c_{C})$. We then feed the prompt into the
Assessment LLM to generate our output $o=A(\phi(n_{k},c_{1},...,c_{C}))$. The
pipeline is similar for ACAN and ACIN, but instead of processing each
individual criterion $c$ separately, we first concatenate them together to
form a single criterion $c_{1}\mathbin{\|}...\mathbin{\|}c_{C}$ and only run
the Pre-Filtering step once.

In practice, we work at the level of “chunks” instead of “notes,” where a
chunk is simply a substring taken from a larger note whose length fits within
the context window of our embedding model $\mathbf{M}$. In our case, this was
512 tokens.

In addition to reducing the number of tokens input into the assessment model
$\mathbf{A}$, this retrieval-based approach would allow our system to scale to
real-world health systems with millions of notes (Jiang et al., 2023), an
important efficiency consideration that was underexplored in prior work.

##  4 Evaluation

We evaluate the ability of LLMs to assess whether a patient meets or does not
meet a set of inclusion criteria.

Table 3: Considering one criterion/note at a time improves performance. “Tokens” includes both prompt and completion tokens (i.e. inputs and outputs). “API Calls” is the total number of times the LLM was queried. “Cost” is based on OpenAI’s pricing as of January 25, 2024 (OpenAI, 2024) Model | Prompt Strategy | Performance | Efficiency
---|---|---|---
| Criteria | Notes | Prec. | Rec. |  Overall Macro-F1 |  Overall Micro-F1 | Cost | API Calls | Tokens
GPT-3.5 | All | All | 0.76 | 0.69 | 0.53 | 0.77 | $1.24 | 86 | 0.64M
| All | Individual | 0.74 | 0.80 | 0.59 | 0.80 | $2.41 | 377 | 1.31M
| Individual | All | 0.76 | 0.82 | 0.63 | 0.82 | $13.19 | 1118 | 6.66M
| Individual | Individual | 0.73 | 0.88 | 0.60 | 0.81 | $17.61 | 4901 | 9.04M
GPT-4 | All | All | 0.89 | 0.87 | 0.80 | 0.90 | $74.77 | 86 | 0.69M
| All | Individual | 0.91 | 0.92 | 0.81 | 0.93 | $133.08 | 377 | 1.32M
| Individual | All | 0.94 | 0.86 | 0.85 | 0.92 | $780.99 | 1118 | 6.61M
| Individual | Individual | 0.92 | 0.89 | 0.82 | 0.92 | $1021.65 | 4901 | 8.93M

###  4.1 Dataset

For our dataset, we use the 2018 n2c2 Clinical Trial Cohort Selection
challenge dataset, as it is one of the more well-studied public benchmarks for
patient matching (Stubbs et al., 2019). The dataset has two parts: patients
and eligibility criteria.

(1) Patients. There are 288 unique patients (subdivided into two splits: 202
train and 86 test). Each patient has 2-5 de-identified clinical notes, with an
average of 2,711 words each. The patients are all diabetic, and were
originally sourced from a combined database of patients from Mass General and
Brigham and Women’s hospitals. Details on the cohort selection process are
provided in Stubbs and Uzuner (2015) and Kumar et al. (2015).

(2) Eligibility criteria. There are 13 inclusion criteria which reflect
commonly used eligibility criteria (Stubbs et al., 2019). These criteria, as
well as their definitions, are listed in Appendix Table 5. Note that there are
no exclusion criteria.

All of this information is provided as unstructured text. Thus, the only way
to match patients is via natural language processing. Note that this benchmark
simulates a synthetic trial – i.e. there is no actual clinical trial that
enrolled these patients. To obtain labels, two medical experts annotated each
patient as to whether the 13 inclusion criteria were “MET” or “NOT MET”. The
average Cohen’s Kappa between the two annotators was 0.54 (Stubbs et al.,
2019).

###  4.2 Metrics

We evaluate our models on their precision, recall, and overall Macro/Micro-F1
scores on the binary classification tasks of deciding whether a patient met or
did not meet each of the 13 inclusion criteria in the 2018 n2c2 benchmark.
“Overall” is defined by the n2c2 benchmark as the simple average of the F1
score calculated for the “met” examples and F1 score for the “not met”
examples (Stubbs et al., 2019; Oleynik et al., 2019). We use this definition
for consistency when reporting Macro/Micro-F1 scores as well.

We also measure several efficiency metrics: (1) total number of tokens (both
prompt and completion) used by the LLM to generate its predictions; (2) total
cost (in USD) of making predictions; (3) and total number of API calls made to
the LLM (as a proxy for time taken).

###  4.3 Baselines

In addition to the LLMs we test, we consider two baselines. First, the best
model from the original 2018 n2c2 competition, which achieved an Overall
Micro-F1 of 0.91 (Oleynik et al., 2019). We refer to this model as “Prior
SOTA.” It utilized hand-crafted rules, negation detection, family history
detection, and regular expressions (Oleynik et al., 2019). We also consider
simply predicting the class prevalence for each criterion, and refer to this
as “Baseline”.

###  4.4 Interpretability

Unlike the rule-based Prior SOTA, our LLMs are able to output natural language
rationales justifying their decisions. We recruit two clinicians to manually
review the rationales generated by our best-performing model (GPT-4).
Specifically, for each of the 13 criteria, we sample 20 patients for which
GPT-4 made an incorrect prediction and 20 patients for which GPT-4 made a
correct prediction. For criterion where GPT-4 made less than 20 incorrect
predictions, we simply include all mispredictions. This gave us a total of 486
rationales.

##  5 Results

Figure 2: Model performance increases as the number ($k$) of retrieved
documents increases, but quickly plateaus with diminishing returns. We test
$k\in\\{1,3,5,10\\}$. Each subfigure is a different prompting strategy. The
y-axis is model performance (Macro/Micro-F1) and the x-axis is the total
number of tokens processed by the model. Orange is GPT-4, blue is GPT-3.5, and
the green line is the prior state-of-the-art. Stars represent each model’s
best performance when feeding in all notes. The MiniLM embedding model is the
dashed line, while BGE is the solid line.

fig:subfigex [Correct decisions] [Incorrect decisions]

Figure 3: Clinician assessment of whether the rationales generated by GPT-4
were correct (green), partially correct (yellow), or incorrect (red), split by
whether GPT-4’s corresponding eligibility decision was evaluated as correct
(a) or incorrect (b) according to the n2c2 benchmark.

We evaluate our LLM-based systems on the 2018 n2c2 Cohort Selection Challenge
dataset (Stubbs et al., 2019).

###  5.1 Zero-Shot Criteria Assessment

For our initial zero-shot evaluation, we feed the entire patient’s medical
history into the LLM and have it predict all criteria at once per the ACIN
prompting strategy. All of the models we test are able to fit each patient’s
history into their context windows. We report our results in Table 1. Despite
not being tuned for trial matching or provided any in-context examples, GPT-4
beats the state-of-the-art by a margin of +6 Macro-F1 and +2 Micro-F1 points.
We provide criterion-level confusion matrices for this model in Appendix
Figure 4.

The open source models perform worse than GPT-3.5, and GPT-3.5 performs
significantly worse than GPT-4 (Bubeck et al., 2023). This aligns with prior
evaluations of LLMs on EHR-based tasks (Fleming et al., 2023), and suggests
that prior trial matching work based on GPT-3.5 has significantly
underestimated the capabilities of LLMs for this task (Jin et al., 2023).

###  5.2 Prompt Engineering

The n2c2 benchmark does not come with a predefined set of prompts, but instead
provides a set of sparse definitions for each criteria (see Appendix Table 5).
As the low inter-rater agreement between clinicians suggests, these
definitions did not fully capture how the dataset was labeled. Thus, to design
our prompts, we randomly sampled three patients who met and did not meet each
eligibility criteria from the training dataset, and used this corpus to
manually refine the definition of each criterion. We provide our “improved”
criteria in Appendix Table 6, and demonstrate in Table 2 that increasing the
specificity of criteria is an essential first step in using an LLM. In a real-
world setting, this step could be avoided by directly using the detailed
protocols provided by trial sponsors (DNDi, 2018; Limited, 2017).

We ablate the 4 prompting strategies detailed in Section 3.2. As shown in
Table 3, the ACAN strategy performs worst, while the rest of the strategies
appear to be relatively equivalent in performance. When cost is considered,
however, the ACIN strategy becomes the clear winner – the ACIN strategy
achieves the best Micro-F1 at almost an order of magnitude lower cost, API
calls, and tokens used than the other prompting strategies. ACIN passes each
note through the LLM once and has it evaluate all criteria simultaneously,
whereas the strategies relying on “Individual Criteria” pass each note through
the LLM 13 times (once for each criterion). Thus, careful prompting can result
in meaningful cost savings with virtually no drop in accuracy.

###  5.3 Retrieval Pipeline

We test different cutoff values of $k$ for the number of chunks to retrieve,
and measure its impact on GPT-4’s performance under the ACAN, ACIN, and ICAN
prompting strategies.

In Figure 2, we plot the performance of the model on the y-axis and the amount
of tokens used on the x-axis. In these plots, the orange line is GPT-4, the
blue line is GPT-3.5, the green line is the prior state-of-the-art, and the
star is the non-retrieval-based implementation of each model. With our best
embedding model, MiniLM, we are able to surpass the prior state-of-the-art on
Macro-F1 using roughly one-third and one-half as many tokens as needed in the
vanilla ICAN and ACAN strategies, respectively. While both methods approach
the state-of-the-art on Micro-F1, neither retrieval-based method is able to
fully achieve it without using the full note. There appears to be an
unresolved gap in performance when relying on a retrieval-based system to
curate patient information. Additionally, the cost benefits of the retrieval
pipeline largely depend on the prompting strategy used – since the ACIN
strategy feeds each chunk separately through the model, once $k$ exceeds the
number of total notes in a patient’s EHR, then the cost of using the retrieval
pipeline ends up exceeding the cost of simply using full notes without
retrieval.

###  5.4 Interpretability

We sample 468 rationales generated by GPT-4 using the ICAN prompting strategy,
and have two clinicians evaluate their veracity. We use the ICAN version of
the system in order to cleanly disambiguate each individual criterion’s
prediction. Each rationale was evaluated on a 3-part scale: Correct, Partially
Correct, and Incorrect, based on how accurately it aligned to the relevant
patient’s EHR.

The results in Figure 3 show that GPT-4 is able to provide legitimate
rationales for most its decisions. When GPT-4 makes a correct eligibility
decision, 89% of its rationales were judged as fully correct, 8% as partially
correct, and 3% as incorrect. When GPT-4 made an incorrect eligibility
decision, its rationales were split 67/8/25% between correct/partially
correct/incorrect. This indicates that either the “ground truth” label of the
n2c2 benchmark is incorrect, or that there is legitimate variation in clinical
judgement as to whether the patient meets a criterion. This nuance is
something that an LLM can better capture through its natural language
generation abilities, whereas traditional rule-based and deep learning models
cannot.

The most common errors noted by our clinician evaluators were pulling outdated
medications and lab test values when mentioned across multiple notes (e.g.
HBA1C values), inaccurately assessing the language of the patient when
mentioned in a note, and over/under-weighting the severity of indicators of
cardiovascular disease.

##  6 Discussion

In this work, we developed a zero-shot LLM-based pipeline to match patients to
clinical trials. We achieved state-of-the-art performance on the 2018 n2c2
cohort selection challenge, achieving gains of +6 Macro-F1 and +2 Micro-F1
points over the prior best model without any feature engineering. Unlike prior
methods, our system also provides interpretable rationale for its predictions,
thereby allowing for human oversight and collaboration. Clinician evaluators
disagreed with the LLM-generated rationales only 3% of the time when the LLM
made a correct eligibility decision, and 25% of the time when it made an
incorrect decision. Finally, we show that a preliminary retrieval step can
preserve high performance while reducing token usage, and that thoughtful
prompting strategies can lead to large cost savings.

In contrast to prior work, we do no fine-tuning or updates to the underlying
LLM’s weights. Nor do we provide in-context few shot examples. This
illustrates the unique scalability advantages that LLMs such as GPT-4 offer,
as they can be seamlessly repurposed for arbitrary trials so long as
definitions for the criteria are known. In contrast to prior rule-based
systems, the LLMs we test demonstrate robust clinical language understanding
and general reasoning capabilities.

If our results generalize, then LLMs represent a viable solution for clinical
trial patient identification given their accuracy, efficiency, and cost-
effectiveness. Our GPT-4-based system is accurate, as shown in Table 1, and is
able to output coherent explanations for the majority of its decisions, as
depicted in Figure 3. It is an order of magnitude faster than today’s clinical
trials operate, as it was able to evaluate all 86 patients in roughly two
hours versus the one patient per hour expected for Phase III trials (Penberthy
et al., 2012). Our pipeline is also more cost-effective, as Table 3 shows that
the cost for using GPT-4 with the ACIN strategy to screen a single patient is
roughly $1.55, whereas prior estimates for a Phase III trial peg the cost per
patient at $34.75111See Table 3, $$38193.70/1099=\$34.75$ cost per patient
screened. (Penberthy et al., 2012), a gap that will likely widen as models
become more efficient.

There are major limitations of our work. First, the size of our dataset is
dwarfed by the scale of most health systems’ EHRs, which underscores the need
for using preliminary retrieval-based pipelines which can reduce the amount of
tokens that an LLM must process to match each patient. Second, the benchmark
we use – the 2018 n2c2 cohort challenge – is a significant simplification of
actual trial matching. It only covers inclusion criteria, only includes
English, and its formatting might not generalize to other EHRs. Third,
although our LLM-based system achieved state-of-the-art results on this
benchmark, further evaluations are required to asses the safety and failure
modes of applying these models in production settings. Fourth, the use of
proprietary models such as GPT-4 may not be feasible for health systems that
do not have HIPAA-compliant Azure instances.

At a higher level, there are also open questions as to whether low enrollment
in trials is a function of not being able to match patients (which this work
addresses) or is instead primarily due to the stringency of the eligibility
criteria themselves (Liu et al., 2021). While beyond the scope of this work,
we believe that improving the design of eligibility criteria presents another
interesting research direction which could benefit from the application of
more sophisticated models such as LLMs (Liu et al., 2021).

##  7 Conclusion

As our results demonstrate, LLMs have the potential to accelerate the matching
of patients to clinical trials through their robust reasoning and natural
language capabilities. Given the central importance of patient recruitment to
drug discovery, we hope this work inspires further exploration of applying
LLMs to clinical trial operations.

This work was supported in part by the Clinical Excellence Research Center at
Stanford Medicine and Technology and Digital Solutions at Stanford Healthcare.
MW is supported by an HAI Graduate Fellowship. AL is supported by an ARC
Institute Graduate Fellowship.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Agrawal et al. (2022) Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag.  Large language models are few-shot clinical information extractors.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1998–2022, 2022.
  * Anil et al. (2023) Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Bouzalmate et al. (2022) Amira Bouzalmate, Paloma Massó Guijarro, Khalid Saeed Khan, Aurora Bueno-Cavanillas, and Naomi Cano-Ibáñez.  Benefits of participation in clinical trials: An umbrella review.  _International Journal of Environmental Research and Public Health_ , 19(22):15368, 2022.
  * Brøgger-Mikkelsen et al. (2020) Mette Brøgger-Mikkelsen, Zarqa Ali, John R Zibert, Anders Daniel Andersen, and Simon Francis Thomsen.  Online patient recruitment in clinical trials: systematic review and meta-analysis.  _Journal of medical Internet research_ , 22(11):e22179, 2020.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ , 2023.
  * Calaprice-Whitty et al. (2020) Denise Calaprice-Whitty, Karim Galil, Wael Salloum, Ashkon Zariv, and Bernal Jimenez.  Improving clinical trial participant prescreening with artificial intelligence (ai): a comparison of the results of ai-assisted vs standard methods in 3 oncology trials.  _Therapeutic Innovation & Regulatory Science_, 54:69–74, 2020.
  * Deloitte (2020) Deloitte.  Intelligent clinical trials.  _Deloitte_ , 2020.  URL https://www2.deloitte.com/content/dam/insights/us/articles/22934_intelligent-clinical-trials/DI_Intelligent-clinical-trials.pdf.
  * DNDi (2018) DNDi.  Double-blind, double-dummy, phase 2 randomized, multicenter, proof-of-concept, safety and efficacy trial to evaluate different oral benznidazole monotherapy and benznidazole/e1224 combination regimens for the treatment of adult patients with chronic indeterminate chagas disease.  2018\.  URL https://dndi.org/wp-content/uploads/2020/07/DNDi-Clinical-Trial-Protocol-BENDITA-V5.pdf.
  * FDA (2020) FDA.  Enhancing the diversity of clinical trial populations—eligibility criteria, enrollment practices, and trial designs guidance for industry.  _Center for Biologics Evaluation and Research Center for Drug Evaluation and Research. https://www. fda. gov/regulatory-information/search-fda-guidance-documents/enhancing-diversity-clinical-trial-populations-eligibility-criteria-enrollment-practices-and-trial. Published_ , 2020.
  * Fleming et al. (2023) Scott L Fleming, Alejandro Lozano, William J Haberkorn, Jenelle A Jindal, Eduardo P Reis, Rahul Thapa, Louis Blankemeier, Julian Z Genkins, Ethan Steinberg, Ashwin Nayak, et al.  Medalign: A clinician-generated dataset for instruction following with electronic medical records.  _arXiv preprint arXiv:2308.14089_ , 2023.
  * Gao et al. (2020) Junyi Gao, Cao Xiao, Lucas M. Glass, and Jimeng Sun.  COMPOSE: Cross-Modal Pseudo-Siamese Network for Patient Trial Matching.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20, pages 803–812, New York, NY, USA, August 2020. Association for Computing Machinery.  ISBN 978-1-4503-7998-4.  10.1145/3394486.3403123.  URL https://doi.org/10.1145/3394486.3403123.
  * Gehrmann et al. (2017) Sebastian Gehrmann, Franck Dernoncourt, Yeran Li, Eric T Carlson, Joy T Wu, Jonathan Welt, John Foote Jr, Edward T Moseley, David W Grant, Patrick D Tyler, et al.  Comparing rule-based and deep learning models for patient phenotyping.  _arXiv preprint arXiv:1703.08705_ , 2017.
  * Goel et al. (2023) Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, et al.  Llms accelerate annotation for medical information extraction.  In _Machine Learning for Health (ML4H)_ , pages 82–100. PMLR, 2023.
  * González et al. (2023) Javier González, Cliff Wong, Zelalem Gero, Jass Bagga, Risa Ueno, Isabel Chien, Eduard Orakvin, Emre Kiciman, Aditya Nori, Roshanthi Weerasinghe, et al.  Trialscope a unifying causal framework for scaling real-world evidence generation with biomedical language models.  _arXiv preprint arXiv:2311.01301_ , 2023.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder.  Universal language model fine-tuning for text classification.  _arXiv preprint arXiv:1801.06146_ , 2018.
  * Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_ , 2024.
  * Jiang et al. (2023) Lavender Yao Jiang, Xujin Chris Liu, Nima Pour Nejatian, Mustafa Nasir-Moin, Duo Wang, Anas Abidin, Kevin Eaton, Howard Antony Riina, Ilya Laufer, Paawan Punjabi, et al.  Health system-scale language models are all-purpose prediction engines.  _Nature_ , pages 1–6, 2023.
  * Jin et al. (2023) Qiao Jin, Zifeng Wang, Charalampos S Floudas, Jimeng Sun, and Zhiyong Lu.  Matching patients to clinical trials with large language models.  _ArXiv_ , 2023.
  * Kang et al. (2017) Tian Kang, Shaodian Zhang, Youlan Tang, Gregory W Hruby, Alexander Rusanov, Noémie Elhadad, and Chunhua Weng.  EliIE: An open-source information extraction system for clinical trial eligibility criteria.  _Journal of the American Medical Informatics Association_ , 24(6):1062–1071, November 2017.  ISSN 1067-5027, 1527-974X.  10.1093/jamia/ocx019.  URL https://academic.oup.com/jamia/article/24/6/1062/3098256.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Kong (2019) Hyoun-Joong Kong.  Managing unstructured big data in healthcare system.  _Healthcare informatics research_ , 25(1):1–2, 2019.
  * Kumar et al. (2015) Vishesh Kumar, Amber Stubbs, Stanley Shaw, and Özlem Uzuner.  Creation of a new longitudinal corpus of clinical narratives.  _Journal of biomedical informatics_ , 58:S6–S10, 2015.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.  Efficient memory management for large language model serving with pagedattention.  In _Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles_ , 2023.
  * Leaman et al. (2015) Robert Leaman, Ritu Khare, and Zhiyong Lu.  Challenges in clinical natural language processing for automated disorder normalization.  _Journal of biomedical informatics_ , 57:28–37, 2015.
  * Limited (2017) Leading Biopharm Limited.  A phase ii, randomized, double-blind, placebo-controlled study to evaluate the safety and efficacy of tj301 (fe 999301) administered intravenously in patients with active ulcerative colitis.  2017\.  URL https://classic.clinicaltrials.gov/ProvidedDocs/52/NCT03235752/Prot_000.pdf.
  * Liu et al. (2021) Ruishan Liu, Shemra Rizzo, Samuel Whipple, Navdeep Pal, Arturo Lopez Pineda, Michael Lu, Brandon Arnieri, Ying Lu, William Capra, Ryan Copping, et al.  Evaluating eligibility criteria of oncology trials using real-world data and ai.  _Nature_ , 592(7855):629–633, 2021.
  * Locock and Smith (2011) Louise Locock and Lorraine Smith.  Personal benefit, or benefiting others? deciding whether to take part in clinical trials.  _Clinical Trials_ , 8(1):85–93, 2011.  10.1177/1740774510392257.  URL https://doi.org/10.1177/1740774510392257.  PMID: 21163854.
  * Mirza et al. (2024) Fatima N Mirza, Oliver Y Tang, Ian D Connolly, Hael A Abdulrazeq, Rachel K Lim, G Dean Roye, Cedric Priebe, Cheryl Chandler, Tiffany J Libby, Michael W Groff, et al.  Using chatgpt to facilitate truly informed medical consent.  _NEJM AI_ , page AIcs2300145, 2024.
  * Nievas et al. (2023) Mauro Nievas, Aditya Basu, Yanshan Wang, and Hrituraj Singh.  Distilling large language models for matching patients to clinical trials.  _arXiv preprint arXiv:2312.09958_ , 2023.
  * Nuttall (2012) A Nuttall.  Considerations for improving patient recruitment into clinical trials.  _Clinical Leader Newsletter_ , 2012.  URL http://vertassets.blob.core.windows.net/download/64c39d7e/64c39d7e-c643-457b-aec2-9ff7b65b3ad2/rdprecruitmentwhitepaper.pdf.
  * Oleynik et al. (2019) Michel Oleynik, Amila Kugic, Zdenko Kasáč, and Markus Kreuzthaler.  Evaluating shallow and deep learning strategies for the 2018 n2c2 shared task on clinical text classification.  _Journal of the American Medical Informatics Association_ , 26(11):1247–1254, 2019.
  * OpenAI (2024) OpenAI.  Openai pricing, 2024.  URL https://openai.com/pricing.
  * Penberthy et al. (2012) Lynne T Penberthy, Bassam A Dahman, Valentina I Petkov, and Jonathan P DeShazo.  Effort required in eligibility screening for clinical trials.  _Journal of Oncology Practice_ , 8(6):365–370, 2012.
  * Peng et al. (2023) Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.  Yarn: Efficient context window extension of large language models.  _arXiv preprint arXiv:2309.00071_ , 2023.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019.  URL https://arxiv.org/abs/1908.10084.
  * Stubbs and Uzuner (2015) Amber Stubbs and Özlem Uzuner.  Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/uthealth corpus.  _Journal of biomedical informatics_ , 58:S20–S29, 2015.
  * Stubbs et al. (2019) Amber Stubbs, Michele Filannino, Ergin Soysal, Samuel Henry, and Özlem Uzuner.  Cohort selection for clinical trials: n2c2 2018 shared task track 1.  _Journal of the American Medical Informatics Association_ , 26(11):1163–1171, 2019.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Tu et al. (2024) Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al.  Towards conversational diagnostic ai.  _arXiv preprint arXiv:2401.05654_ , 2024.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.
  * Weng et al. (2011) C. Weng, X. Wu, Z. Luo, M. R. Boland, D. Theodoratos, and S. B. Johnson.  EliXR: an approach to eligibility criteria extraction and representation.  _Journal of the American Medical Informatics Association_ , 18(Supplement 1):i116–i124, December 2011.  ISSN 1067-5027, 1527-974X.  10.1136/amiajnl-2011-000321.  URL https://academic.oup.com/jamia/article-lookup/doi/10.1136/amiajnl-2011-000321.
  * Willard and Louf (2023) Brandon T Willard and Rémi Louf.  Efficient guided generation for llms.  _arXiv preprint arXiv:2307.09702_ , 2023.
  * Wong et al. (2023) Cliff Wong, Sheng Zheng, Yu Gu, Christine Moung, Jacob Abel, Naoto Usuyama, Roshanthi Weerasinghe, Brian Piening, Tristan Naumann, Carlo Bifulco, et al.  Scaling clinical trial matching using large language models: A case study in oncology.  _arXiv preprint arXiv:2308.02180_ , 2023.
  * Woo (2019) Marcus Woo.  An ai boost for clinical trials.  _Nature_ , 573(7775):S100–S100, 2019.
  * Xian et al. (2018) Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata.  Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(9):2251–2265, 2018.
  * Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff.  C-pack: Packaged resources to advance general chinese embedding, 2023.
  * Yuan et al. (2019) Chi Yuan, Patrick B Ryan, Casey Ta, Yixuan Guo, ZirHe2022RethinkingWRan Li, Jill Hardin, Rupa Makadia, Peng Jin, Ning Shang, Tian Kang, and Chunhua Weng.  Criteria2Query: a natural language interface to clinical databases for cohort definition.  _Journal of the American Medical Informatics Association_ , 26(4):294–305, April 2019.  ISSN 1527-974X.  10.1093/jamia/ocy178.  URL https://academic.oup.com/jamia/article/26/4/294/5308980.
  * Zhang et al. (2020) Xingyao Zhang, Cao Xiao, Lucas M. Glass, and Jimeng Sun.  DeepEnroll: Patient-Trial Matching with Deep Embedding and Entailment Prediction.  In _Proceedings of The Web Conference 2020_ , WWW ’20, pages 1029–1037, New York, NY, USA, April 2020. Association for Computing Machinery.  ISBN 978-1-4503-7023-3.  10.1145/3366423.3380181.  URL https://doi.org/10.1145/3366423.3380181.

##  8 Appendix

###  8.1 Compute Environment

Experiments are performed in a local on-prem university compute environment
using 24 Intel Xeon 2.70GHz CPU cores, 8 Nvidia V100 GPUs, 4 Nvidia A100 GPUs,
and 1.48 TB of RAM. All compute environments supported HIPAA-compliant data
protocols.

Figure 4: Confusion matrices for all criteria for best performing model (GPT-4 using the ACIN prompt). Table 4: Criteria-level breakdown for best model (GPT-4 using the ACIN prompt) Criteria | True Value | Prec. | Rec. | F1 | Support
---|---|---|---|---|---
ABDOMINAL | MET | 0.7097 | 0.7333 | 0.7213 | 30
ABDOMINAL | NOT MET | 0.8545 | 0.8393 | 0.8468 | 56
ADVANCED-CAD | MET | 0.8542 | 0.9111 | 0.8817 | 45
ADVANCED-CAD | NOT MET | 0.8947 | 0.8293 | 0.8608 | 41
ALCOHOL-ABUSE | MET | 0.5000 | 0.6667 | 0.5714 | 3
ALCOHOL-ABUSE | NOT MET | 0.9878 | 0.9759 | 0.9818 | 83
ASP-FOR-MI | MET | 0.8590 | 0.9853 | 0.9178 | 68
ASP-FOR-MI | NOT MET | 0.8750 | 0.3889 | 0.5385 | 18
CREATININE | MET | 0.8333 | 0.8333 | 0.8333 | 24
CREATININE | NOT MET | 0.9355 | 0.9355 | 0.9355 | 62
DIETSUPP-2MOS | MET | 0.9286 | 0.8864 | 0.9070 | 44
DIETSUPP-2MOS | NOT MET | 0.8864 | 0.9286 | 0.9070 | 42
DRUG-ABUSE | MET | 0.7500 | 1.0000 | 0.8571 | 3
DRUG-ABUSE | NOT MET | 1.0000 | 0.9880 | 0.9939 | 83
ENGLISH | MET | 0.9863 | 0.9863 | 0.9863 | 73
ENGLISH | NOT MET | 0.9231 | 0.9231 | 0.9231 | 13
HBA1C | MET | 0.9394 | 0.8857 | 0.9118 | 35
HBA1C | NOT MET | 0.9245 | 0.9608 | 0.9423 | 51
KETO-1YR | MET | 0.0000 | 0.0000 | 0.0000 | 0
KETO-1YR | NOT MET | 1.0000 | 1.0000 | 1.0000 | 86
MAJOR-DIABETES | MET | 0.9730 | 0.8372 | 0.9000 | 43
MAJOR-DIABETES | NOT MET | 0.8571 | 0.9767 | 0.9130 | 43
MAKES-DECISIONS | MET | 0.9651 | 1.0000 | 0.9822 | 83
MAKES-DECISIONS | NOT MET | 0.0000 | 0.0000 | 0.0000 | 3
MI-6MOS | MET | 1.0000 | 0.7500 | 0.8571 | 8
MI-6MOS | NOT MET | 0.9750 | 1.0000 | 0.9873 | 78
Table 5: Eligibility criteria in the 2018 n2c2 cohort selection dataset, along with the percentage of patients who meet each criterion in the test set. Definitions taken verbatim from (Stubbs et al., 2019). Criteria |  Definition |  Prevalence
---|---|---
ABDOMINAL |  History of intra-abdominal surgery, small or large intestine resection, or small bowel obstruction |  0.35
ADVANCED-CAD |  Advanced cardiovascular disease (CAD). For the purposes of this annotation, we define “advanced” as having 2 or more of the following: • Taking 2 or more medications to treat CAD • History of myocardial infarction (MI) • Currently experiencing angina • Ischemia, past or present |  0.52
ALCOHOL-ABUSE |  Current alcohol use over weekly recommended limits |  0.03
ASP-FOR-MI |  Use of aspirin to prevent MI |  0.79
CREATININE |  Serum creatinine $>$ upper limit of normal |  0.28
DIETSUPP-2MOS |  Taken a dietary supplement (excluding vitamin D) in the past 2 months |  0.51
DRUG-ABUSE |  Drug abuse, current or past |  0.03
ENGLISH |  Patient must speak English |  0.85
HBA1C |  Any hemoglobin A1c (HbA1c) value between 6.5% and 9.5% |  0.41
KETO-1YR |  Diagnosis of ketoacidosis in the past year |  0.00
MAJOR-DIABETES |  Major diabetes-related complication. For the purposes of this annotation, we define “major complication” (as opposed to “minor complication”) as any of the following that are a result of (or strongly correlated with) uncontrolled diabetes: • Amputation • Kidney damage • Skin conditions • Retinopathy • nephropathy • neuropathy |  0.50
MAKES-DECISIONS |  Patient must make their own medical decisions |  0.97
MI-6MOS |  MI in the past 6 months |  0.09
Table 6: ”Improved” criteria definitions used in our prompts. Criteria |  Definition
---|---
ABDOMINAL |  History of intra-abdominal surgery. This could include any form of intra-abdominal surgery, including but not limited to small/large intestine resection or small bowel obstruction
ADVANCED-CAD |  Advanced cardiovascular disease (CAD). For the purposes of this annotation, we define “advanced” as having 2 or more of the following: (a) Taking 2 or more medications to treat CAD (b) History of myocardial infarction (MI) (c) Currently experiencing angina (d) Ischemia, past or present. The patient must have at least 2 of these categories (a,b,c,d) to meet this criterion, otherwise the patient does not meet this criterion. For ADVANCED-CAD, be strict in your evaluation of the patient – if they just have cardiovascular disease, then they do not meet this criterion.
ALCOHOL-ABUSE |  Current alcohol use over weekly recommended limits
ASP-FOR-MI |  Use of aspirin for preventing myocardial infarction (MI).
CREATININE |  Serum creatinine level above the upper normal limit
DIETSUPP-2MOS |  Consumption of a dietary supplement (excluding vitamin D) in the past 2 months. To assess this criterion, go through the list of medications_and_supplements taken from the note. If a substance could potentially be used as a dietary supplement (i.e. it is commonly used as a dietary supplement, even if it is not explicitly stated as being used as a dietary supplement), then the patient meets this criterion. Be lenient and broad in what is considered a dietary supplement. For example, a ’multivitamin’ and ’calcium carbonate’ should always be considered a dietary supplement if they are included in this list.
DRUG-ABUSE |  Current or past history of drug abuse
ENGLISH |  Patient speaks English. Assume that the patient speaks English, unless otherwise explicitly noted. If the patientś language is not mentioned in the note, then assume they speak English and thus meet this criteria.
HBA1C |  Any hemoglobin A1c (HbA1c) value between 6.5% and 9.5%
KETO-1YR |  Diagnosis of ketoacidosis within the past year
MAJOR-DIABETES |  Major diabetes-related complication. Examples of “major complication” (as opposed to “minor complication”) include, but are not limited to, any of the following that are a result of (or strongly correlated with) uncontrolled diabetes: • Amputation • Kidney damage • Skin conditions • Retinopathy • nephropathy • neuropathy. Additionally, if multiple conditions together imply a severe case of diabetes, then count that as a major complication.
MAKES-DECISIONS |  Patient must make their own medical decisions. Assume that the patient makes their own medical decisions, unless otherwise explicitly noted. There is no information provided about the patientś ability to make their own medical decisions, then assume they do make their own decisions and therefore meet this criteria
MI-6MOS |  Myocardial infarction (MI) within the past 6 months
Figure 5: The prompt template we use for the All Criteria setting, i.e. the
ACIN and ACAN strategies. The only variables injected into this prompt are
note, section_criteria, and current_date. Respectively, these contain the text
from the patient’s EHR, the definitions of the inclusion criteria being
evaluated, and the date of the last note in the patient’s EHR Figure 6: The
prompt template we use for the Individual Criteria setting, i.e. the ICIN and
ICAN strategies. The only variables injected into this prompt are note,
section_criteria, and current_date. Respectively, these contain the text from
the patient’s EHR, the definitions of the inclusion criteria being evaluated,
and the date of the last note in the patient’s EHR.