# Accelerating Clinical Evidence Synthesis with Large Language Models

Zifeng Wang1, Lang Cao1, Benjamin Danek1, Yichi Zhang1, Qiao Jin2,
Zhiyong Lu2, Jimeng Sun1,3#
1 Department of Computer Science, University of Illinois Urbana-Champaign,
Champaign, IL
2 National Center for Biotechnology Information, National Library of Medicine,
Bethesda, MD
3 Carle Illinois College of Medicine, University of Illinois Urbana-Champaign,
Champaign, IL
#Corresponding authors. Emails<EMAIL_ADDRESS>
###### Abstract

Automatic medical discovery by AI is a dream of many. One step toward that
goal is to create an AI model to understand clinical studies and synthesize
clinical evidence from the literature. Clinical evidence synthesis currently
relies on systematic reviews of clinical trials and retrospective analyses
from medical literature. However, the rapid expansion of publications presents
challenges in efficiently identifying, summarizing, and updating evidence. We
introduce TrialMind, a generative AI-based pipeline for conducting medical
systematic reviews, encompassing study search, screening, and data extraction
phases. We utilize large language models (LLMs) to drive each pipeline
component while incorporating human expert oversight to minimize errors. To
facilitate evaluation, we also create a benchmark dataset TrialReviewBench, a
custom dataset with 870 annotated clinical studies from 25 meta-analysis
papers across various medical treatments. Our results demonstrate that
TrialMind significantly improves the literature review process, achieving high
recall rates (0.897-1.000) in study searching from over 20 million PubMed
studies and outperforming traditional language model embeddings-based methods
in screening (Recall@20 of 0.227-0.246 vs. 0.000-0.102). Furthermore, our
approach surpasses direct GPT-4 performance in result extraction, with
accuracy ranging from 0.65 to 0.84. We also support clinical evidence
synthesis in forest plots, as validated by eight human annotators who
preferred TrialMind over the GPT-4 baseline with a winning rate of 62.5%-100%
across the involved reviews. Our findings suggest that an LLM-based clinical
evidence synthesis approach, such as TrialMind, can enable reliable and high-
quality clinical evidence synthesis to improve clinical research efficiency.

## Introduction

Clinical evidence is crucial for supporting clinical practices and advancing
new drug development. It is primarily gathered through retrospective analysis
of real-world data or through prospective clinical trials that assess new
interventions on humans. Researchers often conduct systematic reviews to
consolidate evidence from various studies in the literature [1]. However,
conducting a systematic literature review is expensive and time-consuming,
requiring five experts to analyze 195 publications over 67 weeks on average
[2]. Moreover, the fast growth of clinical study databases means that the
information in these published clinical reviews becomes outdated rapidly [3].
This situation underscores the urgent need to streamline the systematic review
processes to produce systematic and timely clinical evidence from the
extensive medical literature [4].

Large language models (LLMs) excel at information processing and generating,
showing great promise in streamlining the clinical evidence synthesis process.
Adapting LLMs to new tasks can be done by providing them the task definition
and examples as the text inputs (namely ‘prompts’) without the need of
retraining the model [5]. Researchers have tried to adopt LLMs for individual
tasks in literature review [6, 7, 8, 9, 10]. For example, by enhancing inputs
with multiple papers, LLMs can summarize findings to answer medical questions
[9]. This strategy helps reduce hallucinations but still faces challenges when
the input studies do not adequately support the posed questions, which
requires more effort in literature search and screening steps. Furthermore,
LLMs often demonstrate limitations in reasoning with numerical data found in
clinical studies. The qualitative clinical evidence generated using raw paper
content via prompting can be overly generic, lack critical information, or
misinterpret results [11]. Therefore, we propose developing an LLM-driven
pipeline to assist the entire workflow, including formulating research
questions, conducting literature mining, extracting information, and
synthesizing clinical evidence. This includes a comprehensive evaluation of
LLMs in the entire process, which was not well-explored [12].

Figure 1: The TrialMind framework has four components: a) Utilizing input PICO
elements, TrialMind generates key terms to construct Boolean queries for
retrieving studies from literature databases. b) TrialMind formulates
eligibility criteria, which users can edit to provide context for LLMs during
eligibility predictions. Users can then select studies based on these
predictions and rank their relevance by aggregating them. c) TrialMind
processes the descriptions of target data fields to extract and output the
required information as structured data. d) TrialMind extracts findings from
the studies and collaborates with users to synthesize the clinical evidence.

This study aims to fulfill the potential of AI in helping medical
practitioners with the entire clinical evidence synthesis process using AI. We
demonstrate how our method TrialMind is optimized for the clinical evidence
synthesis task via: (1) generating boolean queries to search from the
literature; (2) building inclusion and exclusion criteria to screen through
the found studies; (3) extracting key information, including study protocols,
methods, participant baselines, etc., from unstructured documents, per users’
request; and (4) synthesizing high-quality clinical evidence. Unlike previous
approaches, TrialMind integrates LLMs into an AI pipeline by breaking down the
task into multiple steps that align with expert practices. This approach
maintains flexibility by involving humans in the loop to monitor, edit, and
verify all intermediate steps and the final synthesized outputs.

This paper curated a benchmark dataset TrialReviewBench for a comprehensive
evaluation. The dataset includes 870 involved clinical studies and more than
50,000 identified studies from 25 meta-analyses. It also consists of manual
annotations of 1,334 study characteristics and 1,049 study results. We show
that the TrialMind is able to 1) retrieve a complete list of target studies
from the literature, 2) follow the specified eligibility criteria to rank the
most relevant studies at the top, and 3) achieve high accuracy in extracting
information and clinical outcomes from unstructured documents based on user
requests. Additionally, the extracted clinical outcomes can be standardized as
input for meta-analysis (e.g., forest plots). We conducted a human evaluation
of the synthesized evidence to demonstrate the potential value of TrialMind in
practice.

## Results

### Creating TrialReviewBench from medical literature

A systematic understanding of cancer treatments is crucial for oncology drug
discovery and development. Using a large list of cancer treatments from the
National Cancer Institute (NCI) [13], we curated a dataset TrialReviewBench of
systematic reviews. To ensure data quality, we crafted comprehensive queries
with automatic filtering and manual screening. For each review, we obtained
the list of studies with their PubMed IDs, retrieved their full content, and
extracted study characteristics and clinical outcomes. We followed PubMed’s
usage policy and guidelines during retrieval. Further manual checks were
performed to correct inaccuracies, eliminate invalid and duplicate papers, and
refine the text for clarity (Methods). The final TrialReviewBench dataset
consists of 870 studies involved in 25 reviews (Fig. 2a), including (1)
Brachytherapy [14, 15, 16]: 3 reviews with 189 involved studies and 5,811
identified studies; (2) Chemotherapy [17, 18, 19]: 3 reviews with 126 involved
studies and 5,874 identified studies; (3) Hyperthermia [20, 21, 22]: 3 reviews
with involved 28 studies and 5,972 identified studies; (4) Hormone Therapy
[23, 24, 25]: 3 reviews with 30 involved studies and 5,970 identified studies;
(5) Cancer Vaccine [26]: one review with 6 involved studies and 1,994 related
studies; (6) Immune Checkpoint Inhibitors [27, 28, 29]: 3 reviews with 187
involved studies and 5,813 identified studies; (7) Immune System Modulators
[30]: one review with 9 involved studies and 1,991 identified studies; (8)
Monoclonal Antibodies [31, 32, 33]: 3 reviews with 228 involved studies and
5,772 identified studies; (9) T-cell Transfer Therapy [34, 35]: 2 reviews with
36 involved studies and 3,964 identified studies; (10) Stem Cell Transplant
[36, 37, 38]: 3 reviews with 31 involved studies and 5,969 identified studies.
The detailed characteristics of these studies are in Extended Table
LABEL:tab:data_stats.

### Build an LLM-driven system for clinical evidence synthesis

Large language models (LLMs) excel in adapting to new tasks when provided with
task-specific prompts while often struggling with complex tasks that require
multi-step reasoning. Additionally, interacting and collaborating with LLMs
can be problematic due to their opaque nature and the complexity of debugging
[39]. In this study, we developed TrialMind that decomposes the clinical
evidence synthesis process into four main tasks (Fig. 1 and Methods).
Initially, using the provided research question enriched with population,
intervention, comparison, and outcome (PICO) elements, TrialMind conducts a
comprehensive search from the literature. It also works with users to build
the criteria for study screening and rank the studies. Next, TrialMind browses
the study details to extract the study characteristics and pertinent findings.
To ensure the accuracy and integrity of the data, each output is linked to the
sources for manual inspection. In the final step, TrialMind standardizes the
clinical outcomes for meta-analysis.

Figure 2: Literature search experiment results. a, Total number of involved
studies and participants of the review papers across different topics. b, The
TrialMind’s interface for users to retrieve studies. c, The Recall of the
search results for reviews across four topics. The bar heights indicate the
Recall, and the star indicates the number of studies found. d, Scatter plots
of the Recall against the number of ground-truth studies. Each scatter
indicates the results of one review. Regression estimates are displayed with
the 95% CIs in blue or purple.

###  TrialMind can make a comprehensive retrieval of studies from the
literature

Finding relevant studies from medical literature like PubMed, which contains
over 37 million entries, can be challenging. Typically, this requires the
research expertise to craft complex queries that comprehensively cover
pertinent studies. The challenge lies in balancing the specificity of queries:
too stringent, and the search may miss relevant studies; too broad, and it
becomes impractical to manually screen the overwhelming number of results.
Previous approaches propose to prompt LLMs to generate the searching query in
one pass [6], which can induce incomplete searching results due to the limited
knowledge of LLMs. In contrast, TrialMind is designed to produce comprehensive
queries through a pipeline that includes query generation, augmentation, and
refinement. It also provides users with the ability to make adjustments (Fig.
2b).

The dataset involving clinical studies spanning ten cancer treatment areas was
used for evaluation (Fig. 2a). For each review, we collected the involved
studies’ PubMed IDs as the ground-truth and measured the Recall, i.e., how
many ground-truth studies are found in the search results. We created two
baselines as the comparison: GPT-4 and Human. The GPT-4 baseline makes a
guided prompt for LLMs to generate the boolean queries [6]. It represents the
common way of prompting LLMs for literature search query generation. The Human
baseline represents a way where the key terms from PICO elements are extracted
manually and expanded, referring to UMLS [40], to construct the search
queries.

Overall, TrialMind achieved a Recall of 0.921 on average for all reviews in
TrialReviewBench, meaning it can capture all target studies most of the time.
By contrast, the GPT-4 baseline yielded Recall = 0.079, and the Human baseline
yielded Recall = 0.230. We divided the search results across four topics
determined by the treatments studied in each review (Fig. 2c). Our analysis
showed that TrialMind can identify many more studies than the baselines. For
instance, TrialMind achieved $\text{Recall}=0.914$ ($N$ studies = 28,863) for
Immunotherapy-related reviews, while the GPT-4 baseline got
$\text{Recall}=0.1$ ($N$ studies = 22), and the Human baseline got
$\text{Recall}=0.154$ ($N$ studies = 153), respectively. In
Radiation/Chemotherapy, TrialMind achieved $\text{Recall}=0.897$, the GPT-4
baseline got $\text{Recall}=0.017$, and the Human baseline got
$\text{Recall}=0.304$. In Hormone Therapy, TrialMind achieved
$\text{Recall}=0.924$, the GPT-4 baseline got $\text{Recall}=0.100$, and the
Human baseline got $\text{Recall}=0.150$. In Hyperthermia, TrialMind achieved
$\text{Recall}=1.000$, the GPT-4 baseline got $\text{Recall}=0.133$, and the
Human baseline got $\text{Recall}=0.683$. These results demonstrate that
regardless of the search task’s complexity, as indicated by the variability in
the Human baseline, TrialMind consistently retrieves nearly all target studies
from the PubMed database. This robust performance provides a solid foundation
for accurately identifying target studies in the screening phase.

Furthermore, we made scatter plots of Recall versus the number of target
studies for each review (Fig. 2d). The hypothesis was that an increase in
target studies correlates with the difficulty of achieving complete coverage.
Our findings reveal that TrialMind consistently maintained a high Recall,
nearing 1 in many instances and never falling below 0.7, significantly
outperforming the best baselines across all 25 reviews. A trend of declining
Recall with an increasing number of target studies was confirmed through
regression analysis. For example, with fewer than 20 target studies, TrialMind
achieved perfect Recall for most reviews, while the GPT-4 baseline struggled,
showing Recall close to 0, and the Human baseline results varied between 0 and
0.85. As the number of target studies increased, the Human and GPT-4
baselines’ Recall decreased to nearly zero. In contrast, TrialMind
demonstrated remarkable resilience, showing minimal variation in performance
despite the increasing number of target studies. For instance, in a review
involving 141 studies, TrialMind achieved a Recall of 0.99, while the GPT-4
and Human baselines obtained a Recall of 0.02 and 0, respectively.

Figure 3: Literature screen experiment results. a, Streamline study screening
using TrialMind with human in the loop. b, Ranking performances for
Recall@20/50 within across therapeutic areas. c, Recall@20 and Recall@50 for
TrialMind and selected baselines. d, Effect of individual criterion on the
ranking results. e, Ranking performance for $\text{Recall}@K$ with varying $K$
in four topics. Shaded areas are $95\%$ confidence interval.

###  TrialMind enhances literature screening and ranking

Typically, human experts manually sift through thousands of retrieved studies
to select relevant ones for inclusion in a systematic review. This process
adheres to the PRISMA statement [41], which involves creating a list of
eligibility criteria and assessing each study’s eligibility. TrialMind
streamlines this task through a three-step approach: (1) it generates a set of
inclusion criteria, which are subject to user’s adjustments; (2) it applies
these criteria to evaluate the study’s eligibility, denoted by $\\{-1,0,1\\}$
where $-1$ and $1$ represent eligible and non-eligible, and $0$ represents
unknown/uncertain, respectively; and (3) it ranks the studies by aggregating
the eligibility predictions, where the aggregation strategy can be specified
by users (Fig. 3a). We took a summation of the criteria-level eligibility
predictions as the study-level relevance prediction scores for ranking. As
such, TrialMind provides a rationale for the relevance scores by detailing the
eligibility predictions for each criterion.

We chose MPNet [42] and MedCPT [43] as the general domain and medical domain
ranking baselines, respectively. These methods compute study relevance by the
cosine similarity between the encoded PICO elements as the query and the
encoded study’s abstracts. We also set a Random baseline that randomly samples
from candidates. We created the evaluation data based on the search results in
the first stage. For each review, we mixed the target studies with the other
found studies to build a candidate set of 2,000 studies for ranking.
Discriminating the target studies from the other candidates is challenging
since all candidates meet the search queries, meaning they most probably
investigate the relevant therapies or conditions. We evaluated the ranking
performance using the Recall@20 and Recall@50 metrics. The concatenation of
the title and abstract of each study is used for all methods as inputs.

We found that TrialMind greatly improved ranking performances, with the fold
changes over the best baselines ranging from 1.4 to 24 across four topics
(Table 3c). For instance, for the Hormone Therapy topic, TrialMind obtained
<EMAIL_ADDRESS>and<EMAIL_ADDRESS>In the Hyperthermia
topic, TrialMind obtained<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>In the Immunotherapy topic, TrialMind obtained
<EMAIL_ADDRESS>and<EMAIL_ADDRESS>In the
Radiation/Chemotherapy topic, TrialMind obtained<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>TrialMind demonstrated the largest advantage over
baselines on the Hormone Therapy topic ($\text{fold change}=24$ and $10.5$ for
$\text{Recall}@20$ and $\text{Recall}@50$ compared to the best baselines). In
contrast, other baselines exhibit significant variability across different
topics. The general domain baseline MPNet was the worst as it performed
similarly to the Random baseline in $\text{Recall}@20$. MedCPT showed marginal
improvement over MPNet in the last three topics, while both failed to capture
any target studies in Hormone Therapy.

Furthermore, TrialMind demonstrated significant improvements over the
baselines across various therapeutic areas (Fig. 3b). For example, in “Cancer
Vaccines” and “Hormone Therapy,” TrialMind substantially increased
$\text{Recall}@50$, achieving 33.33-fold and 10.53-fold improvements,
respectively, compared to the best-performing baseline. TrialMind generally
attained a fold change greater than 2 (ranging from 1.57 to 33.33). Despite
the challenge of selecting from a large pool of candidates ($n=2,000$) where
candidates were very similar, TrialMind identified an average of 43% of target
studies within the top 50.

We compared TrialMind to MedCPT and MPNet for $\text{Recall}@K$ ($K$ in 10 to
200) to gain insight into how $K$ influences the performances (Fig. 3e). We
found TrialMind can capture most of the target studies when $K=200$, as it
obtained<EMAIL_ADDRESS>in Immunotherapy<EMAIL_ADDRESS>in Radiation/Chemotherapy<EMAIL_ADDRESS>in Hormone Therapy,
<EMAIL_ADDRESS>in Hyperthermia, respectively. The improvement over
baselines is especially significant in Hormone Therapy and Hyperthermia, as
the other baselines did not outperform the Random performances all the time.
For instance, in Hormone Therapy, MedCPT’s<EMAIL_ADDRESS>and
MPNet’s<EMAIL_ADDRESS>(Random’s $\text{Recall}@200=0.100$).

To thoroughly assess the quality of these criteria and their impact on ranking
performance, we conducted a leave-one-out analysis to calculate
$\Delta\text{Recall}@200$ for each criterion (Fig. 3d). The
$\Delta\text{Recall}@200$ metric measures the difference in ranking
performance with and without a specific criterion, with a larger value
indicating superior criterion quality. Our findings revealed that most
criteria positively influenced ranking performances, as the negative influence
criteria are $n=1$ in Hormone Therapy, $n=1$ in Hyperthermia, $n=5$ in
Radiation/Chemotherapy, and $n=7$ in Immunotherapy. Additionally, we
identified redundancies among the generated criteria, as those with
$\Delta\text{Recall}@200=0$ were the most frequently observed. This redundancy
likely stems from some criteria covering similar eligibility aspects, thus not
impacting performance when one is omitted.

Figure 4: Data and result extraction experiment results. a, Streamline study
information extraction using TrialMind. b, Data extraction accuracy within
each field type across four topics. c, Confusion matrix showing the
hallucination and missing rates in the data extraction results. d, Result
extraction accuracy across topics. e, Result extraction accuracy across
clinical endpoints. f, Error analysis of the result extraction. g, Streamline
result extraction using TrialMind.

###  TrialMind scales data and result extraction from unstructured documents

TrialMind leverages LLMs to streamline extracting study characteristics such
as target therapies, study arm design, and participants’ baseline information
from involved studies. Specifically, TrialMind refers to the field names and
the descriptions from users and use the full content of the study documents in
PDF or XML formats as inputs (Fig. 4a). When the free full content is
unavailable, TrialMind accepts the user-uploaded content as the input. We
developed an evaluation dataset by converting the study characteristic tables
from each review paper into data points. Our dataset comprises 1,334 target
data points, including 696 on study design, 353 on population features, and
285 on results. We assessed the data extraction performance using the Accuracy
metric.

TrialMind demonstrated strong extraction performance across various topics
(Fig. 4b): it achieved an accuracy of $\text{ACC}=0.78$ (95% confidence
interval (CI) = 0.75–0.81) in the Immunotherapy topic, $\text{ACC}=0.77$ (95%
CI = 0.72-0.82) in the Radiation/Chemotherapy topic, $\text{ACC}=0.72$ (95% CI
= 0.63-0.80) in the Hormone Therapy topic, and $\text{ACC}=0.83$ (95% CI =
0.74-0.90) in the Hyperthermia topic. These results indicate that TrialMind
can provide a solid initial data extraction, which human experts can refine.
Importantly, each output can be cross-checked by the linked original sources,
facilitating verification and further investigation.

Diving deeper into the accuracy across different types of fields, we observed
varying performance levels. It performed best in extracting study design
information, followed by population details, and showed the lowest accuracy in
extracting results (Fig. 4b). For example, in the Immunotherapy topic,
TrialMind achieved an accuracy of $\text{ACC}=0.95$ (95% CI = 0.92-0.96) for
study design, $\text{ACC}=0.74$ (95% CI = 0.67-0.80) for population data, and
$\text{ACC}=0.42$ (95% CI = 0.36-0.49) for results. This variance can be
attributed to the prevalence of numerical data in the fields: fields with more
numerical data are typically harder to extract accurately. Study design is
mostly described in textual format and is directly presented in the documents,
whereas population and results often include numerical data such as the number
of patients or gender ratios. Results extraction is particularly challenging,
often requiring reasoning and transformation to capture values accurately.
Given these complexities, it is advisable to scrutinize the extracted
numerical data more carefully.

We also evaluated the robustness of TrialMind against hallucinations and
missing information (Fig. 4c). We constructed a confusion matrix detailing
instances of hallucinations: false positives (FP) where TrialMind generated
data not present in the input document and false negatives (FN) where it
failed to extract available target field information. We observed that
TrialMind achieved a precision of $\text{Precision}=0.994$ for study design,
$\text{Precision}=0.966$ for population, and $\text{Precision}=0.862$ for
study results. Missing information was slightly more common than
hallucinations, with TrialMind achieving recall rates of $\text{Recall}=0.946$
for study design, $\text{Recall}=0.889$ for population, and
$\text{Recall}=0.930$ for study results. The incidence of both hallucinations
and missing information was generally low. However, hallucinations were
notably more frequent in study results; this often occurred because LLMs could
confuse definitions of clinical outcomes, for example, mistaking ‘overall
response’ for ‘complete response.’ Nevertheless, such hallucinations are
typically manageable, as human experts can easily identify and correct them
while reviewing the referenced material.

The challenges in extracting study results primarily stem from (1) identifying
the locations that describe the desired outcomes from lengthy papers, (2)
accurately extracting relevant numerical values such as patient numbers, event
counts, durations, and ratios from the appropriate patient groups, and (3)
performing the correct calculations to standardize these values for meta-
analysis. In response to these complexities, we developed a specialized
pipeline for result extraction (Fig. 4g), where users provide the interested
outcome and the cohort definition. TrialMind offers a transparent extraction
workflow, documenting the sources of results along with the intermediate
reasoning and calculations.

Figure 5: Human evaluation of the synthesized clinical evidence. a, The study
design compares the synthesized clinical evidence from the baseline and
TrialMind via human evaluation. b, Winning rate of TrialMind against the
GPT-4+Human baseline across studies. c, Violin plots of the ratings across
studies. Each plot is tagged with the mean ratings (95% CI) from all the
annotators. d,Violin plots of the ratings across annotators with different
expertise levels. Each plot is tagged with the mean ratings (95% CI) from all
the studies.

We compared TrialMind against two generalist LLM baselines, GPT-4 and Sonnet,
which were prompted to extract the target outcomes from the full content of
the study documents. Since the baselines can only make text extractions, we
manually convert them into numbers suitable for meta-analysis [44]. This made
very strong baselines since they combined LLM extraction with human post-
processing. We assessed the performance using the Accuracy metric.

The evaluation conducted across four topics demonstrated the superiority of
TrialMind (Fig. 4d). Specifically, in Immunotherapy, TrialMind achieved an
accuracy of $\text{ACC}=0.70$ (95% CI 0.62-0.77), while GPT-4 scored
$\text{ACC}=0.54$ (95% CI 0.45-0.62). In Radiation/Chemotherapy, TrialMind
reached $\text{ACC}=0.65$ (95% CI 0.51-0.76), compared to GPT-4’s
$\text{ACC}=0.52$ (95% CI 0.39-0.65). For Hormone Therapy, TrialMind achieved
$\text{ACC}=0.80$ (95% CI 0.58-0.92), outperforming GPT-4, which scored
$\text{ACC}=0.50$ (95% CI 0.30-0.70). In Hyperthermia, TrialMind obtained an
accuracy of $\text{ACC}=0.84$ (95% CI 0.71-0.92), significantly higher than
GPT-4’s $\text{ACC}=0.52$ (95% CI 0.39-0.65). The breakdowns of evaluation
results by the most frequent types of clinical outcomes (Fig. 4e) showed
TrialMind got fold changes in accuracy ranging from 1.05 to 2.83 and a median
of 1.50 over the best baselines. This enhanced effectiveness is largely
attributable to TrialMind’s ability to accurately identify the correct data
locations and apply logical reasoning, while the baselines often produced
erroneous initial extractions.

We analyzed the error cases in our result extraction experiments and
identified four primary error types (Fig. 4f). The most common error was
‘Inaccurate’ extraction (n=36), followed by ‘Extraction failure’ (n=27),
‘Unavailable data’ (n=10), and ‘Hallucinations’ (n=3). ‘Inaccurate’
extractions often occurred due to multiple sections ambiguously describing the
same field. For example, a clinical study might report the total number of
participants receiving CAR-T therapy early in the document and later provide
outcomes for a subset with non-small cell lung cancer (NSCLC). The specific
results for NSCLC patients are crucial for reviews focused on this subgroup,
yet the presence of general data can lead to confusion and inaccuracies in
extraction. ‘Extraction failure’ and ‘Unavailable data’ both illustrate
scenarios where TrialMind could not retrieve the information. The latter case
particularly showcases TrialMind’s robustness against hallucinations, as it
failed to extract data outside the study’s main content, such as in
appendices, which were not included in the inputs. Furthermore, errors caused
by hallucinations were minor. The outputs were easy to identify and correct
through manual inspection since no references were provided.

###  TrialMind synthesizes clinical evidence from extracted results

We engaged human annotators to assess the quality of synthesized clinical
evidence presented in forest plots, a format commonly used in systematic
reviews to report meta-analysis results (Fig. 5a). We selected five systematic
review studies as benchmarks and referenced the clinical evidence reported in
the target studies. The baseline used GPT-4 with a simple prompting to extract
the relevant text pieces that report the target outcome of interest (Methods).
Manual calculations were necessary to standardize the data for meta-analysis.
In contrast, TrialMind automated the extraction and standardization. Each
annotator was asked to evaluate the evidence quality by comparing it against
the evidence reported in the target review and deciding which method,
TrialMind or the baseline, produced superior results. Additionally, they rated
the quality of the synthesized clinical evidence on a scale of 1 to 5. The
assignment of our method and the baseline was randomized to ensure
objectivity.

The evaluation highlighted TrialMind’s superior performance compared to the
direct use of GPT-4 for clinical evidence synthesis (Fig. 5b). We calculated
the winning rate of TrialMind versus the baseline across the five studies. The
results indicate a consistent preference by annotators for the evidence
synthesized by TrialMind over that of the baseline. Specifically, TrialMind
achieved winning rates of 87.5%, 100%, 62.5%, 62.5%, and 81.2%, respectively.
The baseline’s primary shortcoming stemmed from the initial extraction step,
where GPT-4 often failed to identify the relevant sources without well-crafted
prompting. Therefore, the subsequent manual post-processing was unable to
rectify these initial errors.

In addition, we illustrated the ratings of TrialMind and the baseline across
studies (Fig. 5c). We found TrialMind was competent as the GPT-4+Human
baseline and outperformed the baseline in many scenarios. For example,
TrialMind obtained the mean rating of 4.25 (95% CI 3.93-4.57) in Study #1
while the baseline obtained 3.50 (95% CI 3.13-3.87). In Study #2, TrialMind
yielded 3.50 (95% CI 3.13-3.87) while the baseline yielded 1.25 (95% CI
0.93-1.57). The performance of the two methods was comparable in the remaining
three studies. These results highlight TrialMind as a highly effective
alternative to conventional LLM usage in evidence synthesis, streamlining data
extraction and processing while maintaining the critical benefit of human
oversight.

Finally, we requested that annotators self-assess their expertise level in
clinical studies, classifying themselves into three categories: ‘Basic’,
‘Familiar’, and ‘Advanced’. The typical profile ranges from computer
scientists at the basic level to medical doctors at the advanced level. We
then analyzed the ratings given to both methods across these varying expertise
levels (Fig. 5d). We consistently observed higher ratings for TrialMind than
the baseline across all groups. Annotators with basic knowledge tended to
provide more conservative ratings, while those with more advanced expertise
offered a wider range of evaluations. For instance, the ‘Basic’ group provided
average ratings of 3.67 (95% CI 3.34-3.39) for TrialMind compared to 3.22 (95%
CI 2.79-3.66) for the baseline. The ‘Advanced’ group rated TrialMind at an
average of 3.40 (95% CI 3.16-3.64) and the baseline at 3.07 (95% CI
2.75-3.39).

## Discussion

Clinical evidence forms the bedrock of evidence-based medicine, crucial for
enhancing healthcare decisions and guiding the discovery and development of
new drugs. It often comes from a systematic review of diverse studies found in
the literature, encompassing clinical trials and retrospective analyses of
real-world data. Yet, the burgeoning expansion of literature databases
presents formidable challenges in efficiently identifying, summarizing, and
maintaining the currency of this evidence.

The rapid development of large language models (LLMs) and generative AI
technologies has generated considerable interest in their potential
applications. However, implementing these models in a manner that is
collaborative, transparent, and trustworthy poses significant challenges,
especially in critical areas such as medicine [45]. For instance, when
utilizing LLMs to summarize multiple studies, the summaries often merely echo
the findings verbatim, omit crucial details, and fail to adhere to established
best practices [9].

This study introduces a clinical evidence synthesis pipeline enhanced by LLMs,
named TrialMind. This pipeline is structured in accordance with established
medical systematic review protocols, involving steps such as study searching,
screening, data/result extraction, and evidence synthesis. At each stage,
human experts have the capability to access, monitor, and modify intermediate
outputs. This human oversight helps to eliminate errors and prevents their
propagation through subsequent stages. Unlike approaches that solely depend on
the internal knowledge of LLMs, TrialMind integrates human expertise through
in-context learning and chain-of-thought prompting. Additionally, TrialMind
extends external knowledge sources to its outputs through retrieval-augmented
generation and leveraging external computational tools to enhance the LLM’s
reasoning and analytical capabilities. Comparative evaluations of TrialMind
and traditional LLM approaches have demonstrated the advantages of this system
design in LLM-driven applications within the medical field.

This study also has several limitations. First, despite incorporating multiple
techniques, LLMs may still make errors at any stage. Therefore, human
oversight and verification remain crucial when implementing TrialMind in
practical settings. Second, the prompts used in TrialMind were developed based
on prompt engineering experience, suggesting potential for performance
enhancement through advanced prompt optimization or by fine-tuning the
underlying LLMs to suit specific tasks better. Third, while TrialMind
demonstrated effectiveness in study search, screening, and information
extraction, the dataset used was limited in size due to the high costs
associated with human labeling. Future research could expand on these findings
with larger datasets to further validate the method’s effectiveness. Fourth,
the study coverage was restricted to publicly available sources from PubMed
Central, which provides structured PDFs and XMLs. Many relevant studies are
either not available on PubMed or are in formats that entail OCR algorithms as
preprocessing, indicating a need for further engineering to incorporate
broader data sources. Fifth, although TrialMind illustrated the potential of
using advanced LLMs like GPT-4 to streamline clinical evidence synthesis,
developing techniques to adapt the pipeline for use with other LLMs could
increase its applicability. Finally, while the use of LLMs like GPT-4 can
accelerate study screening and data extraction, the associated costs and
processing times may present bottlenecks in some scenarios. Future
enhancements that improve efficiency or utilize localized, specialized smaller
models could increase practical utility.

LLMs have made significant strides in AI applications. TrialMind exemplifies a
crucial aspect of system engineering in LLM-driven pipelines, facilitating the
practical, robust, and transparent use of LLMs. We anticipate that TrialMind
will benefit the medical AI community by fostering the development of LLM-
driven medical applications and emphasizing the importance of human-AI
collaboration.

## Methods

### Description of the TrialReviewBench Dataset

The overall flowchart for the study identification and screening process in
building TrialReviewBench is illustrated in Extended Fig. 1.

#### Database search and initial filtering

We undertook a comprehensive search on the PubMed database for meta-analysis
papers related to cancer. The Boolean search terms were specifically chosen to
encompass a broad spectrum of cancer-related topics. These terms included
“cancer”, “oncology”, “neoplasm”, “carcinoma”, “melanoma”, “leukemia”,
“lymphoma”, and “sarcoma”. Additionally, we incorporated terms related to
various treatment modalities such as “therapy”, “treatment”, “chemotherapy”,
“radiation therapy”, “immunotherapy”, “targeted therapy”, “surgical
treatment”, and “hormone therapy”. To ensure that our search was exhaustive
yet precise, we also included terms like “meta-analysis” and ”systematic
review” in our search criteria.

This initial search yielded an extensive pool of 46,192 results, reflecting
the vast research conducted in these areas. We applied specific filters to
refine these results and ensure relevance and quality. We focused on articles
where PMC Full text was available and specifically categorized under “Meta-
Analysis”. Further refinement was done by restricting the time frame of
publications to those between January 1, 2020, and January 1, 2023. We also
narrowed our focus to studies conducted on humans and those available in
English. This filtration process was critical in distilling the initial
results into a more manageable and focused collection of 2,691 papers.

#### Refinement

Building upon our initial search, we employed further refinement techniques
using both MeSH terms and specific keywords. The MeSH terms were carefully
selected to target papers precisely relevant to various forms of cancer. These
terms included “cancer”, “tumor”, “neoplasms”, “carcinoma”, “myeloma”, and
“leukemia”. This focused approach using MeSH terms effectively reduced our
selection to 1,967 papers.

To further dive in on papers investigating cancer therapies, we utilized many
keywords derived from the National Cancer Institute’s “Types of Cancer
Treatment” list. This approach was multi-faceted, with each set of keywords
targeting a specific category of cancer therapy. For chemotherapy, we included
terms like “chemotherapy”, “chemo”, and related variations. In the realm of
hormone therapy, we searched for phrases such as ”hormone therapy”, ”hormonal
therapy”, and similar terms. The keyword group for hyperthermia encompassed
terms like “hyperthermia”, “microwave”, “radiofrequency”, and related
technologies. For cancer vaccines, we included keywords such as “cancer
vaccines”, “cancer vaccine”, and other related terms. The search for immune
checkpoint inhibitors and immune system modulators was comprehensive,
including terms like “immune checkpoint inhibitors”, “immunomodulators”, and
various cytokines and growth factors. Lastly, our search for monoclonal
antibodies and T-cell transfer therapy included relevant terms like
“monoclonal antibodies”, “t-cell therapy”, “car-t”, and other related phrases.

The careful application of keyword filtering played a crucial role in
narrowing down our pool of research papers to a more focused and relevant set
of 352. It represents a diverse and meaningful collection of studies in cancer
therapy, highlighting a range of innovative and impactful research within this
field.

#### Manual screening of titles and abstracts

Then, we manually screened titles and abstracts, applying a rigorous
classification and sorting methodology. The remaining papers were first
categorized based on the type of cancer treatment they explored. We then
organized these papers by their citation count to gauge their impact and
relevance in the field. Our selection criteria aimed to enhance the quality
and relevance of our final dataset. We prioritized papers that focused on the
study of treatment effects, such as safety and efficacy, of various cancer
interventions. We preferred studies that compared individual treatments
against a control group, as opposed to those examining the effects of combined
therapies (e.g., Therapy A+B vs. A only). To build a list of representative
meta-analyses, we needed to ensure diversity in the target conditions under
each treatment category.

Further, we favored studies that involved a larger number of individual
studies, providing a broader base of evidence. However, we excluded network
analysis studies and meta-analyses that focused solely on prognostic and
predictive effects, as they did not align with our primary research focus. To
maintain a balanced representation, we limited our selection to a maximum of
three papers per treatment category. This process culminated in a final
dataset comprising 25 papers. This curated collection forms the backbone of
our analysis, ensuring a concentrated and pertinent selection of high-quality
studies directly relevant to our research objectives. The characterstics of
the created dataset is in Extended Table LABEL:tab:data_stats.

### LLM Prompting

Prompting steers LLMs to conduct the target task without training the
underlying LLMs. TrialMind proceeds clinical evidence synthesis in multiple
steps associated with a series of prompting techniques.

#### In-context learning

LLMs exhibit a profound ability to comprehend input requests and adhere to
provided instructions during generation. The fundamental concept of in-context
learning (ICL) is to enable LLMs to learn from examples and task instructions
within a given context at inference time [5]. Formally, for a specific task,
we define $T$ as the task prompt, which includes the task definition, input
format, and desired output format. During a single inference session with
input $X$, the LLM is prompted with $P(T,X)$, where $P(\cdot)$ is a
transformation function that restructures the task definition $T$ and input
$X$ into the prompt format. The output $\hat{X}$ is then generated as
$\hat{X}=\text{LLM}(P(T,X))$.

#### Retrieval-augmented generation

LLMs that rely solely on their internal knowledge often produce erroneous
outputs, primarily due to outdated information and hallucinations. This issue
can be mitigated through retrieval-augmented generation (RAG), which enhances
LLMs by dynamically incorporating external knowledge into their prompts during
generation [46]. We denote $R_{K}(\cdot)$ as the retriever that utilizes the
input $X$ to source relevant contextual information through semantic search.
$R_{K}(\cdot)$ enables the dynamic infusion of tailored knowledge into LLMs at
inference time.

#### Chain-of-thought

Chain-of-though (CoT) guides LLMs in solving a target task in a step-by-step
manner in one inference, hence handling complex or ambiguous tasks better and
inducing more accurate outputs [47]. CoT employs the function
$P_{\text{CoT}}(\cdot)$ to structure the task $T$ into a series of chain-of-
thought steps $\\{S_{1},S_{2},\dots,S_{T}\\}$. As a result, we obtain
$\\{\hat{X}_{S}^{1},\dots,\hat{X}_{S}^{T}\\}=\text{LLM}(P_{\text{CoT}}(T,X))$,
all produced in a single inference session. This is rather critical when we
aim to elicit the thinking process of LLM and urge it in self-reflection to
improve its response. For instance, we may ask LLM to draft the initial
response in the first step and refine it in the second.

#### LLM-driven pipeline

Clinical evidence synthesis involves a multi-step workflow as outlined in the
PRISMA statement [41]. It can be generally outlined as identifying and
screening studies from databases, extracting characteristics and results from
individual studies, and synthesizing the evidence. To enhance each step’s
performance, task-specific prompts can be designed for an LLM to create an
LLM-based module. This results in a chain of prompts that effectively
addresses a complex problem, which we call LLM-driven workflow. Specifically,
this approach breaks down the entire meta-analysis process into a sequence of
$N$ tasks, denoted as $\mathcal{T}=\\{T_{1},\dots,T_{N}\\}$. In the workflow,
the output from one task, $\hat{X}_{n}$, serves as the input for the next,
$\hat{X}_{n+1}=\text{LLM}(P(T_{n},\hat{X}_{n}))$. This modular decomposition
improves LLM performance by dividing the workflow into more manageable
segments, increases transparency, and facilitates user interaction at various
stages.

Incorporating these techniques, the formulation of TrialMind for any subtask
can be represented as:

$\hat{X}_{n+1}=\text{LLM}(P(T_{n},X_{n}),R_{K}(X_{n})),\ \forall n=1,\dots,N,$
(1)

where $R_{K}(\cdot)$ are optional.

### Implementation of TrialMind

All experiments were run in Python v.3.9. Detailed software versions are:
pandas v2.2.2; numpy v1.26.4; scipy v1.13.0; scikit-learn v1.4.1.post1; openai
v1.23.6; langchain v0.1.16; boto3 v1.34.94; pypdf v4.2.0; lxml v5.2.1 and
chromadb v0.5.0 with Python v.3.9.

#### LLMs

We included GPT-4 and Sonnet in our experiments. GPT-4 [48] is regarded as a
state-of-the-art LLM and has demonstrated strong performances in many natural
language processing tasks (version: gpt-4-0125-preview). Sonnet [49] is an LLM
developed by Anthropic, representing a more lightweight but also very capable
LLM (version: anthropic.claude-3-sonnet-20240229-v1:0 on AWS Bedrock). Both
models support long context lengths (128K and 200K), enabling them to process
the full content of a typical PubMed paper in a single inference session.

#### Research question inputs

TrialMind processes research question inputs using the PICO (Population,
Intervention, Comparison, Outcome) framework to define the study’s topic and
scope. In our experiments, the title of the target review paper served as the
general description. Subsequently, we extracted the PICO elements from the
paper’s abstract to detail the specific aspects of the research question.

#### Literature search

TrialMind is tailored to adhere to the established guidelines [41] in
conducting literature search and screening for clinical evidence synthesis. In
the literature search stage, the key is formulating Boolean queries to
retrieve a comprehensive set of candidate studies from databases. These
queries, in general, are a combination of treatment, medication, and outcome
terms, which can be generated by LLM using in-context learning. However,
direct prompting can yield low recall queries due to the narrow range of user
inputs and the LLMs’ tendency to produce incorrect queries, such as generating
erroneous MeSH (Medical Subject Headings) terms [6]. To address these
limitations, TrialMind incorporates RAG to enrich the context with knowledge
sourced from PubMed, and employs CoT processing to facilitate a more
exhaustive generation of relevant terms.

Specifically, the literature search component has two main steps: initial
query generation and then query refinement. In the first step, TrialMind
prompts LLM to create the initial boolean queries derived from the input PICO
to retrieve a group of studies (Prompt in Extended Fig. 3). The abstracts of
these studies then enrich the context for refining the initial queries,
working as RAG. In addition, we used CoT to enhance the refinement by urging
LLMs to conduct multi-step reasoning for self-reflection enhancement (Prompt
in Extended Fig. 4). This process can be described as

$\\{\hat{X}_{S}^{1},\hat{X}_{S}^{2},\hat{X}_{S}^{3}\\}=\text{LLM}(P_{\text{CoT}}(T_{\text{LS}},X,R_{K}(X))),$
(2)

where $X$ denotes the input PICO; $R_{K}(X)$ is the set of abstracts of the
found studies; $T_{\text{LS}}$ is the definition of the query generation task
for literature search. For the output, the first sub-step $\hat{X}_{S}^{1}$
indicates a complete set of terms identified in the found studies; the second
$\hat{X}_{S}^{2}$ indicates the subset of $\hat{X}_{S}^{1}$ by filtering out
the irrelevant; and the third $\hat{X}_{S}^{3}$ indicates the extension of
$\hat{X}_{S}^{2}$ by self-reflection and adding more augmentations. In this
process, LLM will produce the outputs for all three substeps in one pass, and
TrialMind takes $\hat{X}_{S}^{3}$ as the final queries to fetch the candidate
studies.

#### Study screening

TrialMind follows PRISMA to take a transparent approach for study screening.
It creates a set of eligibility criteria based on the input PICO as the basis
for study selection (Prompt in Extended Fig. 5), produced by

$\hat{X}_{\text{EC}}=\text{LLM}(P(T_{\text{EC}},X)),$ (3)

where $\hat{X}_{\text{EC}}=\\{E_{1},E_{2},\dots,E_{M}\\}$ is the $M$ generated
eligibility criteria; $X$ is the input PICO; and $T_{\text{EC}}$ is the task
definition of criteria generation. Users are given the opportunity to modify
these generated criteria, further adjusting to their needs.

Based on $\hat{X}_{\text{EC}}$, TrialMind embarks the parallel processing for
the candidate studies. For $i$-th study $F_{i}$, the eligibility prediction is
made by LLM as (Prompt in Extended Fig. 6)

$\\{I_{i}^{1},\dots,I_{i}^{M}\\}=\text{LLM}(P(F_{i},X,T_{\text{SC}},\hat{X}_{\text{EC}})),$
(4)

where $T_{\text{SC}}$ is the task definition of study screening; $F_{i}$ is
the study $i$’s content; $I_{i}^{m}\in\\{-1,0,1\\},\forall m=1,\dots,M$ is the
prediction of study $i$’s eligibility to the $m$-th criterion. Here, $-1$ and
$1$ mean ineligible and eligible, $0$ means uncertain, respectively. These
predictions offer a convenient way for users to inspect the eligibility and
select the target studies by altering the aggregation strategies. $I_{i}^{m}$
can be aggregated to offer an overall relevance of each study, such as
$\hat{I}_{i}=\sum_{m}I_{i}^{m}$. Users are also encouraged to extend the
criteria set or block the predictions of some criteria to make customized
rankings during the screening phase.

#### Data extraction

Study data extraction is an open information extraction task that requires the
model to extract specific information based on user inputs and handle long
inputs, such as the full content of a paper. LLMs are particularly well-suited
for this task because (1) they can perform zero-shot learning via in-context
learning, eliminating the need for labeled training data, and (2) the most
advanced LLMs can process extremely long inputs. As such, the TrialMind
framework is engineered to streamline data extraction from structured or
unstructured study documents using LLMs.

For the specified data fields to be extracted, TrialMind prompts LLMs to
locate and extract the relevant information (Prompt in Extended Fig. 7). These
data fields include (1) study characteristics such as study design, sample
size, study type, and treatment arms; (2) population baselines; and (3) study
findings. In general, the extraction process can be described as

$\\{\hat{X}^{1}_{\text{EX}},\dots,\hat{X}^{K}_{\text{EX}}\\}=\text{LLM}(P(F,C,T_{\text{EX}})),$
(5)

where $F$ represents the full content of a study; $T_{\text{EX}}$ defines the
task of data extraction; and $C=\\{C_{1},C_{2},\dots,C_{K}\\}$ comprises the
series of data fields targeted for extraction. $C_{k}$ is the user input
natural language description of the target field, e.g., “the number of
participants in the study”. The input content $F$ is segmented into distinct
chunks, each marked by a unique identifier. The outputs, denoted as
$\hat{X}^{k}_{\text{EX}}=\\{V^{k},B^{k}\\}$, include the extracted values $V$
and the indices $B$ that link back to their respective locations in the source
content. Hence, it is convenient to check and correct mistakes made in the
extraction by sourcing the origin. The extraction can also be easily scaled by
making paralleled calls of LLMs.

#### Result extraction

Our analysis indicates that data extraction generally performs well for study
design and population-related fields; however, extracting study results
presents challenges. Errors frequently arise due to the diverse presentation
of results within studies and subtle discrepancies between the target
population and outcomes versus those reported. For instance, the target
outcome is the risk ratios (treatment versus control) regarding the incidence
of adverse events (AEs), while the study reports AEs among many groups
separately. Or, the target outcome is the incidence of severe AEs, which
implicitly correspond to those with grade III and more, while the study
reports all grade AEs. To overcome these challenges, we have refined our data
extraction process to create a specialized result extraction pipeline that
improves clinical evidence synthesis. This enhanced pipeline consists of three
crucial steps: (1) identifying the relevant content within the study (Prompt
in Extended Fig. 8), (2) extracting and logically processing this content to
obtain numerical values (Prompt in Extended Fig. 9), and (3) converting these
values into a standardized tabular format (Prompt in Extended Fig. 10).

Steps (1) and (2) are conducted in one pass using CoT reasoning as

$\\{\hat{X}^{1}_{\text{RE},S},\hat{X}^{2}_{\text{RE},S}\\}=\text{LLM}(P_{\text{CoT}}(X,O,F,T_{\text{RE}})),$
(6)

where $O$ is the natural language description of the clinical endpoint of
interest and $T_{\text{RE}}$ is the task definition of result extraction. In
the outputs, $\hat{X}_{\text{RE},S}^{1}$ represents the raw content captured
from the input content $F$ regarding the clinical outcomes;
$\hat{X}_{\text{RE},S}^{2}$ represents the elicited numerical values from the
raw content, such as the number of patients in the group, the ratio of
patients encountering overall response, etc. In step (3), TrialMind writes
Python code to make the final calculation to convert
$\hat{X}_{\text{RE},S}^{2}$ to the standard tabular format.

$\hat{X}_{\text{RE}}=\texttt{exec}(\text{LLM}(P(X,O,T_{\text{PY}},\hat{X}_{\text{RE},S}^{2})),\hat{X}_{\text{RE},S}^{2}).$
(7)

In this process, TrialMind adheres to the instructions in $T_{\text{PY}}$ to
generate code for data processing. This code is then executed, using
$\hat{X}_{\text{RE},S}^{2}$ as input, to produce the standardized result
$\hat{X}_{\text{RE}}$. An example code snippet made to do this transformation
is shown in Extended Fig. 2. This approach facilitates verification of the
extracted results by allowing for easy backtracking to
$\hat{X}_{\text{RE},S}^{1}$. Additionally, it ensures that the calculation
process remains transparent, enhancing the reliability and reproducibility of
the synthesized evidence.

### Experimental setup

#### Literature search and screening

In our literature search experiments, we assessed performance using overall
Recall, aiming to evaluate the effectiveness of different methods in
identifying all relevant studies from the PubMed database using APIs [50]. For
literature screening, we measured efficacy using Recall@20 and Recall@50,
which gauge how well the methods can prioritize target studies at the top of
the list, thereby facilitating quicker decisions about which studies to
include in evidence synthesis. We constructed the ranking candidate set for
each review paper by initially retrieving studies through TrialMind, then
refining this list by ranking the relevance of these studies to the target
review’s PICO elements using OpenAI embeddings. The top 2000 relevant studies
were kept. We then ensured all target papers were included in the candidate
set to maintain the integrity of our groundtruth data. The final candidate set
was then deduplicated to be ranked by the selected methods.

In the criteria analysis experiment, we utilized Recall@200 to assess the
impact of each criterion. This was done by first computing the relevance
prediction using all eligibility predictions and then recalculating it without
the eligibility prediction for the specific criterion in question. The
difference in Recall@200 between these two relevance predictions, denoted as
$\Delta\text{Recall}$, indicates the criterion’s effect. A larger
$\Delta\text{Recall}$ suggests that the criterion plays a more significant
role in influencing the ranking results.

#### Data extraction and result extraction

To evaluate performance, we measured the accuracy of the values extracted by
TrialMind against the groundtruth. We used the study characteristic tables
from the review papers as our test set. Each table’s column names served as
input field descriptions for TrialMind. We manually downloaded the full
content for the studies listed in the characteristic table. To verify the
accuracy of the extracted values, we enlisted three annotators who manually
compared them against the data reported in the original tables.

We also measured the performance of result extraction using accuracy. The
annotators were asked to carefully read the extracted results and compare them
to the results reported in the original review paper. For the error analysis
of TrialMind, the annotators were asked to check the sources to categorize the
errors for one of the reasons: inaccurate, extraction failure, unavailable
data, or hallucination. We designed a vanilla prompting strategy for GPT-4 and
Sonnet models to set the baselines for the result extraction. Specifically,
the prompt was kept minimal, as “Based on the {paper}, tell me the {outcome}
from the input study for the population {cohort}”, where {paper} is the
placeholder for the paper’s content; {outcome} is the for the target endpoint;
{cohort} is the for the target population’s descriptions, including conditions
and characteristics. The responses from these prompts were typically in free
text, from which annotators manually extracted result values to evaluate the
baselines’ performance.

#### Evidence synthesis

In evidence synthesis, we processed the input data using R and the ‘meta’
package to make the forest plots and the pooled results based on the
standardized result values. This is for both TrialMind and the baselines.
Nonetheless, for the baseline, the annotators also need to manually extract
the result values and standardize the values to make them ready for meta-
analysis, which forms the GPT-4+Human baseline in the experiments.

We engaged two groups of annotators for our evaluation: (1) three computer
scientists with expertise in AI applications for medicine, and (2) five
medical doctors to assess the generated forest plots. Each annotator was asked
to evaluate five review studies. For each review, we randomly presented forest
plots generated by both the baseline and TrialMind. The annotators were
required to determine how closely each generated plot aligned with a reference
forest plot taken from the target review paper. Additionally, they were asked
to judge which method, the baseline or TrialMind, produced better results in a
win/lose assessment. Fig. 5a demonstrates the user interface for this study,
which was created with Google Forms.

Extended Fig. 1: The flowchart of the screening process of meta-analyses
involved in the MetaSyns dataset. Extended Fig. 2: The example Python code
made by TrialMind when converting the extracted result values to standardized
tabular form. Extended Fig. 3: Prompt for generating initial search queries in
the literature search. Extended Fig. 4: Prompt for expanding and refining the
initial search queries in the literature search. Extended Fig. 5: Prompt for
study eligibility criteria generation in the literature screen. Extended Fig.
6: Prompt for study eligibility assessment in the literature screen. Extended
Fig. 7: Prompt for study characteristics extraction in the data extraction.
Extended Fig. 8: Prompt for the initial result extraction in the evidence
synthesis. Extended Fig. 9: Prompt for the result formatting in the evidence
synthesis. Extended Fig. 10: Prompt for the result standardization in the
evidence synthesis.

Table 1: The characteristics of involved meta-analyses papers in the MetaSyns dataset. $N_{1}$: the number of identified studies; $N_{2}$: the number of involved studies after screening; $N_{3}$: the number of involved participants in the studies; P: population; I/C: intervention and comparison; O: outcome measurements. Ref |  Topic | $N_{1}$ | $N_{2}$ | $N_{3}$ |  P |  I/C |  O
---|---|---|---|---|---|---|---
Valle, et al. 202132 |  Brachytherapy | 2553 | 150 | 11322 |  men treated with a number of salvage treatments for radiorecurrent prostate cancer. |  Salvage radical prostatectomy (RP), high-intensity focused ultrasound (HIFU), cryotherapy, stereotactic body radiotherapy (SBRT), low–dose-rate (LDR) brachytherapy, and high-dose-rate (HDR) brachytherapy |  1\. 2-yr and 5-yr Relapse free survival (RFS); 2. severe genitourinary (GU) toxicity; 3. severe gastrointestinal (GI) toxicity
Li et al. 202333 |  Brachytherapy | 11806 | 15 | 12773 |  prostate cancer |  1\. Brachytherapy (BT); 2. External beam radiotherapy (EBRT); 3. BT+EBRT |  1\. risk ratios (RRs) of genitourinary (GU) toxicity; 2. RRs of gastrointestinal (GI) toxicity
Hande et al. 202234 |  Brachytherapy | 5499 | 24 | 5488 |  locally advanced cervical cancer |  1\. Volume-based Brachytherapy (BT); 2. Point-A based BT |  1\. 3-year disease-free survival (DFS); 2. 3-year local control (LC); 3. 3-year overall survival (OS); 4. severe GU toxicity 5. severe GI toxicity
Wahyuhadi et al. 202244 |  Cancer Vaccines | 326 | 6 | 1202 |  Glioblastoma multiforme (GBM) |  1\. active immunotherapy: dendritic cell vaccination, peptide vaccination, DNA vaccine, viral vector-based vaccine, antigen non-specific vaccine, autologous tumor cell therapy; 2. standard therapy (surgery, chemotherapy, radiotherapy) |  1\. Overall survival (OS); 2. Progression-free survival (PFS); 3. post-treatment Karnofsky performance scale (KFS); 4. serious adverse events (AEs); 5. 2-year mortality
Zhou et al. 202045 |  Checkpoint Inhibitors | 2236 | 30 | 4971 |  patients with cancer receiving immune checkpoint inhibitors (ICIs): nivolumab, pembrolizumab, atezolizumab, durvalumab, avelumab, or ipilimumab |  1\. occurrence of immune-related AEs (irAEs); 2. non-occurrence of irAEs |  1\. overall survival (OS); 2. progression-free survival (PFS)
Leone et al. 202246 |  Checkpoint Inhibitors | 984 | 10 | 5257 |  advanced esophageal squamous cell carcinoma (ESCC) |  immune checkpoint inhibitors (ICIs) |  1\. overall survival benefit; 2. progression-free survival; 3. overall response rate
Song et al. 202047 |  Checkpoint Inhibitors | 220 | 147 | 23761 |  cancer patients |  1\. anti-PD-1 & anti-PD-L1 inhibitors; 2. anti-CTLA-4 inhibitors |  ICI-related AEs (Grade 1-5, Grade 3-5)
ABC Group, 202235 |  Chemotherapy | N/A | 10 | 1183 |  participants with biopsy-proven transitional cell carcinoma with muscle-invasive bladder cancer who had not received neoadjuvant chemotherapy |  1\. adjuvant cisplatin-based chemotherapy plus local treatment; 2. local treatment; 3. local treatment then adjuvant cisplatin-based chemotherapy on reccurence |  1\. overall survival; 2. locoregional recurrence-free survival; 3. metastasis-free survival; 4. overall recurrence-free survival
Lacas et al. 202136 |  Chemotherapy | N/A | 107 | 19805 |  squamous cell Head and Neck Cancer (MACH-NC) |  Q1: 1. loco-regional treatment (LRT); 2. LRT+CT; Q2: 1. induction CT + radiotherapy; 2. concomitant CT |  1\. overall survival; 2. event-free survival; 3. loco-regional failure (LRF); 4. distant failure (DF); 5. cancer and non-cancer mortality
Xia et al. 202037 |  Chemotherapy | N/A | 9 | 36480 |  triple-negative breast cancer (TNBC) |  1\. neoadjuvant chemotherapy (NACT); 2. adjuvant chemotherapy (ACT) |  1\. overall survival; 2. disease-free survival (DFS)
Liu et al. 202141 |  Hormone Therapy | 7506 | 16 | 67616 |  high-risk prostate cancer (HRPCa) |  1\. neoadjuvant hormone therapy (NHT) with radical prostatectomy (RP); 2. RP alone |  1\. overall survival; 2. biochemical progression-free survival; 3. cancer-specific survival; 4. disease-free survival; 5. risk rates (RRs) of lymph node involvement; 6. RRs of pathological downstaging; 7. RRs of organ-confinement; 8. RRs of positive surgical margins; 9. RRs of seminal vesicle invasion
Piezzo et al. 202042 |  Hormone Therapy | 685 | 8 | 4580 |  patients with HR-positive/HER2-negative advanced or metastatic breast cancer |  1\. CDK4/6 inhibitors with endocrine therapy (ET); 2. ET only |  1\. progression-free survival; 2. overall survival; 3. objective response rate (ORR)
Peleg Hasson et al. 202143 |  Hormone Therapy | 113 | 6 | 35680 |  HR-positive/HER2-positive early-stage breast cancer |  1\. adjuvant endocrine therapy (tamoxifen); 2. aromatase inhibitors |  disease-free survival
Kim et al. 202140 |  Hyperthermia | 334 | 4 | 653 |  low-risk papillary thyroid microcarcinomas (PTMCs) |  1\. thermal ablation; 2. surgery |  1\. new tumor after treatment; 2. lymph node metastasis; 3. complication
Wang et al. 202239 |  Hyperthermia | 3060 | 9 | 1215 |  T1aN0M0 and T1bN0M0 papillary thyroid carcinoma |  ultrasound-guided thermal ablation (LA, RFA, or MWA) |  1\. volume reduction rate; 2. overall diseasee progress rate; 3. complication rate
Spiliotis et al. 202138 |  Hyperthermia | 716 | 15 | 2169 |  liver cancer |  1\. microwave ablation; 2. radiofrequency ablation |  1\. complete ablation (CA); 2. local tumor prgression (LTP); 3. intrahepatic distant (IDR); 4. complications
Li et al. 202048 |  Immune System Modulators | 682 | 9 | 1752 |  breast cancer receiving chemotherapy |  1\. PEGylated granulocyte colony-stimulating factor (G-CSF); 2. G-CSF |  1\. risk ratios (RRs) of grade ¿= 3 / 4 neutropenia; 2. RRs of febrile neutropenia (FN); 3. time to absolute neutrophil count recovery; 4. grade 4 AEs; 5. RRs of skeletal and/or muscle pain
Wu et al. 202249 |  Monoclonal Antibodies | 116 | 32 | 958 |  HER2-positive patients with NSCLC |  HER2-targeted therapy |  1\. objective response rate (ORR); 2. disease control rate (DCR); 3. progression-free survival (PFS)
Li et al. 202150 |  Monoclonal Antibodies | 5861 | 27 | 15063 |  patients with solid tumors |  1\. anti-PD-1/PD-L1 mAbs; 2. anti-PD-1/PD-L1 mAbs + chemotherapy; 3. standard chemotherapy |  RR of nephrotoxicity
Zhu et al. 202351 |  Monoclonal Antibodies | 2511 | 169 | 22492 |  cancer |  antibody–drug conjugates (ADCs) |  1\. RR of grade ¿= 3 AEs; 2. RR of all grade AE
Masetti et al. 202254 |  Stem Cell Transplant | 2141 | 9 | 1448 |  children with pediatric acute myeloid leukemia (AML) in first complete remission (CR1) |  1\. allogeneic hematopoietic stem cell transplantation (allo-HSCT); 2.chemotherapy alone |  1\. overall survival; 2. relapse rate; 3. disease-free survival
Zeng et al. 202155 |  Stem Cell Transplant | 499 | 15 | 959 |  adult philadelphia chromosome positive acute lymphoblastic leukemia in post-remission |  1\. allogeneic hematopoietic stem cell transplantation (allo-HSCT); 2. tyrosine kinase inhibitor (TKI) combined with chemotherapy |  1\. overall survival; 2. Relapse-free survival (RFS); 3. Odds Ratio of non-relapsed mortality (NRM); 4. Odds Ratio of non-relapsed survival (NRS)
Gagelmann et al. 202156 |  Stem Cell Transplant | 1050 | 7 | 680 |  patients with FLT3-ITD-mutated acute myeloid leukemia (AML) and received allogeneic stem-cell transplation |  tyrosine kinase inhibitor (TKI) maintenance therapy |  1\. relapse-free survival; 2. RRs of relapse; 3. overall survival; 4. non-replase mortality; 5. RRs of chronic graft vs. chronic disease (GVHD)
Yang et al. 202152 |  T-Cell Transfer Therapy | 661 | 23 | 350 |  patients with Relapse or Refractory Multiple Myeloma (RRMM) and treated with CAR-T therapy |  CAR-T therapy |  1\. Overall response (OR); 2. complete response rate (CRR); 3. MRD negativity within responders; 4. relapse rate; 5. progression-free survival; 6. overall survival; 7. severe CRS (sCRS); 8. Neurologic toxicity (NT)
Shahzad et al. 202353 |  T-Cell Transfer Therapy | 677 | 13 | 57 |  relapsed/refractory acute myeloid leukemia (RR-AML) |  CAR-T therapy |  1\. complete remission (CR); 2. overall response rate; 3. incidence of CRS; 4. incidence of ICANs; 5. incidence of graft-versus-host disease (GVHD)

## References

  * [1] Field, A. P. & Gillett, R.  How to do a meta-analysis.  _British Journal of Mathematical and Statistical Psychology_ 63, 665–694 (2010).
  * [2] Borah, R., Brown, A. W., Capers, P. L. & Kaiser, K. A.  Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the prospero registry.  _BMJ open_ 7, e012545 (2017).
  * [3] Hoffmeyer, B. D., Andersen, M. Z., Fonnes, S. & Rosenberg, J.  Most cochrane reviews have not been updated for more than 5 years.  _Journal of evidence-based medicine_ 14, 181–184 (2021).
  * [4] Marshall, I. J. & Wallace, B. C.  Toward systematic review automation: a practical guide to using machine learning tools in research synthesis.  _Systematic reviews_ 8, 1–10 (2019).
  * [5] Brown, T. _et al._ Language models are few-shot learners.  _Advances in Neural Information Processing Systems_ 33, 1877–1901 (2020).
  * [6] Wang, S., Scells, H., Koopman, B. & Zuccon, G.  Can chatgpt write a good boolean query for systematic review literature search?  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 1426–1436 (2023).
  * [7] Wadhwa, S., DeYoung, J., Nye, B., Amir, S. & Wallace, B. C.  Jointly extracting interventions, outcomes, and findings from rct reports with llms.  In _Machine Learning for Healthcare Conference_ , 754–771 (PMLR, 2023).
  * [8] Syriani, E., David, I. & Kumar, G.  Assessing the ability of chatgpt to screen articles for systematic reviews.  _arXiv preprint arXiv:2307.06464_ (2023).
  * [9] Shaib, C. _et al._ Summarizing, simplifying, and synthesizing medical evidence using gpt-3 (with varying success).  In _The 61st Annual Meeting Of The Association For Computational Linguistics_ (2023).
  * [10] Wallace, B. C., Saha, S., Soboczenski, F. & Marshall, I. J.  Generating (factual?) narrative summaries of RCTs: Experiments with neural multi-document summarization.  _AMIA Summits on Translational Science Proceedings_ 2021, 605 (2021).
  * [11] Yun, H., Marshall, I., Trikalinos, T. & Wallace, B. C.  Appraising the potential uses and harms of llms for medical systematic reviews.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , 10122–10139 (2023).
  * [12] Christopoulou, S. C.  Towards automated meta-analysis of clinical trials: An overview.  _BioMedInformatics_ 3, 115–140 (2023).
  * [13] National Cancer Institute.  Types of cancer treatment.  https://www.cancer.gov/about-cancer/treatment/types.  Accessed: 2024-04-24.
  * [14] Valle, L. F. _et al._ A systematic review and meta-analysis of local salvage therapies after radiotherapy for prostate cancer (master).  _European urology_ 80, 280–292 (2021).
  * [15] Li, X. _et al._ Comparison of chronic gastrointestinal and genitourinary toxicities between brachytherapy and external beam radiotherapy for patients with prostate cancer: A systematic review and meta-analysis.  _Technology and Health Care_ 1–16 (2023).
  * [16] Hande, V. _et al._ Point-a vs. volume-based brachytherapy for the treatment of cervix cancer: A meta-analysis.  _Radiotherapy and Oncology_ 170, 70–78 (2022).
  * [17] Burdett, S. _et al._ Adjuvant chemotherapy for muscle-invasive bladder cancer: a systematic review and meta-analysis of individual participant data from randomised controlled trials.  _European urology_ 81, 50–61 (2022).
  * [18] Lacas, B. _et al._ Meta-analysis of chemotherapy in head and neck cancer (mach-nc): An update on 107 randomized trials and 19,805 patients, on behalf of mach-nc group.  _Radiotherapy and Oncology_ 156, 281–293 (2021).
  * [19] Xia, L.-Y., Hu, Q.-L., Zhang, J., Xu, W.-Y. & Li, X.-S.  Survival outcomes of neoadjuvant versus adjuvant chemotherapy in triple-negative breast cancer: a meta-analysis of 36,480 cases.  _World journal of surgical oncology_ 18, 1–8 (2020).
  * [20] Spiliotis, A. E. _et al._ Microwave ablation compared with radiofrequency ablation for the treatment of liver cancer: a systematic review and meta-analysis.  _Radiology and oncology_ 55, 247–258 (2021).
  * [21] Wang, M.-H., Liu, X., Wang, Q. & Zhang, H.-W.  Safety and efficacy of ultrasound-guided thermal ablation in treating t1an0m0 and t1bn0m0 papillary thyroid carcinoma: A meta-analysis.  _Frontiers in Endocrinology_ 13, 952113 (2022).
  * [22] Kim, H. J., Cho, S. J. & Baek, J. H.  Comparison of thermal ablation and surgery for low-risk papillary thyroid microcarcinoma: a systematic review and meta-analysis.  _Korean Journal of Radiology_ 22, 1730 (2021).
  * [23] Liu, W., Yao, Y., Liu, X., Liu, Y. & Zhang, G.-M.  Neoadjuvant hormone therapy for patients with high-risk prostate cancer: a systematic review and meta-analysis.  _Asian Journal of Andrology_ 23, 429–436 (2021).
  * [24] Piezzo, M. _et al._ Progression-free survival and overall survival of cdk 4/6 inhibitors plus endocrine therapy in metastatic breast cancer: a systematic review and meta-analysis.  _International journal of molecular sciences_ 21, 6400 (2020).
  * [25] Hasson, S. P. _et al._ Adjuvant endocrine therapy in her2-positive breast cancer patients: systematic review and meta-analysis.  _ESMO open_ 6, 100088 (2021).
  * [26] Wahyuhadi, J. _et al._ Active immunotherapy for glioblastoma treatment: A systematic review and meta-analysis.  _Cancer Control_ 29, 10732748221079474 (2022).
  * [27] Zhou, X. _et al._ Are immune-related adverse events associated with the efficacy of immune checkpoint inhibitors in patients with cancer? a systematic review and meta-analysis.  _BMC medicine_ 18, 1–14 (2020).
  * [28] Leone, A. _et al._ Efficacy and activity of pd-1 blockade in patients with advanced esophageal squamous cell carcinoma: a systematic review and meta-analysis with focus on the value of pd-l1 combined positive score.  _ESMO open_ 7, 100380 (2022).
  * [29] Song, P., Zhang, D., Cui, X. & Zhang, L.  Meta-analysis of immune-related adverse events of immune checkpoint inhibitor therapy in cancer patients.  _Thoracic cancer_ 11, 2406–2430 (2020).
  * [30] Li, X. _et al._ Is pegylated g-csf superior to g-csf in patients with breast cancer receiving chemotherapy? a systematic review and meta-analysis.  _Supportive Care in Cancer_ 28, 5085–5097 (2020).
  * [31] Wu, H.-X., Zhuo, K.-Q. & Wang, K.  Efficacy of targeted therapy in patients with her2-positive non-small cell lung cancer: a systematic review and meta-analysis.  _British Journal of Clinical Pharmacology_ 88, 2019–2034 (2022).
  * [32] Li, H. _et al._ Nephrotoxicity in patients with solid tumors treated with anti-pd-1/pd-l1 monoclonal antibodies: a systematic review and meta-analysis.  _Investigational new drugs_ 39, 860–870 (2021).
  * [33] Zhu, Y., Liu, K., Wang, K. & Zhu, H.  Treatment-related adverse events of antibody–drug conjugates in clinical trials: A systematic review and meta-analysis.  _Cancer_ 129, 283–295 (2023).
  * [34] Yang, Q. _et al._ Efficacy and safety of car-t therapy for relapse or refractory multiple myeloma: A systematic review and meta-analysis.  _International Journal of Medical Sciences_ 18, 1786 (2021).
  * [35] Shahzad, M. _et al._ Outcomes with chimeric antigen receptor t-cell therapy in relapsed or refractory acute myeloid leukemia: a systematic review and meta-analysis.  _Frontiers in Immunology_ 14, 1152457 (2023).
  * [36] Masetti, R., Muratore, E., Gori, D., Prete, A. & Locatelli, F.  Allogeneic hematopoietic stem cell transplantation for pediatric acute myeloid leukemia in first complete remission: a meta-analysis.  _Annals of Hematology_ 101, 2497–2506 (2022).
  * [37] Zeng, Q., Xiang, B. & Liu, Z.  Comparison of allogeneic hematopoietic stem cell transplantation and tki combined with chemotherapy for adult philadelphia chromosome positive acute lymphoblastic leukemia: a systematic review and meta-analysis.  _Cancer Medicine_ 10, 8741–8753 (2021).
  * [38] Gagelmann, N. _et al._ TKI maintenance after stem-cell transplantation for flt3-itd positive acute myeloid leukemia: a systematic review and meta-analysis.  _Frontiers in Immunology_ 12, 630429 (2021).
  * [39] Wu, T., Terry, M. & Cai, C. J.  Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts.  In _Proceedings of the 2022 CHI conference on human factors in computing systems_ , 1–22 (2022).
  * [40] Bodenreider, O.  The unified medical language system (umls): integrating biomedical terminology.  _Nucleic acids research_ 32, D267–D270 (2004).
  * [41] Page, M. J. _et al._ The prisma 2020 statement: an updated guideline for reporting systematic reviews.  _Bmj_ 372 (2021).
  * [42] Song, K., Tan, X., Qin, T., Lu, J. & Liu, T.-Y.  Mpnet: Masked and permuted pre-training for language understanding (2020).  2004.09297.
  * [43] Jin, Q. _et al._ Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval.  _Bioinformatics_ 39, btad651 (2023).
  * [44] Deeks, J. J. & Higgins, J. P.  Statistical algorithms in review manager 5.  _Statistical methods group of the Cochrane Collaboration_ 1 (2010).
  * [45] Zhang, G. _et al._ Leveraging generative ai for clinical evidence synthesis needs to ensure trustworthiness.  _Journal of Biomedical Informatics_ 104640 (2024).
  * [46] Lewis, P. _et al._ Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ 33, 9459–9474 (2020).
  * [47] Wei, J. _et al._ Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ 35, 24824–24837 (2022).
  * [48] OpenAI.  Gpt-4 technical report (2024).  2303.08774.
  * [49] Anthropic.  Introducing the claude 3 family.  https://www.anthropic.com/news/claude-3-family (2023).  Accessed: 2024-04-24.
  * [50] National Center for Biotechnology Information (NCBI).  Entrez programming utilities help.  https://www.ncbi.nlm.nih.gov/books/NBK25501/ (2008).  Accessed: 2024-04-24.