# Clinical Text Summarization: Adapting Large Language Models Can Outperform
Human Experts

Dave Van Veen† Stanford University  Cara Van Uden  Stanford University  Louis
Blankemeier  Stanford University  Jean-Benoit Delbrouck  Stanford University
Asad Aali‡ Stanford University
Christian Bluethgen  Stanford University  Anuj Pareek  Stanford University
Malgorzata Polacin  Stanford University  William Collins  Stanford University
Neera Ahuja  Stanford University
Curtis P. Langlotz  Stanford University  Jason Hom  Stanford University
Sergios Gatidis  Stanford University  John Pauly  Stanford University  Akshay
S. Chaudhari  Stanford University

###### Abstract

Sifting through vast textual data and summarizing key information imposes a
substantial burden on how clinicians allocate their time. Although large
language models (LLMs) have shown immense promise in natural language
processing (NLP) tasks, their efficacy across diverse clinical summarization
tasks has not yet been rigorously examined. In this work, we employ domain
adaptation methods on eight LLMs, spanning six datasets and four distinct
summarization tasks: radiology reports, patient questions, progress notes, and
doctor-patient dialogue. Our thorough quantitative assessment reveals trade-
offs between models and adaptation methods in addition to instances where
recent advances in LLMs may not lead to improved results. Further, in a
clinical reader study with six physicians, we depict that summaries from the
best adapted LLM are preferable to human summaries in terms of completeness
and correctness. Our ensuing qualitative analysis delineates mutual challenges
faced by both LLMs and human experts. Lastly, we correlate traditional
quantitative NLP metrics with reader study scores to enhance our understanding
of how these metrics align with physician preferences. Our research marks the
first evidence of LLMs outperforming human experts in clinical text
summarization across multiple tasks. This implies that integrating LLMs into
clinical workflows could alleviate documentation burden, empowering clinicians
to focus more on personalized patient care and other irreplaceable human
aspects of medicine.

##  1 Introduction

Text summarization plays an indispensable role in the practice of healthcare.
Currently, clinicians spend significant time summarizing vast amounts of
textual information—whether it be compiling diagnostic reports, writing
progress notes, or synthesizing a patient’s treatment history across different
specialists [29, 2, 23]. Even for experienced physicians with a high level of
expertise, this intricate task naturally introduces the possibility for
errors, which can be detrimental in a field where precision is paramount [78,
5, 27].

The proliferation of electronic health records (EHR) has drastically increased
clinical documentation workload, directly contributing to increasing stress
and clinician burnout [28, 56, 22]. Recent data indicates that physicians
expend two hours on documentation for each hour of patient interaction [62].
Meanwhile, documentation responsibilities for nurses consume up to 60% of
their time and account for significant work stress [36, 21, 8]. These tasks
divert attention from patient care, leading to worse outcomes for patients as
well as disillusionment and decreased job satisfaction for clinicians [2, 59,
57].

In recent years, large language models (LLMs) have gained remarkable traction,
leading to widespread adoption of models such as ChatGPT [6], which excel at
information retrieval, nuanced understanding, and text generation [83, 7].
While excellent LLM benchmarks for general NLP tasks exist [40, 84], they do
not evaluate clinical performance. Addressing this limitation presents a
tremendous opportunity to accelerate the process of clinical text
summarization, hence alleviating documentation burden and improving patient
care.

Crucially, machine-generated summaries must be non-inferior to that of
seasoned clinicians—especially when used to support sensitive clinical
decision-making. Recent work in clinical natural language processing (NLP) has
demonstrated potential on medical text [77, 68], adapting to the medical
domain by either training a new model [61, 71], fine-tuning an existing model
[69, 72], or supplying domain-specific examples in the model prompt [47, 72].
However, non-inferiority has not yet been thoroughly demonstrated for clinical
text summarization. With the overarching objective of bringing LLMs closer to
clinical readiness, we aim to bridge the gap between theoretical promise and
practical utility. This culminates in the following contributions:

  * •

We implement adaptation methods across eight open-source and proprietary LLMs
for four distinct summarization tasks comprising six datasets. To our
knowledge, the subsequent evaluation via NLP metrics is the most comprehensive
assessment of contemporary LLMs for clinical text summarization.

  * •

Our exploration delves into a myriad of trade-offs concerning different models
and adaptation methods, shedding light on scenarios where advancements in
model size, novelty, or domain specificity do not necessarily translate to
superior performance.

  * •

Through a rigorous clinical reader study with six physicians, we demonstrate
that LLM summaries can surpass human summaries in terms of completeness and
correctness. This novel finding affirms the non-inferiority of machine-
generated summaries in a clinical context.

  * •

We qualitatively analyze summaries to pinpoint challenges faced by both LLMs
and humans. Such insights can guide future enhancements and integration of
LLMs into clinical workflows.

  * •

We identify which NLP metrics most correlate with reader scores on key
attributes such as completeness, correctness, and conciseness.

Our results demonstrate that LLMs often outperform human experts for clinical
text summarization across the diverse range of documents we evaluate. This
implies that LLMs could be leveraged to reduce documentation load and thus
support clinicians—not supplant them. Once a summary is provided, clinicians
are essential to make treatment recommendations and final decisions.
Ultimately, such new tools may improve the clinical workflow, resulting in
decreased clinician strain and improved patient care. Accelerating tedious
tasks will enable healthcare providers to dedicate more time to the essential
human facets of medicine, such as fostering patient relationships,
understanding their specific goals, offering personalized advice, and other
elements which artificial intelligence (AI) cannot replace.

##  2 Related Work

Figure 1: Overview. First we quantitatively evaluate each valid combination
($\times$) of LLM and adaptation method across four distinct summarization
tasks comprising six datasets. We then conduct a clinical reader study
comparing the summary of the best model/method against that of a human expert.

Large language models (LLMs) have demonstrated astounding performance,
propelled by both the transformer architecture [73] and increasing scales of
data and compute, resulting in widespread adoption of models such as ChatGPT
[6]. Although several of the more expansive models, such as GPT-4 [51] and
PaLM [11], remain proprietary and provide access via “black-box” interfaces,
there has been a pronounced shift towards open-sourced alternatives such as
Llama-2 [70]. These open-source models grant researchers direct access to
model weights for customization.

Popular transformer models such as BERT [19] and GPT-2 [54] established the
paradigm of self-supervised pretraining on large amounts of general data and
then adapting to a particular domain or task by tuning on specific data. One
approach is customizing model weights via instruction tuning, a process where
language models are trained to generate human-aligned responses given specific
instructions [75]. Examples of clinical instruction-tuned models include Med-
PALM [61] for medical question-answering or Radiology-GPT [43] for radiology
tasks. To enable domain adaptation with limited computational resources,
prefix tuning [39] and low-rank adaptation (LoRA) [31] have emerged as
effective methods that require tuning less than 1% of total parameters over a
small training set. LoRA has been shown to work well for medical question-
answering [69] and summarizing radiology reports [72]. Another adaptation
method, requiring no parameter tuning, is in-context learning: supplying the
LLM with task-specific examples in the prompt [37]. Because in-context
learning does not alter model weights, it can be performed with black-box
model access using only a few training examples [37].

Recent work has adapted LLMs for various medical tasks, demonstrating great
potential for medical language understanding and generation [77, 68, 71, 42].
Specifically, a broad spectrum of methodologies has been applied to clinical
text for specific summarization tasks. One such task is the summarization of
radiology reports, which aims to consolidate detailed findings from
radiological studies into significant observations and conclusions drawn by
the radiologist [35]. LLMs have shown promise on this task [72] and other
tasks such as summarizing daily progress notes into a concise “problem list”
of medical diagnoses [25]. Lastly, there has been significant work on
summarizing extended conversations between a doctor and patient into patient
visit summaries [1, 80, 47].

While the aforementioned contributions incorporate methods to adapt language
models, they often include only a small subset of potential approaches and
models, and/or they predominantly rely on evaluation via standard NLP metrics.
Given the critical nature of medical tasks, demonstrating clinical readiness
requires including human experts in the evaluation process. To address this,
there have been recent releases of expert evaluations for instruction
following [23] and radiology report generation [81]. Other work employs human
experts to evaluate synthesized Cochrane review abstracts, demonstrating that
NLP metrics are not sufficient to measure summary quality [65]. With this in
mind, we extend our comprehensive evaluation of methods and LLMs beyond NLP
metrics to incorporate a rigorous clinical reader study across multiple
summarization tasks. Our results are the first which demonstrate across many
tasks that LLM summaries are comparable to–––and often surpass–––those created
by human experts.

##  3 Approach

This section describes each large language model, adaptation method, and
summarization task as depicted in Figure 1.

###  3.1 Large language models

Table 1: We quantitatively evaluate eight models, including state-of-the-art sequence-to-sequence and autoregressive models. Unless specified, models are open-source (vs. proprietary).  Model | Context | Parameters | Proprietary? | Seq2seq | Autoreg.
---|---|---|---|---|---
FLAN-T5 | 512 | 2.7B | - | ✔ |
FLAN-UL2 | 2,048 | 20B | - | ✔ |
Alpaca | 2,048 | 7B | - | - | ✔
Med-Alpaca | 2,048 | 7B | - | - | ✔
Vicuna | 2,048 | 7B | - | - | ✔
Llama-2 | 4,096 | 7B, 13B | - | - | ✔
GPT-3.5 | 16,384 | 175B | ✔ | - | ✔
GPT-4 | 32,768 | unknown | ✔ | - | ✔

We investigate a diverse collection of transformer-based LLMs for clinical
summarization tasks. This includes two broad approaches to language
generation: sequence-to-sequence (seq2seq) models and autoregressive models.
Seq2seq models use an encoder-decoder architecture to map the input text to a
generated output, often requiring paired datasets for training. These models
have shown strong performance in machine translation [9] and summarization
[60]. In contrast, the typical architecture of autoregressive models is a lone
decoder. They generate tokens sequentially—where each new token is conditioned
on all previous tokens—thus efficiently capturing context and long-range
dependencies. Typically, autoregressive models are trained with unpaired data.
Autoregressive models are particularly useful for NLP tasks such as generating
text generation, question-answering, and dialogue interactions [10, 6].

We include prominent seq2seq models due to their strong summarization
performance [60] and autoregressive models due to their state-of-the-art
performance for many general NLP tasks [84]. As shown in Table 1, our choice
of models varies widely with respect to number of parameters (2.7B to 175B)
and context length (512 to 32K), i.e. the maximum number of tokens a model can
process. We organize our models into three categories:

Open-source seq2seq models. The original T5 “text-to-text transfer
transformer” model [55] demonstrated excellent performance in transfer
learning using the seq2seq architecture. A derivative model, FLAN-T5 [12, 44],
improved performance via instruction prompt tuning. This T5 model family has
proven effective for various clinical NLP tasks [38, 72]. Recently, the FLAN-
UL2 model [67, 13] was introduced, featuring increased context length and a
modified pre-training procedure called unified language learning (UL2).

Open-source autoregressive models. The Llama family of LLMs [70] have enabled
the proliferation of open-source instruction-tuned models that deliver
comparable performance to GPT-3 [6] on many benchmarks despite its smaller
size. Descendants of this original model have taken additional fine-tuning
approaches, such as fine-tuning via instruction following (Alpaca [66]),
medical Q&A data (Med-Alpaca [30]), user-shared conversations (Vicuna [10]),
and reinforcement learning from human feedback (Llama-2 [70]). Llama-2 allows
for twofold longer context lengths (4,096) relative to our other open-source
autoregressive models.

Proprietary autoregressive models. We include GPT-3.5 [50] and GPT-4 [51], the
latter of which is widely regarded as state-of-the-art on general NLP tasks
[84]. Both models offer significantly higher context length than open-source
models. Additionally, the proprietary nature of these models raises an
interesting point for healthcare, where data and model governance is
important—especially if summarization tools are cleared for clinical use by
the FDA.

Note that each model’s context length limits the number of in-context examples
that can be used, as described in the following section.

###  3.2 Adaptation methods

To adapt pre-trained LLMs to our domain-specific clinical summarization tasks,
we consider two proven techniques:

In-context learning (ICL). ICL is a lightweight adaptation method that
requires no altering of model weights; instead, one includes a handful of in-
context examples within the model prompt [37]. This simple approach provides
the model with context, enhancing LLM performance for a particular task or
domain [47, 72]. We implement this by choosing, for each sample in our test
set, the $m$ nearest neighbors training samples in the embedding space of the
PubMedBERT model [14]. Note that choosing “relevant” in-context examples
typically outperforms choosing examples at random [48]. For a given model and
dataset, we use $m=2^{x}$ examples, where $x\in\\{0,1,2,3,...,M\\}$ for $M$
such that no more than $1\%$ of the $s=250$ samples are excluded due to
prompts exceeding the model’s context length.

Quantized low-rank adaptation (QLoRA). Low-rank adaptation (LoRA) [31] has
emerged as an effective, lightweight approach for fine-tuning LLMs by altering
a small subset of model weights—often $<0.1\%$ [72]. This method injects
trainable rank decomposition matrices into the attention layers; then, given a
training set of samples, LoRA performs gradient descent on these injected
weights while keeping the remainder of model weights frozen. Compared to
training model weights from scratch, this method is much more efficient with
respect to both computational requirements and the volume of training data
required. Recently, QLoRA [18] has been introduced as a more memory-efficient
variant of LoRA, employing 4-bit quantization to enable the fine-tuning of
larger LLMs given the same hardware constraints. This quantization does not
impact performance [18]; as such, we use QLoRA for all model training. Note
that QLoRA cannot be used to fine-tune proprietary models on consumer
hardware, as we do not have access to model weights.

###  3.3 Data

Table 2: Description of four distinct summarization tasks comprising six open-source datasets with a wide range of token length and lexical variance, i.e. $\frac{\text{number of unique words}}{\text{number of total words}}$, where a higher ratio indicates higher lexical diversity. |  | Number | Avg. number of tokens | Lexical
---|---|---|---|---
Task (Dataset) | Task description | of samples | Input | Target | variance
Radiol. reports (Open-i) | findings $\rightarrow$ impression | 3.4K | 52 $\pm$ 22 | 14 $\pm$ 12 | 0.11
Radiol. reports (MIMIC-CXR) | findings $\rightarrow$ impression | 128K | 75 $\pm$ 31 | 22 $\pm$ 17 | 0.08
Radiol. reports (MIMIC-III) | findings $\rightarrow$ impression | 67K | 160 $\pm$ 83 | 61 $\pm$ 45 | 0.09
Patient questions (MeQSum) | verbose $\rightarrow$ short question | 1.2K | 83 $\pm$ 67 | 14 $\pm$ 6 | 0.21
Progress notes (ProbSum) | notes $\rightarrow$ problem list | 755 | 1,013 $\pm$ 299 | 23 $\pm$ 16 | 0.15
Dialogue (ACI-Bench) | dialogue $\rightarrow$ assessment | 126 | 1,512 $\pm$ 467 | 211 $\pm$ 98 | 0.04

To robustly evaluate LLM performance on clinical text summarization, we choose
four distinct summarization tasks, comprising six open-source datasets. As
depicted in Table 2, each dataset has a varying number of samples, token
lengths, and lexical variance. Here we calculate lexical variance as
$\frac{\text{number of unique words}}{\text{number of total words}}$ across
the entire dataset; hence a higher ratio indicates less repetition and more
lexical diversity. We describe each task and dataset below.

####  3.3.1 Radiology reports

Radiology report summarization takes as input the findings section of a
radiology study containing detailed exam analysis and results. The goal is to
summarize these findings into an impression section, which concisely captures
the most salient, actionable information from the study. See Figure 8 for
examples.

We consider three datasets for this task, where both reports and findings are
created by attending physicians as part of routine clinical care.

Open-i [17] contains de-identified narrative chest x-ray reports from the
Indiana Network for Patient Care 10 database. From the initial set of 4K
studies, [17] selected a final set of 3.4K reports based on the quality of
imaging views and diagnostic content.

MIMIC-CXR [32] contains chest x-ray studies accompanied by free-text radiology
reports acquired at the Beth Israel Deaconess Medical Center between 2011 and
2016. We utilize a preprocessed version of 128K reports provided by the
RadSum23 shared task at BioNLP 2023 [15, 16].

MIMIC-III [33] contains 67K radiology reports spanning seven anatomies (head,
abdomen, chest, spine, neck, sinus, and pelvis) and two modalities: magnetic
resonance imaging (MRI) and computed tomography (CT). This dataset originated
from patient stays in critical care units of the Beth Israel Deaconess Medical
Center between 2001 and 2012. We utilize a preprocessed version via RadSum23
[15, 16]. Compared to x-rays, MRIs and CT scans portray more information at a
higher resolution. This leads to longer reports (Table 2), rendering MIMIC-III
a more challenging summarization dataset than Open-i or MIMIC-CXR.

####  3.3.2 Patient questions

The task of question summarization consists of generating a condensed question
expressing the minimum information required to find correct answers to the
original question [3]. For this task we employ the MeQSum dataset [3], which
contains (1) original patient health questions of varying verbosity and
coherence selected from the U.S. National Library of Medicine (2)
corresponding condensed questions created by three medical experts such that
the summary allows retrieving complete, correct answers to the original
question without the potential for further condensation. These condensed
questions were then validated by two physicians and verified to have high
inter-annotator agreement. Due to the wide variety of these questions, MeQSum
exhibits the highest lexical variance of our datasets (Table 2). See Figure A5
for examples.

####  3.3.3 Progress notes

The goal of this task is to generate a “problem list,” or condensed list of
diagnoses and medical problems using the provider’s progress notes during
hospitalization. For this we employ the ProbSum dataset [25], extracted from
the MIMIC-III database of de-identified hospital intensive care unit (ICU)
admissions. ProbSum contains (1) progress notes averaging $>1,000$ tokens and
substantial presence of unlabeled numerical data, e.g. dates and test results
(2) corresponding problem lists created by attending medical experts in the
ICU. We access this data via the BioNLP Problem List Summarization shared task
[25, 26, 16] and Physionet [34]. See Figures A6 and A7 for examples.

####  3.3.4 Dialogue

The goal of this task is to summarize a doctor-patient conversation into an
“assessment and plan” paragraph. We employ the ACI-Bench dataset [80, 1, 79],
which contains (1) 207 doctor-patient conversations (2) corresponding patient
visit notes, which were first generated by a seq2seq model and subsequently
corrected and validated by expert medical scribes and physicians. Since ACI-
Bench’s visit notes includes a heterogeneous collection of section headers,
for our analysis we select 126 samples containing an “assessment and plan”
section. Per Table 2, this task entails the largest token counts across
datasets for both the input (dialogue) and target (assessment). See Figure A8
for an example.

##  4 Experiments

We now provide an overview of the evaluation process, which is depicted in
Figure 1. The first step (Section 4.1) includes a quantitative evaluation
across each valid combination of model, adaptation method, and summarization
task using a suite of natural language processing (NLP) metrics. The second
step (Section 4.2) entails an extensive clinical reader study and qualitative
evaluation by physicians.

Table 3: Model prompts consist of a prefix, instruction, and suffix. Top: Instructions for each of the four summarization tasks. Bottom: Prefix and suffix applied to every task. See Section 4.1.1 for further details. Use | Prompt component
---|---
Radiology reports | “Summarize the radiology report findings
into an impression with minimal text.”
Patient questions | “Summarize the patient health query
into one question of 15 words or less.”
Progress notes | “Based on the progress note, generate a list of 3-7 prob-
lems (a few words each) ranked in order of importance.”
Dialogue | “Summarize the patient/doctor dialogue
into an assessment and plan.”
Prefix | “You are a knowledgeable medical professional.”
Suffix (ICL only) | “Use the provided examples to guide word choice.”

###  4.1 Quantitative Evaluation

Building upon the descriptions of models, methods, and tasks in Section 3, we
now specify experimental details such as model prompts, data preparation, and
software implementation. We also describe the NLP metrics used for
quantitative evaluation.

####  4.1.1 Model prompts and temperature

Each of the four summarization tasks uses a slightly different prompt, as
depicted in Table 3. Here all prompts consist of (1) a prefix, nudging the
model to exhibit medical expertise and (2) an instruction, directing the model
to complete a particular task. For ICL (not QLoRA), the prompt further
includes (3) a suffix, directing the model to follow the pattern demonstrated
by the in-context examples and (4) $m$ examples, formatted as such:

> $\begin{array}[]{l}\;\;\vdots\\\ \;\;\texttt{input $i$: \\{input text\\}}\\\
> \;\;\texttt{summary $i$: \\{summarized text\\}}\\\ \;\;\texttt{\\#\\#}\\\
> \;\;\vdots\end{array}$

where $i=1..m$, and # symbols serve as delimiters between examples. Recall $m$
is defined in Section 3.2 for ICL, otherwise $m=0$ for QLoRA. After this
preamble, the final prompt component for all methods is the actual sample we
wish to summarize:

> $\begin{array}[]{l}\;\;\texttt{input $m+1$: \\{input text\\}}\\\
> \;\;\texttt{summary $m+1$: }\\\ \end{array}$

This format clarifies that the model’s task is to create a summary for sample
$m+1$.

Table 4: Results can vary significantly across different model prompt and
parameters. We generally find better performance when (1) using lower
temperature, i.e. generating less random output, as summarization tasks
benefit more from truthfulness than creativity (2) assigning the model
clinical expertise in the prompt. Output generated via GPT-3.5 on the Open-i
radiology report dataset.

Parameter | Value | BLEU | ROUGE-L | BERTScore | MEDCON
---|---|---|---|---|---
| 0.1 | 4.9 | 28.1 | 89.6 | 28.2
Temperature | 0.5 | 4.9 | 27.1 | 89.7 | 27.5
| 0.9 | 4.3 | 25.4 | 89.3 | 25.3
| None | 10.4 | 34.3 | 90.2 | 30.7
Expertise | $\textrm{Physician}^{1}$ | 11.1 | 35.5 | 90.5 | 35.5
| $\textrm{Wizard}^{2}$ | 4.3 | 27.8 | 89.7 | 28.5
1: “You are an experienced physician.” 2: “You are a mystical wizard in Middle
Earth.”

We chose the prompt components (prefix, instruction, and suffix) by following
best practices [58, 4] and qualitatively evaluating a handful of variants for
each component over a small dataset. We note the importance of specifying
desired length in the instruction, e.g. “one question of 15 words or less” for
summarizing patient questions. Without this specification, the model might
generate lengthy outputs—occasionally even longer than the input text. While
in some instances this level of detail may be preferred, we steer the model
toward conciseness given our task of summarization.

We further note the effect of model prompt and parameters in Table 4. For
example, we achieve better performance by nudging the model to have expertise
in medicine than an expertise in wizardry. The model parameter temperature is
also important. Temperature determines conditional probability distributions,
hence impacting how often the model outputs a less likely token. As
temperature increases, the output becomes more random and creative. After a
crude search over temperature values $\\{0.1,0.5,0.9\\}$ using GPT-3.5, we
found the lowest value $0.1$ performs best and thus set temperature to this
value for all models. Intuitively, a lower value seems appropriate given our
goal of factually summarizing text with a high aversion to hallucinations.

####  4.1.2 Experimental Setup

Next, we describe the process for splitting datasets into training,
validation, and test sets. Across each experimental configuration, we
construct test sets using the same $s$ samples for each dataset, where $s=250$
for all datasets except dialogue ($s=100$), which includes only 126 samples in
total. Once these $s$ samples have been selected as a test set, another $s$
are chosen as a validation set for datasets which incorporate fine-tuning. The
remaining samples are used as a training set for ICL examples or QLoRA fine-
tuning.

We leverage PyTorch, using the parameter-efficient fine-tuning [46] and the
generative pre-trained transformers quantization [24] libraries to implement
QLoRA. We fine-tune models with QLoRA for five epochs using the Adam optimizer
with weight decay fix [45]. Our initial learning rate of $1e^{-3}$ decays
linearly to $1e^{-4}$ after a 100-step warm-up; we determined this
configuration after experimenting with different learning rates and
schedulers. To achieve an effective batch size of 24 on each experiment, we
adjust (1) individual batch size and (2) number of gradient accumulation steps
to fit on a single consumer GPU, a NVIDIA Quadro RTX 8000. Deviating from the
recommended QLoRA [18] parameters gave us no improvement in performance. All
open-source models are available on HuggingFace [76].

####  4.1.3 Metrics

We use well-known summarization metrics to assess the quality of generated
summaries. BLEU [52], the simplest metric, calculates the degree of overlap
between the reference and generated texts by considering 1- to 4-gram
sequences. ROUGE-L [41] evaluates similarity based on the longest common
subsequence; it considers both precision and recall, hence being more
comprehensive than BLEU. In addition to these syntactic metrics, we employ
BERTScore, which leverages contextual BERT embeddings to evaluate the semantic
similarity of the generated and reference texts [82]. Lastly, we include
MEDCON [80] to gauge the consistency of medical concepts. This employs
QuickUMLS [63], a tool that extracts biomedical concepts via string matching
algorithms [49]. We restrict MEDCON to specific UMLS semantic groups (Anatomy,
Chemicals & Drugs, Device, Disorders, Genes & Molecular Sequences, Phenomena
and Physiology) relevant for our work. All four metrics range from $[0,100]$
with higher scores indicating higher similarity between the generated and
reference summaries.

###  4.2 Clinical reader study

After identifying the best model and method via NLP quantitative metrics, we
perform a clinical reader study and rigorous qualitative evaluation. This
analysis compares the reference human expert summaries against those generated
by the best model/method identified above. We include three summarization
tasks in this evaluation: radiology reports, patient questions, and progress
notes. The dialogue task is excluded due to the unwieldiness of a human reader
parsing many lengthy transcribed conversations and paragraphs; see Figure A8
for an example and Table 2 for the token count.

Our readers include two sets of physicians: (1) three board-certified
radiologists to evaluate summaries of radiology reports (2) three board-
certified hospitalists (internal medicine physicians) to evaluate summaries of
patient questions and progress notes. For each task, each physician views the
same 100 randomly selected inputs and their A/B comparisons (human vs. model
summaries), which are presented in a blinded and randomized order. An ideal
summary would contain all clinically significant information (completeness)
without any errors (correctness) or superfluous information (conciseness).
Hence we pose the following three questions for readers to evaluate using a
five-point Likert scale.

  * •

Completeness: “Which summary more completely captures important information?”
This compares the summaries’ recall, i.e. the amount of clinically significant
detail retained from the input text.

  * •

Correctness: “Which summary includes less false information?” This compares
the summaries’ precision, i.e. instances of false information due to
hallucination by the model or an error by the human expert.

  * •

Conciseness: “Which summary contains less non-important information?” This
compares which summary is more condensed, as the value of a summary decreases
with superfluous information.

Given this non-parametric, categorical data, we assess the statistical
significance of responses using a Wilcoxon signed-rank test. Additionally,
readers provide comments on notable samples to identify interesting
observations for qualitative analysis.

We create and deploy the reader study via Qualtrics. Figure A4 demonstrates
the user interface. To obfuscate any formatting differences between the human
and model summaries, we apply trivial post-processing to standardize
capitalization, punctuation, newline characters, etc.

Figure 2: Comparing Alpaca vs. Med-Alpaca. As most data points are below the
dashed lines denoting equivalence, we conclude that Med-Alpaca’s fine-tuning
with medical Q&A data results in worse performance for our clinical
summarization tasks. See Section 5.1 for further discussion. Note that each
data point corresponds to the average score of $s=250$ samples for a given
experimental configuration, i.e. {dataset $\times$ $m$ in-context examples}.

##  5 Results and Discussion

###  5.1 Quantitative evaluation

While considering which open-source models to evaluate, we first assess the
benefit of fine-tuning open-source models on medical text. For example, [30]
released Med-Alpaca, a version of Alpaca [66] which was further instruction-
tuned with medical Q&A text, consequently improving performance for the task
of medical question-answering. Despite Med-Alpaca’s adaptation for the medical
domain, Figure 2 shows that it actually performs worse than Alpaca for our
tasks of clinical text summarization. This suggests that—in addition to domain
adaptation–––task adaptation is also important. With this in mind, and
considering that Alpaca is commonly known to perform worse than our other
open-source autoregressive models Vicuna and Llama-2 [84, 10], for simplicity
we exclude Alpaca and Med-Alpaca from further analysis.

Figure 3:  Summarization performance comparing one in-context example (ICL)
vs. QLoRA methods across all open-source models on the Open-i radiology report
dataset. FLAN-T5 achieves best performance on both methods for this dataset.
QLoRA typically outperforms ICL with the better models (FLAN-T5, Llama-2),
although this often shifts with more in-context examples (see Figure A1).
Figure A2 contains similar results with patient health questions.  Figure 4:
MEDCON scores vs. number of in-context examples across models and datasets. We
also include the best model fine-tuned with QLoRA (FLAN-T5) as a horizontal
dashed line for valid datasets. Note the allowable number of in-context
examples varies significantly by model and dataset. See Figure A1 for results
across all four metrics (BLEU, ROUGE-L, BERTScore, MEDCON). Figure 5: Model
win rate: a head-to-head winning percentage of each model combination, where
red/blue intensities highlight the degree to which models on the vertical axis
outperform models on the horizontal axis. GPT-4 generally achieves the best
performance. While FLAN-T5 is more competitive for syntactic metrics such as
BLEU, we note this model is constrained to shorter context lengths (see Table
1). When aggregated across datasets, seq2seq models (FLAN-T5, FLAN-UL2)
outperform open-source autoregressive models (Llama-2, Vicuna) on all metrics.

Next, we compare ICL vs. QLoRA across the remaining open-source models with
the Open-i radiology report dataset in Figure 3 and with patient health
questions in Figure A2. We choose these datasets because their shorter context
lengths allow for training with lower computational cost. Results demonstrate
that FLAN-T5 generally performs best, although Llama-2 is often comparable.
Surprisingly, FLAN-T5 (2.7B) outperforms its fellow seq2seq model FLAN-UL2
(20B), despite being an older model with significantly fewer parameters. For
the better performing models FLAN-T5 and Llama-2, QLoRA outperforms ICL with
one example. As the number of in-context examples increases, however, ICL
outperforms QLoRA on most metrics for Open-i and MIMIC-CXR (Figure A1).

Figure 4 displays MEDCON scores for all models against number of in-context
examples up to the maximum context length permitted by each model and dataset.
This graph also includes the best performing model (FLAN-T5) with QLoRA as a
reference, depicted by a horizontal dashed line. While ICL and QLoRA are
competitive for open-source models, proprietary models GPT-3.5 and GPT-4 far
outperform other models and methods given sufficient in-context examples.
Among open-source models, seq2seq (FLAN-T5, FLAN-UL2) performs better than
autoregressive (Llama-2, Vicuna) models on radiology reports but worse on
patient questions and progress notes. Given that these latter datasets have
higher lexical variance (Table 2) and more heterogeneous formatting compared
to radiology reports, we posit that autoregressive models may perform better
with increasing data heterogeneity and complexity. For a similar graph across
all metrics, see Figure A1.

Figure 6: Clinical reader study. Top: Study design comparing the summarization
of GPT-4 vs. that of human experts on three attributes: completeness,
correctness, and conciseness. Bottom: Results. GPT-4 summaries are rated
higher than human summaries on completeness for all three summarization tasks
and on correctness overall. Radiology reports highlight a trade-off between
correctness (better) and conciseness (worse) with GPT-4. Highlight colors
correspond to a value’s location on the color spectrum. Asterisks denote
statistical significance by Wilcoxon signed-rank test, *p-value $<0.05$,
**p-value $\ll 0.001$.

Figure 5 compares models using win rates, i.e. a head-to-head winning
percentage of each model combination across the same set of samples. GPT-4
further solidifies its position as the best overall. FLAN-T5 performs better
on the syntactical BLEU metric but worse on others, suggesting FLAN-T5 excels
more at matching word choice than matching semantic or conceptual meaning.
Note that FLAN-T5 is constrained to much shorter context length (512) than
GPT-4 (32K).

We conclude the best model and method is GPT-4 with a maximum allowable number
of in-context examples. Next we evaluate this configuration in a clinical
reader study.

###  5.2 Clinical reader study

We conduct a clinical reader study across three distinct summarization tasks
to compare summaries generated by GPT-4 against the reference summaries
created by human experts. Pooled results across our physicians in Figure 6
demonstrate that GPT-4 summaries are more complete and contain fewer errors
compared to human summaries. We additionally display the distribution of
reader responses in Figure 7, showing that in a minority of cases, human
summaries were preferred over GPT-4 summaries. Based on physician feedback, we
provide a rigorous qualitative analysis to illustrate strengths and weaknesses
of summaries by GPT-4 and humans; see Figures 8, A5, A6, and A7. Lastly, Table
A1 contains scores separated by individual readers.

Figure 7:  Distribution of reader scores for each summarization task across
evaluated attributes (completeness, correctness, conciseness). Horizontal axes
denote reader preference between GPT-4 and human summaries as measured by a
five-point Likert scale. Vertical axes denote frequency count, with 900 total
reports for each plot. GPT-4 summaries are more often preferred in terms of
correctness and completeness. While the largest gain in correctness occurs on
radiology reports, this introduces a trade-off with conciseness. See Figure 6
for overall scores.

####  5.2.1 Completeness

Completeness is measured by the reader’s response to the following question:
“Which summary more completely captures important information?” We observe
that GPT-4 summaries are more complete on average than human summaries,
achieving statistical significance across all three summarization tasks with
p-values $\ll 0.001$ (Figure 6). This suggests that GPT-4 excels at
identifying and understanding the most relevant information from the source
text.

We provide intuition for completeness by investigating a specific example in
progress notes summarization. In Figure A6, GPT-4 correctly identified
conditions that were missed by the human expert, such as “hypotension”,
“anemia”, and “COPD”. GPT-4 was more complete in generating its progress note
summary but also missed historical context (a history of “hypertension”, or
“HTN”). While the potential to miss context can be a limitation that affects
completeness, it may be preferable if the goal is to focus on the current
state of a patient’s health. Alternatively, incorporating prior notes into the
model prompt could help capture longitudinal changes.

####  5.2.2 Correctness

Correctness is measured by the reader’s response to the following question:
“Which summary includes less false information?” GPT-4 generated fewer errors
compared to human summaries (Figure 6). This margin was statistically
significant overall with a p-value $\ll 0.001$ and on two of three
summarization tasks. For radiology reports, GPT-4 always matched or
outperformed the human expert; readers identified zero instances (out of 100)
in which the human outperformed GPT-4 (Figure 7).

As an example of GPT-4’s superior correctness performance on the radiology
report summarization task, GPT-4 avoided common human errors related to
lateral distinctions (right vs. left, Figure 8). For the problem list
summarization task, Figure A6 demonstrates that GPT-4 avoided a mistake
(including “UTI”) that was incorrectly documented by the human—for this
example, the physician reader commented that “[the human] is hallucinating,” a
phrase often used to describe mistakes made by LLMs.

Despite this promising performance for radiology report summarization, GPT-4
was not perfect. We see a clear example in Figure A7 where GPT-4 mistakenly
generated (“hallucinated”) several conditions in the problem list that were
false, such as “eosinophilia”.

Both GPT-4 and human experts faced challenges correctly interpreting ambiguous
user queries in patient health questions. Notably, GPT-4’s responses exhibited
a distinctive trait that often affected its correctness specificity. In
Example 1 of Figure A5, when the input question mentioned “diabetes and
neuropathy,” GPT-4 mirrored this phrasing literally. In contrast, the human
expert interpreted it as “diabetic neuropathy.” This highlights GPT-4’s
tendency toward a literal approach without interpretation, which can be either
advantageous or limiting. In Example 2 of Figure A5, GPT-4 simply reformulated
the input question about tests and their locations, while the human inferred a
broader query about tests and treatments. In both cases, GPT-4’s summaries
leaned toward literalness, a trait that readers sometimes favored and
sometimes did not. In future work, a systematic exploration of model
temperature could further illuminate this trade-off.

Figure 8: Annotation of two radiologist report examples from the reader study.
In the top example, GPT-4 performs better due to a laterality mistake by the
human expert. In the bottom example, GPT-4 exhibits a lack of conciseness. The
table (lower left) contains reader scores for these two examples and the task
average across all samples.

####  5.2.3 Conciseness

Conciseness is measured by the reader’s response to the following question:
“Which summary contains less non-important information?” Ideally, summaries
contain minimal information which do not have consequences for clinical care.
Overall, GPT-4 and human summaries were comparable. However, GPT-4 performed
slightly worse on radiology reports, which could suggest a trade-off between
completeness and correctness vs. conciseness (Figure 6). See Figure 8 for an
example in which GPT-4’s summary includes correct information which readers
deemed non-important.

While this aforementioned trade-off may be unavoidable, we suggest it could be
mitigated with improved prompting. Of the task-specific instructions in Table
3, the other two tasks explicitly specify summary length, e.g. “15 words or
less” for patient health questions. These phrases are included so that model
summaries are generated with similar lengths to the human summaries, enabling
a clean comparison. Length specification in the radiology reports instruction
was more vague, i.e. “…with minimal text,” hence imposing a softer constraint
on the model. We leave further study of prompt instructions (“prompt
engineering”) to future work.

Figure 9:  Spearman correlation coefficients between NLP metrics and reader
preference assessing completeness, correctness, and conciseness. The semantic
metric (BERTScore) and conceptual metric (MEDCON) correlate most highly with
correctness. Meanwhile, syntactic metrics BLEU and ROUGE-L correlate most with
completeness. Section 5.3 contains further description and discussion.

###  5.3 Connecting quantitative and clinical evaluations

We now provide intuition connecting NLP metrics and clinical reader scores.
Note that in our work, these tools measure different quantities: NLP metrics
measure the similarity between two summaries, while reader scores measure
which summary is better. Consider an example where two summaries are exactly
the same: NLP metrics would yield the highest possible score (100), while
clinical readers would provide a score of 0 to denote equivalence. As the
magnitude of a reader score increases, the two summaries are increasingly
dissimilar, hence yielding a lower quantitative metric score.

Given this intuition, we now introduce Figure 9 to capture the correlation
between NLP metrics and physicians’ preference. These values are calculated as
the Spearman correlation coefficient between NLP metric scores and the
magnitude of the reader scores. Since these features are inversely correlated,
for clarity we display the negative correlation coefficient values.

Compared to other metrics, BLEU correlates most with completeness and least
with conciseness. Given that BLEU measures sequence overlap, this result seems
reasonable, as more text provides more “surface area” for overlap; more text
also reduces the brevity penalty that BLEU applies on generated sequences
which are shorter than the reference [52]. For correctness, BERTScore and
MEDCON correlate most strongly with reader preference. This implies that the
semantics (BERTScore) and concepts (MEDCON) measured by these metrics
correspond to correctness more effectively than syntactic metrics BLEU and
ROUGE-L.

Ultimately, our reader study highlights the limitations of NLP metrics,
especially as model-generated summaries become increasingly viable. These NLP
metrics rely on a reference, which we’ve demonstrated is inherently fallible.
Further, metrics only account for a partial amount of variance in the
evaluation outcomes. Hence we advocate for the essential component of human
evaluation when assessing the clinical feasibility of new methods.

###  5.4 Pitfalls

We now discuss weaknesses in our work which motivate further study.

In our quantitative analysis, we select state-of-the-art and highly regarded
LLMs with a diverse range of attributes. This includes the 7B-parameter tier
of open-source autoregressive models, despite some models such as Llama-2
having larger versions. We consider the benefit of larger models in Figure A3,
finding this improvement marginal for Llama-2 (13B) compared to Llama-2 (7B).
While there may exist open-source models which perform slightly better than
our selections, we do not believe this would meaningfully alter our
analysis—especially since the clinical reader study employs GPT-4, which is an
established state-of-the-art [84].

Model temperature and prompt phrasing can be very important for a LLM, as we
demonstrate in Table 4. However, we only search over three possible
temperature values. Further, we do not thoroughly engineer our prompts chosen
in Table 3; each was crudely chosen after trying 1-2 options over a small
dataset. While this highlights the potential for improvement, we’re also
encouraged that achieving strong performance does not require a thorough
temperature search or thorough prompt engineering, which has been shown to
significantly improve model performance [64, 74].

The radiology report human summaries occasionally recommend further studies or
refer to prior studies, e.g. “… not significantly changed from prior” in
Figure 8. These instances are out of scope for the LLM, as it does not have
access to prior studies nor the purview to make recommendations. As such,
physicians were told to disregard these instances in the clinical reader
study.

We emphasize that our study does not encompass all clinical document types,
and extrapolating our results is tentative. For instance, our progress notes
task employs ICU notes from a single medical center. These notes may be
structured differently from non-ICU notes or from ICU notes of a different
center. Additionally, more challenging tasks may require summarizing longer
documents or multiple documents of different types. Addressing these cases
demands two key advancements: (1) transcending GPT-4’s current context length
of 32,768 tokens, potentially through multi-query aggregation or methods which
increase context length [53, 20], and (2) introducing open-source datasets
that encompass broader tasks and lengthier documents. We thus advocate for
further development and expanded evaluation of summarization tasks before
incorporating LLMs into the clinical workflow.

##  6 Conclusion

In this research, we exhaustively evaluate the proficiency of LLMs in clinical
text summarization, analyzing eight models across a diverse set of
summarization tasks. Our quantitative results underscore the advantages of
adapting models to specific domains and types of documents. The ensuing
clinical reader study demonstrates that summaries generated by LLMs are often
favored over those created by human experts due to higher scores for
completeness and correctness. The subsequent qualitative exploration provides
deeper insights into the limitations of both LLMs and human experts. Novel
evidence from our research suggests a promising avenue for LLMs—not as
replacements for clinicians, but as tools to minimize documentation burden and
empower clinicians to provide better patient care.

##  7 Reproducibility

In an effort to disseminate these methods for further validation and clinical
impact, we will make our code publicly available no later than October 31st,
2023, at github.com/StanfordMIMI/clin-summ. While all datasets are publicly
available, we will share our pre-processed versions for those which do not
require Physionet [34] access: Open-i [17] (radiology reports), MeQSum [3]
(patient questions), and ACI-Bench [80] (dialogue).

##  8 Acknowledgements

We’re grateful to both Narasimhan Balasubramanian and the Accelerate
Foundation Models Academic Research (AFMAR) program at Microsoft, who both
provided Azure OpenAI credits. Further compute support was provided by One
Medical, which Asad Aali used as part of his summer internship. Curtis
Langlotz is supported by NIH grants R01 HL155410, R01 HL157235, by AHRQ grant
R18HS026886, by the Gordon and Betty Moore Foundation, and by the National
Institute of Biomedical Imaging and Bioengineering (NIBIB) under contract
75N92020C00021. Akshay Chaudhari receives support from NIH grants R01
HL167974, R01 AR077604, R01 EB002524, R01 AR079431, and P41 EB027060; from NIH
contracts 75N92020C00008 and 75N92020C00021; and from GE Healthcare, Philips,
and Amazon.

## References

  * [1] Asma Ben Abacha, Wen-wai Yim, Griffin Adams, Neal Snider and Meliha Yetisgen-Yildiz  “Overview of the MEDIQA-Chat 2023 Shared Tasks on the Summarization & Generation of Doctor-Patient Conversations”  In _Proceedings of the 5th Clinical Natural Language Processing Workshop_ , 2023, pp. 503–513
  * [2] Brian G Arndt, John W Beasley, Michelle D Watkinson, Jonathan L Temte, Wen-Jan Tuan, Christine A Sinsky and Valerie J Gilchrist  “Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations”  In _The Annals of Family Medicine_ 15.5 Annals Family Med, 2017, pp. 419–426
  * [3] Asma Ben Abacha and Dina Demner-Fushman  “On the Summarization of Consumer Health Questions”  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28th - August 2_ , 2019
  * [4] “Best Practices for Prompt Engineering with OpenAI API” Accessed: 2023-09-08, https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api, 2023  OpenAI
  * [5] Sue Bowman  “Impact of electronic health record systems on information integrity: quality and safety implications”  In _Perspectives in health information management_ 10.Fall American Health Information Management Association, 2013
  * [6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry and Amanda Askell  “Language models are few-shot learners”  In _Advances in neural information processing systems_ 33, 2020, pp. 1877–1901
  * [7] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li and Scott Lundberg  “Sparks of artificial general intelligence: Early experiments with gpt-4”  In _arXiv preprint arXiv:2303.12712_ , 2023
  * [8] Chi-Ping Chang, Ting-Ting Lee, Chia-Hui Liu and Mary Etta Mills  “Nurses’ experiences of an initial and reimplemented electronic health record use”  In _CIN: Computers, Informatics, Nursing_ 34.4 LWW, 2016, pp. 183–190
  * [9] Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster and Zhifeng Chen  “The best of both worlds: Combining recent advances in neural machine translation”  In _arXiv preprint arXiv:1804.09849_ , 2018
  * [10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica and Eric P. Xing  “Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality”, 2023  URL: https://lmsys.org/blog/2023-03-30-vicuna/
  * [11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton and Sebastian Gehrmann  “Palm: Scaling language modeling with pathways”  In _arXiv preprint arXiv:2204.02311_ , 2022
  * [12] H.W. Chung, L. Hou and S. Longpre  “Scaling Instruction-Finetuned Language Models”  In _https://doi.org/10.48550/arXiv.2210.11416_ , 2022
  * [13] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani and Siddhartha Brahma  “Scaling instruction-finetuned language models”  In _arXiv preprint arXiv:2210.11416_ , 2022
  * [14] Pritam Deka and Anna Jurek-Loughrey  “Evidence Extraction to Validate Medical Claims in Fake News Detection”  In _International Conference on Health Information Science_ , 2022, pp. 3–15  Springer
  * [15] Jean-Benoit Delbrouck, Maya Varma, Pierre Chambon and Curtis Langlotz  “Overview of the RadSum23 Shared Task on Multi-modal and Multi-anatomical Radiology Report Summarization”  In _Proceedings of the 22st Workshop on Biomedical Language Processing_ Toronto, Canada: Association for Computational Linguistics, 2023
  * [16] Dina Demner-Fushman, Sophia Ananiadou and K Bretonnel Cohen  “The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks”  In _The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks_ , 2023
  * [17] Dina Demner-Fushman, Sameer Antani, Matthew Simpson and George R. Thoma  “Design and development of a multimodal biomedical information retrieval system”  In _Journal of Computing Science and Engineering_ 6.2, 2012, pp. 168–177  DOI: 10.5626/jcse.2012.6.2.168
  * [18] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer  “Qlora: Efficient finetuning of quantized llms”  In _arXiv preprint arXiv:2305.14314_ , 2023
  * [19] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “Bert: Pre-training of deep bidirectional transformers for language understanding”  In _arXiv preprint arXiv:1810.04805_ , 2018
  * [20] Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng and Furu Wei  “LongNet: Scaling Transformers to 1,000,000,000 Tokens”, 2023  arXiv:2307.02486 [cs.CL]
  * [21] William J Duffy, Morris S Kharasch and Hongyan Du  “Point of care documentation impact on the nurse-patient interaction”  In _Nursing Administration Quarterly_ 34.1 LWW, 2010, pp. E1–E10
  * [22] Jesse M Ehrenfeld and Jonathan P Wanderer  “Technology as friend or foe? Do electronic health records increase burnout?”  In _Current Opinion in Anesthesiology_ 31.3 LWW, 2018, pp. 357–360
  * [23] Scott L Fleming, Alejandro Lozano, William J Haberkorn, Jenelle A Jindal, Eduardo P Reis, Rahul Thapa, Louis Blankemeier, Julian Z Genkins, Ethan Steinberg and Ashwin Nayak  “MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records”  In _arXiv preprint arXiv:2308.14089_ , 2023
  * [24] Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh  “Gptq: Accurate post-training quantization for generative pre-trained transformers”  In _arXiv preprint arXiv:2210.17323_ , 2022
  * [25] Yanjun Gao, Dmitriy Dligach, Timothy Miller, Matthew M Churpek and Majid Afshar  “Overview of the Problem List Summarization (ProbSum) 2023 Shared Task on Summarizing Patients’ Active Diagnoses and Problems from Electronic Health Record Progress Notes”  In _arXiv preprint arXiv:2306.05270_ , 2023
  * [26] Yanjun Gao, Timothy Miller, Majid Afshar and Dmitriy Dligach  “BioNLP Workshop 2023 Shared Task 1A: Problem List Summarization”  In _Proceedings of the 22nd Workshop on Biomedical Language Processing_ , 2023
  * [27] Esteban F Gershanik, Ronilda Lacson and Ramin Khorasani  “Critical finding capture in the impression section of radiology reports”  In _AMIA Annual Symposium Proceedings_ 2011, 2011, pp. 465  American Medical Informatics Association
  * [28] Emily Gesner, Priscilla Gazarian and Patricia Dykes  “The burden and burnout in documenting patient care: an integrative literature review”  In _MEDINFO 2019: Health and Wellbeing e-Networks for All_ IOS Press, 2019, pp. 1194–1198
  * [29] Joseph F Golob Jr, John J Como and Jeffrey A Claridge  “The painful truth: The documentation burden of a trauma surgeon”  In _Journal of Trauma and Acute Care Surgery_ 80.5 LWW, 2016, pp. 742–747
  * [30] Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn and Keno K Bressem  “MedAlpaca–An Open-Source Collection of Medical Conversational AI Models and Training Data”  In _arXiv preprint arXiv:2304.08247_ , 2023
  * [31] Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang and Weizhu Chen  “LoRA: Low-Rank Adaptation of Large Language Models”, 2021  arXiv:2106.09685 [cs.CL]
  * [32] Alistair Johnson  “MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports”  In _https://www.nature.com/articles/s41597-019-0322-0_ , 2019
  * [33] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi and Roger Mark  “Mimic-iv”  In _PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021)_ , 2020
  * [34] Alistair Johnson, Tom Pollard and Roger Mark  “MIMIC-III clinical database”  PhysioNet, 2020
  * [35] Charles E Kahn Jr, Curtis P Langlotz, Elizabeth S Burnside, John A Carrino, David S Channin, David M Hovsepian and Daniel L Rubin  “Toward best practices in radiology reporting”  In _Radiology_ 252.3 Radiological Society of North America, Inc., 2009, pp. 852–856
  * [36] Natasha Khamisa, Karl Peltzer and Brian Oldenburg  “Burnout in relation to specific contributing factors and health outcomes among nurses: a systematic review”  In _International journal of environmental research and public health_ 10.6 MDPI, 2013, pp. 2214–2240
  * [37] Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang and Felix Hill  “Can language models learn from explanations in context?”  In _arXiv preprint arXiv:2204.02329_ , 2022
  * [38] Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson and Emily Alsentzer  “Do We Still Need Clinical Language Models?”  In _arXiv preprint arXiv:2302.08091_ , 2023
  * [39] Xiang Lisa Li and Percy Liang  “Prefix-tuning: Optimizing continuous prompts for generation”  In _arXiv preprint arXiv:2101.00190_ , 2021
  * [40] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu and Ananya Kumar  “Holistic evaluation of language models”  In _arXiv preprint arXiv:2211.09110_ , 2022
  * [41] Chin-Yew Lin  “Rouge: A package for automatic evaluation of summaries”  In _Text summarization branches out_ , 2004, pp. 74–81
  * [42] Siru Liu, Allison B McCoy, Aileen P Wright, Babatunde Carew, Julian Z Genkins, Sean S Huang, Josh F Peterson, Bryan Steitz and Adam Wright  “Leveraging Large Language Models for Generating Responses to Patient Messages”  In _medRxiv_ Cold Spring Harbor Laboratory Press, 2023, pp. 2023–07
  * [43] Zhengliang Liu, Aoxiao Zhong, Yiwei Li, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Peng Shu, Cheng Chen and Sekeun Kim  “Radiology-GPT: A Large Language Model for Radiology”  In _arXiv preprint arXiv:2306.08666_ , 2023
  * [44] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei and Adam Roberts  “The Flan Collection: Designing Data and Methods for Effective Instruction Tuning”, 2023  arXiv:2301.13688 [cs.AI]
  * [45] Ilya Loshchilov and Frank Hutter  “Decoupled weight decay regularization”  In _arXiv preprint arXiv:1711.05101_ , 2017
  * [46] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada and Sayak Paul  “PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods”, https://github.com/huggingface/peft, 2022
  * [47] Yash Mathur, Sanketh Rangreji, Raghav Kapoor, Medha Palavalli, Amanda Bertsch and Matthew R Gormley  “SummQA at MEDIQA-Chat 2023: In-Context Learning with GPT-4 for Medical Summarization”  In _arXiv preprint arXiv:2306.17384_ , 2023
  * [48] Feng Nie, Meixi Chen, Zhirui Zhang and Xu Cheng  “Improving few-shot performance of language models via nearest neighbor calibration”  In _arXiv preprint arXiv:2212.02216_ , 2022
  * [49] Naoaki Okazaki and Jun’ichi Tsujii  “Simple and efficient algorithm for approximate dictionary matching”  In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_ , 2010, pp. 851–859
  * [50] OpenAI  “ChatGPT” Accessed: 2023-09-04, 2022  URL: https://openai.com/blog/chatgpt
  * [51] OpenAI  “GPT-4 Technical Report”, 2023  arXiv:2303.08774 [cs.CL]
  * [52] Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu  “Bleu: a method for automatic evaluation of machine translation”  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318
  * [53] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon and Christopher Ré  “Hyena hierarchy: Towards larger convolutional language models”  In _arXiv preprint arXiv:2302.10866_ , 2023
  * [54] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever  “Language models are unsupervised multitask learners”  In _OpenAI blog_ 1.8, 2019, pp. 9
  * [55] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J Liu  “Exploring the limits of transfer learning with a unified text-to-text transformer”  In _The Journal of Machine Learning Research_ 21.1 JMLRORG, 2020, pp. 5485–5551
  * [56] Raj M Ratwani, Erica Savage, Amy Will, Ryan Arnold, Saif Khairat, Kristen Miller, Rollin J Fairbanks, Michael Hodgkins and A Zachary Hettinger  “A usability and safety analysis of electronic health records: a multi-center study”  In _Journal of the American Medical Informatics Association_ 25.9 Oxford University Press, 2018, pp. 1197–1201
  * [57] Kenneth E Robinson and Joyce A Kersey  “Novel electronic health record (EHR) education intervention in large healthcare organization improves quality, efficiency, time, and impact on burnout”  In _Medicine_ 97.38 Wolters Kluwer Health, 2018
  * [58] Elvis Saravia  “Prompt Engineering Guide”  In _https://github.com/dair-ai/Prompt-Engineering-Guide_ , 2022
  * [59] Tait D Shanafelt, Lotte N Dyrbye, Christine Sinsky, Omar Hasan, Daniel Satele, Jeff Sloan and Colin P West  “Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction”  In _Mayo Clinic Proceedings_ 91.7, 2016, pp. 836–848  Elsevier
  * [60] Tian Shi, Yaser Keneshloo, Naren Ramakrishnan and Chandan K Reddy  “Neural abstractive text summarization with sequence-to-sequence models”  In _ACM Transactions on Data Science_ 2.1 ACM New York, NY, USA, 2021, pp. 1–37
  * [61] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis and Stephen Pfohl  “Large Language Models Encode Clinical Knowledge”  In _arXiv preprint arXiv:2212.13138_ , 2022
  * [62] Christine Sinsky, Lacey Colligan, Ling Li, Mirela Prgomet, Sam Reynolds, Lindsey Goeders, Johanna Westbrook, Michael Tutty and George Blike  “Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties”  In _Annals of internal medicine_ 165.11 American College of Physicians, 2016, pp. 753–760
  * [63] Luca Soldaini and Nazli Goharian  “Quickumls: a fast, unsupervised approach for medical concept extraction”  In _MedIR workshop, sigir_ , 2016, pp. 1–4
  * [64] Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister and Alexander M Rush  “Interactive and visual prompt engineering for ad-hoc task adaptation with large language models”  In _IEEE transactions on visualization and computer graphics_ 29.1 IEEE, 2022, pp. 1146–1156
  * [65] Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett and Justin F Rousseau  “Evaluating large language models on medical evidence summarization”  In _npj Digital Medicine_ 6.1 Nature Publishing Group UK London, 2023, pp. 158
  * [66] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang and Tatsunori B. Hashimoto  “Stanford Alpaca: An Instruction-following LLaMA model”  In _GitHub repository_ GitHub, https://github.com/tatsu-lab/stanford_alpaca, 2023
  * [67] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster and Steven Zheng  “Ul2: Unifying language learning paradigms”  In _The Eleventh International Conference on Learning Representations_ , 2022
  * [68] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan and Daniel Shu Wei Ting  “Large language models in medicine”  In _Nature Medicine_ Nature Publishing Group US New York, 2023, pp. 1–11
  * [69] Augustin Toma, Patrick R Lawler, Jimmy Ba, Rahul G Krishnan, Barry B Rubin and Bo Wang  “Clinical Camel: An Open-Source Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding”  In _arXiv preprint arXiv:2305.12031_ , 2023
  * [70] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava and Shruti Bhosale  “Llama 2: Open foundation and fine-tuned chat models”  In _arXiv preprint arXiv:2307.09288_ , 2023
  * [71] Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno and Ira Ktena  “Towards generalist biomedical ai”  In _arXiv preprint arXiv:2307.14334_ , 2023
  * [72] Dave Van Veen, Cara Van Uden, Maayane Attias, Anuj Pareek, Christian Bluethgen, Malgorzata Polacin, Wah Chiu, Jean-Benoit Delbrouck, Juan Manuel Zambrano Chaves and Curtis P Langlotz  “RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models”  In _arXiv preprint arXiv:2305.01146_ , 2023
  * [73] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser and Illia Polosukhin  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [74] Jiaqi Wang, Enze Shi, Sigang Yu, Zihao Wu, Chong Ma, Haixing Dai, Qiushi Yang, Yanqing Kang, Jinru Wu and Huawen Hu  “Prompt engineering for healthcare: Methodologies and applications”  In _arXiv preprint arXiv:2304.14670_ , 2023
  * [75] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai and Quoc V Le  “Finetuned language models are zero-shot learners”  In _arXiv preprint arXiv:2109.01652_ , 2021
  * [76] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf and Morgan Funtowicz  “Transformers: State-of-the-art natural language processing”  In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , 2020, pp. 38–45
  * [77] Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A Pfeffer, Jason Fries and Nigam H Shah  “The shaky foundations of large language models and foundation models for electronic health records”  In _npj Digital Medicine_ 6.1 Nature Publishing Group UK London, 2023, pp. 135
  * [78] Thomas R Yackel and Peter J Embi  “Unintended errors with EHR-based result management: a case series”  In _Journal of the American Medical Informatics Association_ 17.1 BMJ Group BMA House, Tavistock Square, London, WC1H 9JR, 2010, pp. 104–107
  * [79] Wen-wai Yim, Asma Ben Abacha, Neal Snider, Griffin Adams and Meliha Yetisgen  “Overview of the MEDIQA-Sum Task at ImageCLEF 2023: Summarization and Classification of Doctor-Patient Conversations”  In _CLEF 2023 Working Notes_ , CEUR Workshop Proceedings  Thessaloniki, Greece: CEUR-WS.org, 2023
  * [80] Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin and Meliha Yetisgen  “ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation”  In _arXiv preprint arXiv:2306.02022_ , 2023
  * [81] Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, EKU Fonseca, Henrique Lee, Zahra Shakeri and Andrew Ng  “Radiology Report Expert Evaluation (ReXVal) Dataset”, 2023
  * [82] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger and Yoav Artzi  “BERTScore: Evaluating Text Generation with BERT”  In _International Conference on Learning Representations_ , 2020  URL: https://openreview.net/forum?id=SkeHuCVFDr
  * [83] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang and Zican Dong  “A survey of large language models”  In _arXiv preprint arXiv:2303.18223_ , 2023
  * [84] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li and Eric Xing  “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena”  In _arXiv preprint arXiv:2306.05685_ , 2023

##  Appendix A Appendix

###  A.1 Quantitative evaluation

Figure A1:  Metric scores vs. number of in-context examples across models and
datasets. We also include the best model fine-tuned with QLoRA (FLAN-T5) as a
horizontal dashed line. Note the allowable number of in-context examples
varies significantly by model and dataset. Figure A2:  Summarization
performance comparing one in-context example (ICL) vs. QLoRA across all open-
source models on patient health questions. QLoRA typically outperforms ICL
with the better models (FLAN-T5, Llama-2), although this tends to shift when
using more in-context examples (see Figure A1). Figure 3 contains similar
results with the Open-i radiology report dataset.  Figure A3: Comparing
Llama-2 (7B) vs. Llama-2 (13B). As most data points are near or slightly above
the dashed lines denoting equivalence, we conclude that the larger Llama-2
model (13B parameters) delivers marginal improvement for clinical
summarization tasks compared to the 7B model. Note that each data point
corresponds to the average score of $s=250$ samples for a given experimental
configuration, i.e. {dataset $\times$ $m$ in-context examples}.

###  A.2 Clinical reader study and task examples

Table A1: Reader study results evaluating completeness, correctness, conciseness (columns) across individual readers. Scores are on the range [-10, 10], where positive scores denote GPT-4 is preferred to the human reference. Intensity of highlight colors blue (GPT-4 wins) or red (human wins) correspond to the score. See Figure 6 for further details and p-values. Task | Reader | Completeness | Correctness | Conciseness
---|---|---|---|---
| 1 | 7.1 $\pm$ 13.2 | 5.0 $\pm$ 9.3 | -0.6 $\pm$ 10.8
Radiology | 2 | 1.6 $\pm$ 5.8 | 1.2 $\pm$ 6.3 | -3.4 $\pm$ 6.0
reports | 3 | 2.9 $\pm$ 7.9 | 1.1 $\pm$ 4.3 | -1.1 $\pm$ 6.9
| Pooled | 3.9 $\pm$ 9.8 | 2.5 $\pm$ 7.2 | -1.7 $\pm$ 8.3
| 1 | 3.9 $\pm$ 14.2 | 1.5 $\pm$ 6.6 | 0.6 $\pm$ 6.0
Patient | 2 | 1.9 $\pm$ 11.5 | 0.7 $\pm$ 7.2 | 0.7 $\pm$ 7.2
questions | 3 | 3.8 $\pm$ 13.3 | 0.0 $\pm$ 0.0 | 0.0 $\pm$ 0.0
| Pooled | 3.1 $\pm$ 13.0 | 0.7 $\pm$ 5.6 | 0.4 $\pm$ 5.4
| 1 | 5.3 $\pm$ 12.7 | 2.0 $\pm$ 8.8 | 1.9 $\pm$ 7.4
Progress | 2 | 4.1 $\pm$ 13.7 | -1.5 $\pm$ 9.0 | -0.2 $\pm$ 2.4
notes | 3 | 6.8 $\pm$ 15.1 | 1.0 $\pm$ 5.0 | 0.1 $\pm$ 9.0
| Pooled | 5.4 $\pm$ 13.9 | 0.5 $\pm$ 8.0 | 0.6 $\pm$ 6.9
Figure A4:  Reader study user interface. For brevity, we display a short
radiology report example.  Figure A5: Annotation of two patient health
question examples from the reader study. The table (lower left) contains
reader scores for these two examples and the task average across all samples.
Figure A6: Annotation of a progress notes summarization example evaluated in
the reader study. The table (lower right) contains reader scores for this
example and the task average across all samples.  Figure A7: Annotation of a
progress notes summarization example evaluated in the reader study. The table
(lower right) contains reader scores for this example and the task average
across all samples.  Figure A8: Example of the doctor-patient dialogue
summarization task, including “assessment and plan” sections generated by both
a human expert and GPT-4. This task is discluded from the reader study due to
the unwieldiness of a reader parsing many transcribed conversations and
paragraphs with such high number of tokens.


# LAVA: Language Audio Vision Alignment for Data-Efficient Video Pre-Training

Sumanth Gurram  Andy Fang  David Chan  John Canny

###### Abstract

Generating representations of video data is of key importance in advancing the
field of machine perception. Most current techniques rely on hand-annotated
data, which can be difficult to work with, expensive to generate, and hard to
scale. In this work, we propose a novel learning approach based on contrastive
learning, LAVA, which is capable of learning joint language, audio, and video
representations in a self-supervised manner. We pre-train LAVA on the Kinetics
700 dataset using transformer encoders to learn representations for each
modality. We then demonstrate that LAVA performs competitively with the
current state-of-the-art self-supervised and weakly-supervised pre-training
techniques on UCF-101 and HMDB-51 video action recognition while using a
fraction of the unlabeled data.

Machine Learning, ICML

##  1 Introduction

Supervised learning has generally driven the progress video representation
learning, however, labeling datasets is both time-consuming and expensive,
making it especially hard to leverage large amounts of data using supervised
learning. Moreover, while attention-based architectures such as Dosovitskiy et
al. (2020) Arnab et al. (2021); Bertasius et al. (2021) have started to
outperform CNNs on key benchmarks, they often require much larger amounts of
training data than CNNs.

Self-supervised methods have emerged to answer the challenge; as powerful pre-
training strategies for vision tasks, they can scale to larger training
datasets without being constrained by labeling needs. One common approach is
to use data augmentation to learn representational invariants for vision Qian
et al. (2021); Jing et al. (2018). Instead of relying on hand-designed
augmentations for the visual modality, another approach is to exploit the
multi-modal nature of video and learn audio-visual correspondence, as seen
with contrastive methods such as those in Morgado et al. (2020b); Patrick et
al. (2021, 2020a); Morgado et al. (2020a); Korbar et al. (2018). Similarly,
other methods use text metadata from videos to learn joint visual-text
representations in a self/weakly-supervised manner Stroud et al. (2020);
Patrick et al. (2020b); Li & Wang (2020); Sun et al. (2019a); Miech et al.
(2020); Sun et al. (2019b).

Figure 1: For a given sample, the video, audio, and text data are all encoded
into embeddings. LAVA’s pre-training objective involves contrasting video
embeddings from one sample to audio and text embeddings from different samples
(dotted red lines) while aligning embeddings from the same sample (dotted
green lines). Additionally, LAVA will calculate a centroid from audio, video,
and text embeddings for each sample and contrast (solid red lines) or align
(solid green lines) embeddings to these centroids accordingly. Figure 2:
LAVA’s pre-training architecture includes modality-specific feature
extraction, attention-based encoding, cross-modal contrastive losses and
centroid contrastive loss. Implementation details are in Section 3.

A less common but even more label-efficient strategy is to learn audio, visual
and text representations together. While methods such as Akbari et al. (2021);
Alayrac et al. (2020); Chen et al. (2021) pre-train on these three modalities,
they do so leveraging HowTo100M, which has a massive 15 years of unlabeled
video data. Moreover, excepting Akbari et al. (2021), most of the above
techniques use highly modality-specific encoders (e.g. only CNNs for vision),
instead of exploring more generic, attention-based backbones for all
modalities.

These observations are the main motivation for LAVA, which introduces a more
data and label-efficient method for pre-training transformer encoders on
audio, visual, and text modalities for video data. To achieve this, our
proposed method uses novel cross-modal and centroid-based contrastive
objectives seen in Figure 1. We evaluate pre-trained LAVA on UCF-101 and
HMDB-51.

##  2 Pre-Training Approach

### Overview

Given a set of $n$ unlabelled videos $X$, each video $x_{i}\in X$ is
decomposed into different modalities $a_{i}$, $v_{i}$ and $t_{i}$, which are
audio, visual and text features, respectively. Details regarding modality-
specific extraction are in Section 3. Then, LAVA’s three encoders $f_{a}$,
$f_{v}$, and $f_{t}$ produce output embeddings given their respective modality
features $m_{i}$ as inputs. LAVA then uses projection functions to project
these embeddings to various multi-modal latent spaces (e.g. audio-video,
video-text and audio-video-text spaces) in some $\mathbb{R}^{d}$. Let us
denote $z_{m}$ as the embedding for modality $m$. We define projection
function $g_{m,m^{\prime}}$ to project $z_{m}$ and $z_{m^{\prime}}$, to a
multi-modal latent space $m,m^{\prime}$. In these multi-modal latent spaces,
we apply a contrastive framework to jointly compare embeddings using a
similarity function $s$, such that $\forall i,{m^{\prime}}\neq{m}$ we have
high $s(g_{m,m^{\prime}}(z_{m,i}),g_{m,m^{\prime}}(z_{m^{\prime},i})$ and
$\forall i\neq j,{m^{\prime}}\neq{m}$ we have low
$s(f_{m}(m_{i}),f_{m^{\prime}}({m^{\prime}}_{j}))$.

### Cross-Modal Contrastive Loss

We use the function

$s(\cdot,\cdot)=exp(g_{m,m^{\prime}}(z_{m,i})^{T}g_{m,m^{\prime}}(z_{m^{\prime},j})/\tau)$

as the similarity function, where $\tau$ denotes temperature, and the noise
contrastive estimation (NCE) Gutmann & Hyvärinen (2010) as our contrastive
loss, where positive pairs are embeddings from the same instance and negatives
are embeddings from difference instances. This objective is intended to make
LAVA a dictionary, effectively mapping input features $a_{i},v_{i},t_{i}$ to
unified representations $z_{i}$ in a multi-modal latent space. The NCE loss is
formulated below (for simplicity the projection functions are omitted):

$NCE(z_{m},z_{m^{\prime}})=-log(\frac{\sum_{i=0}^{N}exp(z_{m,i}^{T}z_{m^{\prime},i}/\tau)}{\sum_{i=0}^{N}exp(z_{m,i}^{T}z_{m^{\prime},i}/\tau)+\sum_{i=0}^{N}\sum_{j\neq{i}}^{N}exp(z_{m,i}^{T}z_{m^{\prime},j}/\tau)})$

(1)

Following Miech et al. (2020), we use NCE for audio-video and video-text
pairs:

$L_{AV}(z_{a},z_{v})=NCE(g_{av}(z_{a}),g_{av}(z_{v}))$ (2)
$L_{VT}(z_{v},z_{t})=NCE(g_{vt}(z_{v}),g_{vt}(z_{t}))$ (3)

### Centroid Contrastive Loss

LAVA also enforces audio-video-text correspondence via a novel centroid
contrastive loss. Following Chen et al. (2021), for a given instance $x_{i}$,
we calculate a centroid $c_{i}$ by averaging projected LAVA embeddings:
$c_{i}=(g_{avt}(z_{a,i})+g_{avt}(z_{v,i})+g_{avt}(z_{t,i}))/3$. $g_{avt}$
projects modal embeddings to a joint, tri-modal latent space. However, unlike
Chen et al. (2021) we do not k-means cluster the centroids via k-means
clustering and align embeddings to their centroid’s cluster assignment. Thus,
we avoid potential detriments of grouping representations into a fixed K
clusters across all batches and the additional training time needed for
clustering. Instead, we instead directly optimize for alignment between
embeddings and their centroid via the following loss:

$L_{AVT}(c,z_{a},z_{v},z_{t})=\sum_{m\in{a,v,t}}NCE(g_{avt}(z_{m}),c)$ (4)

Combining all losses, we define the total loss for LAVA’s pre-training:

$L_{LAVA}=L_{AV}+L_{VT}+L_{AVT}$ (5)

##  3 Experiments

Method | Dataset (years) | GPU Hours | Mod. | UCF | HMDB
---|---|---|---|---|---
RotNet3D Jing et al. (2018) | K600 (0.1) | - | V | 47.7 | 24.8
CBT Sun et al. (2019a) | K600 (0.1) | 1536 | VT | 54.0 | 29.5
MemDPC Han et al. (2020a) | K600 (0.1) | - | VF | 54.1 | 30.5
AVSF Xiao et al. (2020) | K400 (0.1) | - | AV | 54.1 | 30.5
CoCLR Han et al. (2020b) | K400 (0.1) | - | VF | 77.4 | 44.1
STiCA* Patrick et al. (2021) | K400 (0.1) | 2930 | AV | 77.0 | 48.2
CPD Li & Wang (2020) | IG300 (0.1) | - | VT | 83.7 | 54.7
CVRL Qian et al. (2021) | K600 (0.1) | - | V | 90.8 | 59.7
LAVA* (video only) | K700 (0.1) | 408 | AVT | 81.3 | 50.1
LAVA* (audio+video) | K700 (0.1) | 408 | AVT | 84.3 | -
MIL-NCE Miech et al. (2020) | HT (15) | 4608 | VT | 83.4 | 54.8
ELo Piergiovanni et al. (2020) | Y8M | 4608 | V | - | 64.5
VATT* Akbari et al. (2021) | HT (15) | 18432 | AVT | 89.6 | 65.2
BraVe Recasens et al. (2021) | AS (1) | - | AV | 93.6 | 70.8
MMV† Alayrac et al. (2020) | AS+HT (16) | 2304 | AVT | 95.2 | 75.0
WTS Stroud et al. (2020) | WTS70M (22) | 9216 | VT | 95.8 | 77.7
Table 1: * denotes that the vision backbone has a transformer encoder. †
denotes that the vision backbone is fine-tuned downstream, rather than being
frozen for linear evaluation. The duration of each dataset is also measured in
years.

### Pre-training

We use the training set of Kinetics-700: 480k videos of which 300k videos have
usable audio and web-crawled titles. To the best of our knowledge, LAVA is the
first method to extract this text data for Kinetics-700 and we intend to
release this data for others to use. Video clips are [16 x 224 x 224 x 3] at
10 fps, with random h-flip and space-crop augmentations. Audio features are
[80, 256] log mel-spectrograms augmented with Gaussian noise. Text sequences
have a maximum length of 128 from a 48k vocab size using BPE tokenization.
Most video transformers use ImageNet-pre-trained ViT Arnab et al. (2021);
Bertasius et al. (2021), pre-train on massive video datasets like HowTo100M as
in Akbari et al. (2021) or use convolutions followed transformers as in
Patrick et al. (2021). We follow the latter, except we freeze a ImageNet-pre-
trained ResNet-18 backbone and use it to extract frame-wise patch features for
each clip. Audio features are encoded by an MLP. Text tokens are mapped to
embeddings. As seen in Figure 2, audio, video and text features are then
encoded by transformer encoders $f_{a}$, $f_{v}$, and $f_{t}$, each with 4
layers and a hidden size of 1024. Missing modalities in a batch are masked out
during loss calculation. We use a batch size of 32, 0.07 temperature,
$1e^{-5}$ learning rate, Adam optimizer, and a cosine-learning schedule. Pre-
training is done for 25 epochs on a single Titan X GPU.

### UCF-101 Downstream

The UCF-101 dataset has 13k videos across 101 action categories. We train a
linear classifier on top of the frozen LAVA video encoder using a 1e-4
learning rate and a batch size of 32. Also, we compare the downstream
performance of the classifier when using the frozen video encoder vs. frozen
audio and video encoders. Logits are averaged across multiple clips per test
set video. Top-1 accuracy is averaged across all 3 splits.

### HMDB-51 Downstream

The HMDB-51 dataset has 7k videos across 51 action categories. We follow the
UCF-101 evaluation procedure, except we do not evaluate with audio+video mode
as most HMDB-51 videos have no audio. Otherwise, our procedure follows
UCF-101.

##  4 Results

Pre-train | Downstream | UCF-101 | HMDB-51
---|---|---|---
AV | Video Only | 68.6 | 41.5
AV | Audio+Video | 72.4 | -
AV+VT | Video Only | 74.91 | 49.4
AV+VT | Audio+Video | 79.36 | -
AV+VT+AVT | Video Only | 81.1 | 51.8
AV+VT+AVT | Audio+Video | 84.2 | -
Table 2: All results are split-1 top-1 accuracy. Pre-train denotes which pre-
training losses are used.

### Downstream Tasks

Table 1 compares LAVA linear evaluation performance on UCF-101 and HMDB-51 to
various self-supervised benchmarks. LAVA outperforms all methods trained on
datasets comparable to Kinetics-700 size besides Li & Wang (2020); Qian et al.
(2021). This can be attributed to LAVA’s increased data efficiency as it makes
use of audio, video, and text for each sample, whereas these benchmarks rely
on at most two modalities. Notably, LAVA significantly outperforms Sun et al.
(2019a), which omits audio and uses ASR-extracted captions as text, whereas
LAVA is pre-trained on audio and video titles. LAVA also slightly outperforms
Patrick et al. (2021), which has a R(2+1)-18+transformer video encoder and
ResNet audio encoder, but does not pre-train on text. LAVA performs
competitively with Li & Wang (2020), which also uses video titles as text, and
is outperformed by Qian et al. (2021), which uses the video modality using
extensive spatio-temporal augmentations and a large batch size of 1024.
However, both methods Li & Wang (2020); Qian et al. (2021) use ResNet3D-50 as
their backbone, which is more performant when trained from scratch than
transformer encoders trained with the same amount of data Bertasius et al.
(2021). See Section 5 for plans to directly compare for to these methods.

Interestingly, Miech et al. (2020), despite having been pre-trained on over
150X samples than Kinetics-700, only slightly outperforms LAVA on UCF-101 and
HMDB-51. We believe this is because Miech et al. (2020) uses caption-based
text and does not use audio, whereas LAVA is pre-trained on audio and titles-
based text. Akbari et al. (2021); Alayrac et al. (2020) outperform LAVA using
audio, video and caption-based text, while Stroud et al. (2020) uses video and
titles-based text. While this performance gap can likely be attributed to
LAVA’s reliance on less than 1% of the data used in these benchmarks, it does
seem using audio and titles-based text can increase data efficiency as LAVA’s
performance rivals that of Miech et al. (2020).

Given that LAVA uses Kinetics-700 video titles as text, it can be argued that
this is a form of weak supervision as per Li & Wang (2020); Stroud et al.
(2020). We found that while all HMDB-51 classes are fully covered by
Kinetics-700 text, the following UCF-101 classes are not covered: IceDancing,
PizzaTossing, PommelHorse, SkiJet, StillRings. The 3-split average top-1
accuracy for these classes is 79.9%, which is comparable to LAVA’s overall
81.3% for UCF-101, indicating that LAVA’s representations transfer well to
unseen classes downstream.

Since LAVA is a multi-modal model, we use our novel evaluation strategy to
quantify downstream performance improvements when using embeddings from audio
and video modalities. Table 1 shows that audio-video LAVA outperforms video-
only by 3%, indicating that audio embeddings may provide additional
information for action recognition. See Section 5 for more plans in this
direction.

Lastly, pre-training LAVA takes 408 GPU hours, which is over 70% faster than
all other benchmarks documented in Table 1. Notably, LAVA also completes all
pre-training using just 1 GPU, whereas all other benchmarks use few as 4 and
as many as 256 accelerators. This combination of 70% fewer accelerators and
70% fewer GPU hours means LAVA is significantly more efficient in terms of
computational cost, in addition to label and data-efficiency.

### Ablations

As shown in Table 3, we ablate pre-training duration by evaluating LAVA at 1,
10, and 25 epochs of pre-training; video-only increases from 73.1% to 80.0% to
81.3%, while audio+video increases from 76.0% to 82.3% to 84.3%. Additionally,
we ablate pre-training losses as seen in Table 2. As expected, LAVA pre-
trained without text and centroid contrastive loss performs the worst. LAVA
pre-trained with $L_{AV}$ and $L_{VT}$, but no centroid loss performs worse
than LAVA pre-trained with the centroid loss. This indicates that the text
modality and the centroid loss, both of which increase data efficiency by
making greater use of the same number of samples, significantly improve
downstream video action recognition. The performance boost from using
audio+video embeddings during linear evaluation is also seen in Table 2.

$0$$5$$10$$15$$20$$25$$30$$70$$75$$80$$85$$90$# Pre-training EpochsUCF-101
Top-1 AccuracyVideo OnlyAudio+Video

Figure 3: UCF-101 accuracy as a function of pre-training duration.

##  5 Conclusion

We present LAVA, a novel self-supervised pre-training method for learning
audio, visual, and text representations from unlabeled video data using
transformers. By using multiple modalities and introducing novel cross-modal
and centroid contrastive objectives, LAVA increases data efficiency while
performing competitively with self and weakly supervised benchmarks. As LAVA’s
pre-training effectively combines multi-modal data and generic transformer
encoders, we believe it can scale both in terms of the amount on unlabelled
data and the number of modalities in the data.

### Future Work

To more directly compare LAVA performance to Li & Wang (2020); Qian et al.
(2021), we plan to pre-train the transformer vision encoder on ImageNet first
or use the ResNet3D-50 encoder from scratch, as well as larger batch sizes for
contrastive pre-training and more extensive video augmentation. We also plan
to pre-train LAVA on larger datasets with non-title-based text (e.g.
HowTo100M) to compare more directly with Akbari et al. (2021); Alayrac et al.
(2020). Since contrastive pre-training with audio may dilute information in
the video embeddings, we plan to compare downstream performance between video
embeddings from VT-pretrained LAVA and those of current LAVA. Additionally, we
plan to ablate how different coefficients between various LAVA losses will
affect downstream performance. Lastly, we intend to explore pre-training on
more/different modalities and new downstream tasks (e.g. retrieval,
captioning, video/audio generation) using LAVA.

## References

  * Akbari et al. (2021) Akbari, H., Yuan, L., Qian, R., Chuang, W.-H., Chang, S.-F., Cui, Y., and Gong, B.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Alayrac et al. (2020) Alayrac, J.-B., Recasens, A., Schneider, R., Arandjelović, R., Ramapuram, J., De Fauw, J., Smaira, L., Dieleman, S., and Zisserman, A.  Self-supervised multimodal versatile networks.  _Advances in Neural Information Processing Systems_ , 33:25–37, 2020.
  * Arnab et al. (2021) Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C.  Vivit: A video vision transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 6836–6846, 2021.
  * Bertasius et al. (2021) Bertasius, G., Wang, H., and Torresani, L.  Is space-time attention all you need for video understanding.  _arXiv preprint arXiv:2102.05095_ , 2(3):4, 2021\.
  * Chen et al. (2021) Chen, B., Rouditchenko, A., Duarte, K., Kuehne, H., Thomas, S., Boggust, A., Panda, R., Kingsbury, B., Feris, R., Harwath, D., et al.  Multimodal clustering networks for self-supervised learning from unlabeled videos.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8012–8021, 2021.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Gutmann & Hyvärinen (2010) Gutmann, M. and Hyvärinen, A.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Teh, Y. W. and Titterington, M. (eds.), _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , volume 9 of _Proceedings of Machine Learning Research_ , pp. 297–304, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.  URL https://proceedings.mlr.press/v9/gutmann10a.html.
  * Han et al. (2020a) Han, T., Xie, W., and Zisserman, A.  Memory-augmented dense predictive coding for video representation learning.  In _European conference on computer vision_ , pp. 312–329. Springer, 2020a.
  * Han et al. (2020b) Han, T., Xie, W., and Zisserman, A.  Self-supervised co-training for video representation learning.  _Advances in Neural Information Processing Systems_ , 33:5679–5690, 2020b.
  * Jing et al. (2018) Jing, L., Yang, X., Liu, J., and Tian, Y.  Self-supervised spatiotemporal feature learning via video rotation prediction.  _arXiv preprint arXiv:1811.11387_ , 2018.
  * Korbar et al. (2018) Korbar, B., Tran, D., and Torresani, L.  Cooperative learning of audio and video models from self-supervised synchronization.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Li & Wang (2020) Li, T. and Wang, L.  Learning spatiotemporal features via video and text pair discrimination.  _arXiv preprint arXiv:2001.05691_ , 2020.
  * Miech et al. (2020) Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Sivic, J., and Zisserman, A.  End-to-end learning of visual representations from uncurated instructional videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9879–9889, 2020.
  * Morgado et al. (2020a) Morgado, P., Li, Y., and Nvasconcelos, N.  Learning representations from audio-visual spatial alignment.  _Advances in Neural Information Processing Systems_ , 33:4733–4744, 2020a.
  * Morgado et al. (2020b) Morgado, P., Vasconcelos, N., and Misra, I.  Audio-visual instance discrimination with cross-modal agreement.  _arXiv preprint arXiv:2004.12943_ , 2020b.
  * Patrick et al. (2020a) Patrick, M., Asano, Y. M., Kuznetsova, P., Fong, R., Henriques, J. F., Zweig, G., and Vedaldi, A.  Multi-modal self-supervision from generalized data transformations.  _arXiv preprint arXiv:2003.04298_ , 2020a.
  * Patrick et al. (2020b) Patrick, M., Huang, P.-Y., Asano, Y., Metze, F., Hauptmann, A., Henriques, J., and Vedaldi, A.  Support-set bottlenecks for video-text representation learning.  _arXiv preprint arXiv:2010.02824_ , 2020b.
  * Patrick et al. (2021) Patrick, M., Huang, P.-Y., Misra, I., Metze, F., Vedaldi, A., Asano, Y. M., and Henriques, J. F.  Space-time crop & attend: Improving cross-modal video representation learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 10560–10572, 2021.
  * Piergiovanni et al. (2020) Piergiovanni, A., Angelova, A., and Ryoo, M. S.  Evolving losses for unsupervised video representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 133–142, 2020.
  * Qian et al. (2021) Qian, R., Meng, T., Gong, B., Yang, M.-H., Wang, H., Belongie, S., and Cui, Y.  Spatiotemporal contrastive video representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6964–6974, 2021.
  * Recasens et al. (2021) Recasens, A., Luc, P., Alayrac, J.-B., Wang, L., Strub, F., Tallec, C., Malinowski, M., Pătrăucean, V., Altché, F., Valko, M., et al.  Broaden your views for self-supervised video learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 1255–1265, 2021.
  * Stroud et al. (2020) Stroud, J. C., Lu, Z., Sun, C., Deng, J., Sukthankar, R., Schmid, C., and Ross, D. A.  Learning video representations from textual web supervision.  _arXiv preprint arXiv:2007.14937_ , 2020.
  * Sun et al. (2019a) Sun, C., Baradel, F., Murphy, K., and Schmid, C.  Learning video representations using contrastive bidirectional transformer.  _arXiv preprint arXiv:1906.05743_ , 2019a.
  * Sun et al. (2019b) Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C.  Videobert: A joint model for video and language representation learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 7464–7473, 2019b.
  * Xiao et al. (2020) Xiao, F., Lee, Y. J., Grauman, K., Malik, J., and Feichtenhofer, C.  Audiovisual slowfast networks for video recognition.  _arXiv preprint arXiv:2001.08740_ , 2020.