# A Survey for Biomedical Text Summarization: From Pre-trained to Large
Language Models

Qianqian Xie  Department of Computer Science, University of
ManchesterManchesterUnited Kingdom<EMAIL_ADDRESS>,  Zheheng Luo
Department of Computer Science, University of ManchesterManchesterUnited
Kingdom<EMAIL_ADDRESS>,  Benyou Wang  School of Data
Science, The Chinese University of Hong KongShengzhenChina
<EMAIL_ADDRESS>and  Sophia Ananiadou  Department of Computer Science,
University of ManchesterManchesterUnited Kingdom
<EMAIL_ADDRESS>
(2018)

###### Abstract.

The exponential growth of biomedical texts such as biomedical literature and
electronic health records (EHRs), poses a significant challenge for clinicians
and researchers to access clinical information efficiently. To tackle this
challenge, biomedical text summarization (BTS) has been proposed as a solution
to support clinical information retrieval and management. BTS aims at
generating concise summaries that distill key information from single or
multiple biomedical documents. In recent years, the rapid advancement of
fundamental natural language processing (NLP) techniques, from pre-trained
language models (PLMs) to large language models (LLMs), has greatly
facilitated the progress of BTS. This growth has led to numerous proposed
summarization methods, datasets, and evaluation metrics, raising the need for
a comprehensive and up-to-date survey for BTS. In this paper, we present a
systematic review of recent advancements in BTS, leveraging cutting-edge NLP
techniques from PLMs to LLMs, to help understand the latest progress,
challenges, and future directions. We begin by introducing the foundational
concepts of BTS, PLMs and LLMs, followed by an in-depth review of available
datasets, recent approaches, and evaluation metrics in BTS. We finally discuss
existing challenges and promising future directions in the era of LLMs. To
facilitate the research community, we line up open resources including
available datasets, recent approaches, codes, evaluation metrics, and the
leaderboard in a public project: https://github.com/KenZLuo/Biomedical-Text-
Summarization-Survey/tree/master. We believe that this survey will be a useful
resource to researchers, allowing them to quickly track recent advancements
and provide guidelines for future BTS research within the research community.

Large language models, Pre-trained language models, Biomedical texts, Text
summarization

††copyright: acmcopyright††journalyear: 2018††doi: XXXXXXX.XXXXXXX††ccs:
Information systems Summarization††ccs: Computing methodologies Natural
language generation††ccs: Computing methodologies Language resources††ccs:
Applied computing Bioinformatics

##  1\. Introduction

The rapidly increasing of unstructured clinical text information, such as
biomedical literature (Lu, 2011) and clinical notes (Jensen et al., 2012),
presents a significant challenge for researchers and clinicians seeking
effective access to required information. To tackle the challenge, the field
of biomedical text summarization (BTS) (Maybury, 1999; Xie et al., 2023c) has
emerged to assist users in seeking information efficiently. It aims to shorten
single or multiple lengthy biomedical documents into condensed summaries that
capture the most important information (Mishra et al., 2014). By reading these
summaries, researchers and clinicians can quickly grasp the main ideas of
extensive biomedical texts, saving time and effort. The BTS task has diverse
real-world applications including but not limited to aiding evidence-based
medicine (Przybyła et al., 2018), clinical information management (Esteva et
al., 2021), and clinical decision support (Demner-Fushman et al., 2009).

There has been a remarkable advancement in the field of BST in recent years,
due to the evolution of fundamental NLP techniques ranging from pre-trained
language models (PLMs) (Kenton and Toutanova, 2019) to large language models
(LLMs) (Brown et al., 2020). PLMs, namely language models pre-trained with the
large scale of unlabeled data in a self-supervised manner, possess the ability
to capture the common sense and lexical knowledge inherited in the training
data (Roberts et al., 2020). With the acquired knowledge, PLMs have
significantly enhanced a wide range of NLP tasks with the process of fine-
tuning. Researchers further enhanced the efficiency of PLMs by increasing the
size of models and training datasets, resulting in the emergence of large
language models (LLMs) (Brown et al., 2020; Ouyang et al., 2022). Compared
with PLMs, LLMs possess a remarkable ability for natural language
understanding and generation. Moreover, LLMs possess the emergent ability of
in-context learning, allowing them to perform diverse tasks without the need
for supervised training, simply by following natural language instructions
(Wei et al., 2023). For example, GPT-4 (Bubeck et al., 2023), an LLM released
by OpenAI, has shown human-level capabilities in the zero-shot setting across
various domains, including vision, coding, mathematics, medicine et al. This
growth has revolutionized NLP and also led to numerous proposed summarization
methods, datasets, and evaluation metrics for BTS (Wallace et al., 2021; Luo
et al., 2022; Guo et al., 2021b). In these methods, general-domain PLMs such
as BERT (Kenton and Toutanova, 2019) and BART (Lewis et al., 2020), or domain-
specific PLMs such as BioBERT (Lee et al., 2020) are employed as the backbone
model for encoding input texts and then fine-tuned with the unstructured
biomedical dataset of the BTS task. It allows the semantic knowledge captured
in PLMs to be transferred to the BTS task, resulting in more conclusive and
informative summaries. Latest, there is an increasing number of studies
exploring LLMs for zero-shot summary generation and evaluation of biomedical
texts (Xie et al., 2023b; Shaib et al., 2023), where LLMs have shown
comparable or even better performance than PLMs without any supervised
training.

These advances raise the need for a comprehensive and up-to-date survey for
BTS. Although the availability of previous surveys for BTS with traditional
machine learning and deep learning techniques (Afantenos et al., 2005; Mishra
et al., 2014; Pivovarov and Elhadad, 2015; Wang et al., 2021a), there exists a
literature gap concerning recent developments in BTS based on PLMs and LLMs.
This paper aims to bridge this gap by conducting an in-depth survey of the
recent work exploring PLMs and LLMs for BTS. We conduct a systematic review of
benchmark datasets, summarization approaches, and evaluation methods of the
task. We categorize and delve into a detailed discussion of existing methods
according to their leveraging of PLMs and LLMs, and analyze their limitations
and outlook for future directions. We hope this paper can be a timely survey
for researchers in the research community to quickly track recent progress,
challenges, and promising future directions.

Compared with existing surveys Afantenos et al (Afantenos et al., 2005) was
the earliest survey that summarized traditional NLP and machine learning
methods for medical document summarization. Mishra et al (Mishra et al., 2014)
reviewed text summarization methods for biomedical literature and electronic
health records (EHRs), between January 2000 and October 2013. Pivovarov et al
(Pivovarov and Elhadad, 2015) examined automated summarization methods for
electronic health records. Most methods summarized in these surveys are
traditional machine learning methods based on feature engineering. With the
prosperity of deep learning since 2014, deep neural networks became the
mainstream method for biomedical text summarization. Recently, Wang et al
(Wang et al., 2021a) investigated deep learning-based text summarization
approaches for both biomedical literature and EHRs between January 2013 to
April 2021. Nevertheless, recent research with PLMs and LLMs for biomedical
text summarization was not included in this survey. Wang et al (Wang et al.,
2021b) summarized PLMs-based methods for biomedical NLP, and briefly
introduced PLMs for the biomedical text summarization task as one of the
various tasks. Compared with them, we provide a more comprehensive and focused
overview for PLMs and LLMs on biomedical text summarization including
benchmark datasets, evaluation metrics, and limitations et al. To the best of
our knowledge, our paper is the first review that surveys recent LLMs and
PLMs-based methods for BTS.

Paper collection We collect representative works since 2018 that are published
in conferences and journals of computer science and biomedical science such as
ACL, EMNLP, COLING, NAACL, AAAI, Bioinformatics, BioNLP, JAMIA, AMIA, NPJ
digital medicine et al. We use PubMed, google scholar, ACM Digital Library,
and IEEE Xplore Digital Library as the search engine and the database. We
search with keywords including ”biomedical/medical/clinical summarization”,
”medical dialogue summarization”, ”radiology report summarization”, ”large
language models”, ”pre-trained language models”, ”medical/biomedical/clinical
evaluation”, ”medical/biomedical/clinical datasets”.

Organization of the paper We will first introduce the background of biomedical
text summarization, pre-trained language models, and large language models in
Section 2. Then Section 3 will describe existing benchmark datasets.
Representative methods with PLMs and LLMs will be categorized and discussed in
Section 4. We introduce evaluation methods in Section 5. We next discuss
limitations and future directions in Section 6. Finally, we make a conclusion
in Section 7. Figure 1 shows the proposed overview of biomedical text
summarization with pre-trained language models and large language models.

Figure 1. Overview of biomedical text summarization with PLMs and LLMs. Figure
2. Overview of background. Figure 3. The example of extractive and abstractive
biomedical text summarization.

##  2\. Background

In this section, we first review biomedical text summarization, pre-trained
language models, and large language models, which are essential concepts used
in this survey. The overview of the background section is shown in Figure 2.

###  2.1. Biomedical Text Summarization

Biomedical text summarization (Pivovarov and Elhadad, 2015; Wang et al.,
2021a) aims to shorten single or multiple biomedical documents into a
condensed summary that preserves the most important information. In general,
automated summarization approaches are divided into extractive summarization
methods (Nallapati et al., 2017) and abstractive summarization methods (Lin
and Ng, 2019) according to the output of summaries as shown in Figure 3.
Extractive methods select key sentences from original documents and
concatenate them into a summary, while abstractive methods generate new
sentences as the summary based on the original documents. Compared with
extractive summarization, abstractive summarization presents a more
significant challenge since it involves generating informative sentences from
a large vocabulary, syntactic adjustment, and paraphrasing, to produce
factually consistent summaries. Formally, let’s assume $C$ as a biomedical
corpus with $D$ documents, $d\in C$ is a document consisting of $m$ sentences:
$d=\\{s_{1},\cdots,s_{m}\\}$. We also assume the gold summary of the document
$d$ as $t_{d}$. For biomedical scientific papers, abstract of papers are
generally deemed as their gold summaries.

Automatic biomedical summarization methods are largely facilitated and
inspired by NLP methods in the general domain. The earliest methods are
traditional machine learning methods such as Naive-Bayes classifier (Kupiec et
al., 1995) and graph-based ranking methods such as TextRank (Mihalcea and
Tarau, 2004). With the prosperity of deep learning since 2014, neural network
methods (LeCun et al., 2015) have been the mainstream method for both
extractive and abstractive summarization of biomedical texts. Neural
extractive methods (Nallapati et al., 2017) formulate the extractive task as
the binary classification problem. This involves predicting the labels (1 or
0) of sentences in original documents to select the sentences to be included
in the summary. Neural abstractive methods (Rush et al., 2015) model the
abstractive task as the text generation problem that generates new sentences
based on the sequence-to-sequence (Sutskever et al., 2014) framework.

Extractive summarization For a document $d$ with $m$ sentences, extractive
summarization methods aim to select a subset of $o$ sentences from $d$, $o\ll
m$. Neural extractive methods can be classified into unsupervised methods and
supervised methods. The unsupervised methods (Mihalcea and Tarau, 2004; Erkan
and Radev, 2004) model the extractive task into the sentence ranking problem.
They usually represent sentences with advanced representation techniques such
as word embeddings, and use the unsupervised ranking method to select
important sentences based on their representations. Supervised methods
(Nallapati et al., 2017), adopt human-written abstractive summaries as the
gold summaries whose sentences are not in the original documents. To train the
extractive models, they are first required to generate binary labels for
sentences according to the gold summaries. To this end, they generally adopt
unsupervised sentence selection methods such as the greedy search algorithm to
generate the oracle summary for each document with sentences that are most
semantically similar to the gold summary. Therefore, sentences that are
included in the oracle summary are labeled with $1$, while the remaining
sentences are labeled with $0$.

Most supervised neural extractive methods consist of the neural network-based
encoder and classifier. The neural network-based encoder is used to capture
the contextual information of input documents and generate vector
representations of sentences. The classifier is to predict labels of sentences
according to their vector representations. The objective is to maximize the
log-likelihood of the observed labels of sentences:

(1) $\log p(y|C;\theta)=\sum_{d\in C}\sum_{i=1}^{m}\log p(y_{i}^{d}|d;\theta)$

where $y^{d}_{i}$ is the ground truth label of sentence $s_{i}$ in document
$d$, $\theta$ is the parameter set of the model.

Abstractive summarization Neural abstractive methods build the abstractive
task as the sequence-to-sequence learning problem. Most of them utilize the
encoder-decoder framework (Sutskever et al., 2014), which consists of the
neural network-based encoder and decoder. Similar to extractive methods, the
encoder is used to yield vector representations of input documents. The
decoder is to generate the target summary sequentially with representations
from the encoder. The model is optimized with the objective to maximize the
log-likelihood of target words in the gold summary.

(2) $\log p(t|C;\theta)=\sum_{d\in C}\sum_{i=1}^{n}\log p(t_{i}^{d}|d;\theta)$

where $t^{d}_{i}$ is the $i$-th word in the gold summary $t^{d}$ of the
document $d$, $n\ll m$.

In recent years, PLMs have become the dominant method of biomedical text
summarization. PLMs-based methods (Du et al., 2020; Xie et al., 2022a; Luo et
al., 2023, 2022; Xie et al., 2023c) have a similar framework to neural methods
for extractive and abstractive tasks, while PLMs are more powerful than neural
networks in encoding biomedical texts. We will next introduce the pre-trained
language models.

###  2.2. Pre-trained Language Models

Language model pre-training (Bengio et al., 2000; Mikolov et al., 2013) has
long been an active research area with the aim of learning low-dimensional
vector representations from natural language, which are applicable and
generalizable for downstream tasks. The earliest unidirectional neural
language models such as word2vec (Mikolov et al., 2013) and glove (Pennington
et al., 2014), learn meaningful word embeddings via estimating the probability
of the next word with the sequence of history words. The bidirectional
language models such as ELMo (Peters et al., 2018) are then proposed to
further consider the bidirectional context of words. Bidirectional Encoder
Representations from Transformers (BERT) (Kenton and Toutanova, 2019) is the
breakthrough work that advances the state-of-art of various NLP tasks. BERT
and its variants generally consist of two steps: pre-training and fine-tuning.
It proposes to first pre-train the deep models based on basic neural network
structure such as transformer (Vaswani et al., 2017) on the large scale of
unlabeled data with a self-supervised learning task. Then the pre-trained
parameters of deep models and task-specific parameters are fine-tuned on
labeled data with downstream tasks. To provide a more comprehensive
understanding of PLMs, we will further illustrate their core components. For
more details of PLMs, one can check the review (Qiu et al., 2020).

Model architecture The early language models such as ELMo and its predecessors
(Peters et al., 2017, 2018), generally utilize Bi-LSTM (Hochreiter and
Schmidhuber, 1997) as the backbone network structure, to capture bi-
directional contextual information of texts. However, Bi-LSTM has the
limitation of parallelization and sequential computation with the growth of
sequence length. One breakthrough work is Transformer (Vaswani et al., 2017),
with the self-attention-based model architecture, enabling it to handle
parallel computation and model long-range dependencies of sequences
efficiently. It follows the encoder-decoder architecture with stacked multi-
head self-attention and a point-wise fully connected feed-forward network.
Following the success of the Transformer model, most pre-trained language
models have adopted this architecture, and use deep network architecture. For
example, the base model of BERT has 12 Transformer layers with the hidden size
768 and 12 self-attention heads.

According to different model architectures, existing PLMs can be categorized
into three types: bi-directional encoder language models, auto-regressive
decoder language models, and encoder-decoder language models, as shown in
Figure 4. The bi-directional language models, i.e, BERT and its variants such
as Roberta (Liu et al., 2019), use only the encoder portion of Transformer
without using the decoder structure in Transformer and predict masked tokens
independently with the masked input sequence. However, they are hard to be
used for generation tasks due to the absence of a decoder structure. The auto-
regressive language models such as GPT series (Radford et al., [n. d.], 2019;
Brown et al., 2020), only use the decoder portion of the Transformer. They are
trained to predict tokens in an auto-regressive manner, making them suitable
for text-generation tasks. However, they only consider leftward tokens during
prediction and can only encode input tokens unidirectionally. In contrast,
encoder-decoder language models such as BART (Lewis et al., 2020) use the full
encoder and decoder architecture within Transformer. They incorporate the bi-
directional encoding of masked input tokens and offer enhanced flexibility
since there is no strict need to align the input tokens of the encoder and
output tokens in the decoder.

(a)

(b)

(c)

Figure 4. Comparison of different language models.

Training corpora The majority of pre-trained language models (PLMs) utilize
corpora from the general domain, such as BooksCorpus (Zhu et al., 2015) and
Wikipedia. However, these models often encounter out-of-vocabulary (OOV) words
that are not present in their vocabulary. To tackle this issue, PLMs employ
sub-word tokenization techniques such as Byte-Pair Encoding (BPE) (Sennrich et
al., 2016) or WordPiece (Kudo and Richardson, 2018) to construct their
vocabularies.

Pre-training The pre-training on a large scale of unlabeled data is the key
for language models to learn useful representations and parameters, which can
be fine-tuned to downstream tasks. The pre-training task of most previous
language models follows the unidirectional language model (Bengio et al.,
2000). It aims to maximize the log-likelihood of words conditionally on
history words:

(3) $\mathcal{L}_{lm}=-\sum_{t=1}^{T}\log p(x_{t}|x_{1},x_{2},\cdots,x_{t-1})$

where $X=\\{x_{1},\cdots,x_{T}\\}$ is a given text sequence with $T$ words.
The bidirectional language model is further proposed to capture contextual
information of text from both directions. It combines both the left-to-right
language model and the right-to-left language model:

(4) $\begin{split}\mathcal{L}_{blm}=&-\sum_{t=1}^{T}(\log
p(x_{t}|x_{1},x_{2},\cdots,x_{t-1})\\\ &+\log
p(x_{t}|x_{t+1},x_{t+1},\cdots,x_{T}))\end{split}$

Different from the bidirectional language model, PLMs such as BERT utilize the
masked language model (MLM), which allows bi-directional self-supervised pre-
training more efficiently. It randomly selects 15% tokens of the input text to
be predicted, in which 80% of them are replaced with the special token
”[MASK]”, and 10% of them are replaced with other words in the vocabulary. The
objective is to maximize the log-likelihood of ground-truth words in the
selected positions with masked text sequence:

(5) $\mathcal{L}_{mlm}=-\sum_{\hat{x}\in m(x)}\log p(\hat{x}|X_{m})$

where $X_{m}$ is the masked text sequence and $m(x)$ is the set of masked
words. Pre-training with MLM guides the language model to fully capture the
contextual information embedded in the token sequence and generate more
expressive representations on different levels such as tokens and sentences.

Supervised Fine-tuning Self-supervised pre-training on large-scale corpora
enables language models to acquire common sense knowledge and linguistic
understanding in their pre-trained parameters and contextual representations.
However, to effectively apply these models to specific downstream tasks, it is
crucial to adapt them and their generated representations through supervised
fine-tuning with task-specific objectives and datasets. In the context of
downstream tasks, task-specific inputs are initially fed into PLMs to generate
contextual representations. It requires choosing contextual representations on
different levels and different task-specific layers stacked on top of language
models, for corresponding tasks. For instance, in the case of the extractive
summarization task, previous research has commonly incorporated an additional
classification layer to predict sentence labels based on the sentence-level
representations generated by PLMs. By optimizing the model using the
classification loss, both the parameters of PLMs and the task-specific
parameters are refined. Through this fine-tuning process, the linguistic and
commonsense knowledge present in PLMs is effectively transferred into task-
specific representations. Thus, PLMs have significantly improved the
performance in various NLP tasks.

Biomedical Language Models Inspired by the great success of PLMs on NLP tasks,
much attention has been devoted to applying PLMs to the biomedical domain
including the task of biomedical text summarization. However, most PLMs,
including BERT, variants of BERT, T5 (Raffel et al., 2020) et al, are pre-
trained on texts of the general domain (Zhu et al., 2015). This poses a
significant challenge when directly applying these models to biomedical texts.
The greatest issue is the terminologies and compound words in biomedical
texts, most of which have never been mentioned in the general domain texts,
and thus can’t be covered by the vocabulary of PLMs.

To fill the gap, PLMs for the biomedical domain such as BioBERT (Lee et al.,
2020), BlueBERT (Peng et al., 2019), and ClinicalBERT (Alsentzer et al., 2019)
et al, have been proposed to further pre-train PLMs in the general domain with
biomedical texts. BioBERT (Lee et al., 2020) is the first biomedical language
model that further pre-trains BERT on biomedical scientific texts including
PubMed abstracts and PubMed Central full-text articles (PMC). Following it,
BlueBERT (Peng et al., 2019) conducts continual pre-training on clinical text
corpus MIMIC-III (Johnson et al., 2016) along with biomedical scientific
texts. Yet they still adopt the same vocabulary as BERT, which limits their
ability in encoding biomedical texts. Different from them, SciBERT (Beltagy et
al., 2019) builds a domain-specific vocabulary from scratch and conducts pre-
training on scientific literature, in which 12% articles are from the computer
science domain and 82% articles are from the biomedical domain. PubMedBERT (Gu
et al., 2021) pre-trains their models with scientific papers solely in the
biomedical domain. Compared with PLMs in the general domain, biomedical PLMs
can provide better representations that capture the contextual information of
both normal tokens and domain-specific terms. One can check the survey paper
(Wang et al., 2021b) for more details on pre-trained language models for the
biomedical domain.

###  2.3. Large Language Models

To improve the efficacy of language models, researchers have dedicated to
scaling up language models (Brown et al., 2020; Ouyang et al., 2022), by
increasing the number of model parameters, expanding the size of training
datasets, and enhancing computational resources. These lead to the emergence
of large language models, namely transformer-based language models with
billions of parameters and pre-trained with large amounts of text data. LLMs
usually have the same neural network architecture and pre-training task as
PLMs introduced above. The first representative LLM is GPT-3 (Brown et al.,
2020), proposed by OpenAI in 2020. It is an autoregressive decoder-based
language model with parameters up to 175B, which only uses the decoder part of
the transformer. Subsequently, most LLMs use the autoregressive decoder of the
transformer as the model architecture. With the increased scale, LLMs have
demonstrated remarkable advancements in natural language understanding,
generation, and reasoning compared to smaller PLMs. They exhibit significantly
enhanced abilities in handling complex language tasks. Moreover, researchers
have found that LLMs have the emergent ability (the ability not presented in
small PLMs but LLMs (Wei et al., 2022)) of in-context learning, namely the
capability to perform unseen tasks without any specific training data or with
only a few training samples, by following natural language instructions. Based
on representative LLMs such as GPT-3 (Raffel et al., 2020), recent efforts
have been explored in the following directions to improve the capability of
LLMs:

  * •

Instruction Tuning To enhance the generalization and in-context learning
capabilities of LLMs, researchers such as T0 (Sanh et al., 2022) and FLAN (Wei
et al., 2021), further explored fine-tuning LLMs with diverse tasks expressed
by natural language instruction, namely instruction tuning. The instruction
tuning, makes LLMs to better understand and respond to instructions, enabling
them to generalize more effectively to new tasks. During the instruction
tuning process, researchers (Chung et al., 2022) have found certain factors
including scaling the number of tasks, and model size and using chain-of-
thought tuning data, that contribute to improving LLMs’ generalization ability
and reasoning ability on complex tasks.

  * •

Human Alignment In addition to the capacity to follow instructions, it’s
crucial to ensure that large language models (LLMs) are guided by human
preferences, avoiding generating unfaithful and toxic information (Stiennon et
al., 2020). To achieve this, studies such as InstructGPT (Ouyang et al.,
2022), have explored fine-tuning LLMs with reinforcement learning from human
feedback (RLHF) (Christiano et al., 2017), where the human preferences are
used as the reward signal to fine-tune LLMs. RLHF has demonstrated
effectiveness in improving the truthfulness and toxicity of LLMs.
ChatGPT111https://openai.com/blog/chatgpt and GPT-4 (OpenAI, 2023) are two
representative LLMs trained with RLHF by OpenAI, They have shown remarkable
abilities in communicating with humans and dealing with complex tasks in
diverse domains such as coding, vision, medicine, and law.

  * •

Open Source Due to the high cost of pre-training LLMs, most representative
LLMs such as GPT-3, InstructGPT, ChatGPT, and GPT-4 are developed by
technology companies with enough computation resources for commercial use.
They are not openly accessible for public research or development. In response
to this, efforts have been made to introduce openly available LLMs to aid
future research. Notable examples include LLaMA (Touvron et al., 2023), a
collection of efficient LLMs with parameters from 7B to 65B, RedPajama
(Computer, 2023) with 7B parameters, and
Falcon222https://huggingface.co/blog/falcon with 40B parameters. Despite
having significantly fewer parameters, these openly accessible LLMs have
demonstrated performance on par with larger models like GPT-3, which has 175B
parameters, achieved through pre-training on extensive and high-quality text
data.

LLMs in Biomedical Domain LLMs have shown significant potential in the
biomedical domain (Nori et al., 2023), with applications such as diagnostic
assistance and drug discovery. For example, GPT-4, one of the prominent LLMs,
has outperformed the passing score of the United States Medical Licensing
Examination (USMLE) by 20 points. Studies have been proposed to tailor LLMs to
the biomedical domain by instruction tuning (Singhal et al., 2022) or
continual pre-training with biomedical text data (Wu et al., 2023). While
these techniques have further enhanced the performance of LLMs in medical
tasks, all studies indicate that current LLMs are not yet prepared for
reliable application in the medical field, due to their limitations in
providing accurate, factual, safe, and unbiased information (Xie et al.,
2023a).

##  3\. Datasets

Unstructured biomedical texts used in text summarization methods involve
various types, including biomedical literature, electronic health records
(EHRs), medical conversations, and medical questions. Details of these
datasets are summarized in Table 1.

Dataset | Category | Size | Content | Summarization Task
---|---|---|---|---
PubMed (Cohan et al., 2018) | Biomedical literature | 133,215 | Full contents of articles | Single
SumPubMed (Gupta et al., 2021) | Biomedical literature | 33,772 | Full contents of articles | Single
S2ORC (Bishop et al., 2022) | Biomedical literature | 63,709 | Full contents of articles | Single
CORD-19 (Wang et al., 2020) | Biomedical literature | - | Full contents of articles | Single
PubMedCite (Luo et al., 2023) | Biomedical literature | 192,744 | Full contents of articles, citation graph | Single
CDSR (Guo et al., 2021b) | Biomedical literature | 7,805 | Abstracts of articles | Single
PLOS (Luo et al., 2022) | Biomedical literature | 28,124 | Full contents of articles | Single
RCT (Wallace et al., 2021) | Biomedical literature | 4,528 | Titles and abstracts of articles | Multiple
MSˆ2 (DeYoung et al., 2021) | Biomedical literature | 470,402 | Abstracts of articles | Multiple
MIMIC-CXR (Johnson et al., 2019) | EHRs | 124,577 | Full contents of reports | Single
OpenI (Demner-Fushman et al., 2016) | EHRs | 3,599 | Full contents of reports | Single
HET-MC (Song et al., 2020) | Medical conversation | 109,850 | Multi-turn conversations | Single
MeQSum (Abacha and Demner-Fushman, 2019) | Medical question | 1,000 | Consumer health question | Single
CHQ-Summ (Yadav et al., 2022b) | Medical question | 1,507 | Full contents of question | Single
Table 1. Biomedical text summarization datasets.

Biomedical Literature With the exponentially growing of scientific papers,
developing automated summarization tools for biomedical articles has long
attracted much attention. Biomedical articles are usually written by domain
experts such as researchers and physicians. Compared with general domain texts
such as social media texts or news texts, they are typically less noisy and
exhibit a structured organization with standardized sections such as
”Introduction,” ”Methods,” ”Results,” and others.

For single document summarization, PubMed (Cohan et al., 2018) is one of the
most commonly used datasets, for summarization of long biomedical texts. It
consists of 133K scientific papers collected from the PubMed open access
repositories333https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/. It has
become a popular benchmark dataset for evaluating both general text
summarization methods and biomedical text summarization techniques. It is
noticed that Zhong et al (Zhong et al., 2020) further adapt the dataset that
only uses the introduction of texts as the input. To identify these two
settings on the dataset, we name the original PubMed dataset that uses full
contents of documents as the PubMed-Long444https://github.com/armancohan/long-
summarization, and the new dataset that is adapted by Zhong et al (Zhong et
al., 2020) as the PubMed-Short. Following it, SumPubMed (Gupta et al., 2021)
proposed recently, includes 33,772 documents from Bio Med Central (BMC) of
PubMed archive555https://github.com/vgupta123/sumpubmed. Bishop et al (Bishop
et al., 2022; Xie et al., 2022a) extract the subset from the large scientific
corpus S2ORC (Lo et al., 2020) and build the dataset
S2ORC666https://github.com/jbshp/GenCompareSum which includes 63,709 articles
from the biological and biomedical domain. Most recently, COVID-19 Open
Research Dataset (CORD-19777https://github.com/allenai/cord19) (Wang et al.,
2020) has attracted much attention, for which developing summarization systems
would facilitate relevant research and help against the COVID-19 pandemic.
CORD-19 contains millions of papers related to COVID-19, SARS-CoV-2, and other
coronaviruses. Luo et al (Luo et al., 2023) propose the
PubMedCite888https://github.com/zhehengluoK/PubMedCite-Builder dataset with
the 192K biomedical scientific papers and a citation graph preserving 917K
citation relationships between them, for supporting the biomedical scientific
paper summarization with the citation graph. Moreover, Guo et al (Guo et al.,
2021b) collected the
CDSR999https://github.com/qiuweipku/Plain_language_summarization dataset to
support the task of lay language summarization of biomedical scientific
reviews, which is a special kind of single document summarization that aims to
generate plain language abstract for lay people based on professional
abstracts from expertise. It contains 7,805 abstract pairs of biomedical
scientific reviews, in which professional abstracts of systematic reviews are
deemed as inputs and their corresponding plain language abstracts as target
summaries. Following it, Luo et al (Luo et al., 2022) present the PLOS
corpus101010http://www.nactem.ac.uk/readability/ with 28,124 biomedical papers
along with their technical summaries and plain summaries.

For multi-document summarization in the biomedical domain, Wallace et al
(Wallace et al., 2021) build the RCT111111https://github.com/bwallace/RCT-
summarization-data summarization dataset with 4,528 data samples searched from
PubMed121212https://pubmed.ncbi.nlm.nih.gov. The input of each data sample
includes titles and abstracts of related papers describing randomized
controlled trials (RCTs), while the conclusion section of the systematic
review from Cochrane131313https://www.cochranelibrary.com/ is treated as the
target summary. Similarly, Deyoung et al (DeYoung et al., 2021) developed the
MSˆ2141414https://github.com/allenai/ms2/ for multi-document summarization of
medical studies. It collected 470K papers from Semantic Scholar and 20K
reviews that summarized these papers.

Electronic Health Records Electronic health records have been widely adopted
by hospitals to store and manage medical information of patients, such as
diagnostic codes, medications, laboratory results, clinical notes et al. They
are also written by professionals with professional language and specific
structure. Different from scientific papers that are generally free to access,
EHRs may have restrictions on public access due to privacy issues. Several
publicly available datasets have been released to support the automated
summarization of radiology reports. The task aims to automatically generate
the impressions which highlight key observations of the radiology reports.
Demner et al (Demner-Fushman et al., 2016) collected the
OpenI151515https://openi.nlm.nih.gov/faq#collection datasets containing 3,996
chest x-ray reports from hospitals within the Indiana Network. Compared with
OpenI, MIMIC-CXR (Johnson et al., 2019) is a larger publicly available dataset
including 107,372 radiology reports from Beth Israel Deaconess Medical Center
Emergency Department between
2011–2016161616https://physionet.org/content/mimic-cxr/2.0.0/.

Medical Conversations Medical conversations between patients and doctors from
online healthcare systems have become an important source of medical
information, with the increasing usage of telemedicine. The automated
summarizing of key medical information during long medical conversations can
save much time for doctors and improve healthcare efficiency. Medical
conversations usually involve multi-turn interactions between two parties. The
patients focus on asking questions and solutions to their health problems and
describing their symptoms, while doctors would ask for detailed symptoms of
patients and provide diagnostic suggestions. Similar to EHRs, accessing
medical conversations at telemedicine platforms may have restrictions due to
privacy concerns. Moreover, it is time-consuming and expensive to build the
training data, since it requires professionals to write target summaries
manually. Up to now, although several advanced methods with PLMs for medical
conversation summarization have been proposed (Enarvi et al., 2020; Song et
al., 2020; Chintagunta et al., 2021; Zhang et al., 2021; Yim and Yetisgen-
Yildiz, 2021), publicly available datasets are limited. Song et al (Song et
al., 2020) proposed the Chinese medical conversation summarization
dataset171717https://github.com/cuhksz-nlp/HET-MC with 109,850 conversations
from the online health platform181818https://www.chunyuyisheng.com/.

Medical Questions The consumer health questions produced by healthcare
consumers on the web such as health forums, are another important data source
of clinical information. To find trustworthy answers to their health
questions, healthcare consumers can query the web with long natural language
questions with peripheral details. The peripheral information is useless to
find high-quality answers to health questions. Therefore, summarizing consumer
health questions into concise text with salient information is quite useful
for improving the efficiency of medical question answering. Abacha et al
(Abacha and Demner-Fushman, 2019) build the
MeQSum191919https://github.com/abachaa/MeQSum corpus with 1,000 consumer
health questions as inputs and their manual summaries from three medical
experts. Yadav et al (Yadav et al., 2022b) introduced another dataset CHQ-
Summ202020https://github.com/shwetanlp/Yahoo-CHQ-Summ most recently, which
includes 1,507 consumer health questions and their summaries annotated by
experts. Different from other texts such as biomedical papers with thousands
of words, consumer health question-summary pairs are short texts. For example,
in CHQ-Summ, the average length of questions and their summaries are 200 words
and 15 words respectively.

##  4\. Methods

Given biomedical text datasets, many methods have explored how to make use of
PLMs and LLMs for the biomedical summarization task. Different from previous
reviews (Afantenos et al., 2005; Mishra et al., 2014; Pivovarov and Elhadad,
2015; Wang et al., 2021a) for traditional and deep learning methods that
classify methods according to their inputs (multiple document/single document)
and outputs (extractive/abstractive), we categorize related methods based on
how PLMs and LLMs are leveraged in recent research to improve biomedical text
summarization.

###  4.1. PLMs for Biomedical Text Summarization

For PLMs-based methods, we first categorize them into three major categories:
feature-based, fine-tuning-based, and domain-adaption-with-fine-tuning-based,
according to the ways that they introduce PLMs into biomedical text
summarization. We then categorize methods according to the structure of PLMs
and the dataset they adopt.

As shown in Figure 5,

(a)

(b)

(c)

Figure 5. Comparison of different strategies on using PLMs for BTS.

the feature-based methods independently utilize contextual representations
from PLMs without refining the pre-trained parameters of PLMs. The
representations from PLMs are generally concatenated with the representations
from the encoder to generate the output. Although they are time-saving for
fixing the parameters of PLMs, their performance is limited since they do not
consider task-specific supervised information. The fine-tuning-based methods
employ PLMs as text encoders, where the generated representations are directly
fed into the output layer. The parameters of the PLMs, along with task-
specific parameters, are fine-tuned based on the task loss. These methods
often require higher computing resources and are time-consuming, but they
offer improved and promising performance compared to feature-based methods.
The fine-tuning-with-domain-adaption-based methods first conduct the domain-
adaption for PLMs via continually training PLMs with designed tasks on the
target data. The adapted PLMs, along with task-specific layers, are then fine-
tuned. This domain adaptation process enables the PLMs to better capture
domain-specific knowledge, resulting in improved performance compared to
methods directly fine-tuning PLMs. We will review and discuss these methods in
more detail, as shown in Table 2.

Paper | Strategy | Model | Category | Input | Output | Training | Data
---|---|---|---|---|---|---|---
Moradi et al(Moradi et al., 2020b) | feature-base | encoder | literature | single | extractive | unsupervised | -
Moradi et al(Moradi et al., 2020a) | feature-base | encoder | literature | single | extractive | unsupervised | -
Gharebagh et al(Gharebagh et al., 2020) | feature-base | encoder | EHRs | single | abstractive | supervised | MIMIC-CXR, OpenI
RadBERT(Yan et al., 2022) | feature-base | encoder | EHRs | single | extractive | unsupervised | -
GenCompareSum (Bishop et al., 2022) | feature-base | encoder-decoder | literature | single | extractive | unsupervised | PubMed, CORD-19, S2ORC
Su et al (Su et al., 2020) | fine-tuning,feature-base | encoder, encoder-decoder | literature | mutiple | hybrid | un+supervised | CORD-19
ContinualBERT (Park, 2020) | fine-tuning | encoder | literature | single | extractive | supervised | CORD-19
BioBERTSum (Du et al., 2020) | fine-tuning | encoder | literature | single | extractive | supervised | -
Cai et al (Cai et al., 2022) | fine-tuning | encoder | literature | single | abstractive | supervised | CORD-19, PubMed
Kanwal et al(Kanwal and Rizzo, 2022) | fine-tuning | encoder | EHRs | single | extractive | unsupervised | MIMIC-III
Hu et al (Jingpeng et al., 2022) | fine-tuning | encoder | EHRs | single | abstractive | supervised | MIMIC-CXR,OpenI
HET (Song et al., 2020) | fine-tuning | encoder | conversation | single | extractive | supervised | HET-MC
Esteva et al (Esteva et al., 2021) | fine-tuning | decoder | literature | multiple | abstractive | supervised | CORD-19
Deyoung et al (DeYoung et al., 2021) | fine-tuning | encoder-decoder | literature | multiple | abstractive | supervised | MSˆ2
Luo et al (Luo et al., 2022) | fine-tuning | encoder-decoder | literature | single | abstractive | supervised | PLOS
Zhu et al (Zhu et al., 2021) | fine-tuning | encoder-decoder | EHRs | single | abstractive | supervised | MIMIC-CXR,OpenI
Kondadadi et al (Kondadadi et al., 2021) | fine-tuning | encoder-decoder | EHRs | single | abstractive | supervised | MIMIC-CXR,OpenI
FactReranker (Xie et al., 2023c) | fine-tuning | encoder-decoder | EHRs | single | abstractive | supervised | MIMIC-CXR,OpenI
Xu et al (Xu et al., 2021) | fine-tuning | encoder-decoder | EHRs,question | single | abstractive | supervised | MIMIC-CXR,OpenI,MeQSum
He et al (He et al., 2021) | fine-tuning | encoder-decoder | EHRs,question | single | abstractive | supervised | MIMIC-CXR,OpenI,MeQSum
Yadav et al (Yadav et al., 2021) | fine-tuning | encoder-decoder | question | single | abstractive | supervised | MeQSum
CLUSTER2SENT (Krishna et al., 2021) | fine-tuning | encoder-decoder | conversation | single | abstractive | supervised | -
Zhang et al (Zhang et al., 2021) | fine-tuning | encoder-decoder | conversation | single | abstractive | supervised | -
Navarro et al (Navarro et al., 2022) | fine-tuning | encoder-decoder | conversation | single | abstractive | supervised | -
BioBART (Yuan et al., 2022) | fine-tuning | encoder-decoder | conversation | single | abstractive | supervised | -
KeBioSum (Xie et al., 2022a) | adaption+fine-tuning | encoder | literature | single | extractive | supervised | PubMed, CORD-19, S2ORC
Yalunin et al (Yalunin et al., 2022) | adaption+fine-tuning | encoder | EHRs | single | abstractive | supervised | -
Mahajan et al (Mahajan et al., 2021) | adaption+fine-tuning | encoder | EHRs | single | abstractive | supervised | MIMIC-CXR,OpenI
Yadav et al (Yadav et al., 2022a) | adaption+fine-tuning | encoder | question | single | abstractive | supervised | MeQSum
Kieuvongngam et al (Kieuvongngam et al., 2020) | adaption+fine-tuning | decoder | literature | single | hybrid | supervised | CORD-19
Guo et al (Guo et al., 2021b) | adaption+fine-tuning | encoder, encoder-decoder | literature | single | hybrid | supervised | CDSR
Wallace et al (Wallace et al., 2021) | adaption+fine-tuning | encoder-decoder | literature | multiple | abstractive | supervised | RCT
BDKG (Dai et al., 2021) | adaption+fine-tuning | encoder-decoder | EHRs | single | abstractive | supervised | MIMIC-CXR,OpenI
Mrini et al (Mrini et al., 2021) | adaption+fine-tuning | encoder-decoder | question | single | abstractive | supervised | MeQSum
Table 2. Overview of PLMs-based Methods. Besides how they use PLMs and what
PLMs they used, we also list other features of these methods, according to
types of output summary (extractive, abstractive, hybrid), numbers of input
documents (single, multiple), and types of input documents (biomedical
literature summarization, radiology report summarization, medical dialogue
summarization, and medical question summarization). ”-” means datasets that
are not released.

####  4.1.1. Feature-based Methods

Feature-based methods leverage the contextualized representations of pre-
trained language models (PLMs) as additional features in the summarization
process. These contextualized representations provide extra semantic
information that can’t be captured by text encoders based on non pre-trained
neural networks.

Bi-directional encoder language models For biomedical literature texts, Moradi
et al (Moradi et al., 2020b) propose the unsupervised extractive summarizer
based on hierarchical clustering and PLMs. They conduct sentence clustering
based on sentence representations from PLMs and then select top sentences from
clusters with the ranking method. They investigate different versions of BERT
and BioBERT to yield sentence representations. The proposed methods show
better performance than traditional unsupervised methods such as
TextLexAn212121http://texlexan.sourceforge.net. They find that all versions of
BioBERT (BioBERT-pmc, BioBERT-pubmed, BioBERT-pubmed+pmc) outperform the BERT-
base, but underperform the BERT-large. Moradi et al (Moradi et al., 2020a)
propose the graph ranking based method for biomedical text summarization. They
use the contextualized embeddings of BioBERT to represent sentences and build
graphs for texts. The important sentences are identified with the graph
ranking algorithm from text graphs. The model based on BioBERT-pubmed+pmc
achieves better performance than models based on the other two versions of
BioBERT: BioBERT-pmc and BioBERT-pubmed. Su et al (Su et al., 2020) propose
the query-focused multi-document summarizer with ALBERT (Lan et al., 2019) for
COVID-19 articles. They use ALBERT (Lan et al., 2019) to generate sentence
representations and calculate the cosine similarity between sentences and
queries to select important sentences.

As for clinical notes, Gharebagh et al (Gharebagh et al., 2020) develop
supervised method for abstractive summarization of clinical notes. They
propose to incorporate the contextual embeddings from BERT as the input
embeddings of the Bi-LSTM-based encoder. It shows better performance than
classical sequence-to-sequence-based methods based on recurrent neural
networks. Yan et al (Yan et al., 2022) propose the radiology-specialized
language model RadBERT that is pre-trained on millions of radiology reports.
On the unsupervised extractive summarization of radiology reports, it achieves
better performance than other language models including BERT, BioBERT,
ClinicalBERT, BlueBERT, and BioMed-RoBERTa (Gururangan et al., 2020). Among
all variants: RadBERT-BERT-base, RadBERT-RoBERTa, RadBERT-ClinicalBERT et al,
the RadBERT-BioMed-RoBERTa achieves the best performance.

Encoder-decoder language models Bishop et al (Bishop et al., 2022) present
unsupervised extractive summarization method GenCompareSum for biomedical
literature. GenCompareSum uses the T5 generative model to generate key
snippets for text sections and selects important sentences with BERTScore
(Zhang et al., 2019) between key snippets and sentences. It outperforms
traditional unsupervised methods such as LexRank (Erkan and Radev, 2004),
TextRank (Mihalcea and Tarau, 2004), and also the SOTA supervised method
BERTSum.

####  4.1.2. Fine-tuning based Methods

Fine-tuning is the most common way to use PLMs for downstream tasks. Different
from feature-based methods that freeze the parameters of PLMs, fine-tuning-
based methods refine all parameters of PLMs along with task-specific
parameters.

Bi-directional encoder language models For biomedical literature, Park et al
(Park, 2020) present the ContinualBERT model for adaptive extractive
summarization of covid-19 related literature. ContinualBERT trains two BERT
models with continual learning in order to process texts online. It shows
better performance than the SOTA extractive method BERTSum (Liu and Lapata,
2019). Du et al (Du et al., 2020) propose BioBERTSum model for extractive
summarization of biomedical literature. It uses token embedding, sentence
embedding, and position embedding to embed input texts, and then yields
contextual representations of sentences with BioBERT. BioBERT and the extra
classifier layer are fine-tuned with the cross-entropy loss. It proves the
advantage of using a domain-specific language model for biomedical texts and
outperforms the SOTA method BERTSum that uses BERT in the general domain as
the encoder. Cai et al (Cai et al., 2022) propose a SciBERT-based abstractive
summarization model for COVID-19 scientific papers, that uses the linguistic
information of word co-occurrence encoded by graph attention network to enrich
the SciBERT encoder. They find that their method based on the SciBERT encoder
outperforms that based on BERT and BioBERT encoder.

As for clinical notes, Kanwal et al (Kanwal and Rizzo, 2022) propose the
multi-head attention-based method for extractive summarization. It fine-tunes
BERT on the task of predicting and identifying ICD-9 labels on the ICD-9
labeled MIMIC-III discharge notes. The attention scores of sentences from the
last layer of the BERT model are used to select sentences. Hu et al (Jingpeng
et al., 2022) propose the radiology report summarizer that uses BioBERT as
text encoder and randomly initialized transformer layers as the decoder. They
use the graph encoder and contrastive learning to incorporate extra knowledge
to improve the BioBERT encoder.

Moreover, Song et al (Song et al., 2020) propose the hierarchical encoder-
tagger (HET) model for extractive summarization of medical conversation, which
includes token-level and utterance-level encoders to encode input long
transcripts. They use the Chinese version of BERT as the token-level encoder.

Auto-regressive decoder language models There are abstractive methods that
focus on fine-tuning auto-regressive decoder language models such as GPT-2 and
GPT-3. For COVID-19-related biomedical articles, Esteva et al (Esteva et al.,
2021) design the parallel encoder-decoder framework for abstractive
summarization of multiple COVID-19 articles, that uses BERT as the encoder and
GPT-2 as the decoder. To overcome the problem of limited training data, they
propose to use GPT-3 as the few-shot learner and propose the few-shot fine-
tuning strategy.

Encoder-decoder language models There are many studies explore encoder-decoder
language models like BART, T5, and PEGASUS (Zhang et al., 2020b) for BTS. They
are pre-trained with a specific objective function tailored for abstractive
text summarization, making them well-suited for the summarization task.

For biomedical literature, Deyoung et al (DeYoung et al., 2021) develop the
BART-based method for multi-document summarization of medical studies. To
encode multi-documents, they investigate two encoders. One is using multiple
BART encoders to encode multi-documents separately. Another one is using
LongformerEncoderDecoder (LED) (Beltagy et al., 2020), which can encode long
inputs up to 16K tokens. Luo et al (Luo et al., 2022) fine-tune the
LongformerEncoderDecoder (LED) for generating technical and lay summaries for
biomedical scientific papers. Su et al (Su et al., 2020) fine-tune BART for
multi-document abstractive summarization for COVID-19 articles based on user
queries.

As for EHRs, the MEDIQA 2021 Shared Task (Abacha et al., 2021) at the BioNLP
2021 workshop introduces the abstractive summarization task for radiology
reports and medical question, in which most participating teams propose
methods based on encoder-decoder language models. Among 14 teams that
participate in the radiology reports summarization task, 6 of them (Xu et al.,
2021; Zhu et al., 2021; Kondadadi et al., 2021; Mahajan et al., 2021; He et
al., 2021) use the encoder-decoder language models such as BART, PEGASUS. Most
of them (Xu et al., 2021; Zhu et al., 2021) find that fine-tuning PEGASUS
achieves the best performance, while Kondadadi et al (Kondadadi et al., 2021)
reports that the best performance is achieved by the BART. Moreover, they
report that adapting PEGASUS on the PubMed corpus can lead to worse
performance, which may be due to the gap between biomedical literature and
medical reports. Xie et al (Xie et al., 2023c) propose a second stage
abstractive summarization framework FactReRanker for clinical notes, where the
first stage fine-tunes the BART model to generate multiple candidate summaries
and the second stage selects the best summary based on their factual
consistency score.

For medical conversation summarization, efforts have been proposed to explore
the encoder-decoder language models to address challenges such as limited
labeled data and long transcripts. Krishna et al (Krishna et al., 2021)
develop CLUSTER2SENT, an extractive-abstractive hybrid method on doctor-
patient conversations to generate SOAP notes (long semi-structured clinical
summaries). T5 model is used in the abstractive module of CLUSTER2SENT. Zhang
et al (Zhang et al., 2021) leverage the BART model for automatic summarization
of doctor-patient conversations with limited labeled training data. They
propose the multistage fine-tuning strategy to address the input length
limitation of BART. They find that fine-tuning BART can generate summaries of
good quality even with limited training data. Moreover, they also find that
the BART-based model significantly outperforms the BigBird (Zaheer et al.,
2020) based models that are initialized by RoBERTa-base and PEGASUS-Large.
Navarro et al (Navarro et al., 2022) explore fine-tuning BART, T5, PEGASUS
with zero-shot and few-shot learning strategies for medical dialogue
summarization with small training data. They find that BART achieves the best
performance among these PLMs. Yuan et al (Yuan et al., 2022) develop the first
encoder-decoder language model BioBART in the biomedical domain, which has
shown better performance on medical dialogue summarization than BART.

Moreover, for medical questions, 22 teams participate in the medical question
summarization task of the MEDIQA 2021 Shared Task (Abacha et al., 2021) at the
BioNLP 2021 workshop, and all methods are based on fine-tuning encoder-decoder
language models. The best performance is achieved by the ensemble model (He et
al., 2021) that re-ranks summary outputs of multiple advanced encoder-decoder
language models including BART, T5, and PEGASUS. Yadav et al (Yadav et al.,
2021) present the reinforcement learning based framework for abstractive
summarization of medical questions. They propose two reward functions: the
Question-type Identification Reward (QTR) and Question-focus Recognition
Reward (QFR), which are optimized via learning optimal policy defined by BERT.
They show that the encoder-decoder language model ProphetNet (Qi et al., 2020)
with the proposed reward functions has better performance than other PLMs
including T5, BART, and PEGASUS.

####  4.1.3. Domain Adaption with Fine-tuning based Methods

There are summarization methods that conduct domain adaption before fine-
tuning PLMs, to capture domain and task-specific information.

Bi-directional encoder language models For biomedical literature, Xie et al
(Xie et al., 2022a) propose the KeBioSum for the extractive summarization. It
proposes to refine PLMs with the domain adaption tasks of predicting key
entities and their types based on the lightweight fine-tuning framework, which
aims to incorporate fine-grained medical knowledge into PLMs. It proves that
although biomedical language models such as BioBERT, and PubMedBERT can
capture domain knowledge to some extent, fine-grained medical knowledge is
still beneficial to improve language models. They find that PubMedBERT-based
methods outperform methods based on BERT, RoBERTa, and BioBERT.

As for EHRs, Yalunin et al (Yalunin et al., 2022) present the abstractive
summarizer for patient hospitalization histories, that uses Longformer
(Beltagy et al., 2020) as the encoder and BERT as the decoder. They propose to
pre-train BERT and Longformer with the masked language task on the
hospitalization history dataset before task specific fine-tuning. Dai et al
(Dai et al., 2021) propose the BDKG method, which achieves the best
performance on MEDIQA 2021 Shared Task for radiology reports summarization. It
ensembles results from multiple language models BART, DistillBERT (Sanh et
al., 2019), PEGASUS, and uses other strategies including domain adaption and
text normalization.

Moreover, for medical questions, Yadav et al (Yadav et al., 2022a) investigate
to incorporate the knowledge of ”question-focus” and ”question-type” with PLMs
for abstractive summarization of consumer health questions. To induce PLMs to
capture this knowledge, they adapt PLMs with designed Cloze tasks to capture
the .

Auto-regressive decoder language models Kieuvongngam et al (Kieuvongngam et
al., 2020) use the GPT-2 (Radford et al., 2019) for abstractive summarization
of COVID-19 medical research articles. They take keywords of articles as
inputs and fine-tune GPT-2 on multi-tasks including the language modeling task
and the multiple choice prediction task.

Encoder-decoder language models For biomedical articles, Guo et al (Guo et
al., 2021b) use the BART for automated lay language summarization of
biomedical review articles. They conduct domain-adaption before fine-tuning,
which pre-trains the BART model to reconstruct original PubMed abstracts with
disrupted abstracts. Wallace et al (Wallace et al., 2021) propose the multi-
document abstractive summarization models based on BART for randomized
controlled trials (RCTs). They adapt the BART with the domain-specific pre-
training strategy of generating summaries from full-text articles before fine-
tuning. They also use the ”decoration” strategy to explicitly inform key trial
attributes (the ”PICO” elements) of input articles.

As for medical questions, Mrini et al (Mrini et al., 2021) present the multi-
task learning and data augmentation method on medical question summarization
and recognizing question entailment (RQE) for medical question understanding.
They prove that the multi-task learning between question summarization and RQE
is able to increase the performance of PLMs including BART and BioBERT.

Dataset | Category | Metrics
---|---|---
|  | Automatic metrics
|  | similarity | factuality | others
PubMed (Cohan et al., 2018) | literature |  ROUGE (Bishop et al., 2022; Xie et al., 2022a; Cai et al., 2022; Park, 2020),BERTScore (Xie et al., 2022a) | - | -
SumPubMed (Gupta et al., 2021) | literature |  ROUGE (Gupta et al., 2021) | - | -
S2ORC (Bishop et al., 2022) | literature |  ROUGE (Bishop et al., 2022; Xie et al., 2022a; Cai et al., 2022; Park, 2020),BERTScore (Xie et al., 2022a) | - | -
CORD-19 (Wang et al., 2020) | literature |  ROUGE (Bishop et al., 2022; Xie et al., 2022a; Cai et al., 2022; Park, 2020),BERTScore (Xie et al., 2022a) | - | -
CDSR (Guo et al., 2021b) | literature |  ROUGE (Guo et al., 2021b) | - |  readability (Guo et al., 2021b)
PLOS (Luo et al., 2022) | literature |  ROUGE (Luo et al., 2022) | - |  readability (Luo et al., 2022)
PubMedCite (Luo et al., 2023) | literature |  ROUGE (Luo et al., 2023) | - | -
RCT (Wallace et al., 2021) | literature |  ROUGE (Wallace et al., 2021) | - | -
MSˆ2 (DeYoung et al., 2021) | literature |  ROUGE (DeYoung et al., 2021) |  $\Delta$EI (DeYoung et al., 2021) | -
MIMIC-CXR (Johnson et al., 2019) | EHRs |  ROUGE (Gharebagh et al., 2020; Jingpeng et al., 2022; Zhu et al., 2021; Kondadadi et al., 2021; Xu et al., 2021; He et al., 2021; Mahajan et al., 2021; Dai et al., 2021),BERTScore (Xu et al., 2021; He et al., 2021; Mahajan et al., 2021; Dai et al., 2021) |  CheXbert (Jingpeng et al., 2022; Xu et al., 2021; He et al., 2021; Mahajan et al., 2021; Dai et al., 2021) | -
OpenI (Demner-Fushman et al., 2016) | EHRs |  ROUGE (Gharebagh et al., 2020; Jingpeng et al., 2022; Zhu et al., 2021; Kondadadi et al., 2021; Xu et al., 2021; He et al., 2021; Mahajan et al., 2021; Dai et al., 2021),BERTScore (Xu et al., 2021; He et al., 2021; Mahajan et al., 2021; Dai et al., 2021) |  CheXbert (Jingpeng et al., 2022; Xu et al., 2021; He et al., 2021; Mahajan et al., 2021; Dai et al., 2021) | -
HET-MC (Song et al., 2020) | conversation |  ROUGE (Song et al., 2020) | - | -
MeQSum (Abacha and Demner-Fushman, 2019) | question |  ROUGE (Yadav et al., 2021, 2022a; Mrini et al., 2021) | - | -
CHQ-Summ (Yadav et al., 2022b) | question |  ROUGE,BERTScore (Yadav et al., 2022b) | - | -
Table 3. The usage of automatic evaluation metrics on different biomedical
datasets and methods.

###  4.2. LLMs for Biomedical Text Summarization

Inspired by the impressive capabilities of large language models (LLMs) in
various natural language processing (NLP) tasks, there is a growing interest
in exploring LLMs for biomedical text summarization. These approaches can be
broadly classified into three categories: data augmentation-based methods,
zero-shot-based methods, and domain adaption-based methods, based on how they
leverage LLMs. Data augmentation-based methods utilize LLMs as generators to
produce high-quality training data. By leveraging the generation capabilities
of LLMs, these methods augment the supervised biomedical text summarizers with
additional high-quality data, thereby enhancing their performance. Zero-shot
(and few-shot) methods directly prompt LLMs for zero-shot or few-shot
biomedical text summarization. Through natural language instructions, these
methods enable LLMs to generate summaries without explicit training on the
target task. Domain adaption-based methods focus on continual pre-training or
instruction fine-tuning of publicly available LLMs using biomedical datasets
or task-specific datasets. These approaches aim to enable LLMs to capture
medical knowledge and task-specific information, enhancing their effectiveness
in biomedical text summarization tasks.

Data Augmentation based Methods. For medical dialogue summarization,
Chintagunta et al (Chintagunta et al., 2021) integrate medical knowledge and
GPT-3. They consider GPT-3 as the summary generator and choose the best
summary that captures the most medical concepts. They show that GPT-3 can be a
promising backbone method for generating high-quality training data that can
be incorporated with the training data based on human annotation.

Zero-shot based Methods. Shaib et al (Shaib et al., 2023) show GPT-3 performs
well in summarizing a single medical article but struggles in summarizing
multiple biomedical articles in the zero-shot setting. For radiology reports
summarization, Ma et al (Ma et al., 2023) propose the ImpressionGPT framework,
instructing ChatGPT with dynamically updated prompts, which outperforms
supervised methods without additional fine-tuning of LLMs.

Domain adaption based Methods. For radiology reports summarization, Karn et al
(Karn et al., 2023) pre-train an instruction-tuned LLM Bloomz (Muennighoff et
al., 2022) with radiology reports, which outperforms other supervised methods.
Veen et al (Van Veen et al., 2023) investigate adapting LLMs to the task of
radiology reports summarization (RRS), by pre-training them with clinical
texts and fine-tuning them with examples of RRS. They found that CLIN-T5
(Lehman and Johnson, [n. d.]) achieves the best performance among other LLMs
including FLAN-T5 (Chung et al., 2022), SCIFIVE (Phan et al., 2021), and
CLIN-T5-SCI (Lehman and Johnson, [n. d.]).

##  5\. Evaluations

Assessing the quality of summaries for biomedical texts poses unique
challenges compared to general documents, as biomedical texts tend to be more
technical and complex in terms of length and structure. To evaluate the
effectiveness of summarization methods, we categorize evaluation metrics into
two groups: automatic metrics and human evaluation, based on the level of
human involvement. Automatic metrics in biomedical text summarization can
provide performance measurements without requiring human effort. These metrics
typically rely on lexical and syntactical information to assess the quality of
generated summaries compared to gold (reference) summaries. In contrast, human
evaluation can capture and model semantic information, which is often
challenging to quantify. However, conducting human evaluation in the
biomedical domain is more time-consuming and financially demanding compared to
evaluations in the general domain due to the specific domain expertise
required.

We further divide existing metrics into three different classes, consisting of
similarity, factuality, and others, as shown in Table 3. Similarity metrics
focus on the relevance of generated summaries with gold summaries, which are
generally based on the overlapping of tokens, phrases, and sentences.
Factuality metrics verify the factual agreement of generated summaries with
original documents, which is a critical measurement for the real application
of automatic systems, especially in the biomedical domain. Moreover, there are
other metrics such as 1) readability: how easily can human readers understand
generated summaries, 2) fluency: how fluent and coherent generated summaries
are, and 3) grammaticality: how grammatically correct generated summaries are.

###  5.1. Automatic metrics

Similarity Similar to the general domain, ROUGE (Lin, 2004) is the most widely
used metric for evaluating the relevance between generated summaries and gold
summaries in the biomedical domain, including (1) ROUGE-1: calculating unigram
overlap between generated summaries of summarizers and gold summaries; (2)
ROUGE-2: calculating bigram overlap between generated summaries of summarizers
and gold summaries; and (3) ROUGE-L: calculating the longest common
subsequences between generated summaries of summarizers and gold summaries.
However, ROUGE metrics are limited to relying on shallow lexical overlaps
without considering the paraphrasing and terminology variations when measuring
similarity. To address this issue, BERTScore (Zhang et al., 2019) metric has
been proposed, that calculates the similarity between two sentences as the sum
of cosine similarities between the contextual embeddings of their tokens from
pre-trained language models.

Factuality Compared with extractive methods, it is reported (Zhang et al.,
2018) that abstractive methods struggle to generate factual correct summaries.
It has been a growing awareness that metrics such as ROUGE and BERTScore can
not reflect the factual correctness of generated summaries (Zhang et al.,
2020a). For factuality evaluation, Deyoung et al (DeYoung et al., 2021)
propose the $\Delta$EI metric to calculate the factual agreement of generated
summaries and input medical studies. They propose to calculate the Jensen-
Shannon Distance (JSD) between distributions of generated summaries and input
medical studies in three directions (increase, decrease, no change) of
reported directionality. However, the metric can only be used in the MSˆ2
dataset. Zhang et al (Zhang et al., 2020a) propose the factual F1 score to
evaluate the factual correctness of generated summaries of radiology reports.
They propose to use the CheXbert labeler (Smit et al., 2020) to yield the
binary presence values of disease variables of generated summaries and
references and then calculate the overlap of yielded binary presence values
between them.

Others Guo et al (Guo et al., 2021b) propose to apply the readability
evaluation which verifies if the generated summaries are understandable for
laymen. It utilizes three different metrics including Flesch-Kincaid grade
level (Kincaid et al., 1975), Gunning fog index (Gunning et al., 1952), and
Coleman-Liau index (Coleman and Liau, 1975).

###  5.2. Human evaluation

While manual evaluation of summarization methods is time-consuming and costly,
it provides valuable insights that cannot be captured by automated evaluation
metrics. Human evaluation when conducted by domain experts, can assess various
aspects of summaries, including fluency, coherence, factuality,
grammaticality, and more. Generally, the human evaluation would recruit human
evaluators which are able to read and write in English and participate in
medical training and biology courses. Evaluators are required to score the
generated summaries with designed questions that focus on one of the
aforementioned aspects. Moramarco et al (Moramarco et al., 2022) study the
correlation between human evaluation and 18 automatic evaluation metrics
including text overlap metrics such as ROUGE, CHRF (Popović, 2015), embedding
metrics BertScore, and factual F1 score et al, on generated clinical
consultation notes. They find that simple character-based metrics such as
character-based Levenshtein distance can be more effective than other complex
metrics such as BERTScore, and the choice of human references can largely
influence the performance of automatic evaluation metrics.

Similarity For lay summarization of biomedical literature, Guo et al (Guo et
al., 2021b) also propose to conduct human evaluation to further assess the
quality of generated summaries on the CDSR dataset. Different from other
research, it requires human evaluators to not participate in medical training
and biology courses since its method is designed for laymen. It proposes the
meaning preservation metric for human evaluators, which requires them to
answer whether the generated summaries cover the key information of source
documents, on the 1-5 Likert scale (1 means very poor, 5 means very good). For
multi-document summarization dataset RCT, Wallace et al (Wallace et al., 2021)
ask evaluators who are medical doctors to score the relevance of the generated
summaries to the given topic from mostly off-topic to strongly on-topic.

For medical question summarization, Yadav et al (Yadav et al., 2021) request
two experts in medical informatics to measure the semantics preserved in the
generated summaries, i.e, whether the question intent was mentioned in the
generated summary.

Factual consistency For biomedical literature, Guo et al (Guo et al., 2021b)
ask evaluators to judge the factuality of the generated summaries on the CDSR
dataset following the 1-5 Likert scale. Wallace et al (Wallace et al., 2021)
request evaluators to answer two questions about the factuality of generated
summaries for RCT which concern the factuality degree of the generated
summaries compared with gold summaries. Otmakhova et al (Otmakhova et al.,
2022) define three different metrics for the factuality of the generated
summaries on the MSˆ2 dataset, including (1) PICO correctness: the generated
summary should contain the same patient population, intervention, and outcome
(which are the entity types defined by PICO) as the gold summary; (2)
direction correctness: the generated summary should have the same direction
referring to the intervention’s effect to the outcome as the gold summary,
which can be classified as a positive effect, negative effect, and no effect;
(3) modality: the confidence of the generated summary about the claim should
be the same as the gold summary, which can be defined as a strong claim,
moderate claim, weak claim, no evidence, and no claim.

As for medical questions, Yadav et al (Yadav et al., 2021) ask experts to
verify if all key entities appear in the generated summaries on RCT to assess
their factual consistency.

Other There are several methods that propose new human evaluation metrics to
assess the performance of biomedical summarizers. Moramarco et al (Moramarco
et al., 2021) propose an objective human evaluation based on counting medical
facts for generated summaries of medical reports. Guo et al (Guo et al.,
2021b) conduct human evaluation based on questions for grammatical correctness
and readability respectively for CDSR dataset. Wallace et al (Wallace et al.,
2021) ask evaluators to evaluate the fluency of the generated summaries on the
RCT dataset. Otmakhova et al (Otmakhova et al., 2022) consider the fluency of
the generated summaries and propose three metrics to evaluate it, including
grammatical correctness, lexical correctness, and absence of repetition.

Methods | Strategy | Model | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---|---|---
TextRank (Mihalcea and Tarau, 2004) | - | - | 34.53 | 12.98 | 30.99
BERTSum (Liu and Lapata, 2019) | fine-tuning | BERT | 34.00 | 13.42 | 30.69
PubMedBERTSum (Liu and Lapata, 2019) | fine-tuning | PubMedBERT | 34.98 | 14.22 | 31.37
KeBioSum (Xie et al., 2022a) | domain adaption+fine-tuning | PubMedBERT | 36.39 | 16.27 | 33.28

Table 4. ROUGE F1 score of generated summaries by the SOTA extractive methods
on the PubMed-long dataset, that extract 3 sentences to formulate the final
summary.

Methods | Strategy | Model | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---|---|---
TextRank (Mihalcea and Tarau, 2004) | - | - | 38.15 | 12.99 | 34.77
BERTSum (Liu and Lapata, 2019) | fine-tuning | BERT | 41.09 | 15.51 | 36.85
MatchSum (Zhong et al., 2020) | fine-tuning | RoBERTa | 41.21 | 14.91 | 36.75
KeBioSum (Xie et al., 2022a) | domain adaption+fine-tuning | PubMedBERT | 43.98 | 18.27 | 39.93

Table 5. ROUGE F1 score of generated summaries by the SOTA extractive methods
on the PubMed-short dataset, that extract 6 sentences to formulate the final
summary.

Data | CORD-19 | S2ORC
---|---|---
Methods | Strategy | Model | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-1 | ROUGE-2 | ROUGE-L
TextRank (Mihalcea and Tarau, 2004) | - | - | 32.99 | 10.39 | 24.471 | 36.58 | 13.23 | 33.10
SumBasic (Nenkova and Vanderwende, 2005) | - | - | 33.88 | 8.24 | 30.86 | 36.63 | 10.43 | 33.68
BERTSum (Liu and Lapata, 2019) | fine-tuning | BERT | 38.95 | 12.17 | 35.48 | 43.56 | 17.85 | 40.40
GenCompareSum (Bishop et al., 2022) | feature-base | T5 | 41.02 | 13.79 | 37.25 | 43.39 | 16.84 | 39.82

Table 6. ROUGE F1 score of generated summaries by the SOTA extractive methods
on the CORD-19 and S2ORC datasets, which extract 8 and 9 sentences for the
CORD-19 and S2ORC datasets correspondingly to formulate the final summary.

Methods | Strategy | Model | R-1 | R-2 | R-L | PICO | Direction | Modality | Grammar | Lexical | Non-redundancy
---|---|---|---|---|---|---|---|---|---|---|---
BART (DeYoung et al., 2021) | fine-tuning | BART | 27.56 | 9.40 | 20.80 | 45% | 77% | 45% | 75% | 69% | 85%
LED (DeYoung et al., 2021) | fine-tuning | Longformer | 26.89 | 8.91 | 20.32 | 40% | 75% | 44% | 63% | 73% | 89%

Table 7. ROUGE F1 score (Lin, 2004), factual correctness, grammatical errors
and fluency (Otmakhova et al., 2022) of generated summaries by SOTA
abstractive methods for multiple biomedical document summarization on MSˆ2.
ROUGE-1, ROUGE-2 and ROUGE-L are commonly used for evaluating the relevancy
between gold summaries and generated summaries. PICO, direction, and modality
are used for evaluating the factual correctness of generated summaries.

Metrics | OpenI | MIMIC-CXR
---|---|---
Method | Strategy | Model | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-1 | ROUGE-2 | ROUGE-L |  $F_{1}$CheXbert | RadGraph
LexRank (Erkan and Radev, 2004) | - | - | 14.63 | 4.42 | 14.06 | 18.11 | 7.47 | 16.87 | - | -
TRANSABS (Liu and Lapata, 2019) | - | - | 59.66 | 49.41 | 59.18 | 47.16 | 32.31 | 45.47 | - | -
Hu et al (Jingpeng et al., 2022) | fine-tuning | BioBERT | 64.97 | 55.59 | 64.45 | 51.02 | 35.21 | 46.65 | 70.73 | 45.23
FactReranker (Xie et al., 2023c) | fine-tuning | BART | 66.11 | 56.31 | 65.36 | 55.94 | 40.63 | 51.85 | 76.36 | 53.17
ImpressionGPT (Ma et al., 2023) | zero-shot | ChatGPT | 66.37 | 54.93 | 65.47 | 54.45 | 34.50 | 47.93 | - | -

Table 8. Performance of SOTA abstractive methods for radiology findings
summarization on MIMIC-CXR and OpenI.

Method | Strategy | Model | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---|---|---
Seq2Seq (Sutskever et al., 2014) | - | - | 25.28 | 14.39 | 24.64
BertSum (Liu and Lapata, 2019) | fine-tuning | BERT | 26.24 | 16.20 | 30.59
PEGASUS (Zhang et al., 2020b) | fine-tuning | PEGASUS | 39.06 | 20.18 | 42.05
Yadav et al (Yadav et al., 2022a) | domain adaption+fine-tuning | MiniLM | 45.20 | 28.38 | 48.76
Mrini et al (Mrini et al., 2021) | domain adaption+fine-tuning | BART | 54.5 | 37.9 | 50.2

Table 9. ROUGE F1 score of generated summaries by the SOTA abstractive methods
on the MeQSum dataset.

##  6\. Discussion

In this section, we make a further discussion on existing methods and their
limitations, and then outlook promising future directions.

###  6.1. Comparison

We first present the performance of existing SOTA PLMs and LLMs based methods
on different datasets in Table 9, Table 9, Table 9, Table 9, Table 9, and
Table 9.

How do PLMs and LLMs in biomedical text summarization work? For biomedical
literature, on the PubMed dataset, as shown in Table 9 and Table 9, PLMs-based
methods such as PubMedBERTSum (Liu and Lapata, 2019), and KeBioSum (Xie et
al., 2022a) outperforms TextRank (Mihalcea and Tarau, 2004) without PLMs on
PubMed-long, and GenCompareSum (Bishop et al., 2022), BERTSum (Liu and Lapata,
2019), MatchSum (Zhong et al., 2020), and KeBioSum (Xie et al., 2022a) show a
significant improvement in both ROUGE metrics compared with TextRank without
PLMs on PubMed-short. It is noticed that the PubMed dataset is also used as a
benchmark dataset in the general domain. Although other advanced methods such
as LongT5 (Guo et al., 2021a) have achieved the new SOTA on the dataset, we
mainly focus on comparing methods that are designed for the biomedical domain
here. As for CORD-19 and S2ORC, Table 9 shows that GenCompareSum (Bishop et
al., 2022) and BERTSum (Liu and Lapata, 2019) present great performance
compared with existing non-PLMs-based methods such as TextRank and SumBasic
(Nenkova and Vanderwende, 2005). Moreover, for multiple document summarization
dataset MSˆ2, as shown in Table 9, PLMs-based methods such as BART (DeYoung et
al., 2021) and LED (DeYoung et al., 2021) have shown great performance in both
automatical metrics and human evaluation. We can see that they can generate
summaries that have low grammar and lexical error (less than $31\%$), and low
redundancy (less than $15\%$). However, they have problems generating factual
correct summaries, for example with PICO correctness no higher than $45\%$.

For radiology reports, Table 9 clearly proves the benefits of PLMs via the
comparison of PLMs-based methods such as Factreranker (Xie et al., 2023c) and
non-PLMs-based methods such as TRANSABS (Liu and Lapata, 2019), and TexRank
(Mihalcea and Tarau, 2004). Factreranker presents the best performance on both
MIMIC-CXR and OpenI datasets. Moreover, the ImpressionGPT which instruct
ChatGPT in the zero-shot setting significantly outperforms supervised fine-
tuning method such as Hu et al’s method (Jingpeng et al., 2022), and has
comparable performance with Factreranker on both datasets. This highlights the
great potential of LLMs for radiology report summarization in the zero-shot
setting.

As shown in Table 9, PLMs-based methods such as BertSum, PEGASUS (Zhang et
al., 2020b), Yadav et al (Yadav et al., 2022a), and Mrini et al (Mrini et al.,
2021) both outperform traditional Seq2Seq (Sutskever et al., 2014) method on
the medical question dataset MeQSum.

Overall, we can find that performance on various datasets has been greatly
boosted by advanced methods that make use of PLMs, and existing methods are
able to generate fluent summaries, but have limitations on the factual
correctness of generated summaries. LLMs have shown great potential for zero-
shot BTS without supervised training.

What is the optimal way to introduce PLMs and LLMs in biomedical text
summarization? From Table 9 and Table 9, we can find that domain-adaption with
fine-tuning based methods such as KeBioSum (Xie et al., 2022b) and Mrini et al
(Mrini et al., 2021) present the best performance among three different ways
to introduce PLMs. Fine-tuning based methods generally yield better
performance than feature-based methods, except for methods on the CORD-19 and
S2ORC datasets, as shown in Table 9. For multiple biomedical document
summarization and EHRs summarization, there are only fine-tuning based methods
which are proven to be effective in introducing PLMs. For radiology reports
and medical question summarization, it shows that assembling multiple language
models can achieve the best performance (Dai et al., 2021; He et al., 2021).

What is the difference in the choice of PLMs and LLMs? The choice of PLMs and
LLMs has a significant influence on the performance of biomedical text
summarization. For biomedical literature, it has been proven that the domain-
specific language model BioBERT has better performance than BERT-base, but
underperforms BERT-large (Moradi et al., 2020b). PubMedBERT further
outperforms BioBERT (Xie et al., 2022a). As shown in Table 9, methods using
PubMedBERT which is continually pre-trained on the biomedical texts show a
better performance compared with general PLMs such as BERT and RoBERTa on both
PubMed-long and PubMed-short datasets. On CORD-19 and S2ORC datasets, the
method GenCompareSum with T5 outperforms the method BERTSum with BERT even
though T5 applied in their methods would not be fine-tuned. As for multiple
document summarization, LED based on Longformer which is specially designed
for long texts presents an inferior performance compared with BART, indicating
that it requires more effort to address the input length limitation of PLMs in
the biomedical domain rather than directly applying solutions from the general
domain.

As for EHRs, medical conversations, and medical questions, most methods use
the encoder-decoder language models such as BART, T5, and PEGASUS for
abstractive summarization of these datasets. For EHRs, they find that domain-
specific language models such as BioBERT and PubMedBERT, are not effective for
radiology reports summarization, since they are pre-trained on biomedical
literature (Kondadadi et al., 2021). Language models such as RadBERT which are
pre-trained on radiology reports, are better choices (Yan et al., 2022). For
radiology reports summarization, PEGASUS achieves better performance than BART
and T5 (Xu et al., 2021; Zhu et al., 2021; Dai et al., 2021). For medical
conversation summarization, it shows that GPT-3 and BART are promising methods
with limited training data (Navarro et al., 2022; Chintagunta et al., 2021).
For medical questions, Mrini et al (Mrini et al., 2021) with BART have the
best performance compared with other methods based on BERT, PEGASUS, and
MiniLM.

###  6.2. Limitations

While methods based on PLMs and LLMs have significantly improved the
performance of biomedical text summarization, they still come with a number of
limitations.

Developing high-quality public datasets The development of public datasets for
biomedical text summarization is imbalanced. On one hand, from the perspective
of dataset types, compared with a number of public datasets in biomedical
literature, there are limited released datasets for electronic health records,
medical conversations, and medical questions due to privacy issues, despite
the fact that there is an urgent need for developing automated text
summarization methods in these texts. On the other hand, considering the task
types, there are only two public datasets for multi-document summarization,
while most of the existing datasets focus on single-document summarization.
Moreover, the size of datasets in the biomedical domain such as CHQ-SUmm is
generally much smaller than those in the general domain, due to the high cost
and time-consuming of human annotation which additionally requires domain-
specific knowledge. The lack of high-quality large-scale public datasets can
hinder the development and employment of PLMs whose performance relies on the
amount of data.

Encoding long biomedical texts PLMs and LLMs have limitations on the token
length of input documents (Beltagy et al., 2020) due to the high time and
memory consumption of attention computations in Transformer. Most PLMs and
LLMs-based summarization methods directly truncate input documents and only
take their first 512 tokens following methods in the general domain. However,
biomedical texts such as biomedical scientific papers, usually have thousands
of tokens. The truncating operation losses useful information in the truncated
contents of input documents. Moreover, it also leads to the loss of long-range
dependencies on long biomedical documents. Although there is a method (DeYoung
et al., 2021) that investigates using PLMs that support encoding long
documents such as Longformer, it shows poor performance when compared with
BART on biomedical texts. Therefore, it still requires more effort to deal
with the limitation to encode long biomedical texts based on PLMs and LLMs
efficiently.

Incorporating domain-specific knowledge Domain-specific knowledge is critical
for understanding biomedical texts. Vocabularies, taxonomies, and ontologies
such as UMLS (Bodenreider, 2004) are important sources of biomedical
knowledge. While existing methods with PLMs and LLMs are able to capture
lexical knowledge in biomedical texts, they have no knowledge of words or
entities that have particular domain-specific importance and their relations.
Up to now, limited efforts have been proposed to incorporate external domain-
specific knowledge for the summarization of biomedical literature and EHRs
(Xie et al., 2022a; Jingpeng et al., 2022; Xie et al., 2023c). It is still a
limitation for existing methods on other biomedical texts to capture the
knowledge of sources such as biomedical concepts, relations between concepts,
and lexicographic information et al when leveraging PLMs and LLMs.

Controlling factual consistency The factual correctness of generated summaries
is especially important for the real application of automatic biomedical
summarizers. Despite this, existing methods have traditionally promoted the
unrestrained reconstruction of gold standard summaries, without imposing any
specific word-level restrictions during the text generation process. This
leads to generated summaries that fabricate facts of original inputs due to
freely rephrasing (Kryściński et al., 2020; Zhang et al., 2020a), which may
cause medical errors. It has shown that existing methods based on PLMs such as
BART and LLMs such as ChatGPT struggle with maintaining factual correctness in
their generated summaries (DeYoung et al., 2021; Zhang et al., 2020a; Xie et
al., 2023b). This limitation underscores a critical challenge in the current
landscape of biomedical text summarization: developing methods with PLMs and
LLMs that can effectively control and ensure the factual consistency of the
summaries they generate.

Interpretability and transparency Similar to other deep learning methods, PLMs
and LLMs-based methods have the well-known interpretability problem due to
their black-box nature. For end-users, such as clinicians, it becomes
challenging to comprehend the reasoning behind a model’s selection of specific
words or sentences to compose the final summaries. This lack of transparency
can be particularly problematic when the model consistently generates errors,
making it difficult for users to pinpoint why these inaccuracies occur. The
explanation and transparency of these models, particularly understanding their
inner mechanisms and algorithmic functioning, are crucial for building
trustworthy applications for users (Tjoa and Guan, 2020). However, the
challenge of improving the interpretability of these models, specifically when
implementing PLMs and LLMs, remains largely unexplored in the current research
landscape.

Evaluations Objective and comprehensive evaluation metrics are important to
evaluate summarization methods efficiently and reliably. Most existing methods
only use the ROUGE and BERTScore metrics to evaluate their models
automatically similar to methods in the general domain. However, it has been
reported that they are far from reflecting the quality of generated summaries
accurately such as factual correctness, and key finding directions. While some
efforts have been made toward developing metrics for evaluating factuality,
these typically apply to only one category of biomedical text. Moreover, their
effectiveness and correlation with human evaluations remain largely un-
investigated, indicating a need for further research in this area.

###  6.3. Future Directions

In this section, we further discuss promising future directions, which we hope
can provide guidelines for future research.

New large-scale public biomedical datasets For biomedical text summarization,
the annotation of the dataset is much more expensive and time-consuming than
the general domain since the annotators are required to be domain experts.
Moreover, for datasets such as medical conversations and questions, the
privacy issue is more critical compared with biomedical literature datasets.
Although there is an urgent need for summarization methods to facilitate
information processing, rare attention has been paid to developing high-
quality large-scale public datasets for biomedical summarization, especially
for medical conversations and questions. We believe more efforts should be
devoted to the development of new large-scale public biomedical datasets, i.e,
unsupervised or distant-supervised automatic annotations and federated
learning (Yang et al., 2019) to allow the development of models while keeping
training data on the private side. Besides, more efforts should be proposed in
the future to explore unsupervised, zero-shot, few-shot learning, and data
augmentation techniques, with advanced techniques such as LLMs, for low-
resource biomedical summarization.

Handling long biomedical documents PLMs and LLMs generally are limited to a
given length of text due to the time complexity of the model. It is urgent to
investigate the effective way for PLMs and LLMs in the biomedical domain to
encode the full content of long texts. This could involve investigating
segmentation strategies that preserve the semantic integrity of the long
biomedical text or adapting PLMs and LLMs to deal with longer contexts without
substantial computational costs. The design of domain-specific architecture or
techniques like the sparse attention mechanism might offer a path forward.

Incorporating extra knowledge PLMs and LLMs are shown to be able to capture
common sense knowledge and biomedical knowledge to a certain extent. Although
they can generate summaries that are fluent or grammatically correct, it
proves that most of their generated summaries are illogical or have factual
errors (Otmakhova et al., 2022; DeYoung et al., 2021). Limited medical
knowledge captured by PLMs and LLMs is hard to support the model to generate
desirable summaries. To overcome this limitation, future research could
consider the development of knowledge-aware models, which incorporate
additional domain-specific knowledge like UMLS and techniques such as
Reinforcement Learning (RL), pre-training, and retrieval augmentation, to
enhance the quality and factual correctness of generated summaries.

Controllable generation Existing methods generally yield summaries that ignore
users’ preferences. We believe more efforts should be developed for controlled
summarization of biomedical texts by incorporating PLMs and LLMs with
controlled text generation techniques such as prompt engineering, that meet
the expectations and requirements of users. These enhanced methodologies
should be designed to manipulate various characteristics of the generated
summaries, such as length, readability, and stylistic elements.

Benchmarks To facilitate the development of biomedical NLP, attempts have been
made to create NLP benchmarks in the biomedical domain such as BLUE (Peng et
al., 2019), GLUE (Gu et al., 2021), which include the relation extraction
task, text classification task et al. However, none of the existing benchmarks
includes the biomedical text summarization task. Biomedical texts encompass a
wide array of formats, from scientific papers and Electronic Health Records
(EHRs) to dialogues and queries. Furthermore, the task of summarization itself
is multifaceted, involving extractive, abstractive, and multi-document
summarization. Given this complexity and diversity, we believe it is necessary
to build a unified benchmark to support the development and fair evaluations
of proposed methods.

Multimodality The biomedical field boasts a wealth of multimodal medical
datasets, such as radiology reports paired with associated X-ray images.
Despite this, the majority of existing methods for biomedical text
summarization focus solely on the text inputs, overlooking the potential value
of associated visual data. There is increasing evidence that incorporating
visual features can significantly enhance text generation performance
(Delbrouck et al., 2021). We anticipate a growing interest in multimodal
summarization methods in the future that effectively leverage both textual and
visual information in the biomedical domain to produce richer, more
informative summaries.

##  7\. Conclusion

In this survey, we make a comprehensive overview of biomedical text
summarization with pre-trained language models and large language models. We
systematically review recent approaches that are based on PLMs and LLMs,
benchmark datasets, evaluations of the task. We categorize and compare recent
approaches according to their utilization of PLMs and LLMs and the specific
models they employ. Finally, we highlight the limitations of existing methods
and suggest potential directions for future research. We hope the paper can be
a timely survey to help future researchers.

## Acknowledgement

This research is partially supported by the Alan Turing Institute and the
Biotechnology and Biological Sciences Research Council (BBSRC), BB/P025684/1.

## References

  * (1)
  * Abacha and Demner-Fushman (2019) Asma Ben Abacha and Dina Demner-Fushman. 2019.  On the summarization of consumer health questions. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 2228–2234.
  * Abacha et al. (2021) Asma Ben Abacha, Yassine M’rabet, Yuhao Zhang, Chaitanya Shivade, Curtis Langlotz, and Dina Demner-Fushman. 2021.  Overview of the MEDIQA 2021 shared task on summarization in the medical domain. In _Proceedings of the 20th Workshop on Biomedical Language Processing_. 74–85.
  * Afantenos et al. (2005) Stergos Afantenos, Vangelis Karkaletsis, and Panagiotis Stamatopoulos. 2005.  Summarization from medical documents: a survey.  _Artificial intelligence in medicine_ 33, 2 (2005), 157–177.
  * Alsentzer et al. (2019) Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, WA Redmond, and Matthew BA McDermott. 2019\.  Publicly Available Clinical BERT Embeddings.  _NAACL HLT 2019_ (2019), 72\.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 3615–3620.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020\.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ (2020).
  * Bengio et al. (2000) Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000\.  A neural probabilistic language model.  _Advances in neural information processing systems_ 13 (2000).
  * Bishop et al. (2022) Jennifer Bishop, Qianqian Xie, and Sophia Ananiadou. 2022\.  GenCompareSum: a hybrid unsupervised summarization method using salience. In _Proceedings of the 21st Workshop on Biomedical Language Processing_. 220–240.
  * Bodenreider (2004) Olivier Bodenreider. 2004\.  The unified medical language system (UMLS): integrating biomedical terminology.  _Nucleic acids research_ 32, suppl_1 (2004), D267–D270.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023\.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ (2023).
  * Cai et al. (2022) Xiaoyan Cai, Sen Liu, Libin Yang, Yan Lu, Jintao Zhao, Dinggang Shen, and Tianming Liu. 2022.  COVIDSum: A linguistically enriched SciBERT-based summarization model for COVID-19 scientific papers.  _Journal of Biomedical Informatics_ 127 (2022), 103999.
  * Chintagunta et al. (2021) Bharath Chintagunta, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2021.  Medically aware gpt-3 as a data generator for medical dialogue summarization. In _Machine Learning for Healthcare Conference_. PMLR, 354–372.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017\.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ 30 (2017).
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022\.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ (2022).
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents. In _Proceedings of NAACL-HLT_. 615–621.
  * Coleman and Liau (1975) Meri Coleman and Ta Lin Liau. 1975.  A computer readability formula designed for machine scoring.  _Journal of Applied Psychology_ 60, 2 (1975), 283\.
  * Computer (2023) Together Computer. 2023\.  _RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset_.  https://github.com/togethercomputer/RedPajama-Data
  * Dai et al. (2021) Songtai Dai, Quan Wang, Yajuan Lyu, and Yong Zhu. 2021\.  BDKG at MEDIQA 2021: System Report for the Radiology Report Summarization Task. In _Proceedings of the 20th Workshop on Biomedical Language Processing_. 103–111.
  * Delbrouck et al. (2021) Jean-Benoit Delbrouck, Cassie Zhang, and Daniel Rubin. 2021\.  QIAI at MEDIQA 2021: Multimodal Radiology Report Summarization. In _Proceedings of the 20th Workshop on Biomedical Language Processing_. 285–290.
  * Demner-Fushman et al. (2009) Dina Demner-Fushman, Wendy W Chapman, and Clement J McDonald. 2009.  What can natural language processing do for clinical decision support?  _Journal of biomedical informatics_ 42, 5 (2009), 760–772.
  * Demner-Fushman et al. (2016) Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2016.  Preparing a collection of radiology examinations for distribution and retrieval.  _Journal of the American Medical Informatics Association_ 23, 2 (2016), 304–310.
  * DeYoung et al. (2021) Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Wang. 2021.  MSˆ2: Multi-Document Summarization of Medical Studies. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. 7494–7513.
  * Du et al. (2020) Yongping Du, Qingxiao Li, Lulin Wang, and Yanqing He. 2020\.  Biomedical-domain pre-trained language model for extractive summarization.  _Knowledge-Based Systems_ 199 (2020), 105964.
  * Enarvi et al. (2020) Seppo Enarvi, Marilisa Amoia, Miguel Del-Agua Teba, Brian Delaney, Frank Diehl, Stefan Hahn, Kristina Harris, Liam McGrath, Yue Pan, Joel Pinto, et al. 2020\.  Generating medical reports from patient-doctor conversations using sequence-to-sequence models. In _Proceedings of the first workshop on natural language processing for medical conversations_. 22–30.
  * Erkan and Radev (2004) Günes Erkan and Dragomir R Radev. 2004.  Lexrank: Graph-based lexical centrality as salience in text summarization.  _Journal of artificial intelligence research_ 22 (2004), 457–479.
  * Esteva et al. (2021) Andre Esteva, Anuprit Kale, Romain Paulus, Kazuma Hashimoto, Wenpeng Yin, Dragomir Radev, and Richard Socher. 2021.  COVID-19 information retrieval with deep-learning based semantic search, question answering, and abstractive summarization.  _NPJ digital medicine_ 4, 1 (2021), 1–9.
  * Gharebagh et al. (2020) Sajad Sotudeh Gharebagh, Nazli Goharian, and Ross Filice. 2020\.  Attend to Medical Ontologies: Content Selection for Clinical Abstractive Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 1899–1905.
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ 3, 1 (2021), 1–23.
  * Gunning et al. (1952) Robert Gunning et al. 1952\.  Technique of clear writing.  (1952).
  * Guo et al. (2021a) Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2021a.  Longt5: Efficient text-to-text transformer for long sequences.  _arXiv preprint arXiv:2112.07916_ (2021).
  * Guo et al. (2021b) Yue Guo, Wei Qiu, Yizhong Wang, and Trevor Cohen. 2021b.  Automated Lay Language Summarization of Biomedical Scientific Reviews. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 160–168.
  * Gupta et al. (2021) Vivek Gupta, Prerna Bharti, Pegah Nokhiz, and Harish Karnick. 2021.  SumPubMed: Summarization dataset of PubMed scientific articles. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop_. 292–303.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020.  Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 8342–8360.
  * He et al. (2021) Yifan He, Mosha Chen, and Songfang Huang. 2021.  damo_nlp at MEDIQA 2021: knowledge-based preprocessing and coverage-oriented reranking for medical question summarization. In _Proceedings of the 20th Workshop on Biomedical Language Processing_. 112–118.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Jensen et al. (2012) Peter B Jensen, Lars J Jensen, and Søren Brunak. 2012\.  Mining electronic health records: towards better research applications and clinical care.  _Nature Reviews Genetics_ 13, 6 (2012), 395–405.
  * Jingpeng et al. (2022) Hu Jingpeng, Li Zhuo, Chen Zhihong, Li Zhen, Wan Xiang, and Chang Tsung-Hui. 2022\.  Graph enhanced contrastive learning for radiology findings summarization. In _Proceedings of Association for Computational Linguistics (ACL)_ , Vol. 2.
  * Johnson et al. (2019) Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. 2019\.  MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific data_ 6, 1 (2019), 1–8.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016\.  MIMIC-III, a freely accessible critical care database.  _Scientific data_ 3, 1 (2016), 1–9.
  * Kanwal and Rizzo (2022) Neel Kanwal and Giuseppe Rizzo. 2022.  Attention-based clinical note summarization. In _Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing_. 813–820.
  * Karn et al. (2023) Sanjeev Kumar Karn, Rikhiya Ghosh, Oladimeji Farri, et al. 2023\.  shs-nlp at RadSum23: Domain-Adaptive Pre-training of Instruction-tuned LLMs for Radiology Report Impression Generation.  _arXiv preprint arXiv:2306.03264_ (2023).
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of NAACL-HLT_. 4171–4186.
  * Kieuvongngam et al. (2020) Virapat Kieuvongngam, Bowen Tan, and Yiming Niu. 2020\.  Automatic text summarization of covid-19 medical research articles using bert and gpt-2.  _arXiv preprint arXiv:2006.01997_ (2020).
  * Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975.  _Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel_.  Technical Report. Naval Technical Training Command Millington TN Research Branch.
  * Kondadadi et al. (2021) Ravikumar Kondadadi, Sahil Manchanda, Jason Ngo, and Ronan McCormack. 2021.  Optum at MEDIQA 2021: Abstractive Summarization of Radiology Reports using simple BART Finetuning. In _Proceedings of the 20th Workshop on Biomedical Language Processing_. 280–284.
  * Krishna et al. (2021) Kundan Krishna, Sopan Khosla, Jeffrey P Bigham, and Zachary C Lipton. 2021.  Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 4958–4972.
  * Kryściński et al. (2020) Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the Factual Consistency of Abstractive Text Summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 9332–9346.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. 66–71.
  * Kupiec et al. (1995) Julian Kupiec, Jan Pedersen, and Francine Chen. 1995\.  A trainable document summarizer. In _Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval_. 68–73.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019\.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In _International Conference on Learning Representations_.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015\.  Deep learning.  _nature_ 521, 7553 (2015), 436–444.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ 36, 4 (2020), 1234–1240.
  * Lehman and Johnson ([n. d.]) Eric Lehman and Alistair Johnson. [n. d.].  Clinical-t5: Large language models built using mimic clinical text.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 7871–7880.
  * Lin (2004) Chin-Yew Lin. 2004\.  Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_. 74–81.
  * Lin and Ng (2019) Hui Lin and Vincent Ng. 2019.  Abstractive summarization: A survey of the state of the art. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 9815–9822.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text Summarization with Pretrained Encoders. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 3730–3740.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. 2020.  S2ORC: The Semantic Scholar Open Research Corpus. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 4969–4983.
  * Lu (2011) Zhiyong Lu. 2011\.  PubMed and beyond: a survey of web tools for searching biomedical literature.  _Database_ 2011 (2011).
  * Luo et al. (2022) Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2022\.  Readability Controllable Biomedical Document Summarization. In _Findings of the Association for Computational Linguistics: EMNLP 2022_. 4667–4680.
  * Luo et al. (2023) Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023\.  CitationSum: Citation-aware Graph Contrastive Learning for Scientific Paper Summarization. In _Proceedings of the ACM Web Conference 2023_. 1843–1852.
  * Ma et al. (2023) Chong Ma, Zihao Wu, Jiaqi Wang, Shaochen Xu, Yaonai Wei, Zhengliang Liu, Lei Guo, Xiaoyan Cai, Shu Zhang, Tuo Zhang, et al. 2023\.  ImpressionGPT: an iterative optimizing framework for radiology report summarization with chatGPT.  _arXiv preprint arXiv:2304.08448_ (2023).
  * Mahajan et al. (2021) Diwakar Mahajan, Ching-Huei Tsou, and Jennifer J Liang. 2021\.  IBMResearch at MEDIQA 2021: Toward Improving Factual Correctness of Radiology Report Abstractive Summarization. In _Proceedings of the 20th Workshop on Biomedical Language Processing_. 302–310.
  * Maybury (1999) Mani Maybury. 1999\.  _Advances in automatic text summarization_.  MIT press.
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  Textrank: Bringing order into text. In _Proceedings of the 2004 conference on empirical methods in natural language processing_. 404–411.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ 26 (2013).
  * Mishra et al. (2014) Rashmi Mishra, Jiantao Bian, Marcelo Fiszman, Charlene R Weir, Siddhartha Jonnalagadda, Javed Mostafa, and Guilherme Del Fiol. 2014.  Text summarization in the biomedical domain: a systematic review of recent research.  _Journal of biomedical informatics_ 52 (2014), 457–467.
  * Moradi et al. (2020a) Milad Moradi, Maedeh Dashti, and Matthias Samwald. 2020a.  Summarization of biomedical articles using domain-specific word embeddings and graph ranking.  _Journal of Biomedical Informatics_ 107 (2020), 103452.
  * Moradi et al. (2020b) Milad Moradi, Georg Dorffner, and Matthias Samwald. 2020b.  Deep contextualized embeddings for quantifying the informative content in biomedical text summarization.  _Computer methods and programs in biomedicine_ 184 (2020), 105117\.
  * Moramarco et al. (2021) Francesco Moramarco, Damir Juric, Aleksandar Savkov, and Ehud Reiter. 2021.  Towards Objectively Evaluating the Quality of Generated Medical Summaries. In _Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)_. 56–61.
  * Moramarco et al. (2022) Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Damir Juric, Jack Flann, Ehud Reiter, Anja Belz, and Aleksandar Savkov. 2022.  Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 5739–5754.
  * Mrini et al. (2021) Khalil Mrini, Franck Dernoncourt, Seunghyun Yoon, Trung Bui, Walter Chang, Emilia Farcas, and Ndapandula Nakashole. 2021.  A gradually soft multi-task and data-augmented approach to medical question understanding. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 1505–1515.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022\.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_ (2022).
  * Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017.  Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In _Thirty-first AAAI conference on artificial intelligence_.
  * Navarro et al. (2022) David Fraile Navarro, Mark Dras, and Shlomo Berkovsky. 2022\.  Few-shot fine-tuning SOTA summarization models for medical dialogues. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop_. 254–266.
  * Nenkova and Vanderwende (2005) Ani Nenkova and Lucy Vanderwende. 2005.  The impact of frequency on summarization.  _Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005_ 101 (2005).
  * Nori et al. (2023) Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023\.  Capabilities of gpt-4 on medical challenge problems.  _arXiv preprint arXiv:2303.13375_ (2023).
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Otmakhova et al. (2022) Julia Otmakhova, Karin Verspoor, Timothy Baldwin, and Jey Han Lau. 2022.  The patient is more dead than alive: exploring the current state of the multi-document summarisation of the biomedical literature. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 5098–5111.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.
  * Park (2020) Jong Won Park. 2020\.  Continual bert: Continual learning for adaptive extractive summarization of covid-19 literature.  _arXiv preprint arXiv:2007.03405_ (2020).
  * Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019.  Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. In _Proceedings of the 18th BioNLP Workshop and Shared Task_. 58–65.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Peters et al. (2017) Matthew E Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017.  Semi-supervised sequence tagging with bidirectional language models. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1756–1765.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep Contextualized Word Representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. 2227–2237.
  * Phan et al. (2021) Long N Phan, James T Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Grégoire Altan-Bonnet. 2021\.  Scifive: a text-to-text transformer model for biomedical literature.  _arXiv preprint arXiv:2106.03598_ (2021).
  * Pivovarov and Elhadad (2015) Rimma Pivovarov and Noémie Elhadad. 2015.  Automated methods for the summarization of electronic health records.  _Journal of the American Medical Informatics Association_ 22, 5 (2015), 938–947.
  * Popović (2015) Maja Popović. 2015\.  chrF: character n-gram F-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_. 392–395.
  * Przybyła et al. (2018) Piotr Przybyła, Austin J Brockmeier, Georgios Kontonatsios, Marie-Annick Le Pogam, John McNaught, Erik von Elm, Kay Nolan, and Sophia Ananiadou. 2018.  Prioritising references for systematic reviews with RobotAnalyst: a user study.  _Research synthesis methods_ 9, 3 (2018), 470–488.
  * Qi et al. (2020) Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020\.  ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. 2401–2410.
  * Qiu et al. (2020) Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020\.  Pre-trained models for natural language processing: A survey.  _Science China Technological Sciences_ 63, 10 (2020), 1872–1897.
  * Radford et al. ([n. d.]) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. [n. d.].  Improving language understanding by generative pre-training.  ([n. d.]).
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language Models are Unsupervised Multitask Learners.  (2019).
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020\.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ 21, 140 (2020), 1–67.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020\.  How Much Knowledge Can You Pack Into the Parameters of a Language Model?. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 5418–5426.
  * Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015\.  A Neural Attention Model for Abstractive Sentence Summarization. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. 379–389.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ (2019).
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022\.  Multitask Prompted Training Enables Zero-Shot Task Generalization. In _ICLR 2022-Tenth International Conference on Learning Representations_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016\.  Neural Machine Translation of Rare Words with Subword Units. In _54th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics (ACL), 1715–1725.
  * Shaib et al. (2023) Chantal Shaib, Millicent L Li, Sebastian Joseph, Iain J Marshall, Junyi Jessy Li, and Byron C Wallace. 2023\.  Summarizing, simplifying, and synthesizing medical evidence using gpt-3 (with varying success).  _arXiv preprint arXiv:2305.06299_ (2023).
  * Singhal et al. (2022) Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022\.  Large language models encode clinical knowledge.  _arXiv preprint arXiv:2212.13138_ (2022).
  * Smit et al. (2020) Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew Lungren. 2020\.  Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 1500–1519.
  * Song et al. (2020) Yan Song, Yuanhe Tian, Nan Wang, and Fei Xia. 2020\.  Summarizing medical conversations via identifying important utterances. In _Proceedings of the 28th International Conference on Computational Linguistics_. 717–729.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ 33 (2020), 3008–3021.
  * Su et al. (2020) Dan Su, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham Barezi, and Pascale Fung. 2020\.  CAiRE-COVID: A Question Answering and Query-focused Multi-Document Summarization System for COVID-19 Scholarly Information Management. In _Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020_.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014\.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ 27 (2014).
  * Tjoa and Guan (2020) Erico Tjoa and Cuntai Guan. 2020.  A survey on explainable artificial intelligence (xai): Toward medical xai.  _IEEE transactions on neural networks and learning systems_ 32, 11 (2020), 4793–4813.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Van Veen et al. (2023) Dave Van Veen, Cara Van Uden, Maayane Attias, Anuj Pareek, Christian Bluethgen, Malgorzata Polacin, Wah Chiu, Jean-Benoit Delbrouck, Juan Manuel Zambrano Chaves, Curtis P Langlotz, et al. 2023\.  RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models.  _arXiv preprint arXiv:2305.01146_ (2023).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wallace et al. (2021) Byron C Wallace, Sayantan Saha, Frank Soboczenski, and Iain J Marshall. 2021.  Generating (factual?) narrative summaries of rcts: Experiments with neural multi-document summarization.  _AMIA Summits on Translational Science Proceedings_ 2021 (2021), 605\.
  * Wang et al. (2021b) Benyou Wang, Qianqian Xie, Jiahuan Pei, Prayag Tiwari, Zhao Li, et al. 2021b.  Pre-trained language models in biomedical domain: A systematic survey.  _arXiv preprint arXiv:2110.05006_ (2021).
  * Wang et al. (2020) Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, et al. 2020\.  CORD-19: The COVID-19 Open Research Dataset. In _Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020_.
  * Wang et al. (2021a) Mengqian Wang, Manhua Wang, Fei Yu, Yue Yang, Jennifer Walker, and Javed Mostafa. 2021a.  A systematic review of automatic text summarization for biomedical literature and EHRs.  _Journal of the American Medical Informatics Association_ 28, 10 (2021), 2287–2297.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned Language Models are Zero-Shot Learners. In _International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022\.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ (2022).
  * Wei et al. (2023) Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023\.  Larger language models do in-context learning differently.  _arXiv preprint arXiv:2303.03846_ (2023).
  * Wu et al. (2023) Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023.  Pmc-llama: Further finetuning llama on medical papers.  _arXiv preprint arXiv:2304.14454_ (2023).