# FactReranker: Fact-guided Reranker for Faithful Radiology Report
Summarization

Qianqian Xie<EMAIL_ADDRESS>0000-0002-9588-7454 Department of
Population Health Science,
Weill Cornell Medicine, Cornell University New YorkUSA NY 10065 ,  Jiayu Zhou
<EMAIL_ADDRESS>Michigan State University East LansingUSA ,  Yifan Peng
<EMAIL_ADDRESS>and  Fei Wang<EMAIL_ADDRESS>Department of
Population Health Science,
Weill Cornell Medicine, Cornell University New YorkUSA NY 10065

(2023)

###### Abstract.

Automatic radiology report summarization is a crucial clinical task, whose key
challenge is to maintain factual accuracy between produced summaries and
ground truth radiology findings. Existing research adopts reinforcement
learning to directly optimize factual consistency metrics such as CheXBert or
RadGraph score. However, their decoding method using greedy search or beam
search considers no factual consistency when picking the optimal candidate,
leading to limited factual consistency improvement. To address it, we propose
a novel second-stage summarizing approach FactReranker, the first attempt that
learns to choose the best summary from all candidates based on their estimated
factual consistency score. We propose to extract medical facts of the input
medical report, its gold summary, and candidate summaries based on the
RadGraph schema and design the fact-guided reranker to efficiently incorporate
the extracted medical facts for selecting the optimal summary. We decompose
the fact-guided reranker into the factual knowledge graph generation and the
factual scorer, which allows the reranker to model the mapping between the
medical facts of the input text and its gold summary, thus can select the
optimal summary even the gold summary can’t be observed during inference. We
also present a fact-based ranking metric (RadMRR) for measuring the ability of
the reranker on selecting factual consistent candidates. Experimental results
on two benchmark datasets demonstrate the superiority of our method in
generating summaries with higher factual consistency scores when compared with
existing methods.

Factual consistency, radiology reports summarization, medical knowledge graph

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: ; August 6–August 10, 2023; Long Beach,
CA††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information systems
Summarization††ccs: Computing methodologies Natural language generation††ccs:
Applied computing Bioinformatics

##  1\. Introduction

Radiology reports are an essential component of patient care, as they provide
valuable information to physicians and other healthcare providers to help
guide treatment decisions. They usually include the “Findings” section
illustrating detailed medical observations and findings of medical images, and
the succinct “Impression” section preserving the summary of most preeminent
observations which is the most important information for clinical decision-
making. Automatic radiology report summarization (Zhang et al., 2020b, 2018;
Hu et al., 2021) seeks to condense the ”Findings” section of radiology reports
into the succinct ”Impression” section using the text summarization
techniques, as shown in the Figure 1.

Figure 1. The example of automatic radiology report summarization and the
factual inconsistency problem.

It is a key clinical task with significant application value for reducing the
burden of clinicians and enhancing their efficiency (Wang et al., 2021).
Maintaining the factual correctness between generated summaries and the ground
truth radiology findings is of utmost importance for automated radiology
report summarization. In application, any errors or inaccuracies in the
generated impressions could have significant implications for the patient’s
diagnosis and treatment. To date, there have been few studies (Zhang et al.,
2020b; Delbrouck et al., 2022) that focus on optimizing the factual
correctness of generated summaries with reinforcement learning (RL). Zhang et
al. (2020b) developed a CheXBert (Smit et al., 2020b) based metric to assess
the factual correctness of clinical information in generated summaries and
employ an RL based method to optimize the factual correctness objective based
on the metric. Delbrouck et al. (2022) further developed a RadGraph (Jain et
al., 2021) based metric, which utilizes a broader range of clinical
information from radiology reports than the ChexBERT-based score and is more
aligned with radiologists.

However, there exists a significant gap between the oracle performance of
these models and their actual outputs, due to no factual guidance during the
decoding process. The vast majority of these methods adopt decoding strategies
such as greedy search, beam search, and diverse beam search (Vijayakumar et
al., 2016) et al, which does not factor in factual consistency as a criterion
for selecting the best candidate summary from $k$ generated summaries. These
decoding methods generally maintain a list of top-$k$ best candidates and
output the best one which is sorted according to log-probability.
Nevertheless, the left-out $k-1$ candidates can contain more correct facts. To
highlight the phenomenon, in Table 1, we report the oracle performances (the
highest scores over all candidates) of three popular decoding methods with
different metrics on the benchmark dataset MIMIC-CXR (Johnson et al., 2019).
We can see that the oracle performance when selecting the best candidate
summary with the highest RadGraph score is significantly higher ($+16\%$) than
existing SOTA methods (Hu et al., 2022; Delbrouck et al., 2022) and the base
model $\text{BART}_{BS}$. It indicates a significant opportunity to improve
the factual consistency of generated summaries by identifying better
candidates.

Decoding strategy | R-1 | R-2 | R-L |  $F_{1}$CheXbert | RadGraph
---|---|---|---|---|---
$\text{BART}_{BS}$ | 51.25 | 35.53 | 47.10 | 72.60 | 45.97
$\text{BART}_{BS}$ | 51.25 | 35.53 | 47.10 | 72.60 | 45.97
Hu et al. (2022) | 51.02 | 35.21 | 46.65 | 70.73 | 45.23
Delbrouck et al. (2022) | 51.96 | 35.65 | 47.10 | 74.86 | 48.23
Beam search | 55.09 | 39.59 | 50.69 | 74.63 | 64.35
Diverse beam search | 54.39 | 38.98 | 50.07 | 74.94 | 63.37
Top-k sampling | 53.03 | 37.21 | 47.58 | 73.04 | 49.62
Table 1. Oracle score for the BART (Lewis et al., 2020) backbone model with
different decoding strategies on different evaluation metrics on the benchmark
dataset MIMIC-CXR (Johnson et al., 2019). The last three columns show the
oracle performance of the base model BART with different decoding strategies.
In all decoding methods, we keep 10 candidate summaries. R-1/2/L means the
ROUGE-1/2/L score (Lin, 2004). $F_{1}$CheXbert and RadGraph score are two
metrics for evaluating the factual consistency. Figure 2. The example of the
medical entities and relations in the input findings and its gold summary. We
use different color to identify different type of entities. We also show the
relations between entities such as ”modify” and ”suggestive_of”.

To address these issues, in this paper, we propose a second-stage
summarization method FactReranker, which aims to enhance the factual
correctness of generated summaries by learning to select the optimal summary
from all candidates according to their factual correctness scores. However,
training a robust and effective reranker in the radiology report summarization
scenario is a significant challenge for several reasons. Firstly, different
from general metrics such as ROUGE (Lin, 2004) and Bertscore (Zhang et al.,
2020a) that only consider the semantic similarity between the candidate and
the gold summary, the factual consistency metric such as the RadGraph based
metric (Delbrouck et al., 2022) calculates the overlapping between the medical
entities and relations of the candidate and the gold summary. This means the
reranker should have the ability to identify and evaluate the medical facts
(medical entities and relations) in the candidate summaries, to precisely
select the best summary with the highest factual consistency score. Secondly,
another challenge for the reranker to effectively select the best summary that
aligns with the gold summary in medical facts, arises from the significant gap
of medical facts between the original input and the gold summary. The reranker
only has access to the information in the candidate summaries and the input
radiology reports during inference, while the gold summary can’t be observed.
Yet there is often a significant difference in the medical facts between the
original input and the gold summary, i.e., as shown in Figure 2, the input
findings contain multiple entities and relations that are not mentioned in the
gold summary.

To address the aforementioned two challenges, we propose to extract medical
facts from the input medical report, candidate summaries, and the gold summary
based on the RadGraph schema, and develop the fact-guided reranker with the
factual knowledge graph generation and the factual scorer based on extracted
factual knowledge graphs, to select the optimal summary by explicitly
capturing medical facts and the mapping of medical facts between the input
texts and its gold summary. The RadGraph (Jain et al., 2021) is a dataset
containing annotated clinical entities and relations of radiology reports
based on the information extraction schema designed for structure radiology
reports. We generate a pool of candidates based on the backbone model with
different decoding strategies in the first stage, and extract the factual
knowledge graphs preserving medical facts including medical entities and
relations of the input medical report, all candidates and the gold summary,
based on the RadGraph schema. We further develop the second stage fact-guided
reranker with the factual knowledge graph generation and the factual scorer
based on extracted factual knowledge graphs, that explicitly models the
mapping between the medical facts of the gold summary and that of the input
texts, enabling to select the optimal summary that factually aligns with the
gold summary only based on medical facts of the input text, when the medical
factual information of the gold summary can not be accessed during the
inference. To model the mapping between the medical facts of the gold summary
and that of the input texts, the factual knowledge graph generation is to
generate the factual knowledge graph of the target summary given the original
input text and its factual knowledge graph. Inspired by the generative
knowledge graph construction (Zhang et al., 2021; Ye et al., 2020), we build
the factual knowledge graph generation as the sequence-to-sequence problem,
which takes input as the linearization of RadGraph triplets in the factual
knowledge graph of the input texts and outputs the linearization of expected
RadGraph triplets in that of the target summary, as shown in Figure 3.

Figure 3. The example of the linearization of the RadGraph triplets in both
the factual knowledge graph of input findings and its gold summary. We use the
four types of entity same as the definitions in RadGraph schema and two
simplified relation types as [REL](has any relation) and [NA](no relation),
which is the same as the settings of the RadGraph partial scorer (Delbrouck et
al., 2022).

Specifically, we design a new linearization for RadGraph triplets which starts
with the entity token, followed by the special type tokens indicating the
entity types and ended with the special relation tokens. We also adopt a
special token ([ENT]) to separate each triplet. With the linearization, we can
build a unified sequence for a factual knowledge graph that fully captures its
structure information and label semantics. The factual scorer thus can
estimate the factual consistency score of candidate summaries based on the
alignment between the generated factual knowledge graph of the target summary
and that of candidates, and select the optimal summary with the highest
alignment score. Finally, for evaluating the ranking ability of fact-guided
reranker, we design a fact-based ranking metric (RadMRR), that evaluates how
well our method can select the factual consistent summary from all candidates
according to the inverse of the rank of the summary with the highest
factuality score. Our contributions are as follows:

  1. (1)

We propose the FactReranker, a novel second-stage summarization framework for
faithful radiology report summarization, by learning to select the optimal
summary from all candidates, based on factual consistency and medical factual
knowledge. To the best of our knowledge, this is the first second-state
summarizer for factual enhanced radiology report summarization.

  2. (2)

We propose the novel fact-guided reranker consisting of the factual knowledge
graph generation and factual scorer, which can select the best summary based
on explicitly capturing medical factual knowledge and modeling the mapping of
medical facts between the input texts and its gold summary. We design the new
linearization of the knowledge factual graph which captures the structural
information and label semantics, and enables us to model the factual knowledge
graph generation as the sequence-to-sequence problem. We also present a fact-
based ranking metric (RadMRR) for measuring the ability of the reranker on
selecting factual consistent candidates.

  3. (3)

Experimental results over two benchmark radiology report datasets demonstrate
the superiority of our method by improving the factual consistency score of
generated summaries when compared with existing RL based summarization
methods. Our method improves the RagGraph Score by 4.84%, ROUGE-1 by 3.98%,
ROUGE-2 by 4.98%, ROUGE-L by 4.75%, and $F_{1}$CheXbert by 1.5% on the MIMIC-
CXR dataset, achieving the new SOTA.

##  2\. Related Work

###  2.1. Radiology Report Summarization

In recent years, a growing body of research has focused on automatic radiology
report summarization. Zhang et al. (2018) employed a bi-directional LSTM to
perform sequence-to-sequence generation and reported that 30% of generated
summaries have factual errors. Zhang et al. (2020b) proposed a factual
correctness metric $F_{1}$CheXbert which used CheXbert to assess the factual
consistency of the generated impressions. The metric was then leveraged as the
reward of a reinforcement learning (RL)-based summarizer to guide the
training. Hu et al. (2021) extracted medical entities using the Biomedical and
Clinical English Model Packages, which are further exploited via graph neural
networks. Based on it, Hu et al. (2022) adopted the contrastive learning based
pre-training that masks the entities in the findings to fully embed the domain
knowledge. Most recently, Delbrouck et al. (2022) developed a RadGraph-based
factual consistency measurement, and RL-based method which utilizes the metric
as the reward. However, these methods have no factual guidance during the
decoding process via beam search or greedy search and fail to explicitly model
the domain knowledge in both input documents and target summary via
reinforcement learning. To the best of our knowledge, we are the first second-
stage summarization approach for radiology report summarization, which aims to
rerank the generated candidates based on their factual consistency with the
explicitly predicted medical facts in the target summary.

###  2.2. Second-stage Abstractive Summarization

Numerous research has focused on exploring the second-stage abstractive
summarization techniques. Dou et al. (2020) guided the abstractive
summarization using the salient sentences selected by extractive methods. Sun
and Li (2021) proposed to employ contrastive learning to improve the chance of
generating high-quality summaries and reduce exposure bias. Similarly, Xu et
al. (2021) optimized the similarities between the source document, the gold
summary, and the candidates to improve faithfulness via contrastive learning.
Liu et al. (2021) introduced RefSum, which incorporates a meta-system for
selecting the optimal summary from all candidates. Regarding the reranker
based methods in the second stage, SimCLS (Liu and Liu, 2021) created an
evaluation model with the objective of directly optimizing the evaluation
metric using contrastive learning. Similarly, BRIO (Liu et al., 2022) reused
the generation model as the evaluation model to rank the candidates with
contrastive learning. SummaReranker (Ravaut et al., 2022a) trained a multi-
task learning framework based on mixture-of-experts to optimize multiple
metrics like ROUGE concurrently. SummaFusion (Ravaut et al., 2022b) combined
numerous candidates to provide a more concise summary. Even though these
methods have shown promising performance in abstractive summarization for
directly optimizing evaluation metrics, there have been no prior efforts to
explore the second-stage method for radiology report summarization. Moreover,
it is challenging to directly apply these methods in the general domain to the
radiology report summarization task, since it requires modeling and predicting
the domain knowledge for factual consistency metrics in the medical domain,
unlike commonly used metrics such as ROUGE.

###  2.3. Generative Knowledge Graph Construction

Existing knowledge graph construction (KGC) relies on a series of tasks
including named entity recognition, entity linking, relation extraction, and
event extraction, which suffers from error propagation in the pipeline and
limited extension on different tasks. To address it, a series of studies based
on a sequence-to-sequence framework was proposed. Zeng et al. (2018) was the
first attempt that revisits the KGC as a sequence-to-sequence problem and
designed a copy mechanism to generate knowledge triplets. Despite following
research in the copy mechanism (Zeng et al., 2019b, a; Huang et al., 2021;
Giorgi et al., 2022), there were studies utilizing structural knowledge and
label semantics as unified text sequences. Lu et al. (2021) designed a
linearization of the extracted events for event extraction based on T5. Lou et
al. (2021) further proposed a Multi-Layer Bidirectional Network to capture
document-level relationships and semantics. Zhang et al. (2021) and Ye et al.
(2020) proposed to generate a sequence of triplets separated by tags via
contrastive learning. Based on it, Cabot and Navigli (2021) designed a new
linearization method for triplets that considers the order of triplets
according to the head entity and tail entity. Inspired by these studies, in
this paper, we revisit the factual knowledge graph generation as a sequence-
to-sequence generation task and design a new linearization for triplets of the
factual knowledge graph, which captures the structural information and label
semantics embedded in the triplets.

##  3\. Methodology

In this section, we present the detail of our proposed framework, as shown in
4.

Figure 4. The overall framework.

###  3.1. Framework of Two Stage Summarization Method

Formally, given the input radiology findings with sequence of $N$ tokens
$X=\\{x_{1},x_{2},\dots,x_{N}\\}$, the automatic summarization task on
radiology reports is to generate the summary to best describe the findings of
$X$, with a sequence $Y=\\{y_{1},\dots,y_{i},\dots,y_{L}\\}$, where $L$ is the
length of impressions ($L<<N$) and $y_{i}\in V$ is the generated token of the
vocabulary $V$. Consequently, the generation can be modelled as the
conditional sequence to sequence generation problem:

(1) $p(Y|X)=\prod^{L}_{i=1}p(y_{i}|y_{1},\dots,y_{i-1},X)$

####  3.1.1. First-stage summarizer

For the two-stage summarization method, we first train a base summarization
model with the following objective in the first stage, that maximizes the log-
likelihood of the training data:

(2) $\theta^{*}=\arg\max_{\theta}\sum_{i=1}^{L}\log
p(y_{i}|y_{1},\dots,y_{i-1},X;\theta)$

where $\theta$ is the parameter set of the model. In this paper, we employ the
pre-trained encoder-decoder language model BART (Lewis et al., 2020) which is
pre-trained on the text generation task as our backbone model, and fine-tune
it with the maximum likelihood estimation (MLE) loss as shown in Equation 2 on
the training data used in the radiology report summarization task. For each
document $X$, we then generate a set of candidate summaries with the trained
base model:

(3) $C=\text{BASE}_{\theta^{*}}(X,S)$

where $C=\\{C_{1},C_{2},\dots,C_{m}\\}$, $S$ means the decoding strategy.

####  3.1.2. Second-stage reranker

The reranker aims to rerank all candidate summaries generated by the first-
stage summarizer and select the best candidate according to factual
consistency. Given a factual consistency metric $\mu$, we can calculate the
corresponding score for each candidate
$S_{\mu}=\\{\mu(C_{1}),\mu(C_{2}),\dots,\mu(C_{m})\\}$, based on their
associated gold summaries in the training data. In this paper, we aim to train
a reranker $f_{\theta_{s}}(S_{\mu},X,C)$ parameterized by $\theta_{s}$ that
can identify the best summary candidate $C^{*}_{\mu}$ with the highest factual
consistency:

(4) $C^{*}_{\mu}={\arg\max}_{C_{i}\in C}\\{\mu(C_{1}),\dots,\mu(C_{m})\\}$

###  3.2. RadGraph Score

####  3.2.1. Choice of factual consistency evaluation metric

Up to now, there are two proposed factual consistency metrics for radiologist
reports including the $F_{1}$CheXBert (Smit et al., 2020b) and RadGraph score
(Jain et al., 2021). The $F_{1}$CheXBert computes the $F_{1}$ similarity score
between the indicator vectors of 14 medical pathologies that are extracted
from the generated and golden summaries with the ChexBERT. Similar to
$F_{1}$CheXBert, the RadGraph score is an F-score metric that measures the
factual alignment (medical entities and relations) between the candidates and
the gold summaries. However, it has been proved that the $F_{1}$CheXBert score
fails to align with the radiologists (Yu et al., 2022; Boag et al., 2021;
Anderson et al., 2016), and is limited to a small part of clinical facts which
only covers 14 medical observations, resulting in the poor evaluation of
diverse clinical facts in radiology reports. Compared with the $F_{1}$CheXBert
metric, the RadGraph score covers diverse clinical facts of radiology reports
and has higher alignment with radiologists. Therefore, we use the RadGraph
score as the optimized metric.

####  3.2.2. Extracting medical facts

The RadGraph score of candidates calculates the overlap of their medical facts
with that of the gold summary. Therefore, we first use the PubMedBERT (Gu et
al., 2021) model111We use the pre-trained model checkpoint provided in:
https://physionet.org/content/RadGraph/1.0.0/ pre-trained in annotated data
based on the RadGraph schema, to extract medical facts including the medical
entities and relations to build the RadGraph for the input report, its gold
summary, and all candidates. Formally, we consider the extracted RadGraph of
the input text as a graph $\mathcal{G}(\mathcal{V},\mathcal{E})$, with a set
of entities as nodes $\mathcal{V}=\\{v_{1},v_{2},\dots,v_{|V|}\\}$ and the
relations between nodes as edges
$\mathcal{E}=\\{e_{1},e_{2},\dots,e_{|E|}\\}$, as shown in 5. Each entity node
in the RadGraph has a corresponding label $v_{i_{L}}$ which indicates its
entity type.

Figure 5. The example of the RadGraph of the input text extracted by the
PubMedBERT model.

Entities: There are two types of entities in the RadGraph, including Anatomy
and Observation. For Observation, there are also three different levels of
uncertainty. Consequently, we have four categories of entities: Anatomy (ANAT-
DP), Observation: Definitely Present (OBS-DP), Observation:Uncertain (OBS-U),
and Observation: Definitely Absent (OBS-DA).

Relations: There are three types of relations between entities in the
RadGraph, including the {Suggestive_Of} relation between two Observation
entities, the {Located_At} relation between the Observation entity and the
Anatomy entity, and the {Modify} relation between two Anatomy entities.

To calculate the RadGraph score between candidates and the gold summary, we
further transform the RadGraph into a set of triplets
$T=\\{(v_{i},v_{i_{L}},R)\\}_{i=1:|V|}$, where $R=1$ if there is any edges
connected with $v_{i}$, otherwise $R=0$. Therefore, a triplet contains the
entity, the entity type, and whether or not the entity has a relation. Given
the triplet sets for both the candidate $T_{C_{i}}$ and the gold summary
$T_{Y}$, the RadGraph score is the harmonic mean of precision and recall
between these two sets to measure the overlapping of medical facts between the
candidate and the gold summary. Therefore, we yield the RadGraph score for all
candidates $C_{i}\in C$ for each document in the training data as:

(5) $\mu(C_{i},Y)=RadGraph(C_{i},Y)$

which derives the $F_{1}$ score based on the triplet sets of the candidate
$T_{C_{i}}$ and the gold summary $T_{Y}$.

###  3.3. Fact-guided Reranker

Based on the generated candidates and their RadGraph score calculated above,
we aim to train the robust and effective reranker that can select the best
candidate with the highest factual consistency score, namely the highest
overlap of medical entities and relationships with that of the gold summary.
However, the gold summary is unobserved during the inference, and it is
required to select the candidate summary with the trained reranker
$f_{\theta_{s}}(C_{i},X,T_{C_{i}},T_{X})$, based on the observed candidate
$C_{i}$ and the input text $X$ during the inference. However, the medical
facts of the input text has a significant gap with that of the gold summary,
which leads to great challenge for training a effective reranker that can
select the optimal summary during the inference.

Therefore, we propose the fact-guided reranker which can precisely rank the
generated candidates with estimated factuality scores based on predicted
medical facts in the target summary, when the ground truth medical facts of
the gold summary can not be accessed during the inference. To achieve this, we
decompose the fact-guided reranker into two steps: 1) factual knowledge graph
generation: it aims to generate the factual knowledge graph of the gold
summary given the factual knowledge graph of the input text $X$, which models
the mapping between the medical facts of the text input and its gold summary;
and 2) factual scorer: it calculates the factuality score of each candidate
$C_{i}$ according to the overlapping between its medical facts and that of
generated factual knowledge graph of the gold summary. Inspired by existing
research in knowledge graph construction (Zhang et al., 2021; Ye et al.,
2020), we build the factual knowledge graph generation as a sequence-to-
sequence task to better model the factual connections between entities and
relations. As shown in Figure 3, the input is the RadGraph triplets $T_{X}$
and the output is the RadGraph triplets $T_{Y}$ of the gold summary.

####  3.3.1. Triplets Linearization

We design a new linearization to express RadGraph triplets as a sequence of
tokens, which can be modeled and generated by the backbone model such as the
pre-trained language model to fully leverage its structural information and
label semantics. Given triplets of the input text
$T_{X}=\\{(v_{i},v_{i_{L}},R)\\}_{i=1:|T_{X}|}$, the corresponding sequence
would be:

(6) $<s>v_{0}\ v_{0_{L}}\ R_{0}\ [ENT]\ v_{1}\ v_{1_{L}}\ R_{1}\ [ENT]\ \dots\
v_{i}\ v_{i_{L}}\ R_{i}</s>$

where ¡s¿ and ¡/s¿ is the special token added before and after the generated
sequence, $v_{i}$ is the entity token, $v_{i_{L}}$ is the entity label, and
$R_{i}$ is the relation. For the entity label $v_{i_{L}}$, we introduce four
new special tokens corresponding to the four entity types in the RadGraph:
[ANAT-DP] for Anatomy (ANAT-DP), [OBS-DP] for Observation: Definitely Present
(OBS-DP), [OBS-U] for Observation:Uncertain (OBS-U), and [OBS-DA] for
Observation: Definitely Absent (OBS-DA). We also introduce two new special
relation tokens for $R_{i}$, including [REL] for has_relation and [NA] for
no_relation. Additionally, we add the special token [ENT] between each triplet
to indicate the end of the last triplet and the start of the next triplet.
Similar to (Cabot and Navigli, 2021), we sort the triplets by their first
appearance in the text and linearize the triplets following that order.

####  3.3.2. Factual Knowledge Graph Generation

Given the input sequence as the linearization of the RadGraph triplets
$T_{X}$, we adopt an autoregressive transformer model to generate the output
sequence as the linearization of the expected triplets $T_{Y}$ in the gold
summary. Therefore, the task can also be deemed as a sequence-to-sequence
task, which allows us to reuse and continue training the first-stage
summarizer. Specifically, the model consists of two major components:

Input Encoder: For entity token $v_{i}$, we tokenize the entity token, and
append the introduced special tokens including entity types, relations, and
separation in the tokenizer, which allows the model to not tokenize these
tokens into sub-tokens. The final input representations are the summing of the
corresponding token embedding, position embedding, and segment embedding.

Generating Factual Knowledge Graph: We reuse the first stage summarization
method based on BART and continual fine-tune it to generate the target
sequence $T_{Y}$ of the gold summary based on the input representations on the
training set. Note that we adopt the partial casual masking in the decoder to
perform auto-aggressive generation, which means tokens are only conditioned on
leftward context to capture the order of RadGraph triplets in both the source
document and target summary. It is also notice that since we utilize three
tokens in designed order to represent the triplet, i.e, the method should
identify the token type of the previous entity token when generating the next
token after the entity. We then inference the RadGraph triplets of the target
summary given the source document in the validation and test set based on the
trained model.

####  3.3.3. Factual Scorer

Based on the generated sequence of the gold summary from the generation model,
we can derive the corresponding triplet set $T^{\prime}_{Y}$ of the gold
summary. We filter out the corrupted triplets in the generated sequence, i.e,
whose entity type token and relation type token are not in the given
categories or whose token numbers are not equal to three. Consequently, given
the estimated triplet set $T^{\prime}_{Y}$ and extracted triplet set
$T_{C_{i}}$ for each candidate summary $C_{i}$, we can calculate the estimated
RadGraph score as:

(7)
$\mu^{\prime}(C_{i},Y)=RadGraph(T_{C_{i}},T^{\prime}_{Y})=2*\frac{|T_{C_{i}}\cap
T^{\prime}_{Y}|}{|T_{C_{i}}|+|T^{\prime}_{Y}|}$

Finally, our reranker can rank the generated candidate summaries according to
their estimated RadGraph scores $\mu^{\prime}(C_{i},Y)$, and select the top-1
candidate as the final output during the inference.

###  3.4. RadMRR

Based on the factual consistency metric RadGraph score, inspired by previous
information retrieval research, we design a novel ranking metric named RadMRR
which aims to evaluate the re-ranking performance of the reranker. Given the
observed candidate $C_{i}$, we calculate the reciprocal rank $\frac{1}{rank}$
of the position of the highest-ranked summary according to their RadGraph
score. For example, suppose that we have three candidates, the optimal
candidate with the highest RadGraph score is ranked as the third item by the
method, then the corresponding reciprocal rank would be $\frac{1}{3}$. Given
all examples, we average their reciprocal ranks as the mean reciprocal rank:

(8) $RadMRR=\frac{1}{|C|}\sum_{C_{i}\in C}\frac{1}{rank_{C_{i}}}$

It is easy to compute and interpret and focus on the first summary of all
candidates, which can assess the gap caused by inappropriate ranking.

##  4\. Experiments

In this section, we conduct experiments to answer the following research
questions about our method: RQ1: Does the FactReranker improve the factual
consistency of generated summaries for radiology report summarization? RQ2:
What component in FactReranker contributes to the improvement of factual
consistency? RQ3: How is the quality of the generated RadGraph for the gold
summary via the factual knowledge graph generation model?

###  4.1. Dataset

Following previous methods (Hu et al., 2022), we conduct experiments on two
widely-adopted radiology report datasets as OpenI (Demner-Fushman et al.,
2015) and MIMIC-CXR (Johnson et al., 2019). OpenI contains 3,268 reports
collected by Indiana University, while MIMIC-CXR consists of 124,577 reports.
We also filter out the reports as previous methods that have no findings, no
impressions, findings with less than 10 words, or impressions with less than 2
words. We use the same data splitting as previous methods (Hu et al., 2022),
which divides OpenI into train/validation/test sets as 2400: 292: 576. For
MIMIC-CXR, we adopt the official split in the dataset released by Johnson et
al. (2019). The statistics for two datasets are depicted in Table 2.

DATASET | TYPE | TRAIN | DEV | TEST
---|---|---|---|---
OpenI | REPORT # | 2400 | 292 | 576
AVG.WF | 37.89 | 37.77 | 37.98
AVG.SF | 5.75 | 5.68 | 5.77
AVG.WI | 10.43 | 11.22 | 10.61
AVG.SI | 2.86 | 2.94 | 2.82
MIMIC-CXR | REPORT # | 122,014 | 957 | 1606
AVG.WF | 55.78 | 56.57 | 70.67
AVG.SF | 6.50 | 6.51 | 7.28
AVG.WI | 16.98 | 17.18 | 21.71
AVG.SI | 3.02 | 3.04 | 3.49
Table 2. The statistics of OpenI and MIMIC-CXR, including the number of
reports (REPORT #), the average sentence number for both findings and
impressions (WF & SF), the average word number for both findings and
impressions (WI & SI).

###  4.2. Baselines and Metrics

We compare our methods to state-of-the-art summarization methods for radiology
reports, including extractive summarization methods such as:

  1. (1)

LEXRANK (Erkan and Radev, 2004), a commonly used graph-based ranking method,

  2. (2)

TRANSFORMEREXT (Liu and Lapata, 2019), a method using the Transformer (Vaswani
et al., 2017) network,

and abstractive summarization approaches including:

  1. (1)

PGN (See et al., 2017), an LSTM-based method using the pointer generate
network and the copy mechanism for decoding,

  2. (2)

TRASFROMERABS (Liu and Lapata, 2019), a seq2seq method based on the
Transformer network,

  3. (3)

WGSUM (TRANS+GAT) (Hu et al., 2021), a method seq2seq method incorporating
important words and their relations with the word graph,

  4. (4)

GRAPHSUM (Hu et al., 2022), a pre-trained language model based method
incorporating extra knowledge,

  5. (5)

BART (Lewis et al., 2020), the method fine-tuning the pre-trained language
model BART,

  6. (6)

RadGraph-RL (Delbrouck et al., 2022), the state-of-the-art factual guided
summarization method using reinforcement learning to optimize the RadGraph
score,

Following previous methods, we adopt two types of metrics to evaluate
performance. First, for general summarization metrics, we employ three
different ROUGE metrics including ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L
(R-L) (Lin, 2004). Then, for factual consistency which is most important for
radiology report summarization, we report both the RadGraph score (Delbrouck
et al., 2022) and $F_{1}$CheXbert score (Zhang et al., 2019). To be consistent
with previous methods, we only report the RadGraph score and $F_{1}$CheXbert
score on the MIMIC-CXR dataset, since the RadGraph score and $F_{1}$CheXbert
score have been shown to be ineffective for the OpenI dataset. The former is
based on the overlap of domain entities and relations extracted via RadGraph,
while the latter is the alignment of 14 abnormalities defined in CheXbert
(Smit et al., 2020a).

###  4.3. Implementation Details

In our experiments, we utilize BART implemented in the HuggingFace as the
backbone model in the first stage, and use the parameter setting with learning
rate=2e-5, training epoch=5, training/evaluation batch size=5, warmup
steps=8000, and weight decay=0.01. According to Table 1, we adopt the beam
search as the decoding strategy for inference and generate 10 candidate
summaries for each report. We save the model checkpoint every 500 steps and
select the best checkpoint based on the validation set. For extracting the
RadGraph, we use the model checkpoint based on the
PubMedBERT222https://physionet.org/content/RadGraph/1.0.0/, and the released
code333https://github.com/dwadden/dygiepp. In the reranker, we use the best
checkpoint in the first stage as the backbone method for triplet generation
and set the learning rate to 1e-4, training epochs to 5, and dropout to 0.5.

###  4.4. Main Results

To answer RQ1, in Table 3, we present the performance of all methods on OpenI
and MIMIC-CXR datasets on all metrics.

MODEL | OpenI | MIMIC-CXR
---|---|---
METRIC | R-1 | R-2 | R-3 | R-1 | R-2 | R-3 |  $F_{1}$CheXbert | RadGraph
LEXRANK (Erkan and Radev, 2004) | 14.63 | 4.42 | 14.06 | 18.11 | 7.47 | 16.87 | - | -
TRANSEXT (Liu and Lapata, 2019) | 15.58 | 5.28 | 14.42 | 31.00 | 16.55 | 27.49 | - | -
PGN (See et al., 2017) | 63.71 | 54.23 | 63.38 | 46.41 | 32.33 | 44.76 | - | -
TRANSABS (Liu and Lapata, 2019) | 59.66 | 49.41 | 59.18 | 47.16 | 32.31 | 45.47 | - | -
WGSUM (Hu et al., 2021) | 61.63 | 50.98 | 61.73 | 48.37 | 33.34 | 46.68 | 69.15 | 44.20
GRAPHSUM (Hu et al., 2022) | 64.97 | 55.59 | 64.45 | 51.02 | 35.21 | 46.65 | 70.73 | 45.23
RadGraph-RL (Delbrouck et al., 2022) | - | - | - | 51.96 | 35.65 | 47.10 | 74.86 | 48.23
BART (Lewis et al., 2020) | 63.68 | 54.28 | 62.96 | 50.50 | 35.27 | 46.61 | 71.00 | 45.10
FactRanker | 64.24 | 52.61 | 63.42 | 55.94 | 40.63 | 51.85 | 76.36 | 53.17
FactRanker(w/o RC) | 63.69 | 54.28 | 62.97 | 50.51 | 35.28 | 46.62 | 71.00 | 45.08
Table 3. The ROUGE F1, $F_{1}$CheXbert and RadGraph score of different methods on both OpenI and MIMIC-CXR datasets. Some results are referenced from (Hu et al., 2021, 2022; Delbrouck et al., 2022) TYPE | TEXT | RadGraph |  | Oracle
---
| Beam
---
| Original
---
| FactRanker
---
| w/o RC
---
GOLD | No evidence of acute cardiopulmonary process. | - | - | - | - | - | - |
CANDIDATE | No evidence of acute cardiopulmonary process. | 100 | 1 | 2 | 7 | 1 | 2 |
No acute cardiopulmonary process. | 100 | 2 | 4 | 4 | 2 | 4 |
| No evidence of acute intrathoracic process.
---
66.67 | 3 | 3 | 2 | 3 | 3 |
No acute intrathoracic process. | 66.67 | 4 | 5 | 8 | 4 | 5 |
No evidence of acute cardiopulmonary abnormality. | 66.67 | 5 | 7 | 3 | 5 | 7 |
No evidence of acute process. | 40.0 | 6 | 9 | 9 | 6 | 9 |
Unremarkable chest radiographic examination. | 0.0 | 7 | 1 | 5 | 7 | 1 |
Normal chest radiographic examination. | 0.0 | 8 | 6 | 10 | 8 | 6 |
Unremarkable chest x-ray. | 0.0 | 9 | 8 | 6 | 9 | 8 |
Unremarkable chest examination. | 0.0 | 10 | 10 | 1 | 10 | 10 |
Table 4. The ranking example of all decoding methods. | DECODING
---
METHODS
RadMRR | RadGraph
Beam Search | 51.8 | 44.3
Oracle | 99.9 | 60.1
OriginalReranker | 44.0 | 38.2
FactReranker | 72.6 | 53.2
FactReranker(w/o RC) | 50.4 | 45.0
Table 5. The ranking results of all decoding methods. We report the RadMRR and
RadGraph scores.

On the MIMIC-CXR dataset, our method shows a significant improvement compared
with other methods in the factual consistency metric RadGraph score and
$F_{1}$CheXbert. This demonstrates the superiority of our proposed method in
enhancing factual consistency for reranking and selecting the optimal target
summary according to the estimated target RadGraph score rather than the log
probability. FactReranker also yields the highest ROUGE-1, ROUGE-2, and
ROUGE-L scores among all methods, indicating that the improvement of the
factual consistency would also benefit the fluency of the generated summary
for generating accurate key entity words and semantic relations. Specifically,
when compared with the SOTA factual-guided summarizer RadGraph-RL, our method
yields a higher improvement than RadGraph-RL over non-factual-guided methods.
It proves the essential for explicitly capturing medical facts in the target
summary to select the optimal summary, rather than directly optimizing the
factuality score with RL and but select the summary based on the log
probability. Our method outperforms the BART backbone model which selects the
top-1 candidate with the beam search decoding by $8\%$ in RadGraph score,
which further proves the effectiveness of the factual-guided reranker. Both
our method and RadGraph-RL outperform non-factual-guided methods such as
GraphSum and WGSUM on RadGraph score, although they have shown more powerful
ability than traditional methods like PGN and TRANSEXT on Rouge score.

As for the OpenI dataset, our method also presents a competitive performance
in ROUGE-1, ROUGE-2, and ROUGE-L metrics compared with existing methods. It
has been reported that existing factual consistency metrics such as the
RadGraph score and $F_{1}$CheXbert are ineffective for OpenI since the
extraction method is trained on MIMIC-CXR. Therefore all previous methods and
our method don’t report the factual consistency results on this dataset.
Compared with the backbone method BART, our method shows a slight improvement
in ROUGE scores, which indicates again that the RadGraph score is not
effective in this dataset since the extractor would fail to extract correct
triplets. We believe more factual evaluation metrics should be developed in
this field, that can be used in various datasets including the OpenI dataset.

|  Original Text: The (lungs, ANAT-DP, NA) are (well, OBS-DP, REL) (expanded,
OBS-DP, REL) and (clear, OBS-DP, REL).
---
(Cardiomediastinal, ANAT-DP, NA) and (hilar, ANAT-DP, NA) (contours, ANAT-DP,
REL) are (unremarkable, OBS-DP, REL).
There is no (pleural, ANAT-DP, NA) (effusion, OBS-DA, REL) or (pneumothorax,
OBS-DA, NA). (Sternotomy, OBS-DP, REL)
(wires, OBS-DP, NA) are again noted, with (facture, OBS-DP, REL) of the (two,
OBS-DP, REL) (upper, OBS-DP, REL)
(wires, OBS-DP, NA) (unchanged, OBS-DP, REL) from prior exam.
Gold Summary: No evidence of (acute, OBS-DA, REL) (cardiopulmonary, ANAT-DP,
NA) (process, OBS-DA, REL).
FactReranker: ¡s¿acute [OBS-DA] [REL] [ENT] cardiopulmonary [ANAT-DP] [NA]
[ENT] process [OBS-DA] [REL]¡/s¿
FactReranker(w/o RC): ¡s¿[ [ [REL]¡/s¿
|  Original Text: A (hazy, OBS-DP, REL) (opacity, OBS-DP, REL) is present in
the (right, ANAT-DP, REL) (lung, ANAT-DP, NA)
---
which may represent (aspiration, OBS-U, NA) (pleural, ANAT-DP, NA) (effusion,
OBS-U, REL) or (hemorrhage, OBS-U, NA).
(Retrocardiac, ANAT-DP, NA) (opacity, OBS-DP, REL) at the (left, ANAT-DP, REL)
(base, ANAT-DP, NA) is (unchanged, OBS-DP, REL).
(Moderate, OBS-DP, REL) (cardiomegaly, OBS-DP, NA) is (stable, OBS-DP, REL).
(Slight, OBS-DP, REL) (prominence, OBS-DP, REL)
of the (pulmonary, ANAT-DP, REL) (vasculature, ANAT-DP, NA) with
(cephalization, OBS-DP, NA) and (enlarged, OBS-DP, REL)
(pulmonary, ANAT-DP, REL) (arteries, ANAT-DP, NA) are consistent with (mild,
OBS-DP, REL) (pulmonary, ANAT-DP, NA)
(edema, OBS-DP, REL). (Tracheostomy, OBS-DP, REL) (tube, OBS-DP, NA) is (in
place, OBS-DP, REL).
There are no (displaced, OBS-DA, REL) (rib, ANAT-DP, NA) (fractures, OBS-DA,
REL).
Gold Summary:1. (Hazy, OBS-DP, REL) (opacity, OBS-DP, REL) in the (right,
ANAT-DP, REL) (lung, ANAT-DP, NA) which may
represent (aspiration, OBS-U, NA) versus (pleural, ANAT-DP, NA) (effusion,
OBS-U, REL) or (hemorrhage, OBS-U, NA)
2\. (Mild, OBS-DP, REL) (pulmonary, ANAT-DP, NA) (edema, OBS-DP, REL)
3\. No (displaced, OBS-DA, REL) (rib, ANAT-DP, NA) (fractures, OBS-DA, REL)
FactReranker: ¡s¿Hazy [OBS-DP] [REL] [ENT] opacity [OBS-DP] [REL] [ENT] right
[ANAT-DP] [REL] [ENT]
lung [ANAT-DP] [NA] [ENT] aspiration [OBS-U] [NA] [ENT] pleural [ANAT-DP] [NA]
[ENT] effusion [OBS-U] [REL] [ENT]
hemorrhage [OBS-U] [NA] [ENT] Mild [OBS-DP] [REL] [ENT] pulmonary [ANAT-DP]
[NA] [ENT] edema [OBS-DP] [REL]¡/s¿
FactReranker(w/o RC): ¡s¿¡/s¿
Table 6. The generated triplet sequences from FactRanker and the ablation FactRanker(w/o RC), along with source documents and gold summaries annotated with RadGraph triplets using red text color. MODEL | MIMIC-CXR
---|---
METRIC | R-1 | R-2 | R-3 | BertScore
FactRanker | 81.72 | 74.82 | 80.52 | 94.87
FactRanker(w/o RC) | 0.10 | 0.10 | 0.10 | 81.48
Table 7. The ROUGE score and BertScore of FactRanker and FactRanker(w/o RC) on
MIMIC-CXR

###  4.5. Reranking Performance

In this section, we further investigate RQ1 with a detailed question: Does our
second-stage reranker possess the ability to rank candidates according to
their factual consistency? To answer it, we conduct experiments on three
different ranking methods: 1) Beam Search: it generates 10 candidate sequences
and selects the optimal one with the highest log-likelihood; 2) Oracle: it
selects the optimal candidate according to its factual consistency with the
gold summary; 3) OriginalRadRanker: it first calculates the RadGraph scores of
all candidates with the original texts and chooses the one with the highest
RadGraph score. 4) Ours: Similar to OriginalRadRanker, our method calculates
the RadGraph scores of all candidates with the predicted factual knowledge
graph of the gold summary and chooses the one with the highest RadGraph score.
We compare the results of these methods in Table 5, which utilizes our
designed ranking metric RadMRR to measure the ranking ability. As
aforementioned in Section 3.4, the higher RadMRR, the higher position of the
optimal candidate in the ranked list of all candidates. We also present the
RadGraph score of the selected summary from different ranking methods. The
results clearly prove the ability of our method to rerank candidates according
to their factual relevance to the target summary. In comparison with Beam
search, which is generally adopted by previous methods, our method yields a
21% higher RadMRR and results in a significantly better RadGraph score, which
is competitive with the results of Oracle. This is also proved when Oracle has
a 20% improvement over Beam Search, which is almost two times larger than the
improvement in the previously proposed research. Our method also outperforms
OrigianlRadRanker that utilizes the source document with no regard for the
semantic gap in the medical facts between the source document and the target
summary. In fact, OriginalRadRanker has the worst ranking performance among
all ranking methods, which indicates a significant difference between the
medical facts of source texts and target summaries. In Table 4, we
additionally provide an example of a list of candidates and their
corresponding positions generated by each ranking method. Compared with
existing ranking methods, our method has the same ranking result with that of
the Oracle. It can generally assign high-factual consistent candidates to
higher positions and otherwise lower positions, leading to a more robust and
faithful summarization process.

###  4.6. Ablation Study

To answer RQ2 and attribute the ranking ability of our reranker, we further
perform experiments on our method along with w/o RC where we directly adopt
the first-stage summarizer to generate the RadGraph triplets of the gold
summary without further fine-tuning. First, as shown in Table 5, our method of
fine-tuning the RadGraph generation presents better ranking performance and
higher factual correctness, which indicates the importance of continuous
training of the existing summarizer for RadGraph generation. Compared with our
method, w/o RC presents a 22% decrease in RadMRR and 8% in RadGraph score,
which is almost the same performance as the beam search method. This proves
that the improvement of our method over beam search mostly attributes to the
RadGraph generation which explicitly models the medical facts in the target
summary, rather than the first-stage summarizer as the backbone model. As
shown in Table 5 and Table 4, with wrong medical facts in the gold summary,
w/o RC presents the comparable ranking performance with the first-stage
summarizer based on the beam search.

###  4.7. Case Study

In this section, we further investigate the performance of our triplet
generation model on generating the medical facts of the gold summary to answer
RQ3. In Table 6, we present the generated sequences of our method and the
ablation, along with the ground truth RadGraph of the original text and its
gold summary. The first example in the first row clearly shows that the
RadGraph generation model in our reranker can generate the correct RadGraph
triplets of the gold summary, that covers the groud truth medical facts in the
gold summary. Even for the difficult sample as shown in the second row, the
fine-tuned RadGraph generation method in our reranker can still predict the
RadGraph triplets, that has high overlap with the ground truth RadGraph
triplets of the gold summary. In contrast, directly applying summarization
models without fine-tuning would result in sequences with broken triplets.
Consequently, the model would select the first candidate as the beam search
since all candidates have the same score due to an empty generated triplet
set. Moreover, we adopt several metrics to evaluate the generated sequences
for triplet generation, including ROUGE and BertScore. As shown in Table 7,
FactRanker with fine-tuning on the RadGraph constrcution yields a relatively
high ROUGE score and BertScore, indicating the ability of our generation model
to generate faithful RadGraph triplets of the target summary.

##  5\. Conclusion

In this paper, we present the first second-state approach FactReranker for
faithful radiology report summarization that prioritizes factual consistency
of the generated summary. We propose the fact-guided reranker by extracting
medical facts based on the RadGrap and incorporating them into the reranking
process. We decompose the fact-guided reranker into the RadGraph generation
and factual scorer, which allows the reranker to precisely estimate the
RadGraph scores of candidate summaries and choose the optimal summary
according to the predicted RadGraph of the gold summary. Our experimental
results show that FactReranker outperforms existing state-of-the-art radiology
report summarization methods and provides a promising solution to the
challenge of ensuring factual accuracy in this task. For future work, we will
explore dealing with limitations in our method: 1) In the first stage, we only
use the BART backbone model with the beam search decoding method for
generating candidates. We will further investigate and analyze the influence
of different backbone models and decoding strategies on the performance 2) In
the second stage, we only adopt the RadGraph triplets from both the source
document and target summary. We will further investigate using the natural
language texts to further improve the generation performance base don the
prompt learning. Moreover, we will further explore developing a reranker that
can learn to select the optimal candidate based on multiple metrics.

## Acknowledgment

This work was supported by the National Library of Medicine grant
4R00LM013001, NSF CAREER Award No. 2145640, and Amazon Research Award. We
would like to thank Jinpeng Hu for providing the processed dataset and useful
feedback for the paper. We also would like to thank Jimin Huang for his useful
suggestions on improving the paper.

## References

  * (1)
  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  SPICE: Semantic Propositional Image Caption Evaluation. In _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V_ _(Lecture Notes in Computer Science, Vol. 9909)_ , Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer, 382–398.
  * Boag et al. (2021) William Boag, Hassan Kané, Saumya Rawat, Jesse Wei, and Alexander Goehler. 2021.  A Pilot Study in Surveying Clinical Judgments to Evaluate Radiology Report Generation. In _FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021_ , Madeleine Clare Elish, William Isaac, and Richard S. Zemel (Eds.). ACM, 458–465.
  * Cabot and Navigli (2021) Pere-Lluís Huguet Cabot and Roberto Navigli. 2021.  REBEL: Relation Extraction By End-to-end Language generation. In _Conference on Empirical Methods in Natural Language Processing_.
  * Delbrouck et al. (2022) Jean-Benoit Delbrouck, Maya Varma, and Curt P. Langlotz. 2022\.  Toward expanding the scope of radiology report summarization to multiple anatomies and modalities.  _ArXiv_ abs/2211.08584 (2022).
  * Demner-Fushman et al. (2015) Dina Demner-Fushman, Marc D. Kohli, Marc B. Rosenman, Sonya E. Shooshan, Laritza M. Rodriguez, Sameer Kiran Antani, George R. Thoma, and Clement J. McDonald. 2015\.  Preparing a collection of radiology examinations for distribution and retrieval.  _Journal of the American Medical Informatics Association : JAMIA_ 23 2 (2015), 304–10.
  * Dou et al. (2020) Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2020.  GSum: A General Framework for Guided Neural Abstractive Summarization.  _ArXiv_ abs/2010.08014 (2020).
  * Erkan and Radev (2004) Günes Erkan and Dragomir R. Radev. 2004.  LexRank: Graph-based Lexical Centrality as Salience in Text Summarization.  _J. Artif. Intell. Res._ 22 (2004), 457–479.
  * Giorgi et al. (2022) John Giorgi, Gary D Bader, and Bo Wang. 2022.  A sequence-to-sequence approach for document-level relation extraction.  _ArXiv_ abs/2204.01098 (2022).
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ 3, 1 (2021), 1–23.
  * Hu et al. (2021) Jinpeng Hu, Jianling Li, Zhihong Chen, Yaling Shen, Yan Song, Xiang Wan, and Tsung-Hui Chang. 2021.  Word Graph Guided Summarization for Radiology Findings. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. 4980–4990.
  * Hu et al. (2022) Jinpeng Hu, Zhuo Li, Zhihong Chen, Zhuguo Li, Xiang Wan, and Tsung-Hui Chang. 2022\.  Graph Enhanced Contrastive Learning for Radiology Findings Summarization. In _Annual Meeting of the Association for Computational Linguistics_.
  * Huang et al. (2021) Kung-Hsiang Huang, Sam Tang, and Nanyun Peng. 2021.  Document-level Entity-based Extraction as Template Generation.  _ArXiv_ abs/2109.04901 (2021).
  * Jain et al. (2021) Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Q. H. Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P. Lungren, Andrew Y. Ng, Curtis P. Langlotz, and Pranav Rajpurkar. 2021.  RadGraph: Extracting Clinical Entities and Relations from Radiology Reports. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_ , Joaquin Vanschoren and Sai-Kit Yeung (Eds.).
  * Johnson et al. (2019) Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih ying Deng, Roger G. Mark, and Steven Horng. 2019.  MIMIC-CXR: A large publicly available database of labeled chest radiographs.  _ArXiv_ abs/1901.07042 (2019).
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 7871–7880.
  * Lin (2004) Chin-Yew Lin. 2004\.  ROUGE: A Package for Automatic Evaluation of Summaries. In _Annual Meeting of the Association for Computational Linguistics_.
  * Liu et al. (2021) Yixin Liu, Zi-Yi Dou, and Pengfei Liu. 2021.  RefSum: Refactoring Neural Summarization. In _North American Chapter of the Association for Computational Linguistics_.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text Summarization with Pretrained Encoders.  _ArXiv_ abs/1908.08345 (2019).
  * Liu and Liu (2021) Yixin Liu and Peng Liu. 2021.  SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization. In _Annual Meeting of the Association for Computational Linguistics_.
  * Liu et al. (2022) Yixin Liu, Pengfei Liu, Dragomir R. Radev, and Graham Neubig. 2022\.  BRIO: Bringing Order to Abstractive Summarization. In _Annual Meeting of the Association for Computational Linguistics_.
  * Lou et al. (2021) Dongfang Lou, Zhilin Liao, Shumin Deng, Ningyu Zhang, Ningyu Zhang, and Huajun Chen. 2021\.  MLBiNet: A Cross-Sentence Collective Event Detection Network. In _Annual Meeting of the Association for Computational Linguistics_.
  * Lu et al. (2021) Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, M. Liao, and Shaoyi Chen. 2021.  Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction.  _ArXiv_ abs/2106.09232 (2021).
  * Ravaut et al. (2022a) Mathieu Ravaut, Shafiq R. Joty, and Nancy F. Chen. 2022a.  SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization. In _Annual Meeting of the Association for Computational Linguistics_.
  * Ravaut et al. (2022b) Mathieu Ravaut, Shafiq R. Joty, and Nancy F. Chen. 2022b.  Towards Summary Candidates Fusion.  _ArXiv_ abs/2210.08779 (2022).
  * See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017\.  Get To The Point: Summarization with Pointer-Generator Networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1073–1083.
  * Smit et al. (2020a) Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, A. Ng, and Matthew P. Lungren. 2020a.  CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. In _Conference on Empirical Methods in Natural Language Processing_.
  * Smit et al. (2020b) Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y. Ng, and Matthew P. Lungren. 2020b.  CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT.  _CoRR_ abs/2004.09167 (2020).
  * Sun and Li (2021) Shichao Sun and Wenjie Li. 2021.  Alleviating Exposure Bias via Contrastive Learning for Abstractive Text Summarization.  _ArXiv_ abs/2108.11846 (2021).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Vijayakumar et al. (2016) Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016\.  Diverse beam search: Decoding diverse solutions from neural sequence models.  _arXiv preprint arXiv:1610.02424_ (2016).
  * Wang et al. (2021) Benyou Wang, Qianqian Xie, Jiahuan Pei, Prayag Tiwari, Zhao Li, et al. 2021\.  Pre-trained language models in biomedical domain: A systematic survey.  _arXiv preprint arXiv:2110.05006_ (2021).
  * Xu et al. (2021) Shusheng Xu, Xingxing Zhang, Yi Wu, and Furu Wei. 2021\.  Sequence Level Contrastive Learning for Text Summarization. In _AAAI Conference on Artificial Intelligence_.
  * Ye et al. (2020) Hongbin Ye, Ningyu Zhang, Shumin Deng, Mosha Chen, Chuanqi Tan, Fei Huang, and Huajun Chen. 2020.  Contrastive Triple Extraction with Generative Transformer. In _AAAI Conference on Artificial Intelligence_.
  * Yu et al. (2022) Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y Ng, et al. 2022\.  Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation.  _medRxiv_ (2022).
  * Zeng et al. (2019b) Daojian Zeng, Haoran Zhang, and Qianying Liu. 2019b.  CopyMTL: Copy Mechanism for Joint Extraction of Entities and Relations with Multi-Task Learning.  _ArXiv_ abs/1911.10438 (2019).
  * Zeng et al. (2019a) Xiangrong Zeng, Shizhu He, Daojian Zeng, Kang Liu, Shengping Liu, and Jun Zhao. 2019a.  Learning the Extraction Order of Multiple Relational Facts in a Sentence with Reinforcement Learning. In _Conference on Empirical Methods in Natural Language Processing_.
  * Zeng et al. (2018) Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018.  Extracting Relational Facts by an End-to-End Neural Model with Copy Mechanism. In _Annual Meeting of the Association for Computational Linguistics_.
  * Zhang et al. (2021) Ningyu Zhang, Ningyu Zhang, Hongbin Ye, Shumin Deng, Chuanqi Tan, Mosha Chen, Songfang Huang, Fei Huang, and Huajun Chen. 2021.  Contrastive Information Extraction With Generative Transformer.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 29 (2021), 3077–3088.
  * Zhang et al. (2020a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a.  BERTScore: Evaluating Text Generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_.
  * Zhang et al. (2018) Yuhao Zhang, Daisy Yi Ding, Tianpei Qian, Christopher D. Manning, and Curtis P. Langlotz. 2018\.  Learning to Summarize Radiology Findings. In _Proceedings of the Ninth International Workshop on Health Text Mining and Information Analysis_. Association for Computational Linguistics, Brussels, Belgium, 204–213.
  * Zhang et al. (2019) Yuhao Zhang, Derek Merck, Emily B Tsai, Christopher D. Manning, and C. Langlotz. 2019.  Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports. In _Annual Meeting of the Association for Computational Linguistics_.
  * Zhang et al. (2020b) Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D. Manning, and Curtis P. Langlotz. 2020b.  Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, 5108–5120.