:
1 2022 Machine Learning for Health (ML4H) 2022

# Toward expanding the scope of radiology report summarization to multiple
anatomies and modalities

Jean-Benoit Delbrouck<EMAIL_ADDRESS>
Maya Varma<EMAIL_ADDRESS>
Curtis P. Langlotz<EMAIL_ADDRESS>
Stanford University

###### Abstract

Radiology report summarization is a growing area of research. Given the
Findings and/or Background sections of a radiology report, the goal is to
generate a summary (called an Impression section) that highlights the key
observations and conclusions of the radiology study. Recent efforts have
released systems that achieve promising performance as measured by widely used
summarization metrics such as BLEU and ROUGE. However, the research area of
radiology report summarization currently faces important limitations. First,
most of the results are reported on private datasets. This limitation prevents
the ability to reproduce results and fairly compare different systems and
solutions. Secondly, to the best of our knowledge, most research is carried
out on chest X-rays. Sometimes, studies even omit to mention the concerned
modality and anatomy in the radiology reports used for their experiments. To
palliate these limitations, we propose a new dataset of six different
modalities and anatomies based on the MIMIC-III database. We further release
our results and the data splits used to carry out our experiments. Finally, we
propose a simple report summarization system that outperforms the previous
replicable research on the existing dataset.

###### keywords:

summarization; impressions; findings; radiology reports; language generation

fig:main

Figure 1: Illustration of our summarization pipeline. The model generates an
impression from the findings and RadGraph creates semantic annotations used to
evaluate the factual correctness of the impression.

##  1 Introduction

The radiology report documents and communicates crucial findings in a
radiology study. A standard radiology report usually consists of a Background
section that describes the exam and patient information, a Findings section,
and an Impression section (Kahn Jr et al., 2009). In a typical workflow, a
radiologist first dictates the detailed findings into the report and then
summarizes the salient findings into the more concise Impression section based
also on the condition of the patient. Automating this summarization task is
critical because the Impression section is the most important part of a
radiology report, and manual summarization can be time-consuming and error-
prone.

Despite its importance, we identify three weaknesses in the ongoing work on
radiology report summarization.

First, the most recent studies (Zhang et al., 2018, 2020; Hu et al., 2022) and
organized challenges (Ben Abacha et al., 2021) on automated radiology report
summarization systems solely focus on chest X-rays. The reason is that the
only two open-access and curated datasets, namely MIMIC-CXR (Johnson et al.,
2019) and Open-i Chest X-ray (Demner-Fushman et al., 2012), exclusively
contain chest-X ray radiology reports. In some rarer cases, researchers omit
to disclose the modality and anatomy of the radiology reports used for their
experiments (Karn et al., 2022).

Secondly, existing models are optimized to generate summaries that achieve
high performance on the ROUGE metric (Lin, 2004). As investigated in previous
studies, this does not guarantee factually correct summaries (Zhang et al.,
2020). So far, only one ”factually-oriented” metric has been proposed, based
on CheXbert (Smit et al., 2020). While this addition is a good step towards
evaluating factual correctness of the summaries, it is limited to chest
X-rays.

Finally, new proposed models (Karn et al., 2022; Hu et al., 2022) present an
increased complexity in architecture that offers only marginal improvements on
the existing evaluation metrics for summarization. This, in turn, makes the
replication of studies more difficult.

To address these three limitations, we consequently present three
contributions:

  * •

We release a pre-processed and curated dataset of radiology reports for new
modalities (MR and CT) and anatomies (Chest, Head, Neck, Sinus, Spine,
Abdomen, Pelvis). Our dataset is based on the MIMIC-III database (Johnson et
al., 2016) and suitable for radiology report summarization: each report
contains a clear findings and impression section.

  * •

We present a new summarization metric, called the RadGraph score, that
evaluates the factual completeness and correctness of the generated radiology
impressions. We show that this score is suitable for every modality and
anatomy of our new dataset. We also show that the RadGraph score can be
turned, without further modification, into a reward suitable for Reinforcement
Learning (RL) optimization.

  * •

We present a new simple summarization system that not only acts a strong
baseline on the new datasets but also outperforms the previous replicable
research on chest X-rays. Because our new system is small (few trainable
parameters) and fast (low FLOPs), it makes it suitable for RL which is, by
essence, computationally expensive.

Our paper is structured as follows: we first start by describing our three
contribution, namely our new MIMIC-III summarization dataset
(sec:mimiciiisum), the RadGraph score (sec:radgraph_section) and our baseline
model (sec:model). We then proceed to outline the experiments carried out
(sec:experiments) and present our results (sec:results).

CT Abd-pelv | CT Chest | CT Head
---|---|---
15,989 | 12,786 | 31,402
CT Spine | MR Head | CT Neck
5,517 | 7,313 | 1,140
CT Sinus | Mr Spine | MR Abdomen
1,267 | 2,821 | 1,061
MR Neck | MR Pelvis |
230 | 253 |
Table 1: Number of radiology reports (containing findings and impression
pairs) for each new modality-anatomy, totaling 79,779 samples.

##  2 MIMIC-III summarization dataset

MIMIC-III (Johnson et al., 2016) is a large, freely-available database
comprising de-identified health-related data associated with over forty
thousand patients who stayed in critical care units of the Beth Israel
Deaconess Medical Center between 2001 and 2012. This data comprises radiology
reports from a wide range of modality (medical imaging techniques) and anatomy
(body parts). To create a new radiology report summarization dataset, we first
chose 5 of the most frequent modality/anatomy pairs in the pool of MIMIC-III
reports, namely CT Head, CT Spine, CT Chest, CT Abdomen-Pelvis and MR head. We
discard chest X-rays as they are included in the MIMIC-CXR dataset (Johnson et
al., 2019). The number of samples per pair are available in Table 1 and
constitute enough data to train a deep learning model. We also pick 6 less
represented modality/anatomy pairs that could act as out-of-domain (OOD) test-
sets, namely CT neck, CT sinus, MR Pelvis, MR Neck, MR Abdomen, MR Spine. This
set of pairs represents two levels of OOD: 1) the modality has not been seen
during training (one could train on CT neck and test on MR Neck) 2) the
anatomy has not been seen during training (for example, CT Sinus is the only
“sinus” dataset).

For each report, we extract the findings and impression section. However, the
findings section is not always stated as such. With the help of one board-
certified radiologist, and for each modality/anatomy pair, we create a mapping
of the section header that acts as “findings”. As an example, for CT head,
findings could be referred as “non-contrast head ct”, “ct head”, “ct head
without contrast”, “ct head without iv contrast”, “head ct”, “head ct without
iv contrast” or “cta head”. This “findings” mapping contains up to 537
candidate sections for our whole dataset. We also discarded reports where
multiple studies are pooled in the same radiology reports, leading to multiple
intricate observations in the impression section. We release our mapping as
well as the code to recreate the dataset from scratch (Appendix B).

Figure 2: Section length and vocabulary size per anatomy-modality pairs.

In addition, a few comments can be made from fig:stats. As expected, for all
anatomy-modality pairs, the findings section is significantly longer than the
impression section (up to +315% for MR abdomen). The findings section of all
our new pairs are also much longer than chest X-rays: the MIMIC-CXR dataset
averages 49 words per findings. In contrast, MR Abdomen and MR Pelvis average
205 and 174 words respectively. We can see note that CT Chest, CT Head and CT
Abdomen-Pelvis have a relatively large vocabulary size (given their sample
size) with respectively 20,909, 19,813 and 18,933. Surprisingly, the CT
Abdomen-Pelvis impressions contains more vocabulary than the findings, as
opposed to MR pelvis and MR abdomen impressions that contain respectively 36%
and 37% less words than their findings counterpart.

MR Neck | CT Spine | MR Head
---|---|---
|  |

Table 2: Visualization of RadGraph’s model output (relations omitted) for a MR
Neck (left), CT Spine (middle) and MR Neck (right) report.

##  3 RadGraph score

The goal of our RadGraph score is to provide an evaluation of the factual
correctness and completeness of a generated impression. It does so by
evaluating two semantic components of the impression: the correctness of the
named entities and the relation between two entities.

We divide this section in two subsections: we first present RadGraph in
section sec:radgraph and then describe how we leverage RadGraph to compute the
proposed RadGraph score in section sec:score.

###  3.1 RadGraph

Figure 3: Example of the RadGraph annotations. Figure taken from Jain et al.
(2021).

To design our new evaluation metric, we leverage the RadGraph dataset (Jain et
al., 2021) containing board-certified radiologist annotations of chest X-ray
reports, which correspond to 14,579 entities and 10,889 relations. RadGraph
has released a PubMedBERT model (Gu et al., 2021) pretrained on these
annotations to annotate new reports. An example of annotation can be seen in
fig:ddd. Before moving on to the next section, we quickly describe the concept
of entities and relations:

#### Entities

An entity is defined as a continuous span of text that can include one or more
adjacent words. Entities in RadGraph center around two concepts: Anatomy and
Observation. Three uncertainty levels exist for Observation, leading to four
different entities: Anatomy (ANAT-DP), Observation: Definitely Present (OBS-
DP), Observation: Uncertain (OBS-U), and Observation: Definitely Absent (OBS-
DA).

#### Relations

A relation is defined as a directed edge between two entities. Three levels
exist: Suggestive Of (., .), Located At (., .), and Modify (., .).

Figure 4: Graph view of the RadGraph annotations for the report in fig:ddd.

###  3.2 Score

Using RadGraph annotation scheme and pretrained model, we design a F-score
style reward that measures the factual consistency and completeness of the
generated impression (also called hypothesis impression) compared to the
reference impression.

To do so, we treat the RadGraph annotations of an impression as a graph
$\mathcal{G}(V,E)$ with the set of nodes $V=\\{v_{1},v_{2},\ldots,v_{|V|}\\}$
containing the entities and the set of edges
$E=\\{e_{1},e_{2},\ldots,e_{|E|}\\}$ the relations between pairs of entities.
The graph is directed, meaning that the edge
$e=(v_{1},v_{2})\neq(v_{2},v_{1})$. An example is depicted in fig:radgraph.
Each node or edge of the graph also has a label, which we denote as
$v_{i_{L}}$ for an entity $i$ (for example “OBS-DP” or “ANAT”) and
$e_{ij_{L}}$ for a relation $e=(v_{i},v_{j})$ (such as “modified” or “located
at”).

To design our RadGraph score, we focus on the nodes $V$ and whether or not a
node has a relation in $E$. For a hypothesis impression $y$, we create a new
set of triplets ${T}_{y}=\\{{(v_{i},v_{i_{L}},\mathcal{R})}\\}_{i=1:{|V|}}$.
The value $\mathcal{R}$ is $1$ if $(v_{i},v_{j})_{j=1:{|E|},i\neq j}\in E$,
$0$ otherwise. In other words, a triplet contains an entity, the entity label
and whether or not this entity has a relation. We proceed to construct the
same set for the reference report $\hat{y}$ and denote this set
${T}_{\hat{y}}$.

Finally, our score is defined as the harmonic mean of precision and recall
between the hypothesis set ${T}_{{y}}$ and the reference set $T_{\hat{y}}$,
giving a value between $0$ and $100$. As an illustration, we provide in
Appendix C the set $V$, $E$ and $T$ of the graph $\mathcal{G}$ in Figure 4.

###  3.3 Generalization to other modalities and anatomies

As mentioned in sec:radgraph, the RadGraph model is trained on chest X-rays
entities and relations. Therefore, it is not obvious if RadGraph can be ported
to other modalities and anatomies. As a partial validation of the use of
RadGraph for our experiments, we ask one board-certified radiologist to
subjectively evaluate the entities of the RadGraph model on two randomly
selected reports for each anatomy-modality pairs. Three examples of those
reports are shown in Table 2. In the selected reports, no entities were
omitted by the RadGraph model.

##  4 Model

In this section, we detail our third contribution. We first describe in
sec:achi our simple baseline by detailing the architecture of our findings-to-
impression model and then explain in sec:training how this model can be
trained using RL to directly optimize our RadGraph score.

###  4.1 Architecture

To encode the findings, we use a BERT encoder (Vaswani et al., 2017; Devlin et
al., 2019) and use its final representation as textual features $\bm{H}$. To
generate impressions, we use a BERT decoder with cross-attention over the
textual features. More formally, the cross-attention of the decoder
transformer layer is written:

Cross-Attention $\displaystyle(Q,K,V)$ (1)
$\displaystyle=\text{softmax}(\frac{QK^{\top}}{\sqrt{d}})V$

where $Q$ is the decoder hidden state of size $d$ and $K$ and $V$ are the
textual features $\bm{H}$. We denote the number of layers in the BERT encoder
and decoder as $L$.

The encoder can be pre-trained (i.e. using the pre-trained BioMed-RoBERTa
(Gururangan et al., 2020) as encoder and decoder, for example) in which case
the number of layers $L$ is defined by the pre-trained model. We can also
train both the encoder and decoder from scratch with a custom number of layers
$L$. More details are available in our Experiment section.

###  4.2 Training

If we denote $\bm{\theta}$ as the model parameters, then $\bm{\theta}$ is
learned by maximizing the likelihood of the hypothesis impression
$\bm{Y}=(\bm{y}_{1},\bm{y}_{2},\cdots,\bm{y}_{n})$ or in other words by
minimizing the negative log likelihood (NLL). The objective function is given
by:

$\mathcal{L}(\bm{\theta})=-\sum\limits_{t=1}^{n}\log
p_{\bm{\theta}}(\bm{y}_{t}|\bm{y}<t,\bm{H})$ (2)

After the NLL training, we start a RL training by optimizing our RadGraph
score. The loss function in eq:5 is now given by:

$\mathcal{L}(\bm{\theta})=-\mathbb{E}_{\bm{Y}\sim
p_{\bm{\theta}}}\bm{r}(\bm{Y})$ (3)

where $\bm{r}(\bm{Y})$ is the reward given of the generated report. We use the
SCST algorithm (Rennie et al., 2017) to approximate the expected gradient of
our non-differentiable reward function. The expression becomes:

$\nabla_{\bm{\theta}}\mathcal{L}(\bm{\theta})\approx-(\bm{r}(\bm{Y})-\bm{r}(\bar{\bm{Y}}))\nabla_{\bm{\theta}}\log_{p_{\bm{\theta}}}(\bm{Y})$
(4)

Here $\bm{r}(\bar{\bm{Y}})$ acts a baseline (Sutton et al., 1998) to reduce
the variance of $\bm{r}(\bm{Y})$. In our case, $\bm{r}(\bar{\bm{Y}})$ is the
expected reward by sampling from the model during training.

##  5 Experiments

In this section, we detail the experiments carried out to evaluate our three
contributions.

Model | Test Scores |
---|---|---
| ROUGE1 | ROUGE2 | ROUGEL |  $\text{F}_{1}$CheXbert | RadGraph
MIMIC-CXR |  |  |  |  |
Liu and Lapata (2019) | 47.16 | 32.31 | 45.47 | — | —
Hu et al. (2021) | 48.37 | 33.34 | 46.68 | 69.15 | 44.20
Hu et al. (2022) | 51.02 | 35.21 | 46.65 | 70.73 | 45.23
BioMedRoBERTa (NLL) | 45.86 | 29.80 | 42.10 | 65.90 | 38.31
PubMedBERT (NLL) | 48.30 | 33.18 | 46.25 | 68.20 | 44.80
L2 (NLL) | 48.84 | 34.20 | 44.86 | 70.01 | 44.09
L4 (NLL) | 50.12 | 35.12 | 46.82 | 71.25 | 45.95
L8 (NLL) | 51.14 | 34.38 | 46.25 | 69.12 | 44.94
L4 (RL) | 51.96 | 35.65 | 47.10 | 74.86 | 48.23
MIMIC-III |  |  |  |  |
L4 (NLL, MR Head) | 35.84 | 16.20 | 27.50 | — | 22.50
L4 (NLL, CT Abd/Pel) | 27.62 | 7.85 | 19.69 | — | 13.02
L4 (NLL, CT Chest) | 33.62 | 13.13 | 25.01 | — | 22.19
L4 (NLL, CT Neck) | 24.36 | 8.95 | 20.78 | — | 10.06
L4 (NLL, CT Head) | 41.09 | 23.25 | 35.29 | — | 32.62

Table 3: Summary of our results discussed throughout sec:results. Each
reported score is the average of 5 independent runs. NLL refers to eq:5, RL
refers to eq:rl.

###  5.1 Model

For our BERT encoder-decoder of sec:achi, we use two pre-trained model:
BioMed-RoBERTa (Gururangan et al., 2020) and PubMedBERT (Gu et al., 2021), and
also train from scratch using $L=2$, $L=4$ and $L=8$. When training from
scratch, the rest of the model parameters can be found in Appendix D.

The details of NLL and RL training can be found in Appendix E.

###  5.2 Datasets

MIMIC-CXR To compare our baselines with the state-of-the-art, we run
summarization experiments on the MIMIC-CXR dataset (Johnson et al., 2019),
using the official splits. MIMIC-CXR contains 128,003 reports with findings
and impression split as follows: 125,388 for train, 991 for validation and
1,624 for testing.

MIMIC-III For training, we use the CT adbomen/pelvis, CT Chest, CT Neck, CT
Spine, CT Head and MR Head splits as they contain enough training data to
train our models. The last splits, namely MR Pelvis, MR Spine, MR Neck, MR
Abdomen and CT Sinus are used as test-sets.

###  5.3 Metrics

We proceed to evaluate our systems using the ROUGE1, ROUGE2 and ROUGEL metrics
(Lin, 2004) to be consistent with prior work. We also report the RadGraph
score (sec:radgraph_section) for both the MIMIC-CXR and MIMIC-III experiments.

For MIMIC-CXR, we also use F1CheXbert (Zhang et al., 2020) score alongside our
RadGraph score to evaluate the factual correctness of the generated
impressions. This metric uses CheXbert (Smit et al., 2020), a Transformer-
based model trained to output abnormalities of chest X-rays given a radiology
report (or an impression) as input. F1CheXbert is the f1-score between the
prediction of CheXbert over the hypothesis impression $\hat{y}$ and the
corresponding reference impression $y$. The f1-score is calculated over the 14
abnormalities to be consistent with Hu et al. (2022). Because the
abnormalities are specific to chest X-rays, we only use it for MIMIC-CXR.

##  6 Results

This section is divided in two parts. First, we show the results of our
preliminary experiments on MIMIC-CXR to validate our baselines and training
setup (sec:prelim. We then discuss the results on our new MIMIC-III dataset.

###  6.1 Preliminary experiments on MIMIC-CXR

The results of our preliminary experiments on MIMIC-CXR can be found in Table
3.

(NLL) When training with NLL, our best performing model is $L=4$, meaning our
BERT encoder-decoder trained from scratch with four hidden layers. For the
factual-correctness metrics, this model achieves 71.25 points of
$\text{F}_{1}$CheXbert and 45.95 points of RadGraph score. This difference is
respectively of +0.52 and +1.15 points compared to Hu et al. (2022) and +1.13
and +1.01 compared to our biggest model $L=8$. The model $L=4$ also reports
the highest ROUGE-L: 46.82. These metrics can be further improved when
ensembling models:

Surprisingly, using BioMedRoBERTa and PubMedBERT as pre-training didn’t help
improve our results. On the contrary, they report the lowest score of all our
model variants. Nevertheless, this aligns with the low results from Liu and
Lapata (2019) where authors also used pre-trained BERT encoder and decoder for
their summarization. In addition, in the task of Radiology Report Generation
(X-ray to impression), Miura et al. (2021) reported the strongest performances
using a one-layered BERT decoder.

(RL) Using RL to directly optimize the RadGraph score further improved the
metrics accross the board. Compared to L4 (NLL), L4 (RL) reports an
improvement of +2.28 (+4.96%) RadGraph score, +3.61 (+5.1%)
$\text{F}_{1}$CheXbert but only +0.28 (+0.5%) ROUGEL. fig:rl_ent shows that
our L4 (RL) model is also encouraged to generate more entities and relations
than its NLL counterpart. However, L4 (RL) generated more ”ANAT-DP” entities
and ”located at” relations than found in the reference. Conversely, it didn’t
generate any ”OSB-U” and ”suggestive-of”. One hypothesis is that, because
”OSB-U” and ”suggestive-of” are under-represented in the dataset, the RL model
couldn’t learn to use it correctly, leading to a negative reward, making in
turn the model less and less likely to use it.

Figure 5: Number of entities and relations generated by our models L4 (NLL)
and L4 (RL) on the MIMIC-CXR test set, compared to the reference (blue)

Given these encouraging results, we decide to carry out the remaining of our
experiments using the L4 model.

###  6.2 Experiments on MIMIC-III

The results of our experiments on MIMIC-III can be found in Table 3 (NLL) and
Table 5 (RL).

(NLL) Overall, the scores are lower than the MIMIC-CXR experiments. Two
reasons explain this trend: 1) all MIMIC-III datasets contain fewer training
samples, and 2) as shown in fig:stats, the findings and impressions in the
MIMIC-III datasets are substantially longer than MIMIC-CXR. Hence, the
summarization task is more difficult. As an example, we highlight in Table 4
generated impression of by two of our models. Such examples show that our
model L4 (NLL) has learned, to some extent, to summarize effectively. A more
in-depth study would be required to evaluate the output on the whole test-set,
for different modalities and anatomies.

(RL) As shown in Table 5, directly optimizing the RadGraph score using
Reinforcement Learning improves the ROUGE and RadGraph score for all
modalities and anatomies. It shows convincingly that RadGraph can be used,
either as a score to evaluate the radiology reports or as reward to optimize
factual correctness, for various types of reports.

L4 (NLL): | no significant change in degree
---|---
| of intraventricular hemorrhage
| compared to prior study
L4 (RL): | 1 no evidence of acute
| intraventricular hemorrhage or
| mass effect 2 no evidence of
| acute bleed
Reference: | stable appearance of large
| intraventricular hemorrhage
| with no evidence of obstructive
| hydrocephalus or new bleed.

Table 4: Cherry picked generated CT HEAD impressions from our models and its
reference.

| ROUGE | RadGraph score
---|---|---
MR Head | 29.10 (+5.8%) | 24.19 (+7.5%)
CT ab/p | 21.32 (+8.2%) | 15.10 (+14.3%)
CT Chest | 25.50 (+1.5%) | 25.06 (+12.9%)
CT Neck | 21.18 (+1.9%) | 12.56 (+24.8%)
CT Head | 35.52 (+0.6%) | 34.95 (+6.1%)

Table 5: Performance of our L4 (RL) on the five largest sets of our MIMIC-III
summarization dataset with improvements in % compared to their NLL counterpart
from Table 3

##  7 Conclusion

In this paper, we present three original contributions to address the current
weakness of the task of Radiology Report Summarization. First, we release a
pre-processed and curated dataset of radiology reports for new modalities (MR
and CT) and anatomies (Chest, Head, Neck, Sinus, Spine, Abdomen, Pelvis) based
MIMIC-III database (sec:mimiciiisum). This allows future research to extend
the scope of this task beyond chest x-rays. Then, we presented a new
summarization metric, called the RadGraph score (sec:radgraph_section), that
evaluates the factual completeness and correctness of the generated radiology
impressions. We showed theoretically (sec:generalization) and empircally
(sec:exp:mimiciii) that this new evaluation metric was suitable for every
modality and anatomy of our new dataset.

Finally, we presented a new simple summarization system (sec:achi) that not
only acted a strong baseline on the new datasets but also outperforms the
previous replicable research on chest X-rays (sec:results). Our model, the
dataset, and the score can be found and replicated in Appendix B.

##  8 Future work

RadGraph has shown to be an efficient tool and could be used in different ways
than presented in this paper. One could train a general model on the most
frequent modalities to test its performance on the ood test-sets, namely CT
neck, CT sinus, MR Pelvis, MR Neck, MR Abdomen, MR Spine. The goal is to
evaluate if knowledge can be transferred to unseen modalities and anatomies.

## References

  * Ben Abacha et al. (2021) Asma Ben Abacha, Yassine Mrabet, Yuhao Zhang, Chaitanya Shivade, Curtis Langlotz, and Dina Demner-Fushman.  Overview of the MEDIQA 2021 shared task on summarization in the medical domain.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 74–85, Online, June 2021. Association for Computational Linguistics.  10.18653/v1/2021.bionlp-1.8.  URL https://aclanthology.org/2021.bionlp-1.8.
  * Delbrouck et al. (2022) Jean-benoit Delbrouck, Khaled Saab, Maya Varma, Sabri Eyuboglu, Pierre Chambon, Jared Dunnmon, Juan Zambrano, Akshay Chaudhari, and Curtis Langlotz.  ViLMedic: a framework for research at the intersection of vision and language in medical AI.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 23–34, Dublin, Ireland, May 2022. Association for Computational Linguistics.  10.18653/v1/2022.acl-demo.3.  URL https://aclanthology.org/2022.acl-demo.3.
  * Demner-Fushman et al. (2012) Dina Demner-Fushman, Sameer Antani, Matthew Simpson, and George R Thoma.  Design and development of a multimodal biomedical information retrieval system.  _Journal of Computing Science and Engineering_ , 6(2):168–177, 2012.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ , 3(1):1–23, 2021.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of ACL_ , 2020.
  * Hu et al. (2021) Jinpeng Hu, Jianling Li, Zhihong Chen, Yaling Shen, Yan Song, Xiang Wan, and Tsung-Hui Chang.  Word graph guided summarization for radiology findings.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4980–4990, Online, August 2021. Association for Computational Linguistics.  10.18653/v1/2021.findings-acl.441.  URL https://aclanthology.org/2021.findings-acl.441.
  * Hu et al. (2022) Jinpeng Hu, Zhuo Li, Zhihong Chen, Zhen Li, Xiang Wan, and Tsung-Hui Chang.  Graph enhanced contrastive learning for radiology findings summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4677–4688, 2022\.
  * Jain et al. (2021) Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew Lungren, Andrew Ng, Curtis Langlotz, Pranav Rajpurkar, and Pranav Rajpurkar.  Radgraph: Extracting clinical entities and relations from radiology reports.  In J. Vanschoren and S. Yeung, editors, _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1, 2021.
  * Johnson et al. (2019) Alistair E. W. Johnson, Tom J. Pollard, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G. Mark, Seth J. Berkowitz, and Steven Horng.  MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs.  _arXiv e-prints_ , art. arXiv:1901.07042, January 2019.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9, 2016.
  * Johnson et al. (2019) Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng.  Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific data_ , 6(1):1–8, 2019.
  * Kahn Jr et al. (2009) Charles E Kahn Jr, Curtis P Langlotz, Elizabeth S Burnside, John A Carrino, David S Channin, David M Hovsepian, and Daniel L Rubin.  Toward best practices in radiology reporting.  _Radiology_ , 252(3):852–856, 2009.
  * Karn et al. (2022) Sanjeev Kumar Karn, Ning Liu, Hinrich Schütze, and Oladimeji Farri.  Differentiable multi-agent actor-critic for multi-step radiology report summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1542–1553, 2022\.
  * Lin (2004) Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81, 2004.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China, November 2019. Association for Computational Linguistics.  10.18653/v1/D19-1387.  URL https://aclanthology.org/D19-1387.
  * Miura et al. (2021) Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, and Dan Jurafsky.  Improving factual completeness and consistency of image-to-text radiology report generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5288–5304, 2021.
  * Rennie et al. (2017) Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7008–7024, 2017.
  * Smit et al. (2020) Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew Lungren.  Combining automatic labelers and expert annotations for accurate radiology report labeling using bert.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1500–1519, 2020.
  * Sutton et al. (1998) Richard S Sutton, Andrew G Barto, et al.  Introduction to reinforcement learning.  _MIT press Cambridge_ , 1998.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , pages 38–45, 2020.
  * Zhang et al. (2018) Yuhao Zhang, Daisy Yi Ding, Tianpei Qian, Christopher D Manning, and Curtis P Langlotz.  Learning to summarize radiology findings.  In _Proceedings of the Ninth International Workshop on Health Text Mining and Information Analysis_ , pages 204–213, 2018.
  * Zhang et al. (2020) Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D Manning, and Curtis Langlotz.  Optimizing the factual correctness of a summary: A study of summarizing radiology reports.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5108–5120, 2020.
  * Zhang et al. (2021) Yuhao Zhang, Yuhui Zhang, Peng Qi, Christopher D Manning, and Curtis P Langlotz.  Biomedical and clinical english model packages for the stanza python nlp library.  _Journal of the American Medical Informatics Association_ , 28(9):1892–1899, 2021.

##  Appendix A Related work

We restrict this section to Radiology Report Summarization.
The first attempt at automatic summarization of radiology finding into natural
language impression statements is proposed by Zhang et al. (2018). Their
contribution is to propose a first baseline on the task, using a
bidirectional-LSTM as encoder and decoder. Importantly, they found that about
30% of the radiology summaries generated from neural models contain factual
errors. Subsequently, Zhang et al. (2020) proposed the $\text{F}_{1}$CheXbert
score to evaluate the factual correctness of the generated impression. They
also use RL to directly optimize that metric. Finally, both Hu et al. (2021)
and Hu et al. (2022) used the Biomedical and Clinical English Model Packages
in the Stanza Python NLP Library (Zhang et al., 2021) to extract medical
entites. The former study used the entities to construct a Graph Neural
Network, used as input in their summarization pipeline, while the latter study
used the entites to mask the findings in a contrastive pre-training.

We believe this paper is an original contribution to the aforementioned work.
As instigated by Zhang et al. (2018), our goal is to release new summarization
corpus and baselines on new modalities and anatomies. We do so by releasing 11
new anatomy-modality pairs. Similarly to Zhang et al. (2020), we continue the
effort in proposing a new metric that evaluate the factual correctness and
completeness of the generated impression, namely the RadGraph score. Finally,
we improve the work of Hu et al. (2021) and Hu et al. (2022) in two ways: 1)
we use semantic annotations from a pre-trained model supervised on board-
certified radiologists annotations, as opposed to Stanza that levarages
unsupervised biomedical and clinical text data 2) we leverage relation
annotations between entities, a feature that was not available to prior work.

##  Appendix B Code and data release

To help with further research, we also make our code publicly available. More
specifically, we release the code of the RadGraph score as well as the
training of our baseline. We also release the script to download, pre-process
and split the radiology reports of the MIMIC-III database as per our
experiments.

To download the MIMIC-III database, researchers are required to formally
request access via a process documented on the MIMIC website. There are two
key steps that must be completed before access is granted:

1) the researcher must complete a recognized course in protecting human
research participants that includes Health Insurance Portability and
Accountability Act (HIPAA) requirements.

2) the researcher must sign a data use agreement, which outlines appropriate
data usage and security standards, and forbids efforts to identify individual
patients.

Our research has been carried out using the ViLMedic library (Delbrouck et
al., 2022). Our code is available at https://github.com/jbdel/vilmedic. This
link is anonymized and complies with the double blind review process.

##  Appendix C Set $\bar{V}$ of Figure 4

$V=\\{$mild, fluid, overload, overt, pulmonary, edema$\\}$
$E=\\{$(mild,overload), (overload, fluid), (edema, pulmonary)$\\}$

$T=\\{$(mild, obs-dp, 1), (fluid, obs-dp, 0), (overload, obs-dp, 1), (overt,
obs-da, 0), (pulmonary, anat-dp, 0), (edema, obs-da, 1)$\\}$

##  Appendix D Model hyper-parameters

Our model is built using HuggingFace (Wolf et al., 2020). For reference, the
following configuration declares 40.77M training parameters for $L=2$.

⬇

{

encoder:

attention_probs_dropout_prob: 0.1

hidden_act: gelu

hidden_dropout_prob: 0.1

hidden_size: 768

initializer_range: 0.02

intermediate_size: 3072

layer_norm_eps: 1e-05

max_position_embeddings: 514

num_attention_heads: 12

num_hidden_layers: L

position_embedding_type: absolute

decoder:

add_cross_attention: true

attention_probs_dropout_prob: 0.1

hidden_act: gelu

hidden_dropout_prob: 0.1

hidden_size: 768

initializer_range: 0.02

intermediate_size: 3072

layer_norm_eps: 1e-05

max_position_embeddings: 514

num_attention_heads: 12

num_hidden_layers: L

position_embedding_type: absolute

}

Listing 1: Model parameters

##  Appendix E Training hyper-parameters

For NLL and RL training, we use the following hyper-parameters (number of bert
layers $L=4$):

⬇

{

batch_size (mimic-cxr): 128

batch_size (mimic-iii): 24

optimizer: Adam

optim_params:

lr: 1e-4

weight_decay: 0.

lr_decay: ReduceLROnPlateau

lr_decay_params:

factor: 0.8

patience: 1

min_lr: 0.000001

threshold_mode: abs

}

Listing 2: Training parameters

The ReduceLROnPlateau scheduler is on the ROUGE metric for NLL training and
RadGraph score for RL training.

During RL training, the reward $\bm{r}(\bm{Y})$ at eq:reward is computed as
$0.01*\text{NLL loss}+0.495*\text{BERTSCORE}+0.495*\text{RadGraph score}$ as
suggested by Miura et al. (2021). If one optimizes a factual metric based on
entities alone, then the model is not encouraged to keep a correct grammatical
structure.