#  ICon: Improving Inter-Report Consistency of Radiology Report Generation via
Lesion-aware Mix-up Augmentation

Wenjun Hou1,2, Yi Cheng1∗, Kaishuai Xu1∗, Yan Hu2, Wenjie Li1†, Jiang Liu2†
1Department of Computing, The Hong Kong Polytechnic University, HKSAR, China
2Research Institute of Trustworthy Autonomous Systems and
Department of Computer Science and Engineering,
Southern University of Science and Technology, Shenzhen, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Previous research on radiology report generation has made significant progress
in terms of increasing the clinical accuracy of generated reports. In this
paper, we emphasize another crucial quality that it should possess, i.e.,
inter-report consistency, which refers to the capability of generating
consistent reports for semantically equivalent radiographs. This quality is
even of greater significance than the overall report accuracy in terms of
ensuring the system’s credibility, as a system prone to providing conflicting
results would severely erode users’ trust. Regrettably, existing approaches
struggle to maintain inter-report consistency, exhibiting biases towards
common patterns and susceptibility to lesion variants. To address this issue,
we propose ICon, which Improves the inter-report Consistency of radiology
report generation. Aiming at enhancing the system’s ability to capture the
similarities in semantically equivalent lesions, our approach involves first
extracting lesions from input images and examining their characteristics.
Then, we introduce a lesion-aware mix-up augmentation technique to ensure that
the representations of the semantically equivalent lesions align with the same
attributes, by linearly interpolating them during the training phase.
Extensive experiments on three publicly available chest X-ray datasets verify
the effectiveness of our approach, both in terms of improving the consistency
and accuracy of the generated reports111Our codes and model checkpoints are
available at https://github.com/wjhou/ICon.

ICon: Improving Inter-Report Consistency of Radiology Report Generation via
Lesion-aware Mix-up Augmentation

Wenjun Hou1,2, Yi Cheng1∗, Kaishuai Xu1∗, Yan Hu2, Wenjie Li1†, Jiang Liu2†
1Department of Computing, The Hong Kong Polytechnic University, HKSAR, China
2Research Institute of Trustworthy Autonomous Systems and Department of
Computer Science and Engineering, Southern University of Science and
Technology, Shenzhen, China<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

$\ast$$\ast$footnotetext: Equal Contribution.$\dagger$$\dagger$footnotetext:
Corresponding authors. Figure 1: Given two semantically equivalent cases
(i.e., Case A and Case B), an example to illustrate the difference between
three radiology report generation systems: a consistent and accurate system
(i.e., System $\alpha$) and a consistently inaccurate system (i.e., System
$\beta$), and an inconsistent system (i.e., System $\gamma$).

##  1 Introduction

Being part of the diagnostic process, radiology report generation Shin et al.
(2016); Zhang et al. (2017); Jing et al. (2018) has garnered significant
attention within the research community, due to its large potential to
alleviate the heavy strain of radiologists. Recent research Nishino et al.
(2022); Tanida et al. (2023); Hou et al. (2023b) has made noteworthy progress
in enhancing the clinical accuracy of the generated reports.

However, constructing a credible report generation system goes beyond the
overall accuracy. There is another crucial quality for report generation
systems that has been largely overlooked in the existing literature of medical
report generation, which is, inter-report consistency Elazar et al. (2021). To
illustrate the disparity between accuracy and inter-report consistency, we
exemplify two semantically equivalent cases as shown in Figure 1.
Specifically, System $\alpha$ demonstrates the ability to maintain both inter-
report consistency and factual accuracy for two similar cases (i.e., "mildly
enlarged" for positive Cardiomegaly), whereas other systems (i.e., $\beta$ and
$\gamma$) fail to meet these criteria. These systems might have overfitted to
ordinary cases and could be vulnerable to noise or attack. In terms of
enhancing the system’s credibility, inter-report consistency might even hold
greater significance than the overall accuracy, since a system prone to
providing conflicting results would severely undermine users’ trust Qayyum et
al. (2020); Asan et al. (2020). Regrettably, existing report generation
systems struggle to maintain this important quality. They tend to exhibit
biases towards common patterns, primarily describing normal observations, and
are extremely susceptible to lesion variants and context noise Chen et al.
(2020); Qin and Song (2022); Ma et al. (2021); Kaviani et al. (2022). We argue
that this is largely due to their limited capability of capturing shared
attributes of similar patterns, which arises from the data scarcity of
distributed lesions and their semantically equivalent variants, rendering it
challenging for neural models to accurately locate and describe abnormalities.

In this paper, we propose ICon, which aims to Improves inter-report
Consistency of radiology report generation. Our proposed method involves first
extracting lesions from given input images, followed by examining the
attributes of these lesions. Subsequently, both the radiographs and their
associated attributes are utilized as inputs for report generation. To further
enhance the inter-report consistency, we introduce a lesion-aware mix-up
augmentation technique by learning from linearly interpolated lesions and
attributes that belong to the same observation. In summary, the contributions
of this paper are as follows:

  * •

We propose ICon which improves the consistency of the radiology report
generation system by capturing abnormalities at the lesion level. ICon only
requires coarse-grained labels (i.e., image labels) to extract lesions, which
can be easily acquired and transferred to other modalities, compared with
bounding boxes Tanida et al. (2023).

  * •

We introduce a lesion-aware mix-up augmentation to capture the shared features
between two chest X-ray lesions to further enhance consistency. Besides, two
metrics (i.e., Con and R-Con) are devised to quantify the inter-report
consistency.

  * •

Extensive experiments are conducted on three publicly available datasets, and
experimental results demonstrate the effectiveness of ICon in terms of
improving both the consistency and accuracy of the generated reports.

##  2 Priliminaries

###  2.1 Problem Formulation

Given a set of radiographs $\mathcal{X}=\\{X_{1},\dots,X_{L}\\}$ in one study,
along with its historical records
$\mathcal{X}^{p}=\\{X_{1}^{p},\dots,X_{|p|}^{p}\\}$ or
$\mathcal{X}^{p}=\emptyset$, and its report
$\mathcal{Y}=\\{y_{1},\dots,y_{T}\\}$, the task of radiology report generation
(RRG) is formulated as $p(\mathcal{Y}|\mathcal{X},\mathcal{X}^{p})$. We
elaborate on the justification of using the historical records as context in
Appendix A.7. Our proposed method, denoted as ICon, decomposes the RRG task
into two stages, i.e., Lesion Extraction (Stage 1) and Report Generation
(Stage 2). Specifically, given the input images $\mathcal{X}$, ICon first
extracts lesions $\mathcal{Z}=\\{Z_{1},\dots,Z_{|O|}\\}$ from $\mathcal{X}$,
where the probability of a region $R_{i,j}$ from image $X_{i}$ being
identified as a lesion $Z_{k}$ is estimated as $p(Z_{k}|X_{i})$. Subsequently,
in Stage 2, ICon generates a report based on both the input images and the
extracted lesions, modeled as
$P(\mathcal{Y}|\mathcal{X},\mathcal{X}^{p},\mathcal{Z})$. Finally, our
framework aims to maximize the following probability:

$\begin{split}P(\mathcal{Y}|\mathcal{X},\mathcal{X}^{p})\propto&\underbrace{p(\mathcal{Z}|\mathcal{X})}_{\text{Stage
1}}\cdot\underbrace{P(\mathcal{Y}|\mathcal{X},\mathcal{X}^{p},\mathcal{Z})}_{\text{Stage
2}}.\end{split}$

###  2.2 Observation Annotation for Lesion Extraction

In order to perform lesion extraction, ICon requires report-level labels for
lesion extraction, whereas benchmarking datasets only provide token-level
labels. Therefore, it is necessary to annotate the reports to obtain higher-
level labels. In this work, we adopt CheXbert Smit et al. (2020) for this
purpose. Specifically, CheXbert first annotates a report with 14 observation
categories $O=\\{o_{1},\dots,o_{14}\\}$ (refer to Appendix A.1 for data
statistics). Each observation is then assigned one of four statuses (i.e.,
Present, Absent, Uncertain, and Blank). During training and evaluation,
Present and Uncertain are merged into the Positive category, which represents
abnormal observation. Note that for the observation category No Finding, only
two statuses, Present or Absent, are applicable. Finally, observation
information is utilized for lesion extraction as will be illustrated in §3.2.

Figure 2: Overview of the ICon framework, which first extracts lesions from
radiographs and then generates reports.

###  2.3 Inter-Report Consistency Metrics

To assess the inter-report consistency of a model, we introduce two metrics,
i.e., Con and R-Con, similar to Elazar et al. (2021). We start by gathering
samples with semantically equivalent images based on their reports’
similarity. To be specific, given a corpus $\mathcal{D}$ filtered out samples
without abnormality, we first build a BM25 retriever Robertson and Zaragoza
(2009) based on the training data, and then the top-$N$ ($N=5$) relevant
reports are selected as the semantically equivalent samples
$\mathcal{K}_{i}=\\{K_{i},\dots,K_{N}\\}$ for each report $Q_{i}$ in
$\mathcal{D}$. We also collect the corresponding outputs of $\mathcal{K}_{i}$
from a model, denoted as
$\hat{\mathcal{K}_{i}}=\\{\hat{K_{1}},\dots,\hat{K_{N}}\\}$. Afterward, the
Jaccard Index Jaccard (1901); Wikipedia (2024) is adopted to measure the
similarity between two generated reports $(\hat{Q_{i}},\hat{K_{j}})$:

$\mathtt{Jaccard}(\hat{Q_{i}},\hat{K_{j}})=\frac{|\hat{t_{i}}\cap\hat{t_{j}}|}{|\hat{t_{i}}\cup\hat{t_{j}}|},$

where $\hat{t}_{i}$ and $\hat{t}_{j}$ are tokens222Stopwords are removed in
advanced. in $\hat{Q_{i}}$ and $\hat{K_{j}}$, respectively. Then, the
consistency is defined as:

$\textsc{Con}=\frac{1}{|D|\times
N}\sum_{i=1}^{|D|}\sum_{\begin{subarray}{c}j=1\end{subarray}}^{N}\mathtt{Jaccard}(\hat{Q_{i}},\hat{K_{j}}).$

As Con only considers the consistency among the model’s outputs, ignoring the
qualities concerning the references. To consider both consistency and
accuracy, for each output $\hat{Q_{i}}$, we plug a factor
$\tau_{i}=\mathtt{Jaccard}(\hat{Q_{i}},Q_{i})$, which measures the similarity
between the hypothesis $\hat{Q_{i}}$ and reference $Q_{i}$. Then, Con is
modified as R-Con:

$\textsc{R-Con}=\frac{1}{|D|\times
N}\sum_{i=1}^{|D|}\sum_{j=1}^{N}\tau_{i}\cdot\mathtt{Jaccard}(\hat{Q_{i}},\hat{K_{j}}).$

We report the Con and R-Con results in Table 3.

##  3 Methodology

###  3.1 Visual Encoding

Given an image $X_{l}$, an image processor is first utilized to split $X_{l}$
into $N$ patches, and then a visual encoder $f_{\theta}$ (e.g., Swin
Transformer Liu et al. (2021d)) is employed to extract visual representations
$\bm{X}_{l}$ and the pooler output $\bm{P}_{l}\in\mathbb{R}^{h}$:

$\begin{split}[\bm{P}_{l},\bm{X}_{l}]=f_{\theta}(X_{l}),\end{split}$

where $\bm{X}_{l}=\\{\bm{x}_{l,i},\dots,\bm{x}_{l,N}\\}$ and
$\bm{x}_{l,i}\in\mathbb{R}^{h}$ is the $i$-th visual representation.

###  3.2 Stage 1: Extracting Lesions via Observation Classification (Zoomer)

Observation Classification. A Zoomer is a visual encoder parameterized by
$\theta_{\text{Z}}$ and trained to classify a given input $\mathcal{X}$ into
abnormal observations as mentioned in §2.2:

$p(o_{i})=\textsc{Zoomer}(\mathcal{X}).$

Specifically, Zoomer first encodes images
$\mathcal{X}=\\{X_{1},\dots,X_{L}\\}$ as outlined in §3.1, and then takes the
averaged pooler output for classification, following these steps:

${}_{l}\begin{gathered}=f_{\theta_{\text{Z}}}(X_{l}),\\\
\bm{P}=\frac{1}{L}\sum\bm{P}_{l},\\\
p(o_{i})=\sigma(\bm{W}_{i}\bm{{P}}+b_{i}),{}\end{gathered}$

where $\bm{W}_{i}\in\mathbb{R}^{h}$ is the weight for $i$-th observation,
$b_{i}\in\mathbb{R}$ is its bias, and $\sigma$ is the Sigmoid function.

Zooming In for Lesion Extraction. Upon completing training Zoomer, we can use
it to extract lesions without the need for object detectors Ren et al. (2015).
It is worth noting that our method does not require fine-grained labels, such
as bounding boxes Tanida et al. (2023), making it easily adaptable to other
modalities, e.g., FFA images Li et al. (2021).

For an image $X_{l}$, a sliding window with a $0.375$ ratio of $X_{l}$ is
applied to extract $M$ region candidates
$\mathcal{R}_{l}=\\{R_{l,1},\dots,R_{l,M}\\}$ from $X_{l}$, as shown in the
left side of Figure 2. These regions are then sequentially fed into Zoomer for
classification. Further details on the extraction of these regions can be
found in Appendix A.5. The probability of a region $R_{l,j}$ being classified
as an abnormal observation $o_{i}$ is:

$\begin{split}p_{l,j}(o_{i})=\textsc{Zoomer}(R_{l,j}).\end{split}$

For each study, all images in $\mathcal{X}$ are iterated, and only the region
with the highest $p_{l,j}(o_{i})$ is chosen as a lesion $Z_{i}$ corresponding
to the observation $o_{i}$. Finally, the set of lesions is denoted as
$\mathcal{Z}=\\{Z_{1},\dots,Z_{|O|}\\}$.

Training Zoomer. Zoomer is optimized using the binary cross-entropy (BCE)
loss. To handle the class-imbalanced issue (refer to Appendix A.1 for
details), a weight factor $\alpha_{j}$ is plugged for each abnormal
observation and the loss function $\mathcal{L}_{\text{S1}}$ is:

$\begin{split}\mathtt{BCE}(p(o_{*}),O)=&-\frac{1}{|O|}\sum_{j}[\alpha_{j}\cdot
o_{j}\cdot\mathtt{log}\>p(o_{j})\\\
&+(1-o_{j})\mathtt{log}(1-p(o_{j}))],\end{split}$

where $o_{j}\in\\{0,1\\}$ is the label,
$\alpha_{j}=1+\mathtt{log}(\frac{|\mathcal{D}_{\text{train}}|-w_{j}}{w_{j}})$,
and $|\mathcal{D}_{\text{train}}|$ and $w_{j}$ are the number of sample and
the number of $j$-th observation in the training set, respectively.

Dataset | Model | NLG Metrics | CE Metrics (CheXbert)
---|---|---|---
B-1 | B-2 | B-3 | B-4 | MTR | R-L | P | R | F1
MIMIC -ABN | R2Gen | $0.290$ | $0.157$ | $0.093$ | $0.061$ | $0.105$ | $0.208$ | $0.266$ | $0.320$ | $0.272$
R2GenCMN | $0.264$ | $0.140$ | $0.085$ | $0.056$ | $0.098$ | $0.212$ | $0.290$ | $0.319$ | $0.280$
ORGan | $0.314$ | $0.180$ | $0.114$ | $0.078$ | $0.120$ | $0.234$ | $0.271$ | $0.342$ | $0.293$
Recap | $0.321$ | $0.182$ | $0.116$ | $0.080$ | $0.120$ | $0.223$ | $0.300$ | $0.363$ | $0.305$
ICon (Ours) | $\bm{0.337}$ | $\bm{0.195}$ | $\bm{0.126}$ | $\bm{0.086}$ | $\bm{0.129}$ | $\bm{0.236}$ | $\bm{0.332}$ | $\bm{0.430}$ | $\bm{0.360}$
MIMIC -CXR | R2Gen | $0.353$ | $0.218$ | $0.145$ | $0.103$ | $0.142$ | $0.270$ | $0.333$ | $0.273$ | $0.276$
R2GenCMN | $0.353$ | $0.218$ | $0.148$ | $0.106$ | $0.142$ | $0.278$ | $0.344$ | $0.275$ | $0.278$
$\mathcal{M}^{2}$Tr | $0.378$ | $0.232$ | $0.154$ | $0.107$ | $0.145$ | $0.272$ | $0.240$ | $0.428$ | $0.308$
KnowMat | $0.363$ | $0.228$ | $0.156$ | $0.115$ | $-$ | $0.284$ | $0.458$ | $0.348$ | $0.371$
CMM-RL | $0.381$ | $0.232$ | $0.155$ | $0.109$ | $0.151$ | $0.287$ | $0.342$ | $0.294$ | $0.292$
CMCA | $0.360$ | $0.227$ | $0.156$ | $0.117$ | $0.148$ | $0.287$ | ${0.444}$ | $0.297$ | $0.356$
KiUT | $0.393$ | $0.243$ | $0.159$ | $0.113$ | $0.160$ | $0.285$ | $0.371$ | $0.318$ | $0.321$
DCL | $-$ | $-$ | $-$ | $0.109$ | $0.150$ | $0.284$ | $0.471$ | $0.352$ | $0.373$
METrans | $0.386$ | $0.250$ | $0.169$ | $0.124$ | $0.152$ | $0.291$ | $0.364$ | $0.309$ | $0.311$
RGRG | $0.373$ | $0.249$ | $0.175$ | $\bm{0.126}$ | $0.168$ | $0.264$ | $0.380$ | $0.319$ | $0.305$
ORGan | $0.407$ | $0.256$ | $0.172$ | $0.123$ | $0.162$ | $\bm{0.293}$ | $0.416$ | $0.418$ | $0.385$
Recap | $0.429$ | $\bm{0.267}$ | $\underline{0.177}$ | ${0.125}$ | $\underline{0.168}$ | $0.288$ | $0.389$ | $\underline{0.443}$ | $\underline{0.393}$
ICon (Ours) | $\bm{0.429}$ | $\underline{0.266}$ | $\bm{0.178}$ | $\bm{0.126}$ | $\bm{0.170}$ | $0.287$ | $\underline{0.445}$ | $\bm{0.505}$ | $\bm{0.464}$
|  |  |  |  |  |  |  |  |  |

Table 1: Experimental results of our model and baselines on the MIMIC-ABN and
MIMIC-CXR datasets. The best results are in boldface, and the underlined are
the second-best results.

###  3.3 Stage 2: Inspecting Lesions (Inspector)

Inspecting Lesions with Attributes. Given that lesions of the same observation
can exhibit different characteristics, it is crucial to inspect each lesion
and match it with corresponding attributes to differentiate it from other
variations. We adopt the attributes released by Hou et al. (2023a) as labels,
and for each observation, the top-$K$ attributes are selected as candidates.
In specific, an Inspector is a visual encoder parameterized by $\theta_{I}$,
similar to §3.2. $\textsc{Inspector}(\bm{{P}}^{p},\bm{{P}},Z_{j})$ takes prior
and current visits chest X-rays as context, along with a lesion region as
input:

${}_{j}\begin{gathered}=f_{\theta_{I}}(Z_{j}),\\\
p_{j}(a_{k})=\sigma(\mathtt{MLP}(\bm{{P}}^{p},\bm{{P}},\bm{P}_{Z_{j}})),{}\end{gathered}$

where $\mathtt{MLP}$ is a two-layer perception with non-linear activation, and
$\bm{{P}}^{p},\bm{{P}},\bm{P}_{Z_{j}}\in\mathbb{R}^{h}$ are pooler output of
prior images, current images, and the lesion, respectively. The lesion
features $\bm{\mathcal{Z}}=\\{\bm{Z}_{1},\dots,\bm{Z}_{|O|}\\}$ are then
collected for report generation. For image encoding, we use another visual
encoder $f_{\theta_{V}}$ to encoder $\mathcal{X}$ into $\bm{\mathcal{X}}$ and
$\mathcal{X}^{p}$ into $\bm{\mathcal{X}}^{p}$.

Figure 3: Overview of our proposed lesion-aware mix-up augmentation.

Lesion-aware Mix-up Augmentation. To further improve the consistency of the
generated outputs, we adopt the Mix-up Augmentation method Zhang et al. (2018)
and devise a Lesion-aware Mix-up during the training phase. To be specific,
for a lesion-attribute pair $(Z_{j},A_{j})$, we retrieve a similar pair
$(Z_{k},A_{k})$ with the same observation from the training data based on
report similarity. These lesions are combined using linear interpolation, as
illustrated in Figure 3:

${Z}^{*}_{j}=\lambda Z_{j}+(1-\lambda)Z_{k},$

where $\lambda$ is set to $0.75$. Note that during training, $Z^{*}_{j}$ is
used for both Inspector and Generator.

Training Inspector. Similar to §3.2, we adopt a linearly interpolated BCE loss
to optimize Inspector:

$\mathcal{L}_{\text{I}}=\lambda\mathtt{BCE}_{j}+(1-\lambda)\mathtt{BCE}_{k},$

where $\mathtt{BCE}_{j}$ and $\mathtt{BCE}_{k}$ take $A_{j}$ and $A_{k}$ as
their respective labels. Notably, only the attributes that are shared between
$Z_{j}$ and $Z_{k}$ are fully optimized. Consequently, our lesion-aware mix-up
augmentation technique facilitates the improvement of output consistency for
two semantically equivalent lesions.

###  3.4 Generating Consistent Radiology Report (Generator)

Lesion-Attribute Alignment. To bridge the modality gap between lesion
representations and text-based attributes, we leverage a BART Lewis et al.
(2020) encoder to extract attribute representations. The attributes associated
with each lesion are formulated as a prompt: <s>$o_{j}$</s>$A_{j}$</s>, as
depicted in the upper part of Figure 2. Then, a cross-attention module Vaswani
et al. (2017) is inserted after every self-attention module. This module
aligns the lesion representations with the attribute representations by
querying visual representations using attribute representations, similar to
Q-Former Li et al. (2023a):

$\bm{H}^{a}_{j}=\mathtt{Attention}(\bm{H}^{s}_{j},\bm{Z}_{j},\bm{Z}_{j}),$

where $\bm{H}^{a}_{j},\bm{H}^{s}_{j}\in\mathbb{R}^{h}$ are the aligned
attribute representation and the self-attended representation of $A_{j}$,
respectively. All prompts are encoded and the attribute representations of
$\bm{\mathcal{Z}}$ are denoted as $\bm{\mathcal{H}}^{a}$.

Report Generation. Given the input images $\bm{\mathcal{X}}$, images of prior
visit $\bm{\mathcal{X}}^{p}$, the lesions $\bm{\mathcal{Z}}$, and attribute
$\bm{\mathcal{H}}^{a}$, we utilize a BART decoder in conjunction with the
Fusion-in-Decoder (FiD; Izacard and Grave (2021)) technique that simply
concatenates multiple context sequences for report generation. Then, the
probability of $t$-th step is expressed as:

$\begin{gathered}\bm{h}_{t}=\mathtt{FiD}([\bm{\mathcal{X}};\bm{\mathcal{X}}^{p};\bm{Z};\bm{\mathcal{H}}^{a}],\bm{h}_{<t}),\\\
p(y_{t}|\mathcal{X},\mathcal{X}^{p},\mathcal{Z},\mathcal{Y}_{<t})=\mathtt{Softmax}(\bm{W}_{g}\bm{h}_{t}+\bm{b}_{g}),\end{gathered}$

where $\bm{h}_{t}\in\mathbb{R}^{h}$ is the $t$-th hidden representation,
$\bm{W}_{g}\in\mathbb{R}^{|\mathcal{V}|\times h}$ is the weight matrix,
$\bm{b}_{g}\in\mathbb{R}^{|\mathcal{V}|}$ is the bias vector, and
$\mathcal{V}$ is the vocabulary.

Training Generator. The generation process is optimized using the negative
log-likelihood loss, given each token’s probability
$p(y_{t}|\mathcal{X},\mathcal{X}^{p},\mathcal{Z},\mathcal{Y}_{<t})$:

$\begin{split}\mathcal{L}_{\text{G}}=-&\sum^{T}_{t=1}\mathtt{log}\>p(y_{t}|\mathcal{X},\mathcal{X}^{p},\mathcal{Z},\mathcal{Y}_{<t}).\end{split}$

The loss function of Stage 2 is:
$\mathcal{L}_{\text{S2}}=\mathcal{L}_{\text{I}}+\mathcal{L}_{\text{G}}$.

##  4 Experiments

###  4.1 Datasets

Three public datasets are used to evaluate our models, i.e., IU
X-ray333https://openi.nlm.nih.gov/ Demner-Fushman et al. (2016), MIMIC-
CXR444https://physionet.org/content/mimic-cxr-jpg/2.0.0/ Johnson et al.
(2019), and MIMIC-ABN555https://github.com/zzxslp/WCL Ni et al. (2020). We
follow previous research Chen et al. (2020) to preprocess these datasets, and
provide other details in Appendix A.6.

  * •

IU X-ray consists of 3,955 reports. We follow previous research Chen et al.
(2020) and split the dataset into train/validation/test sets with a ratio of
7:1:2.

  * •

MIMIC-CXR consists of 377,110 chest X-ray images and 227,827 reports.

  * •

MIMIC-ABN is modified from the MIMIC-CXR dataset and its reports only contain
abnormal part. We adopt the data-split as used in Hou et al. (2023a), and the
data-split is 71,786/546/806 for train/validation/test sets.

Unlike previous research Chen et al. (2020); Hou et al. (2023a) which only
used one view for report generation on MIMIC-CXR and MIMIC-ABN, we collect all
views for each visit in experiments. The justification is provided in Appendix
A.7.

Dataset | Model | NLG Metrics | CE Metrics (RadGraph)
---|---|---|---
B-4 | R-L | $\textbf{RG}_{\text{E}}$ | $\textbf{RG}_{\text{ER}}$ | $\textbf{RG}_{\overline{\text{ER}}}$
IU X-ray | R2Gen | $\bm{0.120}$ | $0.298$ | $-$ | $-$ | $-$
$\mathcal{M}^{2}$Tr | $0.121$ | $0.288$ | $-$ | $-$ | $-$
$\mathcal{T}_{\text{NLL}}$ | $0.114$ | $-$ | $0.230$ | $0.202$ | $0.153$
ICon | $0.098$ | $\bm{0.320}$ | $\bm{0.342}$ | $\bm{0.312}$ | $\bm{0.246}$
MIMIC -CXR | $\mathcal{T}_{\text{NLL}}$ | $0.105$ | $0.253$ | $0.230$ | $0.202$ | $0.153$
ORGan | $0.123$ | $\bm{0.293}$ | $0.303$ | $0.275$ | $0.199$
Recap | $0.125$ | $0.288$ | $0.307$ | $0.276$ | $\bm{0.205}$
ICon | $\bm{0.126}$ | $0.287$ | $\bm{0.312}$ | $\bm{0.278}$ | $0.197$
|  |  |  |  |  |

Table 2: Radgraph evaluation results on the IU X-ray and MIMIC-CXR datasets.
Results of $\mathcal{T}_{\text{NLL}}$ are cited from Delbrouck et al. (2022).

###  4.2 Evaluation Metrics and Baselines

NLG Metrics. To assess the quality of generated reports, we adopt several
natural language generation (NLG) metrics for evaluation. BLEU Papineni et al.
(2002), METEOR Banerjee and Lavie (2005), and ROUGE Lin (2004) are selected as
NLG Metrics, and we use the MS-COCO caption evaluation
tool666https://github.com/tylin/coco-caption to compute the results.

CE Metrics. Following previous research Chen et al. (2020, 2021), we adopt
macro-weighted clinical efficacy (CE) metrics to evaluate the observation-
level factual accuracy, and CheXbert Smit et al. (2020) is used in this paper.
To measure the entity-level factual accuracy, we leverage the RadGraph Jain et
al. (2021); Delbrouck et al. (2022) and temporal entity matching (TEM) scores
for evaluation.

Consistency Metrics. Con and R-Con (mentioned in §2.3) are utilized to measure
the inter-report consistency.

Baselines. We compare our models with the following state-of-the-art
baselines: R2Gen Chen et al. (2020), R2GenCMN Chen et al. (2021), KnowMat Yang
et al. (2021), $\mathcal{M}^{2}$Tr Nooralahzadeh et al. (2021), CMM-RL Qin and
Song (2022), CMCA Song et al. (2022), CXR-RePaiR-Sel/2 Endo et al. (2021),
BioViL-T Bannur et al. (2023), DCL Li et al. (2023b), METrans Wang et al.
(2023c), KiUT Huang et al. (2023), RGRG Tanida et al. (2023), ORGan Hou et al.
(2023b), and Recap Hou et al. (2023a).

Model | MIMIC-ABN | MIMIC-CXR
---|---|---
Con | R-Con | Con | R-Con
Majority | $100.0$ | $-$ | $100.0$ | $-$
R2Gen | $16.89$ | $15.60$ | $8.40$ | $9.52$
R2GenCMN | $16.10$ | $17.12$ | $9.22$ | $11.17$
ORGan | ${20.15}$ | $26.80$ | ${18.65}$ | $25.09$
Recap | $17.02$ | $21.42$ | $16.81$ | $23.80$
ICon (Ours) | $17.48$ | $\bm{31.15}$ | ${16.72}$ | $\bm{28.36}$
ICon w/o Zoom | $11.38$ | $18.93$ | $10.66$ | $15.65$
ICon w/o Inspect | $12.82$ | $18.83$ | $13.21$ | $19.59$
ICon w/o Mix-up | $15.72$ | $23.94$ | $15.88$ | $26.93$
|  |  |  |

Table 3: The Con score (in %) and the R-Con score (in ‰). Majority denotes
only one generated report is used for evaluation.

###  4.3 Implementation Details

The small and tiny version of Swin Transformer V2 Liu et al. (2022) are used
as the visual backbone for Zoomer and Inspector, respectively. The Generator
is initialized with the base version of BART pretrained on biomedical corpus
Yuan et al. (2022). Other parameters are randomly initialized. For Stage 2
training, the learning rate is $5e-5$ with linear decay, the batch size is
$32$, and the models are trained for $20$ and $5$ epochs on MIMIC-ABN and
MIMIC-CXR with early stopping, respectively. Since the number of samples in IU
X-ray is too small to train a multimodal model, we only provide results
produced by models trained on MIMIC-CXR as a reference, similar to Delbrouck
et al. (2022). For other training details (e.g., training Zoomer), and the
resources used in this paper, we list them in Appendix A.2.

Dataset | Model | NLG Metrics | CE Metrics
---|---|---|---
B-1 | B-2 | B-3 | B-4 | MTR | R-L | P | R | F1
MIMIC -ABN | ICon | $0.337$ | $0.195$ | $0.126$ | $0.086$ | $0.129$ | $0.236$ | $0.332$ | $0.430$ | $0.360$
ICon w/o Zoom | $0.310$ | $0.181$ | $0.119$ | $0.084$ | $0.120$ | $0.243$ | $0.306$ | $0.353$ | $0.306$
ICon w/o Inspect | $0.315$ | $0.182$ | $0.117$ | $0.081$ | $0.121$ | $0.236$ | $0.338$ | $0.401$ | $0.352$
ICon w/o Mix-up | $0.335$ | $0.192$ | $0.124$ | $0.085$ | $0.129$ | $0.239$ | $0.332$ | $0.413$ | $0.356$
MIMIC -CXR | ICon | ${0.429}$ | ${0.266}$ | ${0.178}$ | ${0.126}$ | ${0.170}$ | $0.287$ | ${0.445}$ | ${0.505}$ | ${0.464}$
ICon w/o Zoom | $0.377$ | $0.237$ | $0.162$ | $0.119$ | $0.149$ | $0.288$ | $0.363$ | $0.280$ | $0.278$
ICon w/o Inspect | $0.399$ | $0.248$ | $0.168$ | $0.122$ | $0.157$ | $0.287$ | $0.444$ | $0.447$ | $0.423$
ICon w/o Mix-up | $0.427$ | $0.264$ | $0.176$ | $0.124$ | $0.169$ | $0.285$ | $0.444$ | $0.502$ | $0.462$
|  |  |  |  |  |  |  |  |  |

Table 4: Ablation results of our model and its variants on the MIMIC-ABN and
MIMIC-CXR datasets. ICon w/o Zoom (§3.2) is the standard encoder-decoder
model, w/o Inspect stands for without Inspector (§3.4), and w/o Mix-up stands
for without lesion-aware Mix-up augmentation (§3.4).

Model | B-4 | R-L | CE-F1 | TEM
---|---|---|---|---
CXR-RePaiR-2 | $0.021$ | $0.143$ | $0.281$ | $0.125$
BioViL-NN | $0.037$ | $0.200$ | $0.283$ | $0.111$
BioViL-T-NN | $0.045$ | $0.205$ | $0.290$ | $0.130$
BioViL-AR | $0.075$ | $0.279$ | $0.293$ | $0.138$
BioViL-T-AR | $0.092$ | $\bm{0.296}$ | $0.317$ | $0.175$
Recap | $0.118$ | $0.279$ | $0.400$ | $0.304$
ICon (Ours) | $\bm{0.120}$ | $0.279$ | $\bm{0.468}$ | $\bm{0.335}$
|  |  |  |

Table 5: Progression modeling results on the MIMIC-CXR dataset. The results of
BioViL-* models are cited from Bannur et al. (2023).

##  5 Results

###  5.1 Quantitative Analysis

NLG Results. The NLG results are presented in the left section of Table 1. Our
proposed method, ICon, achieves SOTA performance, demonstrating significant
improvements on the MIMIC-ABN dataset. Regarding the MIMIC-CXR dataset, our
model achieves BLEU-3 and METEOR scores of $0.178$ and $0.170$ respectively,
surpassing previous top-performing models, while maintaining strong
performance on other metrics. Additionally, we provide experimental results on
the IU X-ray dataset as a reference in Table 2.

Inter-Report Consistency Analysis. Table 3 provides Con and R-Con scores,
where our models achieve the highest R-Con for both datasets. In terms of the
Con score, ICon demonstrates competitive performance when compared with ORGan.
However, it is important to note that the Majority baseline achieves perfect
(100%) consistency when generating only one report. Hence, the Con score
primarily serves as a reference.

---

Reference A: in comparison with the study of there is little overall change.
continued enlargement of the cardiac silhouette with pulmonary vascular
congestion and bilateral pleural effusions with compressive atelectasis.
central catheter remains in place. ICon w/o Zoom: as compared to the previous
radiograph the patient has received a right-sided picc line. the course of the
line is unremarkable the tip of the line projects over the mid svc. there is
no evidence of complications notably no pneumothorax. otherwise the radiograph
is unchanged. ICon: in comparison with the study of there is little overall
change. continued enlargement of the cardiac silhouette with pulmonary
vascular congestion and mild to moderate cardiomegaly. small bilateral pleural
effusions with areas of compressive atelectasis. the right picc line has been
removed. nasogastric tube remains in place.

---

Reference B: in comparison with the study of there is again enlargement of the
cardiac silhouette with extensive bilateral pleural effusions and compressive
atelectasis combined with pulmonary vascular congestion. ICon w/o Zoom: as
compared to the previous radiograph there is no relevant change. moderate
cardiomegaly with bilateral pleural effusions and subsequent areas of
atelectasis. moderate pulmonary edema. no newly appeared focal parenchymal
opacity suggesting pneumonia. ICon: as compared to the previous radiograph
there is unchanged evidence of moderate cardiomegaly with pulmonary vascular
congestion and moderate pulmonary edema. unchanged moderate bilateral pleural
effusions with areas of compressive atelectasis at the lung bases. no new
parenchymal opacities. no pneumothorax.

Figure 4: A case study of ICon on two semantically equivalent cases (i.e.,
Case A and Case B), given their radiographs and lesions. Spans with the same
color (Cardiomegaly, Pleural Effusion, Atelectasis, and Others) represent the
same positive observation. Consistent outputs are highlighted with underline.

CE (CheXbert and RadGraph) Results. In the right section of Table 1, we
observe that our proposed model, ICon, achieves SOTA clinical accuracy,
increasing CE $\text{F}_{1}$ from $0.393$ to $0.464$ on the MIMIC-CXR dataset
and rising it by $5.5\%$ on the MIMIC-ABN dataset. These results indicate that
our model is capable of generating accurate and consistent radiology reports.
Furthermore, Table 2 presents the RadGraph F1 on both the IU X-ray and MIMIC-
CXR dataset. Our model achieves competitive performance compared with the non-
RL-optimized baselines.

Temporal Modeling Results. Since ICon takes longitudinal information as
context, we present the results of our temporal modeling analysis in Table 5.
Notably, ICon exhibits significant improvements over other baselines in terms
of BLEU score, clinical accuracy, and TEM score while maintaining competitive
performance on the ROUGE score, indicating its enhanced capacity to
effectively utilize historical records.

---
① Pneumonia/False Neg
② Lung Opacity/False POS

Reference: pa and lateral views of the chest. there are new opacities in the
superior segment of the left lower lobe and in the right lower lobe most
consistent with ① multifocal pneumonia. no pleural effusion or pneumothorax.
cardiomediastinal and hilar contours are normal. ICon: ……the heart size
remains unchanged and is within normal limits. …the pulmonary vasculature is
not congested. no signs of acute or chronic parenchymal infiltrates are
present and the lateral and posterior pleural sinuses are free. no
pneumothorax in the apical area on frontal view. when comparison is made with
the next preceding study there is a new area of ② increased opacity in the
left.

Figure 5: An error case, where AP and Lateral views are available, produced by
ICon. The span and the span denote false negative observation and false
positive observation, respectively.

Ablation Results. The ablation results for MIMIC-ABN and MIMIC-CXR are listed
in Table 3 and Table 4. The performance of the ablated model w/o Zoom drops
significantly for both datasets, while the variant w/o Inspect achieves
competitive results on clinical accuracy. This suggests that the Zoomer
effectively extracts lesions and provides relevant abnormal information for
report generation. In addition, the variant w/o Mix-up further improves the
performance, which demonstrates the effectiveness of Inspector in transforming
concise lesion information into precise free-text reports. Moreover,
introducing lesion-aware mix-up augmentation strengthens the consistency of
generated outputs, indicating the effectiveness of ICon.

###  5.2 Qualitative Analysis

Case Study. Figure 4 showcases two semantically equivalent cases, i.e., Case A
and Case B, extracted from the test set of MIMIC-CXR. In both instances, ICon
successfully identifies abnormal observations (e.g., Cardiomegaly, Pleural
Effusion, and Atelectasis) and generates consistent phrases including
"pulmonary vascular congestion", "bilateral pleural effusions", and
"compressive atelectasis." Conversely, variant w/o Zoom fails to produce these
descriptions given Case A. This demonstrates that Zoomer plays a crucial role
in identifying lesions and also highlighting the ability of the mix-up
augmentation to ensure the alignment of lesions with their corresponding
attributes.

Error Analysis. Figure 5 presents an error case produced by ICon. Although
Zoomer successfully identifies Pneumonia in the given radiographs, Generator
fails to realize it into descriptions "multifocal pneumonia" (i.e., false
negative observation). We notice that the lesion of this observation is
inaccurately identified. Additionally, Zoomer outputs a false positive
observation Lung Opacity, leading to an inaccurate phrase "increased opacity".
In light of this, a better Zoomer trained with larger datasets could be a way
to mitigate it.

##  6 Related Works

Radiology report generation Jing et al. (2018); Li et al. (2018) has gained
much attention recently. Prior research either devised various memory
mechanisms to record key information Chen et al. (2020, 2021); Qin and Song
(2022); Wang et al. (2023c); Zhao et al. (2023) or proposed different learning
methods to enhance the performance Liu et al. (2021c, a, b). In addition, Yang
et al. (2021); Li et al. (2023b); Huang et al. (2023); Yan et al. (2023)
proposed to utilize knowledge graphs for report generation. Liu et al. (2019);
Lovelace and Mortazavi (2020); Miura et al. (2021); Nishino et al. (2022);
Delbrouck et al. (2022) designed various rewards for reinforcement learning to
improve clinical accuracy. Tanida et al. (2023) proposed an explainable
framework for report generation. Hou et al. (2023b) proposed to introduce
observations to improve factual accuracy. Kale et al. (2023) proposed a
template-based approach to improve the quality and accuracy of radiology
reports. Additionally, Ramesh et al. (2022); Bannur et al. (2023); Hou et al.
(2023a); Dalla Serra et al. (2023) focus on exploring the temporal structure.
Wang et al. (2023b, a) utilize CLIP Radford et al. (2021) to bridge the
modality gap. Mix-up augmentation is also closely related to this research
Zhang et al. (2018), and this method has been adopted in various NLP research
Sun et al. (2020); Yoon et al. (2021); Yang et al. (2022).

##  7 Conclusion and Future Works

In this paper, we propose ICon comprising three components to improve the
inter-report consistency between semantically equivalent lesions. ICon first
extracts lesions and then matches fine-grained attributes for report
generation. A lesion-aware mix-up augmentation method is devised for attribute
alignment. Experimental results on three datasets demonstrate the
effectiveness of ICon. In the future, we plan to explore incorporating large
language models (LLMs) into our framework, given their advanced capabilities
in planning and generation, to further enhance the performance of the task of
radiology report generation. Leveraging the strengths of LLMs could provide
more refined signals to enhance the performance of Zoomer.

## Limitations

Although ICon can improve the consistency of radiology report generation, it
still exhibits some limitations. Since our lesion extraction method is based
on coarse-grained labels (i.e., image labels), training such a model requires
annotations for images. However, annotations could be difficult to obtain in
some medical settings. Recent advances in foundation vision models Kirillov et
al. (2023) and open-set learning Zara et al. (2023) could be a potential
direction to handle this issue. In addition, since our framework consists of
two stages, prediction errors will propagate through the pipeline, and as a
result, the final performance of our framework largely depends on Stage 1.
Reinforcement learning Nishino et al. (2022) that takes factual improvement as
a reward could be a solution to optimize the framework in an end-to-end
manner.

## Ethics Statement

The IU X-ray Demner-Fushman et al. (2016), MIMIC-ABN Ni et al. (2020), and
MIMIC-CXR Johnson et al. (2019) are publicly available datasets and have been
automatically de-identified to protect patient privacy. Our goal is to enhance
the inter-report consistency of radiology report generation systems. Despite
the substantial improvement of our framework over SOTA baselines, the
performance still lags behind the requirements of real-world deployment and
could lead to unexpected failures in an untested environment. Thus, we urge
the readers of this paper and the potential users of this system to cautiously
check the generated outputs and should inquire about suggestions of experts
when using it.

## References

  * Asan et al. (2020) Onur Asan, Alparslan Emrah Bayrak, and Avishek Choudhury. 2020.  Artificial intelligence and human trust in healthcare: Focus on clinicians.  _J Med Internet Res_ , 22(6):e15154.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Bannur et al. (2023) Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle, and Ozan Oktay. 2023.  Learning to exploit temporal structure for biomedical vision-language processing.
  * Chen et al. (2021) Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. 2021.  Cross-modal memory networks for radiology report generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 5904–5914. Association for Computational Linguistics.
  * Chen et al. (2020) Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. 2020.  Generating radiology reports via memory-driven transformer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Dalla Serra et al. (2023) Francesco Dalla Serra, Chaoyang Wang, Fani Deligianni, Jeff Dalton, and Alison O’Neil. 2023.  Controllable chest X-ray report generation from longitudinal representations.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 4891–4904, Singapore. Association for Computational Linguistics.
  * Delbrouck et al. (2022) Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, and Curtis Langlotz. 2022.  Improving the factual correctness of radiology report generation with semantic rewards.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4348–4360, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Demner-Fushman et al. (2016) Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2016.  Preparing a collection of radiology examinations for distribution and retrieval.  _Journal of the American Medical Informatics Association_ , 23(2):304–310.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021.  Measuring and improving consistency in pretrained language models.  _Transactions of the Association for Computational Linguistics_ , 9:1012–1031.
  * Endo et al. (2021) Mark Endo, Rayan Krishnan, Viswesh Krishna, Andrew Y. Ng, and Pranav Rajpurkar. 2021.  Retrieval-based chest x-ray report generation using a pre-trained contrastive language-image model.  In _Proceedings of Machine Learning for Health_ , volume 158 of _Proceedings of Machine Learning Research_ , pages 209–219. PMLR.
  * Hou et al. (2023a) Wenjun Hou, Yi Cheng, Kaishuai Xu, Wenjie Li, and Jiang Liu. 2023a.  Recap: Towards precise radiology report generation via dynamic disease progression reasoning.
  * Hou et al. (2023b) Wenjun Hou, Kaishuai Xu, Yi Cheng, Wenjie Li, and Jiang Liu. 2023b.  ORGAN: Observation-guided radiology report generation via tree reasoning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8108–8122, Toronto, Canada. Association for Computational Linguistics.
  * Huang et al. (2023) Zhongzhen Huang, Xiaofan Zhang, and Shaoting Zhang. 2023.  Kiut: Knowledge-injected u-transformer for radiology report generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 19809–19818.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Jaccard (1901) Paul Jaccard. 1901.  Distribution de la flore alpine dans le bassin des Dranses et dans quelques régions voisines.  _Bulletin de la Société Vaudoise des Sciences Naturelles_ , 37:241 – 272.
  * Jain et al. (2021) Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Q. H. Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P. Lungren, Andrew Y. Ng, Curtis P. Langlotz, and Pranav Rajpurkar. 2021.  Radgraph: Extracting clinical entities and relations from radiology reports.  _CoRR_ , abs/2106.14463.
  * Jing et al. (2018) Baoyu Jing, Pengtao Xie, and Eric P. Xing. 2018.  On the automatic generation of medical imaging reports.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , pages 2577–2586. Association for Computational Linguistics.
  * Johnson et al. (2019) Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. 2019.  Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.  _arXiv preprint arXiv:1901.07042_.
  * Kale et al. (2023) Kaveri Kale, Pushpak Bhattacharyya, and Kshitij Jadhav. 2023.  Replace and report: NLP assisted radiology report generation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 10731–10742, Toronto, Canada. Association for Computational Linguistics.
  * Kaviani et al. (2022) Sara Kaviani, Ki Jin Han, and Insoo Sohn. 2022.  Adversarial attacks and defenses on ai in medical imaging informatics: A survey.  _Expert Systems with Applications_ , 198:116815.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. 2023.  Segment anything.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 4015–4026.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
  * Li et al. (2021) Mingjie Li, Wenjia Cai, Rui Liu, Yuetian Weng, Xiaoyun Zhao, Cong Wang, Xin Chen, Zhong Liu, Caineng Pan, Mengke Li, yingfeng zheng, Yizhi Liu, Flora D. Salim, Karin Verspoor, Xiaodan Liang, and Xiaojun Chang. 2021.  FFA-IR: Towards an explainable and reliable medical report generation benchmark .  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Li et al. (2023b) Mingjie Li, Bingqian Lin, Zicong Chen, Haokun Lin, Xiaodan Liang, and Xiaojun Chang. 2023b.  Dynamic graph enhanced contrastive learning for chest x-ray report generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3334–3343.
  * Li et al. (2018) Yuan Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. 2018.  Hybrid retrieval-generation reinforced agent for medical image report generation.  In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 1537–1547.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2021a) Fenglin Liu, Shen Ge, and Xian Wu. 2021a.  Competence-based multimodal curriculum learning for medical report generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 3001–3012. Association for Computational Linguistics.
  * Liu et al. (2021b) Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. 2021b.  Exploring and distilling posterior and prior knowledge for radiology report generation.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_ , pages 13753–13762. Computer Vision Foundation / IEEE.
  * Liu et al. (2021c) Fenglin Liu, Changchang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun. 2021c.  Contrastive attention for automatic chest x-ray report generation.  In _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_ , volume ACL/IJCNLP 2021 of _Findings of ACL_ , pages 269–280. Association for Computational Linguistics.
  * Liu et al. (2019) Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew B. A. McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019.  Clinically accurate chest x-ray report generation.  _CoRR_ , abs/1904.02633.
  * Liu et al. (2022) Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. 2022.  Swin transformer v2: Scaling up capacity and resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12009–12019.
  * Liu et al. (2021d) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021d.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 10012–10022.
  * Lovelace and Mortazavi (2020) Justin Lovelace and Bobak Mortazavi. 2020.  Learning to generate clinically coherent chest X-ray reports.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1235–1243, Online. Association for Computational Linguistics.
  * Ma et al. (2021) Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. 2021.  Understanding adversarial attacks on deep learning based medical image analysis systems.  _Pattern Recognition_ , 110:107332.
  * Miura et al. (2021) Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, and Dan Jurafsky. 2021.  Improving factual completeness and consistency of image-to-text radiology report generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5288–5304, Online. Association for Computational Linguistics.
  * Ni et al. (2020) Jianmo Ni, Chun-Nan Hsu, Amilcare Gentili, and Julian McAuley. 2020.  Learning visual-semantic embeddings for reporting abnormal findings on chest X-rays.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1954–1960, Online. Association for Computational Linguistics.
  * Nishino et al. (2022) Toru Nishino, Yasuhide Miura, Tomoki Taniguchi, Tomoko Ohkuma, Yuki Suzuki, Shoji Kido, and Noriyuki Tomiyama. 2022.  Factual accuracy is not enough: Planning consistent description order for radiology report generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , Online. Association for Computational Linguistics.
  * Nooralahzadeh et al. (2021) Farhad Nooralahzadeh, Nicolas Perez Gonzalez, Thomas Frauenfelder, Koji Fujimoto, and Michael Krauthammer. 2021.  Progressive transformer-based generation of radiology reports.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2824–2832, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Qayyum et al. (2020) Adnan Qayyum, Junaid Qadir, Muhammad Bilal, and Ala Al-Fuqaha. 2020.  Secure and robust machine learning for healthcare: A survey.  _IEEE Reviews in Biomedical Engineering_ , 14:156–180.
  * Qin and Song (2022) Han Qin and Yan Song. 2022.  Reinforced cross-modal alignment for radiology report generation.  In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 448–458. Association for Computational Linguistics.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.
  * Ramesh et al. (2022) Vignav Ramesh, Nathan Andrew Chi, and Pranav Rajpurkar. 2022.  Improving radiology report generation systems by removing hallucinated references to non-existent priors.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: towards real-time object detection with region proposal networks.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ , NIPS’15, page 91–99, Cambridge, MA, USA. MIT Press.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389.
  * Shin et al. (2016) Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner-Fushman, Jianhua Yao, and Ronald M Summers. 2016.  Learning to read chest x-rays: Recurrent neural cascade model for automated image annotation.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2497–2506.
  * Smit et al. (2020) Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Ng, and Matthew Lungren. 2020.  Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1500–1519, Online. Association for Computational Linguistics.
  * Song et al. (2022) Xiao Song, Xiaodan Zhang, Junzhong Ji, Ying Liu, and Pengxu Wei. 2022.  Cross-modal contrastive attention model for medical report generation.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 2388–2397, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Sun et al. (2020) Lichao Sun, Congying Xia, Wenpeng Yin, Tingting Liang, Philip Yu, and Lifang He. 2020.  Mixup-transformer: Dynamic data augmentation for NLP tasks.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3436–3440, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Tanida et al. (2023) Tim Tanida, Philip Müller, Georgios Kaissis, and Daniel Rueckert. 2023.  Interactive and explainable region-guided radiology report generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7433–7442.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
  * Wang et al. (2023a) Siyuan Wang, Zheng Liu, and Bo Peng. 2023a.  A self-training framework for automated medical report generation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 16443–16449, Singapore. Association for Computational Linguistics.
  * Wang et al. (2023b) Siyuan Wang, Bo Peng, Yichao Liu, and Qi Peng. 2023b.  Fine-grained medical vision-language representation learning for radiology report generation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 15949–15956, Singapore. Association for Computational Linguistics.
  * Wang et al. (2023c) Zhanyu Wang, Lingqiao Liu, Lei Wang, and Luping Zhou. 2023c.  Metransformer: Radiology report generation by transformer with multiple learnable expert tokens.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11558–11567.
  * Wikipedia (2024) Wikipedia. 2024.  Jaccard index — Wikipedia, the free encyclopedia.  http://en.wikipedia.org/w/index.php?title=Jaccard%20index&oldid=1196092673.  [Online; accessed 12-February-2024].
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yan et al. (2023) Benjamin Yan, Ruochen Liu, David Kuo, Subathra Adithan, Eduardo Reis, Stephen Kwak, Vasantha Venugopal, Chloe O’Connell, Agustina Saenz, Pranav Rajpurkar, and Michael Moor. 2023.  Style-aware radiology report generation with RadGraph and few-shot prompting.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 14676–14688, Singapore. Association for Computational Linguistics.
  * Yang et al. (2022) Huiyun Yang, Huadong Chen, Hao Zhou, and Lei Li. 2022.  Enhancing cross-lingual transfer by manifold mixup.  In _International Conference on Learning Representations_.
  * Yang et al. (2021) Shuxin Yang, Xian Wu, Shen Ge, Shaohua Kevin Zhou, and Li Xiao. 2021.  Knowledge matters: Radiology report generation with general and specific knowledge.  _CoRR_ , abs/2112.15009.
  * Yoon et al. (2021) Soyoung Yoon, Gyuwan Kim, and Kyumin Park. 2021.  SSMix: Saliency-based span mixup for text classification.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3225–3234, Online. Association for Computational Linguistics.
  * Yuan et al. (2022) Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. 2022.  BioBART: Pretraining and evaluation of a biomedical generative language model.  In _Proceedings of the 21st Workshop on Biomedical Language Processing_ , pages 97–109, Dublin, Ireland. Association for Computational Linguistics.
  * Zara et al. (2023) Giacomo Zara, Subhankar Roy, Paolo Rota, and Elisa Ricci. 2023.  Autolabel: Clip-based framework for open-set video domain adaptation.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_.
  * Zhang et al. (2017) Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason McGough, and Lin Yang. 2017.  Mdnet: A semantically and visually interpretable medical image diagnosis network.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3549–3557.
  * Zhao et al. (2023) Guosheng Zhao, Yan Yan, and Zijian Zhao. 2023.  Normal-abnormal decoupling memory for medical report generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 1962–1977, Singapore. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Abnormal Observation Statistics

The abnormal observation statistics of MIMIC-ABN, MIMIC-CXR, and IU X-ray are
listed in Table 6.

#Observation | MIMIC-ABN | MIMIC-CXR | IU X-ray
---|---|---|---
No Finding | 5002/32/22 | 64,677/514/229 | 744/108/318
Cardiomegaly | 16,312/118/244 | 70,561/514/1,602 | 244/38/61
Pleural Effusion | 10,502/80/186 | 56,972/477/1,379 | 60/13/15
Pneumothorax | 1,452/24/4 | 8,707/62/106 | 9/2/5
Enlarged Card. | 5,202/40/90 | 49,806/413/1,140 | 159/29/28
Consolidation | 4,104/36/96 | 14,449/119/384 | 17/1/3
Lung Opacity | 22,598/166/356 | 67,714/497/1,448 | 295/35/57
Fracture | 4,458/32/76 | 11,070/59/232 | 84/6/15
Lung Lesion | 5,612/54/112 | 11,717/123/300 | 85/14/17
Edema | 8,704/76/168 | 33,034/257/899 | 28/2/7
Atelectasis | 19,132/134/220 | 68,273/515/1,210 | 143/15/37
Support Devices | 9,886/58/196 | 60,455/450/1,358 | 89/20/16
Pneumonia | 17,826/138/260 | 23,945/184/503 | 20/2/1
Pleural Other | 2,850/30/62 | 7,296/70/184 | 32/4/7
|  |  |

Table 6: Observation distribution in train/valid/test split of three datasets.
Enlarged Card. refers to Enlarged Cardiomediastinum.

###  A.2 Implementation Details and Related Pretrained Models

For Stage 1, all three datasets use the same hyper-parameters for training
Zoomer, with a learning rate of $1e-4$, batch size of $128$, and dropout rate
of $0.1$, and the number of training epochs is adjusted accordingly. We train
Zoomer for 5, 10, and 15 epochs on MIMIC-ABN, MIMIC-CXR, and IU X-ray,
respectively. During training, several data augmentation methods are applied.
The input resolution of Swin Transformer is $256\times 256$, and we first
resize an image to $288\times 288$, and then randomly crop it to $256\times
256$ with random horizontal flip. All experiments are conducted using one
NVIDIA-3090 GTX GPU. For Stage 2, no data augmentation is applied, and we
conduct experiments on MIMIC-ABN and IU X-ray using two NVIDIA-3090 GTX GPUs,
and on MIMIC-CXR using four NVIDIA-V100 GPUs, both with half precision. Our
model has 328.38M trainable parameters, and the implementations are based on
the HuggingFace’s Transformers Wolf et al. (2020). Here are the pretrained
models we used:

  * •

Small version of Swin Transformer V2:
https://huggingface.co/microsoft/swinv2-small-patch4-window8-256

  * •

Tiny version of Swin Transformer V2:
https://huggingface.co/microsoft/swinv2-tiny-patch4-window8-256

  * •

Base Version of Biomedical BART:
https://huggingface.co/GanjinZero/biobart-v2-base

###  A.3 Detailed CE Results of ICon on the MIMIC-CXR Dataset

Observation | P | R | F1
---|---|---|---
Enlarged Card. | $0.442$ | $0.525$ | $0.428$
Cardiomegaly | $0.630$ | $0.822$ | $0.714$
Lung Opacity | $0.542$ | $0.563$ | $0.552$
Lung Lesion | $0.321$ | $0.177$ | $0.228$
Edema | $0.464$ | $0.784$ | $0.583$
Consolidation | $0.275$ | $0.162$ | $0.204$
Pneumonia | $0.341$ | $0.350$ | $0.345$
Atelectasis | $0.539$ | $0.620$ | $0.577$
Pneumothorax | $0.400$ | $0.444$ | $0.421$
Pleural Effusion | $0.721$ | $0.827$ | $0.770$
Pleural Other | $0.295$ | $0.315$ | $0.304$
Fracture | $0.225$ | $0.164$ | $0.190$
Support Devices | $0.785$ | $0.784$ | $0.785$
No Finding | $0.263$ | $0.535$ | $0.352$
Macro Average | $0.445$ | $0.505$ | $0.464$
|  |  |
Table 7: Experimental results of each observation on the MIMIC-CXR dataset.

###  A.4 Experimental Results of Stage 1

The experimental results are provided in Table 8. Results on the IU X-ray
dataset are only provided for reference.

Dataset | P | R | F1
---|---|---|---
IU X-ray | $0.223$ | $0.243$ | $0.225$
MIMIC-ABN | $0.379$ | $0.472$ | $0.411$
MIMIC-CXR | $0.454$ | $0.550$ | $0.491$
|  |  |
Table 8: Abnormal observation prediction results of Zoomer at Stage 1.

###  A.5 Lesion Extraction

There are two steps in extraction lesions: candidate generation and candidate
classification. Given an image with a resolution of 1024$\times$1024, padding
if needed, we apply a sliding window of 384$\times$384, with a step size of
128 to extract candidates for classification. This operation results in $36$
regions. Then, each region is fed into the Zoomer for classification, and only
the top-$1$ lesion is selected for each observation. Note that before
extracting lesions, each input case is first assigned with their observations
by Zoomer, and as a result, the number of lesions corresponds to the number of
observations.

The No Finding observation is excluded for lesion extraction, as it estimates
the overall conditions of a patient, which makes it difficult to locate at
specific regions.

###  A.6 Other Preprocessing Details

We adopt the same preprocessing setup used in Chen et al. (2020), and the
minimum count of each token is set to 3/3/10 for IU X-ray/MIMIC-ABN/MIMIC-CXR,
respectively. Other tokens are replaced with a special token <unk>.

###  A.7 Justifications for Additional Data Processing

Justification for Using Historical Records. As stated in Hou et al. (2023a),
without historical information, it is unreasonable to generate reports with
comparisons between two consecutive visits and will lead to hallucinations
Ramesh et al. (2022). As a result, we include historical records as context
information for report generation.

Justification for Using All Views. Prior research Chen et al. (2020, 2021);
Hou et al. (2023b, a) treated different views of radiographs in one visit as
different samples. However, this is unreasonable to generate a report with
only one view position, since different diseases could be observed from
different view positions. For example, most of the devices can not be observed
from a Lateral view. Given a lateral view radiograph, writing a sentence of "A
right chest tube is in unchanged position." is not acceptable.

In addition, some reports describe how many views are provided at the
beginning, e.g., "PA and lateral views are provided." Above all, we have
justified reasons to use all the views in one visit of a patient to generate
the target report. Note that previous work treated each image as a sample and
their settings have more samples than ours. For a fair comparison, each
generated output of a study with $L$ images is duplicated $L$ times so that
the number of samples in evaluation is consistent with previous research.