11institutetext:  1School of ECE, Peking University, China
2Tencent Medical AI Lab, China 3Harbin Chest Hospital, China
11email: {diyou<EMAIL_ADDRESS>{shenge<EMAIL_ADDRESS>
11email: {xiaoxiaxie_hrb<EMAIL_ADDRESS>
# AlignTransformer: Hierarchical Alignment of
Visual Regions and Disease Tags for
Medical Report Generation

Di You1 Equal Contributions. Fenglin Liu1⋆ Shen Ge2 Xiaoxia Xie3 Jing Zhang3
Xian Wu2 Corresponding Author.

###### Abstract

Recently, medical report generation, which aims to automatically generate a
long and coherent descriptive paragraph of a given medical image, has received
growing research interests. Different from the general image captioning tasks,
medical report generation is more challenging for data-driven neural models.
This is mainly due to 1) the serious data bias: the normal visual regions
dominate the dataset over the abnormal visual regions, and 2) the very long
sequence. To alleviate above two problems, we propose an AlignTransformer
framework, which includes the Align Hierarchical Attention (AHA) and the
Multi-Grained Transformer (MGT) modules: 1) AHA module first predicts the
disease tags from the input image and then learns the multi-grained visual
features by hierarchically aligning the visual regions and disease tags. The
acquired disease-grounded visual features can better represent the abnormal
regions of the input image, which could alleviate data bias problem; 2) MGT
module effectively uses the multi-grained features and Transformer framework
to generate the long medical report. The experiments on the public IU-Xray and
MIMIC-CXR datasets show that the AlignTransformer can achieve results
competitive with state-of-the-art methods on the two datasets. Moreover, the
human evaluation conducted by professional radiologists further proves the
effectiveness of our approach.

###### Keywords:

Medical Report Generation Data Bias Transformer.

##  1 Introduction

Medical images, e.g., radiology and pathology images, as well as their
corresponding reports (see Fig. 1) are widely-used for diagnosis [9, 12]. In
clinical practice, writing a medical report can be time-consuming and tedious
for experienced radiologists, and error-prone for inexperienced radiologists
[5, 29, 28]. As a result, automatic medical report generation systems can
reduce the workload of radiologists by assisting them in clinical decision-
making, and are thus in urgent need [17, 22, 16, 21, 42, 44, 4, 38, 45, 7, 29,
28, 24].

Figure 1: An example of ground truth report and reports generated by a state-
of-the-art approach R2Gen [7] and our approach. The  Red bounding boxes and
Red colored text indicate the abnormalities in images and reports,
respectively. As we can see, in this example, the R2Gen fails to depict the
rare but important abnormalities and generates some error sentences ( Blue
colored text) and repeated sentences ( Purple colored text).

Most existing medical report generation models follow the standard image
captioning approaches [41, 40, 43, 1, 27, 26, 25, 32, 8] and employ the
encoder-decoder framework, e.g., a CNN-based image encoder followed by a LSTM-
based report decoder. However, directly applying image captioning approaches
to medical images has the following problems: 1) Data bias: the normal images
dominate the dataset over the abnormal ones [36]. Furthermore, for each
abnormal image, the normal regions dominate the image over the abnormal ones
[24]. As shown in Fig. 1, abnormal regions (Red bounding boxes) only occupy a
small part of the entire image. As a result, existing models tend to generate
plausible general reports with no prominent abnormal narratives, failing to
depict the rare but important abnormalities [16, 22, 44]. 2) Very long
sequence: Image captioning models are designed to generate one single and
short sentence, which only describes the most prominent visual contents, while
medical report generation requires to generate a long paragraph, including
multiple structural sentences with each one focusing on a specific medical
observation. For the widely-used LSTM [14] in image captioning, it is hard to
model such long sequences or paragraphs due to vanishing or exploding
gradients [34]. Recent medical report generation models [17, 16, 44] rely on
hierarchical LSTM [20], which have similar long sequence modeling problems. As
shown in Fig. 1, even a state-of-the-art model R2Gen [7] still generates some
repeated sentences of normalities.

To alleviate above problems, we propose the AlignTransformer, which includes
two major modules: 1) Align Hierarchical Attention (AHA) works as the encoder
to extract visual features. To focus on the abnormal regions, AHA first
predicts the disease tags from the input image and then aligns these tags with
the corresponding visual regions. To fit for both coarse- and fine-grained
visual features, AHA introduces a hierarchically alignment model [25]. In
detail, the disease tags are firstly used to find the most relevant visual
regions, extracting disease-grounded visual features for each disease tag.
Then the extracted visual features are further used to find the most relevant
disease tags and filter out the irrelevant disease tags for each visual
feature. By conducting the above process at different granularity level, the
visual receptive fields gradually concentrate on salient abnormal regions
under the guidance of the disease tags. In this way, we can obtain the
features of abnormal regions, which can be used to alleviate data bias; 2)
Multi-Grained Transformer (MGT) works as the decoder to generate final
reports. MGT adaptively exploit coarse- and fine-grained disease-grounded
visual features through a learned gating mechanism. In addition, MGT
introduces the Transformer [39, 8, 7, 28] to generate the reports. Benefiting
from the visual features from multiple granularities and the Transformer, as
shown in Fig. 1, our approach are a better fit for long report.

Overall, the contributions of this paper are as follows:

  * •

In this paper, we propose the AlignTransformer framework, which aims to
alleviate the data bias problem and model the very long sequence for medical
report generation.

  * •

The experiments on the benchmark IU-Xray [10] and MIMIC-CXR [18] datasets
prove the effectiveness of our approach, which achieves results competitive
with the existing state-of-the-art methods. We also verify the advantage of
AlignTransformer from the manual evaluation of radiologists.

##  2 Related Works

The related works are introduced from: 1) Image Captioning and Visual
Paragraph Generation and 2) Medical Report Generation.

Image Captioning and Visual Paragraph Generation Image captioning [6] aims to
understand the given images and generate corresponding descriptive sentences
[41, 40, 1, 27, 26, 25, 32, 8]. However, the sentence generated by
AlignTransformer is usually short and describes the most prominent visual
contents, which cannot fully convey the rich feature information of the image.
Recently, visual paragraph generation [20], which aims to generate long and
coherent reports or stories to describe visual contents, has attracted
increasing research interests. To conduct the task, the Hierarchical LSTM
structure (HLSTM) [20] is widely-used. However, in paragraph generation for
medical images, the correctness of generating abnormalities should be
emphasized more than other normalities, while in paragraphs of natural images
each sentence has equal importance. Besides, due to the data bias in the
medical domain, the widely-used HLSTM in the visual paragraph generation does
not perform very well in medical report generation and is tend to produce
normal reports [42, 22, 44, 16, 29, 28, 24].

Medical Report Generation Inspired by the success of deep learning models on
image captioning, a lot of encoder-decoder based frameworks have been proposed
[17, 16, 44, 42, 22, 45, 21, 30, 7, 29, 28, 24] to automatically generate a
fluent medical report. However, due to the data bias and the very long
sequence, some errors occur in the generated reports of the existing methods,
like duplicate reports, inexact descriptions, etc [42, 44, 17]. To this end,
[16, 22, 30] and [28, 45, 21] introduced the reinforcement learning and
medical knowledge graph, respectively. [29] introduced the contrastive
attention to compare the abnormal image with the normal images to better
capture the abnormalities. Meanwhile, [28, 7] further introduced the
Transformer [39] to model the very long sequence. In our work, different from
[28, 7], we conduct the alignments between visual regions and disease tags in
an iterative manner and propose a multi-grained transformer to combine
disease-grounded visual features at different depths.

##  3 Approach

We first formulate the problem; Next, we introduce our framework, which
includes the Align Hierarchical Attention and Multi-Grained Transformer.

###  3.1 Problem Formulation

Given a medical image encoded as visual features $V$ and disease tags $T$, the
goal of our framework is to generate a coherent report $R$ that describes the
observations in details of both normal and abnormal regions. In
implementations, for the visual features $V$, we follow [16, 21, 22, 29, 28,
24] to adopt the ResNet-50 [13] pre-trained on ImageNet [11] and fine-tuned on
CheXpert dataset [15] to extract the 2,048 $7\times 7$ feature maps, which are
further projected into 512 $7\times 7$ feature maps, denoted as
$V=\\{v_{1},v_{2},\dots,v_{N_{\text{V}}}\\}\in\mathbb{R}^{N_{\text{V}}\times
d}$ ($N_{\text{V}}=49,d=512$). Moreover, we follow [17] to further predict the
disease tags $T$ of the input image. Specifically, we further feed the
extracted visual features $V$ into a multi-label classification network, which
is pre-trained as multi-label classification task on the downstream datasets
to generate a distribution over all of the pre-defined tags. Finally, the
embeddings of the $N_{\text{T}}=10$ most likely tags
$T=\\{t_{1},t_{2},\dots,t_{N_{\text{T}}}\\}\in\mathbb{R}^{N_{\text{T}}\times
d}$ are used as disease tags of current input image. Based on the extracted
$V$ and $T$, our framework first adopt the AHA to align the visual regions $V$
and disease tags $T$ in a coarse-to-fine manner, acquiring multi-grained
disease-grounded visual features $\hat{V}$. Next, MGT is introduced to
generate final report based on the $\hat{V}$. In brief, our AlignTransformer
framework is formulated as:

$\footnotesize\text{AHA}:\\{V,T\\}\to\hat{V};\quad\text{MGT}:\hat{V}\to R.$
(1)

Through the above process, our framework gets the ability to alleviate the
data bias problem and model the very long sequence.

###  3.2 Align Hierarchical Attention

####  3.2.1 Basic Module

To align the visual regions $V$ and the disease tags $T$, we adopt the Multi-
Head Attention (MHA) from [39], which allows probabilistic many-to-many
relations instead of monotonic relations, as in [25, 41]. Thus, the MHA can
compute the association weights between different features. The MHA consists
of $n$ parallel heads and each head is defined as a scaled dot-product
attention:

$\displaystyle\text{Att}_{i}(X,Y)$
$\displaystyle=\text{softmax}\left(X\text{W}_{i}^{\text{Q}}(Y\text{W}_{i}^{\text{K}})^{T}\right)Y\text{W}_{i}^{\text{V}}$
(2) $\displaystyle\text{MHA}(X,Y)$
$\displaystyle=[\text{Att}_{1}(X,Y);\dots;\text{Att}_{n}(X,Y)]\text{W}^{\text{O}}$
(3)

where $X\in\mathbb{R}^{l_{x}\times d}$ and $Y\in\mathbb{R}^{l_{y}\times d}$
denote the Query matrix and the Key/Value matrix, respectively; the divisor
$\sqrt{{d}_{n}}$ (${d}_{n}=d/n$) in Eq. (2) is omitted in equations for
conciseness, please see [39] for details;
$\text{W}_{i}^{\text{Q}},\text{W}_{i}^{\text{K}},\text{W}_{i}^{\text{V}}\in\mathbb{R}^{d\times
d_{n}}$ and $\text{W}^{\text{O}}\in\mathbb{R}^{d\times d}$ are learnable
parameters. $[\cdot;\cdot]$ stands for concatenation operation.

Conventionally, a Fully-Connected Network (FCN) is followed by the MHA:

$\displaystyle\text{FCN}(X)=\max(0,X\text{W}^{\text{f}}+\text{b}^{\text{f}})\text{W}^{\text{ff}}+\text{b}^{\text{ff}}$
(4)

where $\max(0,*)$ represents the ReLU activation function;
$\text{W}^{\text{f}}\in\mathbb{R}^{d\times 4d}$ and
$\text{W}^{\text{ff}}\in\mathbb{R}^{4d\times d}$ are learnable matrices;
$\text{b}^{\text{f}}$ and $\text{b}^{\text{ff}}$ are the bias terms. It is
worth noting that both the MHA and the FCN are followed by an operation
sequence of dropout [37], skip connection [13] and layer normalization (Norm)
[2].

Figure 2: Illustration of our proposed AlignTransformer framework, which
consists of the Align Hierarchical Attention (AHA) and Multi-Grained
Transformer (MGT).

Align Attention Since the MHA can compute the association weights between
different features, if we apply the MHA between visual regions $V$ and the
disease tags $T$, we can learn the correlation and relationship between $V$
and $T$, resulting in the alignment between visual regions and disease tags.
Therefore, based on the MHA, as shown in Fig. 2, the align attention is
conducted as:

$\displaystyle V^{\prime}=\text{FCN}(\text{MHA}(T,V));\quad
T^{\prime}=\text{FCN}(\text{MHA}(V^{\prime},T))$ (5)

In implementation, the disease tags $T$ are first used as the Query to find
the most relevant visual regions in $V$, generating the disease related visual
features $V^{\prime}$. Then the $V^{\prime}$ are further used as the Query to
find the most relevant disease tags and filter out the irrelevant disease tags
in $T$. As a result, we can obtain alignment between visual regions and
disease tags via above processes. Finally, since $V^{\prime}$ and $T^{\prime}$
are aligned, we can add them up to get the disease-grounded visual features
$\hat{V}$ that exploits the best of their respective advantages:

$\displaystyle\hat{V}=\text{LayerNorm}(V^{\prime}+T^{\prime})$ (6)

where the LayerNorm denotes the layer normalization [2]. In particular, since
$T$ contains the disease tags, the disease-grounded visual features $\hat{V}$
can be referred as the features of abnormal regions, which are useful in
alleviating data bias.

Full Model To improve the performance and acquire the multi-grained disease-
grounded visual features, we propose to perform Align Attention
hierarchically, which we call Align Hierarchical Attention. Specifically, the
process in Eq. (5) that uses the original features is considered as the first
round:

$\displaystyle V_{1}=\text{FCN}(\text{MHA}(T_{0},V_{0}));\quad
T_{1}=\text{FCN}(\text{MHA}(V_{1},T_{0}))$ (7)

where $V_{0}=V$, $T_{0}=T$, $V_{1}$ and $T_{1}$ denote original visual
features, original disease tags, aligned visual features, and aligned disease
tags, respectively. By repeating the same process for $N$ times, we obtain the
final outputs of the two stacks:

$\displaystyle V_{N}=\text{FCN}(\text{MHA}(T_{N-1},V_{N-1}));$
$\displaystyle\quad T_{N}=\text{FCN}(\text{MHA}(V_{N},T_{N-1}))$ (8)
$\displaystyle\hat{V}_{N}=\text{Layer}$
$\displaystyle\text{Norm}(V_{N}+T_{N})$ (9)

In this way, we obtain the multi-grained disease-grounded visual features
$\\{\hat{V}_{i}\\}=\\{\hat{V}_{1},\hat{V}_{2},\dots,\hat{V}_{N}\\}$.

###  3.3 Multi-Grained Transformer

We propose the Multi-Grained Transformer (MGT), which includes $L=3$ decoder
layers, to exploit the $\\{\hat{V}_{i}\\}$ and generate a proper report $R$.
In implementations, for generating each word $y_{t}$, given the embedding of
current input word $x_{t}^{(0)}=w_{t}+e_{t}$ ($w_{t}$: word embedding, and
$e_{t}$: fixed position embedding). For the $l^{\text{th}}$ decoder layer, it
takes $x_{t}^{(l-1)}$ as input to obtain
$h_{t}^{(l)}=\text{MHA}(x_{t}^{(l-1)},x_{1:t}^{(l-1)})$. Then, inspired by
[8], we introduce the Adaptive Exploiting Attention (AEA) module to adaptively
exploit multi-grained disease-grounded visual features:

$\displaystyle\hat{h}_{t}^{(l)}=$
$\displaystyle\text{AEA}(h_{t}^{(l)},\\{\hat{V}_{i}\\})=\sum_{i=1}^{N}\lambda_{i}\odot\text{MHA}(h_{t}^{(l)},\hat{V}_{i})$
(10) $\displaystyle\lambda_{i}$
$\displaystyle=\sigma\left(\left[h_{t}^{(l)};\text{MHA}(h_{t}^{(l)},\hat{V}_{i})\right]\text{W}_{i}+b_{i}\right)$
(11)

where $\text{W}_{i}\in\mathbb{R}^{2d\times d}$ is a learnable parameter and
$b_{i}$ is a learnable bias vector. $\odot$, $\sigma$ and $[\cdot;\cdot]$
denote the element-wise multiplication, the sigmoid function and the
concatenation, respectively. The computed $\\{\lambda_{i}\\}$ weights the
expected importance of different grained disease-grounded visual features for
each target word. Then, we adopt the FCN to acquire the output of
$l^{\text{th}}$ decoder layer: $x_{t}^{(l)}=\text{FCN}(\hat{h}_{t}^{(l)})$ It
is worth noting that both the MHA, the AEA and the FCN are followed by an
operation sequence of dropout [37], skip connection [13] and memory-driven
conditional layer normalization (MCLN) [7].

Finally, the output of last decoder layer $x_{t}^{(L)}$ is used to predict the
next word: $y_{t}\sim
p_{t}=\text{softmax}(x_{t}^{(L)}\text{W}^{p}+\text{b}^{p})$, where the
$\text{W}^{p}$ and $\text{b}^{p}$ are the learnable parameters. Given the
ground truth report
$R^{*}=\\{y^{*}_{1},y^{*}_{2},\dots,y^{*}_{N_{\text{R}}}\\}$ provided by the
radiologists, we can train our AlignTransformer framework by minimizing the
widely-used cross-entropy loss:
$L_{\text{CE}}(\theta)=-\sum_{i=1}^{N_{\text{R}}}\log\left(p_{\theta}\left(y_{i}^{*}\mid
y_{1:i-1}^{*}\right)\right)$.

##  4 Experiments

####  4.0.1 Datasets, Metrics and Settings

In our work, we conduct experiments on two public datasets, i.e., a widely-
used benchmark IU-Xray [10] and a recently released large-scale MIMIC-CXR
[18]. IU-Xray is widely-used to evaluate the performance of medical report
generation methods. It contains 7,470 chest X-ray images associated with 3,955
reports. Following previous works [7, 16, 21, 22, 29, 28, 24], we randomly
split the dataset into 70%-10%-20% training-validation-testing splits. MIMIC-
CXR is the recently released largest dataset to date and consists of 377,110
chest X-ray images and 227,835 reports from 64,588 patients. Following [7, 29,
28, 24], we use the official splits to report our results, resulting in
368,960 in the training set, 2,991 in the validation set and 5,159 in the test
set.

To test the performance, we adopt the widely-used BLEU [33], METEOR [3] and
ROUGE-L [23], which are reported by the evaluation toolkit [6]. Since our
approach is based on the Transformer [39], we follow the original settings in
[39] and set $d=512,n=8$. Based on the average performance on the validation
set, both the $N$ and $L$ in our Align Hierarchical Attention and Multi-
Grained Transformer, respectively, are set to $3$. We use Adam optimizer [19]
with a batch size of 16 and a learning rate of 2e-4 within maximum 100 epochs
for parameter optimization. We also use momentum of 0.8 and weight decay of
0.999.

Table 1: Results of our AlignTransformer and existing models on the MIMIC-CXR and IU-Xray datasets. Hit represents the human evaluation results and is measured by the pick-up percentages (%). The differences between their sum and 100% represent the percentage of ties. Higher value denotes better performance in all columns.  Datasets | Methods | Year | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | Hit
---|---|---|---|---|---|---|---|---|---
MIMIC -CXR |  CNN-RNN [40] | 2015 | 0.299 | 0.184 | 0.121 | 0.084 | 0.124 | 0.263 | -
AdaAtt [31] | 2017 | 0.299 | 0.185 | 0.124 | 0.088 | 0.118 | 0.266 | -
Att2in [35] | 2017 | 0.325 | 0.203 | 0.136 | 0.096 | 0.134 | 0.276 | -
Up-Down [1] | 2018 | 0.317 | 0.195 | 0.130 | 0.092 | 0.128 | 0.267 | -
R2Gen [7] | 2020 | 0.353 | 0.218 | 0.145 | 0.103 | 0.142 | 0.277 | 29
PPKED [28] | 2021 | 0.360 | 0.224 | 0.149 | 0.106 | 0.149 | 0.284 | -
AlignTransformer | Ours | 0.378 | 0.235 | 0.156 | 0.112 | 0.158 | 0.283 | 54
IU-Xray |  HRGR-Agent [22] | 2018 | 0.438 | 0.298 | 0.208 | 0.151 | - | 0.322 | -
CMAS-RL [16] | 2019 | 0.464 | 0.301 | 0.210 | 0.154 | - | 0.362 | -
SentSAT [45] | 2019 | 0.445 | 0.289 | 0.200 | 0.143 | - | 0.359 | -
SentSAT+KG [45] | 2020 | 0.441 | 0.291 | 0.203 | 0.147 | - | 0.367 | -
R2Gen [7] | 2020 | 0.470 | 0.304 | 0.219 | 0.165 | 0.187 | 0.371 | 22
PPKED [28] | 2021 | 0.483 | 0.315 | 0.224 | 0.168 | 0.190 | 0.376 | -
AlignTransformer | Ours | 0.484 | 0.313 | 0.225 | 0.173 | 0.204 | 0.379 | 63

Automatic Evaluation Several representative models, including three most
recently published state-of-the-art models, i.e., PPKED [28], R2Gen [7] and
SentSAT + KG [45], are selected for comparison. The results on the test of two
datasets are shown in Table 1. As we can see, our AlignTransformer achieves
the competitive results with the previous state-of-the-art models on the two
datasets under all metrics, which verify the effectiveness of our approach.

Human Evaluation Moreover, we invite two professional radiologists to evaluate
the perceptual quality of 200 randomly selected reports generated by our
AlignTransformer and the R2Gen [7]. The radiologists are unaware of which
model generates these reports and are encouraged to select a more accurate
report for each pair. The results of last column in Table 1 show that our
approach is better than R2Gen in clinical practice with winning pick-up
percentages.

Overall, the results verify that our proposed framework can generate accurate
and proper reports, which could assist radiologists in making decisions.

Table 2: Ablation study of our method, which includes the AHA and the MGT. The $N$ denotes the number of alignments in AHA. The ‘Baseline’ and ‘w/ AHA+MGT’ denote the ResNet-50 image encoder [13] equipped with the Transformer report decoder in [7] and our proposed AlignTransformer, respectively.  Datasets | Methods | $N$ | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L
---|---|---|---|---|---|---|---|---
IU-Xray | Baseline | - | 0.442 | 0.285 | 0.195 | 0.138 | 0.176 | 0.357
w/ AHA | 1 | 0.467 | 0.301 | 0.210 | 0.159 | 0.185 | 0.368
w/ AHA | 2 | 0.473 | 0.305 | 0.217 | 0.163 | 0.188 | 0.372
w/ AHA | 3 | 0.476 | 0.308 | 0.219 | 0.164 | 0.192 | 0.374
w/ AHA | 4 | 0.471 | 0.303 | 0.215 | 0.160 | 0.187 | 0.370
w/ AHA+MGT | 3 | 0.484 | 0.313 | 0.225 | 0.173 | 0.204 | 0.379

##  5 Analysis

In this section, we conduct analysis on the benchmark IU-Xray dataset [10]
from different perspectives to better understand our proposed framework.

Ablation Study We investigate the contribution of each component in our
approach. Table 2 shows that our AHA can significantly promote the performance
over all metrics, with up to 18.8% gain in BLEU-4 ($N=3$). This indicates that
the disease-grounded visual features extracted by our AHA contain sufficient
accurate abnormal information, which is vital in alleviating the data
deviation problem and improving the performance of medical report generation.
For the MGT, Table 2 shows that the MGT can further boost the performance
under all metrics, which demonstrates the effectiveness of our MGT to
adaptively exploit the multi-grained disease-grounded visual features.

Visualization and Example In Fig. 1, we give the visualization of the AHA and
an intuitive example to better understand our approach. As we can see, for the
current state-of-the-art model R2Gen [7], it produces some inexact
descriptions and repeated sentences of normalities, which is due to the
overwhelming normal images in the dataset, i.e., data deviation [36, 16, 28,
24]. Since our AHA can efficiently capture the abnormal regions by aligning
visual regions and disease tags, we can generate more accurate report for the
abnormal regions. For example, the generated report correctly describes “The
lungs are hyperlucent”, “There is left lower lobe airspace disease suspicious
for pneumonia” and “There is a moderate left sided pleural effusion”.

##  6 Conclusions

In this paper, we propose an AlignTransformer to alleviate the data bias
problem and model the very long sequence for medical report generation. The
experiments prove the effectiveness of our method, which not only generates
meaningful medical reports with accurate abnormal descriptions and regions,
but also achieves competitive results with state-of-the-art models on two
public datasets with the best human preference. The results justify the
usefulness of our framework in assisting radiologists in clinical decision-
making, reducing their workload.

## Acknowledgments

We sincerely thank all the anonymous reviewers and chairs for their
constructive comments and suggestions that substantially improved this paper.
Xian Wu is the corresponding author of this paper.

## References

  * [1] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and VQA. In: CVPR (2018)
  * [2] Ba, L.J., Kiros, R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
  * [3] Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: IEEvaluation@ACL (2005)
  * [4] Biswal, S., Xiao, C., Glass, L., Westover, M.B., Sun, J.: Clinical report auto-completion. In: WWW (2020)
  * [5] Brady, A., Laoide, R.Ó., Mccarthy, P., Mcdermott, R.: Discrepancy and error in radiology: Concepts, causes and consequences. The Ulster Medical Journal 81, 3 – 9 (2012)
  * [6] Chen, X., Fang, H., Lin, T., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.L.: Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015)
  * [7] Chen, Z., Song, Y., Chang, T., Wan, X.: Generating radiology reports via memory-driven transformer. In: EMNLP (2020)
  * [8] Cornia, M., Stefanini, M., Baraldi, L., Cucchiara, R.: Meshed-memory transformer for image captioning. In: CVPR (2020)
  * [9] Delrue, L., Gosselin, R., Ilsen, B., Van Landeghem, A., de Mey, J., Duyck, P.: Difficulties in the interpretation of chest radiography. Comparative Interpretation of CT and Standard Radiography of the Chest pp. 27–49 (2011)
  * [10] Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S.K., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Medical Informatics Assoc. 23(2), 304–310 (2016)
  * [11] Deng, J., Dong, W., Socher, R., Li, L., Li, K., Li, F.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009)
  * [12] Goergen, S.K., Pool, F.J., Turner, T.J., Grimm, J.E., Appleyard, M.N., Crock, C., Fahey, M.C., Fay, M.F., Ferris, N.J., Liew, S.M., et al.: Evidence-based guideline for the written radiology report: Methods, recommendations and implementation challenges. Journal of medical imaging and radiation oncology 57(1), 1–7 (2013)
  * [13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
  * [14] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
  * [15] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R.L., Shpanskaya, K.S., Seekins, J., Mong, D.A., Halabi, S.S., Sandberg, J.K., Jones, R., Larson, D.B., Langlotz, C.P., Patel, B.N., Lungren, M.P., Ng, A.Y.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: AAAI (2019)
  * [16] Jing, B., Wang, Z., Xing, E.P.: Show, describe and conclude: On exploiting the structure information of chest x-ray reports. In: ACL (2019)
  * [17] Jing, B., Xie, P., Xing, E.P.: On the automatic generation of medical imaging reports. In: ACL (2018)
  * [18] Johnson, A.E.W., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C., Mark, R.G., Horng, S.: MIMIC-CXR: A large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 (2019)
  * [19] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2014)
  * [20] Krause, J., Johnson, J., Krishna, R., Fei-Fei, L.: A hierarchical approach for generating descriptive image paragraphs. In: CVPR (2017)
  * [21] Li, C.Y., Liang, X., Hu, Z., Xing, E.P.: Knowledge-driven encode, retrieve, paraphrase for medical image report generation. In: AAAI (2019)
  * [22] Li, Y., Liang, X., Hu, Z., Xing, E.P.: Hybrid retrieval-generation reinforced agent for medical image report generation. In: NeurIPS (2018)
  * [23] Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: ACL (2004)
  * [24] Liu, F., Ge, S., Wu, X.: Competence-based multimodal curriculum learning for medical report generation. In: ACL (2021)
  * [25] Liu, F., Liu, Y., Ren, X., He, X., Sun, X.: Aligning visual regions and textual concepts for semantic-grounded image representations. In: NeurIPS (2019)
  * [26] Liu, F., Ren, X., Liu, Y., Lei, K., Sun, X.: Exploring and distilling cross-modal information for image captioning. In: IJCAI (2019)
  * [27] Liu, F., Ren, X., Liu, Y., Wang, H., Sun, X.: simnet: Stepwise image-topic merging network for generating detailed and comprehensive image captions. In: EMNLP (2018)
  * [28] Liu, F., Wu, X., Ge, S., Fan, W., Zou, Y.: Exploring and distilling posterior and prior knowledge for radiology report generation. In: CVPR (2021)
  * [29] Liu, F., Yin, C., Wu, X., Ge, S., Zhang, P., Sun, X.: Contrastive attention for automatic chest x-ray report generation. In: ACL (Findings) (2021)
  * [30] Liu, G., Hsu, T.H., McDermott, M.B.A., Boag, W., Weng, W., Szolovits, P., Ghassemi, M.: Clinically accurate chest x-ray report generation. In: MLHC (2019)
  * [31] Lu, J., Xiong, C., Parikh, D., Socher, R.: Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In: CVPR (2017)
  * [32] Pan, Y., Yao, T., Li, Y., Mei, T.: X-linear attention networks for image captioning. In: CVPR (2020)
  * [33] Papineni, K., Roukos, S., Ward, T., Zhu, W.: BLEU: a Method for automatic evaluation of machine translation. In: ACL (2002)
  * [34] Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: ICML (2013)
  * [35] Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: CVPR (2017)
  * [36] Shin, H., Roberts, K., Lu, L., Demner-Fushman, D., Yao, J., Summers, R.M.: Learning to read chest x-rays: Recurrent neural cascade model for automated image annotation. In: CVPR (2016)
  * [37] Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (2014)
  * [38] Syeda-Mahmood, T.F., Wong, K.C.L., Gur, Y., Wu, J.T., Jadhav, A., Kashyap, S., Karargyris, A., Pillai, A., Sharma, A., Syed, A.B., Boyko, O.B., Moradi, M.: Chest x-ray report generation through fine-grained label learning. In: MICCAI (2020)
  * [39] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NIPS (2017)
  * [40] Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: CVPR (2015)
  * [41] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: ICML (2015)
  * [42] Xue, Y., Xu, T., Long, L.R., Xue, Z., Antani, S.K., Thoma, G.R., Huang, X.: Multimodal recurrent model with attention for automated radiology report generation. In: MICCAI (2018)
  * [43] You, Q., Jin, H., Wang, Z., Fang, C., Luo, J.: Image captioning with semantic attention. In: CVPR (2016)
  * [44] Yuan, J., Liao, H., Luo, R., Luo, J.: Automatic radiology report generation based on multi-view image fusion and medical concept enrichment. In: MICCAI (2019)
  * [45] Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A.L., Xu, D.: When radiology report generation meets knowledge graph. In: AAAI (2020)