[ type=editor, auid=000,bioid=1,orcid=0000-0002-7163-1809 ] [1]

1] organization=The Australian e-Health Research Centre, CSIRO Health and
Biosecurity, Brisbane, Australia,

[]

[]

[cor1]Corresponding author

# Improving Chest X-Ray Report Generation by Leveraging Warm Starting

Aaron Nicolson<EMAIL_ADDRESS>[  Jason Dowling  Bevan Koopman

###### Abstract

Automatically generating a report from a patient’s Chest X-Rays (CXRs) is a
promising solution to reducing clinical workload and improving patient care.
However, current CXR report generators—which are predominantly encoder-to-
decoder models—lack the diagnostic accuracy to be deployed in a clinical
setting. To improve CXR report generation, we investigate warm starting the
encoder and decoder with recent open-source computer vision and natural
language processing checkpoints, such as the Vision Transformer (ViT) and
PubMedBERT. To this end, each checkpoint is evaluated on the MIMIC-CXR and IU
X-Ray datasets. Our experimental investigation demonstrates that the
Convolutional vision Transformer (CvT) ImageNet-21K and the Distilled
Generative Pre-trained Transformer 2 (DistilGPT2) checkpoints are best for
warm starting the encoder and decoder, respectively. Compared to the state-of-
the-art ($\mathcal{M}^{2}$ Transformer Progressive), CvT2DistilGPT2 attained
an improvement of 8.3% for CE F-1, 1.8% for BLEU-4, 1.6% for ROUGE-L, and 1.0%
for METEOR. The reports generated by CvT2DistilGPT2 have a higher similarity
to radiologist reports than previous approaches. This indicates that
leveraging warm starting improves CXR report generation. Code and checkpoints
for CvT2DistilGPT2 are available at https://github.com/aehrc/cvt2distilgpt2.

###### keywords:

Chest X-ray report generation Image captioning Multi-modal learning warm
starting

##  1 Introduction

Chest X-Ray (CXR) report generation is the task of automatically generating a
radiology report from a given patient’s CXR. It has the potential to improve
radiologist workflows, reduce the burden of radiology reporting, and improve
patient outcomes (Thrall et al., 2018). The most popular method of CXR report
generation is with a deep learning model, specifically, an encoder-to-decoder
model as shown in Figure 1 (Pavlopoulos et al., 2021). First, the encoder
extracts visual features from a given CXR. Next, the decoder autoregressively
generates each word (or subword) of the radiology report based on the
previously generated words and the visual features. While current CXR report
generation methods are promising, a significant improvement in diagnostic
accuracy is required before clinical consideration. One cause is that the
publicly available datasets used to develop CXR report generators (e.g.,
MIMIC-CXR (Johnson et al., 2019a) and IU X-Ray (Demner-Fushman et al., 2015))
are relatively smaller and of lesser quality than general-domain image
captioning datasets (Chen et al., 2015). For example, CXR report generators
perform poorly on underrepresented abnormalities within MIMIC-CXR (Liu et al.,
2019).

Figure 1:  Encoder-to-decoder model for CXR report generation. The encoder
extracts visual features from the CXR. Following this, the decoder
autoregressively generates each subword of the radiology report based on the
previously generated subwords and the visual features from the encoder. The
example is DICOM 02aa804e-bde0afdd-112c0b34-7bc16630-4e384014 from study
50414267 of patient 10000032 from MIMIC-CXR. [BOS] is the beginning-of-
sentence special token.

A potential solution that has not been thoroughly investigated for CXR report
generation is warm starting. Warm starting refers to the initialisation of a
models parameters with those from a pre-trained model, that is, an identical
model that has been trained on some task that is related to the task of
concern (where the related task is otherwise known as a pre-training task). By
warm starting a model, the transfer of knowledge from the pre-training task to
the task of concern can provide a significant performance boost. Warm starting
is particularly effective when the dataset for the task of concern is lacking
in size or quality, the domain of the pre-training task is similar to the task
of concern, and the pre-training dataset is of considerable size or quality
(Zhuang et al., 2021).

When warm starting, the collection of learned parameters of a pre-trained
model from a specific point during its training is used, otherwise known as a
checkpoint. Checkpoints are saved at set intervals during model training, and
the checkpoint that is selected for further use (in this case, warm starting)
is usually the one that attains the best score for a chosen metric. For an
encoder-to-decoder model, either the encoder, decoder, or both can be warm
started with different encoder-only or decoder-only checkpoints. This gives
rise to a vast number of different encoder and decoder checkpoint combinations
that can be considered.

There are many publicly available checkpoints that can be used for warm
starting, with two main types frequently available. There are checkpoints that
have been trained on a large amount of general-domain data. There are also
those trained in a specific domain on a smaller amount of data than their
general-domain counterparts, yet larger than the data available for the task
of concern. For example, BERT is a general-domain checkpoint trained on
English Wikipedia and BookCorpus (Devlin et al., 2019), and ClinicalBERT is a
domain-specific checkpoint whose last stage of pre-training is on MIMIC-III
(Alsentzer et al., 2019). Together, English Wikipedia and BookCorpus form a
significantly larger dataset than MIMIC-III, yet the domain of the Intensive
Care Unit (ICU) Electronic Health Records (EHRs) from MIMIC-III is closer to
radiology reports. This makes it difficult to discern which compromise is
appropriate when selecting a checkpoint for a specific task such as CXR report
generation.

In this work, we study the impact of warm starting on CXR report generation
with the aim of generating reports that are more similar to radiologists’
reports than previous approaches. This is motivated by the performance gains
that can be attained through warm starting, especially when there are
deficiencies with the dataset for a task (Zhuang et al., 2021). Given the
deficiencies associated with MIMIC-CXR and IU X-Ray, transferring knowledge
from a pre-trained model via warm starting may be beneficial. We do this by
evaluating various general-domain and domain-specific checkpoints for the
encoder and decoder on the MIMIC-CXR and IU X-Ray datasets (described in
Section 5) using several objective metrics (described in Section 7.2). From
this evaluation, we determine which checkpoints are most suitable for warm
starting the encoder and decoder of a CXR report generator. Using these
checkpoints, we aim to generate reports that are more similar to radiologists’
reports than previous approaches in the literature. We also aim to better
understand what influences the effectiveness of warm starting, such as the
domain and size of the pre-training data and the model architecture of the
checkpoint. This investigation also enables us to give insightful
recommendations on how to further improve CXR report generation.

##  2 Background

The chest radiograph or CXR is an indispensable tool for diagnosing diseases
of the cardiovascular and respiratory systems (Kelly, 2012). While it is the
most commonly performed radiologic examination, the CXR is difficult to
interpret (Kanne et al., 2005). Even though CXRs are analysed by clinicians of
all types, radiologists demonstrate the highest diagnostic accuracy (Satia et
al., 2013). Alarmingly, the workload of radiologists has increased
significantly over the last couple of decades—mostly due to increases in
cross-sectional imaging and understaffing (McDonald et al., 2015; Liu et al.,
2017). As a consequence, burnout is common amongst radiologists, with fatigue
leading to a reduction in their diagnostic accuracy (Harolds et al., 2016;
Krupinski et al., 2010). Radiologists also exhibit interobserver and
intraobserver variability between their diagnoses (Balabanova et al., 2005).
Currently, radiologists communicate their findings to referring clinicians
through a report. However, there are inconsistencies in reporting styles, with
structured reports recommended (European Society of Radiology, 2011).
Moreover, failing to report in a clear and concise manner—which is exacerbated
by fatigue—can also lead to sub-optimal patient care (Siegal et al., 2017;
Vosshenrich et al., 2021).

Deep learning is an important tool for medical image analysis, with recent
applications to tasks such as segmentation (Wang et al., 2022), COVID-19
diagnosis (Li et al., 2021), and disease classification (Li et al., 2023). It
has been applied to a range of modalities, for example, radiography,
ultrasonography, computed tomography, magnetic resonance, and radiomic data.
An emerging solution for CXR interpretation is the use of deep learning to
automatically generate a report from a given CXR (Pavlopoulos et al., 2021).
CXR report generation has become the most popular medical image captioning
task with more datasets than any other modality or anatomical region (Ayesha
et al., 2021). Medical image captioning is a domain-specific version of
general-domain image captioning—a topic that has garnered a high amount of
attention over the last couple of years (Yang et al., 2022; Ji et al., 2021).

CXR report generation has the potential to improve radiologist workflows; by
providing a pre-filled radiology report ready for modification, the burden of
radiology reporting could be reduced. Such a system could also increase the
diagnostic accuracy of clinicians who have a lower diagnostic confidence
(Thrall et al., 2018; Alexander et al., 2020). A hosted system would also be
able to serve the radiology demands of an entire health system, as it
possesses the ability to remotely interpret multiple CXRs simultaneously in
seconds. Moreover, the system would not suffer from intraobserver variability
or fatigue—which could result in more consistent reporting (Leeuwen et al.,
2021). While current CXR report generation approaches are promising, a
significant improvement in diagnostic accuracy is needed before adoption into
a clinical setting (Kelly et al., 2019).

As mentioned previously, the encoder-to-decoder model is the most popular
approach to automatic CXR report generation (Pavlopoulos et al., 2021). Visual
features from the CXR are extracted with the encoder, were the visual features
are a high-level representation of regions in the CXR. During generation, the
decoder is conditioned on the visual features via its cross-attention modules
and on the previous words of the report via its self-attention modules. The
encoder is typically a Convolutional Neural Network (CNN), such as a Residual
or Densely-connected Network (ResNet or DenseNet, respectively) (He et al.,
2016; Huang et al., 2017). A CNN includes multiple layers of two-dimensional
convolutional kernels that have an inductive bias towards local spatial
features—a characteristic that makes CNNs suitable for Computer Vision (CV)
tasks. For the decoder, a Transformer decoder is most commonly employed
(Vaswani et al., 2017). A Transformer consists of layers that have multiple
attention heads, where attention produces an output based on which inputs it
deems important. For Natural Language Processing (NLP), each input corresponds
to a word. Along with the encoder-to-decoder model, several deep learning
techniques have been investigated for CXR report generation.

For CXR interpretation, warm starting the encoder with a general-domain CV
checkpoint—typically a ResNet or DenseNet ImageNet-1K checkpoint—provides a
significant performance boost over random parameter initialisation (Ke et al.,
2021; Russakovsky et al., 2015; Pavlopoulos et al., 2021). More recently,
DistilGPT2—a general-domain NLP checkpoint—was used to warm start the decoder
(Alfarghaly et al., 2021). However, a direct comparison to a randomly
initialised decoder was not provided. While warm starting the encoder has
become standard practice, and warm starting the decoder is beginning to
receive traction, there exists a plethora of general-domain and domain-
specific CV and NLP checkpoints that have yet to be explored for CXR report
generation.

General-domain CV checkpoints have received a considerable amount of attention
as of late (Kolesnikov et al., 2020). Various pre-training tasks, such as
distillation and self-supervised learning, have produced checkpoints that
provide a significant performance boost on general-domain tasks (Bao et al.,
2021; Touvron et al., 2021). Along with these, several prominent CV models and
their checkpoints have not been investigated for CXR report generation.
ImageNet-1K checkpoints for EfficientNet are one such example; EfficientNet is
a CNN that can outperform ResNets and DenseNets while consuming drastically
fewer parameters (Tan and Le, 2019). Another promising checkpoint is the
Vision Transformer (ViT) and its improvements (Dosovitskiy et al., 2020; Wu et
al., 2021; El-Nouby et al., 2021). ViT is a Transformer that has been trained
on large general-domain image classification datasets, where the set of words
as input are replaced with a set of patches from an image. Transformers
possess several benefits over CNNs, such as no pooling, a full-receptive field
at each layer, and robustness to occlusion (Naseer et al., 2021). More recent
checkpoints that use distillation have outperformed EfficientNet on
ImageNet-1K, namely the Data-efficient image Transformer (DeiT) (Touvron et
al., 2021).

While DistilGPT2 has been investigated for CXR report generation, several
other prominent NLP checkpoints have not. General-domain NLP checkpoints
formed from large corpora, such as BERT and GPT2, have been shown to boost
performance in Natural Language Understanding (NLU) (Devlin et al., 2019) and
Natural Language Generation (NLG) (Radford et al., 2019) tasks, respectively.
NLU focuses on comprehending natural language through grammar and context
while NLG focuses on constructing natural language based on a given input. As
BERT is an NLU checkpoint, one would assume that it is less suitable than GPT2
for warm starting a decoder tasked with NLG. However, Rothe et al. (2020,
BERT2BERT vs. BERT2GPT2) disproved this by demonstrating that BERT is more apt
than GPT2 for warm starting the decoder of a sequence-to-sequence model.
Following BERT, several NLU checkpoints were developed from large biomedical
natural language corpora. Warm starting with a biomedical NLU checkpoint
instead of BERT was shown to improve performance in biomedical NLU tasks (Lee
et al., 2019). As highlighted by Kaur et al. (2021), warm starting the decoder
of a CXR report generator with a biomedical NLU checkpoint is a promising
approach that warrants investigation. Hence, an investigation determining if
NLG and NLU checkpoints can be effectively fine-tuned to model not only CXR
reports but also visual features of CXRs is warranted.

##  3 Related work

###  3.1 CXR report generation

In this section, we summarise both foundational and recent automatic CXR
report generation approaches in the literature. For a more exhaustive review
of CXR report generation, we refer the reader to the survey conducted by Kaur
et al. (2021). As highlighted by Pavlopoulos et al. (2021), the MIMIC-CXR and
IU X-Ray datasets are consistently used in the literature to evaluate CXR
report generation (both are described in Subsection 5). The following
approaches are evaluated using at least one or both of these datasets.

Wang et al. (2018) was the first to use an encoder-to-decoder model for CXR
report generation—a ResNet-502RNN with attention, where RNN refers to a
Recurrent Neural Network.111Henceforth, the naming convention for encoder-to-
decoder models will be the name of the encoder, followed by 2, followed by the
name of the decoder. For example, a model with a CNN encoder and an RNN
decoder will be named CNN2RNN. The authors employed multi-task learning of CXR
report generation (IU X-Ray) and multi-label abnormality classification
(ChestX-ray14 abnormality labels). The ResNet was warm started using an
ImageNet-1K checkpoint. The proposed approach outperformed a general-domain
image captioning approach, where several NLG metrics were used for evaluation.
However, many false negative abnormality predictions were evident in the
generated reports (Wang et al., 2018, Figure 4).

While NLG metrics capture the similarity between the predicted and ground-
truth reports, they do not always capture diagnostic accuracy (Pavlopoulos et
al., 2021). Motivated by this, Liu et al. (2019) proposed the Clinical
Efficacy (CE) metrics. The CE metrics make use of the CheXpert labeler—a tool
developed to extract 14 observations from the reports of the CheXpert dataset
(12 of which are abnormalities) (Irvin et al., 2019). For each observation,
the CheXpert labeler predicts whether each observation was mentioned as
positive, negative, or uncertain, or if it was not mentioned. For the CE
metric, observations are first extracted from the generated and ground-truth
reports using the CheXpert labeler. Following this, the precision, recall, and
F-1 score between the observations of the generated and ground-truth reports
are calculated to give the scores of the CE metric. Liu et al. (2019) used a
reward derived from the CE metric for Self-Critical Sequence Training (SCST),
in place of an NLG metric. SCST enables the decoder to be fine-tuned with its
own outputs as input—something that is not possible with standard training
schema (Rennie et al., 2017). This is achieved by using a reinforcement
learning algorithm that uses the score between the generated and ground-truth
reports as the reward. This was able to outperform the approach by Wang et al.
(2018) for all tested NLG metrics. Moreover, using SCST was shown to improve
the diagnostic accuracy of the generated reports.

To embed expert knowledge, Zhang et al. (2020) incorporated a Graph
Convolutional Network (GCN)—whose nodes consisted of abnormalities—into the
encoder. This was able to outperform the aforementioned CXR report generation
approaches on multiple NLG metrics. The authors also developed a diagnostic
accuracy metric based on the graph CNN—which showed that the graph CNN
improves diagnostic accuracy.

More recently, Lovelace and Mortazavi employed a CNN followed by a series of
Transformer layers as the encoder and a Transformer decoder. To improve
diagnostic accuracy, the CheXpert labeler was used to extract observations
from the generated report in a differentiable manner, where the loss between
the observations of the generated and ground-truth reports was added to the
training loss. This outperformed multiple approaches that employed an RNN as
the decoder, where multiple NLG metrics and the CE metrics were used for
evaluation (Lovelace and Mortazavi, 2020).

Chen et al. proposed a ‘memory-driven’ Transformer decoder (R2Gen) (Chen et
al., 2020), and later built upon this by proposing the ‘Cross-modal Memory
Network’ (CMN) (Chen et al., 2021). Using multiple NLG metrics and the CE
metric, it was determined that the proposed approaches were able to outperform
several previous CXR report generation approaches in the literature, each of
which used an RNN as the decoder (Jing et al., 2019; Li et al., 2018; Jing et
al., 2018).

Alfarghaly et al. (2021) employed CheXNet (Rajpurkar et al., 2017)—a
DenseNet-121 ChestX-ray14 checkpoint—as the encoder and a Transformer as the
decoder. The decoder was warm started with DistilGPT2, a general-domain NLG
checkpoint. However, cross-attention was not used with each layer of the
decoder, rather the embeddings of each predicted abnormality of CheXNet were
fed as input to DistilGPT2. This method outperformed multiple approaches that
utilised an RNN as the decoder for multiple NLG metrics.

Liu et al. (2021b) proposed the Contrastive Attention model which compares the
visual features of the current CXR to a pool of CXRs that have no
abnormalities, where a ResNet-50 warm started with a CheXpert checkpoint was
used to extract the visual features. Contrastive Attention comprises two
modules, ‘aggregate attention’ and ‘difference attention’. Aggregate attention
finds the normal CXRs from the pool that are closest to the CXR in question.
Difference attention involves two steps; for the first step, common features
between the current CXR and the pool of normal CXRs are found. For the second
step, the common features are subtracted from the features of the current CXR
to capture ‘contrastive information’. This was able to outperform R2Gen, where
multiple NLG metrics and the CE metrics were used for evaluation.

With the aim to imitate the interpretation process of radiologists, Liu et al.
(2021a) proposed the a Posterior-and-Prior Knowledge Exploring-and-Distilling
approach (PPKED). Along with a CXR, the model consumes embeddings of the most
common abnormalities found in the reports from the training set. It also
consumes encoded reports from CXRs of the training set that have similar
visual features to the current CXR. Finally, the model consumes the embedding
from a knowledge graph that represents the most common abnormalities found in
the training set. This outperformed R2Gen, where multiple NLG metrics and the
CE metrics were used for evaluation.

Nooralahzadeh et al. (2021) proposed an approach that generated a report from
detected abnormalities in the CXR. First, a DenseNet-121 extracted visual
features from a CXR, where a pre-trained CheXpert checkpoint was used for warm
starting. Next, an encoder-to-decoder model called the meshed Transformer with
memory ($\mathcal{M}^{2}$ Transformer) (Cornia et al., 2020) predicted the
abnormalities from the visual features. The labels for the $\mathcal{M}^{2}$
Transformer were extracted from the CXR using the GCN of Zhang et al. (2020).
These were fed to BART to generate the report, where BART is a sequence-to-
sequence encoder-to-decoder model (Lewis et al., 2020). The final method,
named $\mathcal{M}^{2}$ Transformer Progressive, outperformed R2Gen, where
multiple NLG metrics and the CE metrics were used for evaluation.

###  3.2 CV checkpoints

As highlighted in the previous subsection, the encoder is typically warm
started with a ResNet or DenseNet ImageNet-1K or CXR checkpoint. These are
summarised in this section, along with more recent CV models and their
associated checkpoints.

Residual and dense aggregations of layer outputs have been found to benefit
training. Residual aggregations simplify the landscape of the loss function
and prevent the vanishing and exploding gradient problems (He et al., 2016).
This allows the training of very deep CNNs, called ResNets. The dense
aggregations of DenseNet offer direct feature re-usage, as deeper layers have
access to the outputs of shallower layers (Huang et al., 2017). Following
this, Tan and Le (2019) focused on efficiently scaling the depth, width, and
input image size of CNNs, forming EfficientNets. As a result, EfficientNets
are able to significantly outperform ResNets and DenseNets on ImageNet-1K,
while remaining parameter efficient. An EfficientNet as the encoder, warm
started or not, has not been investigated for CXR report generation. Domain-
specific CXR checkpoints for CNNs also exist in the literature. Rajpurkar et
al. (2017) proposed CheXNet, a DenseNet-121 checkpoint for multi-label
classification of 14 abnormalities. The checkpoint was formed by additionally
training an ImageNet-1K checkpoint on ChestX-Ray14.

More recently, Transformer encoders have been investigated for CV. Dosovitskiy
et al. (2020) proposed the Vision Transformer (ViT)—a Transformer checkpoint
pre-trained on large general-domain image classification datasets which takes
as input a set of patches from an image. ViT possesses several appealing
features for CV, including no pooling, a full-receptive field, and robustness
to occlusion (Naseer et al., 2021). However, ViT does not posses the same
inductive bias that makes CNNs an attractive option for medical CV tasks—a
bias towards local spatial features. While the self-attention weights of a ViT
head are able to model the relationship between patches, they do not model the
relationship between pixels. This may be detrimental for CXR report
generation, as many anatomical features of the chest—such as the pulmonary
arteries—are represented in fine detail by the CXR. Moreover, ViT was only
able to outperform ResNet when warm starting with checkpoints that have been
trained using 30 million or more images (Dosovitskiy et al., 2020, Figure 4,
ViT-B/32 vs. ResNet50x1 (BiT)). This lead to Dosovitskiy et al. concluding
that ViT does not generalise well when trained on insufficient amounts of
data.

There are multiple improvements to ViT in the literature that either use a
self-supervised pre-training task or modify the Transformer to manually inject
an inductive bias towards local spatial features. The Convolutional vision
Transformer (CvT) replaces the linear layers of each self-attention head with
two-dimensional convolutional layers, thus introducing an inductive bias to
local spatial features into each head. This enabled CvT to outperform ViT on
ImageNet-1K (Wu et al., 2021). The Data-efficient image Transformer (DeiT) is
an ImageNet-1K checkpoint that incorporated knowledge distillation into its
set of pre-training tasks. Knowledge distillation involves two models; a
smaller model—called the student—and a larger model—called the teacher. The
student is trained to replicate the categorical distribution of the teacher.
For DeiT, the teacher was a recent CNN. On ImageNet-1K, DeiT was able to
outperform ViT and EfficientNet (Touvron et al., 2021).

The Cross-Covariance image Transformer (XCiT) utilises a transposed version of
the self-attention mechanism. This results in the attention weights modelling
the relationship between feature channels rather than patches. To model the
relationship between local patches, each layer of XCiT employs a two-
dimensional convolutional kernel. XCiT was able to outperform both
EfficientNet and DeiT on ImageNet-1K (El-Nouby et al., 2021). Inspired by
BERT, Bao et al. (2021) used a self-supervised task, namely Masked Image
Modelling (MIM), to form a Bidirectional Encoder representation from an image
Transformer (BEiT). For MIM, the objective for each randomly masked patch is
to predict its corresponding discrete visual token. The token for each patch
was produced by a discrete Variational Auto-Encoder (VAE). BEiT was able to
outperform both ViT and DeiT on ImageNet-1K.

###  3.3 NLP checkpoints

As detailed in Subsection 3.1, the Transformer is the standard decoder used by
recent CXR report generation methods. In this subsection, we summarise the
Transformer and its checkpoints. Vaswani et al. (2017) proposed the
Transformer—a sequence-to-sequence model for NLG. Its layers employ multiple
scaled dot-product attention heads, each of which model the complex
dependencies between its inputs. Due to this, Transformers more efficiently
model long-range dependencies between subword tokens than RNNs and Temporal
Convolutional Networks (TCNs) (Vaswani et al., 2017, Table 1).

Building upon this, checkpoints for the encoder and decoder of the Transformer
were formed with self-supervised pre-training tasks and extremely large,
unlabelled corpora. Masked Language Modelling (MLM) and Next Sentence
Prediction (NSP) are self-supervised pre-training tasks that were used to form
Bidirectional Encoder Representations from Transformers (BERT)—a general-
domain NLU checkpoint created from BookCorpus (Zhu et al., 2015) and English
Wikipedia. As the encoder is non-causal, an output for BERT depends on all
input tokens during MLM and NSP, and not just previous tokens. This enabled
BERT to outperform previous Transformer checkpoints on multiple NLU tasks
(Devlin et al., 2019). The second version of the Generative Pre-trained
Transformer (GPT2) is a checkpoint for the decoder of the Transformer. Using
language modelling as the pre-training task and the WebText corpus (Radford et
al., 2019), GPT2 achieved state-of-the-art performance on several zero-shot
NLG tasks (Radford et al., 2019).

Sanh et al. (2019) proposed DistilBERT, a distilled version of BERT that
consumes 40% fewer parameters while retaining 97% of its performance.
Knowledge distillation was incorporated into the training task, where
DistilBERT was the student and BERT was the teacher. The authors also produced
DistilGPT2 in a similar fashion.

General (e.g. English Wikipedia) Biomedical (e.g. PubMed) CXR reports (e.g. MIMIC-CXR) ICU EHRs (e.g. MIMIC-III) Figure 2: Non-proportional Venn diagram of the vocabulary of each natural language domain. As CXR reports are included in Intensive Care Unit (ICU) Electronic Health Records (EHRs), the vocabulary of CXR reports is a subset of the vocabulary of ICU EHRs. The vocabulary of ICU EHRs is a subset of the vocabulary of biomedical natural language, which is a subset of the vocabulary of the general domain. An example of a corpus belonging to each domain is also given. Table 1: Datasets sized for pre-training. The number of examples for Semantic Scholar are those used to pre-train SciBERT (Beltagy et al., 2019). Dataset | No. of examples
---|---
ImageNet-21K (Deng et al., 2009) | 14M images
ImageNet-1K (Russakovsky et al., 2015) | 1.3M images
CheXpert (Irvin et al., 2019) | 224K images
BookCorpus (Zhu et al., 2015) | 0.8B words
English Wikepedia | 2.5B words
WebText (Radford et al., 2019) | 8M documents
Semantic Scholar (Fricke, 2018) | 1.14M papers
PubMed | 4.5B words
PMC | 13.5B words
MIMIC-III (Johnson et al., 2016) | 2M notes

The self-supervised tasks used to form BERT (namely MLM and NSP) have been
used to form domain-specific NLU checkpoints. Beltagy et al. (2019) formed a
scientific NLU checkpoint from 1.14M documents of Semantic Scholar—a corpus of
scientific publications. The resulting checkpoint, called SciBERT,
outperformed BERT on several scientific NLU tasks. SciBERT also featured a
domain-specific vocabulary constructed from Semantic Scholar. Multiple domain-
specific checkpoints have been formed from biomedical corpora, particularly
PubMed and PubMed Central (PMC). As seen in Table 1, a benefit of PubMed and
PMC is that they are both larger in size than the corpora used to train BERT
and GPT2 (BookCorpus, English Wikipedia, and WebText). BioBERT is one such
example, which outperformed BERT on several biomedical NLU tasks. However,
BioBERT does not have a domain-specific vocabulary, instead it is inherited
from BERT. This may be less than optimal, as medical terms not captured by
BERT’s vocabulary would be represented by multiple subwords (Lee et al.,
2019). PubMedBERT improves upon BioBERT by having a domain-specific vocabulary
built from PubMed and PMC. The authors also demonstrate that training from
scratch rather than additionally training a general-domain checkpoint such as
BERT is best for biomedical NLU tasks—if there is sufficient data (Gu et al.,
2020).

There also exists multiple checkpoints formed from MIMIC-III—an Intensive Care
Unit (ICU) Electronic Health Record (EHR) corpus. As seen in Figure 2, the
vocabulary of EHRs is a subset of the vocabulary of biomedical natural
language. ClinicalBERT is an EHR NLU checkpoint formed by further training
BioBERT with MLM and NSP on MIMIC-III (Alsentzer et al., 2019). A similar
checkpoint is BlueBERT, which is formed in two stages: BERT is further trained
with MLM and NSP on PubMed. The resulting checkpoint is further trained with
MLM and NSP on MIMIC-III, finally forming BlueBERT (Peng et al., 2019). Both
ClinicalBERT and BlueBERT were able to outperform BioBERT on several EHR NLU
tasks. However, ClinicalBERT and BlueBERT do not have domain-specific
vocabularies and instead rely on the vocabulary of BERT. Moreover, MIMIC-III
is considerably smaller in size than the other corpora listed in Table 1,
raising concerns as to whether ClinicalBERT and BlueBERT can be successfully
fine-tuned to CXR report generation.

##  4 Contributions

Other than determining which checkpoints are most suitable for warm starting
the encoder and decoder of a CXR report generator, we aim to answer the
following Research Questions (RQs):

RQ1:

Are Transformer encoder CV checkpoints better than CNN checkpoints for warm
starting the encoder?

RQ2:

Can an NLP checkpoint be effectively fine-tuned to model not only natural
language but also visual features?

RQ3:

Are NLU checkpoints (e.g., BERT) better for warm starting the decoder than NLG
checkpoints (e.g., GPT2)?

RQ4:

Are domain-specific checkpoints better for warm starting than general-domain
checkpoints?

Our study differs from previous studies as follows:

  * •

We investigate the impact of publicly available checkpoints on CXR report
generation.

  * •

We investigate CV models and checkpoints not previously considered for CXR
report generation, namely, EfficientNet and Transformer encoder CV models.

  * •

We investigate NLP checkpoints, specifically, NLU checkpoints such as BERT,
and NLG checkpoints such as GPT2.

  * •

We investigate domain-specific CV and NLP checkpoints.

  * •

We present a case study of the final model that includes the attention weights
of its cross-attention heads—to reveal what the model attends to when
generating a report.

  * •

We present a fine-grained evaluation on individual abnormalities, something
that has been lacking in recent studies.

##  5 Datasets

The Medical Information Mart for Intensive Care CXR dataset (MIMIC-CXR)
consists of $377{,}110$ CXRs in both DICOM and JPEG formats, and $227{,}835$
English radiology reports associated with $64{,}588$ patients. The reports and
CXRs were automatically de-identified (Johnson et al., 2019a, b). Demner-
Fushman et al. (2015) released a dataset called IU X-Ray that consisted of
$3{,}955$ English radiology reports and $7{,}470$ CXRs in both DICOM and PNG
formats, where both the reports and CXRs were de-identified automatically.
Each report is associated with a single patient. We refer the reader to the
survey conducted by Pavlopoulos et al. (2021, Subsection 2.1) for a more
detailed analysis of MIMIC-CXR and IU X-Ray.

In order to compare to previous studies, we adopt the dataset splits and
labels of Chen et al. (2020) for both MIMIC-CXR and IU X-Ray.222The MIMIC-CXR
and IU X-Ray subsets used by Chen et al. (2020) are available at:
https://github.com/cuhksz-nlp/R2Gen. For MIMIC-CXR, the splits are formed from
$276{,}778$ of the CXRs. A portion of the reports were associated with
multiple CXRs, meaning that they were the label for multiple examples. The
subset splits are detailed in Table 2; the $276{,}778$ CXRs are split into
$270{,}790$, $2{,}130$, and $3{,}858$ for training, validation, and testing,
respectively. For IU X-Ray, the subsets are formed from $2{,}955$ of the
reports, where each report is associated with a frontal and lateral view. This
means that the model consumes two CXRs per example for IU X-Ray. The $2{,}955$
reports are split into $2{,}069$, $296$, and $590$ for training, validation,
and testing, respectively. A training example for IU X-Ray is shown in Figure
3 (right); the frontal and lateral views are given as input to the encoder and
the label is the ground-truth report produced by a radiologist. For both
MIMIC-CXR and IU X-Ray, each patient is included in only one of the subsets.

Table 2: Subset splits of MIMIC-CXR and IU X-Ray from Chen et al. (2020). Dataset | No. of examples | Pixel depth | Views per example
---|---|---|---
Training | Validation | Test
MIMIC-CXR (splits from Chen et al. (2020)) | $270{,}790$ | $2{,}130$ | $3{,}858$ | 8-bit | 1
IU X-Ray (splits from Chen et al. (2020)) | $2{,}069$ | 296 | 590 | 8-bit | 2
Figure 3: CXR report generation framework for (left) MIMIC-CXR and (right) IU
X-Ray as in Chen et al. (2020). [BOS] is the beginning-of-sentence special
token.

##  6 Problem formulation

Given a set of $N$ CXRs
$\mathcal{X}=\left\\{\boldsymbol{X}_{0},\dots,\boldsymbol{X}_{N-1}\right\\}$
where $\boldsymbol{X}_{i}\in\mathbb{R}^{C\times W\times H}$ and $C$, $W$, and
$H$ denote the number of channels, the width, and the height of each CXR,
respectively, the aim is to generate a report ($\hat{R}$) whose target is the
ground-truth report $R$. The encoder-to-decoder model generates $\hat{R}$ in
multiple stages; first, the encoder $E$ produces a set of visual features from
each of the CXRs
$\mathcal{V}=\left\\{\boldsymbol{V}_{0},\dots,\boldsymbol{V}_{N-1}\right\\}$,
where the encoder processes each CXR independently
($E:\boldsymbol{X}_{i}\rightarrow\boldsymbol{V}_{i}$),
$\boldsymbol{V}_{i}\in\mathbb{R}^{S\times F}$, $S$ is the number of spatial
positions, and $F$ is the number of features for each spatial position. Here,
the visual features correspond to the last hidden state of the encoder. For
some of the CV models, particularly the CNNs, the spatial positions are
distributed over two axes—which are flattened to give $S$. Moreover, for some
CV models, the spatial position and feature axes of the last hidden state are
opposite (i.e. $\boldsymbol{V}_{i}\in\mathbb{R}^{F\times S}$) and must be
transposed. Next, the visual features for each CXR are concatenated along the
spatial position axis:
$\boldsymbol{V}_{concat}=\texttt{concat}\left(\boldsymbol{V}_{0},\dots,\boldsymbol{V}_{N-1}\right)\in\mathbb{R}^{D\times
F}$, where concat is the concatenation operation and $D=S\times N$. Next, the
visual features are projected to the hidden state size of the decoder $H$
using learned projection matrix $\boldsymbol{P}\in\mathbb{R}^{F\times H}$:
$\boldsymbol{V}_{project}=\boldsymbol{V}_{concat}\cdot\boldsymbol{P}\in\mathbb{R}^{D\times
H}$.

Table 3: CV checkpoints for warm-starting the encoder. Configurations for the same model are separated by ampersands and commas. The symbol $\to$ indicates that a checkpoint was transferred to a new domain. The configurations for ResNet, DenseNet, and CvT indicate the number of layers; the configurations for EfficientNet indicate the size of the model with B4 being the smallest and B7 being the largest; the configurations for ViT, DeiT, and BEiT indicate the patch size, where distillation is the pre-training task for DeiT; the configurations for XCiT are determined as follows: the letter and number combination before the forward slash indicates the size of the model, $\Upsilon$ indicates that distillation was used as a pre-training task, and the number between the forward slash and $\Upsilon$ indicates the width and height of the patch size. Model | Configuration/s |  Image width ($W$) | Pre-training data
---|---|---|---
ResNet | 101 & 152 | 224 & 224 | ImageNet-1K
DenseNet | 121, 169, & 201 | 224, 224, & 224 | ImageNet-1K
CheXNet | DenseNet-121 | 224 |  ImageNet-1K $\to$ CheXpert
EfficientNet | B4, B5, B6, & B7 | 380, 456, 528, & 600 | ImageNet-1K
$\rm ViT_{BASE}$ | 16x16 | 384 |  ImageNet-21K $\to$ ImageNet-1K
CvT | 13 & 21 | 384 & 384 | ImageNet-21K
$\rm DeiT_{BASE}$ | 16x16 with distillation | 384 | ImageNet-1K
XCiT |  S12/16$\Upsilon$, S12/8$\Upsilon$, S24/16$\Upsilon$, S24/8$\Upsilon$, & M24/16$\Upsilon$ | 384, 384, 384, 384, & 384 | ImageNet-1K
$\rm BEiT_{BASE}$ | 16x16 | 384 |  ImageNet-21K $\to$ ImageNet-1K
Table 4: NLP checkpoints for warm-starting the decoder. The symbol $\to$ indicates that a checkpoint was transferred to a new domain. The Transformer is identical in configuration to DistilBERT, except that its parameters are randomly initialised rather than warm-started. PubMed and PubMed Central (PMC) are available at https://pubmed.ncbi.nlm.nih.gov/ and https://www.ncbi.nlm.nih.gov/pmc/, respectively. Model | Cased/uncased | Pre-training data | Vocabulary
---|---|---|---
Transformer | Uncased | – | 30K BookCorpus + English Wikipedia
GPT2 | Cased | WebText | 50k WebText
$\rm BERT_{BASE}$ | Uncased | BookCorpus + English Wikipedia | 30K BookCorpus + English Wikipedia
$\rm DistilBERT_{BASE}$ | Uncased | BookCorpus + English Wikipedia | 30K BookCorpus + English Wikipedia
DistilGPT2 | Cased | WebText | 50k WebText
$\rm BioBERT_{BASE}$ | Cased |  $\rm BERT_{BASE}$ $\to$ PubMed + PMC | 30K BookCorpus + English Wikipedia
$\rm SciBERT_{BASE}$ | Uncased | Semantic Scholar | 30K Semantic Scholar
$\rm ClinicalBERT_{BASE}$ | Cased |  $\rm BioBERT_{BASE}$ $\to$ MIMIC-III | 30K BookCorpus + English Wikipedia
$\rm BlueBERT_{BASE}$ | Cased |  $\rm BERT_{BASE}$ $\to$ PubMed $\to$ MIMIC-III | 30K BookCorpus + English Wikipedia
$\rm PubMedBERT_{BASE}$ | Uncased | PubMed + PMC | 30K PubMed + PMC

The projected visual features are fed to the decoder via a randomly
initialised multi-head cross-attention module, which is inserted between the
masked multi-head self-attention module and the feedforward neural network
module of each decoder layer (Vaswani et al., 2017, Section 3.1, Decoder). The
decoder then generates the report from the projected visual features in an
autoregressive fashion ($D:\boldsymbol{V}_{project}\rightarrow\hat{R}$). As
shown in Figure 3, $N=1$ for MIMIC-CXR and $N=2$ for IU X-Ray. This means that
the concatenation operation is not required for MIMIC-CXR. For IU X-Ray, the
two CXRs for an example correspond to a frontal and lateral view, whereas for
MIMIC-CXR, the single CXR for an example is either a frontal or a lateral
view, as in Chen et al. (2020).

##  7 Methodology

###  7.1 Checkpoints

In this study, we investigate the publicly available CV checkpoints described
in Table 3 and the NLP checkpoints described in Table 4 for warm starting CXR
report generation. The hyperparameters for each model were determined by those
from their publicly available checkpoint. The training data that formed each
checkpoint, the vocabulary of each NLP checkpoint, and the image width of each
CV checkpoint is given. Note that the image height is equal to the image width
for each CV checkpoint. The configuration of each CV model is also indicated.
For each decoder checkpoint, if available, we use the uncased version, else,
the cased version. This was due to the lowercase format of the ground-truth
reports, as described in Subsection 7.4. Finally, the number of parameters for
each checkpoint are given in Figure 4.

Figure 4: The number of parameters for each checkpoint.

###  7.2 Evaluation Metrics

As in previous CXR report generation studies, we employ several NLG metrics
and the CE metrics to evaluate the generated reports (Pavlopoulos et al.,
2021). The NLG metrics are word overlap measures that compute a similarity
score based on the number of words shared between the generated and ground-
truth reports. The Bi-Lingual Evaluation Understudy (BLEU-n) measure computes
the word n-gram overlap between the generated and ground-truth reports, for
example, BLEU-3 considers trigrams (Papineni et al., 2002). The Metric for
Evaluation of Translation with Explicit ORdering (METEOR) builds upon BLEU-1
by instead computing the $F_{\beta}$ score between unigrams (where recall is
weighted higher than precision). METEOR also employs stemming and synonymy
matching (Banerjee and Lavie, 2005).

LinearConv. token embeddingCvT-21stride=2Multi-head attention*flatten……reshape
& padNormMLPNormflattenflatten$\displaystyle\times 3$ Stages
$\displaystyle\times N_{l}$ Layersflattenstride=2 Masked multi-head  attention
NormHeadMulti-head attentionNormMLPNorm$\displaystyle\times 6$
LayersDistilGPT2position embeddings[BOS] the patient is status post median
stern—otomyToken embeddingsthe patient is status post median stern—otomy
andQKVQQKKVV

Figure 5: High-level view of the architecture of CvT-212DistilGPT2. See Figure
1 for a high-level depiction of the CXR report generation process with an
encoder-to-decoder model. Here, it is configured for MIMIC-CXR with the Chen
et al. (2020) splits. $Q$, $K$, and $V$ are the queries, keys, and values,
respectively, for multi-head attention (Vaswani et al., 2017). * indicates
that the linear layers for $Q$, $K$, and $V$ are replaced with the
convolutional layers depicted below the multi-head attention module. [BOS] is
the beginning-of-sentence special token. $N_{l}$ is the number of layers for
each stage, where $N_{l}=1$, $N_{l}=4$, and $N_{l}=16$ for the first, second,
and third stage, respectively. The head for DistilGPT2 is the same used for
language modelling. Subwords produced by DistilGPT2 are separated by a
vertical bar.

The Recall-Oriented Understudy for Gisting Evaluation with Longest common
subsequence-based statistics (ROUGE-L) is the harmonic mean of ROUGE-L Recall
and Precision. ROUGE-L Recall is ratio of the length of the longest common
n-gram shared by the generated and ground-truth reports, to the number of
words in the ground-truth report. ROUGE-L Precision is identical to ROUGE-L
Recall, except that the denominator is the number of words in the generated
report (Lin and Och, 2004). The Consensus-based Image Description Evaluation
(CIDEr) measure computes the cosine similarity between Term Frequency-Inverse
Document Frequency (TF-IDF) n-grams of the generated and ground-truth reports.
Cosine similarities are calculated for 1 to 4-grams and their average is
returned as the final score. By using TF-IDFs, terms that are infrequent in
the corpus are rewarded, while terms that are common (e.g., stopwords) are
penalised (Vedantam et al., 2015).

It should be noted that the aforementioned word overlap measures do not
necessarily capture diagnostic accuracy (Babar et al., 2021). To more
effectively evaluate this, we use the CE metrics that was previously employed
to evaluate CXR report generators (Liu et al., 2019; Chen et al., 2020). In
place of the CheXpert labeler, we use CheXbert—a BERT-based approach to
extracting the 14 CheXpert observations from a given report. CheXbert
demonstrated a statistically significant improvement in performance over the
CheXpert labeler (CheXbert had a macro-averaged F-1 score improvement of 0.055
over the CheXpert labeler), while being 120 times faster (when a GPU is
available) (Smit et al., 2020). The CE classification scores are calculated as
follows; CheXbert first determines the class of each of the 14 observations
from the generated and ground-truth reports as either positive, negative,
uncertain, or no mention.333The 14 observations include enlarged
cardiomediastinum, cardiomegaly, lung opacity, lung lesion, edema,
consolidation, pneumonia, atelectasis, pneumothorax, pleural effusion, pleural
other, fracture, support devices, and no finding. Next, the multi-class
classification task for each observation is converted into a binary
classification task; observations that are positive are considered positive
results, and observations that are negative, uncertain, or no mention are all
considered negative results. Following this, true positives, true negatives,
and false negatives are found by comparing the observations of the generated
reports to that of the ground-truth reports. From this, the example-based
precision, recall, and F-1 scores are found over the 14 observations, while
the label-based precision, recall, and F-1 scores are found for each of the 14
observations (Sorower, 2010, Section 7).

###  7.3 CXR pre-processing and augmentation

Following Chen et al. (2020), we adopted the CXRs in JPEG and PNG formats for
MIMIC-CXR and IU X-Ray, respectively, resulting in an 8-bit pixel depth and
three identical channels ($C=3$). This was required as each CV checkpoint was
configured to take three channels as input (even CheXNet). The following pre-
processing and augmentation steps were then applied to each CXR; first, a
given CXR was resized using bilinear interpolation so that its smallest side
had a length of $W+64$, where $W$ is given in Table 3, and its largest side
was set such that it maintained the aspect ratio. Next, the resized CXR was
cropped to a size of $\mathbb{R}^{3\times W\times H}$, where $W=H$. The crop
location was random during training and centered during testing. After this
and only during training, the CXR was rotated around its centre where the
angle of rotation was sampled from $\mathcal{U}{[{-5^{\circ},5^{\circ}}]}$.
Finally, the CXR was standardised using the mean and standard deviation of
each channel provided with each CV checkpoint.

###  7.4 Report pre-processing and generation

In order to compare to previous CXR report generators, we formatted the
ground-truth reports identically to Chen et al. (2020). This was achieved by
allowing a maximum of 60 words per report (words after the $60^{th}$ word were
removed), changing upper-cased letters to lowercase, removing special
characters, and replacing words that occurred less than three times in the
corpus with a special unknown token. During testing, the maximum amount of
subwords that the decoder could generate was set to 128, as each word could be
represented by multiple subwords. Beam search with a beam size of four was
used during testing when generating the reports. During validation a beam size
of one was used (i.e., greedy search).

###  7.5 Fine-tuning

Teacher forcing was used for fine-tuning (Williams and Zipser, 1989). Each
model was implemented in PyTorch version 1.9.0 and trained with
$4\times$NVIDIA P100 16GB GPUs with automatic mixed precision. To select the
best epoch for a model, we use the highest CIDEr validation score. Due to
phenomena such as mild overparameterisation, not every random initialisation
of a models parameters will lead to a global minimum during gradient descent
(Simsek et al., 2021). This leads to large performance variability between
training runs. To account for this, we performed multiple training runs for
each model. Finally, only models fine-tuned on the MIMIC-CXR training set were
tested on the MIMIC-CXR test set and models fine-tuned on the IU X-Ray
training set were tested on the IU X-Ray test set. The following configuration
was used to fine-tune each model with teacher forcing:

  * •

Categorical cross-entropy as the loss function.

  * •

AdamW optimiser for gradient descent optimisation (Loshchilov and Hutter,
2019). An initial learning rate of $1e-5$ and $1e-4$ for the encoder and all
other parameters, respectively, following Chen et al. (2020). All other
hyperparameters for AdamW were set to their defaults.

  * •

A mini-batch size of 16.

  * •

Early stopping with a patience of 10 epochs and a minimum delta of $1e-4$.

  * •

The validation CIDEr score was the monitored metric for early stopping.

Table 5: Mean NLG metric scores on the MIMIC-CXR test set with the labels of Chen et al. (2020). If available, the 95% confidence intervals are reported. $n=5$ indicates the mean over five training runs. * is the training run that scored the highest validation CIDEr score. Model | Natural language generation metrics
---|---
BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | CIDEr
R2Gen |  [HTML]f97d6e0.3530 |  [HTML]f97d6e |  [HTML]f97d6e |  [HTML]f8696b0.2180 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]f8696b0.1450 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]f8696b0.1030 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]f8696b0.1420 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]fba1750.2770 |  [HTML]fba175 |  [HTML]fba175 |  - |  |
CMN |  [HTML]f97d6e0.3530 |  [HTML]f97d6e |  [HTML]f97d6e |  [HTML]f8696b0.2180 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]fa99740.1480 |  [HTML]fa9974 |  [HTML]fa9974 |  [HTML]fcb77a0.1060 |  [HTML]fcb77a |  [HTML]fcb77a |  [HTML]f8696b0.1420 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]fbac780.2780 |  [HTML]fbac78 |  [HTML]fbac78 |  - |  |
Contrastive Attention |  [HTML]f8696b0.3500 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]f8766d0.2190 |  [HTML]f8766d |  [HTML]f8766d |  [HTML]feda800.1520 |  [HTML]feda80 |  [HTML]feda80 |  [HTML]f7e9840.1090 |  [HTML]f7e984 |  [HTML]f7e984 |  [HTML]e1e3830.1510 |  [HTML]e1e383 |  [HTML]e1e383 |  [HTML]fee5820.2830 |  [HTML]fee582 |  [HTML]fee582 |  - |  |
PPKED |  [HTML]fbad780.3600 |  [HTML]fbad78 |  [HTML]fbad78 |  [HTML]fcb77a0.2240 |  [HTML]fcb77a |  [HTML]fcb77a |  [HTML]fbaa770.1490 |  [HTML]fbaa77 |  [HTML]fbaa77 |  [HTML]fcb77a0.1060 |  [HTML]fcb77a |  [HTML]fcb77a |  [HTML]feda800.1490 |  [HTML]feda80 |  [HTML]feda80 |  [HTML]ede6830.2840 |  [HTML]ede683 |  [HTML]ede683 |  - |  |
$\mathcal{M}^{2}$ Transformer Progressive |  [HTML]c9dc810.3780 |  [HTML]c9dc81 |  [HTML]c9dc81 |  [HTML]e2e3830.2320 |  [HTML]e2e383 |  [HTML]e2e383 |  [HTML]f7e9840.1540 |  [HTML]f7e984 |  [HTML]f7e984 |  [HTML]fdd17f0.1070 |  [HTML]fdd17f |  [HTML]fdd17f |  [HTML]fa99740.1450 |  [HTML]fa9974 |  [HTML]fa9974 |  [HTML]f8696b0.2720 |  [HTML]f8696b |  [HTML]f8696b |  - |  |
CvT-212DistilGPT2 ($n\texttt{=}5$) |  [HTML]75c47d0.3918 |  [HTML]75c47d± |  [HTML]75c47d0.00008 |  [HTML]7ec67d0.2454 |  [HTML]7ec67d± |  [HTML]7ec67d0.00008 |  [HTML]81c77d0.1685 |  [HTML]81c77d± |  [HTML]81c77d0.00008 |  [HTML]81c77d0.1236 |  [HTML]81c77d± |  [HTML]81c77d0.00009 |  [HTML]b2d5800.1525 |  [HTML]b2d580± |  [HTML]b2d5800.00004 |  [HTML]d7e0820.2846 |  [HTML]d7e082± |  [HTML]d7e0820.00007 |  [HTML]f8696b0.3614 |  [HTML]f8696b± |  [HTML]f8696b0.00052
CvT-212DistilGPT2* |  [HTML]63be7b0.3928 |  [HTML]63be7b± |  [HTML]63be7b0.00013 |  [HTML]63be7b0.2478 |  [HTML]63be7b± |  [HTML]63be7b0.00013 |  [HTML]63be7b0.1713 |  [HTML]63be7b± |  [HTML]63be7b0.00013 |  [HTML]63be7b0.1267 |  [HTML]63be7b± |  [HTML]63be7b0.00013 |  [HTML]63be7b0.1545 |  [HTML]63be7b± |  [HTML]63be7b0.00007 |  [HTML]63be7b0.2863 |  [HTML]63be7b± |  [HTML]63be7b0.00012 |  [HTML]63be7b0.3892 |  [HTML]63be7b± |  [HTML]63be7b0.00077
Table 6: Mean example-based CE metric scores on the MIMIC-CXR test set with the labels of Chen et al. (2020). If available, the 95% confidence intervals are reported. $n=5$ indicates the mean over five training runs. * is the training run that scored the highest validation CIDEr score. Model | Example-based CE metrics
---|---
Precision | Recall | F-1
R2Gen |  [HTML]fdd47f0.3330 |  [HTML]fdd47f |  [HTML]fdd47f |  [HTML]f8696b0.2730 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]f8696b0.2760 |  [HTML]f8696b |  [HTML]f8696b
CMN |  [HTML]fdd67f0.3340 |  [HTML]fdd67f |  [HTML]fdd67f |  [HTML]f86a6b0.2750 |  [HTML]f86a6b |  [HTML]f86a6b |  [HTML]f8716c0.2780 |  [HTML]f8716c |  [HTML]f8716c
Contrastive Attention |  [HTML]ffeb840.3520 |  [HTML]ffeb84 |  [HTML]ffeb84 |  [HTML]f9806f0.2980 |  [HTML]f9806f |  [HTML]f9806f |  [HTML]fdd6800.3030 |  [HTML]fdd680 |  [HTML]fdd680
$\mathcal{M}^{2}$ Transformer Prog. |  [HTML]f8696b0.2400 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]63be7b0.4280 |  [HTML]63be7b |  [HTML]63be7b |  [HTML]ffeb840.3080 |  [HTML]ffeb84 |  [HTML]ffeb84
CvT-212DistilGPT2 ($n\texttt{=}5$) |  [HTML]afd4800.3597 |  [HTML]afd480± |  [HTML]afd4800.0003 |  [HTML]ffeb840.4122 |  [HTML]ffeb84± |  [HTML]ffeb840.0003 |  [HTML]70c27c0.3842 |  [HTML]70c27c± |  [HTML]70c27c0.0002
CvT-212DistilGPT2* |  [HTML]63be7b0.3670 |  [HTML]63be7b± |  [HTML]63be7b0.0004 |  [HTML]c2da810.4184 |  [HTML]c2da81± |  [HTML]c2da810.0004 |  [HTML]63be7b0.3910 |  [HTML]63be7b± |  [HTML]63be7b0.0004
Table 7: Mean NLG metric scores on the IU X-Ray test set with the labels of Chen et al. (2020). If available, the 95% confidence intervals are reported. $n=5$ indicates the mean over five training runs. * is the training run that scored the highest validation CIDEr score. Model | Natural language generation metrics
---|---
BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | CIDEr
R2Gen |  [HTML]fbb1780.4700 |  [HTML]fbb178 |  [HTML]fbb178 |  [HTML]fcc17b0.3040 |  [HTML]fcc17b |  [HTML]fcc17b |  [HTML]fbb0780.2190 |  [HTML]fbb078 |  [HTML]fbb078 |  [HTML]f8696b0.1650 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]f8696b0.1870 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]f97c6e0.3710 |  [HTML]f97c6e |  [HTML]f97c6e |  - |  |
CMN |  [HTML]fede810.4750 |  [HTML]fede81 |  [HTML]fede81 |  [HTML]f3e8840.3090 |  [HTML]f3e884 |  [HTML]f3e884 |  [HTML]fedc810.2220 |  [HTML]fedc81 |  [HTML]fedc81 |  [HTML]ede6830.1700 |  [HTML]ede683 |  [HTML]ede683 |  [HTML]fdcc7e0.1910 |  [HTML]fdcc7e |  [HTML]fdcc7e |  [HTML]ffeb840.3750 |  [HTML]ffeb84 |  [HTML]ffeb84 |  - |  |
Contrastive Attention |  [HTML]63be7b0.4920 |  [HTML]63be7b |  [HTML]63be7b |  [HTML]7bc57d0.3140 |  [HTML]7bc57d |  [HTML]7bc57d |  [HTML]fedc810.2220 |  [HTML]fedc81 |  [HTML]fedc81 |  [HTML]ffeb840.1690 |  [HTML]ffeb84 |  [HTML]ffeb84 |  [HTML]f4e8840.1930 |  [HTML]f4e884 |  [HTML]f4e884 |  [HTML]63be7b0.3810 |  [HTML]63be7b |  [HTML]63be7b |  - |  |
PPKED |  [HTML]bdd8810.4830 |  [HTML]bdd881 |  [HTML]bdd881 |  [HTML]63be7b0.3150 |  [HTML]63be7b |  [HTML]63be7b |  [HTML]dce1820.2240 |  [HTML]dce182 |  [HTML]dce182 |  [HTML]fdca7d0.1680 |  [HTML]fdca7d |  [HTML]fdca7d |  [HTML]fcb3790.1900 |  [HTML]fcb379 |  [HTML]fcb379 |  - |  |  |  [HTML]f8696b0.3510 |  [HTML]f8696b |  [HTML]f8696b
$\mathcal{M}^{2}$ Transformer Progressive |  [HTML]9fd07f0.4860 |  [HTML]9fd07f |  [HTML]9fd07f |  [HTML]63be7b0.3150 |  [HTML]63be7b |  [HTML]63be7b |  [HTML]dce1820.2240 |  [HTML]dce182 |  [HTML]dce182 |  [HTML]ffeb840.1690 |  [HTML]ffeb84 |  [HTML]ffeb84 |  [HTML]fee5820.1920 |  [HTML]fee582 |  [HTML]fee582 |  [HTML]fcb3790.3730 |  [HTML]fcb379 |  [HTML]fcb379 |  - |  |
CvT-212DistilGPT2 ($n\texttt{=}5$) |  [HTML]f8696b0.4620 |  [HTML]f8696b± |  [HTML]f8696b0.00038 |  [HTML]f8696b0.2945 |  [HTML]f8696b± |  [HTML]f8696b0.00030 |  [HTML]f8696b0.2141 |  [HTML]f8696b± |  [HTML]f8696b0.00030 |  [HTML]f86a6b0.1650 |  [HTML]f86a6b± |  [HTML]f86a6b0.00031 |  [HTML]fceb840.1924 |  [HTML]fceb84± |  [HTML]fceb840.00022 |  [HTML]f8696b0.3703 |  [HTML]f8696b± |  [HTML]f8696b0.00024 |  [HTML]fdd57f0.5868 |  [HTML]fdd57f± |  [HTML]fdd57f0.00335
CvT-212DistilGPT2* |  [HTML]fdce7e0.4732 |  [HTML]fdce7e± |  [HTML]fdce7e0.00045 |  [HTML]fcc07b0.3039 |  [HTML]fcc07b± |  [HTML]fcc07b0.00056 |  [HTML]63be7b0.2242 |  [HTML]63be7b± |  [HTML]63be7b0.00061 |  [HTML]63be7b0.1754 |  [HTML]63be7b± |  [HTML]63be7b0.00065 |  [HTML]63be7b0.1997 |  [HTML]63be7b± |  [HTML]63be7b0.00029 |  [HTML]eae5830.3758 |  [HTML]eae583± |  [HTML]eae5830.00045 |  [HTML]63be7b0.6935 |  [HTML]63be7b± |  [HTML]63be7b0.00495

###  7.6 Statistical analysis

The Confidence Intervals (CIs) in Tables 5, 6, 7, and 8 were found with
bootstrapping; the test set was resampled with replacement $1{,}000$ times,
where the size of each sample was equal to the size of the respective test set
(as indicated in Table 1) (Efron, 1979). The $95\%$ CIs were then calculated
from the mean scores of each sample. In Subsections 8.4, 8.5, and 8.6 we
perform statistical tests on the NLG metric scores where the checkpoint type
was the factor and the scores for each example of the MIMIC-CXR test set and
for each of the training runs were the dependent variables. For each test, we
used a p-value of 0.05. First, a Levene’s test revealed that the variances of
the scores were not homogeneous. This lead us to using a one-way Welch’s ANOVA
to determine if there was a significant difference between the scores of the
checkpoints. If a significant difference existed, Games-Howell tests were used
to perform a post hoc analysis.

##  8 Results and discussion

In this section, we first compare the final model shown in Figure 5, whose
encoder and decoder is warm started with the CvT-21 ImageNet-21K and
DistilGPT2 checkpoints, respectively, (i.e., CvT-212DistilGPT2) to current CXR
report generators in the literature (Subsection 8.1). Next, we evaluate the
diagnostic performance of CvT-212DistilGPT2 on 14 observations (Subsection
8.2). Following this, the case study in Subsection 8.3 provides insight as to
how CvT-212DistilGPT2 interprets a CXR when generating a report. In Subsection
8.4, we compare the CV checkpoints and determine which is best for warm
starting the encoder. Moreover, we compare the NLP checkpoints in Subsection
8.5 and determine which is best for warm starting the decoder. We also answer
RQ1-RQ4 in Subsections 8.4 and 8.5. Finally, we provide the reader with the
limitations of this investigation and several future recommendations.

###  8.1 Comparison to current methods

Here, we compare CvT-212DistilGPT2 to other CXR report generators in the
literature that were evaluated on the labels of Chen et al. (2020).444Scores
for the other CXR report generators in the literature were taken from their
respective articles. The NLG metric scores on the MIMIC-CXR test set are shown
in Table 5. CvT-212DistilGPT2 attained the highest mean scores for all NLG
metrics. This indicates that the reports generated by CvT-212DistilGPT2 are
more similar to the reports produced by radiologists than those of previous
approaches.

For the example-based CE scores in Table 6, $\mathcal{M}^{2}$ Transformer
Progressive had the highest recall—but also the lowest precision. The authors
of $\mathcal{M}^{2}$ Transformer Progressive speculated that the high false
positive rate was due to its generated reports having a longer length on
average than the ground-truth reports (Nooralahzadeh et al., 2021, Section 4).
CvT-212DistilGPT2 was able to attain the highest precision and the second
highest recall—leading to the highest F-1 score. This indicates that
CvT-212DistilGPT2 is more diagnostically accurate than previous approaches.

The NLG metric scores on the IU X-Ray test set are shown in Table 7.
CvT-212DistilGPT2 attained the highest mean scores for BLEU-3, BLEU-4, METEOR,
and CIDEr. However, Contrastive Attention attained the highest mean BLEU-1 and
ROUGE-L scores, while PPKED and $\mathcal{M}^{2}$ Transformer Progressive both
attained the highest mean BLEU-2 score. This could indicate that the training
set size of IU X-Ray is too small for CvT-212DistilGPT2. Alternatively,
concatenating the visual features of the two CXRs—as shown in Figure 3
(right)—may not be the best multi-source combination technique, in fact,
Libovickỳ et al. (2018) found it to be the worst technique for a Transformer
decoder.

###  8.2 Performance on different observations

Next, to get an indication of the diagnostic accuracy of CvT-212DistilGPT2 for
different abnormalities, we analyse its performance on each of the 14 CheXpert
observations. Scores for the label-based CE metrics are given in Table 8,
along with the macro- and micro-averaged scores over the observations.
CvT-212DistilGPT2 attained the highest precision and recall for support
devices—which is expected as it was one of the most frequent observations in
the training set. CvT-212DistilGPT2 performed well for pleural effusion and
cardiomegaly, of which both are frequently observed in the training set. For
no finding, CvT-212DistilGPT2 demonstrated high precision but poor recall.
Oppositely, it had a high recall and low precision for lung opacity. The
performance of CvT-212DistilGPT2 for consolidation, enlarged
cardiomediastinum, lung lesion, fracture, and pleural other was poor—likely
due to the infrequency of these observations in the training set. This was
also the case with previous CXR report generators, for example, the approach
by Liu et al. (2019, Table 3) performed poorly on the rarer abnormalities of
MIMIC-CXR. This suggests that the class imbalance of the observations in the
MIMIC-CXR training set leads to poor performance. Interestingly, recall was
higher for all observations except no finding and support devices, as
reflected by the macro- and micro-averaged scores for precision and recall.
This shows that CvT-212DistilGPT2 has a higher false positive rate but a lower
false negative rate.

Table 8: Label-based CE metric scores of CvT-212DistilGPT2 for each observation on the MIMIC-CXR test set from Chen et al. (2020). The 95% confidence intervals are reported for the averaged scores. The count for each observation indicates the number of times the observation was positive over all of the ground-truth reports of the training and test sets. The positive observation count for the training set is from Johnson et al. (2019b, Table 2). The positive observation count for the test set of Chen et al. (2020) was found with CheXbert (Smit et al., 2020). Observation | Count | CvT-212DistilGPT2
---|---|---
Train |  Test | Precision | Recall | F-1
No Finding |  75,163 |  323 |  [HTML]83c77d0.681 |  [HTML]83c77d |  [HTML]83c77d |  [HTML]f98c710.173 |  [HTML]f98c71 |  [HTML]f98c71 |  [HTML]fee4820.276 |  [HTML]fee482 |  [HTML]fee482
Support Dev. |  65,637 |  1,345 |  [HTML]63be7b0.795 |  [HTML]63be7b |  [HTML]63be7b |  [HTML]63be7b0.734 |  [HTML]63be7b |  [HTML]63be7b |  [HTML]63be7b0.763 |  [HTML]63be7b |  [HTML]63be7b
Pleural Effus. |  53,188 |  1,056 |  [HTML]c1d9810.454 |  [HTML]c1d981 |  [HTML]c1d981 |  [HTML]76c47d0.692 |  [HTML]76c47d |  [HTML]76c47d |  [HTML]aad3800.548 |  [HTML]aad380 |  [HTML]aad380
Lung Opacity |  50,916 |  1,392 |  [HTML]ffeb840.227 |  [HTML]ffeb84 |  [HTML]ffeb84 |  [HTML]b5d6800.551 |  [HTML]b5d680 |  [HTML]b5d680 |  [HTML]f5e8840.321 |  [HTML]f5e884 |  [HTML]f5e884
Atelectasis |  45,088 |  841 |  [HTML]eae5830.306 |  [HTML]eae583 |  [HTML]eae583 |  [HTML]feeb840.388 |  [HTML]feeb84 |  [HTML]feeb84 |  [HTML]eee6830.342 |  [HTML]eee683 |  [HTML]eee683
Cardiomegaly |  39,094 |  1,271 |  [HTML]b1d5800.512 |  [HTML]b1d580 |  [HTML]b1d580 |  [HTML]a3d17f0.591 |  [HTML]a3d17f |  [HTML]a3d17f |  [HTML]aad3800.549 |  [HTML]aad380 |  [HTML]aad380
Edema |  26,455 |  563 |  [HTML]feea830.224 |  [HTML]feea83 |  [HTML]feea83 |  [HTML]dae1820.468 |  [HTML]dae182 |  [HTML]dae182 |  [HTML]fbea840.303 |  [HTML]fbea84 |  [HTML]fbea84
Pneumonia |  15,769 |  165 |  [HTML]fa9e750.097 |  [HTML]fa9e75 |  [HTML]fa9e75 |  [HTML]fcc37c0.296 |  [HTML]fcc37c |  [HTML]fcc37c |  [HTML]fba7770.146 |  [HTML]fba777 |  [HTML]fba777
Consolidation |  10,487 |  176 |  [HTML]f98a710.063 |  [HTML]f98a71 |  [HTML]f98a71 |  [HTML]fbaa770.239 |  [HTML]fbaa77 |  [HTML]fbaa77 |  [HTML]fa91720.099 |  [HTML]fa9172 |  [HTML]fa9172
Pneumothorax |  9,317 |  75 |  [HTML]fcb4790.133 |  [HTML]fcb479 |  [HTML]fcb479 |  [HTML]e0e3830.455 |  [HTML]e0e383 |  [HTML]e0e383 |  [HTML]fcc37c0.206 |  [HTML]fcc37c |  [HTML]fcc37c
Enlarged Card. |  7,004 |  320 |  [HTML]f98c710.066 |  [HTML]f98c71 |  [HTML]f98c71 |  [HTML]f8696b0.093 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]f986700.077 |  [HTML]f98670 |  [HTML]f98670
Lung Lesion |  6,129 |  199 |  [HTML]f86a6b0.010 |  [HTML]f86a6b |  [HTML]f86a6b |  [HTML]f989710.167 |  [HTML]f98971 |  [HTML]f98971 |  [HTML]f86b6b0.019 |  [HTML]f86b6b |  [HTML]f86b6b
Fracture |  3,768 |  148 |  [HTML]f8696b0.007 |  [HTML]f8696b |  [HTML]f8696b |  [HTML]fdd37f0.333 |  [HTML]fdd37f |  [HTML]fdd37f |  [HTML]f8696b0.013 |  [HTML]f8696b |  [HTML]f8696b
Pleural Other |  1,961 |  122 |  [HTML]f86e6c0.016 |  [HTML]f86e6c |  [HTML]f86e6c |  [HTML]f989710.167 |  [HTML]f98971 |  [HTML]f98971 |  [HTML]f8706c0.030 |  [HTML]f8706c |  [HTML]f8706c
Macro-average | - | - |  [HTML]f7e9840.256 |  [HTML]f7e984± |  [HTML]f7e9840.0013 |  [HTML]fee9830.382 |  [HTML]fee983± |  [HTML]fee9830.0003 |  [HTML]faea840.307 |  [HTML]faea84± |  [HTML]faea840.0006
Micro-average | - | - |  [HTML]d0de820.398 |  [HTML]d0de82± |  [HTML]d0de820.0004 |  [HTML]cedd820.497 |  [HTML]cedd82± |  [HTML]cedd820.0004 |  [HTML]cddd820.442 |  [HTML]cddd82± |  [HTML]cddd820.0003

###  8.3 Case study

Here, we observe the attention weights of a subset of the cross-attention
heads of CvT-212DistilGPT2 as they collectively generate a word (or subword)
for the case study in Figure 6 (where subwords are separated by a vertical
bar). The cross-attention layers in the decoder are the means of conditioning
the report generation process on the visual features of the CXR, where each
layer comprises multiple cross-attention heads. Cross-attention heads are
parallel units within the layer that compute an output based on the
relationship between its two inputs, in this case, the visual features and the
previously generated words. Here, the visual features are a high-level
representation of the regions of the CXR. For a cross-attention head, a high-
level representation of both the visual features and words are compared via
their dot product, giving the attention weights. A higher weight indicates a
higher similarity between the CXR region and the word. Through this, we can
get an indication of what words and CXR regions the model has learnt to
correctly or incorrectly relate to one another. This case study was selected
to highlight both strengths and weaknesses of the model and of the current
paradigm of CXR report generators. It should be noted that higher layers
(e.g., Layer 6) are considered higher-level representations of lower layers
(e.g., Layer 1)—which typically model more primitive relationships. Because we
are interested in the anatomical aspects of the CXR, we direct the readers
attention to the higher layers.

Ground truthpatient is status post median sternotomy and cabg. left-sided
pacemaker device is noted with leadsterminating in the right atrium and right
ventricle unchanged. the heart remains mildly enlarged but stable. the aorta
is unfolded. there is mild pulmonary vascular congestion which is improved
when compared to the prior exam. no new focal consolidation.Generatedthe
patient is status post median stern—otomy and cab—g. left—-—sided
dual—-—ch—amber pac—emaker device is noted with leads terminating in the right
at—rium and right vent—ric—le. moderate enlarg—ement of the cardiac silhouette
is unchanged. the medi—ast—inal and hilar cont—ours are similar. pulmonary
vas—cul—ature is not eng—or—ged. no focal consolidation ple—ural eff—usion or
pneum—oth—or—ax is present.Layer 6 Head 12Layer 5Head 6Layer 4Head 6Layer
3Head 7Layer 2Head 10Layer 1Head
1cab—pac——riumsilhouettepneum——cul—hilar—usionCXRLow cross-attention
weightHigh cross-attention weight

Figure 6:  Case study of CXR a0578edb-12a640ca-1ddab351-089c4d4c-00bb6f19 from
study s54265960 of patient p18615099 from the MIMIC-CXR test set. The ‘ground-
truth’ report was produced by a radiologist, while the ‘generated’ report was
produced by CvT-212DistilGPT2. The CXR displayed has been pre-processed for
testing. Subwords produced by DistilGPT2 are separated by a vertical bar. Each
cross-attention weight matrix is min-max normalised and then scaled element
wise using $\exp\left(1-a^{-1}\right)$ for ease of interpretation (where $a$
is an attention weight). A beam size of one (i.e., a greedy search) was used
to produce the report. One cross-attention head from each layer of DistilGPT2
was selected for analysis.

Comparing the generated and ground-truth reports, CvT-212DistilGPT2 correctly
predicted that the patient has had Coronary Artery Bypass Graft (CABG) surgery
via a median sternotomy incision, although it is unclear from both the
generated and ground-truth reports which of the coronary arteries were
bypassed. It can be seen that the heads attended to the vascular clips when
generating the subword ‘cab—’—a key indicator of CABG. Next, CvT-212DistilGPT2
correctly identified the left-side pacemaker with its leads terminating in the
right atrium and ventricle. Note that in the ground-truth report, it was
mentioned that the positioning of the leads remained ‘unchanged’—something
that is impossible for CvT-212DistilGPT2 to infer as it has not observed the
previous study. It can be seen that Head 12 of Layer 6 and Head 1 of Layer 1
attend to the pacemaker when generating the subword ‘pac—’. However, not one
of the presented heads attend to the lead in the right atrium when generating
the subword ‘—rium’.

CvT-212DistilGPT2 indicated mild cardiomegaly with the phrase “a moderate
enlargement of the cardiac silhouette”. However, CvT-212DistilGPT2 specified
that the mild cardiomegaly was unchanged, referring to a previous, unobserved
study. This is because CvT-212DistilGPT2 has learnt to often describe
abnormalities in the context of disease progression due to the ground-truth
reports of the training set frequently referring to previous studies. When
generating the word ‘silhouette’, Head 6 of Layers 4 and 5 attend to regions
of the cardiac silhouette. Moreover, CvT-212DistilGPT2 missed the widened and
decreased curvature of the aortic arch indicating that the aorta is unfolded.
CvT-212DistilGPT2 also mentioned that the mediastinal and hilar contours are
similar, which is not a valid interpretation due to the word similar. Not one
of the heads in Figure 6 attends to the right or left hilar points when
generating the word ‘hilar’.

CvT-212DistilGPT2 also incorrectly predicted no pulmonary vascular congestion,
while it is reported as mild and an improvement over a previous study in the
ground-truth report. Head 7 of Layer 3 does pay some attention to the
pulmonary vessels when generating the subword ‘—cul—’. However, it is
difficult to determine an increased prominence of the pulmonary vessels
without a prior normal CXR of the patient. Lastly, CvT-212DistilGPT2 mentions
which abnormalities were not present in the CXR; no focal consolidation,
pleural effusion, or pneumothorax. Some of the presented heads attend to
regions of the pleura when generating the subword ‘—usion’, especially Head 6
of Layer 5. When looking for signs of pneumothorax, the heads did not attend
to the position of the trachea, however, some attention was paid to each of
the hemidiaphragms.

Figure 7: NLG metric scores for each CV checkpoint when warm starting the
encoder. DistilGPT2 was used to warm start the decoder. The black dots and the
error bars indicate the mean and standard deviation over the training runs,
respectively. Figure 8: Label-based CE scores for each CV checkpoint when warm
starting the encoder. DistilGPT2 was used to warm start the decoder. The black
dots and the error bars indicate the mean and standard deviation over the
training runs, respectively.

###  8.4 Best checkpoint for warm starting the encoder

In this subsection, we determine which CV checkpoint is best for warm starting
the encoder. To succeed, the pre-training task of the CV checkpoint must be
transferable to that of extracting salient visual features from CXRs. To make
this investigation more tractable, the following methodology was adopted; for
training, we use the 50K subset of MIMIC-CXR’s training set described in
Subsection 8.6; we make the assumption that the best CV checkpoint is
independent of the NLP checkpoint used to warm start the decoder. Following
this, the decoder of each encoder-to-decoder model is warm started with
DistilGPT2. Eight training runs are performed for each CV checkpoint to
account for the variability introduced by randomly initialising the parameters
of $\boldsymbol{P}$ and the cross-attention modules of the decoder. The scores
for the NLG metrics are shown in Figure 7, while the label-based CE metrics
are shown in Figure 8. A one-way Welch’s ANOVA revealed a statistically
significant difference between the NLG metric scores of each CV checkpoint,
showing that they have an effect on performance.

CvT-13 attained the highest mean BLEU-4 score; CvT-21 the highest mean METEOR
and CIDEr scores; BEiT the highest mean ROUGE-L score; ResNet-101 the highest
macro-averaged precision; DeiT the highest macro-averaged recall and micro-
averaged precision and recall. Between the two best performing checkpoints,
namely, CvT-21 and DeiT, CvT-21 attained higher mean BLEU-4, METEOR, ROUGE-L,
CIDEr, and macro-averaged precision scores, while DeiT achieved a higher
macro-averaged recall and micro-averaged precision and recall. However, Games-
Howell tests revealed no significant difference between their NLG metric
scores. As CvT-21 and DeiT cannot be separated based on their performance, we
instead select based on their parameter efficiency. Hence, we select CvT-21
over DeiT as it consumes drastically fewer parameters (31.6M vs. 86M).

Comparing the CNN ImageNet-1K checkpoints, EfficientNet (B6 and B7) attained
the highest scores for each metric (except for macro-averaged precision). In
fact, Games-Howell tests revealed a significant difference between the METEOR
scores of EfficientNet-B7 and each of the DenseNets. This is likely due to the
improvements proposed by Tan and Le (2019) to scale the depth, width, and
input image size of the network to more efficiently use the networks
parameters. Next, we compare the DenseNet-121 ImageNet-1K checkpoint to its
domain-specific equivalent, namely CheXNet. CheXNet demonstrates a marked
improvement, attaining higher mean scores for all metrics with Games-Howell
tests revealing a statistically significant difference between their NLG
metric scores.

Answer to RQ4 (for CV checkpoints): CheXNet demonstrates that there is a clear
advantage to warm starting the encoder with a domain-specific CV checkpoint
over a general-domain CV checkpoint.

ViT lacked performance, attaining lower mean scores for all metrics (except
macro-averaged precision) than EfficientNet-B5, B6, and B7. Moreover, Games-
Howell tests confirmed a significant difference between the METEOR scores of
EfficientNet-B5, B6, and B7 versus ViT. Its hard to pinpoint the cause of the
performance difference as many variables exist, such as the number of
parameters and the training dataset. Another reason could be due to ViTs
inability to model intra-patch visual features with its self-attention
weights. The performance of DeiT was on par with EfficientNet, with DeiT
attaining a higher macro- and micro-averaged precision and recall than
EfficientNet-B7, whereas EfficientNet-B7 attained higher mean BLEU-4, METEOR,
ROUGE-L, and CIDEr scores. Additionally, Games-Howell tests revealed no
significant difference between the NLG metric scores of the two. The
performance of BEiT was also on par with EfficientNet, with BEiT attaining
higher mean BLEU-4, METEOR, ROUGE-L, and micro-averaged precision than
EfficientNet-B7, whereas EfficientNet-B7 attained higher CIDEr, macro-averaged
precision and recall, and micro-averaged recall. Moreover, Games-Howell tests
revealed a significant difference between only the ROUGE-L scores of BEiT and
EfficientNet-B4, B5, and B6 (in favour of BEiT), but not for B7. While having
a similar architecture to ViT, the pre-training tasks of DeiT and BEiT enable
them to perform comparatively to EfficientNet.

Finally, we analyse adaptations of the Transformer, namely XCiT and CvT.
XCiT-S-24-P8 attained higher mean ROUGE-L and CIDEr scores than the
EfficientNets, as well as a higher macro- and micro-averaged precision.
However, Games-Howell tests only revealed significant difference for ROUGE-L
between EfficientNet (B4, B5, and B6), but not for B7 (in favour of
XCiT-S-24-P8). Moreover, EfficientNet-B6 and B7 attained higher mean BLEU-4
and METEOR scores as well as a higher macro- and micro-averaged recall. CvT-21
attained higher mean scores than the EfficientNets for all metrics, indicating
that CvT-21 is the only Transformer-based CV checkpoint that can consistently
outperform EfficientNet. However, Games-Howell tests revealed no statistically
significant difference between the NLG metric scores of EfficientNet-B7 and
CvT-21.

Answer to RQ1: Currently, it seems that Transformers require pre-training with
a task such as MIM or distillation to perform comparatively to CNNs. However,
the performance of CvT suggests that incorporating convolutional layers into
the Transformer alleviates it from having to learn the inductive bias of local
spatial feature processing, allowing it to outperform EfficientNet.

###  8.5 Best checkpoint for warm starting the decoder

This subsection is identical in methodology to the previous subsection, except
that now we determine which NLP checkpoint is best for warm starting the
decoder. A ResNet-101 ImageNet-1K checkpoint was employed to warm start the
encoder of each encoder-to-decoder model following the assumption that the
best NLP checkpoint is independent of the CV checkpoint used to warm start the
encoder. To succeed, the pre-training task of an NLP checkpoint must be
transferable to that of generating reports from visual features. The scores
for the NLG metrics are shown in Figure 9, while the label-based CE metrics
are shown in Figure 10. A one-way Welch’s ANOVA revealed a statistically
significant difference between the NLG metric scores of each checkpoint,
showing that they have an effect on performance.

Figure 9: NLG metric scores for each NLP checkpoint when warm starting the
decoder. A ResNet-101 ImageNet-1K checkpoint was used to warm start the
encoder. The black dots and the error bars indicate the mean and standard
deviation over the training runs, respectively. Figure 10: Label-based CE
scores for each NLP checkpoint when warm starting the decoder. A ResNet-101
ImageNet-1K checkpoint was used to warm start the encoder. The black dots and
the error bars indicate the mean and standard deviation over the training
runs, respectively.

DistilGPT2 attained the highest mean BLEU-4, METEOR, ROUGE-L, and CIDEr scores
while GPT2 attained the highest macro-averaged recall and micro-averaged
precision and recall. Furthermore, Games-Howell tests revealed a statistically
significant difference between the scores of both GPT2 and DistilGPT2 versus
the remaining checkpoints for both METEOR and CIDEr, indicating that GPT2 and
DistilGPT2 are the most suitable checkpoints for warm starting the decoder.
Comparing GPT2 to DistilGPT2, Games-Howell tests revealed no significant
difference between their NLG metric scores, showing that distilling GPT2 to
66% of its parameters has no effect on performance.555Games-Howell tests also
revealed no significant difference between the NLG metric scores of BERT and
DistilBERT, again demonstrating that distillation does not impact performance.
Due to its parameter efficiency, we select DistilGPT2 over GPT2 as the best
checkpoint for warm starting the decoder.

Answer to RQ2: GPT2 demonstrated that an NLP checkpoint can be effectively
fine-tuned to model not only natural language, but also visual features (i.e.,
a pre-trained NLP checkpoint is able to outperform a randomly initialised
decoder).

Comparing BERT and GPT2, Games-Howell tests revealed a significant difference
between their METEOR, ROUGE-L, and CIDEr scores (in favour of GPT2). GPT2 also
attained higher label-based CE scores than BERT. This finding is opposite to
that of Rothe et al. (2020, BERT2BERT vs. BERT2GPT2); however, the encoder in
their case was an NLP checkpoint rather than a CV checkpoint. Some of the main
differences between BERT and GPT2 include their number of parameters (110M for
BERT vs. 124M for GPT2), their training data (BookCorpus and English Wikipedia
for BERT vs. WebText for GPT2), their vocabulary size and source (30K formed
from BookCorpus and English Wikipedia for BERT vs. 50K formed from WebText for
GPT2), and their token embeddings (BERT uses WordPiece (Devlin et al., 2019)
while GPT2 uses byte-pair encodings (Radford et al., 2019)). However, we
hypothesise that these differences are relatively minor and have only a slight
effect on the performance difference. Instead, we speculate that their pre-
training tasks are the main cause of the performance difference and suggest
that GTP2’s language modelling pre-training task is better than BERT’s MLM and
NSP pre-training tasks for CXR report generation.

Answer to RQ3: GPT2 is able to outperform BERT. Our hypothesis as to why this
is the case is due to their different pre-training tasks (MLM and NSP vs.
language modelling).

Amongst the domain-specific NLU checkpoints, PubMedBERT performed best. Games-
Howell tests revealed a significant difference between the METEOR and ROUGE-L
scores of BERT and PubMedBERT (in favour of PubMedBERT). PubMedBERT also
attained higher label-based CE metric scores. This indicates that a biomedical
NLU checkpoint is better than a general-domain NLU checkpoint for warm
starting the decoder. Opposite to PubMedBERT, BioBERT lacked performance, with
Games-Howell tests revealing a significant difference between all of the NLG
metric scores of BioBERT and PubMedBERT (in favour of PubMedBERT).
Additionally, PubMedBERT achieved higher label-based CE metric scores. Unlike
BioBERT, PubMedBERT was pre-trained from scratch on PubMed and PMC and has a
domain-specific vocabulary, both of which are cited as the reason why
PubMedBERT outperforms BioBERT on biomedical NLU tasks (Gu et al., 2020).

For the EHR NLU checkpoints, BlueBERT was outperformed by both BERT and
PubMedBERT, attaining lower mean scores for each metric (except for ROUGE-L,
where BlueBERT attained a higher mean ROUGE-L score than BERT).666BlueBERT
outperformed ClinicalBERT, attaining higher mean scores for each metric, with
Games-Howell tests revealing a significant difference between their NLG metric
scores. Furthermore, Games-Howell tests revealed a significant difference
between all the NLG metric scores of BlueBERT and PubMedBERT, and a
significant difference between the METEOR and CIDEr scores of BlueBERT and
BERT. Several factors could be causing the deficit in performance to the
general-domain and biomedical NLU checkpoints. Unlike BERT or PubMedBERT,
ClinicalBERT and BlueBERT are not pre-trained from scratch, rather, each
involves three stages of pre-training, as shown in Table 4. Moreover, MIMIC-
III is significantly smaller than the datasets used to pre-train BERT and
PubMedBERT (Table 1). Finally, both ClinicalBERT and BlueBERT do not have
domain-specific vocabularies.

Answer to RQ4 (for NLP checkpoints): While PubMedBERT outperformed BERT, the
EHR NLU checkpoints—which are closer in domain to CXR reports—did not. This is
more likely due to factors other than the choice of domain, such as the size
of the pre-training datasets and the use of domain-specific vocabularies.

Figure 11: Different subset sizes of MIMIC-CXR’s training set.
ResNet-1012DistilBERT was used for the comparison. The black dots and the
error bars indicate the mean and standard deviation over the training runs,
respectively.

###  8.6 No. of training examples

In order to make the experiments in Subsections 8.4 and 8.5 tractable, we
sought to find a subset size of the MIMIC-CXR training set that was large
enough that it produced similar results to the full training set, yet small
enough to reduce training time. Hence, we investigate different subset sizes
of the MIMIC-CXR training set, as presented in Figure 11. For each size, each
training run used the same random sample (without replacement) of the MIMIC-
CXR training set. We selected ResNet-1012DistilBERT as the model. Eight
training runs were conducted for each training size, where only teacher
forcing was employed during fine-tuning. A one-way Welch’s ANOVA revealed a
significant difference between the scores of each NLG metric over the
different sizes. Games-Howell tests revealed a significant difference between
the scores for 10K and 20K and the full training set. This was not the case
with 50K and 100K and the full training set. Hence, we use the 50K subset of
MIMIC-CXR’s training set in Subsections 8.4 and 8.5, as it is the smallest
tested size with no statistically significant difference to the full training
set.

##  9 Limitations and future recommendations

There are several limitations of our work which lead to recommendations we
propose for future investigation:

  1. 1.

We did not consider every combination of encoder and decoder checkpoint,
instead, we assumed that the choice of encoder does not impact the decoder and
vice versa. This assumption may be wrong and could be considered for future
investigation.

  2. 2.

We did not consider CXRs in DICOM format. The formats that we used for the
CXRs (JPEG and PNG) meant that the pixel depth was 8-bit, rather than the
12-bits available from the DICOM format. Reducing the quantisation error could
improve performance.

  3. 3.

Downsampling each CXR to $384\times 384$ increases the risk of missing fine
details. Hence, the downsampled CXRs should be closer in resolution to the
original CXRs.

  4. 4.

With MIMIC-CXR, reports are typically generated from only a single CXR of a
study. However, studies with multiple CXRs often have multiple views. Certain
abnormalities are easier to detect when interpreting both frontal and lateral
views simultaneously. A CXR report generator that can accommodate a variable
number of CXRs may improve performance.

  5. 5.

Radiologists frequently reference previous studies of a patient in the ground-
truth reports. We hypothesise that observing both the generated reports and
the visual features of previous studies will further improve CXR report
generation. This will be especially important for monitoring disease
progression.

  6. 6.

Motivated by Section 8.2, we hypothesise that an under or over sampling
technique could reduce the impact of the class imbalance of the MIMIC-CXR
training set.

  7. 7.

There also exists more recent CV checkpoints which are not yet open-source
that may outperform CvT. For example, a ViT checkpoint was proposed that had
the image patches as input replaced with the output of a small number of
stacked stride-two 3$\times$3 convolutions ($\textrm{ViT}_{c}$). By injecting
this inductive bias towards local spatial feature processing,
$\textrm{ViT}_{c}$ is able to outperform ViT and EfficientNet (Xiao et al.,
2021).

  8. 8.

As demonstrated by CheXNet outperforming the DenseNet-121 ImageNet-21K
checkpoint, a domain-specific CvT-21 checkpoint may be more apt for warm
starting the encoder than the CvT-21 ImageNet-21K checkpoint.

  9. 9.

The recent rise of Large Language Models (LLMs) has produced Transformer
decoder checkpoints with considerable capabilities. An investigation into LLMs
as the decoder for CXR report generation should be considered.

  10. 10.

This study indicates that the following modifications to DistilGPT2 checkpoint
should be investigated: i) pre-training on PubMed and PMC from scratch before
distillation, and ii) using a domain-specific vocabulary.

  11. 11.

An important consideration is the level of diagnostic accuracy that a CXR
report generator must attain before retrospective or prospective clinical
trials are deemed appropriate. Understanding the diagnostic accuracy of humans
will help establish this, for example, a study by Satia et al. (2013, Table 1)
found that the diagnostic accuracy of clinicians ranged from 66% to 83%
depending on expertise. As demonstrated in Table 8, current CXR report
generators demonstrate poor diagnostic accuracy for multiple abnormalities,
indicating that further research and development is required before reaching
such a threshold.

  12. 12.

We were not able to include Receiver Operating Characteristics (ROC) curves.
In future work, we will consider ROC curves to provide a more thorough
investigation.

  13. 13.

Metrics that evaluate the clinical decisions inferred from the generated
reports should be considered—as clinical decisions ultimately dictate patient
care.

##  10 Conclusion

In this study, we investigate warm starting the encoder and decoder of a CXR
report generator with recent publicly available CV and NLP checkpoints. Our
investigation led us to the CvT-21 ImageNet-21K checkpoint—which possesses the
advantages of both CNNs and Transformers—as the best CV checkpoint for warm
starting the encoder. Moreover, we find that DistilGPT2—a distilled general-
domain NLG checkpoint—is best for warm starting the decoder. The results
indicate that the reports generated by CvT-212DistilGPT2 are more
diagnostically accurate and have a higher similarity to radiologist reports
than previous approaches. Compared to $\mathcal{M}^{2}$ Transformer
Progressive, CvT-212DistilGPT2 attained an improvement of 8.3% for CE F-1,
1.8% for BLEU-4, 1.6% for ROUGE-L, and 1.0% for METEOR.

Our investigation also reveals several important findings about warm starting
the encoder and decoder of a CXR report generator. The first is that a
Transformer-based CV checkpoint that incorporates convolutional layers is
better than a CNN checkpoint for warm starting the encoder. Moreover, an NLP
checkpoint can be effectively fine-tuned to model not only natural language
but also visual features. Furthermore, we find that GPT2 is better for warm
starting the decoder than BERT. Finally, we found that domain-specific
checkpoints are better for warm starting than general-domain checkpoints—if
the size and quality of the pre-training dataset is sufficient. In general,
the best checkpoint for a task depends on multiple variables; for example, the
pre-training task, the size and quality of the dataset, the vocabulary, and
the model architecture.

Our results indicate that leveraging warm starting improves CXR report
generation. The future outlook on CXR report generation is promising; a CXR
report generator that has been clinically validated through retrospective and
prospective trails—that also meets regulatory requirements—could have a
significant impact on radiology. Automatic CXR report generation could provide
more consistent and reliable reporting, as well as cheaper running costs. It
could also reduce the burden placed on overworked radiologists. A secondary
use for such a technology could be to conduct a retrospective analysis on
previous reports, which in turn could be used for other tasks such as question
answering or population research. CvT-212DistilGPT2 and its MIMIC-CXR
checkpoint are available at https://github.com/aehrc/cvt2distilgpt2.

## Acknowledgments

This work was partially funded by CSIRO’s Machine Learning and Artificial
Intelligence Future Science Platform.

## References

  * Alexander et al. (2020) Alexander, A., Jiang, A., Ferreira, C., Zurkiya, D., 2020\.  An Intelligent Future for Medical Imaging: A Market Outlook on Artificial Intelligence for Medical Imaging.  Journal of the American College of Radiology 17, 165–170.  doi:10.1016/j.jacr.2019.07.019.
  * Alfarghaly et al. (2021) Alfarghaly, O., Khaled, R., Elkorany, A., Helal, M., Fahmy, A., 2021.  Automated radiology report generation using conditioned transformers.  Informatics in Medicine Unlocked 24, 100557.  doi:10.1016/j.imu.2021.100557.
  * Alsentzer et al. (2019) Alsentzer, E., Murphy, J., Boag, W., Weng, W.H., Jindi, D., Naumann, T., McDermott, M., 2019.  Publicly Available Clinical BERT Embeddings, in: Proceedings of the 2nd Clinical Natural Language Processing Workshop, Association for Computational Linguistics, Minneapolis, Minnesota, USA. pp. 72–78.  URL: https://aclanthology.org/W19-1909, doi:10.18653/v1/W19-1909.
  * Ayesha et al. (2021) Ayesha, H., Iqbal, S., Tariq, M., Abrar, M., Sanaullah, M., Abbas, I., Rehman, A., Niazi, M.F.K., Hussain, S., 2021.  Automatic medical image interpretation: State of the art and future directions.  Pattern Recognition 114, 107856\.  doi:10.1016/j.patcog.2021.107856.
  * Babar et al. (2021) Babar, Z., van Laarhoven, T., Zanzotto, F.M., Marchiori, E., 2021\.  Evaluating diagnostic content of AI-generated radiology reports of chest X-rays.  Artificial Intelligence in Medicine 116, 102075.  doi:10.1016/j.artmed.2021.102075.
  * Balabanova et al. (2005) Balabanova, Y., Coker, R., Fedorin, I., Zakharova, S., Plavinskij, S., Krukov, N., Atun, R., Drobniewski, F., 2005\.  Variability in interpretation of chest radiographs among Russian clinicians and implications for screening programmes: observational study.  BMJ 331, 379–382.  doi:10.1136/bmj.331.7513.379.
  * Banerjee and Lavie (2005) Banerjee, S., Lavie, A., 2005\.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments, in: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Association for Computational Linguistics, Ann Arbor, Michigan. pp. 65–72.  URL: https://aclanthology.org/W05-0909.
  * Bao et al. (2021) Bao, H., Dong, L., Wei, F., 2021.  BEiT: BERT Pre-Training of Image Transformers.  arXiv:2106.08254 [cs.CV] arXiv:2106.08254.
  * Beltagy et al. (2019) Beltagy, I., Lo, K., Cohan, A., 2019.  SciBERT: A Pretrained Language Model for Scientific Text, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics. pp. 3615--3620.  doi:10.18653/v1/d19-1371.
  * Chen et al. (2015) Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.L., 2015\.  Microsoft COCO captions: Data collection and evaluation server.  arXiv1504.00325 [cs.CV] .
  * Chen et al. (2021) Chen, Z., Shen, Y., Song, Y., Wan, X., 2021.  Cross-modal Memory Networks for Radiology Report Generation, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics. pp. 5904--5914.  doi:10.18653/v1/2021.acl-long.459.
  * Chen et al. (2020) Chen, Z., Song, Y., Chang, T., Wan, X., 2020.  Generating Radiology Reports via Memory-driven Transformer, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics. pp. 1439--1449.  doi:10.18653/v1/2020.emnlp-main.112.
  * Cornia et al. (2020) Cornia, M., Stefanini, M., Baraldi, L., Cucchiara, R., 2020\.  Meshed-Memory Transformer for Image Captioning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10578--10587.
  * Demner-Fushman et al. (2015) Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R., McDonald, C.J., 2015\.  Preparing a collection of radiology examinations for distribution and retrieval.  Journal of the American Medical Informatics Association 23, 304--310.  doi:10.1093/jamia/ocv080.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L., 2009.  ImageNet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, IEEE. pp. 248--255.  doi:10.1109/cvpr.2009.5206848.
  * Devlin et al. (2019) Devlin, J., Chang, M., Lee, K., Toutanova, K., 2019\.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota. pp. 4171--4186.  URL: https://www.aclweb.org/anthology/N19-1423, doi:10.18653/v1/N19-1423.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N., 2020\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  arXiv:2010.11929 [cs.CV] arXiv:2010.11929.
  * Efron (1979) Efron, B., 1979.  Bootstrap Methods: Another Look at the Jackknife.  The Annals of Statistics 7.  doi:10.1214/aos/1176344552.
  * El-Nouby et al. (2021) El-Nouby, A., Touvron, H., Caron, M., Bojanowski, P., Douze, M., Joulin, A., Laptev, I., Neverova, N., Synnaeve, G., Verbeek, J., Jegou, H., 2021.  XCiT: Cross-Covariance Image Transformers.  arXiv:2106.09681 [cs.CV] arXiv:2106.09681.
  * European Society of Radiology (2011) European Society of Radiology, 2011.  Good practice for radiological reporting. Guidelines from the European Society of Radiology (ESR).  Insights into Imaging 2, 93--96.  doi:10.1007/s13244-011-0066-7.
  * Fricke (2018) Fricke, S., 2018.  Semantic Scholar.  Journal of the Medical Library Association: JMLA 106, 145--147.  URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5764585/, doi:10.5195/jmla.2018.280.
  * Gu et al. (2020) Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H., 2020.  Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing.  arXiv:2007.15779 [cs.CL] arXiv:2007.15779.
  * Harolds et al. (2016) Harolds, J.A., Parikh, J.R., Bluth, E.I., Dutton, S.C., Recht, M.P., 2016.  Burnout of Radiologists: Frequency, Risk Factors, and Remedies: A Report of the ACR Commission on Human Resources.  Journal of the American College of Radiology 13, 411--416.  doi:10.1016/j.jacr.2015.11.003.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep Residual Learning for Image Recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE. pp. 770--778.  doi:10.1109/cvpr.2016.90.
  * Huang et al. (2017) Huang, G., Liu, Z., Maaten, L.V.D., Weinberger, K.Q., 2017\.  Densely Connected Convolutional Networks, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE. pp. 4700--4708.  doi:10.1109/cvpr.2017.243.
  * Irvin et al. (2019) Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., Seekins, J., Mong, D.A., Halabi, S.S., Sandberg, J.K., Jones, R., Larson, D.B., Langlotz, C.P., Patel, B.N., Lungren, M.P., Ng, A.Y., 2019\.  CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison, in: Proceedings of the AAAI Conference on Artificial Intelligence, Association for the Advancement of Artificial Intelligence (AAAI). pp. 590--597.  doi:10.1609/aaai.v33i01.3301590.
  * Ji et al. (2021) Ji, J., Du, Z., Zhang, X., 2021.  Divergent-convergent attention for image captioning.  Pattern Recognition 115, 107928\.  doi:10.1016/j.patcog.2021.107928.
  * Jing et al. (2019) Jing, B., Wang, Z., Xing, E., 2019.  Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics. pp. 6570--6580.  doi:10.18653/v1/p19-1657.
  * Jing et al. (2018) Jing, B., Xie, P., Xing, E., 2018.  On the Automatic Generation of Medical Imaging Reports, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics. pp. 2577--2586.  doi:10.18653/v1/p18-1240.
  * Johnson et al. (2019a) Johnson, A.E.W., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., ying Deng, C., Mark, R.G., Horng, S., 2019a.  MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.  Scientific Data 6.  doi:10.1038/s41597-019-0322-0.
  * Johnson et al. (2019b) Johnson, A.E.W., Pollard, T.J., Greenbaum, N.R., Lungren, M.P., ying Deng, C., Peng, Y., Lu, Z., Mark, R.G., Berkowitz, S.J., Horng, S., 2019b.  MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs.  arXiv:1901.07042 [cs.CV] arXiv:1901.07042.
  * Johnson et al. (2016) Johnson, A.E.W., Pollard, T.J., Shen, L., Lehman, L.H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., Mark, R.G., 2016\.  MIMIC-III, a freely accessible critical care database.  Scientific Data 3, 160035\.  URL: https://doi.org/10.1038/sdata.2016.35, doi:10.1038/sdata.2016.35.
  * Kanne et al. (2005) Kanne, J.P., Thoongsuwan, N., Stern, E.J., 2005.  Common Errors and Pitfalls in Interpretation of the Adult Chest Radiograph.  Clinical Pulmonary Medicine 12, 97--114.  doi:10.1097/01.cpm.0000156704.33941.e2.
  * Kaur et al. (2021) Kaur, N., Mittal, A., Singh, G., 2021.  Methods for automatic generation of radiological reports of chest radiographs: a comprehensive survey.  Multimedia Tools and Applications doi:10.1007/s11042-021-11272-6.
  * Ke et al. (2021) Ke, A., Ellsworth, W., Banerjee, O., Ng, A.Y., Rajpurkar, P., 2021.  CheXtransfer: performance and parameter efficiency of ImageNet models for chest X-Ray interpretation, in: Proceedings of the Conference on Health, Inference, and Learning, ACM. pp. 116--124.  doi:10.1145/3450439.3451867.
  * Kelly (2012) Kelly, B., 2012.  The chest radiograph.  The Ulster medical journal 81, 143--148.  URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3632825/.
  * Kelly et al. (2019) Kelly, C.J., Karthikesalingam, A., Suleyman, M., Corrado, G., King, D., 2019.  Key challenges for delivering clinical impact with artificial intelligence.  BMC Medicine 17.  doi:10.1186/s12916-019-1426-2.
  * Kolesnikov et al. (2020) Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., Houlsby, N., 2020.  Big Transfer (BiT): General Visual Representation Learning, in: Computer Vision – ECCV 2020. Springer International Publishing, pp. 491--507.  doi:10.1007/978-3-030-58558-7_29.
  * Krupinski et al. (2010) Krupinski, E.A., Berbaum, K.S., Caldwell, R.T., Schartz, K.M., Kim, J., 2010.  Long Radiology Workdays Reduce Detection and Accommodation Accuracy.  Journal of the American College of Radiology 7, 698--704.  doi:10.1016/j.jacr.2010.03.004.
  * Lee et al. (2019) Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J., 2019.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics doi:10.1093/bioinformatics/btz682.
  * Leeuwen et al. (2021) Leeuwen, K.G.V., de Rooij, M., Schalekamp, S., van Ginneken, B., Rutten, M.J.C.M., 2021.  How does artificial intelligence in radiology improve efficiency and health outcomes?  Pediatric Radiology doi:10.1007/s00247-021-05114-8.
  * Lewis et al. (2020) Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L., 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language generation, translation, and comprehension, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online. pp. 7871--7880.  URL: https://aclanthology.org/2020.acl-main.703, doi:10.18653/v1/2020.acl-main.703.
  * Li et al. (2018) Li, C.Y., Liang, X., Hu, Z., Xing, E.P., 2018.  Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation, in: Proceedings of the 32nd International Conference on Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY, USA. p. 1537–1547.
  * Li et al. (2021) Li, J., Zhao, G., Tao, Y., Zhai, P., Chen, H., He, H., Cai, T., 2021.  Multi-task contrastive learning for automatic CT and X-ray diagnosis of COVID-19.  Pattern Recognition 114, 107848\.  doi:10.1016/j.patcog.2021.107848.
  * Li et al. (2023) Li, Z., Li, H., Ralescu, A.L., Dillman, J.R., Parikh, N.A., He, L., 2023.  A novel collaborative self-supervised learning method for radiomic data.  NeuroImage 277, 120229\.  URL: https://www.sciencedirect.com/science/article/pii/S1053811923003804, doi:https://doi.org/10.1016/j.neuroimage.2023.120229.
  * Libovickỳ et al. (2018) Libovickỳ, J., Helcl, J., Mareček, D., 2018.  Input Combination Strategies for Multi-Source Transformer Decoder, in: Proceedings of the Third Conference on Machine Translation (WMT), Association for Computational Linguistics. pp. 253--260.
  * Lin and Och (2004) Lin, C., Och, F.J., 2004.  Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics, in: Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), Barcelona, Spain. pp. 605--612.  URL: https://aclanthology.org/P04-1077, doi:10.3115/1218955.1219032.
  * Liu et al. (2021a) Liu, F., Wu, X., Ge, S., Fan, W., Zou, Y., 2021a.  Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13753--13762.
  * Liu et al. (2021b) Liu, F., Yin, C., Wu, X., Ge, S., Zhang, P., Sun, X., 2021b.  Contrastive Attention for Automatic Chest X-ray Report Generation, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Association for Computational Linguistics, Online. pp. 269--280.  URL: https://aclanthology.org/2021.findings-acl.23, doi:10.18653/v1/2021.findings-acl.23.
  * Liu et al. (2019) Liu, G., Hsu, T.H., McDermott, M., Boag, W., Weng, W., Szolovits, P., Ghassemi, M., 2019.  Clinically Accurate Chest X-Ray Report Generation, in: Doshi-Velez, F., Fackler, J., Jung, K., Kale, D., Ranganath, R., Wallace, B., Wiens, J. (Eds.), Proceedings of the 4th Machine Learning for Healthcare Conference, PMLR, Ann Arbor, Michigan. pp. 249--269.  URL: http://proceedings.mlr.press/v106/liu19a.html.
  * Liu et al. (2017) Liu, J.X., Goryakin, Y., Maeda, A., Bruckner, T., Scheffler, R., 2017.  Global Health Workforce Labor Market Projections for 2030.  Human Resources for Health 15.  doi:10.1186/s12960-017-0187-2.
  * Loshchilov and Hutter (2019) Loshchilov, I., Hutter, F., 2019\.  Decoupled Weight Decay Regularization, in: International Conference on Learning Representations, pp. 1--18.  URL: https://openreview.net/forum?id=Bkg6RiCqY7.
  * Lovelace and Mortazavi (2020) Lovelace, J., Mortazavi, B., 2020\.  Learning to Generate Clinically Coherent Chest X-Ray Reports, in: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics. pp. 1235--1243.  URL: https://www.aclweb.org/anthology/2020.findings-emnlp.110, doi:10.18653/v1/2020.findings-emnlp.110.
  * McDonald et al. (2015) McDonald, R.J., Schwartz, K.M., Eckel, L.J., Diehn, F.E., Hunt, C.H., Bartholmai, B.J., Erickson, B.J., Kallmes, D.F., 2015\.  The Effects of Changes in Utilization and Technological Advancements of Cross-Sectional Imaging on Radiologist Workload.  Academic Radiology 22, 1191--1198.  doi:10.1016/j.acra.2015.05.007.
  * Naseer et al. (2021) Naseer, M., Ranasinghe, K., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., 2021\.  Intriguing Properties of Vision Transformers.  arXiv:2105.10497 [cs.CV] arXiv:2105.10497.
  * Nooralahzadeh et al. (2021) Nooralahzadeh, F., Gonzalez, N.P., Frauenfelder, T., Fujimoto, K., Krauthammer, M., 2021.  Progressive Transformer-Based Generation of Radiology Reports.  arXiv:2102.09777 [cs.CL] arXiv:2102.09777.
  * Papineni et al. (2002) Papineni, K., Roukos, S., Ward, T., Zhu, W., 2002.  Bleu: a Method for Automatic Evaluation of Machine Translation, in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Philadelphia, Pennsylvania, USA. pp. 311--318.  URL: https://www.aclweb.org/anthology/P02-1040, doi:10.3115/1073083.1073135.
  * Pavlopoulos et al. (2021) Pavlopoulos, J., Kougia, V., Androutsopoulos, I., Papamichail, D., 2021\.  Diagnostic Captioning: A Survey.  arXiv:2101.07299 [cs.CV] arXiv:2101.07299.
  * Peng et al. (2019) Peng, Y., Yan, S., Lu, Z., 2019.  Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets, in: Proceedings of the 18th BioNLP Workshop and Shared Task, Association for Computational Linguistics. pp. 58--65.  doi:10.18653/v1/w19-5006.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., 2019\.  Language Models are Unsupervised Multitask Learners.
  * Rajpurkar et al. (2017) Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., Lungren, M.P., Ng, A.Y., 2017\.  CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning.  arXiv:1711.05225 [cs.CV] arXiv:1711.05225.
  * Rennie et al. (2017) Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V., 2017.  Self-Critical Sequence Training for Image Captioning, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE. pp. 7008--7024.  doi:10.1109/cvpr.2017.131.
  * Rothe et al. (2020) Rothe, S., Narayan, S., Severyn, A., 2020.  Leveraging Pre-trained Checkpoints for Sequence Generation Tasks.  Transactions of the Association for Computational Linguistics 8, 264--280.  doi:10.1162/tacl_a_00313.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L., 2015\.  ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision 115, 211--252.  doi:10.1007/s11263-015-0816-y.
  * Sanh et al. (2019) Sanh, V., Debut, L., Chaumond, J., Wolf, T., 2019\.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  arXiv:1910.01108 [cs.CL] arXiv:1910.01108.
  * Satia et al. (2013) Satia, I., Bashagha, S., Bibi, A., Ahmed, R., Mellor, S., Zaman, F., 2013\.  Assessing the accuracy and certainty in interpreting chest X-rays in the medical division.  Clinical Medicine 13, 349--352.  doi:10.7861/clinmedicine.13-4-349.
  * Siegal et al. (2017) Siegal, D., Stratchko, L.M., DeRoo, C., 2017.  The role of radiology in diagnostic error: a medical malpractice claims review.  Diagnosis 4, 125--131.  doi:10.1515/dx-2017-0025.
  * Simsek et al. (2021) Simsek, B., Ged, F., Jacot, A., Spadaro, F., Hongler, C., Gerstner, W., Brea, J., 2021.  Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances, in: Meila, M., Zhang, T. (Eds.), Proceedings of the 38th International Conference on Machine Learning, PMLR. pp. 9722--9732.  URL: https://proceedings.mlr.press/v139/simsek21a.html.
  * Smit et al. (2020) Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A., Lungren, M., 2020\.  Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics. pp. 1500--1519.  doi:10.18653/v1/2020.emnlp-main.117.
  * Sorower (2010) Sorower, M., 2010.  A literature survey on algorithms for multi-label learning.  Oregon State University, Corvallis 18, 1--25.
  * Tan and Le (2019) Tan, M., Le, Q., 2019.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, in: Chaudhuri, K., Salakhutdinov, R. (Eds.), Proceedings of the 36th International Conference on Machine Learning, PMLR. pp. 6105--6114.  URL: http://proceedings.mlr.press/v97/tan19a.html.
  * Thrall et al. (2018) Thrall, J.H., Li, X., Li, Q., Cruz, C., Do, S., Dreyer, K., Brink, J., 2018\.  Artificial Intelligence and Machine Learning in Radiology: Opportunities, Challenges, Pitfalls, and Criteria for Success.  Journal of the American College of Radiology 15, 504--508.  doi:10.1016/j.jacr.2017.12.026.
  * Touvron et al. (2021) Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jegou, H., 2021\.  Training data-efficient image transformers & distillation through attention, in: Meila, M., Zhang, T. (Eds.), Proceedings of the 38th International Conference on Machine Learning, PMLR. pp. 10347--10357.  URL: https://proceedings.mlr.press/v139/touvron21a.html.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, u., Polosukhin, I., 2017\.  Attention is All You Need, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY, USA. p. 6000–6010.
  * Vedantam et al. (2015) Vedantam, R., Lawrence Zitnick, C., Parikh, D., 2015.  CIDEr: Consensus-Based Image Description Evaluation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4566--4575.
  * Vosshenrich et al. (2021) Vosshenrich, J., Brantner, P., Cyriac, J., Boll, D.T., Merkle, E.M., Heye, T., 2021\.  Quantifying radiology resident fatigue: Analysis of preliminary reports.  Radiology 298, 632--639.  doi:10.1148/radiol.2021203486.
  * Wang et al. (2022) Wang, K., Zhang, X., Zhang, X., Lu, Y., Huang, S., Yang, D., 2022\.  EANet: Iterative edge attention network for medical image segmentation.  Pattern Recognition 127, 108636\.  doi:10.1016/j.patcog.2022.108636.
  * Wang et al. (2018) Wang, X., Peng, Y., Lu, L., Lu, Z., Summers, R.M., 2018\.  TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE. pp. 9049--9058.  doi:10.1109/cvpr.2018.00943.
  * Williams and Zipser (1989) Williams, R.J., Zipser, D., 1989\.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks.  Neural Computation 1, 270--280.  doi:10.1162/neco.1989.1.2.270.
  * Wu et al. (2021) Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., Zhang, L., 2021\.  CvT: Introducing Convolutions to Vision Transformers.  arXiv:2103.15808 [cs.CV] arXiv:2103.15808.
  * Xiao et al. (2021) Xiao, T., Singh, M., Mintun, E., Darrell, T., Dollár, P., Girshick, R., 2021\.  Early Convolutions Help Transformers See Better.  arXiv:2106.14881 [cs.CV] arXiv:2106.14881.
  * Yang et al. (2022) Yang, Z., Wang, P., Chu, T., Yang, J., 2022.  Human-Centric Image Captioning.  Pattern Recognition 126, 108545\.  doi:10.1016/j.patcog.2022.108545.
  * Zhang et al. (2020) Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A., Xu, D., 2020.  When Radiology Report Generation Meets Knowledge Graph.  Proceedings of the AAAI Conference on Artificial Intelligence 34, 12910--12917.  doi:10.1609/aaai.v34i07.6989.
  * Zhu et al. (2015) Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., Fidler, S., 2015\.  Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 19--27.
  * Zhuang et al. (2021) Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., He, Q., 2021.  A Comprehensive Survey on Transfer Learning.  Proceedings of the IEEE 109, 43--76.  doi:10.1109/JPROC.2020.3004555.