# Automated Radiology Report Generation:
A Review of Recent Advances

Phillip Sloan  Philip Clatworthy  Edwin Simpson  and Majid Mirmehdi P. Sloan
was supported by the UK Research and Innovation (UKRI) Centre for Doctoral
Training (CDT) in Interactive Artificial Intelligence Award (EP/S022937/1)P.
Sloan is with the Interactive Artificial Intelligence CDT at the School of
Computer Science, University of Bristol, UK (e-mail:
phillip.sloan@bristol.ac.uk). P. Clatworthy is with the Bristol Medical
School, University of Bristol, UK (e-mail: phil.clatworthy@bristol.ac.uk).E.
Simpson is with the Intelligent Systems Labs, School of Engineering
Mathematics and Technology, University of Bristol, UK (email:
edwin.simpson@bristol.ac.uk).M. Mirmehdi is with the School of Computer
Science, University of Bristol, UK (email: majid@cs.bris.ac.uk).

###### Abstract

Increasing demands on medical imaging departments are taking a toll on the
radiologist’s ability to deliver timely and accurate reports. Recent
technological advances in artificial intelligence have demonstrated great
potential for automatic radiology report generation (ARRG), sparking an
explosion of research. This survey paper conducts a methodological review of
contemporary ARRG approaches by way of (i) assessing datasets based on
characteristics, such as availability, size, and adoption rate, (ii) examining
deep learning training methods, such as contrastive learning and reinforcement
learning, (iii) exploring state-of-the-art model architectures, including
variations of CNN and transformer models, (iv) outlining techniques
integrating clinical knowledge through multimodal inputs and knowledge graphs,
and (v) scrutinising current model evaluation techniques, including commonly
applied NLP metrics and qualitative clinical reviews. Furthermore, the
quantitative results of the reviewed models are analysed, where the top
performing models are examined to seek further insights. Finally, potential
new directions are highlighted, with the adoption of additional datasets from
other radiological modalities and improved evaluation methods predicted as
important areas of future development.

Artificial Intelligence, Biomedical Imaging, Computer Vision, Deep Learning,
Image Captioning, Language Models, Natural Language Processing, Radiology
Report Generation, Transformers, Vision Language Models.

##  1 Introduction

Figure 1: Illustration of the rapid growth of published papers within the ARRG
domain. The dashed line represents a forecast of anticipated future
publications.

Radiology is critical to modern healthcare and is the main diagnostic tool
used by clinicians for monitoring treatment and predicting outcomes. Radiology
includes various imaging modalities, and after each examination has been
performed, a radiological report is produced to guide patient management. The
diagnostic accuracy of such reports is critical in ensuring optimal care,
however it has been reported that there is a discrepancy rate of 3–5% [1]
while expressions of uncertainty have been found in 35% of reports [2]. This
can be compounded by a shortage of radiologists in some regions, for example
97% of imaging departments in the UK report that they can not keep up with
demand [3]. Such understaffing can cause reporting delays leading to
clinicians making critical decisions without the benefit of a radiological
opinion and potentially reaching a different conclusion compared to that of an
interpreting radiologist [4].

Healthcare represents an important application of deep learning research and
with radiology producing 90% of healthcare data [5], it is an attractive area
for deep learning researchers. Methods from computer vision (CV) [6] and
natural language processing (NLP) [7, 8] are currently being adopted within
the healthcare domain to improve the affordability and standards of care. One
rapidly developing healthcare application of deep learning is automated
radiology report generation (ARRG), a vision-language task that has
similarities to the broader area of image captioning [9]. With its ability to
augment radiologists’ capabilities, ARRG has significant clinical value and
could alleviate time pressures by reporting relatively simple examinations. It
could also be used as a patient specific draft template and for an
inexperienced radiologist it could act as a second reader [10] by
automatically highlighting any potential abnormalities [11].

Figure 2: The road map of this paper – First relevant ARRG datasets are
discussed, before we visit recent model training approaches. Next, the range
and extent of deep learning architectures deployed or developed are discussed,
followed by the “who and how” of works that have used knowledge and multiple
modalities. Finally, we consider assessment and evaluation methods employed
in, or specifically crafted for, ARRG.

There have been prior surveys of the ARRG domain [12, 14, 13] and of the
automated medical report generation domain [16, 17, 15] (see Table 1 for
summary). While these earlier papers offered an analysis of the initial growth
of ARRG, there has since been a surge of papers released over the past 2-3
years as can be observed in Figure 1. We anticipate this trend to continue,
driven by the rising number of researchers in this field who seek to harness
technological advances in various areas, such as language modelling [18, 19],
knowledge graphs [20, 21], and contrastive learning [22, 23, 24, 25].
Moreover, in early 2024, a new 3D chest computed tomography (CT) dataset was
made available [26]. Although no research has yet been published using this
dataset, it is anticipated that researchers will seek to leverage the
additional information it offers, potentially resulting in an even greater
influx of papers in ARRG.

Our review seeks to cover the abundance of recent works since the latest
review papers of Kaur et al. [14] and Messina et al. [15]. In particular, Kaur
et al. [14] provided a comprehensive survey of the ARRG domain specific to
chest radiographs. They classified research into four categories including
encoder–decoder architectures, attention mechanisms, reinforcement learning,
and graphs. While we also cover these areas in our review, there has been an
influx of research into new areas, e.g. training methods such as curriculum
learning [27] and contrastive learning [22] and the development of qualitative
evaluation methods such as assessment by clinical experts [28]. As their
review includes papers up to and including 2021, we seek to complement their
survey by reviewing these new avenues of research only from the last few
years.

Liao et al [13]’s survey spanned publications released by 2021 in relation to
datasets, training, architecture, interpretability and evaluation metrics.
Since the publication of their review, there has been a deluge in research
investigating injecting semantic knowledge into the model through the use of
knowledge graphs, such as RadGraph [29]. Our review builds on their review,
but also evaluates approaches injected with semantic knowledge, e.g. [9, 31,
30] and ARRG contributions with a multimodal input, where the input contained
not only images, but other information such as clinical indications [32, 33,
34, 36, 35] or temporal features [38, 37, 36, 39].

The main contribution of this work is a comprehensive review of the most
recent literature on automated radiology report generation since 2020,
examining the research by way of five different categories: datasets,
training, architecture, utilising knowledge and multiple modalities, and
evaluation methods (see road map in Figure 2). We offer carefully curated
figures and tables to distinguish papers based on the alignment of their
research contributions with these five classifications. The articles reviewed
in this study are predominantly sourced from high impact journals and
conferences, such as Neural Information Processing Systems (NeurIPS), Computer
Vision Pattern Recognition (CVPR), Empirical Methods in Natural Language
Processing (EMNLP), Institute of Electrical and Electronics Engineers(IEEE)
and Medical Image Computing and Computer Assisted Interventions (MICCAI).
Occasionally, when significant and helpful to the narrative, we include work
prior to 2020.

The rest of the paper is organised as follows. Section 2 provides a background
to the ARRG task. Section 3 outlines the datasets used for ARRG. Section 4
reviews articles which seek to improve different aspects of model training.
Section 5 discusses contributions which develop model architecture. Section 6
explores research that utilises domain knowledge or multiple modalities within
their models. Section 7 considers the various evaluation methods used within
the ARRG domain. Section 8 discusses the ARRG landscape. We contrast the
reported model performances on commonly employed NLP metrics. Section 9
concludes the paper, discussing potential future directions in the field and
outlining the potential limitations of this review.

Author | Year | Coverage | # Cited | # ARRG
---|---|---|---|---
Allaouzi et al. [16] | 2018 | 2015-2018 | 50 | 10
Pavlopoulos et al. [17] | 2019 | 2015-2019 | 65 | 7
Monshi et al. [12] | 2020 | 2015-2019 | 105 | 14
Kaur et al. [14] | 2022 | 2015-2021 | 83 | 19
Messina et al. [15] | 2022 | 2016-2021 | 167 | 40
Liao et al. [13] | 2023 | 2016-2021 | 154 | 41
Our Review | 2023 | 2020-2023 | 155 | 84
Table 1: Recent ARRG review articles and their coverage including total number
of citations and the total number of articles related to ARRG or medical
report generation.

##  2 Background

The purpose of a radiology report is to provide an accurate interpretation of
an imaging study relating to the findings that have been observed within the
study, irrespective of whether they are anticipated or unexpected. Reports
should also be specific in answering the clinical question asked by the
referring clinician who then uses information gleaned from the report to
provide appropriate care for the affected patient. While there is no
universally agreed structure to radiology reports, the European Society of
Radiology (ESR) [40] have outlined good practice guidelines for the
standardised structure of a radiological report:

  * •

Clinical referral – A brief summary of the indications and reasons for the
referral. This often includes the clinical question asked by the referring
clinician.

  * •

Technique – A description of what investigation was performed and which
techniques or sequences were utilised.

  * •

Findings – A comprehensive description of all observations within the image,
starting with the most relevant to the request. The findings should note any
incidental findings not related to the indications.

  * •

Conclusion/Impression – A concise diagnosis of the examination or a ranked
list of potential differential diagnoses to help guide patient treatment.

Figure 3: An example x-ray and report pair from Indiana University X-ray
dataset [41]. ARRG models seek to generate the Findings and Impression
sections.

The ARRG task takes one or more radiographic images as input and attempts to
produce a sequence of text that closely resembles the report created by a
radiologist.

Deep learning approaches for ARRG have generally adopted generative models,
with an encoder-decoder architecture being the most commonly used. Within this
architecture the encoder is used to extract meaningful features from the input
radiographic image, and subsequently, the decoder used the extracted features
to generate the radiology report. The majority of reviewed research trains in
a supervised fashion where the associated report is used as a ground truth for
training and evaluation. An example of a training sample can be seen in Figure
3.

Early ARRG research coincided with the 2016 release of the Indiana University
X-ray Dataset [41], the first dataset that contained both radiology images and
their associated radiology reports. Shin et al. [42] used this dataset to
develop a method to automatically annotate medical images using a
convolutional neural network (CNN) for their encoder and a recurrent neural
network (RNN) for their decoder. In 2018, research into ARRG started gaining
popularity [43, 44], however the generated reports from these CNN-RNN
methodologies are overly rigid [9] and often repeat phrases from the training
set at inappropriate times, leading to summaries that appear less human-like.
The development of the transformer architecture [45] has created new avenues
to be explored and stimulated significant amounts of research in the ARRG
field.

ARRG can be considered to have its roots in image captioning. From the CV
perspective, unlike natural images such as images of cats and dogs,
radiological images closely resemble each other with differences between two
radiographic images being quite nuanced [9]. A pathology could occupy only a
limited section of an image, thereby increasing the burden on the encoder.
These subtle changes have a high impact on the generated report and if missed,
will turn an abnormal case into a normal one. From an NLP perspective, the
generated text is longer than a traditional caption. In addition, differences
in reports from patient to patient are fine-grained, meaning there are a lot
of similar sentences describing the normal appearances, while the sentences
related to observed abnormalities could be overlooked [9].

##  3 Datasets

Name | Year | Availability | Modality |  Views | Patients | E/R | Images | Language | Adoption
---|---|---|---|---|---|---|---|---|---
IU-Xray [41] | 2016 | Public | CXR | Frontal/Lateral | 3,955 | 3,955 | 7,470 | English | 50
MIMIC-CXR [52] | 2019 | Restricted | CXR | Frontal/Lateral | 65,379 | 227,827 | 377,110 | English | 45
MIMIC-ABM [53] | 2020 | Restricted | CXR | Frontal/Lateral | —- | —- | 38,551 | English | 0
Chest ImaGenome [54] | 2021 | Restricted | CXR | Frontal | —- | —- | 242,072 | English | 1
CXR-PRO [55] | 2022 | Restricted | CXR | Frontal/Lateral | 65,379 | —- | 374,139 | English | 1
MS-CXR [56] | 2022 | Restricted | CXR | Frontal | 851 | —- | 1047 | English | 1
MS-CXR-T [37] | 2023 | Restricted | CXR | Frontal | 800 | —- | 1326 | English | 1
PadChest [57] | 2019 | Restricted | CXR | Frontal/Lateral | 67,625 | 109,931 | 160,868 | Spanish | 0
CH-Xray [21] | 2022 | Private | CXR | Frontal | —- | 11,049 | 11,049 | Chinese | 3
CX-CXR [58] | 2018 | Restricted | CXR | Frontal/Lateral | 33,236 | 33,236 | 45,598 | Chinese | 2
COV-CTR [20] | 2022 | Public | CT | Axial | —- | 728 | 728 | English | 1
JLiverCT [38] | 2022 | Private | CT | Axial | —- | 1,083 | 1,083 | Japanese | 1
CT-RATE [26] | 2024 | Public | CT | Axial | 21,304 | 25,692 | 50,188 | English | 1
Table 2: Datasets used in the Literature. Restricted means an application
process is required to gain access to the associated dataset. A ‘—-’ indicates
that the corresponding value was not reported for this dataset. E/R represents
the number of Examinations and Reports. The adoption field quantities how many
of the reviewed articles use a dataset.

Ethical and confidentiality issues surrounding the public release of medical
data [46] means medical datasets with associated radiology reports are scarce.
In this section, after considering two publicly available datasets that are
used most frequently by researchers in ARRG, we shall briefly mention a number
of other datasets that have been harnessed by ARRG researchers to improve
their models. A list of the most significant datasets encountered in our
search of the existing literature, where actual reports beyond simple patient
metadata are available, is available in Table 2.

###  3.1 Indiana University X-ray Dataset (IU-Xray)

The Indiana University X-ray Dataset dataset (IU-Xray) [41], also known as the
OpenI Dataset, was published in 2016. It is publicly available and contains
$7,470$ images and $3,955$ radiology reports. Each report is comprised of four
sections: indication, comparison, findings and impression, and thus can be
considered to be roughly aligned with ESR guidelines. IU-Xray has also been
labelled with Medical Text Indexer (MTI) tags which are keywords generated
from the diagnostic sentences within the report. While there is no official
split, some more recent researchers [50, 48, 47, 49] have adopted the split
provided by Chen et al. [51] for reproducibility.

When reviewing Table 2, IU-Xray emerges as the most prevalent dataset, with
$50$ of the reviewed articles using this dataset for training and evaluation.
This can be attributed to its easy accessibility and relatively small size
which allows quick prototyping and training. It’s relatively small size has
however been noted as a potential limitation, with Chen et al. [51] stating
that its size can result in significant variance of results. Additionally, IU-
Xray is composed of only one examination per patient, meaning there is no
longitudinal dimension for researchers to leverage. Moreover, the data
collection exclusively included outpatient examinations, thereby potentially
impeding the accuracy and generalisability of any trained models due to the
absence of certain projections and artefacts, such as portable x-rays and
central lines, which are out of distribution.

While IU-Xray is a very popular dataset within the ARRG domain, it has been
noted that researchers sometimes employ the dataset as a secondary testing set
[51].

###  3.2 Medical Information Mart for Intensive Care CXR (MIMIC-CXR)

The Medical Information Mart for Intensive Care Chest X-ray (MIMIC-CXR) [52]
stands as the largest ARRG dataset to date, comprising $377,110$ radiographic
images and $227,835$ radiology reports collected from $65,379$ individuals.
These records span the years 2011-2016 and originate from the Beth Israel
Deaconess Medical Center Emergency Department in Boston, MA. In addition to
the images and reports, the dataset also contains medical subject headings
(MeSH), which are labels that are automatically generated, annotating 13
important medical tags for each sample. The original release of MIMIC-CXR was
in Digital Imaging and Communications in Medicine (DICOM) format, however the
dataset has since been converted into the JPEG image format. MIMIC-CXR has an
official train/test/validation split, which has easily enabled comparisons
between different ARRG models. Researchers who wish to access MIMIC-CXR are
required to be credentialed on PhysioNet 111https://physionet.org/ and to sign
a data use agreement.

When examining the datasets listed in Table 2, it becomes apparent from the
adoption column that MIMIC-CXR is one of the main choices among researchers in
the field of ARRG. It also stands out as the largest available dataset for
ARRG which is advantageous because larger datasets often lead to reduced model
variance [59]. The ratio of patients to examinations suggests that each
patient has on average $3.5$ studies within the dataset, which highlights the
temporal aspect of MIMIC-CXR, presenting researchers with significant
potential for exploitation. MIMIC-CXR is also part of the larger MIMIC
database which contains various other related datasets that could potentially
be used in conjunction with with MIMIC-CXR.

###  3.3 MIMIC Derivatives

Due to its rich data and popularity among researchers, several derivative
datasets have been developed from MIMIC-CXR. These datasets are located on
PhysioNet and similarly to MIMIC-CXR, require researchers to be credentialed
before they can gain access. Ni et al. [53] developed a subset called MIMIC-
ABM which is constructed from samples of MIMIC-CXR that have at least one
abnormal finding in the associated report. Ramesh et al [55] implemented
another derivative called CXR-PRO that omits from reports any reference to
prior radiological studies through the use of a BERT-based token classifier.

Chest ImaGenome was constructed by Wu et al. [54], which contains
automatically constructed scene graphs for the MIMIC-CXR images. Each scene
graph describes one frontal chest X-ray image and contains bounding box
coordinates for $29$ unique anatomical regions in the chest, as well as
sentences describing each region if they exist in the corresponding radiology
report. The dataset is split according to the official MIMIC split. Boecking
et al. [56] developed a dataset called MS-CXR which contains $1047$ images
with a total of $1162$ image–sentence pairs of bounding boxes and
corresponding phrases, collected across eight different cardiopulmonary
radiological findings, with an approximately equal number of image-sentence
pairs for each finding.

Taking advantage of the longitudinal nature of MIMIC, Bannur et al [37]
released MS-CXR-T, a multi-modal benchmark dataset for evaluating biomedical
vision-language models on two distinct temporal tasks in radiology: image
classification and sentence similarity. The image classification task provides
labels across five findings that can be considered to be in one of three
states: improving, stable or worsening. The sentence similarity task contains
$361$ pairs of sentences that are either semantically equivalent or
contradictions. The datasets aim was to provide an opportunity to evaluate
models on temporal tasks, which has not previously been available within the
ARRG domain.

###  3.4 Other Datasets

Aurelia et al. [57] compiled PadChest which is a dataset of 160,868 images and
109,931 Spanish reports from the San Juan Hospital in Spain. Although PadChest
contains public reports, researchers have not embraced it, and the reason for
this lack of adoption remains unclear. Typically, having datasets from various
languages is deemed advantageous for assessing whether a method generalises
across languages. To gain access to PadChest, a researcher will need to apply
to the principle researchers through an application form. Li et al. [20]
publicly released COV-CTR, a COVID-19 CT Report dataset containing 728
samples, which contains reports both in English and Chinese. CX-CHR[58] is a
private collection of chest x-ray images with corresponding Chinese reports
consisting of 33,236 samples. While not publicly available, the paper states
that researchers can apply for academic usage after signing a confidentiality
agreement.

CT-RATE [26] is a recently published, 3D CT Chest dataset, containing 25,692
non-contrast chest CT volumes. It has been expanded to 50,188 volumes through
various reconstructions and encompasses data from 21,304 unique patients ,
accompanied by the corresponding English radiology text reports. CT-RATE
provides an important step forward for the ARRG domain as it is the first
large scale dataset in a radiology modality other than 2D chest x-rays.
Similar to IU-Xray [41] and MIMIC-CXR [52], the reports contain clinical
indications, findings and impressions. The dataset is publicly available
through a HuggingFace dataset repository 222https://huggingface.co/datasets.

There are very few other datasets used in ARRG research often limited by their
restricted (or complete lack of) accessibility. CH-Xray [21] is a private
dataset with Chinese language reports containing 11,049 samples. The private
JLiverCT dataset [38] contains 1,083 samples, with the reports written in
Japanese. Liu et al. [60] released a public dataset of patients experiencing
acute and early sub-acute stroke, comprising of 2,888 multi-sequence clinical
MRIs along with metadata (age, sex and other clinical factors). Subsequently,
Liu et al. [61] generated reports for this dataset, but these are not
accessible to the public at present which inhibits the wider ARRG community
from adopting the dataset.

There are some datasets that do not offer free-text reports, but have been
used to help augment a model’s ability to handle ARRG. The National Institutes
of Health Chest X-Ray Dataset (NIH-CXR) [62] is composed of 112,120 images
from 30,805 examinations, with labels text-mined from radiology reports,
initially containing 8 labels and improved to 14 on a later release. CheXpert
[63] consists of 224,316 images obtained from the Stanford University Medical
Center between 2002–2017 and also uses an NLP method to mine 14 thoracic
pathological labels.

##  4 Training

Training a computational model to learn how to make decisions and generate
knowledge on its input data is a key stage of building such a model. We now
categorise, loosely under specific topics, various training strategies that
were originally developed for other applications, but have been either adopted
or developed further for ARRG.

###  4.1 Supervised Learning

Supervised learning is defined by training a model using datasets containing
ground truth labels. In ARRG, the label is the radiology report which is the
target output the model is trained to predict. For ARRG datasets, some
training samples are more challenging than others as the radiographic image
could contain a subtle pathology or the radiology report could be particularly
long. Curriculum learning is a training strategy where, instead of randomly
shuffling the entire dataset, the model is exposed to increasingly complex
examples over time. This method of training a model is thought to closely
resemble how a radiologist learns throughout their career, enabling a
clinician to gradually become more comfortable with more complex cases. The
novelty of curriculum learning techniques relates to how a researcher measures
how challenging an example case is.

Liu et al. [27] developed a competence-based multimodal curriculum learning
strategy which evaluated the complexity of each sample in the dataset using
heuristic metrics to quantify the visual and textual difficulty. To measure
the degree of visual difficulty, they implemented a ResNet-50 architecture,
fine-tuned on the CheXpert dataset. Their model extracted the normal image
embeddings of all normal training images from ResNet-50’s last average pooling
layer. Then, given an input image, an image embedding was obtained from their
model which was compared against the normal images using average cosine
similarity to provide a heuristic measure of the visual complexity of that
training instance. For the textual difficulty, they utilised a count of the
number of sentences describing abnormal findings within the report to
establish how arduous it is to generate. Their method considered sentences
without the keywords “no”, “normal”, “clear” or “stable” as sentences
describing abnormal findings.

Figure 4: An illustration of (A) contrastive learning and (B) contrastive
language-image pre-training (CLIP) methods used within ARRG. (A) demonstrates
SimCLR [64] as implemented for example by [22, 65]. In this framework, image
augmentations such as cropping and Gaussian noise were used to create
additional positive samples while other radiographic images were considered as
negative samples. (B) demonstrates CLIP, which was implemented by works such
as [67, 66]. The dotted lines present novel pathways from ARRG research
contributions: “hard” negatives were used by [28, 23] to enable more robust
representations to be learnt and [24] developed extra positive samples at the
embedding level through the use of dropout and cutoff.

A multi-criteria supervised approach to training was proposed by Wang et al.
[9], with auxiliary objectives embedded into their training strategy. An
image-text matching objective is used to better correlate image and text
features while an image classification objective is also developed to improve
the feature extraction capabilities of their model. For the classification
objective, the MeSH labels provided with the MIMIC-CXR dataset are adopted as
ground truths. Li et al. [20] also implemented auxiliary objectives within
their model, naming the process auxiliary signal‐guided learning. Their method
implemented internal auxiliary signals where their model generated
segmentation masks of abnormal regions which are used to crop the radiographic
images. These cropped images serve as a secondary input to the model which
enabled the model to attend to abnormal areas in more detail. Their model also
employed external auxiliary signals to correctly phrase and learn medical
knowledge. The medical knowledge is mined from an unnamed, medical website
which provided medical knowledge on the symptoms, manifestations and other
information regarding thoracic diseases. This medical text is divided into
tokens, which are embedded before being encoded with a single-layer gated-
recurrent unit.

###  4.2 Unsupervised Learning

Unsupervised learning trains models using unlabeled datasets, seeking to find
hidden patterns and insights from the given data. During our literature
search, we encountered less than a handful significant ARRG research
contributions concerning unsupervised learning for the main ARRG task which
suggests the domain is either under-explored or self-supervised approaches
have already extinguished the flame for such approaches.

Liu et al. [68] leveraged a pre-constructured knowledge graph as a shared
embedding space to bridge the visual and textual domains with their knowledge-
driven encoder adopting uncoupled image-report pairs in IU-Xray and MIMIC-CXR
as queries, projecting them to corresponding coordinates in the latent space.
These embedding positions within the latent space are used to measure the
relationship between images and reports. A variational autoencoder-based
unsupervised clustering method was proposed by Yan et al. [69] to glean
additional knowledge from input radiographic images. The input images go
through a ResNet-101 feature extractor, and then their refinement is carried
out by a Transformer encoder. Subsequently, clustering is applied to the
refined patches, effectively capturing and representing the additional
knowledge. Specifically, additional knowledge is represented by an additive
Gaussian distribution which is learned in an unsupervised manner. This prior
knowledge is then used as an additional input to their decoder to generate
medically fluent reports. Li et al [70] also utilised clustering to obtain
potential knowledge from reports, adopting HDBSCAN to identify clusters of
similar knowledge. Rather than randomly initialising their memory matrix
module, Wang et al. [71] employed K-means clustering on visual and textual
features extracted from the training set of the MIMIC-CXR dataset to calculate
the initial starting values for their matrix.

###  4.3 Contrastive Learning

A common form of training implemented in the ARRG domain is contrastive
learning [22, 66, 23, 71, 72, 73, 74, 75] which focuses on learning
representations by comparing similar and dissimilar instances within the data
itself. By contrasting positive and negative pairs of example data, it
generates features for similar pairs closer together in a feature embedding
space, while dissimilar pairs are registered farther apart. Depending on how
the positive/negative pairings are generated, contrastive learning can be
defined as either supervised or unsupervised training, often being classified
as self-supervised training due to the lack of explicit labels.

Contrastive learning approaches have become popular due to the scarcity of
data within the medical deep learning domain with works differing in how they
select positive and negative samples. Some works have focused on contrasting
radiographic images [22, 23, 71] while others implemented the Contrastive
Language–Image Pre-training (CLIP) approach [76], where both radiographic
images and reports are used in some fashion [28, 67, 65, 24, 66, 77, 25]. An
illustration of contrastive learning and CLIP can be seen in Figure 4.

Yan et al. [28] developed a weakly supervised contrastive learning method,
comparing the images and radiology reports against each other. The novelty of
their approach was through the identification of “hard” negative samples which
are cases that are close semantically to the report of the corresponding
radiographic image. K-means clustering is used to select these “hard” samples
which originate from the same cluster as the reference report. Li et al.’s
[74] proposed framework also identified “hard” negative samples during
training, using cosine similarity to locate the nearest samples in the
embedding space.

Leonardi et al. [66] utilised contrastive learning on radiographic images and
reports as pairs. An image and its associated report are used as positive
pairs while an image and an alternative report is considered a negative pair.
The approach implemented several methods for image or textual augmentations to
generate extra positive samples from the reference image/report pair.

Tanwani et al.[65] incorporated SimCLR in their proposed network RepsNet, a
visual question answering model which is also shown to be capable at ARRG.
Image augmentations such as colour jittering, normalisation and random erasing
are used during their approach to generate new positive samples. Their model
used a bilinear attention network to fuse image and question/text features
together after which bidirectional contrastive learning [64] is adopted,
pulling together a given image-text pair while pushing away observations that
correspond to a different image-text pairs. Hou et al. [22] also drew
inspiration from the SimCLR framework [64] and generated positive samples
through augmentation of the radiographic images with transforms such as
rotation and Gaussian blurring. These augmented samples, along with any
samples from the same patient, are considered positive samples while all other
samples are considered negative samples. Wu et al. [24] generated extra
samples through augmenting report sentences at the embedding level using
methods such as dropout and cutoff. Augmented sentences are considered
positive instances while other sentences within the same batch are considered
negative instances.

Figure 5: An overview of the three main encoder-decoder architectures used
within the ARRG domain: (A) demonstrates a typical CNN-LSTM network as
utilised for example by [93, 90, 82, 92, 38, 89, 58, 91], (B) illustrates the
most popular architecture in use today, such as in [51, 94, 73, 47] while (C)
embodies the pure transformer architecture used for example by [9, 33, 95].

The CheXpert labeller [63] is an NLP tool that extracts observations from the
reports which have been used in various contrastive learning approaches to
help distinguish between positive and negative samples [67, 71, 77, 23]. For
example, Wang et al. [71] introduced a multi-label contrastive loss to support
their cross-modal prototype model in learning class-related and informative
cross-modal patterns. CheXbert was then implemented to generate a pseudo label
for each image-text pair, considering a sample as positive if it shares at
least one label and negative if it does not share any common labels. Jeong et
al. [23] generated hard negative samples to encourage their model to learn
more robust representations. Instead of K-means sampling, they applied the
CheXbert [78] labeller to generate embeddings and create negative and “hard”
negative samples by examining their Manhattan distance, with “hard” samples
having smaller distances. Their model is trained against the hardest negative
sample.

As a rule-based labeller, an issue with CheXpert is that it is non-
differentiable which prevents it being used with techniques such as
backpropagation [79]. To be able to directly train their model to produce
clinically accurate reports, Lovelace and Mortazavi [80] developed a method to
differentiably extract clinical observations from generated reports and
utilised this differentiability to fine-tune their model in a supervised
fashion. They trained CNN and long short-term memory (LSTM) models to
approximate the CheXpert labeler [63] during training.

###  4.4 Triplet Loss

Although not as commonly applied as contrastive loss, triplet loss [81] has
been used in ARRG for training the model encoder, and as an auxiliary task to
improve a model’s performance. While both triplet loss and contrastive loss
compare samples, the mechanism is a little more nuanced. Triplet loss, in
particular, incorporates an additional sample to form a triplet, which
includes a positive and negative sample for contrasting against the anchor.
The aim is then to minimize the distance between the anchor and positive
samples while maximizing the distance between the anchor and the negative
samples to create more distinct clusters within the latent space.

Wang et al. [82] proposed a framework that uses triplet loss to help boost its
performance during training. The framework consists of two branches where the
main branch tackled the standard report generation task and the auxiliary
branch performed image-text matching with triplet loss to learn text
correlated visual features. Their approach trained the branches in an
alternating manner and sought to boost training in three ways: (i) the feature
extractor was used by both branches, providing twice as many training steps;
(ii) the report encoder of the auxiliary branch is also used by the main
branch to evaluate and minimise the feature-level loss between the generated
and the ground-truth reports; (iii) the reports generated by the main branch
were used as “hard” negative samples while training the auxiliary branch,
which as the network trained, became semantically closer to the reference
report.

Two types of triplets were applied in Yang et al. [83]’s contrastive triplet
network (CTN). The first triplet was the traditional positive, anchor,
negative sample approach, while the second type of triplets reversed this,
with abnormal cases as the anchor and positive class, and normal cases as the
negative class. These triplets were used during training, where the visual
embedding from the encoder and the semantic embedding of the reports were
contrasted to encourage the similarity between positive and anchor cases and
the dissimilarity between negative and anchor cases.

###  4.5 Reinforcement Learning

Methodologies utilising reinforcement learning (RL) have been effective within
the ARRG domain [84, 85, 86] with most contributions defining rewards for
agents to optimise.

Two reinforcement rewards were proposed by Miura et al. [87]: $fact_{ENT}$, or
exact entity match reward, sought to capture factual completeness by
extracting entities from both the generated and corresponding ground truth
report, after which the reward function checks for matching entities, and
$fact_{ENTNLI}$ or entailing entity match reward extended $fact_{ENT}$ by
giving weight to the context of entities within sentences rather than just
checking for an occurrence. The reward searched for contradictions between the
two reports, such as a negation changing the meaning of the sentence. With
$fact_{ENTNLI}$ a match only occurs if both sentences have the same semantic
meaning, providing a more robust metric than $fact_{ENT}$.

“RadGraph reward” was introduced by Delbrouck et al. [84] which involved the
use of three RL rewards. The approach harnessed the PubMedBERT model [88],
trained on the RadGraph dataset [29], to extract entities and relations from
reports, where entities represent anatomical and clinical observations and
relations denote the connections between the entities, such as “suggestive of”
or “located at”. For each case the PubMedBERT model created two semantic
graphs, one generated from their ARRGs model’s output and the other from the
associated ground truth report. Their rewards evaluated the similarities
between the two generated graphs, comparing the entities and relations that
were extracted. By using RL, a simpler model was able to be quickly trained
and provided equivalent performance to more complex architectures.

Imbalanced token distributions naturally exist within radiology reports, and
this disparity has been found to cause ARRG models to overfit on frequent
tokens, leading to lower performance on uncommon tokens. To combat this, Wu et
al. [86] reduced token frequency effects by adopting RL to dynamically adapt
the token set during training. Their reward consisted of two components, a
text generation reward and an imbalanced evaluation reward which broke down
the comparison of performance differences into four groups based on how often
the tokens appear. The F1 score was then calculated for each group of tokens,
balancing performance to ensure the tokens from the groups are given more
equal weights.

Other works of note that utilise RL have generally employed NLP metrics as
their reward [82, 85, 31, 38] to reward lexical similarity to a reference
report.

##  5 Architecture

The architecture of ARRG models determine how they process and interpret
radiographic images and reports and generate radiology reports from unseen
radiographic images. To date, there are two dominant architectures used,
encoder-decoder models and retrieval models. This section discusses various
avenues of research which has sought to provide novel contributions to such
architectures. Figure 5 illustrates the three main architectures of encoder-
decoder networks that are utilised within ARRG.

###  5.1 CNN-LSTM

Table 3 demonstrates that encoder-decoder models are the most frequently
adopted architecture type for the ARRG task. Within this architecture, the
encoder extracts features from the input radiographic image which are in turn
used by the decoder to generate a free text radiology report. As discussed in
Section 2, early research employed CNN-RNN or CNN-LSTM models, which are still
in use in more current works [93, 82, 90, 92, 38, 89, 96, 58, 91, 97], which
can be seen in Figure 5(A). The novelty of these approaches generally comes
from additionally proposed modules or strategies within their networks.

The following approaches harnessed pretrained CNNs for their encoder and a
Hierarchical LSTM for their decoder composed of two layers: a sentence LSTM
and a word LSTM. The sentence LSTM generates topics for each sentence which
are transferred to a word-LSTM to generate sentences on a word by word basis,
using the topics as inspiration. Both Zhang et al. [93] and Wang et al. [92]
employed pretrained DenseNet-121 [98] CNN encoders with their novelty being
grounded in their graph convolutional neural network modules. Wang et al. [92]
also introduced a knowledge graph within their architecture which assists the
generation process by providing medical knowledge to the model.

A VGG19 encoder was employed by Kaur & Mittal [91] for extracting image
features and their distinctive contribution lies in the reduction of model
size and complexity achieved through parameter pruning. Wang et al. [82]
harnessed a ResNet-101 [99] for their encoder48 with their novelty being
introduced through their self-boosting training regime. Liu et al. [90]
employed ResNet-50 as their encoder and introduced a contrastive attention
module as part of their contribution.

Other works implemented a Bi-Directional LSTM (Bi-LSTM) for their decoder
where the two layers process information in opposite directions. To improve
the performance of their model, Wang et al. [58] proposed a hybrid model which
unified retrieval and sentence generation approaches. Their adaptive generator
selected between retrieving a sentence from a corpus of reports or utilising
their LSTM decoder to generate sentences. Gajbhiye et al. [89] integrated an
adaptive multilevel multi-attention module to learn the association between
visual and textual features. Their Bi-LSTM decoder explored these embeddings
to generate a radiological report.

Model | Year | Feature Extractor | Model Type | Model Architecture | Model Output
---|---|---|---|---|---
R2Gen [51] | 2020 | ResNet | Encoder-Decoder | Transformer | Findings
Syeda-Mahmood et al. [103] | 2020 | CNN | Retrieval | CNN | Both
Zhang et al. [93] | 2020 | DenseNet | Encoder-Decoder | GraphCNN-LSTM | Both
Lovelace & Mortazavi [80] | 2020 | Densenet | Encoder-Decoder | Transformer | Findings
Boag et al. [117] | 2020 | Densenet | Various | Various | Findings
VTI [100] | 2022 | Densenet | Encoder-Decoder | CNN-LSTM | Both
CMCL [27] | 2021 | ResNet | Encoder-Decoder | Various | Findings
CA [90] | 2021 | ResNet | Encoder-Decoder | Various | Findings
Nooralahzadeh et al. [104] | 2021 | Densenet | Encoder-Decoder | Transformer | Findings
Yan et al. [28] | 2021 | ResNet | Encoder-Decoder | Transformer | Findings
AlignTransformer [118] | 2021 | ResNet | Encoder-Decoder | Transformer | Findings
PPKED [48] | 2021 | ResNet | Encoder-Decoder | Transformer | Findings
KGAE [68] | 2021 | ResNet | Encoder-Decoder | Transformer | Findings
R2GenCMN [50] | 2021 | ResNet | Encoder-Decoder | Transformer | Findings
Nguyen et al. [32] | 2021 | ResNet | Encoder-Decoder | Transformer | Findings
MedWriter [119] | 2021 | Densenet | Various | CNN-LSTM | Findings
CXR-RePaiR [67] | 2021 | ResNet / ViT | Retrieval | ResNet / ViT | Both
CDGPT2 [120] | 2021 | ResNet | Encoder-Decoder | CNN-Transformer | Both
Wang et al. [82] | 2021 | ResNet | Encoder-Decoder | CNN-LSTM | Findings
Muira et al. [87] | 2021 | Densenet | Encoder-Decoder | Transformer | Findings
Medical-VLBERT [121] | 2021 | DenseNet | Encoder | BERT | Findings
Zhao et al. [21] | 2022 | ResNet | Encoder-Decoder | Transformer | Findings
Wang et al. [9] | 2022 | ViT | Encoder-Decoder | Transformer | Findings
Radgraph Reward [84] | 2022 | Densenet | Encoder | BERT | Findings
Yang et al. [83] | 2022 | ResNet | Encoder-Decoder | Transformer | Findings
Qin and Song [85] | 2022 | ResNet | Encoder-Decoder | Transformer | Findings
Yang et al. [11] | 2022 | ResNet | Various | CNN-Transformer | Findings
VTI-TRS [101] | 2022 | Densenet | Encoder-Decoder | CNN-LSTM/Transformer | Both
XPRONET [71] | 2022 | ResNet | Encoder-Decoder | CNN-Transformer | Findings
CMCA [122] | 2022 | Densenet | Various | CNN-Transformer | Findings
MaKG [123] | 2022 | Densenet | Encoder-Decoder | Transformer | Findings
Zhang et al. [30] | 2022 | Densenet | Encoder-Decoder | CNN-Transformer | Findings
Nazarov et al. [77] | 2022 | Various | Encoder-Decoder | Various | Impression
MATNet [33] | 2022 | Densenet | Encoder-Decoder | Transformer | Findings
Dalla Serra et al. [34] | 2022 | ResNet | Encoder-Decoder | Transformer | Findings
SGF [70] | 2022 | ResNet | Encoder-Decoder | Transformer | Findings
JPG [72] | 2022 | ResNet | Encoder-Decoder | Transformer | Findings
MedViLL [124] | 2022 | ResNet | Encoder | BERT | Both
AMLMA [89] | 2022 | Densenet | Encoder-Decoder | CNN-LSTM | Findings
Relation-paraNet [58] | 2022 | VGG19 / Densenet | Encoder-Decoder | CNN-LSTM | Findings
Clinical-BERT [125] | 2022 | Densenet | Encoder | BERT | Both
RepsNet [65] | 2022 | ResNeXt | Encoder | BERT | Findings
MSAT [31] | 2022 | CLIP | Encoder-Decoder | Transformer | Findings
Wang et al. [92] | 2022 | Densenet | Encoder-Decoder | CNN-LSTM | Findings
CoPlan [38] | 2022 | CNN | Encoder-Decoder | CNN-LSTM | Findings
Leonardi et al. [66] | 2022 | CNN | Encoder-Decoder | CNN-Transformer | Findings
Yan et al. [69] | 2022 | ResNet | Encoder-Decoder | Transformer | Findings
Wang et al. [126] | 2022 | ResNet | Encoder-Decoder | Transformer | Findings
MRCL [73] | 2022 | ResNet | Encoder-Decoder | CNN-LSTM/BERT | Both
Boecking et al. [56] | 2022 | ResNet | Encoder | BERT | Findings
DeltaNet [96] | 2022 | ResNet | Various | CNN | LSTM
TrMRG [95] | 2023 | CNN | Encoder-Decoder | Transformer | Both
CheXPrune [91] | 2023 | VGG19 | Encoder-Decoder | CNN-LSTM | Both
Kim et al. [127] | 2023 | Densenet | Encoder-Decoder | Transformer | Findings
MKCL [22] | 2023 | ResNet | Encoder-Decoder | Transformer | Findings
CvT-212DistilGPT2 [128] | 2023 | Various | Encoder-Decoder | Various | Findings
ASGK [20] | 2023 | Densenet | Encoder-Decoder | CNN-Transformer | Findings
BioViL-T [37] | 2023 | ResNet | Encoder-Decoder | Transformer | Findings
ORGAN [94] | 2023 | ResNet | Encoder-Decoder | Transformer | Findings
X-REM [23] | 2023 | ALBEF [135] | Retrieval | ALBEF [135] | Both
MRCL [73] | 2023 | CNN | Encoder-Decoder | CNN-LSTM/Transformer | Both
KiUT [47] | 2023 | ResNet | Encoder-Decoder | Transformer | Findings
TIMER [86] | 2023 | ResNet | Encoder-Decoder | Transformer | Findings
Yang et al. [129] | 2023 | ResNet | Encoder-Decoder | Transformer | Findings
Kale et al. [116] | 2023 | ResNet | Encoder-Decoder | Transformer | Findings
METransformer [49] | 2023 | ViT | Encoder-Decoder | Transformer | Findings
ATAG [130] | 2022 | Densenet | Encoder-Decoder | CNN-LSTM/Transformer | Both
Selivanov et al. [137] | 2023 | CNN | Encoder-Decoder | CNN-LSTM/Transformer | Findings
RGRG [110] | 2023 | Faster R-CNN | Encoder-Decoder | Faster RCNN-Transformer | Findings
Gu et al. [131] | 2023 | CNN | Encoder-Decoder | CNN-LSTM | Findings
UAR [74] | 2023 | dVAE | Encoder-Decoder | dVAE-Transformer | Findings
Dalla Serra et al. [36] | 2023 | Faster R-CNN | Encoder-Decoder | Faster RCNN-Transformer | Findings
Zhu et al. [39] | 2023 | CNN | Encoder-Decoder | CNN-Transformer | Findings
Nguyen et al. [132] | 2023 | ResNet | Various | CNN-Transformer | Findings
Table 3: Architectures employed by ARRG Research with a breakdown of feature
extractors, conceptual model type and architectures of the proposed models.
Models are designed to generate content for either the findings section, the
impression section, or both sections of a radiology report.

Variational Topic Inference (VTI) was proposed by Najdenkoska et al. [100] who
introduced latent variables in the form of a set of topics to guide the
sentence generation process. Their model used a Hierarchical LSTM for sentence
generation. Upon scrutinising their findings, they observed instances where
sentences exhibited missing words which they attributed to the LSTM’s handling
of long-term dependencies. Building on this work, Najdenkoska et al. [101]
proposed VTI-TRS, taking their original proposal and adopted the transformer
architecture. Each attention head created a specific topic representation,
which is used to create a sentence within the generated report. Within their
new research they trained models employing either an LSTM or a transformer
decoder. Ablation studies revealed that the transformer decoder yielded the
most optimal results.

###  5.2 Encoder-decoder Transformers

Many recent works have leveraged the conventional transformer architecture
[45], which was initially evaluated for NLP tasks such as translating text
from one language to another (see Table 3). These approaches have often
adopted a CNN feature extractor to manipulate the radiographic image into an
appropriate input for the traditional transformer encoder. An illustration of
a typical architecture of this type can be seen in Figure 5(B). As depicted in
Figure 5(C), other researchers [9, 33, 95] utilised the vision transformer
[102] to integrate the image directly into their model.

A radiology report goes beyond identifying anatomical structures and
pathological findings, and also provides detailed information about the size,
location, and severity of any findings. Syeda-Mahmood et al. [103] sought to
extract this additional information from the radiographic images, denoting
these findings as fine finding labels (FFL). They trained two networks, one to
extract traditional labels and another to extract FFLs, the features from
which were then consolidated into a pattern vector. The closest matching
report was then selected by finding the most semantically similar report
vector within their database.

Rather than simply extracting visual features from the images, some works have
introduced an intermediate step to the report generation process.
Nooralahzadeh et al. [104]’s first step employed a Meshed-Memory Transformer
[105] for generating high level textual concepts to be reformed into finer and
more fluent reports by their second step, a transformer language model called
BART [106] which contains a BERT [107] encoder and GPT-2 [108] decoder. Dalla
Serra et al. [34] also divided the ARRG task into two components: factual
triple extraction and report generation. To train their model, ground truth
triples in the form $(entity1,relation,entity2)$ were generated by merging
outputs from RadGraph [29] and ScispaCy [109], which were exploited, along
with the associated clinical indications, to train their transformer encoder
for triplets production. Their report generation stage of the network
incorporated the triplets, in conjunction with the input radiographic image,
to generate the radiology report.

As an alternative to extracting image-level features, Tanida et al. [110]’s
approach, Region-Guided Radiology Report Generation, focused on leveraging a
FasterRCNN to extract 29 distinct anatomical regions of the chest. Their
region selection module then predicted (by binary classification) whether a
sentence should be generated for each anatomical region. Their decoder, a
transformer-based language model that was pretrained on medical abstracts,
then generated sentences for each of the salient anatomical regions.
Similarily, Wang et al. [111] proposed AGFNet, which employed a global and
region feature extractor and a self-adaptive fusion gate to improve their ARRG
model. Their feature extractor produced region features by utilising a Faster
R-CNN head and region of interest pooling. These region features, along with
the global features, were processed by their self-adaptive fusion gate which
used multi-head attention as its main mechanism. An image segmentation
approach was adopted by Nimalsiri et al. [112] within their transformer-based
architecture, incorporating a U-Net architecture [113] to first extract lung
images, from which image features were then derived. Wang et al [114]
clustered discrete grid features with similar semantics into groups to obtain
key region features with the vector of locally aggregated descriptors method
[115] being used to determine similar regions.

The Knowledge-injected U-Transformer is a framework developed by Huang et al.
[47] which focused on extracting and distilling multi-level information and
multiple injected knowledge. Their model contained three main components: a
cross-modal U-Transformer, the injected knowledge distiller, and the region
relationship encoder. Their cross-modal U-Transformer is the main architecture
of the model which contains connections between each level of the transformer,
similar to skip connections in U-Nets [113].

Wang et al. [49] incorporated “expert” tokens (a learnable embedding) as
queries in their expert bilinear attention module, and employed these tokens
in both their transformer encoder and decoder. For their encoder, each expert
token interacted with both vision tokens and other expert tokens to learn to
attend different image regions for image representation to capture
complementary information by an orthogonal loss that minimizes their overlap.
In their decoder, each attended expert token guided the cross-attention
between input words and visual tokens, influencing the generated report. A
metrics-based expert voting strategy generated the final report.

Exploiting the fact that the majority of sentences within a radiology report
describe normal findings, Kale et al. [116] developed a novel method that
incorporated a “normal” template report. Their ResNet-50 visual extractor was
trained to predict MeSH or CheXpert labels from IU-Xray and MIMIC-CXR,
respectively, aiming to identify abnormal areas in the input image. Should any
of these regions be deemed abnormal, their transformer-based text generator
generated a sentence describing the observed pathology. Subsequently, their
replacement module, a BERT [107] architecture, determined whether to replace a
“normal” sentence or append the sentence as an additional statement within the
template report.

Figure 6: A sample prompt and GPT-4V’s responses from Chaoyi et al.’s [133]
qualitative analysis. The chest x-ray demonstrates a case of military
tuberculosis. GPT-4V was requested to generate a radiology report. The
resulting findings section shows that while GPT-4V is capable of conducting a
comprehensive analysis of the chest x-ray, its descriptions and assessment of
the pathology are not accurate.

###  5.3 Pretrained Language Models

Due to their impressive performance in a variety of NLP tasks [18, 19], there
has been a surge of interest investigating pretrained language models in the
ARRG domain, beginning with models such as BERT [107] and GPT [108] which are
based on the transformer encoder and decoder respectively.

BERT-related architectures are currently the most frequently used pretrained
language model within the ARRG domain, either as a stand alone model [84, 124,
125] or as a module within their architecture [104, 82, 65, 24, 23, 116].

Approaches that use BERT as a stand alone model seek to generate the report in
an autoregressive manner [84, 124, 125]. Being an encoder-based model, it
inherently lacks generative capabilities, but it can execute this task by
treating it as a sequential classification task. Generating reports in this
fashion is initiated with a “MASK” token, a placeholder token where the model
is predicting the most likely word. Once predicted, another “MASK” token is
added to the end of the sequence until the model forecasts a “SEP” token,
indicating the conclusion of the sequence. Moon et al. [124] trained their
model using masked language modelling (MLM), where part of a report was masked
with a “MASK” token with the model being tasked to predict the token, and
image-report matching which was used to encourage the model to learn both
visual and textual features to predict whether an image and a report are a
matching pair.

Approaches that integrated BERT into their network applied the module in a
variety of ways. Wang et al. [82] implemented Sentence-BERT [134] in their
self-boosting network, with the module performing an auxiliary task of image-
text matching. Kale et al. [116] trained a BERT-based multilabel text
classifier to identify the relevant sentences within their “normal” template
report so they can be replaced with the generated sentences that describe
pathologies that are present within the input radiographic image. Jeong et al.
[23] adopted the Align Before Fuse (ALBEF) model [135] for their retrieval
based approach to the ARRG task. ALBEF is an encoder model with three
components: a vision transformer [102] for encoding the radiographic images
and two BERT models are used for their text and multimodal encoders.

The introduction of large language models (LLMs), particularly GPT-3 [136],
marked quite a significant shift due to the models impressive generalisation
in zero-shot settings, which is due to an increase in the number of parameters
and training data. While ARRG research with generative LLMs is not yet as
widespread as research focusing on encoder-related LLMs, the availability of
publicly available foundational models such as LLaMA [18] and Falcon [19] have
enhanced their accessibility to the research audience. As such, it is
anticipated that these foundational models will be investigated for the ARRG
task in the near future.

Selivanov et al. [137] sought to improve the quality of the encoder-decoder
generated radiology reports by using the GPT-3 language model. Their model
consists of two parts, show, attend and tell (SAT) [138] and GPT-3 [136]. They
proposed two approaches for combining the models. The first model forced the
models to learn a joint word distribution by concatenating their outputs and
using a feed-forward network to obtain a final score for each token. Their
second approach utilised them in a stacked approach where SAT took the input
radiographic image and created an initial report after which GPT-3 refined the
initial report to create a more fluent report, with this stacked approach
proving more effective.

Chaoyi et al. [133] performed qualitative analysis of OpenAI’s latest model,
GPT4-V [139], which supports multimodal inputs such as images. An example case
study can be seen in Figure 6. The study evaluated the models capacity for
medical diagnosis in various radiology modalities. They assessed GPT-4V’s
efficacy at identifying anatomical structures, disease diagnosis and the
generation of reports for various parts of the body, including the chest. Zero
shot learning was used with images obtained from Radiopaedia
333https://radiopaedia.org/.GPT-4V was observed to be proficient at
identifying between image modalities and anatomy, but faces challenges in
disease diagnosis and report generation. Due to the limitation that OpenAI
have not yet released an API for this model, a quantitative use is not yet
possible.

Other papers of significance where various approaches to ARRG were introduced
by way of pretrained language models are [120, 121, 125, 84, 56, 37, 132].

###  5.4 Attention Modules

Due to the fine-grained differences within radiographic images, researchers
have often sought to develop novel attention mechanisms. These attention
modules aim to more effectively capture the subtle changes within the images,
ultimately enhancing the robustness of features and improving the decoder’s
ability to generate accurate reports. Various different solutions for the
attention mechanism have been developed, for example [89, 122, 31, 69].

To effectively obtain the visual features of pathologies, Song et al. [122]
proposed a cross-modal contrastive attention (CMCA) model which compared the
input features extracted by their encoder against the most similar historical
case retrieved from their database, where each case contained global and
spatial visual features along with the corresponding report. The contrastive
features created by their CMCA module were then used, along with the retrieved
report, to guide the report generation process.

Adaptive Multilevel Multi-Attention was developed by Gajbhiye et al. [89]
which contained two separate attention mechanisms, adaptive visual semantic
attention (AVSA) and visual based language attention (VBLA). AVSA adaptively
selected the weight of visual global information and semantic patterns to
attend to their local visual features while VBLA module was responsible for
learning the correlation and inter-dependency between the consecutive words
and sentences. Align Hierarchical Attention was proposed by You et al. [118]
which was integrated into their AlignTransformer model to help correspond
visual features with disease tags.

Gu et al. [131] leveraged the multi-perspective nature of radiology images to
introduce cross-view attention. This operated through two branches, a single-
view branch preserving single-view features, and a multi-view branch combining
features extracted from both frontal and lateral images using squeeze-
excitation attention to create multi-view features. The single-view features,
multi-view features and medical concepts (MTI tags from IU-Xray dataset) were
then used by their Medical Visual-Semantic LSTM to help generate a radiology
report.

Memory-augmented Sparse Attention (MSA) was proposed by Wang et al. [31] with
the aim of capturing second or even higher-order interactions by incorporating
bilinear pooling into self-attention. In their MSA module, global features
were considered as the query, while local features served as the key and
value. Additional ”memory slots” were also introduced to the keys and values,
facilitating the encoding and collection of features from all preceding
processes with the intention of generating long reports more accurately.
Sparse attention was also utilised by Yan et al. [69] who developed a two-
tiered approach consisting of region sparse attention and word sparse
attention. To enhance semantic alignment in their model, they harnessed the
MeSH labels from the MIMIC-CXR dataset. Their region sparse attention acquired
region features corresponding to each MeSH label, while their word sparse
attention assigned increased weights to the MeSH words present in a report.

Figure 7: A representation of the retrieval based architecture for the ARRG
task where, given an input radiographic image, the most similar report, which
is calculated through a distance function, is retrieved from a collection of
reports. Several researchers have a corpus that contains both sentences and
reports, for example [67, 119, 58, 23].

###  5.5 Retrieval-based methods

An alternative solution to the ARRG task is the retrieval approach where
models consider a corpus of radiological knowledge to find similar reports or
sentences to create their reports. An illustration of a retrieval-based method
can be seen in Figure 7. Yang et al. [119] developed MedWriter, which
incorporated a hierarchical retrieval mechanism comprising, (i) a visual-
language retrieval module operated on the report level, using visual features
to source the most semantically similar report template in their retrieval
pool, (ii) a language-language retrieval module working on the sentence level,
where a set of candidate sentences are retrieved from the retrieval pool that
have potential to be the next sentence, and (iii) a hierarchical-LSTM decoder
fusing the features from the previous modules to generate their report. Wu et
al. [96] devised a comparable approach to Yang et al. [119], which either
retrieved the image-report pair with the most similar visual features or, if
prior imaging studies were available for a patient, then these were retrieved
instead. Their model demonstrated the capability to process multiple prior
imaging studies to assist with the generation of the current radiology report.

A retrieval-based approach called CXR-RePaiR was developed by Endo et al. [67]
which used a CLIP model with a retrieval corpus generated by collating the
reports and sentences of the MIMIC-CXR dataset. The models were first trained
on natural image-text pairs, followed by training on radiology image-report
pairs where the model was trained to produce higher dot-product similarity
values for a matching image-report pair. Their ablation studies found that
sentence retrieval resulted in improved performance over report retrieval as
sentence retrieval was considered to have a greater set of possible outputs to
select from. Ramesh et al. [55] retrained CXR-RePaiR [67] on their version of
MIMIC, CXR-PRO, finding their curated dataset improved their model’s
performance. Similarly to Endo et al. [67], Jeong et al [23] utilised CLIP to
improve the image-text matching score of their model, enabling a more accurate
measure of the similarity of a chest X-ray image and radiology report for
report retrieval. Their approach used a model called ALBEF [135] which
contained three fused encoders – an image, text and a multimodal encoder.
Instead of employing the dot-product, they applied cosine similarity as their
distance measure function.

Wang et al. [58] developed Relation-paraNet which unified retrieval and report
generation through the use of their relational-topic encoder by taking image
features and sentence embeddings of the previously generated sentences as
input. The contextual topic embeddings, produced by their encoder, were used
by their adaptive generator to determine whether to retrieve a template from
their template database or generate a new sentence.

Wu et al. [24] proposed an LSTM module focused on creating the impression
section of a report, and a BERT-based module that retrieved a set of
appropriate sentences for the findings section. Both modules employed the same
image encoder but each task was associated with its own final, fully connected
layer to tailor the visual features specific to each task.

##  6 Utilising Knowledge and Multiple Modalities

Despite the improvements in models and architectures, e.g. the rewards
harvested by the transformer, significant room for improvements remain in the
ARRG domain. Babar et al. [140] investigated the efficacy of the encoder-
decoder model and found that the evaluated ARRG models learnt an unconditioned
model, not effectively attending to the image features during report
generation. One of their recommendations was to utilise medical knowledge
within the model.

Various ARRG researchers have developed methods to inject medical knowledge
into their models [38, 89, 70, 127, 94]. Kim et al. [127] argued that due to
the lack of prior knowledge, current approaches generated results containing
references to non-existent prior examinations. They introduced comparison
priors which were extracted from reports through a rule-based classifier.
During training, reports were classified as being a first (negative) or a
follow up examination (positive). They integrated their labeler into two
publicly available models, R2Gen [50] and M2Tr [105], improving both models
scores on common NLG metrics by reducing the occurrence of incorrectly
mentioned prior examinations. Wang et al. [31] created a Medical Concepts
Generation Network. using the entities from RadGraph [29] as medical semantic
concepts, and selecting 768 concepts according to their occurrence frequency.
Multi-label classification was then performed to determine which of the
concepts were contained within the input radiographic image.

###  6.1 Multimodal Inputs

ARRG is a multimodal task where both radiographic images and reports are used
at various points throughout a contribution’s pipeline. Reports have been used
for implementing training approaches, such as CLIP [67, 65, 77, 24], for
example providing extra information to improve feature extraction [103, 34,
36, 35] or to create knowledge graphs [68, 21, 94]. In the inference phase
however, the majority of ARRG architectures typically feature a unimodal
encoder that takes radiographic images as input. Information such as clinical
indications or prior examinations are often used by radiologists while
reporting as they provide additional information about a patient’s condition.
To emulate this working style, some researchers have attempted to leverage
this additional information to enhance their models during inference as this
information is readily available at the time of reporting. Instances of
research incorporating prior information during inference can be seen in
Figure 8.

Harnessing multimodal inputs, Nguyen et al. [32] incorporated radiographic
images and clinical details, which were fused into contextualised disease-
related embeddings by their classification module. Similarily, Shang et al.
[33] also included clinical referrals to provide their encoder with better
learning capability. Dalla Serra et al. [34] used clinical indications to
assist their feature extractor with creating more impactful features. Nguyen
et al. [132] leveraged clinical indications alongside the classifications from
their ResNet encoder to assist their model with generating reports. Through
the use of a transformer-based backbone, Mondal et al. [35] incorporated early
fusion of radiology images and clinical indications to assist in classifying
their samples.

Rather than having separate visual and textul encoders, Jeong et al. [23]
harnessed the ALBEF model [135], a multimodal encoder, to create multimodal
representations of the radiographic image and reports. They utilised a
trainable, shared matrix to learn new representations and used contrastive
learning to help create more meaningful embeddings. Wang et al. [126] utilised
their cross-modal fusion network to fuse visual features and disease tags of
the current radiographic image.

Using the JLiverCT dataset, Nishino et al [38] introduced a temporal aspect to
their model. This dataset consists of multi-phase CT examinations, where
scanning was performed sequentially over a period of time, often tracking a
contrast medium perfusing throughout a patients body. Their model exploited
this temporal information to generate chronologically accurate reports.
Instead of examining each sample individually, Bannur et al. [37] took
advantage of the longitudinal nature of MIMIC-CXR by leveraging the images and
reports from patients’ previous imaging studies as a secondary input to their
model. These additional images and reports provided their model with prior
information that enabled the model to generate sentences on the status of a
noted pathology, and if its condition was improving or deteriorating. Dalla
Serra et al. [36] also leveraged the temporal nature of MIMIC-CXR, using a
Faster-R CNN object detector to extract anatomical tokens of the current and
prior images. These tokens were aligned, concatenated and projected into joint
representations before being passed alongside the clinical indications to
their transformer based decoder to generate a report. Prior imaging was also
used by Zhu et al. [39] to assist with generating the findings section of a
radiology report with their transformer decoder being supplied with the
longitudinal multimodal embeddings of the prior images and reports alongside
the current image embedding.

Figure 8: Illustration of diverse forms of multimodal input used within ARRG
research during inference. (A) demonstrates the most common, unimodal form of
input to an ARRG model, the radiographic image. (B) represents models which
adopted a multimodal input of images and text, with the text often being the
clinical indications found within the image’s corresponding reports [32, 33,
34]. (C) shows a model which implemented temporal information from prior
radiographic images [37].

###  6.2 Memory Modules

Some works have introduced memory mechanisms to provide a way of storing
knowledge to boost a model’s ability to generate clinically accurate reports
[129] Chen et al. [51] employed a relational memory module in their
transformer-based architecture to record information from previous generation
processes and a novel memory-driven conditional layer normalization to
integrate relational memory into their transformer architecture. This research
contribution is one of the most frequently compared against models [68, 9, 34,
31, 70, 94, 22, 49, 47, 127, 20]. Chen et al. [50] later expanded the R2Gen
design by introducing the cross-modal memory network module. Rather than
mapping between images and text directly from their encoded representations,
they used a shared memory block as a medium to record the alignment between
images and reports.

Wang et al. [71] harnessed prior information from the training set of the
MIMIC-CXR dataset to guide the initialisation of their semantic cross-modal
prototype matrix. Their model, XPRONET, proposed a cross-modal prototype
network, distilling image and textual information into single-model features.
Central to their module was the prototype memory matrix, which contained image
pseudo-labels.

Yang et al. [129] developed a memory module to be used as a knowledge base
which was used to model the relationships between the image and textual
features. Multi-head attention was the main mechanism used to update their
knowledge bases’ memory values.

###  6.3 Knowledge-based Graphs

Knowledge graphs were popularised by Google in 2012 as a method to achieve a
more intelligent search engine. Since its conception, it has gained popularity
in deep learning research as a useful method to inject semantic knowledge into
a model. A knowledge graph is essentially a structured base of knowledge with
a directed graph structure where the nodes of the graph represent concepts
while the edges of the graph represent semantic relations between the nodes.
Due to this, a knowledge graph can well represent the relationship between
nodes [30], which has been utilised by many researchers to improve the
performance of their models.

Amongst approaches which have manually built their knowledge graphs to hand
pick the most pertinent categories, Zhang et al. [93] constructed a chest
abnormality graph based on prior knowledge. They selected 20 categories which
facilitated the classification and report generation of their model. Visual
features were extracted from the radiographic images, and then attention
mechanisms and graph convolutions were applied to learn the graph embedded
features.

Liu et al [48] utilised a graph for a module in their transformer-based
architecture, called the posterior-and-prior knowledge exploring-and-
distilling model. They built their graph from the training splits of MIMIC-CXR
and IU-Xray, where common topics are defined as nodes and are grouped by organ
or body part. In another work, Liu et al. [68] again generated their knowledge
graph from the training set of the MIMIC-CXR dataset, while Zhao et al. [21]
used IU-Xray and CH-Xray datasets. The majority of approaches, such as [48,
68, 21, 58, 20], consider abnormalities as nodes while edge weights are
calculated by the normalised co-occurrence of different nodes within their
corpus, often a training set of an ARRG dataset.

Zhang et al. [30] constructed their knowledge graph at first by using CheXpert
[63] to extract and classify the given medical report for 14 common diseases,
and then used Sentencepiece [141] to mine reports and extract any additional
disease labels not found by CheXpert. Edges were defined by the amount of co-
occurrences, with two nodes considered to be associated if they occur together
more than the calculated average co-occurrenced rate. Hou et al. [94] adopted
CheXbert [78] to help create their observation graph which were then aligned
with the visual features from the input radiographic image. Once trained, the
observation graph and radiographic images are considered as a pair during
inference to guide the report generation process.

Yang et al. [11] implemented a CNN-Transformer architecture, enhanced with two
forms of knowledge graphs, general and specific. The general knowledge graph
was based on RadGraph [29] while the specific knowledge graph was extracted
from reports retrieved from their corpus of radiology reports, after which the
RadGraph relation extractor was applied to extract entities and relations. The
graphs were fused with the input using knowledge-enhanced multi-head
attention. Wang et al. [126]’s approach, a Transformer architecture with
graph-based distinctive attention, retrieved the $K$ most relevant images and
used them to construct a knowledge graph. The latent representation of each
image served as a node, with the $\mathcal{L}_{2}$ distance score between
paired images representing the weight of an edge in the graph, which was
leveraged to extract more meaningful semantic features. Wang et al. [92]
injected prior knowledge through a knowledge graph that was constructed
through text mining the RadLex Lexicon [142].

Yan introduced MaKG [123], a memory aligned knowledge graph of clinical
abnormalities which sought to better learn the visual patterns of
abnormalities and their relationships. Similar to Chen et al. [51], they
implemented memory slots to record the features of abnormal regions from which
graph embeddings were computed through the use of multi-headed attention
followed by graph attentional layer. Each node embedding was used to predict
the probability of an abnormality being present. Their transformer-based
decoder then generated a report by attending to their graph’s node embeddings.

Shang et al. [33] used a fully-connected graph to guide their multimodal
encoder, with each image patch and text token treated as an independent, fully
connected node. The weight of the edges are decided by an attention mechanism
called optional cross attention. Yan et al. [130] introduced a method to
automatically generate a fine-grained, attributed abnormality graph,
consisting of interconnected abnormality nodes and attribute nodes. Their
method generated the graph through dataset annotations, radiological reports,
and the RadLex lexicon [142].

##  7 Evaluation Methods

Model | Year | BLEU | ROUGE | CIDEr | METEOR | fact [87] | AUC |
Clinical Efficacy |
Case Study |
Clinical Validation | Reproducible
---|---|---|---|---|---|---|---|---|---|---|---
R2Gen [51] | 2020 | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  | ✓
Syeda-Mahmood et al. [103] | 2020 | ✓ | ✓ |  | ✓ |  |  |  |  |  |
Zhang et al. [93] | 2020 | ✓ | ✓ | ✓ |  |  | ✓ |  |  |  |
Lovelace & Mortazavi [80] | 2020 | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |  |  | ✓
Boag et al. [117] | 2020 | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |  |  | ✓
VTI [100] | 2021 | ✓ | ✓ |  | ✓ |  | ✓ | ✓ |  |  | ✓
CMCL [27] | 2021 | ✓ | ✓ | ✓ |  |  |  |  |  |  |
CA [90] | 2021 | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  | ✓
Nooralahzadeh et al. [104] | 2021 | ✓ | ✓ |  | ✓ |  |  | ✓ |  |  | ✓
Yan et al. [28] | 2021 | ✓ | ✓ |  | ✓ |  |  |  |  |  ✓ | ✓
AlignTransformer [118] | 2021 | ✓ | ✓ |  | ✓ |  |  |  | ✓ |  ✓ |
PPKED [48] | 2021 | ✓ | ✓ | ✓ |  |  |  |  | ✓ |  |
KGAE [68] | 2021 | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  |
R2GenCMN [50] | 2021 | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  | ✓
Nguyen et al. [32] | 2021 | ✓ | ✓ |  | ✓ |  | ✓ | ✓ |  |  ✓ | ✓
MedWriter [119] | 2021 | ✓ | ✓ | ✓ |  |  | ✓ |  |  |  ✓ |
CXR-RePaiR [67] | 2021 | BLEU-2 |  |  |  |  |  | ✓ | ✓ |  | ✓
CDGPT2 [120] | 2021 | ✓ | ✓ | ✓ | ✓ |  |  |  | ✓ |  ✓ | ✓
Wang et al. [82] | 2021 | ✓ | ✓ | ✓ |  |  |  |  | ✓ |  |
Muira et al. [87] | 2021 | ✓ |  | ✓ |  |  | ✓ |  | ✓ |  | ✓
Medical-VLBERT [121] | 2021 | ✓ | ✓ | ✓ |  |  |  |  | ✓ |  |
Zhao et al. [21] | 2022 | ✓ | ✓ |  | ✓ |  |  |  | ✓ |  |
Wang et al. [9] | 2022 | ✓ | ✓ | ✓ |  |  |  |  | ✓ |  |
Radgraph Reward [84] | 2022 | BLEU-4 | ✓ |  |  | ✓ |  | ✓ | ✓ |  |
Yang et al. [83] | 2022 | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ | ✓ |  |
Qin and Song [85] | 2022 | ✓ | ✓ |  | ✓ |  |  | ✓ |  |  |
Yang et al. [11] | 2022 | ✓ | ✓ | ✓ |  |  |  | ✓ |  |  |
VTI-TRS [101] | 2022 | ✓ | ✓ |  | ✓ |  |  |  | ✓ |  | ✓
XPRONET [71] | 2022 | ✓ | ✓ |  | ✓ |  |  |  | ✓ |  | ✓
CMCA [122] | 2022 | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  |
MaKG [123] | 2022 | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ |  |  |
Zhang et al. [30] | 2022 | ✓ | ✓ |  | ✓ |  | ✓ | ✓ | ✓ |  |
Nazarov et al. [77] | 2022 | ✓ | ✓ | ✓ | ✓ |  |  | ✓ | ✓ |  | ✓
MATNet [33] | 2022 | ✓ | ✓ |  | ✓ |  | ✓ | ✓ |  |  |
Dalla Serra et al. [34] | 2022 | ✓ | ✓ |  | ✓ |  |  | ✓ |  |  ✓ |
SGF [70] | 2022 | ✓ | ✓ |  | ✓ |  |  |  | ✓ |  |
JPG [72] | 2022 | ✓ | ✓ |  | ✓ |  |  |  |  |  |
MedViLL [124] | 2022 |  |  |  |  |  |  |  |  |  | ✓
AMLMA [89] | 2022 | ✓ | ✓ | ✓ | ✓ |  |  |  | ✓ |  |
Relation-paraNet [58] | 2022 | ✓ | ✓ | ✓ |  |  |  |  | ✓ |  |
Clinical-BERT [125] | 2022 | ✓ | ✓ | ✓ | ✓ |  |  | ✓ | ✓ |  |
RepsNet [65] | 2022 | ✓ |  |  |  |  |  |  | ✓ |  | ✓
MSAT [31] | 2022 | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  | ✓
Wang et al. [92] | 2022 | ✓ | ✓ | ✓ |  | ✓ |  | ✓ |  |  | ✓
CoPlan [38] | 2022 | BLEU-4 | ✓ |  |  |  | ✓ |  |  |  |
Yan et al. [69] | 2022 | ✓ | ✓ | ✓ | ✓ |  |  | ✓ | ✓ |  |
Wang et al. [126] | 2022 | ✓ | ✓ |  | ✓ |  |  |  | ✓ |  |
MRCL [73] | 2022 | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  |
Boecking et al. [56] | 2022 |  |  |  |  |  |  |  |  |  | ✓
DeltaNet [96] | 2022 | ✓ | ✓ | ✓ |  |  |  | ✓ | ✓ |  | ✓
TrMRG [95] | 2023 | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  |
CheXPrune [91] | 2023 | ✓ | ✓ | ✓ |  |  |  |  | ✓ |  |
Kim et al. [127] | 2023 | ✓ | ✓ | ✓ |  |  |  |  |  |  |
MKCL [22] | 2023 | ✓ | ✓ | ✓ |  |  |  |  | ✓ |  | ✓
CvT-212DistilGPT2 [128] | 2023 | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |  |  | ✓
ASGK [20] | 2023 | ✓ | ✓ | ✓ |  |  |  |  |  |  | ✓
BioViL-T [37] | 2023 | BLEU-2 |  |  |  |  |  | ✓ |  |  | ✓
ORGAN [94] | 2023 | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  |
X-REM [23] | 2023 | BLEU-2 |  |  |  |  |  | ✓ |  |  | ✓
MRCL [73] | 2023 | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  |
KiUT [47] | 2023 | ✓ | ✓ |  | ✓ |  |  |  | ✓ |  | ✓
TIMER [86] | 2023 | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  | ✓
Yang et al. [129] | 2023 | ✓ | ✓ | ✓ |  |  |  | ✓ | ✓ |  |
Kale et al. [116] | 2023 | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |  |  ✓ |
METransformer [49] | 2023 | ✓ | ✓ | ✓ | ✓ |  |  | ✓ | ✓ |  |
ATAG [130] | 2023 | ✓ | ✓ | ✓ |  |  | ✓ | ✓ |  |  |
Selivanov et al. [137] | 2023 | ✓ | ✓ | ✓ |  |  | ✓ | ✓ |  |  |
RGRG [110] | 2023 | ✓ | ✓ | ✓ | ✓ |  |  | ✓ | ✓ |  | ✓
Gu et al. [131] | 2023 | ✓ | ✓ | ✓ |  |  |  |  | ✓ |  ✓ |
UAR [74] | 2023 | ✓ | ✓ | ✓ | ✓ |  |  |  | ✓ |  |
Dalla Serra et al. [36] | 2023 | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  |
Zhu et al. [39] | 2023 | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  |
Nguyen et al. [132] | 2023 | BLEU-2 |  |  |  |  |  | ✓ | ✓ |  |
Table 4: Various recent methods and how they assess the performance of their
model. Most ARRG researchers use NLP metrics such as BLEU, ROUGE, CIDEr and
METEOR or ARRG specific methods such as Clinical Efficacy metrics. Other
metrics such as fact [87] and Area under the Curve are less frequently used.
In terms of qualitative evaluation, a significant number of approaches perform
their own case study, while some seek clinical experts to externally validate
their models. Empty cells suggest an approach did not use that evaluation
method.

The aim of ARRG is to generate a radiology report that is semantically
equivalent to the report a radiologist would create. Assessing the quality of
the generated report is a challenging and subjective task [143]. Generated
reports are required to be fluent and grammatically and factually correct.
Table 4 demonstrates the metrics that are commonly adopted in the ARRG
literature, noting many were originally designed for NLP tasks, such as
machine translation [144, 145], summarisation [146] and image captioning
[147].

NLP Metrics – The common NLP evaluation metrics used are:

  * •

BLEU - Bilingual Evaluation Understudy [144] – BLEU is a precision-based
metric originally designed for machine translation by analysing n-grams up to
a length of four (hence for eaxample BLEU-1 for unigrams or BLEU-3 for
trigrams etc). It primarily penalises or rewards the generated text depending
on how it matches the reference text in length, word choice, and order.

  * •

METEOR - Metric for Evaluation of Translation with Explicit ORdering [145] –
This is a precision and recall-based measure which extends BLEU-1. Originally
designed for machine translation, it matches unigrams of the generated text to
the reference text based on their exact form, stemmed form and meaning. It
computes the harmonic mean of unigram precision and recall, biased towards
recall. A multiplicative factor is used to reward identically ordered
unigrams.

  * •

ROUGE - Recall-Oriented Understudy for Gisting Evaluation [146] – ROUGE is a
set of metrics, of which ROUGE-L is employed within ARRG. It is a recall-
oriented metric originally designed for summarisation tasks and measures the
length of the longest common subsequence between two texts. ROGUE-L calculates
the weighted harmonic mean of precision and recall, favouring recall. It does
not require consecutive matches but in-sequence matches that reflect sentence
level word order as n-grams.

  * •

CIDEr – Consensus-based Image Description Evaluation [147] – This is also a
precision and recall based metric originally developed to evaluate image
captioning. CIDEr evaluates the likeness between the generated text and
reference text by measuring the cosine similarity between the Term Frequency-
Inverse Document Frequency weighting of n-grams, stressing the importance of
content words in the evaluation. The metric also introduces a Gaussian penalty
factor that rewards length similarity between the generated and reference
texts.

The metrics described above focus on evaluating the lexical similarity between
a generated and reference report, but there is an ongoing debate that it is
more important to assign higher scores to semantically equivalent reports
[117, 140]. For example, the sentences “The heart is within normal size and
contour.” and “No observed cardiomegally.” are semantically equivalent but
would score zero on the NLP metrics commonly used.

An interesting metric that has not been picked up in ARRG, but has been
implemented for NLP tasks, such as machine translation, summarisation and
image captioning, is BERTScore [148], where rather than assessing the
generated sentences, token similarity is computed using contextual embeddings
which enables a more robust assessment for semantically equivalent reports. Yu
et al. [143] quantitatively examined the correlation between automated metrics
(such as BLEU and BERTScore) and the scoring of radiology reports by
radiologists to understand how to meaningfully measure progress on ARRG. They
found that n-gram matching methods performed worse than semantic methods such
as BERTScore in evaluating the false prediction of a finding, concluding that
BERTScore also had better radiologist alignment.

###  7.1 Clinical Efficacy

A number of works have attempted to address the issue of evaluating semantic
equivalence by developing clinical efficacy (CE) metrics that focus on the
medical relevance and accuracy of the generated content. Chen et al. [51]
developed an evaluation method employing the rule-based labeller CheXpert
[63], which generates 14 labels for the generated and reference reports
relating to thoracic pathologies and support devices. If two reports are
semantically equivalent then the derived CE score will be high as both reports
will have received the same labels from the CheXpert labeller. Chen et al
[51]’s method has been utilised to assess an ARRG models performance by
various researchers [50, 34, 124, 11, 49, 104, 130, 67]. Endo et al. [67]
utilised CheXpert to create their own measure, $S_{emb}$, to evaluate semantic
equivalence through cosine similarity. It is important to note that a
limitation of these CheXpert-based metrics is that they can only be utilised
for models using the MIMIC-CXR dataset, as the labeling schema of CheXpert was
designed with this dataset in mind.

Yan et al. [130] proposed a metric called Radiology Report Quality Index
(RadRQI) to evaluate the accuracy of radiology related abnormalities with the
clinical attributes. Given a radiology report, keywords related to
abnormalities and their attributes are extracted using RadLex [142]. The
association between abnormalities and their attributes are then contextualised
by RadGraph [29], ensuring negations are considered. The proposed RadRQI-F1
score aimed to reflect the correctness of mentioned abnormality with
associated attributes. In addition, the number of abnormality categories with
non-zero F1 score, denoted as RadRQI-Hits, is also reported to show the
coverage of distinct abnormality categories in the generated reports.

Model |  Journal or Conference | Year | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE | CIDEr | METEOR
---|---|---|---|---|---|---|---|---|---
Lovelace & Mortazavi [80] | EMNLP | 2020 | 0.415 | 0.272 | 0.193 | 0.146 | 0.318 | 0.316 | 0.159
R2Gen [51] | EMNLP | 2020 | 0.470 | 0.304 | 0.219 | 0.165 | 0.371 |  | 0.187
Syeda-Mahmood et al. [103] | MICCAI | 2020 | 0.560 | 0.510 | 0.500 | 0.490 | 0.580 |  | 0.580
VTI [100] | MICCAI | 2021 | 0.493 | 0.360 | 0.291 | 0.154 | 0.375 |  | 0.281
CDGPT2 [120] | IMU | 2021 | 0.387 | 0.245 | 0.166 | 0.111 | 0.289 | 0.257 | 0.164
CMCL [27] | IJCNLP | 2021 | 0.473 | 0.305 | 0.217 | 0.162 | 0.378 |  | 0.186
Nooralahzadeh et al. [104] | EMNLP | 2021 | 0.486 | 0.317 | 0.232 | 0.173 | 0.390 |  | 0.192
Wang et al. [82] | CVPR | 2021 | 0.487 | 0.346 | 0.270 | 0.208 | 0.359 | 0.452 |
CA [90] | IJCNLP | 2021 | 0.492 | 0.314 | 0.222 | 0.169 | 0.381 |  | 0.193
AlignTransformer [118] | MICCAI | 2021 | 0.484 | 0.313 | 0.225 | 0.173 | 0.379 |  | 0.204
PPKED [48] | CVPR | 2021 | 0.483 | 0.315 | 0.224 | 0.168 | 0.376 | 0.351 |
KGAE [68] | NeurIPS | 2021 | 0.512 | 0.327 | 0.240 | 0.179 | 0.383 |  | 0.195
R2GenCMN [50] | IJCNLP | 2021 | 0.475 | 0.309 | 0.222 | 0.170 | 0.375 |  | 0.191
Nguyen et al. [32] | EMNLP | 2021 | 0.515 | 0.387 | 0.293 | 0.235 | 0.436 |  | 0.219
MedWriter [119] | IJCNLP | 2021 | 0.471 | 0.336 | 0.238 | 0.166 | 0.382 | 0.345 |
Zhao et al. [21] | ICCPR | 2021 | 0.502 | 0.326 | 0.232 | 0.176 | 0.381 |  | 0.199
Wang et al. [92] | AMIA | 2022 | 0.450 | 0.301 | 0.213 | 0.158 | 0.384 | 0.340 |
Radgraph Reward [84] | EMNLP | 2022 |  |  |  | 0.139 | 0.327 |  |
Yang et al. [83] | Neurocomputing | 2022 | 0.491 | 0.334 | 0.242 | 0.18 | 0.397 | 0.469 | 0.212
Qin and Song [85] | ACL | 2022 | 0.494 | 0.321 | 0.235 | 0.109 | 0.384 |  | 0.201
Yang et al. [11] | Med Image Anal. | 2022 | 0.496 | 0.327 | 0.238 | 0.178 | 0.381 | 0.382 |
VTI-TRS [101] | Med Image Anal. | 2022 | 0.503 | 0.394 | 0.302 | 0.170 | 0.390 |  | 0.230
SGF [70] | MICCAI | 2022 | 0.467 | 0.334 | 0.261 | 0.215 | 0.415 |  | 0.201
XPRONET [71] | ECCV | 2022 | 0.525 | 0.357 | 0.262 | 0.199 | 0.411 | 0.359 | 0.22
JPG [72] | ICCL | 2022 | 0.479 | 0.319 | 0.222 | 0.174 | 0.377 |  | 0.193
CMCA [122] | ICCL | 2022 | 0.497 | 0.349 | 0.268 | 0.215 | 0.392 |  | 0.209
MaKG [123] | JBHI | 2022 |  |  |  |  | 0.353 | 0.681 | 0.392
AMLMA [89] | CMPB | 2022 | 0.471 | 0.315 | 0.231 | 0.172 | 0.376 | 0.381 | 0.247
MaKG [123] | BioNLP | 2022 |  |  |  |  | 0.353 | 0.523 | 0.378
Zhang et al. [30] | Appl. Sci. | 2022 | 0.505 | 0.379 | 0.303 | 0.251 | 0.446 |  | 0.218
ATAG [130] | IEEE TMI. | 2022 |  |  |  |  | 0.341 | 0.380 |
Relation-paraNet [58] | IEEE TC | 2022 | 0.505 | 0.329 | 0.230 | 0.168 | 0.372 | 0.317 |
Clinical-BERT [125] | AAAI | 2022 | 0.495 | 0.330 | 0.231 | 0.170 | 0.376 | 0.432 |
MATNet [33] | IEEE SPL | 2022 | 0.518 | 0.387 | 0.308 | 0.254 | 0.446 |  | 0.222
RepsNet [65] | MICCAI | 2022 | 0.580 | 0.440 | 0.320 | 0.270 |  |  |
DeltaNet [96] | ICCL | 2022 | 0.485 | 0.324 | 0.238 | 0.184 | 0.379 | 0.802 |
TrMRG [95] | IEEE Access | 2023 | 0.532 | 0.344 | 0.233 | 0.158 | 0.387 | 0.500 | 0.218
KiUT [47] | CVPR | 2023 | 0.525 | 0.36 | 0.251 | 0.185 | 0.409 |  | 0.242
CheXPrune [91] | JAIHC | 2023 | 0.543 | 0.445 | 0.374 | 0.320 | 0.598 | 0.322 |
MRCL [73] | JAIHC | 2023 | 0.458 | 0.324 | 0.238 | 0.180 | 0.369 | 0.287 | 0.206
Kim et al. [127] | BioNLP | 2023 | 0.438 | 0.28 | 0.201 | 0.155 | 0.351 | 0.631 |
ORGAN [94] | JBI | 2023 | 0.510 | 0.346 | 0.255 | 0.195 | 0.399 |  | 0.205
MKCL [22] | JBI | 2023 | 0.490 | 0.311 | 0.222 | 0.167 | 0.385 | 0.523 | 0.206
TIMER [86] | JMLR | 2023 | 0.493 | 0.325 | 0.238 | 0.186 | 0.383 |  | 0.204
CvT-212DistilGPT2 [128] | AIMI | 2023 | 0.473 | 0.304 | 0.224 | 0.175 | 0.376 | 0.694 | 0.200
Yang et al. [129] | Med Image Anal. | 2023 | 0.497 | 0.319 | 0.230 | 0.174 | 0.399 | 0.407 |
ASGK [20] | WWWJ | 2023 |  |  |  | 0.125 | 0.279 | 0.306 |
METransformer [49] | CVPR | 2023 | 0.483 | 0.322 | 0.228 | 0.172 | 0.380 | 0.435 | 0.192
Gu et al. [131] | Bioengineering | 2023 | 0.460 | 0.294 | 0.207 | 0.152 | 0.385 | 0.409 |
UAR [74] | JMLR | 2023 | 0.530 | 0.365 | 0.263 | 0.200 | 0.405 | 0.501 | 0.218
Table 5: IU-Xray Dataset: Comparative Model Performance. All values were extracted from the original papers. Best results are in bold and the second-best underlined. Empty cells denote metrics not evaluated by the model. The table is grouped by year. Model |  Journal or Conference | Year | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE | CIDEr | METEOR
---|---|---|---|---|---|---|---|---|---
Lovelace & Mortazavi [80] | EMNLP | 2020 | 0.415 | 0.272 | 0.193 | 0.146 | 0.318 | 0.316 | 0.159
Boag et al. [117] | NeurIPS | 2020 | 0.305 | 0.201 | 0.137 | 0.092 |  | 0.850 |
R2Gen [51] | EMNLP | 2020 | 0.470 | 0.304 | 0.219 | 0.165 | 0.371 |  | 0.187
VTI [100] | MICCAI | 2021 | 0.418 | 0.293 | 0.152 | 0.109 | 0.302 |  | 0.177
CMCL [27] | IJCNLP | 2021 | 0.344 | 0.217 | 0.14 | 0.097 | 0.281 |  | 0.133
CA [90] | IJCNLP | 2021 | 0.350 | 0.219 | 0.152 | 0.109 | 0.283 |  | 0.151
Nooralahzadeh et al. [104] | EMNLP | 2021 | 0.378 | 0.232 | 0.154 | 0.107 | 0.272 |  | 0.145
Yan et al. [28] | EMNLP | 2021 | 0.373 |  |  | 0.107 | 0.274 |  | 0.144
AlignTransformer [118] | MICCAI | 2021 | 0.378 | 0.235 | 0.156 | 0.112 | 0.283 |  | 0.158
PPKED [48] | CVPR | 2021 | 0.360 | 0.224 | 0.149 | 0.106 | 0.284 |  | 0.149
KGAE [68] | NeurIPS | 2021 | 0.369 | 0.231 | 0.156 | 0.118 | 0.295 |  | 0.153
R2GenCMN [50] | EMNLP | 2021 | 0.353 | 0.218 | 0.148 | 0.106 | 0.278 |  | 0.142
Nguyen et al. [32] | EMNLP | 2021 | 0.495 | 0.360 | 0.278 | 0.224 | 0.222 |  | 0.219
MedWriter [119] | IJCNLP | 2021 | 0.438 | 0.297 | 0.216 | 0.164 | 0.332 | 0.306 |
CXR-RePaiR [67] | JMLR | 2021 | 0.438 | 0.297 | 0.216 | 0.164 | 0.332 | 0.306 |
Wang et al. [9] | IEEE TMI | 2022 | 0.373 | 0.235 | 0.162 | 0.120 | 0.282 | 0.299 | 0.143
Radgraph Reward [84] | EMNLP | 2022 |  |  | 0.259 | 0.116 |  |  |
Yang et al. [83] | Neurocomputing | 2022 | 0.362 | 0.224 | 0.15 | 0.108 | 0.276 | 0.157 | 0.142
Qin and Song [85] | ACL | 2022 | 0.381 | 0.232 | 0.155 | 0.109 | 0.287 |  | 0.151
Yang et al. [11] | Med Image Anal. | 2022 | 0.363 | 0.228 | 0.156 | 0.115 | 0.284 | 0.203 |
VTI-TRS [101] | Med Image Anal. | 2022 | 0.475 | 0.314 | 0.196 | 0.136 | 0.315 |  | 0.191
XPRONET [71] | ECCV | 2022 | 0.344 | 0.215 | 0.146 | 0.105 | 0.279 |  | 0.138
CMCA [122] | COLING | 2022 | 0.360 | 0.227 | 0.156 | 0.117 | 0.287 |  | 0.148
MaKG [123] | BioNLP | 2022 |  |  |  |  | 0.228 | 0.120 | 0.284
Zhang et al. [30] | Appl. Sci. | 2022 | 0.491 | 0.358 | 0.278 | 0.225 | 0.389 |  | 0.218
ATAG [130] | IEEE TMI. | 2022 |  |  |  |  | 0.225 | 0.160 |
Nazarov et al. [77] | BMVC | 2022 | 0.299 | 0.182 | 0.124 | 0.090 | 0.238 | 0.136 | 0.123
MATNet [33] | IEEE SPL | 2022 | 0.506 | 0.370 | 0.288 | 0.233 | 0.395 |  | 0.221
Dalla Serra et al. [34] | IJCNLP | 2022 | 0.363 | 0.245 | 0.178 | 0.161 | 0.313 |  |
MSAT [31] | MICCAI | 2022 | 0.373 | 0.235 | 0.162 | 0.120 | 0.282 | 0.299 | 0.143
DeltaNet [96] | ICCL | 2022 | 0.361 | 0.225 | 0.154 | 0.114 | 0.277 | 0.281 |
Kim et al. [127] | BioNLP | 2023 | 0.342 | 0.222 | 0.152 | 0.110 | 0.301 | 0.166 |
ORGAN [94] | J. Biomed. Inform. | 2023 | 0.407 | 0.256 | 0.172 | 0.123 | 0.293 |  | 0.162
X-REM [23] | MIDL | 2023 |  | 0.22 |  |  |  |  |
MRCL [73] | JAIHC | 2023 | 0.340 | 0.212 | 0.145 | 0.103 | 0.270 | 0.109 | 0.137
KiUT [47] | CVPR | 2023 | 0.393 | 0.243 | 0.159 | 0.113 | 0.285 |  | 0.16
TIMER [86] | JMLR | 2023 | 0.383 | 0.225 | 0.146 | 0.104 | 0.28 |  | 0.147
CvT-212DistilGPT2 [128] | AIMI | 2023 | 0.393 | 0.248 | 0.171 | 0.127 | 0.286 | 0.389 | 0.155
Yang et al. [129] | Med Image Anal. | 2023 | 0.386 | 0.237 | 0.157 | 0.111 | 0.274 | 0.111 |
METransformer [49] | CVPR | 2023 | 0.386 | 0.250 | 0.169 | 0.124 | 0.291 | 0.362 | 0.362
RGRG [110] | CVPR | 2023 | 0.373 | 0.249 | 0.175 | 0.126 | 0.264 | 0.495 | 0.168
UAR [74] | JMLR | 2023 | 0.363 | 0.229 | 0.158 | 0.107 | 0.289 | 0.246 | 0.157
Dalla Serra et al. [36] | EMNLP | 2023 | 0.486 | 0.366 | 0.295 | 0.246 | 0.423 |  | 0.216
Zhu et al. [39] | MICCAI | 2023 | 0.343 | 0.210 | 0.140 | 0.099 | 0.271 |  | 0.137
Nguyen et al. [132] | JMLR | 2023 |  | 0.137 |  |  |  |  |
Table 6: MIMIC-CXR Dataset: Comparative Model Performance. All values were
extracted from the original papers. Best results are in bold and the second-
best underlined. Empty cells denote metrics not evaluated by the model.

Miura et al. [87] developed two factually oriented metrics called $fact_{ENT}$
and $fact_{ENTNLI}$ which evaluate the factual correctness and completeness of
the generated findings section of a report, for a more comprehensive
explanation of this work please see Section 4.5. While Miura et al. [87]
primarily used these metrics as rewards to train their model using
reinforcement learning, they have been applied by some during evaluation [87,
84].

###  7.2 Diversity Metrics

The process of writing radiology reports is inherently ambiguous, stemming
from the diverse levels of expertise, experience, and expressive styles among
radiologists [101]. To address this, certain ARRG researchers have endeavored
to quantify these differences by developing metrics that gauge the
distinctiveness of a generated report in comparison to others. Gajbhiye et al.
[89] introduced the metric Unique Index to demonstrate a model’s ability to
generate unique reports. This is calculated by taking the fraction of the
total number of a unique reports a model generates against the total number of
unique reports in the test set. For example, if a model created 100 unique
reports from a test set of 500 unique reports, then its score would be $0.2$.
Similarly, Najdenkoska et al. [101] used evaluation protocols from image
captioning for measuring diversity within the generated captions. The %Novel
metric they adopt measured the percentage of generated sentences that are not
contained within the model’s training set.

###  7.3 Qualitative Evaluation

Adequately evaluating the performance of medical texts can be challenging due
to the complexity of the human language [149]. To combat this, some
researchers have evaluated their models in a qualitative fashion. Christy et
al. [150, 151] performed evaluations via Amazon Mechanical Turk by asking
participants to choose amongst reports generated by different models that
match with the ground truth report in terms of language fluency, content
selection, and correctness of abnormal medical findings. More recent research
has utilised clinical experts to evaluate their models [28, 68, 118, 119, 120,
34, 116]. For example, Li et al. [20] engaged three radiologists to evaluate
the outputs of their methods, which is explained in Section 4, against CoAtt
and Vision-BERT.

Although using human clinical experts to evaluate medical machine learning
models is considered the most reliable method [149], it is time consuming and
costly [143]. Asking healthcare professionals to evaluate the generated
reports provides more work for an already overstretched workforce. Finding
methods to accurately evaluate not only the fluency of the generated reports,
but also the medical semantics are an important avenue of research.

##  8 Performance Comparison

In this section, we scrutinise the results of the state of the art in ARRG by
comparing their reported metrics, including BLEU, ROUGE, and METEOR. All
values referred to have been extracted from the original papers, and where the
information was missing, it was assumed the same training and test set from
the respective dataset (IU-Xray and MIMIC-CXR) were used for each.

For the IU-Xray dataset, Table 5 illustrates that the most effective model
according to several metrics is by Syeda-Mahmood et al. [103]. Their model
extracted not just core findings relating to pathologies, but also findings
relating to location, size and severity which they coined fine-finding labels.
These labels were used to retrieve the most similar report from their report
corpus, after which the report was edited to more closely align with the
generated labels. Regrettably, their code has not been made public, posing a
challenge for exact reproducibility and their evaluation does not contain an
ablation study which would have helped quantify the effectiveness of their
contribution. Their paper did provide a small case study which compared their
model’s generated reports against the ground truth and a select few other
models which highlighted their reports’ succinct nature, possibly due to their
report editing process. CheXPrune by Kaur & Mittal [91] also performed well,
achieving the second highest score on most NLP metrics. Their method adopted a
multi-attention mechanism and utilised parameter pruning to create a more
lightweight model. Their approach applied a hierarchical-LSTM for decoding
which has previously been accused of generating inflexible reports [9, 89].
Unfortunately, Kaur & Mittal [91] have not released their code for
reproducibility and their own evaluations did not provide any ablation studies
or other case studies, hindering an in-depth evaluation of their work. Despite
this, their performance implies that the LSTM architecture, coupled with
appropriate supporting modules, can generate reports on a par, and above, the
more recent architectures.

RepsNet [65] also performed strongly, leveraging publicly available models to
obtain the highest BLEU-1 score. They used ResNeXt [152] and BERT [107] as
their encoders and GPT-2 [108] as their decoder to generate their final
reports, training with contrastive loss. During their discussion, Tanwani et
al. [65] evaluated the effectiveness of their model through the use of
ablation and qualitative case studies. Their ablation study presented results
for a base model before incrementally adding two techniques, contrastive
learning and inserting prior context from retrieved reports. They observed
that their model performed best when using their complete methodology,
demonstrating that LLMs based methods such as RepsNet are capable of producing
state of the art results within the ARRG domain.

Table 6 demonstrates performances on the MIMIC-CXR dataset with the best
scoring model being produced by Dalla Serra et al. [36], whose contribution
leveraged images from previous scans and clinical indications to improve their
model’s performance. While there is also no code released for this work, they
performed an indepth performance analysis. Their ablation study investigated
the effectiveness of incorporating prior scans as input and adopting sentence-
anatomy (SA) dropout during training, where a random subset of anatomical
tokens was dropped from the input and omitted from the generated radiology
report. The findings from this investigation demonstrated a marked improvement
in performance from the baseline with the addition of either technique. There
was also a further improvement when their model implemented both techniques
which demonstrates the effectiveness of both their training and multimodal
input strategies. They also investigated the effectiveness of their model with
regard to initial and follow up scans by splitting the test set respectively.
This study highlighted that including priors improved their model’s
performance on not only follow up examinations, but also on imaging studies
with no prior scans. This improvement was reasoned to occur because their
model could infer which samples are initial scans from the zero filled vectors
that were used as a placeholder instead of an image. MATNet by Shang et al.
[33] also performed well, scoring second in five metrics. Similarily to Dalla
Serra et al. [36], their approach also tried to incorporate extra knowledge
into their model in the form of the clinical indications provided with the
MIMIC-CXR dataset as a secondary input to their encoder. Unfortunately, Shang
et al. [33] did not publically release their code which makes difficult to
replicate their results but their ablation study demonstrated the
effectiveness of the individual components of their model. They provided
results for a base model on the IU-xray dataset and then added components,
including their multi-modal Encoder, disease classifier, and adaptive decoder.
The performance of their model drastically improved with the addition of all
three components, with the best performance being achieved when all three
additional components are used.

Reflecting on the results, it is vital that the reporting of dataset usage
within research is transparent and standardised especially in terms of the
data split. While the MIMIC-CXR dataset provided its own, IU-Xray does not
contain an official split, and our recommendation for IU-Xray is to follow the
split provided by Chen et al [51] to facilitate accurate comparison of results
across different research. Furthermore, when examining Table 5 and Table 6, it
becomes evident that there is considerable variation in the NLP metrics
employed, often hindering accurate model comparisons. To ensure reliable
comparisons, it is advisable to adopt the NLP metrics BLEU, ROUGE-L, CIDEr,
and METEOR as a minimum benchmark.

Among the reviewed models, it is important to note that IU-Xray is a much
smaller dataset than MIMIC-CXR and as such, IU-Xray’s variance in results
could be significant [50]. As MIMIC is larger, and has an official split which
can be used to standardise results, it is considered to be a more accurate
baseline for the current state of the art within the field. On the MIMIC-CXR
dataset, the two highest performing models for the MIMIC dataset both utilised
multimodal inputs to improve their scores which had been validated by their
own ablation and case studies within their papers. Due to this, it is reasoned
that adding extra information, either through temporal or multimodal
approaches, can be considered the current state of the art approach within
ARRG.

##  9 Conclusion

ARRG poses an inherently intricate challenge for deep learning, addressing
complexities from both computer vision and natural language generation. In
this review, we have conducted a methodological assessment of deep learning
research methods addressing the ARRG task. We offer a description of the ARRG
task, followed by an overview of the datasets commonly employed in the current
literature. The majority of approaches utilised the two main datasets, IU-Xray
and MIMIC-CXR, with a minority exploring datasets that are not publicly
available. For the review, we analysed 70 ARRG contributions which were
published between 2020 to 2023, categorising their contributions into specific
topic areas, including training methods, architecture, utilising knowledge and
multiple modalities and evaluation methods.

Given the lack of a universally accepted standard for evaluation, we are
unable to advocate for an ideal model architecture, however from the papers
reviewed we can infer some recommendations. Methods have highlighted the
benefits of utilising auxiliary tasks, either in the pre-training phase or
throughout the training process. In terms of architecture, by observing Table
3 we can see that the use of the transformer is growing in popularity. This
trend is expected to continue, with foundational models such as LLaMA [18]
being adopted within the domain. Exploring ways to integrate knowledge into
ARRG models has emerged as a central focus for numerous studies within the
field.

Methods for evaluating ARRG systems is still an open research question.
Commonly used NLP metrics are not domain specific and cannot be utilised to
measure the diagnostic quality of generated radiology reports [117]. Clinical
efficacy metrics [51] are a step in the right direction, but these measures
depend on specific methods used to construct the rule-based classifier,
meaning the metric is specific to a single dataset [140]. The design of
metrics that can better handle semantics and assess clinical evaluations is
essential. Our evaluation section provides insights into the performance of
the designed models and showcases their results. We found that despite
encoder-decoder models being the more prevalent architecture, retrieval based
approaches perform at a similar level on NLP metrics. Another insight is that
the introduction of multimodal inputs have improved performance.

###  9.1 Future Directions

Finally, we highlight unresolved issues in the literature that create
opportunities for potential future research directions within the ARRG field.

Better Utilisation of Clinical Experts – If the ultimate objective of ARRG is
to create a radiology report generation system, it must adhere to the same
rigorous standards expected of radiologists and other reporting clinical
experts. As these experts are required to be qualified and experienced in
order to report, it makes sense to take advantage of their abilities within
the ARRG workflow where possible. A minority of approaches already
incorporated reviews by clinical experts into their studies [118, 28, 120,
119, 32, 34, 116] but providing an objectively fair comparison between the
approaches is challenging due to the lack of standardisation of these reviews.
Consequently, establishing a standardised approach to qualitative evaluation
would facilitate a more straightforward comparison of approaches that adopt
these assessments by clinical experts.

Reinforcement learning is becoming more prevalent in ARRG, however an area
that is currently unexplored is reinforcement learning with human feedback
[153] which has been successfully used to improve the performance of LLMs
[139]. Learning a reward function from the feedback data of clinical experts
is a promising avenue for ARRG as their feedback can promote considerations
such as safety and ethics which are big areas of concern within the medical
field, while also reducing overall model training costs.

Improving Quantitative Evaluation – While qualitative evaluation from a
clinical expert is beneficial, it is both costly and impractical for these
experts to assess all cases. Having an agreed set of quantitative metrics that
evaluate the semantics, quality and correctness of generated reports is
crucial. As seen in Table 4, the most commonly adopted assessment criteria are
NLP metrics which have been found to fall short for ARRG assessment [117].

Several clinical metrics have been proposed, but a universally accepted gold
standard metric has not yet been established. Consequently, the challenge of
developing metrics to assess medical accuracy and correctness remains an
ongoing research challenge. Metrics that have been used within machine
translation and summarisation could be used within the ARRG domain to compare
generated and reference reports. Metrics such as BERTScore [148], which use
embeddings to estimate semantic distance rather than relying on exact matches
between n-grams, have been found to be more effective than metrics such as
ROUGE, BLEU and METEOR.

Application of Large Language Models – While examining current research, it
was observed that there exists a gap in the literature concerning the
application of state-of-the-art large language models. These models have been
used and evaluated for similar tasks such as simplifying radiology reports or
generating impressions from findings [154, 155]. However, the adoption of LLM
decoder-only models within ARRG research has been restricted to models from
previous generations, such as GPT-2 [120] or the utilisation of the ChatGPT
web interface [133]. Foundational models such as LLaMA [18] and Falcon [19]
have been released in multiple versions, with LLaMA ranging from 7 to 65
billion parameters. These open-source models offer greater accessibility to
the research community and have not yet been incorporated into the ARRG task.
Given their demonstrated state-of-the-art performance in numerous NLP tasks,
it is expected that the incorporation of these models into research is
imminent.

Adoption of new datasets – The ARRG research community has focused their
efforts on two datasets: IU-Xray and MIMIC-CXR. Both datasets share the same
imaging modality, chest X-rays. Given the availability of these datasets, the
community’s emphasis has been generating reports specifically for chest
X-rays. As a resource, PadChest remains largely untapped, and incorporating it
into research could enhance the ability of future models to generalise.
Hamamci et al [26] have recently compiled a new public 3D dataset from an
alternative imaging modality, CT. This dataset offers significant value to the
ARRG community, affording them the opportunity to broaden the domain’s scope
and explore new avenues of research.

Extending model capabilities – The bulk of the examined research has been
focused on generating the findings of a radiology report, with only a limited
number looking into generating the impressions section. Extending models to
generate the impression section is an interesting area to explore and could be
approached from various perspectives. For instance, this could involve
concatenation of the findings and impression sections to be used as a target
in generation, or using NLP summarization techniques on the generated findings
to formulate an impression section.

## Data access statement

No new data was created during this review article.

## Acknowledgment

The 1st Author wishes to acknowledge and thank the financial support of the
UKRI (Grant ref EP/S022937/1) and the University of Bristol.

## References

  * [1] A. P. Brady, “Error and discrepancy in radiology: inevitable or avoidable?”, _Insights into Imaging_. Vol. 8, No. 1, pp. 171–182, 2017.
  * [2] T. Mabotuwana  _et al._ , “Detecting Technical Image Quality in Radiology Reports.”, _AMIA Annu. Symp. Proc._ , pp. 171–182, 2018.
  * [3] A. Rimmer, “Radiologist shortage leaves patient care at risk, warns royal college” _BMJ (Online)_ Vol.359, 2017.
  * [4] B. Petinaux  _et al._ , “Accuracy of radiographic readings in the emergency department.” _Am. J. Emerg. Med._ Vol.29, No.1, pp.18–-25, 2011.
  * [5] S. K. Zhou  _et al._ , “A Review of Deep Learning in Medical Imaging: Imaging Traits, Technology Trends, Case Studies With Progress Highlights, and Future Promises,”  _Proc. IEEE Inst Electr. Electron. Eng._ , vol. 109, no. 5, pp.  _820–838_ , 2021.
  * [6] L. Cai  _et al._ , “A review of the application of deep learning in medical image classification and segmentation.” _Ann. Transl. Med._ Vol.8, No.11, 2020.
  * [7] Y. Zhang  _et al._ , “Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports.”  _Proc. Assoc. Comp. Ling._ , pp. 5108–5120, 2020.
  * [8] S. Y. Chng  _et al._ , “Automated labelling of radiology reports using natural language processing: comparison of traditional and newer methods.” _Health Care Sci._ Vol.2, pp. 120-–128, 2023.
  * [9] Z. Wang  _et al._ , “Automated Radiographic Report Generation Purely on Transformer: A Multicriteria Supervised Approach,” _IEEE Trans. Med Imaging._ , Vol. 41, No. 10, pp. 2803–2813, 2022.
  * [10] Royal College of Radiologists, “The Standards for interpretation and reporting of imaging investigations, 2nd Ed”, 2018. [Online] Available from: https://www.rcr.ac.uk/publication/standards-interpretation-and-reporting-imaging-investigations-second-edition
  * [11] S. Yang  _et al._ , “Knowledge matters: Chest radiology report generation with general and specific knowledge” _Med. Image Anal._ , Vol. 80, No 102510. 2022.
  * [12] M. M. A. Monshi  _et al._ , “Deep learning in generating radiology reports: A survey.” _Art. Int. Med._ Vol. 106, 2020.
  * [13] Y. Liao  _et al._ , “Deep learning approaches to automatic radiology report generation: A systematic review.” _Inform. Med. Unlocked._ Vol. 39 2023.
  * [14] N. Kaur  _et al._ , “Methods for automatic generation of radiological reports of chest radiographs: a comprehensive survey.” _Multimed. Tools. Appl.,_ Vol. 81, No. 10, pp. 13409-–13439, 2022.
  * [15] P. Messina  _et al._ “A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images.” _ACM Comput. Surv._ , Vol. 54, No. 10, 2022
  * [16] I. Allaouzi  _et al._ , “Automatic Caption Generation for Medical Images.” _Proc. Int. Smart Cities Conf._ , Article 86, 1–-6. 2018.
  * [17] J. Pavlopoulos  _et al._ , “A Survey on Biomedical Image Captioning.” _Proc. of the Second Workshop on Shortcomings in Vis. Lang._ , pp. 26–36. 2019
  * [18] H. Touvron  _et al._ “LLaMA: Open and Efficient Foundation Language Models”  _arXiv preprint_. 2023.
  * [19] G. Penedo  _et al._ “The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only”  _arXiv preprint_. 2023.
  * [20] M. Li  _et al._ , “Auxiliary signal‐guided knowledge encoder‐decoder for medical report generation”  _World Wide Web_ vol. 26, pp.  _253–270_ , 2023.
  * [21] H. Zhao  _et al._ , “Automatic Generation of Medical Report with Knowledge Graph”  _Proc. IEEE Comput. Soc. Int. Comput. Vis. Pattern Recognit._ , pp.  _217–223_ , 2022.
  * [22] X. Hou  _et al._ , “MKCL: Medical Knowledge with Contrastive Learning model for radiology report generation”  _J. of Biomed. Inform._ , No. 104496, 2023.
  * [23] J. Jeong  _et al._ , “Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generation”,  _Med. Imaging with Deep Learn._ , 2023. pp.  _1–-13_
  * [24] X. Wu  _et al._ , “Multimodal contrastive learning for radiology report generation,”  _J. Ambient. Intell. Humaniz. Comput._ , vol. 14, pp.  _11185–11194_ , 2023.
  * [25] S. Ye  _et al._ “Semantic-Driven Global-Local Cooperative Contrastive Learning for Medical Report Generation”  _Int. Conf. Digit. Image Comput. Tech. Appl._ , pp.  _251–257_ , 2023.
  * [26] I.E. Hamamci  _et al._ , “A foundation model utilizing chest CT volumes and radiology reports for supervised-level zero-shot detection of abnormalities”  _arXiv preprint_ , 2024.
  * [27] F. Liu  _et al._ , “Competence-based Multimodal Curriculum Learning for Medical Report Generation.” _Proc. Int Jt. Conf. on Nat. Lang. Process._ , pp. 3001–3012, 2021.
  * [28] A. Yan  _et al._ , “Weakly Supervised Contrastive Learning for Chest X-Ray Report Generation”  _Find. In Proc. Conf. Empir. Methods Nat. Lang. Process._ pp.  _4009–4015_. 2021.
  * [29] S. Jain  _et al._ , “RadGraph: Extracting Clinical Entities and Relations from Radiology Reports (v1.0.0).” _PhysioNet._ , 2021.
  * [30] D. Zhang  _et al._ , “Improving Medical X-ray Report Generation by Using Knowledge Graph,”  _Applied Sciences_ , vol. 12, no. 21, 2022.
  * [31] Z. Wang  _et al._ , “A Medical Semantic-Assisted Transformer for Radiographic Report Generation.”,  _Med. Image Comput. Assist Interv.,_ Vol 13433, pp.  _655–664_ , 2022.
  * [32] H. Nguyen  _et al._ , “Automated Generation of Accurate & Fluent Medical X-ray Reports”,  _Proc. Conf. Empir. Methods Nat. Lang. Process._ , pp.  _3552–3569_ , 2021.
  * [33] C. Shang  _et al._ , “MATNet: Exploiting Multi-Modal Features for Radiology Report Generation,”  _IEEE Signal Process. Lett._ , vol. 29, pp.  _2692–2696_ , 2022.
  * [34] F. Dalla Serra  _et al._ , “Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations,”  _Proc. Conf. Int. Joint Conf. Nat. Lang. Process._ , pp.  _615–624_ , 2022.
  * [35] C. Mondal  _et al._ “Transformers Are All You Need to Generate Automatic Report from Chest X-ray Images.,”  _Int. Conf. Digit. Image Comput. Tech. Appl._ , pp.  _387–394_ , 2023.
  * [36] F. Dalla Serra  _et al._ “Controllable Chest X-Ray Report Generation from Longitudinal Representations”,  _Find. In Proc. Conf. Empir. Methods Nat. Lang. Process._ , pp  _4891–4904_ , 2023.
  * [37] S. Bannur  _et al._ “Learning To Exploit Temporal Structure for Biomedical Vision-Language Processing”  _Proc. IEEE Conf. Comput. Vis. Pat. Rec._ , pp.  _15016–15027_ , 2023.
  * [38] T. Nishino  _et al.,_ “Factual Accuracy is not Enough: Planning Consistent Description Order for Radiology Report Generation”  _In Proc. Conf. Empir. Methods Nat. Lang. Process._ pp.  _7123–7138_ , 2022.
  * [39] Q. Zhu  _et al._ “Utilizing Longitudinal Chest X-Rays and Reports to Pre-fill Radiology Reports. ”  _Med. Image Comput. Assist Interv.,_ pp.  _189–198_ , 2023.
  * [40] European Society of Radiology (ESR), “Good practice for radiological reporting. Guidelines from the European Society of Radiology (ESR)”. _Insights in Imaging_ Vol. 2, pp. 93–-96 2011.
  * [41] D. Demner-Fushman  _et al._ , “Preparing a collection of radiology examinations for distribution and retrieval”. _J. Am. Med. Inform. Assoc._ , Vol. 23, No 2, pp. 304–310. 2016.
  * [42] H. Shin  _et al._ , “Learning to read chest X-rays: recurrent neural cascade model for automated image annotation.” _Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit._ pp. 2497–2506, 2016.
  * [43] Y. Xue  _et al._ , “Multimodal Recurrent Model with Attention for Automated Radiology Report Generation.” _Med. Image Comput. Comput. Assist Interv._ , vol 11070, pp. 457–-466, 2018.
  * [44] B. Jing  _et al._ , “On the Automatic Generation of Medical Imaging Reports.” _Proc. Assoc Comp. Lingu.,_ , pp. 2577–-2586, 2018.
  * [45] A. Vaswani  _et al._ “Attention is All You Need,” _Proc. Neural Inf. Process_ , Vol. 30, pp. 6000–6010, 2017.
  * [46] A. Fernandez-Quilez, “Deep learning in radiology: ethics of data and on the value of algorithm transparency, interpretability and explainability.” _AI Ethics_ Vol. 3, pp.257–265, 2023.
  * [47] Z. Huang  _et al._ , “KiUT: Knowledge-Injected U-Transformer for Radiology Report Generation”  _Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit._ pp. 19809–19818, 2023.
  * [48] F. Liu  _et al._ , “Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation,”  _Proc. IEEE Conf. Comput. Vis. Pat. Rec._ , pp.  _13748-13757_ , 2021.
  * [49] Z. Wang  _et al._ , “METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens,”  _Proc. IEEE Conf. Comput. Vis. Pat. Rec._ , pp.  _11558-11567_ , 2023.
  * [50] Z. Chen  _et al._ , “Cross-modal Memory Networks for Radiology Report Generation”  _Proc. Int. Joint Conf. Nat. Lang. Process._ , pp.  _5904–5914_ 2021\.
  * [51] Z. Chen  _et al._ , “Generating Radiology Reports via Memory-driven Transformer”  _In Proc. Conf. Empir. Methods Nat. Lang. Process._ , pp.  _1439–1449_ , 2020.
  * [52] A. E. W. Johnson  _et al._ , “MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.” _Sci. Data._ , Vol. 6, No. 317, 2019.
  * [53] J. Ni  _et al._ , “Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rays”  _Proc. Conf. Emp. Methods Nat. Lang. Process._ , pp.  _1954–1960_ , 2020.
  * [54] J. Wu  _et al._ “Chest ImaGenome Dataset (version 1.0.0)”  _PhysioNet_ , 2021.
  * [55] Ramesh, V.  _et al._ , “Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors”,  _Proc. Mach. Learn. Res._ , Vol. 193, pp. /emph456–-473, 2022.
  * [56] B. Boecking  _et al._ “Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing.”,  _Comp. Vis. – ECCV_ vol. 13696, pp.  _1–21_ , 2022.
  * [57] A. Bustos  _et al._ , “PadChest: A large chest x-ray image dataset with multi-label annotated reports.” _Med. Image Anal._ Vol. 66, No 101797, 2020.
  * [58] F. Wang  _et al._ , “Unifying Relational Sentence Generation and Retrieval for Medical Image Report Composition,”  _IEEE Trans. Cybern.,_ vol. 52, no. 6, pp.  _5015-5025_ , 2022.
  * [59] X. Ying “An Overview of Overfitting and its Solutions”  _J. Phys. Conf. Ser._ , vol. 1168, 2019.
  * [60] C. F. Liu  _et al._ “A large public dataset of annotated clinical MRIs and metadata of patients with acute stroke.”  _Sci. Data_ vol. 10, No. 548, 2023.
  * [61] C. Liu  _et al._ , “Automatic comprehensive radiological reports for clinical acute stroke MRIs.”  _Commun. Med._ , Vol 3. No. 95, 2023.
  * [62] X. Wang  _et al._ , “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.” _IEEE Conf. Comp. Vis. Pat. Rec._ , pp 2097–-2106. 2017.
  * [63] J. A. Irvin  _et al._ , “CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.” _Proc. AAAI Conf. Artif. Intell._ , 2019
  * [64] T. Chen  _et al._ , “A Simple Framework for Contrastive Learning of Visual Representations.”  _Proc. Int. Conf. on Mach. Learn._ Ed.Vol. 119. pp.  _1597–1607._ 2020\.
  * [65] A. K. Tanwani  _et al._ , “RepsNet: Combining Vision with Language for Automated Medical Reports.”  _Med. Image Comput. Comput. Assist Interv._ , vol. 13435, 2022.
  * [66] G. Leonardi  _et al._ , “A multimodal approach to automated generation of radiology reports using contrastive learning”  _Proc AIxIA Works. Art. Intel. Health._ , pp.  _16-23_ , 2022.
  * [67] M. Endo  _et al._ , “Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model”,  _Proc. Mach. Learn. Res._ vol. 158, pp  _209–219_ , 2021.