# A Systematic Review of Natural Language Processing Applied to Radiology
Reports

Arlene Casey  School of Literatures, Languages and Cultures (LLC), University
of Edinburgh  Corresponding author:arlene.casey AT ed.ac.uk  Emma Davidson
Centre for Clinical Brain Sciences, University of Edinburgh  Michael Poon
Centre for Clinical Brain Sciences, University of Edinburgh  Hang Dong  Centre
for Medical Informatics, Usher Institute of Population Health Sciences and
Informatics, University of Edinburgh  Health Data Research, UK  Daniel Duma
School of Literatures, Languages and Cultures (LLC), University of Edinburgh
Andreas Grivas  Institute for Language, Cognition and Computation, School of
Informatics, University of Edinburgh  Claire Grover  Institute for Language,
Cognition and Computation, School of Informatics, University of Edinburgh
Víctor Suárez-Paniagua  Centre for Medical Informatics, Usher Institute of
Population Health Sciences and Informatics, University of Edinburgh  Health
Data Research, UK  Richard Tobin  Institute for Language, Cognition and
Computation, School of Informatics, University of Edinburgh  William Whiteley
Centre for Clinical Brain Sciences, University of Edinburgh  Nuffield
Department of Population Health, University of Oxford  Honghan Wu  Health Data
Research, UK  Institute of Health Informatics, University College of London
Beatrice Alex  School of Literatures, Languages and Cultures (LLC), University
of Edinburgh  Edinburgh Futures Institute, University of Edinburgh

###### Abstract

Background Natural language processing (NLP) has a significant role in
advancing healthcare and has been found to be key in extracting structured
information from radiology reports. Understanding recent developments in NLP
application to radiology is of significance but recent reviews on this are
limited. This study systematically assesses and quantifies recent literature
in NLP applied to radiology reports.

Methods We conduct an automated literature search yielding 4,799 results using
automated filtering, metadata enriching steps and citation search combined
with manual review. Our analysis is based on 21 variables including radiology
characteristics, NLP methodology, performance, study, and clinical application
characteristics.

Results We present a comprehensive analysis of the 164 publications retrieved
with publications in 2019 almost triple those in 2015. Each publication is
categorised into one of 6 clinical application categories. Deep learning use
increases in the period but conventional machine learning approaches are still
prevalent. Deep learning remains challenged when data is scarce and there is
little evidence of adoption into clinical practice. Despite 17% of studies
reporting greater than 0.85 F1 scores, it is hard to comparatively evaluate
these approaches given that most of them use different datasets. Only 14
studies made their data and 15 their code available with 10 externally
validating results.

Conclusions Automated understanding of clinical narratives of the radiology
reports has the potential to enhance the healthcare process and we show that
research in this field continues to grow. Reproducibility and explainability
of models are important if the domain is to move applications into clinical
use. More could be done to share code enabling validation of methods on
different institutional data and to reduce heterogeneity in reporting of study
properties allowing inter-study comparisons. Our results have significance for
researchers in the field providing a systematic synthesis of existing work to
build on, identify gaps, opportunities for collaboration and avoid
duplication.

## Background

Medical imaging examinations interpreted by radiologists in the form of
narrative reports are used to support and confirm diagnosis in clinical
practice. Being able to accurately and quickly identify the information stored
in radiologists’ narratives has the potential to reduce workloads, support
clinicians in their decision processes, triage patients to get urgent care or
identify patients for research purposes. However, whilst these reports are
generally considered more restricted in vocabulary than other electronic
health records (EHR), e.g. clinical notes, it is still difficult to access
this efficiently at scale (Bates et al., 2016). This is due to the
unstructured nature of these reports and Natural Language Processing (NLP) is
key to obtaining structured information from radiology reports (Pons et al.,
2016).

NLP applied to radiology reports is shown to be a growing field in earlier
reviews (Pons et al., 2016; Cai et al., 2016). In recent years there has been
an even more extensive growth in NLP research in general and in particular
deep learning methods which is not seen in the earlier reviews. A more recent
review of NLP applied to radiology-related research can be found but this
focuses on one NLP technique only, deep learning models (Sorin et al., 2020).
Our paper provides a more comprehensive review comparing and contrasting all
NLP methodologies as they are applied to radiology.

It is of significance to understand and synthesise recent developments
specific to NLP in the radiology research field as this will assist
researchers to gain a broader understanding of the field, provide insight into
methods and techniques supporting and promoting new developments in the field.
Therefore, we carry out a systematic review of research output on NLP
applications in radiology from 2015 onward, thus, allowing for a more up to
date analysis of the area. An additional listing of our synthesis of
publications detailing their clinical and technical categories along with
anatomical scan regions can be made available on request. Also different to
the existing work, we look at both the clinical application areas NLP is being
applied in and consider the trends in NLP methods. We describe and discuss
study properties, e.g. data size, performance, annotation details, quantifying
these in relation to both the clinical application areas and NLP methods.
Having a more detailed understanding of these properties allows us to make
recommendations for future NLP research applied to radiology datasets,
supporting improvements and progress in this domain.

## Related Work

Amongst pre-existing reviews in this area, (Pons et al., 2016) was the first
that was both specific to NLP on radiology reports and systematic in
methodology. Their literature search identified 67 studies published in the
period up to October 2014. They examined the NLP methods used, summarised
their performance and extracted the studies’ clinical applications, which they
assigned to five broad categories delineating their purpose. Since Pons et
al.’s paper, several reviews have emerged with the broader remit of NLP
applied to electronic health data, which includes radiology reports.
(Kreimeyer et al., 2017) conducted a systematic review of NLP systems with a
specific focus on coding free text into clinical terminologies and structured
data capture. The systematic review by (Spasic and Nenadic, 2020) specifically
examined machine learning approaches to NLP (2015-2019) in more general
clinical text data, and a further methodical review was carried out by (Wu et
al., 2020) to synthesise literature on deep learning in clinical NLP (up to
April 2019) although the did not follow the PRISMA guideline completely. With
radiology reports as their particular focus, (Cai et al., 2016) published, the
same year as Pons et al.’s review, an instructive narrative review outlining
the fundamentals of NLP techniques applied in radiology. More recently, (Sorin
et al., 2020) published a systematic review focused on deep learning
radiology-related research. They identified 10 relevant papers in their search
(up to September 2019) and examined their deep learning models, comparing
these with traditional NLP models and also considered their clinical
applications but did not employ a specific categorisation. We build on this
corpus of related work, and most specifically Pons et al.’s work. In our
initial synthesis of clinical applications we adopt their application
categories and further expand upon these to reflect the nature of subsequent
literature captured in our work. Additionally, we quantify and compare
properties of the studies reviewed and provide a series of recommendations for
future NLP research applied to radiology datasets in order to promote
improvements and progress in this domain.

## Methods

Our methodology followed the Preferred Reporting Items for Systematic Reviews
and Meta-Analysis (PRISMA) (Moher et al., 2015), and the protocol is
registered on protocols.io.

### Eligibility for Literature Inclusion and Search Strategy

We included studies using NLP on radiology reports of any imaging modality and
anatomical region for NLP technical development, clinical support, or
epidemiological research. Exclusion criteria included: (1) case reports; (2)
published before 2015; (3) in language other than English; (4) processing of
radiology images; (5) reviews, conference abstracts, comments, patents, or
editorials; (6) not reporting outcomes of interest; (7) not radiology reports;
(8) not using NLP methods; (9) not available in full text; (10) duplicates.

We used Publish or Perish (Harzing A. W., 2007), a citation retrieval and
analysis software program, to search Google Scholar. Google Scholar has a
similar coverage to other databases (Gehanno et al., 2013) and is easier to
integrate into search pipelines. We conducted an initial pilot search
following the process described here, but the search terms were too specific
and restricted the number of publications. However, we did include papers
found in the pilot search in full-text screening. We use the following search
query restricted to research articles published in English between January
2015 and October 2019. (”radiology” OR ”radiologist”) AND (”natural language”
OR ”text mining” OR ”information extraction” OR ”document classification” OR
”word2vec”) NOT patent. We automated the addition of publication metadata and
applied filtering to remove irrelevant publications. These automated steps are
described in Table 1 & Table 2.

Table 1: Metadata enriching steps undertaken for each publication Metadata
Enriching Steps
---
1\. Match the paper with its DOI via the Crossref API
2\. If DOI matched, check Semantic Scholar for metadata/abstract
3\. If no DOI match and no abstract, search PubMed for abstract
4\. Search arXiv(for a pre-print)
5\. If no PDF link, search Unpaywall for available open access versions
6\. If PDF but no separate abstract via Semantics Scholar/PubMed, extract
abstract from the PDF
Table 2: Automated filtering steps to remove irrelevant publications Automated
Filtering Steps
---
1\. Document language is English
2\. Word ’patent’ in title or URL
3\. Year of publication out of range (<2015)
4\. The words ’review’ or ’overview’ in the title, ’this review’ in the
abstract
5\. Image keywords in title or abstract
6\. No radiology keywords in title or abstract
7\. No NLP terminology in abstract

In addition to query search, another method to find papers is to conduct a
citation search (Briscoe et al., 2020). The citation search compiled a list of
publications that cite the Pons et al. review and the articles cited in the
Pons’ review. To do this, we use a snowballing method (Wohlin, 2014) to follow
the forward citation branch for each publication in this list, i.e. finding
every article that cites the publications in our list. The branching factor
here is large, so we filter at every stage and automatically add metadata.

### Manual Review of Literature

Four reviewers (three NLP researchers [AG,DD and HD] and one epidemiologist
[MTCP]) independently screened all titles and abstracts with the Rayyan online
platform and discussed disagreements. Fleiss’ kappa (Fleiss, 1971) agreement
between reviewers was 0.70, indicating substantial agreement (Landis and Koch,
1977). After this screening process, each full-text article was reviewed by a
team of eight (six NLP researchers and two epidemiologists) and double
reviewed by a NLP researcher. We resolved any discrepancies by discussion in
regular meetings.

### Data Extraction for Analysis

We extracted data on: primary clinical application and technical objective,
data source(s), study period, radiology report language, anatomical region,
imaging modality, disease area, dataset size, annotated set size,
training/validation/test set size, external validation performed, domain
expert used, number of annotators, inter-annotator agreement, NLP technique(s)
used, best-reported results (recall, precision and F1 score), availability of
dataset, and availability of code.

## Results

The literature search yielded 4,799 possibly relevant publications from which
our automated exclusion process removed 4,402, and during both our screening
processes, 233 were removed, leaving 164 publications. See Figure 1 for
details of exclusions at each step.

Figure 1: PRISMA diagram for search publication retrieval

### General Characteristics

2015 and 2016 saw similar numbers of publications retrieved (22 and 21
respectively) with the volume increasing almost three-fold in 2019 (55),
noting 2019 only covers 10 months (Figure 2). Imaging modality (Table 4)
varied considerably and 38 studies used reports from multiple modalities. Of
studies focusing on a single modality, the most featured were CT scans (36)
followed by MRI (16), X-Ray (18), Mammogram (5) and Ultrasound (4). Forty-
seven studies did not specifying scan modality. For the study samples (Table
4), 33 papers specified that they used consecutive patient images, 38 used
non-consecutive image sampling and 93 did not clearly specify their sampling
strategy. The anatomical regions for scans varied (Table 6) with mixed being
the highest followed by Thorax and Head/neck. Disease categories are presented
in Table 6 with the largest disease category being Oncology. The majority of
reports were in English (142) and a small number in other languages e.g.,
Chinese (5), Spanish (4), German (3) (Table 7).

Table 3: Scan modality  Scan Modality | No. Studies
---|---
Multiple Modalities | 38
MRI | 16
CT | 36
X-Ray | 18
Mammogram | 5
Ultrasound | 4
Not specified | 47
TOTAL | 164

Table 4: Image sampling method  Sampling Method No. Studies |
---|---
Consecutive Images | 33
Non-Consecutive Images | 38
Not specified | 93
TOTAL | 164

Table 5: Anatomical region scanned  Anatomical Region | No. Studies
---|---
Mixed | 45
Thorax | 31
Head/Neck | 25
Abdomen | 15
Breast | 15
Extremities | 8
Spine | 5
Other | 1
Unspecified | 19
TOTAL | 164

Table 6: Disease category  Disease Category | No. Studies
---|---
Not specific disease related | 40
Oncology | 39
Various | 20
Musculoskeletal | 10
Cerebrovascular | 13
Other | 13
Respiratory | 10
Trauma | 7
Cardiovascular | 6
Gastrointestinal | 3
Hepatobiliary | 2
Genitourinary | 1
TOTAL | 164

Table 7: Radiology report language Report Language | No. Studies
---|---
English | 142
Chinese | 5
Spanish | 4
German | 3
Italian | 2
French | 2
Hebrew | 1
Polish | 1
Brazilian Portuguese | 1
Unspecified | 3
TOTAL | 164
Table 8: Clinical Application Category by Technical Objective Application Category |  Information Extraction (n=81) |  Report/ Sentence Classification (n=73) |  Lexicon/ Ontology Discovery (n=9) |  Clustering (n=1)
---|---|---|---|---
Disease Information & Classification | 14 | 31 - | - |
Diagnostic Surveillance | 28 | 17 | - | -
Quality Compliance | 7 | 14 | - | -
Cohort-Epid. | 6 | 10 | - | -
Language Discovery & Knowledge | 13 | 4 | 9 | 1
Technical NLP | 6 | 4 | - | -

### Clinical Application Categories

In synthesis of the literature each publication was classified by the primary
clinical purpose. Pons’ work in 2016 categorised publications into 5 broad
categories: Diagnostic Surveillance, Cohort Building for Epidemiological
Studies, Query-based Case Retrieval, Quality Assessment of Radiological
Practice and Clinical Support Services. We found some changes in this
categorisation schema and our categorisation consisted of six categories:
Diagnostic Surveillance, Disease information and classification, Quality
Compliance, Cohort/Epidemiology, Language Discovery and Knowledge Structure,
Technical NLP. The main difference is we found no evidence for a category of
Clinical Support Services which described applications that had been
integrated into the workflow to assist. Despite the increase in the number of
publications, very few were in clinical use with more focus on the category of
Disease Information and Classification. We describe each clinical application
area in more detail below and where applicable how our categories differ from
the earlier findings. A listing of all publications and their corresponding
clinical application category can be made available on request. Table 8 shows
the clinical application category by the technical classification and Figure 2
shows the breakdown of clinical application category by publication year.
There were more publications in 2019 compared with 2015 for all categories
except Language Discovery & Knowledge Structure, which fell by $\approx$ 25%
(Figure 2).

Figure 2: Clinical application of publication by year

#### Diagnostic Surveillance

A large proportion of studies in this category focused on extracting disease
information for patient or disease surveillance e.g. investigating tumour
characteristics (Peng et al., 2019; Bozkurt et al., 2019); changes over time
(Hassanpour et al., 2017) and worsening/progression or improvement/response to
treatment (Kehl et al., 2019; Chen et al., 2018); identifying correct
anatomical labels (Cotik et al., 2018); organ measurements and temporality
(Sevenster et al., 2015a). Studies also investigated pairing measurements
between reports (Sevenster et al., 2015b) and linking reports to monitoring
changes through providing an integrated view of consecutive examinations
(Oberkampf et al., 2016). Studies focused specifically on breast imaging
findings investigating aspects, such as BI-RADS MRI descriptors (shape, size,
margin) and final assessment categories (benign, malignant etc.) e.g., (Liu et
al., 2019; Gupta et al., 2018; Castro et al., 2017; Short et al., 2019; Lacson
et al., 2017, 2015). Studies focused on tumour information e.g., for liver
(Yim et al., 2016b) and hepatocellular carcinoma (HPC) (Yim et al., 2017,
2016a) and one study on extracting information relevant for structuring
subdural haematoma characteristics in reports (Pruitt et al., 2019).

Studies in this category also investigated incidental findings including on
lung imaging (Farjah et al., 2016; Karunakaran et al., 2017; Tan et al.,
2018), with (Farjah et al., 2016) additionally extracting the nodule size; for
trauma patients (Trivedi et al., 2019); and looking for silent brain
infarction and white matter disease (Fu et al., 2019). Other studies focused
on prioritising/triaging reports, detecting follow-up recommendations, and
linking a follow-up exam to the initial recommendation report, or bio-
surveillance of infectious conditions, such as invasive mould disease.

#### Disease Information and Classification

Disease Information and Classification publications use reports to identify
information that may be aggregated according to classification systems. These
publications focused solely on classifying a disease occurrence or extracting
information about a disease with no focus on the overall clinical application.
This category was not found in Pons’ work. Methods considered a range of
conditions including intracranial haemorrhage (Jnawali et al., 2019; Banerjee
et al., 2017), aneurysms (Kłos et al., 2018), brain metastases (Deshmukh et
al., 2019), ischaemic stroke (Kim et al., 2019; Garg et al., 2019), and
several classified on types and severity of conditions e.g., (Deshmukh et al.,
2019; Shin et al., 2017; Wheater et al., 2019; Gorinski et al., 2019; Alex et
al., 2019). Studies focused on breast imaging considered aspects such as
predicting lesion malignancy from BI-RADS descriptors (Bozkurt et al., 2016),
breast cancer subtypes (Patel et al., 2017), and extracting or inferring BI-
RADS categories, such as (Banerjee et al., 2019a; Miao et al., 2018). Two
studies focused on abdominal images and hepatocellular carcinoma (HCC) staging
and CLIP scoring. Chest imaging reports were used to detect pulmonary embolism
e.g., (Dunne et al., 2015; Banerjee et al., 2019b; Chen et al., 2017),
bacterial pneumonia (Meystre et al., 2017), and Lungs-RADS categories (Beyer
et al., 2017). Functional imaging was also included, such as echocardiograms,
extracting measurements to evaluate heart failure, including left ventricular
ejection fractions (LVEF). Other studies investigated classification of
fractures and abnormalities and the prediction of ICD codes from imaging
reports.

#### Language Discovery and Knowledge Structure

Language Discovery and Knowledge Structure publications investigate the
structure of language in reports and how this might be optimised to facilitate
decision support and communication. Pons et al. reported on applications of
Query-based retrieval which has similarities to Language Discovery and
Knowledge Structure but it is not the same. Their category contains studies
that retrieve cases and conditions that are not predefined and in some
instances could be used for research purposes or are motivated for educational
purposes. Our category is broader and encompasses papers that investigated
different aspects of language including variability, complexity simplification
and normalising to support extraction and classification tasks.

Studies focus on exploring lexicon coverage and methods to support language
simplification for patients looking at sources, such as the consumer health
vocabulary (Qenam et al., 2017) and the French lexical network (JDM)
(Lafourcade and Ramadier, 2017). Other works studied the variability and
complexity of report language comparing free-text and structured reports and
radiologists. Also investigated was how ontologies and lexicons could be
combined with other NLP methods to represent knowledge that can support
clinicians. This work included improving report reading efficiency (Hong and
Zhang, 2015); finding similar reports (Comelli et al., 2015); normalising
phrases to support classification and extraction tasks, such as entity
recognition in Spanish reports (Cotik et al., 2015); imputing semantic classes
for labelling (Johnson et al., 2015), supporting search (Mujjiga et al., 2019)
or to discover semantic relations (Lafourcade and Ramadier, 2016).

#### Quality and Compliance

Quality and Compliance publications use reports to assess the quality and
safety of practice and reports similar to Pons’ category. Works considered how
patient indications for scans adhered to guidance e.g., (Shelmerdine et al.,
2019; Mabotuwana et al., 2018b; Dalal et al., 2020; Bobbin et al., 2017; Kwan
et al., 2019; Mabotuwana et al., 2018a) or protocol selection (Brown and
Marotta, 2017; Trivedi et al., 2018; Zhang et al., 2018; Brown and Marotta,
2018; Yan et al., 2016) or the impact of guideline changes on practice, such
as (Kang et al., 2019). Also investigated was diagnostic utilisation and
yield, based on clinicians or on patients, which can be useful for hospital
planning and for clinicians to study their work patterns e.g.(Brown and
Kachura, 2019). Other studies in this category looked at specific aspects of
quality, such as, classification for long bone fractures to support quality
improvement in paediatric medicine (Grundmeier et al., 2016), automatic
identification of reports that have critical findings for auditing purposes
(Heilbrun et al., 2019), deriving a query-based quality measure to compare
structured and free-text report variability (Maros et al., 2018), and (Minn et
al., 2015) who describe a method to fix errors in gender or laterality in a
report.

#### Cohort and Epidemiology

This category is similar to Pons’ earlier review but we treated the studies in
this category slightly attempting to differentiate which papers described
methods for creating cohorts for research purposes, and those which also
reported the outcomes of an epidemiological analysis. Ten studies use NLP to
create specific cohorts for research purposes and six reported the performance
of their tools. Out of these papers, the majority (n=8) created cohorts for
specific medical conditions including fatty liver disease (Goldshtein et al.,
2020; Redman et al., 2017) hepatocellular cancer (Sada et al., 2016), ureteric
stones (Li and Elliot, 2019), vertebral facture (Tan and Heagerty, 2019),
traumatic brain injury (Yadav et al., 2016; Mahan et al., 2019), and
leptomeningeal disease secondary to metastatic breast cancer (Brizzi et al.,
2019). Five papers identified cohorts focused on particular radiology findings
including ground glass opacities (GGO) (Van Haren et al., 2019), cerebral
microbleeds (CMB) (Noorbakhsh-Sabet et al., 2018), pulmonary nodules (Gould et
al., 2015), (Huhdanpaa et al., 2018), changes in the spine correlated to back
pain (Bates et al., 2016) and identifying radiological evidence of people
having suffered a fall. One paper focused on identifying abnormalities of
specific anatomical regions of the ear within an audiology imaging database
(Masino et al., 2016) and another paper aimed to create a cohort of people
with any rare disease (within existing ontologies - Orphanet Rare Disease
Ontology ORDO and Radiology Gamuts Ontology RGO). Lastly, one paper took a
different approach of screening reports to create a cohort of people with
contraindications for MRI, seeking to prevent iatrogenic events. Amongst the
epidemiology studies there were various analytical aims, but they primarily
focused on estimating the prevalence or incidence of conditions or imaging
findings and looking for associations of these conditions/findings with
specific population demographics, associated factors or comorbidities. The
focus of one study differed in that it applied NLP to healthcare evaluation,
investigating the association of palliative care consultations and measures of
high-quality end-of-life (EOL) care (Brizzi et al., 2019).

#### Technical NLP

This category is for publications that have a primary technical aim that is
not focused on radiology report outcome, e.g. detecting negation in reports,
spelling correction (Zech et al., 2019), fact checking (Zhang et al., 2019;
Steinkamp et al., 2019) methods for sample selection, crowd source annotation
(Cocos et al., 2017). This category did not occur in Pons’ earlier review.

### NLP Methods in Use

NLP methods capture the different techniques an author applied broken down
into rules, machine learning methods, deep learning, ontologies, lexicons and
word embeddings. We discriminate machine learning from deep learning, using
the former to represent traditional machine learning methods.

Over half of the studies only applied one type of NLP method and just over a
quarter of the studies compared or combined methods in hybrid approaches. The
remaining studies either used a bespoke proprietary system or focus on
building ontologies or similarity measures (Figure 3). Rule-based method use
remains almost constant across the period, whereas use of machine learning
decreases and deep learning methods rises, from five publications in 2017 to
twenty-four publications in 2019 (Figure 4).

Figure 3: NLP method breakdown Figure 4: NLP method by year Table 9: Breakdown of NLP Method ML (n=74) | No studies | Deep Learning (n=36) | No studies
---|---|---|---
SVM | 34 | RNN variants | 14
Logistic Regression | 23 | CNN | 10
Random Forest | 18 | Other | 5
Naïve Bayes | 17 | Compare CNN, RNN | 4
Maximum Entropy | 7 | Combine CNN+RNN | 3
Decision Trees | 4 |  |

A variety of machine classifier algorithms were used, with SVM and Logistic
Regression being the most common (Table 9). Recurrent Neural Networks (RNN)
variants were the most common type of deep learning architectures. RNN methods
were split between long short-term memory (LSTM) and bidirectional-LSTM (Bi-
LSTM), bi-directional gated recurrent unit (Bi-GRU), and standard RNN
approaches. Four of these studies additionally added a Conditional Random
Field (CRF) for the final label generation step. Convolutional Neural Networks
(CNN) were the second most common architecture explored. Eight studies
additionally used an attention mechanism as part of their deep learning
architecture. Other neural approaches included feed-forward neural networks,
fully connected neural networks and a proprietary neural system IBM Watson
(Trivedi et al., 2018) and Snorkel (Ratner et al., 2018). Several studies
proposed combined architectures, such as (Zhu et al., 2019; Short et al.,
2019).

### NLP Method Features

Most rule-based and machine classifying approaches used features based on bag-
of-words, part-of-speech, term frequency, and phrases with only two studies
alternatively using word embeddings. Three studies use feature engineering
with deep learning rather than word embeddings. Thirty-three studies use
domain-knowledge to support building features for their methods, such as
developing lexicons or selecting terms and phrases. Comparison of embedding
methods is difficult as many studies did not describe their embedding method.
Of those that did, Word2Vec (Mikolov et al., 2013) was the most popular
(n=19), followed by GLOVE embeddings (Pennington et al., 2014) (n=6), FastText
(Mikolov et al., 2018) (n=3), ELMo (Peters et al., 2018) (n=1) and BERT
(Devlin et al., 2018) (n=1). Ontologies or lexicon look-ups are used in 100
studies; however, even though publications increase over the period in real
terms, 20% fewer studies employ the use of ontologies or lexicons in 2019
compared to 2015. The most widely used resources were UMLS (National Library
of Medicine, 2021b) (n=15), Radlex (RSNA, 2021) (n=20), SNOMED-CT (National
Library of Medicine, 2021a) (n=14). Most studies used these as features for
normalising words and phrases for classification, but this was mainly those
using rule-based or machine learning classifiers with only six studies using
ontologies as input to their deep learning architecture. Three of those
investigated how existing ontologies can be combined with word embeddings to
create domain-specific mappings, with authors pointing to this avoiding the
need for large amounts of annotated data. Other approaches looked to extend
existing medical resources using a frequent phrases approach, e.g. (Bulu et
al., 2018). Works also used the derived concepts and relations visualising
these to support activities, such as report reading and report querying (e.g.
(Hassanpour and Langlotz, 2016; Zhao et al., 2018))

### Annotation and Inter-Annotator Agreement

Eighty-nine studies used at least two annotators, 75 did not specify any
annotation details, and only one study used a single annotator. Whilst 69
studies use a domain expert for annotation (a clinician or radiologist) only
56 studies report the inter-annotator agreement. Some studies mention
annotation but do not report on agreement or annotators. Inter-annotator
agreement values for Kappa range from 0.43 to perfect agreement at 1. Whilst
most studies reported agreement by Cohen’s Kappa (Cohen, 1960) some reported
precision, and percent agreement. Studies reported annotation data sizes
differently, e.g., on the sentence or patient level. Studies also considered
ground truth labels from coding schemes such as ICD or BI-RADS categories as
annotated data. Of studies which detailed human annotation at the radiology
report level, only 45 specified inter-annotator agreement and/or the number of
annotators. Annotated report numbers for these studies varies with 15 papers
having annotated less than 500, 12 having annotated between 500 and less than
1,000, 15 between 1,000 and less than 3,000, and 3 between 4,000 and 8,288
reports.

### Data Sources and Availability

Only 14 studies reported that their data is available, and 15 studies reported
that their code is available. Most studies sourced their data from medical
institutions, a number of studies did not specify where their data was from,
and some studies used publicly available datasets: MIMIC-III (n=5), MIMIC-II
(n=1), MIMIC-CXR (n=1); Radcore (n=5) or STRIDE (n=2). Four studies used
combined electronic health records such as clinical notes or pathology
reports.

Table 10: NLP Method by data size properties, minimum data size, maximum data size and median value, studies reporting in numbers of radiology reports NLP Method | Min Size | Max Size | Median
---|---|---|---
Compare Methods | 513 | 2,167,445 | 2,845
Hybrid Methods | 40 | 34,926 | 918
Deep Learning (Only) | 120 | 1,567,581 | 5,000
Machine Learning (Only) | 101 | 2,977,739 | 2,531
Rules (Only) | 31 | 10,000,000 | 8,000
Other | 25 | 12,377,743 | 10,000
Table 11: Grouped data size and number of studies in each group, only for studies reporting in numbers of radiology reports Data Size Group | No. Studies (%)
---|---
<200 | 9 (6.7)
200 <500 | 6 (4.4)
500 <1,000 | 18 (13.3)
1,000 <2,000 | 17 (12.6)
2,000 <5,000 | 17 (12.6)
5,000 <10,000 | 12 (8.9)
10,000+ | 53 (39.3)
Unspecified | 3 (2.2)

Reporting on data size and splits differed across studies with some not giving
exact data sizes and others reporting numbers of sentences, patients, or mixed
data sources rather than radiology reports. Data sizes for those reporting at
the radiology report level is n=135 or 82.32% of the studies (Table 10). The
biggest variation of data size by NLP Method is in studies that apply other
methods or are rule-based. Machine learning also varies in size; however, the
median value is lower compared to rule-based methods. The median value for
deep learning is considerably higher at 5,000 reports compared to machine
learning or those that compare or create hybrid methods. Of the studies
reporting on radiology reports numbers, 39.3% used over 10,000 reports and
this increases to over 48% using more than 5,000 reports. However, a small
number of studies, 14%, are using comparatively low numbers of radiology
reports, less than 500 (Table 11).

### NLP Performance and Evaluation Measures

Performance metrics applied for evaluation of methods vary widely with authors
using precision (positive predictive value (PPV)), recall (sensitivity),
specificity, the area under the curve (AUC) or accuracy. We observed a wide
variety in evaluation methodology employed concerning test or validation
datasets. Different approaches were taken in generating splits for testing and
validation, including k-fold cross-validation. Ninety-nine studies reported on
training and test data splits, of which only 59 studies included a validation
set. Only 10 studies validated their algorithm using an external dataset from
another institution, another modality, or a different patient population. The
most widely used metrics for reporting performance were precision (PPV) and
recall (sensitivity) reported in 47% of studies. However, even though many
studies compared methods and reported on the top-performing method, very few
studies carried out significance testing on these comparisons. Issues of
heterogeneity make it difficult and unrealistic to compare performance between
methods applied, hence, we use summary measures as a broad overview (Figure
5). Performance reported varies, but both the mean and median values for the
F1 score appear higher for methods using rule-based only or deep learning only
methods. Whilst differences are less discernible between F1 scores for
application areas, Diagnostic Surveillance looks on average lower than other
categories.

Figure 5: Application Category and NLP Method, Mean and Median Summaries. Mean
value is indicated by a vertical bar, the box shows error bars and the
asterisk is the median value.

## Discussion and Future Directions

Our work shows there has been a considerable increase in the number of
publications using NLP on radiology reports over the recent time period.
Compared to 67 publications retrieved in the earlier review of (Pons et al.,
2016), we retrieved 164 publications. In this section we discuss and offer
some insight into the observations and trends of how NLP is being applied to
radiology and make some recommendations that may benefit the field going
forward.

### Clinical Applications and NLP Methods in Radiology

The clinical applications of the publications is similar to the earlier review
of Pons et al. but whilst we observe an increase in research output we also
highlight that there appears to be even less focus on clinical application
compared to their review. Like many other fields applying NLP the use of deep
learning has increased, with RNN architectures being the most popular. This is
also observed in a review of NLP in clinical text(Wu et al., 2020). However,
although deep learning use increases, rules and traditional machine
classifiers are still prevalent and often used as baselines to compare deep
learning architectures against. One reason for traditional methods remaining
popular is their interpretability compared to deep learning models.
Understanding the features that drive a model prediction can support decision-
making in the clinical domain but the complex layers of non-linear data
transformations deep learning is composed of does not easily support
transparency (Shickel et al., 2018). This may also help explain why in
synthesis of the literature we observed less focus on discussing clinical
application and more emphasis on disease classification or information task
only. Advances in interpretability of deep learning models are critical to its
adoption in clinical practice.

Other challenges exist for deep learning such as only having access to small
or imbalanced datasets. Chen et al. (Chen et al., 2019) review deep learning
methods within healthcare and point to these challenges resulting in poor
performance but that these same datasets can perform well with traditional
machine learning methods. We found several studies highlight this and when
data is scarce or datasets imbalanced, they introduced hybrid approaches of
rules and deep learning to improve performance, particularly in the Diagnostic
Surveillance category. Yang et al. (Yang et al., 2018) observed rules
performing better for some entity types, such as time and size, which are
proportionally lower than some of the other entities in their train and test
sets; hence they combine a bidirectional-LSTM and CRF with rules for entity
recognition. Peng et al. (Peng et al., 2019) comment that combining rules and
the neural architecture complement each other, with deep learning being more
balanced between precision and recall, but the rule-based method having higher
precision and lower recall. The authors reason that this provides better
performance as rules can capture rare disease cases, particularly when multi-
class labelling is needed, whilst deep learning architectures perform worse in
instances with fewer data points.

In addition to its need for large-scale data, deep learning can be
computationally costly. The use of pre-trained models and embeddings may
alleviate some of this burden. Pre-trained models often only require fine-
tuning, which can reduce computation cost. Language comprehension pre-learned
from other tasks can then be inherited from the parent models, meaning fewer
domain-specific labelled examples may be needed (Wood et al., 2020). This use
of pre-trained information also supports generalisability, e.g., (Banerjee et
al., 2019b) show that their model trained on one dataset can generalise to
other institutional datasets.

Embedding use has increased which is expected with the application of deep
learning approaches but many rule-based and machine classifiers continue to
use traditional count-based features, e.g., bag-of-words and n-grams. Recent
evidence (Ong et al., 2020) suggests that the trend to continue to use feature
engineering with traditional machine learning methods does produce better
performance in radiology reports than using domain-specific word embeddings.

Banerjee et al. (Banerjee et al., 2017) found that there was not much
difference between a uni-gram approach and a Word2vec embedding, hypothesising
this was due to their narrow domain, intracranial haemorrhage. However, the
NLP research field has seen a move towards bi-directional encoder
representations from transformers (BERT) based embedding models not reflected
in our analysis, with only one study using BERT generated embeddings (Deshmukh
et al., 2019). Embeddings from BERT are thought to be superior as they can
deliver better contextual representations and result in improved task
performance. Whilst more publications since our review period have used BERT
based embeddings with radiology reports e.g. (Wood et al., 2020; Smit et al.,
2020a) not all outperform traditional methods (Grivas et al., 2020). Recent
evidence shows that embeddings generated by BERT fail to show a generalisable
understanding of negation (Ettinger, 2020), an essential factor in
interpreting radiology reports effectively. Specialised BERT models have been
introduced such as ClinicalBERT (Alsentzer et al., 2019) or BlueBERT (Smit et
al., 2020a). BlueBERT has been shown to outperform ClinicalBERT when
considering chest radiology (Smit et al., 2020b) but more exploration of the
performance gains versus the benefits of generalisability are needed for
radiology text.

All NLP models have in common that they need large amounts of labelled data
for model training (Yasaka and Abe, 2018). Several studies (Percha et al.,
2018; Tahmasebi et al., 2019; Banerjee et al., 2018) explored combining word
embeddings and ontologies to create domain-specific mappings, and they suggest
this can avoid a need for large amounts of annotated data. Additionally,
(Percha et al., 2018; Tahmasebi et al., 2019) highlight that such combinations
could boost coverage and performance compared to more conventional techniques
for concept normalisation.

The number of publications using medical lexical knowledge resources is still
relatively low, even though a recent trend in the general NLP field is to
enhance deep learning with external knowledge (Young et al., 2018). This was
also observed by (Wu et al., 2020), where only 18% of the deep learning
studies in their review utilised knowledge resources. Although pre-training
supports learning previously known facts it could introduce unwanted bias,
hindering performance. The inclusion of domain expertise through resources
such as medical lexical knowledge may help reduce this unwanted bias (Wu et
al., 2020). Exploration of how this domain expertise can be incorporated with
deep learning architectures in future could improve the performance when
having access to less labelled data.

### Task Knowledge

Knowledge about the disease area of interest and how aspects of this disease
are linguistically expressed is useful and could promote better performing
solutions. Whilst (Donnelly et al., 2019) find high variability between
radiologists, with metric values (e.g. number of syntactic, clinical terms
based on ontology mapping) being significantly greater on free-text than
structured reports, (Xie et al., 2019) who look specifically at anatomical
areas find less evidence for variability. Zech et al. (Zech et al., 2018)
suggest that the highly specialised nature of each imaging modality creates
different sub-languages and the ability to discover these labels (i.e. disease
mentions) reflects the consistency with which labels are referred to. For
example, edema is referred to very consistently whereas other labels are not,
such as infarction/ischaemic. Understanding the language and the context of
entity mentions could help promote novel ideas on how to solve problems more
effectively. For example, (Yim et al., 2017) discuss how the accuracy of
predicting malignancy is affected by cues being outside their window of
consideration and (Yim et al., 2018) observe problems of co-reference
resolution within a report due to long-range dependencies. Both these studies
use traditional NLP approaches, but we observed novel neural architectures
being proposed to improve performance in similar tasks specifically capturing
long-range context and dependency learning, e.g., (Zhu et al., 2019; Short et
al., 2019). This understanding requires close cooperation of healthcare
professionals and data scientists, which is different to some other fields
where more disconnection is present (Chen et al., 2019).

### Study Heterogeneity, a Need for Reporting Standards

Most studies reviewed could be described as a proof-of-concept and not
trialled in a clinical setting. Pons et al. (Pons et al., 2016) hypothesised
that a lack of clinical application may stem from uncertainty around minimal
performance requirements hampering implementations, evidence-based practice
requiring justification and transparency of decisions, and the inability to be
able to compare to human performance as the human agreement is often an
unknown. These hypotheses are still valid, and we see little evidence that
these problems are solved.

Human annotation is generally considered the gold standard at measuring human
performance, and whilst many studies reported that they used annotated data,
overall, reporting was inconsistent. Steps were undertaken to measure inter-
annotator agreement (IAA), but in many studies, this was not directly
comparable to the evaluation undertaken of the NLP methods. The size of the
data being used to draw experimental conclusions from is important and
accurate reporting of these measures is essential to ensure reproducibility
and comparison in further studies. Reporting on the training, test and
validation splits was varied with some studies not giving details and not
using held-out validation sets.

Most studies use retrospective data from single institutions but this can lead
to a model over-fitting and, thus, not generalising well when applied in a new
setting. Overcoming the problem of data availability is challenging due to
privacy and ethics concerns, but essential to ensure that performance of
models can be investigated across institutions, modalities, and methods.
Availability of data would allow for agreed benchmarks to be developed within
the field that algorithm improvements can be measured upon. External
validation of applied methods was extremely low, although, this is likely due
to the availability of external datasets. Making code available would enable
researchers to report how external systems perform on their data. However,
only 15 studies reported that their code is available. To be able to compare
systems there is a need for common datasets to be available to benchmark and
compare systems against.

Whilst reported figures in precision and recall generally look high more
evidence is needed for accurate comparison to human performance. A wide
variety of performance measures were used, with some studies only reporting
one measure, e.g., accuracy or F1 scores, with these likely representing the
best performance obtained. Individual studies are often not directly
comparable for such measures, but none-the-less clarity and consistency in
reporting is desirable. Many studies making model comparisons did not carry
out any significance testing for these comparisons.

The make the following recommendations to help move the field forward, enable
more inter-study comparisons, and increase study reproducibility:

  1. 1.

Clarity in reporting study properties is required: (a) Data characteristics
including size and the type of dataset should be detailed, e.g., the number of
reports, sentences, patients, and if patients how many reports per patient.
The training, test and validation data split should be evident, as should the
source of the data. (b) Annotation characteristics including the methodology
to develop the annotation should be reported, e.g., annotation set size,
annotator details, how many, expertise. (c) Performance metrics should include
a range of metrics: precision, recall, F1, accuracy and not just one overall
value.

  2. 2.

Significance testing should be carried out when a comparison between methods
is made.

  3. 3.

Data and code availability are encouraged. While making data available will
often be challenging due to privacy concerns, researchers should make code
available to enable inter-study comparisons and external validation of
methods.

  4. 4.

Common datasets should be used to benchmark and compare systems.

### Limitations of Study

Publication search is subject to bias in search methods and it is likely that
our search strategy did inevitably miss some publications. Whilst trying to be
precise and objective during our review process some of the data collected and
categorising publications into categories was difficult to agree on and was
subjective. For example, many of the publications could have belonged to more
than one category. One of the reasons for this was how diverse in structure
the content was which was in some ways reflected by the different domains
papers were published in. It is also possible that certain keywords were
missed in recording data elements due to the reviewers own biases and research
experience.

## Conclusions

This paper presents an systematic review of publications using NLP on
radiology reports during the period 2015 to October 2019. We show there has
been substantial growth in the field particularly in researchers using deep
learning methods. Whilst deep learning use has increased, as seen in NLP
research in general, it faces challenges of lower performance when data is
scarce or when labelled data is unavailable, and is not widely used in
clinical practice perhaps due to the difficulties in interpretability of such
models. Traditional machine learning and rule-based methods are, therefore,
still widely in use. Exploration of domain expertise such as medial lexical
knowledge must be explored further to enhance performance when data is scarce.
The clinical domain faces challenges due to privacy and ethics in sharing data
but overcoming this would enable development of benchmarks to measure
algorithm performance and test model robustness across institutions. Common
agreed datasets to compare performance of tools against would help support the
community in inter-study comparisons and validation of systems. The work we
present here has the potential to inform researchers about applications of NLP
to radiology and to lead to more reliable and responsible research in the
domain.

## Acknowledgements

Not applicable

## Funding

This research was supported by the Alan Turing Institute, MRC, HDR-UK and the
Chief Scientist Office. B.A.,A.C,D.D.,A.G. and C.G. have been supported by the
Alan Turing Institute via Turing Fellowships (B.A,C.G.) and Turing project
funding (ESPRC grant EP/N510129/1). A.G. was also funded by a MRC Mental
Health Data Pathfinder Award (MRC-MCPC17209). H.W. is MRC/Rutherford Fellow
HRD UK (MR/S004149/1). H.D. is supported by HDR UK National Phemomics Resource
Project. V.S-P. is supported by the HDR UK National Text Analytics
Implementation Project. W.W. is supported by a Scottish Senior Clinical
Fellowship (CAF/17/01).

## Abbreviations

NLP - natural language processing
e.g. - example
ICD - international classification of diseases
BI-RADS - Breast Imaging-Reporting and Data System
IAA - inter-annotator agreement
No. - number
UMLS - unified medical language system
ELMo - embeddings from Language Models
BERT - bidirectional encoder representations form transformers
SVM - support vector machine
CNN - convolutional neural network
LSTM - long short-term memory
Bi-LSTM - bi-directional long short-term memory
Bi-GRU - bi-directional gated recurrent unit
CRF - conditional random field
GLOVE - Global Vectors for Word Representation

## References

  * Alex et al. [2019] Beatrice Alex, Claire Grover, Richard Tobin, Cathie Sudlow, Grant Mair, and William Whiteley.  Text mining brain imaging reports.  _Journal of Biomedical Semantics_ , 10(1):23, November 2019.  ISSN 2041-1480.  doi: 10.1186/s13326-019-0211-7.  URL https://doi.org/10.1186/s13326-019-0211-7.
  * Alsentzer et al. [2019] Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott.  Publicly Available Clinical BERT Embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota, USA, June 2019\. Association for Computational Linguistics.  doi: 10.18653/v1/W19-1909.  URL https://www.aclweb.org/anthology/W19-1909.
  * Banerjee et al. [2017] Imon Banerjee, Sriraman Madhavan, Roger Eric Goldman, and Daniel L. Rubin.  Intelligent Word Embeddings of Free-Text Radiology Reports.  _AMIA Annual Symposium Proceedings_ , pages 411–420, 2017.  ISSN 1942-597X.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5977573/.
  * Banerjee et al. [2018] Imon Banerjee, Matthew C. Chen, Matthew P. Lungren, and Daniel L. Rubin.  Radiology report annotation using intelligent word embeddings: Applied to multi-institutional chest CT cohort.  _Journal of Biomedical Informatics_ , 77:11–20, January 2018.  ISSN 1532-0464.  doi: 10.1016/j.jbi.2017.11.012.  URL http://www.sciencedirect.com/science/article/pii/S1532046417302575.
  * Banerjee et al. [2019a] Imon Banerjee, Selen Bozkurt, Emel Alkim, Hersh Sagreiya, Allison W. Kurian, and Daniel L. Rubin.  Automatic inference of BI-RADS final assessment categories from narrative mammography report findings.  _Journal of Biomedical Informatics_ , 92:103137, April 2019a.  ISSN 1532-0464.  doi: 10.1016/j.jbi.2019.103137.  URL http://www.sciencedirect.com/science/article/pii/S1532046419300553.
  * Banerjee et al. [2019b] Imon Banerjee, Yuan Ling, Matthew C. Chen, Sadid A. Hasan, Curtis P. Langlotz, Nathaniel Moradzadeh, Brian Chapman, Timothy Amrhein, David Mong, Daniel L. Rubin, Oladimeji Farri, and Matthew P. Lungren.  Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification.  _Artificial Intelligence in Medicine_ , 97:79–88, June 2019b.  ISSN 0933-3657.  doi: 10.1016/j.artmed.2018.11.004.  URL http://www.sciencedirect.com/science/article/pii/S0933365717306255.
  * Bates et al. [2016] Jonathan Bates, Samah J. Fodeh, Cynthia A. Brandt, and Julie A. Womack.  Classification of radiology reports for falls in an HIV study cohort.  _Journal of the American Medical Informatics Association_ , 23(e1):e113–e117, April 2016.  ISSN 1067-5027.  doi: 10.1093/jamia/ocv155.  URL https://academic.oup.com/jamia/article/23/e1/e113/2379897.
  * Beyer et al. [2017] Sebastian E. Beyer, Brady J. McKee, Shawn M. Regis, Andrea B. McKee, Sebastian Flacke, Gilan El Saadawi, and Christoph Wald.  Automatic Lung-RADS™ classification with a natural language processing system.  _Journal of Thoracic Disease_ , 9(9):3114–3122, September 2017.  ISSN 2072-1439.  doi: 10.21037/jtd.2017.08.13.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5708435/.
  * Bobbin et al. [2017] Mark D. Bobbin, Ivan K. Ip, V. Anik Sahni, Atul B. Shinagare, and Ramin Khorasani.  Focal Cystic Pancreatic Lesion Follow-up Recommendations After Publication of ACR White Paper on Managing Incidental Findings.  _Journal of the American College of Radiology_ , 14(6):757–764, June 2017.  ISSN 1546-1440.  doi: 10.1016/j.jacr.2017.01.044.  URL http://www.sciencedirect.com/science/article/pii/S1546144017301771.
  * Bozkurt et al. [2016] Selen Bozkurt, Francisco Gimenez, Elizabeth S. Burnside, Kemal H. Gulkesen, and Daniel L. Rubin.  Using automatically extracted information from mammography reports for decision-support.  _Journal of Biomedical Informatics_ , 62:224–231, August 2016.  ISSN 1532-0464.  doi: 10.1016/j.jbi.2016.07.001.  URL http://www.sciencedirect.com/science/article/pii/S1532046416300557.
  * Bozkurt et al. [2019] Selen Bozkurt, Emel Alkim, Imon Banerjee, and Daniel L. Rubin.  Automated Detection of Measurements and Their Descriptors in Radiology Reports Using a Hybrid Natural Language Processing Algorithm.  _Journal of Digital Imaging_ , 32(4):544–553, August 2019.  ISSN 1618-727X.  doi: 10.1007/s10278-019-00237-9.  URL https://doi.org/10.1007/s10278-019-00237-9.
  * Briscoe et al. [2020] Simon Briscoe, Alison Bethel, and Morwenna Rogers.  Conduct and reporting of citation searching in Cochrane systematic reviews: A cross-sectional study.  _Research Synthesis Methods_ , 11(2):169–180, 2020.  ISSN 1759-2887.  doi: 10.1002/jrsm.1355.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1355.
  * Brizzi et al. [2019] Kate Brizzi, Sophia N. Zupanc, Brooks V. Udelsman, James A. Tulsky, Alexi A. Wright, Hanneke Poort, and Charlotta Lindvall.  Natural Language Processing to Assess Palliative Care and End-of-Life Process Measures in Patients With Breast Cancer With Leptomeningeal Disease.  _American Journal of Hospice and Palliative Medicine_ , 37(5):371–376, 2019.  doi: https://doi.org/10.1177/1049909119885585.  URL https://journals.sagepub.com/doi/abs/10.1177/1049909119885585.
  * Brown and Kachura [2019] A. D. Brown and J. R. Kachura.  Natural Language Processing of Radiology Reports in Patients With Hepatocellular Carcinoma to Predict Radiology Resource Utilization.  _Journal of the American College of Radiology_ , 16(6):840–844, June 2019.  ISSN 1546-1440.  doi: 10.1016/j.jacr.2018.12.004.  URL http://www.sciencedirect.com/science/article/pii/S1546144018315539.
  * Brown and Marotta [2017] Andrew D. Brown and Thomas R. Marotta.  A Natural Language Processing-based Model to Automate MRI Brain Protocol Selection and Prioritization.  _Academic Radiology_ , 24(2):160–166, February 2017.  ISSN 1076-6332.  doi: 10.1016/j.acra.2016.09.013.  URL http://www.sciencedirect.com/science/article/pii/S1076633216303270.
  * Brown and Marotta [2018] Andrew D. Brown and Thomas R. Marotta.  Using machine learning for sequence-level automated MRI protocol selection in neuroradiology.  _Journal of the American Medical Informatics Association_ , 25(5):568–571, May 2018.  ISSN 1067-5027.  doi: 10.1093/jamia/ocx125.  URL https://academic.oup.com/jamia/article/25/5/568/4569611.
  * Bulu et al. [2018] Hakan Bulu, Dorothy A. Sippo, Janie M. Lee, Elizabeth S. Burnside, and Daniel L. Rubin.  Proposing New RadLex Terms by Analyzing Free-Text Mammography Reports.  _Journal of Digital Imaging_ , 31(5):596–603, October 2018.  ISSN 1618-727X.  doi: 10.1007/s10278-018-0064-0.  URL https://doi.org/10.1007/s10278-018-0064-0.
  * Cai et al. [2016] Tianrun Cai, Andreas A. Giannopoulos, Sheng Yu, Tatiana Kelil, Beth Ripley, Kanako K. Kumamaru, Frank J. Rybicki, and Dimitrios Mitsouras.  Natural Language Processing Technologies in Radiology Research and Clinical Applications.  _RadioGraphics_ , 36(1):176–191, January 2016\.  ISSN 0271-5333.  doi: 10.1148/rg.2016150080.  URL https://pubs.rsna.org/doi/full/10.1148/rg.2016150080.
  * Castro et al. [2017] Sergio M. Castro, Eugene Tseytlin, Olga Medvedeva, Kevin Mitchell, Shyam Visweswaran, Tanja Bekhuis, and Rebecca S. Jacobson.  Automated annotation and classification of BI-RADS assessment from radiology reports.  _Journal of Biomedical Informatics_ , 69:177–187, May 2017\.  ISSN 1532-0464.  doi: 10.1016/j.jbi.2017.04.011.  URL http://www.sciencedirect.com/science/article/pii/S1532046417300813.
  * Chen et al. [2019] David Chen, Sijia Liu, Paul Kingsbury, Sunghwan Sohn, Curtis B. Storlie, Elizabeth B. Habermann, James M. Naessens, David W. Larson, and Hongfang Liu.  Deep learning and alternative learning strategies for retrospective real-world clinical data.  _npj Digital Medicine_ , 2(1):1–5, May 2019.  ISSN 2398-6352.  doi: 10.1038/s41746-019-0122-0.  URL https://www.nature.com/articles/s41746-019-0122-0.
  * Chen et al. [2017] Matthew C. Chen, Robyn L. Ball, Lingyao Yang, Nathaniel Moradzadeh, Brian E. Chapman, David B. Larson, Curtis P. Langlotz, Timothy J. Amrhein, and Matthew P. Lungren.  Deep Learning to Classify Radiology Free-Text Reports.  _Radiology_ , 286(3):845–852, November 2017.  ISSN 0033-8419.  doi: 10.1148/radiol.2017171115.  URL https://pubs.rsna.org/doi/full/10.1148/radiol.2017171115.
  * Chen et al. [2018] Po-Hao Chen, Hanna Zafar, Maya Galperin-Aizenberg, and Tessa Cook.  Integrating Natural Language Processing and Machine Learning Algorithms to Categorize Oncologic Response in Radiology Reports.  _Journal of Digital Imaging_ , 31(2):178–184, April 2018.  ISSN 1618-727X.  doi: 10.1007/s10278-017-0027-x.  URL https://doi.org/10.1007/s10278-017-0027-x.
  * Cocos et al. [2017] Anne Cocos, Ting Qian, Chris Callison-Burch, and Aaron J. Masino.  Crowd control: Effectively utilizing unscreened crowd workers for biomedical data annotation.  _Journal of Biomedical Informatics_ , 69:86–92, May 2017\.  ISSN 1532-0464.  doi: 10.1016/j.jbi.2017.04.003.  URL http://www.sciencedirect.com/science/article/pii/S1532046417300746.
  * Cohen [1960] Jacob Cohen.  A Coefficient of Agreement for Nominal Scales.  _Educational and Psychological Measurement_ , 20(1):37–46, April 1960.  ISSN 0013-1644.  doi: 10.1177/001316446002000104.  URL https://doi.org/10.1177/001316446002000104.
  * Comelli et al. [2015] A. Comelli, L. Agnello, and S. Vitabile.  An ontology-based retrieval system for mammographic reports.  In _2015 IEEE Symposium on Computers and Communication (ISCC)_ , pages 1001–1006, Larnaca, July 2015. IEEE.  doi: 10.1109/ISCC.2015.7405644.
  * Cotik et al. [2015] Viviana Cotik, Dario Filippo, and Jose Castano.  An Approach for Automatic Classification of Radiology Reports in Spanish.  _Studies in Health Technology and Informatics_ , 216:634–638, jan 2015.  ISSN 0926-9630, 1879-8365.  URL https://europepmc.org/article/med/26262128.
  * Cotik et al. [2018] Viviana Cotik, Horacio Rodríguez, and Jorge Vivaldi.  Spanish Named Entity Recognition in the Biomedical Domain.  In Juan Antonio Lossio-Ventura, Denisse Muñante, and Hugo Alatrista-Salas, editors, _Information Management and Big Data_ , volume 898 of _Communications in Computer and Information Science_ , pages 233–248, Lima, Peru, 2018. Springer International Publishing.  ISBN 978-3-030-11680-4.  doi: 10.1007/978-3-030-11680-4-23.
  * Dalal et al. [2020] Sandeep Dalal, Vadiraj Hombal, Wei-Hung Weng, Gabe Mankovich, Thusitha Mabotuwana, Christopher S. Hall, Joseph Fuller, Bruce E. Lehnert, and Martin L. Gunn.  Determining Follow-Up Imaging Study Using Radiology Reports.  _Journal of Digital Imaging_ , 33(1):121–130, February 2020.  ISSN 1618-727X.  doi: 10.1007/s10278-019-00260-w.  URL https://doi.org/10.1007/s10278-019-00260-w.
  * Deshmukh et al. [2019] Neil Deshmukh, Selin Gumustop, Romane Gauriau, Varun Buch, Bradley Wright, Christopher Bridge, Ram Naidu, Katherine Andriole, and Bernardo Bizzo.  Semi-Supervised Natural Language Approach for Fine-Grained Classification of Medical Reports.  _arXiv:1910.13573 [cs.LG]_ , November 2019.  URL http://arxiv.org/abs/1910.13573.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Donnelly et al. [2019] Lane F. Donnelly, Robert Grzeszczuk, Carolina V. Guimaraes, Wei Zhang, and George S. Bisset III.  Using a Natural Language Processing and Machine Learning Algorithm Program to Analyze Inter-Radiologist Report Style Variation and Compare Variation Between Radiologists When Using Highly Structured Versus More Free Text Reporting.  _Current Problems in Diagnostic Radiology_ , 48(6):524–530, November 2019.  ISSN 0363-0188.  doi: 10.1067/j.cpradiol.2018.09.005.  URL http://www.sciencedirect.com/science/article/pii/S0363018818302081.
  * Dunne et al. [2015] Ruth M. Dunne, Ivan K. Ip, Sarah Abbett, Esteban F. Gershanik, Ali S. Raja, Andetta Hunsaker, and Ramin Khorasani.  Effect of Evidence-based Clinical Decision Support on the Use and Yield of CT Pulmonary Angiographic Imaging in Hospitalized Patients.  _Radiology_ , 276(1):167–174, February 2015.  ISSN 0033-8419.  doi: 10.1148/radiol.15141208.  URL https://pubs.rsna.org/doi/full/10.1148/radiol.15141208.
  * Ettinger [2020] Allyson Ettinger.  What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48, January 2020.  doi: 10.1162/tacl“˙a“˙00298.  URL https://doi.org/10.1162/tacl_a_00298.
  * Farjah et al. [2016] Farhood Farjah, Scott Halgrim, Diana S.M. Buist, Michael K. Gould, Steven B. Zeliadt, Elizabeth T. Loggers, and David S. Carrell.  An Automated Method for Identifying Individuals with a Lung Nodule Can Be Feasibly Implemented Across Health Systems.  _eGEMs_ , 4(1):1254, August 2016.  ISSN 2327-9214.  doi: 10.13063/2327-9214.1254.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5013935/.
  * Fleiss [1971] Joseph L. Fleiss.  Measuring nominal scale agreement among many raters.  _Psychological Bulletin_ , 76(5):378–382, 1971\.  ISSN 1939-1455(Electronic),0033-2909(Print).  doi: 10.1037/h0031619.
  * Fu et al. [2019] Sunyang Fu, Lester Y. Leung, Yanshan Wang, Anne-Olivia Raulli, David F. Kallmes, Kristin A. Kinsman, Kristoff B. Nelson, Michael S. Clark, Patrick H. Luetmer, Paul R. Kingsbury, David M. Kent, and Hongfang Liu.  Natural Language Processing for the Identification of Silent Brain Infarcts From Neuroimaging Reports.  _JMIR Medical Informatics_ , 7(2):e12109, 2019\.  doi: 10.2196/12109.  URL https://medinform.jmir.org/2019/2/e12109/.
  * Garg et al. [2019] Ravi Garg, Elissa Oh, Andrew Naidech, Konrad Kording, and Shyam Prabhakaran.  Automating Ischemic Stroke Subtype Classification Using Machine Learning and Natural Language Processing.  _Journal of Stroke and Cerebrovascular Diseases_ , 28(7):2045–2051, July 2019.  ISSN 1052-3057.  doi: 10.1016/j.jstrokecerebrovasdis.2019.02.004.  URL http://www.sciencedirect.com/science/article/pii/S1052305719300485.
  * Gehanno et al. [2013] Jean-François Gehanno, Laetitia Rollin, and Stefan Darmoni.  Is the coverage of google scholar enough to be used alone for systematic reviews.  _BMC Medical Informatics and Decision Making_ , 13:7, 2013\.  doi: 10.1186/1472-6947-13-7.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3544576/.
  * Goldshtein et al. [2020] Inbal Goldshtein, Gabriel Chodick, Ilan Kochba, Nitsan Gal, Muriel Webb, and Oren Shibolet.  Identification and Characterization of Nonalcoholic Fatty Liver Disease.  _Clinical Gastroenterology and Hepatology_ , 18(8):1887–1889, July 2020.  ISSN 1542-3565.  doi: 10.1016/j.cgh.2019.08.007.  URL http://www.sciencedirect.com/science/article/pii/S1542356519308638.
  * Gorinski et al. [2019] Philip John Gorinski, Honghan Wu, Claire Grover, Richard Tobin, Conn Talbot, Heather Whalley, Cathie Sudlow, William Whiteley, and Beatrice Alex.  Named Entity Recognition for Electronic Health Records: A Comparison of Rule-based and Machine Learning Approaches.  _arXiv:1903.03985 [cs.CL]_ , June 2019.  URL http://arxiv.org/abs/1903.03985.
  * Gould et al. [2015] Michael K. Gould, Tania Tang, In-Lu Amy Liu, Janet Lee, Chengyi Zheng, Kim N. Danforth, Anne E. Kosco, Jamie L. Di Fiore, and David E. Suh.  Recent Trends in the Identification of Incidental Pulmonary Nodules.  _American Journal of Respiratory and Critical Care Medicine_ , 192(10):1208–1214, July 2015.  ISSN 1073-449X.  doi: 10.1164/rccm.201505-0990OC.  URL https://www.atsjournals.org/doi/full/10.1164/rccm.201505-0990OC.
  * Grivas et al. [2020] A. Grivas, B. Alex, C. Grover, Tobin, R., and Whiteley, W.  Not a cute stroke: Analysis of Rule- and Neural Network-Based Information Extraction Systems for Brain Radiology Reports.  In _Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis_ , 2020.
  * Grundmeier et al. [2016] Robert W. Grundmeier, Aaron J. Masino, T. Charles Casper, Jonathan M. Dean, Jamie Bell, Rene Enriquez, Sara Deakyne, James M. Chamberlain, and Elizabeth R. Alpern.  Identification of Long Bone Fractures in Radiology Reports Using Natural Language Processing to Support Healthcare Quality Improvement.  _Applied Clinical Informatics_ , 7(4):1051–1068, November 2016.  ISSN 1869-0327.  doi: 10.4338/ACI-2016-08-RA-0129.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5228143/.
  * Gupta et al. [2018] Anupama Gupta, Imon Banerjee, and Daniel L. Rubin.  Automatic information extraction from unstructured mammography reports using distributed semantics.  _Journal of Biomedical Informatics_ , 78:78–86, February 2018.  ISSN 1532-0464.  doi: 10.1016/j.jbi.2017.12.016.  URL http://www.sciencedirect.com/science/article/pii/S1532046417302903.
  * Harzing A. W. [2007] Harzing A. W.  _Publish or Perish_.  2007\.  URL Availablefromhttps://harzing.com/resources/publish-or-perish.
  * Hassanpour and Langlotz [2016] Saeed Hassanpour and Curtis P. Langlotz.  Unsupervised Topic Modeling in a Large Free Text Radiology Report Repository.  _Journal of Digital Imaging_ , 29(1):59–62, February 2016.  ISSN 1618-727X.  doi: 10.1007/s10278-015-9823-3.  URL https://doi.org/10.1007/s10278-015-9823-3.
  * Hassanpour et al. [2017] Saeed Hassanpour, Graham Bay, and Curtis P. Langlotz.  Characterization of Change and Significance for Clinical Findings in Radiology Reports Through Natural Language Processing.  _Journal of Digital Imaging_ , 30(3):314–322, June 2017.  ISSN 1618-727X.  doi: 10.1007/s10278-016-9931-8.  URL https://doi.org/10.1007/s10278-016-9931-8.
  * Heilbrun et al. [2019] Marta E. Heilbrun, Brian E. Chapman, Evan Narasimhan, Neel Patel, and Danielle Mowery.  Feasibility of Natural Language Processing–Assisted Auditing of Critical Findings in Chest Radiology.  _Journal of the American College of Radiology_ , 16(9, Part B):1299–1304, September 2019.  ISSN 1546-1440.  doi: 10.1016/j.jacr.2019.05.038.  URL http://www.sciencedirect.com/science/article/pii/S1546144019306386.
  * Hong and Zhang [2015] Yi Hong and Jin Zhang.  Investigation of Terminology Coverage in Radiology Reporting Templates and Free‐text Reports.  _International Journal of Knowledge Content Development & Technology_, 5:5–14, 2015.  doi: 10.5865/IJKCT.2015.5.1.005.
  * Huhdanpaa et al. [2018] Hannu T. Huhdanpaa, W. Katherine Tan, Sean D. Rundell, Pradeep Suri, Falgun H. Chokshi, Bryan A. Comstock, Patrick J. Heagerty, Kathryn T. James, Andrew L. Avins, Srdjan S. Nedeljkovic, David R. Nerenz, David F. Kallmes, Patrick H. Luetmer, Karen J. Sherman, Nancy L. Organ, Brent Griffith, Curtis P. Langlotz, David Carrell, Saeed Hassanpour, and Jeffrey G. Jarvik.  Using Natural Language Processing of Free-Text Radiology Reports to Identify Type 1 Modic Endplate Changes.  _Journal of Digital Imaging_ , 31(1):84–90, February 2018.  ISSN 1618-727X.  doi: 10.1007/s10278-017-0013-3.  URL https://doi.org/10.1007/s10278-017-0013-3.
  * Jnawali et al. [2019] K. Jnawali, M. R. Arbabshirani, A. E. Ulloa, N. Rao, and A. A. Patel.  Automatic Classification of Radiological Report for Intracranial Hemorrhage.  In _2019 IEEE 13th International Conference on Semantic Computing (ICSC)_ , pages 187–190, Newport Beach, CA, USA, January 2019. IEEE.  doi: 10.1109/ICOSC.2019.8665578.
  * Johnson et al. [2015] E. Johnson, W. C. Baughman, and G. Ozsoyoglu.  A method for imputation of semantic class in diagnostic radiology text.  In _2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , pages 750–755, Washington, DC, November 2015. IEEE.  doi: 10.1109/BIBM.2015.7359780.
  * Kang et al. [2019] Stella K. Kang, Kira Garry, Ryan Chung, William H. Moore, Eduardo Iturrate, Jordan L. Swartz, Danny C. Kim, Leora I. Horwitz, and Saul Blecker.  Natural Language Processing for Identification of Incidental Pulmonary Nodules in Radiology Reports.  _Journal of the American College of Radiology_ , 16(11):1587–1594, November 2019.  ISSN 1546-1440.  doi: 10.1016/j.jacr.2019.04.026.  URL http://www.sciencedirect.com/science/article/pii/S1546144019305332.
  * Karunakaran et al. [2017] B. Karunakaran, D. Misra, K. Marshall, D. Mathrawala, and S. Kethireddy.  Closing the loop — Finding lung cancer patients using NLP.  In _2017 IEEE International Conference on Big Data (Big Data)_ , pages 2452–2461, Boston, MA, December 2017. IEEE.  doi: 10.1109/BigData.2017.8258203.
  * Kehl et al. [2019] Kenneth L. Kehl, Haitham Elmarakeby, Mizuki Nishino, Eliezer M. Van Allen, Eva M. Lepisto, Michael J. Hassett, Bruce E. Johnson, and Deborah Schrag.  Assessment of Deep Natural Language Processing in Ascertaining Oncologic Outcomes From Radiology Reports.  _JAMA Oncology_ , 5(10):1421–1429, October 2019\.  ISSN 2374-2437.  doi: 10.1001/jamaoncol.2019.1800.  URL https://doi.org/10.1001/jamaoncol.2019.1800.
  * Kim et al. [2019] Chulho Kim, Vivienne Zhu, Jihad Obeid, and Leslie Lenert.  Natural language processing and machine learning algorithm to identify brain MRI reports with acute ischemic stroke.  _PLOS ONE_ , 14(2):e0212778, February 2019.  ISSN 1932-6203.  doi: 10.1371/journal.pone.0212778.  URL https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0212778.
  * Kreimeyer et al. [2017] Kory Kreimeyer, Matthew Foster, Abhishek Pandey, Nina Arya, Gwendolyn Halford, Sandra F. Jones, Richard Forshee, Mark Walderhaug, and Taxiarchis Botsis.  Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic review.  _Journal of Biomedical Informatics_ , 73:14–29, 2017.  ISSN 1532-0480.  doi: 10.1016/j.jbi.2017.07.012.
  * Kwan et al. [2019] Janice L. Kwan, Darya Yermak, Lezlie Markell, Narinder S. Paul, Kaveh J. Shojania, and Peter Cram.  Follow Up of Incidental High-Risk Pulmonary Nodules on Computed Tomography Pulmonary Angiography at Care Transitions.  _Journal of Hospital Medicine_ , 14(6):349–352, June 2019.  doi: 10.12788/jhm.3128.  URL https://europepmc.org/article/med/30794133.
  * Kłos et al. [2018] Monika Kłos, Jarosław Żyłkowski, and Dominik Spinczyk.  Automatic Classification of Text Documents Presenting Radiology Examinations.  In Ewa Pietka, Pawel Badura, Jacek Kawa, and Wojciech Wieclawek, editors, _Proceedings 6th International Conference Information Technology in Biomedicine(ITIB’2018)_ , Advances in Intelligent Systems and Computing, pages 495–505. Springer International Publishing, 2018\.  ISBN 978-3-319-91211-0.  doi: 10.1007/978-3-319-91211-0-43.
  * Lacson et al. [2015] Ronilda Lacson, Kimberly Harris, Phyllis Brawarsky, Tor D. Tosteson, Tracy Onega, Anna N. A. Tosteson, Abby Kaye, Irina Gonzalez, Robyn Birdwell, and Jennifer S. Haas.  Evaluation of an Automated Information Extraction Tool for Imaging Data Elements to Populate a Breast Cancer Screening Registry.  _Journal of Digital Imaging_ , 28(5):567–575, October 2015.  ISSN 1618-727X.  doi: 10.1007/s10278-014-9762-4.  URL https://doi.org/10.1007/s10278-014-9762-4.
  * Lacson et al. [2017] Ronilda Lacson, Martha E. Goodrich, Kimberly Harris, Phyllis Brawarsky, and Jennifer S. Haas.  Assessing Inaccuracies in Automated Information Extraction of Breast Imaging Findings.  _Journal of Digital Imaging_ , 30(2):228–233, April 2017.  ISSN 1618-727X.  doi: 10.1007/s10278-016-9927-4.  URL https://doi.org/10.1007/s10278-016-9927-4.
  * Lafourcade and Ramadier [2017] M. Lafourcade and Lionel Ramadier.  Radiological text simplification using a general knowledge base.  In _18th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2017)_ , CICLing 2017. Budapest, Hungary, 2017.  doi: https://doi.org/10.1007/978-3-319-77116-8“˙46.
  * Lafourcade and Ramadier [2016] Mathieu Lafourcade and Lionel Ramadier.  Semantic RelationExtraction with Semantic Patterns: Experiment on Radiology Report.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_ , LREC 2016 Proceedings, Portorož, Slovenia, 2016. European Language Resources Association (ELRA).  ISBN 978-2-9517408-9-1.  URL https://hal.archives-ouvertes.fr/hal-01382320.
  * Landis and Koch [1977] J. Richard Landis and Gary G. Koch.  The Measurement of Observer Agreement for Categorical Data.  _Biometrics_ , 33(1):159–174, 1977.  ISSN 0006-341X.  doi: 10.2307/2529310.  URL https://www.jstor.org/stable/2529310.
  * Li and Elliot [2019] Andrew Yu Li and Nikki Elliot.  Natural language processing to identify ureteric stones in radiology reports.  _Journal of Medical Imaging and Radiation Oncology_ , 63(3):307–310, 2019.  ISSN 1754-9485.  doi: 10.1111/1754-9485.12861.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/1754-9485.12861.
  * Liu et al. [2019] Yi Liu, Li-Na Zhu, Qing Liu, Chao Han, Xiao-Dong Zhang, and Xiao-Ying Wang.  Automatic extraction of imaging observation and assessment categories from breast magnetic resonance imaging reports with natural language processing.  _Chinese Medical Journal_ , 132(14):1673–1680, July 2019.  ISSN 0366-6999.  doi: 10.1097/CM9.0000000000000301.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6759110/.
  * Mabotuwana et al. [2018a] Thusitha Mabotuwana, Christopher S Hall, Joel Tieder, and Martin L. Gunn.  Improving Quality of Follow-Up Imaging Recommendations in Radiology.  _AMIA Annual Symposium Proceedings_ , 2017:1196–1204, April 2018a.  ISSN 1942-597X.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5977608/.
  * Mabotuwana et al. [2018b] Thusitha Mabotuwana, Vadiraj Hombal, Sandeep Dalal, Christopher S. Hall, and Martin Gunn.  Determining Adherence to Follow-up Imaging Recommendations.  _Journal of the American College of Radiology_ , 15(3, Part A):422–428, March 2018b.  ISSN 1546-1440.  doi: 10.1016/j.jacr.2017.11.022.  URL http://www.sciencedirect.com/science/article/pii/S1546144017314758.
  * Mahan et al. [2019] Margaret Mahan, Daniel Rafter, Hannah Casey, Marta Engelking, Tessneem Abdallah, Charles Truwit, Mark Oswood, and Uzma Samadani.  tbiExtractor: A framework for extracting traumatic brain injury common data elements from radiology reports.  _bioRxiv 585331_ , 2019.  doi: 10.1101/585331.  URL https://www.biorxiv.org/content/10.1101/585331v1.
  * Maros et al. [2018] Máté E. Maros, Ralf Wenz, Alex Förster, Matthias F. Froelich, Christoph Groden, Wieland H. Sommer, Stefan O. Schönberg, Thomas Henzler, and Holger Wenz.  Objective Comparison Using Guideline-based Query of Conventional Radiological Reports and Structured Reports.  _In Vivo_ , 32(4):843–849, January 2018.  ISSN 0258-851X, 1791-7549.  doi: 10.21873/invivo.11318.  URL http://iv.iiarjournals.org/content/32/4/843.
  * Masino et al. [2016] Aaron J. Masino, Robert W. Grundmeier, Jeffrey W. Pennington, John A. Germiller, and E. Bryan Crenshaw.  Temporal bone radiology report classification using open source machine learning and natural langue processing libraries.  _BMC Medical Informatics and Decision Making_ , 16(1):65, June 2016.  ISSN 1472-6947.  doi: 10.1186/s12911-016-0306-3.  URL https://doi.org/10.1186/s12911-016-0306-3.
  * Meystre et al. [2017] Stephane Meystre, Ramkiran Gouripeddi, Joel Tieder, Jeffrey Simmons, Rajendu Srivastava, and Samir Shah.  Enhancing Comparative Effectiveness Research With Automated Pediatric Pneumonia Detection in a Multi-Institutional Clinical Repository: A PHIS+ Pilot Study.  _Journal of Medical Internet Research_ , 19(5):e162, 2017.  doi: 10.2196/jmir.6887.  URL https://www.jmir.org/2017/5/e162/.
  * Miao et al. [2018] Shumei Miao, Tingyu Xu, Yonghui Wu, Hui Xie, Jingqi Wang, Shenqi Jing, Yaoyun Zhang, Xiaoliang Zhang, Yinshuang Yang, Xin Zhang, Tao Shan, Li Wang, Hua Xu, Shui Wang, and Yun Liu.  Extraction of BI-RADS findings from breast ultrasound reports in Chinese using deep learning approaches.  _International Journal of Medical Informatics_ , 119:17–21, November 2018.  ISSN 1386-5056.  doi: 10.1016/j.ijmedinf.2018.08.009.  URL http://www.sciencedirect.com/science/article/pii/S1386505618309225.
  * Mikolov et al. [2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  _Efficient Estimation of Word Representations in Vector Space_.  2013\.  URL http://arxiv.org/abs/1301.3781.
  * Mikolov et al. [2018] Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin.  Advances in Pre-Training Distributed Word Representations.  In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_ , 2018.
  * Minn et al. [2015] Matthew J. Minn, Arash R. Zandieh, and Ross W. Filice.  Improving Radiology Report Quality by Rapidly Notifying Radiologist of Report Errors.  _Journal of Digital Imaging_ , 28(4):492–498, August 2015.  ISSN 1618-727X.  doi: 10.1007/s10278-015-9781-9.  URL https://doi.org/10.1007/s10278-015-9781-9.
  * Moher et al. [2015] David Moher, Larissa Shamseer, Mike Clarke, Davina Ghersi, Alessandro Liberati, Mark Petticrew, Paul Shekelle, and Lesley A Stewart.  Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.  _Systematic Reviews_ , 4(1):1, December 2015.  ISSN 2046-4053.  doi: 10.1186/2046-4053-4-1.  URL https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/2046-4053-4-1.
  * Mujjiga et al. [2019] Srikanth Mujjiga, Vamsi Krishna, Kalyan Chakravarthi, and Vijayananda J.  Identifying Semantics in Clinical Reports Using Neural Machine Translation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):9552–9557, July 2019.  ISSN 2374-3468.  doi: 10.1609/aaai.v33i01.33019552.  URL https://www.aaai.org/ojs/index.php/AAAI/article/view/5015.
  * National Library of Medicine [2021a] National Library of Medicine.  _SNOMED CT_.  2021a.  URL @miscumls,author=NationalLibraryofMedicine,title=SNOMEDCT,url=https://www.nlm.nih.gov/healthit/snomedct/index.html,year=2021,urldate=Accessed07Feb2021.
  * National Library of Medicine [2021b] National Library of Medicine.  _Unified Medical Language System_.  2021b.  URL https://www.nlm.nih.gov/research/umls/index.html.
  * Noorbakhsh-Sabet et al. [2018] Nariman Noorbakhsh-Sabet, Georgios Tsivgoulis, Shima Shahjouei, Yirui Hu, Nitin Goyal, Andrei V. Alexandrov, and Ramin Zand.  Racial Difference in Cerebral Microbleed Burden Among a Patient Population in the Mid-South United States.  _Journal of Stroke and Cerebrovascular Diseases_ , 27(10):2657–2661, October 2018.  ISSN 1052-3057.  doi: 10.1016/j.jstrokecerebrovasdis.2018.05.031.  URL http://www.sciencedirect.com/science/article/pii/S1052305718302714.
  * Oberkampf et al. [2016] Heiner Oberkampf, Sonja Zillner, James A. Overton, Bernhard Bauer, Alexander Cavallaro, Michael Uder, and Matthias Hammon.  Semantic representation of reported measurements in radiology.  _BMC Medical Informatics and Decision Making_ , 16(1):5, January 2016.  ISSN 1472-6947.  doi: 10.1186/s12911-016-0248-9.  URL https://doi.org/10.1186/s12911-016-0248-9.
  * Ong et al. [2020] Charlene Jennifer Ong, Agni Orfanoudaki, Rebecca Zhang, Francois Pierre M. Caprasse, Meghan Hutch, Liang Ma, Darian Fard, Oluwafemi Balogun, Matthew I. Miller, Margaret Minnig, Hanife Saglam, Brenton Prescott, David M. Greer, Stelios Smirnakis, and Dimitris Bertsimas.  Machine learning and natural language processing methods to identify ischemic stroke, acuity and location from radiology reports.  _PLOS ONE_ , 15(6):e0234908, June 2020.  ISSN 1932-6203.  doi: 10.1371/journal.pone.0234908.  URL https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0234908.
  * Patel et al. [2017] Tejal A. Patel, Mamta Puppala, Richard O. Ogunti, Joe E. Ensor, Tiancheng He, Jitesh B. Shewale, Donna P. Ankerst, Virginia G. Kaklamani, Angel A. Rodriguez, Stephen T. C. Wong, and Jenny C. Chang.  Correlating mammographic and pathologic findings in clinical decision support using natural language processing and data mining methods.  _Cancer_ , 123(1):114–121, January 2017.  ISSN 1097-0142.  doi: 10.1002/cncr.30245.
  * Peng et al. [2019] Y. Peng, K. Yan, V. Sandfort, R. M. Summers, and Z. Lu.  A self-attention based deep learning method for lesion attribute detection from CT reports.  In _2019 IEEE International Conference on Healthcare Informatics (ICHI)_ , pages 1–5, Xi’an, China, June 2019. IEEE Computer Society.  doi: 10.1109/ICHI.2019.8904668.
  * Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543, 2014.
  * Percha et al. [2018] Bethany Percha, Yuhao Zhang, Selen Bozkurt, Daniel Rubin, Russ B. Altman, and Curtis P. Langlotz.  Expanding a radiology lexicon using contextual patterns in radiology reports.  _Journal of the American Medical Informatics Association_ , 25(6):679–685, June 2018.  doi: 10.1093/jamia/ocx152.  URL https://academic.oup.com/jamia/article/25/6/679/4797401.
  * Peters et al. [2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  _CoRR_ , abs/1802.05365, 2018.  URL http://arxiv.org/abs/1802.05365.  _eprint: 1802.05365.
  * Pons et al. [2016] Ewoud Pons, Loes M. M. Braun, M. G. Myriam Hunink, and Jan A. Kors.  Natural Language Processing in Radiology: A Systematic Review.  _Radiology_ , 279(2):329–343, April 2016.  ISSN 0033-8419.  doi: 10.1148/radiol.16142770.  URL https://pubs.rsna.org/doi/10.1148/radiol.16142770.
  * Pruitt et al. [2019] Peter Pruitt, Andrew Naidech, Jonathan Van Ornam, Pierre Borczuk, and William Thompson.  A natural language processing algorithm to extract characteristics of subdural hematoma from head CT reports.  _Emergency Radiology_ , 26(3):301–306, June 2019\.  ISSN 1438-1435.  doi: 10.1007/s10140-019-01673-4.  URL https://doi.org/10.1007/s10140-019-01673-4.
  * Qenam et al. [2017] Basel Qenam, Tae Youn Kim, Mark J. Carroll, and Michael Hogarth.  Text Simplification Using Consumer Health Vocabulary to Generate Patient-Centered Radiology Reporting: Translation and Evaluation.  _Journal of Medical Internet Research_ , 19(12):e417, 2017.  doi: 10.2196/jmir.8536.  URL https://www.jmir.org/2017/12/e417/.
  * Ratner et al. [2018] Alex Ratner, Braden Hancock, Jared Dunnmon, Roger Goldman, and Christopher Ré.  Snorkel MeTaL: Weak Supervision for Multi-Task Learning.  In _Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning_ , volume 3 of _DEEM’18_ , pages 1–4, Houston, TX, USA, 2018. ACM.  ISBN 978-1-4503-5828-6.  doi: 10.1145/3209889.3209898.  URL https://doi.org/10.1145/3209889.3209898.
  * Redman et al. [2017] Joseph S. Redman, Yamini Natarajan, Jason K. Hou, Jingqi Wang, Muzammil Hanif, Hua Feng, Jennifer R. Kramer, Roxanne Desiderio, Hua Xu, Hashem B. El-Serag, and Fasiha Kanwal.  Accurate Identification of Fatty Liver Disease in Data Warehouse Utilizing Natural Language Processing.  _Digestive Diseases and Sciences_ , 62(10):2713–2718, October 2017.  ISSN 1573-2568.  doi: 10.1007/s10620-017-4721-9.  URL https://doi.org/10.1007/s10620-017-4721-9.
  * RSNA [2021] RSNA.  _RadLex_.  2021\.  URL http://radlex.org/.
  * Sada et al. [2016] Yvonne Sada, Jason Hou, Peter Richardson, Hashem El-Serag, and Jessica Davila.  Validation of Case Finding Algorithms for Hepatocellular Cancer from Administrative Data and Electronic Health Records using Natural Language Processing.  _Medical care_ , 54(2):e9–e14, February 2016\.  ISSN 0025-7079.  doi: 10.1097/MLR.0b013e3182a30373.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3875602/.
  * Sevenster et al. [2015a] M. Sevenster, J. Buurman, P. Liu, J. F. Peters, and P. J. Chang.  Natural Language Processing Techniques for Extracting and Categorizing Finding Measurements in Narrative Radiology Reports.  _Applied Clinical Informatics_ , 06(3):600–610, 2015a.  ISSN 1869-0327.  doi: 10.4338/ACI-2014-11-RA-0110.  URL http://www.thieme-connect.de/DOI/DOI?10.4338/ACI-2014-11-RA-0110.
  * Sevenster et al. [2015b] Merlijn Sevenster, Jeffrey Bozeman, Andrea Cowhy, and William Trost.  A natural language processing pipeline for pairing measurements uniquely across free-text CT reports.  _Journal of Biomedical Informatics_ , 53:36–48, February 2015b.  ISSN 1532-0464.  doi: 10.1016/j.jbi.2014.08.015.  URL http://www.sciencedirect.com/science/article/pii/S1532046414001968.
  * Shelmerdine et al. [2019] S. C. Shelmerdine, M. Singh, W. Norman, R. Jones, N. J. Sebire, and O. J. Arthurs.  Automated data extraction and report analysis in computer-aided radiology audit: practice implications from post-mortem paediatric imaging.  _Clinical Radiology_ , 74(9):733.e11–733.e18, September 2019.  ISSN 0009-9260.  doi: 10.1016/j.crad.2019.04.021.  URL http://www.sciencedirect.com/science/article/pii/S0009926019302181.
  * Shickel et al. [2018] B. Shickel, P. J. Tighe, A. Bihorac, and P. Rashidi.  Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis.  _IEEE Journal of Biomedical and Health Informatics_ , 22(5):1589–1604, September 2018.  ISSN 2168-2208.  doi: 10.1109/JBHI.2017.2767063.
  * Shin et al. [2017] B. Shin, F. H. Chokshi, T. Lee, and J. D. Choi.  Classification of radiology reports using neural attention models.  In _2017 International Joint Conference on Neural Networks (IJCNN)_ , pages 4363–4370, Anchorage, AK, May 2017. IEEE.  doi: 10.1109/IJCNN.2017.7966408.
  * Short et al. [2019] Ryan G. Short, John Bralich, Dave Bogaty, and Nicholas T. Befera.  Comprehensive Word-Level Classification of Screening Mammography Reports Using a Neural Network Sequence Labeling Approach.  _Journal of Digital Imaging_ , 32(5):685–692, October 2019.  ISSN 1618-727X.  doi: 10.1007/s10278-018-0141-4.  URL https://doi.org/10.1007/s10278-018-0141-4.
  * Smit et al. [2020a] Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Ng, and Matthew Lungren.  Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1500–1519, Online, November 2020a. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.117.  URL https://www.aclweb.org/anthology/2020.emnlp-main.117.
  * Smit et al. [2020b] Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y. Ng, and Matthew P. Lungren.  CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT.  _CoRR_ , abs/2004.09167, 2020b.  URL https://arxiv.org/abs/2004.09167.  _eprint: 2004.09167.
  * Sorin et al. [2020] V. Sorin, Y. Barash, E. Konen, and E. Klang.  Deep Learning for Natural Language Processing in Radiology-Fundamentals and a Systematic Review.  _Journal of the American College of Radiology : JACR_ , 17(5):639–648, 2020.  doi: 10.1016/j.jacr.2019.12.026.
  * Spasic and Nenadic [2020] Irena Spasic and Goran Nenadic.  Clinical Text Data in Machine Learning: Systematic Review.  _JMIR medical informatics_ , 8(3):e17984, March 2020.  ISSN 2291-9694.  doi: 10.2196/17984.
  * Steinkamp et al. [2019] Jackson M. Steinkamp, Charles Chambers, Darco Lalevic, Hanna M. Zafar, and Tessa S. Cook.  Toward Complete Structured Information Extraction from Radiology Reports Using Machine Learning.  _Journal of Digital Imaging_ , 32(4):554–564, August 2019.  ISSN 1618-727X.  doi: 10.1007/s10278-019-00234-y.  URL https://doi.org/10.1007/s10278-019-00234-y.
  * Tahmasebi et al. [2019] Amir M. Tahmasebi, Henghui Zhu, Gabriel Mankovich, Peter Prinsen, Prescott Klassen, Sam Pilato, Rob van Ommering, Pritesh Patel, Martin L. Gunn, and Paul Chang.  Automatic Normalization of Anatomical Phrases in Radiology Reports Using Unsupervised Learning.  _Journal of Digital Imaging_ , 32(1):6–18, February 2019.  ISSN 1618-727X.  doi: 10.1007/s10278-018-0116-5.  URL https://doi.org/10.1007/s10278-018-0116-5.
  * Tan and Heagerty [2019] W. Katherine Tan and Patrick J. Heagerty.  Surrogate-guided sampling designs for classification of rare outcomes from electronic medical records data.  _arXiv:1904.00412 [stat.ME]_ , March 2019.  URL http://arxiv.org/abs/1904.00412.
  * Tan et al. [2018] W. Katherine Tan, Saeed Hassanpour, Patrick J. Heagerty, Sean D. Rundell, Pradeep Suri, Hannu T. Huhdanpaa, Kathryn James, David S. Carrell, Curtis P. Langlotz, Nancy L. Organ, Eric N. Meier, Karen J. Sherman, David F. Kallmes, Patrick H. Luetmer, Brent Griffith, David R. Nerenz, and Jeffrey G. Jarvik.  Comparison of Natural Language Processing Rules-based and Machine-learning Systems to Identify Lumbar Spine Imaging Findings Related to Low Back Pain.  _Academic Radiology_ , 25(11):1422–1432, November 2018.  ISSN 1076-6332.  doi: 10.1016/j.acra.2018.03.008.  URL http://www.sciencedirect.com/science/article/pii/S1076633218301211.
  * Trivedi et al. [2019] Gaurav Trivedi, Charmgil Hong, Esmaeel R. Dadashzadeh, Robert M. Handzel, Harry Hochheiser, and Shyam Visweswaran.  Identifying incidental findings from radiology reports of trauma patients: An evaluation of automated feature representation methods.  _International Journal of Medical Informatics_ , 129:81–87, September 2019.  ISSN 1386-5056.  doi: 10.1016/j.ijmedinf.2019.05.021.  URL http://www.sciencedirect.com/science/article/pii/S138650561831061X.
  * Trivedi et al. [2018] Hari Trivedi, Joseph Mesterhazy, Benjamin Laguna, Thienkhai Vu, and Jae Ho Sohn.  Automatic Determination of the Need for Intravenous Contrast in Musculoskeletal MRI Examinations Using IBM Watson’s Natural Language Processing Algorithm.  _Journal of Digital Imaging_ , 31(2):245–251, April 2018.  ISSN 1618-727X.  doi: 10.1007/s10278-017-0021-3.  URL https://doi.org/10.1007/s10278-017-0021-3.
  * Van Haren et al. [2019] Robert M. Van Haren, Arlene M. Correa, Boris Sepesi, David C. Rice, Wayne L. Hofstetter, Reza J. Mehran, Ara A. Vaporciyan, Garrett L. Walsh, Jack A. Roth, Stephen G. Swisher, and Mara B. Antonoff.  Ground Glass Lesions on Chest Imaging: Evaluation of Reported Incidence in Cancer Patients Using Natural Language Processing.  _The Annals of Thoracic Surgery_ , 107(3):936–940, March 2019.  ISSN 0003-4975.  doi: 10.1016/j.athoracsur.2018.09.016.  URL http://www.sciencedirect.com/science/article/pii/S0003497518315315.
  * Wheater et al. [2019] Emily Wheater, Grant Mair, Cathie Sudlow, Beatrice Alex, Claire Grover, and William Whiteley.  A validated natural language processing algorithm for brain imaging phenotypes from radiology reports in UK electronic health records.  _BMC Medical Informatics and Decision Making_ , 19(1):184, September 2019.  ISSN 1472-6947.  doi: 10.1186/s12911-019-0908-7.  URL https://doi.org/10.1186/s12911-019-0908-7.
  * Wohlin [2014] Claes Wohlin.  Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering.  In _Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering_ , EASE ’14, New York, NY, USA, 2014. Association for Computing Machinery.  ISBN 978-1-4503-2476-2.  doi: 10.1145/2601248.2601268.  URL https://doi.org/10.1145/2601248.2601268.  event-place: London, England, United Kingdom.
  * Wood et al. [2020] David A. Wood, Jeremy Lynch, Sina Kafiabadi, Emily Guilhem, Aisha Al Busaidi, Antanas Montvila, Thomas Varsavsky, Juveria Siddiqui, Naveen Gadapa, Matthew Townend, Martin Kiik, Keena Patel, Gareth Barker, Sebastian Ourselin, James H. Cole, and Thomas C. Booth.  Automated Labelling using an Attention model for Radiology reports of MRI scans (ALARM).  _arXiv:2002.06588 [cs.CV]_ , 2020.  URL http://arxiv.org/abs/2002.06588.
  * Wu et al. [2020] Stephen Wu, Kirk Roberts, Surabhi Datta, Jingcheng Du, Zongcheng Ji, Yuqi Si, Sarvesh Soni, Qiong Wang, Qiang Wei, Yang Xiang, Bo Zhao, and Hua Xu.  Deep learning in clinical natural language processing: a methodical review.  _Journal of the American Medical Informatics Association: JAMIA_ , 27(3):457–470, 2020.  ISSN 1527-974X.  doi: 10.1093/jamia/ocz200.
  * Xie et al. [2019] Zhe Xie, Yuanyuan Yang, Mingqing Wang, Ming Li, Haozhe Huang, Dezhong Zheng, Rong Shu, and Tonghui Ling.  Introducing Information Extraction to Radiology Information Systems to Improve the Efficiency on Reading Reports.  _Methods of Information in Medicine_ , 58(2-03):94–106, 2019.  ISSN 2511-705X.  doi: 10.1055/s-0039-1694992.
  * Yadav et al. [2016] Kabir Yadav, Efsun Sarioglu, Hyeong-Ah Choi, Walter B. Cartwright, Pamela S. Hinds, and James M. Chamberlain.  Automated Outcome Classification of Computed Tomography Imaging Reports for Pediatric Traumatic Brain Injury.  _Academic Emergency Medicine_ , 23(2):171–178, 2016.  ISSN 1553-2712.  doi: 10.1111/acem.12859.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/acem.12859.
  * Yan et al. [2016] Zihao Yan, Ivan K. Ip, Ali S. Raja, Anurag Gupta, Joshua M. Kosowsky, and Ramin Khorasani.  Yield of CT Pulmonary Angiography in the Emergency Department When Providers Override Evidence-based Clinical Decision Support.  _Radiology_ , 282(3):717–725, September 2016\.  ISSN 0033-8419.  doi: 10.1148/radiol.2016151985.  URL https://pubs.rsna.org/doi/full/10.1148/radiol.2016151985.
  * Yang et al. [2018] Hongmei Yang, Lin Li, Ridong Yang, and Yi Zhou.  Towards Automated Knowledge Discovery of Hepatocellular Carcinoma: Extract Patient Information from Chinese Clinical Reports.  In _Proceedings of the 2nd International Conference on Medical and Health Informatics_ , ICMHI ’18, pages 111–116, New York, NY, USA, June 2018. ACM.  ISBN 978-1-4503-6389-1.  doi: 10.1145/3239438.3239445.  URL https://doi.org/10.1145/3239438.3239445.
  * Yasaka and Abe [2018] Koichiro Yasaka and Osamu Abe.  Deep learning and artificial intelligence in radiology: Current applications and future directions.  _PLOS Medicine_ , 15(11):e1002707, November 2018\.  ISSN 1549-1676.  doi: 10.1371/journal.pmed.1002707.  URL https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002707.
  * Yim et al. [2016a] Wen-wai Yim, Tyler Denman, Sharon W. Kwan, and Meliha Yetisgen.  Tumor information extraction in radiology reports for hepatocellular carcinoma patients.  _AMIA Summits on Translational Science Proceedings_ , 2016:455–464, July 2016a.  ISSN 2153-4063.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5001784/.
  * Yim et al. [2016b] Wen-wai Yim, Sharon W. Kwan, and Meliha Yetisgen.  Tumor reference resolution and characteristic extraction in radiology reports for liver cancer stage prediction.  _Journal of Biomedical Informatics_ , 64:179–191, December 2016b.  ISSN 1532-0464.  doi: 10.1016/j.jbi.2016.10.005.  URL http://www.sciencedirect.com/science/article/pii/S1532046416301381.
  * Yim et al. [2017] Wen-wai Yim, Sharon W. Kwan, and Meliha Yetisgen.  Classifying tumor event attributes in radiology reports.  _Journal of the Association for Information Science and Technology_ , 68(11):2662–2674, 2017.  ISSN 2330-1643.  doi: 10.1002/asi.23937.  URL https://asistdl.onlinelibrary.wiley.com/doi/abs/10.1002/asi.23937.
  * Yim et al. [2018] Wen-wai Yim, Sharon W Kwan, Guy Johnson, and Meliha Yetisgen.  Classification of hepatocellular carcinoma stages from free-text clinical and radiology reports.  _AMIA Annual Symposium Proceedings_ , 2017:1858–1867, April 2018.  ISSN 1942-597X.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5977638/.
  * Young et al. [2018] T. Young, D. Hazarika, S. Poria, and E. Cambria.  Recent Trends in Deep Learning Based Natural Language Processing [Review Article].  _IEEE Computational Intelligence Magazine_ , 13(3):55–75, August 2018.  ISSN 1556-6048.  doi: 10.1109/MCI.2018.2840738.
  * Zech et al. [2018] John Zech, Margaret Pain, Joseph Titano, Marcus Badgeley, Javin Schefflein, Andres Su, Anthony Costa, Joshua Bederson, Joseph Lehar, and Eric Karl Oermann.  Natural Language–based Machine Learning Models for the Annotation of Clinical Radiology Reports.  _Radiology_ , 287(2):570–580, January 2018.  ISSN 0033-8419.  doi: 10.1148/radiol.2018171093.  URL https://pubs.rsna.org/doi/full/10.1148/radiol.2018171093.
  * Zech et al. [2019] John Zech, Jessica Forde, Joseph J. Titano, Deepak Kaji, Anthony Costa, and Eric Karl Oermann.  Detecting insertion, substitution, and deletion errors in radiology reports using neural sequence-to-sequence models.  _Annals of Translational Medicine_ , 7(11), June 2019.  ISSN 2305-5839.  doi: 10.21037/atm.2018.08.11.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6603352/.
  * Zhang et al. [2018] A. Y. Zhang, S. S. W. Lam, N. Liu, Y. Pang, L. L. Chan, and P. H. Tang.  Development of a Radiology Decision Support System for the Classification of MRI Brain Scans.  In _2018 IEEE/ACM 5th International Conference on Big Data Computing Applications and Technologies (BDCAT)_ , pages 107–115, December 2018.  doi: 10.1109/BDCAT.2018.00021.
  * Zhang et al. [2019] Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D. Manning, and Curtis P. Langlotz.  Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports.  _arXiv:1911.02541 [cs.CL]_ , 2019.  URL http://arxiv.org/abs/1911.02541.
  * Zhao et al. [2018] Yiqing Zhao, Nooshin J. Fesharaki, Hongfang Liu, and Jake Luo.  Using data-driven sublanguage pattern mining to induce knowledge models: application in medical image reports knowledge representation.  _BMC Medical Informatics and Decision Making_ , 18(1):61, July 2018.  ISSN 1472-6947.  doi: 10.1186/s12911-018-0645-3.  URL https://doi.org/10.1186/s12911-018-0645-3.
  * Zhu et al. [2019] Henghui Zhu, Ioannis Ch. Paschalidis, Christopher Hall, and Amir Tahmasebi.  Context-Driven Concept Annotation in Radiology Reports: Anatomical Phrase Labeling.  _AMIA Summits on Translational Science Proceedings_ , 2019:232–241, May 2019.  ISSN 2153-4063.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6568085/.