# MedKLIP: Medical Knowledge Enhanced Language-Image
Pre-Training in Radiology

Chaoyi Wu1,2, Xiaoman Zhang1,2, Ya Zhang1,2, Yanfeng Wang1,2, Weidi Xie1,2
1Cooperative Medianet Innovation Center, Shanghai Jiao Tong University
2Shanghai AI Laboratory
{wtzxxxwcy02, xm99sjtu, ya_zhang, wangyanfeng<EMAIL_ADDRESS>
https://chaoyi-wu.github.io/MedKLIP/

###### Abstract

In this paper, we consider enhancing medical visual-language pre-training
(VLP) with domain-specific knowledge, by exploiting the paired image-text
reports from the radiological daily practice. In particular, we make the
following contributions: First, unlike existing works that directly process
the raw reports, we adopt a novel triplet extraction module to extract the
medical-related information, avoiding unnecessary complexity from language
grammar and enhancing the supervision signals; Second, we propose a novel
triplet encoding module with entity translation by querying a knowledge base,
to exploit the rich domain knowledge in medical field, and implicitly build
relationships between medical entities in the language embedding space; Third,
we propose to use a Transformer-based fusion model for spatially aligning the
entity description with visual signals at the image patch level, enabling the
ability for medical diagnosis; Fourth, we conduct thorough experiments to
validate the effectiveness of our architecture, and benchmark on numerous
public benchmarks e.g., ChestX-ray14, RSNA Pneumonia, SIIM-ACR Pneumothorax,
COVIDx CXR-2, COVID Rural, and EdemaSeverity. In both zero-shot and fine-
tuning settings, our model has demonstrated strong performance compared with
the former methods on disease classification and grounding.

##  1 Introduction

Figure 1: Our method mainly considers combining medical knowledge with VLP. We
propose Triplet Extraction and Entity Translation modules, so that the network
can be supervised with detailed entity-level signals.

With the rapid development of deep learning, numerous works have been proposed
to facilitate computer-aided diagnosis in the medical field [44, 20, 53, 19].
Despite the tremendous progress, these models are normally trained to
recognize or segment the structures that fall into a certain closed set of
anatomical or disease categories, whenever a new disease comes to be of
interest, a costly procedure for data annotation, model re-training are
required, fundamentally limiting its practical values. As an alternative,
recent research considers to train the model on the corpus, consisting of
large amount of multi-modal data, that is generated from daily clinical
routine, for instance, the most common example is the dataset of X-ray images
with paired radiological reports [18, 28, 31].

This paper presents our preliminary investigation on vision-language
representation learning in the medical domain, with the goal of better zero-
shot disease diagnosis (classification) and grounding. Undoubtedly, these
tasks have also been widely investigated in the computer vision community,
with significant progress made on developing Foundational Models in the past
years, for example, CLIP [48], ALBEF [33], BLIP[32], etc. However, to achieve
such a goal in the medical domain, different challenges must be resolved, that
requires research efforts from the community: First, data availability,
training Foundation Models in computer vision normally require over millions
of image-text pairs, while in the medical domain, only a few hundred thousand
pairs are available [31]. The limited data challenges language models to
understand the reports in free form [6]. Second, the problem considered in
computer-aided diagnosis is naturally fine-grained, that requires
distinguishing the medical concepts to understand the disease, as a
consequence, domain knowledge is essential; Third, robustness is crucial, it
is, therefore, preferable to have explainability, where diagnosis results come
along with the visual grounding, to help radiologists understand the system,
and build trust between human and machines.

Existing work in medical VLP (Vision-Language Pre-training) [66, 45, 25, 6]
follows a straightforward training paradigm by matching raw reports with image
scans, as shown in Fig.1A, ignoring the medical prior knowledge, and, thus, we
propose a novel knowledge-enhanced visual-language model as shown in Fig. 1B.
First, we propose a triplet extraction module to extract useful medical
entities (keywords) from raw reports, and simplify each report into sets of
triplets, denoted as $\\{{\rm entity,position,exist}\\}$. Decomposing reports
into triplets leads to an effective representation of the reports with minimal
information loss due to the structural prior in reports; Second, we translate
the medical entities into fine-grained descriptions by leveraging a well-
defined medical word knowledge base, that tends to explain diseases with
common vocabulary. Thus, computing text embeddings for these descriptions
enables to implicitly establish relationships between medical entities; Third,
we view the entities as a query set and adopt a transformer-based architecture
for aligning the image patches with entity descriptions, that enables explicit
supervision signals at entity level. Consequently, we can simultaneously infer
the likelihood of certain diseases with the visual evidence in the form of a
spatial heatmap, i.e., providing rough grounding for explainability.

We pre-train the model on one widely-used medical image-report dataset MIMIC-
CXR [31], and rigorously evaluate on the task of disease diagnosis across
numerous public benchmarks, e.g., ChestX-ray14 [56], RSNA Pneumonia [49],
SIIM-ACR Pneumothorax [1], COVIDx CXR-2 [46], COVID Rural [52, 15], and
EdemaSeverity [7]. We get state-of-the-art performance on zero-shot
classification and grounding on different diseases, spanning different image
distributions, with further fine-tuning, our model still exceeds previous
models significantly.

##  2 Related Work

General Vision-Language Pre-training (VLP) Models. In computer vision, such
line of research has gained tremendous success in the recent literature,
generally speaking, the developed architectures can either be two-stream [4,
33, 30], i.e., dual encoders, or those based on single-stream methods [35, 9],
that favors visual-language fusion. In particular, several works [13, 37, 64]
consider considers to combine the commonsense knowledge into the vision-
language pre-training, however, in this paper, we focus on medical domain,
which is clearly more fine-grained and requires significantly more expertise.

Medical Named-Entity-Recognition (NER) Models. Various natural language
processing (NLP) approaches have been proposed to extract information from
radiology reports [47, 28, 42, 50]. These early methods considered only the
disease, thus causing information loss. Further state-of-the-art works [29,
59] are proposed to extract relationship between different entities without
demand of pre-defined close disease set, retaining most of useful information
with high accuracy. In weakly supervision [65] and report generation fields
[14], NER methods have shown great impact, and greatly inspired us for more
effective vision-language pre-training with medical domain knowledge injected.

Medical Knowledge Enhanced Models. Leveraging external medical knowledge to
enhance deep learning models is not a new topic [60]. Depending the approaches
of using medical knowledge, They can be classified into model-based or input-
based. In model-based approaches, the authors aim to imitate the radiological
or diagnosis practice to design models [34, 23, 55, 26, 43, 21, 12]. While in
input-based approaches, the knowledge is treated as an extra input for
computing features [63, 61, 51, 11] or to guide the final training loss [8,
27, 36, 38, 41], commonly used in report generation tasks [57, 62, 39, 11].
However, none of these works are targeting on vision-language pre-training in
medical domain with image-report.

Concurrent Works in Medical VLP. Existing medical VLP methods follow the two-
stream flow [66, 25, 7, 45, 58, 10], i.e., use contrastive learning and
without fusion module, for example, ConVIRT [66] initially proposed to use
contrastive loss as a proxy task for aligning the medical scan and
corresponding reports, LoVT and GLoRIA then focus on improving the local
alignment performance [25, 7]. BioViL notices the language pattern in reports
is different from natural texts and re-designs the language model [6]. Align
[10] improves the performance of VLP model on medical Visual Question
Answering (VQA). The recent arxiv preprint, MedCLIP [58], considers leveraging
unpaired data to make up data scarcity. The most related to ours is CheXzero
[54], which targets at zero-shot diagnosis. Despite significant contribution
has been made by existing work [66, 25, 7, 45, 58, 54], they still treat
medical texts and images as common natural data and do not explicitly leverage
the rich prior knowledge from medical domain. In this paper, we consider to
incorporate domain knowledge, re-design the pre-training pipeline delicately
and target at accurate diagnosis in X-ray scans.

Figure 2: The whole framework of our method. We first pre-process the report
into triplets leveraging triplet extraction module. Then we encode the
extracted triplets and it is worth emphasizing that we translate the entities
into detailed descriptions during encoding, by querying the medical knowledge
base. Finally we change the training flow with triplets, _i.e._ , we query a
transformer-based fusion module at entity level, which provides more detailed
supervision signals.

##  3 Method

In this section, we start by describing the considered problem scenario in
Sec. 3.1, followed by our report pre-process operation with triplet extraction
in Sec. 3.2. Then we introduce our proposed knowledge-enhanced architecture in
Sec. 3.3, including, visual encoder, knowledge-enhanced triplet encoder, and
the fusion module for aligning visual-language signals. In Sec. 3.4, we
describe the training procedure with the paired image-reports sourced from the
daily routine X-ray scans and, in Sec.3.5, we introduce the procedure for
inference.

###  3.1 Problem Scenario

Assuming we are given a training set with $N$ samples, i.e.,
$\mathcal{D}_{\text{train}}=\\{(\mathcal{X}_{1},\mathcal{T}_{1}),\dots,(\mathcal{X}_{N},\mathcal{T}_{N})\\}$,
where $\mathcal{X}_{i},\mathcal{T}_{i}$ refer to the X-ray image and its
corresponding medical report generated in the daily routine scans,
respectively, our goal is to train a visual-language model that enables us to
diagnose the existence of certain diseases and localize the visual evidence
spatially. Specifically, at inference time, we can freely ask the system to
identify the likelihood of the patient getting a certain disease (seen or
unseen during training):

$\displaystyle\hat{s}_{i},\hat{m}_{i}=\Phi_{\text{fusion}}(\Phi_{\text{visual}}(\mathcal{X}_{i}),\Phi_{\text{textual}}(\text{[description]})),$
(1)

where $\mathcal{X}_{i}\in\mathbb{R}^{H\times W\times 3}$ refers to an image
sample from the test set, with $H,W$ denoting height and width respectively.
$\hat{s}_{i}\in[0,1]$ refers to the inferred likelihood of the patient having
a certain disease indicated by the input description, and
$\hat{m}_{i}\in\mathbb{R}^{H\times W\times 1}$ denotes a predicted spatial
heatmap, with high activation on pixels that potentially provide the visual
indication for such disease. In the following section, we will introduce our
report pre-process operation with triplet extraction.

###  3.2 Report Pre-processing

To start with, we propose to pre-process medical reports with a Triplet
Extraction module by removing the unnecessary complexity from language
grammar. Note that, we hereon only consider single sampled image-reports pair
$(\mathcal{X}_{i},\mathcal{T}_{i})$, and ignore the subscript in notations for
simplicity.

We condense the original reports with an off-shelf medical Named Entity
Recognition (NER) method, namely RadGraph [29, 65], transforming reports into
a set of triplets, as shown in Figure 2A. In detail, the medical key words can
be extracted and classified as “entity” or “position” with the NER module.
“Entity” refers to some clinical observations, like “Opacity”. “Position”
refers to the anatomical body part that occurs in a radiology report, like
“right lower lobe”. Besides, the NER module will also provide an “exist” label
to conclude whether an entity is claimed to be exist, absent or uncertain in
reports. Based on this, we can use a set of triplets, _i.e._ ,
$\\{\text{entity},\text{position},\text{exist}\\}$, to re-formulate the
sentence in reports, for example, the triplet $\\{\text{Opacity},\text{Right
lower lobe},\text{True}\\}$ represents “It is true that there is opacity
located at right lower lobe”. Note that, the triplets with a specific
“position” are not always termed as True in “exist” as radiologists may point
out entities absent at some specific position.

Therefore, given a report ${\mathcal{T}}$ with multiple sentences,
${\mathcal{T}}=\left\\{s_{1},s_{2},...,s_{M}\right\\}$, the extraction module
independently operates on each of the sentences, and construct a number of
triplets from the report:

$\displaystyle\Phi_{\text{ex}}({s_{j}})=\\{\text{entity}_{n},\text{position}_{n},\text{exist}_{n}\\},n\in[0,t_{j}],$
(2)

where $t_{j}$ represents the total number of entities contained in one
sentence, with $n=0$ indicating the special case that there is no valid
entity. After the triplet extraction, each report is equal to a set of
triplets.

Discussion. In contrast to natural texts, information in medical reports tends
to be more condensed, with radiologists pointing out the existence of
abnormality and their positions in the image. Meanwhile, medical terminologies
tend to be professional, and within certain vocabulary (mostly listed in UMLS
[5]), specially designed NER methods [29] demonstrate great performance on
reports. Therefore, adopting the Triplet Extraction operation in medical VLP
can avoid unnecessary complexity from understanding grammar, while still
retaining the useful information in reports.

###  3.3 Architecture

In this section, we detail our proposed framework, consisting of three
components, namely, visual encoding, knowledge-enhanced triplet encoding, and
fusion module, as shown in Fig. 2B and Fig. 2C.

####  3.3.1 Visual Encoding

Given an X-ray image scan $\mathcal{X}\in\mathbb{R}^{H\times W\times 3}$, we
can compute the features with a visual backbone:

$\displaystyle\mathcal{V}=\Phi_{\text{visual}}(\mathcal{X})\in\mathbb{R}^{h\times
w\times d},$ (3)

$h,w,d$ refer to the height, width, and feature dimension of the output, in
our case, we adopt ResNet-50 as the visual backbone, and take the output from
the 4th residual block. Note that, we make the architectural choice for fair
comparison with existing work [66, 45, 25, 6], while other visual backbones,
e.g., ViT [17], can equally be applied.

####  3.3.2 Knowledge-enhanced Triplet Encoding

The goal of this module is to encode the triplets extracted from reports by
incorporating medical domain knowledge as shown in Fig.2B.

Given a triplet as $\\{\text{entity},\text{position},\text{exist}\\}$, it is
easy to code the “exist” as it only has three outcomes. We use
$l\in\\{0,1,-1\\}$ to tokenize it, $1$ for True, $0$ for False, $-1$ for
uncertain. For the “entity” words, we translate them into detailed
descriptions by querying some easy-access medical knowledge bases 111Wikipedia
https://en.wikipedia.org/wiki/222UMLS [5]
https://www.nlm.nih.gov/research/umls/, _e.g._ ,
Description([“Pneumonia”])=“It is a condition of the lung primarily …present
with opacities and pleural effusion …”. Despite its simplicity, converting the
entities into descriptions is crucial for more reliable and zero-shot
diagnosis, as it further decomposes the professional medical entities into
basic attributes that are shared by different diseases, encouraging the model
to capture a deep understanding of the visual evidence. For the “position”
words, we use a prompt as “It is located at {position}” to form a sentence.
Finally, we use ClinicalBERT [3] as a pre-trained text encoder, to compute the
embedding for the “entity” and “position”, and then adopt a linear MLP to
project the embedding to desired dimensions:

$\displaystyle e=\Phi_{\text{textual}}(\text{Description(\text{\\{entity\\})
})}\in\mathbb{R}^{d},$ (4) $\displaystyle p=\Phi_{\text{textual}}(\text{``It
is located at \\{position\\}''})\in\mathbb{R}^{d^{\prime}}.$ (5)

Each triplet has now been embedded into $\\{e,p,l\\}$.

Discussion. The extracted entities are medical terminologies that are only
understandable to audiences with a medical background, while enriching them
with detailed descriptions helps the model to capture a deep understanding of
visual evidence for diseases. Such patterns can be generalized across
diseases, as many attribute descriptions tend to be shared, enabling the model
to build implicit relationships on seen classes and understand descriptions
for unseen ones.

####  3.3.3 Fusion Module

With the triplets from reports, we can supervise the model on the entity level
instead of the entire report level. The “position” and “exist” parts in
triplets can be naturally seen as more fine-grained supervision labels.
Specifically, we adopt a Transformer-based architecture, use the embedding of
entities as query, iteratively attending the image embeddings, and output
exist and position predictions of entities.

In detail, we select the top $|Q|$ most commonly appearing entities’
embeddings in all training reports, to form an entity query set
$Q=\\{e_{1},e_{2},...,e_{|Q|}\\}$. The details of the entity query set is
provided in the supplementary material (Sec. A). Then $Q$ will be passed into
a fusion module with the image representation $\mathcal{V}$ for alignment. The
fusion module consists of multiple Transformer Decoder layers, with $Q$ as
Query, and $\mathcal{V}$ as Key and Value. The outputs are further fed into
two MLPs, independently infer the existence of the entity and the entity’s
position:

$\displaystyle\\{\hat{s},\text{ }\hat{p},\text{
}\hat{m}\\}=\Phi_{\text{fusion}}(\mathcal{V},\text{ }Q),$ (6)

where $\hat{s}\in\mathbb{R}^{|Q|}$ represents the existence prediction for
each entity query, and $\hat{p}\in\mathbb{R}^{|Q|\times d^{\prime}}$
represents the predicted position for all entities. Note that,
$\hat{m}\in\mathbb{R}^{H\times W}$ denotes the average of the cross-attention
maps sourced from Transformer layers and is up-sampled to the size of input
image with nearest interpolation. $\hat{m}$ is used for grounding at
inference, as it naturally acts as a segmentation heatmap. During training, we
will not directly calculate any loss on it.

Discussion. Adopting Transformer decoder enables to compute correspondences
between entities and images at patch-level. Consequently, image features
$\mathcal{V}$ are more suitable for downstream segmentation tasks and the
average of cross-attention maps in each layers can be used directly for zero-
shot grounding, providing explainable for diagnosis. Besides, the default
self-attention on queries in Transformer structure can also build
relationships across entities.

###  3.4 Training

Given a set of encoded triplets $\\{e,p,l\\}$ extracted form the pairing
reports ${\mathcal{T}}$, we can compute training loss on the output of fusion
module. For the existence prediction $\hat{s}$, we use binary cross-entropy
with the corresponding “exist” labels $l$, and if $l$ is $-1$ we just pass
this label, denoted as $\mathcal{L}_{\text{cls}}$. To supervise the position
prediction for each entity query, we adopt contrastive learning. We form a
position set with top $|P|$ common position embeddings as a position set,
$P=\\{p_{1},p_{2},\cdots,p_{|P|}\\}$, randomly sample $M$ negative position
embeddings from it, and use the corresponding position embedding $p$ from
triplets as positive:

$\begin{split}\mathcal{L}_{\text{loc}}=-\frac{1}{|Q|}\sum_{k=1}^{|Q|}\frac{e^{\langle\hat{p}_{k},p_{k}\rangle}}{e^{\langle\hat{p}_{k},p_{k}\rangle}+\sum_{u=1}^{M}e^{\langle\hat{p}_{k},P_{\mathcal{I}(k,u)}\rangle}},\end{split}$
(7)

where $\langle\cdot,\cdot\rangle$ represents the inner product of two vectors
and $\mathcal{I}(\cdot,\cdot)$ is a random index sampling function. The
position embeddings are un-normalized in calculation. Note that, some entities
may not be mentioned in the report and thus, we can not find corresponding
labels in triplets. We simply ignore the corresponding predictions while
computing loss.

The final loss is the sum of the two:

$\mathcal{L}_{\text{total}}=\alpha_{1}\mathcal{L}_{\text{loc}}+\alpha_{2}\mathcal{L}_{\text{cls}},$
(8)

where $\alpha_{1},\alpha_{2}$ refer to two hyper-parameters controlling the
ratio of the two losses, and we set them to be $1.0$ by default.

Discussion. In contrast to the existing approaches [66] that align images with
entire reports, our training paradigm with triplets provides supervision at a
more fine-grained entity level, rather than the global alignment between image
and reports as has often done in existing approaches.

###  3.5 Inference

At inference time, given a test image, we can directly infer the existence of
certain entities/disease, and ground their visual evidence. In particular, for
the entities that have appeared in the entity query set $Q$, we simply adopt
the corresponding elements from $Q$, while for those unseen ones, we replace
the entity with a brief description provided by the user, and treat it as an
extra query added to entity query set $Q$, resembling zero-shot inference. The
existence output $\hat{s}$ can be directly applied for classification, the
average cross-attention $\hat{m}$ between the target entity and the visual
features are used for grounding.

Dataset | RSNA Pneumonia | SIIM-ACR Pneumothorax | ChestX-ray14
---|---|---|---
Methods |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$ |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$ |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$
ConVIRT [66] | 0.8042 | 0.5842 | 0.7611 | 0.6431 | 0.4329 | 0.5700 | 0.6101 | 0.1628 | 0.7102
GLoRIA [25] | 0.7145 | 0.4901 | 0.7129 | 0.5342 | 0.3823 | 0.4047 | 0.6610 | 0.1732 | 0.7700
BioViL [6] | 0.8280 | 0.5833 | 0.7669 | 0.7079 | 0.4855 | 0.6909 | 0.6912 | 0.1931 | 0.7916
CheXzero [54] | 0.8579 | 0.6211 | 0.7942 | 0.6879 | 0.4704 | 0.5466 | 0.7296 | 0.2141 | 0.8278
Ours | 0.8694 | 0.6342 | 0.8002 | 0.8924 | 0.6833 | 0.8428 | 0.7676 | 0.2525 | 0.8619
Table 1: Comparison with other state-of-the-art methods on zero-shot
classification task. AUC, F1 and ACC scores are reported. For ChestX-ray14,
the metrics all refer to the macro average on the 14 diseases.

Prompt Type | Direct Covid-19 | Covid-19 Description
---|---|---
Methods |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$ |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$
ConVIRT [66] | 0.6159 | 0.7057 | 0.6113 | 0.5208 | 0.6902 | 0.5266
GLoRIA [25] | 0.6319 | 0.6938 | 0.5710 | 0.6659 | 0.7007 | 0.6083
BioViL [6] | 0.6137 | 0.6958 | 0.5461 | 0.5382 | 0.6910 | 0.5375
CheXzero [54] | 0.6462 | 0.7369 | 0.6629 | 0.6667 | 0.6400 | 0.6578
Ours | 0.6561 | 0.7066 | 0.5917 | 0.7396 | 0.7670 | 0.7006

Table 2: Comparison with other state-of-the-art methods on zero-shot Covid-19
classification task. AUC, F1 and ACC scores are reported. “Direct covid-19”
refers to directly use “Covid-19” to construct the prompt sentence while
“Covid-19 Description” refers to replace the name “Covid-19” with its
description.

##  4 Experiment

In this section, we start by introducing the dataset used for experiments,
e.g., pre-training, and various downstream datasets. Then we describe the
implementation details and the considered baselines.

###  4.1 Pre-training Dataset

MIMIC-CXR v2 [31, 22] consists of over 227k studies of paired image-report
data, they are sourced from 65,379 patients at different scanning. Each study
can have one or two images (different scan views), totaling 377,110 images.

###  4.2 Datasets for Downstream Tasks

ChestX-ray14 [56] contains 112,120 frontal-view X-ray images of 30,805 unique
patients, collected from the year of 1992 to 2015 by NIH(National Institutes
of Health), with labels of 14 common diseases provided. We split the dataset
into $0.8/0.1/0.1$ for train/valid/test.

RSNA Pneumonia [49] contains more than 260k frontal-view chest X-rays with
corresponding pneumonia opacity masks collected by RSNA (Radiological Society
of North America). Commonly, it is treated as a classification tasks [25, 6].
We split the dataset into $0.6/0.2/0.2$ for train/valid/test.

SIIM-ACR Pneumothorax [1] contains more than 12k frontal-view chest X-rays
with pneumothorax masks collected by SIIM-ACR (Society for Imaging Informatics
in Medicine and American College of Radiology). Similarly to RSNA Pneumonia
dataset, it can be both used as classification and segmentation tasks. We
split the dataset into $0.6/0.2/0.2$ for train/valid/test.

COVIDx CXR-2 [46] and COVID Rural [52, 15] aim to evaluate on diagnosing
COVID-19. COVIDx CXR-3 contains 29,986 images from 16,648 patients with
COVID-19 classification labels. We use it as a classification dataset and
split it into $0.7/0.2/0.1$ for train/valid/test. Additionally, we use COVID
Rural dataset for COVID-19 segmentation. It contains more than 200 chest
X-rays with segmentation masks, and we split it into $0.6/0.2/0.2$ for
train/valid/test.

Edema Severity [7] contains 6,524 examples from MIMIC-CXR with pulmonary edema
severity labels (0 to 3, increasing severity) extracted from the radiology
reports. Of these, 141 radiologists were examined by radiologists, and
consensus was reached on severity level. It can be seen as a typical fine-
grained classification task. We split the dataset into $0.6/0.2/0.2$ for
train/valid/test.

###  4.3 Implementation

This section describes the implementation for architectures. In Pre-training,
the triplets extraction module and text encoders used in triplets encoding are
all fixed, while the visual encoder and fusion module are trained end-to-end
on the image-text pairs. In Fine-tuning, we adopt ResNet50 [24] initialized
with image encoder for classification, and ResUNet [16] initialize its encoder
with our pre-trained image encoder for segmentation. More details about exact
values of different parameters and training progress can be found in
supplementary material (Sec. B)

###  4.4 Baselines

We compare with various existing state-of-the-art medical image-text pre-train
methods, namely, ConVIRT [66], GLoRIA [25], BioViL [6] and CheXzero [54].
Since ConVIRT and GLoRIA are pre-trained on an in-house dataset, we re-train
their models on MIMIC-CXR dataset for fair comparison. For BioViL, we use the
officially released models by the authors. For zero-shot setting, we use the
prompt as mentioned by BioViL [6] and compare to the very recent method
(CheXzero [54]) that has shown to have better zero-shot diagnosis ability than
radiologists. For fine-tuning, we all use the same setting as described in
Sec. 4.3.

###  4.5 Metrics

AUC, F1 and ACC are measured for classification tasks. F1 comprehensively
measures the recall and precision of the model, and ACC is the short of
Accuracy. The final binary prediction threshold is chosen to maximise the F1
score. The ACC score is also calculated under this threshold.

Pointing Game is used for evaluating the grounding performance. In specific,
we extract the region with max response in the output heat-map, for one
instance, if the region hit the ground-truth mask, it is considered a positive
prediction, otherwise negative. Finally, accuracy can be calculated as the
pointing game score.

Dice and IOU are commonly used for segmentation tasks. For zero-shot
segmentation, we search the segmentation threshold with 0.01 interval for all
methods, and report the maximal Dice score for each model.

Precision and Recall refer to the detection Precision and Recall. For medical,
it is important that lesions are detected even without fine segmentation.
Additionally, in some hard cases, especially for the zero-shot setting, Dice
and IOU may be too strict to reflect the performance difference. Precision and
recall scores can compensate for these. We choose the IOU threshold as 0.1 to
calculate the scores.

Methods |  Pointing Game$\uparrow$ |  Recall$\uparrow$ |  Precision$\uparrow$ |  IoU$\uparrow$ |  Dice$\uparrow$
---|---|---|---|---|---
GLoRIA [25] | 0.7607 | 0.8330 | 0.1621 | 0.2182 | 0.3468
BioViL [6] | 0.8342 | 0.8521 | 0.5034 | 0.3029 | 0.4386
Ours | 0.8721 | 0.8661 | 0.6420 | 0.3172 | 0.4649
(a) Zero-shot grounding on Pneumonia

Methods |  Pointing Game$\uparrow$ |  Recall$\uparrow$ |  Precision$\uparrow$
---|---|---|---
GLoRIA [25] | 0.0651 | 0.2377 | 0.0585
BioViL [6] | 0.0252 | 0.1963 | 0.1429
Ours | 0.1975 | 0.3562 | 0.1940
(b) Zero-shot grounding on Pneumothorax

Table 3: Comparison with other state-of-the-art methods on zero-shot region grounding tasks. (a) shows the results on RSNA Pneumonia dataset. (b) shows the results on SIIM-ACR Pneumothorax dataset. The pneumothorax region tends to be thin and narrow and much more challenging for grounding, we thus only consider pointing game, recall, and precision. Our method can achieve better performance on different metrics, especially on the pointing game score. ConVIRT and CheXzero can not realize this function. Prompt Type | Direct covid-19 | Covid-19 Description
---|---|---
Methods |  Pointing Game$\uparrow$ |  Recall$\uparrow$ |  Precision$\uparrow$ |  IoU$\uparrow$ |  Dice$\uparrow$ |  Pointing Game$\uparrow$ |  AR$\uparrow$ |  AP$\uparrow$ |  IoU$\uparrow$ |  Dice$\uparrow$
GLoRIA [25] | 0.0364 | 0.2906 | 0.1073 | 0.0645 | 0.1141 | 0.2727 | 0.2821 | 0.1336 | 0.0596 | 0.1075
BioViL [6] | 0.4000 | 0.2564 | 0.2703 | 0.1198 | 0.1967 | 0.1818 | 0.2393 | 0.1637 | 0.0861 | 0.1427
Ours | 0.1818 | 0.1880 | 0.1497 | 0.0747 | 0.1289 | 0.5818 | 0.5214 | 0.4959 | 0.1373 | 0.2278
Table 4:  Comparison with other state-of-the-art methods on zero-shot covid-19 opacity region grounding task. “Direct covid-19” refers to directly use “Covid-19” to construct the prompt sentence for entity encoding while “Covid-19 Description” refers to replace the name “Covid-19“ with its description. Our method can achieve better performance on different metrics.  Dataset | Pneumonia | Pneumothorax | Covid-19 | ChestX-ray14
---|---|---|---|---
Data Portion | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100%
Scratch | 0.7107 | 0.8150 | 0.8626 | 0.4347 | 0.6120 | 0.6571 | 0.7861 | 0.9162 | 0.9554 | 0.6005 | 0.7365 | 0.7924
ConVIRT [66] | 0.8398 | 0.8562 | 0.8761 | 0.7134 | 0.7826 | 0.9004 | 0.8675 | 0.9541 | 0.9726 | 0.6615 | 0.7658 | 0.8128
GLoRIA [25] | 0.8599 | 0.8666 | 0.8846 | 0.7439 | 0.8538 | 0.9014 | 0.9065 | 0.9381 | 0.9728 | 0.6710 | 0.7642 | 0.8184
BioViL [6] | 0.8233 | 0.8538 | 0.8836 | 0.6948 | 0.7775 | 0.8689 | 0.8989 | 0.9529 | 0.9729 | 0.6952 | 0.7527 | 0.8245
Ours | 0.8731 | 0.8799 | 0.8931 | 0.8527 | 0.9071 | 0.9188 | 0.9224 | 0.9657 | 0.9729 | 0.7721 | 0.7894 | 0.8323
Table 5:  Comparison of AUC scores with other state-of-the-art methods on fine-tuning classification task. The macro average of AUC scores on 14 diseases are reported for ChestX-ray14 dataset. Diseases | Pneumonia | Pneumothorax | Covid-19
---|---|---|---
Data Portion | 1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100%
Scratch | 0.4347 | 0.6047 | 0.7068 | 0.2133 | 0.3323 | 0.7447 | 0.1481 | 0.2367 | 0.3228
ConVIRT [66] | 0.5706 | 0.6491 | 0.7201 | 0.5406 | 0.6121 | 0.7352 | 0.1995 | 0.2724 | 0.3737
GLoRIA [25] | 0.6555 | 0.6907 | 0.7328 | 0.5673 | 0.5778 | 0.7694 | 0.1889 | 0.2809 | 0.3869
BioViL [6] | 0.6824 | 0.7038 | 0.7249 | 0.6267 | 0.6998 | 0.7849 | 0.2113 | 0.3239 | 0.4162
Ours | 0.7064 | 0.7162 | 0.7579 | 0.6659 | 0.7210 | 0.7937 | 0.2445 | 0.3539 | 0.4399
Table 6:  Comparison of Dice scores with other state-of-the-art methods on fine-tuning segmentation tasks. Three diseases are reported, and for each disease, three data portions, $1\%$, $10\%$, $100\%$ are adopted to show the performance change under different data amounts. Methods | 0 | 1 | 2 | 3 | AVG
---|---|---|---|---|---
AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$ |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$ |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$ |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$ |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$
Scratch | 0.7631 | 0.7036 | 0.6738 | 0.5383 | 0.3593 | 0.3223 | 0.6692 | 0.4328 | 0.7012 | 0.8420 | 0.5694 | 0.8770 | 0.7031 | 0.5163 | 0.6436
ConVIRT [66] | 0.8453 | 0.7769 | 0.7793 | 0.6099 | 0.3938 | 0.4629 | 0.7202 | 0.4843 | 0.6445 | 0.9047 | 0.6154 | 0.8809 | 0.7700 | 0.5676 | 0.6919
GLoRIA [25] | 0.8304 | 0.7577 | 0.7520 | 0.6208 | 0.3991 | 0.4922 | 0.7339 | 0.4958 | 0.7037 | 0.9246 | 0.6667 | 0.9102 | 0.7774 | 0.5798 | 0.7145
BioViL [6] | 0.8034 | 0.7378 | 0.7148 | 0.6035 | 0.3912 | 0.4570 | 0.6860 | 0.4497 | 0.6777 | 0.9229 | 0.6500 | 0.9160 | 0.7540 | 0.5572 | 0.6914
Ours | 0.8502 | 0.7646 | 0.7539 | 0.6641 | 0.4140 | 0.5392 | 0.7605 | 0.5266 | 0.7031 | 0.8845 | 0.6250 | 0.9160 | 0.7898 | 0.5826 | 0.7280
Table 7:  Comparison with other state-of-the-art methods on fine-tuning edema
severity grading multi-class classification task. AUC score is reported in the
Table. “0,1,2,3” in the table represents the severity level and final average
scores are reported.

##  5 Results

In this section, we will report the experimental results. In general, we split
the results into two parts: zero-shot setting and fine-tuning setting. In the
zero-shot case (Sec. 5.1), we carry out the ablation study and compare it with
the other SOTA image-text pre-train methods. We mainly consider classification
and segmentation tasks; In the fine-tuning case (Sec. 5.2), we evaluate the
model’s transferability by fine-tuning the model with $1\%$, $10\%$, and
$100\%$ data portion. Additionally, we also add a disease grading downstream
task, which can be seen as a fine-grade classification task, showing that our
pre-trained model can be transferred to the downstream tasks at ease.

###  5.1 Zero-shot Evaluation

In this section, we compare our method with other state-of-the-art methods
under zero-shot setting, on classification and grounding. Due to the space
limitation, we include the entire ablation study in the supplementary material
(Sec. C), referring to it for more details and analysis, and all comparisons
here are made using our best model with position contrastive loss and entity
description encoder.

####  5.1.1 Classification

Seen Diseases. As shown in Tab. 1, we compare with existing methods on three
widely-used datasets, demonstrating consistent performance improvement.
Specifically, on pneumonia and pneumothorax datasets, despite the images being
collected by different clinics with different diseases, our model improves the
AUC score from $0.83$ to $0.87$ on RSNA pneumonia dataset and from $0.71$ to
$0.89$ on SIIM-ACR pneumothorax dataset, as shown in Tab. 1. This shows that
our method can better deal with the multi-center and multi-disease data
distribution in medical. While on ChestX-ray14 dataset, we improve the average
AUC scores from $0.69$ to $0.77$, we refer the reader to supplementary
material (Sec. D) for detailed comparison of 14 diseases.

Unseen Diseases. Here, we are considering a strict setting for openset
classification, in particular, we use covid-19 to evaluate the systems.
Covid-19 is a new disease that only appeared in 2019, MIMIC-CXR reports
collected in the year 2015 do not include any entity of covid-19, thus it
requires the system to have the ability to diagnose truly unseen diseases. As
shown in Tab. 2, existing approaches that only rely on disease name struggles
to make the correct diagnosis. While, our proposed approach, after introducing
medical knowledge, i.e., using entity descriptions, can understand the complex
medical entity descriptions unseen in the training set, and significantly
boost the performance from $0.66$ to $0.74$ on AUC and from $0.59$ to $0.70$
on ACC, demonstrating entity translation is vital for unseen diseases.

####  5.1.2 Grounding

In addition to the plain diagnosis, explainability can be equally critical in
healthcare, improving the reliability and trustiness of the machine learning
systems. Here, we consider providing explainability by grounding the
abnormality in the prediction and compare against the existing approaches.
Similarly, we split the diseases into seen and unseen ones, depending on
whether their names have appeared in the medical reports. Specifically,
“Pneumonia” and “Pneumothorax” are treated as seen, and “Covid-19” is treated
as unseen. Due to the space limitation, we include visualization results in
supplementary material (Sec. 4).

Seen Diseases. We show the results for grounding on RSNA Pneumonia opacity and
SIIM-ACR Pneumothorax collapse in Tab. 3. As shown in Tab. 3(a), our proposed
model surpasses existing approaches on all metrics, for example, we improve
the pointing game score from $0.83$ to $0.87$, the detection Recall from
$0.85$ to $0.87$, the detection precision from $0.50$ to $0.64$, the IOU from
$0.30$ to $0.32$ and the Dice from $0.44$ to $0.46$. While on SIIM-ACR dataset
(Tab. 3(b)), the pneumothorax region tends to be thin and narrow, localizing
it can often be more challenging than that of opacity grounding [6], we thus
only consider pointing game, recall, and precision. Similarly, our method can
achieve significantly better performance than prior approaches.

Unseen Diseases. We also conduct the zero-shot grounding experiment on the
unseen disease, namely, Covid-19, as shown in Tab. 4. Our model has shown
consistent improvements in all metrics, e.g., boosting the pointing game score
from $0.40$ to $0.58$. One observation to be noticed is that, results in Tab.
4 are mostly consistent with those in Tab. 2, i.e., better classification
results tend to lead to better grounding. Overall, our model with knowledge-
enhanced language encoding has facilitated the visual encoder to learn
underlying evidence relating to the diseases, therefore, leading to more
interpretable representations than prior approaches.

###  5.2 Fine-tuning Evaluation

In this section, we consider the fine-tuning scenario, with the pre-trained
model as initialization, and trained end-to-end on the downstream tasks. We
test on three different tasks, namely, classification, segmentation, and
grading. In classification and segmentation, the test splits and metrics are
the same as in the “zero-shot” section. Grading is a new task we introduce in
fine-tuning setting, which can be seen as a fine-grained classification task.

####  5.2.1 Classification

We experiment on four different datasets, using $1\%$, $10\%$, $100\%$ of the
data for fine-tuning, that is consistent with the existing work [66, 25, 6].
As shown in Tab. 5, our model has demonstrated substantial improvements in the
AUC scores over the existing approaches across all datasets, reflecting that
our pre-trained representation is of higher quality than existing models. We
refer the readers to the supplementary material (Sec. D) for more detailed
comparison results.

####  5.2.2 Segmentation

In Tab. 6, we conduct fine-tuning experiments on three different diseases for
segmentation. We pick $1\%$, $10\%$, $100\%$ of the data for fine-tuning. For
all three different diseases with different image distributions, our methods
surpass the existing state-of-the-art methods by a large margin, especially
under the low-data regime.

####  5.2.3 Grading

Besides diagnosis, grading the disease severity level also plays an important
role. Here, we adopt our pre-trained features and train them for the multi-
class classification task, with $0$ to $3$ representing different severity
levels. As shown in Tab. 7, for each level, the AUC, F1, and ACC scores are
calculated as one class against all other ones, for example, $0$ vs
$\left\\{1,2,3\right\\}$. Final macro average scores of four levels are
computed. On the majority of severity levels, our method can achieve the best
results.

##  6 Conclusion

In this paper, we introduce a novel medical knowledge enhanced VLP model.
First, we propose a triplet extraction module to extract useful medical-
related triplets as more useful supervision signals, simplifying complex raw
reports with minimal information loss. Second, we translate the entities in
extracted triplets into detailed medical descriptions and embed them with a
text encoder enabling the network to understand complex medical expert-level
knowledge. Finally, a transformer-based structure is proposed to do local
region alignment. In experiments, We evaluate our method on different datasets
under various settings. Our method shows strong zero-shot classification and
grounding abilities, even facing unseen diseases. Additionally, with fine-
tuning, our method still outperforms state-of-the-art methods significantly,
showing the superiority of our method.

## References

  * [1] Society for imaging informatics in medicine: Siim-acr pneumothorax segmentation.  https://www.kaggle.com/c/siim-acr-pneumothorax-segmentation.  2019\.
  * [2] Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott.  Publicly available clinical BERT embeddings.  In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics.
  * [3] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, WA Redmond, and Matthew BA McDermott.  Publicly available clinical bert embeddings.  NAACL HLT 2019, page 72, 2019.
  * [4] Federico Bianchi, Giuseppe Attanasio, Raphael Pisoni, Silvia Terragni, Gabriele Sarti, and Sri Lakshmi.  Contrastive language-image pre-training for the italian language.  arXiv preprint arXiv:2108.08688, 2021.
  * [5] Olivier Bodenreider.  The unified medical language system (umls): integrating biomedical terminology.  Nucleic acids research, 32(suppl_1):D267–D270, 2004.
  * [6] Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, et al.  Making the most of text semantics to improve biomedical vision–language processing.  In European conference on computer vision, pages 1–21, 2022.  Official Implementation: https://github.com/microsoft/hi-ml/tree/main/hi-ml-multimodal.
  * [7] Geeticka Chauhan, Ruizhi Liao, William Wells, Jacob Andreas, Xin Wang, Seth Berkowitz, Steven Horng, Peter Szolovits, and Polina Golland.  Joint modeling of chest radiographs and radiology reports for pulmonary edema assessment.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 529–539. Springer, 2020.
  * [8] Sihong Chen, Jing Qin, Xing Ji, Baiying Lei, Tianfu Wang, Dong Ni, and Jie-Zhi Cheng.  Automatic scoring of multiple semantic attributes with multi-task feature leverage: a study on pulmonary nodules in ct images.  IEEE transactions on medical imaging, 36(3):802–814, 2016.
  * [9] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In European conference on computer vision, pages 104–120. Springer, 2020.
  * [10] Zhihong Chen, Guanbin Li, and Xiang Wan.  Align, reason and learn: Enhancing medical vision-and-language pre-training with knowledge.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 5152–5161, 2022.
  * [11] Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan.  Generating radiology reports via memory-driven transformer.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1439–1449, 2020.
  * [12] Hui Cui, Yiyue Xu, Wanlong Li, Linlin Wang, and Henry Duh.  Collaborative learning of cross-channel clinical attention for radiotherapy-related esophageal fistula prediction from ct.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 212–220. Springer, 2020.
  * [13] Yuhao Cui, Zhou Yu, Chunqi Wang, Zhongzhou Zhao, Ji Zhang, Meng Wang, and Jun Yu.  Rosita: Enhancing vision-and-language semantic alignments via cross-and intra-modal knowledge integration.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 797–806, 2021.
  * [14] Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, and Curtis P Langlotz.  Improving the factual correctness of radiology report generation with semantic rewards.  arXiv preprint arXiv:2210.12186, 2022.
  * [15] Shivang Desai, Ahmad Baghal, Thidathip Wongsurawat, Piroon Jenjaroenpun, Thomas Powell, Shaymaa Al-Shukri, Kim Gates, Phillip Farmer, Michael Rutherford, Geri Blake, et al.  Chest imaging representing a covid-19 positive rural us population.  Scientific data, 7(1):1–6, 2020.
  * [16] Foivos I Diakogiannis, François Waldner, Peter Caccetta, and Chen Wu.  Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data.  ISPRS Journal of Photogrammetry and Remote Sensing, 162:94–114, 2020.
  * [17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [18] Jared A Dunnmon, Alexander J Ratner, Khaled Saab, Nishith Khandwala, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew P Lungren, Daniel L Rubin, et al.  Cross-modal data programming enables rapid medical machine learning.  Patterns, 1(2):100019, 2020.
  * [19] Bradley J Erickson, Panagiotis Korfiatis, Zeynettin Akkus, and Timothy L Kline.  Machine learning for medical imaging.  Radiographics, 37(2):505, 2017.
  * [20] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun.  Dermatologist-level classification of skin cancer with deep neural networks.  nature, 542(7639):115–118, 2017.
  * [21] Leyuan Fang, Chong Wang, Shutao Li, Hossein Rabbani, Xiangdong Chen, and Zhimin Liu.  Attention to lesion: Lesion-aware convolutional neural network for retinal optical coherence tomography image classification.  IEEE transactions on medical imaging, 38(8):1959–1970, 2019.
  * [22] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley.  Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals.  circulation, 101(23):e215–e220, 2000.
  * [23] Ivan Gonzalez-Diaz.  Dermaknet: Incorporating the knowledge of dermatologists to convolutional neural networks for skin lesion diagnosis.  IEEE journal of biomedical and health informatics, 23(2):547–559, 2018.
  * [24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [25] Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung.  Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942–3951, 2021.  Official Implementation: https://github.com/marshuang80/gloria.
  * [26] Xin Huang, Yu Fang, Mingming Lu, Fengqi Yan, Jun Yang, and Yilu Xu.  Dual-ray net: automatic diagnosis of thoracic diseases using frontal and lateral chest x-rays.  Journal of Medical Imaging and Health Informatics, 10(2):348–355, 2020.
  * [27] Sarfaraz Hussein, Kunlin Cao, Qi Song, and Ulas Bagci.  Risk stratification of lung nodules using 3d cnn-based multi-task learning.  In International conference on information processing in medical imaging, pages 249–260. Springer, 2017.
  * [28] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590–597, 2019.
  * [29] Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Truong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, Curtis Langlotz, et al.  Radgraph: Extracting clinical entities and relations from radiology reports.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  * [30] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [31] AEWP Johnson, Tom Pollard, Roger Mark, Seth Berkowitz, and Steven Horng.  Mimic-cxr database.  PhysioNet10, 13026:C2JT1Q, 2019.
  * [32] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  arXiv preprint arXiv:2201.12086, 2022.
  * [33] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  Advances in neural information processing systems, 34:9694–9705, 2021.
  * [34] Liu Li, Mai Xu, Xiaofei Wang, Lai Jiang, and Hanruo Liu.  Attention based glaucoma detection: a large-scale database and cnn model.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10571–10580, 2019.
  * [35] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [36] Xiaomeng Li, Xiaowei Hu, Lequan Yu, Lei Zhu, Chi-Wing Fu, and Pheng-Ann Heng.  Canet: cross-disease attention network for joint diabetic retinopathy and diabetic macular edema grading.  IEEE transactions on medical imaging, 39(5):1483–1493, 2019.
  * [37] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, pages 121–137. Springer, 2020.
  * [38] Qing Liao, Ye Ding, Zoe L Jiang, Xuan Wang, Chunkai Zhang, and Qian Zhang.  Multi-task deep convolutional neural network for cancer diagnosis.  Neurocomputing, 348:66–73, 2019.
  * [39] Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou.  Exploring and distilling posterior and prior knowledge for radiology report generation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13753–13762, 2021.
  * [40] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2018.
  * [41] Gabriel Maicas, Andrew P Bradley, Jacinto C Nascimento, Ian Reid, and Gustavo Carneiro.  Training medical image analysis systems like radiologists.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 546–554. Springer, 2018.
  * [42] Matthew BA McDermott, Tzu Ming Harry Hsu, Wei-Hung Weng, Marzyeh Ghassemi, and Peter Szolovits.  Chexpert++: Approximating the chexpert labeler for speed, differentiability, and probabilistic output.  In Machine Learning for Healthcare Conference, pages 913–927. PMLR, 2020.
  * [43] Masahiro Mitsuhara, Hiroshi Fukui, Yusuke Sakashita, Takanori Ogata, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi.  Embedding human knowledge into deep neural network via attention map.  arXiv preprint arXiv:1905.03540, 2019.
  * [44] Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, and Dan Jurafsky.  Improving factual completeness and consistency of image-to-text radiology report generation.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5288–5304, 2021.
  * [45] Philip Müller, Georgios Kaissis, Congyu Zou, and Daniel Rückert.  Joint learning of localized representations from medical images and reports.  arXiv preprint arXiv:2112.02889, 2021.
  * [46] Maya Pavlova, Naomi Terhljan, Audrey G Chung, Andy Zhao, Siddharth Surana, Hossein Aboutalebi, Hayden Gunraj, Ali Sabri, Amer Alaref, and Alexander Wong.  Covid-net cxr-2: An enhanced deep convolutional neural network design for detection of covid-19 cases from chest x-ray images.  Frontiers in Medicine, 9, 2022.
  * [47] Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers, and Zhiyong Lu.  Negbio: a high-performance tool for negation and uncertainty detection in radiology reports.  AMIA Summits on Translational Science Proceedings, 2018:188, 2018\.
  * [48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [49] George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, et al.  Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia.  Radiology. Artificial intelligence, 1(1), 2019.
  * [50] Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew Lungren.  Combining automatic labelers and expert annotations for accurate radiology report labeling using bert.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1500–1519, 2020.
  * [51] Jiaxing Tan, Yumei Huo, Zhengrong Liang, and Lihong Li.  Expert knowledge-infused deep learning for automatic lung nodule detection.  Journal of X-ray Science and Technology, 27(1):17–35, 2019.
  * [52] Haiming Tang, Nanfei Sun, and Yi Li.  Deep learning segmentation model for automated detection of the opacity regions in the chest x-rays of the covid-19 positive patients and the application for disease severity.  medRxiv preprint, 2020.
  * [53] Joseph J Titano, Marcus Badgeley, Javin Schefflein, Margaret Pain, Andres Su, Michael Cai, Nathaniel Swinburne, John Zech, Jun Kim, Joshua Bederson, et al.  Automated deep-neural-network surveillance of cranial images for acute neurologic events.  Nature medicine, 24(9):1337–1341, 2018.
  * [54] Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar.  Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning.  Nature Biomedical Engineering, pages 1–8, 2022.
  * [55] Kun Wang, Xiaohong Zhang, Sheng Huang, Feiyu Chen, Xiangbo Zhang, and Luwen Huangfu.  Learning to recognize thoracic disease in chest x-rays with knowledge-guided deep zoom neural networks.  IEEE Access, 8:159790–159805, 2020.
  * [56] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017.
  * [57] Zhanyu Wang, Mingkang Tang, Lei Wang, Xiu Li, and Luping Zhou.  A medical semantic-assisted transformer for radiographic report generation.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part III, pages 655–664. Springer, 2022\.
  * [58] Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun.  Medclip: Contrastive learning from unpaired medical images and text.  arXiv preprint arXiv:2210.10163, 2022.
  * [59] Joy T Wu, Nkechinyere Nneka Agu, Ismini Lourentzou, Arjun Sharma, Joseph Alexander Paguio, Jasper Seth Yao, Edward Christopher Dee, William G Mitchell, Satyananda Kashyap, Andrea Giovannini, et al.  Chest imagenome dataset for clinical reasoning.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  * [60] Xiaozheng Xie, Jianwei Niu, Xuefeng Liu, Zhengsu Chen, Shaojie Tang, and Shui Yu.  A survey on incorporating domain knowledge into deep learning for medical image analysis.  Medical Image Analysis, 69:101985, 2021.
  * [61] Yutong Xie, Yong Xia, Jianpeng Zhang, Yang Song, Dagan Feng, Michael Fulham, and Weidong Cai.  Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest ct.  IEEE transactions on medical imaging, 38(4):991–1004, 2018.
  * [62] Shuxin Yang, Xian Wu, Shen Ge, S Kevin Zhou, and Li Xiao.  Knowledge matters: Chest radiology report generation with general and specific knowledge.  Medical Image Analysis, 80:102510, 2022.
  * [63] Wenkai Yang, Juanjuan Zhao, Yan Qiang, Xiaotang Yang, Yunyun Dong, Qianqian Du, Guohua Shi, and Muhammad Bilal Zia.  Dscgans: Integrate domain knowledge in training dual-path semi-supervised conditional generative adversarial networks and s3vm for ultrasonography thyroid nodules classification.  In International conference on medical image computing and computer-assisted intervention, pages 558–566. Springer, 2019.
  * [64] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang.  Ernie-vil: Knowledge enhanced vision-language representations through scene graphs.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3208–3216, 2021.
  * [65] Ke Yu, Shantanu Ghosh, Zhexiong Liu, Christopher Deible, and Kayhan Batmanghelich.  Anatomy-guided weakly-supervised abnormality localization in chest x-rays.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 658–668. Springer, 2022.
  * [66] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz.  Contrastive learning of medical visual representations from paired images and text.  In Machine Learning for Healthcare, 2022.  Highest Starred Implementation: https://github.com/edreisMD/ConVIRT-pytorch.

##  Appendix A The Entity Description Base and Position Set

We leverage the rad-graph NER extraction results provided by [65] and further
add extra descriptions. Tab. LABEL:Table:_Description_Base shows the
descriptions we used to translate different entities. We have kept $75$
entities in entity query set $Q$, following [65], which covers $90\%$ entities
in reports. “Tail_abnorm_obs” entity represents some tailed entities and
“exluded_obs” represents some entities useless for diagnosis. The last
“covid-19” row is only referred for inference since it never appear in pre-
train reports.

Table 8: The Entity description used for translate single entity name. The description can be easily found from website. Entity |  Description
---|---
normal |  It means the absence of diseases and infirmity, indicating the structure is normal.
clear |  The lungs are clear and normal. No evidence for other diseases on lung.
sharp |  This means that an anatomical structure s boundary or edge is clear and normal, meaning it is free of diseases.
sharply |  “Sharply seen means that an anatomical structure is clearly visible.
unremarkable |  This represents some anatomical structures are normal, usually modifying cardiac and mediastinal silhouettes.
intact |  The bonny structure is complete and normal, meaning no fractures.
stable |  The modified anatomical structures are normal and stable. No evidence for diseases.
free |  It usually refers to free air and is associate with pneumothorax, atelectasis, pneumoperitoneum and emphysema.
effusion |  A pleural effusion is accumulation of excessive fluid in the pleural space, the potential space that surrounds each lung. A pleural effusion infiltrates the space between the visceral pleura and the parietal pleura.
opacity |  It is defined as an area of hazy opacification due to air displacement by fluid, airway collapse, fibrosis, or a neoplastic process. It is causes include infections, interstitial lung disease, and pulmonary edema.
pneumothorax |  A pneumothorax is an abnormal collection of air in the pleural space between the lung and the chest wall. It may be caused by pneumonia or fibrosis and other diseases.
edema |  Pulmonary edema, also known as pulmonary congestion, is excessive liquid accumulation in the tissue and air spaces of the lungs. It will show fluid in the alveolar walls.
atelectasis |  It is the collapse or closure of a lung resulting in reduced or absent gas exchange. Findings can include lung opacification and loss of lung volume.
tube |  It is a surgical drain that is inserted through the chest wall and into the pleural space or the mediastinum to remove undesired substances such as air.
consolidation |  It is a region of normally compressible lung tissue that has filled with liquid instead of air. Consolidation must be present to diagnose pneumonia: the signs of lobar pneumonia are characteristic and clinically referred to as consolidation.
process |  Acute process means there is abnormality in the anotomy structure.
abnormality |  It means the exist of diseases and infirmity, indicating the structure is abnormal.
enlarge |  It usually modifies cardiac silhouette and heart. Cardiomegaly is a medical condition in which the heart is enlarged.
tip |  It refers to the top head of the tube.
low |  The presence of low lung volumes may be a sign of a restrictive lung condition such as pulmonary fibrosis or sarcoidosis.
pneumonia |  Pneumonia is an inflammatory condition of the lung primarily small air sacs known as alveoli. Pneumonia may present with opacities. Complications such as pleural effusion may also be found increasing the diagnostic accuracy of lung consolidation and pleural effusion
line |  It refers to venous access line ot PICC lines.
congestion |  Pulmonary congestion is defined as accumulation of fluid in the lungs, resulting in impaired gas exchange and arterial hypoxemia.
catheter |  catheter is a tube placed in the body to drain and collect urine from the bladder
cardiomegaly |  Cardiomegaly (sometimes megacardia or megalocardia) is a medical condition in which the heart is enlarged.
fracture |  Fracture is a break in a rib bone.
air |  It refers to the free air or gas in pleural space, indicating pneumothorax. Air displacement by fluid may lead to opacity.
tortuous |  The Aorta is slightly tortuous. Sometimes it may refer to varicose veins
lead |  It refers to the leading head of the tube.
disease |  It means the exist of diseases and abnormalty, indicating the structure is abnormal.
calcification |  Pulmonary calcification is a common asymptomatic finding. Pulmonary calcifications are caused mainly by two mechanisms: the dystrophic form and the metastatic form
prominence |  It means the exist of some observation.
device |  It refer to some equipments like picc tub, valve catheter, pacemaker hardware, arthroplastmarker icd defib, device support equipment and mediport
engorgement |  Pulmonary vascular engorgement means obstruction of the normal flux of blood within the blood vessel network of the lung resulting in engorgement of pulmonary vessels
picc |  A peripherally inserted central catheter (PICC), also called a PICC line, is a long, thin tube that s inserted through a vein in your arm and passed through to the larger veins near your heart.
clip |  Surgical clips or vascular clips usually represent the one kind of medical equipments.
elevation |  If tissues or anatomical structures are elevated, they are raised up higher than the normal location.
expand |  It means the lungs are normally expanded and clear, indicating the absence of pneumothorax.
nodule |  A lung nodule or pulmonary nodule is a relatively small focal density in the lung. it may be confused with the projection of a structure of the chest wall or skin, such as a nipple, a healing rib fracture or lung cancer.
wire |  Sternotomis wires means the center line of the chest.
fluid |  It refers to the water of liquid in the lung and it may indicate edema and other diseases.
degenerative |  Degenerative disease is the result of a continuous process based on degenerative cell changes
pacemaker |  pacemaker device usually represents the one kind of medical equipments.
thicken |  Pleural thickening is an increase in the bulkiness of one or both of the pulmonary pleurae. It may cause by pulmonary Infection, empyema, tuberculosis or lung cancer.
marking |  It represents interstitial markings or bronchovascular markings
scar |  A scar (or scar tissue) is an area of fibrous tissue that replaces normal tissues after an injury.
hyperinflate |  Hyperinflated lungs are larger-than-normal lungs as a result of trapped air.
blunt |  Blunting of the costophrenic angles is usually caused by a pleural effusion, as already discussed. Other causes of costophrenic angle blunting include lung disease in the region of the costophrenic angle, and lung hyperexpansion.
loss |  The etiology of lung volume loss can be listed as follow: airway obstruction or compression, obesity, scoliosis, restrictive diseases such as pulmonary fibrosis and interstitial lung disease, tuberculosis.
widen |  The mediastinum is not widened or enlarged.
collapse |  Collapse lung refers to pneumothorax or atelectasis.
density |  The density (more precisely, the volumetric mass density; also known as specific mass), of a substance is its mass per unit volume.
emphysema |  Emphysema, or pulmonary emphysema, is a lower respiratory tract disease, characterized by air-filled spaces (pneumatosis) in the lungs, that can vary in size and may be very large.
aerate |  Aeration (also called aerification or aeriation) is the process by which air is circulated through, mixed with or dissolved in a liquid or other substances that act as a fluid (such as soil).
mass |  A lung mass is an abnormal growth or area in the lungs and it can also view as lung cancer.
crowd |  Crowding of the bronchovascular structures is an important direct sign of volume loss. The atelectatic lung enhances densely after contrast administration because of closeness of the pulmonary arteries and arterioles within the collapsed lobe.
infiltrate |  A pulmonary infiltrate is a substance denser than air, such as pus, blood, or protein, which lingers within the parenchyma of the lungs. Pulmonary infiltrates are associated with pneumonia, tuberculosis and sarcoidosis.
obscure |  Some anatomy structures are not clear and is difficult to understand or see.
deformity |  It means some body parts are abnormal or unjuried.
hernia |  Lung hernia (Sibson hernia) is a protrusion of lung outside of thoracic wall. the hernia is noted after chest trauma, thoracic surgery or certain pulmonary diseases.
drainage |  Tube drainage represents the one kind of medical equipment.
distention |  Distension generally refers to an enlargement, dilation, or ballooning effect. It may refer to: Abdominal distension.
shift |  The mediastinal shift is the deviation of the mediastinal structures towards one side of the chest cavity, usually seen on chest radiograph. It indicates a severe asymmetry of intrathoracic pressures.
stent |  Tracheal stent represents the one kind of medical equipments
pressure |  Pulmonary venous pressure is intermediate between mean PAP and LAP over all physiologic pressures
lesion |  Lung nodules, pulmonary nodules, white spots, lesions—these terms all describe the same phenomenon: an abnormality in the lungs.
finding |  Some observation on body parts, usually indicating abnormalty.
borderline |  Borderline size of the cardiac silhouette means the cardiac silhouette is not enlarged and normal.
hardware |  It represents the one kind of medical equipments.
dilation |  The state of being larger or more open than normal
chf |  Heart failure — sometimes known as congestive heart failure — occurs when the heart muscle doesn’t pump blood as well as it should. When this happens, blood often backs up and fluid can build up in the lungs, causing shortness of breath.
redistribution |  If the pulmonary edema is due to heart failure or fluid overload, you may also see cardiomegaly and distension of the pulmonary veins, particularly in the upper lung fields.
aspiration |  Aspiration pneumonia occurs when food or liquid is breathed into the airways or lungs, instead of being swallowed.
tail_abnorm_obs |  Some very rare diseases.
excluded_obs |  Some meaningless observations.
covid-19 |  It is a contagious disease caused by a virus. Ground-glass opacities, consolidation, thickening, pleural effusions commonly appear in infection.

Additionally, we keep $51$ positive positions, following [65], to form the
position set $P$, as {trachea, left_hilar, right_hilar, hilar_unspec,
left_pleural, right_pleural, pleural_unspec, heart_size, heart_border,
left_diaphragm, right_diaphragm, diaphragm_unspec, retrocardiac,
lower_left_lobe, upper_left_lobe, lower_right_lobe middle_right_lobe,
upper_right_lobe, left_lower_lung, left_mid_lung, left_upper_lung
left_apical_lung, left_lung_unspec, right_lower_lung, right_mid_lung,
right_upper_lung right_apical_lung, right_lung_unspec, lung_apices,
lung_bases, left_costophrenic right_costophrenic, costophrenic_unspec,
cardiophrenic_sulcus, mediastinal, spine clavicle, rib, stomach, right_atrium,
right_ventricle, aorta, svc, interstitium, parenchymal, cavoatrial_junction,
cardiopulmonary, pulmonary, lung_volumes, unspecified, other}. “Other” is used
to reprepresnt some tailed positions.

##  Appendix B Implementation Details

Model architecture. As input to the model, images are resized into $224\times
224\times 3$. We use the first four layers of ResNet50 [24] as our visual
backbone ($\Phi_{\text{visual}}$), and adopt a MLP layer to transform the
output feature dimension into $d=256$. As a result, the output feature maps
from visual encoder is $\mathcal{V}\in\mathbb{R}^{14\times 14\times 256}$. On
the report side, we extract the triplets with a pre-trained NER module, as
described in [29], and compute the entity and position embedding with a pre-
trained ClinicalBERT [2], its default embedding dim is $d^{\prime}=768$. We
obtain $|Q|=75$ entities and $|P|=51$ positions that most frequently appear in
the reports, following [65]. We sample $M=7$ negative positions for each
entity to calculate contrastive loss. In the fusion module, We adopt 4
Transformer Decoder layers with $4$ heads.

Pre-training. At this stage, both the pre-process operation and language
encoding use pre-trained networks, while the visual encoder and fusion module
are trained end-to-end. We use AdamW [40] optimizer with $lr=1\times 10^{-4}$
and $lr_{\text{warm up}}=1\times 10^{-5}$. We train on a GeForce RTX 3090 GPU
with batch size 32 for 60 epochs. First $5$ epochs are set for warming up.

Fine-tuning. For the downstream tasks, with large amount of training data, we
can fine-tune the model end-to-end, with our pre-trained visual backbone as
initialization. Specifically, for image classification task, we adopt ResNet50
[24] and initialize its first four layers with our pre-trained visual encoder.
For image segmentation task, we use ResUNet [16] as backbone and initialize
its encoder with our pre-trained image encoder.

##  Appendix C Ablation Study

Our final method mainly contains three key parts, transformer-based entity-
query fusion module, position location contrastive loss (PosCL), and entity
translation (ET) encoding. We gradually remove the modules to analyze their
effectiveness. “w/o (ET)” refers to removing ET module and “w/o (PosCL + ET)”
refers to only maintaining the fusion module with basic CE loss. We cannot
dismiss the transformer-based entity-query fusion module as it is the most
basic module to support our pre-training. Tab. 9 and Tab. 10 shows the
quantitative results.

Entity-query Fusion module. The lines about “w/o (PosCL + ET)” in tables
demonstrate the performance of the basic model modified only by base entity
existence CE loss. This model can exceed many former methods. This proves our
assumption that the complex syntax will hurt the network to capture the useful
entities significantly and our pre-process operation and entity-level
supervision can greatly relieve the problem.

Position Contrastive Loss. The PosCL can significantly help the network to
ground the abnormalities. As shown in the results by adding PosCL the
classification results can be further improved, e.g., from $0.75$ to $0.76$ on
AUC in ChestX-ray14 dataset. Besides classification, location contrastive loss
can bring more gain in grounding. These results show position is another vital
element in reports especially for grounding tasks. Our extracted triplets can
conclude and clean the reports with little information loss and make the
network learn the report information more straightforward.

Entity Translation. By adding entity translation, we want to realize two
goals. First, in addition to just learning from the image-report data, the
network can actively learn the relationship between different entities based
on the entity descriptions. As shown in tables, adding descriptions in most
scenarios can help the network better understand the entity and bring gain to
the final metric scores. Second, more importantly, the entity translation
enables our model to handle openset new diseases. If excluding TE and
prompting the entities only with their names as former works, the performance
of our method will drop significant when facing unseen diseases which is
discussed in zero-shot classification for Covid-19 at main body.

Dataset | RSNA Pneumonia | SIIM-ACR Pneumothorax | ChestX-ray14
---|---|---|---
Methods |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$ |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$ |  AUC$\uparrow$ |  F1$\uparrow$ |  ACC$\uparrow$
w/o (PosCL + ET) | 0.8532 | 0.6079 | 0.7669 | 0.8768 | 0.6672 | 0.8187 | 0.7502 | 0.2374 | 0.8541
w/o (ET) | 0.8537 | 0.6241 | 0.8146 | 0.9017 | 0.7008 | 0.8584 | 0.7621 | 0.2452 | 0.8606
Ours | 0.8694 | 0.6342 | 0.8002 | 0.8924 | 0.6833 | 0.8428 | 0.7676 | 0.2525 | 0.8619
Table 9: Ablation study on zero-shot classification task. AUC, F1 and ACC
scores are reported. For ChestX-ray 14, the metrics all refer to the macro
average on the 14 diseases.

Methods |  Pointing Game$\uparrow$ |  Recall$\uparrow$ |  Precision$\uparrow$ |  IoU$\uparrow$ |  Dice$\uparrow$
---|---|---|---|---|---
w/o (PosCL + ET) | 0.7979 | 0.8961 | 0.4036 | 0.2783 | 0.4230
w/o (ET) | 0.8424 | 0.8226 | 0.6520 | 0.3118 | 0.4610
Ours | 0.8721 | 0.8661 | 0.6420 | 0.3172 | 0.4649
(a) Zero-shot grounding on Pneumonia

Methods |  Pointing Game$\uparrow$ |  Recall$\uparrow$ |  Precision$\uparrow$
---|---|---|---
w/o (PosCL + ET) | 0.1786 | 0.3151 | 0.1336
w/o (ET) | 0.2080 | 0.3178 | 0.1711
Ours | 0.1975 | 0.3562 | 0.1940
(b) Zero-shot grounding on Pneumothorax

Table 10: Ablation study on zero-shot grounding tasks. (a) shows the results
on RSNA Pneumonia dataset. (b) shows the results on SIIM-ACR Pneumothorax
dataset.

##  Appendix D Detailed results on ChestX-ray14

We further show the detailed performance of 14 different diseases on ChestX-
ray14 dataset. Tab. 11 shows the results on the zero-shot setting. Our method
can exceed the former methods for most diseases. The radar Fig. 3Y shows more
visually how our model compares with other solutions under the zero-shot
setting. Our method can exceed the former methods for most diseases. Under
$100\%$ fine-tuning settings, we achieved similarly excellent results as shown
in Tab. 12.

Methods | Ate. | Car. | Eff. | Inf. | Mas. | Nod. | Pna. | Pnx. | Con. | Ede. | Emp. | Fib. | Thi. | Her. | AVG
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ConVIRT [66] | 0.4533 | 0.4601 | 0.7262 | 0.6238 | 0.6790 | 0.6322 | 0.6097 | 0.6698 | 0.6855 | 0.7699 | 0.4701 | 0.5293 | 0.6098 | 0.6220 | 0.6101
GLoRIA [25] | 0.6680 | 0.7647 | 0.7975 | 0.6159 | 0.6722 | 0.5293 | 0.6755 | 0.4785 | 0.7306 | 0.8212 | 0.6033 | 0.5104 | 0.6721 | 0.7144 | 0.6610
BioViL [6] | 0.5026 | 0.6328 | 0.7914 | 0.5791 | 0.7029 | 0.6126 | 0.6866 | 0.7516 | 0.7455 | 0.8533 | 0.7136 | 0.6751 | 0.6560 | 0.7692 | 0.6909
CheXzero [54] | 0.7426 | 0.7956 | 0.8415 | 0.6223 | 0.7095 | 0.6666 | 0.7263 | 0.7679 | 0.7866 | 0.8862 | 0.6451 | 0.6402 | 0.6134 | 0.7704 | 0.7296
w/o (PosCL + ET) | 0.7131 | 0.8100 | 0.8635 | 0.6361 | 0.7776 | 0.6740 | 0.6903 | 0.8124 | 0.7915 | 0.8869 | 0.7480 | 0.6780 | 0.6429 | 0.7784 | 0.7502
w/o (ET) | 0.7420 | 0.8270 | 0.8663 | 0.6336 | 0.7867 | 0.6974 | 0.7238 | 0.8310 | 0.8037 | 0.8887 | 0.7865 | 0.6715 | 0.5414 | 0.8691 | 0.7621
ours | 0.7506 | 0.8299 | 0.8636 | 0.6280 | 0.7885 | 0.6947 | 0.7236 | 0.8361 | 0.8079 | 0.8888 | 0.7950 | 0.6511 | 0.5783 | 0.9097 | 0.7676
Table 11:  Comparison with other state-of-the-art methods on zero-shot ChestX-ray 14 diseases classification task. For each disease, AUC score is reported and the macro average AUC score is also reported. We use the first three letters to represent one disease but for “pneumonia” and “pneumothorax” we use the first two and the last letters.  Methods | Ate. | Car. | Eff. | Inf. | Mas. | Nod. | Pna. | Pnx. | Con. | Ede. | Emp. | Fib. | Thi. | Her. | AVG
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Scratch | 0.7835 | 0.8116 | 0.8563 | 0.6537 | 0.7788 | 0.6912 | 0.7004 | 0.8561 | 0.8090 | 0.8869 | 0.8564 | 0.7534 | 0.7454 | 0.9106 | 0.7924
ConVIRT [66] | 0.8012 | 0.8360 | 0.8511 | 0.6613 | 0.8004 | 0.7490 | 0.6998 | 0.8666 | 0.8079 | 0.9023 | 0.9014 | 0.7933 | 0.7468 | 0.9627 | 0.8128
GLoRIA [25] | 0.8263 | 0.8326 | 0.8596 | 0.6641 | 0.8179 | 0.7348 | 0.7104 | 0.8452 | 0.8129 | 0.8977 | 0.9310 | 0.7886 | 0.7608 | 0.9750 | 0.8184
BioViL [6] | 0.8185 | 0.8543 | 0.8607 | 0.6660 | 0.8302 | 0.7633 | 0.7090 | 0.8595 | 0.8287 | 0.9031 | 0.9251 | 0.7912 | 0.7638 | 0.9696 | 0.8245
ours | 0.8291 | 0.8594 | 0.8719 | 0.6565 | 0.8382 | 0.7647 | 0.7378 | 0.8807 | 0.8275 | 0.9083 | 0.9224 | 0.7977 | 0.7784 | 0.9796 | 0.8323
Table 12:  Comparison with other state-of-the-art methods on fine-tuning
ChestX-ray 14 diseases classification task. For each disease, AUC score is
reported and the macro average AUC score is also reported. We use the first
three letters to represent one disease but for “pneumonia” and “pneumothorax”
we use the first two and the last letters.

Figure 3: The radar figure of our method and other methods of ChestX-ray14 14
diseases. AUC scores are reported and, as shown, our method exceeds the
previous state-of-the-art on most diseases.

##  Appendix E Visualization Results

Figure 4: The visualization of zero-shot grounding results of our method. Each
column represents the results on one disease and the left in it is the ground-
truth and right is the heatmap predication of our model. The brighter the red
on the figure, the more likely the model considering this region to be
associated with abnormalities.

Fig. 4 shows visualization results of our model on zero-shot grounding task.
As shown in figure, the ground truth of “Pneumonia” is given by bounding box
and generally related to a large area region. Thus the metrics on this are
higher than other two datasets. Our network captures its regions very well.
For “Pneumothorax”, its abnormality pattern is different from other diseases,
which aim to capturing the collapsed part of the lung, rendering darker areas
on the images rather than brighter opacity. Its ground-truth masks are
generally thin and narrow while our network can still highlight its location.
For “Covid-19”, its image textual was similar to “Pneumonia”, but since this
is a totally new disease, grounding its regions is much more challenging. It
requires the model to build relationships between them based on their complex
definition and symptoms. The visualization results suggest that our model
successfully achieve this, supporting that, for other unseen diseases, our
model can also understand their complex descriptions.


# Partial-Attribution Instance Segmentation for Astronomical Source Detection
and Deblending

Ryan Hausen
Department of Computer Science and Engineering
University of California, Santa Cruz
Santa Cruz, CA 95064
<EMAIL_ADDRESS>
Brant Robertson
Department of Astronomy and Astrophysics
University of California, Santa Cruz
Santa Cruz, CA 95064
<EMAIL_ADDRESS>

###### Abstract

Astronomical source deblending is the process of separating the contribution
of individual stars or galaxies (sources) to an image comprised of multiple,
possibly overlapping sources. Astronomical sources display a wide range of
sizes and brightnesses and may show substantial overlap in images.
Astronomical imaging data can further challenge off-the-shelf computer vision
algorithms owing to its high dynamic range, low signal-to-noise ratio, and
unconventional image format. These challenges make source deblending an open
area of astronomical research, and in this work, we introduce a new approach
called Partial-Attribution Instance Segmentation that enables source detection
and deblending in a manner tractable for deep learning models. We provide a
novel neural network implementation as a demonstration of the method.

##  1 Introduction

Astronomical images can contain tens of thousands of stars and galaxies
(sources). Forthcoming telescopes including the Vera Rubin Observatory [11,
12], James Webb Space Telescope [26], and Nancy Grace Roman Space Telescope
[24, 25] will push the current limits of observational astronomy and
dramatically increase the number of sources to analyze. To measure accurate
properties for these sources, we must detect sources by identifying
statistically significant local maxima in an image and deblend sources by
isolating the potentially overlapping flux distributions of each object.
Consider a background-subtracted astronomical image
$\mathcal{I}\in\mathbb{R}^{h\times w\times b}$ in which $n$ sources are
observed, where $h$ is the height, $w$ is the width, and $b$ indicates the
number of astronomical passbands. The image $\mathcal{I}$ can be decomposed
into a sum of individual object contributions as

$\mathcal{I}=\sum_{i=1}^{N}S_{i}+\epsilon$ (1)

where $S_{i}\in\mathbb{R}^{h\times w\times b}$ represents the flux contributed
to $\mathcal{I}$ by source $i$, and $\epsilon\in\mathcal{N}(0,\sigma)$ is the
approximate noise distribution in the image. The process of decomposing an
image into the form of Equation 1 represents the core challenge of source
deblending. This submission presents a deep learning-based method to perform
detection and deblending on astronomical images.

###  1.1 Related Work

Source detection and deblending are well-studied problems in astronomy, and
many approaches have been developed. Below, we highlight some popular and
recent methods for source detection and deblending and point the interested
reader to the review by Masias et al. [18].

Detection and deblending methods can be characterized by their detection
capacity and deblend type. The detection capacity represents the number of
sources a method can detect within a single image. For Equation 1, a detection
capacity of $N$ would indicate that a method could detect all sources
appearing in an image. The deblend type indicates how the flux in a single
pixel may be split between overlapping (blended) sources. A disjoint deblender
assigns all flux in a pixel to a single source exclusively. An
intersecting/discrete deblender can assign the flux to more than one source
with uniform weighting across pixels. Finally, an intersecting/continuous
deblender can assign the flux to more than one source with variable weighting
across pixels.

Astronomical analysis methods vary in their detection and deblending methods.
Bertin & Arnouts [2] introduced SExtractor that uses a convolution and
thresholding approach for detection, and an isophotal analysis using binned
pixel intensity for deblending. Hausen & Robertson [7] introduced Morpheus, a
U-Net [23] style convolutional neural network (CNN) model that filters out
background pixels, uses a thresholding approach for detection, and combines
watershed and peak finding algorithms for deblending. Another U-Net based
model called blend2mask [3] performs detection and deblending using the U-Net
alone. Reiman & Göhre [22] use a modified Super-Resolution GAN (SRGAN) [16] to
deblend overlapping sources. Burke et al. [4] trained a Mask R-CNN [8] model
to detect and deblend sources. SCARLET [19] deblends sources using constrained
matrix factorization.

Table 1 summarizes the features of these previous methods, none of which have
a detection capacity of $N$ and an intersecting/continuous deblend type. We
now present a deep learning-based intersecting/continuous deblending algorithm
with a detection capacity of $N$.

Table 1: Detection and deblending method categorization Name | Detection Capacity | Deblend Type
---|---|---
SExtractor[2] | $N$ | Disjoint
Morpheus[7] | $N$ | Disjoint
Mask R-CNN[4] | $N$ | Intersecting/Discrete
blend2mask2flux[3] | $2$ | Intersecting/Discrete
Modified SRGAN[22] | 0 | Intersecting/Continuous
SCARLET[19] | 0 | Intersecting/Continuous
This Work | $N$ | Intersecting/Continuous

##  2 Partial-Attribution Instance Segmentation

Partial-Attribution Instance Segmentation (PAIS) is a new extension of the
instance segmentation paradigm that allows for weighted, overlapping
segmentation maps. PAIS differs from other segmentation schemes like cell
segmentation [28], interacting surface segmentation [27], and amodal instance
segmentation [17]. PAIS aims to isolate objects appearing in an image while
preserving their measurable quantities within areas of overlap. For PAIS, we
can approximate Equation 1 as

$\tilde{\mathcal{I}}=\sum_{i=1}^{N}M_{i}\odot\mathcal{I}$ (2)

where $\tilde{\mathcal{I}}\in\mathbb{R}^{h\times w\times b}$ estimates the
background-subtracted flux image $\mathcal{I}$ in Equation 1,
$M_{i}\in[0,1]^{h\times w\times b}st.\sum_{i}^{N}M_{i,jkl}=1$ constitutes the
pixel-level fractional contribution of source $i$ to $\mathcal{I}$, and
$\odot$ symbolizes the Hadamard product. Equation 2 is tractable for deep
learning models, allowing the model to learn the bounded quantities $M_{i}$
rather than the unbounded source images $S_{i}$. The $N$ number of sources
setting the upper limit of the sum in Equation 2 can differ for each image.

To construct a PAIS format that can be represented by a CNN, we have to
construct an encoding for the $M_{i}$ in Equation 2. Inspired by Cheng et al.
[5] and Kendall et al. [13], we propose an encoding for the $M_{i}$ components
called Partial Contribution Representation (PCR). The goal of PCR is to
encode, for any single pixel $(j,k,l)$, the fractional contribution to its
intensity from the closest $n$ sources. Using PCR, a variable number $N$ of
sources can be encoded per image. PCR consists of three tensors: the Center-
of-mass $C^{c}\in\\{0,1\\}^{h\times w}$, Contribution-vectors
$C^{v}\in\mathbb{R}^{h\times w\times n\times 2}$ and Contribution-maps
$C^{m}\in[0,1]^{h\times w\times b\times n}$. The center-of-mass encodes the
locations of all the sources in an image. For any pixel, we set $C^{c}_{jk}=1$
if that location indicates the center of a source and $C^{c}_{jk}=0$
otherwise. The contribution-vector $C^{v}_{jk}$ encodes the Cartesian offset
to the closest $n$ sources. The contribution-map $C^{m}_{jkl}$ connects the
fractional contribution of the $n$ sources with the associated contribution-
vectors $C^{v}_{jk}$. The fixed dimensionality of $C^{c}$, $C^{v}$, and
$C^{m}$ make PCR tractable for deep learning algorithms.

##  3 Our Approach

Our approach consists of making a PAIS dataset leveraging PCR and is
implemented using a novel neural network architecture. We summarize our
dataset, model, and training method below.

###  3.1 Dataset

To generate the PAIS input samples, we used the Hubble Legacy Fields (HLF)
GOODS-South F160W ($1.6\mu\mathrm{m}$) flux images [10], along with the 3D-HST
source catalog [20]. The HLF images were split into training and test sets of
$256\times 256$ pixel subregions, with 2,000 training samples and 500 test
samples. The input labels, as described in Section 2, consist of the center-
of-mass images $C^{c}$, the contribution-vectors $C^{v}$, and the
contribution-maps $C^{m}$. The center-of-mass training images are generated in
a manner similar to Cheng et al. [5], by placing pixelated 2D Gaussians with
standard deviation $\sigma=8(\textrm{pixels})$ at the locations of sources in
the 3D-HST catalog. The contribution-vectors, an extension to the method by
Cheng et al. [5], are generated from the Cartesian offset to the nearest $n=3$
sources to each pixel. The contribution-maps require the $M_{i}$ values from
Equation 2. To determine $M_{i}$, we use SCARLET [19] with the F125W, F160W,
F606W, and F850LP flux and weight images from the HLF GOODS-South data and the
TinyTim point-spread functions [15] to deblend the sources from the 3D-HST
catalog. We then use PCR to encode the $M_{i}$ from SCARLET. The complete
dataset generation routine can be found in our project repository
(https://github.com/ryanhausen/morpheus-deblend/ ).

To evaluate the efficacy of PCR to encode $M_{i}$, we define two metrics. We
use the mean difference between the total flux determined by the SCARLET
encoded $M_{i}$ for each input source and that recovered by our encoding. We
also use a two-sample Kolmogorov–Smirnov (KS) test to compare the normalized
cumulative surface brightness profile within the radius encompassing 90% of
the total flux of each source to evaluate the encoding of the spatial flux
distribution. Table 2 reports the results and demonstrates that PCR encoding
approximately preserves both the total flux and the spatial flux distribution
for each source. With this verification, we can train a network to recover the
PCR encoding for each input HST F160W image.

Table 2: Partial Contribution Representation encoding efficacy Test | Value
---|---
Total Source Flux [e/s] (MAE) | $1.97\pm 15.43$
Two-Sample KS Test p-value | $0.93\pm 0.22$

###  3.2 Model

To recover the PCR for training images, we developed a novel neural network
architecture inspired by Cheng et al. [5], based on the Fast Attention Network
[9] and implemented in TensorFlow [1]. The model features two decoders that
share a single encoder. The first decoder, called the spatial decoder,
predicts values for $C^{c}$ and $C^{v}$. The second decoder, called the
attribution decoder, predicts values for $C^{m}$. The complete model code can
be found in the repository for this project
(https://github.com/ryanhausen/morpheus-deblend/ ). An end-to-end example of
the model can be seen in Figure 1.

Figure 1: End-to-end example using our method to detect and deblend sources.
Starting from the left: A flux image is input to the Model (see Section 3.2).
The Model outputs the deblended image in the Partial Contribution
Representation (PCR; see Section 2).The output from the model is then decoded
using the non-learned PCR Decoder algorithm into separate deblended source
images. The deblended source images have their total flux within $r_{90}$
annotated. The deblended source images are then added together to generate the
reconstructed image which has an $L_{1}$ total flux difference of $1.52\times
10^{-5}$ with the original input image.

###  3.3 Training

To train the model to recover the PCR of the input images, we use the Adam
Optimizer [14] with a learning rate of $5\times 10^{-5}$, $\beta_{1}=0.9$,
$\beta_{2}=0.9999$, $\epsilon=1\times 10^{-7}$, and a batch size of 100\. The
model was trained for 1000 epochs using an NVIDIA V100 32GB GPU, taking 31
hours. The loss function for the model is composed of three functions. The
spatial decoder outputs for $C^{c}$ and $C^{v}$ are penalized according to
mean squared error (MSE) and the mean absolute error (MAE), respectively. The
attribution decoder output $C^{m}$ is penalized using cross-entropy loss with
an additional entropy regularization term. In practice, we found that the
additional entropy regularization helped incentivize the network to learn
information about multiple sources in $C^{m}$. Each loss term is weighted and
combined into a single loss function described by

$\mathcal{L}_{\mathrm{total}}=\lambda_{C^{c}}\mathcal{L}_{C^{c}}+\lambda_{C^{v}}\mathcal{L}_{C^{v}}+\lambda_{C^{m}}\mathcal{L}_{C^{m}}+\lambda_{S}\mathcal{L}_{S},$
(3)

where $\mathcal{L}_{C^{c}}$ is the MSE loss calculated between the model
output and input label with $\lambda_{C^{c}}=15$, $\mathcal{L}_{C^{v}}$ is the
MAE loss calculated between the model output and input label with
$\lambda_{C^{v}}=0.06$, $\mathcal{L}_{C^{m}}$ is a cross-entropy loss
calculated between the model output and input label with $\lambda_{C^{m}}=4$,
and $\mathcal{L}_{S}$ is the entropy regularization on the model $C^{m}$
output with $\lambda_{S}=2$. See Table 3 for a summary of the training
results, demonstrating a good balance between test and training error. A
complete log of training experiments is available at
(https://www.comet.ml/ryanhausen/morpheus-deblend/ ).

Table 3: Training metric results Metric | Training | Test
---|---|---
MAE | $27.0183\pm 1.0658$ | $28.5090\pm 0.3386$
MSE | $0.0114\pm 0.0001$ | $0.0124\pm 0.0006$
cross-entropy | $0.9485\pm 0.0069$ | $1.0806\pm 0.0098$

##  4 Discussion and Future Work

In this work, we introduced the Partial Attribution Instance Segmentation
(PAIS) scheme for astronomical source debelending. We presented Partial
Contribution Representation (PCR) as a method for implementing PAIS within
deep learning-based models. We demonstrated the efficacy of PCR for encoding
the results of existing astronomical deblenders, and developed a novel neural
network architecture to recover the PCR from input flux images. While we
demonstrated deblending for single band (F160W) images, PCR can be extended to
multiband images. As with many supervised methods, our model requires labeled
training data. To apply this method on other survey datasets may require the
use of transfer learning [6, 21] or retraining.

## Acknowledgments and Disclosure of Funding

##  5 Acknowledgements

RDH would like to thank Roberto Manduchi for helpful conversations. BER
acknowledges support from NASA contract NNG16PJ25C and grant 80NSSC18K0563.
The authors acknowledge use of the lux supercomputer at UC Santa Cruz, funded
by NSF MRI grant AST 1828315.

##  6 Broader Impact

This work develops a novel method for separating source signals in
astronomical images. Due to the specialized format and problem setting, the
authors do not see any broader negative societal impacts as a result of this
work.

## References

  * Abadi et al. [2016] Abadi, M., Agarwal, A., Barham, P., et al. 2016, arXiv e-prints, arXiv:1603.04467.  https://arxiv.org/abs/1603.04467
  * Bertin & Arnouts [1996] Bertin, E., & Arnouts, S. 1996, A&AS, 117, 393, doi: 10.1051/aas:1996164
  * Boucaud et al. [2020] Boucaud, A., Huertas-Company, M., Heneka, C., et al. 2020, MNRAS, 491, 2481, doi: 10.1093/mnras/stz3056
  * Burke et al. [2019] Burke, C. J., Aleo, P. D., Chen, Y.-C., et al. 2019, MNRAS, 490, 3952, doi: 10.1093/mnras/stz2845
  * Cheng et al. [2019] Cheng, B., Collins, M. D., Zhu, Y., et al. 2019, arXiv e-prints, arXiv:1911.10194.  https://arxiv.org/abs/1911.10194
  * Domínguez Sánchez et al. [2019] Domínguez Sánchez, H., Huertas-Company, M., Bernardi, M., et al. 2019, MNRAS, 484, 93, doi: 10.1093/mnras/sty3497
  * Hausen & Robertson [2020] Hausen, R., & Robertson, B. E. 2020, ApJS, 248, 20, doi: 10.3847/1538-4365/ab8868
  * He et al. [2017] He, K., Gkioxari, G., Dollár, P., & Girshick, R. 2017, arXiv e-prints, arXiv:1703.06870.  https://arxiv.org/abs/1703.06870
  * Hu et al. [2020] Hu, P., Perazzi, F., Caba Heilbron, F., et al. 2020, arXiv e-prints, arXiv:2007.03815.  https://arxiv.org/abs/2007.03815
  * Illingworth et al. [2016] Illingworth, G., Magee, D., Bouwens, R., et al. 2016, arXiv e-prints, arXiv:1606.00841.  https://arxiv.org/abs/1606.00841
  * Ivezić et al. [2008] Ivezić, Ž., Kahn, S. M., Tyson, J. A., et al. 2008, arXiv e-prints.  https://arxiv.org/abs/0805.2366
  * Ivezić et al. [2019] Ivezić, Ž., Kahn, S. M., Tyson, J. A., et al. 2019, ApJ, 873, 111, doi: 10.3847/1538-4357/ab042c
  * Kendall et al. [2017] Kendall, A., Gal, Y., & Cipolla, R. 2017, arXiv e-prints, arXiv:1705.07115.  https://arxiv.org/abs/1705.07115
  * Kingma & Ba [2014] Kingma, D. P., & Ba, J. 2014, ArXiv e-prints.  https://arxiv.org/abs/1412.6980
  * Krist et al. [2011] Krist, J. E., Hook, R. N., & Stoehr, F. 2011, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 8127, Optical Modeling and Performance Predictions V, ed. M. A. Kahan, 81270J, doi: 10.1117/12.892762
  * Ledig et al. [2016] Ledig, C., Theis, L., Huszar, F., et al. 2016, arXiv e-prints, arXiv:1609.04802.  https://arxiv.org/abs/1609.04802
  * Li & Malik [2016] Li, K., & Malik, J. 2016, arXiv e-prints, arXiv:1604.08202.  https://arxiv.org/abs/1604.08202
  * Masias et al. [2012] Masias, M., Freixenet, J., Lladó, X., & Peracaula, M. 2012, MNRAS, 422, 1674, doi: 10.1111/j.1365-2966.2012.20742.x
  * Melchior et al. [2018] Melchior, P., Moolekamp, F., Jerdee, M., et al. 2018, Astronomy and Computing, 24, 129, doi: 10.1016/j.ascom.2018.07.001
  * Momcheva et al. [2016] Momcheva, I. G., Brammer, G. B., van Dokkum, P. G., et al. 2016, ApJS, 225, 27, doi: 10.3847/0067-0049/225/2/27
  * Pratt [1993] Pratt, L. Y. 1993, in Advances in Neural Information Processing Systems 5, ed. S. J. Hanson, J. D. Cowan, & C. L. Giles (Morgan-Kaufmann), 204–211.  http://papers.nips.cc/paper/641-discriminability-based-transfer-between-neural-networks.pdf
  * Reiman & Göhre [2019] Reiman, D. M., & Göhre, B. E. 2019, MNRAS, 485, 2617, doi: 10.1093/mnras/stz575
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., & Brox, T. 2015, arXiv e-prints, arXiv:1505.04597.  https://arxiv.org/abs/1505.04597
  * Spergel et al. [2013] Spergel, D., Gehrels, N., Breckinridge, J., et al. 2013, arXiv e-prints, arXiv:1305.5422.  https://arxiv.org/abs/1305.5422
  * Spergel et al. [2015] Spergel, D., Gehrels, N., Baltay, C., et al. 2015, arXiv e-prints, arXiv:1503.03757.  https://arxiv.org/abs/1503.03757
  * Williams et al. [2018] Williams, C. C., Curtis-Lake, E., Hainline, K. N., et al. 2018, ApJS, 236, 33, doi: 10.3847/1538-4365/aabcbb
  * Xie et al. [2020] Xie, H., Pan, Z., Zhou, L., et al. 2020, arXiv e-prints, arXiv:2007.01259.  https://arxiv.org/abs/2007.01259
  * Zhou et al. [2020] Zhou, Y., Chen, H., Lin, H., & Heng, P.-A. 2020, arXiv e-prints, arXiv:2007.10787.  https://arxiv.org/abs/2007.10787