# DrasCLR: A Self-supervised Framework of Learning Disease-related and
Anatomy-specific Representation for 3D Medical Images

Ke Yu∗
School of Computing and Information
University of Pittsburgh
<EMAIL_ADDRESS>
&Li Sun∗
Department of Electrical and Computer Engineering
Boston University
<EMAIL_ADDRESS>
Junxiang Chen
Department of Biomedical Informatics
University of Pittsburgh
&Max Reynolds
Department of Biomedical Informatics
University of Pittsburgh
&Tigmanshu Chaudhary
Department of Biomedical Informatics
University of Pittsburgh

&Kayhan Batmanghelich
Department of Electrical and Computer Engineering
Boston University

###### Abstract

Large-scale volumetric medical images with annotation are rare, costly, and
time prohibitive to acquire. Self-supervised learning (SSL) offers a promising
pre-training and feature extraction solution for many downstream tasks, as it
only uses unlabeled data. Recently, SSL methods based on instance
discrimination have gained popularity in the medical imaging domain. However,
SSL pre-trained encoders may use many clues in the image to discriminate an
instance that are not necessarily _disease-related_. Moreover, pathological
patterns are often subtle and heterogeneous, requiring the ability of the
desired method to represent _anatomy-specific_ features that are sensitive to
abnormal changes in different body parts. In this work, we present a novel SSL
framework, named DrasCLR, for 3D medical imaging to overcome these challenges.
We propose two domain-specific contrastive learning strategies: one aims to
capture subtle disease patterns inside a local anatomical region, and the
other aims to represent severe disease patterns that span larger regions. We
formulate the encoder using conditional hyper-parameterized network, in which
the parameters are dependant on the anatomical location, to extract
anatomically sensitive features. Extensive experiments on large-scale computer
tomography (CT) datasets of lung images show that our method improves the
performance of many downstream prediction and segmentation tasks. The patient-
level representation improves the performance of the patient survival
prediction task. We show how our method can detect emphysema subtypes via
dense prediction. We demonstrate that fine-tuning the pre-trained model can
significantly reduce annotation efforts without sacrificing emphysema
detection accuracy. Our ablation study highlights the importance of
incorporating anatomical context into the SSL framework.

**footnotetext: These authors contributed equally to this work

_K_ eywords Self-supervised learning $\cdot$ Contrastive learning $\cdot$
Label-efficient learning $\cdot$ 3D Medical imaging data

##  1 Introduction

While deep learning approaches have significantly advanced computer vision and
many other fields [1, 2, 3], efforts to apply these advancements to medical
image analysis are still hampered by the scarcity of large-scale annotated
datasets. Annotating medical images requires domain expertise and is a
laborious and costly process, especially for 3D volumetric medical data.
However, massive amounts of unlabeled raw images have been collected and
stored in hospitals’ picture archiving and communication systems (PACS) for
decades. Recently, self-supervised learning (SSL) has become increasingly
popular as a way to alleviate the annotation burden by exploiting the readily
available unlabeled data [4, 5]. However, unlike supervised approaches, which
use experts’ annotations (e.g., disease labels, lesion segmentation masks) as
supervision, self-supervised models are trained with limited supervision
derived from the data itself, making it far more difficult to identify
disease-related features from the data. Furthermore, certain lesions (e.g.,
early-stage tumors) may occupy only a small region in high-resolution
volumetric medical images, and their visual patterns may vary depending on
where they are located in the body. Thus, the desired self-supervised learning
algorithm should be sensitive enough to capture local anatomical deformities.
In this research, we propose DrasCLR: a novel framework for self-supervised
learning of disease-related and anatomy-specific representation of 3D medical
imaging. DrasCLR learns a patch-based dense representation that conditionally
depends on the anatomical location of the center voxel. We extensively
evaluate our method on chest computed tomography (CT) imaging because of its
prominent role in the prevention, diagnostics and treatment of lung diseases.

Self-supervised learning methods aim to provide useful feature representations
for downstream tasks without human supervision, which is typically achieved by
optimizing the model to solve a _proxy_ task. When designing a proxy task, the
primary consideration is: _what information in the data is important and what
is not to the downstream tasks?_ Early self-supervised approaches use
heuristic-based pretext tasks to learn representations invariant to
transformations that do not change the semantic meaning of the target labels
[6, 7, 8]. More recent contrastive learning approaches [9, 10] use _instance
discrimination task_ , which consider every instance as a class of its own and
train deep neural networks to discriminate pairs of similar inputs (augmented
views of the same instance) from a selection of dissimilar pairs (different
instances). In this setting, data augmentation guided by prior knowledge often
plays a vital role in preserving task-relevant information [11]. The sampling
strategy for negative pairs is also crucial for the performance of contrastive
learning methods. Recent studies [12, 13] show that hard negative sampling
guided by domain knowledge helps in preventing trivial solutions and improving
the alignment of extracted features with human semantics.

Self-supervised representation learning of disease-related features in medical
images is particularly challenging for two reasons. First, since disease-
related features are often represented through subtle changes, an effective
self-supervised learning method should be able to ignore large but irrelevant
and non-informative information and focus on representing fine-grained
features [14]. More specifically, this requires the encoder to be capable of
representing small deviations from the appearance of normal anatomy. Second,
because pathological tissues may only scatter in a few small regions,
adequately representing local content is crucial for dense (voxel-level)
prediction tasks such as anomaly detection and segmentation. Several self-
supervised learning methods [15, 16, 17] have been developed to learn local
representations of 3D medical images. These methods use sub-volumes sampled
from random locations in the image as inputs and train a single encoder with
parameters shared across all locations. However, disease types and their
visual patterns are often associated with anatomical locations. For example,
pulmonary emphysema can be divided into three major subtypes (i.e.,
centrilobular, paraseptal, and panlobular) based on their visual
characteristics and anatomical locations within the lung [18]. A more
sophisticated framework for learning local representations should incorporate
anatomical locations as prior information to account for the spatial
heterogeneity of anatomical and pathological patterns.

In this research, we take inspiration from the aforementioned challenges and
propose a novel contrastive learning framework for 3D medical imaging. In
order to represent disease-related imaging features, we propose to combine two
domain-specific contrasting strategies. The first strategy leverages the
similarity across patients at the same anatomical location and aims to
represent small disease patterns within a local (anchor) region. The second
strategy takes advantage of anatomical similarities between the anchor and its
nearby anatomical regions, with the goal of compensating for the first
strategy by learning larger disease patterns that expand beyond the local
region. We use small 3D patches to represent local anatomical regions. The
effectiveness of both strategies depends on the difficulty of instance
discrimination; as the anatomical similarity between the query and negative
patches becomes greater, the encoder is forced to rely on subtle and disease-
related features rather than normal anatomical features. To that end, we use
image registration to obtain _hard negative_ patches from different subjects
that are anatomically best aligned to the query patch. In particular, we
obtain point-by-point correspondence between image pairs by mapping them to
the same anatomical atlas. The coordinates in the atlas image can then be
viewed as a standard set of anatomical locations. To incorporate anatomical
locations into learned representations, we further develop a novel 3D
convolutional layer whose kernels are conditionally parameterized through a
routing function that takes the coordinates in atlas space as inputs. We call
our unified framework Disease-related anatomy-specific Contrastive Learning
Representation (DrasCLR). The overview of our proposed approach is illustrated
in Fig. 1. We conduct experiments on large-scale lung CT datasets. The results
empirically show that our method outperforms baseline methods on both image-
level and voxel-level tasks. Our code and pre-trained weights will be publicly
available.

Figure 1: Schematic diagram of DrasCLR. Left panel: We represent a volumetric
image with a collection of 3D patches registered with distinctive anatomical
landmarks defined in an atlas image. We develop an encoder that generates
location-specific representation using the patch and location of associated
anatomical landmark as inputs. Our contrastive learning framework comprises
two objectives. Middle panel: The first one aims to learn local representation
from a single patch. Right panel: The second one aims to learn representations
of larger patterns across neighboring patches. Both contrasting strategies
incentivize the encoder to learn disease-related features by using patches of
similar anatomy as _hard_ negative samples.

In summary, the major contributions of this paper are:

  1. 1.

We propose a novel framework for contrastive learning of disease-related
representation for 3D medical images.

  2. 2.

We propose a novel 3D convolutional layer that encodes anatomical location-
dependent information.

  3. 3.

We extensively validate our model on large-scale lung CT datasets and show
that our method outperforms existing baselines for a wide range of image-level
tasks.

  4. 4.

We demonstrate the application of our method for voxel-wise emphysema
detection and show that using our pre-trained model can significantly cut
annotation costs without compromising detection accuracy.

The paper is organized as follows: we first briefly introduce the related
works in Section 2. In Section 3, we present the details of our proposed
methodology. Implementation details and experimental results are described in
Sections 4 and 5. Finally, we discuss the key findings and limitations of our
work in Section 6 followed by conclusions in Section 7.

##  2 Related Work

We review related works in four areas: (1) self-supervised learning
approaches, including pretext task-based methods and contrastive learning
methods, (2) applications of self-supervised learning in medical image
analysis, (3) self-supervised learning methods that exploit anatomical context
in medical images, and (4) conditionally parameterized networks.

###  2.1 Self-supervised Learning

Self-supervised learning has been shown to be an effective approach for
learning semantically useful representations from large-scale unlabeled data
without requiring human annotation [4, 5]. To generate supervisory signals
from the data itself, a popular strategy is to present the model with various
pretext tasks to solve. Commonly used pretext tasks include image inpainting
[19], image colorization [7], relative position prediction [6], image jigsaw
puzzle [20], patch cut and paste [21], temporal order verification [22],
geometric transformation recognition [8], cross-modal correspondence [23, 24],
and so on. These methods all have one thing in common: they build predictive-
based pretext tasks using data’s inherent structures, such as context
similarity, spatial correlation, and temporal order. High-level semantic
features are extracted in the process of accomplishing these tasks.

More recently, contrastive learning methods have emerged as one of the most
popular self-supervised approaches due to their empirical success in computer
vision [25, 26]. The objective of contrastive training is to push learned
representations to be similar for positive (similar) pairs and dissimilar for
negative (dissimilar) pairs. This task is called _instance discrimination_
[27] and is often formulated using the InfoNCE loss [28, 29]. A variety of
contrastive learning frameworks have been proposed, such as SimCLR [9], which
uses the augmented view of the same input as positive samples and the
augmented views of other samples in a minibatch as negative samples, and MoCo
[10], which uses a slow moving average (_momentum_) encoder and a dictionary
that stores old negative representations to enable constructing very large
batches of negative pairs. We extend MoCo as our contrastive learning
paradigm. Rather than using a global dictionary, we develop a conditional
memory bank that maintains distinct dictionaries of negative representations
for each anatomical location. The design of sampling strategies for positive
and negative pairs is a key driver to the success of contrastive learning [30,
31]. Previous studies have demonstrated that encoders trained with harder
negative pairs can represent more challenging features [12, 13, 32, 33, 34].
We create negative pairs from examples with highly similar local anatomy to
force the model to solve instance discrimination using more subtle visual
features (e.g. deviation from normal appealing tissues).

###  2.2 SSL Applications in Medical Imaging

Self-supervised learning is particularly useful for medical imaging analysis,
in which labels are expensive to collect. Several studies have shown the
effectiveness of self-supervised approaches in a variety of medical imaging
analysis tasks such as disease diagnosis [35, 36, 37], detection and
localization [38, 39], image segmentation [16, 40, 41, 42], and image
registration [43]. Contrastive learning frameworks have been employed to
leverage large-scale, unlabeled medical imaging data to produce pre-trained
models. For example, [44] adopted MoCo as a pre-training approach to obtain
high-quality representations for detecting diseases in chest X-rays; [37]
extended SimCLR to train robust representations for dermatology condition
classification and thoracic disease classification by using the availability
of multiple views of the same pathology from the same patients. Our
contrastive learning framework is built upon MoCo and is specifically designed
to learn voxel-level representations that are sensitive to local anatomical
deformities.

###  2.3 Leveraging Anatomical Structure in SSL

Medical images have consistent anatomy across patients, providing domain-
specific cues for self-supervised representation learning. [45] introduced the
Models Genesis, which learns image representation by recovering anatomical
patterns from transformed sub-volumes extracted from CT images. [17] extended
the Models Genesis framework by adding a self-classification objective to
enable the encoder to learn common anatomical semantics at similar body
locations across patients. [46] presented an anatomical position prediction
task for learning segmentation features from cardiac magnetic resonance
images. [16] enhanced SimCLR by integrating two domain-specific contrasting
strategies: incentivizing similar representations for volumetric image slices
coming from similar anatomical areas, and incentivizing distinctive local
representations for different anatomical regions coming from the same image.
Our approach differs from the reviewed methods in two aspects: (1) We use
image registration to improve the alignment of anatomical structures across
patients, and (2) We leverage local anatomical similarity to create _hard
negative_ samples to gain more information about tissue abnormalities.

###  2.4 Conditionally Parameterized Networks

In comparison to conventional neural networks with fixed weights,
conditionally parameterized networks have weights computed as a function of
the inputs. One such network is hypernetwork [47], which is typically a small
network that outputs weights for a primary network. It has been used in
functional image representation [48] and hyperparameter optimization [49, 50,
51]. Another prominent conditional parameterization technique is CondConv
[52], which creates convolutional kernels as a linear combination of experts
with scalar weights dependent on input images. In our framework, the CondConv
is modified to compute convolutional kernels as a function of the input
anatomical locations. As a result, the learned representation at a given voxel
depends on both its surrounding patch and its anatomical location.

##  3 Method

We propose DrasCLR, a novel contrastive learning framework for 3D medical
data. Our goal is to learn location-specific representations that are
sensitive to tissue abnormalities. We start by aligning images to an
_anatomical atlas_ using image registration and treating the image of each
patient as a collection of 3D patches centered at a common set of anatomical
locations. Our contrasting strategies are motivated by two domain-specific
similarity cues: one leverages the similarity between patients at the same
anatomical location, and the second leverages the similarity between nearby
anatomical locations on the same image. In the following sections, we explain
each component separately. The schematic diagram of the proposed method is
shown in Fig. 1. The notations used in this paper are summarized in Table 1.

Table 1: Important notations in this paper. Models
---
$e(\cdot,\cdot;\theta_{1})$ | Image encoder.
$g(\cdot;\theta_{2})$ | MLP projection head.
$f(\cdot,\cdot;\theta)$ | Network composed of $e$ and $g$, where $\theta=\\{\theta_{1},\theta_{2}\\}$.
$r(\cdot)$ | Routing function used in Loc-CondConv.
Functions
$\phi_{i}(\cdot)$ | Transformation from the $i$-th image to the atlas space.
$\phi^{-1}_{i}(\cdot)$ | Inverse transformation from atlas space to the $i$-th image.
$\tilde{t}(\cdot)$ | Random augmentations.
Variables
$p^{j}$ | Location of the $j$-th anatomical landmark in the atlas space.
$p_{i}^{j}$ | Location of the $j$-th anatomical landmark mapped in the $i$-th subject.
$x_{i}^{j}$ | Patch of the $i$-th subject centering at the $j$-th anatomical landmark.
$y_{i}^{j}$ | Representation of $j$-th patch in the $i$-th image used in downstream tasks.
$y_{i}$ | Representation of the $i$-th image used in downstream tasks.
$q_{i}^{j}$ | MoCo embedding of $j$-th patch in the $i$-th query image.
$k_{+}$ | MoCo embedding of the positive sample.
$k_{-}$ | MoCo embedding of the negative sample.
$r_{i}^{l}$ | MoCo embedding of $l$-th neighboring patch in the $i$-th image.
${\mathcal{N}(j)}$ | Neighboring patches of the $j$-th patch.
$X_{Atlas}$ | The atlas image.

###  3.1 Anatomical Alignment via Image Registration

We represent each volumetric image as a collection of 3D patches centered at a
standard set of anatomical locations pre-defined on an anatomical atlas, with
each patch corresponding to a distinct anatomical region of the lung. To align
anatomical structures among patients, we first choose an image of a healthy
subject to serve as the anatomical atlas, and then use image registration to
obtain the subject-specific transformations that establish the point-by-point
correspondence between the patients’ images and the atlas image. Let
$X_{\text{Atlas}}$ denote the atlas image, $x_{i}$ denote the image of patient
$i$, the transformation $\phi_{i}$ is obtained by solving the optimization
problem as follows:

$\operatorname*{argmin}_{\phi_{i}}\hskip
1.49994pt\texttt{Sim}\big{(}\phi_{i}(x_{i}),X_{\text{Atlas}}\big{)}+\texttt{Reg}(\phi_{i}),$
(1)

where $\texttt{Sim}(\cdot,\cdot)$ is mutual information similarity function
and $\texttt{Reg}(\phi_{i})$ is a regularization term to ensure the
transformation is smooth. We perform the image registration using the Advanced
Neuroimaging Tools (ANTs) [53].

After registration, we divide the lung region of the atlas image into $J$
evenly spaced three-dimensional patches with some overlap and define the
patches’ centers as the anatomical landmarks, denoted by
$\\{p^{j}\\}_{j=1}^{J}$, where $j$ is the patch index and each
$p^{j}\in\mathbb{R}^{3}$ is a coordinate in the _Atlas space_. We apply the
inverse transformation $\phi^{-1}_{i}$ to locate the anatomical landmarks on
each patient’s image and extract the corresponding patches for training.
Formally, each patient’s image $x_{i}$ is partitioned into a set of patches
$\\{x_{i}^{j}\\}_{j=1}^{J}$ centered at $\\{p_{i}^{j}\\}_{j=1}^{J}$,
respectively, where $x_{i}^{j}\in\mathbb{R}^{d\times d\times d}$,
$p_{i}^{j}=\phi_{i}^{-1}(p^{j})$ and $d$ is the dimension of patch. It is
straightforward to show that patches with the same index across all patients
correspond to the same anatomical region on the anatomical atlas:

$\phi_{i}(p_{i}^{j})=\phi_{i}\big{(}\phi_{i}^{-1}(p^{j})\big{)}=p^{j}.$ (2)

###  3.2 Conditionally Parameterized Convolutional Layer

Image patches from different anatomical locations have distinctive anatomical
features and may be associated with different diseased tissue patterns.
Standard convolutional layers that apply the same kernels throughout the
entire image may not be sufficient to accommodate spatial heterogeneity among
patches at different locations. Inspired by CondConv [52], we propose Loc-
CondConv, a location-dependent, conditionally parameterized convolutional
layer. Instead of using static convolutional kernels, we compute convolutional
kernels as a function of the anatomical location. In particular, we
parameterize the kernels in Loc-CondConv as a linear combination of $n$
convolutional kernels:

$W=\alpha_{1}W_{1}+\cdots+\alpha_{N}W_{N},$ (3)

where $\\{W_{n}\\}_{n=1}^{N}$ are the same-sized convolutional kernels as in
the regular convolutional layer and $\\{\alpha_{n}\\}_{n=1}^{N}$ are scalar
weights computed via a routing function taking anatomical location as input.
Specifically, we construct the routing function $r(\cdot)$ using a fully-
connected layer followed by a Sigmoid activation function:

$r(p^{j})=\sigma(p^{j}\times W_{r}),$ (4)

where $p^{j}$ is a coordinate in the _Atlas space_ and $W_{r}$ is a learnable
weight matrix with dimension $3\times N$, and $\sigma$ represents the sigmoid
function. Fig. 2 illustrates the architecture of Loc-CondConv. In the DrasCLR
models, we replace all static convolutional layers with Loc-CondConv layers.

Figure 2: The architecture of the Loc-CondConv layer. The kernels $W$ are
conditionally parameterized for each anatomical location $p^{j}$. The symbols
$\alpha_{n}$ denote the routing weights. $x$ denotes the input from the
previous layer.

###  3.3 Local Contrastive Loss

In contrastive learning, the model is trained to discriminate pairs of
positive inputs from a selection of negative pairs. Recent studies show that
selecting harder negative pairs is critical for the success of contrastive
learning [30, 32]. The anatomical similarity between patients in the same lung
region provides domain-specific cues for selecting hard negatives. More
specifically, after registration alignment, any pair of patches centered at
the same anatomical landmark, e.g. $x_{i}^{j},x_{v}^{j}$ ($i\neq v$), have
highly similar local anatomy, forcing the encoder to discriminate them using
more subtle visual features, such as pathological tissues, rather than
shortcuts, such as the overall anatomical background or boundaries.

With this motivation, we propose a _local contrasting strategy_. Formally,
given a patch $x_{i}^{j}$, we generate two augmented views
$\tilde{x}_{i}^{j}=\tilde{t}(x_{i}^{j})$, where $\tilde{t}$ is random
augmentations sampled from a set of transformations $\mathcal{T}$. These two
augmented patches are considered as a positive pair. Each negative sample
$\tilde{x}_{v}^{j}$ is generated as $\tilde{x}_{v}^{j}=\tilde{t}(x_{v}^{j})$
by randomly sampling a patch in the same anatomical region $j$ from a
different patient ($v\neq i$) and random augmentations
$\tilde{t}\sim\mathcal{T}$. We adopt the MoCo [10] as our contrastive learning
paradigm. Specifically, we train two networks $f_{\theta_{q}}$,
$f_{\theta_{k}}$ to map the positive pair
$(\tilde{x}_{i}^{j},\tilde{x}_{i}^{j})$ and the negative pair
$(\tilde{x}_{i}^{j},\tilde{x}_{v}^{j})$ to corresponding embeddings as
follows:

$q_{i}^{j}=f(\tilde{x}_{i}^{j},p^{j};\theta_{q}),\;k_{+}=f(\tilde{x}_{i}^{j},p^{j};\theta_{k}),\;k_{-}=f(\tilde{x}_{v}^{j},p^{j};\theta_{k}),$
(5)

where $\theta_{k}=m\theta_{k}+(1-m)\theta_{q}$ and $m\in[0,1)$ is a momentum
coefficient. The network $f(\cdot,\cdot;\theta_{q})$ is comprised of a feature
extractor function $e(\cdot,\cdot;\theta_{1})$, which accepts both patches and
their corresponding anatomical landmarks as inputs, and a multilayer
perceptron (MLP) projection head $g(\cdot;\theta_{2})$, which maps the patch
representations to the space where contrastive loss is applied. The equation
can be written as
$f(x_{i}^{j},p^{j};\theta_{q})=g(e(x_{i}^{j},p^{j};\theta_{1});\theta_{2})$,
where $\theta_{q}=\\{\theta_{1},\theta_{2}\\}$. Finally, the _local
contrastive loss_ per location is defined as:

$\mathcal{L}_{l}^{j}=-\log\frac{\exp(q_{i}^{j}\cdot
k_{+}/\tau)}{\exp(q_{i}^{j}\cdot k_{+}/\tau)+\sum^{{K}^{-}}\exp(q_{i}^{j}\cdot
k_{-}/\tau)},$ (6)

where $K^{-}$ denotes the number of negative pairs and $\tau$ denotes the
_temperature_ hyperparameter.

###  3.4 Neighboring Contrastive Loss

The _local contrastive loss_ incentivizes representations to be sensitive to
tissue abnormalities within local anatomical regions. Pathological tissues,
however, may expand beyond the borders of a single patch. We develop a
complementary contrasting strategy - _neighboring contrasting_ to allow the
same encoder to learn disease patterns that may spread across multiple
anatomical regions. For a given anatomical region $j$, we denote the indices
of its $\ell$ nearest neighboring regions by $\mathcal{N}(j)$, its neighboring
anatomical landmarks by $\\{p^{l}\\}_{l\in\mathcal{N}(j)}^{\ell}$, and the
neighboring patches of $x_{i}^{j}$ on the same image by
$\\{x_{i}^{l}\\}_{l\in\mathcal{N}(j)}^{\ell}$. The corresponding embeddings of
the neighboring patches are given by:

$r_{i}^{l}=f(\tilde{x}_{i}^{l},p^{l};\theta_{q}),\;{l\in\mathcal{N}(j)},$ (7)

where $\tilde{x}_{i}^{l}=\tilde{t}(x_{i}^{l})$ and $\tilde{t}\sim\mathcal{T}$.
Instead of constructing positive and negative pairs, we construct positive and
negative _sets_ , specifically,

$\displaystyle\text{positive
set}:\left\\{\\{\tilde{x}_{i}^{l}\\}_{l\in\mathcal{N}(j)}^{\ell},\>\tilde{x}_{i}^{j}\right\\},$
$\displaystyle\text{negative
set}:\left\\{\\{\tilde{x}_{i}^{l}\\}_{l\in\mathcal{N}(j)}^{\ell},\>\tilde{x}_{v}^{j}\right\\},\>v\neq
i.$

The _neighboring contrastive loss_ per location is define as:

$\mathcal{L}_{n}^{j}=-\log\frac{\sum_{l}^{\mathcal{N}(j)}\exp(r_{i}^{l}\cdot
k_{+}/\tau)}{\sum_{l}^{\mathcal{N}(j)}\exp(r_{i}^{l}\cdot
k_{+}/\tau)+\sum_{l}^{\mathcal{N}(j)}\sum_{k_{-}}^{\mathcal{K}^{-}}\exp(r_{i}^{l}\cdot
k_{-}/\tau)},$ (8)

where $k_{+}$ and $k_{-}$ are the same as defined in Eqn. 5. Minimizing this
loss forces the encoder to extract similar visual features of the disease
spreading across the patch $x_{i}^{j}$ and its neighboring patches
$\\{x_{i}^{l}\\}_{l\in\mathcal{N}(j)}^{\ell}$. Additionally, by selecting
random patches in the same anatomical region as the hard negatives, the
encoder is prevented from using mismatched anatomy as a shortcut to perform
this _instance discrimination_ task.

###  3.5 Overall Model

We train our model end-to-end by minimizing the combined _local contrastive
loss_ and _neighboring contrastive loss_ and looping through each anatomical
landmark. The overall loss function per location is defined as:

$\mathcal{L}^{j}=\mathcal{L}_{l}^{j}+\mathcal{L}_{n}^{j}.$ (9)

During inference time, the voxel-level representation can be obtained by:

$y_{i}^{j}=e(x_{i}^{j},p^{j};\theta_{1}),$ (10)

where $e(\cdot,\cdot;\theta_{1})$ is the trained encoder with Loc-CondConv
layers. The image-level representation $y_{i}$ is obtained by averaging the
representations of patches across all the anatomical landmarks. Formally, the
representation at the image level is given by:

$y_{i}=\frac{1}{J}\sum_{j=1}^{J}e(x_{i}^{j},p^{j};\theta_{1}).$ (11)

Note that, at the time of inference, $p^{j}$ can be any point inside the atlas
space and is not restricted to the predefined anatomical landmarks. In our
experiments, we obtain image-level representations using only predetermined
anatomical markers for computational efficiency.

##  4 Implementation Details

We begin by extracting the lung regions from each CT scan using the lung
segmentation method proposed by [54]. We then choose the image of one healthy
subject as the anatomical atlas and partition it into a grid of 3D patches
with some overlap. This results in 581 patches, each with a size of $32\times
32\times 32$, that fully cover the lung in the atlas image. Anatomical
landmarks are defined as the centers of these 581 patches on the atlas
coordinate system. We use the image registration toolkit ANTs [53] to obtain
the forward and inverse affine transformations between each subject’s (moving)
image and the atlas (fixed) image.

We construct the encoder $e(\cdot,\cdot;\theta_{1})$ using Loc-CondConv layers
as the building blocks. Each Loc-CondConv layer contains $N$ 3D-convolutional
kernels with size $3\times 3\times 3$ and is zero-padded on each side of the
inputs by one pixel. We adopt batch normalization (BN) [55] and ELU [56]
activation following each Loc-CondConv. For the projection head
$g(\cdot;\theta_{2})$, we adopt a 2-layer MLP with ReLU activation. We set the
number of nearest neighbors used in the neighboring contrastive loss as 2
based on an ablation study (Sec. 5.4.3). We create data augmentations using
MONAI [57] package. The data augmentation includes random affine transforms
(applied in the order of rotation, translation, and scale), Gaussian noise,
and random image contrast adjustments. We optimize the networks using SGD with
momentum = 0.9 and weight decay = $10^{-4}$. The learning rate is set to be
$10^{-2}$ and is updated using a cosine schedule. We choose the batch size of
128. Following the practice in MoCo-v2 [58], we set temperature $\tau$ to 0.2
and momentum coefficient to $0.999$. Unlike regular MoCo, which uses a single
dictionary for negative samples, we develop a conditional memory bank that
maintains separate dictionaries for anatomical landmarks, each of which has a
size of 4096. For training, we select negative samples from the corresponding
dictionary, which stores patch embeddings from the same anatomical location as
the query patch. We perform self-supervised pretraining on the full dataset
using four NVIDIA Tesla V100 GPUs, each with 32GB memory, for 48 hours or 20
epochs, whichever comes first.

##  5 Experiments

In this section, we take our DrasCLR pre-trained models and evaluate their
performance in medical imaging tasks at both image and voxel levels. At the
image level, we evaluate the effectiveness of the learned representation in
disease phenotype prediction, disease severity classification, and survival
analysis. At the voxel level, we first describe how our model can be used to
produce voxel-wise segmentation masks. Using this approach, we then present
the quantitative and qualitative results of subtype emphysema detection.
Finally, we perform ablation studies to validate the importance of the
proposed components in DrasCLR.

###  5.1 Datasets

We conduct the experiments on two large-scale datasets of 3D medical images,
including the COPDGene dataset [59] and the MosMed dataset [60]. We apply the
same data preprocessing procedure for images in both datasets. We begin by re-
sampling all images into $1mm^{3}$ isotropic resolution. We then threshold the
Hounsfield Units (HU) to the intensity window of $[-1024,240]$ and normalize
the intensity range to $[-1,1]$ by linear scaling.

####  5.1.1 COPDGene Dataset

Chronic Obstructive Pulmonary Disease (COPD) is a chronic inflammatory lung
disease that causes obstruction of lung airflow and is one of the leading
causes of death worldwide. The COPDGene Study [59] is a multi-center
observational study that collects imaging data, genetic biomarkers, and
relevant phenotypes from a large cohort of subjects. In our study, we use a
large set of 3D thorax CT images from 9,180 subjects for self-supervised pre-
training. We use the spirometry measures, disease-related phenotypes, and
survival status of the same cohort as the image-level labels in our
experiments. On a subset of these CT scans, an experienced pulmonologist
annotated the bounding boxes of subtypes of emphysema by clicking on locations
surrounded by the pathological tissues [61, 62]. This procedure created 696
centrilobular emphysema bounding boxes from 153 subjects, and 243 paraseptal
emphysema bounding boxes from 69 subjects. All these bounding boxes are of the
same size ($32mm^{3}$). We use this annotated subset to examine the
performance of the DrasCLR pre-trained model for subtype emphysema detection.

####  5.1.2 MosMed Dataset

The MosMed dataset contains 3D thorax CT images of 1,110 subjects from the
municipal hospitals in Moscow, Russia [60]. Subjects in this dataset are
classified into five grades (“Zero”, “Mild”, “Moderate”, “Severe”, and
“Critical”) based on COVID-19 related CT findings and physiological measures,
such as body temperature, respiration rate, blood oxygen saturation level
(SpO2) and so on. Triage decisions are made based on the severity levels of
the patients. For example, patients in the “Moderate” category only need to be
followed up at home by a primary care physician, whereas patients in the
“Critical” category are immediately transferred to the intensive care unit. We
use the CT images in MosMed for model pre-training and use COVID-19 severity
grades as classification labels in downstream analysis.

###  5.2 Image Level Evaluation

To assess how much disease-related information is preserved by the proposed
method, we use the learned image-level representation to predict a wide range
of clinical variables measured at the subject level, such as spirometry
measurements, disease phenotypes, disease staging, and patients’ survival
rates.

####  5.2.1 COPD Phenotype Prediction

We begin by performing self-supervised pre-training with DrasCLR on the
COPDGene dataset. Then, we use the learned image-level representations in
downstream prediction tasks in a _linear readout_ fashion. In particular, we
train linear regression models to predict two pulmonary function measures on
the log scale, which are percent predicted values of Forced Expiratory Volume
in one second (FEV1pp) and its ratio with Forced vital capacity
($\text{FEV}_{1}/\text{FVC}$). We use $R^{2}$ scores as an evaluation metric
for the regression analysis. In addition, we train multi-class logistic
regression models to predict four categorical outcomes: (1) Global Initiative
for Chronic Obstructive Lung Disease (GOLD) spirometric stage, a four-grade
categorical variable indicating the severity of airflow limitation, (2)
Centrilobular emphysema visual score (CLE), a six-grade categorical variable
indicating the extent of emphysema in centrilobular, (3) Paraseptal emphysema
visual score (Paraseptal), a three-grade categorical variable indicating the
severity of paraseptal emphysema, and (4) Acute Exacerbation history (AE
history), a binary variable indicating whether the patient has encountered at
least one exacerbation event before enrolling in the study. For all
classification tasks, we use accuracy as the evaluation metric. To account for
human variability in annotation, for GOLD, CLE, and Paraseptal scores, we also
report the proportion of times the predicted class fell within one class of
the true score (denoted as 1-off).

We compare the performance of DrasCLR against both unsupervised and supervised
approaches. The unsupervised baselines include: Models Genesis [45],
MedicalNet [63], MoCo (3D version on the entire volume) [10], Context SSL
[64], Divergence-based feature extractor [65], K-means algorithm applied to
image features extracted from local lung regions [65], and Low Attenuation
Area (LAA), a commonly used clinical descriptor. The supervised baselines
include convolutional neural networks (CNN) that were separately trained to
predict FEV1pp, GOLD and CLE scores using 2D slices as inputs (2D CNN) [66],
and Subject2Vec [67], where a patch-based CNN model was first trained with
FEV1 and $\text{FEV}_{1}/\text{FVC}$ as joint supervised information, and the
learned image representations were then used in other prediction tasks. We
perform five-fold cross-validation for all experiments and report the average
results along with standard deviations. Table 2 shows that the DrasCLR pre-
trained model outperforms unsupervised baseline models in all metrics, with
the exception of 1-off accuracy for Paraseptal emphysema, where the difference
is within one standard deviation. Our DrasCLR pre-trained model also
outperforms the supervised baseline models, including Subject2Vec and 2D CNN,
in terms of CLE, Paraseptal, and AE History predictions. For spirometry and
COPD Staging, on which Subject2Vec were trained, the performance gap of our
model is smaller compared to other unsupervised baseline models.

Overall, these results suggest that image-level features extracted by the
DrasCLR pre-trained model preserve richer information about COPD severity than
other unsupervised baselines. When compared to supervised methods, our
proposed method learns more generalizable features as it achieves higher
predictive performance for a broader range of clinical variables, such as
emphysema visual scores and AE history.

Table 2: Results of phenotype prediction on the COPDGene dataset. We use
R-Square for continuous measurements and accuracy for discrete scores. Results
including the mean and standard deviation (mean$\pm$s.d.) are derived from
5-fold cross validation. Our DrasCLR model has the best or competitive
performance on all phenotype prediction tasks when compared to seven
unsupervised methods, and it generalizes better than the supervised method for
predicting visual scores and AE history.

Method Supervised Spirometry COPD Staging Visual scores Acuity $\log$FEV1pp
$\log$$\text{FEV}_{1}/\text{FVC}$ GOLD GOLD _1-off_ CLE CLE _1-off_ Paraseptal
Paraseptal _1-off_ AE History Metric R-Square % Accuracy LAA-950 ✗
$0.44_{\pm.02}$ $0.60_{\pm.01}$ $55.8$ $75.7$ $32.9$ $77.7$ $33.3$ $87.6$
$73.8$ K-Means ✗ $0.55_{\pm.03}$ $0.68_{\pm.02}$ $57.3$ $82.3$ - - - - -
Divergence-based ✗ $0.58_{\pm.03}$ $0.70_{\pm.02}$ $58.9$ $84.2$ - - - - -
MedicalNet ✗ $0.47_{\pm.10}$ $0.59_{\pm.06}$ $57.0_{\pm 1.3}$ $75.4_{\pm.9}$
$40.3_{\pm 1.9}$ $69.6_{\pm 1.6}$ $53.1_{\pm 0.7}$ $81.8_{\pm 0.8}$ $78.7_{\pm
1.3}$ ModelsGenesis ✗ $0.58_{\pm.01}$ $0.64_{\pm.01}$ $59.5_{\pm 2.3}$
$82.9_{\pm 1.3}$ $41.8_{\pm 1.4}$ $77.0_{\pm 1.5}$ $52.7_{\pm.5}$ $85.3_{\pm
1.1}$ $77.8_{\pm.8}$ MoCo ✗ $0.40_{\pm.02}$ $0.49_{\pm.02}$ $52.7_{\pm 1.1}$
$67.6_{\pm 1.4}$ $36.5_{\pm.7}$ $61.9_{\pm.9}$ $52.5_{\pm 1.4}$ $79.7_{\pm
1.2}$ $78.6_{\pm.9}$ Context SSL ✗ $0.62_{\pm.01}$ $0.70_{\pm.01}$ $63.2_{\pm
1.1}$ $83.6_{\pm.9}$ $50.4_{\pm 1.3}$ $81.5_{\pm 1.1}$ $56.2_{\pm 1.1}$
$84.9_{\pm 1.2}$ $78.8_{\pm 1.3}$ 2D CNN ✓ $0.53$ - $51.1$ - - $60.4$ - -
Subject2Vec ✓ $0.67_{\pm.03}$ $0.74_{\pm.01}$ $65.4$ $89.1$ $40.6$ $74.7$
$52.8$ $83.0$ $76.9$ Ours ✗ $0.63_{\pm.01}$ $0.71_{\pm.01}$ $65.0_{\pm.6}$
$85.6_{\pm.6}$ $53.9_{\pm.8}$ $86.3_{\pm.7}$ $58.4_{\pm.8}$ $87.0_{\pm.8}$
$78.9_{\pm 1.3}$ \- indicates not reported. Some baseline methods only report
mean value without standard deviation in original manuscript. The bold font is
used to highlight the highest value for each column among _all_ methods. The
underline is used to highlight the highest value for each column among
_unsupervised_ methods.

####  5.2.2 Survival Analysis of COPD Patients

We evaluate the effectiveness of DrasCLR in survival analysis for the COPDGene
population. We employ the Cox proportional hazards (CPH) model [68] to predict
patients’ survival using the learned image-level representations while
controlling for five potential confounders, including age, gender, race,
smoking status, and packyear (calculated by multiplying the number of packs of
cigarettes smoked per day by the number of years the person has smoked). We
compare the performance of features extracted by our method against: (1) hand-
designed imaging features, (2) imaging features retrieved by other machine
learning methods, and (3) relevant clinical features, such as spirometry
measures and the BODE index [69]. The hand-designed imaging features include
CT metrics of emphysema, gas trapping, average wall thickness of hypothetical
airway, and wall area percentage of segmental airways [70]. All comparison
baselines use the same CPH model and are controlled by including the same five
confounding variables.

We report the results in terms of time-dependent concordance index ($C^{td}$),
which estimates the model’s risk ranking ability, at each of the censoring
period quantiles. Table 3 shows that the survival model with our imaging
features achieved concordance scores of 0.76, 0.75, and 0.74 at the 25, 50,
and 75 quantiles, respectively, outperforming baselines with imaging-only
features retrieved by Models Gensis [45], Subject2Vec [67] and the hand-
designed model. In comparison to clinical features, our method outperformed
spirometry measures (0.76 vs 0.74) for risk stratification of near-term events
before the 25 quantile, slightly underperformed the BODE index (0.74 vs 0.75)
at the 75 quantile of censoring time, and achieved comparable accuracy
otherwise. We also developed survival models with combined imaging and
clinical features. The bottom rows of Table 3 show that the model using both
our imaging features and BODE index achieved the highest concordance scores of
0.78, 0.77, and 0.77 at the 25, 50, and 75 quantiles, respectively,
demonstrating that the imaging representation learned by DrasCLR provides
incremental predictive value for survival analysis of COPD patients.

Table 3: Time-dependent concordance index on the COPDGene dataset. Results are
averages over five runs with bootstrapped standard errors. Our DrasCLR model
performs the best when compared to other imaging representation approaches,
and it provides incremental predictive value to clinical features.

Feature Method Concordance Index $t=\nth{25}$ $t=\nth{50}$ $t=\nth{75}$
Imaging Hand-designed $0.74_{\pm 0.02}$ $0.73_{\pm 0.01}$ $0.74_{\pm 0.01}$
Models Genesis $0.72_{\pm 0.02}$ $0.7_{\pm 0.01}$ $0.72_{\pm 0.01}$
Subject2Vec $0.72_{\pm 0.02}$ $0.72_{\pm 0.01}$ $0.72_{\pm 0.03}$ Ours
$0.76_{\pm 0.02}$ $0.75_{\pm 0.01}$ $0.74_{\pm 0.01}$ Clinical Spirometry
$0.74_{\pm 0.02}$ $0.75_{\pm 0.01}$ $0.74_{\pm 0.01}$ BODE $0.76_{\pm 0.01}$
$0.75_{\pm 0.01}$ $0.75_{\pm 0.00}$ Imaging + Clinical Hand-designed + BODE
$0.76_{\pm 0.02}$ $0.76_{\pm 0.01}$ $0.76_{\pm 0.01}$ Ours + Spirometry
$0.77_{\pm 0.02}$ $0.76_{\pm 0.00}$ $0.76_{\pm 0.01}$ Ours + BODE
$\bm{0.78_{\pm 0.01}}$ $\bm{0.77_{\pm 0.01}}$ $\bm{0.77_{\pm 0.00}}$

####  5.2.3 COVID-19 Severity Prediction

We first pre-train a model with DrasCLR on the CT scans in the MosMed dataset.
Then, we freeze the encoder and train a linear classifier to predict the
severity of COVID-19, a categorical variable with five grades. The DrasCLR
pre-trained model is compared to unsupervised baseline models, including
MedicalNet [63], Models Genesis [45], MoCo [10], Context SSL [64], and one
supervised CNN model that uses entire 3D images as inputs (3D CNN). To
evaluate classification performance, we perform five-fold cross-validation and
report the average test accuracy along with standard deviations.

Table 4: Classification of 5-grade COVID-19 severity on the MosMed dataset. The results are the means and standard deviations of accuracy for 5-fold cross validation. Our DrasCLR model leads the best performance over both unsupervised and supervised approaches. Method | Supervised | % Accuracy
---|---|---
3D CNN | ✓ | $61.2_{\pm 3.5}$
MedicalNet | ✗ | $62.1_{\pm 3.3}$
Models Genesis | ✗ | $62.0_{\pm 3.5}$
MoCo | ✗ | $62.1_{\pm 3.3}$
Context SSL | ✗ | $65.3_{\pm 3.2}$
Ours w/o Neighbor Contrast | ✗ | $62.6_{\pm 2.4}$
Ours | ✗ | $65.4_{\pm 2.5}$

Table 4 shows that the DrasCLR pre-trained model outperforms the unsupervised
baseline models with the exception of ContextSSL, whose average test accuracy
is close to ours (65.3 vs. 65.4). Both ContextSSL and our methods leverage the
context between neighboring anatomical regions for representation learning.
ContextSSL incorporates this information via a graph neural network, whereas
our method uses a neighboring contrasting strategy. The ablation study results
in Table 4’s bottom rows show that leveraging anatomical context from large
regions is useful for categorizing COVID-19 severity. With the neighboring
contrastive loss, the COVID-19 severity prediction accuracy increases by
$2.8\%$. Interestingly, we found that the supervised 3D CNN model performs the
worst, suggesting that directly extracting features from the entire volume may
have resulted in the loss of fine-grained information at local anatomy. It is
also possible that the supervised model may not converge properly or becomes
overfitted due to the small amount of training data.

###  5.3 Voxel Level Evaluation

To show that the DrasCLR pre-trained model encodes fine-grained information at
local anatomy, we demonstrate its ability to detect two subtypes of emphysema
(i.e., centrilobular and paraseptal emphysema), which are prevalent in
different pulmonary regions. We perform the experiments in three aspects.
First, we conduct a quantitative evaluation for emphysema detection via voxel-
wise classification. Second, we show the qualitative results of predicted
emphysema masks for COPD patients at different stages. Third, we perform
emphysema detection on a group of randomly selected subjects and show the
relationship between the detected emphysema volume and the patient’s COPD
stage. In addition, we demonstrate that our method can reduce annotation
efforts through transfer learning.

####  5.3.1 Emphysema Detection via Dense Classification

We propose to use dense or voxel-level classification for emphysema detection.
In particular, we first pre-train a model with DrasCLR on all CT scans from
the COPDGene dataset. We then fine-tune the model in a binary classification
task to discriminate between emphysema-annotated patches and healthy patches.
The healthy patches are sampled from random subjects with the criteria that
the subjects’ GOLD scores are equal to zero, and no centrilobular or
paraseptal emphysema is found based on their image-level visual scores. We
have evaluated two fine-tuning schemes: (1) employing the pre-trained encoder
as a fixed feature extractor and training a new linear classifier (named as
linear readout), and (2) appending a linear classifier to the pre-trained
encoder and fine-tuning all layers in the network (named as full fine-tuning).
During inference time, the fine-tuned model is used to perform voxel-wise
classification of emphysema. For a given patch, the detection is positive if
$25\%$ voxels within it have a predicted probability of emphysema greater than
0.5; otherwise, the detection is negative.

Table 5 presents the quantitative performance of our method and selected
baselines for emphysema detection. Under the linear readout scheme, we compare
the performance of our model to logistic regression models using patch-level
LAA features and learned features from Models Genesis [45]. Under the full
fine-tuning scheme, we compare our model’s performance to a patch-based CNN
that is trained on the same set of patches from scratch. The results in Table
5 show that, compared to unsupervised baselines, our method achieves superior
F1 scores for both centrilobular and paraseptal emphysema detection under
linear readout. Through full fine-tuning, our method achieves the best
emphysema detection performance across all evaluation metrics.

Table 5: Evaluation for subtype emphysema detection. Results are the means and standard deviations (mean$\pm$s.d.) of F1, precision and recall scores for 5-fold cross validation. With linear readout, our DrasCLR outperforms unsupervised baselines. With full fine-funing, our DrasCLR outperforms the supervised baseline. |  | Centrilobular | Paraseptal
---|---|---|---
| Supervised | F1 | Precision | Recall | F1 | Precision | Recall
Patch LAA | ✗ | $0.77_{\pm.03}$ | $0.94_{\pm.02}$ | $0.65_{\pm.04}$ | $0.70_{\pm.06}$ | $0.83_{\pm.06}$ | $0.61_{\pm.09}$
Models Genesis | ✗ | $0.71_{\pm.01}$ | $0.58_{\pm.02}$ | $0.93_{\pm.03}$ | $0.67_{\pm.03}$ | $0.64_{\pm.16}$ | $0.82_{\pm.23}$
Ours w/ Linear Readout | ✗ | $0.82_{\pm.03}$ | $0.75_{\pm.04}$ | $0.91_{\pm.02}$ | $0.82_{\pm.03}$ | $0.73_{\pm.05}$ | $0.95_{\pm.01}$
Patch-based CNN | ✓ | $0.91_{\pm.03}$ | $0.92_{\pm.01}$ | $0.89_{\pm.05}$ | $0.82_{\pm.04}$ | $0.77_{\pm.05}$ | $0.88_{\pm.02}$
Ours w/ Full Fine-tuning | ✓ | $0.98_{\pm.01}$ | $0.97_{\pm.01}$ | $0.99_{\pm.01}$ | $0.99_{\pm.01}$ | $0.99_{\pm.02}$ | $1.00_{\pm.00}$
The bold font is used to highlight the highest value for each column among
_all_ methods.
The underline is used to highlight the highest value for each column among
_unsupervised_ methods.

To qualitatively demonstrate the outcomes of our method, we create voxel-wise
emphysema segmentation for subjects at different stages of COPD. In
particular, we first use the full fine-tuning model to estimate the
probability of emphysema in a sliding-window fashion with a step size of 1
voxel. We then use a 0.5 threshold to map voxels with emphysema probability
greater than or equal to the threshold to 1 and all other voxels to 0. Fig. 3
shows the predicted segmentation masks of two subtypes of emphysema of varying
COPD stage in coronal and 3D views. We find that as the COPD severity increase
(higher GOLD score), the volume of detected emphysema region increases in both
subtypes. Furthermore, segmentation masks of GOLD scores 1 and 2 show a clear
heterogeneity in the regional distribution of emphysema in the lung between
these two subtypes. The regions of predicted segmentation are consistent with
the clinical definition of emphysema subtypes, where centrilobular emphysema
is commonly described as an abnormal enlargement of airspaces centered on the
respiratory bronchiole [71] and paraseptal emphysema refers to emphysematous
change adjacent to a pleural surface [72].

Figure 3: Examples of predicted dense emphysema binary masks for subjects with
different GOLD scores. The top three rows show the predicted regions of
centrilobular emphysema, and the bottom three rows show the predicted regions
of paraseptal emphysema. The intensity range is set as [-1060, -825] to better
illustrate the emphysema. The predicted emphysema regions are plotted in red,
and the lung regions are plotted in blue. As the severity of COPD increases
(higher GOLD score), the detected region increases in both subtypes of
emphysema. In addition, the predicted emphysema regions correspond to the
clinical description of their subtypes.

Finally, we analyze the correlation between the total emphysema detected in 3D
images and the subjects’ COPD stages. In particular, we randomly select 500
subjects from the COPDGene dataset and use the fully fine-tuned model to make
voxel-wise emphysema classification on their CT scans. Then, we aggregate all
voxels in a CT scan to determine the fraction of voxels with a predicted
probability of emphysema greater than 0.5. The box plots in Fig. 4 represent
the distributions of detected emphysema proportion against GOLD scores, as
well as the group with preserved ratio impaired spirometry (PRISm). Subplots
of both centrilobular and paraseptal emphysema show positive correlations
between the detected emphysema and patient’s COPD severity.

Figure 4: Comparison of predicted volume proportion of centrilobular (left)
and paraseptal (right) emphysema for subjects with different GOLD scores. A
higher GOLD score indicates a more severe stage of COPD. The results show that
with an increasing GOLD score, the predicted emphysema volume proportion in CT
scan becomes higher.

####  5.3.2 Improve Annotation Efficiency via Transfer Learning

Transfer learning makes use of the knowledge of underlying data structure
learned by the pre-trained models and has been demonstrated to be beneficial
in medical imaging analysis, where the amount of annotated data is often
limited. We simulate the scenarios of using a subset of annotated data to
investigate the power of our method in transfer learning. Specifically, we
fine-tune the DrasCLR pre-trained model by starting with $10\%$ annotated
emphysema patches and gradually increasing the amount of annotations by $10\%$
in subsequent experiments. Fig. 5 shows the results of transfer learning on
two target tasks. The performance of centrilobular detection learning from
scratch with the entire dataset can be surpassed using DrasCLR with only
$50\%$ of the dataset, hence doubling the annotation efficiency. The
performance of paraseptal detection learning from scratch with the entire
dataset can be surpassed using DrasCLR with only $20\%$ of the dataset, thus
improving the annotation efficiency by five times. These results demonstrate
how DrasCLR can significantly reduce the cost of manual image annotation,
ultimately leading to more label-efficient deep learning.

Figure 5: Results of fine-tuning with different amounts of data. We perform
evaluations for centrilobular (left) and paraseptal (right) emphysema
detection. Compared to the model fine-tuned with full data from scratch
(random initialization), the DrasCLR pre-trained model only needs $50\%$ and
$20\%$ annotated data to achieve the same performance for centrilobular and
paraseptal emphysema detection, respectively.

###  5.4 Ablation Study

In this section, we conduct ablation experiments to validate the effects of
several DrasCLR components.

####  5.4.1 Design Choices for Incorporating Anatomical Location

We demonstrate the effects of various designs for incorporating anatomical
location information into DrasCLR. Specifically, we compare the following four
approaches: (1) No conditioning, which uses a standard CNN to extract features
from image patches without taking their anatomical locations into account; (2)
Concatenation [64], which concatenates the features from the last layer of the
standard CNN with patch location and then fuses them through fully connected
layers; (3) HyperNetwork [47], which uses a separate neural network that takes
an anatomical location as input to generate weights in the standard CNN; (4) A
CNN consists of our proposed Loc-CondConv layers. All models in this ablation
experiment are trained using only local contrastive loss. As shown in Table.
6, the simple concatenation approach outperforms the standard CNN in three
image-level tasks, suggesting that including anatomical location enriches the
learned representations and enhances the performance of the downstream
analysis. Furthermore, Table. 6 shows that the CNN with Loc-CondConv layers
achieves the best performances in all target tasks, demonstrating it is a
superior design for incorporating anatomical location information.

Table 6: Ablation study for how to incorporate anatomical location. We report R-Square for $\log$FEV1pp, accuracy scores for the GOLD and CLE scores. The mean and standard deviation values are calculated via 5-fold cross-validation. Method |  $\log$FEV1pp | GOLD | CLE
---|---|---|---
No conditioning | $0.57_{\pm.04}$ | $61.8_{\pm 1.1}$ | $48.0_{\pm.9}$
Concatenation | $0.60_{\pm.01}$ | $62.5_{\pm 1.0}$ | $49.2_{\pm 1.1}$
HyperNetwork | $0.60_{\pm.01}$ | $58.6_{\pm 1.7}$ | $44.1_{\pm 1.3}$
Loc-CondConv (Ours) | $0.62_{\pm.02}$ | $63.4_{\pm 1.0}$ | $50.3_{\pm.9}$

####  5.4.2 Impact of Anatomical Location for Disease Detection

We have thus far validated the importance of anatomical location as well as
the effectiveness of Loc-CondConv layer for image-level prediction tasks. To
investigate whether anatomical location is an important factor in representing
fine-grained local features, we explore the impact of perturbing the input
locations of DrasCLR on the performance of emphysema detection. In particular,
instead of extracting features from a given patch using its correct anatomical
location as input, we extract features from the patch using a random location
in the lung. We employ a linear readout fine-tuning scheme on the same set of
patches as described in Section 5.3.1. As shown in Table 7, by using random
anatomical locations as inputs, the detection accuracy drops by $6\%$ and
$16\%$ for centrilobular and paraseptal emphysema, respectively, showing
statistically significant decreases (p-value$<0.005$, one-sided two sample
t-test). The results indicate that the DrasCLR pre-trained encoder is
sensitive to anatomical locations and captures anatomy-specific features.

Table 7: Sensitivity of the pre-trained DrasCLR model to anatomical location perturbation. The results are the means and standard deviations of emphysema detection accuracy for 5-fold cross validation. Subtype | Random location | Patch location
---|---|---
Centrilobular emphysema | $73.3_{\pm 2.6}$ | $79.2_{\pm 1.4}$
Paraseptal emphysema | $56.4_{\pm 4.1}$ | $72.1_{\pm 4.3}$

####  5.4.3 Neighboring Contrastive Loss

In this experiment, we investigate the effect of neighboring contrastive loss
as well as the impact of the number of neighbors used. We pre-train DrasCLR
models with no neighboring contrast as well as with different numbers of
neighbors. Using a linear readout scheme, we benchmark the performance of the
pre-trained models in downstream image-level tasks. As shown in Table 8, the
incorporation of spatial context from neighboring patches enhances the
performance of image-level tasks in most situations. The improvement is
particularly notable in the prediction of the centrilobular visual score,
which ranges from $5\%$ to $8\%$ depending on the number of neighbors. This is
likely due to the fact that CLE grades are determined by the extent to which
the lung’s center is damaged by the disease, and the neighboring contrasting
strategy encourages the learning of common disease patterns that span multiple
anatomical regions.

Table 8:  Ablation study for neighboring contrastive loss. We report R-Square for $\log$FEV1pp, accuracy scores for GOLD and CLE scores. The mean and standard deviation values are calculated via 5-fold cross-validation. Method |  $\log$FEV1pp |  $\log$FEV1pp/FVC | CLE
---|---|---|---
No Neighboring Contrast | $0.62_{\pm.02}$ | $0.69_{\pm.02}$ | $50.3_{\pm 0.9}$
# Neighbors = 1 | $0.61_{\pm.01}$ | $0.70_{\pm.01}$ | $53.0_{\pm 1.3}$
# Neighbors = 2 (Ours) | $0.63_{\pm.01}$ | $0.71_{\pm.01}$ | $53.9_{\pm 0.8}$
# Neighbors = 3 | $0.63_{\pm.01}$ | $0.71_{\pm.01}$ | $52.5_{\pm 0.5}$

##  6 Discussions

###  6.1 Does DrasCLR learn disease-related features?

In this paper, we propose a novel self-supervised method, DrasCLR to learn
_disease-related_ representation of 3D medical images. Medical images have
recurring and similar anatomy across patients. While previous self-supervised
methods [45, 17] used this knowledge to learn common anatomical
representation, our method uses this knowledge to create hard negative samples
in contrastive learning. As a result, our method is more sensitive to tissue
abnormalities and can encode more disease-related information. As seen in
Table 2, our method demonstrates excellent performance in predicting a wide
range of clinical variables, such as spirometry measures and COPD phenotypes,
which are closely associated with the degree of lung impairments in COPD
patients. Emphysema is a hallmark of COPD. However, visually assessing
emphysema at CT is time-consuming and subject to human variability. Our method
demonstrates superior performance for this task and may provide a data-driven
way of quantifying the visual score of emphysema from CT imaging.

Moreover, as evidenced by Table 3, imaging features learned by our method add
incremental value to the BODE index for survival analysis in COPD patients,
suggesting that our method is capable of capturing complementary risk factors
from CT imaging. In addition to the evaluation on the COPD cohort, we
demonstrate, as shown in Table 4, that DrasCLR can learn robust features
associated with COVID-19 severity. The ability to represent disease-related
information from medical images in an unsupervised manner is particularly
useful during pandemic outbreaks, when labeled data is rarely available.

###  6.2 Does DrasCLR extract location-specific features?

A conventional convolutional layer is designed to be transitionally invariant.
However, pathological patterns tend to be heterogeneous across locations in
the human body, and an _one-size-fits-all_ design may not be sufficient to
learn the variety of tissue abnormalities at different anatomical locations.
In this paper, we incorporate anatomical context into representation learning
via two components in DrasCLR. First, image registration is used to provide a
unified anatomical coordinate system, and all images are aligned to the atlas.
Second, a novel Loc-CondConv layer is introduced to have modifiable weights
that are conditionally dependent on anatomical location.

We have explored different ways to incorporate anatomical location into
representation learning. As seen in Table 6, the Loc-CondConv layer is
superior to simple concatenation and hypernetwork. To further validate the
effect of our design, we investigate the impact of perturbing the input
locations of DrasCLR on emphysema detection. As shown in Table. 7, the
emphysema detection accuracy decreases when random location is used,
demonstrating that lack of correct anatomical context is detrimental.

###  6.3 Why do we choose sliding-window based approach for emphysema
segmentation?

The most well-known architecture for medical image segmentation is U-Net [73],
which is composed of opposing convolution and deconvolution layers, and
spatial information is provided through skip connections to each decoder layer
to recover fine-grained details. However, training U-Net requires the data to
have complete pixel/voxel-level annotation, which is difficult to obtain. The
emphysema annotation in the COPDGene dataset was collected as bounding boxes
by a physician clicking on regions surrounding by pathological tissues, thus
the annotation is not complete at the voxel level. Our method can leverage
these partially annotated images and produce voxel-wise emphysema
classification in a sliding-window fashion. Moreover, the DrasCLR pre-trained
model is capable of utilizing spatial information by modifying kernels’
weights based on the input anatomical location. As illustrated by Fig. 3, our
method can produce high-quality emphysema segmentation in which the regional
distribution of detected emphysema matches with the clinical description of
emphysema subtypes. The box plots in Fig. 4 show quantitative evidence that
the detected emphysema volume correlates with the subjects’ COPD stages.

##  7 Conclusion

In this paper, we present a novel method for anatomy-specific self-supervised
representation learning on 3D medical images. We propose two domain-specific
contrasting strategies to learn disease-related representations, including a
local contrasting loss to capture small disease patterns and a neighboring
contrasting loss to learn anomalies spanning across larger anatomical regions.
In addition, we introduce a novel conditional encoder for location-specific
feature extraction. The experiments on multiple datasets demonstrate that our
proposed method is effective, generalizable, and can be used to improve
annotation efficiency for supervised learning.

## Acknowledgments

This work was partially supported by NIH Award Number 1R01HL141813-01 and NSF
1839332 Tripod+X. We are grateful for the computational resources provided by
Pittsburgh SuperComputing grant number TG-ASC170024.

## References

  * [1] Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis.  Deep learning for computer vision: A brief review.  Computational intelligence and neuroscience, 2018, 2018.
  * [2] Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen.  Deep learning for generic object detection: A survey.  International journal of computer vision, 128(2):261–318, 2020\.
  * [3] Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, and Sundaraja S Iyengar.  A survey on deep learning: Algorithms, techniques, and applications.  ACM Computing Surveys (CSUR), 51(5):1–36, 2018.
  * [4] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  IEEE transactions on pattern analysis and machine intelligence, 43(11):4037–4058, 2020.
  * [5] Kriti Ohri and Mukesh Kumar.  Review on self-supervised image recognition using deep neural networks.  Knowledge-Based Systems, 224:107090, 2021.
  * [6] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In Proceedings of the IEEE international conference on computer vision, pages 1422–1430, 2015.
  * [7] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In European conference on computer vision, pages 649–666. Springer, 2016.
  * [8] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [10] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [11] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning?  Advances in Neural Information Processing Systems, 33:6827–6839, 2020.
  * [12] SouYoung Jin, Aruni RoyChowdhury, Huaizu Jiang, Ashish Singh, Aditya Prasad, Deep Chakraborty, and Erik Learned-Miller.  Unsupervised hard example mining from videos for improved object detection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 307–324, 2018.
  * [13] Sangryul Jeon, Dongbo Min, Seungryong Kim, and Kwanghoon Sohn.  Mining better samples for contrastive learning of temporal correspondence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1034–1044, 2021.
  * [14] Olle G Holmberg, Niklas D Köhler, Thiago Martins, Jakob Siedlecki, Tina Herold, Leonie Keidel, Ben Asani, Johannes Schiefelbein, Siegfried Priglinger, Karsten U Kortuem, and Fabian J Theis.  Self-supervised retinal thickness prediction enables deep learning from unlabelled data to boost classification of diabetic retinopathy.  Nature Machine Intelligence, 2(11):719–726, 2020.
  * [15] Zongwei Zhou, Vatsal Sodha, Md Mahfuzur Rahman Siddiquee, Ruibin Feng, Nima Tajbakhsh, Michael B Gotway, and Jianming Liang.  Models genesis: Generic autodidactic models for 3d medical image analysis.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 384–393. Springer, 2019.
  * [16] Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu.  Contrastive learning of global and local features for medical image segmentation with limited annotations.  Advances in Neural Information Processing Systems, 33:12546–12558, 2020.
  * [17] Fatemeh Haghighi, Mohammad Reza Hosseinzadeh Taher, Zongwei Zhou, Michael B Gotway, and Jianming Liang.  Transferable visual words: Exploiting the semantics of anatomical patterns for self-supervised learning.  IEEE transactions on medical imaging, 40(10):2857–2868, 2021.
  * [18] Benjamin M Smith, John HM Austin, John D Newell Jr, Belinda M D’Souza, Anna Rozenshtein, Eric A Hoffman, Firas Ahmed, and R Graham Barr.  Pulmonary emphysema subtypes on computed tomography: the mesa copd study.  The American journal of medicine, 127(1):94–e7, 2014.
  * [19] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [20] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European Conference on Computer Vision, pages 69–84. Springer, 2016.
  * [21] Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister.  Cutpaste: Self-supervised learning for anomaly detection and localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9664–9674, 2021.
  * [22] Ishan Misra, C Lawrence Zitnick, and Martial Hebert.  Shuffle and learn: unsupervised learning using temporal order verification.  In European Conference on Computer Vision, pages 527–544. Springer, 2016.
  * [23] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Cooperative learning of audio and video models from self-supervised synchronization.  Advances in Neural Information Processing Systems, 31, 2018.
  * [24] Relja Arandjelovic and Andrew Zisserman.  Look, listen and learn.  In Proceedings of the IEEE International Conference on Computer Vision, pages 609–617, 2017.
  * [25] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707–6717, 2020.
  * [26] Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer.  Compressive visual representations.  Advances in Neural Information Processing Systems, 34, 2021.
  * [27] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733–3742, 2018.
  * [28] Michael Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297–304. JMLR Workshop and Conference Proceedings, 2010.
  * [29] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv e-prints, pages arXiv–1807, 2018.
  * [30] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar.  A theoretical analysis of contrastive unsupervised representation learning.  In International Conference on Machine Learning, pages 5628–5637. PMLR, 2019.
  * [31] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In European conference on computer vision, pages 776–794. Springer, 2020.
  * [32] Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka.  Contrastive learning with hard negative samples.  arXiv preprint arXiv:2010.04592, 2020.
  * [33] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus.  Hard negative mixing for contrastive learning.  Advances in Neural Information Processing Systems, 33:21798–21809, 2020.
  * [34] Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra.  Can contrastive learning avoid shortcut solutions?  Advances in Neural Information Processing Systems, 34, 2021.
  * [35] Saeed Shurrab and Rehab Duwairi.  Self-supervised learning methods and applications in medical imaging analysis: A survey.  arXiv preprint arXiv:2109.08685, 2021.
  * [36] Xiaomeng Li, Xiaowei Hu, Xiaojuan Qi, Lequan Yu, Wei Zhao, Pheng-Ann Heng, and Lei Xing.  Rotation-oriented collaborative self-supervised learning for retinal disease diagnosis.  IEEE Transactions on Medical Imaging, 40(9):2284–2294, 2021.
  * [37] Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, Vivek Natarajan, and Mohammad Norouzi.  Big self-supervised models advance medical image classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3478–3488, October 2021.
  * [38] Nima Tajbakhsh, Yufei Hu, Junli Cao, Xingjian Yan, Yi Xiao, Yong Lu, Jianming Liang, Demetri Terzopoulos, and Xiaowei Ding.  Surrogate supervision for medical image analysis: Effective deep learning from limited quantities of labeled data.  In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pages 1251–1255. IEEE, 2019.
  * [39] Jianbo Jiao, Richard Droste, Lior Drukker, Aris T Papageorghiou, and J Alison Noble.  Self-supervised representation learning for ultrasound video.  In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 1847–1850. IEEE, 2020.
  * [40] Aiham Taleb, Winfried Loetzsch, Noel Danz, Julius Severin, Thomas Gaertner, Benjamin Bergner, and Christoph Lippert.  3d self-supervised methods for medical imaging.  Advances in Neural Information Processing Systems, 33:18158–18172, 2020.
  * [41] Tobias Ross, David Zimmerer, Anant Vemuri, Fabian Isensee, Manuel Wiesenfarth, Sebastian Bodenstedt, Fabian Both, Philip Kessler, Martin Wagner, Beat Müller, Hannes Kenngott, Stefanie Speidel, Annette Kopp-Schneider, Klaus Maier-Hein, and Lena Maier-Hein.  Exploiting the potential of unlabeled endoscopic video data with self-supervised learning.  International journal of computer assisted radiology and surgery, 13(6):925–933, 2018.
  * [42] Jiuwen Zhu, Yuexiang Li, Yifan Hu, Kai Ma, S Kevin Zhou, and Yefeng Zheng.  Rubik’s cube+: A self-supervised feature learning framework for 3d medical image analysis.  Medical image analysis, 64:101746, 2020.
  * [43] Hongming Li and Yong Fan.  Non-rigid image registration using self-supervised fully convolutional networks without training data.  In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 1075–1078. IEEE, 2018.
  * [44] Hari Sowrirajan, Jingbo Yang, Andrew Y Ng, and Pranav Rajpurkar.  Moco pretraining improves representation and transferability of chest x-ray models.  In Medical Imaging with Deep Learning, pages 728–744. PMLR, 2021\.
  * [45] Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael B Gotway, and Jianming Liang.  Models genesis.  Medical image analysis, 67:101840, 2021.
  * [46] Wenjia Bai, Chen Chen, Giacomo Tarroni, Jinming Duan, Florian Guitton, Steffen E Petersen, Yike Guo, Paul M Matthews, and Daniel Rueckert.  Self-supervised learning for cardiac mr image segmentation by anatomical position prediction.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 541–549. Springer, 2019.
  * [47] David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  arXiv preprint arXiv:1609.09106, 2016.
  * [48] Sylwester Klocek, Łukasz Maziarka, Maciej Wołczyk, Jacek Tabor, Jakub Nowak, and Marek Śmieja.  Hypernetwork functional image representation.  In International Conference on Artificial Neural Networks, pages 496–510. Springer, 2019.
  * [49] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston.  Smash: one-shot model architecture search through hypernetworks.  arXiv preprint arXiv:1708.05344, 2017.
  * [50] Jonathan Lorraine and David Duvenaud.  Stochastic hyperparameter optimization through hypernetworks.  arXiv preprint arXiv:1802.09419, 2018.
  * [51] Andrew Hoopes, Malte Hoffmann, Bruce Fischl, John Guttag, and Adrian V Dalca.  Hypermorph: amortized hyperparameter learning for image registration.  In International Conference on Information Processing in Medical Imaging, pages 3–17. Springer, 2021.
  * [52] Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam.  Condconv: Conditionally parameterized convolutions for efficient inference.  Advances in Neural Information Processing Systems, 32, 2019.
  * [53] Nicholas J Tustison, Philip A Cook, Arno Klein, Gang Song, Sandhitsu R Das, Jeffrey T Duda, Benjamin M Kandel, Niels van Strien, James R Stone, James C Gee, and Brain B Avants.  Large-scale evaluation of ants and freesurfer cortical thickness measurements.  Neuroimage, 99:166–179, 2014.
  * [54] Johannes Hofmanninger, Forian Prayer, Jeanny Pan, Sebastian Röhrich, Helmut Prosch, and Georg Langs.  Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem.  European Radiology Experimental, 4(1):1–13, 2020.
  * [55] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [56] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  arXiv preprint arXiv:1511.07289, 2015.
  * [57] MONAI Consortium.  MONAI: Medical Open Network for AI, 3 2020.
  * [58] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [59] Elizabeth A Regan, John E Hokanson, James R Murphy, Barry Make, David A Lynch, Terri H Beaty, Douglas Curran-Everett, Edwin K Silverman, and James D Crapo.  Genetic epidemiology of copd (copdgene) study design.  COPD: Journal of Chronic Obstructive Pulmonary Disease, 7(1):32–43, 2011.
  * [60] SP Morozov, AE Andreychenko, NA Pavlov, AV Vladzymyrskyy, NV Ledikhova, VA Gombolevskiy, Ivan A Blokhin, PB Gelezhe, AV Gonchar, and V Yu Chernina.  Mosmeddata: Chest ct scans with covid-19 related findings dataset.  arXiv preprint arXiv:2005.06465, 2020.
  * [61] Peter J Castaldi, Raúl San José Estépar, Carlos S Mendoza, Craig P Hersh, Nan Laird, James D Crapo, David A Lynch, Edwin K Silverman, and George R Washko.  Distinct quantitative computed tomography emphysema patterns are associated with physiology and function in smokers.  American journal of respiratory and critical care medicine, 188(9):1083–1090, 2013.
  * [62] Carlos S Mendoza, George R Washko, James C Ross, Alejandro A Diaz, David A Lynch, James D Crapo, Edward K Silverman, Begoña Acha, Carmen Serrano, and R San José Estépar.  Emphysema quantification in a multi-scanner hrct cohort using local intensity distributions.  In 2012 9th IEEE International Symposium on Biomedical Imaging (ISBI), pages 474–477. IEEE, 2012.
  * [63] Sihong Chen, Kai Ma, and Yefeng Zheng.  Med3d: Transfer learning for 3d medical image analysis.  arXiv preprint arXiv:1904.00625, 2019.
  * [64] Li Sun, Ke Yu, and Kayhan Batmanghelich.  Context matters: Graph-based self-supervised representation learning for medical images.  In Proceedings of the… AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, volume 35, pages 4874–4882. NIH Public Access, 2021.
  * [65] Jenna Schabdach, William M Wells, Michael Cho, and Kayhan N Batmanghelich.  A likelihood-free approach for characterizing heterogeneous diseases in large-scale studies.  In International Conference on Information Processing in Medical Imaging, pages 170–183. Springer, 2017.
  * [66] Germán González, Samuel Y Ash, Gonzalo Vegas-Sánchez-Ferrero, Jorge Onieva Onieva, Farbod N Rahaghi, James C Ross, Alejandro Díaz, Raúl San José Estépar, and George R Washko.  Disease staging and prognosis in smokers using deep learning in chest computed tomography.  American journal of respiratory and critical care medicine, 197(2):193–203, 2018.
  * [67] Sumedha Singla, Mingming Gong, Siamak Ravanbakhsh, Frank Sciurba, Barnabas Poczos, and Kayhan N Batmanghelich.  Subject2vec: generative-discriminative approach from a set of image patches to a vector.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 502–510. Springer, 2018.
  * [68] David R Cox.  Regression models and life-tables.  Journal of the Royal Statistical Society: Series B (Methodological), 34(2):187–202, 1972.
  * [69] Bartolome R Celli, Claudia G Cote, Jose M Marin, Ciro Casanova, Maria Montes de Oca, Reina A Mendez, Victor Pinto Plata, and Howard J Cabral.  The body-mass index, airflow obstruction, dyspnea, and exercise capacity index in chronic obstructive pulmonary disease.  New England Journal of Medicine, 350(10):1005–1012, 2004.
  * [70] Fernando J Martinez, Gregory Foster, Jeffrey L Curtis, Gerard Criner, Gail Weinmann, Alfred Fishman, Malcolm M DeCamp, Joshua Benditt, Frank Sciurba, Barry Make, Zab Mohsenifar, Philip Diaz, Eric Hoffman, Robert Wise, and NETT Research Group.  Predictors of mortality in patients with emphysema and severe airflow obstruction.  American journal of respiratory and critical care medicine, 173(12):1326–1334, 2006.
  * [71] JG Leopold and J Gough.  The centrilobular form of hypertrophic emphysema and its relation to chronic bronchitis.  Thorax, 12(3):219, 1957.
  * [72] BE Heard, V Khatchatourov, H Otto, NV Putov, and L Sobin.  The morphology of emphysema, chronic bronchitis, and bronchiectasis: definition, nomenclature, and classification.  Journal of clinical pathology, 32(9):882, 1979.
  * [73] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.