# Lung Swapping Autoencoder: Learning a Disentangled Structure-texture
Representation of Chest Radiographs

Lei Zhou<EMAIL_ADDRESS>Joseph Bae  Huidong Liu  Gagandeep Singh
Jeremy Green
Amit Gupta  Dimitris Samaras  Prateek Prasanna  Department of Computer
Science, Stony Brook University, NY, USA  Department of Biomedical
Informatics, Stony Brook University, NY, USA  Department of Radiology, Newark
Beth Israel Medical Center, NJ, USA  Department of Radiology, University
Hospitals Cleveland Medical Center, OH, USA

###### Abstract

Well-labeled datasets of chest radiographs (CXR) are often difficult to
acquire due to the high cost of annotation. Thus, it is desirable to learn a
robust and transferable representation in an unsupervised manner to benefit
downstream tasks that lack annotated data. Unlike natural images, medical
images often have their own domain prior; e.g., we observe that many pulmonary
diseases, such as the Coronavirus Disease 2019 (COVID-19), manifest as
characteristic changes in the lung tissue texture rather than the anatomical
structure. Therefore, we hypothesize that studying only the tissue texture
without the influence of possible structure variations would be advantageous
for downstream prognostic and predictive modeling tasks. In this paper, we
propose a generative framework, the Lung Swapping Autoencoder (LSAE), that
learns a factorized representation of a CXR to disentangle the tissue texture
representation from the anatomic structure representation. Specifically, by
adversarial training, the LSAE is optimized to generate a hybrid image that
preserves the lung shape in one image but inherits the lung texture of another
image. To demonstrate the effectiveness of the disentangled texture
representation, we evaluate the texture encoder $Enc^{t}$ in LSAE on a large-
scale thoracic disease dataset, ChestX-ray14 (N=112,120), and our own multi-
institutional COVID-19 outcome prediction dataset, COVOC (N=340 (Subset-1) +
53 (Subset-2)). On both datasets, we reach or surpass the state-of-the-art by
finetuning the texture encoder in LSAE with a model that is 77$\%$ smaller
than a baseline Inception v3 architecture. Additionally, in semi-supervised
and linear evaluation settings with a similar model budget, the texture
encoder in LSAE is also competitive with the state-of-the-art, MoCo. By “re-
mixing” the disentangled texture and shape factors, we generate meaningful
hybrid images that can augment the training set. The proposed data
augmentation method can further improve COVOC prediction performance.
Moreover, the improvement is consistent even when we directly evaluate the
Subset-1 trained model on Subset-2 without additional finetuning. The code is
available at: https://github.com/cvlab-stonybrook/LSAE

###### keywords:

Chest Radiographs , Disentanglement , Lung Swapping Autoencoder , Unsupervised
Learning

††journal: Journal of LaTeX Templates

##  1 Introduction

Chest radiographs (CXRs) are the primary imaging modality for front-line
evaluation of suspected lung diseases, especially in the COVID-19 pandemic.
They are widely available, portable, and have reduced risk of cross infection
compared to Computed Tomography (CT). Despite these advantages, CXRs are also
less sensitive to subtle disease changes compared to CT scans [Litmanovich et
al. [2020]]. For CXRs, it is often challenging to collect large-scale well-
annotated datasets, especially for emerging diseases like COVID-19, due to the
high cost of expert annotation. Therefore, it is important to develop
algorithms that are able to learn a robust and transferable representation in
an unsupervised/self-supervised way. Most self-supervised methods are designed
for natural images and do not exploit any domain-specific priors that might
exist for medical images.

COVID-19 manifests as infiltrates such as ground-glass opacities on CXR
[Litmanovich et al. [2020]]. These findings are examples of changes in texture
that appear as fuzzy-white areas within the lung fields. However, chest
anatomy remains mostly unchanged throughout the disease course of an
individual patient. Therefore, we hypothesize that disease information is more
related to lung tissue texture rather than the anatomical structure of the
lung. To be concise, we use the terms texture and structure in the rest of the
article.

Figure 1: Lung Swapping Result. Two examples of lung swapping between images
in column $I_{1}$ and images in column $I_{2}$. The Lung Swapping Autoencoder
(LSAE) is able to successfully transfer target lung textures without affecting
the lung shape. The swapping results are shown in the second and the third
columns.

Motivated by the above observation, we propose the Lung Swapping AutoEncoder
(LSAE), that learns a factorized representation of a CXR to disentangle the
texture factor from the structure factor. LSAE shares the same core idea as
the recently-proposed Swapping AutoEncoder (SAE) [Park et al. [2020b]], namely
that a successful disentanglement model should be able to generate a realistic
hybrid image that merges the structure of one image with the texture of
another. To achieve this goal, images are encoded as a combination of two
latent codes representing structure and texture respectively. The SAE is
trained to generate realistic images from the swapped codes of arbitrary image
pairs in an adversarial learning manner. To supervise the synthesis of target
texture, SAE is further forced to mimic the pattern of patches sampled from
the target texture image. However, this vanilla SAE does not work well for CXR
disentanglement (see Fig 2). First, by sampling texture patches from the whole
target image, irrelevant out-of-lung textures diminish the effect of the in-
lung texture transfer (see the upper row in Fig 2). Second, because texture
supervision is derived from image patches, irrelevant structure clues may leak
into the hybrid image, resulting in undesired lung shape distortion and
interference with successful disentanglement (see the lower row in Fig 2).

We address the above two problems with two contributions. The first
contribution is based on the observation that the infiltrates of interest are
located within lung zones. Hence, sampling patches from the lung area instead
of the whole image for texture supervision can alleviate the disturbance
introduced by irrelevant texture patterns. The second contribution is applying
a patchwise contrastive loss which explicitly forces out-of-lung local patches
in the hybrid image to mimic the corresponding patch in the structure image.
Thus, we can prevent structure information in texture patches from leaking
into the hybrid image which can cause undesired lung shape distortion
visually. We introduce this SAE equipped with the new designs as the Lung
Swapping Autoencoder (LSAE). LSAE, trained on a large public CXR dataset,
ChestX-ray14 [Wang et al. [2017]], can generate realistic, plausible hybrid
CXRs with one patient’s lung structure and another patient’s disease texture
(see Fig 1. Please refer to Fig 7 for additional lung swapping examples). We
further demonstrate disentanglement performance quantitatively through disease
texture distance, lung segmentation metrics, and radiologist feedback.

If textures do indeed represent disease infiltrates, we expect the texture
encoder $Enc^{t}$ in LSAE to be discriminative in downstream CXR semantic
tasks. We evaluate the learned representation on two datasets, the large-scale
ChestX-ray14 database (N=112,120) and our multi-institutional COVID-19 outcome
prediction dataset, COVOC (N=340(Subset-1)+53(Subset-2)). On ChestX-ray14, we
compare $Enc^{t}$ in LSAE with state-of-the-art methods in linear evaluation,
semi-supervised, and fully-supervised settings. Our method achieves
competitive results with a model of almost half the size of ResNet18. Then we
treat the $Enc^{t}$ as a pre-trained model on ChestX-ray14, and perform
transfer learning to COVOC (Subset-1). $Enc^{t}$ in LSAE reduces the error
rate by 13% compared with a strong baseline Inception v3. Moreover, using
LSAE, we generate hybrid images with textures and labels from COVOC training
data and lung structures from ChestX-ray14. Augmenting with these hybrid
images further improve outcome (ventilation) prediction on COVOC. All
experiments show that $Enc^{t}$ in LSAE learns effective and transferable
representations of lung diseases.

We summarize our contributions as follows: 1) LSAE is the first approach to
disentangle chest CXRs into structure and texture representations. LSAE
achieves this complex task by leveraging explicit in-lung texture supervision
and out-of-lung distortion suppression. 2) We achieve competitive performance
compared with state-of-the-art approaches on the large-scale ChestX-ray14
dataset in both fully-supervised and semi-supervised settings for predicting
pulmonary diseases. 3) We achieve superior performance for COVID-19 outcome
prediction with an efficient model. 4) We propose a hybrid image augmentation
technique to further improve clinical outcome prediction.

Figure 2: Two Problems in SAE. The first problem is the irrelevant out-of-lung
texture transfer as shown in the upper row. As indicated by the red arrows,
these two out-of-lung regions show similar texture patterns which is
undesired. The second problem is the lung shape distortion as shown in the
lower row. The  green contour outlines the shape of lung in $I_{1}$ which
should be preserved in the middle hybrid image. However, the lung shape in the
middle hybrid image outlined in  red looks obviously distorted from the  green
contour.

##  2 Related Work

##### Image-to-image Translation and Image Manipulation

Image-to-image translation can be viewed as a conditional generation task. It
takes a source image as input and transforms it into another target domain,
such as a semantic image or manipulated image, as demonstrated in the
following works [Isola et al. [2017], Zhu et al. [2017], Huang et al. [2018],
Choi et al. [2018], Liu et al. [2019]]. These methods tackle the image-to-
image translation task under different experimental settings, such as
supervised, unsupervised, multi-modal, and few-shot conditions. We can
categorize these approaches with respect to different architectures and loss
functions. Most image-to-image models adopt U-Net or Autoencoder
architectures. The key difference between the two genres lies in the skip
connection. Equipped with skip connections, U-Net-based models can be
optimized efficiently and accurately. However, Autoencoders have the advantage
that a compact latent space is preserved. Specifically, an Autoencoder is
usually composed of an encoder and a decoder. The encoder maps from the source
image to a latent space, and the decoder maps from the latent space to the
target image. As for loss functions, most methods adopt a reconstruction loss.
To synthesize more realistic images, adversarial loss can sometimes be applied
as well. Additionally, cycle-consistency loss [Zhu et al. [2017]] is useful
when no paired data is available.

To manipulate the image, the latent space of the Autoencoder needs to be
disentangled and semantically meaningful. After that, by interpolating along a
particular latent direction, the decoder can generate target/manipulated
images. Our baseline model Swapping Autoencoder (SAE) [Park et al. [2020b]]
also follows the above routine for image manipulation. We will introduce SAE
in the methodology section.

##### Self-supervised Representation Learning

Current self-supervised representation learning methods primarily fall in two
bins, contrastive learning and siamese non-contrastive learning. Generally,
self-supervision originates from the invariance of representation with respect
to augmentation transformations. For contrastive learning [Chen et al.
[2020a]], a network is trained to attract features of views from one image but
repulse views from different images. However, contrastive training is hungry
for negative pairs which requires a very large batch to achieve a good result.
To alleviate the heavy memory consumption burden, Wu et al. [2018] proposed a
memory bank containing only feature vectors for a large batch of data.
Alternatively, MoCo [He et al. [2020], Chen et al. [2020b]] utilizes a queue
of representation vectors produced by a momentum encoder. Recently, siamese
networks have been proposed that can learn effective representations without
contrastive loss. Specifically, Grill et al. [2020] proposed a simple model
called BYOL (Bootstrap Your Own Latent) which removes the negative pair
repelling in training. Surprisingly, BYOL does not collapse to a trivial
solution due to the absence of negative pairs. Instead, with a momentum
encoder and the siamese architecture, BYOL achieves even better performance
than contrastive counterparts. After BYOL, Chen & He [2021] proposed a simpler
version by removing momentum encoder with direct copying in BYOL, and proved
empirically that a stop gradient with respect to the target branch made the
siamese non-contrastive learning work particularly well. Moreover, Tian et al.
[2021] analyzed these non-contrastive siamese frameworks from a theoretical
view, justifying some key designs including the stop gradient and asymmetric
predictor. Compared with contrastive learning, non-contrastive learning does
not require a very large min-batch to work well. However, medical images have
their own domain priors. For example, as mentioned above, CXRs for a patient
have similar lung structures but different textures. All of the above methods
are designed for natural images which do not consider the domain knowledge of
medical images.

##### COVID-19 Imaging Studies

Previous medical image analysis in COVID-19 has mostly focused on disease
diagnosis [Hu et al. [2021], López-Cabrera et al. [2021]]. CT scan based
models have done better in predicting COVID-19 disease outcomes compared to
CXR-based models [Chassagnon et al. [2021], López-Cabrera et al. [2021], Li et
al. [2021], Bae et al. [2020]]. This is largely due to the lack of COVID-19
CXR datasets with relevant endpoints and the limited information that CXRs
contain relative to CT scans. Several data augmentation techniques have been
proposed for CT scans in the COVID-19 setting [Li et al. [2021]], but CXR
approaches have continued to rely on publicly sourced datasets [Cohen et al.
[2020], López-Cabrera et al. [2021]]. These datasets tend to be homogeneous
within COVID-19 positive and negative classes, and often lack disease outcome
labels (hospitalization, mechanical ventilation requirement, etc.) [López-
Cabrera et al. [2021], Cohen et al. [2020]].

##  3 Methodology

First, we briefly review SAE in Section 3.1. Then, in Section 3.2, we present
our Lung Swapping Autoencoder (LSAE) model and emphasize our key designs.
Lastly, in Section 3.3, we illustrate how to generate hybrid images for data
augmentation.

###  3.1 Swapping AutoEncoder (SAE)

Recently, Swapping AutoEncoder (SAE) [Park et al. [2020b]] is proposed for
image manipulation by disentanglement. It consists of an encoder $Enc$ and a
decoder $Dec$, where $Enc$ is composed of a structure branch $Enc^{s}$ and a
texture branch $Enc^{t}$ to encode the input into structure and texture latent
codes $z^{s}$ and $z^{t}$ separately.

####  3.1.1 Latent Code Swapping.

To achieve the goal of disentanglement, SAE aims to generate realistic images
from swapped latent codes of arbitrary image pairs. Specifically, two sampled
images $I_{1}$ and $I_{2}$ are first encoded as $(z_{1}^{s},z_{1}^{t})$ and
$(z_{2}^{s},z_{2}^{t})$. Then, the latent codes are swapped to get a hybrid
code $(z_{1}^{s},z_{2}^{t})$. Finally, the hybrid code is decoded to yield a
hybrid image $I_{\texttt{hybrid}}$ which is expected to maintain the structure
of $I_{1}$ but present the texture of $I_{2}$. We use $G$ to denote the
composite generation process as

$G(I_{1},I_{2})\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}Dec(Enc^{s}(I_{1}),Enc^{t}(I_{2}))$

To optimize the network, we try to minimize the reconstruction error and also
take advantage of adversarial training by introducing an auxiliary
discriminator.

$\displaystyle\mathcal{L}_{\texttt{recon}}=\mathop{\mathbb{E}}_{I_{1}\sim\mathbf{X}}\|G(I_{1},I_{1})-I_{1}\|_{1}$
$\displaystyle\mathcal{L}_{\texttt{G}}=\mathop{\mathbb{E}}_{{I_{1},I_{2}}\sim\mathbf{X}}-\log\big{(}D\big{(}G(I_{1},I_{1})\big{)}\big{)}-\log\big{(}D\big{(}G(I_{1},I_{2})\big{)}\big{)}$

where $\mathbf{X}$ denotes the image training set, and $D$ is a discriminator
[Goodfellow et al. [2014]]. Note that the complete GAN loss also includes a
discriminator part. To be concise, we only show the generator part.

####  3.1.2 Texture Supervision.

To ensure $I_{\texttt{hybrid}}$ texture matches that of $I_{2}$, SAE samples
patches from $I_{2}$ to supervise the $I_{\texttt{hybrid}}$ texture. Another
auxiliary network, patch discriminator $D_{\texttt{patch}}$, is trained to
distinguish patches in $I_{\texttt{hybrid}}$ from patches in $I_{2}$.
Adversarially, SAE is trained to generate a $I_{\texttt{hybrid}}$ whose
patches can confuse $D_{\texttt{patch}}$ by mimicking the texture of $I_{2}$.
The texture loss is formulated as

$\mathcal{L}_{\texttt{tex}}=\mathbb{E}_{\begin{subarray}{c}\tau_{1},\tau_{2}\sim\mathcal{T}\\\
I_{1},I_{2}\sim\mathbf{X}\end{subarray}}\big{[}-\log\big{(}D_{\texttt{patch}}\big{(}\tau_{1}\big{(}I_{2}\big{)},\tau_{2}\big{(}G(I_{1},I_{2})\big{)}\big{)}\big{)}\big{]}$

where $\mathcal{T}$ is the distribution of multi-scale random cropping, and
$\tau_{1}$ and $\tau_{2}$ are two random operations sampled from
$\mathcal{T}$.

###  3.2 Lung Swapping AutoEncoder (LSAE)

Figure 3: Lung Swapping AutoEncoder (LSAE) consists of a structure encoder, a
texture encoder, and a decoder. After inputting images $(I_{1},I_{2})$, the
LSAE generates a hybrid image with factorized representations from $I_{1}$ and
$I_{2}$ respectively. To ensure the texture in $I_{\texttt{hybrid}}$ matches
$I_{2}$, we adversarially train a patch discriminator $D_{\texttt{patch}}$ to
supervise texture synthesis within lungs. To ensure $I_{\texttt{hybrid}}$
maintains the structure of $I_{1}$, we apply a patchwise contrastive loss
outside the lungs to minimize structural distortion.

Using SAE directly to generate a hybrid CXR from an image pair does not work
as desired. First, the disease level in $I_{\texttt{hybrid}}$ is usually
diminished when compared with $I_{2}$. We hypothesize that the out-of-lung
irrelevant texture patterns may hamper target texture synthesis in
$I_{\texttt{hybrid}}$. Results in Table 4 support our analysis. Please refer
to the experimental part for detailed discussion on the results. Second, since
there is no structure supervision in SAE, $I_{\texttt{hybrid}}$ can show
undesired lung shape distortion towards $I_{2}$ which indicates a suboptimal
disentanglement. These problems lead us to design two new strategies, in-lung
texture supervision and out-of-lung structural distortion suppression. We name
the SAE model equipped with the new designed strategies the Lung Swapping
AutoEncoder (LSAE).

####  3.2.1 In-lung Texture Supervision.

As infiltrates on CXRs are observed inside lung regions, we should cast the
focus of texture synthesis in the in-lung region. Hence, we can adapt SAE by
sampling texture patches from within the lung zone instead of the whole image.
Guided by a lung segmentation mask which can be computed by a pre-trained
model, we rewrite the texture supervision loss as,

$\mathcal{L}_{\texttt{inTex}}=\mathop{\mathbb{E}}_{\begin{subarray}{c}\tau_{1},\tau_{2}\sim\mathcal{T}_{\texttt{LungMask}}^{\texttt{in}}\\\
I_{1},I_{2}\sim\mathbf{X}\end{subarray}}\bigg{[}-\log\bigg{(}D_{\texttt{patch}}\Big{(}\tau_{1}\big{(}I_{2}\big{)},\tau_{2}\big{(}G(I_{1},I_{2})\big{)}\Big{)}\bigg{)}\bigg{]}$
(1)

where most items are the same as $\mathcal{L}_{\texttt{tex}}$ except
$\mathcal{T}$ in $\mathcal{L}_{\texttt{tex}}$, which is now replaced by
$\mathcal{T}_{\texttt{LungMask}}^{\texttt{in}}$, indicating that we only
sample patches from in-lung regions to supervise the synthesis of target
texture in $I_{2}$.

####  3.2.2 Out-of-lung Structural Distortion Suppression.

To suppress the undesired structural distortion, we further introduce a
patchwise contrastive loss [Park et al. [2020a]] in LSAE. The patchwise
contrastive loss, based on Noise Contrastive Estimation [Gutmann & Hyvärinen
[2010]], is designed to preserve image content in image-to-image translation
tasks. Hence, we adapt the patchwise contrastive loss to help our task by
applying it in the following manner: We first encode $I_{\texttt{hybrid}}$
with structure encoder $Enc^{s}$. Then, we randomly sample a feature vector
$q_{i}$ from the $h^{th}$ layer’s output $Enc^{s}_{h}(I_{\texttt{hybrid}})$,
and a bag of feature vectors
$\\{p^{+},p^{-}_{1},p^{-}_{2},\cdots,p^{-}_{N-1}\\}$ from
$Enc^{s}_{h}(I_{1})$, where $p^{+}$ is from the corresponding position with
$q_{i}$, and $\\{p^{-}\\}$ from other positions. The loss objective is to
force $q_{i}$ from $I_{\texttt{hybrid}}$ to attract to the corresponding
feature vector $p^{+}$ but repel the others $\\{p^{-}\\}_{N-1}$ from $I_{1}$.
Therefore, we write it as a cross-entropy loss by regarding the process as an
$N$-way classification problem where $q_{i}$ is the query and $p^{+}$ is the
target:

$l_{\texttt{NCE}}(q_{i},\\{p\\}_{N})=-\log\left(\frac{\exp(q_{i}\cdot
p^{+}/\alpha)}{\exp(q_{i}\cdot p^{+}/\alpha)+\sum_{j=1}^{N-1}\exp(q_{i}\cdot
p^{-}_{j}/\alpha)}\right)$ (2)

where $\alpha$ is the temperature, and
$\\{p\\}_{N}=\\{p^{+}\\}\cup\\{p^{-}\\}_{N-1}$. Since we found experimentally
that applying $l_{\texttt{NCE}}$ to the whole spatial region can disturb the
texture synthesis within lungs, we only apply it outside the lung region.
Thus, the final structural distortion suppression loss can be expressed as

$\mathcal{L}_{\texttt{sup}}=\mathop{\mathbb{E}}_{\begin{subarray}{c}\\{\tau\\}_{N}\sim\mathcal{T}^{\texttt{out}}_{\texttt{LungMask}}\\\
I_{1},I_{2}\in\mathbf{X}\end{subarray}}\sum_{h}^{H}\sum_{i}^{N}l_{\texttt{NCE}}\Big{(}\tau_{i}\big{(}Enc^{s}_{h}(I_{\texttt{hybrid}})),\\{\tau\big{(}Enc^{s}_{h}(I_{1})\big{)}\\}_{N}\Big{)}$
(3)

where $\mathcal{T}^{\texttt{out}}_{\texttt{LungMask}}$ is the distribution of
position sampling outside the lung region guided by the mask, and $H$ is the
number of layers we apply the loss to.

####  3.2.3 The Overall Objective.

Combining the core designs in LSAE, we term the final loss as

$\mathcal{L}=\mathcal{L}_{\texttt{recon}}+\lambda_{1}\mathcal{L}_{\texttt{G}}+\lambda_{2}\mathcal{L}_{\texttt{inTex}}+\lambda_{3}\mathcal{L}_{\texttt{sup}}$
(4)

where $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are tunable hyper-
parameters. The selected hyper-parameters in experiments are provided further
in Sec. 4.2.

###  3.3 Hybrid Image Augmentation

If the texture in $I_{2}$ can be transferred to $I_{\texttt{hybrid}}$, we
assume the label (e.g., ventilation) of $I_{2}$ is also attached to
$I_{\texttt{hybrid}}$, even if the lung structure is different. Based on this
hypothesis, we design a new method for data augmentation: Given an image
$I^{\texttt{dst}}$ in the target domain, e.g., COVID-19 Outcome prediction
dataset, COVOC, in our context, we sample $K$ images
$\\{I^{\texttt{{src}}}\\}$ from a source domain, e.g., ChestX-ray14. Then, we
use LSAE to take $I^{\texttt{dst}}$ as the texture template and images from
$\\{I^{\texttt{{src}}}\\}$ as structure templates to generate $K$ hybrid
images $\\{I_{\texttt{hybrid}}\\}$. We label $\\{I_{\texttt{hybrid}}\\}$ with
the label of $I^{\texttt{dst}}$. Following this protocol, we can enlarge the
training set in the target domain to $(K+1)$ times of origial size.
Experimental results show that training with the additional generated hybrid
images can improve performance further on the COVID-19 ventilation prediction
task.

##  4 Experimental Design and Results

###  4.1 Dataset Description

ChestX-ray14 [Wang et al. [2017]] is a large-scale CXR database consisting of
112,120 frontal-view CXRs from 32,717 patients. We report all the results
based on the official split which consists of training ($\sim 70\%$),
validation ($\sim 10\%$), and testing ($\sim 20\%$) sets. Images from the same
patient will only appear in one of the sets.

COVID-19 Outcome (COVOC) is a COVID-19 CXR dataset curated from three
institutes. Each CXR in COVOC is labeled based upon whether the patient
required mechanical ventilation (henceforth ventilation) or not. Specifically,
we split COVOC into two subsets. Subset-1 constitutes of 340 CXRs from 327
COVID-19 patients acquired in the first two institutes upon disease
presentation. We further separate this subset randomly into 3 splits. Each
split has 250 samples for training, 30 for validation, and 60 for testing;
Subset-2 are composed of 53 images from the third institute. We only use this
subset for testing in order to demonstrate the effectiveness of our method
when generalizing to a new domain without adapting. As for positive/negative
cases distribution, we list the statistics in Table 1.

|  Subset-1 Train/Val/Test/Subtotal |  Subset-2 Subtotal(Test) | Total
---|---|---|---
positive | 81/10/20/111 | 27 | 138
negative | 169/20/40/229 | 26 | 255
Table 1: Class-wise data distribution on COVOC. for prediction of ventilation
requirement. There are two subsets in COVOC. Subset-1 is used to train,
validate, and test models. Subset-2 is reserved only for evaluating the
generalizing ability of models (usually trained on Subset-1 training set) to a
new domain without any adaptation.

###  4.2 Implementation

Architecture. In LSAE, both decoder and discriminator architectures follow
StyleGAN2. Encoders are built with residual blocks [He et al. [2016]]. Among
them, the texture encoder outputs a flattened vector while structure encoder’s
output preserves spatial dimension.
Preprocessing. The input image size of both encoder and discriminator is
$256\times 256$. Patch sizes sampled for texture supervision vary from
$16\times 16$ to $64\times 64$. We also randomly rotate the cropped patches up
to 60 degrees. We preprocess all CXRs with histogram equalization. The lung
masks are pre-computed by a pre-trained segmentation model from [Konwer et al.
[2021]].
Hyper-parameters. The temperature $\alpha$ in Eq. 2 is set to 0.07. We set
$\lambda_{1}=0.5$, $\lambda_{2}=1$, and $\lambda_{3}=1$ in Eq. 4.

Questions | Images
---|---
Q1: Is this a real or a fake image? |
Q2: Is this patch cropped from a real image? |
Q3: Which CXR (A or B) has a more similar disease level to the reference one? |
Q4: Which CXRs do you think have fibrosis/pneumonia/infiltration? |
Table 2: Survey Questions. The first two questions seek to assess the image synthesis quality in both the image and patch level. The third question is designed to verify the correctness of the proposed Masked SIFID by measuring the matching rate between radiologists’ judgements and Masked SIFID’s measurements. The last question aims to prove whether LSAE can transfer the target disease texture correctly.  |
---|---
(a) | (b)
|
(c) | (d)
Table 3: Radiologists’ Survey Results. The x-axes of all the plots denote
indices of radiologists with the corresponding years of radiology training.
(a), (b), (c), and (d) show the results for Q1, Q2, Q3, and Q4, respectively.
In the upper row, we also list the confusion rates on real images for
reference. The match rate in lower-left plot is computed by
$\frac{\\#QsHaveSameAnswers}{\\#Qs}$. The success rate in lower-right plot is
computed by
$\frac{\\#QsBothHaveDisease}{\\#QsBothHaveDisease+\\#QsOnlyRefHasDisease}$.
The final averaged results are reported in Sec. 4.3.3.

Training. LSAE is optimized by Adam with a learning rate of 1e-3. Each
training batch contains 16 images that are distributed in 2 NVIDIA GeForce RTX
2080 Ti GPUs. To regularize both discriminators, we apply R1 regularization
every 16 iterations. When finetuning the texture encoder on COVOC, we add a
randomly-initialized linear layer for prediction, and set the batch size to
56. The texture encoder is still optimized by Adam. We adopt an exponential
decay schedule for learning rate which starts from 0.004 and ends at 0.001.
Our code is developed based on [Seonghyeon ] and runs on PyTorch 1.7. We use
Wandb [Biewald [2020]] to monitor the training process.

###  4.3 Hybrid CXR Generation

Our first experiments utilize the large ChestX-ray14 dataset to pre-train LSAE
by learning to generate hybrid CXRs from swapped latent code in an
unsupervised way.

####  4.3.1 Experimental Settings.

We train LSAE on the training set of ChestX-ray14 with a batch of 16 images
for 150K iterations. To evaluate the performance, we define a lung swapping
task as follows, First, we create two sets of 9,000 images, $\\{I^{+}\\}$ and
$\\{I^{-}\\}$, from the test set of ChestX-ray14. $\\{I^{+}\\}$ are sampled
from the images diagnosed with at least one of the diseases: infiltration,
pneumonia, and fibrosis, have been described to share similar imaging findings
with COVID-19 [Salehi et al. [2020]]. In contrast, $\\{I^{-}\\}$ are sampled
from healthy lungs. We pair each image in $\\{I^{+}\\}$ with an image in
$\\{I^{-}\\}$ in a random order. Thus, we can now generate two hybrid images
by mixing texture and structure for each image pair in both directions, i.e.,
$G(I^{+},I^{-})$ and $G(I^{-},I^{+})$. We will measure the distance of the set
of hybrid images between both the target texture image set and the structure
image set.

Figure 4: Comparison with SAE. As indicated in Table 4, LSAE can suppress lung
shape distortion while synthesizing target texture. Here we compare LSAE with
SAE in a qualitative manner. Similar to Fig. 1, the first and last columns are
real images for mixing. The second column is the hybrid image generated by the
baseline SAE while the third column is generated by our LSAE. It is obvious
that SAE suffers from shape distortion while LSAE can suppress the undesired
distortion effectively.

####  4.3.2 Evaluation Protocol

Quantitative evaluation. First, to measure the disease level distance between
$I_{\texttt{hybrid}}$ and $I_{2}$, we propose a new metric, Masked SIFID.
Masked SIFID is designed based on SIFID [Rott Shaham et al. [2019]] which
calculates the FID distance between two images in the Inception v3 feature
space. To customize it to disease level distance, we only consider features
within the lung region. Additionally, we use the ChestX-ray14 pre-trained
Inception v3 (mAUC: 79.56$\%$) to infer features. Second, to quantify
structural distortion, we use lung segmentation metrics as surrogates. Given a
lung segmentation model $Seg$ [Konwer et al. [2021]], we can compute
segmentation metrics by treating $Seg(G(I_{1},I_{2}))$ as the query and
$Seg(I_{1})$ as the ground truth.

Qualitative evaluation. To evaluate the results qualitatively, we solicited
feedback from 5 radiologists through a 4-question survey. The questions are
shown in Table. 2. In particular, the first two questions seek to assess the
image synthesis quality in both the image and patch level. The third question
is designed to verify the correctness of the proposed Masked SIFID. The last
question aims to prove whether LSAE can transfer the target disease texture
correctly.

Method  | Masked SIFID $\downarrow$ | mIoU $\uparrow$ | Pixel Acc $\uparrow$ | Dice $\uparrow$
---|---|---|---|---
Init  | 0.0335 | 0.60 | 0.82 | 0.72
SAE  | 0.0257 | 0.76 | 0.91 | 0.85
LSAE  | 0.0245 | 0.91 | 0.97 | 0.95
Table 4: Evaluation of Hybrid Image Generation. We report Masked SIFID for evaluation of disease transgfer. To evaluate distortion suppression, we report mIoU, pixel accuracy, and Dice. LSAE surpasses SAE in both texture synthesis and structure maintenance. We report the average results of texture transfer over two directions, i.e., texture transfer from $\\{I^{+}\\}$ to $\\{I^{-}\\}$, and from $\\{I^{-}\\}$ to $\\{I^{+}\\}$. Pre-train | Method | Params | mAUC $\uparrow$
---|---|---|---
Supervised | CXR14-R50 [Wang et al. [2017]] | 23M | 0.745
ChestNet [Wang & Xia [2018]] | 60M | 0.781
CheXNet∗ [Rajpurkar et al. [2017]] | 7M | 0.789
Inception v3 | 22M | 0.796
Unsupervised | MoCo-R18 [He et al. [2020]] | 11M | 0.786
$Enc^{t}$ in LSAE | 5M | 0.790
Table 5: Performance comparison on ChestX-ray14. The texture encoder in LSAE achieves competitive results with a smaller model size. ∗Note that the data split in [Rajpurkar et al. [2017]] is not released. We reimplemented CheXNet on the official split. Note that CheXNet follows the structure of DenseNet-121. The inception v3 model is also self-implemented. Lung Diseases | ChestNet [Wang & Xia [2018]] |  $Enc^{t}$ in LSAE
---|---|---
Atelectasis | 0.7433 | 0.7545
Cardiomegaly | 0.8748 | 0.8668
Effusion | 0.8114 | 0.8214
Infiltration | 0.6772 | 0.6927
Mass | 0.7833 | 0.7948
Nodule | 0.6975 | 0.7380
Pneumonia | 0.6959 | 0.7088
Pneumothorax | 0.8098 | 0.8442
Consolidation | 0.7256 | 0.7297
Edema | 0.8327 | 0.8401
Emphysema | 0.8222 | 0.8547
Fibrosis | 0.8041 | 0.8078
Pleural Thickening | 0.7513 | 0.7489
Hernia | 0.8996 | 0.8667
mAUC | 0.781 | 0.7906
Table 6: Class-wise classification performance on ChestX-ray14. We report
class-wise AUC metric of our model, i.e., the texture encoder in a pre-trained
LSAE. The mean AUC corresponds to the results reported in Table. 5. We also
list class-wise AUC of ChestNet for a reference.

####  4.3.3 Results

Disease Transfer. In Table 4, to get reference values, we first report initial
results by directly comparing $\\{I^{-}\\}$ with $\\{I^{+}\\}$. LSAE achieves
Masked SIFID of 0.0245 when compared with 0.0257 using SAE and the initial
distance 0.0335, demonstrating that our design of in-lung texture supervision
works as expected.

Distortion Suppression. LSAE also outperforms SAE by an average of 12.7% in
all segmentation metrics, achieving over 90$\%$ in all segmentation metrics,
which proves out-of-lung patchwise contrastive loss can effectively suppress
structural distortion. In Fig. 4, we visualize the image swapping results of
both the baseline SAE and our LSAE. It can be observed qualitatively that LSAE
does perform better with respect to the suppression of lung shape distortion.

Radiologists’ Survey. The results of radiologists’ survey are shown in Fig. 3.
The x-axes of all the plots denote indices of radiologists with the
corresponding years of residency. Respectively, (a), (b), (c), and (d) sub-
figures shows the results for Q1, Q2, Q3, and Q4. In the first two questions
regarding image quality, radiologists misconstrued 56$\%$ of the generated
images as real, and 74$\%$ of the hybrid image patches as real on average. To
get a reference, we also list the confusion rates of real images in the upper
row of Fig. 3 as blue bars. The third question attempted to verify a
correlation between Masked SIFID and disease level distance. After the
radiologists selected the query image closer to the reference, we compute the
proportion of radiologists’ answers matching with Masked SIFID. Specifically,
match rate is computed by $\frac{\\#QsHaveSameAnswers}{\\#Qs}$. On average,
the match rate in Q3 is 78.67$\%$. This justifies the proposed Masked SIFID
for evaluating disease transfer. The fourth question was to ascertain whether
our method can transfer disease correctly. When the radiologists were shown
the original image and the hybrid image generated with the same texture, we
compute the success rate by
$\frac{\\#QsBothHaveDisease}{\\#QsBothHaveDisease+\\#QsOnlyRefHasDisease}$.
Specifically, the success rate of 60$\%$ in Q4 means 60$\%$ of the hybrid
images are identified to have the same lung disease as the original texture
image on average.

###  4.4 Semantic Prediction in CXRs by Texture Encoder

Based on our hypothesis that pulmonary diseases are tightly related to CXR
texture, the texture encoder in a well-trained LSAE should be discriminative
on CXR semantic tasks. To verify this, we evaluate texture encoder $Enc^{t}$
on both lung disease classification and COVID-19 outcome prediction tasks.

Method | Params | 1% labels ft | 10% labels ft | linear evaluation
---|---|---|---|---
MoCo-R18 | 11M | 61.1% | 71.8% |  73.0%
$Enc^{t}$ in LSAE | 5M |  61.3% |  73.2% | 69.0%
Table 7: Comparison with MoCo (ResNet18) in self-supervised (linear
evaluation) and semi-supervised settings. Our method shows superior
performance when compared with MoCo in semi-supervised settings while MoCo
still ourperforms ours in the linear evaluation setting.

####  4.4.1 Disease Classification on ChestX-ray14.

First, to test the extreme performance of our model, we finetune $Enc^{t}$ on
the whole training set of ChestX-ray14 with all the available 14 disease
labels, and report the class average AUC in Table 5. We achieve competitive
results compared with both fully-supervised pre-trained and self-supervised
pre-trained methods (e.g., MoCo), albeit with a smaller model size. We further
provide class-wise AUC in Table 6.

It should be noted that, of the fourteen diseases classified in the ChestX-
ray14 dataset, cardiomegaly and hernia are not lung pathologies. Indeed, as
can be observed in Table 6, the texture encoder $Enc^{t}$ in LSAE achieves
poorer results than ChestNet in these two classes.

Second, to compare with other state-of-the-art self-supervised and semi-
supervised methods, we examine our model in three restricted settings, i.e.,
linear evaluation, 1% labeled data fine-tuning, and 10% labeled data fine-
tuning. Specifically, linear evaluation freezes the whole backbone network but
only updates the final linear classifier. 1% labeled data fine-tuning updates
the whole network but with only 1% of the labeled data. Similarly, 10% labeled
data fine-tuning updates the whole network with 10% of the labeled data. We
report results of the three settings of both our method and the recently-
proposed method MoCo [He et al. [2020]] in Table. 7. As the results show,
$Enc^{t}$ in LSAE can outperform MoCo in semi-supervised settings (i.e, 1%
labels finetuning and 10% labels finetuning). However, MoCo still outperforms
LSAE in the linear evaluation setting.

| BER(%)$\downarrow$ | Inception v3 |  $Enc^{t}$ in SAE |  $Enc^{t}$ in LSAE
---|---|---|---
split 1 | 20.25 $\pm$ 1.46 | 20.25 $\pm$ 1.63 |  19.00 $\pm$ 1.84
split 2 | 19.25 $\pm$ 3.67 | 20.50 $\pm$ 1.12 |  17.75 $\pm$ 1.66
split 3 | 17.75 $\pm$ 3.48 | 14.00 $\pm$ 1.85 |  12.75 $\pm$ 2.15
Avg | 19.08 | 18.25 | 16.50
| mAUC(%)$\uparrow$ | Inception v3 |  $Enc^{t}$ in SAE |  $Enc^{t}$ in LSAE
---|---|---|---
split 1 | 85.45 $\pm$ 1.89 | 89.03 $\pm$ 2.16 |  89.17 $\pm$ 0.68
split 2 | 86.02 $\pm$ 1.27 | 85.63 $\pm$ 1.77 |  87.07 $\pm$ 1.91
split 3 | 89.12 $\pm$ 1.38 | 92.60 $\pm$ 1.25 |  95.00 $\pm$ 0.29
Avg | 86.86 | 89.09 | 90.41
Table 8: COVOC (Subset-1) Outcome Prediction Results. We evaluate models with
Balanced Error Rate (BER) and average AUC (mAUC). The texture encoder in LSAE
surpasses Inception v3 by a large margin. It also outperforms the texture
encoder in baseline model SAE, which demonstrates that better disentanglement
does lead to better discrimination. We report mean and std of 5 random runs.

####  4.4.2 Outcome Prediction on COVOC.

We further evaluate our model by transferring the pre-trained model to COVOC
in two steps. First, on Subset-1 of COVOC, we evaluate the performance of
$Enc^{t}$ in LSAE by adapting and finetuning it. Then, we test the Subset-1
model directly on Subset-2, which evaluates the generalizability of our model
without any adaptation and finetuning.
Step 1: Adapting and Finetuning on Subset-1. Considering the possible domain
discrepancy between ChestX-ray14 and COVOC, we first adapt LSAE to the new
domain by training to generate hybrid images on COVOC for 10K iterations.
Then, we evaluate the texture encoder $Enc^{t}$ on the outcome prediction task
by further finetuning. As COVOC is imbalanced, we report the ventilation
prediction Balanced Error Rate (BER) together with mAUC in Table 8. Compared
with Inception v3, $Enc^{t}$ reduces BER by 13.5$\%$, and improves mAUC by
4.1$\%$. When comparing with the texture encoder in baseline model SAE, LSAE
also performs better. It demonstrates that better disentanglement leads to
better discrimination.
Step 2: Directly Generalizing to Subset-2. To further demonstrate the
effectiveness of our method, we directly generalize the model trained on
Subset-1 training set to Subset-2. As shown in Table 9, although the
discrepancy between different domains decreases the performance relatively,
LSAE can still achieve reasonable results and consistently outperform the
baseline method.

| BER(%)$\downarrow$ | mAUC(%)$\uparrow$
---|---|---
Inception v3 (baseline) | 32.83 $\pm$ 7.34 | 76.01 $\pm$ 6.85
$Enc^{t}$ in LSAE | 32.02 $\pm$ 4.22 | 77.58 $\pm$ 2.75
$Enc^{t}$ in LSAE + Our Aug |  24.70 $\pm$ 2.77 |  79.94 $\pm$ 1.18
Table 9: Generalization on COVOC Subset-2. To better demonstrate the
robustness of our method, we directly apply the model which is trained on
Subset-1 training set to Subset-2 of COVOC. Our method can achieve reasonable
results without any adaptation. It also outperforms the baseline Inception v3.

###  4.5 Data Augmentation with Hybrid Images

Figure 5: Grad-CAM results of $Enc^{t}$ in LSAE. To get insights from the
Grad-CAM visualizations, we collect corresponding annotations from
radiologists (denoted by red arrows). We show the results of three correctly
classified images in the first three rows, and one mis-classified image in the
last row. Please refer to Sec. 4.6 for detailed radiologists’ analysis.

As described in Section 3.3, we generate hybrid images to augment the training
data in COVOC. To control the training budget, we set $K=2$, i.e., the
training data of COVOC is augmented to 2 times. To avoid introducing
irrelevant diseases, we only sample structure images from the healthy lungs in
ChestX-ray14. With the same experimental setup with Table 8, the augmentation
method can reduce error rate further from 16.50$\%$ to 15.67$\%$, and improve
mAUC from 90.41$\%$ to 92.04$\%$ on the ventilation prediction task on average
over 3 splits. Moreover, to compare with others, we implement Mixup [Zhang et
al. [2017]] independently for training on COVOC which achieves 16.41%
BER/90.82% mAUC. Thus, our method still shows superior performance. As shown
in Table 9, when directly generalizing on the Subset-2 data, our augmentation
method can also help $Enc^{t}$ in LSAE get better results.

###  4.6 Visualizations and Radiologist Interpretations

Figure 6: t-SNE results of features from $Enc^{t}$ in LSAE on COVOC
(Subset-1). The three plots are feature clustering results of split1, split2,
and split3 respectively. Most samples distribute in two clusters, one for
positive and the other for negative, which indicates the effectiveness of the
learned representations.

To better understand our model, we use Grad-CAM [Selvaraju et al. [2017]] to
highlight the key areas that contribute to the prediction on both ChestX-ray14
and COVOC. To get more insight, we also obtained radiologists’ interpretations
regarding the visualization results. Considering the time cost of annotations,
in Chest-Xray14, we only pick up one COVID-related disease, i.e., Pneumonia.
The Grad-CAM results are shown in the second and fourth columns in Fig 5 while
the corresponding radiologists’ annotations are listed in the first and third
columns. Generally speaking, for the cases which are correctly classified,
Grad-CAM’s highlighted regions match well with radiologists’ annotations. In
contrast, Grad-CAM results on misclassified images usually localize wrong
areas of diseases.

We further use t-SNE to visualize the representation vectors of $Enc^{t}$ in
LSAE on the three splits of COVOC Subset-1. As shown in Fig 6, in the
representation space, although not perfectly, most samples are distributed
around two clusters, one for positive and the other for negative, which
indicates the effectiveness of the learned representations.

Figure 7: Lung swapping results on ChestX-ray14. The first column shows
$I_{1}$ images that provide structure templates. The third column shows
$I_{2}$ images that provide texture targets. The middle column presents the
hybrid images that mix the structure of $I_{1}$ and the texture of $I_{2}$.

##  5 Discussion

###  5.1 Texture Hypothesis

In this paper, we propose a self-supervised representation framework, Lung
Swapping Autoencoder (LSAE), for CXRs. LSAE is based on the assumption that
texture is tightly related to specific lung diseases while lung structures are
regarded as less important in their diagnosis and prognosis. The texture
encoder $Enc^{t}$ in LSAE outperforms ChestNet in disease classification for
all but three classes listed in the ChestX-ray14 dataset. Notably, two of
these three disease classes (hernia and cardiomegaly) are not primarily
pathologies of the lung, and therefore might not be expected to be suitable
for analysis by LSAE.

###  5.2 Comparison with larger network

In Table 5 and Table 7, we have compared with the state-of-the-art self-
supervised learning method MoCo [He et al. [2020]]. To guarantee a fair
comparison, we re-implemented MoCo with a ResNet-18 (model size: 11M) backbone
which has comparable model size as $Enc^{t}$ (model size: 5M). For discussion,
we also re-implemented a MoCo-v2 with ResNet-50. Compared to MoCo-R18,
MoCo-v2-R50 has a one time larger backbone network (model size: 23M), and
introduces an auxiliary projection sub-network during training. The
performance of MoCo-v2-R50 on ChestX-ray14 across different supervision
settings is listed below.

Method | Params | epoch | 1% | 10% | 100% | linear eval
---|---|---|---|---|---|---
MoCo-v2-R50 | 23M | 200 |  65.1% | 72.8% |  79.4% |  73.4%
$Enc^{t}$ in LSAE | 5M | 57 | 61.3% |  73.2% | 79.0% | 69.8%
Table 10: Results in self-supervised (linear evaluation) and semi-supervised
settings

Although MoCo-v2-R50 shows best performance over almost all the supervision
settings, it is worth mentioning that $Enc^{t}$ in LSAE can still win in the
track of 10% labeled data supervision with a 4 times smaller model size.

###  5.3 Radiologist Survey

1). In Table 3, we list the survey results of each radiologist. Q1 and Q2
focus on the quality assessment of generated images. It is worth noting that
radiologists are easier to be confused in the patch level than the image
level. This is because there are more clues in the image level to provide
evidence of synthesis, such as tags and pacemakers. When only showing patches,
the radiologists can only depend on the in-lung texture to make a decision.

2). Along x-axis, we also provide radiologists’ experiences, i.e., the
residency years shown in the parentheses along x-axis. Except for Q1 where the
most experienced radiologist also achieves the lowest confusion rate, we do
not observe a significant correlation between the feedback and the experience
in the other 3 questions.

Overall, the results of this radiologist survey support our proposal of Masked
SFID as a measurement of texture similarity, and demonstrate the effectiveness
of our image generation techinque.

###  5.4 Hybrid Image Augmentation

We have introduced a data augmentation method by generating hybrid images.
There is a hyper-parameter $K$ in the method that indicates how many times the
dataset is augmented. In this paper, enlarging the dataset by 2 times ($K=2$)
can reduce the error rate from 16.50% to 15.67%. However, when augmenting the
dataset to a larger scale, we did not observe a consistent improvement of
performance. We provide a possible explanation here for discussion: Our
proposed augmentation strategy can increase the diversity of structure
information. During training, high structural diversity can make the model
more robust to the potential structure variations in test cases. However, it
can hardly introduce more diversity of texture patterns. Therefore, the
improvement can reach the limit after the model has exploited all the
advantages of structural diversity. Nonetheless, this hybrid image generation
method and its promising application to COVID-19 datasets demonstrates its
potential in future settings of rare or novel lung pathologies for which few
images may exist for machine learning.

##  6 Conclusion

To learn a transferable and clinically effective representation of CXRs in a
self-supervised way, we proposed LSAE to disentangle texture from structure in
CXR images, enabling analysis of disease-associated textural changes present
in pulmonary diseases, such as COVID-19. Experiments on ChestX-ray14 show LSAE
is able to learn an effective representation in fully-supervised, semi-
supervised, and linear evaluation settings across many common pulmonary
diseases. We also created a data augmentation technique, wherein synthetic
images created using structural and textural information from two distinct
CXRs can be used to expand datasets for machine learning applications, such as
COVID-19 outcome prediction. Our resulting predictive model for mechanical
ventilation in COVID-19 patients outperforms conventional methods and may have
clinical significance in enabling improved decision making.

## Acknowledgement

Research reported in this publication was funded by the Office of the Vice
President for Research and Institute for Engineering-Driven Medicine Seed
Grants, 2019 at Stony Brook University.

## References

  * Bae et al. [2020] Bae, J., Kapse, S., Singh, G., Phatak, T., Green, J., Madan, N., & Prasanna, P. (2020).  Predicting mechanical ventilation requirement and mortality in covid-19 using radiomics and deep learning on chest radiographs: A multi-institutional study.  arXiv preprint arXiv:2007.08028, .
  * Biewald [2020] Biewald, L. (2020).  Experiment tracking with weights and biases.  URL: https://www.wandb.com/ software available from wandb.com.
  * Chassagnon et al. [2021] Chassagnon, G., Vakalopoulou, M., Battistella, E., Christodoulidis, S., Hoang-Thi, T.-N., Dangeard, S., Deutsch, E., Andre, F., Guillo, E., Halm, N., El Hajj, S., Bompard, F., Neveu, S., Hani, C., Saab, I., Campredon, A., Koulakian, H., Bennani, S., Freche, G., Barat, M., Lombard, A., Fournier, L., Monnier, H., Grand, T., Gregory, J., Nguyen, Y., Khalil, A., Mahdjoub, E., Brillet, P.-Y., Tran Ba, S., Bousson, V., Mekki, A., Carlier, R.-Y., Revel, M.-P., & Paragios, N. (2021).  AI-driven quantification, staging and outcome prediction of COVID-19 pneumonia.  Medical Image Analysis, 67, 101860.
  * Chen et al. [2020a] Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020a).  A simple framework for contrastive learning of visual representations.  In International conference on machine learning (pp. 1597–1607).  PMLR.
  * Chen et al. [2020b] Chen, X., Fan, H., Girshick, R., & He, K. (2020b).  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, .
  * Chen & He [2021] Chen, X., & He, K. (2021).  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15750–15758).
  * Choi et al. [2018] Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., & Choo, J. (2018).  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  * Cohen et al. [2020] Cohen, J. P., Morrison, P., Dao, L., Roth, K., Duong, T. Q., & Ghassemi, M. (2020).  Covid-19 image data collection: Prospective predictions are the future.  arXiv 2006.11988, .
  * Goodfellow et al. [2014] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014).  Generative adversarial nets.  In Advances in Neural Information Processing Systems.
  * Grill et al. [2020] Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G. et al. (2020).  Bootstrap your own latent: A new approach to self-supervised learning.  arXiv preprint arXiv:2006.07733, .
  * Gutmann & Hyvärinen [2010] Gutmann, M., & Hyvärinen, A. (2010).  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (pp. 297–304).  JMLR Workshop and Conference Proceedings.
  * He et al. [2020] He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020).  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9729–9738).
  * He et al. [2016] He, K., Zhang, X., Ren, S., & Sun, J. (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).
  * Hu et al. [2021] Hu, Q., Drukker, K., & Giger, M. L. (2021).  Role of standard and soft tissue chest radiography images in COVID-19 diagnosis using deep learning.  In Medical Imaging 2021: Computer-Aided Diagnosis (p. 1159704).  International Society for Optics and Photonics volume 11597.
  * Huang et al. [2018] Huang, X., Liu, M.-Y., Belongie, S., & Kautz, J. (2018).  Multimodal unsupervised image-to-image translation.  In ECCV.
  * Isola et al. [2017] Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017).  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Konwer et al. [2021] Konwer, A., Bae, J., Singh, G., Gattu, R., Ali, S., Green, J., Phatak, T., Gupta, A., Chen, C., Saltz, J. et al. (2021).  Predicting covid-19 lung infiltrate progression on chest radiographs using spatio-temporal lstm based encoder-decoder network.  In Medical Imaging with Deep Learning.
  * Li et al. [2021] Li, Z., Zhao, W., Shi, F., Qi, L., Xie, X., Wei, Y., Ding, Z., Gao, Y., Wu, S., Liu, J., Shi, Y., & Shen, D. (2021).  A novel multiple instance learning framework for COVID-19 severity assessment via data augmentation and self-supervised learning.  arXiv:2102.03837 [cs, eess], .
  * Litmanovich et al. [2020] Litmanovich, D. E., Chung, M., Kirkbride, R. R., Kicska, G., & Kanne, J. P. (2020).  Review of Chest Radiograph Findings of COVID-19 Pneumonia and Suggested Reporting Language.  Journal of Thoracic Imaging, 35, 354–360.
  * Liu et al. [2019] Liu, M.-Y., Huang, X., Mallya, A., Karras, T., Aila, T., Lehtinen, J., & Kautz, J. (2019).  Few-shot unsupervised image-to-image translation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 10551–10560).
  * López-Cabrera et al. [2021] López-Cabrera, J. D., Orozco-Morales, R., Portal-Diaz, J. A., Lovelle-Enríquez, O., & Pérez-Díaz, M. (2021).  Current limitations to identify COVID-19 using artificial intelligence with chest X-ray imaging.  Health and Technology, 11, 411–424.
  * Park et al. [2020a] Park, T., Efros, A. A., Zhang, R., & Zhu, J.-Y. (2020a).  Contrastive learning for unpaired image-to-image translation.  In European Conference on Computer Vision.
  * Park et al. [2020b] Park, T., Zhu, J.-Y., Wang, O., Lu, J., Shechtman, E., Efros, A. A., & Zhang, R. (2020b).  Swapping autoencoder for deep image manipulation.  arXiv preprint arXiv:2007.00653, .
  * Rajpurkar et al. [2017] Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K. et al. (2017).  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  arXiv preprint arXiv:1711.05225, .
  * Rott Shaham et al. [2019] Rott Shaham, T., Dekel, T., & Michaeli, T. (2019).  Singan: Learning a generative model from a single natural image.  In Computer Vision (ICCV), IEEE International Conference on.
  * Salehi et al. [2020] Salehi, S., Abedi, A., Balakrishnan, S., & Gholamrezanezhad, A. (2020).  Coronavirus Disease 2019 (COVID-19): A Systematic Review of Imaging Findings in 919 Patients.  American Journal of Roentgenology, 215, 87–93.  Publisher: American Roentgen Ray Society.
  * Selvaraju et al. [2017] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017).  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision (pp. 618–626).
  * [28] Seonghyeon, K. ().  swapping-autoencoder-pytorch.  URL: https://github.com/rosinality/swapping-autoencoder-pytorch.
  * Tian et al. [2021] Tian, Y., Chen, X., & Ganguli, S. (2021).  Understanding self-supervised learning dynamics without contrastive pairs.  arXiv preprint arXiv:2102.06810, .
  * Wang & Xia [2018] Wang, H., & Xia, Y. (2018).  Chestnet: A deep neural network for classification of thoracic diseases on chest radiography.  arXiv preprint arXiv:1807.03058, .
  * Wang et al. [2017] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017).  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2097–2106).
  * Wu et al. [2018] Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018).  Unsupervised feature learning via non-parametric instance discrimination.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3733–3742).
  * Zhang et al. [2017] Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017).  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, .
  * Zhu et al. [2017] Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017).  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision (pp. 2223–2232).