# GAN-based disentanglement learning for chest X-ray rib suppression

Luyi Han Yuanyuan Lyu Cheng Peng S.Kevin Zhou<EMAIL_ADDRESS>Department of Radiology and Nuclear Medicine, Radboud University Medical
Center, Geert Grooteplein 10, 6525 GA, The Netherlands  Department of
Radiology, Netherlands Cancer Institute (NKI), Plesmanlaan 121, 1066CX,
Amsterdam, The Netherlands  Z2Sky Technologies Inc., Suzhou, 215123, China
Artificial Intelligence for Engineering and Medicine Lab, Johns Hopkins
University, MD, USA  School of Biomedical Engineering & Suzhou Institute for
Advanced Research, University of Science and Technology of China, Suzhou,
215123, China  Key Lab of Intelligent Information Processing of Chinese
Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing,
100190, China

###### Abstract

Clinical evidence has shown that rib-suppressed chest X-rays (CXRs) can
improve the reliability of pulmonary disease diagnosis. However, previous
approaches on generating rib-suppressed CXR face challenges in preserving
details and eliminating rib residues. We hereby propose a GAN-based
disentanglement learning framework called Rib Suppression GAN, or RSGAN, to
perform rib suppression by utilizing the anatomical knowledge embedded in
unpaired computed tomography (CT) images. In this approach, we employ a
residual map to characterize the intensity difference between CXR and the
corresponding rib-suppressed result. To predict the residual map in CXR
domain, we disentangle the image into structure- and contrast-specific
features and transfer the rib structural priors from digitally reconstructed
radiographs (DRRs) computed by CT. Furthermore, we employ additional adaptive
loss to suppress rib residue and preserve more details. We conduct extensive
experiments based on 1,673 CT volumes, and four benchmarking CXR datasets,
totaling over 120K images, to demonstrate that (i) our proposed RSGAN achieves
superior image quality compared to the state-of-the-art rib suppression
methods; (ii) combining CXR with our rib-suppressed result leads to better
performance in lung disease classification and tuberculosis area detection.

###### keywords:

CXR rib suppression, domain adaptation, disentanglement learning

††journal: Medical Image Analysis

##  1 Introduction

Chest X-ray (CXR), a 2D projection of a 3D scene from an X-ray source, and
computed tomography (CT), which reconstructs a 3D scene based on a collection
of 2D X-ray projections, are the widely used modalities to diagnose lung
disease. As compared to CT, CXR examinations induce up to 120 times lower
radiation dose and are more affordable [6]. However, diagnosis with CXR alone
is challenging and misses lung cancer findings [15]. That is because CXR
represents a 2D projection of the 3D chest and contains overlapped anatomies
and ambiguous structure details. Thereby, it is clinically significant to
improve the diagnostic reliability of CXR for better clinical decision making.

Fig. 1: Rib suppression. RSGAN proposes to predict the residual map for a CXR
without label. The final rib-suppressed image is obtained by subtracting it
from the input CXR.

Clinical evidence indicates that bone-suppressed CXR can improve the
interpretation of the CXR images and diagnostic reliability by markedly
reducing the visibility of particular superimposed structures in chest
radiographs [12, 23]. Rib suppression is also an important preprocessing step
for a computer‐aided diagnosis (CAD) system contributing to lung segmentation
and tuberculosis detection [19, 28]. Automatically detecting and removing the
rib structures in a CXR by applying image-processing techniques can simplify
the feature extraction and analysis stage in a CAD system. Fig. 1 shows an
example of rib suppression. Dual-energy (DE) CXRs can provide bone-free
images, but motion artifacts are unavoidable due to cardiac motion and breath.
In a post-processing setting, various methods are proposed to generate bone-
suppressed CXRs and can be categorized as: (1) physical model based and (2)
deep learning based. The physical model-based methods mimic the bone
structures depending on various physical models and obtain bone-free results
via subtracting bone from the original CXR. These approaches require manually
annotated bone mask for each CXR image [12, 5, 31, 14, 37].

Learning-based methods can be divided into two subcategories according to the
source of manual ground truth: (1) DE CXR based and (2) digitally
reconstructed radiography (DRR) based. For DE CXR based learning models, the
suppression of bones in CXR images is learned with a specifically designed
artificial neural network [33, 8] or convolutional neural network (CNN) [39,
43, 9]. However, limited amount of DE CXR images impedes the sufficient
learning of CNNs. DRR based learning methods attempt to utilize the structural
prior knowledge in CT domain. Clinically, the bone component in the DRR domain
paired with original chest DRR is readily accessible through projecting bone
from a set of CT images, making it possible to learn a bone-suppressed model
[26]. Although DRR appears similar to CXR, they have different contrasts due
to simulation assumptions. Domain adaptation based on CycleGAN [44] is
employed to reduce the gap through transferring image from CXR domain to DRR
domain. After that, bone component of CXR image can be generated by a model
trained on large annotated DRR data with less domain gap. To obtain rib-
suppressed results, Li et al. [25] subtracts bone decomposition from the
original high-resolution CXR. By histogram matching on CXR in the area of
inner-rib mask, it can produce high-resolution results with inter-rib
information unchanged, which is meaningful for clinical diagnosis.

###  1.1 Related Work

Recently, GAN-based methods have achieved success in the domain adaptation.
The principle of GAN is to introduce a discriminator to distinguish the
generated image domain from the real image domain, while enforce the generator
to trick the discriminator in order to generate an image belonging to the
domain of real image. And the performance of domain adaptation improves during
the adversarial learning between the generator and the discriminator.
Particularly, CycleGAN [44] constructs a forward- and backward-generation
cycle to ensure the bilateral consistency. And this cycle-consistency
constrain enables unpaired data transfer from each other [41, 20, 42].
However, the cycle-consistency in CycleGAN is limited to image-level,
resulting in details drop at the backward generation stage.

To improve the preservation of details in performing domain adaptation, MUNIT
[17] and DRIT [24] disentangle the latent space in the generator into content-
and style-specific representation. In this way, only the domain-variant
features will be exchanged and the domain-invariant feature will be well-
preserved. Many recent medical applications, like synthesis [4] and cross-
modality segmentation [38], adopt the concept of feature disentanglement to
improve the performance. A perfect example can be found in Chartsias et al.
[7], which improved the high-level representation on 2D medical images by
disentangling latent space into spatial anatomical and non-spatial modality
representation. Despite the success in generating more details via feature
disentanglement, these methods still confront the blur issue in the generated
images. It is because that the recovery capacity of the decoder is limited and
it is hard to reconstruct realistic noise distribution. This motives us to
propose an easy-to-generate output of the decoder. Instead of directly
generating images in the target domain, our RSGAN aims at generating a
residual map, which contains domain-invariant anatomical structure and
residual intensity in a specific domain.

The annotations of CT volume, e.g. lung and bone, can be projected to the DRR
domain. By leveraging the information from DRR images, several studies
developed research in X-ray decomposition [1, 26], lung enhancement [10, 26],
and rib-suppression [26, 25] tasks. To mitigate the domain gap between CXR and
DRR, some of these methods [26, 25] consist of the domain adaptation between
DRR and CXR images. Specially, DecGAN [26] proposes a CycleGAN-based network
and decomposes input into different components (bone, lung and other soft-
tissue structures), then adaptively combining the decomposed components to
implement the rib-suppression or lung enhancement. However, the rib-suppressed
DRR is not strictly equal to the linear combination with the projection of
lung and other soft-tissue structures. The realistic suppression indicates
that the rib area in the CT volume should be replaced with intensities of soft
tissue. In DecGAN [26], this part of soft tissue is omitted and considered to
be air. Furthermore, the CycleGAN-based framework drops details and results in
a low-resolution generation. Li et al. [25], a further work of DecGAN [26],
attempt to generate more details in a coarse-to-fine manner. The generated
rib-mask would be subtracted from the CXR images with ad-hoc histogram
matching. However, the performance of rib suppression is affected by the
accuracy of generated rib mask. A less accurate rib mask would result in sharp
intensity changes at the rib edges in the rib-suppressed image. This approach
has the following drawbacks:

  1. 1.

lacking structure-consistency in CXR during domain adaptation;

  2. 2.

neglecting the difference between the decomposed bone and the intensity
residue resulting from a CXR image subtracted by the corresponding rib-
suppressed prediction;

  3. 3.

lacking a formulation to promote a consistent rib mask, which may result in
sharp changes around rib edges.

###  1.2 Contributions

To address the above challenges and obtain better rib suppression results for
real CXR, we propose a generative adversarial network (GAN) based
disentanglement learning framework called Rib Suppression GAN (RSGAN). Our
contributions are three folds:

  1. 1.

We transfer the prior knowledge from the DRR domain to the CXR domain with
disentangled structure- and contrast-specific generators.

  2. 2.

We predict residual map of ribs instead of the rib-suppressed image, enabling
features to be effectively recovered from a typical decoder.

  3. 3.

We formulate an adaptive loss function to enhance intensity-consistency at the
inter-rib regions and preserve the details overlapped by ribs.

The remainder of this paper is organized as follows. In Section 2, we detail
the proposed rib-suppressed method–RSGAN. In Section 3, we describe the
dataset and the metrics for evaluating the performance of rib suppression. In
Section 4, we show predicted rib-suppressed images for each competing method
and the feasibility of RSGAN in downsteam applications. In Section 5, we
discuss the improvement of RSGAN in detail. And we conclude in Section 6.

Fig. 2: Structure of the disentangling generator: rib-suppressed, contrast,
rib bone features are extracted from both DRR and CXR, and decoded to lung
mask, rib-suppressed image, residual map, and bone projection.

##  2 Methods

###  2.1 Overview

Denoting a CXR image by $I$. It is assumed that

$I=Q+R,$ (1)

where $Q$ is the rib-suppressed image and $R$ is the rib residual image. While
direct approaches explicitly predicts the rib-suppressed image $Q$ from $I$,
we attempt to estimate $R$ from $I$ instead and then subtract it from $I$ to
finally get $Q$.

We propose a GAN-based disentanglement learning framework RSGAN to suppress
rib bones in CXR images by leveraging anatomical knowledge from unpaired
CT/DRR images. Although the contrast is different between CXR and DRR images,
the anatomical structures are similar in both images. Based on this fact, our
framework disentangles structure and contrast with separate generators for
both CXR and DRR images. The rib bone features learned from the DRR domain are
then combined with the contrast feature of the CXR domain to construct the
CXR-based residual map. The final rib-suppressed CXR is achieved through
subtracting transferred ribs from the input CXR image as shown in Fig. 1. To
facilitate to the description of the proposed approach, the mathematical
setting and notations are listed in Table 1.

Table 1: The mathematical setting and notations used in this paper. $I_{x}$ | CXR image
---|---
$I_{d}$ | DRR image
$Q_{d}$ | Rib-suppressed image of DRR
$R_{d}$ | Residual map of DRR, where $R_{d}=I_{d}-Q_{d}$
$B_{d}$ | Bone component projection of DRR
$L_{d}$ | Lung mask of DRR
$F^{C}_{x}$, $F^{C}_{d}$ | Contrast feature extracted by encoder $E_{C}$ for CXR and DRR
$F^{B}_{x}$, $F^{B}_{d}$ | Rib bone feature extracted by encoder $E_{B}$ for CXR and DRR
$F^{S}_{x}$, $F^{S}_{d}$ | Rib-suppressed feature extracted by encoder $E_{S}$ for CXR and DRR
$Q_{x}^{\prime}$, $Q_{d}^{\prime}$ | Rib-suppressed image generated by decoder $G_{Q}$ for CXR and DRR
$R_{x}^{\prime}$, $R_{d}^{\prime}$ | Residual map generated by decoder $G_{R}$ for CXR and DRR
$B_{x}^{\prime}$, $B_{d}^{\prime}$ | Bone component projection generated by decoder $G_{B}$ for CXR and DRR
$L_{x}^{\prime}$, $L_{d}^{\prime}$ | Lung mask generated by decoder $G_{L}$ for CXR and DRR
$I_{x}^{\prime}$, $I_{d}^{\prime}$ | Reconstructed CXR and DRR, where $I_{x}^{\prime}=Q_{x}^{\prime}+R_{x}^{\prime}$ and $I_{d}^{\prime}=Q_{d}^{\prime}+R_{d}^{\prime}$
$I_{x}^{\prime\prime}$, $I_{d}^{\prime\prime}$ | Cycle reconstructed CXR and DRR
$I^{\prime}_{x\rightarrow d}$, $I^{\prime}_{d\rightarrow x}$ | Domain transferred image between CXR and DRR
$Q^{\prime}_{d\rightarrow x}$ | Predicted rib-suppressed CXR transferred from DRR, where $Q^{\prime}_{d\rightarrow x}=G_{Q}(E_{C}(I_{x}),E_{S}(I_{d}))$
$M^{\prime}_{x}$ | Bone mask for CXR generated by bilateral filter and threshold segmentation, where $M^{\prime}_{x}=[F_{bilateral}(B_{x}^{\prime})>\theta_{thresh}]$

###  2.2 Disentangled representation in generator

Fig. 2 shows the disentangled representations in our generative model. In the
generator, each input is disentangled into three components, including
anatomical content, image contrast, and rib bones, with three encoders
$E_{S}$, $E_{C}$, and $E_{B}$, respectively. The output has four predictions
(lung mask, rib-suppressed image, residual map of rib, and bone component
projection) from four decoders $G_{L}$, $G_{Q}$, $G_{R}$, and $G_{B}$,
respectively.

In the decoding path, $G_{Q}$ inputs with contrast and rib-suppressed features
and outputs the rib-suppressed image. Improving from Li et al. [26], we
inpaint the rib-removed area in CT volume based on its surrounding tissue
intensities to simulate more real suppression performance, which projected as
the ground truth for rib-suppressed DRR. $G_{R}$ takes the contrast and rib
bone feature as inputs, outputs a residual map. The residual map is utilized
to generate a rib-suppressed image with more details by subtracting it from
the input image. The input image can be reconstructed by adding the rib-
suppressed image and the residual map. $G_{B}$ only takes the rib bone feature
as input and outputs the bone component projection. The ground truth of the
bone component projection for the DRR image is generated by projecting a
separate CT rib bone component obtained in a 3D volume. By generating bone
projection, better rib bone features can be obtained for residual map
prediction, and a bone mask is available due to the clearer rib edge than the
residual map. $G_{L}$ inputs rib-suppressed features and outputs the lung
mask, which can help the model pay more attention to the lung area.

For CXR images, it is difficult to achieve the desirable rib-suppressed
results due to the overlap of multiple tissues (e.g., ribs, clavicles, lung).
To relieve the interference from other organs, the specific representation of
ribs should be well constructed. CT scans provide the 3D structure of the ribs
that are without any overlap and hence easily annotated. Thus tailored rib-
suppressed DRR images can be delivered by projecting from the corresponding
rib-suppressed CT slices. The rib features of CXR images then can be
transferred from those of DRR images. As illustrate in Fig. 2, the mapping
mode of rib-suppression is learned from the DRR domain and supervised from
three aspects, including rib-suppression, residual map of ribs, and bone
component. The supervised loss is given as,

$\displaystyle\mathcal{L}_{su}$
$\displaystyle=\left\|Q_{d}^{\prime}-Q_{d}\right\|_{1}+\left\|R_{d}^{\prime}-R_{d}\right\|_{1}+\left\|B_{d}^{\prime}-B_{d}\right\|_{1},$
(2)

where $\left\|\cdot\right\|_{1}$ is a $L_{1}$ loss, $Q_{d}$ and
$Q_{d}^{\prime}$ are the ground truth and prediction of rib-suppressed DRR
image, respectively, $R_{d}$ and $R_{d}^{\prime}$ are the ground truth and
predicted residual map of bone in DRR image, respectively, and $B_{d}$ and
$B_{d}^{\prime}$ are the ground truth and prediction of bone component
projection in DRR image, respectively.

Fig. 3: Flowchart of domain adaptation via contrast exchanging (CE) block, the
input image is transformed to the target domain by the first CE block and back
to the input domain by the second one.

###  2.3 CXR disentanglement via domain adaptation

Directly disentangling ribs from CXR under the supervision of the annotated
rib in DRR is ineffective due to the domain gap. To reduce the impact of
domain variance in rib-suppression, our RSGAN, shown in Fig. 3, employs cycle-
consistency learning to implement CXR disentanglement via domain adaption.
Particularly, we separate the learning into domain-invariant and domain-
specific features. Here, we assume that the rib bone features and the rib-
suppressed features disentangled from CXR images or DRR images are domain-
invariant, while the contrast features are domain-specific. In this way, a
domain-specific image can be generated by exchanging the contrast feature from
the target domain via contrast exchanging (CE) block.

The requirement of transferring the rib-suppressed map from the DRR domain to
the CXR domain is to achieve the specific style of these two domains. Here,
two discriminators are involved to evaluate the generations for DRR style and
CXR style, respectively. Additionally, the third discriminator is utilized to
assess the predicted difference of bone projection from CXR and DRR images, so
that to sufficiently and precisely transfer the rib structures from DRR to
CXR. Adversarial learning is employed to minimize the domain distance between
real image and prediction, and adversarial losses of the generator and
discriminator are defined as,

$\displaystyle\mathcal{L}_{D_{adv}}$
$\displaystyle=\mathbb{E}_{I_{x}}(D_{x}(I_{x})-1)^{2}+\mathbb{E}_{I^{\prime}_{d\rightarrow
x}}(D_{x}(I^{\prime}_{d\rightarrow x}))^{2}$ (3)
$\displaystyle+\mathbb{E}_{I_{d}}(D_{d}(I_{d})-1)^{2}+\mathbb{E}_{I^{\prime}_{x\rightarrow
d}}(D_{d}(I^{\prime}_{x\rightarrow d}))^{2}$
$\displaystyle+\mathbb{E}_{B_{d}^{\prime}}(D_{B}(B_{d}^{\prime})-1)^{2}+\mathbb{E}_{B_{x}^{\prime}}(D_{B}(B_{x}^{\prime}))^{2},$
$\displaystyle\mathcal{L}_{G_{adv}}$
$\displaystyle=\mathbb{E}_{I^{\prime}_{d\rightarrow
x}}(D_{x}(I^{\prime}_{d\rightarrow
x})-1)^{2}+\mathbb{E}_{I^{\prime}_{x\rightarrow
d}}(D_{d}(I^{\prime}_{x\rightarrow d})-1)^{2}$
$\displaystyle+\mathbb{E}_{B_{x}^{\prime}}(D_{B}(B_{x}^{\prime})-1)^{2},$

where $I_{x}$ and $I_{d}$ denote the CXR and DRR image.
$I^{\prime}_{d\rightarrow x}$ denotes the predicted CXR image transferred from
DRR domain. $I^{\prime}_{x\rightarrow d}$ denotes the predicted DRR image
transferred from CXR domain. $B_{d}^{\prime}$ and $B_{x}^{\prime}$ refer to
the prediction of bone component projection in DRR image and CXR image,
respectively. Two discriminators $D_{x}$ and $D_{d}$ are leveraged to evaluate
the performance of the reconstructed domain-transfer images in CXR domain and
DRR domain, respectively, which retain anatomical structure but with exchanged
contrast. $D_{B}$ refers the bone component discriminator to distinguish the
predicted bone projection from CXR to DRR.

To improve transferability between two domains, we constrain learning on both
feature- and image-level reconstructions. At the feature level, we employ
contrast- and structure-consistency losses. Contrast-consistency loss
$\mathcal{L}_{c}$ is used to keep the contrast feature identical within the
same domain. The structure-consistency loss $\mathcal{L}_{s}$ is used to
protect the structures from deterioration when the contrast changes.
$\mathcal{L}_{c}$ and $\mathcal{L}_{s}$ are defined as follows:

$\mathcal{L}_{c}=\left\|E_{C}(I_{x})-E_{C}(I^{\prime}_{d\rightarrow
x})\right\|_{1}+\left\|E_{C}(I_{d})-E_{C}(I^{\prime}_{x\rightarrow
d})\right\|_{1},$ (4) $\displaystyle\mathcal{L}_{s}$
$\displaystyle=\left\|E_{S}(I_{x})-E_{S}(I^{\prime}_{x\rightarrow
d})\right\|_{1}+\left\|E_{S}(I_{d})-E_{S}(I^{\prime}_{d\rightarrow
x})\right\|_{1}$ (5)
$\displaystyle+\left\|E_{B}(I_{x})-E_{B}(I^{\prime}_{x\rightarrow
d})\right\|_{1}+\left\|E_{B}(I_{d})-E_{B}(I^{\prime}_{d\rightarrow
x})\right\|_{1},$

where $E_{C}$ refers to the contrast encoder, $E_{S}$ refers to the rib-
suppressed encoder, $E_{B}$ refers to the rib bone encoder, $I_{x\rightarrow
d}$ means transferring the CXR image to the DRR domain, and $I_{d\rightarrow
x}$ denotes the reverse transfer.

To ensure the generation be consistent with the real images, a pixel-wise
$L_{1}$ constraint is used to yield a reconstructed image close to the real
one at the image-level. Meanwhile, a cycle-consistency $L_{1}$ constraint is
introduced to ensure the pixel similarity between an original input and the
corresponding cycle-transferred generation as shown in Fig. 3. The pixel-wise
reconstruction loss $\mathcal{L}_{rec}$ and the cycle-consistency loss
$\mathcal{L}_{cyc}$ are utilized to ensure the image-to-image translation,
which are written as follows,

$\displaystyle\mathcal{L}_{rec}=\left\|I_{x}^{\prime}-I_{x}\right\|_{1}+\left\|I_{d}^{\prime}-I_{d}\right\|_{1},$
(6)
$\displaystyle\mathcal{L}_{cyc}=\left\|I_{x}^{\prime\prime}-I_{x}\right\|_{1}+\left\|I_{d}^{\prime\prime}-I_{d}\right\|_{1},$
(7)

where $I_{x}^{\prime}$ and $I_{d}^{\prime}$ refer to the forward prediction of
$I_{x}$ and $I_{d}$, respectively, and $I_{x}^{\prime\prime}$ and
$I_{d}^{\prime\prime}$ denote the cycle predictions, respectively.
$I_{x}^{\prime}$ and $I_{x}^{\prime\prime}$ are formulated as

$\displaystyle I_{x}^{\prime}$
$\displaystyle=G_{Q}(E_{C}(I_{x}),E_{S}(I_{x}))+G_{R}(E_{C}(I_{x}),E_{B}(I_{x})),$
(8) $\displaystyle I_{x}^{\prime\prime}$
$\displaystyle=G_{Q}(E_{C}(I^{\prime}_{d\rightarrow
x}),E_{S}(I^{\prime}_{x\rightarrow d}))$
$\displaystyle+G_{R}(E_{C}(I^{\prime}_{d\rightarrow
x}),E_{B}(I^{\prime}_{x\rightarrow d})),$

$I_{d}^{\prime}$ and $I_{d}^{\prime\prime}$ are similarly formulated.

Fig. 4: Rib-suppressed adaptive loss functions used in our proposed RSGAN,
consist of adversarial learning on the gradient map of rib-suppressed image,
and inner-rib and inter-rib constrains.

###  2.4 Rib-suppression learning on CXR

The above domain adaptation learning is built up using the global intensity
distribution. The translation of local details is ignored. Hence, as shown in
Fig. 4, additional local constraints are applied on the rib-suppression
learning on CXR images from three aspects: (1) handling rib residue; (2)
inter-rib intensity consistency; and (3) inner-rib detail preservation.

##### Handling rib residue

Ribs may reside in the rib-suppression prediction during the domain adaptation
learning. Here, we employ a gradient-based constraint within the lung regions
to eliminate the influence of intensity various across images and enhance
smooth translation at the rib regions for a rib-suppression prediction.
Gradient supervision is applied on predicted rib-suppression of CXR images and
the corresponding residual maps of ribs during the adversarial learning, which
is defined as,

$\displaystyle\mathcal{L}_{D_{\bigtriangledown}}$
$\displaystyle=\mathbb{E}_{Q^{\prime}_{d\rightarrow
x}}(D_{\bigtriangledown}(\bigtriangledown Q^{\prime}_{d\rightarrow x}\cdot
L^{\prime}_{d})-1)^{2}$ (9)
$\displaystyle+\mathbb{E}_{R^{\prime}_{x}}(D_{\bigtriangledown}(\bigtriangledown(I_{x}-R^{\prime}_{x})\cdot
L^{\prime}_{x}))^{2},$ $\displaystyle\mathcal{L}_{G_{\bigtriangledown}}$
$\displaystyle=\mathbb{E}_{R^{\prime}_{x}}(D_{\bigtriangledown}(\bigtriangledown(I_{x}-R^{\prime}_{x})\cdot
L^{\prime}_{x})-1)^{2},$

where $\bigtriangledown$ refers to the gradient operator,
$D_{\bigtriangledown}$ refers to a discriminator aiming to regularize the
gradient map of rib-suppressed image for a smooth transition on rib edges,
$Q^{\prime}_{d\rightarrow x}$ is the predicted rib-suppressed CXR images
transferred from the DRR domain, $R^{\prime}_{x}$ denotes the predicted
residual map of ribs in CXR and $L^{\prime}_{x}$ and $L^{\prime}_{d}$ refer to
the predicted lung mask in CXR and DRR image, respectively. The lung mask
decoder is trained with a Dice loss on a set of annotated CXR images and is
frozen afterward.

##### Inter-rib intensity consistency

Inter-rib intensities should be consistent between the input CXR image and the
corresponding rib-suppressed prediction. The intensity-consistency loss is
defined as:

$\mathcal{L}_{inter}=\left\|R^{\prime}_{x}\cdot[(1-M^{\prime}_{x})\cup(1-L^{\prime}_{x})]\right\|_{1},$
(10)

where $M^{\prime}_{x}$ denotes the corresponding bone mask of $I_{x}$, which
is generated from $B^{\prime}_{x}$ by utilizing bilateral filter and threshold
segmentation, and $\cup$ denotes the union region of two input masks. Note
that, the bone mask $M^{\prime}_{x}$ can not be generated from the residual
map $R^{\prime}_{x}$. Because it will bring unstable texture changes to the
residual map when introducing $\mathcal{L}_{G_{\bigtriangledown}}$. The bone
mask $M^{\prime}_{x}$ will be influenced by the disturbance of the residual
map and enlarge the bias during the training stage. Instead, the bone
component projection $B^{\prime}_{x}$ is more stable and stops the gradient
from $\mathcal{L}_{G_{\bigtriangledown}}$, which is necessary to generate the
bone mask.

##### Inner-rib detail preservation

To preserve the trachea overlapped with ribs in the predicted residual map of
ribs, we employed a Laplacian-based loss $\mathcal{L}_{\bigtriangleup}$ to
eliminate the overlapped regions in the residual prediction. Laplacian as a
regularization method encourages generating smoother residual maps [25].
$\mathcal{L}_{\bigtriangleup}$ is defined as,

$\mathcal{L}_{\bigtriangleup}=\left\|\bigtriangleup
R^{\prime}_{x}\cdot(M^{\prime}_{x}\cup L^{\prime}_{x})\right\|_{1},$ (11)

where $\bigtriangleup$ represents the Laplacian operator.

###  2.5 Total loss function

The above extra adaptive loss functions are employed to refine the performance
of rib suppression on local details, which only work on the fine-tuning stage
and need to be removed in the initial stage. The total loss function of the
generative model in the initial stage and in the fine-tuning stage can be
summarized as,

$\displaystyle\mathcal{L}_{D_{init}}$
$\displaystyle=\lambda_{adv}\mathcal{L}_{D_{adv}},$ (12)
$\displaystyle\mathcal{L}_{G_{init}}$
$\displaystyle=\mathcal{L}_{su}+\lambda_{adv}\mathcal{L}_{G_{adv}}$
$\displaystyle+\lambda_{f}(\mathcal{L}_{c}+\mathcal{L}_{s})+\lambda_{i}(\mathcal{L}_{rec}+\mathcal{L}_{cyc}),$
$\displaystyle\mathcal{L}_{D_{fine}}$
$\displaystyle=\lambda_{adv}\mathcal{L}_{D_{adv}}+\lambda_{G_{\bigtriangledown}}\mathcal{L}_{{D}_{\bigtriangledown}},$
(13) $\displaystyle\mathcal{L}_{G_{fine}}$
$\displaystyle=\mathcal{L}_{su}+\lambda_{adv}\mathcal{L}_{G_{adv}}$
$\displaystyle+\lambda_{f}(\mathcal{L}_{c}+\mathcal{L}_{s})+\lambda_{i}(\mathcal{L}_{rec}+\mathcal{L}_{cyc})$
$\displaystyle+\lambda_{G_{\bigtriangledown}}\mathcal{L}_{G_{\bigtriangledown}}+\lambda_{inter}\mathcal{L}_{inter}+\lambda_{\bigtriangleup}\mathcal{L}_{\bigtriangleup},$

where we set $\lambda_{adv}=1$, $\lambda_{f}=1$, $\lambda_{i}=10$,
$\lambda_{G_{\bigtriangledown}}=10$, $\lambda_{inter}=500$, and
$\lambda_{\bigtriangleup}=1$ with experimental experience.

##  3 Experimental Settings

###  3.1 Data

We utilize two CT datasets and four CXR datasets in our experiment. CT
datasets involve 896 CT volumes from LIDC-IDRI [2] and 777 CT volumes from
2017 111https://tianchi.aliyun.com/competition/entrance/231601/introduction
and 2019
222https://tianchi.aliyun.com/competition/entrance/231724/introduction TianChi
AI Competition for Healthcare organized by Alibaba and all the CT volumes are
selected with a thickness less than 2.5 mm. CXR datasets involve 11,200 CXRs
of size $512\times 512$ from TBX11K [27], 112,120 CXRs of size $1024\times
1024$ from chest-14 [36], 138 CXRs of size around $4020\times 4892$ from
Montgomery County (MC) [18], and 662 CXRs of size around $3000\times 3000$
from Shenzhen Hospital [18]. As for the CT dataset, we train a U-Net model for
rib segmentation on 199 CT volume selected from LIDC-IDRI with manual labels
and generate rib masks for the whole CT dataset. Then, based on the rib masks,
we suppress the region of ribs on CT volumes with an inpainting method [34].
To generate the DRRs, we utilize DeepDRR [35] to project each CT volume into
42 projection slices which are uniformly sampled from $-10^{\circ}$ to
$10^{\circ}$ in azimuth and elevation angles, respectively. Similarly, we
generate the projection of rib-suppressed volume, rib region, and lung area
with the same projection strategy. All the DRRs are resized to $320\times 320$
based on bilinear interpolation for training. As for the CXR dataset, all the
CXRs are resized to $320\times 320$ when input into the network and back to
their original resolution for rib suppression. We utilize the training set of
TBX11K, Chest-14, and MontgomerySet for training, and others for testing.

Fig. 5: Implementation details of blocks in the generator.

###  3.2 Implementation details

Fig. 5 illustrates the details of three kind of blocks in the generator. The
contrast encoder $E_{C}$ is consisted of four convolutional layers with a
kernel size of $3\times 3$ and a stride of 2, followed with a global average
pooling (GAP) and a fully connection layer of 256 output channels. The rib-
suppressed encoder $E_{S}$ and the rib bone encoder $E_{B}$ have the same
structure, consisting of two convolutional layers with a kernel size of
$3\times 3$ and a stride of 2, followed with a residual block of 256 output
channels. In this paper, the size of features encoded from $E_{S}$ and $E_{B}$
is set to be $80\times 80$ with the channel of 256. The rib-suppressed image
decoder $G_{Q}$ and the residual map decoder $G_{R}$ have the same structure
as shown in Fig. 5, involving three weight demodulation blocks [21], and each
weight demodulation block is followed with an upsampling layer to pass a
higher dimension of feature to the next block. The scale factor of the
upsampling layer is 2. Three different resolutions of feature maps in the
decoder are set to be $80\times 80$, $160\times 160$, and $320\times 320$.
Every block includes another sub-branch of convolutional layer to generate an
output image in a higher resolution. The lower resolution image is merged with
the higher resolution one outputed by next block via an upsampling layer and
an element-wise addition. The contrast feature is fed into each weight
demodulation block as shown by the dotted arrow. The bone component decoder
$G_{B}$ has similar structure with $G_{Q}$ and $G_{R}$, replacing the
demodulation block with residual block, and the final output is followed with
a Tanh activation. The lung mask decoder $G_{L}$ is the same as $G_{B}$, but
outputs with a Sigmoid activation to predict the mask of lung. Note it, the
contrast feature is not fed into $G_{B}$ and $G_{L}$.

Four discriminators are utilized in our proposed RSGAN. All discriminators are
built with PatchGAN [44], consisting of four convolutional layers with a
kernel size of 4 and a stride of 2 and one convolutional layer with a kernel
size of 1.

The proposed network is developed with PyTorch and trained on a GeForce RTX
2080 Ti GPU. All the networks are trained using Adam optimizer with a learning
rate of $1\times 10^{-5}$. We train the proposed network with 40,000
iterations in the initial stage, and 10,000 iterations for the fine-tuning
stage with a batch size of 1.

###  3.3 Evaluation metrics

Four metrics are utilized to evaluate rib suppression quality, including Weber
Contrast [13], Learned Perceptual Image Patch Similarity (LPIPS) [40], Peak
Signal Noise Rate (PSNR), and Structural Similarity Index Measure (SSIM).

Weber Contrast [13] provides an estimation of rib-suppression performance on
the boundaries for CXR images by calculating the contrast gap between the rib-
suppressed region and the background. It is defined as
$C_{w}=I_{r}/I_{b}~{}-~{}1$, where $I_{r}$ denotes the average intensity of
the manually annotated rib region. $I_{b}$ denotes the confined background,
corresponding to the surrounded region of ribs in a radius of fewer than 5
pixels. Lower Weber Contrast means higher similarity between the rib-
suppressed region and the background, suggesting better visualization of the
predicted rib-suppressed CXR.

LPIPS [40] is a metric to measure the perceptual similarity at feature level,
enabling to evaluate the difference between the generated and real images at
multiple feature space. LPIPS is defined as follow,

$d(x,x_{0})=\sum_{l}\frac{1}{H_{l}W_{l}}\sum_{h,w}\left\|\omega_{l}\odot(\hat{y}^{l}_{hw}-\hat{y}^{l}_{0hw})\right\|^{2}_{2},$
(14)

where $x$, $x_{0}$ denote the reference and generated patch, respectively.
$\hat{y}^{l}$, $\hat{y}^{l}_{0}\in\mathbb{R}^{H_{l}\times W_{l}\times C_{l}}$
correspond to the unit-normalized features at $l$th layer in a feature-
extraction network when inputting $x$, $x_{0}$.
$\omega_{l}\in\mathbb{R}^{C_{l}}$ represents a weighted vector to control
different channels, $\odot$ refers to a channel-wise multiplication, and
$\left\|\cdot\right\|_{2}$ is a $L_{2}$ loss. Note that, AlexNet [22] is
chosen as the feature-extraction network, and all weights in vector
$\omega_{l}$ are set to $1$.

Besides, PSNR and SSIM, the widely used image-level similarity metrics, are
employed to estimate the details remained in the rib-suppressed prediction.
PSNR presents the total pixel errors, and SSIM quantifies the structure
similarity based on brightness, contrast, and structures. All the above-
mentioned three similarity metrics are calculated over the manually labeled
lung regions without bones. Lower LPIPS, higher PNSR and SSIM refer to a
better prediction.

Other two metrics are introduced to evaluate the quality of two key components
in RSGAN – residual map mechanism and domain adaptation, including Mean
Absolute Error for reconstruction consistency ($\mathbf{MAE}_{rec}$) and Mean
Absolute Error for cycle consistency ($\mathbf{MAE}_{cyc}$). They are defined
as $\mathbf{MAE}_{rec}=\left\|I^{\prime}_{x}-I_{x}\right\|_{1}$ and
$\mathbf{MAE}_{cyc}=\left\|I^{\prime\prime}_{x}-I_{x}\right\|_{1}$. Lower
$\mathbf{MAE}_{rec}$ and $\mathbf{MAE}_{cyc}$ refer to a better performance of
the generator.

Fig. 6: Rib-suppressed results for alternative methods. Case 1 and Case 2 are
normal CXR images. Case 3 and Case 4 are with pulmonary tuberculosis. The
first line of each case is the CXR images, and the second line is the colormap
for the difference in the inter-rib area between the rib-suppressed image and
the original CXR. The red region in the CXR colormap indicates the manually
labeled mask of the inter-rib area. Rectangles in different colors refer to
different areas in the image, and the areas are magnified at the right side of
the image. Red arrows indicate some notable differences in the local area.

##  4 Results

###  4.1 Alternative comparisons

We compare our method with U-Net (DRR), U-Net (Cycle), and Li et al. [25] with
the above-mentioned metrics. For the setting of U-Net (DRR), a U-Net [30] is
trained on annotated DRR images and directly used to predict the residual map
of ribs for CXR. For U-Net (Cycle), the CXR image is first translated to the
DRR domain by CycleGAN [44] and fed back to the CXR domain after rib
suppression by U-Net [30]. The comparisons are based on 192 CXRs with manually
annotated rib and lung region, in which 99 CXRs are randomly selected from the
Shenzhen hospital dataset [18] and 93 come from Chest-xray14 dataset [36].

Table 2 and Fig. 6 show the quantitative and visual results of alternative
methods, respectively. RSGAN achieves state-of-the-art performance with the
lowest Weber Contrast of $1.96\times 10^{-2}$, LPIPS of $0.62\times 10^{-2}$
and the highest PSNR of 34.62, SSIM of 0.996. The LPIPS, PSNR and SSIM results
of RSGAN are much better than those of U-Net (DRR) and U-Net (Cycle). This is
probably because RSGAN poses a constraint on the inter-rib area and ensures
the region between ribs unchanged. The proposed RSGAN achieves slightly better
Weber Contrast, PSNR and SSIM than those of Li et al. [25], but obtains a
$0.56\times 10^{-2}$ drop of LPIPS. It is indicated that RSGAN introduces
better perceptual similarity on the edge of the rib area. As illustrated in
Fig. 6, we randomly choose four CXRs including two normal and two pulmonary
tuberculosis cases from Shenzhen hospital dataset [18]. The rib-suppressed
results by RSGAN have cleaner visualization in both overview and details than
other methods and also have a lower difference of inter-rib area in the
colormap. In the blue rectangle of case 1, U-Net (DRR) and U-Net (Cycle) can
not suppress the ribs completely and details are smoothed out. Due to the
invalidation of histogram matching, a sharp difference appears in the result
of Li et al. [25]. Among all the methods, RSGAN removes the most rib residue
and retains the richest details, which proves that RSGAN can obtain a better
and robust rib-suppressed image via the disentanglement approach. As for the
details remaining in the rib-suppressed results, the proposed RSGAN could make
the detail much clearer than other methods and retain both big nidus
(tuberculosis in red rectangles) and very small objects (small vascular and
trachea, e.g. in blue rectangles), but Li et al. [25] leverage a large
residual of clavicle because of bad rib prediction. All these advantages
contribute to reducing the reading difficulty and the chance of misdiagnosis.

###  4.2 Ablation study

We investigate the effectiveness of two key components in our RSGAN: (i)
residual map mechanism and domain adaptation (RMDA); and (ii) different extra
adaptive loss functions. Residual map mechanism is accompanied with the domain
adaption, and we take them as an indivisible part. The experiments with and
without RMDA are performed to demonstrate the effectiveness of component RMDA.
The detailed experimental settings are RSGAN not using residual map (nRM),
RSGAN using residual map (RM) , and RSGAN using RMDA. In the experiment of
RSGAN (nRM), the finally rib-suppressed image is directly predicted from the
generator $G_{Q}$ in Fig. 2. And the generator $G_{Q}$ is trained with the
following loss function,

$\displaystyle\mathcal{L}_{G_{nRM}}=\left\|Q_{d}^{\prime}-Q_{d}\right\|_{1},$
(15)

where $Q_{d}$ and $Q_{d}^{\prime}$ are the ground truth and prediction of rib-
suppressed DRR image, respectively.

In the experiment of RSGAN (RM), all the generators are preserved without the
discriminators. The generators with residual map mechanism are trained with
the following loss function,

$\displaystyle\mathcal{L}_{G_{RM}}=\mathcal{L}_{su}+\lambda_{i}\mathcal{L}_{rec},$
(16)

where $\lambda_{i}$ is set to $10$, which is identical with RSGAN.

For RSGAN (RMDA), all the generators and discriminators are available, and all
the setting are the same as the proposed RAGAN expect using the extra adaptive
loss functions. RSGAN (RMDA) is equal to the RSGAN only trained after the
initial stage mentioned in Section 2.5, whose loss function is written as Eq.
12.

Then we set RSGAN (RMDA) as the baseline model, and provide an ablation study
on the effect of proposed extra loss functions, including rib-suppressed
adversarial loss $\mathcal{L}_{G_{\bigtriangledown}}$, background consistency
loss $\mathcal{L}_{inter}$, and residual smooth loss
$\mathcal{L}_{\bigtriangleup}$. We compare the quantitative results with
adding the above components in turn, and evaluate with the metrics of Weber
Contrast, LPIPS, PSNR, and SSIM.

Table 2 and Fig. 7 show the quantitative and visual results of the ablation
study of two key components–residual map mechanism and domain adaptation.
Compared with RSGAN (RM) and RSGAN (nRM), the LPIPS is reduced by nearly
$29\times 10^{-2}$, the PSNR and SSIM increase of 2.65 and 0.008,
respectively, which illustrate that introducing residual map mechanism can
improve the performance of inter-rib details. The metrics of RSGAN (RM) are
also better than those of U-Net (DRR), which shows that our proposed
disentangled network has better ability of generation than that of U-Net.
After introducing the domain adaptation structure, RSGAN (RMDA) achieves lower
Weber contrast of $2.45\times 10^{-2}$ and lower $\mathbf{MAE}_{rec}$ of 0.027
than those of RSGAN (RM), and has much better $\mathbf{MAE}_{cyc}$ of 0.041
than that of U-Net (Cycle) and Li et al. [25]. It may because the proposed
domain adaptation method can reduce the impact of domain variance between DRR
and CXR, and can avoid the changing of the inter-rib area in the procedure of
domain transform.

Table 3 shows the quantified comparisons over the different extra adaptive
loss functions. The first row lists the baseline results from the typical
adversarial learning loss function. We can see that, adding the suppressed
loss $\mathcal{L}_{G_{\bigtriangledown}}$ can reduce the Weber contrast by
$0.74\times 10^{-2}$, illustrating that the intensity difference decreases
between the inner region of rib and background. Despite this,
$\mathcal{L}_{G_{\bigtriangledown}}$ results in the decrease of PSNR and the
increase of LPIPS, indicating the loss of details during rib suppression.
Further introducing the inter-rib constraint loss $\mathcal{L}_{inter}$,
LPIPS, PSNR and SSIM achieve obvious improvement, suggesting that
$\mathcal{L}_{inter}$ facilitates the preservation of the inter-rib details
when suppressing ribs. Additionally, the complementary Laplacian-based loss
$\mathcal{L}_{\bigtriangleup}$ can result in marginal improvement on the
listed metrics and retains clearer details in the inner-rib area of the rib-
suppressed result.

As shown in Fig. 8, three CXRs are chosen from Shenzhen hospital dataset [18]
to visually compare the rib-suppressed performance by utilizing different loss
functions. When adding with the $\mathcal{L}_{G_{\bigtriangledown}}$, we
obtain the rib-suppressed result of CXR with fewer rib residues than that in
the baseline, but with fewer details remaining. Comparing with
$\mathcal{L}_{G_{\bigtriangledown}}$, $\mathcal{L}_{inter}$ helps keep the
intensity in inter-rib area unchanged, and $\mathcal{L}_{\bigtriangleup}$
improves the preserving of the details in the inner-rib area.

Table 2: The quantitative results of CXR rib suppression of manual annotated image from Shenzhen hospital and Chest-xray14 dataset. The comparison methods involve state-of-art methods and ablation study with different key components in RSGAN (nRM indicates not using a residual map, RM indicates using a residual map, and RMDA indicates using both residual map and domain adaption). The best result is in bold and the second best one is underlined. Method |  $C_{w}$ ($10^{-2}$)$\downarrow$ | LPIPS ($10^{-2}$)$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | $\mathbf{MAE}_{rec}\downarrow$ | $\mathbf{MAE}_{cyc}\downarrow$
---|---|---|---|---|---|---
CXR | 4.04 | - | - | - | - | -
U-Net (DRR) | 3.49 | 2.96 | 26.73 | 0.982 | - | -
U-Net (Cycle) | 2.76 | 8.19 | 25.31 | 0.959 | - | 0.055
Li et al. [25] | 2.36 | 1.18 | 34.17 | 0.995 | - | 0.064
RSGAN (nRM) | 3.53 | 31.5 | 25.09 | 0.978 | - | -
RSGAN (RM) | 2.92 | 2.81 | 27.74 | 0.986 | 0.032 | -
RSGAN (RMDA) | 2.45 | 1.51 | 30.15 | 0.992 | 0.027 | 0.041
RSGAN | 1.96 | 0.62 | 34.62 | 0.996 | 0.026 | 0.039
Fig. 7: Rib-suppressed results for ablation study of residual map mechanism and domain adaptation. The first line of each case is the CXR images, and the second line is the colormap for the difference in the inter-rib area between the rib-suppressed image and the original CXR. The red region in the CXR colormap indicates the manually labeled mask of the inter-rib area. Rectangles in different colors refer to different areas in the image, and the areas are magnified at the right side of the image. Red arrows indicate some notable differences in the local area. Fig. 8: Rib-suppressed results for ablation study of extra adaptive loss functions. The first line of each case is the CXR images, and the second line is the colormap for the difference in the inter-rib area between the rib-suppressed image and the original CXR. The red region in the CXR colormap indicates the manually labeled mask of the inter-rib area. Rectangles in different colors refer to different areas in the image, and the areas are magnified at the right side of the image. Red arrows indicate some notable differences in the local area. Table 3: Ablation study with different extra adaptive loss functions. The best result is in bold and the second best one is underlined. Baseline | $\mathcal{L}_{G_{\bigtriangledown}}$ | $\mathcal{L}_{inter}$ | $\mathcal{L}_{\bigtriangleup}$ |  $C_{w}$ ($10^{-2}$)$\downarrow$ | LPIPS ($10^{-2}$)$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---|---|---|---|---
✓ |  |  |  | 2.45 | 1.51 | 30.15 | 0.992
✓ | ✓ |  |  | 1.71 | 2.08 | 28.54 | 0.989
✓ | ✓ | ✓ |  | 2.05 | 0.83 | 34.26 | 0.996
✓ | ✓ | ✓ | ✓ | 1.96 | 0.62 | 34.62 | 0.996

###  4.3 Statistical tests

Fig. 9 illustrates the box plots for Weber contrast, LPIPS, PSNR, and SSIM of
RSGAN and compared methods. Paired T-tests show that the Weber contrast,
LPIPS, PSNR and SSIM improvement given by RSGAN are statistically significant
($p<0.05$) against U-Net (DRR), U-Net (Cycle), Li et al. [25], RSGAN (nRM),
RSGAN (RM), RSGAN (RMDA) and ”$+\mathcal{L}_{G_{\bigtriangledown}}$”. Although
RSGAN does not give statistically significant improvements on Weber contrast
and SSIM against ”$+\mathcal{L}_{G_{\bigtriangledown}}+\mathcal{L}_{inter}$”
($p=0.141$ and $p=0.900$), RSGAN achieves statistically significant
improvements on LPIPS and PSNR ($p<0.05$).

Fig. 9: Box plots of Weber contrast, LPIPS, PSNR, and SSIM for different methods. For statistically significant improvements between RSGAN and compared methods, methods marked with *, **, and *** indicate $0.01<p<0.05$, $0.001<p<0.01$, and $p<0.001$, respectively. Table 4: The area under curve (AUC) of predicting 14 lung diseases on Chest-xray14. The best result is in bold and the second best one is underlined. Method | DenseNet-121 | DenseNet-121 | DenseNet-121 | DenseNet-121 | DenseNet-121
---|---|---|---|---|---
[36] | \+ Li et al. [25] | \+ Li et al. [25] (Mix) | \+ RSGAN | \+ RSGAN (Mix)
Atelectasis | 0.816 | 0.809 | 0.817 | 0.815 | 0.827
Cardiomegaly | 0.901 | 0.902 | 0.906 | 0.900 | 0.906
Effusion | 0.877 | 0.881 | 0.883 | 0.884 | 0.881
Infiltration | 0.705 | 0.698 | 0.701 | 0.706 | 0.717
Mass | 0.838 | 0.832 | 0.860 | 0.839 | 0.859
Nodule | 0.774 | 0.738 | 0.786 | 0.749 | 0.795
Pneumonia | 0.753 | 0.759 | 0.770 | 0.759 | 0.769
Pneumothorax | 0.868 | 0.867 | 0.884 | 0.878 | 0.884
Consolidation | 0.807 | 0.805 | 0.812 | 0.810 | 0.809
Edema | 0.892 | 0.891 | 0.896 | 0.894 | 0.900
Emphysema | 0.925 | 0.914 | 0.931 | 0.934 | 0.935
Fibrosis | 0.831 | 0.828 | 0.844 | 0.847 | 0.850
Pleural Thickening | 0.783 | 0.770 | 0.778 | 0.776 | 0.784
Hernia | 0.886 | 0.954 | 0.940 | 0.942 | 0.942
Average | 0.833 | 0.832 | 0.844 | 0.838 | 0.847
Table 5: CXR image classification results on TBX11K test set. The best result is in bold and the second best one is underlined. Method | Accuracy | AUC (TB) | Sensitivity | Specificity | Ave. Prec. | Ave. Rec.
---|---|---|---|---|---|---
FRCNN [27] | 86.49 | 94.83 | 75.58 | 94.35 | 84.72 | 84.85
FRCNN + Li et al. [25] | 86.95 | 95.21 | 78.67 | 93.52 | 84.94 | 85.75
FRCNN + Li et al. [25] (Mix) | 87.83 | 95.54 | 81.30 | 93.11 | 85.77 | 87.06
FRCNN + RSGAN | 87.28 | 95.39 | 75.73 | 95.86 | 86.30 | 85.66
FRCNN + RSGAN (Mix) | 89.19 | 96.32 | 83.31 | 95.25 | 88.04 | 88.65
Table 6: TB area detection results on TBX11K test set. CA TB denotes class-agnostic TB. The best result is in bold and the second best one is underlined. Method | CA TB | Active TB | Latent TB
---|---|---|---
AP50 | mAP | AP50 | mAP | AP50 | mAP
FRCNN [27] | 55.76 | 25.65 | 51.35 | 23.55 | 5.81 | 1.71
FRCNN + Li et al. [25] | 50.77 | 20.61 | 46.69 | 18.80 | 6.99 | 1.78
FRCNN + Li et al. [25] (Mix) | 55.67 | 22.66 | 51.28 | 21.21 | 3.86 | 1.08
FRCNN + RSGAN | 55.77 | 23.95 | 50.56 | 22.07 | 5.35 | 1.58
FRCNN + RSGAN (Mix) | 60.28 | 24.59 | 54.14 | 22.11 | 8.52 | 2.36

###  4.4 Downstream applications

We employ two downstream applications, including lung disease classification
and tuberculosis detection, to evaluate the quality of the imputed rib-
suppressed CXR images.

####  4.4.1 Lung disease classification

To quantify the contribution of rib suppression in lung disease
classification, we conduct experiments on Chest-xray14 dataset [36] and TBX11K
[27].

Referring to the experiments in [36], we utilize DenseNet-121 [16] as
classification network to predict 14 lung disease in Chest-xray14 dataset
[36], and conduct experiments on different input combinations. We propose five
combinations including (1) DenseNet-121: only input with original CXR; (2)
DenseNet-121 + Li: only input with rib-suppressed CXR generated by Li et al.
[25]; (3) DenseNet-121 + Li (Mix): input with the concatenation of two
original CXR channels and one rib-suppressed CXR channel generated by Li et
al. [25]; (4) DenseNet-121 + RSGAN: only input with rib-suppressed CXR
generated by our method; (5) DenseNet-121 + RSGAN (Mix): input with the
concatenation of two original CXR channels and one rib-suppressed CXR channel
generated by our method. We compare the five input combinations using the
common metrics of Area Under Curve (AUC) on predicting 14 lung diseases on
Chest-xray14 dataset [36]. The results are illustrated in Table 4, and our
method achieves state-of-the-art results on lung disease classification.
Concatenating original CXR and rib-suppressed images generated by our method
obtains a performance boost of 0.014 in average AUC than single original CXR.
Only utilizing our rib-suppressed CXR achieves 1% AUC higher in pneumothorax
and 1.6% AUC higher in fibrosis than original CXR because these lung diseases
cause appearance changes in lung textures and rib suppression helps highlight
the lung texture.

Referring to the experiments in Liu et al. [27], we utilize Faster R-CNN
(FRCNN) as a classification network to predict CXR into three classes
(healthy, unhealthy but non-tuberculosis, tuberculosis). Similarly, we propose
five combination including (1) FRCNN; (2) FRCNN + Li et al. [25]; (3) FRCNN +
Li et al. [25] (Mix); (4) FRCNN + RSGAN; and (5) FRCNN + RSGAN (Mix). Here we
compare the five input combinations using the metrics of Accuracy, AUC,
Sensitivity, Specificity, Average Precision, and Average Recall on predicting
three classes. The comparisons are based on TBX11K dataset [27] using the
official splits. As illustrated in Table 5, RSGAN achieves state-of-the-art
results on CXR image classification with an increase of 2.7% in accuracy and
1.49% in AUC than those of input without rib-suppressed images. It is
supported that rib suppression can help improve CXR image classification
accuracy for clinical diagnosis.

####  4.4.2 Tuberculosis detection

In order to demonstrate the effectiveness of RSGAN for tuberculosis (TB) area
detection, we utilize FRCNN as a detection network to predict TB area and
classify each bounding box into category-agnostic tuberculosis (CA TB), active
TB, and latent TB. Similar to classification tasks, we combine five types of
input to train the network: (1) FRCNN; (2) FRCNN + Li et al. [25]; (3) FRCNN +
Li et al. [25] (Mix); (4) FRCNN + RSGAN; and (5) FRCNN + RSGAN (Mix).

For the evaluation of tuberculosis detection, we utilize two metrics of the
average precision of the bounding box (AP). AP50 refers to AP at the IoU
(intersection-over-union) threshold of 0.5. mAP denotes the average AP with
the IoU threshold from 0.5 to 0.95 with a step of 0.05. The higher the AP50
and mAP are, the higher accuracy of tuberculosis detection the method gets.
The comparisons are based on TBX11K dataset [27] according to the official
splits. The results are illustrated in Table 6. Combining the original CXR
image with its rib-suppressed result can obtain the best detection results
than the other types of input, and achieves approximately 5% increase on AP50
of CA TB than that of input only with original CXR image. It might be because
that RSGAN could suppress the clavicles and ribs overlapped on the
tuberculosis lesion, which is illustrated in the red rectangles of case 3 and
case 4 in Fig. 6.

##  5 Discussion

In this study, we develop the RSGAN for suppressing ribs in the CXR images.
Our proposed RSGAN is especially designed to handle with the challenges —
overlapped anatomies and unknown ground truth — in CXR rib-suppression. To
obtain the rib structural features, we borrow the rib-suppression knowledge
from DRR images. The structural-specific features of ribs are learned from our
disentangled GAN framework. These features are then transferred to the CXR
images for rib-suppression using the domain adaption technique. Besides, to
ensure the inter-rib intensities (i.e. those in the cavity regions) intact, we
incorporate a residual map mechanism, which characterizes the intensity
difference between a CXR and its corresponding rib-suppressed image, into
domain adaption as a constraint for the contrast-specific features.
Furthermore, extra adaptive loss functions are introduced to handle rib
residue and preserve details in both inter-rib and inner-rib regions.

We demonstrate the effectiveness of our proposed RSGAN by comparing with the
state-of-the-art methods. Sufficient ablation studies are conducted to
validate the improvement of each key component in RSGAN.

##### Residual Map—

ResNet [11] demonstrates that the residual representation can simplify the
optimization for deep CNN. Many researches [29, 32, 3] prove that long-term
residual connection benefits medical image synthesis. It is because that the
residual maps have intensity distribution closed to zero, making CNN easier to
be trained.

Quantitative results listed in Table 2 and predictions shown in Fig. 7 prove
that the residual map mechanism is efficient to improve the accuracy of
predicting the rib-suppressed image. For the rib suppression task, the
residual maps involve many pixels with the intensity of zero due to most areas
except ribs on the image keep unchanged. On the other hand, the residual map
is the combination of the projection for rib and tissue components, which is
simpler than that of the rib-suppressed CXR. Thus the rib residual map is
easier to be predicted than rib-suppressed CXR, which simplifies the task. As
shown in Fig. 6, Fig. 7 and Table 2, U-Net (DRR) and RSGAN (RM), the methods
directly predict the residual map instead of the rib-suppressed image, achieve
superior performance than that of RSGAN (nRM) from both statistical and visual
results. Especially, the methods without using residual map are ineffective,
resulting in blur boundaries and dropped details as shown in Fig. 7.

##### Domain Adaption—

Domain adaption is a key factor of suppressing the ribs in CXR images based on
DRR images. In the proposed RSGAN, the structural features of DRR images are
adaptively transferred to those of CXR images via the disentangled structure-
and contrast-specific generators. CycleGAN [44] is a widely used solution for
domain adaption by constructing the mapping of the source-target domains. But
the CycleGAN-based methods only learn the relationship between the CXR and DRR
images and neglect the domain variance between original and rib-suppressed
images. The disentanglement network provides a reconstruction consistency for
the process of image decomposition and reconstruction of a single image and a
cycle consistency for the domain transferring processes between CXR and DRR.
Aided by these two consistency constraints, the methods equipped with a
disentangled architecture achieve clearer details and less blur in the inter-
rib area than U-Net (Cycle). It is demonstrated that disentangled domain
adaptation can make up for the shortcoming of CycleGAN-based methods by
implicit constraints. As shown in Table 2, Li et al. [25] achieves higher
$\mathbf{MAE}_{cyc}$ but better rib suppression performance than those of
U-Net (Cycle). Worse $\mathbf{MAE}_{cyc}$ of Li et al. [25] is because that
the LoG transformation is applied on both CXR and DRR images which can make
bone component shaper and enlarge the consistency error. Different from U-Net
(Cycle), Li et al. [25] avoids transferring the rib-suppressed image from DRR
to CXR by generating a rib mask from domain-transferred image to solve the
problem of the CycleGAN-based methods. But it is difficult to guarantee that
the predicted binary rib masks are accurate at the boundary of the rib. As
shown in Fig. 6, there is a plaque remained in the Case 2 of Li et al. [25],
which is caused by a wrong binary rib mask.

##### Adaptive Loss Function—

An effective loss function is highly important for the convergence and
performance of a neural network. The extra adaptive loss functions in our
RSGAN are introduced to address the drawbacks in the method of Li et al. [25].
Li et al. [25] predicts the residual map from the bone component projection
based on histogram matching and then substracts the residual map from the
original CXR image constrained within the area of predicted binary mask. This
method neglects the difference between the decomposed bone and the intensity
residue and lacks a formulation to promote a consistent rib mask, which may
result in sharp changes around rib edges. The loss function
$\mathcal{L}_{G_{\bigtriangledown}}$ in RSGAN provides GAN-based learning to
handle rib residue adaptively rather than binary segmentation. As shown in
Table 2 and Table 3, baseline with $\mathcal{L}_{G_{\bigtriangledown}}$
achieves the lowest Weber Contrast than that of RSGAN and Li et al. [25].
However, LPIPS, PSNR, and SSIM perform worse when only introducing
$\mathcal{L}_{G_{\bigtriangledown}}$. Weber Contrast is the metric to evaluate
the average difference between the rib and its surrounding region. Because the
rib-suppressed DRR images remain fewer details of the trachea than CXR,
adversarial learning with $\mathcal{L}_{G_{\bigtriangledown}}$ blurs the
details in the inter-rib area of CXR images, which leads to lower Weber
contrast but worse LIPIS, PSNR, and SSIM. To solve this issue,
$\mathcal{L}_{inter}$ and $\mathcal{L}_{\bigtriangleup}$ help keep the
intensity in inter-rib area unchanged and improves the preserving of the
details in the inner-rib area, respectively.

Besides the performance of rib suppression, we also compare our proposed RSGAN
with Li et al. [25] in downstream applications, including lung disease
classification and tuberculosis detection. As shown in Table 4, using only
rib-suppressed image achieves a lower AUC than that of using only original CXR
images. And combining CXR with its corresponding rib-suppressed image obtains
better result than using either of them. It is demonstrated that the rib
suppression can improve the reliability of pulmonary disease diagnosis by
reducing overlapped anatomies and ambiguous structure details, but there might
be some details missing in the rib suppressed image, which cause a drop of AUC
in lung disease classification. Compared with the result of Li et al. [25],
combing the CXR with rib-suppressed image predicted by our proposed RSGAN
obtains the best performance in both lung disease classification and
tuberculosis detection tasks.

##  6 Conclusion

In this paper, we propose a GAN-based disentanglement learning network to
obtain the rib suppression result of a CXR image automatically. The proposed
approach aims to suppress ribs by predicting the residual map between the
input CXR and its rib-suppressed image. Specifically, we train the model on
annotated DRRs for rib suppression and transfer structural priors derived from
unpaired CT/DRR images into the CXR domain. Furthermore, we propose three rib
suppression loss functions based on prior knowledge to improve the performance
of the generated residual map. Experimental results based on multiple
benchmarking CXR datasets demonstrate that the performances of automatic lung
disease classification and TB area detection are boosted with the aid of rib-
suppressed images produced by our approach. The limitation of our RSGAN is
that minor pixel-level rib residues may appear in the predicted rib-suppressed
CXR images caused by interpolation bias when resampling a low-resolution image
to the high-resolution image. A future study will focus on the generation of a
high-resolution residual map in order to achieve a more accurate rib-
suppressed image.

## References

  * Albarqouni et al. [2017] Albarqouni, S., Fotouhi, J., Navab, N., 2017.  X-ray in-depth decomposition: Revealing the latent structures, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 444–452.
  * Armato III et al. [2011] Armato III, S.G., McLennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P., Zhao, B., Aberle, D.R., Henschke, C.I., Hoffman, E.A., et al., 2011\.  The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans.  Medical physics 38, 915–931.
  * de Bel et al. [2021] de Bel, T., Bokhorst, J.M., van der Laak, J., Litjens, G., 2021\.  Residual cyclegan for robust domain transformation of histopathological tissue slides.  Medical Image Analysis 70, 102004\.
  * Ben-Cohen et al. [2019] Ben-Cohen, A., Mechrez, R., Yedidia, N., Greenspan, H., 2019\.  Improving cnn training using disentanglement for liver lesion classification in ct, in: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE. pp. 886–889.
  * von Berg et al. [2016] von Berg, J., Young, S., Carolus, H., Wolz, R., Saalbach, A., Hidalgo, A., Giménez, A., Franquet, T., 2016\.  A novel bone suppression method that improves lung nodule detection.  International journal of computer assisted radiology and surgery 11, 641–655.
  * Brenner and Hall [2007] Brenner, D.J., Hall, E.J., 2007\.  Computed tomography—an increasing source of radiation exposure.  New England Journal of Medicine 357, 2277–2284.
  * Chartsias et al. [2019] Chartsias, A., Joyce, T., Papanastasiou, G., Semple, S., Williams, M., Newby, D.E., Dharmakumar, R., Tsaftaris, S.A., 2019\.  Disentangled representation learning in cardiac image analysis.  Medical image analysis 58, 101535\.
  * Chen and Suzuki [2013] Chen, S., Suzuki, K., 2013.  Separation of bones from chest radiographs by means of anatomically specific multiple massive-training anns combined with total variation minimization smoothing.  IEEE transactions on medical imaging 33, 246–257.
  * Chen et al. [2019] Chen, Y., Gou, X., Feng, X., Liu, Y., Qin, G., Feng, Q., Yang, W., Chen, W., 2019.  Bone suppression of chest radiographs with cascaded convolutional networks in wavelet domain.  IEEE Access 7, 8346–8357.
  * Gozes and Greenspan [2018] Gozes, O., Greenspan, H., 2018\.  Lung structures enhancement in chest radiographs via ct based fcnn training, in: Image Analysis for Moving Organ, Breast, and Thoracic Images. Springer, pp. 147–158.
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Hogeweg et al. [2013a] Hogeweg, L., Sánchez, C.I., van Ginneken, B., 2013a.  Suppression of translucent elongated structures: applications in chest radiography.  IEEE transactions on medical imaging 32, 2099–2113.
  * Hogeweg et al. [2013b] Hogeweg, L., Sánchez, C.I., van Ginneken, B., 2013b.  Suppression of translucent elongated structures: applications in chest radiography.  IEEE transactions on medical imaging 32, 2099–2113.
  * Hogeweg et al. [2012] Hogeweg, L., Sánchez, C.I., de Jong, P.A., Maduskar, P., van Ginneken, B., 2012.  Clavicle segmentation in chest radiographs.  Medical image analysis 16, 1490–1502.
  * Hossain et al. [2018] Hossain, R., Wu, C.C., de Groot, P.M., Carter, B.W., Gilman, M.D., Abbott, G.F., 2018\.  Missed lung cancer.  Radiologic Clinics 56, 365–375.
  * Huang et al. [2017] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., 2017.  Densely connected convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708.
  * Huang et al. [2018] Huang, X., Liu, M.Y., Belongie, S., Kautz, J., 2018\.  Multimodal unsupervised image-to-image translation, in: Proceedings of the European conference on computer vision (ECCV), pp. 172–189.
  * Jaeger et al. [2014] Jaeger, S., Candemir, S., Antani, S., Wáng, Y.X.J., Lu, P.X., Thoma, G., 2014\.  Two public chest x-ray datasets for computer-aided screening of pulmonary diseases.  Quantitative imaging in medicine and surgery 4, 475.
  * Jaeger et al. [2013] Jaeger, S., Karargyris, A., Candemir, S., Siegelman, J., Folio, L., Antani, S., Thoma, G., 2013.  Automatic screening for tuberculosis in chest radiographs: a survey.  Quantitative imaging in medicine and surgery 3, 89.
  * Kamnitsas et al. [2017] Kamnitsas, K., Baumgartner, C., Ledig, C., Newcombe, V., Simpson, J., Kane, A., Menon, D., Nori, A., Criminisi, A., Rueckert, D., et al., 2017\.  Unsupervised domain adaptation in brain lesion segmentation with adversarial networks, in: International conference on information processing in medical imaging, Springer. pp. 597–609.
  * Karras et al. [2020] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T., 2020\.  Analyzing and improving the image quality of stylegan, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119.
  * Krizhevsky et al. [2012] Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems 25, 1097–1105.
  * Laskey [1996] Laskey, M.A., 1996.  Dual-energy x-ray absorptiometry and body composition.  Nutrition 12, 45–51.
  * Lee et al. [2018] Lee, H.Y., Tseng, H.Y., Huang, J.B., Singh, M., Yang, M.H., 2018.  Diverse image-to-image translation via disentangled representations, in: Proceedings of the European conference on computer vision (ECCV), pp. 35–51.
  * Li et al. [2020] Li, H., Han, H., Li, Z., Wang, L., Wu, Z., Lu, J., Zhou, S.K., 2020.  High-resolution chest x-ray bone suppression using unpaired ct structural priors.  IEEE transactions on medical imaging 39, 3053–3063.
  * Li et al. [2019] Li, Z., Li, H., Han, H., Shi, G., Wang, J., Zhou, S.K., 2019.  Encoding ct anatomy knowledge for unpaired chest x-ray image decomposition, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 275–283.
  * Liu et al. [2020] Liu, Y., Wu, Y.H., Ban, Y., Wang, H., Cheng, M.M., 2020\.  Rethinking computer-aided tuberculosis diagnosis, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2646–2655.
  * Maduskar et al. [2013] Maduskar, P., Hogeweg, L., Philipsen, R., Schalekamp, S., Van Ginneken, B., 2013.  Improved texture analysis for automatic detection of tuberculosis (tb) on chest radiographs with bone suppression images, in: Medical Imaging 2013: Computer-Aided Diagnosis, International Society for Optics and Photonics. p. 86700H.
  * Nie et al. [2018] Nie, D., Trullo, R., Lian, J., Wang, L., Petitjean, C., Ruan, S., Wang, Q., Shen, D., 2018.  Medical image synthesis with deep convolutional adversarial networks.  IEEE Transactions on Biomedical Engineering 65, 2720–2730.
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., Brox, T., 2015.  U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, Springer. pp. 234–241.
  * Simkó et al. [2009] Simkó, G., Orbán, G., Máday, P., Horváth, G., 2009\.  Elimination of clavicle shadows to help automatic lung nodule detection on chest radiographs, in: 4th European Conference of the International Federation for Medical and Biological Engineering, Springer. pp. 488–491.
  * Sun et al. [2020] Sun, L., Wang, J., Huang, Y., Ding, X., Greenspan, H., Paisley, J., 2020.  An adversarial learning approach to medical image synthesis for lesion detection.  IEEE journal of biomedical and health informatics 24, 2303–2314.
  * Suzuki et al. [2006] Suzuki, K., Abe, H., MacMahon, H., Doi, K., 2006\.  Image-processing technique for suppressing ribs in chest radiographs by means of massive training artificial neural network (mtann).  IEEE Transactions on medical imaging 25, 406–416.
  * Telea [2004] Telea, A., 2004.  An image inpainting technique based on the fast marching method.  Journal of graphics tools 9, 23–34.
  * Unberath et al. [2019] Unberath, M., Zaech, J.N., Gao, C., Bier, B., Goldmann, F., Lee, S.C., Fotouhi, J., Taylor, R., Armand, M., Navab, N., 2019\.  Enabling machine learning in x-ray-based procedures via realistic simulation of image formation.  International journal of computer assisted radiology and surgery 14, 1517–1528.
  * Wang et al. [2017] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M., 2017.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097–2106.
  * Wu et al. [2012] Wu, D., Liu, D., Puskas, Z., Lu, C., Wimmer, A., Tietjen, C., Soza, G., Zhou, S.K., 2012.  A learning based deformable template matching method for automatic rib centerline extraction and labeling in ct images, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE. pp. 980–987.
  * Yang et al. [2019] Yang, J., Dvornek, N.C., Zhang, F., Chapiro, J., Lin, M., Duncan, J.S., 2019\.  Unsupervised domain adaptation via disentangled representations: Application to cross-modality liver segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 255–263.
  * Yang et al. [2017] Yang, W., Chen, Y., Liu, Y., Zhong, L., Qin, G., Lu, Z., Feng, Q., Chen, W., 2017.  Cascade of multi-scale convolutional neural networks for bone suppression of chest radiographs in gradient domain.  Medical image analysis 35, 421–433.
  * Zhang et al. [2018a] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O., 2018a.  The unreasonable effectiveness of deep features as a perceptual metric, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595.
  * Zhang et al. [2018b] Zhang, Y., Miao, S., Mansi, T., Liao, R., 2018b.  Task driven generative modeling for unsupervised domain adaptation: Application to x-ray image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 599–607.
  * Zhang et al. [2018c] Zhang, Z., Yang, L., Zheng, Y., 2018c.  Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network, in: Proceedings of the IEEE conference on computer vision and pattern Recognition, pp. 9242–9251.
  * Zhou et al. [2018] Zhou, B., Lin, X., Eck, B., Hou, J., Wilson, D., 2018\.  Generation of virtual dual energy images from standard single-shot radiographs using multi-scale and conditional adversarial network, in: Asian Conference on Computer Vision, Springer. pp. 298–313.
  * Zhu et al. [2017] Zhu, J.Y., Park, T., Isola, P., Efros, A.A., 2017\.  Unpaired image-to-image translation using cycle-consistent adversarial networks, in: Proceedings of the IEEE international conference on computer vision, pp. 2223–2232.