# Breaking the Dilemma of Medical Image-to-image Translation

Lingke Kong111Equal contribution.
Manteia Tech
<EMAIL_ADDRESS>
&Chenyu Lian11footnotemark: 1
Xiamen University
<EMAIL_ADDRESS>
Detian Huang
Huaqiao University
<EMAIL_ADDRESS>
&Zhenjiang Li
Shandong University
<EMAIL_ADDRESS>
&Yanle Hu222Corresponding author.
Mayo Clinic Arizona
<EMAIL_ADDRESS>
&Qichao Zhou22footnotemark: 2
Manteia Tech
<EMAIL_ADDRESS>

###### Abstract

Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that
dominate the field of medical image-to-image translation. However, neither
modes are ideal. The Pix2Pix mode has excellent performance. But it requires
paired and well pixel-wise aligned images, which may not always be achievable
due to respiratory motion or anatomy change between times that paired images
are acquired. The Cycle-consistency mode is less stringent with training data
and works well on unpaired or misaligned images. But its performance may not
be optimal. In order to break the dilemma of the existing modes, we propose a
new unsupervised mode called RegGAN for medical image-to-image translation. It
is based on the theory of "loss-correction". In RegGAN, the misaligned target
images are considered as noisy labels and the generator is trained with an
additional registration network to fit the misaligned noise distribution
adaptively. The goal is to search for the common optimal solution to both
image-to-image translation and registration tasks. We incorporated RegGAN into
a few state-of-the-art image-to-image translation methods and demonstrated
that RegGAN could be easily combined with these methods to improve their
performances. Such as a simple CycleGAN in our mode surpasses latest NICEGAN
even though using less network parameters. Based on our results, RegGAN
outperformed both Pix2Pix on aligned data and Cycle-consistency on misaligned
or unpaired data. RegGAN is insensitive to noises which makes it a better
choice for a wide range of scenarios, especially for medical image-to-image
translation tasks in which well pixel-wise aligned data are not available.
Code and data used in this study can be found at https://github.com/Kid-
Liet/Reg-GAN.

##  1 Introduction

Generative adversarial networks (GANs)[1] is a framework that simultaneously
trains a generator $G$ and a discriminator $D$ through an adversarial process.
The generator is used to translate the distribution of source domain images
$X$ to the distribution of target domain images $Y$. The discriminator is used
to determine if the target domain images are likely from the generator or from
the real data.

$\displaystyle\min_{G}\max_{D}\mathcal{L}_{Adv}\left(G,D\right)=\mathbb{E}_{y}\left[log\left(D\left(y\right)\right)\right]+\mathbb{E}_{x}\left[log\left(1-D\left(G\left(x\right)\right)\right)\right]$
(1)

Supervised Pix2Pix[2] and unsupervised Cycle-consistency[3] are the two
commonly used modes in GANs. Pix2Pix updates the generator ($G:X\rightarrow
Y$) by minimizing pixel-level $L1$ loss between the source image $x$ and the
target image $y$. Therefore, it requires well aligned paired images, where
each pixel has a corresponding label.

$\displaystyle\min_{G}\mathcal{L}_{L1}\left(G\right)=\mathbb{E}_{x,y}\left[\|y-G\left(x\right)\|_{1}\right]$
(2)

Well aligned paired images, however, are not always available in real-world
scenarios. To address the challenges caused by misaligned images, Cycle-
consistency was developed which was based on the assumption that the generator
$G$ from the source domain $X$ to the target domain $Y$ ($G:X\rightarrow Y$)
was the reverse of the generator $F$ from $Y$ to $X$ ($F:Y\rightarrow X$).
Compared to the Pix2Pix mode, the Cycle-consistency mode works better on
misaligned or unpaired images.

$\displaystyle\min_{G}\min_{F}\mathcal{L}_{Cyc}\left(G,F\right)=\mathbb{E}_{x}\left[\|F\left(G\left(x\right)\right)-x\|_{1}\right]+\mathbb{E}_{y}\left[\|G\left(F\left(y\right)\right)-y\|_{1}\right]$
(3)

The Cycle-consistency mode, however, has its limitations. In the field of
medical image-to-image translation, it requires not only the style translation
between image domains, but also the translation between specific pair of
images. The optimal solution should be unique. For example, the translated
images should maintain the anatomical features of the original images as much
as possible. It is known that the Cycle-consistency mode may produce multiple
solutions[4, 5], meaning that the training process may be relatively
perturbing and the results may not be accurate. The pix2pix mode is not ideal
either. Even though it has a unique solution, it is difficult to satisfy the
requirement asking for well aligned paired images. With misaligned images, the
errors are propagated through the Pix2Pix mode which may result in
unreasonable displacements on the final translated images.

As of today, there is no image-to-image translation mode that can outperform
both the Pix2Pix mode on aligned data and the Cycle-consistency mode on
misaligned or unpaired data. Inspired by[6, 7, 8, 9, 10], we consider the
misaligned target images as noisy labels, which means that the existing
problem is regarded as supervised learning with noisy labels. So we introduce
a new image-to-image translation mode called RegGAN. Figure 1 provides a
comparison of the three modes: Pix2Pix, Cycle-consistency and RegGAN. To
facilitate reading, we summarize our contributions as follows.

  * •

We demonstrate the feasibility of RegGAN from the theoretical perspective of
"loss-correction". Specifically, we train the generator using an additional
registration network to fit the misaligned noise distribution adaptively, with
the goal to search for the common optimal solution for both image-to-image
translation and registration tasks.

  * •

RegGAN eliminates the requirement for well aligned paired images and searches
unique solution in training process. Based on our results, RegGAN outperformed
both Pix2Pix on aligned data and Cycle-consistency on misaligned or unpaired
data.

  * •

RegGAN can be integrated into other methods without changing the original
network architecture. Compared to Cycle-consistency with two generators and
discriminators, RegGAN can provide better performance using less network
parameters.

Figure 1: Comparison among the modes of Pix2Pix, CycleGAN and RegGAN.

##  2 Related Work

Image-to-image Translation: Generative adversarial networks (GANs) have shown
great potential in the field of image-to-image translation[11, 12, 13, 14, 15,
16]. It has been successfully implemented in medical image analysis like
segmentation[17], registration[18, 19] and dose calculation[20]. The existing
modes, however, have their limitations. Specifically, the Pix2Pix mode[2]
requires well aligned paired images which may not always be available. The
Cycle-consistency mode can achieve unsupervised image-to-image translation.
With a Cycle-consistency loss, it can be used for misaligned images. Based on
Cycle-consistency, many methods[3, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]
have been developed including CycleGAN[3] and its variants such as MUNIT[31]
and UNIT[32] in which both image content and style information are used to
decouple and reconstruct the image-to-image translation task; U-gat-it[33]
with a self-attention mechanism added; and NICEGAN[34] proposed to reuse the
discriminator for encoding. The main limitation of Cycle-consistency is that
it may produce multiple solutions and therefore is sensitive to perturbation,
making it difficult to meet the high accuracy requirements of medical image-
to-image translation tasks.

Learning From Noisy Labels: Neural network anti-noise training has made great
progress. Current research are mainly focused on: estimating the noise
transition matrix[7, 35, 36, 37, 38, 39, 40], designing a robust loss
function[41, 42, 43, 44], correcting the noise label[45, 46, 47, 48, 49, 50],
sampling importance weighting[51, 52, 53, 54, 55] and meta-learning[56, 57,
58, 59]. Our work is in the category of estimating the noise transition
matrix. Compared to conventional noise transition estimation, we mitigate the
issue and simplify the task by acquiring prior knowledge of noise
distribution.

Deformable Registration: Traditional image registration methods have gained
widespread acceptance, such as Demons[60], B-spline[61] and elastic
deformation model[62]. One of the most popular deep learning methods is
Voxelmorph[63]. In this study, a CNN model was trained to predict the
deformable vector filed (DVF). The time-consuming gradient derivation process
was thus skipped to improve the calculation efficiency. Affine registration
and a vector momentum-parameterized stationary velocity field (vSVF)[64] was
implemented to get better transformation regulation. Fast Symmetric method[65]
used symmetric maximum similarity. Deep flash[66] outperformed other models in
terms of training and calculation time.

Closest to our work, Arar.M et al[67] introduced a multi-modal registration
method for natural images based on geometry preserving. But their work focused
only on registration and did not demonstrate results of image-to-image
translation or discuss the relationship between registration and image-to-
image translation. The key insight of our work is that we demonstrated the
feasibility of using registration to significantly improve the performance of
image-to-image translation because the noise could be eliminated adaptively
during the joint training process. What we propose in the paper is a
completely new mode for medical image-to-image translation.

##  3 Methodology

###  3.1 Theoretical Motivation

If we consider misaligned target images as noisy labels, the training for
image-to-image translation becomes a supervised learning process with noisy
labels. Given a training dataset
$\left\\{(x_{n},\widetilde{y}_{n})\right\\}_{n=1}^{N}$ with $N$ noisy labels
in which $x_{n}$, $\widetilde{y}_{n}$ are images from two modalities and
assume $y_{n}$ is the correct label for $x_{n}$, but it is unknown in real-
world scenarios. Our goal is to train a generator using the dataset
$\left\\{(x_{n},\widetilde{y}_{n})\right\\}_{n=1}^{N}$ with noisy labels and
achieve the performance equivalent to trained on clean dataset
$\left\\{(x_{n},y_{n})\right\\}_{n=1}^{N}$ as much as possible. Direct
optimization based on Equations 4 usually does not work and can lead to bad
results because the generator cannot squeeze out the influence of noise.

$\displaystyle\hat{G}=\mathop{\arg\min}_{G}\frac{1}{N}\sum_{n=1}^{N}\mathcal{L}\left(G\left(x_{n}\right),\widetilde{y}_{n}\right)$
(4)

To address the noise issue, we propose a solution based on "loss-
correction"[7] shown in Equations 5. Our solution corrects the output of the
generator $G(x_{n})$ by modeling a noise transition $\phi$ to match the noise
distribution. Previously, Patrini et al[7] proved mathematically that the
model trained with the noisy labels could be equivalent to the model trained
with the clean labels, if the noise transition $\phi$ matches the noise
distribution.

$\displaystyle\hat{G}=\mathop{\arg\min}_{G}\frac{1}{N}\sum_{n=1}^{N}\mathcal{L}\left(\phi\circ
G\left(x_{n}\right),\widetilde{y}_{n}\right)$ (5)

To achieve this, Goldberger et al[36] proposed to view the correct label as a
latent random variable and explicitly model the label noise as a part of the
network architecture, denoted by $R$. Then, Equations 5 can be rewritten in
the form of log-likelihood, which is used as the loss function for neural
network training.

$\displaystyle\mathcal{L}\left(G,R\right)$
$\displaystyle=-\sum_{n=1}^{N}log\left(p\left(\widetilde{y}_{n}|y_{n};R\right)p\left({y}_{n}|x_{n};G\right)\right)$
(6)
$\displaystyle=-\sum_{n=1}^{N}log\left(p\left(\widetilde{y}_{n}|x_{n};G,R\right)\right)$

###  3.2 RegGAN

Compared to existing methods that use expectation-maximum[7, 36], fully
connected layers[35], anchor point estimate[37] and Drichlet-distribution[38]
to solve Equations 6. In our problem, the type of noise distribution is
clearer, it can be expressed as displacement error: $\widetilde{y}=y\circ T$.
Here $T$ is expressed as a random deformation field, which produces random
displacement for each pixel. So we adopt a registration network $R$ after the
generator $G$ as label noise model to correct the results. The Correction loss
is shown Equations 7:

$\displaystyle\min_{G,R}\mathcal{L}_{Corr}\left(G,R\right)=\mathbb{E}_{x,\widetilde{y}}\left[\|\widetilde{y}-G\left(x\right)\circ
R\left(G\left(x\right),\widetilde{y}\right)\|_{1}\right]$ (7)

where, $R\left(G\left(x\right),\widetilde{y}\right)$ is the deformation field
and $\circ$ represents the resamples operation. The registration network is
based on U-Net[68]. A smoothness loss[63] is defined in Equations 8 to
evaluate the smoothness of the deformation field and minimize the gradient of
the deformation field.

$\displaystyle\min_{R}\mathcal{L}_{Smooth}\left(R\right)=\mathbb{E}_{x,\widetilde{y}}\left[\|\nabla
R\left(G\left(x\right),\widetilde{y}\right)\|^{2}\right]$ (8)

Finally, we add the Aversarial loss between the generator and the
discriminator (Equations 1), and the total loss is expressed in Equations 9.

$\displaystyle\min_{G,R}\max_{D}\mathcal{L}_{Total}\left(G,R,D\right)=\mathcal{L}_{Corr}+\mathcal{L}_{Smooth}+\mathcal{L}_{Adv}$
(9)

##  4 Experiments

Performance evaluation of RegGAN was conducted through three investigations to
1) demonstrate the feasibility and superiority of the RegGAN mode in various
methods, and 2) assess RegGAN’s sensitivity to noise, and 3) explore the
availability of the RegGAN on unpaired data.

###  4.1 Dataset

The open-access dataset (BraTS 2018[69]) was used to evaluate the proposed
RegGAN mode. The training dataset and testing dataset contained 8457 and 979
pairs of T1 and T2 MR images, respectively. BraTS 2018 was selected because
the original images were paired and well aligned. We created misaligned images
by randomly adding different levels of rotation, translation and rescaling to
the original images. And we randomly sample one image from T1 and the other
one from T2 when training on unpaired images. The availability of well aligned
paired images, misaligned paired images, and unpaired images allow us to
evaluate the performances of all three modes (Pix2Pix, Cycle-consistency and
RegGAN).

###  4.2 Performances in Different Methods

The primary motivation of introducing RegGAN was to address challenges caused
by misaligned data. Therefore, in this section, misaligned data were used in
model training to demonstrate the feasibility and superiority of RegGAN. We
selected the most popular CycleGAN[3] and its variants MUNIT[31], UNIT[32],
and NICEGAN[34] as the methods for evaluation and compared the following four
modes for each method.

  * •

C(Cycle-consistency): The most primitive mode of all methods, with Cycle-
consistency loss (Equations 3) as the main constraint. Two generators and two
discriminators are required in this mode.

  * •

C+R (Cycle-consistency + Registration): The RegGAN mode is combined with the
mode C. Registration network ($R$) and Correction loss (Equations 7) are added
to the constraints.

  * •

NC(Non Cycle-consistency): Only Adversarial loss (Equations 1) is used for
updating. Compared to the mode C, Cycle-consistency loss is removed. Only one
generator and one discriminator are required in this mode.

  * •

NC+R(Non Cycle-consistency + Registration): A registration network ($R$) and
Correction loss (Equations 7) are added to the mode NC. It is the proposed
RegGAN mode.

Table 1: Comparison of CycleGAN, MUNIT, UNIT and NICEGAN using four training modes(C, C+R, NC and NC+R). Index Methods Modes |  | CycleGAN | MUNIT | UNIT | NICEGAN
---|---|---|---|---|---
NMAE $\downarrow$ | C | 0.089 | 0.11 | 0.087 | 0.082
C+R |  (-0.012)0.077 |  (-0.022)0.088 |  (-0.013)0.074 |  (-0.011)0.071
NC | 0.11 | 0.10 | 0.098 | 0.089
NC+R |  (-0.038)0.072 |  (-0.021)0.079 |  (-0.027)0.071 |  (-0.019)0.070
PSNR $\uparrow$ | C | 23.5 | 20.6 | 24.6 | 25.2
C+R |  (+0.3)23.8 |  (+2.1)22.7 |  (+0.7)25.3 |  (+0.9)26.1
NC | 20.2 | 21.5 | 23.7 | 23.5
NC+R |  (+5.4)25.6 |  (+2.3)23.8 |  (+1.8)25.5 |  (+2.8)26.3
SSIM $\uparrow$ | C | 0.83 | 0.80 | 0.84 | 0.83
C+R |  (+0.02)0.85 |  (+0.03) 0.83 |  (+0.02)0.86 |  (+0.03)0.86
NC | 0.79 | 0.81 | 0.83 | 0.84
NC+R |  (+0.07)0.86 |  (+0.04)0.85 |  (+0.03) 0.86 |  (+0.02)0.86

To evaluate the performance of each method on misaligned data, we randomly
added [-5, +5] degrees of angle rotation, [-5, +5] percent of translation, and
[-5, +5] percent of rescaling to the original T1 and T2 images on the training
dataset.

To ensure fair comparison, we used the same training strategy and
hyperparameters for all methods and modes (see supplementary materials for
details). The Normalized Mean Absolute Error (NMAE), Peak Signal to Noise
Ratio (PSNR) and Structural Similarity (SSIM) were used as metrics to evaluate
the performances of trained models based on the testing dataset. To avoid
false high results of index, we excluded the image background from the
calculation. Table 1 summarized the results for all methods and modes under
the current investigation.

Based on the results from the Table 1, we can reach several conclusions.
First, adding the registration network (+R) significantly improves the
performances of the methods. This is true for all methods in both C and NC
modes. It clearly demonstrates that RegGAN can be incorporated in various
methods or combined with different network architectures to improve the
performances. Second, the C mode is in general better than the NC mode for
most of methods. Adding the registration network (+R) improves the performance
of the NC mode more than that of the C mode. In fact, our results show that
the NC+R mode is even better than the C+R mode, implying that "Cycle-
consistency loss" may play a negative role when it is combined with RegGAN.
Compared with the commonly used C mode with two generators and two
discriminators, RegGAN has fewer parameters but provides better performance.
The simple CycleGAN method in the NC+R mode outperforms the current state-of-
the-art method NICEGAN in the C mode by 0.01, 0.4, 0.03 for NMAE, PSNR and
SSIM, respectively. The NC+R mode can also be used to improve the performance
of NICEGAN. In fact, the performance of NICEGAN in the NC+R mode is the best
among all combinations of the 4 methods and 4 modes.

Figure 2 shows representative results from various combinations of the 4
methods (CycleGAN, MUNIT, UNIT and NICEGAN) and 4 modes (C, C+R, NC and NC+R).
For all aspects of the image (from the tumor areas and the details), the
combinations that use the registration network (+R) always provide more
realistic and accurate results than those that do not use the registration
network (+R).

Figure 2: The errors of different modes in different methods.

###  4.3 Performances in Different Noise Levels

To evaluate the sensitivity of RegGAN to noise, we selected a simple network
architecture.(CycleGAN) with the intention to minimize interference from other
factors. The same network architecture was used for all three modes:
CycleGAN(C), Pix2Pix and RegGAN. Seven levels of noise were used in the
evaluation. Table 2 lists the specific noise setting and range for each noise
level. Noise.0 means the original dataset with no added noise. Noise.5 is the
highest level of noise. At Noise.5, the data are likely from different
patients. In addition, we also made non-affine noise(Noise.NA) settings. The
non-affine noise is generated by spatially transforming T1 and T2 using
elastic transformations on control points followed by Gaussian smoothing.
Figure 3 shows example images at different levels of introduced noise.

Figure 3: Example images at seven different levels of introduced noise. Table 2: Comparison of the NMAE, PSNR and SSIM for CycleGAN(C), Pix2Pix and RegGAN under 7 levels of noise. | Noise.0 | Noise.1 | Noise.2 | Noise.3 | Noise.4 | Noise.5 | Noise.NA
---|---|---|---|---|---|---|---
Setting | Rotate | $0^{\circ}$ | $\pm{1}^{\circ}$ | $\pm{2}^{\circ}$ | $\pm{3}^{\circ}$ | $\pm{4}^{\circ}$ | $\pm{5}^{\circ}$ | ✗
Translation | $0\%$ | $\pm{2}\%$ | $\pm{4}\%$ | $\pm{6}\%$ | $\pm{8}\%$ | $\pm{10}\%$ | ✗
Rescaling | $0\%$ | $\pm{2}\%$ | $\pm{4}\%$ | $\pm{6}\%$ | $\pm{8}\%$ | $\pm{10}\%$ | ✗
CycleGAN(C) | NMAE $\downarrow$ | 0.084 | 0.095 | 0.087 | 0.094 | 0.087 | 0.110 | 0.091
PSNR $\uparrow$ | 23.9 | 22.5 | 23.7 | 23.3 | 23.9 | 23.7 | 23.5
SSIM $\uparrow$ | 0.83 | 0.83 | 0.82 | 0.81 | 0.82 | 0.79 | 0.83
Pix2Pix | NMAE $\downarrow$ | 0.075 | 0.103 | 0.139 | 0.161 | 0.175 | 0.181 | 0.086
PSNR $\uparrow$ | 25.6 | 22.3 | 18.9 | 16.2 | 15.3 | 15.0 | 21.1
SSIM $\uparrow$ | 0.85 | 0.82 | 0.78 | 0.76 | 0.74 | 0.74 | 0.82
RegGAN | NMAE $\downarrow$ | 0.071 | 0.073 | 0.071 | 0.072 | 0.072 | 0.072 | 0.071
PSNR $\uparrow$ | 26 | 25.6 | 25.9 | 25.7 | 25.4 | 25.2 | 25.9
SSIM $\uparrow$ | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 0.85 | 0.86

Table 2 lists the quantitative evaluation metrics from 3 modes at 7 levels of
noise. It is clear that RegGAN outperforms CycleGAN(C) under all noise levels.
Figure 4(a) shows the test results from each epoch during the training process
for both RegGAN and CycleGAN(C). Curves of different colors corresponds to
different levels of noise. We notice that CycleGAN(C) is not very stable
during the training process. The test results fluctuate significantly and
cannot converge well. This may be caused by the fact that the solution of
CycleGAN (C) is not unique. As a comparison, RegGAN is quite stable. Although
the results from different levels of noise may vary at the beginning of
training, all curves converge to a similar result after multiple epoches of
training, indicating that RegGAN is more robust to noise compared to CycleGAN
(C).

Figure 4: Quantitative evaluation metrics at different epochs in the training
process. (a) Comparison of CycleGAN and RegGAN at different levels of noise.
(b) Comparison of Pix2Pix and RegGAN at Noise.0 (i.e., no noise). (c) RegGAN’s
Smoothness loss under different levels of noise.

Based on Table 2, we notice that the performance of Pix2Pix deteriorates
rapidly as the noise increases. This is as expected because Pix2Pix requires
well aligned paired images. Surprisingly, the performances of RegGAN at all
noise levels exceed those of Pix2Pix with no noise. Figure 4(b) shows the test
results at each epoch of RegGAN and Pix2Pix under Noise.0 (i.e., no noise).
Theoretically, the performances of RegGAN and Pix2Pix should be similar on
perfectly aligned paired datasets because the registration network of RegGAN
does not help and RegGAN is equivalent to Pix2Pix. A possible explanation to
our results is that in the medical field, the perfectly pixel-wise aligned
dataset may not practically exist. Even for BraTS 2018[63] which is recognized
as well aligned, it is still possible that there exists slight misalignment.
As a result, adding the registration network is always likely to improve the
performances in real-world scenarios. To verify our explanation, we plotted
the Smoothness loss of RegGAN under different noise levels as shown in Figure
4(c). Large Smoothness loss corresponds to large deformation field
displacement. First, we notice that the Smoothness loss under Noise.0 never
completely goes to 0, indicating the existence of misalignment and potential
usefulness of the registration network. Second, the noise level and Smoothness
loss show a step-like positive correlation, which means that RegGAN can
adaptively handle the noise distribution, i.e., the registration network can
determine the range of deformation according to the noise level. In addition,
we can see that even under the setting of non-affine noise, the above
conclusion still holds. Because what the registration network corrects is
deformation noise.

Figure 5: The misalignment of orginal image pairs and corresponding
deformation fields.

we also show some original image pairs and visualize the corresponding
deformation fields output by registration network in Figure 5. Obviously,
there is some misalignment between the original T1 and T2 images, and such
misalignment is represented by the deformation fields (highlighted by red
circle).

###  4.4 Performances on Unpaired Dataset

So far, our investigations are based on paired datasets. We also want to
explore how RegGAN performs using unpaired datasets. In practice, this is not
recommended because even different patients may have similarities in their
body tissues of adjacent layers. For unpaired datasets, we can conduct rigid
registration first in 3D space and then use RegGAN for training. Unpaired data
can be treated as having larger scale noise. If the correction capability is
strong enough, RegGAN can still work effectively. The comparison of the
performances of three modes on the unpaired dataset is shown in Figure 6.

Mode Index | NMAE$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---
Pix2Pix | 0.180 | 15.5 | 0.71
CycleGAN(C) | 0.094 | 23.6 | 0.83
RegGAN | 0.086 | 24.0 | 0.83

Figure 6: Performance comparison of the three modes (CycleGAN(C), Pix2Pix and
RegGAN) on unpaired dataset.

With unpaired datasets, Pix2Pix no longer considers the characteristics of the
input T1 images and thus has the worst performance. Due to the challenges in
fitting the noise, the performance improvement from replacing CycleGAN(C) with
RegGAN using unpaired datasets may not be as dramatic as that demonstrated
using paired datasets, but RegGAN still has the best performance under
unpaired conditions. In Figure 7, we show some examples of how RegGAN corrects
noise on unpaired dataset. It can be seen that RegGAN will try its best to
eliminate the influnce of noise through registration.

Figure 7: Display of RegGAN’s output on unpaired data. T1 and T2 are unpaired
images. The Translated represents the translation result of T1 to T2.
Registered represents the registration result of the translated images. D.F
represents deformation fields.

Based on our results, it is reasonable to reach the conclusions below. In all
circumstances, RegGAN demonstrates better performance compared to Pix2Pix and
CycleGAN(C).

  * •

For paired and aligned conditions, RegGAN $\geq$ Pix2Pix $>$ CycleGAN(C).

  * •

For paired but misaligned conditions, RegGAN $>$ CycleGAN(C) $>$Pix2Pix.

  * •

For unpaired conditions, RegGAN $>$ CycleGAN(C) $>$Pix2Pix.

## Conclusion

In this study, we introduced a new image-to-image translation mode RegGAN to
the medical community that can break the dilemma of image-to-image translation
task. Using a public BraTS 2018 dataset, we demonstrated the feasibility of
RegGAN and its superior performance compared to Pix2Pix and Cycle-consistency.
We validated that RegGAN can be incorporated into various existing methods to
improve their performances. We also evaluated the sensitivity of RegGAN to
noise. Our results confirmed that RegGAN could adapt well to various scenarios
from no noise to large-scale noise. The superior performance of RegGAN makes
it a better choice over Pix2Pix and Cycle-consistency whether datasets are
aligned or not. However, this mode may not work well on natural images. The
noise may cannot be considered simply as deformation errors due to the
differences in natural images are much greater than those in medical images.

## Broader Impact

Image-to-image translation has been one of the main focuses in medical image
analysis, as it aids in diagnosis and treatment. Previously, physicians had to
use different medical imaging equipments if they wanted to get different image
sequences of a patient, which was time-consuming and expensive. Pix2Pix mode
is expected to solve this problem by its outstanding performance in image-to-
image translation. In most of clinical scenarios, however, it is not practical
to create such a large well aligned dataset for Pix2Pix mode. Cycle-
consistence mode does not need well aligned dataset but can not meet the high-
precision requirements of medical image analysis. Our work aims to provide a
general image-to-image translation mode, which not only has no strict
requirements on the dataset, but also can meet the clinical requirements in
terms of image quality. In the future, we will attempt to obtain multi-modal
dataset(eg MR-CT) for clinical verification. We foresee positive impacts if
the mode is applied to diagnosis in radiology, treatment planning and
research.

## References

  * Goodfellow et al. [2014] Goodfellow I, Pouget-Abadie J, Mirza M, et al.  Generative adversarial nets[C]//Ghahramani Z, Welling M, Cortes C, et al.  Advances in Neural Information Processing Systems.  Curran Associates, Inc., 2014.
  * Isola et al. [2017] Isola P, Zhu J Y, Zhou T, et al.  Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.  2017: 1125-1134.
  * Zhu et al. [2017] Zhu J Y, Park T, Isola P, et al.  Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE international conference on computer vision.  2017: 2223-2232.
  * Sim et al. [2019] Sim B, Oh G, Lim S, et al.  Optimal transport, cyclegan, and penalized ls for unsupervised learning in inverse problems[J].  2019\.
  * Moriakov et al. [2020] Moriakov N, Adler J, Teuwen J.  Kernel of cyclegan as a principal homogeneous space[C]//International Conference on Learning Representations.  2020\.
  * Huang et al. [2020] Huang L, Zhang C, Zhang H.  Self-adaptive training: beyond empirical risk minimization[J].  Advances in Neural Information Processing Systems, 2020, 33.
  * Patrini et al. [2017] Patrini G, Rozza A, Krishna Menon A, et al.  Making deep neural networks robust to label noise: A loss correction approach[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.  2017: 1944-1952.
  * Dgani et al. [2018] Dgani Y, Greenspan H, Goldberger J.  Training a neural network based on unreliable human annotation of medical images[C]//2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).  IEEE, 2018: 39-42.
  * Yao et al. [2020] Yao Y, Liu T, Han B, et al.  Dual t: Reducing estimation error for transition matrix in label-noise learning[C/OL]//Larochelle H, Ranzato M, Hadsell R, et al.  Advances in Neural Information Processing Systems: volume 33.  Curran Associates, Inc., 2020: 7260-7271.  https://proceedings.neurips.cc/paper/2020/file/512c5cad6c37edb98ae91c8a76c3a291-Paper.pdf.
  * Zhang et al. [2020a] Zhang L, Tanno R, Xu M C, et al.  Disentangling human error from ground truth in segmentation of medical images[C/OL]//Larochelle H, Ranzato M, Hadsell R, et al.  Advances in Neural Information Processing Systems: volume 33.  Curran Associates, Inc., 2020a: 15750-15762.  https://proceedings.neurips.cc/paper/2020/file/b5d17ed2b502da15aa727af0d51508d6-Paper.pdf.
  * wang et al. [2020] wang y, Yu L, van de Weijer J.  Deepi2i: Enabling deep hierarchical image-to-image translation by transferring from gans[C/OL]//Larochelle H, Ranzato M, Hadsell R, et al.  Advances in Neural Information Processing Systems: volume 33.  Curran Associates, Inc., 2020: 11803-11815.  https://proceedings.neurips.cc/paper/2020/file/88855547570f7ff053fff7c54e5148cc-Paper.pdf.
  * Zhang et al. [2020b] Zhang P, Zhang B, Chen D, et al.  Cross-domain correspondence learning for exemplar-based image translation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.  2020b: 5143-5153.
  * Kim et al. [2017] Kim T, Cha M, Kim H, et al.  Learning to discover cross-domain relations with generative adversarial networks[C]//International Conference on Machine Learning.  PMLR, 2017: 1857-1865.
  * Cao et al. [2020] Cao Y, Wan X.  Divgan: Towards diverse paraphrase generation via diversified generative adversarial network[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings.  2020: 2411-2421.
  * Choi et al. [2018] Choi Y, Choi M, Kim M, et al.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.  2018: 8789-8797.
  * Choi et al. [2020] Choi Y, Uh Y, Yoo J, et al.  Stargan v2: Diverse image synthesis for multiple domains[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.  2020: 8188-8197.
  * Chen et al. [2019] Chen C, Dou Q, Chen H, et al.  Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation[C]//Proceedings of the AAAI Conference on Artificial Intelligence: volume 33.  2019: 865-872.
  * Fan et al. [2019] Fan J, Cao X, Wang Q, et al.  Adversarial learning for mono-or multi-modal registration[J].  Medical image analysis, 2019, 58:101545.
  * Qin et al. [2019] Qin C, Shi B, Liao R, et al.  Unsupervised deformable registration for multi-modal images via disentangled representations[C]//International Conference on Information Processing in Medical Imaging.  Springer, 2019: 249-261.
  * Sahiner et al. [2019] Sahiner B, Pezeshk A, Hadjiiski L M, et al.  Deep learning in medical imaging and radiation therapy[J].  Medical physics, 2019, 46(1):e1-e36.
  * Hoffman et al. [2018] Hoffman J, Tzeng E, Park T, et al.  Cycada: Cycle-consistent adversarial domain adaptation[C]//International conference on machine learning.  PMLR, 2018: 1989-1998.
  * Li et al. [2018] Li M, Huang H, Ma L, et al.  Unsupervised image-to-image translation with stacked cycle-consistent adversarial networks[C]//Proceedings of the European conference on computer vision (ECCV).  2018: 184-199.
  * Deng et al. [2018] Deng W, Zheng L, Ye Q, et al.  Image-image domain adaptation with preserved self-similarity and domain-dissimilarity for person re-identification[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.  2018: 994-1003.
  * Li et al. [2019a] Li J, Chen E, Ding Z, et al.  Cycle-consistent conditional adversarial transfer networks[C]//Proceedings of the 27th ACM International Conference on Multimedia.  2019a: 747-755.
  * Schmidt et al. [2019] Schmidt V, Luccioni A, Mukkavilli S K, et al.  Visualizing the consequences of climate change using cycle-consistent adversarial networks[J].  arXiv preprint arXiv:1905.03709, 2019.
  * Chen et al. [2020a] Chen Z, Li J, Luo Y, et al.  Canzsl: Cycle-consistent adversarial networks for zero-shot learning from natural language[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.  2020a: 874-883.
  * Ren et al. [2020] Ren C X, Ziemann A, Theiler J, et al.  Cycle-consistent adversarial networks for realistic pervasive change generation in remote sensing imagery[C]//2020 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI).  IEEE, 2020: 42-45.
  * Liu et al. [2020] Liu J, Ding Y, Xiong J, et al.  Multi-cycle-consistent adversarial networks for ct image denoising[C]//2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).  IEEE, 2020: 614-618.
  * Zheng et al. [2021] Zheng C, Pan L, Wu P.  Camu: Cycle-consistent adversarial mapping model for user alignment across social networks[J].  IEEE Transactions on Cybernetics, 2021.
  * Shah et al. [2019] Shah M, Chen X, Rohrbach M, et al.  Cycle-consistency for robust visual question answering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.  2019: 6649-6658.
  * Huang et al. [2018] Huang X, Liu M Y, Belongie S, et al.  Multimodal unsupervised image-to-image translation[C]//Proceedings of the European conference on computer vision (ECCV).  2018: 172-189.
  * Liu et al. [2017] Liu M Y, Breuel T, Kautz J.  Unsupervised image-to-image translation networks[C/OL]//Guyon I, Luxburg U V, Bengio S, et al.  Advances in Neural Information Processing Systems: volume 30.  Curran Associates, Inc., 2017.  https://proceedings.neurips.cc/paper/2017/file/dc6a6489640ca02b0d42dabeb8e46bb7-Paper.pdf.
  * Kim et al. [2020] Kim J, Kim M, Kang H, et al.  U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation[C/OL]//International Conference on Learning Representations.  2020\.  https://openreview.net/forum?id=BJlZ5ySKPH.
  * Chen et al. [2020b] Chen R, Huang W, Huang B, et al.  Reusing discriminators for encoding: Towards unsupervised image-to-image translation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.  2020b: 8168-8177.
  * S. Sukhbaatar [2015] S. Sukhbaatar M P L B R F, J. Bruna.  Training convolutional networks with noisy labels[C]//International Conference on Learning Representations.  2015\.
  * Goldberger et al. [2016] Goldberger J, Ben-Reuven E.  Training deep neural-networks using a noise adaptation layer[J].  2016\.
  * Xia et al. [2019] Xia X, Liu T, Wang N, et al.  Are anchor points really indispensable in label-noise learning?[C/OL]//NeurIPS.  2019: 6835-6846.  http://papers.nips.cc/paper/8908-are-anchor-points-really-indispensable-in-label-noise-learning.
  * Yao et al. [2019] Yao J, Wu H, Zhang Y, et al.  Safeguarded dynamic label regression for noisy supervision[C]//Proceedings of the AAAI Conference on Artificial Intelligence: volume 33.  2019: 9103-9110.
  * Xiao et al. [2015] Xiao T, Xia T, Yang Y, et al.  Learning from massive noisy labeled data for image classification[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.  2015: 2691-2699.
  * Misra et al. [2016] Misra I, Lawrence Zitnick C, Mitchell M, et al.  Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.  2016: 2930-2939.
  * Tanno et al. [2019] Tanno R, Saeedi A, Sankaranarayanan S, et al.  Learning from noisy labels by regularized estimation of annotator confusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.  2019: 11244-11253.
  * Rodrigues et al. [2018] Rodrigues F, Pereira F.  Deep learning from crowds[C]//Proceedings of the AAAI Conference on Artificial Intelligence: volume 32.  2018\.
  * Branson et al. [2017] Branson S, Van Horn G, Perona P.  Lean crowdsourcing: Combining humans and machines in an online system[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.  2017: 7474-7483.
  * Izadinia et al. [2015] Izadinia H, Russell B C, Farhadi A, et al.  Deep classifiers from image tags in the wild[C]//Proceedings of the 2015 Workshop on Community-Organized Multimodal Mining: Opportunities for Novel Solutions.  2015: 13-18.
  * Jaehwan et al. [2019] Jaehwan L, Donggeun Y, Hyo-Eun K.  Photometric transformer networks and label adjustment for breast density prediction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.  2019: 0-0.
  * Veit et al. [2017] Veit A, Alldrin N, Chechik G, et al.  Learning from noisy large-scale datasets with minimal supervision[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.  2017: 839-847.
  * Tanaka et al. [2018] Tanaka D, Ikami D, Yamasaki T, et al.  Joint optimization framework for learning with noisy labels[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.  2018: 5552-5560.
  * Zheng et al. [2020] Zheng S, Wu P, Goswami A, et al.  Error-bounded correction of noisy labels[C]//International Conference on Machine Learning.  PMLR, 2020: 11447-11457.
  * Han et al. [2019] Han J, Luo P, Wang X.  Deep self-learning from noisy labels[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.  2019: 5138-5147.
  * Durand et al. [2019] Durand T, Mehrasa N, Mori G.  Learning a deep convnet for multi-label classification with partial labels[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.  2019: 647-657.
  * Wu et al. [2018] Wu X, He R, Sun Z, et al.  A light cnn for deep face representation with noisy labels[J].  IEEE Transactions on Information Forensics and Security, 2018, 13(11):2884-2896.
  * Huang et al. [2019] Huang J, Qu L, Jia R, et al.  O2u-net: A simple noisy label detection approach for deep neural networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.  2019: 3326-3334.
  * Li et al. [2020] Li J, Socher R, Hoi S C.  Dividemix: Learning with noisy labels as semi-supervised learning[C/OL]//International Conference on Learning Representations.  2020\.  https://openreview.net/forum?id=HJgExaVtwr.
  * Yan et al. [2016] Yan Y, Xu Z, Tsang I, et al.  Robust semi-supervised learning through label aggregation[C]//Proceedings of the AAAI Conference on Artificial Intelligence: volume 30.  2016\.
  * Jiang et al. [2018] Jiang J, Ma J, Wang Z, et al.  Hyperspectral image classification in the presence of noisy labels[J].  IEEE Transactions on Geoscience and Remote Sensing, 2018, 57(2):851-865.
  * Li et al. [2019b] Li J, Wong Y, Zhao Q, et al.  Learning to learn from noisy labeled data[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.  2019b: 5051-5059.
  * Li et al. [2017] Li Y, Yang J, Song Y, et al.  Learning from noisy labels with distillation[C]//Proceedings of the IEEE International Conference on Computer Vision.  2017: 1910-1918.
  * Algan et al. [2021] Algan G, Ulusoy I.  Meta soft label generation for noisy labels[C]//2020 25th International Conference on Pattern Recognition (ICPR).  IEEE, 2021: 7142-7148.
  * Garcia et al. [2016] Garcia L P, de Carvalho A C, Lorena A C.  Noise detection in the meta-learning level[J].  Neurocomputing, 2016, 176:14-25.
  * Thirion [1998] Thirion J P.  Image matching as a diffusion process: an analogy with maxwell’s demons[J].  Medical image analysis, 1998, 2:243260.
  * Rueckert et al. [1999] Rueckert D, Sonoda L I, Hayes C, et al.  Nonrigid registration using free-form deformations: application to breast mr images[J].  IEEE transactions on medical imaging, 1999, 18(8):712-721.
  * Shen et al. [2002] Shen D, Davatzikos C.  Hammer: hierarchical attribute matching mechanism for elastic registration[J].  IEEE transactions on medical imaging, 2002, 21(11):1421-1439.
  * Balakrishnan et al. [2019] Balakrishnan G, Zhao A, Sabuncu M R, et al.  Voxelmorph: a learning framework for deformable medical image registration[J].  IEEE transactions on medical imaging, 2019, 38(8):1788-1800.
  * Shen et al. [2019] Shen Z, Han X, Xu Z, et al.  Networks for joint affine and non-parametric image registration[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.  2019: 4224-4233.
  * Mok et al. [2020] Mok T C, Chung A.  Fast symmetric diffeomorphic image registration with convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.  2020: 4644-4653.
  * Wang et al. [2020] Wang J, Zhang M.  Deepflash: An efficient network for learning-based medical image registration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.  2020: 4444-4452.
  * Arar et al. [2020] Arar M, Ginger Y, Danon D, et al.  Unsupervised multi-modal image registration via geometry preserving image-to-image translation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.  2020: 13410-13419.
  * Ronneberger et al. [2015] Ronneberger O, Fischer P, Brox T.  U-net: Convolutional networks for biomedical image segmentation[C]//International Conference on Medical image computing and computer-assisted intervention.  Springer, 2015: 234-241.
  * Menze et al. [2014] Menze B H, Jakab A, Bauer S, et al.  The multimodal brain tumor image segmentation benchmark (brats)[J].  IEEE transactions on medical imaging, 2014, 34(10):1993-2024.

##  Appendix A Training Details

All the experiments were implemented in Pytorch software on 64-bit Ubuntu
Linux system with 96GB RAM and 24GB Nvidia Titan RTX GPU. All the images were
normalized to [-1, 1] and resampled to 256$\times$256\. We train all methods
using the Adam optimizer with the learning rate of 1e-4 and ($\beta_{1}$
,$\beta_{2}$) = (0.5, 0.999). The batch size was set to 1 with weight decay
1e-4. The training process includes totally 80 epochs and over 640K
iterations. We also set different weights for different loss functions, as
shown in Table 3.

Table 3: The different weights for different loss functions. Weight Loss | $\mathcal{L}_{adv}$ | $\mathcal{L}_{L1}$ | $\mathcal{L}_{cyc}$ | $\mathcal{L}_{corr}$ | $\mathcal{L}_{smooth}$
---|---|---|---|---|---
$\lambda$ | 1 | 100 | 10 | 20 | 10

CycleGAN uses two downsampling convolution blocks, nine residual blocks, two
up-sampling deconvolution blocks and four discriminator layers. Codes are on
https://github.com/junyanz/pytorchCycleGAN-and-pix2pix.

MUNIT assumes that the image representation can be decomposed into a content
code and a style code. It splits a generator into two encoders and a decoder.
Codes are on https://github.com/NVlabs/MUNIT.

UNIT assumes that the different modality of an image should share the same
latent coding space. It shares the weight of the high-level layer stage of the
encoder and decoder. Codes are on https://github.com/mingyuliutw/UNIT.

NICEGAN reuses discriminators for encoding specifically for unsupervised
image-to-image translation. By such a reusing, a more compact and more
effective architecture is derived. Codes are on
https://github.com/alpc91/NICE-GAN-pytorch.

##  Appendix B Theoretical Analysis

In this section, we will analyze the generalization form for the proposed
mode.

Definition: In the image-to-image translation task, the goal is to optimize
and get a generator $G:\begin{aligned}
\mathop{\arg\min}_{G}\mathbb{E}_{x~{}y}\mathcal{L}(G(x),y)\end{aligned}$.
Where $(x,y)$ are paired and aligned multi-modal images,
${\forall}x,y\in\mathbb{R}^{H\times W}$ image space. $\mathcal{L}$ is the loss
function. But in practice, we can only get noisy labels $(x,\widetilde{y})$,
and the correct label $y$ is unknown. The relationship between $\widetilde{y}$
and $y$ can be expressed as displacement error: $\widetilde{y}=y\circ T$. Here
$T$ is expressed as a random deformation field, which produces displacement
for each pixel, ${\forall}T\in\mathbb{R}^{2\times H\times W}$. If we can build
an unbiased estimator of model $R$, it can fit the noise distribution $T$
well, such that under expected label noise the corrected loss equals the
original one computed on clean data.

######  Theorem 1.

Suppose the deformation field $T$ is smooth enough for: $T\circ T^{-1}\equiv
I$. $I$ represents identical transformation. Then, the minimizer of the
corrected loss under the noisy distribution is the same as the minimizer of
the original loss under the clean distribution:

$\displaystyle\mathop{\arg\min}_{G}\mathbb{E}_{x~{}\widetilde{y}}\mathcal{L}(G(x)\circ
T,\widetilde{y})=\mathop{\arg\min}_{G}\mathbb{E}_{x~{}y}\mathcal{L}(G(x),y)$
(10)

###### Proof.

Put $\widetilde{y}=y\circ T$ in the left-hand side of the above equation:

$\displaystyle\mathop{\arg\min}_{G}\mathbb{E}_{x~{}\widetilde{y}}\mathcal{L}(G(x)\circ
T,\widetilde{y})$
$\displaystyle=\mathop{\arg\min}_{G}\mathbb{E}_{x~{}y}\mathcal{L}(G(x)\circ
T,y\circ T)$ (11)
$\displaystyle=\mathop{\arg\min}_{G}\mathbb{E}_{x~{}y}\mathcal{L}(G(x\circ
T^{-1})\circ T,y\circ T\circ T^{-1})$
$\displaystyle=\mathop{\arg\min}_{G}\mathbb{E}_{x~{}y}\mathcal{L}(G(x)\circ
T\circ T^{-1},y\circ T\circ T^{-1})$
$\displaystyle=\mathop{\arg\min}_{G}\mathbb{E}_{x~{}y}\mathcal{L}(G(x),y)$

∎

##  Appendix C More Rusults

The results in Section 4.3 show that RegGAN mode is superior to Pix2Pix mode
at all noise levels. Here, we give a comparison of error mappings of Pix2Pix
and RegGAN under Noise.0(Figure 8) and Noise.5(Figure 9) respectively. In
Figure 8, the RegGAN’s result has smaller errors and smoother texture details
compared to Pix2Pix, but the overall difference is not significant. In Figure
9, Pix2Pix is no longer effective, while RegGAN maintains a good performance.

Figure 8: Qualitative results under Noise.0 (o noise). We show eight examples,
which are randomly selected from total test dataset.

Figure 9: Qualitative results under Noise.5(largest noise in our experiments).
We show eight examples, each one is randomly selected from total test dataset.


11institutetext: Manonmaniam Sundaranar University, Tirunelveli, India
11email: {kiruba.briskline,ampetchiammal}<EMAIL_ADDRESS>

# COVID-19 Disease Identification on Chest-CT images using CNN and VGG16

Briskline Kiruba S  Petchiammal A  D. Murugan

###### Abstract

A newly identified coronavirus disease called COVID-19 mainly affects the
human respiratory system. COVID-19 is an infectious disease caused by a virus
originating in Wuhan, China, in December 2019. Early diagnosis is the primary
challenge of health care providers. In the earlier stage, medical
organizations were dazzled because there were no proper health aids or
medicine to detect a COVID-19. A new diagnostic tool RT-PCR (Reverse
Transcription Polymerase Chain Reaction), was introduced. It collects swab
specimens from the patient’s nose or throat, where the COVID-19 virus gathers.
This method has some limitations related to accuracy and testing time. Medical
experts suggest an alternative approach called CT (Computed Tomography) that
can quickly diagnose the infected lung areas and identify the COVID-19 in an
earlier stage. Using chest CT images, computer researchers developed several
deep learning models identifying the COVID-19 disease. This study presents a
Convolutional Neural Network (CNN) and VGG16-based model for automated
COVID-19 identification on chest CT images. The experimental results using a
public dataset of 14320 CT images showed a classification accuracy of 96.34%
and 96.99% for CNN and VGG16, respectively.

###### Keywords:

Image processing Deep Learning Convolutional Neural Network (CNN) Computed
Tomography COVID-19

##  1 Introduction

COVID-19 is a coronavirus disease that can cause respiratory illness in
humans. COVID‑19 spreads when people breathe in air contaminated by droplets
and tiny airborne particles containing the virus. This largest pandemic
affects more than 500 million people around the world and causes 62.7 lakhs of
people deaths. In the starting year of COVID-19, RT-PCR (Reverse Transcription
Polymerase Chain Reaction) laboratory test is used to identify the disease
[21]. This test collects swab specimens from the human nose or throat where
the virus stayed. It takes 4 to 6 hours to generate results and has low
sensitivity [17]. Apart from this, the testing kit has a severe problem and
low efficiency. For all these issues, it uses the radiology method to identify
the clear findings of the COVID-19 disease.

Medical experts and organizations suggest that radiology imaging shows
apparent lung abnormalities and lung involvement findings. Radiology imaging-
based imaging test methods are X-ray, and CT defines human chest involvement
[8]. Among the two, X-ray has some limitations due to less sensitivity and
does not show any abnormalities in the earlier stage. Due to these problems,
CT(Computed Tomography) is the best diagnosis method of COVID-19, showing the
evident lung abnormalities in the earlier step. With the help of
radiographers, computer researchers are involved in the study of CT images
implemented in deep learning models and detect lung infections efficiently.

This paper presents a deep learning-based diagnosis system for the COVID-19
virus infection assessment based on lung CT images. COVID-19 infected and
normal chest CT images are collected from various public sources consisting of
14320 images. Two deep learning-based models, CNN and VGG16 are implemented to
compare their performance. These models efficiently identified a COVID-19
disease and achieved an accuracy of 96.34% and 97.40% for CNN and VGG16,
respectively.

The remaining sections of the paper is organized as follows: Section 2
describes the related work, followed by the dataset details in Section 3.
Section 4 presents the methodology and Section 5 summarizes the results.
Finally, conclusions are given in Section 6.

##  2 Related Work

In medical imaging, many studies related to COVID-19 identification use CT
images with the help of deep learning models. Models are separated by
classification, and segmentation is discussed here in a related study of
COVID. In [18], Residual Network (ResNet) built with various layers, looks
like a pyramidal structure to classify the CT images with an accuracy of
86.70%. Within the model of DenseNet121, they are using 757 CT image to
accurately extract the infected lung areas with an accuracy result of 84.07%
[11].

To classify the high dimensional convolutional features with less time using
the AlexNet model processed 7500 CT images also found other diseases like lung
tumors to predict the disease with an accuracy of 98.25% [20]. EfficientNet
[14] uniformly scales all the dimensions using compound co-efficient with
87.68% using more than 3000 CT images. In DCNN [16], using 746 CT image
reduces the dimensional size and computational cost. U-net [6] specifies the
encoding and decoding transformations performed are also recommended for
medical imaging segmentation to detect COVID. In this study, a combined
dataset with a large number of samples is used to validate the performance of
two deep learning models.

##  3 Dataset

Several public COVID-19 CT image datasets are available in various data
repositories like Kaggle, Git-Hub, and IEEEDataPort. Using this database, we
collected 14320 chest CT images split into two classes as covid and normal,
namely as All COVID Dataset - Splitted. This section describes the five
datasets combined in this work are shown in Table 1.

###  3.1 COVID-19 Lung CT Scans

COVID-19 Lung CT Scans:This dataset consists of 746 chest CT images split into
COVID and normal. Among these 349 COVID CT images from 216 studies and 347
normal CT images. Covid CT image resolution is 819x460 in size from China and
is described in [19].

###  3.2 SARS-COV-2 CT-Scan Dataset

This dataset [15] contains 2481 chest CT images of 1252 COVID CT images and
1229 normal with 819x460 dimensions taken from the Public Hospital of
Government Employees of Sao Paulo, Brazil.

###  3.3 COVID 19 X-ray and CT Scan Image Dataset

This public dataset [3] contains 8055 CT images of 5427 COVID, and 2628 normal
resolution is 512x512 in PNG format.

###  3.4 COVID-19DATASET

This dataset [2]consists of a total of 1964 CT images. There are 982 images
with COVID infections and the remaining 982 images from normal patients. The
standard resolution of CT image size of 256x256 in png format.

###  3.5 COVID-19 CT image

This dataset [5] consists of 1051 CT images in total. There are 501 images
with COVID symptoms and the remaining 550 images from normal patients. The
resolution of CT images in this dataset is 512x512 pixels in PNG format.

The final combined dataset used in this study consists of 5,785 normal and
8,535 COVID-19 infected chest CT images. This final dataset was used to
validate the performance of the CNN and VGG16 models.

Table 1: List of COVID-19 chest CT datasets Dataset name | COVID | Normal | Total | Image Resolution
---|---|---|---|---
COVID-CT-Dataset [19] | 349 | 463 | 746 | 819 x 460
SARS-COV-2 CT-Scan Dataset [15] | 1252 | 1229 | 2481 | 819 x 460
COVID 19 X-ray and CT Scan Dataset [3] | 5427 | 2628 | 8055 | 512 x 512
COVID-19DATASET [2] | 982 | 982 | 1964 | 256 x 256
COVID-19 CT image [5] | 501 | 550 | 1051 | 512 x 512
Combined dataset | 8535 | 5785 | 14320 | 256 x 256

##  4 Methodology

In this study, the proposed methodology defines automated COVID-19 detection
based on lung CT images using CNN and VGG16 deep learning models. We used a
large-scale combined split dataset to train the models for classifying whether
the CT images are COVID or normal.

###  4.1 Pre-processing

Creating a medical image is diagnosed with the interior parts of the human
body. One of the test methods is the CT imaging modality, which is used to
detect COVID-19. There is a need to preprocess the CT images due to low
contrast, intensity, and clarity to prevent a false diagnosis of the COVID-19
disease [12]. The reason to preprocess the image is to remove irrelevant data
and increase the accuracy level performed by image resizing, segmentation, and
enhancement.

###  4.2 Image Augmentation

The image augmentation techniques are performed for image detection purposes
in this dataset. Expand more features from All COVID Dataset images from
different dimensions. It generally performs zoom, horizontal flip, shear, and
shifting. It is mainly applied to the training dataset images; either position
augmented where the pixel position of the image changes or color augmented,
where the color of an image is modified by changing its pixel values. Color
augmentation can be performed by varying an image’s brightness, contrast, and
saturation. From the above, all operations are performed to combined the
images on an Combined Dataset of covid and normal. Finally, this dataset can
be uploaded to Kaggle public data repository to evaluate the methodology.
Using CT dataset was applied in CNN and VGG16 models to generate better
accuracy.

###  4.3 Convolutional Neural Network (CNN) architecture

A CNN has more convolution layers and handles large volumes of image data. The
main work of the model is COVID CT image taken as input and assigned weights
and biases for various aspects of an image and able to differentiate one
another from finally extracted features from the CT image. CNN architecture
has a stack of layers, namely the convolutional layer, Pooling layer, ReLU
layer, and fully connected layer [12]. We proposed this model for using CT
images to classify its normal or COVID, and it mainly satisfied the
clinician’s expectations. The CNN architecture is shown in Figure 1.

Figure 1: CNN Architecture for COVID-19 Classification

In this CNN architecture, the input is a CT image size of 256 x 256 fed into
the convolutional and pooling layers network, which performs feature
extraction. The fully connected layer acts as a classifier that performs non-
linear transformations of feature extraction [10]. The CT image is an input,
and it will predict whether it’s a COVID or a normal CT image as an output.
The layers are designed to filter size is 32, and the padding is the same with
a kernel size of 3 and the activation function as ReLU. Here, the First max-
pooling layer has a pool size of 2. The flat layers convert all the pooled
layers into a single column. In the end, ReLU and softmax are the two dense
layers formed to perform the classification as COVID or normal [9].

###  4.4 VGG 16 Architecture

The VGG16 architecture was first developed in 2014 [4]. It has 16 layers where
13 convolutional layers are stacked with 3x3 filters and 2x2 max-pooling
layers. A network contains three fully connected layers, and classification is
performed using the softmax function [13], and the architecture is followed by
the arrangement of convolutional and pooling layers. Finally, there are two
fully connected layers, a softmax for output. VGG16 needed to be pre-trained
on a large labeled image dataset. It can notably reduce training time and
computational load.

Figure 2: VGG16 architecture for COVID-19 classification. (Image source:
www.learndatasci.com)

The architecture of VGG16 structure is shown in Figure 2. The convolutional
layer receives the input image of 224 × 224 and then propagates through a set
of convolutional layers with a receptive field of 3 × 3. These are followed by
a down sampling process and have five max-pooling layers through a set of
specified convolutional layers and two fully connected layers with a fixed
channel size [1]. Every neuron is the fully connected layer that accepts the
activations input from the preceding layer neuron. In VGG16, the average
pooling layer AveragePooling2D layer, dropout layer, and fully connected layer
are added [7]. Finally, an FC layer created a new set of classifications,
namely COVID-19 and normal.

##  5 Experiments and Results

###  5.1 Implementation

Performance metrics like Accuracy, AUC, F1-Score, Precision, and Recall are
used to perform an experimental evaluation model. It is more helpful for
evaluating the COVID-19 predictions. The formulae that calculate the
performance metrics are shown below

$Accuracy=\frac{TP+TN}{TP+FP+TN+FN}$

We developed a public dataset, namely the All COVID Dataset, consisting of
14320 images grouped into 8535 COVID CT images and 5785 normal CT images. A
whole dataset can be split into train, test, and validate to run the python
programming code. This experimentation is done using the Kaggle data science
environment freely. Both models are implemented using the python Keras library
with TensorFlow as the backend. The models are evaluated four times with data
split like 60% training, 20% testing, and 20% validating with different
epochs. The evaluated model behaviors are observed by accuracy.

###  5.2 Results

The experimental results of CNN and VGG 16 are shown in Figure 3, Figure 4
more helpful for automated COVID-19 identification on CT images. This study
comprises 60% training images and 20% of testing, and 20% validating images
and setting the epoch value of 25 and 50 for CNN and VGG 16 models.
Classifying the CT images for COVID identification achieved a prediction
accuracy of 96.34%, 96.99% are shown in Table 2. The benefit of the proposed
work is time requirements for CT image generation, and then models are trained
and predicted, reducing the testing time more helpful to COVID-19 patients.

Table 2: Comparison of accuracy of CNN and VGG16 models. Model |  Dataset Size |  Accuracy
---|---|---
CNN |  14,320 (5,785 normal and 8,535 COVID-19) |  96.34%
VGG16 |  14,320 (5,785 normal and 8,535 COVID-19) |  96.99%

##  6 Conclusion

Faster diagnosis of COVID-19 is the primary challenge for physicians today.
COVID-19 patients can be quickly identified, which may contribute to
addressing the current pandemic situations. For this purpose, we developed two
deep learning models for identifying COVID-19 disease using chest CT images.
We collected 14320 original COVID-19 CT images from various public databases
and validated the performance of two deep learning models namely CNN and
VGG16. We use these models to classify the CT image as COVID or normal. The
accuracy of CNN and VGG16 models are 96.34% and 97.40%, respectively.

## References

  * [1] Arora, V., Ng, E.Y.K., Leekha, R.S., Darshan, M., Singh, A.: Transfer learning-based approach for detecting covid-19 ailment in lung ct scan. Computers in biology and medicine 135, 104575 (2021)
  * [2] Chetoui, M., Akhloufi, M.A.: Automated detection of covid-19 cases using recent deep convolutional neural networks and ct images. In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). pp. 3297–3300. IEEE (2021)
  * [3] COVID, M.D.A.: X-ray images dataset (19)
  * [4] Dansana, D., Kumar, R., Bhattacharjee, A., Hemanth, D.J., Gupta, D., Khanna, A., Castillo, O.: Early diagnosis of covid-19-affected patients based on x-ray and computed tomography images using deep learning algorithm. Soft Computing pp. 1–9 (2020)
  * [5] Elmuogy, S., Hikal, N.A., Hassan, E.: An efficient technique for ct scan images classification of covid-19. Journal of Intelligent & Fuzzy Systems 40(3), 5225–5238 (2021)
  * [6] Gozes, O., Frid-Adar, M., Sagie, N., Zhang, H., Ji, W., Greenspan, H.: Coronavirus detection and analysis on chest ct with deep learning. arXiv preprint arXiv:2004.02640 (2020)
  * [7] Halder, A., Datta, B.: Covid-19 detection from lung ct-scan images using transfer learning approach. Machine Learning: Science and Technology 2(4), 045013 (2021)
  * [8] Horry, M.J., Chakraborty, S., Paul, M., Ulhaq, A., Pradhan, B., Saha, M., Shukla, N.: Covid-19 detection through transfer learning using multimodal imaging data. Ieee Access 8, 149808–149824 (2020)
  * [9] Kogilavani, S., Prabhu, J., Sandhiya, R., Kumar, M.S., Subramaniam, U., Karthick, A., Muhibbullah, M., Imam, S.B.S.: Covid-19 detection based on lung ct scan using deep learning techniques. Computational and Mathematical Methods in Medicine 2022 (2022)
  * [10] Lokwani, R., Gaikwad, A., Kulkarni, V., Pant, A., Kharat, A.: Automated detection of covid-19 from ct scans using convolutional neural networks. arXiv preprint arXiv:2006.13212 (2020)
  * [11] Mishra, A.K., Das, S.K., Roy, P., Bandyopadhyay, S.: Identifying covid19 from chest ct images: a deep convolutional neural networks based approach. Journal of Healthcare Engineering 2020 (2020)
  * [12] Polsinelli, M., Cinque, L., Placidi, G.: A light cnn for detecting covid-19 from ct scans of the chest. Pattern recognition letters 140, 95–100 (2020)
  * [13] Shah, V., Keniya, R., Shridharani, A., Punjabi, M., Shah, J., Mehendale, N.: Diagnosis of covid-19 using ct scan images and deep learning techniques. Emergency radiology 28(3), 497–505 (2021)
  * [14] Silva, P., Luz, E., Silva, G., Moreira, G., Silva, R., Lucio, D., Menotti, D.: Covid-19 detection in ct images with deep learning: A voting-based scheme and cross-datasets analysis. Informatics in medicine unlocked 20, 100427 (2020)
  * [15] Soares, E., Angelov, P., Biaso, S., Froes, M.H., Abe, D.K.: Sars-cov-2 ct-scan dataset: A large dataset of real patients ct scans for sars-cov-2 identification. MedRxiv (2020)
  * [16] Wang, S., Kang, B., Ma, J., Zeng, X., Xiao, M., Guo, J., Cai, M., Yang, J., Li, Y., Meng, X., et al.: A deep learning algorithm using ct images to screen for corona virus disease (covid-19). European radiology 31(8), 6096–6104 (2021)
  * [17] Wang, W., Xu, Y., Gao, R., Lu, R., Han, K., Wu, G., Tan, W.: Detection of sars-cov-2 in different types of clinical specimens. Jama 323(18), 1843–1844 (2020)
  * [18] Xu, X., Jiang, X., Ma, C., Du, P., Li, X., Lv, S., Yu, L., Ni, Q., Chen, Y., Su, J., et al.: A deep learning system to screen novel coronavirus disease 2019 pneumonia. Engineering 6(10), 1122–1129 (2020)
  * [19] Yang, X., He, X., Zhao, J., Zhang, Y., Zhang, S., Xie, P.: Covid-ct-dataset: a ct scan dataset about covid-19. arXiv preprint arXiv:2003.13865 (2020)
  * [20] Zhou, T., Lu, H., Yang, Z., Qiu, S., Huo, B., Dong, Y.: The ensemble deep learning model for novel covid-19 on ct images. Applied soft computing 98, 106885 (2021)
  * [21] Zhu, N., Zhang, D., Wang, W., Li, X., Yang, B., Song, J., Zhao, X., Huang, B., Shi, W., Lu, R., et al.: A novel coronavirus from patients with pneumonia in china, 2019. New England journal of medicine (2020)