#  toon2real: Translating Cartoon Images to Realistic Images

K. M. Arefeen Sultan1, Mohammad Imrul Jubair2,
MD. Nahidul Islam3, Sayed Hossain Khan4  Department of Computer Science and
Engineering,
Ahsanullah University of Science and Technology, Bangladesh
{1krsultan069, 3nahidul19967<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In terms of Image-to-image translation, Generative Adversarial Networks (GANs)
has achieved great success even when it is used in the unsupervised dataset.
In this work, we aim to translate cartoon images to photo-realistic images
using GAN. We apply several state-of-the-art models to perform this task;
however, they fail to perform good quality translations. We observe that the
shallow difference between these two domains causes this issue. Based on this
idea, we propose a method based on CycleGAN model for image translation from
cartoon domain to photo-realistic domain. To make our model efficient, we
implemented Spectral Normalization which added stability in our model. We
demonstrate our experimental results and show that our proposed model has
achieved the lowest Fréchet Inception Distance score and better results
compared to another state-of-the-art technique, UNIT.

###### Index Terms:

GANs, Image-to-image-translation, Cartoon-to-real,

##  I Introduction

Cartoons occupy a huge part in our entertainment sector. Film industries, in
recent days, are remaking movies from the popular past cartoons and presenting
them for current generation. Such an example is – the The Lion King (2019)
from The Lion king (1994). Therefore, we realize the necessity of recreating
realistic images from the cartoons which can contribute to photo-realistic
rendering in computer graphics as well as in film industries. In this paper,
we propose an approach which converts images from cartoons into their
corresponding photo-realistic images. From Figure 1, we can see an outcome of
our work where the cartoon scene is translated into a photo-realistic one.

Image-to-image translation using Generative Adversarial Network (GAN)[1] has
been one of the most desiring fields of deep learning research lately. In a
GAN architecture, a discriminator network tries to measure the probability of
whether an image has come from an authentic data source or a fake generated
source of the generator. On the other hand, the generator tries to maximize
the probability of the discriminator’s making mistake and while doing that, it
learns to generate more accurate data as much as close to real data.
Tremendous success of GANs [1], led other researchers to work on unsupervised
settings of datasets, such as [2], [3]. Although these models have succeeded
to translate one domain of image to another in general, there hasn’t been any
specific research for generating images of the photo-realistic domain from
cartoon domain. This is an extremely hard task; the reason is the domain gap
between these two distributions is too shallow.

(a) Cartoon scene

(b) Our result

Figure 1: An example of cartoon to real world translation. (a) Input image:
which is from the animated film ”The Wind Rises”. (b) Our result: transforming
the cartoon image (a) to real world image.

As a result, the discriminator can be easily erroneous to determine the
generated data as real ones. This is the reason why most state-of-the-art
models tend to fail in case of generating cartoon to real images. We
illustrate in our result section that several models intent to keep the
original content of cartoon domain while generating photo-realistic images.

To satisfy our objective on this task, we have taken an approach built on
CycleGAN[2]. We implemented spectral normalization technique[4] which helps
our model to converge faster. Our approach also keeps the content of photo-
realistic domain. In addition to these, we also created our own dataset for
our model. We show that our method has the lowest FID score than the other
baseline models, and also it tends to show more stabilization in training than
the others.

##  II Related Works

In this section, we review on different relevant variations of GAN and several
past works on image-to-image translations.

GANs have achieved great results in various image generation tasks, which are
image super-resolution[5], image-to-image translation[2, 6, 7], text-to-image
synthesis[8, 9] etc. Recently, GAN[1] based approach has given tremendous
results in image-to-image translation tasks. Zhu et al.[2] proposed a cycle
consistency loss to reduce the infinite mappings of input images to any
distribution in the target domain. Adversarial loss alone can’t solve the
random permutation mappings of target distribution, rather it helps the input
image to be translated into target domain. Similar to CycleGAN, Kim et al. [3]
proposed a method for preserving the key attributes between the input and the
transformed image while maintaining a cycle consistency criterion. Similarly,
Yi et al. [10] proposed dual-GAN mechanism based on dual learning from natural
language translation[11]. In UNIT[7] framework, Liu et al. proposed a shared-
latent space assumption, which denotes that the pair of corresponding images
in different domains can be mapped to a same latent representation in a
shared-latent space. Liu et al. used the combination of generative adversarial
network (GAN), based on CoGAN[12] and variational autoencoders (VAEs)[13, 14,
15].
For stabilizing and improving the training of GAN, several works were proposed
such as adding weight normalization and regularization techniques [16, 4],
designing new generative architectures[17, 18] to improve visual results and
modifying learning objectives [19, 20]. Miyato et al.[4] first proposed
spectral normalization technique which constrains the Lipschitz constant of
the discriminator network by limiting the spectral norm of each layer. The
authors argue that spectral normalization can improve the quality of training
GANs better than weight normalization[21] and gradient penalty[16]. In this
paper, we have utilized this property of spectral norm to improve the training
of our GAN.
In our previous work Cartoon-to-real [22], CycleGAN based model was
implemented to transform cartoon images into photo-realistic domain. While all
these methods achieve compelling results, they take too much time for
training. The reason behind it is that these models are not stabilized during
training. In the next section, we discuss the approach we took to solve these
fundamental issues and to obtain better outcomes.

##  III Formulation

Our main objective is to transform cartoon images to photo-realistic images by
learning the mapping of a cartoon domain $C$ to the photo-realistic domain
$R$. In this section we discuss about the approaces we have taken to
accomplish the task.

(a) Generator Network

(b) Discriminator Network

Figure 2: Architectures of the generator and discriminator of our proposed
model, in which k is the kernel size and s is the stride in each convolutional
layer.

(a) Input

(b) UNIT

(c) Cartoon-to-real

(d) toon2real

Figure 3: Detailed comparisons in terms of contrast and content preservation.
(a) Input images of cartoon scenes. (b) Result of UNIT[7] which shows the
lacking of content than (c) and (d). (c) Result of our previous work(Cartoon-
to-real)[22] which closely preserves the texture of input domain rather than
translating into realistic one. (d) Result of our toon2real shows more
contrast on content, compared to other works. The images were selected
randomly.

Dataset Collection: Due to the lack of paired data between cartoon domain and
photo-realistic domain, we took an approach to collect unpaired dataset for
both domains. As deep learning is data hungry, initially, for realistic
domain, we scraped scenery images from Flickr and many other sources which
were tagged as scenery, sunrise, sunset, sea, sky & beach and collected around
$5000$ samples. Besides, for cartoon domain, we extracted images from various
Japanese anime movies and collected about $5000$ images. We extracted the
scenery images from these movies consisting of sunsets, sea, sky, trees etc.
We excluded the frames which are darker to see, and the first and last few
frames—as the introductory and credits part tend to be textual in a movie.
After hand-picking the appropriate images, in order to approximate with the
size of the realistic domain, we collected images from more than $15$ cartoon
movies and clips, consisting of genres romance, spiritual, war, supernatural &
science-fiction. For both the domain, images were of $128\times 128$
dimension. For the validation set, we took about $2500$ images from animation
images and $2000$ from real-world photos.

Adversarial Loss: Although in [1], a binary cross-entropy based Adversarial
Loss function was proposed, we use a Least Squares Loss(LSGAN) function for
our training. According to Mao et. al [23], we have explored that LSGAN
performs better in the case of vanishing gradient problem and thus shows more
stability during training and produces much higher quality images in the case
of Image-to-image Translation. So, our adversarial loss stands as follows -

$For\,Generator\,G_{r},\,\mathcal{L}_{G_{r}}\,=\,\frac{1}{m}\sum^{m}_{i=1}(1-D_{r}(G_{r}(c)))^{2}$
(1)

However, due to the deep similarities between cartoon and photo-realistic
images, we observed that using only a single generator fails to map the
differences between these $two$ domains. To resolve this issue, we use $two$
additional networks in our model, where a generator, $G_{c}$ tries to generate
images of Cartoon domain and a discriminator, $D_{c}$ tries to discriminate
the generated image from cartoon domain. The additional networks also perform
according to the previously mentioned loss function.

Reconstruction Loss: Adding an additional generator solves the issue of
mapping differences; however, it still lacks in content preservation of input
domain. For this reason, we’ve used an additional loss function, by using the
technique of forward and backward loss[3, 2]. The motive of this function is
that an image generated from an input can be reconstructed back to the input
again such that $x=F(G(x))$, where $F$ and $G$ are generators and thus, it is
able to map an image of target domain which is as close as possible to the
image of input domain. In our paper, we call it Reconstruction Loss. The
equation is as follows -

$Forward\,Consistency\,Loss,\mathcal{L}_{f\\_cyc}=\frac{1}{m}\sum^{m}_{i=1}(F_{r}(G_{r}(c))-c)$
(2)
$Backward\,Consistency\,Loss,\mathcal{L}_{b\\_cyc}=\frac{1}{m}\sum^{m}_{i=1}(G_{r}(G_{c}(r))-r)$
(3)

Discriminator Normalization: Training GAN with efficiency is a hard nut to
crack. Prior to previous works, it is known that discriminator tends to make
the training slower and show more inconsistency during training. We used
Spectral Normalization technique, which was first proposed by Miyato et
al.[4], to stabilize our training. The benefit of spectral normalization is
that it doesn’t need extra hyper-parameter tuning. Also, the computational
cost is relatively small compared to other weight normalization techniques.
Miyato et al.[4] found better or same results with image generation tasks by
utilizing this normalization technique. We can see from Figure 4 that, using
this technique stabilized the training, where Figure 4(b) is ours, which shows
a much smoother curve of FID scores than Figure 4(a), which is the FID score-
graph of our previous work(Cartoon-to-real)[22]. Also, we can see that ours
achieved the least FID score within $145$ epoch, whereas our previous
model[22] took more epochs for that.

PatchGAN: As discriminators, we used PatchGAN which was first proposed in
Isola et al. [24]. The intuition of using this discriminator is that it works
best for extracting the high-frequency details of the distribution. Another
beneficial feature is, due to working on $N\times N$ patches, it takes fewer
parameters and thus decreases the computation cost.

##  IV Implementations & Analysis

In this section, we discuss the implementation of our approach followed by
illustrating its results.

Network Structure: For generative networks we implemented the architecture
from Johnson et al. [5] who achieved amazing results for neural style transfer
and super-resolution. The network includes two stride-2 convolutions, $6$
residual blocks[25], and two fractional strided convolutions with stride
$1/2$. We used instance normalization technique. For the discriminator
network, we used $70\times 70$ PatchGANs[24]. The network architecture of our
proposed model is shown in Fig 2.

Evaluation Metric: We chose the Fréchet Inception Distance (FID) [26] for
quantitative evaluation. As FID score measures the difference between the
generated dataset and the target dataset, it has shown more consistency with
human evaluation. Samples from P and Q are gone through an Inception-v3
network to transform it into feature space. Then, we calculate the
Wasserstein-2 distance between the translated image and the real world images
from an intermediate layer of an Inception-v3 network. The distance can be
calculated as

$\centering
FID=\left\|\mu_{x}-\mu_{y}\right\|_{2}^{2}+Tr(\Sigma_{x}+\Sigma_{y}-2(\Sigma_{x}\Sigma_{y})^{\frac{1}{2}})\@add@centering$
(4)

where $(\mu_{x},\Sigma_{x})$, and $(\mu_{x},\Sigma_{y})$ are the mean and
covariance of the feature space of samples from P and Q. Lower the FID score,
the closer the distance between translated image and real domain images. As
our task is image-to-image translation where we want our output to have the
content of input cartoon images and the style of real-world images, we
calculated a weighted average between them, where we used $80\%$ weight for
target data and $20\%$ weight for input data. From Table I we can see that our
work has shown the least FID score compared to other state of the art model,
i.e UNIT.

Methods | FID Score
---|---
Our work | 39.2574
Cartoon-to-real | 45.2566
UNIT | 55.9214

TABLE I: FID score of our proposed model, in comparison with our previous work
(Cartoon-to-real)[22] and UNIT model[7].

(a) Cartoon-to-real

(b) Our proposed method

Figure 4: Here, FID scores for our previous work (Cartoon-to-real) (a) and for
our method (b) are shown from $20$ epochs up to $200$ epochs.

Evaluation of Discriminator Normalization: By utilizing spectral normalization
technique on discriminator network shown in Figure 4(b), we started to gain a
lower FID score from the very initial of training compared to our previous
model. Spectral normalization is used on discriminator network which is shown
in 4(b). From 4(b), the quality of transforming images doesn’t improve
monotonically during training. For example, the FID score of our work starts
to drop at the $37$th epoch. On the contrary, previous model’s FID score
starts to rise after $125$th epoch and it crosses the initial FID scores,
whereas, in our work, the scores didn’t rise as the previous model did. From
this, we can clarify that we achieved a more stabilized model and better
scores. We can also clarify from Figure 4 that the stabilization technique
also takes fewer training epochs to achieve better scores.

Comparison with state of the art models: We compared our work with state of
the art technique, i.e UNIT [7] and our previous work(Cartoon-to-real)[22]. In
Figure 3 we show a close-up view of an example, explaining that our work
preserves much better vibrance and content preservation, whereas UNIT’s[7]
output shows a lacking of preserving content and information in image. Also
our previous work[22] closely shows similar characteristics of input domain
rather than realistic one.

(a) Cartoon scene

(b) Our result

Figure 5:  A failure case of our method. In the output image (b), the cat
remains cartoonish as in input (a) and is not translated into a realistic one.

Limitations: Despite achieving better FID score of all, it is still too high
to be a perfect image translation score. In fact, we can see from Figure 5
that, the output fails to achieve the meaningful (semantically and
geometrically) structure of real-world objects—in this example a cat. This
problem is also common in UNIT[7] and other image-to-image translation models.

##  V Conclusion

In this paper, we showed a GAN based approach to translate images from cartoon
domain to photo-realistic domain. We implemented our model based on
CycleGAN[2], where we used Reconstruction Loss for content preservation of
input image and the PatchGAN for better texture extraction. By implementing
spectral normalization technique on discriminator network, we showed that our
model achieves better training stability and the lowest FID score of all the
other models. Our future plan is to lessen our current limitations by
investigating more geometry and content aware model to improve the texture so
that the gap with the photo-realistic domain decreases. In addition to FID
score, we have plans to arrange human-involved and perceptual evaluation
processes to assess the correctness of our outcomes.

## References

  * [1] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , 2014, pp. 2672–2680. [Online]. Available: http://papers.nips.cc/paper/5423-generative-adversarial-nets
  * [2] J. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_ , 2017, pp. 2242–2251. [Online]. Available: https://doi.org/10.1109/ICCV.2017.244
  * [3] T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim, “Learning to discover cross-domain relations with generative adversarial networks,” in _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , 2017, pp. 1857–1865. [Online]. Available: http://proceedings.mlr.press/v70/kim17a.html
  * [4] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, “Spectral normalization for generative adversarial networks,” _CoRR_ , vol. abs/1802.05957, 2018. [Online]. Available: http://arxiv.org/abs/1802.05957
  * [5] J. Johnson, A. Alahi, and F. Li, “Perceptual losses for real-time style transfer and super-resolution,” _CoRR_ , vol. abs/1603.08155, 2016. [Online]. Available: http://arxiv.org/abs/1603.08155
  * [6] X. Yu, X. Cai, Z. Ying, T. H. Li, and G. Li, “Singlegan: Image-to-image translation by a single-generator network using multiple generative adversarial learning,” _CoRR_ , vol. abs/1810.04991, 2018. [Online]. Available: http://arxiv.org/abs/1810.04991
  * [7] M. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” _CoRR_ , vol. abs/1703.00848, 2017. [Online]. Available: http://arxiv.org/abs/1703.00848
  * [8] H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. N. Metaxas, “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” _CoRR_ , vol. abs/1612.03242, 2016. [Online]. Available: http://arxiv.org/abs/1612.03242
  * [9] S. E. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” _CoRR_ , vol. abs/1605.05396, 2016. [Online]. Available: http://arxiv.org/abs/1605.05396
  * [10] Z. Yi, H. R. Zhang, P. Tan, and M. Gong, “Dualgan: Unsupervised dual learning for image-to-image translation.” in _ICCV_ , 2017, pp. 2868–2876.
  * [11] D. He, Y. Xia, T. Qin, L. Wang, N. Yu, T.-Y. Liu, and W.-Y. Ma, “Dual learning for machine translation,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 820–828.
  * [12] M.-Y. Liu and O. Tuzel, “Coupled generative adversarial networks,” in _Advances in neural information processing systems_ , 2016, pp. 469–477.
  * [13] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [14] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther, “Autoencoding beyond pixels using a learned similarity metric,” _arXiv preprint arXiv:1512.09300_ , 2015.
  * [15] D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic backpropagation and variational inference in deep latent gaussian models,” in _International Conference on Machine Learning_ , vol. 2, 2014.
  * [16] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, “Improved training of wasserstein gans,” _CoRR_ , vol. abs/1704.00028, 2017\. [Online]. Available: http://arxiv.org/abs/1704.00028
  * [17] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _CoRR_ , vol. abs/1511.06434, 2015. [Online]. Available: http://arxiv.org/abs/1511.06434
  * [18] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” _arXiv preprint arXiv:1710.10196_ , 2017.
  * [19] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” _CoRR_ , vol. abs/1701.07875, 2017.
  * [20] L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein, “Unrolled generative adversarial networks,” _arXiv preprint arXiv:1611.02163_ , 2016.
  * [21] T. Salimans and D. P. Kingma, “Weight normalization: A simple reparameterization to accelerate training of deep neural networks,” in _Advances in neural information processing systems_ , 2016, pp. 901–909.
  * [22] K. M. A. Sultan, L. K. Rupty, N. I. Pranto, S. K. Shuvo, and M. I. Jubair, “Cartoon-to-real: An approach to translate cartoon to realistic images using GAN,” _CoRR_ , vol. abs/1811.11796, 2018. [Online]. Available: http://arxiv.org/abs/1811.11796
  * [23] X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley, “Least squares generative adversarial networks,” in _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_ , 2017, pp. 2813–2821. [Online]. Available: https://doi.org/10.1109/ICCV.2017.304
  * [24] _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 2017. [Online]. Available: http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=8097368
  * [25] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _CoRR_ , vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385
  * [26] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a nash equilibrium,” _CoRR_ , vol. abs/1706.08500, 2017. [Online]. Available: http://arxiv.org/abs/1706.08500