# Review Neural Networks about Image Transformation Based on IGC Learning
Framework with Annotated Information

Yuanjie Yan, Suorong Yang, Yan Wang, Jian Zhao, Furao Shen Manuscript received
April 15, 2021; revised xx, xx, xxxx; accepted xx, xx, xxxx. Date of
publication xx, xx, xxxx; date of current version April 15, 2021. This work is
supported in part by the National Natural Science Foundation of China under
Grant 61876076. The associate editor coordinating the review of this
manuscript and approving it for publication was Prof. xx. (Corresponding
author: Furao Shen and Jian Zhao.) Y. Yan, H., S. Yang, Y. Wang and F. Shen
are with the National Key Laboratory for Novel Software Technology, Department
of Computer Science and Technology, Nanjing University, Nanjing 210046, China
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>).J. Zhao is with the School of
Electronic Science and Engineering, Nanjing University, Nanjing 210046, China
(e-mail: jianzhao@nju.edu.cn). He is also with the National Key Laboratory for
Novel Software Technology, Nanjing University, China.

###### Abstract

Image transformation, a class of vision and graphics problems whose goal is to
learn the mapping between an input image and an output image, develops rapidly
in the context of deep neural networks. In Computer Vision (CV), many problems
can be regarded as the image transformation task, e.g., semantic segmentation
and style transfer. These works have different topics and motivations, making
the image transformation task flourishing. Some surveys only review the
research on style transfer or image-to-image translation, all of which are
just a branch of image transformation. However, none of the surveys summarize
those works together in a unified framework to our best knowledge. This paper
proposes a novel learning framework including Independent learning, Guided
learning, and Cooperative learning, called the IGC learning framework. The
image transformation we discuss mainly involves the general image-to-image
translation and style transfer about deep neural networks. From the
perspective of this framework, we review those subtasks and give a unified
interpretation of various scenarios. We categorize related subtasks about the
image transformation according to similar development trends. Furthermore,
experiments have been performed to verify the effectiveness of IGC learning.
Finally, new research directions and open problems are discussed for future
research.

###### Index Terms:

Image transformation, image segmentation, image-to-image translation, style
transfer, IGC learning framework

##  I Introduction

Digital image transformation, which has a long history in the field of
computer graphics and computer vision, is to transform the original image into
another form according to the needs. Many mathematical methods, such as image
rendering [3] and image spatial filters [4], are traditional image
transformation algorithms. However, these methods can not handle more complex
and creative tasks. Image transformation using deep neural networks, also
known as image-to-image translation, can complete more sophisticated and
artistic tasks, such as semantic segmentation [1] and style transfer [2].
Semantic segmentation can be regarded as an image transformation task from the
original image to a semantic segmentation mask. The development of semantic
segmentation has an important influence on image transformation with neural
networks. Style transfer is the conversion of the content image to an image
with another style, which is a problem of art and re-creation. These two
typical image transformation scenes are traditionally studied independently in
context of neural networks.

Deep learning has made breakthroughs in many sophisticated scenes, such as
Computer Vision (CV) [3, 4, 5], and Neural Language Process (NLP) [6, 7]. With
the development of deep learning, it is common for a neural network to have
millions or billions of parameters, which makes it more and more difficult to
train. Building large labelled training data sets is a good method to overcome
the notorious over-fitting problem in machine learning. In CV, ImageNet [8]
and MS COCO[9] are constructed and published for image classification [3],
semantic segmentation [4], instance segmentation [10]. In NLP, General
Language Understanding Evaluation (GLUE) [11] also summarizes nine datasets
for different subtasks about neural language. However, it takes a lot of time
and effort to label data manually. In addition, it is impossible to label the
dataset manually for some CV tasks, such as style transfer which is one of the
themes of this article as shown in Fig. 1. There is no clear label for style
transfer, but it is necessary to generate an image which combines content and
style images. Therefore, learning neural network model from unlabelled data or
a small number of labelled samples is the current research trend. On the other
hand, some self-supervised methods of neural networks can utilize the
unlabelled dataset in special tasks, e.g., Generative Adversarial Network
(GAN) [12] and adversarial examples generation [13]. GAN maps the random
vector to a specific image via the discriminator network to judge the true and
false of the generator output. In adversarial examples generation, the
adversarial image is iteratively derived from the target network model about
the original image. In the above two methods, other models are used to label
the unlabelled data, so as to realize the target network training. Besides,
meta-learning [14] and few-shot learning [15] are also proposed to deal with
the problem of fewer labelled samples.

The image-to-image translation is a broad issue in the literature of CV and
graphics where many subtasks have been researched individually. We will not
discuss all the work of image-to-image translation but will reveal the
relationships between different tasks with a unified perspective in this
article. Therefore, we mainly discuss general image-to-image translation tasks
and specifically the subtask about image style. The development of image style
transfer is related to the development of CV. Gatys et al. [16] first
demonstrates the power of Convolution Neural Networks (CNNs) to render a
natural image with different painting styles in Fig. 1. They apply the pre-
trained CNN model to analyse the image content and image style as the
annotated label. The annotation information from the model is different from
that annotated manually. The work of Gatys et al. not only opens up a new
field called style transfer (NST) but also demonstrates a way where a neural
network can learn from another pretrained model without manual labels. Since
then, there have been lots of follow-up studies to improve the NST methods
from various aspects in academia [17] and many successful industrial
applications in industry [18]. In addition, the concept of neural image style
transfer has been extended to the field of NLP. Style transfer in NLP
represents variations of a text without modifying its meaning [19].

In this paper, we aim to provide a new learning framework based on annotated
information. We mainly discuss the image transformation methods using neural
networks, referred to as image transformation for short. Such a learning
framework is called IGC learning, which comprises three learning paradigms,
namely independent learning, guided learning, and cooperative learning. Note
that those learning paradigms are related to other learning paradigms
presented in other works [20, 21], but we provide a unified definition based
on the annotated information of labels. In addition, we review and analyse
some representative works about image transformation from the perspective of
the IGC learning framework. Then, we use the same method to organize image
transformation and style transfer. We show that there is a close relationship
between the development of image translation and style transfer. In order to
verify the effectiveness of each learning paradigm, we perform experiments and
compare related methods on various datasets. What’s more, we also introduce
related applications of image transformation. Finally, we discuss some
problems of image transformation, and extend the IGC learning framework to
other areas. The main contributions of this paper can be summarized as
follows:

  * •

We propose a new learning framework called IGC learning in term of the
annotated information.

  * •

From the perspective of the IGC learning framework, we analyse representative
models and categorize more works in research of image transformation about
neural networks.

  * •

We first demonstrate that the development of image-to-image translation is
highly related to the development of style transfer.

  * •

We discuss open problems of the image transformation and extend the IGC
learning framework for future research.

The rest of the paper is organized as follows. First, we introduce the
background of image transformation and style transfer in Section II. Then, we
propose the novel IGC learning framework in detail in Section III. Section IV
reviews some representative image transformation models from the perspective
of the IGC learning framework. In Section V, we categorize more works in the
research of image transformation and survey their motivations and
contributions. In Section VI, we perform relevant experiments and discuss the
learning methods in IGC. The applications of image transformation are
introduced in Section VII. Moreover, we discuss open problems in image
transformation and extend our proposed IGC learning framework in Section VIII.
We delineate several promising directions for future research. Finally,
Section IX summarizes our themes and contributions in this article.

Figure 1: A example of image style transfer from [16]. A) The original
photograph depicting the Neckarfront in Tübingen, Germany (Photo: Andreas
Praefcke). The painting that provided the style for the respective generated
image is shown in the bottom left corner of each panel. B) The Shipwreck of
the Minotaur by J.M.W. Turner, 1805. C) The Starry Night by Vincent van Gogh,
1889. D) Der Schrei by Edvard Munch, 1893.

##  II Background of Image Transformation

The goal of image transformation is to learn a mapping from an input image to
an output image. In the image semantic segmentation task, the each pixel of
the original image is classified according to whether it belongs to the same
object. Aligned image pairs are required to train the network. With the help
of sufficient aligned data, the methods about semantic segmentation based on
neural networks surpass other traditional algorithms, e.g., Conditional Random
Field (CRF) [22] to achieve state of the art. Semantic segmentation can be
seen as a special task of image transformation on aligned data.

However, the cost of a large-scale aligned image dataset is tremendous.
Inspired by GAN, some works are proposed to solve the lack of aligned images.
To a certain extent, researchers construct a new problem about the image
transformation on two unaligned datasets. As the flexibility of GAN, there are
many studies to follow up, e.g., DualGAN [23] and UNIT [24]. These methods as
variants of GAN can solve the general image transformation problems. Besides,
some interesting research has been derived by combining the image
transformation and objection detection. Researches introduce conditional GAN
(cGAN) [25] models to control or modify partial areas in image. Partial image
transformation (e.g., face swapping [26] and lip swapping [27]) have attracted
great attention in academia and also produced popular applications in
industry. Those methods also work on unaligned datasets.

Figure 2: The general image transformation. The sample cases belong to the
aligned dataset in the first line. The second line contains the unaligned
sample of image transformation. The case in the third line shows the multiple
transformations from one original image.

Furthermore, style transfer, one sub-task of the image transformation,
accomplishes an artistic image-to-image translation to repaint a content image
with a kind of style from another image. This method exploits the pre-trained
neural network model to extract the image feature and style as supervised
information. Consequently, there is no need for tagging data or even datasets.
Only a content image and a style image are needed. It is very different from
the aforementioned methods of image transformation by GAN.

Figure 3: The IGC learning framwork. The figure only shows independent learning, guided learning between a single student model and a teacher model, and cooperative learning between two student models. TABLE I: Datasets about image transformation tasks Tasks | Dataset A | Dataset B | Pattern | Description
---|---|---|---|---
Semantic segmentation | Semantic labels | Photo | Aligned | from Cityscapes dataset [28]
Semantic segmentation: | Architectural labels | Photo | Aligned | from CMP Facades dataset [29]
Colourization | Black and white picture | Colour photos | Aligned | from dataset [8]
Photo generation | Edges | Photo | Aligned | from [30]
Object transfiguration | the class images of object A | the class images of object B | Unaligned | the object class from ImageNet [8]
Season transfer | Winter photos of Yosemite | Summer photos of Yosemite | Unaligned | collected in [31].
Style transfer | the content dataset | the style dataset | Unaligned | from [2]
Style transfer | the content image | the style image | Two images | from [2]
Style transfer | the content dataset | the style image | Dataset & image | from [31]
Style transfer | the content dataset | the style image dataset | Unaligned | from [32]

To summarize the datasets on image transformation, we classify the three
dataset types: aligned between two datasets, unaligned between two datasets
and only two image examples without datasets. Different types of datasets are
built for different granularities of image transformation tasks. The aligned
dataset is for image transformation tasks with pixel-level granularity. The
unaligned dataset is aimed at target-level granular image transformation
tasks. Only two image examples without datasets are often referred to as style
transfer, which relates to perception by a human from the image-level
granularity. In Table LABEL:dataset, we summarize some datasets about image
transformation tasks.

Currently, most works focus on the unaligned datasets. There are two main
advantages. First, compared with aligned datasets, unaligned datasets are
easier to create. Second, in the context of unaligned datasets, the ability of
image transformation methods is flexible and its application is wider. We
define these works as the general image transformation. For style transfer,
due to the discrepancy among the above methods, it has also been studied
individually. Consequently, we mainly focus on general image transformation
and style transfer in this paper.

###  II-A General Image Transformation

General image transformation contains various image-to-image translations on
aligned and unaligned datasets as shown in Fig. 2. General image
transformation mostly focuses on the objects in the image. Therefore, it is
crucial to identify and locate objects. Image classification and object
detection as the basic problems in CV have been researched widely. Some
popular neural networks have been proposed like ResNet [3] and VggNet [33].
The architecture of those network has been adopted and applied in a wide range
of image problems. Follow those works, most of image transformation methods
use similar networks to build transformation models. At the beginning of the
research on image-to-image translation, researchers have directly applied the
semantic segmentation models to aligned datasets. Then, the general image
transformation on the unaligned dataset has been proposed, which has attracted
widespread attentions. In this paper, we discuss the general image
transformation on aligned and unaligned datasets. The solutions on the two
types of datasets have similar network models but there are discrepancies in
learning methods.

###  II-B Style Transfer

The image style comes from subjective feelings by humans and has no clear
definition. The image style is related to color, texture, lighting, etc. The
traditional method mainly adopts spatial filter, which achieves various styles
by defining transformations on RGB channels, e.g., comic strip filter [34] and
nostalgia filter [35]. Since most image styles can not be expressed
explicitly, we describe the style using a style image.

The work of Gatys et al. utilizes a pretrained convolution neural network to
extract the style of images and achieve fusion images like Fig. 1. Since then,
many works about style transfer have been proposed from different
perspectives. Survey [17] categorizes those works according to the image-
optimization-based online neural method and model-optimization-based offline
neural method. The image-optimization-based online neural method means that it
generates the target image iteratively with the original image and the style
image. The model-optimization-based offline neural method means that it uses
the transform neural network, which has been trained for a certain style
before, to generate the target image directly from the original image.
However, it did not reveal the relationship between the two methods. In this
survey, we try to adopt a unified architecture to demonstrate their
relationship.

##  III IGC Learning Framework

With the development of hardware, we can solve more complex problems by
designing larger neural network models which contains millions or billions
parameters. To avoid the overfitting problem, a large training dataset is
needed in the corresponding large model. However, a large amount of data from
the internet is unlabelled. Labelling data manually is tedious in time and
cost, even impossible in some tasks like the image style transfer.
Fortunately, some neural network models can use other models to annotate data.
Inspired by their works, we summary and propose a new learning framework
called IGC learning, which comprises three learning paradigms, namely
Independent learning, Guided learning, and Cooperative learning. Single or
multiple models can be included under the IGC learning framework. According to
whether the model can be learned, it is divided into student model and teacher
model, which is consistent with the relevant description in knowledge
distilling [36].

We define the following mathematical symbols about the model in the IGC
learning framework.

  * •

Student model: we denote $S^{i}_{w}$ as the $i$-th student model which is
needed to learn parameters $w$.

  * •

Teacher model: we denote $T^{i}$ as the $i$-th teacher model which does not
need to learn the parameters.

For specific problems, we can construct any learning model and decompose it
into several student models and teacher models in the perspective of the IGC
learning framework. Then, depending on how the data is labelled, we define
three learning paradigms about the IGC learning.

  * •

Independent learning is the learning process of the student model based on
data.

  * •

Cooperative learning is the learning process among some student models.

  * •

Guided learning is the learning process between the student models and teacher
models.

Fig.3 shows the IGC learning framework with two student models and two teacher
models. We believe that the IGC learning framework is more suitable for neural
network models than the mainstream supervised and unsupervised learning
frameworks. First, multiple models are often involved in a complex image task
and the IGC learning framework is capable of analysing their relationships.
Second, the IGC learning framework guides us to build more suitable models by
introducing new models and learning methods. Note that IGC framework is mainly
proposed for neural network models, but it is not limited to it. We can also
apply this framework to the analysis of the combination of traditional models
and neural network models [37]. Furthermore, since the three learning
paradigms are independent from each other, the student model can perform three
learning methods simultaneously. Each learning method will improve the
performance of the student model to a certain extent. We discuss more details
of the learning paradigm about the IGC and give some examples of related
models.

###  III-A Independent Learning

Independent learning is the mainstream way in deep neural networks. When there
is only one model, we denote it as $S_{w}$. For most neural networks, the
student model $S_{w}$ is trained on the labelled dataset. Multimodal learning
[38] involving a variety of networks is to improve the performance of models
via the cooperation on some related models. Joint representations [39] is a
typical multi-model method. As shown in Fig. 4, the model can divide some
student models that denote as $S^{1}_{w},S^{2}_{w},\cdots,S^{n}_{w}$, where
$n$ is the number of networks. Multimodal learning just using the manually
labelled data belongs to independent learning. Self-supervised learning
methods that are different from supervised learning, e.g., Autoencoder[40] and
BERT[41], also belong to independent learning in the context of the dataset.
The annotations of data are not from other models but from the data itself.
Therefore, independent learning is not just a subset of supervised learning.

Figure 4: The framework of joint representations in multimodal learning

###  III-B Cooperative Learning

Cooperative learning is the learning process among some student models. The
purpose of cooperative learning is to share knowledge or confront each other
among student models to improve their ability. GAN is most relevant to the
cooperation learning we proposed. The typical GAN model has a generator model
and a discriminator model, which can be seen as student model $S^{1}_{w}$ and
$S^{2}_{w}$, respectively. In the image generation task, the $S^{1}_{w}$ model
maps random vectors to real images, while the $S^{2}_{w}$ model classifies
images as real images or generated images. The original adversarial loss and
its variants [42] are proposed to train the models $S^{1}_{w}$ and
$S^{2}_{w}$. The Conditional Generative Adversarial Networks [25] replace
random vectors with the the limited input to control the generated output as
the student model $S^{1}_{w}$. Most image transformations adopt the
architecture of conditional GAN to accomplish the reconstruction from the
source image to the target image. Besides GAN, Dual Learning [43] where two
models learn their parameters through duality tasks is also a typical
cooperation learning method. The way of cooperative learning is to accomplish
the spreading of data and loss among models. As shown in Fig. 5, for
$S^{1}_{w}$, it communicates with $S^{2}_{w}$ to learn its parameters through
data spreading and label spreading as follows,

  * •

Data spreading means that the outputs of $S^{1}_{w}$ are the inputs of the
$S^{2}_{w}$.

  * •

Label spreading means that the outputs of $S^{2}_{w}$ are the labels as the
outputs of $S^{1}_{w}$.

The learning of $S^{2}_{w}$ is not related to any other models. We call
$S^{1}_{w}$ as the main student and $S^{2}_{w}$ as the associate student. For
GAN, the generator is the main student and the discriminator is the associate
student. In the next Section, we analyse some models that make full use of
these two cooperative methods to strengthen the learning among students.

Figure 5:  Data spreading and label spreading between $S^{1}_{w}$ and
$S^{2}_{w}$.

###  III-C Guided Learning

Guided learning is rather related to independent learning. Teacher model $T$
can come from the student model $S$ after independent learning, but it is not
limited to this. The paper [37] uses the traditional clustering model as the
teacher model. Moreover, the teacher model $T$ is also called the pre-trained
model in the fine-tuning method [44]. However, unlike the fine-tuning method,
we try to learn knowledge from the pre-trained model instead of adjusting it.
Knowledge distillation [45], an effective way of neural network compression,
is a typical guided learning. The task of knowledge distillation is to train a
small network from the original big network but maintain the accuracy of the
original model. Another important research of guided learning is feature
visualization [46]. The typical feature visualization method is to iteratively
modify the random noise using the image features extracted from the teacher
model $T$, which can be seen as the source of the image style transfer. Guided
learning is also similar to the cooperative learning. Student models learn
from teacher models through data spreading and label spreading. Teacher model
can be seen as an excellent student model without learning.

##  IV Review Representative image transformation Methods based on the IGC
Learning Framework

Our proposed IGC learning framework is based on the annotation from datasets
or models. This framework does not focus on the details of models, but on the
learning methods among different models. From the perspective of the IGC
learning framework, we analyse some representative works on image
transformation.

###  IV-A Independent Learning of image transformation

Independent learning works on the aligned datasets. Much work of the image
transformation on the aligned dataset is related to semantic segmentation.
Various aligned datasets have been established to solve practical problems in
Table LABEL:dataset. A large number of models have been proposed, e.g., Fully
Convolutional Networks (FCN) [47] and U-Net [48]. They have a similar
structure shown in Fig. 6. FCN adopts a basic autoencoder to map images to
semantic masks through the downsampling to the upsampling. U-Net also provides
an easy but efficient method to locate the pixels of the mask by adding some
skip connections from feature extraction to reconstruction. Based on the IGC
learning framework, the student model denotes as $S_{w}$ to learn the
parameters $w$ from the labelled dataset. $S_{w}$ is trained using a per-pixel
loss between the output and ground-truth image. A per-pixel loss is defined as
follows,

$\mathcal{L}_{pixel}(S_{w};x,y)=(S_{w}(x)-y)^{2}$ (1)

where $x$ means a image and $y$ is the semantic mask of $x$.

Although we do not care much about the details of the model under the IGC
learning framework, a well-structured model is an important research direction
related to semantic segmentation. Various efficient models about independent
learning are proposed in the literature of semantic segmentation in survey
[49]. We only present two methods for semantic segmentation: FCN and U-Net,
which are also adopted by other models that will be introduced later.

Figure 6: A general model of semantic segmentation. The feature extraction can
be seen as an encoder to extract various objects in the image. The
reconstruction is to realize the mapping from object feature to mask. U-Net
improves the accuracy of position for each pixel through skip layer connection
(the dotted line in the figure).

####  IV-A1 FCN

FCN is the first model to achieve an end-to-end and pixel-to-pixel neural
network model in semantic segmentation. FCN is made up of fully convolutional
layers that can take the input with arbitrary size and produce
correspondingly-sized output. Using the upsampling convolution, FCN achieves
the feature extraction and reconstructs the mask with the same size as the
original image.

####  IV-A2 U-Net

U-Net is a kind of convolution networks for biomedical image segmentation. It
is widely adopted for simplicity but efficiency in small datasets. Compared
with FCN, the biggest difference is that U-Net uses skip connections between
feature extraction and mask reconstruction, which greatly enhances the
accuracy of pixels after reconstructing the mask.

###  IV-B Cooperation Learning of image transformation

Consider the opposite problem of semantic segmentation, converting the
semantic mask into a realistic image. This problem can be solved by previous
independent learning methods. However, the researchers found that the
generated images are vague due to a lack of details. Inspired by GAN which can
generate realistic and vivid images, more and more image-to-image translation
tasks adopted the conditional GAN. As mentioned in Section III-B, conditional
GAN is considered a kind of cooperative learning under the framework of IGC
learning. At the beginning of the image transformation, the works of
cooperative learning are aimed to the aligned dataset. Compared with
independent learning, cooperative learning has significantly improved the
details of the generated images, e.g., colourization and edge to photo in
Table LABEL:dataset. An aligned dataset restricts the applications of the
image transformation. Some researchers pay attention to accomplishing the
object transformation of images on unaligned datasets. The survey [50]
summarizes the methods for two types of datasets in the context of GAN.

In this section, we analyse typical image transformation methods about
cooperation learning in the IGC learning framework. We review the PixeltoPixel
[51] on aligned dataset and CycleGAN [24] on unaligned dataset.

Figure 7: Review the Pixeltopixel model based on IGC learning framework

####  IV-B1 PixeltoPixel

PixeltoPixel is a model to solve general image transformation problems in Fig.
2. PixeltoPixel, a typical conditional GAN, has a generator as image
transformation network and an discriminator as a classifier. As shown in Table
LABEL:dataset, PixeltoPixel solves some transformation tasks on aligned
dataset.

Based on the IGC learning framework, PixeltoPixel can be regarded as two
students $S^{1}_{w}$ and $S^{2}_{w}$, as shown in Fig. 7. The $S^{1}_{w}$ is
generator as the main student. The $S^{2}_{w}$ is discriminator as the
associate student.

The annotation of $S^{1}_{w}$ output comes from the $S^{2}_{w}$. The label of
$S^{2}_{w}$ output is defined by the source the input.

The cGAN loss is defined as follows,

$\mathcal{L}_{cgan}(S^{1}_{w},S^{2}_{w};x,y)=\log(S^{2}_{w}(x,y))+\log(1-S^{2}_{w}(S^{1}_{w}(x),y)).$
(2)

The $S^{1}_{w}$ and $S^{2}_{w}$ are related to cGAN loss. To optimize the
parameters of $S^{1}_{w}$, we minimize the cGAN loss when fixing the
$S^{2}_{w}$. From the original paper, PixeltoPixel uses $L_{1}$ distance
rather than $L_{2}$ as $L_{1}$ encourages less blurring.

$\mathcal{L}_{1}(S^{1}_{w}(x),y)=\|S^{1}_{w}(x)-y\|$ (3)

When we fix the parameters of $S^{2}_{w}$, $\mathcal{L}_{cgan}$ can be
regarded as as a part of the loss of $S^{1}_{w}$. The total loss for
$S^{1}_{w}$ is

$\mathcal{L}^{1}_{total}(S^{1}_{w};x,y,S^{2}_{w})=\mathcal{L}_{cgan}(S^{1}_{w};x,y,S^{2}_{w})+\lambda\mathcal{L}_{1}(S^{1}_{w};x,y)$
(4)

where $\lambda$ is a hyperparameter to balance the two losses.

When we fix the $S^{1}_{w}$ to optimize the $S^{2}_{w}$, the total loss can be
defined as a cross-entropy loss which is the same as the negative
$\mathcal{L}_{cgan}$.

$\mathcal{L}^{2}_{total}(S^{2}_{w};x,y,S^{1}_{w})=-\mathcal{L}_{cgan}(S^{2}_{w};x,y,S^{1}_{w}).$
(5)

The cooperative learning between $S^{1}_{w}$ and $S^{2}_{w}$ is that
$S^{1}_{w}$ passes the generated images to $S^{2}_{w}$ and $S^{2}_{w}$ labels
the outputs of $S^{1}_{w}$ then passes it back to $S^{1}_{w}$ through (2).
With the cGAN’s loss, $S^{1}_{w}$ and $S^{2}_{w}$ accomplish the data
spreading and label spreading. Meanwhile, $S^{1}_{w}$ also conducts the
individual learning through $\mathcal{L}_{1}$ loss.

####  IV-B2 CycleGAN

CycleGAN is a general model to accomplish image to image translation on
unaligned datasets. Suppose there are two image datasets $A$ and $B$, each of
which contains a series of images about a specific object. The task is to
replace the object in $A$ with the object in $B$, and vice versa. As is shown
in Fig. 8, CycleGAN is composed of four student models $S^{1}_{w}$,
$S^{2}_{w}$, $S^{3}_{w}$ and $S^{4}_{w}$. The $S^{1}_{w}$ and $S^{2}_{w}$
models are generator and discriminator respectively in cGAN. The model
$S^{1}_{w}$ achieves the transformation of images from $A$ to $B$. Similar to
that, the model $S^{2}_{w}$ judges whether the image comes from $A$. The model
$S^{3}_{w}$ achieves the transformation of images from $B$ to $A$. The model
$S^{4}_{w}$ judges whether the image comes from $B$.

Figure 8: Review the CycleGAN model based on IGC learning framework

The sample image in $A$ is denoted as $x_{a}$ and the sample image in $B$ is
denoted as $x_{b}$. The $S^{1}_{w}$ is related to $S^{2}_{w}$ and $S^{3}_{w}$.
The loss of $S^{1}_{w}$ related $S^{2}_{w}$ is called cGAN loss as follows,

$\mathcal{L}^{1}_{cgan}(S^{1}_{w},S^{2}_{w};x_{a},x_{b})=\log(S^{2}_{w}(x_{b}))+\log(1-S^{2}_{w}(S^{1}_{w}(x_{a})))$
(6)

We defined the $z_{a}$ and $z_{b}$ for simplicity,

$\displaystyle z_{a}$ $\displaystyle=S^{3}_{w}(S^{1}_{w}(x_{a}))$ (7)
$\displaystyle z_{b}$ $\displaystyle=S^{1}_{w}(S^{3}_{w}(x_{b}))$

The loss of $S^{1}_{w}$ related $S^{3}_{w}$ is called the cycle consistency
loss as follows,

$\mathcal{L}^{1}_{cycle}(S^{1}_{w};S^{3}_{w},x_{a})=L_{p}(z_{a},x_{a})=\left(\sum_{l=1}^{n}\left|z_{a}^{(l)}-x_{a}^{(l)}\right|^{p}\right)^{\frac{1}{p}}$
(8)

where $p$ sets the dimension of the metric space. When $p=1$, the loss
$\mathcal{L}_{cycle}$ is equivalent to the loss $\mathcal{L}_{1}$ in (3). When
$p=2$, the loss $\mathcal{L}_{cycle}$ is equivalent to the loss
$\mathcal{L}_{pixel}$ in (1).

The total loss of the $S^{1}_{w}$ is summarized as follows,

$\displaystyle\mathcal{L}^{1}_{total}(S^{1}_{w};S^{2}_{w},S^{3}_{w},x_{a},x_{b})=$
$\displaystyle\mathcal{L}^{1}_{cgan}(S^{1}_{w};S^{2}_{w},x_{a},x_{b})$ (9)
$\displaystyle+$
$\displaystyle\lambda\mathcal{L}^{1}_{cycle}(S^{1}_{w};S^{3}_{w},x_{a})$

The total loss of the $S^{2}_{w}$ is similar to (5).

$\mathcal{L}^{2}_{total}(S^{2}_{w};S^{1}_{w},x_{a},x_{b})=-\mathcal{L}^{2}_{cgan}(S^{2}_{w};S^{1}_{w},x_{a},x_{b}).$
(10)

The total loss of the $S^{3}_{w}$ is similar to the loss of $S^{1}_{w}$ in
(9).

$\displaystyle\mathcal{L}^{3}_{total}(S^{3}_{w};S^{1}_{w},S^{4}_{w},x_{a},x_{b})=$
$\displaystyle\mathcal{L}^{3}_{cgan}(S^{3}_{w};S^{4}_{w},x_{a},x_{b})$ (11)
$\displaystyle+$
$\displaystyle\lambda\mathcal{L}^{3}_{cycle}(S^{3}_{w};S^{1}_{w},x_{b}).$

###  IV-C Guided Learning of style transfer

Style transfer is the application of deep neural networks to transform a
content image with a new style. A specific style is often defined by a style
image. The goal of the style transfer is to create a new image by combining a
content image and a style image. The content image can be any picture,
including landscape, portraits, etc., and the style image is required a
certain style like in Fig. 1. The first method by Gatey et al. [16] is to
accomplish the style transfer iteratively. This method is flexible enough to
transform a content image with a variety of different styles. However, when a
series of content images need to be processed with a fixed style, the speed of
the generating images becomes a bottleneck. Following the work of Johnson et
al. [52], they provide an online generative model to solve this problem. They
train a convolutional network to accomplish a content image translation with a
certain style. Compared to the Gatey et al.’s method, it produces similar
qualitative results but is three orders of magnitude faster. According to
their contributions, more and more works follow up and develop in two
directions [17]. Based on our proposed IGC learning framework, we believe that
the difference between the two method lies in the student model. Therefore, we
introduce the iterative (online) generative model and the non-iterative
(offline) generative model in a unified guided learning framework.

####  IV-C1 Iterative generative model

The method proposed by Gatey et al. is a typical iterative generative method.
The iterative generative method can be seen as an optimization-based method.
From the perspective of IGC, we categorize this method as guided learning and
decompose it into a student model $S_{w}$ and a teacher model $T$
respectively.

A content image is denoted as $x_{c}$ and a style image is denoted as $x_{s}$.
Surprisingly, the $S_{w}$ is plain enough to be defined as a linear model.

$S_{w}=w=x^{\prime}$ (12)

where $w$ is initialized as white noise in accordance with the independent
Gaussian distribution and $x^{\prime}$ is the fused image combining a content
image and a style image. There is no input to $S_{w}$ and the output is the
optimized parameters for $S_{w}$. The $T$ is a pretrained convolution neural
network as a classifier model on ImageNet. Vgg network [53] was adopted as $T$
in [16], but other models like GoogleNet [54] are still adequately
appreciative.

We first define the content loss for $x^{\prime}$ and $x_{c}$ between $S_{w}$
and $T$ in guide learning.

$\mathcal{L}^{l}_{c}(S_{w};T,x_{c})=\sum_{l}(T^{l}(x^{\prime})-T^{l}(x_{c}))^{2}$
(13)

where $l$ means the $l$-th layer of the $T$ and $T^{l}(\cdot)$ denotes the
activation of the $l$-th layer for inputs. Before defining the style loss, we
define the Gram matrix in the $k$-th layer of the $T$ for $x_{s}$.

$G^{k}_{ij}(x_{s})=\sum_{k}{T^{k}_{i}(x_{c})T^{k}_{j}(x_{c})}$ (14)

where $G^{k}_{ij}$ is the inner product between the vectorised feature maps
$i$ and $j$ in the $k$-th layer. Then, we define the style loss for
$x^{\prime}$ and $x_{style}$ as follows,

$\mathcal{L}^{k}_{s}(S_{w};T,x_{s})=\sum_{k}(G^{k}_{ij}(x^{\prime})-G^{k}_{ij}(x_{s}))$
(15)

Finally, the total loss is defined for $S_{w}$.

$\mathcal{L}_{total}{(S_{w};T,x_{c},x_{s})}=\sum_{l\in
L}\mathcal{L}^{l}_{c}(S_{w};T,x_{c})+\alpha\sum_{k\in
K}\mathcal{L}^{k}_{s}{(S_{w};T,x_{s})}$ (16)

where $L$ is a set of $l$, $K$ is a set of $k$ and $\alpha$ is a balance
hyperparameter.

####  IV-C2 Non-iterative generative model

As the iterative generative model with a specific style picture, student model
$S_{w}$ needs to be iteratively trained for each new content image. Another
effective way is that the $S_{w}$ is pretrained for a certain style. When
solving a new content image, the fused image $x^{\prime}$ can be directly
generated through $S_{w}$. For this purpose, Johnson et al. [52] replace the
linear $S_{w}$ with a neural network and pretrain it to accomplish a non-
iterative generative method.

The replaced $S_{w}$ has a similar structure in Fig. 6. As is shown in Fig. 9,
we redrawn its model diagram based on IGC learning framework. By training an
image dataset composed of content images, student model $S_{w}$ about the
offline generative model can complete a certain style of end-to-end image
transformation.

The loss is also similar to (16). They define perceptual loss functions that
measure perceptual differences in content and style between images. Compared
with the iterative generative model, the non-iterative generative model is
only different in the structure and training method of the student model from
the perspective of the IGC learning framework.

Figure 9: The model of an off-line generative model on style transfer. The
figure is modified from [52] to fit the IGC learning framework.

##  V A Taxonomy of image transformation Methods

There are many methods about image transformation from different perspectives.
According to the IGC learning framework in Table LABEL:table_methods, we
roughly categorize the general image transformation methods from two domains
to multiple domains. Correspondingly, we also divide the image style transfer
works into multiple-style and arbitrary-style methods. The research of general
image transformation and style transfer are developed independently. Wu et al.
[50] presented a survey of image transformation about synthesis and editing
with GANs, ignored the style transfer yet. Jing et al. [17] investigated style
transfer in recent years but did not cover general image transformation.

In this article, we review the general image transformation and style transfer
along with a similar research trend. To the best of our knowledge, we first
organize the researches on general image transformation and style transfer
from the unified perspective of our IGC learning framework. The development of
general image transformation has a strong relationship with the development of
style transfer. According to above discussion, we summary some methods of
general image transformation and style transfer in Table LABEL:table_methods.
First, we briefly introduce the problem to be solved by each method. Then we
discuss their models and learning methods under the IGC learning framework.
More details about each method can refer to the original paper.

TABLE II: The taxonomy of image transformation methods Methods |  Dataset Type |  Learning Paradigm |  Loss |  Description
---|---|---|---|---
FCN [47] |  Aligned |  Independent learning |  $\mathcal{L}_{pixel}$ |  Image semantic segmentation.
U-Net [48] |  Aligned |  Independent learning |  $\mathcal{L}_{pixel}$ |  Image semantic segmentation.
PixeltoPixel [51] |  Aligned |  Cooperative learning & Independent learning |  $\mathcal{L}_{pixel}$ & $\mathcal{L}_{cgan}$ |  General image translation.
TextureGAN [55] |  Aligned |  Cooperative learning & Independent learning |  $\mathcal{L}_{pixel}$ & $\mathcal{L}_{cgan}$ & Local texture loss |  General image translation, especially in texture.
VidtoVid [56] |  Aligned |  Cooperative learning & Independent learning |  $\mathcal{L}_{cgan}$ & $\mathcal{L}_{style}$ (feature matching loss) |  Applying the video-to-video synthesis.
CycleGAN [24] |  Unaligned |  Cooperative learning |  $\mathcal{L}_{pixel}$ & $\mathcal{L}_{1}$ & $\mathcal{L}_{cgan}$ & $\mathcal{L}_{cycle}$ |  General image translation.
DiscoGAN [57] |  Unaligned |  Cooperative learning |  $\mathcal{L}_{pixel}$ & $\mathcal{L}_{gan}$ & $\mathcal{L}_{cycle}$ |  Similar to CycleGAN.
DualGAN [58] |  Unaligned |  Cooperative learning |  $\mathcal{L}_{pixel}$ & $\mathcal{L}_{cgan}$ & $\mathcal{L}_{cycle}$ |  Applying dual learning to image translation.
DA-GAN [59] |  Unaligned |  Cooperative learning |  $\mathcal{L}_{gan}$ & $\mathcal{L}_{cycle}$ & Attention loss |  Apply the attention to two domains transformation.
UNIT [24] |  Unaligned |  Cooperative learning |  $\mathcal{L}_{gan}$ & $\mathcal{L}_{cycle}$ |  Two domains transformation from probability distributions.
MUNIT [60] |  Unaligned |  Cooperative leaning |  $\mathcal{L}_{cgan}$ & $\mathcal{L}_{cycle}$ |  Multiple domains transformation.
StarGAN [61] |  Unaligned |  Cooperative learning |  $\mathcal{L}_{cgan}$ & $\mathcal{L}_{cycle}$ |  Multiple domains transformation, especially in face.
AugCGAN [62] |  Unaligned |  Cooperative learning |  $\mathcal{L}_{cgan}$ & $\mathcal{L}_{cycle}$ |  Extendingt the CycleGAN to multiple domains transformation.
Gatys et al. [2] |  Two images |  Guided learning |  $\mathcal{L}_{content}$ & $\mathcal{L}_{style}$ |  Iterative method to transfer the content image with any styles.
Johnson et al. [52] |  Content images and style images dataset |  Guided learning |  $\mathcal{L}_{content}$ & $\mathcal{L}_{style}$ |  Non-iterative method to accomplish a special style transformation with a network, one-style method.
Ulyanov et al. [63] |  Content images and style images dataset |  Guided learning |  $\mathcal{L}_{content}$ & $\mathcal{L}_{style}$ |  Propose the instance normalization to improve the performance, one-style method.
StyleBank [64] |  Content images dataset and some style images |  Guided learning |  $\mathcal{L}_{content}$ & $\mathcal{L}_{style}$ |  Propose filter banks for multiple styles. Every filter bank should be trained for one style, multi-style method.
Dumoulin et al. [65] |  Content images dataset and some style images |  Guided learning |  $\mathcal{L}_{content}$ & $\mathcal{L}_{style}$ |  Propose a conditional instance normalization (CIN), multi-style methods.
Roy et al. [66] |  Content images and style images dataset |  Guided learning |  $\mathcal{L}_{content}$ & $\mathcal{L}_{style}$ |  Propose to learn a style network to adapt the affine parameters of CIN , arbitrary-style method.
AdaIN [67] |  Content images and style images dataset |  Guided learning |  $\mathcal{L}_{content}$ & $\mathcal{L}_{style}$ |  Propose to learn the channel-wise mean and variance feature statistics in activations to adapt the affine parameters of CIN , arbitrary-style method.
DIN [66] |  Content images and style images dataset |  Guided learning |  $\mathcal{L}_{content}$ & $\mathcal{L}_{style}$ |  The idea is similar to the Ghiasi et al. but it improves the speed of style transfer, arbitrary-style method.
WCT [66] |  Content images and style images dataset |  Guided learning |  $\mathcal{L}_{content}$ & $\mathcal{L}_{style}$ |  Demonstrates that the covariance of the feature as the style reflects the whitening and coloring transformations, arbitrary-style method.
Lu et al. [68] |  Content images and style images dataset |  Guided learning |  $\mathcal{L}_{content}$ & $\mathcal{L}_{style}$ |  Derive a closed-form solution named Optimal Style Transfer (OST) and give a theoretical analysis in style transfer, arbitrary-style method.

###  V-A Methods of General Image Transformation

As mentioned in Section II-A and Section IV-B, most image transformation tasks
focus on the objects in images. PixeltoPixel implements pixel-level image-to-
image transformation on aligned datasets while CycleGAN implements object-
level image transformation on unaligned datasets. Both methods use the GAN as
the basic architecture, but the learning methods are different. From the view
of the IGC learning framework, in PixeltoPixel, the tasks of $S^{1}_{w}$ and
$S^{2}_{w}$ are different, where $S^{1}_{w}$ is the main task to implement
image transformation and $S^{2}_{w}$ is the associate task to determine
whether the image is forged. CycleGAN can be regarded as a model with two
PixeltoPixels, the $S^{1}_{w}$ and $S^{2}_{w}$ as one, $S^{3}_{w}$ and
$S^{4}_{w}$ as the other. The $S^{1}_{w}$ and $S^{3}_{w}$ are mutually dual
models to realize the transformation from A to B and B to A respectively. The
cycle loss as the pixel loss between $S^{1}_{w}$ and $S^{3}_{w}$ is adopted to
train their parameters on unaligned datasets.

However, both PixeltoPixel and CycleGAN have limited scalability, which means
that one model only solves the image translation in two domains. The general
image transformation among $N$ domains requires $N*(N-1)$ models. To improve
the flexibility and diversity, some researchers pay attention to accomplishing
general image transformation in multiple domains with single model. In this
section, we first discuss the general image transformation in two domains.
Then, the methods of multiple domains transformation are introduced.

####  V-A1 Two domains transformation

At the beginning, semantic segmentation is widely studied as a task to
comprehend the content of images for decades. The researchers expanded the
problem to general image transformation in two domains. According to whether
it is applicable to the aligned and unaligned datasets, we divide the model
into two parts, one part is about individual learning, the other part is about
cooperative learning.

For aligned datasets, most methods have similar structures as shown in Fig. 6.
$\mathcal{L}_{1}$ and $\mathcal{L}_{style}$ loss are also commonly used. Some
methods improve the PixeltoPixel from different aspects. TextureGAN [55]
investigates the image synthesis guided by texture and develop a local texture
loss to generate more realistic images. Studying the loss function on images
can improve the quality of image translation under individual learning. Wang
et al. [56] extend the image-to-image synthesis problem and investigate the
potential research of the video-to-video synthesis. The most critical thing in
video translation is the speed of image-to-image translation. The efficiency
of image-to-image translation is also a problem that neural network
researchers focus on. However, these methods on the alignment datasets have
the same problem, i.e., it is difficult to extend to general image
transformations due to the cost of labelling datasets.

For unaligned datasets, learning from unlabelled data is the trend of deep
image research. It is difficult for the aforementioned CycleGAN to clearly
confirm whether it is the first work. The idea of CycleGAN about cycle loss
was also proposed by other models in the same period, e.g., DiscoGAN [57] and
DualGAN [58]. DiscoGAN and DualGAN both solve the general image transformation
in two domains and have similar architectures which related to the four
student models like CycleGAN. The difference is that the details of student
model and experiments on the image-to-image translation. The experiment of
DiscoGAN verified the necessity of cycle loss that ensures the correlation
between domain A and domain B. Inspired by dual learning from natural language
translation [43], DualGAN developed a novel dual-GAN mechanism to accomplish
image-to-image translation between two domains. The DualGAN also utilizes
Wasserstein GAN [69] to improve the stability of training and the results of
experiments. To improve the quality of general image transformation, some
researchers have proposed new losses under cooperative learning. The
unsupervised image-to-image translation (UNIT) [24] learns the marginal
distributions to infer the joint distribution of images in different domains.
DA-GAN [59] learns the instance-level correspondences by attention mechanism
[70].

Figure 10: Comparison between models about two domains and StarGAN about
multiple domains [61]. (a) To handle multiple domains, cross-domain models
should be built for every pair of image domains. (b) StarGAN has capable of
learning mappings among multiple domains using a single generator.

####  V-A2 Multiple domains transformation

Multiple domains transformation aims to convert images with multiple types in
only one model. By solving the poor flexibility on the two domains
transformation, the methods about multiple domains transformation have
received extensive attentions and researches. Inspired by cGAN, most methods
try to adapt to the migration of images on multiple domains by conditional
input.

However, due to the lack of annotation information, the quality of image
transformation on cooperative learning is not as good as that on individual
learning. There are constraints and connections among multiple domains.
StarGAN [61] aims to achieve transformation among multiple domains of facial
attributes and expressions. Fig. 10 shows the two domains transformation
method and StarGAN. The model of StarGAN determines the image translation
domain according to the conditional input. It utilizes a mask vector as the
conditional input to control all available domain labels, and then achieves
the mapping between arbitrary domains with a single model. Improved from UNIT,
Multimodel Unsupervised Image-to-Image Translation (MUNIT) [60] applies one
model to infer the joint probabilities among multiple image domains. MUNIT
assumes that the image representation can be decomposed into a content code
that is domain-invariant and a style code that captures domain-specific
properties. In the considerations of the learning [71], MUNIT explicitly maps
the image to a content and a style and accomplishes image-to-image translation
in multiple domains by combining different styles with special content. MUNIT
implements image transformation among multiple similar objects in Fig 11.
Augmented CycleGAN (AugCGAN) [62] extends CycleGAN to many-to-many mappings
among image domains. AugCGAN adds a latent variable to the original image as
input and also outputs a new latent variable with the translation image. As
shown in Fig. 12, multiple translation images can be generated from the same
source image with additional information. It successfully accomplishes the
diversity of image translation. We summary two main objectives in multiple
domains transformation. One is to convert the attributes about object, the
other is to transfer among similar objects.

Figure 11: Examples of UNIT. It accomplished Image-to-image translation among
similar objects. The figure comes from [24].

###  V-B Methods of Style Transfer

Most methods about style transfer are related to the guided learning in the
IGC learning framework. Just like the development of general image
transformation, we survey the methods of style transfer according to similar
classification. When the content image is appointed, iterative methods like
Gaty et al. can realize the transformation of any styles. However, due to the
efficiency of iterative methods, most proposed methods are non-iterative
methods. For example, the student model need to pretrained on a special style
in Johnson et al. A simple method of multiple-style transfer is to pretrained
multiple student models of different styles. The adaptability and efficiency
of style transfer methods are crucial for their applications yet. The problem
is similar to the aforementioned general image transformation among multiple
domains. Many follow-up researchers study non-iterative methods to achieve the
transfer about multiple styles, even arbitrary styles.

Before discussing those methods about non-iterative methods, we introduce
important developments in iterative generative methods. Many subsequent
studies are also based on those works. Instance normalization [72] is proposed
instead of batch normalization [73]. Instance normalization, which is
equivalent to batch normalisation when the batch size is set to $1$, leads to
a significant improvement in stylisation quality [63]. Instance normalization
methods rely on affine transformations to produce arbitrary image style
transfers, of which the parameters are computed in a pre-defined way. Dynamic
Instance Normalization (DIN) [17] introduces a sophisticated style encoder to
automatically learn the affine parameters of transformations to provide
flexible support for image style transfer. The iterative generation method can
generally be regarded as an arbitrary style method.

As for non-iterative methods, most methods focus on improving student model
from the respective of IGC learning framework. For multiple styles, one of the
main purposes is to use a single student model to transfer multiple styles
according to the addition input conditions. For arbitrary styles, a student
model can accomplish content image transformation with arbitrary style image,
just like iterative methods. Since most solutions try to modify the student
model, we mainly introduce the student model of each methods. The teacher
model and learning methods are similar to the work of Johnson et al. from the
respective of IGC learning framework. The student model of style transfer is
also based on the encoder and decoder. Most methods propose a style network
model that extracts the style from the image and embed it between the encoder
and the decoder. In this Section, we mainly focus on the no-iterative
generative methods and also summary them in Table LABEL:table_methods.

####  V-B1 Multiple-Style methods

To handle multiple styles with a single student model, a way is to separate
image content and combine them when a special style is appointed. StyleBank
[64] is composed of multiple convolution filter banks and each filter bank
explicitly represents one style. StyleBank explicitly divides the student
model into encoder, style filter bank, and decoder. To transfer an image to a
specific style, the corresponding filter bank is operated on top of the
intermediate feature embedding produced by a single encoder. StyleBank
realizes multiple styles from content image to fusion image by defining
different style bank filters in Fig. 13. Although StyleBank avoids training a
whole model, it still needs to train special filter bank for each style.

Inspired by instance normalization, Dumoulin et al. [65] propose a conditional
instance normalization (CIN) that learns a different set of parameters
$\alpha_{s}$ and $\beta_{s}$ for each style $s$. The conditional instance
normalization is defined as,

${CIN}_{\mathcal{F}_{i}(x_{c})}=\alpha_{s}\frac{(\mathcal{F}_{i}(x_{c})-\mu(\mathcal{F}_{i}(x_{c}))}{\sigma(\mathcal{F}_{i}(x_{c}))}+\beta_{s},$
(17)

where $\mathcal{F}$ is the activation of the $i$-th feature layer of the
student model. They found that different styles are accomplished by using the
same convolution parameters but different affine parameters in instance
normalization layers. This critical discovery makes it possible to learn
multiple styles in a network with stacked CINs.

Figure 12: Example results of Augmented CycleGAN. StochcGAN means the
stochastic CycleGAN as the comparative method. The figure is credited from
[62] Figure 13: StyleGAN consists of three modules: image encoder
$\mathcal{E}$, StyleBank layer $\mathcal{K}$ and image decoder $\mathcal{D}$.
The figure comes from [64]

(a) Semantic Mask

(b) Ground Truth

(c) By U-Net

(d) By Pixel2Pixel

Figure 14: Some example results of image-to-image translation generated by
U-Net in Fig. 14(c) and Pixel2Pixel in Fig. 14(d). The ground truth is in Fig.
14(b).

####  V-B2 Arbitrary-Style methods

The style of any image is extracted through the deep convolution network, and
as a part of a student model, arbitrary styles are synthesized on content
image. Following conditional instance normalization, Ghiasi et al. [74] train
a style transfer network to supply a set of affine parameters. Since its style
transfer network is trained on corpus of style images and content images, it
has a certain generalization to extract styles for unknown images. Besides,
AdaIN [75] also solves a similar problem. Instead of training an entire style
network, AdaIN proposed adaptive instance normalization layer embedded in the
original autoencoder. Modifying the (17), the adaptive instance normalization
is defined as,

$\begin{array}[]{l}AdaIN_{\mathcal{F}_{i}}(x_{c},x_{s})=\\\
\quad\quad\sigma(\mathcal{F}_{i}(x_{s}))\frac{(\mathcal{F}_{i}(x_{c})-\mu(\mathcal{F}_{i}(x_{c}))}{\sigma(\mathcal{F}_{i}(x_{c}))}+\mu(\mathcal{F}_{i}(x_{s})).\end{array}$
(18)

AdaIN transfers the channel-wise mean and variance feature statistics between
content and style feature activations. AdaIN does not learn fixed affine
parameters, but calculates them adaptively from the pattern image. Drawing on
the advantages of AdaIN and Ghiasi et al. ’ method, Dynamic Instance
Normalization (DIN) [76] introduces a sophisticated style encoder and a
compact lightweight content encoder for fast inference. DIN uses a lightweight
architecture to achieve arbitrary style transfer, which reduces the
computational cost by more than $20$ times.

However, only adjusting the mean and variance of feature statistics makes it
hard to synthesise complicated style patterns with rich details and local
structures. Whitening and Colouring Transform (WCT) [66] demonstrates that
whitening and colouring transformations reflect a direct matching of feature
covariance of the content image to a given style image. WCT adopts feature
transformation network to accomplish whitening and colouring, which is
embedded to decoder as the style for image composition. Lu et al. [68] give a
unique closed-form solution to image style transfer as an optimal transport
problem. They also demonstrate the relations of their formulation with former
works like AdaIN and WCT. Both AdaIN and WCT holistically calculate their
style feature distributions and the parametrize feature statistics. Avatar-Net
[77] proposed zero-shot artistic style transfer that means the same as
transferring arbitrary style into content images. The key ingredient of
avatar-Net is a style decorator that makes up the content features by
semantically aligning style features from an arbitrary style image. Avatar-Net
does not only holistically match their feature distributions but also
preserves detailed style patterns in the decorated features.

##  VI Experiment

In this section, we perform experiments about image transformation on three
learning paradigms of the IGC learning framework. We select several models on
three types of datasets from the IGC learning framework. We qualitatively
evaluate each model through metrics and visualization to show the superiority
in context of the IGC learning.

###  VI-A General Image Transformation with Independent Learning

In Table LABEL:dataset, we choose one aligned datasets: CMP Facades. We choose
the U-Net as a independent learning model and PixeltoPixel as a cooperative
learning model. The codes 111https://github.com/junyanz/pytorch-CycleGAN-and-
pix2pix come from the open source library by official.

####  VI-A1 Experiment details

The CMP Facades consists of $606$ images. Each image is cropped as 256 x 256 x
3 as the input. The dataset is divided into training, validation, and testing
datasets with $400$, $100$, and $106$ images respectively. U-Net is composed
of $4$ skip connection blocks like in Fig 6. Batch normalization [73] is
adopted instead of dropout layer [78] in each blocks. U-Net as a student model
is trained by (3). Taking U-Net as the main student, PixeltoPixel adds
PatchGAN [79] as an associate student. PatchGAN encourages U-Net to generate
greater color diversity but has no effect on spatial statistics by classifying
whether a pixel is real or not. In PixeltoPixel, the U-Net and PatchGAN is
trained by (4) and (5) respectively. We set $\lambda=10$ in (4) and use the
Adam solver [80] with a batch size of $1$. All networks are trained from
scratch with a learning rate of $0.0002$. We also decay linearly the learning
rate of each parameter group by $\gamma=0.1$ with every $50$ epochs. U-Net and
PixeltoPixel are trained totally for $200$ epochs on training dataset.

####  VI-A2 Evaluation metrics

This task is to transform a label image into a realistic image. Evaluating the
quality of synthesized images is an open and difficult problem. We will
discuss more about the evaluation of image transformation in Section VIII-A.
In this experiment, the per-pixel loss in (1) can be seen as the metric to
access the quality between the generated image and ground truth. However, it
cannot measure the realism of the image. Therefore, we visualize some
transistorized images for intuitive comparison.

####  VI-A3 Visualization and analysis

Table LABEL:losstable1 shows the mean $L_{1}$ loss in (3) and per-pixel loss
in (1) of these models on testing dataset. With distance-based loss, the
difference between the generated image and the ground truth is small. We can
not judge the quality of the images from the loss. As is shown in Fig. 14, we
visualize some generated images by different models with the semantic label
and ground truth. From Fig. 14, the generated images by U-Net is blurred and
can easily be distinguished among those images. The images generated by
PixeltoPixel have more details, making it difficult to distinguish from the
ground truth. From the perspective of the IGC learning, PatchGAN as an
assistant student does provide some annotation for the main student by
cooperative learning. Visual analysis is more in line with human intuition,
while distance-based loss is more difficult to evaluate the performance of
image transformation methods.

TABLE III: The mean $L_{1}$ loss for U-Net and PixeltoPixel models. Models |  $L_{1}$ Loss | Per-pixel Loss
---|---|---
U-Net | 0.4891 | 0.3699
PixeltoPixel | 0.4921 | 0.3713

(a) Original Image

(b) Object Transformation

(c) Original Image

(d) Object Transformation

Figure 15: Some example results of image transformation with cooperative
learning generated by CycleGAN on appletoorange.

(a) G_A and D_A Losses

(b) Cycle_A and Idt_A Losses

Figure 16: The loss of CycleGAN when trained on appletoorange dataset. G_A
demonstrates the $\mathcal{L}^{1}_{cgan}$ of $S^{1}_{w}$ and D_A shows the
$\mathcal{L}^{2}_{cgan}$ in Fig. 16(a). As shown in Fig. 16(b), cycle_A means
the $\mathcal{L}^{1}_{cycle}$ of $S^{1}_{w}$ in (8) and idt_A represents the
$\mathcal{L}^{1}_{identity}$ in (19).

With the help of GAN on cooperative learning, the assistant model instructs
the main model to generate more texture details. It is difficult to describe
the details of texture by loss defined on the main model. Therefore,
cooperative learning is a good way to learn knowledge among models.
Furthermore, since the main students of PixeltoPixel are the same as U-Net,
the cost of image transformation is similar to each other. Cooperative
learning in the training process does not harm the efficiency of the model in
inference. Under the guidance of the IGC framework, we can study to apply
multiple learning methods to the previous single learning method. Many works
introduce GANs to improve the main model. Combined with individual learning,
cooperative learning can provide additional information to guide the main
students’ learning.

###  VI-B General Image Transformation with Cooperative Learning

In realistic image scenes, the collected data is often not aligned. For
example, we can obtain a bunch of misaligned images through a deep
classification network. In this section, we mainly experimented the general
image-to-image translation on unaligned data under cooperative learning.
According to Table LABEL:dataset, we choose one unaligned datasets:
apple2orange 222
https://people.eecs.berkeley.edu/~taesung_park/CycleGAN/datasets/apple2orange.zip.
The dataset is collected by [31]. We verify the experiment of the CycleGAN
model and discuss its performance and shortcomings.

####  VI-B1 Experiment details

The apple2orange dataset contains $1261$ images of apples and $1267$ images of
oranges. The images are crawled from the web with different resolutions, but
uniformly cropped to $256$ x $256$ and normalized as the input of the
CycleGAN. The details of CycleGAN are discussed in Section IV-B2. However, in
the implementation, the identity loss is also defined for $S^{1}_{w}$ as
following.

$\mathcal{L}^{1}_{identity}(S^{1}_{w};x_{a})=\left|\left|S^{1}_{w}(x_{a})-x_{a}\right|\right|_{1}$
(19)

The identity loss encourages the mapping to preserve color composition between
the input and output. Besides, we also use adam with the batch size of $1$ and
the learning rate of $0.0002$. We iteratively train the model for $200$
epochs. The losses related $S^{1}_{w}$ in the training is shown in Fig. 16 and
the losses of $S^{3}_{w}$ is similar to it. We can estimate the training of
CycleGAN by monitor the loss in 16(b).

Figure 17: Some samples of aforementioned methods about image style transfer
for qualitative evaluation. The content images are from the benchmark dataset
proposed by Mould and Rosin [81]. The figure comes from [17].

####  VI-B2 Evaluation method

For unaligned datasets, we cannot directly utilize the ground truth to
evaluate the generated images. Evaluating the quality of generated images is a
challenging problem, which we will discuss more in section VIII-A. Following
the previous methods, we visually analyse the results of CycleGAN as a typical
cooperative learning in the unaligned dataset.

####  VI-B3 Visualization and analysis

Fig. 15 demonstrates some examples about the transformation from orange to
apple for the trained CycleGAN. The transformation object in Fig. 15(b) is a
typical example, which represents the result of most images. However, we think
the transformation is not good enough to consider as an apple. The
transformation occurs more in the color of the object, but the shape of the
object changes less. This problem also exists but it was ignored in the
original paper. They experimented on datasets such as horses to zebras, which
have highly similar shapes and different textures. From the IGC learning, we
believe that they adopt a naive cycle consistency loss in (8). As is shown in
Fig. 15(c), the cycle consistency restricts the object transfiguration to
generate the blur images. The cycle consistency can be seen as the extension
of $\mathcal{L}_{1}$ and $\mathcal{L}_{2}$ on unaligned datasets. Some
researchers utilize the new classification network as a teacher model to
strengthen the modification of target shapes like face transformation [82].
The new methods of cooperative learning or guided learning are proposed to
improve the performance of main student.

###  VI-C Style Transfer with Guided Learning

The survey of style transfer in [17] makes sufficient experiments to show the
results of the related methods. Therefore, we summary their works and review
them with the IGC learning framework. The relationships among the models are
discussed in this subsection.

####  VI-C1 Experiment details

Jing et al.select and test ten style images. Some style transfer method is
offlined and trained on MS-COCO dataset. Similar to what we classify in
Section IV-C, they also divide style transfer methods into two categories:
1)Image-Optimisation-Based Online Image transformation (IOB), 2)Model-
Optimisation-Based Offline Image transformation (MOB). The method proposed by
Gatys et al. belongs to IOB and other methods proposed by Ulyanov et al. [83],
Johnson et al. [52] and Li & Wand [84] belong to MOB. The works of Gatys et
al. and Li & Wand are introduced as the typical iterative generative and non-
iterative generative methods in Section IV-C. Ulyanov et al.focus on the
synthesis of textures to improve the IOB on style transfer. Furthermore,
following this work, they also improve their method by applying the instance
normalization to maximize quality and diversity of style transformation in
[63]. Li & Wand achieve the style transfer by precomputed real-time texture
synthesis with markovian generative adversarial networks. From our IGC
learning view, Ulyanov et al. optimize the structure of the main student to
accomplish the better style transfer and Li & Wand [84] introduce a
adversarial network as a new assistant student to improve the style transfer
network through cooperative learning. We can evaluate image style in terms of
the quality of generated images.

####  VI-C2 Visualization and discussion

Fig. 17 shows the results of related methods. The IOB method is
computationally expensive whose results are appealing in visually. The method
of Gatys et al. usually is regarded as the gold-standard method in the
community of style transfer. For the MOB methods, Ulyanov et al. and Johnson
et al. have a similar idea but differ in their detailed network architectures.
Li & Wand use cGAN to transmit the image, which has some strange rendering, as
shown in the middle area of the image in the group one of Fig. 17. The student
model of IOB method is too simple to complete the transfer of image styles in
one step. It needs to learn the transfer style step by step through the
guidance of the teacher model. The student models of MOB methods can learn
under the guidance teacher model and independently accomplish the style
transfer after training. The IOB and MOB methods are essentially the same in
the student models.

##  VII Application

With the development of general image transformation and style transfer in
neural network, the researchers also apply those technology to related
projects. Some works create more complex tools to extend image processing.
Those tools bring great convenience to image and video processors. In the
field of image and video creation, we can generate the forgery image and video
based on the general image transformation. Furthermore, some applications are
prevalent by younger to provide social communication and entertainment. As
following, we briefly introduce three aspects of application in image
transformation.

Figure 18: Face editing on CelebA [85]. The beard on the face has been edited
in each row. The original figure is from [86].

###  VII-A Tools for Image Transformation Works

image transformation have a broad application about various tasks of image
restoration and manipulation. As aforementioned the general image
transformation methods in Section V, more and more researchers pay attention
to applying them to actual application scenarios.

Image manipulation in CV is a little different from the image processing in
graphic. Image manipulation deals with more complex tasks, which are usually
difficult to define exactly by mathematics. Image inpainting [87, 88] and
Super-resolution [89] are two important applications about image manipulation.
Image inpainting is a task of reconstructing missing regions in an image. For
example, as is shown in Fig. 14 in Section VI , the upper left and lower right
parts of the ground truth in Fig. 14(b) are blank, but the generated images
supplement missing parts. Image super-resolution refers to the restoration of
a high-resolution image from a low-resolution image or image sequence. Both of
these tasks need to infer more information from their own information. A lot
of works from neural network try to solve the problem. Dong et al. [90] first
propose a deep learning method for single image super-resolution. Ledig et al.
[91] combine the perceptual loss and adversarial loss to train the network to
obtain more realistic high-resolution images.

Furthermore, it is important to learn interpretable decomposition
representations from network to achieve image editing. Higgins et al. [92]
introduce $\beta$-VAE, a new framework for automated discovery of
interpretable factorised latent representations from raw image data in a
completely unsupervised manner. Ricky et al. [86] inspire $\beta$-TCVAE (Total
Correlation Change Auto-Encoder) to learn the untangled representation,
thereby achieving controllable facial editing to a certain extent. Pan et al.
[93] achieve general image restoration and manipulation by learning a good
long-term prior of image. As shown in Fig. 18, Liu et al. successfully
realized user interactive visual content filling into the image editing
program These studies are closely related to the research of general image
transformation.

###  VII-B Image and Video Forgery about Face

Portraits, especially facial shots, have a significant proportion of the
images. Many applications about human faces, such as face recognition, facial
expression recognition, etc., are widely used in society. The image
transformation discussed earlier is mostly for general images. For facical
images, there are also massive studies to improve the aforementioned general
methods. Derived from the general image translation, I. Korshunova [94]
proposed the face-swap using similar network and loss, which realized the
replacement of different faces in the same image in Fig. 19. Face-swap can
also provide social communication and entertainment. TikTok [95] realizes the
replacement of faces in one video on mobile phones. which has attracted a huge
audience of $1.5$ billion active users.

These technologies make it difficult to identity fake images or videos. Some
malicious images and videos have been forged. For example, the images of
celebrities, politicians and others have been maliciously modified to damage
their reputations. The detection of tampered images is currently a hot topic,
and we will introduce relevant research in Section VIII-B as the future
challenge.

Figure 19: (a) The input image. (b) the result of face swapping with nicolas
cage using our method. (c) the result of a manual face swap (source:
http://niccageaseveryone.blogspot.com). The figure comes from [94]

###  VII-C Social Communication and Entertainment

style transfer can create impressive image, such as learning the painting
styles of famous painters like Van Gogh. As is shown in Fig. 1, some excellent
pictures are attractive and widely spread on social media. More and more
people would like to implement custom style transfer for their images. The web
Ostagram 333 https://www.ostagram.me makes it easy for anyone to accomplish
style transfer by uploading two pictures. The created high-quality pictures
can be publicly used for social communication like in Fig. 20. Prisma 444
https://www.prisma.io is a mobile application that provides similar services.
TikTok attracts a huge audience of $1.5$ billion active users, mostly children
and teenager as the fastest-growing application [95]. Although most social
attention is focused on leading platforms such as Twitter and Facebook, TikTok
is increasingly being used by young people under the age of 13. Use
technologies to attract user participation, these applications are playing
roles about social communication and entertainment.

Figure 20: The artist image combines the content image and style image from
Ostagram.

##  VIII Future Challenge

Currently, although many creative and artistic tasks about image translations
have been solved by neural networks, they extend and flourish the traditional
research of image transformation. However, new research has brought new
challenges. First, due to the lack of data annotation, it is difficult to
evaluate the results of the image transformation and style transfer. Second,
when the technologies of image transformation end-to-end makes it easier to
forge images, it is crucial to detect or classifier synthetic images. Third,
we believe that our proposed IGC learning framework is a broad learning
framework beyond the topic of image transformation. In this section, we
discuss two issues of image transformation about evaluation and detection, and
broaden our proposed IGC learning framework for future research.

###  VIII-A Evaluation of Image Transformation

The evaluation of the quality of synthesized images is an open and challenging
problem [96], because there is no correctly ground truth in some image
transformation tasks. Specifically, most evaluation methods of style transfer
algorithms are qualitative. For example, the usual evaluation process is to
recruit some users from Amazon Mechanical Turk. The synthesized and real
images are judged by humans. The evaluation process by humans is subjective
and costly. The results vary with different observers. Therefore, the auto
evaluation by the program is the current direction of research.

For the general image transformation methods, we evaluate them from a variety
of perspectives. According to the student model, we can compare its size and
computational cost. The efficiency of inference is the most concerned in the
deployment phase, especially on resource-constrained devices, e.g., mobile
devices. Multiple-domain transformation methods improve the efficiency of
image-to-image translation among objects. As for evaluating the quality of
generated image, we can utilize the ground truth on aligned dataset. The
$\mathcal{L}_{pixel}$ and $\mathcal{L}_{1}$ are used to measure the difference
between the generated image and the label. However, it’s not good metrics to
evaluate the synthetic images, which we experiment in Section VI-A.
Furthermore, the metrics do not work on unaligned dataset. More and more
researchers adopt the Intersection Over Union (IOU) [97] as an evaluation
metric. IOU is to calculate relatively the intersection pixels about a certain
object between the ground truth and the output of image transformation, as
defined.

$IoU=\frac{\text{Area of Overlap}}{\text{Area of Union}}$ (20)

When the general image transformation task on cooperative learning is related
to object transformation, the classifier network and object detection network
in the CV are used to evaluate the quality of images. Table LABEL:resultcyc
shows the classification accuracy and IOU about CycleGAN. The ablation
experiment verifies that cooperative learning improves the student model of
image-to-image translation.

TABLE IV: The ablation experiments of CycleGAN. The results evaluated on Cityscapes from label to photo. Loss | $\mathcal{L}_{pixel}$ | Per-class acc | Class IOU
---|---|---|---
Cycle alone | 0.22 | 0.07 | 0.02
GAN alone | 0.51 | 0.11 | 0.12
CycleGAN | 0.52 | .17 | 0.11

For style transfer methods with guided learning, it is more difficult to
quantitatively evaluate the quality of images. In the work of Jing et al.,
they experimented style transfer methods with $10$ styles images and content
images. Then, the visualization of the experimental results demonstrates that
each method accomplishes the image fusion task and has its own
characteristics. Although the style itself originates from subjective
feelings, researchers also evaluate those methods from other aspects, such as
the speed and diversity of transformation. In Section V-B, we introduce some
methods which deal with various content image and style images in one student
model. These methods reduce the time of inference and improve the diversity of
styles. On resource-constrained mobile platforms, the consumption of resource
in the model is also used as an evaluation metric. DIN achieves arbitrary
style transfer via MobileNet [98], and reduces the computational cost by more
than $20$ times compared with the existence methods.

In summary, although we can evaluate the model from the consumption of model,
inference time, and diversity, we are still hard to quantitatively assess the
quality of transformation images. More contributory work is worth looking
forward to in the future.

###  VIII-B Detection of Image Transformation

image transformation makes it easy to fake images and even videos. When it
comes to the security field, it is important to detect whether the image has
been fabricated. The Deepfake Detection Challenge (DFDC) Dataset [99], which
contains more than $100000$ videos, is created to promote the detection of
image transformation. Wang et al. [100] present a neural network method to
detect the manipulation by Photoshop and apply it to real artist-created image
manipulations. Rossler et al. [101] propose a method to examine the realism of
state-of-the-art image manipulations, especially on the facial images.

However, the existing detection methods rely on specific datasets and
generation algorithms, which leads to the weakness of their generalization.
Therefore, to enhance the capability of models for detection of image
transformation, the models should be exposed to more image transformation
methods, which helps models generalize on more transformation images. The
traditional detection methods can learn from the guidance learning, i.e., the
detection model is the student model, and the image transformation model is
the teacher model. Nguyen et al. proposed multi-task learning to detect and
segment manipulated facial images and videos, which is corresponding to the
individual learning of the IGC framework. The model consisted of multiple
student models have stronger the generalization of detection. Some researches
on the adversarial examples provided the development of active defence
algorithms [102]. This kind of detection methods are designed for unknown type
of transformed images. Employing adversarial examples is a typical cooperative
learning method. The detection of image transformation is also a hot topic.
The manipulation and detection are a zero-sum game, and the development of
forgery technologies continuously launch a new round of challenge to detection
field.

###  VIII-C Extension of the IGC Learning Framework

From the previous models and experiments, many models can be unified on the
IGC learning model framework. Although each learning method is relatively
independent, a mixed model of multiple learning methods can often be used in
specific problems. We believe that the exploration of effectively learning
ways among multiple models is a challenging research direction.

The paper [103] proposes a new capsule network. A capsule is a group of
neurons whose activity vector represents the instantiation parameters of a
specific type of entity (such as an object or a part of object). They divide
the different parts of the object whose each part is represented by a capsule.
The capsule can been seen as a student network from our IGC learning
framework. A series of capsules have their own independent capacities, and
they can be connected to learn parameters by appropriate methods, e.g.,
dynamic routing mechanism in [103]. Those methods classified as cooperative
learning enrich the research of the IGC framework.

Reinforcement learning [104] differs from supervised and unsupervised learning
but is contained in the IGC learning framework. In the absence of a training
dataset, reinforcement learning is bound to learn from its experience, which
is the study of decisions to make over time with consequences. The agent and
environment in reinforcement learning. The learning methods in reinforcement
learning can be regarded as guided learnings in the IGC framework.

In summary, we believe the IGC learning framework based on annotated
information is a unified framework to research neural network Systematically.
Massive neural network research can be incorporated into it. More works
deserve to be reconsidered and analysed with the IGC framework in the future.

##  IX Conclusion

In this article, we reviewed the image transformation from the perspective of
the IGC learning framework that we summarize and categorize the annotation to
train the neural network. Based on that, we analysed some image-to-image
translation methods on an aligned dataset with independent learning, the
general image translation on an unaligned dataset with cooperative learning,
and the image style transfer with guided learning. In addition, we summarized
the development of image translation and style transfer following one model to
accomplish a single image or multiple image transformations. To our best
knowledge, we are the first to propose a unified perspective called IGC
learning which is adopted to analyse image transformation and style transfer
about neural network image transformation. We experimented with related
methods based on the learning methods of IGC on three types of datasets and
analysed the advantages and disadvantages of their respective learning.
Furthermore, we discussed some popular applications and related topics. At the
end, we proposed possible directions to be research in future works.

## References

  * [1] E. Shelhamer, J. Long, and T. Darrell, _Fully Convolutional Networks for Semantic Segmentation_. IEEE Computer Society, 2017.
  * [2] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 2414–2423.
  * [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * [4] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2017.
  * [5] S.-H. Chou, C. Sun, W.-Y. Chang, W.-T. Hsu, M. Sun, and J. Fu, “360-indoor: Towards learning real-world objects in 360$\circ$ indoor equirectangular images,” in _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2020, pp. 834–842.
  * [6] e. a. Tang, Hao, “Attention-guided generative adversarial networks for unsupervised image-to-image translation,” in _International Joint Conference on Neural Networks (IJCNN)_ , 2019, pp. 1–8.
  * [7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krüger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” _ArXiv_ , vol. abs/2005.14165, 2020\.
  * [8] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [9] M. Everingham, L. Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International Journal of Computer Vision_ , vol. 88, pp. 303–338, 2009.
  * [10] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 2980–2988.
  * [11] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” _EMNLP_ , p. 353, 2018.
  * [12] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” in _International Conference on Learning Representations (ICLR)_ , 2016.
  * [13] N. Akhtar and A. Mian, “Threat of adversarial attacks on deep learning in computer vision: A survey,” _IEEE Access_ , vol. 6, pp. 14 410–14 430, 2018.
  * [14] T. M. Hospedales, A. Antoniou, P. Micaelli, and A. J. Storkey, “Meta-learning in neural networks: A survey,” _arXiv preprint arXiv:2004.05439_ , 2020.
  * [15] N. Bendre, H. Terashima-Marín, and P. Najafirad, “Learning from few samples: A survey.” _arXiv preprint arXiv:2007.15484_ , 2020.
  * [16] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 2414–2423.
  * [17] Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song, “Neural style transfer: A review,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 26, no. 11, p. 3365–3385, Nov 2020.
  * [18] A. J. Champandard. (2015) Deep forger: Paint photos in the style of famous artists,. [Online]. Available: http://deepforger.com
  * [19] Z. Fu, X. Tan, N. Peng, D. Zhao, and R. Yan, “Style transfer in text: Exploration and evaluation,” _arXiv preprint arXiv:1711.06861_ , 2017\.
  * [20] P. Luo, G. Wang, L. Lin, and X. Wang, “Deep dual learning for semantic image segmentation,” in _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 2737–2745.
  * [21] Z. Yu, L. Chen, Z. Cheng, and J. Luo, “Transmatch: A transfer-learning scheme for semi-supervised few-shot learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 856–12 864.
  * [22] R. Vemulapalli, O. Tuzel, M. Liu, and R. Chellappa, “Gaussian conditional random field network for semantic segmentation,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 3224–3233.
  * [23] Z. Yi, H. Zhang, P. Tan, and M. Gong, “Dualgan: Unsupervised dual learning for image-to-image translation,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 2868–2876.
  * [24] M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” in _Advances in Neural Information Processing Systems_ , 2017, p. 700–708.
  * [25] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [26] R. Tolosana, R. Vera-Rodriguez, J. Fierrez, A. Morales, and J. Ortega-Garcia, “Deepfakes and beyond: A survey of face manipulation and fake detection,” _Information Fusion_ , vol. 64, pp. 131–148, 2020.
  * [27] S. Agarwal, H. Farid, Y. Gu, M. He, K. Nagano, and H. Li, “Protecting world leaders against deep fakes,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2019, pp. 38–45.
  * [28] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, p. 3213–3223.
  * [29] R. Tylecek and R. Sára, _Spatial Pattern Templates for Recognition of Objects with Regular Structure_. Springer Berlin Heidelberg, 2013.
  * [30] J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros, “Generative visual manipulation on the natural image manifold,” in _European Conference on Computer Vision_ , 2016, p. 597–613.
  * [31] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [32] F. Luan, S. Paris, E. Shechtman, and K. Bala, “Deep photo style transfer,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4990–4998.
  * [33] X. Zhang, J. Zou, K. He, and J. Sun, “Accelerating very deep convolutional networks for classification and detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 10, p. 1943–1955, Oct 2016.
  * [34] J. S. Park, D. H. Kim, and M. S. Chung, “Anatomy comic strips,” _Anatomical Sciences Education_ , vol. 4, no. 5, pp. 275–279, 2011.
  * [35] G. Bartholeyns, “The instant past: Nostalgia and digital retro photography,” _Springer_ , pp. 51–69, 2014.
  * [36] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _Computer Ence_ , vol. 14, no. 7, pp. 38–39, 2015.
  * [37] Y. Yan, H. Hao, B. Xu, J. Zhao, and F. Shen, “Image clustering via deep embedded dimensionality reduction and probability-based triplet loss,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 5652–5661, 2020.
  * [38] Y. Xu, “Deep learning in multimodal medical image analysis,” in _International Conference on Health Information Science_ , 2019, p. 193–200.
  * [39] H. Mao, M. Cheung, and J. She, “Deepart: Learning joint representations of visual arts,” in _Proceedings of the 25th ACM international conference on Multimedia_ , 2017, pp. 1183–1191.
  * [40] X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, and P. Abbeel, “Variational lossy autoencoder,” _ArXiv_ , vol. abs/1611.02731, 2017.
  * [41] J. Devlin, M.-W. Chang, K. Lee, and K. N. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2018, pp. 4171–4186.
  * [42] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in _International conference on machine learning_. PMLR, 2017, pp. 214–223.
  * [43] D. He, Y. Xia, T. Qin, L. Wang, N. Yu, T.-Y. Liu, and W.-Y. Ma, “Dual learning for machine translation,” in _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , ser. NIPS’16. Red Hook, NY, USA: Curran Associates Inc., 2016, p. 820–828.
  * [44] S. Thorvaldsen and O. Hössjer, “Using statistical methods to model the fine-tuning of molecular machines and systems,” _Journal of Theoretical Biology_ , vol. 501, p. 110352, 2020.
  * [45] T. Chen, B. Ji, Y. Shi, T. Ding, B. Fang, S. Yi, and X. Tu, “Neural network compression via sparse optimization,” _ArXiv_ , vol. abs/2011.04868, 2020\.
  * [46] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” _Lecture Notes in Computer Science_ , p. 818–833, 2014.
  * [47] E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, pp. 640–651, 2017.
  * [48] O. Ronneberger, P. Fischer, e. N. Brox, Thomas”, J. Hornegger, W. M. Wells, and A. F. Frangi, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2015, pp. 234–241.
  * [49] F. Lateef and Y. Ruichek, “Survey on semantic segmentation using deep learning techniques,” _Neurocomputing_ , vol. 338, pp. 321–348, 2019.
  * [50] X. Wu, K. Xu, and P. Hall, “A survey of image synthesis and editing with generative adversarial networks,” _Tsinghua Science and Technology_ , vol. 22, no. 6, pp. 660–674, 2017.
  * [51] P. Isola, J. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 5967–5976.
  * [52] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” _Lecture Notes in Computer Science_ , p. 694–711, 2016.
  * [53] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [54] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 1–9.
  * [55] W. Xian, P. Sangkloy, V. Agrawal, A. Raj, J. Lu, C. Fang, F. Yu, and J. Hays, “Texturegan: Controlling deep image synthesis with texture patches,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , Jun 2018\.
  * [56] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, G. Liu, A. Tao, J. Kautz, and B. Catanzaro, “Video-to-video synthesis,” in _Advances in Neural Information Processing Systems_ , vol. 31, 2018, pp. 1144–1156.
  * [57] T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim, “Learning to discover cross-domain relations with generative adversarial networks,” in _Proceedings of the 34th International Conference on Machine Learning (ICML)_ , 2017, pp. 1857–1865.
  * [58] Z. Yi, H. Zhang, P. Tan, and M. Gong, “Dualgan: Unsupervised dual learning for image-to-image translation,” in _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 2868–2876.
  * [59] S. Ma, J. Fu, C. W. Chen, and T. Mei, “Da-gan: Instance-level image translation by deep attention generative adversarial networks,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5657–5666.
  * [60] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz, “Multimodal unsupervised image-to-image translation,” _Lecture Notes in Computer Science_ , p. 179–196, 2018.
  * [61] Y. Choi, M. Choi, M. Kim, J. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8789–8797.
  * [62] A. Almahairi, S. Rajeswar, A. Sordoni, P. Bachman, and A. C. Courville, “Augmented cyclegan: Learning many-to-many mappings from unpaired data,” in _International Conference on Machine Learning_ , 2018, pp. 195–204.
  * [63] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 4105–4113.
  * [64] D. Chen, L. Yuan, J. Liao, N. Yu, and G. Hua, “Stylebank: An explicit representation for neural image style transfer,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2770–2779.
  * [65] V. Dumoulin, J. Shlens, and M. Kudlur, “A learned representation for artistic style,” _arXiv preprint arXiv:1610.07629_ , 2016.
  * [66] S. Roy, A. Siarohin, and N. Sebe, “Unsupervised domain adaptation using full-feature whitening and colouring,” in _International Conference on Image Analysis and Processing_. Springer, 2019, pp. 225–236.
  * [67] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 1510–1519.
  * [68] M. Lu, H. Zhao, A. Yao, Y. Chen, F. Xu, and L. Zhang, “A closed-form solution to universal style transfer,” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 5952–5961.
  * [69] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved training of wasserstein gans,” _arXiv preprint arXiv:1704.00028_ , 2017.
  * [70] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , 2017, pp. 6000–6010.
  * [71] F. Locatello, S. Bauer, M. Lučić, G. Rätsch, S. Gelly, B. Schölkopf, and O. F. Bachem, “Challenging common assumptions in the unsupervised learning of disentangled representations,” in _36th International Conference on Machine Learning (ICML)_ , 2019, pp. 4114–4124.
  * [72] D. Ulyanov, A. Vedaldi, and V. S. Lempitsky, “Instance normalization: The missing ingredient for fast stylization.” _arXiv preprint arXiv:1607.08022_ , 2016.
  * [73] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proceedings of The 32nd International Conference on Machine Learning_ , 2015, pp. 448–456.
  * [74] G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens, “Exploring the structure of a real-time, arbitrary neural artistic stylization network,” _arXiv preprint arXiv:1705.06830_ , 2017.
  * [75] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1501–1510.
  * [76] Y. Jing, X. Liu, Y. Ding, X. Wang, E. Ding, M. Song, and S. Wen, “Dynamic instance normalization for arbitrary style transfer,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 4, 2020, pp. 4369–4376.
  * [77] Q. Shen, L. Zou, F. Wang, and Z. Huang, “A scale-adaptive color preservation neural style transfer method,” in _Proceedings of the 5th International Conference on Mathematics and Artificial Intelligence_ , 2020, pp. 5–9.
  * [78] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” _Journal of Machine Learning Research_ , vol. 15, no. 56, pp. 1929–1958, 2014\.
  * [79] C. Li and M. Wand, “Precomputed real-time texture synthesis with markovian generative adversarial networks,” in _European conference on computer vision_. Springer, 2016, pp. 702–716.
  * [80] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [81] D. Mould and P. L. Rosin, “A benchmark image set for evaluating stylization,” in _Proceedings of the Joint Symposium on Computational Aesthetics and Sketch Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering_ , 2016, p. 11–20.
  * [82] I. Perov, D. Gao, N. Chervoniy, K. Liu, S. Marangonda, C. Umé, Dpfks, C. S. Facenheim, L. Rp, J. Jiang, S. Zhang, P. Wu, B. Zhou, and W. Zhang, “Deepfacelab: A simple, flexible and extensible face swapping framework.” _arXiv preprint arXiv:2005.05535_ , 2020.
  * [83] D. Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempitsky, “Texture networks: Feed-forward synthesis of textures and stylized images,” in _Proceedings of the 33rd International Conference on International Conference on Machine Learning_ , 2016, p. 1349–1357.
  * [84] C. Li and M. Wand, “Precomputed real-time texture synthesis with markovian generative adversarial networks,” in _European Conference on Computer Vision_ , 2016, pp. 702–716.
  * [85] Z. Liu, P. Luo, X. Wang, and X. Tang, “Large-scale celebfaces attributes (celeba) dataset,” _Retrieved August_ , vol. 15, no. 2018, p. 11, 2018.
  * [86] R. T. Q. Chen, X. Li, R. Grosse, and D. Duvenaud, “Isolating sources of disentanglement in variational autoencoders,” _Advances in Neural Information Processing Systems_ , p. 2610–2620, 2018.
  * [87] O. Elharrouss, N. Almaadeed, S. Al-Maadeed, and Y. Akbari, “Image inpainting: A review,” _Neural Processing Letters_ , vol. 51, no. 2, p. 2007–2028, Dec 2019.
  * [88] L. Zhao, Q. Mo, S. Lin, Z. Wang, Z. Zuo, H. Chen, W. Xing, and D. Lu, “Uctgan: Diverse image inpainting based on unsupervised cross-space translation,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 5740–5749.
  * [89] Z. Wang, J. Chen, and S. C. Hoi, “Deep learning for image super-resolution: A survey,” _IEEE transactions on pattern analysis and machine intelligence_ , 2020.
  * [90] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 2, p. 295–307, Feb 2016.
  * [91] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 105–114.
  * [92] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, “beta-vae: Learning basic visual concepts with a constrained variational framework,” in _International Conference on Learning Representations (ICLR)_ , 2017.
  * [93] X. Pan, X. Zhan, B. Dai, D. Lin, C. C. Loy, and P. Luo, “Exploiting deep generative prior for versatile image restoration and manipulation.” in _European Conference on Computer Vision_ , 2020, pp. 262–277.
  * [94] I. Korshunova, W. Shi, J. Dambre, and L. Theis, “Fast face-swap using convolutional neural networks,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 3697–3705.
  * [95] G. Weimann and N. Masri, “Research note: spreading hate on tiktok,” _Studies in Conflict & Terrorism_, pp. 1–14, 2020.
  * [96] D. V. Ruiz, G. Salomon, and E. Todt, “Can giraffes become birds? an evaluation of image-to-image translation for data generation,” _arXiv preprint arXiv:2001.03637_ , 2020.
  * [97] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 658–666.
  * [98] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [99] B. Dolhansky, R. Howes, B. Pflaum, N. Baram, and C. C. Ferrer, “The deepfake detection challenge (dfdc) preview dataset,” _arXiv preprint arXiv:1910.08854_ , 2019.
  * [100] S. Wang, O. Wang, R. Zhang, A. Owens, and A. Efros, “Detecting photoshopped faces by scripting photoshop,” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 10 071–10 080.
  * [101] A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Niessner, “Faceforensics++: Learning to detect manipulated facial images,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 1–11.
  * [102] S. Khan, A. Artusi, and H. Dai, “Adversarially robust deepfake media detection using fused convolutional neural network predictions,” _ArXiv_ , vol. abs/2102.05950, 2021.
  * [103] S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” in _Advances in Neural Information Processing Systems_ , vol. 30, 2017, pp. 3856–3866.
  * [104] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, pp. 26–38, 2017.