# A Systematic Survey of Regularization and Normalization in GANs

Ziqiang Li, Xintian Wu, Muhammad Usman, Rentuo Tao, Pengfei Xia, Huanhuan
Chen2, , Bin Li2 This paper was submitted on November 16, 2020. This work was
partially supported by National Natural Science Foundation of China under
grand No.U19B2044, No.61836011 and No.91746209. Ziqiang Li, Muhammad Usman,
Rentuo Tao, and Pengfei Xia are with the CAS Key Laboratory of Technology in
Geo-spatial Information Processing and Application Systems, University of
Science and Technology of China, Anhui, China. (E-mail: iceli,muhammadusman,
trtmelon,xpengfei@mail.ustc.edu.cn)Xintian Wu is with computer science and
technology, Zhejiang University. (Email:hsintien@zju.edu.cn)Huanhuan Chen
(Corresponding author) is with the School of Computer Science and Technology,
University of Science and Technology of China, Anhui, China. (E-mail:
hchen@ustc.edu.cn)Bin Li (Corresponding author) is with the CAS Key Laboratory
of Technology in Geo-spatial Information Processing and Application Systems,
University of Science and Technology of China, Anhui, China. (E-mail:
<EMAIL_ADDRESS>
###### Abstract

Generative Adversarial Networks (GANs) have been widely applied in different
scenarios thanks to the development of deep neural networks. The original GAN
was proposed based on the non-parametric assumption of the infinite capacity
of networks. However, it is still unknown whether GANs can generate realistic
samples without any prior information. Due to the overconfident assumption,
many issues remain unaddressed in GANs’ training, such as non-convergence,
mode collapses, gradient vanishing. Regularization and normalization are
common methods of introducing prior information to stabilize training and
improve discrimination. Although a handful number of regularization and
normalization methods have been proposed for GANs, to the best of our
knowledge, there exists no comprehensive survey which primarily focuses on
objectives and development of these methods, apart from some in-comprehensive
and limited scope studies. In this work, we conduct a comprehensive survey on
the regularization and normalization techniques from different perspectives of
GANs training. First, we systematically describe different perspectives of
GANs training and thus obtain the different objectives of regularization and
normalization. Based on these objectives, we propose a new taxonomy.
Furthermore, we compare the performance of the mainstream methods on different
datasets and investigate the regularization and normalization techniques that
have been frequently employed in SOTA GANs. Finally, we highlight potential
future directions of research in this domain.

###### Index Terms:

Generative Adversarial Networks (GANs), Regularization, Normalization, Survey,
Lipschitz

##  I Introduction

Generative adversarial networks (GANs) [1] have been widely used in computer
vision, such as image inpainting [2, 3, 4], style transfer [5, 6, 7, 8], text-
to-image translations [9, 10, 11], and attribute editing [12, 13, 14, 15].
GANs training is a two-player zero-sum game between a generator and a
discriminator, which can be understood from different perspectives: (i) "Real
& Fake" [1, 16], (ii) "Fitting distribution" [17, 18], and (iii) "Training
dynamics" [19, 20]. GANs training suffers from several issues, for instance:
non-convergence [21, 22], mode collapses [23], gradient vanishing [24],
overfitting [25], discriminator forgetting [26] and deficiency [27], and
hyperparameters sensitivity [28]. Many solutions to mitigate these issues have
been proposed, focusing on designing new architectures [29, 30], loss
functions [18, 31, 26, 32], optimization methods [20, 30], regularization [31,
19], and normalization [46]. Among them, regularization and normalization
techniques are compatible with loss functions, model structures, and tasks,
which has attracted the attention of scholars.

Regularization and normalization are widely applied in neural networks
training. For supervised tasks, regularization in literature has been proposed
to introduce prior information leading to advantages like overfitting
prevention [33, 34], semi-supervised assumptions [35], manifold assumptions
[36, 37], feature selection [38], and low rank representation [39]. On the
other hand, normalization [40, 41] is advantageous for the Stochastic Gradient
Descent (SGD) [42], accelerating convergence and improving accuracy. Unlike
the icing on the cake of supervisory tasks, regularization and normalization
are utilized inevitably in weak-supervised and unsupervised tasks. GANs’
training is a two-player zero-sum game having a solution to Nash equilibrium.
The proposal of standard GAN is based upon the non-parametric assumption of
the infinite capacity of networks, an unsupervised learning task. Likewise, a
good number of research studies targeting GANs training from different
perspectives argue that unconstrained training causes unstable training
(generator [43] and discriminator [31]) and significant bias between real
images and fake images (attributes domain [44] and frequency domain [27, 45]).
Specifically, Jacobi regularization [19] needs to be used to achieve local
convergence from the perspective of "Training dynamics"; Gradient penalty
[31], weight normalization [46], and weight regularization need to be used to
fulfill Lipschitz continuity and stabilize training from the perspective of
"Fitting distribution"; Data augmentation [47] and preprocessing [45],
consistency regularization [48], and self-supervision [26] need to be used to
mitigate overfitting, improve the representation of discriminator, and avoid
discriminator forgetting from the perspective of "Real & Fake". Therefore,
advanced regularization and normalization techniques with rich assumptions are
indispensable for GANs training.

Figure 1: The summary of the regularization and normalization for GANs.

Regularization and normalization are effectively used to stabilize training
and improve the performance of GANs in existing literature [31, 49, 50]. Due
to diverse nature of the topic, there is a need for systematic literature
survey. There exist some literature studies [51, 28, 52], however, these
studies either lack comprehensive coverage of the topic, miss detailed
background information and theoretical analysis, or do not correlate different
methods. In this paper, based on the different perspectives of GANs training,
we propose a new taxonomy, denoted as "Training dynamics", "Fitting
distribution", "Real & Fake", and "Other methods", for a better understanding
of regularization and normalization during the GANs training as depicted in
Figure 1. Furthermore, we divide each group into different categories based on
implementation strategies. Specifically, due to non-convexity, global
convergence is hard to achieve during GANs’ training. Hence, Jacobian
regularization methods have been proposed to achieve local convergence in
"Training dynamics"; Gradient penalty, weight normalization, and weight
regularization are used to fulfill Lipschitz continuity and ensure training
stability of discriminator in "Fitting distribution"; Data augmentation and
preprocessing111Data augmentation and preprocess introduce additional data and
prior, which is similar to regularization. More importantly, both consistency
regularization and self-supervision need different data transformation
operations. Hence, this paper also discusses some works on this., consistency
regularization, and self-supervision are proposed to mitigate overfitting,
improve the representation of discriminator, and avoid discriminator
forgetting by introducing additional supervised information and data in "Real
& Fake"; Finally, "Other methods" containing layer normalization and inverse
gradient penalty are used for conditional generation and easing mode collapse.

In summary, we make the following contributions in this survey:

  * •

Comprehensive analysis of GANs training. In this study, we analyze the GANs
training from three perspectives including "Real & Fake", "Fitting
distribution", and "Training dynamics". To the best of our knowledge, this
survey is the first in this domain with comprehensive analysis.

  * •

New taxonomy. Based on the analysis of GANs training from different
perspectives, we propose a novel taxonomy and contextualize the regularization
and normalization of GANs comprehensively.

  * •

Comparison and analysis. Following the taxonomy, we also provide quantitative
and qualitative analysis and comparison for each type of regularization and
normalization techniques, which has helped the researchers and practitioners
navigate this space.

The rest of this paper is organized as follows: Section 2 introduces the
background and different training perspectives of GANs. Section 3, 4, 5, and 6
describe regularization and normalization methods in different groups,
respectively. Furthermore, we investigate the regularization and normalization
techniques that have been frequently employed in SOTA GANs in Section 7 and
discuss the current problems and prospects for future work in Section 8.

##  II Background and Three Perspectives of GANs Training

GANs are two-player zero-sum games, where generator (G) and discriminator
($D$) try to optimize opposing loss functions to find the global Nash
equilibrium. In general, GANs can be formulated as follows:

$\mathop{\min}\limits_{\phi}\mathop{\max}\limits_{\theta}f(\phi,\theta)=\mathop{\min}\limits_{\phi}\mathop{\max}\limits_{\theta}\mathbb{E}_{x\sim
p_{r}}[g_{1}(D_{\theta}(x))]+\mathbb{E}_{z\sim
p_{z}}[g_{2}(D_{\theta}(G_{\phi}(z)))],$ (1)

where $\phi$ and $\theta$ are parameters of the generator $G$ and the
discriminator $D$, respectively. $p_{r}$ and $p_{z}$ represent the real
distribution and the latent distribution, respectively.

In this section, we first introduce the background of the regularization and
prior. Furthermore, we elaborate the training of GANs from three perspectives:
low level: the perspective of "Making sample real", middle level: the
perspective of "Fitting distribution", and high level: the perspective of
"Training dynamic".

###  II-A Regularization and Prior

Regularization is a technique to control the complexity of learning models. It
has been widely used in learning models. Weight decay [53] is a typical method
to minimize the square of weights together with the training loss in the
training of neural networks [54, 55], which can be used to improve
generalization. In Bayesian learning methods, regularization is termed as
prior distribution, such as relevance vector machine [56], probabilistic
classification vector machines [57, 58], and others [59]. Specifically, L2
regularization [34] is equivalent to introducing Gaussian prior to the
parameters, and L1 regularization [33] is equivalent to introducing Laplace
prior to the parameters. The theoretical connection between regularization and
prior information has been investigated in neural network ensembles research
[60, 61]. Regularization can not only control overfitting but also provide
other characteristics like semi-supervised assumptions [35], manifold
assumptions [36, 37], feature selection [38], low rank representation [39,
62], and consistency assumptions [48, 63].

###  II-B Low Level: The Perspective of "Making sample real"

In this level, GANs can be considered as "counterfeiters-police" competition,
where the generator (G) can be thought of counterfeiters, trying to produce
fake currency and use it undetected, while the discriminator (D) is analogous
to the police, trying to detect the counterfeit currency. This competition
drives both teams to upgrade their methods until the fake currency is
indistinguishable from the real ones. Generally, D estimates the real
probability of both real and fake samples, which is very similar to the bi-
classification task, while G generates fake samples similar to real ones.
Hence, the loss function in Eq (1) can be formulated as:

$\begin{split}\mathop{\min}\limits_{\phi}\mathop{\max}\limits_{\theta}f(\phi,\theta)&=\mathop{\min}\limits_{\phi}\mathop{\max}\limits_{\theta}\mathbb{E}_{x\sim
p_{r}}[\log(D_{\theta}(x))]\\\ &+\mathbb{E}_{z\sim
p_{z}}[\log(1-D_{\theta}(G_{\phi}(z)))],\end{split}$ (2)

where $f(\phi,\theta)$ is a binary cross-entropy function, commonly used in
binary classification problems. Eq (2) is proposed in original GAN [1] and can
be optimized by alternate training. The training of discriminator is:

$\mathop{\max}\limits_{\theta}\mathbb{E}_{x\sim
p_{r}}[\log(D_{\theta}(x))]+\mathbb{E}_{z\sim
p_{z}}[\log(1-D_{\theta}(G_{\phi}(z)))],$ (3)

which is the same as a bi-classification task between real images and
generated images. Hence, we can apply some techniques used in the
classification task, such as new loss functions and some regularization
methods, to the training of discriminator. Specifically, to overcome the
gradients vanishing problem, Mao et al. [16] propose the LSGANs which adopts
the least squares loss function for the discriminator. The least squares loss
function can move the fake samples toward the decision boundary even though
they are correctly classified. Based on this property, LSGANs is able to
generate samples that are closer to real ones. Then the loss functions of
LSGANs can be defined as follows:

$\displaystyle\mathop{\min}\limits_{\theta}\mathbb{E}_{x\sim
p_{r}}[(D_{\theta}({x})-b)^{2}]+\mathbb{E}_{{z}\sim
p_{{z}}}[(D_{\theta}(G_{\phi}({z}))-a)^{2}],$ (4)
$\displaystyle\mathop{\min}\limits_{\phi}\mathbb{E}_{z\sim
p_{z}}[(D_{\theta}(G_{\phi}({z}))-c)^{2}],$

where $b$ and $a$ are values that $D$ for the training of real and fake
samples respectively, $c$ denotes the value that $G$ wants $D$ to believe for
fake sample. Gradients vanishing problem of the LSGANs only appears with
$D_{\theta}(G_{\phi}({z}))=c$, which is hard. Furthermore, Lin et al. [64] use
SVM separating hyperplane that maximizes the margin. They use the Hinge loss
to train the models, which can be formulated as:

$\displaystyle\mathop{\min}\limits_{\theta}\mathbb{E}_{x\sim
p_{r}}[(1-D_{\theta}({x})]_{+}+\mathbb{E}_{{z}\sim
p_{{z}}}[1+D_{\theta}(G_{\phi}({z}))]_{+},$ (5)
$\displaystyle\mathop{\min}\limits_{\phi}-\mathbb{E}_{z\sim
p_{z}}D_{\theta}(G_{\phi}({z})),$

where $[x]_{+}=\mathop{\max}\\{0,x\\}$.

The motivation of GANs is to train the generator based on the output of the
discriminator. Unlike the direct training objective of the classification task
(Minimizing cross-entropy loss), the objective of generator is indirect (With
the help of the discriminator output). Hence, discriminator should provide a
richer representation on the truth or false of samples compared to the
classification task. More prior information and additional supervision tasks
are urgent during the training process of the discriminator. Based on these,
some regularization methods, such as data augmentation and preprocessing,
consistency regularization, and self-supervision are proposed to improve the
stability and generalizability [65] of discriminator.

###  II-C Middle Level: The Perspective of "Fitting distribution"

At this level, generator $G(z)$ is considered as a distribution mapping
function that maps latent distribution $z$ to generated distribution
$P_{g}(x)$, and the discriminator $D(x)$ is a distribution distance that
evaluates the distance between the target distribution $P_{r}(x)$ and the
generated distribution $P_{g}(x)$. The framework can be illustrated in Fig.2.
For the optimal discriminator, the generator $G(z)$ tries to minimize the
distance between $P_{r}(x)$ and $P_{g}(x)$. Specifically, generator of the
vanilla GAN222Vanilla GAN, also known as standard GAN, is the first GAN model.
[1] and $f$-GAN333 $f$-GAN is a collective term for a type of GAN models whose
discriminator minimizes $f$ divergence. $f$ divergence is the general form of
KL divergence. It can be demonstrated as: $D_{f}(P||Q)=\int
q(x)f\big{(}\frac{p(x)}{q(x)}\big{)}\rm{d}x$, where $f$ is a mapping function
from non-negative real numbers to real numbers
($\mathbb{R}^{*}\rightarrow\mathbb{R}$) that satisfies: (1) $f(1)=0$. (2) $f$
is a convex function. To be more specific, KL divergence corresponds to
$f(u)=u\log u$ and JS divergence corresponds to
$f(u)=-\frac{u+1}{2}\log\frac{1+u}{2}+\frac{u}{2}\log u$. More details can be
viewed in [17] [17] can be considered to minimize Jensen–Shannon (JS)
divergence and $f$ divergence3, respectively. When the conditions of LSGANs
loss are set to $b-c=1$ and $b-a=2$, the generator of the LSGAN can be
considered to minimize the Pearson ${\chi}^{2}$ divergence. Also, generator of
the WGAN-div444Different from WGAN-div, WGAN [18] minimize Wasserstein
distance, not Wasserstein divergence. [66] and GAN-QP [67] can be considered
to minimize the Wasserstein divergence and Quadratic divergence, respectively.

Figure 2: The framework of GANs. $P_{z}$ is a latent space distribution,
$P_{r}$ and $P_{g}$ represent the real distribution and the generated
distribution, respectively.

For the GANs that is defined by Eq (1), Different choices of $g_{1}(t)$ and
$g_{2}(t)$ lead to different GAN models. Specifically, vanilla GAN [1] can be
described with $g_{1}(t)=g_{2}(-t)=-\log(1+e^{-t})$, $f$-GAN [17] and WGAN
[18] can be written as $g_{1}(t)=-e^{-t},g_{2}(t)=1-t$ and
$g_{1}(t)=g_{2}(-t)=t$, respectively.

Generator is a transportation map from $z$ to $p_{g}(x)$. In this section, we
introduce the optimal transport and the optimal transport with regular term,
which can lead to the form of Wasserstein GANs with gradient penalty [31]
(WGAN-GP) and Wasserstein GANs with Lipschitz penalty [49] (WGAN-LP),
respectively. Wasserstein distance is the most popular distance in GANs and it
corresponds to the optimal transport of the generator. To solve the dual
problem of Wasserstein distance, Lipschitz continuity is introduced, which is
the reason why gradient penalty and weight normalization technologies are
proposed in the GANs training. The following is a more detailed description.

Optimal transport [68] was proposed in the 18th century to minimize the
transportation cost while preserving the measure quantities. Given the space
with probability measures $(X,\mu)$ and $(Y,\upsilon)$, if there is a map
$T:X\rightarrow Y$ which is measure-preserving, then for any $B\subset Y$,
having:

$\int_{T^{-1}(B)}\mathrm{d}\mu(x)=\int_{B}\mathrm{d}\upsilon(y).$ (6)

Writing the measure-preserving map as $T_{*}(\mu)=\upsilon$. For any $x\in X$
and $y\in Y$, the transportation distance can be defined as $c(x,y)$, then the
total transportation cost is given by:

$C(T):=\int_{X}c(x,T(x))\mathrm{d}\mu(x).$ (7)

In the 18th century, Monge proposed the Optimal Mass Transportation Map that
corresponds to the smallest total transportation cost: $C(T)$. The
transportation cost corresponding to the optimal transportation map is called
the Wasserstein distance between probability measures $\mu$ and $\upsilon$:

$W_{c}(\mu,\upsilon)=\mathop{\min}\limits_{T}\left\\{\int_{X}c(x,T(x))\mathrm{d}\mu(x)\
|\ T_{*}(\mu)=\upsilon\right\\}.$ (8)

In 1940s, Kantorovich proved the existence and uniqueness of the solution for
Monge problem [69], and according to the duality of linear programming, he
obtained the Kantorovich-Rubinstein (KR) duality of Wasserstein distance:

$W_{c}(\mu,\upsilon)=\mathop{\max}\limits_{\varphi,\psi}\left\\{\int_{X}\varphi\mathrm{d}\mu\
+\int_{Y}\psi\mathrm{d}\upsilon\ |\ \varphi(x)+\psi(y)\leq c(x,y)\right\\}.$
(9)

This dual problem is constrained, defining the c-transform:
$\psi(y)=\varphi^{c}(y):=inf_{x}\\{c(x,y)-\varphi(x)\\}$, and the Wasserstein
distance becomes:

$W_{c}(\mu,\upsilon)=\mathop{\max}\limits_{\varphi}\left\\{\int_{X}\varphi\mathrm{d}\mu\
+\int_{Y}\varphi^{c}\mathrm{d}\upsilon\right\\},$ (10)

where $\varphi$ is called the Kantorovich potential. It can be shown that if
$c(x,y)=|x-y|$ and Kantorovich potential satisfies the 1-Lipschitz continuity,
then $\varphi^{c}=-\varphi$. At this time, Kantorovich potential can be fitted
by a deep neural network, which is recorded as $\varphi_{\xi}$. Wasserstein
distance is:

$W_{c}(\mu,\upsilon)=\mathop{\max}\limits_{||\varphi_{\xi}||_{L}\leq
1}\left\\{\int_{X}\varphi_{\xi}\mathrm{d}\mu\
-\int_{Y}\varphi_{\xi}\mathrm{d}\upsilon\right\\}.$ (11)

If $X$ is the generated image space, $Y$ is the real sample space, $Z$ is
latent space and $g_{\theta}$ is the geneartor, then the Wasserstein GANs
(WGAN) can be formulated as a min-max problem:

$\mathop{\min}\limits_{\theta}\mathop{\max}\limits_{||\varphi_{\xi}||_{L}\leq
1}\left\\{\int_{Z}\varphi_{\xi}(g_{\theta}(z))\mathrm{d}z\
-\int_{Y}\varphi_{\xi}(y)\mathrm{d}y\right\\}.$ (12)

In the optimization process, the generator and the Kantorovich potential
function (discriminator) are independent of each other, which can be optimized
in a step-by-step iteration.

If $c(x,y)=\frac{|x-y|^{2}}{2}$, there is a convex function $u$ that is called
Brenier potential [70]. The optimal transportation map is given by the
gradient map of Brenier potential: $T(x)=\nabla u(x)$. There exists a
relationship between Kantorovich potential and Brenier potential [71]:

$u(x)=\frac{|x|^{2}}{2}-\varphi(x).$ (13)

From the previous discussion, we know that the optimal transportation map
(Brenier potential) corresponds to the generator, and Kantorovich potential
corresponds to the discriminator. After the discriminator is optimized, the
generator can be directly derived without having to go through the
optimization process [71].

The transportation cost of Eq (3) can be defined as the form of two
distribution distances:

$OT(P||Q)=\mathop{inf}\limits_{\pi}\int\pi(x,y)c(x,y)\mathrm{d}x\mathrm{d}y,$
(14)

where $\pi(x,y)$ is the joint distribution, satisfying
$\int_{y}\pi(x,y)dy=P(x)$ and $\int_{x}\pi(x,y)dx=Q(y)$. The dual form of Eq
(10) is derived as follows::

$\begin{split}OT(P||Q)=&\mathop{\max}\limits_{\varphi,\psi}\\{\int_{x}\varphi(x)P(x)\mathrm{d}x\
\\\ &+\int_{y}\psi(y)Q(y)\mathrm{d}y\ |\ \varphi(x)+\psi(y)\leq
c(x,y)\\}.\end{split}$ (15)

Considering the optimal transportation with regular terms, Peyré et al. [72]
added the entropic regularization for optimal transportation that transforms
the dual problem into a smooth unconstrained convex problem. The regularized
optimal transport is defined as:

$OT_{c}(P||Q)=\mathop{\min}\limits_{\pi}\int\pi(x,y)c(x,y)\mathrm{d}x\mathrm{d}y+\epsilon
E(\pi).$ (16)

If
$E(\pi)=\int_{x}\int_{y}\pi(x,y)\log(\frac{\pi(x,y)}{P(x)Q(y)})\mathrm{d}x\mathrm{d}y$,
Eq (12) can be written as:

$\begin{split}OT_{c}(P||Q)=&\mathop{\min}\limits_{\pi}\int\pi(x,y)c(x,y)\mathrm{d}x\mathrm{d}y+\\\
&\epsilon\int_{x}\int_{y}\pi(x,y)\log\left(\frac{\pi(x,y)}{P(x)Q(y)}\right)\mathrm{d}x\mathrm{d}y\\\
s.t.\int_{y}\pi(x,y)&\mathrm{d}y=P(x),\int_{x}\pi(x,y)\mathrm{d}x=Q(y).\end{split}$
(17)

The dual form of Eq (13) becomes:

$\begin{split}OT_{c}(P||Q)&=\mathop{\max}\limits_{\varphi,\psi}\int_{x}\varphi(x)P(x)\mathrm{d}x\
+\int_{y}\psi(y)Q(y)\mathrm{d}y\\\
&+\frac{\epsilon}{e}\int_{x}\int_{y}\exp\left(\frac{-\left(c(x,y)+\varphi(x)+\psi(y)\right)}{\epsilon}\right)\mathrm{d}x\mathrm{d}y.\end{split}$
(18)

This section introduces the optimal transportation and optimal transportation
with regular terms for WGAN and Lipschitz continuity, an important property
for optimal transportation. It is pertinent to note that gradient penalty is a
simple and effective way to implement the Lipschitz continuity.

###  II-D Lipschitz Continuity and Matrix Norm

WGAN is the most popular generative adversarial network. From the optimal
transport in the last subsection, to obtain Eq (11), the discriminator must
satisfy the 1-Lipschitz continuity. This section introduces the form of the
Lipschitz constant and shows that the spectral norm and the Lipschitz constant
have the same meaning.

1-Lipschitz continuity can be represented as:

$||D(x_{1})-D(x_{2})||\leq||x_{1}-x_{2}||.$ (19)

Generally, considering the K-Lipschitz for a neural network $f(x)$:

$f(x)=g_{N}\circ\cdots g_{2}\circ g_{1}(x),$ (20)

where $g_{i}(x)=\sigma(W_{i}x+b_{i})$. And K-Lipschitz continuity for $f(x)$
is:

$||f(x_{1})-f(x_{2})||\leq\mathrm{K}||x_{1}-x_{2}||,$ (21)

where K is Lipschitz constant of the function $f$. Due to the consistency of
Lipschitz $||h\circ g||_{Lip}\leq||h||_{Lip}\cdot||g||_{Lip}$, $g_{i}$ needs
to satisfy the C-Lipschitz continuity ($\mathrm{C}=\sqrt[N]{\mathrm{K}}$) so
that $f$ can satisfy the K-Lipschitz continuity:

$||g_{i}(x_{1})-g_{i}(x_{2})||\leq\mathrm{C}||x_{1}-x_{2}||,$ (22)
$||\sigma(Wx_{1}+b)-\sigma(Wx_{2}+b)||\leq\mathrm{C}||x_{1}-x_{2}||.$ (23)

When $x_{1}\rightarrow x_{2}$, the Taylor expansion of Eq (19):

$||\frac{\partial\sigma}{\partial
x}W(x_{1}-x_{2})||\leq\mathrm{C}||x_{1}-x_{2}||.$ (24)

Normally, $\sigma$ is a function with limited derivatives such as Sigmoid, so
the $\mathrm{C^{\prime}}$-Lipschitz continuity can be written as:

$||W(x_{1}-x_{2})||\leq\mathrm{C^{\prime}}||x_{1}-x_{2}||,$ (25)

where $\mathrm{C^{\prime}}$ is a limited constant, which is determined by
$\frac{\partial\sigma}{\partial x}$ and $\mathrm{C}$. Similarly, the spectral
norm of matrix is defined by:

$||W||_{2}=\mathop{\max}\limits_{x\not=0}\frac{||Wx||}{||x||}.$ (26)

From now, the spectral norm $||W||_{2}$ can be used to represent the Lipschitz
constant $\mathrm{C^{\prime}}$. The Lipschitz continuity can be achieved by
normalizing the spectral norm of the weight, approximately. Hence, weight
normalization and weight regularization can also be used to enable the
Lipschitz continuity of the discriminator.

###  II-E High Level: The Perspective of "Training dynamic"

GANs training is a two-player zero-sum game, which exists the Nash equilibrium
solution. At this level, we analyze the convergence of GANs by understanding
the optimization process. Based on these, some regularization technologies are
proposed to guide the GANs model to reach the theoretical equilibrium point,
and thus improve the effectiveness of GANs.

Reconsidering the Eq (1) in Section 2, the training of GANs is achieved by
solving a two-player zero-sum game via Simultaneous Gradient Descent (SimGD)
[1, 18]. The updates of the SimGD are given as:

$\begin{split}\phi^{(k+1)}=\phi^{(k)}-h\nabla_{\phi}f(\phi^{(k)},\theta^{(k)}),\\\
\theta^{(k+1)}=\theta^{(k)}+h\nabla_{\theta}f(\phi^{(k)},\theta^{(k)}).\end{split}$
(27)

Assuming that the objectives of GANs are convex, many research works discussed
their global convergence characteristics [17, 75]. However, due to the high
non-convexity of deep networks, even a simple GAN does not satisfy the
convexity assumption [76]. A recent study [77] shows that it is unrealistic to
obtain approximate global convergence under the assumption of the optimal
discriminator, so the community considers local convergence. It hopes that the
trajectory of the dynamic system can enter a local convergence point with
continuity iterations, that is, Nash equilibrium:

$\begin{split}\bar{\phi}=\mathop{\arg\max}_{\phi}-f(\phi,\bar{\theta}),\\\
\bar{\theta}=\mathop{\arg\max}_{\theta}f(\bar{\phi},\theta).\end{split}$ (28)

If the point $(\bar{\phi},\bar{\theta})$ is called the local Nash-equilibrium,
Eq (28) holds in a local neighborhood of $(\bar{\phi},\bar{\theta})$. For this
differentiable two-player zero-sum games, a vector is defined as below:

$v(\phi,\theta)=\left(\begin{aligned} -\nabla_{\phi}f(\phi,\theta)\\\
\nabla_{\theta}f(\phi,\theta)\end{aligned}\right).$ (29)

The Jacobian matrix is:

$v^{{}^{\prime}}(\phi,\theta)=\left(\begin{aligned}
-&\nabla^{2}_{\phi,\phi}f(\phi,\theta)\quad-\nabla^{2}_{\phi,\theta}f(\phi,\theta)\\\
&\nabla^{2}_{\phi,\theta}f(\phi,\theta)\quad\nabla^{2}_{\theta,\theta}f(\phi,\theta)\end{aligned}\right).$
(30)

Lemma 2.1: For zero-sum games, $v^{{}^{\prime}}$is negative semi-definite for
any local Nash-equilibrium. Conversely, if $v(\bar{x})=0$ and
$v^{{}^{\prime}}$is negative definite, then $\bar{x}$ is a local Nash-
equilibrium.

Proof 2.1: Refer to [19] $\square$

Lemma 2.1 [19] gives the conditions for the local convergence of GANs, which
is converted into the negative semi-definite problem of the Jacobian matrix.
Negative semi-definite of the Jacobian matrix corresponds to its eigenvalue
less than or equal to 0. If the eigenvalue of the Jacobian matrix at a certain
point is a negative real number, the training process can converge; but if the
eigenvalue is complex and the real part of the eigenvalue is small and the
imaginary part is relatively large, the training process is difficult to
converge unless the learning rate is very small.

Proposition 2.2: Let $F:\Omega\rightarrow\Omega$ be a continuously
differentiable function on an open subset $\Omega$ of $R^{n}$ and let
$\bar{x}\in\Omega$ be so that:

1\. $F(\bar{x})=\bar{x}$ and

2\. the absolute values of the eigenvalues of the Jacobian
$F^{{}^{\prime}}(x)$ are all smaller than 1.

There is an open neighborhood $U$ of $\bar{x}$ so that for all $x_{0}\in U$,
the iterates $F^{(k)}(x_{0})$ converge to $\bar{x}$. The rate of convergence
is at least linear. More precisely, the error $||F^{(k)}(x_{0})-\bar{x}||$ is
in $\mathcal{O}(|\lambda_{max}|^{k})$ for $k\rightarrow\infty$ where
$\lambda_{max}$ is the eigenvalue of $F^{{}^{\prime}}(\bar{x})$ with the
largest absolute value.

Proof 2.2: Refer to Section 3 in [19] and Proposition 4.4.1 in [78]. $\square$

From Proposition 2.2: Under the premise of asymptotic convergence, the local
convergence of GAN is equivalent to the absolute value of all eigenvalues of
the Jacobian matrix at the fixed point $(v(\bar{\phi},\bar{\theta})=0)$ being
less than 1. To get this condition, some Jacobian regularization [19, 79, 80,
81] need to be used.

##  III Regularization and Normalization of "Training dynamic"

Assuming the objectives of GANs are convex-concave, some works have provided
the global convergence of GANs [17, 82]. However, these theoretical
convergence analyses can only be applied to the GANs with existing the optimal
discriminator. It is not capable to achieve the optimal discriminator, thus,
more works focus on analyzing the local convergence of GANs. According to
Nagarajan et al. [79] and Mescheder et al. [19], under some assumptions, GANs
dynamics are locally convergent. However, if these assumptions are not
satisfied, especially if the data distributions are not continuous, GANs
dynamics can not always be locally convergent unless some regularization
technologies are used.

Initially, Jacobian regularization technologies [19, 79], minimizing Jacobi
matrix, are proposed to achieve local convergence. In this section, we survey
these methods. Subsequently, Mescheder et al. [80] propose a simplified
gradient penalties method, named zero-centered gradient penalties (zc-GP),
that guarantees the local convergence under suitable assumptions. Since it is
similar to 0-GP, we will cover it in detail in Section 4.

###  III-A Jacobian Regularization

In Proposition 2.2 of Section 2: absolute values of all eigenvalues of the
Jacobian matrix of the $V(\phi,\theta)$ are expected to be less than 1 at the
fixed point, which is equivalent to the real part of the eigenvalue being
negative. Additionally, the learning rate must be relatively low [19]. To meet
these requirements, Mescheder et al. [19] used the Consensus Optimization
(ConOpt) to make the real part of the eigenvalue negative. Its regularized
updates are:

$\begin{split}\phi^{(k+1)}=\phi^{(k)}+h\nabla_{\phi}\left(-f(\phi^{(k)},\theta^{(k)})-\gamma
L(\phi^{k},\theta^{k})\right),\\\
\theta^{(k+1)}=\theta^{(k)}+h\nabla_{\theta}\left(f(\phi^{(k)},\theta^{(k)})-\gamma
L(\phi^{k},\theta^{k})\right),\end{split}$ (31)

where
$L(\phi^{k},\theta^{k})=\frac{1}{2}||v(\phi^{k},\theta^{k})||^{2}=\frac{1}{2}\left(||\nabla_{\phi}f(\phi^{k},\theta^{k})||^{2}+||\nabla_{\theta}f(\phi^{k},\theta^{k})||^{2}\right)$
is the regularization of the Jacobian matrix.

Apart from Mescheder et al, Nagaraja et al. [79] also analyze the relationship
between local convergence of GANs and all eigenvalues of the Jacobian of the
gradient vector field. They prove the local convergence for absolutely
continuous generator and data distributions under certain regularity
assumptions. This requires the loss function of the GANs to be strictly
concave, which is not the case for some GANs. Based on this, a simple
regularization technology that regularized the generator using the gradient of
the discriminator is proposed by Nagaraja et al. [79]. The regularized updates
for the generator can be expressed as:

$\phi^{(k+1)}=\phi^{(k)}-h\nabla_{\phi}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\phi}||\nabla_{\theta}f(\phi^{k},\theta^{k})||^{2}.$
(32)

Herein, the update of the discriminator is similar to SimGD. Furthermore, Nie
et al. [76] propose a method that only regularizes the discriminator. The
regularized update of the discriminator in this case is given by:

$\theta^{(k+1)}=\theta^{(k)}+h\nabla_{\theta}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\theta}||\nabla_{\phi}f(\phi^{k},\theta^{k})||^{2}.$
(33)

The update of the generator is the same as SimGD. Nie et al. [76] propose
JAcobian REgularization (JARE) that regularizes both the generator and the
discriminator. The regularized updates for the generator and the discriminator
are:

$\begin{split}\phi^{(k+1)}=\phi^{(k)}-h\nabla_{\phi}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\phi}||\nabla_{\theta}f(\phi^{k},\theta^{k})||^{2},\\\
\theta^{(k+1)}=\theta^{(k)}+h\nabla_{\theta}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\theta}||\nabla_{\phi}f(\phi^{k},\theta^{k})||^{2}.\end{split}$
(34)

The key difference between JARE and ConOpt is that JARE does not contain the
Hessians $\nabla^{2}_{\phi,\phi}f(\phi^{k},\theta^{k})$ and
$\nabla^{2}_{\theta,\theta}f(\phi^{k},\theta^{k})$ in the regularization term,
which avoids two factors777Intuitively, a reason for not introducing Hessians
is to avoid the risk of reversing the gradient flows, which may diverge the
GAN training dynamics (see Appendix C in [76] for a detailed explanation). of
the Jacobian in the GANs training dynamics simultaneously. (i) the Phase
Factor, i.e., the Jacobian has complex eigenvalues with a large imaginary-to-
real ratio; (ii) the Conditioning Factor, i.e., the Jacobian is ill-
conditioned.

The above discussions of local convergence during GANs training involve a
premise: absolutely continuous data and generator distributions. Indeed, the
assumption of absolute continuity is not true for common cases of GANs, where
both distributions, specially the data distribution, may lie on lower-
dimensional manifolds [83]. More generally, Mescheder et al. [80] extend the
convergence proof by Nagaraja et al. [79] to the case where the generator and
data distribution do not locally have the same support. Based on this, a
simplified zero-centered gradient penalties (zc-GP) method is proposed, which
guarantees the local convergence under suitable assumptions. Zc-GP is obtained
from the training dynamic, which is similar to 0-GP methods mentioned in
Section 4. We will cover it in detail in that section.

In summary, Jacobian regularization technologies are obtained from the
training dynamic of GANs, which are used for achieving local convergence. The
summary of the Jacobian regularization methods is demonstrated oin Table I.
From the form of the updating, Jacobian regularization is similar to the
Gradient penalty. In general, zc-GP is used in many SOTA methods, which are
demonstrated in Section 7.

TABLE I: The summary of the Jacobian regularization Method | regularized updates of generator | regularized updates of discriminator
---|---|---
SimGD [1] | $\phi^{(k+1)}=\phi^{(k)}-h\nabla_{\phi}f(\phi^{(k)},\theta^{(k)})$ | $\theta^{(k+1)}=\theta^{(k)}+h\nabla_{\theta}f(\phi^{(k)},\theta^{(k)})$
ConOpt [19] | $\phi^{(k+1)}=\phi^{(k)}-h\nabla_{\phi}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\phi}||v(\phi^{(k)},\theta^{(k)})||^{2}$ | $\theta^{(k+1)}=\theta^{(k)}+h\nabla_{\theta}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\theta}||v(\phi^{(k)},\theta^{(k)})||^{2}$
Generator [79] | $\phi^{(k+1)}=\phi^{(k)}-h\nabla_{\phi}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\phi}||\nabla_{\theta}f(\phi^{(k)},\theta^{(k)})||^{2}$ | $\theta^{(k+1)}=\theta^{(k)}+h\nabla_{\theta}f(\phi^{(k)},\theta^{(k)})$
Discriminator [76] | $\phi^{(k+1)}=\phi^{(k)}-h\nabla_{\phi}f(\phi^{(k)},\theta^{(k)})$ | $\theta^{(k+1)}=\theta^{(k)}+h\nabla_{\theta}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\theta}||\nabla_{\phi}f(\phi^{(k)},\theta^{(k)})||^{2}$
JARE [76] | $\phi^{(k+1)}=\phi^{(k)}-h\nabla_{\phi}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\phi}||\nabla_{\theta}f(\phi^{(k)},\theta^{(k)})||^{2}$ | $\theta^{(k+1)}=\theta^{(k)}+h\nabla_{\theta}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\theta}||\nabla_{\phi}f(\phi^{(k)},\theta^{(k)})||^{2}$
zc-GP [80] | $\phi^{(k+1)}=\phi^{(k)}-h\nabla_{\phi}f(\phi^{(k)},\theta^{(k)})$ | $\theta^{(k+1)}=\theta^{(k)}+h\nabla_{\theta}f(\phi^{(k)},\theta^{(k)})-\frac{1}{2}h\gamma\nabla_{\theta}||\nabla D_{\theta}(x)||^{2}$

##  IV Regularization and Normalization of "Fitting distribution"

From the perspective of "Fitting distribution", generator is considered as a
distribution mapping function and the optimal discriminator is considered to
be the distribution distance. Among them, Wasserstein distance is the most
popular in GANs and it corresponds to the optimal transport of the generator.
To solve the dual problem of Wasserstein distance, Lipschitz continuity is
introduced, which is the first time that Lipschitz continuity has been
introduced into the training of GANs. The Wasserstein distance-based GANs
(WGAN and WGAN-GP) have achieved remarkable results during the training.
However, some works [84, 85, 21] suggest that the success of WGAN-GP is not
due to the Wasserstein distance and the Lipschitz constraint of discriminator
may improve the performance and stability of GANs training regardless of the
statistical distance used as a loss function. Based on these, the Lipschitz
continuity of discriminator is an essential condition during GANs training. As
acknowledged, gradient penalty, weight normalization, and weight
regularization are widely applied in GANs training for fulfilling Lipschitz
continuity, which will be summarized at this section.

###  IV-A Gradient Penalty

TABLE II: The Gradient penalty of the Discriminator. $\mu$ and $v$ are real and generated distribution, respectively. Method |  $\mathcal{L}_{GP}$ | $\pi$ | Lipschitz continuity
---|---|---|---
WGAN-GP [31] | $\mathbb{E}_{\hat{x}\sim\pi}(||\nabla D_{\theta}(\hat{x})||_{2}-1)^{2}$ | $t\cdot p_{r}+(1-t)\cdot p_{g}$ | $||D_{\theta}||_{Lip}\to 1$
DRAGAN [21] | $\mathbb{E}_{\hat{x}\sim\pi}(||\nabla D_{\theta}(\hat{x})||_{2}-1)^{2}$ | $p_{r}+\epsilon$ | $||D_{\theta}||_{Lip}\to 1$
Max-GP [86] | $\left(\mathop{\max}\limits_{\hat{x}\sim\pi}||\nabla D_{\theta}(\hat{x})||_{2}-1\right)^{2}$ | $t\cdot p_{r}+(1-t)\cdot p_{g}$ | $||D_{\theta}||_{Lip}\to 1$
ALP [87] | $\mathbb{E}_{\hat{x}\sim\pi}(||\nabla D_{\theta}(\hat{x})||_{2}-1)^{2}$ | $p_{r}\cup p_{g}$ | $||D_{\theta}||_{ALP-Lip}\to 1$
Banach-GP [88] | $\mathbb{E}_{\hat{x}\sim\pi}(||\nabla D_{\theta}(\hat{x})||_{B^{*}}-1)^{2}$ | $t\cdot p_{r}+(1-t)\cdot p_{g}$ | $||D_{\theta}||_{Lip}\to 1$
WGAN-LP [49] | $\mathbb{E}_{\hat{x}\sim\pi}\left[\left(\mathop{\max}\\{0,||\nabla D_{\theta}(\hat{x})||_{2}-1\\}\right)^{2}\right]$ | $t\cdot p_{r}+(1-t)\cdot p_{g}$ | $||D_{\theta}||_{Lip}\leq 1$
SWGAN [89] | $\mathbb{E}_{\hat{x}\sim\pi}\left[\left(\mathop{\max}\\{0,||\nabla D_{\theta}(\hat{x})||^{2}-1\\}\right)^{2}\right]$ | $t\cdot p_{r}+(1-t)\cdot p_{g}$ | $||D_{\theta}||_{Lip}\leq 1$
zc-GP [66, 80, 90] | $\mathbb{E}_{\hat{x}\sim\pi}||\nabla D_{\theta}(\hat{x})||^{2}_{2}$ | $p_{r}\cup p_{g}$ | $||D_{\theta}||_{Lip}\to 0$
GAN-QP [67] | $\mathcal{L}_{GP}=\mathbb{E}_{x_{r},x_{g}\sim\pi}\frac{\left(D_{\theta}(x_{r})-D_{\theta}(x_{f})\right)^{2}}{||x_{r}-x_{f}||}$ | $\pi=p_{r}\cdot p_{g}$ | $\frac{\left(D_{\theta}(x_{r})-D_{\theta}(x_{f})\right)^{2}}{||x_{r}-x_{f}||}\to 0$
ZP-Max [91] | $\mathop{\max}\limits_{\hat{x}\sim\pi}||\nabla D_{\theta}(\hat{x})||^{2}_{2}$ | $t\cdot p_{r}+(1-t)\cdot p_{g}$ | $||D_{\theta}||_{Lip}\to 0$
ZP [92] | $\mathbb{E}_{\hat{x}\sim\pi}||\nabla D_{\theta}(\hat{x})||^{2}_{2}$ | $t\cdot p_{r}+(1-t)\cdot p_{g}$ | $||D_{\theta}||_{Lip}\to 0$

Gradient penalty is a simple and direct way to fulfill Lipschitz continuity.
Specifically, K-Lipschitz continuity (mentioned in Eq (21)) of the function
$f$ can be accessed by $\mathop{\min}\mathbb{E}_{\hat{x}\sim\pi}(||\nabla
f(\hat{x})||_{2}-{\rm K})^{2}$. According to the optimal transport theory
mentioned in Section 2.3, gradient penalty can be used for the approximation
of $W_{c}(\mu,\upsilon)$ (Eq (11)) in WGANs, named WGAN-GP [31]. Specifically,
WGAN-GP fulfills the 1-Lipschitz continuity of the discriminator by
$\mathop{\min}\mathbb{E}_{\hat{x}\sim\pi}(||\nabla
D_{\theta}(\hat{x})||_{2}-{\rm 1})^{2}$, which limits the gradient of the
discriminator to 1. Although WGAN-GP solves the instability of GANs training
to some extent, the assumption of optimal transport is a constrained linear
programming problem. Overly strict restriction reduces the exploratory of the
discriminator.

In contrast, the optimal transport with the regular term mentioned in section
2.3 is an unconstrained optimization problem. Petzka et al. [49] set
$c(x,y)=||x-y||_{2}$ in Eq (18), and the dual form of optimal transport with
the regular term can be expressed as:

$\begin{split}&\mathop{\sup}\limits_{\varphi,\psi}\\{\mathbb{E}_{x\sim{p(x)}}[\varphi(x)]-\mathbb{E}_{y\sim
q(y)}[\psi(y)]\\\
&-\frac{4}{\epsilon}\int\int\mathop{\max}\\{0,(\varphi(x)-\psi(y)-||x-y||_{2})\\}^{2}\mathrm{d}p(x)\mathrm{d}q(y)\\}.\end{split}$
(35)

Similar to dealing with a single function, one can replace $\varphi=\psi$ in
Eq (35), which leads to the objective of minimum:

$\begin{split}&\mathbb{E}_{y\sim q(y)}[\varphi(y)]-\mathbb{E}_{x\sim
p(x)}[\varphi(x)]\\\
&+\frac{4}{\epsilon}\int\int\mathop{\max}\\{0,(\varphi(x)-\varphi(y)-||x-y||_{2})\\}^{2}\mathrm{d}p(x)\mathrm{d}q(y).\end{split}$
(36)

Similar to optimal transport corresponds to 1-Lipschitz continuity, the
optimal transport with the regular term corresponds to k-Lipschitz continuity
(${\rm k}\leq 1$) of the discriminator, named WGAN-LP [49], which is
implemented by
$\mathop{\min}\mathbb{E}_{\hat{x}\sim\pi}\left[\left(\mathop{\max}\\{0,||\nabla
D_{\theta}(\hat{x})||_{2}-1\\}\right)^{2}\right]$. WGAN-LP with using a weaker
regularization term enforcing the Lipschitz constraint of the discriminator
achieves the preferable performance.

WGAN-GP and WGAN-LP introduce Wasserstein distance into GANs framework. Due to
the gap between limited input samples and the strict Lipschitz constraint on
the whole input sample domain, the approximate of the Wasserstein distance is
challenging. To this end, WGAN-div [66] introduces a Wasserstein divergence
into GANs training. The objective of WGAN-div can be smoothly derived as:

$\mathbb{E}_{y\sim q(y)}[\varphi(y)]-\mathbb{E}_{x\sim
p(x)}[\varphi(x)]+k\mathbb{E}_{\hat{x}\sim\pi}\left[||\varphi(\hat{x})||^{p}\right].$
(37)

The objective of WGAN-div is similar to WGAN-GP and WGAN-LP. It can be
considered as achieving 0-Lipschitz continuity of discriminator by adopting
$\mathop{\min}\mathbb{E}_{\hat{x}\sim\pi}\left[||\nabla
D_{\theta}(\hat{x})||^{p}\right]$.

Generally, Wasserstein distance and Wasserstein divergence, reliable ways of
measuring the difference between fake and real data distribution, lead to
stable training of WGAN-based algorithms. However, recent work [84] shows that
the c-transform method [93] achieves a much better estimation of Wasserstein
divergence but obtains worse performance compared to the gradient penalty
method. The results demonstrate that the success of WGAN-based methodologies
can not in truth be attributed to approximate the Wasserstein distance and the
gradient penalty methods improve the performance indeed. Furthermore, some
works [85, 21, 94] also demonstrate that gradient penalty methods of
discriminator, such as 1-GP, k-GP (${\rm k}\leq 1$), and 0-GP stabilize the
training and improve the performance of GANs remarkably regardless of the loss
functions. Based on these observations, stabilizing GANs training using
gradient penalty is widely applied by the community for various losses of
GANs. In the rest of this section, we will discuss gradient penalty methods
regardless of the loss function and divide them into three parts: 1-GP:
$\mathop{\min}\mathbb{E}_{\hat{x}\sim\pi}(||\nabla D_{\theta}(\hat{x})||-{\rm
1})^{p}$, k-GP (${\rm k}\leq 1$):
$\mathop{\min}\mathbb{E}_{\hat{x}\sim\pi}\left[\left(\mathop{\max}\\{0,||\nabla
D_{\theta}(\hat{x})||-1\\}\right)^{p}\right]$, and 0-GP:
$\mathop{\min}\mathbb{E}_{\hat{x}\sim\pi}\left[||\nabla
D_{\theta}(\hat{x})||^{p}\right]$, where $\pi$ is the distribution of
different image space (entire image space or part of image space) and
$||\cdot||$ represents the norm of the gradient. Generally, the loss function
of the discriminator with GP can be formulated as:

$\mathcal{L}_{D}=f(\phi,\theta)+\lambda\mathcal{L}_{GP},$ (38)

where $f(\phi,\theta)$ is the uniform loss function defined in Eq (1) and
$\mathcal{L}_{GP}$ is the gradient penalty regularization.

####  IV-A1 1-GP

1-GP is first applied by Gulrajani et al. [31] during the training of GANs,
named WGAN-GP. WGAN-GP uses the 2-norm gradient penalty across the entire
image domain, which can be formulated as:

$\mathcal{L}_{GP}=\mathbb{E}_{\hat{x}\sim\pi}(||\nabla
D_{\theta}(\hat{x})||_{2}-1)^{2},$ (39)

where $\pi$ is the distribution of entire image space approximated by the
interpolation of real distribution ($p_{r}$) and generated distribution
($p_{g}$): $\pi=t\cdot p_{r}+(1-t)\cdot p_{g}$ for $t\sim U[0,1]$. Although,
WGAN-GP stabilizes the training of GANs astonishingly, the overly strict
gradient penalty limits the exploratory of discriminator. To loosen the
penalty, many efforts of $\pi$, $||\cdot||$, and gradient direction are
proposed.

For relaxing the image distribution, Kodali et al. [21] track the training
process of GANs and find that the decrease of the Inception Score (IS) is
accompanied by a sudden change of the discriminator’s gradient around the real
images. They propose DRAGAN and only restrict the Lipschitz constant around
the real images $\pi=p_{r}+\epsilon$, where $\epsilon\sim N_{d}(0,cI)$.

For relaxing the gradient direction, Zhou [86] et al. believe that restricting
the global Lipschitz constant might be unnecessary. Therefore, only maximum
gradient is necessary to be penalized:

$\mathcal{L}_{GP}=\left(\mathop{\max}\limits_{\hat{x}\sim\pi}||\nabla
D_{\theta}(\hat{x})||_{2}-1\right)^{2},$ (40)

where $\pi=t\cdot p_{r}+(1-t)\cdot p_{g}$; Furthermore, inspired by Virtual
Adversarial Training (VAT) [95], Dávid et al. [87] propose a method, called
Adversarial Lipschitz Regularization (ALR), which restricts the 1-Lipschitz
continuity at $\pi=p_{r}\cup p_{g}$ with the direction of adversarial
perturbation. The proposed ALP shows the SOTA performance in terms of
Inception Score and Fréchet Inception Distance among non-progressive growing
methods trained on CIFAR-10 dataset.

Different from the above methods penalizing the gradient in the Euclidean
space, Adler et al. [88] extended the $L_{p}(p=2)$ space with gradient penalty
to Banach space that contains the $L_{p}$ space and Sobolev space. For the
Banach space B, the Banach norm $||.||_{B}^{*}$ can be defined as:

$||x^{*}||_{B^{*}}=\sup\limits_{x\in B}\frac{x^{*}(x)}{||x||_{B}}.$ (41)

Thus, the gradient penalty of Banach wasserstein GAN can be expressed as:

$\mathcal{L}_{GP}=\mathbb{E}_{\hat{x}\sim\pi}(||\nabla
D_{\theta}(\hat{x})||_{B^{*}}-1)^{2},$ (42)

where $\pi=t\cdot p_{r}+(1-t)\cdot p_{g}$.

####  IV-A2 k-GP (${\rm k}\leq 1$)

k-GP (${\rm k}\leq 1$) is first applied by Gulrajani et al. [49] during the
training of GANs, named WGAN-LP. WGAN-LP also uses the 2-norm gradient penalty
across the entire image domain, which can be formulated as:

$\mathcal{L}_{GP}=\mathbb{E}_{\hat{x}\sim\pi}\left[\left(\mathop{\max}\\{0,||\nabla
D_{\theta}(\hat{x})||_{2}-1\\}\right)^{2}\right],$ (43)

where $\pi$ is the distribution of entire image space approximated by the
interpolation of real distribution ($p_{r}$) and generated distribution
($p_{g}$): $\pi=t\cdot p_{r}+(1-t)\cdot p_{g}$ for $t\sim U[0,1]$.
Furthermore, Xu et al [89] show a more general dual form of the Wasserstein
distance compared to KR duality (mentioned in section 2.3), named Sobolev
duality, which relaxes the Lipschitz constraint but still maintains the
favorable gradient property of the Wasserstein distance. They also show that
the KR duality is a special case of the proposed Sobolev duality. Based on the
Sobolev duality, the relaxed gradient penalty of the proposed SWGAN can be
formulated as:

$\mathcal{L}_{GP}=\mathbb{E}_{\hat{x}\sim\pi}\left[\left(\mathop{\max}\\{0,||\nabla
D_{\theta}(\hat{x})||^{2}-1\\}\right)^{2}\right],$ (44)

where $\pi=t\cdot p_{r}+(1-t)\cdot p_{g}$ for $t\sim U[0,1]$. It is clear that
both the WGAN-LP and the SWGAN have the same form of gradient penalty.
Different relaxation methods yield the same form of regularization, which is
exciting.

####  IV-A3 0-GP

0-GP is first implemented by Wu et al. [66], where Wasserstein divergence is
introduced. According to [96], Wasserstein divergence can be solved by
minimizing:

$\mathcal{L}_{GP}=\mathbb{E}_{\hat{x}\sim\pi}||\nabla
D_{\theta}(\hat{x})||^{2},$ (45)

where $\pi$ is both the real distribution ($p_{r}$) and the generated
distribution ($p_{g}$): $\pi=p_{r}\cup p_{g}$. Furthermore, Mescheder et al.
[80] also demonstrate that the optimization of unregularized GAN is not always
locally convergent and some simplified zero centered gradient penalty (zc-GP)
technologies, implemented by minimizing Eq (45), can be used to achieve local
convergence of GANs.

Besides, some other 0-GP methods [67, 90, 91, 92] are derived by different
theoretical derivations. Specifically, Su et al. [67] propose a Quadratic
Potential (QP) for GANs training, which can the formulated as:

$\mathcal{L}_{GP}=\mathbb{E}_{x_{r},x_{g}\sim\pi}\frac{\left(D_{\theta}(x_{r})-D_{\theta}(x_{f})\right)^{2}}{||x_{r}-x_{f}||},$
(46)

where $\pi$ is the joint distribution of the real and generated distributions:
$\pi=p_{r}\cdot p_{g}$; Zhang et al. [90] combine a Total Variational (TV)
regularizing term into the training of GANs, that is
$|D_{\theta}(x_{r})-D_{\theta}(x_{f})-\delta|$. According to the [90], the TV
term can be approximated by Eq (45), which is exhilarating; Zhou et al. [91]
propose the Lipschitz GANs, with the maximum of the gradients penalty for
guaranteeing the gradient informativeness:

$\mathcal{L}_{GP}=\mathop{\max}\limits_{\hat{x}\sim\pi}||\nabla
f(\hat{x})||^{2}_{2},$ (47)

where $\pi=t\cdot p_{r}+(1-t)\cdot p_{g}$; Thanh-Tung et al. [92] also propose
the 0-GP with gradients penalty at $\pi=t\cdot p_{r}+(1-t)\cdot p_{g}$:

$\mathcal{L}_{GP}=\mathbb{E}_{\hat{x}\sim\pi}||\nabla f(\hat{x})||^{2}_{2}.$
(48)

In summary, gradient penalty technologies are widely used in the GANs training
to achieve Lipschitz continuity of discriminator. As shown in TABLE II, too
many similar techniques are proposed based on different theories and
phenomena. But as far as we know, there is no fair and comprehensive work
comparing the performance of these gradient penalty methods. To compare the
performance of various methods intuitively, a comparative experiment on the
CIFAR-10 dataset is conducted.

To compare the performance of various methods intuitively, a comparative
experiment on the CIFAR10 dataset is conducted. The results of its Inception
Score (IS) [97, 98] and FID [20] are shown in TABLE III. The results validate
the conclusion in studies [84, 85, 21], that the Lipschitz constraint of
discriminator may improve the performance and stability of GANs training
regardless of the statistical distance used as a loss function. All gradient
penalty methods improve the performance of GANs upon all three loss functions.
Among them, zc-GP [66, 80, 90] obtains the best performance and is widely used
in SOTA methods as illustrated in Table X.

TABLE III: The experimental results of different Gradient penalty Method | Inception Score | FID
---|---|---
WGAN-GP [31] | 7.869 | 16.62
WGAN-LP [49] | 7.98 | 15.85
WGAN-ALP [87] | 8.247 | 14.19
WGAN-GP-Max [86] | 7.956 | 18.43
WGAN-ZP-Max [91] | 7.908 | 17.97
WGAN-ZP-Sample [80] | 8.013 | 15.87
WGAN-ZP [92] | 7.957 | 16.08
TABLE IV: The summary of the weight normalization and wight regularization Method | Implementation | Motivation
---|---|---
Spectral normalization (SN) [46] | $W_{\sigma}=W/||W||_{2}$ | $||D||_{Lip}\to 1$
F normalization [46] | $W_{\sigma}=W/||W||_{F}$ | $||D||_{Lip}\leq 1$
Mixed normalization [99] | $W_{\sigma}=W/\sqrt{||W||_{1}||W||_{\infty}}$ | $||D||_{Lip}\leq 1$
Spectral increment normalization [100] | $W_{\sigma}=W/||W||_{2}+\nabla W/||W||_{2}$ | $||D||_{Lip}\to 1$
Off-Diagonal Orthogonal regularization (Off-Diagonal OR) [30] | $R_{o}(W)=\beta\left\|W^{\top}W\odot(\mathbf{1}-I)\right\|_{\mathrm{F}}^{2}$ | Smoothing G

###  IV-B Weight Normalization and Weight Regularization

####  IV-B1 Weight Normalization

Lipschitz continuity is important for GANs training. From Section 2.4,
spectral norm of the weight and the Lipschitz constant express the same
concept. To meet the Lipschitz continuity, in addition to the gradient
penalty, weight normalization can also be used. Spectral normalization of the
weight can limit the Lipschitz constant to 1. Certainly, upper bound of the
spectral norm can be used to normalize the weights, achieving $k(k\leq 1)$
Lipschitz continuity. The following lemmas put forward some upper bounds of
the spectral norm.

Lemma 3.1: If $\lambda_{1}\leq\lambda_{2}\leq\cdots\leq\lambda_{M}$ are the
eigenvalues of the $W^{\top}W$, then the spectral
norm$||W||_{2}=\sqrt{\lambda_{M}}$; The Frobenius
norm$||W||_{F}=\sqrt{\sum_{i=1}^{M}\lambda_{i}}$

Proof 3.1: See [101] and [46] $\square$

Lemma 3.2: For a $n\times m$ matrix,
$||W||_{1}=\mathop{\max}\limits_{j}\sum_{i=1}^{n}|a_{i,j}|$,
$||W||_{\infty}=\mathop{\max}\limits_{i}\sum_{j=1}^{m}|a_{i,j}|$, then
$||W||_{2}\leq\sqrt{||W||_{1}||W||_{\infty}}$

Proof 3.2: See [101] $\square$

Lemma 3.3: For a $n\times m$ matrix,
$||W||_{F}=\sqrt{\left(\sum_{j=1}^{m}\sum_{i=1}^{n}|a_{i,j}|^{2}\right)}$,
then $||W||_{2}\leq||W||_{F}$

Proof 3.3: See [101] $\square$

1-Lipschitz continuity can be expressed by the spectral normalization. Miyato
et al. [46] control the Lipschitz constant through spectral normalization
$W_{\sigma}=\frac{W}{||W||_{2}}$ of each layer for D, leading to a better
result than WGAN-GP. Similarly, according to the optimal transport with
regular term, Lipschitz constant of discriminator should be less than or equal
to 1. Correspondingly, upper bound of the spectral norm can be utilized to
normalize the weight ($||W_{\sigma}||_{2}\leq 1$), achieving $k(k\leq 1)$
Lipschitz continuity. In terms of Lemma 3.2 and Lemma 3.3,
$\sqrt{||W||_{1}||W||_{\infty}}$ and Frobenius norm ($||W||_{F}$) are simple
upper bound of the spectral norm ($||W||_{2}$) and can be used to normalize
the weight. Specifically, Zhang et al. [99] use the
$\sqrt{||W||_{1}||W||_{\infty}}$, seeking for an approximation of the spectral
norm that is easy to calculate. Miyato et al. [46] explain that the Frobenius
norm is a restriction on all eigenvalues. It is different from the spectral
norm, which only constrains the maximum eigenvalue. They only propose a
conjecture that Frobenius normalization will affect the network’s ability to
express, but no experiments are reported to compare it with the spectral
normalization. Liu et al. [100] find that the mode collapse is often
accompanied by the collapse of the eigenvalue of the discriminator. Because
the spectral normalization only limits the maximum eigenvalue, and the
eigenvalue collapse means the remaining eigenvalues suddenly decreasing.
Therefore, they adopt the following methods to prevent the collapse of the
eigenvalues:

$W_{\sigma}=\frac{W+\nabla W}{||W||_{2}}=\frac{W}{||W||_{2}}+\frac{\nabla
W}{||W||_{2}}.$ (49)

The results demonstrate that this simple method can effectively prevent mode
collapses. Experiments are reported in this study, yet theoretical proofs are
not absent. Therefore the relationship between the matrix eigenvalues and GAN
performance is not clear.

Few researches focus on weight normalization, whose summary is demonstrated in
TABLE IV. Among them, spectral normalization is widely applied in some SOTA
methods, which is demonstrated in Section 7.

####  IV-B2 Weight Regularization

Compared with spectral normalization similar to 1-GP, spectral norm
regularization is similar to the 0-GP. Therefore, Kurach et al. [51] use the
$\mathcal{L}_{R}=||W||_{2}$ to regularize the loss function. Besides, Zhou et
al. [102] also use the $L_{P}$-norm ($P=1,F,\infty$) to regularize the
discriminator. However, researchers are not involved in the related study of
these methods as they lead to worse performance than spectral normalization.

In addition to the above methods being used to implement Lipschitz continuity
for discriminators, Brock et al. [30] apply Off-Diagonal Orthogonal
Regularization (Off-Diagonal OR) to the generator directly enforcing the
orthogonality condition:

$R_{o}(W)=\beta\left\|W^{\top}W\odot(\mathbf{1}-I)\right\|_{\mathrm{F}}^{2},$
(50)

where W is a weight matrix, $\beta$ is a hyperparameter and $\mathbf{1}$
denotes a matrix with all elements set to 1. The Off-Diagonal OR makes G to be
smooth, so that the full space of $z$ will map to good output samples.

##  V Regularization and Normalization of "Making sample real"

From the perspective of the "Making sample real", generator is counterfeiter
designed to deceive the discriminator, while discriminator is police designed
to distinguish between real and fake samples. The motivation of GANs is to
train the generator based on the loss of the discriminator. Unlike the direct
training objective of the classification task (Minimizing cross-entropy loss),
the objective of GANs training is indirect. Hence, only one-dimensional output
of the discriminator does not provide a complete representation on truth or
false of samples. Some works have shown that the present discriminators
contain some significant deficiencies in the frequency domain [27] and
attribute domain [44], which are evidence of the lacking discrimination for
discriminators. Excessive shortage of discrimination makes the generator lack
incentives from the discriminator to learn useful information of the data. To
alleviate this situation, many regularization methods and additional
supervision tasks have been proposed in the community, and they can be divided
into three categories: data augmentation and preprocessing, consistency
regularization, and self-supervision.

###  V-A Data Augmentation and Preprocessing

Figure 3: Framework of data augmentation and preprocessing for updating D
(left) and G (right). (Coming from [47])

Data Augmentation has played an important role in deep learning algorithms. It
can increase the diversity of the training data naturally, thus reduce the
overfitting in many computer vision and graphics applications [103, 104]. Date
augmentation adopts different data transformation technologies ($T$) to
increase the number of training samples. According to the different forms,
data transformations can be divided into two types. One type is spatial
transformation of data, such as $zoomout$, $zoomin$, $translation$,
$translationx$, $translationy$, $cutout$ [105], $cutmix$ [106]; The other is
visual transformation, such as $brightness$, $redness$, $greenness$,
$blueness$, $mixup$ [107].

Similarly, the performance of GANs heavily deteriorates given a limited amount
of training data [108]. However, recent works [48, 47, 109, 110, 111] observe
that augmenting only real images (Only applying $T$ to (i) in Figure 3), only
generated images (Only applying $T$ to (ii) in Figure 3), and only
discriminator (Both applying $T$ to (i) and (ii) in Figure 3) do not help with
GANs training. Naturally, one problem needs to be considered: whether the
overfitting exists in GANs’ training? Some works [47, 110] demonstrate that,
even big dataset and state of the art models, the training of GANs still
suffers from severe overfitting. And the less training data there is, the
earlier this happens. Recently, some works [47, 109, 110, 111] on data
augmentation for GANs training have been proposed simultaneously. They all
argue that the classical data augmentation approach could mislead the
generator to learn the distribution of the augmented data, which could be
different from that of the original data. To deal with this problem, they
augment both real and fake samples and let gradients propagate through the
augmented samples to G (Applying $T$ to (i), (ii), and (iii) in Figure 3).
Through adding the data augmentation to all processes of GANs training, the
performance of GANs has been significantly improved. However, which
augmentation is most beneficial for GANs training? Figure 4 shows the FID
comparisons (demonstrated on [109]) of BigGAN on CIFAR-10 dataset. For only
data augmentation (represented by ‘vanilla_rf’), the operations in spatial
augmentation such as $translation$, $zoomout$, and $zoomin$, are much more
effective than the operations in visual augmentation, such as $brightness$,
$colorness$ ($redness$ and $greenness$), and $mixup$. The result is that
augmentations resulting in spatial changes improve the GANs performance more
than those inducing mostly visual changes. It is easy to understand that
generated images are significantly lacking in detail compared to the real
images, and spatial augmentation improves the ability of the generator to fit
detailed textures through spatial changes.

In addition to data augmentation, Li et al. [45] indicate that high-frequency
components between real images and fake images are different, which is not
conducive to the training of GANs. They propose two preprocessing methods
eliminating high-frequency differences in GANs training: High-Frequency
Confusion (HFC) and High-Frequency Filter (HFF). The proposed methods are
applied in places (i), (ii), and (iii) in Figure 3 and improve the performance
of GANs with a fraction of the cost.

Figure 4: FID mean and std of BigGAN on CIFAR-10. The blue dashed horizontal
line shows the baseline FID=14.73 of BigGAN trained without augmentation.
‘vanilla_rf’ represents training vanilla BigGAN with both real and fake images
augmented. ‘bcr’ corresponds to training BigGAN with bCR on augmented real and
fake images. (Figure comes from [109])

In summary, both data augmentation and data preprocessing improve the
performance of GANs with little cost. Data augmentation uses different
transformations to improve discrimination and avoid overfitting. Among them,
spatial augmentations achieve better performance than visual augmentations.
More specifically, Zhao et al. [47] demonstrate that hybrid augmentation with
$Color+Translation+Cutout$ is especially effective, and which is widely used
in other works [112, 113]. Besides, data preprocessing is also a remarkable
method. We expect more work to be presented in this area.

Figure 5: Overview of consistency regularization, where image consistency
regularization updates the G (left) and network consistency regularization
updates the D (right). $H$ is the feature mapping function and $T$ is
different data transformation technologies.

###  V-B Consistency Regularization

For semi-supervised or unsupervised learning, consistency regularization has
been widely used in [114, 115, 116, 117]. It is motivated by the fact that
models should produce consistent predictions given input and their semantics-
preserving augmentations, such as image rotating, and adversarial attacks. As
acknowledged, the supervision of GANs training is weak. To increase the
discrimination of discriminator, some consistency regularization technologies
have also been used. Due to different goals, we divide them into two parts:
image consistency and network consistency. The overviews of them are
demonstrated in Figure 5.

####  V-B1 Image Consistency

The purpose of GANs is to generate fake images similar to real ones. In GANs,
the discriminator is generally used to distinguish real images and generated
images. However, outputs of the discriminator with only one dimension hardly
portray the authenticity of the image completely. To improve the
representation of the discriminator, some works extend the outputs of the
discriminator, for example, relativistic discriminator [118, 119],
distribution discriminator [120], and cascading rejection [121]. While some
works reduce the training difficulty of discriminators by introducing prior
information. Regularizing the distance between the generated and real images
with different measurements, namely image consistency, is the focus of this
paper. The overview of it is demonstrated in left part of Figure 5, where
consistency regularization is used to update generator (G) and can be
formulated as:

$\mathcal{L}_{C}=\mathbb{E}_{x\sim p_{r},z\sim p_{z}}C(H(x),H(G(z))),$ (51)

where $H$ is the feature mapping function and $C$ is the consistency
measurement function. Different image consistency regularization have
different $H$ and $C$. Specifically, Salimans et al. [97] recommend that the
generator should be trained using a feature matching procedure. The objective
is:

$\mathcal{L}_{C}=||\mathbb{E}_{x\sim p_{r}}f(x)-\mathbb{E}_{z\sim
p_{z}}f(G(z))||^{2}_{2},$ (52)

where $f(x)$ denotes the intermediate layer of the discriminator. Similarly,
the intermediate layer of another pre-trained classification model is also
alternative. The empirical results indicate that feature matching is indeed
effective in situations where normal GAN becomes unstable. Unlike the above
work only using mean feature matching to train generators, Mroueh et al. [122]
propose McGAN, which trains both the generator and discriminator using the
mean and covariance feature matching. The objective is:

$\begin{split}\mathcal{L}_{C}&=\mathcal{L}_{\mu}+\mathcal{L}_{\sigma}\\\
&=||\mu(p_{r})-\mu(G(p_{z}))||_{q}+||\sum\left(p_{r}\right)-\sum(p(G(z)))||_{k},\end{split}$
(53)

where $\mu(p_{r})=\mathbb{E}_{x\sim p_{r}}f(x)$ and
$\sum(p_{r})=\mathbb{E}_{x\sim p_{r}}f(x)\cdot f(x)^{\mathrm{T}}$ represent
the mean and the covariance of the feature layer $f(x)$, respectively. Apart
from statistical differences, some works [123, 27] focus on the difference in
frequency domain between the generated and real image. Specifically, Durall et
al. [123] find that the deep generative models based on up-convolution are
failing to reproduce spectral distributions, which leads to considerable
differences in the spectral distributions between real images and generated
images. Thus, the spectral regularization has been proposed as follows:

$\begin{split}\mathcal{L}_{C}&=\frac{1}{M/2-1}\sum_{i=0}^{M/2-1}AI^{real}_{i}\cdot\log(AI^{fake}_{i})\\\
&+(1-AI^{real}_{i})\cdot\log(1-AI^{fake}_{i}),\end{split}$ (54)

where $M$ is the image size and $AI$ is the spectral representation from the
Fourier transform of the images. Corresponding to Eq (51), $H$ and $C$ are
implemented with $AI$ and cross-entropy, respectively.

Contrary to this, the research study [27] uses hard example mining to improve
the discriminatory of the discriminator based on the difference between real
and generated samples under different metrics. Although this paradigm is
different from the paradigm of image consistency regularization, both cases
are motivated by obtaining generated samples similar to real images under
different distance measures, so we integrate them. Chen et al. [27] consider
both downsampling strategies: downsampling with anti-aliasing and downsampling
without anti-aliasing, leads to high frequencies missing in the discriminator.
High frequencies missing leads to high frequency deviation between real and
generated images. To mitigate this issue, authors propose SSD-GAN, which
introduces an additional spectral classifier to detect frequency spectrum
discrepancy between real and generated images and integrate it into the
discriminator of GANs. The overall realness of sample x is represented as:

$D^{ss}(x)=\lambda D(x)+(1-\lambda)C(\phi(x)),$ (55)

where the enhanced discriminator $D^{ss}$ consists of two modules, a vanilla
discriminator $D$ that measures the spatial realness, and a spectral
classifier $C$. $\lambda$ is a hyperparameter that controls the relative
importance of the spatial realness and the spectral realness. The adversarial
loss of the framework can be written as:

$\mathcal{L}_{D}=\mathbb{E}_{x\sim p_{\text{data }}(x)}\left[\log
D^{ss}(x)\right]+\mathbb{E}_{x\sim
p_{g}(x)}\left[\log\left(1-D^{ss}(x)\right)\right].$ (56)

TABLE V: The summary of the consistency regularization Method |  Consistency regularization term $L_{C}$
---|---
Mean regularization [97] | $||\mathbb{E}_{x\sim p_{r}}f(x)-\mathbb{E}_{z\sim p_{z}}f(G(z))||_{q}$
Mean and Convariance regularization [122] | $||\mathbb{E}_{x\sim p_{r}}f(x)-\mathbb{E}_{z\sim p_{z}}f(G(z))||_{q}+||\mathbb{E}_{x\sim p_{r}}f(x)\cdot f(x)^{\mathrm{T}}-\mathbb{E}_{z\sim p_{z}}f(G(z))\cdot f(G(z))^{\mathrm{T}}||_{k}$
Spectral regularization [123] | $\mathcal{L}_{C}=\frac{1}{M/2-1}\sum_{i=0}^{M/2-1}AI^{real}_{i}\cdot\log(AI^{fake}_{i})+(1-AI^{real}_{i})\cdot\log(1-AI^{fake}_{i})$
CR-GAN [48, 124] | $\mathbb{E}_{x\sim p_{r}}||D(x)-D(T(x))||_{2}$
bCR-GAN [32] | $\lambda_{real}\mathbb{E}_{x\sim p_{r}}||D(x)-D(T(x))||_{2}+\lambda_{fake}\mathbb{E}_{z\sim p_{z}}||D(G(z))-D(T(G(z)))||_{2}$
zCR-GAN[32] | $\lambda_{dis}\mathbb{E}_{z\sim p_{z}}||D(G(z))-D(G(T(z)))||_{2}-\lambda_{gen}\mathbb{E}_{z\sim p_{z}}||G(z)-G(T(z))||_{2}$

The summary of the image consistency regularization is given in Table V. In
summary, image consistency considers that the real images and the generated
images are similar not only in the output of discriminator, but also in
statistical information and frequency domain. The analysis of biases between
real and generated images using different metrics will be an interesting
future research direction.

####  V-B2 Network Consistency

Network consistency regularization can be regarded as Lipschitz continuity on
semantics-preserving transformation. Specifically, we hope discriminator is
insensitive to semantics-preserving transformation, which drives the
discriminator to pay more attention to the authenticity of the images. For
example, in the image domain, the reality of images should not change if we
flip the image horizontally or translate the image by a few pixels. To resolve
this, Zhang et al. [48] propose the Consistency Regularization GAN (CR-GAN)
that uses the consistency regularization on the discriminator during GANs
training:

$\mathcal{L}_{C}=\mathbb{E}_{x\sim p_{r}}||D(x)-D(T(x))||_{2},$ (57)

where $T(x)$ represents a transformation (shift, flip, cutout, etc.) of
images. One key problem with the CR-GAN is that the discriminator might occur
the ’mistakenly believe’. ’mistakenly believe’ considers that the
transformations are actual features of the target dataset, due to only
applying these transformations on real images. This phenomenon is not easy to
notice for certain types of transformations (e.g. image shifting and
flipping). However, some types of transformations, such as cutout
transformations, contain visual artifacts not belonging to real images, which
effects greatly limits the choice of advanced transformations we could use. To
address this issue, Zhao et al. [32] propose Balanced Consistency
Regularization (bCR-GAN) that uses regulation with respect to both real and
fake images and balances the training of discriminator between real images and
fake images by $\lambda_{real}$ and $\lambda_{fake}$:

$\begin{split}\mathcal{L}_{C}&=\lambda_{real}\mathbb{E}_{x\sim
p_{r}}||D(x)-D(T(x))||_{2}\\\ &+\lambda_{fake}\mathbb{E}_{z\sim
p_{z}}||D(G(z))-D(T(G(z)))||_{2}.\end{split}$ (58)

The overview of bCR is demonstrated in right part of Figure 5.

Contrary to the methods which focus on consistency regularization with respect
to transformations in image space, Zhao et al. [32] also propose Latent
Consistency Regularization (zCR) that considers the consistency regularization
on transformations in latent space. They expect that output of the
discriminator ought not to change much with respect to the small enough
perturbation $\Delta z$ and modify the discriminator loss by enforcing:

$\mathcal{L}^{D}_{C}=\lambda_{dis}\mathbb{E}_{z\sim
p_{z}}||D(G(z))-D(G(T(z)))||_{2},$ (59)

where $T(z)$ represents the added small perturbation noise. However, only this
loss added onto the GAN loss, mode collapse can easily appear in the training
of generators. To avoid this, an inverse gradient penalty (We will describe it
in detail in section 6.2) is added to modify the loss function for generator.
Hence, we modify the generator loss by enforcing:

$\mathcal{L}^{D}_{C}=-\lambda_{gen}\mathbb{E}_{z\sim
p_{z}}||G(z)-G(T(z))||_{2}.$ (60)

Naturally, putting both bCR and zCR together, Improved Consistency
Regularization (ICR) is also proposed by Zhao et al. [32]. In addition, there
are some applications where cyclic consistency regularization [124] is used
for unpaired image-to-image translation. The summary of network consistency
regularization are demonstrated in TABLE V.

In summary, network consistency considers that the networks, especially the
discriminator, should be insensitive to semantics-preserving transformation
($T$). The results in [48] demonstrate that random shift and flip is the best
way to perform image transformation on the CIFAR-10 dataset. And the FID
results with CR, bCR, zCR, and ICR (where transformation is flipping
horizontally and shifting by multiple pixels) demonstrated on [32] are shown
in Table VI. The results demonstrate that network consistency regularization
can significantly improve the performance of GANs. However, which
transformation is best for consistency regularization? Zhao et al [109]
compare the effect of different data transformation technologies (mentioned in
Section 5.1) on bCR. Figure 4 shows the FID results (demonstrated on [109]) of
BigGAN adding bCR (represented by ’bcr’) on CIFAR-10 dataset. From the
results, the best BigGAN FID 8.65 is with transformation technology
$translation$ of strength $\lambda=0.4$, outperforming the corresponding FID
10.54 reported in Zhao et al. [32]. Moreover, spatial transforms, which retain
the major content while introducing spatial variances, can substantially
improve GANs performance together with bCR. While visual transforms, which
retain the spatial variances, can not further improve the performance of GANs
compared with data augmentation only. Furthermore, bCR with stronger
transformation (larger value of $\lambda_{aug}$) does not improve the
performance of GANs, the optimal value of $\lambda_{aug}$ is uncertain for
different data transformation technologies.

TABLE VI: FID scores for class conditional image generation of the network consistency regularization (Data come from [32]) Models |  CIFAR-10 |  ImageNet
---|---|---
SNGAN |  17.50 |  27.62
BigGAN |  14.73 |  8.73
CR-BigGAN |  11.48 |  6.66
bCR-BigGAN |  10.54 |  6.24
zCR-BigGAN |  10.19 |  5.87
ICR-BigGAN |  $\mathbf{9.21}$ |  $\mathbf{5.38}$
Figure 6: Overview of the predict self supervised learning of GANs, where $C$
performs the predict classification task and shares the weights with the
discriminator except for the last layer, $T$ is different data transformation
technologies. Furthermore, the self-supervised task of generated images is
used to update the generator (left) and the self-supervised task of real
images is used to update the classification (right).

###  V-C Self-Supervision

Self-supervised learning aims to learn representations from the data itself
without explicit manual supervision. Recently, some self-supervised works
[125, 126, 117] provide competitive results on ImageNet classification and the
representations learned from which transfer well to downstream tasks. Self-
supervised learning can outperform its supervised pre-training counterpart in
many tasks, such as detection and segmentation, sometimes surpassing it by
large margins. This suggests that self-supervised learning can obtain more
representational features and significantly improve the representation of
networks. Based on this, self-supervised learning can be introduced into the
training of GANs, and we divide them into two categories according to
different self-supervision tasks: predictive self-supervised learning and
contrastive self-supervised learning.

####  V-C1 Predictive Self-Supervised Learning

Predictive self-supervised learning is a popular method to improve the
representation of neural networks by introducing additional supervised tasks,
such as context prediction [127] and rotations prediction [128, 129, 130,
131]. Predictive self-supervised learning is introduced into GANs by Chen et
al. [26] to avoid discriminator forgetting. Discriminator forgetting means
that the discriminator can not remember all tasks at the same time during the
training process, for example, learning varying levels of detail, structure,
and texture, which causes the discriminator to fail to get a comprehensive
representation of the current images. "If the outcome is your focus, then it’s
easy to look for shortcuts. And ultimately shortcuts keep you from seeing the
truth, it drains your spark for life. What matters most is your will to seek
the truth despite the outcome."888Come from ”JoJo’s Bizarre Adventure:Golden
Wind” -Araki Hirohiko.. The same is true for GANs, which are only driven by
the loss of discriminator, which is easy to distinguish between real images
and generated images through shortcuts, instead of the texture and structural
features we need. Predictive self-supervised learning solves this problem by
introducing new generalization tasks, which can also be considered to prevent
overfitting. The overview of the predictive self-supervised learning of GANs
are demonstrated on Figure 6. Depending on the data transformation function
$T$, we can design different self-supervised tasks.

Chen et al. [26] used the predictive self-supervision in GANs training for the
first time. They adopt the rotation prediction as the expanding task to
prevent the discriminator from forgetting. Besides, plenty of other prediction
tasks have also been proposed to improve the discrimination. Huang et al.
[132] exploit the feature exchange to make the discriminator learn the proper
feature structure of natural images. Baykal et al. [133] introduce a
reshuffling task to randomly arrange the structural blocks of the images, thus
helping the discriminator increase its expressive capacity for spatial
structure and realistic appearance. Different from the methods described above
for designing tasks at the image or feature level, Patel et al. [134] propose
a self-supervised task with latent transformation detection, which identifies
whether the latent transformation applied in the given pair is the same as
that of the other pair. All above methods have designed different self-
supervised tasks, and their loss functions can be formulated as:

$\begin{split}\mathcal{L}_{D,C}=-\lambda_{r}\mathbb{E}_{x\sim
p^{T}_{r}}\mathbb{E}_{T_{k}\sim{T}}\log\big{(}C_{k}(x)\big{)}\quad\text{ for
}k=1,\ldots,K,\\\ \mathcal{L}_{G}=-\lambda_{g}\mathbb{E}_{x\sim
p^{T}_{g}}\mathbb{E}_{T_{k}\sim{T}}\log\big{(}C_{k}(x)\big{)}\quad\text{ for
}k=1,\ldots,K,\end{split}$ (61)

where ${T}$ represents the different types of image transfer, such as rotation
and reshuffling. Furthermore, $T_{k}$ represents different forms of the
transfer ${T}$, such as $0^{\circ},90^{\circ},180^{\circ},270^{\circ}$ for
rotation task, $K$ is the number of transformed forms, $C_{k}$ is the k-th
output of the classifier $C$ that shares parameters with discriminator except
for two different heads, $P^{T}_{r}$ and $P^{T}_{g}$ are the mixture
distributions of real and generated images, respectively. For rotation
conversion task [26], $K=4$, and the classifier $C$ predicts the rotation
angle; For feature exchange task [132], $K=2$, and the classifier $C$ predicts
whether the swap has occurred; For block reshuffling task [133], the image is
divided into $9$ blocks and the number of the permutations is $9!$, which is
unnecessarily huge. Thirty different permutations are selected in terms of the
Hamming distances between the permutations in [135]. As a result, $K$ is set
to 30, and classifier $C$ predicts the Hamming distances of different
permutations; For the latent transformation task, $K=2$, and the classifier
$C$ predicts whether the transformations parameterized by the same $\epsilon$
or different.

TABLE VII: The summary of the Self-supervision Method | Description | Types
---|---|---
Rotation Prediction [26, 136] | Predicting the angle of rotation ($0^{\circ},90^{\circ},180^{\circ},270^{\circ}$) | Predictive self-supervised
Feature Exchange Detection [132] | Predicting if some exchanges have occurred at the feature level (yes or not) | Predictive self-supervised
Block Reshuffling Prediction [133] | Predicting the Hamming distances of different reshuffling in image level (Total 30 categories) | Predictive self-supervised
Latent Transformation Detection [134] | Predicting if some exchanges have occurred at the latent space level (yes or not) | Predictive self-supervised
InfoMax-GAN [137] |  Positive pairs: Global and local features of an image (both real and fake images) Negative pairs: Global and local features of different images (both real fake images) | Contrastive self-supervised
Cntr-GAN [109] |  Positive pairs: Two different data transformations of the same image (both real and fake images). Negative pairs: Otherwise | Contrastive self-supervised
ContraD [112] |  Positive pairs: Two different data transformations of the same image (real images only) + Two fake images Negative pairs: Otherwise | Contrastive self-supervised

The above methods design different kinds of self-supervised prediction tasks
and participate in the training of the discriminator or generator,
independently, which have “loophole” that, during generator learning, $G$
could exploit to minimize $\mathcal{L}_{G}$ without truly learning the data
distribution. To address this issue, Ngoc-TrungTran et al. [136] introduce
true or false judgment along with self-supervised prediction. The number of
classification is $K+1$, while the loss function can be expressed as:

$\displaystyle\mathcal{L}_{D,C}=$
$\displaystyle-\lambda_{r}\Bigg{(}\mathbb{E}_{x\sim
p^{T}_{r}}\mathbb{E}_{T_{k}\sim{T}}\log\big{(}C_{k}(x)\big{)}+\mathbb{E}_{x\sim
p^{T}_{r}}\mathbb{E}_{T_{k}\sim{T}}\log\big{(}C_{K+1}(x)\big{)}\Bigg{)}$ (62)
$\displaystyle\text{ for }k=1,\ldots,K,$ $\displaystyle\mathcal{L}_{G}=$
$\displaystyle-\lambda_{g}\Bigg{(}\mathbb{E}_{x\sim
p^{T}_{g}}\mathbb{E}_{T_{k}\sim{T}}\log\big{(}C_{k}(x)\big{)}-\mathbb{E}_{x\sim
p^{T}_{g}}\mathbb{E}_{T_{k}\sim{T}}\log\big{(}C_{k+1}(x)\big{)}\Bigg{)}$
$\displaystyle\text{ for }k=1,\ldots,K,$

where $C_{k}$ is a classifier that predicts the rotation angles and $C_{K+1}$
is a classifier that predicts the truth of the images. The new self-supervised
rotation-based GANs use the multi-class minimax game to avoid the mode
collapse, which is better than the original predictive self-supervised
paradigm.

In summary, predictive self-supervised learning improves the discrimination by
designing different self-supervised prediction tasks, among them, rotation
prediction [26] is widely used ([47, 109]) for its simplicity and
practicality. The summary is illustrated in TABLE VII.

Figure 7: Overview of the contrastive self supervised learning, where x is
real of fake images, $v^{(1)}$ and $v^{(2)}$ are different views of the image
x, s is the score function (Usually discriminator in GANs) that measures the
similarity, the square on the middle part is the label of InfoNCE (blue,
white, and black squares are labels with 1, 0, and undefined respectively.),
and the square on the right part is the label of SimCLR. Obviously, SimCLR
defines more negative pairs to improve the sample utilization.

####  V-C2 Contrastive Self-Supervised Learning

Contrastive self-supervised Learning [125, 126, 117], as the name implies,
learn representations by contrasting positive and negative examples. They led
to great empirical success in computer vision tasks with unsupervised
contrastive pre-training. More and more works demonstrate that self-supervised
learning can outperform its supervised pre-training counterpart in many tasks,
which indicates contrastive self-supervised learning can lead to more
expressive features. Consider two views $(v^{(1)}$ and $v^{(2)})$, contrastive
self-supervised learning aims to identify whether two views are dependent or
not. More specifically, maximize the mutual information of positive pairs. To
this end, Oord et al. [138] propose to minimize InfoNCE loss, which turns out
to maximize a lower bound of mutual information. the InfoNCE loss is defined
by:

$L_{\mathrm{NCE}}\left({v}_{i}^{(1)};{v}^{(2)},s\right):=-\log\frac{\exp\left(s\left({v}_{i}^{(1)},{v}_{i}^{(2)}\right)\right)}{\sum_{j=1}^{K}\exp\left(s\left({v}_{i}^{(1)},{v}_{j}^{(2)}\right)\right)},$
(63)

where $s(,)$ is the score function that measure the similarity, positive pairs
($v_{i}^{(1)}$ and $v_{i}^{(2)}$) are different views of the same sample, and
negative pairs ($v_{i}^{(1)}$ and $v_{j}^{(2)}$, $(i\neq j)$) are different
views of different samples. InfoNCE loss is the cornerstone of contrastive
self-supervised learning and its overview is demonstrated on Figure 7.

Many advanced self-supervised methods are implemented by modifying the views
of images $v^{(1)}$, $v^{(2)}$ and score function $s(,)$. Specifically, Deep
InfoMAX [139] maximizes the mutual information between local and global
features, that is, image $x$ passes through the encoder
$E_{\psi}=f_{\psi}\circ C_{\psi}$, producing local feature map $C_{\psi}(x)$
and global feature vector $E_{\psi}(x)$. To maximize the lower bound of the
InfoMax: $\mathcal{I}\big{(}C_{\psi}(x),E_{\psi}(x)\big{)}$, the theoretical
InfoMAX loss has been defined as:

$\displaystyle L_{InfoMAX}(X)=$ (64) $\displaystyle-\mathbb{E}_{x\in
X}\mathbb{E}_{i\in\mathcal{A}}\big{[}\log\frac{\exp\left(g_{\theta,\omega}\left(C_{\psi}^{(i)}(x),E_{\psi}(x)\right)\right)}{\sum_{\left(x^{\prime},i\right)\in
X\times\mathcal{A}}\exp\left(g_{\theta,\omega}\left(C_{\psi}^{(i)}\left(x^{\prime}\right),E_{\psi}(x)\right)\right)}\big{]},$
$\displaystyle
g_{\theta,\omega}\left(C_{\psi}^{(i)}(x),E_{\psi}(x)\right)=\phi_{\theta}\left(C_{\psi}^{(i)}(x)\right)^{T}\phi_{\omega}\left(E_{\psi}(x)\right),$

where $X=\\{x_{1},\cdots,x_{N}\\}$ is a set of random images and
$\mathcal{A}=\\{0,1,\cdots,M^{2}-1\\}$ represents indices of a $M\times M$
spatial sized local feature map. Based on this, positive sample pairs are
$C^{(i)}_{\phi}(x)$ and $E_{\phi}(x)$, and negative sample pairs are
$C^{(i)}_{\phi}(x^{{}^{\prime}})$ and $E_{\phi}(x)$), where $x^{{}^{\prime}}$
is a different image from $x$. SimCLR [117] is another popular contrast
learning framework, which applys two independent transformations, namely
$t_{1}$ and $t_{2}$, to obtain the different views
$v^{(1)},v^{(2)}=t_{1}(x),t_{2}(x)$. Then the loss function of SimCLR is
defined as:

$\displaystyle
L_{\mathrm{SimCLR}}\left({v}^{(1)},{v}^{(2)}\right)=\frac{1}{N}\sum_{i=1}^{N}\left(L_{\mathrm{NCE}}\left({v}_{i}^{(1)};\left[{v}^{(2)};{v}_{-i}^{(1)}\right],s_{\mathrm{SimCLR}}\right)\right),$
(65)

where ${v}_{-i}:={v}\backslash\left\\{{v}_{i}\right\\}$ and
$s_{\mathrm{SimCLR}}$ is defined as:

$s_{\text{SimCLR
}}\left({v}^{(1)},{v}^{(2)};f,h\right)=\frac{h\left(f\left({v}^{(1)}\right)\right)\cdot
h\left(f\left({v}^{(2)}\right)\right)}{\tau\cdot\left\|h\left(f\left({v}^{(1)}\right)\right)\right\|_{2}\left\|h\left(f\left({v}^{(2)}\right)\right)\right\|_{2}}.$
(66)

As shown in Figure 7, SimCLR defines more negative pairs to improve the sample
utilization compared to InfoNCE.

The self-supervised methods mentioned above are also widely applied to the
training of GANs. Inspired by Deep InfoMax [139], Lee et al. [137] propose
InfoMax-GAN maximizing the mutual information between local and global
features of real and fake images. The regularization of discriminator can be
expressed as:

$\mathcal{L}_{\mathrm{InfoMax-
GAN}}=\lambda_{d}\\{L_{\mathrm{InfoMAX}}(x_{r})+L_{\mathrm{InfoMAX}}(x_{f})\\}.$
(67)

where $x_{r}$ and $x_{f}$ represent sets of real and fake images,
respectively.

Inspired by SimCLR [117], some methods [112, 109] introduce different data
transformation technologies to create positive and negative pairs during GANs
training. Zhao et al [109] propose Cntr-GAN, where SimCLR loss is used to
regularize the discriminator on two random augmented copies of both real and
fake images. The regularization of discriminator for transformation $T$ is:

$\mathcal{L}_{\mathrm{Cntr-
GAN}}=\lambda_{d}\\{L_{\mathrm{SimCLR}}(x_{r},T(x_{r}))+L_{\mathrm{SimCLR}}(x_{f},T(x_{f}))\\}.$
(68)

They also compare the effect of different data transformation technologies
(mentioned in Section 5.1) on Cntr-GAN. Figure 8 shows the FID results
(demonstrated on [109]) of BigGAN adding SimCLR loss on CIFAR-10 dataset. The
results illustrate that spatial transformations still work better than visual
transformations and the best FID of 11.87 is achieved by applying adjusted
SimCLR transformations with the cropping/resizing strength of 0.3. Although
this simply regularizing auxiliary SimCLR loss improves GAN training, but
could not outperform existing methods based on simple data augmentations,
$e.g.$, bCR (demonstrated in Figure 4).

To improve the efficiency of contrastive learning, Jeong et al. [112] propose
Contrastive Discriminator (ContraD), a way of training discriminators of GANs
using improved SimCLR. Different from Cntr-GAN with SimCLR loss on both real
and generated images, ContraD uses the SimCLR loss on the real images and the
supervised contrastive loss on the generated images. Supervised contrastive
loss adopts the contrastive between real and generated images, which is needed
as a GAN discriminator. More concretely, for two views
$v^{(1)},v^{(2)}=t_{1}(x),t_{2}(x)$ with ${t}_{1},{t}_{2}\sim{T}$, the loss of
real images are:

$L_{\text{con
}}^{+}\left(D,h_{{r}}\right)=L_{\mathrm{SimCLR}}\left({v}_{{r}}^{(1)},{v}_{{r}}^{(2)};D,h_{{r}}\right),$
(69)

where $h_{r}$ is a projection head for this loss. However, the loss for
generated images, an extended version of contrastive loss to support
supervised learning by allowing more than one view to be positive. More
concretely, they assume all the views from fake samples have the same label
against those from real samples. Formally, for each $v_{i}^{(1)}$, the
positive views are represented by $V_{i}^{(2)}$ that is a subset of $v^{(2)}$.
Then the supervised contrastive loss is defined by:

$\begin{split}&L_{\text{SupCon
}}\left({v}_{i}^{(1)},{v}^{(2)},V_{i+}^{(2)}\right)=\\\
&-\frac{1}{\left|V_{i+}^{(2)}\right|}\sum_{{v}_{i+}^{(2)}\in
V_{i+}^{(2)}}\log\frac{\exp\left(s_{{SimCLR}}\left({v}_{i}^{(1)},{v}_{i+}^{(2)}\right)\right)}{\sum_{j}\exp\left(s_{{SimCLR}}\left({v}_{i}^{(1)},{v}_{j}^{(2)}\right)\right)}.\end{split}$
(70)

Using the notation,the ContraD loss for fake samples are:

$\begin{split}&L_{\text{con }}^{-}\left(D,h_{f}\right)=\\\
&\frac{1}{N}\sum_{i=1}^{N}L_{\text{SupCon
}}\left({v}_{f,i},\left[{v}_{f,-i};{v}_{{r}}^{(1)};{v}_{{r}}^{(2)}\right],\left[{v}_{f,-i}\right];D,h_{{f}}\right),\end{split}$
(71)

where $v_{f}=t_{3}(G(z))$ is a random view of fake samples ($t_{3}\sim T$),
and $v_{-i}=v\backslash\\{v_{i}\\}$ is subset of $v$ that does not contain
$v_{i}$. Remark that they also use an independent projection header $h_{f}$ in
this loss instead of $h_{r}$ in $L_{\text{con }}^{+}\left(D,h_{{r}}\right)$.

To sum up, ContraD learns its contrastive representation by minimizing the
following regularization loss:

$L_{\text{con }}\left(D,h_{{r}},h_{{f}}\right)=L_{\text{con
}}^{+}\left(D,h_{{r}}\right)+\lambda_{\text{con
}}L_{{con}}^{-}\left(D,h_{{f}}\right).$ (72)

The experimental results show that ContraD consistently improves the
performance of GANs compared to other methods, such as Cntr-GAN, DiffAug, bCR,
and CR. However, ContraD with different data transformations is not discussed
further.

Figure 8: BigGAN regularized by SimCLR loss with different image
augmentations. The blue dashed horizontal line shows the baseline FID=14.73 of
BigGAN trained without augmentation. Here they adjust the strength of
cropping-resizing in the default simclr. Cntr-GAN consistently outperforms
vanilla GAN with preferance on spatial augmentations. (Figure comes from
[109])

In summary, contrastive self-supervised learning designs different positive
and negative pairs and maximizes the mutual information of positive pairs
according to the InfoNCE loss. Different from classification and segmentation
tasks, two types of samples (real and fake images) exist for generating
adversarial networks, which adds more possibilities to the definition of
positive and negative pairs. In the future, score-based contrastive learning
may be proposed during the training of GANs. The summary of contrastive self-
supervised regularization technologies of GANs are illustrated in TABLE VII.

##  VI Regularization and Normalization of "Other methods"

In addition to the three groups mentioned above, this section summarizes and
discusses the remaining regularization and normalization techniques, namely,
layer normalization and inverse gradient penalty. Specifically, layer
normalization consists of unconditional-based layer normalization and
conditional-based layer normalization, the former inspired by supervised
learning is used to accelerate training, but its impact on the GANs is small
and sometimes affects the performance, while the latter is used in the
conditional generation and significantly improves the performance of
conditional generation; Furthermore, inverse gradient penalty mitigates mode
collapse by maximizing the Lipschitz constant of the generator.

###  VI-A Layer Normalization

Data in machine learning is expected to be independent and identically
distributed ($i.i.d$). However, in terms of deep learning, because of the
Internal Covariate Shift (ICS) [40], inputs of each neuron do not satisfy the
$i.i.d$, making the training of the deep neural networks hard and unstable.
Layer normalization999 layer normalization is different from the Layer
Normalization (LN), where layer normalization is a general term for a class of
methods such as BN, LN.  has been proposed to avoid such problems. The general
form of the layer normalization is (the difference between the normalization
methods lies in the choice of $h$ and the calculation of $\mathbb{E}[h]$ and
$var[h]$):

$h_{N}=\frac{x-\mathbb{E}[h]}{\sqrt{var[h]+\epsilon}}\cdot\gamma+\beta.$ (73)

For GANs, the layer normalization can be divided into two parts:
unconditional-based layer normalization and conditional-based layer
normalization. Unconditional-based layer normalizations are used for
unconditional generation similar to the other deep neural networks. On the
other hand, conditional-based layer normalizations are used for the generator
of the conditional generation, where the shift and scale parameters
($\gamma,\beta$) depend on the condition information. The form can be defined
as:

$h_{N}=\frac{x-\mathbb{E}[h]}{\sqrt{var[h]+\epsilon}}\cdot\gamma(c)+\beta(c).$
(74)

####  VI-A1 Unconditional-based layer Normalization

Unconditional-based layer normalization is used for both the generator and
discriminator with the same motivation as in other deep neural networks. Ioffe
et al. [40] proposed the first normalization for neural networks, namely,
Batch Normalization (BN). Batch normalization adopts the data of the mini-
batch to compute the mean and variance, making the data distribution of each
mini-batch approximately the same. Miyato et al. [46] used the BN in GANs. BN
normalizes at the mini-batch level, which destroys the difference between
pixels during the generation on account of image generation being a pixel-
level task. Contrary to BN which normalizes the same channel with different
images, Layer Normalization9 (LN) [41] normalizes different channels of a
single image that also destroys the diversity between channels for the pixel-
by-pixel generative model [46]. Instance Normalization (IN) [140] has also
been proposed for style transformation that is adopted for a single channel of
a single image. Moreover, Group Normalization (GN) [141] sits between LN and
IN, which first divides the channel into many groups, and normalizes different
groups of a single image. Compared to normalization of input of neural
networks in BN, LN, IN and GN, Weight Normalization (WN) [142] normalizes the
weight matrix of neural networks. Miyato et al. [46] also used this
normalization in GANs.

In summary, unconditional-based layer normalization in GANs is similar to
other neural networks. The related summaries are shown in Table IX. To the
best of our knowledge, no study compares the performance of these methods,
therefore, we demonstrate the FID results101010The base framework comes from
the SNGAN in https://github.com/kwotsin/mimicry for different normalization
methods on CIFAR-10 and CIFAR-100 datasets in Table VIII. Among them, LN and
GN obtained better performance than the most popular normalization method:
Spectral normalization (mentioned in Section 4.2) and other methods
significantly affect the stability of GANs training.

TABLE VIII:  FID results for different normalization methods on CIFAR-10 and CIFAR-100 datasets. ( The structure is the same as SNGAN except that the discriminator uses different normalization methods) methods | CIFAR-10 | CIFAR-100
---|---|---
None | 40.91 | 45.44
BN | 37.63 | 44.45
LN | 19.21 | 21.15
IN | 34.14 | 43.64
GN | 19.31 | 20.80
WN | 24.28 | 29.96
SN | 19.75 | 22.89

####  VI-A2 Conditional-based layer Normalization

Conditional-based layer normalization is only used for the generator of the
conditional generation. It aims to introduce conditional information to each
layer of the generator, which helps to improve the quality of the generated
images. $\gamma(c)$ and $\beta(c)$ in Eq (44) are calculated with different
features or class labels as input to the neural network in different methods.
Miyato et al. [143] and Zhang et al. [144] used the Conditional Batch
Normalization (CBN) to encode class labels, thereby improving the quality of
conditional generation. Huang et al. [145] and Karras et al. [146] used the
Adaptive Instance Normalization (AdaIN) with target images to improve the
accuracy of style transfer. Park et al. [147] used the Spatially-Adaptive (de)
Normalization (SPADE) with semantic segmentation image to incorporate semantic
information into all layers. Wang et al. [148] used the Attentive
Normalization (AN) to model long-range dependent attention, which is similar
to self-attention GAN [144].

In summary, the main difference between these conditional-based normalizations
is the content of conditional inputs (c in Eq (45)). As the information of
inputs is gradually enriched, the performance of conditional generation is
gradually improved. The related summaries are shown in TABLE IX.

TABLE IX: The summary of the layer normalization Method | Reference | Classification |  Inputs of $\gamma(c)$ and $\beta(c)$
---|---|---|---
Batch Normalization (BN) |  2018 [46, 149] | unconditional-based | -
Layer Normalization (LN) |  2018 [46] | unconditional-based | -
Instance Normalization (IN) |  2018 [46] | unconditional-based | -
Group Normalization (GN) |  2018 [141] | unconditional-based | -
Weight Normalization (WN) |  2018 [46, 149] | unconditional-based | -
Conditional Batch Normalization (CBN) |  2018 [143, 144] | conditional-based | class label
Adaptive Instance Normalization (AdaIN) |  2017 [145],2019 [146] | conditional-based | target images
Spatially-adaptive (de) Normalization (SPADE) |  2019 [147] | conditional-based | sematic segmentation map
Attentive Normalization(AN) |  2020 [148] | conditional-based | self

###  VI-B Inverse Gradient Penalty

Mode collapse is a common phenomenon in GANs’ training. Specifically, little
changing of the generated space $G(z)$ along with the change in given latent
variable $z$. Geometrically, the phenomenon means that all the tangent vectors
of the manifold are no longer independent of each other - some tangent vectors
either disappear or become linearly correlated with each other. Intuitively,
we can solve this problem by maximizing the Lipschitz constant of the
generator, which is opposite of the gradient penalty of the discriminator
described in the previous section. Based on this, inverse gradient penalty of
the generator has been proposed. Concretely, under the little perturbation of
the latent space, the generator needs to produce different images. Yang et al.
[150] use it in conditional generation, especially for tasks that are rich in
conditional information, such as inpainting and super-resolution.

$\mathop{\max}\limits_{G}\mathcal{L}_{z}(G)=\mathop{\max}\mathbb{E}_{z_{1},z_{2}}\left[\mathop{\min}\left(\frac{||G(y,z_{1})-G(y,z_{2})||}{||z_{1}-z_{2}||},\tau\right)\right],$
(75)

where $y$ is the class label and $\tau$ is the bound to ensure numerical
stability. Unlike the intuition-based work described above, Odena et al. [151]
demonstrate that the decreasing of singular value in the Jacobian matrix of
the generator is the main reason for the mode collapse during GANs training.
Furthermore, the singular value can be approximated by the gradient, so
Jacobian clamping is used to limit singular values to
$[\lambda_{min},\lambda_{\max}]$. The loss can be expressed as:

$\mathop{\min}\limits_{G}\mathcal{L}_{z}(G)=\big{(}\mathop{\max}(Q,\lambda_{\max})-\lambda_{\max}\big{)}^{2}+\big{(}\mathop{\min}(Q,\lambda_{\min})-\lambda_{\min}\big{)}^{2},$
(76)

where $Q=||G(z)-G(z^{\prime})||/||z-z^{\prime}||$.

In summary, the above two methods [150, 151] are similar and they can all
mitigate the model collapse of generator to some extent. The key point is to
improve the sensitivity of the generator to latent space.

##  VII Regularization and Normalization in SOTA GANs

In this section, we investigate the regularization and normalization
techniques that have been frequently employed in SOTA and popular GANs. We
select five methods categorized into two classes according to different tasks:
Unconditional Generation and Conditional Generation. The selected methods and
analysis are shown in Table X. PGGAN [29] is the most popular GAN model in
recent years, which grows the size of both the generator and discriminator
progressively. PGGAN empowers high-resolution image generation. Since PGGAN
was proposed in 2017, only some simple regularization techniques were applied:
WGAN-GP [31], BN [46], and LN [46]; AutoGAN [152] is the first study
introducing the Neural architecture search (NAS) to GANs. It defines the
search space for the generator architecture and adopts Inception score as the
reward to discover the best architecture. The discovered generator does not
have any normalization, so AutoGAN only comprises SN [46]; BigGAN [30] is a
popular conditional generative adversarial networks, which uses many
regularization and normalization techniques, such as zc-GP[80], SN [46], Off-
Diagonal OR [30], and CBN [143]; DiffAugment-CR-BigGAN [47] applies some
latest regularization techniques to BigGAN and achieves better performance.
Apart from the regularization and normalization techniques applied to BigGAN,
CR [48] and data augmentation with $Translation+Cutout$ are also applied in
DiffAugment-CR-BigGAN; Furthermore, DiffAugment-StyleGAN2 [47] adds data
augmentation with $Color+Cutout$ to StyleGAN2 [153], where StyleGAN2 is also a
popular conditional GANs. StyleGAN2 produces photorealistic images with large
varieties and is widely used in image conditional generation tasks, such as
Image Completion [154], Image-to-Image Translation [155]. Apart from the zc-GP
and IN, path length regularization is also used in StyleGAN2.

In summary, many regularization and normalization techniques have been used in
SOTA GANs with zc-GP and SN being more attractive to researchers. Furthermore,
CR and Data augmentation are used to further improve the performance of GANs.
They are orthogonal to and can be combined with other methods.

TABLE X: The survey of the Regularization and Normalization technologies used in SOTA GANs Method | Task |  Gradient Penalty |  Weight normalization and regularization |  Jacobian Regularization |  Data augmentation and preprocessing |  Consistency regularization |  Self supervision |  Layer normalization |  Inverse gradient penalty
---|---|---|---|---|---|---|---|---|---
PGGAN (2017[29]) |  Unconditinal Generation | WGAN-GP |  None | None |  None | None | None | BN: G,LN: D | None
BigGAN (2018 [30]) |  Conditinal Generation | zc-GP |  SN: G, D Off-Diagonal OR | None | None | None | None | CBN | None
AutoGAN (2019 [152]) |  Unconditinal Generation | None |  SN:D | None | None | None | None | None | None
DiffAugment -CR-BigGAN (2020[47]) |  Conditinal Generation | zc-GP |  SN: G, D Off-Diagonal OR | None |  Translation Cutout | CR | None | CBN | None
DiffAugment- StyleGAN2 (2020[47]) |  Conditinal Generation | zc-GP | None | None |  Color Cutout | None | None | IN | None

##  VIII Summary and Outlook

###  VIII-A Summary

Recently, significant achievements of GANs have been made in generation tasks
and the network has been widely used in many computer vision tasks, such as
image inpainting, style transfer, text-to-image translations, and attribute
editing. However, due to the overconfident assumptions, the training faces
many challenges, such as non-convergence, mode collapse, gradient vanishing,
and overfitting. To mitigate these problems, many solutions focus on designing
new architectures, new loss functions, new optimization methods, and
regularization and normalization techniques.

In this paper, we study GANs training from three perspectives and propose a
new taxonomy, denoted as "Training dynamics", "Fitting distribution", "Real &
Fake", and "Other methods", to survey the different regularization and
normalization techniques during GANs training. Our study provides a systematic
and comprehensive analysis of the reviewed methods to serve researchers of the
community. In addition, we also demonstrate the motivation and objectives of
different methods and compare the performance of some popular methods in a
fair manner quantitatively, which has implications for future research in
selecting their research topics or developing their approaches.

###  VIII-B Outlook

In the review of regularization and normalization of GANs, the following
questions and thoughts are proposed based on different perspectives of GANs
training:

  * 1)

What is a good distance metric, and which divergence should be used in GANs
training? The priority in the training process of GANs is to find a suitable
divergence to measure the distance between the generated distribution and the
true distribution. Wasserstein divergence is important for the training of
GANs. However, it is uncertain whether the next proposed divergence performs
better.

  * 2)

What is the main difference between real images and generated images? During
the training of unconstrained and unprioritized GANs, if we can quantitatively
represent the difference between real images and generated images from
different perspectives, the efficient regularization methods can be designed
based on this.

  * 3)

How to avoid real images forgetting111111Real images forgetting is caused by
not introducing real images while training the generator, which is different
from discriminator forgetting.? As acknowledged, real images do not directly
participate in the training of the generator, thus the discriminator needs to
remember the characteristics of the real images to optimize the generator
indirectly. We call this the real images forgetting. We conjecture that real
images forgetting may exist, and which may increase the difficulty of GANs
training. Some works might serve as basis to prove this hypothesis and propose
effective solutions.

  * 4)

Recent studies show that discriminator suffers from overfitting and
discriminator forgetting. It is a common problem of neural networks, which is
caused by the shortcut of the loss driven method. Some new methods, such as
contrastive learning, representation learining, can be proposed to improve the
generalization of the discriminator.

## Acknowledgment

The work is partially supported by the National Natural Science Foundation of
China under grand No.U19B2044, No.61836011 and No.91746209.

## References

  * [1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in Neural Information Processing Systems_ , 2014, pp. 2672–2680.
  * [2] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, “Generative image inpainting with contextual attention,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5505–5514.
  * [3] U. Demir and G. Unal, “Patch-based image inpainting with generative adversarial networks,” _arXiv preprint arXiv:1803.07422_ , 2018.
  * [4] K. Javed, N. U. Din, S. Bae, and J. Yi, “Image unmosaicing without location information using stacked gan,” _IET Computer Vision_ , vol. 13, no. 6, pp. 588–594, 2019.
  * [5] A. Gonzalez-Garcia, J. Van De Weijer, and Y. Bengio, “Image-to-image translation for cross-domain disentanglement,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 1287–1298.
  * [6] A. Royer, K. Bousmalis, S. Gouws, F. Bertsch, I. Mosseri, F. Cole, and K. Murphy, “Xgan: Unsupervised image-to-image translation for many-to-many mappings,” in _Domain Adaptation for Visual Understanding_. Springer, 2020, pp. 33–49.
  * [7] H.-Y. Lee, H.-Y. Tseng, Q. Mao, J.-B. Huang, Y.-D. Lu, M. Singh, and M.-H. Yang, “Drit++: Diverse image-to-image translation via disentangled representations,” _International Journal of Computer Vision_ , pp. 1–16, 2020.
  * [8] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, “Stargan v2: Diverse image synthesis for multiple domains,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8188–8197.
  * [9] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 5907–5915.
  * [10] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He, “Attngan: Fine-grained text to image generation with attentional generative adversarial networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1316–1324.
  * [11] T. Qiao, J. Zhang, D. Xu, and D. Tao, “Mirrorgan: Learning text-to-image generation by redescription,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1505–1514.
  * [12] Y. Shen, J. Gu, X. Tang, and B. Zhou, “Interpreting the latent space of gans for semantic face editing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9243–9252.
  * [13] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8789–8797.
  * [14] Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen, “Attgan: Facial attribute editing by only changing what you want,” _IEEE Transactions on Image Processing_ , vol. 28, no. 11, pp. 5464–5478, 2019.
  * [15] R. Tao, Z. Li, R. Tao, and B. Li, “Resattr-gan: Unpaired deep residual attributes learning for multi-domain face image translation,” _IEEE Access_ , vol. 7, pp. 132 594–132 608, 2019.
  * [16] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2794–2802.
  * [17] S. Nowozin, B. Cseke, and R. Tomioka, “f-gan: Training generative neural samplers using variational divergence minimization,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 271–279.
  * [18] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” _arXiv preprint arXiv:1701.07875_ , 2017.
  * [19] L. Mescheder, S. Nowozin, and A. Geiger, “The numerics of gans,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 1825–1835.
  * [20] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” _arXiv preprint arXiv:1706.08500_ , 2017.
  * [21] N. Kodali, J. Abernethy, J. Hays, and Z. Kira, “On convergence and stability of gans,” _arXiv preprint arXiv:1705.07215_ , 2017.
  * [22] W. Nie and A. Patel, “Jr-gan: Jacobian regularization for generative adversarial networks,” _arXiv preprint arXiv:1806.09235_ , 2018.
  * [23] A. Srivastava, L. Valkov, C. Russell, M. U. Gutmann, and C. Sutton, “Veegan: Reducing mode collapse in gans using implicit variational learning,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 3308–3318.
  * [24] M. Arjovsky and L. Bottou, “Towards principled methods for training generative adversarial networks,” _Stat_ , 2017.
  * [25] Y. Yazici, C.-S. Foo, S. Winkler, K.-H. Yap, and V. Chandrasekhar, “Empirical analysis of overfitting and mode drop in gan training,” in _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2020, pp. 1651–1655.
  * [26] T. Chen, X. Zhai, M. Ritter, M. Lucic, and N. Houlsby, “Self-supervised gans via auxiliary rotation loss,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 12 154–12 163.
  * [27] Y. Chen, G. Li, C. Jin, S. Liu, and T. Li, “Ssd-gan: Measuring the realness in the spatial and spectral domains,” _arXiv preprint arXiv:2012.05535_ , 2020\.
  * [28] K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly, “The gan landscape: Losses, architectures, regularization, and normalization,” 2018.
  * [29] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” _arXiv preprint arXiv:1710.10196_ , 2017.
  * [30] A. Brock, J. Donahue, and K. Simonyan, “Large scale gan training for high fidelity natural image synthesis,” _arXiv preprint arXiv:1809.11096_ , 2018\.
  * [31] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, “Improved training of wasserstein gans,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 5767–5777.
  * [32] Z. Zhao, S. Singh, H. Lee, Z. Zhang, A. Odena, and H. Zhang, “Improved consistency regularization for gans,” _arXiv preprint arXiv:2002.04724_ , 2020.
  * [33] T. Park and G. Casella, “The bayesian lasso,” _Journal of the American Statistical Association_ , vol. 103, no. 482, pp. 681–686, 2008.
  * [34] A. E. Hoerl and R. W. Kennard, “Ridge regression: Biased estimation for nonorthogonal problems,” _Technometrics_ , vol. 12, no. 1, pp. 55–67, 1970\.
  * [35] R. G. Soares, H. Chen, and X. Yao, “Semisupervised classification with cluster regularization,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 23, no. 11, pp. 1779–1792, 2012.
  * [36] J. Hong, Y. Li, and H. Chen, “Variant grassmann manifolds: A representation augmentation method for action recognition,” _ACM Transactions on Knowledge Discovery from Data (TKDD)_ , vol. 13, no. 2, pp. 1–23, 2019.
  * [37] H. Zhao, J. Zheng, W. Deng, and Y. Song, “Semi-supervised broad learning system based on manifold regularization and broad network,” _IEEE Transactions on Circuits and Systems I: Regular Papers_ , vol. 67, no. 3, pp. 983–994, 2020.
  * [38] B. Jiang, C. Li, M. D. Rijke, X. Yao, and H. Chen, “Probabilistic feature selection and classification vector machine,” _ACM Transactions on Knowledge Discovery from Data_ , vol. 13, no. 2, pp. 1–27, 2019.
  * [39] Z. Hu, F. Nie, R. Wang, and X. Li, “Low rank regularization: A review,” _Neural Networks_ , 2020.
  * [40] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _arXiv preprint arXiv:1502.03167_ , 2015.
  * [41] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [42] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in _Proceedings of International Conference on Computational Statistics_. Springer, 2010, pp. 177–186.
  * [43] Y. Wu, P. Zhou, A. G. Wilson, E. P. Xing, and Z. Hu, “Improving gan training with probability ratio clipping and sample reweighting,” _arXiv preprint arXiv:2006.06900_ , 2020.
  * [44] S. Zhao, H. Ren, A. Yuan, J. Song, N. Goodman, and S. Ermon, “Bias and generalization in deep generative models: An empirical study,” _arXiv preprint arXiv:1811.03259_ , 2018.
  * [45] Z. Li, P. Xia, X. Rui, Y. Hu, and B. Li, “Are high-frequency components beneficial for training of generative adversarial networks,” _arXiv preprint arXiv:2103.11093_ , 2021.
  * [46] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, “Spectral normalization for generative adversarial networks,” _arXiv preprint arXiv:1802.05957_ , 2018\.
  * [47] S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han, “Differentiable augmentation for data-efficient gan training,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [48] H. Zhang, Z. Zhang, A. Odena, and H. Lee, “Consistency regularization for generative adversarial networks,” _arXiv preprint arXiv:1910.12027_ , 2019\.
  * [49] H. Petzka, A. Fischer, and D. Lukovnicov, “On the regularization of wasserstein gans,” _arXiv preprint arXiv:1709.08894_ , 2017.
  * [50] X. Wei, B. Gong, Z. Liu, W. Lu, and L. Wang, “Improving the improved training of wasserstein gans: A consistency term and its dual effect,” _arXiv preprint arXiv:1803.01541_ , 2018.
  * [51] K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly, “A large-scale study on regularization and normalization in gans,” _arXiv preprint arXiv:1807.04720_ , 2018.
  * [52] M. Lee and J. Seok, “Regularization methods for generative adversarial networks: An overview of recent studies,” _arXiv preprint arXiv:2005.09165_ , 2020.
  * [53] A. Krogh and J. A. Hertz, “A simple weight decay can improve generalization,” in _Advances in Neural Information Processing Systems_ , 1992, pp. 950–957.
  * [54] H. Chen, “Diversity and regularization in neural network ensembles,” Ph.D. dissertation, University of Birmingham, 2008.
  * [55] J. Kukačka, V. Golkov, and D. Cremers, “Regularization for deep learning: A taxonomy,” _arXiv preprint arXiv:1710.10686_ , 2017.
  * [56] M. E. Tipping, “Sparse bayesian learning and the relevance vector machine,” _Journal of Machine Learning Research_ , vol. 1, no. Jun, pp. 211–244, 2001\.
  * [57] H. Chen, P. Tino, and X. Yao, “Probabilistic classification vector machines,” _IEEE Transactions on Neural Networks_ , vol. 20, no. 6, pp. 901–914, 2009\.
  * [58] H. Chen, P. Tiňo, and X. Yao, “Efficient probabilistic classification vector machine with incremental basis function selection,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 25, no. 2, pp. 356–369, 2013.
  * [59] S. Lyu, X. Tian, Y. Li, B. Jiang, and H. Chen, “Multiclass probabilistic classification vector machine,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2019.
  * [60] H. Chen and X. Yao, “Regularized negative correlation learning for neural network ensembles,” _IEEE Transactions on Neural Networks_ , vol. 20, no. 12, pp. 1962–1979, 2009.
  * [61] ——, “Multiobjective neural network ensembles based on regularized negative correlation learning,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 22, no. 12, pp. 1738–1751, 2010.
  * [62] J. Wen, X. Fang, Y. Xu, C. Tian, and L. Fei, “Low-rank representation with adaptive graph regularization,” _Neural Networks_ , vol. 108, pp. 83–96, 2018.
  * [63] S. Zhou, F. Wang, Z. Huang, and J. Wang, “Discriminative feature learning with consistent attention regularization for person re-identification,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 8040–8049.
  * [64] J. H. Lim and J. C. Ye, “Geometric gan,” _arXiv preprint arXiv:1705.02894_ , 2017.
  * [65] K. Than and N. Vu, “Generalization of gans under lipschitz continuity and data augmentation,” _arXiv preprint arXiv:2104.02388_ , 2021.
  * [66] J. Wu, Z. Huang, J. Thoma, D. Acharya, and L. Van Gool, “Wasserstein divergence for gans,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 653–668.
  * [67] J. Su, “Gan-qp: A novel gan framework without gradient vanishing and lipschitz constraint,” _arXiv preprint arXiv:1811.07296_ , 2018.
  * [68] N. Bonnotte, “From knothe’s rearrangement to brenier’s optimal transport map,” _SIAM Journal on Mathematical Analysis_ , vol. 45, no. 1, pp. 64–87, 2013.
  * [69] L. V. Kantorovich, “On a problem of monge,” _J. Math. Sci.(NY)_ , vol. 133, p. 1383, 2006.
  * [70] Y. Brenier, “Polar factorization and monotone rearrangement of vector-valued functions,” _Communications on Pure and Applied Mathematics_ , vol. 44, no. 4, pp. 375–417, 1991.
  * [71] N. Lei, K. Su, L. Cui, S.-T. Yau, and X. D. Gu, “A geometric view of optimal transportation and generative model,” _Computer Aided Geometric Design_ , vol. 68, pp. 1–21, 2019.
  * [72] G. Peyré, M. Cuturi _et al._ , “Computational optimal transport,” _Foundations and Trends in Machine Learning_ , vol. 11, no. 5-6, pp. 355–607, 2019.
  * [73] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , July 2017.
  * [74] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [75] A. Yadav, S. Shah, Z. Xu, D. Jacobs, and T. Goldstein, “Stabilizing adversarial nets with prediction methods,” _arXiv preprint arXiv:1705.07364_ , 2017.
  * [76] W. Nie and A. Patel, “Towards a better understanding and regularization of gan training dynamics,” _arXiv preprint arxiv:1806.09235_ , 2019.
  * [77] J. Li, A. Madry, J. Peebles, and L. Schmidt, “On the limitations of first-order approximation in gan dynamics,” _arXiv preprint arXiv:1706.09884_ , 2017.
  * [78] O. L. Mangasarian, _Nonlinear programming_. SIAM, 1994.
  * [79] V. Nagarajan and J. Z. Kolter, “Gradient descent gan optimization is locally stable,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 5585–5595.
  * [80] L. Mescheder, A. Geiger, and S. Nowozin, “Which training methods for gans do actually converge?” _arXiv preprint arXiv:1801.04406_ , 2018.
  * [81] K. Roth, A. Lucchi, S. Nowozin, and T. Hofmann, “Stabilizing training of generative adversarial networks through regularization,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 2018–2028.
  * [82] G. Gidel, H. Berard, G. Vignoud, P. Vincent, and S. Lacoste-Julien, “A variational inequality perspective on generative adversarial networks,” _arXiv preprint arXiv:1802.10551_ , 2018.
  * [83] M. Arjovsky and L. Bottou, “Towards principled methods for training generative adversarial networks,” 2017.
  * [84] J. Stanczuk, C. Etmann, L. M. Kreusser, and C.-B. Schonlieb, “Wasserstein gans work because they fail (to approximate the wasserstein distance),” _arXiv preprint arXiv:2103.01678_ , 2021.
  * [85] W. Fedus, M. Rosca, B. Lakshminarayanan, A. M. Dai, S. Mohamed, and I. Goodfellow, “Many paths to equilibrium: Gans do not need to decrease a divergence at every step,” _arXiv preprint arXiv:1710.08446_ , 2017.
  * [86] Z. Zhou, J. Shen, Y. Song, W. Zhang, and Y. Yu, “Towards efficient and unbiased implementation of lipschitz continuity in gans,” _arXiv preprint arXiv:1904.01184_ , 2019.
  * [87] D. Terjék, “Virtual adversarial lipschitz regularization,” _arXiv preprint arXiv:1907.05681_ , 2019.
  * [88] J. Adler and S. Lunz, “Banach wasserstein gan,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 6754–6763.
  * [89] M. Xu, Z. Zhou, G. Lu, J. Tang, W. Zhang, and Y. Yu, “Towards generalized implementation of wasserstein distance in gans,” 2021.
  * [90] L. Zhang, Y. Zhang, and Y. Gao, “A wasserstein gan model with the total variational regularization,” _arXiv preprint arXiv:1812.00810_ , 2018.
  * [91] Z. Zhou, J. Liang, Y. Song, L. Yu, H. Wang, W. Zhang, Y. Yu, and Z. Zhang, “Lipschitz generative adversarial nets,” _arXiv preprint arXiv:1902.05687_ , 2019.
  * [92] H. Thanh-Tung, T. Tran, and S. Venkatesh, “Improving generalization and stability of generative adversarial networks,” _arXiv preprint arXiv:1902.03984_ , 2019.
  * [93] A. Mallasto, G. Montúfar, and A. Gerolin, “How well do wgans estimate the wasserstein metric?” _arXiv preprint arXiv:1910.03875_ , 2019.
  * [94] G.-J. Qi, “Loss-sensitive generative adversarial networks on lipschitz densities,” _International Journal of Computer Vision_ , vol. 128, no. 5, pp. 1118–1140, 2020.
  * [95] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, “Virtual adversarial training: a regularization method for supervised and semi-supervised learning,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 8, pp. 1979–1993, 2018.
  * [96] L. C. Evans, “Partial differential equations and monge-kantorovich mass transfer,” _Current developments in mathematics_ , vol. 1997, no. 1, pp. 65–126, 1997.
  * [97] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 2234–2242.
  * [98] S. Barratt and R. Sharma, “A note on the inception score,” _arXiv preprint arXiv:1801.01973_ , 2018.
  * [99] Z. Zhang, Y. Zeng, L. Bai, Y. Hu, M. Wu, S. Wang, and E. R. Hancock, “Spectral bounding: Strictly satisfying the 1-lipschitz property for generative adversarial networks,” _Pattern Recognition_ , p. 107179, 2019.
  * [100] K. Liu, W. Tang, F. Zhou, and G. Qiu, “Spectral regularization for combating mode collapse in gans,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 6382–6390.
  * [101] R. Mathias, “The spectral norm of a nonnegative matrix,” _Linear Algebra and its Applications_ , vol. 139, pp. 269–284, 1990.
  * [102] C. Zhou, J. Zhang, and J. Liu, “Lp-wgan: Using lp-norm normalization to stabilize wasserstein generative adversarial networks,” _Knowledge-Based Systems_ , vol. 161, pp. 415–424, 2018.
  * [103] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, 2017.