# Towards Generalized Implementation of
Wasserstein Distance in GANs

Minkai Xu1,2, Zhiming Zhou3, Guansong Lu4, Jian Tang2,5,6, Weinan Zhang4, Yong
Yu4
1University of Montreal 2Mila - Quebec AI Institute
3Shanghai University of Finance and Economics 4Shanghai Jiao Tong University
5HEC Montreal 6CIFAR AI Research Chair
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Wasserstein GANs (WGANs), built upon the Kantorovich-Rubinstein (KR) duality
of Wasserstein distance, is one of the most theoretically sound GAN models.
However, in practice it does not always outperform other variants of GANs.
This is mostly due to the imperfect implementation of the Lipschitz condition
required by the KR duality. Extensive work has been done in the community with
different implementations of the Lipschitz constraint, which, however, is
still hard to satisfy the restriction perfectly in practice. In this paper, we
argue that the strong Lipschitz constraint might be unnecessary for
optimization. Instead, we take a step back and try to relax the Lipschitz
constraint. Theoretically, we first demonstrate a more general dual form of
the Wasserstein distance called the _Sobolev duality_ , which relaxes the
Lipschitz constraint but still maintains the favorable gradient property of
the Wasserstein distance. Moreover, we show that the KR duality is actually a
special case of the Sobolev duality. Based on the relaxed duality, we further
propose a generalized WGAN training scheme named Sobolev Wasserstein GAN
(SWGAN), and empirically demonstrate the improvement of SWGAN over existing
methods with extensive experiments.111Code is available at
https://github.com/MinkaiXu/SobolevWassersteinGAN.

##  1 Introduction

Generative adversarial networks (GANs) [13] have attracted huge interest in
both academia and industry communities due to its effectiveness in a variety
of applications. Despite its effectiveness in various tasks, a common
challenge for GANs is the training instability [14]. In literature, many works
have been developed to mitigate this problem [3, 23, 17, 25, 26, 39].

By far, it is well known that the problem of training instability of the
original GANs mainly comes from the ill-behaving distance metric [3], i.e.,
the Jensen-Shannon divergence metric, which remains constant when two
distributions are disjoint. The Wasserstein GAN [4] improves this by using the
Wasserstein distance, which is able to continuously measure the distance
between two distributions. Such a new objective has been shown to be effective
in improving training stability.

In practice, since the primal form of the Wasserstein distance is difficult to
optimize, the WGAN model [4] instead proposed to optimize it with the
Kantorovich-Rubinstein (KR) duality [37]. However, though the new WGAN scheme
is theoretically more principled, it does not yield better performance in
practice compared to other variants of GANs [23]. The main obstacle is that
the WGAN requires the discriminator (or the critic) to be a Lipschitz
function. However, this is very hard to satisfy in practice though a variety
of different implementations have been tried such as weight clipping [4],
gradient penalty (GP) [15], Lipschitz penalty (LP) [34] and spectral
normalization (SN) [28]. As a result, WGAN is still unable to always achieve
very compelling results.

In this paper, we argue that the strong Lipschitz condition might be
unnecessary in the inner optimization loop for WGAN’s critic. Intuitively, a
looser constraint on the critic, which results in a larger function space, can
simplify the practical constrained optimization problem of the restricted
critic, and the better-trained critic would further benefit the training of
the generator. Therefore, instead of developing new methods to impose the
Lipschitz constraint, in this paper we propose to relax this constraint. In
other words, we move our attention from “how to better implement the Lipschitz
constraint” to “how to loosen the Lipschitz constraint”. More specifically, in
this paper we demonstrate a new dual form of the Wasserstein distance where
the Lipschitz constraint is relaxed to the Sobolev constraint [1, 30]. We
further show that the new duality with the relaxed constraint indeed is a
generalization of the KR duality, and it still keeps the gradient property of
the Wasserstein distance. Based on this relaxed duality, we propose a
generalized WGAN model called Sobolev Wasserstein GAN. To the best of our
knowledge, among the restricted GAN models [4, 15, 30, 29, 7, 2], Sobolev
Wasserstein GAN is the most relaxed one that can still avoid the training
instability problem.

The main contributions of this paper can be summarized as follows:

  * •

We demonstrate the Sobolev duality of Wasserstein distance and demonstrate
that the new duality is also capable of alleviating the training instability
problem in GANs (Section 3.1). We further clarify the relation between Sobolev
duality and other previous metrics and highlight that by far Sobolev duality
is the most relaxed metric that can still avoid the non-convergence problem in
GANs (Section 3.2).

  * •

Based on Sobolev duality, we introduce the Inequality Constraint Augmented
Lagrangian Method [32] to build the practical Sobolev Wasserstein GAN (SWGAN)
training algorithm (Section 4).

We conduct extensive experiments to study the practical performance of SWGAN.
We find that generally our proposed model achieves better sample quality and
is less sensitive to the hyper-parameters. We also present a theoretical
analysis of the minor sub-optimal equilibrium problem common in WGAN family
models, and further propose an improved SWGAN with a better convergence.

##  2 Preliminaries

###  2.1 Generative adversarial networks

Generative adversarial networks [13] perform generative modeling via two
competing networks. The generator network $G$ learns to map samples from a
noise distribution to a target distribution, while the discriminator network
$D$ is trained to distinguish between the real data and the generated samples.
Then the generator $G$ is trained to output images that can fool the
discriminator $D$. The process is iterated. Formally, the game between the
generator $G$ and the discriminator $D$ leads to the minimax objective:

$\displaystyle\min_{G}\max_{D}\Big{\\{}$ $\displaystyle\mathbb{E}_{x\sim
P_{r}}[\log(D(x))]+$ $\displaystyle\mathbb{E}_{z\sim
P_{z}}[\log(1-D(G(z))]\Big{\\}},$ (1)

where $P_{r}$ denotes the distribution of real data and $P_{z}$ denotes the
noise distribution.

This objective function is proven to be equivalent to the Jensen-Shannon
divergence (JSD) between the real data distribution $P_{r}$ and fake data
distribution $P_{g}$ when the discriminator is optimal. Assuming the
discriminator is perfectly trained, the optimal discriminator is as follows:

$D^{*}(x)=\frac{P_{r}}{P_{r}+P_{g}}.$ (2)

However, recently [42] points out that the gradients provided by the optimal
discriminator in vanilla GAN cannot consistently provide meaningful
information for the generator’s update, which leads to the notorious training
problems in GANs such as gradient vanishing [13, 3] and mode collapse [9, 27,
21, 6]. This view would be clear when checking the gradient of optimal
discriminator in Eq. (2): the value of the optimal discriminative function
$D^{*}(x)$ at each point is independent of other points and only reflects the
local densities of $P_{r}(x)$ and $P_{g}(x)$, thus, when the supports of the
two distributions are disjoint, the gradient produced by a well-trained
discriminator is uninformative to guide the generator [42].

###  2.2 Wasserstein distance

Let $P_{r}$ and $P_{g}$ be two data distributions on $\mathbb{R}^{n}$. The
Wasserstein-$1$ distance between $P_{r}$ and $P_{g}$ is defined as

$\displaystyle
W(P_{r},P_{g})=\inf_{\pi\in\Pi(P_{r},P_{g})}\mathbb{E}_{(x_{i},x_{j})\sim\pi}[\|x_{i}-x_{j}\|],$
(3)

where the coupling $\pi$ of $P_{r}$ and $P_{g}$ is the probability
distribution on $\mathbb{R}^{n}\times\mathbb{R}^{n}$ with marginals $P_{r}$
and $P_{g}$, and $\Pi(P_{r},P_{g})$ denotes the set of all joint distributions
$\pi$. The Wasserstein distance can be interpreted as the minimum cost of
transporting one probability distribution to another. The Kantorovich-
Rubinstein (KR) duality [37] provides a new way to evaluate the Wasserstein
distance between distributions. The duality states that

$\displaystyle W$
$\displaystyle(P_{r},P_{g})={\sup}_{f}\,\Big{\\{}\mathbb{E}_{x\sim
P_{r}}[f(x)]-\mathbb{E}_{x\sim P_{g}}[f(x)]\Big{\\}},$ (4)
$\displaystyle\emph{s.t. }\,f(x_{i})-f(x_{j})\leq\|x_{i}-x_{j}\|,\,\,\forall
x_{i},\forall x_{j}.$

where the supremum is taken over all functions $f$:
$\mathcal{X}\rightarrow\mathbb{R}$ whose Lipschitz constant is no more than
one.

###  2.3 Wasserstein GAN

The training instability issues of vanilla GAN is considered to be caused by
the unfavorable property of distance metric [3], i.e., the JSD remains
constant when the two distributions are disjoint. Accordingly, [4] proposed
Wasserstein distance in the form of KR duality Eq. (4) as an alternative
objective.

The Wasserstein distance requires to enforce the Lipschitz condition on the
critic network $D$. It has been observed in previous work that imposing
Lipschitz constraint in the critic leads to improved stability and sample
quality [4, 21, 12, 11]. Besides, some researchers also found that applying
Lipschitz continuity condition to the generator can benefit the quality of
generated samples [40, 33]. Formally, in WGAN family, with the objective being
Wasserstein distance, the optimal critic $f^{*}$ under Lipschitz constraint
holds the following property [15]:

######  Proposition 1.

Let $\pi^{*}$ be the optimal coupling in Eq. (3), then the optimal function
$f^{*}$ in KR duality Eq. (4) satisfies that: let $x_{t}=tx_{i}+(1-t)x_{j}$
with $0\leq t\leq 1$, if $f^{*}$ is differentiable and $\pi^{*}(x,x)=0$ for
all $x$, then it holds that $P_{(x_{i},x_{j})\sim\pi^{*}}[\nabla
f^{*}(x_{t})=\frac{x_{i}-x_{j}}{\|x_{i}-x_{j}\|}]=1$.

This property indicates that for each coupling of generated datapoint $x_{j}$
and real datapoint $x_{i}$ in $\pi^{*}$, the gradient at any linear
interpolation between $x_{i}$ and $x_{j}$ is pointing towards the real
datapoint $x_{i}$ with unit norm. Therefore, guided by the gradients, the
generated sample $x_{j}$ would move toward the real sample $x_{i}$. This
property provides the explanation, from the gradient perspective, on why WGAN
can overcome the training instability issue.

###  2.4 Sobolev space

Let $\mathcal{X}$ be a compact space in $\mathbb{R}^{n}$ and let $\mu$(x) to
be a distribution defined on $\mathcal{X}$ as a dominant measure. Functions in
the Sobolev space $W^{1,2}(\mathcal{X},\mu)$ [1] can be written as:

$\displaystyle
W^{1,2}(\mathcal{X},\mu)=\Big{\\{}f:\mathcal{X}\rightarrow\mathbb{R},\int_{\small{\mathcal{X}}}\|\nabla_{x}f(x)\|^{2}\mu(x)dx<\infty\Big{\\}}.$
(5)

Restrict functions to the Sobolev space $W^{1,2}(\mathcal{X},\mu)$ vanishing
at the boundary and denote this space by $W^{1,2}_{0}(\mathcal{X},\mu)$, then
the semi-norm in $W^{1,2}_{0}(\mathcal{X},\mu)$ can be defined as:

$\displaystyle\|f\|_{W^{1,2}_{0}(\mathcal{X},\mu)}=\sqrt{\int_{\mathcal{X}}\|\nabla_{x}f(x)\|^{2}\mu(x)dx}.$
(6)

Given the notion of semi-norm, we can define the Sobolev unit ball constraint
as follows:

$\displaystyle\mathcal{F}_{S}(\mathcal{X},\mu)=\Big{\\{}$ $\displaystyle
f:\mathcal{X}\rightarrow\mathbb{R},\,f\in W^{1,2}_{0}(\mathcal{X},\mu),$ (7)
$\displaystyle\|f\|_{W^{1,2}_{0}(\mathcal{X},\mu)}\leq 1\Big{\\}}.$

Sobolev unit ball is a function class that restricts the square root of
integral of squared gradient norms according to the dominant measure $\mu(x)$.

###  2.5 Sobolev GAN

After WGAN, many works are devoted to improving GAN model by imposing
restrictions on the critic function. Typical instances are the GANs based on
Integral Probability Metric (IPM) [29, 7]. Among them, Sobolev GAN (SGAN) [30]
proposed using Sobolev IPM as the metric for training GANs, which restricts
the critic network $D$ in $\mathcal{F}_{S}(\mathcal{X},\mu)$:

$\displaystyle\mathcal{S}_{\mu}(P_{r},P_{g})=\sup_{f\in\mathcal{F}_{S}(\mathcal{X},\mu)}\Big{\\{}\mathbb{E}_{x\sim
P_{r}}[f(x)]-\mathbb{E}_{x\sim P_{g}}[f(x)]\Big{\\}}.$ (8)

The following choices of measure $\mu$ for $\mathcal{F}_{S}$ are considered,
which we will take as our baselines:

  1. (a)

$\mu=\frac{P_{r}+P_{g}}{2}$: the mixed distribution of $P_{r}$ and $P_{g}$;

  2. (b)

$\mu_{gp}$: $x=tx_{i}+(1-t)x_{j}$, where $x_{i}\sim P_{r}$, $x_{j}\sim P_{g}$
and $t\sim\text{U}[0,1]$, i.e., the distribution defined by the interpolation
lines between $P_{r}$ and $P_{g}$ as in [15].

Let $F_{P_{r}}$ and $F_{P_{g}}$ be the cumulative distribution functions (CDF)
of $P_{r}$ and $P_{g}$ respectively, and assume that the $n$ partial
derivatives of $F_{P_{r}}$ and $F_{P_{g}}$ exist and are continuous. Define
the differential operator $D^{-}=(D^{-1},...,D^{-n})$ where $D^{-i}$ =
$\dfrac{\partial^{n-1}}{\partial x_{1}...\partial x_{i-1}\partial
x_{i+1}...\partial x_{n}}$, which computes $(n-1)$ high-order partial
derivative excluding the $i$-th dimension. Let
$x^{-i}=(x_{1},...,x_{i-1},x_{i+1},...,x_{d})$. According to [30], the Sobolev
IPM in Eq. (8) has the following equivalent form:

$\small\mathcal{S}_{\mu}(P_{r},P_{g})=\frac{1}{n}\sqrt{\mathbb{E}_{x\sim\mu}\sum_{i=1}^{n}\bigg{(}\frac{D^{-i}F_{P_{r}}(x)-D^{-i}F_{P_{g}}(x)}{\mu(x)}\bigg{)}^{2}}.$
(9)

Note that, for each $i$, $D^{-i}F_{P}(x)$ is the cumulative distribution of
the variable $X_{i}$ given the other variables $X^{-i}=x^{-i}$ weighted by the
density function of $X^{-i}$ at $x^{-i}$, _i.e._ ,

$\displaystyle
D^{-i}F_{P}(x)=P_{[X^{-i}]}(x^{-i})F_{P_{[X_{i}|X^{-i}=x^{-i}]}}(x_{i}).$ (10)

Thus, the Sobolev IPM can be seen as a comparison of coordinate-wise
conditional CDFs. Furthermore, [30] also proves that the optimal critic
$f^{*}$ in SGAN holds the following property:

$\small\nabla_{x}f^{*}(x)=\frac{1}{n\mathcal{S}_{\mu}(P_{r},P_{g})}\frac{D^{-}F_{P_{g}}(x)-D^{-}F_{P_{r}}(x)}{\mu(x)}.$
(11)

##  3 Sobolev duality of Wasserstein distance

###  3.1 Sobolev duality

Let $x_{i}$ and $x_{j}$ be two points in $\mathbb{R}^{n}$. The linear
interpolation between $x_{i}$ and $x_{j}$ can be written as
$x=tx_{i}+(1-t)x_{j}$ with $0\leq t\leq 1$. Regarding $x$ as a random variable
on the line between $x_{i}$ and $x_{j}$, we can then define its probability
distribution as $\mu^{x_{i},x_{j}}(x)$, which we will later use as the
dominant measure for Sobolev space. Formally, let $t$ be the random variable
that follows the uniform distribution U$[0,1]$. Then $\mu^{x_{i},x_{j}}(x)$
can be written as:

$\mu^{x_{i},x_{j}}(x)=\begin{cases}\dfrac{1}{\|x_{i}-x_{j}\|},&x=tx_{i}+(t-1)x_{j},\vspace{4pt}\cr
0,&otherwise.\end{cases}$ (12)

With the above defined notation, we propose our new dual form of Wasserstein
distance as follows, which we call _Sobolev duality_ 222We provide the proofs
of _Sobolev duality_ and Proposition 2 in Appendix.

$\displaystyle W$
$\displaystyle(P_{r},P_{g})={\sup}_{f}\,\Big{\\{}\mathbb{E}_{x\sim
P_{r}}[f(x)]-\mathbb{E}_{x\sim P_{g}}[f(x)]\Big{\\}},$ (13)
$\displaystyle\emph{s.t.
}\,f\in\mathcal{F}_{S}(\mathcal{X},\mu^{x_{i},x_{j}}),\,\,\forall x_{i}\sim
P_{r},\forall x_{j}\sim P_{g},$

where $\mathcal{F}_{S}(\mathcal{X},\mu^{x_{i},x_{j}})$ denotes the Sobolev
unit ball of

$\|f\|_{W^{1,2}_{0}(\mathcal{X},\mu^{x_{i},x_{j}})}=\sqrt{\int_{\mathcal{X}}\|\nabla_{x}f(x)\|^{2}\mu^{x_{i},x_{j}}(x)dx}\leq
1.$ (14)

Note that the support of $\mu^{x_{i},x_{j}}(x)$ is the straight line between
$x_{i}$ and $x_{j}$. Thus $\|f\|_{W^{1,2}(\mathcal{X},\mu^{x_{i},x_{j}})}$ is
the square root of the path integral of squared gradient norms from $x_{i}$ to
$x_{j}$. In other words, the constraint is restricting the gradient integral
on each line between $P_{r}$ and $P_{g}$ to be no more than 1.

Corresponding to Proposition 1 of KR duality, we highlight the following
property of the proposed Sobolev duality:

######  Proposition 2.

Let $\pi^{*}$ be the optimal coupling in Eq. (3), then the optimal function
$f^{*}$ in Sobolev duality Eq. (13) satisfies that: let
$x_{t}=tx_{i}+(1-t)x_{j}$ with $0\leq t\leq 1$, if $f^{*}$ is differentiable
and $\pi^{*}(x,x)=0$ for all $x$, then it holds that
$P_{(x_{i},x_{j})\sim\pi^{*}}[\nabla
f^{*}(x_{t})=\frac{x_{i}-x_{j}}{\|x_{i}-x_{j}\|}]=1$.

That is, with Sobolev duality, the gradient direction for every fake datum is
the same as WGAN. Hence, enforcing the Sobolev duality constraint on
discriminator can be an effective alternative of the Lipschitz condition to
guarantee a stable training for GAN model.

###  3.2 Relation to other metrics

Relation to KR duality in Eq. (4). As indicated by Proposition 2, the optimal
critic $f^{*}$ of Sobolev duality actually holds the same gradient property as
KR duality in Proposition 1. However, as clarified below, the constraint in
Sobolev duality is indeed looser than KR duality, which would potentially
benefit the optimization.

In the classic KR duality , $f$ is restricted under Lipschitz condition, i.e.,
the gradient norms of all points in the metric space are enforced to no more
than $1$. By contrast, in our Sobolev duality, we restrict the integral of
squared gradient norms over each line between $P_{r}$ and $P_{g}$. This
implies that Lipschitz continuity is a sufficient condition of the constraint
in Sobolev duality. In summary, Sobolev duality is a generalization of KR
duality where the constraint is relaxed, while still keeps the same property
of training stability.

Relation to Sobolev IPM in Eq. (8). We now clarify the difference between
Sobolev IPM in Eq. (8) and Sobolev duality of Wasserstein distance in Eq.
(13). In the former metric, when implementing $\mu_{gp}$ (defined in Section
2.5), the total integral of squared gradient norms on all interpolation lines
between $P_{r}$ and $P_{g}$ is enforced to no more than $1$; while in the
latter metric, the integral over each interpolation line between $P_{r}$ and
$P_{g}$ is restricted. Therefore, Sobolev duality enforces stronger constraint
than Sobolev IPM.

However, we should also note that the stronger constraint is necessary to
ensure the favorable gradient property in Proposition 2. By contrast, as shown
in Eq. (11), Sobolev IPM measures coordinate-wise conditional CDF, which
cannot always provide gradients as good as the optimal transport plan in
Wasserstein distance. A toy example is provided in Appendix to show the case
that Sobolev IPM is sometimes insufficiently constrained to ensure the
convergence.

##  4 Sobolev Wasserstein GAN

Now we define the GAN model with Sobolev duality, which we name as Sobolev
Wasserstein GAN (SWGAN). Formally, SWGAN can be written as:

$\displaystyle\min_{G}\max_{D}\mathcal{L}_{S}$
$\displaystyle(D_{w},G_{\theta})$ $\displaystyle=\mathbb{E}_{x\sim
P_{r}}D_{w}(x)-\mathbb{E}_{z\sim P_{z}}D_{w}(G_{\theta}(z)),$ (15)

with the constraint that

$\displaystyle\mathbb{E}_{x\sim\mu^{x_{i},x_{j}}}\|\nabla_{x}D_{w}(x)\|^{2}\leq
1,\forall x_{i}\sim P_{r},\forall x_{j}\sim P_{g},$ (16)

where $\mu^{x_{i},x_{j}}$ is the interpolation distribution on lines between
pairs of points $x_{i}$ and $x_{j}$ as defined in Eq. (12).

Let $\Omega_{ij}$ denote
$1-\mathbb{E}_{x\sim\mu^{x_{i},x_{j}}}\|\nabla_{x}D_{w}(x)\|^{2}$, then the
constraint is to restrict $\Omega_{ij}$ to be greater than or equal to 0 for
all the pairs of $(x_{i},x_{j})$. Inspired by [30], we define the following
Augmented Lagrangian inequality regularization [32] corresponding to SWGAN
ball constraints:

$\displaystyle\mathcal{L}^{(ij)}_{al}(w,\theta,\alpha)=\alpha(\Omega_{ij}-s_{ij})-\frac{\rho}{2}(\Omega_{ij}-s_{ij})^{2},$
(17) $\displaystyle\mathcal{L}_{al}(w,\theta,\alpha)=\mathbb{E}_{x_{i}\sim
P_{r}}\mathbb{E}_{x_{j}\sim P_{g}}\mathcal{L}^{(ij)}_{al}(w,\theta,\alpha).$

where $\alpha$ is the Lagrange multiplier, $\rho$ is the quadratic penalty
weight and $s_{ij}$ represents the slack variables. Practically, $s_{ij}$ is
directly substituted by its optimal solution:

$s^{*}_{ij}=\max\Big{\\{}\Omega_{ij}-\frac{\alpha}{\rho}\,,\,0\Big{\\}}.$ (18)

As in [4] and [30], the regularization term in Eq. (17) is added to the loss
only when training the critic. To be more specific, the training process is:
given the generator parameters $\theta$, we train the discriminator by
maximizing $\mathcal{L}_{S}+\mathcal{L}_{al}$; then given the discriminator
parameters $w$, we train the generator via minimizing $\mathcal{L}_{S}$. We
leave the detailed training procedure in Appendix.

##  5 Experiments

We tested SWGAN on both synthetic density modeling and real-world image
generation task.

Figure 1: 1D Training comparison. Upper: WGAN. Lower: SWGAN. Orange: real
data, sampled from $N(0,1)$. Green: fake data, sampled from
$\frac{1}{2}(N(-5,1)+N(5,1))$. Blue: the re-scaled critic, which is normalized
to $[0,0.5]$. From left to right, different columns correspond to iteration
$0,30,60,90,120,180,240,300$ respectively. The critic of SWGAN holds a faster
and smoother convergence.

###  5.1 Synthetic density modeling

1D distribution modeling. Displaying the level sets is a standard qualitative
approach to evaluate the learned critic function for two-dimensional data sets
[15, 20, 34]. Here, we consider both real data distribution $P_{r}$ and
generated fake distribution $P_{g}$ are fixed simple one-dimensional Gaussian
distributions. Our goal is to investigate whether the critics of both WGAN and
SWGAN can be efficiently optimized to provide the favorable gradients
presented in Proposition 1 and 2. We observed that while both critics can be
trained to the theoretical optimum, the latter one can always enjoy a faster
convergence compared with the former one. An empirical example is visualized
in Fig. 1. As shown here, with the same initial states and hyper-parameters,
the critic of SWGAN holds a faster and smoother convergence towards the
optimal state. This meaningful observation verifies our conjecture that a
larger function space of the critic would benefit the training.

(a) 8 Gaussians

(b) 25 Gaussians

(c) Swiss Roll

Figure 2: Level sets of SWGANs critic. Yellow corresponds to high values and
purple to low. The training samples are indicated in red and the generated
distribution is fixed at the real data plus Gaussian noise.

Level sets of the critic In this section we give another 2D level sets
visualization. As analyzed in Section 3.1, Sobolev constraint in SWGAN is a
generalization of Lipschitz constraint. Therefore, theoretically SWGAN critic
should also be capable of modeling more challenging real and fake data and
providing meaningful gradients. To demonstrate this, we train SWGAN critics to
optimality on several toy distributions. The value surfaces of the critics are
plotted in Figure 2, which shows good fitness of the distribution following
our theoretical analysis.

###  5.2 Real-world image generation

####  5.2.1 Experimental setup

[b]  GANs CIFAR-10 Tiny-ImageNet IS FID IS FID WGAN-GP∗ 7.85$\pm$.07
18.21$\pm$.12 8.17$\pm$.03 18.70$\pm$.05 WGAN-GP with $m=8$ 7.88$\pm$.09
18.08$\pm$.22 8.17$\pm$.04 18.69$\pm$.10 WGAN-AL 7.79$\pm$.09 17.86$\pm$.16
8.26$\pm$.03 18.70$\pm$.06 WGAN-AL with $m=8$ 7.89$\pm$.09 17.52$\pm$.27
8.31$\pm$.02 18.61$\pm$.09 SGAN∗, $\mu=\frac{P_{r}+P_{g}}{2}$ 7.81$\pm$.11
17.89$\pm$.27 8.30$\pm$.04 18.90$\pm$.04 SGAN∗, $\mu=\mu_{GP}$ 7.83$\pm$.10
18.03$\pm$.24 8.31$\pm$.03 18.90$\pm$.08 SGAN, $\mu=\mu_{GP}$ with $m=8$
7.86$\pm$.09 17.74$\pm$.24 8.33$\pm$.03 18.75$\pm$.07 SWGAN-GP 7.98$\pm$.08
17.50$\pm$.19 8.38$\pm$.03 18.50$\pm$.03 SWGAN-AL 7.93$\pm$.09 16.75$\pm$.24
8.41$\pm$.03 18.32$\pm$.05

  * *

denotes the vanilla version of our baselines.

Table 1: Performance of GANs on CIFAR-10 and Tiny-ImageNet.

Controlled variables. To make the comparisons more convincing, we also include
extended versions of the existing GAN models to control the contrastive
variables. The controlled variables includes:

  * •

Sampling size. In SWGAN-AL we need to sample $m$ points on each interpolation
line between $P_{r}$ and $P_{g}$, while in WGAN-GP [15] and SGAN [30] only one
point is sampled. To yield a more fair comparison, we perform additional
experiments of WGAN and SGAN with the sampling size equal to $m$.

  * •

Optimization method. In our baseline WGAN-GP [15], the restriction is imposed
by Penalty Method (PM). By contrast, SGAN and SWGAN-AL use Augmented
Lagrangian Method (ALM). ALM is a more advanced algorithm than PM for strictly
imposing the constraint. To see the practical difference, we add experiment
settings of SWGAN with penalty regularization term (named SWGAN-GP). Formally,
the penalty can be written as:

$\displaystyle\mathcal{L}_{gp}$
$\displaystyle(w,\theta)=-\lambda\;\mathbb{E}_{x_{i}\sim
P_{r}}\mathbb{E}_{x_{j}\sim P_{g}}\Omega_{ij}^{2}(D_{w},G_{\theta}),$ (19)

where $\lambda$ is the gradient penalty coefficient. $\mathcal{L}_{gp}$ is the
alternative term of the ALM penalty $\mathcal{L}_{al}$ in Eq. (17) for the
training of SWGAN-GP.

Baselines. For comparison, we also evaluated the WGAN-GP [15] and Sobolev GAN
(SGAN) [30] with different sampling sizes and penalty methods. The choice of
baselines is due to their close relation to SWGAN as analyzed in Section 3.2.
We omit other previous methods since as a representative of state-of-the-art
GAN model, WGAN-GP has been shown to rival or outperform a number of former
methods, such as the original GAN [13], Energy-based generative adversarial
network [41], the original WGAN with weight clipping [4], Least Squares GAN
[24], Boundary equilibrium GAN [8] and GAN with denoising feature matching
[38].

Evaluation metrics. Since GAN lacks the capacity to perform reliable
likelihood estimations [36], we instead concentrate on evaluating the quality
of generated images. We choose to compare the maximal Frechet Inception
Distances (FID) [18] and Inception Scores [35] reached during training
iterations, both computed from 50K samples. A high image quality corresponds
to high Inception and low FID scores. Specifally, IS is defined as
$\exp(\mathbb{E}_{x}\operatorname{KL}(p(y|x)||p(y)))$, where $p(y|x)$ is the
distribution of label $y$ conditioned on generated data $x$, and $p(y)$ is the
marginal distribution. IS combines both the confidence of the class
predictions for each synthetic images (quality) and the integral of the
marginal probability of the predicted classes (diversity). The classification
probabilities were estimated by the Inception model [35], a classifier pre-
trained upon the ImageNet dataset [10]. However, in practice we note that IS
is hard to detect the mode collapse problems. FID use the same Inception model
to capture computer-vision-specific features of a collection of real and
generated images, and then calculate the Frechet distance (also called
$Wasserstein$-$2$ distance) [5] between two activation distributions. The
intuition of IS is that high-quality images should lead to high confidence in
classification, while FID is aiming to measure the computer-vision-specific
similarity of generated images to real ones through Frechet distance (a.k.a.,
Wasserstein-$2$ distance) [5].

Data. We test different GANs on CIFAR-10 [22] and Tiny-ImageNet [10] , which
are standard datasets widely used in GANs literatures. Both datasets consist
of tens of thousands of real-world color images with class labels.

Network architecture. For all experimental settings, we follow WGAN-GP [15]
and adopt the same Residual Network (ResNet) [16] structures and
hyperparameters. Details of the architecture are provided in Appendix.

Other implementation details. For SWGAN metaparameter, we choose $8$ as the
sample size $m$. Adam optimizer [19] is set with learning rate decaying from
$2\cdot 10^{-4}$ to $0$ over 100K iterations with $\beta_{1}=0,\beta_{2}=0.9$.
We used $5$ critic updates per generator update, and the batch size used was
$64$.

Figure 3: Generated CIFAR-10 samples.

Figure 4: Generated Tiny-ImageNet samples.

Figure 5: Training curves on CIFAR-10.

Figure 6: The comparison of SWGANs with different regularization terms and
parameters. Top: Inception scores. Bottom: Frechet Inception Distances.

####  5.2.2 Results

We here also introduce WGAN with Augmented Lagrangian Method (WGAN-AL) for
further comparison, which is similar to SGAN [30]. Scores in terms of FID and
IS on CIFAR-10 and Tiny-ImageNet are reported in Table 1. Some representative
samples from the resulting generator of SWGAN are provided in Fig. 4 and Fig.
4. Some representative samples from the resulting generator of SWGAN are
provided in Appendix. In experiments, we note that IS is remarkably unstable
during training and among different initializations, while FID is fairly
stable. So we plot the training curves in terms of FID on CIFAR-10 in Figure 6
to show the training process of different GANs.

From Table 1 and Figure 6, we can see that SWGANs generally work better than
the baseline models. The experimental results also show that WGAN and SGAN
tend to have slightly better performance when using ALM or sample more
interpolation points. However, compared with SWGAN-AL and SWGAN-GP, the
performances in these cases are still not competitive enough. This indicates
that the larger sampling size and ALM optimization algorithm are not the key
elements for the better performance of SWGAN, i.e., these results evidence
that it is the relaxed constraint in Sobolev duality that leads to the
improvement, which is in accordance with our motivation that a looser
constraint would simplify the constrained optimization problem and lead to a
stronger GAN model.

We further test SWGAN with different regularization terms and parameters on
CIFAR-10. The scores are shown in Figure 6. As shown in Figure 6, generally
ALM is a better choice when considering FID, while GP is better for IS. A
meaningful observation is that SWGAN is not sensitive to different values of
penalty weights $\rho$ and $\lambda$. By contrast, a previous large scale
study reported that the performance of WGAN-GP holds strong dependence on the
penalty weight $\lambda$ (see Figure 8 and 9 in [23]). This phenomenon
demonstrates a more smooth and stable convergence and well-behaved critic
performance throughout the whole training process of SWGAN.

##  6 Local Convergence Analysis

(a) SimGD

(b) AltGD($n_{d}$=$1$)

(c) AltGD($n_{d}$=$5$)

(d) SimGD

(e) AltGD($n_{d}$=$1$)

(f) AltGD($n_{d}$=$5$)

Figure 7: Training behavior of SWGANs with different gradient penalty and
gradient desent method. The starting iterate is marked in red. Upper: SWGAN-
GP. Bottom: SWGAN-CP. First column: update the generator and critic
simultaneously. Second and third column: $n_{d}$ critic updates per generator
update.

Some literature pay attention to the sub-optimal convergence problem of GANs
[25, 26, 6, 11], which is an important direction that deserves further
investigation. Moreover, [26] recently shows that in some toy case that
distributions that are not absolutely continuous, WGAN based models are not
always convergent.

We employ the same prototypical counterexample to further analyze the
convergence of SWGAN: the real data distribution $P_{r}$ is defined as a
Dirac-distribution concentrated at 0; generator distribution is set as
$P_{g}=\delta_{\theta}$ and the linear critic is set as $D_{w}(x)=w\cdot x$.
We also follow them to focus on investigating the gradient penalty version
(SWGAN-GP). In the toy case, the regularizer term $\mathcal{L}_{gp}$ in Eq.
(19) can be simplified to:

$\mathcal{L}_{gp}(w)=-\lambda\left(w^{2}-g_{0}\right)^{2},$ (20)

where $g_{0}$ is the gradient penalty target that set as $1$ in SWGAN-GP.
Following the previous work, we set the object $\mathcal{L}(w,\theta)$ as
$\mathcal{L}_{S}(w,\theta)+\mathcal{L}_{gp}(w)$ and then study the sub-optimal
convergence problem through the gradient vector field:

$v_{gp}(w,\theta):=\left(\begin{array}[]{c}{-\nabla_{\theta}\mathcal{L}(w,\theta)}\\\
{\nabla_{w}\mathcal{L}(w,\theta)}\end{array}\right)=\left(\begin{array}[]{c}{-w}\\\
{\theta-4w\lambda\left(w^{2}-g_{0}\right)}\end{array}\right).$ (21)

In practice, GANs are usually trained through either Simultaneous or
Alternating Gradient Descent (SimGD and AltGD), which means updating the
parameters $(w,\theta)$ according to the gradient field simultaneously or
alternately. [26] proves that the unique equilibrium point in the above
defined case is $w=\theta=0$. However, in Eq. (21) the gradient field shows a
discontinuity at the equilibrium point, which indicates that similar to WGAN-
GP, SWGAN-GP also cannot converge locally in this case (Fig. 7(a), 7(b), and
7(c)). To bypass this minor sub-optimal problem, we design a zero-centered
gradient penalty (SWGAN-CP) by simply setting the $g_{0}$ in Eq. (20) as $0$
and adding another penalty on the squared norm of the gradients (Fig. 7(d),
7(e), and 7(f)). The proof for the local convergence of SWGAN-CP is provided
in Appendix.

However, as acknowledged by [26], despite these minor ill-behaviors, WGAN has
been successfully applied in practice. Furthermore, in our real-world
experiments, we also noticed that there are few positive improvements when
taking zero-centered penalty as the regularization term.

##  7 Conclusion

In this paper, we proposed a new dual form of Wasserstein distance with the
Lipschitz constraint relaxed and demonstrate that it is still capable of
eliminating the training instability issues. This new dual form leads to a
generalized WGAN model. We built Sobolev Wasserstein GAN based on the proposed
duality and provided empirical evidence that our GAN model outperforms the
previous approaches, which either impose the strong Lipschitz penalty or
cannot theoretically guarantee the convergence.

This work was motivated by the intuition that with a less restricted function
space, the critic would be easier to be trained to optimum, thus benefiting
the training of GANs. To the best of our knowledge, Sobolev Wasserstein GAN is
the GAN model with the most relaxed restriction that can still avoid the
training instability problem. In the future, we hope that practitioners can
take a step back and investigate whether we can further relax the constraint
imposed on the function space of the critic and what the minimal requirement
for the convergence guarantee could be.

## References

  * Adams and Fournier [2003] Robert A Adams and John JF Fournier.  Sobolev spaces, volume 140.  Elsevier, 2003.
  * Adler and Lunz [2018] Jonas Adler and Sebastian Lunz.  Banach wasserstein gan.  In NIPS, 2018.
  * Arjovsky and Bottou [2017] Martin Arjovsky and Léon Bottou.  Towards principled methods for training generative adversarial networks.  In ICLR, 2017.
  * Arjovsky et al. [2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In ICML, 2017.
  * Aronov et al. [2006] Boris Aronov, Sariel Har-Peled, Christian Knauer, Yusu Wang, and Carola Wenk.  Fréchet distance for curves, revisited.  In European Symposium on Algorithms. Springer, 2006.
  * Arora et al. [2017] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang.  Generalization and equilibrium in generative adversarial nets (gans).  arXiv preprint arXiv:1703.00573, 2017.
  * Bellemare et al. [2017] Marc G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos.  The cramer distance as a solution to biased wasserstein gradients.  arXiv preprint arXiv:1705.10743, 2017.
  * Berthelot et al. [2017] David Berthelot, Thomas Schumm, and Luke Metz.  Began: Boundary equilibrium generative adversarial networks.  arXiv preprint arXiv:1703.10717, 2017.
  * Che et al. [2016] Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li.  Mode regularized generative adversarial networks.  arXiv preprint arXiv:1612.02136, 2016.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR. Ieee, 2009.
  * Farnia and Tse [2018] Farzan Farnia and David Tse.  A convex duality framework for gans.  In NIPS, 2018.
  * Fedus et al. [2017] William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M Dai, Shakir Mohamed, and Ian Goodfellow.  Many paths to equilibrium: Gans do not need to decrease a divergence at every step.  arXiv preprint arXiv:1710.08446, 2017.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NIPS, 2014.
  * Goodfellow [2016] Ian Goodfellow.  Nips 2016 tutorial: Generative adversarial networks.  arXiv preprint arXiv:1701.00160, 2016.
  * Gulrajani et al. [2017] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein gans.  In NIPS, 2017.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * Heusel et al. [2017a] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NIPS, 2017.
  * Heusel et al. [2017b] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a nash equilibrium.  arXiv preprint arXiv:1706.08500, 2017.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * Kodali et al. [2017a] Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira.  How to train your dragan.  arXiv preprint arXiv:1705.07215, 2(4), 2017.
  * Kodali et al. [2017b] Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira.  On convergence and stability of gans.  arXiv preprint arXiv:1705.07215, 2017.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  Technical report, Citeseer, 2009.
  * Lucic et al. [2017] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet.  Are gans created equal? a large-scale study.  arXiv preprint arXiv:1711.10337, 2017.
  * Mao et al. [2017] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In CVPR, 2017.
  * Mescheder et al. [2017] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger.  The numerics of gans.  In NIPS, 2017.
  * Mescheder et al. [2018] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.  Which training methods for gans do actually converge?  In ICML, 2018.
  * Metz et al. [2016] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein.  Unrolled generative adversarial networks.  arXiv preprint arXiv:1611.02163, 2016.
  * Miyato et al. [2018] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In ICLR, 2018.
  * Mroueh and Sercu [2017] Youssef Mroueh and Tom Sercu.  Fisher gan.  In NIPS, 2017.
  * Mroueh et al. [2018] Youssef Mroueh, Chun-Liang Li, Tom Sercu, Anant Raj, and Yu Cheng.  Sobolev GAN.  In ICLR, 2018.
  * Nagarajan and Kolter [2017] Vaishnavh Nagarajan and J Zico Kolter.  Gradient descent gan optimization is locally stable.  In NIPS, 2017.
  * Nocedal and Wright [2006] Jorge Nocedal and Stephen Wright.  Numerical optimization.  Springer Science & Business Media, 2006.
  * Odena et al. [2018] Augustus Odena, Jacob Buckman, Catherine Olsson, Tom B Brown, Christopher Olah, Colin Raffel, and Ian Goodfellow.  Is generator conditioning causally related to gan performance?  arXiv preprint arXiv:1802.08768, 2018.
  * Petzka et al. [2018] Henning Petzka, Asja Fischer, and Denis Lukovnikov.  On the regularization of wasserstein GANs.  In ICLR, 2018.
  * Salimans et al. [2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In NIPS, 2016.
  * Theis et al. [2015] Lucas Theis, Aäron van den Oord, and Matthias Bethge.  A note on the evaluation of generative models.  arXiv preprint arXiv:1511.01844, 2015.
  * Villani [2008] C. Villani.  Optimal Transport: Old and New.  Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2008.
  * Warde-Farley and Bengio [2016] David Warde-Farley and Yoshua Bengio.  Improving generative adversarial networks with denoising feature matching.  2016\.
  * Yadav et al. [2017] Abhay Yadav, Sohil Shah, Zheng Xu, David Jacobs, and Tom Goldstein.  Stabilizing adversarial nets with prediction methods.  arXiv preprint arXiv:1705.07364, 2017.
  * Zhang et al. [2018] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  arXiv preprint arXiv:1805.08318, 2018.
  * Zhao et al. [2016] Junbo Zhao, Michael Mathieu, and Yann LeCun.  Energy-based generative adversarial network.  arXiv preprint arXiv:1609.03126, 2016.
  * Zhou et al. [2019] Zhiming Zhou, Jiadong Liang, Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang.  Lipschitz generative adversarial nets.  In ICML, 2019.

##  Appendix A Algorithm for SWGAN-AL

Algorithm 1 Sobolev Wasserstein GAN with Augmented Lagrangian

Input: $\rho$ penalty weight, $\eta$ learning rate, $k$ number of critic
iterations per generator iteration, $n$ batch size, $m$ number of points
sampled between each pair of fake and real data

Initial: critic parameters $w$, generator parameters $\theta$, Lagrange
multiplier $\alpha$

1: while $\theta$ has not converged do

2: for $l=1,...,k$ do

3: Sample a minibatch $x_{i}$, $i=1,...,n$, $x_{i}\sim P_{r}$

4: Sample a minibatch $z_{i}$, $i=1,...,n$, $z_{i}\sim P_{z}$

5: $\widetilde{x}_{i}\leftarrow G_{\theta}(z_{i})$, $i=1,...,n$

6: for $i=1,...,n$ do

7: Sample a minibatch $\hat{x}_{ij}=t_{j}x_{i}+(1-t_{j})\widetilde{x}_{i}$,
$j=1,...,m$, $t_{j}\sim\text{U}(0,1)$

8: $\Omega^{(i)}\leftarrow
1-\frac{1}{m}\sum_{j=1}^{m}\|\nabla_{x}D(\hat{x}_{ij})\|^{2}$

9:
$s^{(i)}\leftarrow\max\big{\\{}\Omega^{(i)}-\frac{\alpha}{\rho}\,,\,0\big{\\}}$.

10:
$\mathcal{L}_{al}^{(i)}\leftarrow\alpha(\Omega^{(i)}-s^{(i)})-\frac{\rho}{2}(\Omega^{(i)}-s^{(i)})^{2}$

11: end for

12:
$\mathcal{L}\leftarrow\frac{1}{n}\sum_{i=1}^{n}\big{\\{}D_{w}(x_{i})-D_{w}(\widetilde{x}_{i})+\mathcal{L}_{al}^{(i)}\big{\\}}$

13:
$(g_{w},g_{\alpha})\leftarrow(\nabla_{w}\mathcal{L},\nabla_{\alpha}\mathcal{L})(w,\theta,\alpha)$

14: $w\leftarrow w+\eta\text{ADAM}(w,g_{w})$

15: $\alpha\leftarrow\max\big{\\{}\alpha-\rho g_{\alpha},0\big{\\}}$

16: end for

17: Sample a minibatch $z_{i}$, $i=1,...,n$, $z_{i}\sim P_{z}$

18:
$g_{\theta}\leftarrow-\nabla_{\theta}\frac{1}{n}\sum_{i=1}^{n}D_{w}(G_{\theta}(z_{i}))$

19: $\theta\leftarrow\theta-\eta\text{ADAM}(\theta,g_{\theta})$

20: end while

##  Appendix B Proofs for _Sobolev duality_

###  B.1 Proof of the _Sobolev duality_ of Wasserstein Distance

We here provide a proof for our new dual form of Wasserstein distance.

The Wasserstein distance is given as follows

$W_{1}(P_{r},P_{g})=\inf_{\pi\in\Pi(P_{r},P_{g})}\,\mathbb{E}_{(x_{i},x_{j})\sim\pi}\,[\|x_{i}-x_{j}\|],$
(22)

where $\Pi(P_{r},P_{g})$ denotes the set of all probability measures with
marginals $P_{r}$ and $P_{g}$ on the first and second factors, respectively.
The Kantorovich-Rubinstein (KR) dual is written as

$\displaystyle W_{KR}$
$\displaystyle(P_{r},P_{g})={\sup}_{f}\,\,\Big{\\{}\mathbb{E}_{x_{i}\sim
P_{r}}\,[f(x_{i})]-\mathbb{E}_{x_{j}\sim P_{g}}\,[f(x_{j})]\Big{\\}},\,$ (23)
$\displaystyle\emph{s.t.}\,f(x_{i})-f(x_{j})\leq\|x_{i}-x_{j}\|,\,\,\forall
x_{i},\forall x_{j}.$

We will prove that Wasserstein distance in its dual form can also be written
in _Sobolev duality_ :

$\displaystyle W_{S}$
$\displaystyle(P_{r},P_{g})={\sup}_{f}\,\Big{\\{}\mathbb{E}_{x\sim
P_{r}}[f(x)]-\mathbb{E}_{x\sim P_{g}}[f(x)]\Big{\\}},$ (24)
$\displaystyle\emph{s.t.}\,f\in\mathcal{F}_{S}(\mathcal{X},\mu^{x_{i},x_{j}}),\,\,\forall
x_{i}\sim P_{r},\forall x_{j}\sim P_{g},$

where

$\mu^{x_{i},x_{j}}(x)=\begin{cases}\dfrac{1}{\|x_{i}-x_{j}\|},&x=tx_{i}+(t-1)x_{j},\vspace{+5pt}\cr
0,&otherwise.\end{cases}$ (25)

which relaxes the Lipschitz constraint in the KR dual form of Wasserstein
distance.

######  Theorem 1.

Given $W_{KR}(P_{r},P_{g})=W_{1}(P_{r},P_{g})$, we have
$W_{KR}(P_{r},P_{g})=W_{S}(P_{r},P_{g})=W_{1}(P_{r},P_{g})$.

###### Proof.

(i) As illustrated in Section 3.2.1, Lipschitz continuity is a sufficient
condition of the constraint in Sobolev dual. Therefor, for any $f$ that
satisfies “$f(x_{i})-f(x_{j})\leq\|x_{i}-x_{j}\|,\,\forall x_{i},\forall
x_{j}$”, it must satisfy
“$f\in\mathcal{F}_{S}(\mathcal{X},\mu^{x_{i},x_{j}}),\,\forall x_{i}\sim
P_{r},\forall x_{j}\sim P_{g}$”. Thus, $W_{KR}(P_{r},P_{g})\leq
W_{S}(P_{r},P_{g})$.

(ii) Let a curve $\mathcal{C}$ be the straight line from $x_{j}$ to $x_{i}$
with its tangent vector at some chosen point defined as $\mathbf{v}$, i.e.,
$\mathbf{v}=\frac{x_{i}-x_{j}}{\|x_{i}-x_{j}\|}$. Let $P_{t}$ be U$[0,1]$ and
let $t$ be the random variable that follows $P_{t}$. Let
$\mathbf{r}(t)=tx_{i}+(1-t)x_{j}$, which is a bijective parametrization of the
curve $\mathcal{C}$ such that $\mathbf{r}(0)$ and $\mathbf{r}(1)$ give the
endpoints of $\mathcal{C}$. Given
$f\in\mathcal{F}_{S}(\mathcal{X},\mu^{x_{i},x_{j}})$, we have:

$\displaystyle f(x_{i})-f(x_{j})$
$\displaystyle=\int_{\mathcal{C}}\mathbf{v}\cdot\nabla_{x}f(\mathbf{r})d\mathbf{r}$
$\displaystyle\leq\int_{\mathcal{C}}\|\nabla_{x}f(\mathbf{r})\|d\mathbf{r}$
(26) (Directional derivative is less than norm of gradient)
$\displaystyle=\int_{\mathcal{C}}\|\nabla_{x}f(\mathbf{r})\|\sqrt{\mu^{x_{i},x_{j}}(\mathbf{r})}\frac{1}{\sqrt{\mu^{x_{i},x_{j}}(\mathbf{r})}}d\mathbf{r}$
(27)
$\displaystyle\leq\sqrt{\int_{\mathcal{C}}\|\nabla_{x}f(\mathbf{r})\|^{2}\mu^{x_{i},x_{j}}(\mathbf{r})d\mathbf{r}}\sqrt{\int_{\mathcal{C}}{\frac{1}{\mu^{x_{i},x_{j}}(\mathbf{r})}d\mathbf{r}}}$
(28) (Cauchy–Schwarz inequality)
$\displaystyle\leq\sqrt{\int_{\mathcal{C}}\frac{1}{\mu^{x_{i},x_{j}}(\mathbf{r})}d\mathbf{r}}$
(29) ($f\in\mathcal{F}_{S}(\mathcal{X},\mu^{x_{i},x_{j}})$, see Eq. (14) in
paper)
$\displaystyle=\sqrt{\int_{0}^{1}\frac{1}{\mu^{x_{i},x_{j}}(\mathbf{r(t)})}|\mathbf{r}^{\prime}(t)|dt}$
$\displaystyle=\sqrt{\int_{0}^{1}\|x_{i}-x_{j}\||\mathbf{r}^{\prime}(t)|dt}$
(see Eq. (25) ) $\displaystyle=\|x_{i}-x_{j}\|$
($\mathbf{r}(t)=tx_{i}+(1-t)x_{j}$)

Let $F_{S}=\\{f|\,f(x_{i})-f(x_{j})\leq\|x_{i}-x_{j}\|,\,\,\forall x_{i}\sim
P_{r},\forall x_{j}\sim P_{g}\\}$.

Let
$A=\\{(x_{i},x_{j})\,|\,x_{i}\in\mathcal{S}_{r},x_{j}\in\mathcal{S}_{g}\\}$
and $I_{A}=\begin{cases}1,\quad(x_{i},x_{j})\in A;\\\ 0,\quad
otherwise\end{cases}$.

Let $A^{c}$ denote the complementary set of $A$ and define $I_{A^{c}}$
accordingly.

For $\forall\pi\in\Pi(P_{r},P_{g})$, we have the following:

$\displaystyle W_{S}(P_{r},P_{g})$ $\displaystyle\leq{\sup}_{f\in
F_{S}}\,\,\mathbb{E}_{x\sim P_{r}}\,[f(x)]-\mathbb{E}_{x\sim P_{g}}\,[f(x)]$
(31) $\displaystyle={\sup}_{f\in
F_{S}}\,\,\mathbb{E}_{(x_{i},x_{j})\sim\pi}[f(x_{i})-f(x_{j})]$
$\displaystyle={\sup}_{f\in
F_{S}}\,\,\mathbb{E}_{(x_{i},x_{j})\sim\pi}[(f(x_{i})-f(x_{j}))I_{A}]+\mathbb{E}_{(x_{i},x_{j})\sim\pi}[(f(x_{i})-f(x_{j}))I_{A^{c}}]$
$\displaystyle={\sup}_{f\in
F_{S}}\,\,\mathbb{E}_{(x_{i},x_{j})\sim\pi}[(f(x_{i})-f(x_{j}))I_{A}]$
$\displaystyle\leq\mathbb{E}_{(x_{i},x_{j})\sim\pi}[\|x_{j}-x_{i}\|I_{A}]$
$\displaystyle\leq\mathbb{E}_{(x_{i},x_{j})\sim\pi}[\|x_{i}-x_{j}\|].$

$W_{S}(P_{r},P_{g})\leq\mathbb{E}_{(x,y)\sim\pi}[\|x_{i}-x_{j}\|],\forall\pi\in\Pi(P_{r},P_{g})$

$\Rightarrow
W_{S}(P_{r},P_{g})\leq\inf_{\pi\in\Pi(P_{r},P_{g})}\,\mathbb{E}_{(x,y)\sim\pi}\,[\|x_{i}-x_{j}\|]=W_{1}(P_{r},P_{g})$.

(iii) Combining (i) and (ii), we have $W_{KR}(P_{r},P_{g})\leq
W_{S}(P_{r},P_{g})\leq W_{1}(P_{r},P_{g})$.

Given $W_{KR}(P_{r},P_{g})=W_{1}(P_{r},P_{g})$, we have
$W_{KR}(P_{r},P_{g})=W_{S}(P_{r},P_{g})=W_{1}(P_{r},P_{g})$. ∎

###  B.2 Proof of Proposition 1

######  Proposition 3.

Let $\pi^{*}$ be the optimal coupling in Eq. (22), then the optimal function
$f^{*}$ in Sobolev duality Eq. (24) satisfies that: let
$x_{t}=tx_{i}+(1-t)x_{j}$ with $0\leq t\leq 1$, if $f^{*}$ is differentiable
and $\pi^{*}(x,x)=0$ for all $x$, then it holds that
$P_{(x_{i},x_{j})\sim\pi^{*}}[\nabla
f^{*}(x_{t})=\frac{x_{i}-x_{j}}{\|x_{i}-x_{j}\|}]=1$.

######  Lemma 1.

Let $\pi^{*}$ be the optimal transport plan in Eq. (22) and $f^{*}$ be the
optimal discriminative function in Eq. (24). It holds that

${P_{(x_{i},x_{j})\sim\pi^{*}}}\Big{[}f^{*}(x_{i})-f^{*}(x_{j})=\|x_{i}-x_{j}\|\Big{]}=1.$
(32)

###### Proof.

(i) According to [37], the optimal $f$ in KR duality Eq. (23) satisfies the
property in Eq. (32).

In addition, as illustrated in Section 3.2.1, 1-Lipschitz condition in Eq.
(23) is the sufficient condition of Sobolev duality Eq. (24). Therefore, the
function space of $f$ in KR duality Eq. (23) is a subspace of the function
space defined in Sobolev duality Eq. (24).

(ii) Recently, [42] proposed another relaxed dual form of Wasserstein
distance. It states that:

$\displaystyle W_{L}$
$\displaystyle(P_{r},P_{g})={\sup}_{f}\,\,\Big{\\{}\mathbb{E}_{x_{i}\sim
P_{r}}\,[f(x_{i})]-\mathbb{E}_{x_{j}\sim P_{g}}\,[f(x_{j})]\Big{\\}},\,$ (33)
$\displaystyle\emph{s.t.}\,f(x_{i})-f(x_{j})\leq\|x_{i}-x_{j}\|,\,\,\forall
x_{i}\sim\mathcal{S}_{r},\forall x_{j}\sim\mathcal{S}_{g}.$

It is worthy noticing that any $f^{*}$ under this constraint corresponds to
one $f^{*}$ in Eq. (23) with the value of $f^{*}$ on $\mathcal{S}_{r}$ and
$\mathcal{S}_{g}$ unchanged. Thus, any $f^{*}$ under the constraint in Eq.
(33) also holds the property in Eq. (32).

According to Part (ii) of our proof in Section B.1, given the constraint
$f\in\mathcal{F}_{S}(\mathcal{X},\mu^{x_{i},x_{j}})$ in Sobolev duality Eq.
(24), we can get the constraint in Eq. (33) that
$f(x_{i})-f(x_{j})\leq\|x_{i}-x_{j}\|$. Therefore, the restriction in Sobolev
duality is the sufficient condition of Eq. (33), i.e., the function space of
$f$ in Sobolev duality Eq. (24) is a subspace of the function space defined in
Eq. (33).

(iii) Combining (i) and (ii), we have that

  * •

The function space of $f$ in KR duality Eq. (23) is a subspace of the function
space defined in Sobolev duality Eq. (24), and the function space of $f$ in
Sobolev duality Eq. (24) is a subspace of the function space defined in Eq.
(33).

  * •

$f^{*}$ in both Eq. (23) and Eq. (33) satisfies the property in Eq. (32).

Therefore, we conclude that the optimal $f^{*}$ in Sobolev duality Eq. (24)
also holds this property. ∎

######  Theorem 2.

If $f\in\mathcal{F}_{S}(\mathcal{X},\mu^{x_{i},x_{j}})$, given
$f^{*}(x_{i})-f^{*}(x_{j})=\|x_{i}-x_{j}\|$, then for
$x_{t}=tx_{i}+(1-t)x_{j}$ with $0\leq t\leq 1$, we have
$P_{(x_{i},x_{j})\sim\pi^{*}}[\nabla
f^{*}(x_{t})=\frac{x_{i}-x_{j}}{\|x_{i}-x_{j}\|}]=1$.

###### Proof.

Given $f^{*}(x_{i})-f^{*}(x_{j})=\|x_{i}-x_{j}\|$, then the inequality in Eq.
(26), Eq. (28) and Eq. (29) are turned to equations:

(i) The first equation corresponding to Eq. (26) is that
$\int_{\mathcal{C}}\mathbf{v}\cdot\nabla_{x}f(\mathbf{r})d\mathbf{r}=\int_{\mathcal{C}}\|\nabla_{x}f(\mathbf{r})\|d\mathbf{r}$.
It indicates that

$\nabla_{x}f(\mathbf{r})=k\mathbf{v}=k\frac{x_{i}-x_{j}}{\|x_{i}-x_{j}\|},$
(34)

where $k$ is a scalar, i.e., we have that

$\nabla_{x}f(\mathbf{r})\parallel\frac{x_{i}-x_{j}}{\|x_{i}-x_{j}\|}$ (35)

(ii) The second equation corresponding to Eq. (28) is that
$\int_{\mathcal{C}}\|\nabla_{x}f(\mathbf{r})\|\sqrt{\mu^{x_{i},x_{j}}(\mathbf{r})}\frac{1}{\sqrt{\mu^{x_{i},x_{j}}(\mathbf{r})}}d\mathbf{r}=\sqrt{\int_{\mathcal{C}}\|\nabla_{x}f(\mathbf{r})\|^{2}\mu^{x_{i},x_{j}}(\mathbf{r})d\mathbf{r}}\sqrt{\int_{\mathcal{C}}{\frac{1}{\mu^{x_{i},x_{j}}(\mathbf{r})}d\mathbf{r}}}$.
According to Cauchy–Schwarz inequality, it indicates that
$\|\nabla_{x}f(\mathbf{r})\|\sqrt{\mu^{x_{i},x_{j}}(\mathbf{r})}$ and
$\frac{1}{\sqrt{\mu^{x_{i},x_{j}}(\mathbf{r})}}$ are linearly dependent. More
precisely, we have:

$\displaystyle\|\nabla_{x}f(\mathbf{r})\|\sqrt{\mu^{x_{i},x_{j}}(\mathbf{r})}=k\frac{1}{\sqrt{\mu^{x_{i},x_{j}}(\mathbf{r})}},$
(36)
$\displaystyle\|\nabla_{x}f(\mathbf{r})\|=k\frac{1}{\mu^{x_{i},x_{j}}(\mathbf{r})}=k\|x_{i}-x_{j}\|,$

where $k$ is a scalar.

(iii) The third equation corresponding to Eq. (29) is that

$\sqrt{\int_{\mathcal{X}}\|\nabla_{x}f(\mathbf{r})\|^{2}\mu^{x_{i},x_{j}}(\mathbf{r})dt}=1$
(37)

Combine Eq. (LABEL:sobolev-scalar) and Eq. (37), we can get the solution of
$k$ in Eq. (LABEL:sobolev-scalar): $k=\frac{1}{\|x_{i}-x_{j}\|}$.

Assign the value of $k$ into Eq. (LABEL:sobolev-scalar) we get the solution of
the gradient norms:

$\|\nabla_{x}f(\mathbf{r})\|=1.$ (38)

Combining the gradient direction in Eq. (34) and norm in Eq. (38), we have
that:

$\nabla_{x}f(\mathbf{r})=\frac{x_{i}-x_{j}}{\|x_{i}-x_{j}\|}$ (39)

To conclude, we showed that if $(x_{i},x_{j})$ have the property that
$f^{*}(x_{i})-f^{*}(x_{j})=\|x_{i}-x_{j}\|$, then
$\nabla_{x}f(x_{t})=\frac{x_{i}-x_{j}}{\|x_{i}-x_{j}\|}$. Since this happens
with probability 1 under $\pi$, we know that

$P_{(x_{i},x_{j})\sim\pi^{*}}[\nabla
f^{*}(x_{t})=\frac{x_{i}-x_{j}}{\|x_{i}-x_{j}\|}]=1$ (40)

and we finished the proof. ∎

##  Appendix C Example for the convergence of Sobolev IPM

##### Example 1.

Let $P_{r}$ and $P_{g}$ be two disjoint 2-dimensional uniform distributions.
Consider the following case:

$\displaystyle P_{r}(x_{1},x_{2})$ $\displaystyle=\begin{cases}1,&-1\leq
x_{1}\leq 0\ and\ -1\leq x_{2}\leq 0,\cr 0,&otherwise.\end{cases}$ (41)
$\displaystyle P_{g}(x_{1},x_{2})$ $\displaystyle=\begin{cases}1,&0\leq
x_{1}\leq 1\ and\ 0\leq x_{2}\leq 1,\cr 0,&otherwise.\end{cases}$

Eq. (11) in the paper states the property of the optimal critic $f^{*}$ in
Sobolev IPM that

$\nabla_{x}f^{*}(x)=\frac{1}{n\mathcal{S}_{\mu}(P_{r},P_{g})}\frac{D^{-}F_{P_{g}}(x)-D^{-}F_{P_{r}}(x)}{\mu(x)}.$
(42)

According to this property, we have that

$\displaystyle\frac{\partial}{\partial x_{1}}f^{*}(a,b)=\ $ $\displaystyle
k\Big{(}D^{-1}F_{g}(a,b)-D^{-1}F_{r}(a,b)\Big{)}$ $\displaystyle=\ $
$\displaystyle k\frac{\partial}{\partial x_{2}}(F_{g}(a,b)-F_{r}(a,b))$
$\displaystyle=\ $ $\displaystyle
k\int_{-\infty}^{a}\Big{(}P_{g}(x_{1},b)-P_{r}(x_{1},b)\Big{)}dx_{1}$
$\displaystyle=\ $ $\displaystyle k\int_{-\infty}^{a}P_{g}(x_{1},b)dx_{1}>0$
(43)

where $k$ is a scalar equal to
$\frac{1}{n\mathcal{S}_{\mu}(P_{r},P_{g})\mu(a,b)}$. Similarly, we also have
that $\frac{\partial}{\partial x_{2}}f^{*}(a,b)>0$. Thus, for any point
$(a,b)$ from $P_{g}$, we have

$\displaystyle\frac{\partial}{\partial x_{1}}f^{*}(a,b)>0,\ \
\frac{\partial}{\partial x_{2}}f^{*}(a,b)>0.$ (44)

This implies in this case the gradients guide fake data distribution $P_{g}$
to move along the positive direction of the $x$ and $y$ axes, which is the
opposite direction to converge to $P_{r}$. Thus we have shown that Sobolev IPM
will fail to converge in some cases. By contrast, according to Proposition 3,
the gradients of the proposed Sobolev duality always follow the optimal
transport plan which guarantees the convergence.

##  Appendix D Proof for the local convergence of SWGAN-CP

In this section, we present the proof of the local convergence property of
proposed SWGAN-CP. This proof is build upon the results from the theory of
discrete dynamical systems on previous works [31, 25, 26]. The discrete
version of a basic convergence theorem for continuous dynamical systems from
[31] can be found in Appendix A.1 in [25]. This theorem allows us to make
statements about training algorithms for GANs for finite learning rates.
Besides, [25] analysed the convergence properties of simultaneous and
alternating gradient descent, and [26] states some eigenvalue bounds that were
derived.

Following the notation of [31], the training objective for the two players in
GANs can be described by an objective function of the form

$\mathcal{L}(\theta,w)=\mathbb{E}_{p(z)}\left[f(D_{w}(G_{\theta}(z)))\right]\\\
+\mathbb{E}_{p_{data}(x)}\left[f(-D_{w}(x))\right]$ (45)

for some real-valued function $f$. Then in the toy case defined in Section 6,
this is simplified to:

$\mathcal{L}(\theta,w)=f(w\theta)+f(0).$ (46)

Now we consider to add the GAN loss with the gradient penalty regularization
term used in SWGAN-CP

$\mathcal{L}_{cp}=-\lambda(w^{4}+w^{2}),$ (47)

then we have the following lemma:

######  Lemma 2.

The eigenvalues of the Jacobian of the gradient vector field for the SWGAN-CP
at the equilibrium point are given by

$\gamma_{1/2}=-\lambda\pm\sqrt{\lambda^{2}-f^{\prime}(0)^{2}}.$ (48)

In particular, for $\lambda>0$ all eigenvalues have negative real part. Hence,
simultaneous and alternating gradient descent are both locally convergent for
small enough learning rates.

###### Proof.

The gradient vector field for SWGAN-CP becomes

$\tilde{v}(\theta,w)=\left(\begin{array}[]{c}-f^{\prime}(\theta w)w\\\
f^{\prime}(\theta w)\theta-\lambda(4w^{3}+2w)\end{array}\right).$ (49)

The Jacobian $\tilde{v}^{\prime}(\theta,w)$ is therefore given by

$\begin{pmatrix}-f^{\prime\prime}(\theta w)w^{2}&-f^{\prime}(\theta
w)-f^{\prime\prime}(\theta w)\theta w\\\ f^{\prime}(\theta
w)+f^{\prime\prime}(\theta w)\theta w&f^{\prime\prime}(\theta
w)\theta^{2}-\lambda(12w^{2}+2)\end{pmatrix}.$ (50)

Evaluating it at $\theta=w=0$ yields

$\tilde{v}^{\prime}(0,0)=\left(\begin{array}[]{cc}0&-f^{\prime}(0)\\\
f^{\prime}(0)&-2\lambda\end{array}\right)$ (51)

whose eigenvalues are given by

$\gamma_{1/2}=-\lambda\pm\sqrt{\lambda^{2}-f^{\prime}(0)^{2}}.$ (52)

∎

##  Appendix E Algorithm for SWGAN-GP and WGAN-AL

The training procedures of SWGAN-GP and WGAN-AL that we used in Section 5.2
are formally presented in Algorithm 2 and Algorithm 3 respectively.

Algorithm 2 Sobolev Wasserstein GAN with Gradient Penalty

Input: $\rho$ penalty weight, $\eta$ learning rate, $k$ number of critic
iterations per generator iteration, $n$ batch size, $m$ number of points
sampled between each pair of fake and real data

Initial: critic parameters $w$, generator parameters $\theta$

1: while $\theta$ has not converged do

2: for $k=1,...,k$ do

3: Sample a minibatch $x_{i}$, $i=1,...,n$, $x_{i}\sim P_{r}$

4: Sample a minibatch $z_{i}$, $i=1,...,n$, $z_{i}\sim P_{z}$

5: $\widetilde{x}_{i}\leftarrow G_{\theta}(z_{i})$, $i=1,...,n$

6: for $i=1,...,n$ do

7: Sample a minibatch $\hat{x}_{ij}=t_{j}x_{i}+(1-t_{j})\widetilde{x}_{i}$,
$j=1,...,m$, $t_{j}\sim\text{U}(0,1)$

8: $\Omega^{(i)}\leftarrow
1-\frac{1}{m}\sum_{j=1}^{m}\|\nabla_{x}D(\hat{x}_{ij})\|^{2}$

9:
$\mathcal{L}^{(i)}_{gp}\leftarrow-\lambda\,\big{(}\max\\{-\Omega^{(i)}(D_{w},G_{\theta}),0\\}\big{)}^{2}$

10: end for

11:
$\mathcal{L}\leftarrow\frac{1}{n}\sum_{i=1}^{n}\big{\\{}D_{w}(x_{i})-D_{w}(\widetilde{x}_{i})+\mathcal{L}_{gp}^{(i)}\big{\\}}$

12: $g_{w}\leftarrow\nabla_{w}\mathcal{L}(w,\theta,\alpha)$

13: $w\leftarrow w+\eta\text{ADAM}(w,g_{w})$

14: end for

15: Sample a minibatch $z_{i}$, $i=1,...,n$, $z_{i}\sim P_{z}$

16:
$d_{\theta}\leftarrow-\nabla_{\theta}\frac{1}{n}\sum_{i=1}^{n}D_{w}(G_{\theta}(z_{i}))$

17: $\theta\leftarrow\theta-\eta\text{ADAM}(\theta,d_{\theta})$

18: end while

Algorithm 3 Wasserstein GAN with Augmented Lagrangian

Input: $\rho$ penalty weight, $\eta$ learning rate, $k$ number of critic
iterations per generator iteration, $n$ batch size, $m$ number of points
sampled between each pair of fake and real data

Initial: critic parameters $w$, generator parameters $\theta$, Lagrange
multiplier $\alpha$

1: while $\theta$ has not converged do

2: for $k=1,...,k$ do

3: Sample a minibatch $x_{i}$, $i=1,...,n$, $x_{i}\sim P_{r}$

4: Sample a minibatch $z_{i}$, $i=1,...,n$, $z_{i}\sim P_{z}$

5: $\widetilde{x}_{i}\leftarrow G_{\theta}(z_{i})$, $i=1,...,n$

6: for $i=1,...,n$ do

7: Sample a minibatch $\hat{x}_{ij}=t_{j}x_{i}+(1-t_{j})\widetilde{x}_{i}$,
$j=1,...,m$, $t_{j}\sim\text{U}(0,1)$

8: $h^{(ij)}=1-\|\nabla_{x}D_{w}(\hat{x}_{ij})\|$

9: $\mathcal{L}_{al}^{(j)}\leftarrow\sum_{j=0}^{m}\big{\\{}\alpha
h^{(ij)}-\frac{\rho}{2}(h^{(ij)})^{2}\big{\\}}$

10: end for

11:
$\mathcal{L}\leftarrow\frac{1}{n}\sum_{i=1}^{n}\big{\\{}D_{w}(x_{i})-D_{w}(\widetilde{x}_{i})+\mathcal{L}_{al}^{(i)}\big{\\}}$

12:
$(g_{\alpha},g_{w})\leftarrow(\nabla_{\alpha}\mathcal{L},\nabla_{w}\mathcal{L})(w,\theta,\alpha)$

13: $\alpha\leftarrow\alpha-\rho g_{\alpha}\ $ (SGD rule on $\alpha$ with
learning rate $\rho$)

14: $w\leftarrow w+\eta\text{ADAM}(w,g_{w})$

15: end for

16: Sample a minibatch $z_{i}$, $i=1,...,n$, $z_{i}\sim P_{z}$

17:
$d_{\theta}\leftarrow-\nabla_{\theta}\frac{1}{n}\sum_{i=1}^{n}D_{w}(G_{\theta}(z_{i}))$

18: $\theta\leftarrow\theta-\eta\text{ADAM}(\theta,d_{\theta})$

19: end while

##  Appendix F ResNet architecture

We adopt the same ResNet network structures and hyperparameters as [15]. The
generator and critic are residual networks. [15] use pre-activation residual
blocks with two $3\times 3$ convolutional layers each and ReLU nonlinearity.
Batch normalization is used in the generator but not the critic. Some residual
blocks perform downsampling (in the critic) using mean pooling after the
second convolution, or nearest-neighbor upsampling (in the generator) before
the second convolution.

Formally, we present our ResNet architecture in Table 2. Further architectural
details can be found in our open-source model in supplementary material.

Table 2: ResNet architecture. Generator $G(z)$
---
Operation | Kernel size | Resample | Output Dims
Noise | N/A | N/A | $128$
Linear | N/A | N/A | $128\times 4\times 4$
Residual block | $[3\times 3]\times 2$ | Up | $128\times 8\times 8$
Residual block | $[3\times 3]\times 2$ | Up | $128\times 16\times 16$
Residual block | $[3\times 3]\times 2$ | Up | $128\times 32\times 32$
Conv, tanh | $3\times 3$ | N/A | $3\times 32\times 32$
Critic $D(x)$
Operation | Kernel size | Resample | Output Dims
Residual block | $[3\times 3]\times 2$ | Down | $128\times 16\times 16$
Residual block | $[3\times 3]\times 2$ | Down | $128\times 8\times 8$
Residual block | $[3\times 3]\times 2$ | N/A | $128\times 8\times 8$
Residual block | $[3\times 3]\times 2$ | N/A | $128\times 8\times 8$
ReLU, mean pool | N/A | N/A | $128$
Linear | N/A | N/A | $1$

##  Appendix G More generated samples

###  G.1 MNIST samples

Figure 8: Generated MNIST samples of SWGAN-AL.

###  G.2 CIFAR-10 samples

Figure 9: Generated CIFAR-10 samples of SWGAN-AL. Left: Samples generated by
our unsupervised model. Right: Samples generated by our conditional model. We
add AC-GAN conditioning to our unconditional model. Samples from the same
class are displayed in the same column.

Figure 10: Generated CIFAR-10 samples of SWGAN-GP. Left: Samples generated by
our unsupervised model. Right: Samples generated by our conditional model. We
add AC-GAN conditioning to our unconditional model. Samples from the same
class are displayed in the same column.

###  G.3 Tiny-ImageNet samples

Figure 11: Generated Tiny-ImageNet samples of SWGAN-AL. Figure 12: Generated
Tiny-ImageNet samples of SWGAN-GP.