subsection 8in5.5in* **1.1 **1 2 *2*

# Properties of f-divergences and f-GAN training

Matt Shannon
Google Research
<EMAIL_ADDRESS>

(August 2020)

###### Abstract

In this technical report we describe some properties of f-divergences and
f-GAN training. We present an elementary derivation of the f-divergence lower
bounds which form the basis of f-GAN training. We derive informative but
perhaps underappreciated properties of f-divergences and f-GAN training,
including a gradient matching property and the fact that all f-divergences
agree up to an overall scale factor on the divergence between nearby
distributions. We provide detailed expressions for computing various common
f-divergences and their variational lower bounds. Finally, based on our
reformulation, we slightly generalize f-GAN training in a way that may improve
its stability.

##  1 The family of f-divergences

We start by reviewing the definition of an _f-divergence_ (or _$\phi$
-divergence_) (Csiszár, 1967; Ali and Silvey, 1966) and establishing some
basic properties.

###  1.1 Definition

Given a strictly convex twice continuously differentiable function
$f\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}_{>0}\to\mathbb{R}$, the _$f$
-divergence_ between probability distributions with densities111Most results
also hold for “discrete” probability distributions.  $p$ and $q$ over
$\mathbb{R}^{K}$ is defined as

$D_{f}(p,q)=\int q(x)f\left(\frac{p(x)}{q(x)}\right)\operatorname{d\\!}x$ (1)

For simplicity, we assume the probability distributions $p$ and $q$ are
suitably nice, e.g. absolutely continuous with respect to the Lebesgue measure
on $\mathbb{R}^{K}$, $p(x),q(x)>0$ for $x\in\mathbb{R}^{K}$, and $p$ and $q$
continuously differentiable. These assumptions are discussed in §3.4. We refer
to $f$ as the _defining function_ of the divergence $D_{f}$.

There is some redundancy in the above definition. Adding an affine-linear term
to the defining function just adds a constant to the divergence: if
$g(u)=f(u)+a+bu$ for $a,b\in\mathbb{R}$ then $D_{g}(p,q)=D_{f}(p,q)+a+b$ for
all $p$ and $q$. Typically we do not care about an overall additive shift and
would regard $D_{f}$ and $D_{g}$ as essentially the same divergence. Thus we
have identified two unnecessary degrees of freedom in the specification of the
defining function, and these may be removed by fixing $f(1)=f^{\prime}(1)=0$.
This results in no loss of generality since any defining function can be put
in this form by adding a suitable affine-linear term.222This affine-linear
term also does not affect the various bounds and finite sample approximations
derived below, as long as the reparameterization trick is used for the
generator gradient as is standard practice. In the discrete case or if other
finite sample approximation such as naive REINFORCE is used then adding an
affine-linear term to $f$ may affect the variance of the finite sample
approximation of the generator gradient.  From here on we assume
$f(1)=f^{\prime}(1)=0$ as part of our definition of an f-divergence. The
choice $f(1)=0$ ensures $D_{f}(p,q)=0$ when $p=q$. The choice
$f^{\prime}(1)=0$ has the ancillary benefit of (1) being non-negative even if
$p$ and $q$ are positive functions that do not integrate to one.

There is also a multiplicative degree of freedom in the defining function that
is often irrelevant: multiplying the defining function by a constant $k>0$
just multiplies the divergence by the same constant. Removing this superficial
source of variation makes different f-divergences easier to compare. We refer
to an f-divergence as being _canonical_ if $f^{\prime\prime}(1)=1$. Any
f-divergence can be made canonical by scaling appropriately. Intuitively,
fixing $f^{\prime\prime}(1)$ corresponds to fixing the behavior of the
f-divergence in the region $u\approx 1$, corresponding to $p\approx q$. This
will be made precise below.

###  1.2 Properties

f-divergences satisfy several mathematical properties:

  * •

$D_{f}$ is linear in $f$.

  * •

$D_{f}(p,q)\geq 0$ for all distributions $p$ and $q$ with equality iff $p=q$.
This justifies referring to $D_{f}$ as a divergence.

  * •

$D_{f}$ uniquely determines $f$.

  * •

All f-divergences agree up to an overall scale factor on the divergence
between nearby distributions: If $p\approx q$ then $D_{f}(p,q)\approx
f^{\prime\prime}(1)\operatorname{KL}(p\,\|\,q)$ to second order. In particular
all canonical f-divergences agree when $p\approx q$.

  * •

For many common f-divergences, $f^{\prime\prime}$ has a simpler algebraic form
than $f$ and is easier to work with.

Linearity is straightforward to verify. If $D_{f}$ and $D_{g}$ are two
f-divergences and $k>0$ then $D_{f+g}=D_{f}+D_{g}$ and $D_{(kf)}=kD_{f}$. If
$f$ and $g$ are strictly convex then so is $f+g$ and $kf$, and
$(f+g)(1)=f(1)+g(1)=0$ and $(kf)(1)=kf(1)=0$, and similarly for $f^{\prime}$,
so $D_{f+g}$ and $D_{kf}$ are valid f-divergences.

The non-negativity of $D_{f}$ follows from the convexity of $f$. Since
$f(1)=f^{\prime}(1)=0$ and $f$ is strictly convex, $f(u)\geq 0$ with equality
iff $u=1$. Thus

$\int q(x)f\left(\frac{p(x)}{q(x)}\right)\operatorname{d\\!}x\geq 0$ (2)

Thus $D_{f}(p,q)\geq 0$ for all $p$ and $q$. In general, if $\int
g(x)\operatorname{d\\!}x=0$ for a continuous integrable non-negative function
$g\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{K}\to\mathbb{R}$ then
$g(x)=0$ for all $x$. Applying this to $g(x)=q(x)f(p(x)/q(x))$ we see that we
have equality in (2) iff $p(x)/q(x)=1$ for all $x$. Thus $D_{f}(p,q)=0$
implies $p=q$. The non-negativity of $D_{f}$ can also be seen by plugging the
constant function $u(x)=1$ into (14).

We now show that $D_{f}$ completely determines $f$. Suppose $D_{f}=D_{g}$. We
wish to show that $f=g$. Consider first the discrete case where $p$ and $q$
are distributions over a two-point set $\\{0,1\\}$. Given $u>1$, choose
$p_{u}$ and $q_{u}$ such that $p_{u}(0)/q_{u}(0)=u$ and
$p_{u}(1)/q_{u}(1)=\tfrac{1}{2}$. It is straightforward to show that such
$p_{u}$ and $q_{u}$ exist and are given by

$\displaystyle p_{u}(0)$ $\displaystyle=\frac{u}{2u-1}$ $\displaystyle
q_{u}(0)$ $\displaystyle=\frac{1}{2u-1}$ (3) $\displaystyle p_{u}(1)$
$\displaystyle=\frac{u-1}{2u-1}$ $\displaystyle q_{u}(1)$
$\displaystyle=\frac{2(u-1)}{2u-1}$ (4)

Thus

$(2u-1)D_{f}(p_{u},q_{u})=f(u)+2(u-1)f(\tfrac{1}{2})$ (5)

Subtracting the equivalent equation for $g$ and using $D_{f}=D_{g}$, we see
that $f$ and $g$ differ only in an affine-linear term: $g(u)=f(u)+a(u-1)$ for
all $u>1$ for some $a\in\mathbb{R}$. Therefore $g^{\prime}(u)=f^{\prime}(u)+a$
for $u>1$, and taking the limit as $u\to 1$ we have
$g^{\prime}(1)=f^{\prime}(1)+a$. But $f^{\prime}(1)=g^{\prime}(1)=0$, so
$a=0$, so $f(u)=g(u)$ for $u>1$. A similar argument applies for $0<u<1$. In
this case we choose $p_{u}(0)/q_{u}(0)=u$ and $p_{u}(1)/q_{u}(1)=2$, and we
have

$\displaystyle p_{u}(0)$ $\displaystyle=\frac{u}{2-u}$ $\displaystyle
q_{u}(0)$ $\displaystyle=\frac{1}{2-u}$ (6) $\displaystyle p_{u}(1)$
$\displaystyle=\frac{2(1-u)}{2-u}$ $\displaystyle q_{u}(1)$
$\displaystyle=\frac{1-u}{2-u}$ (7)

and

$(2-u)D_{f}(p_{u},q_{u})=f(u)+(1-u)f(2)$ (8)

From here we apply the same argument as above. Thus $f(u)=g(u)$ for $0<u<1$.
Combining this with the above result for $u>1$ and the fact that $f(1)=g(1)$,
we have $f=g$ as desired. For the continuous case where $p$ and $q$ are
densities over $\mathbb{R}^{K}$, we reduce this to the discrete case by
considering mixtures of Gaussians with shrinking covariances. Consider the
$u>1$ case. Fix any two distinct points $\mu_{0},\mu_{1}\in\mathbb{R}^{K}$.
Let
$\tilde{p}_{u\sigma}(x)=p_{u}(0)\,\mathcal{N}(x;\mu_{0},\sigma^{2}I)+p_{u}(1)\,\mathcal{N}(x;\mu_{1},\sigma^{2}I)$
and
$\tilde{q}_{u\sigma}(x)=q_{u}(0)\,\mathcal{N}(x;\mu_{0},\sigma^{2}I)+q_{u}(1)\,\mathcal{N}(x;\mu_{1},\sigma^{2}I)$
where $p_{u}$ and $q_{u}$ are as specified for the discrete $u>1$ case above.
As $\sigma\to 0$, $D_{f}(\tilde{p}_{u\sigma},\tilde{q}_{u\sigma})\to
D_{f}(p_{u},q_{u})$, so
$(2u-1)D_{f}(\tilde{p}_{u\sigma},\tilde{q}_{u\sigma})\to
f(u)+2(u-1)f(\tfrac{1}{2})$, and similarly for $g$. But $D_{f}=D_{g}$, so we
have the same quantity tending to two limits, so the limits must be equal.
Thus $g(u)=f(u)+a(u-1)$ for some $a\in\mathbb{R}$ and the remainder of the
argument is as above. A similar argument applies for $0<u<1$. Thus $f=g$ also
holds in the continuous case.

Different f-divergences may behave very differently when $p$ and $q$ are far
apart but are essentially identical when $q\approx p$. One way to make this
precise is to consider a parametric family
$\\{q_{\lambda}\mathrel{\mathop{\mathchar 58\relax}}\lambda\in\Lambda\\}$ of
densities. A Taylor expansion of
$D_{f}\left(q_{\lambda},q_{\lambda+\varepsilon v}\right)$ in terms of
$\epsilon$ shows that

$D_{f}\left(q_{\lambda},q_{\lambda+\varepsilon
v}\right)=\tfrac{1}{2}\varepsilon^{2}f^{\prime\prime}(1){v}^{\mathsf{T}}F(\lambda)v+O(\varepsilon^{3})$
(9)

where $\varepsilon\in\mathbb{R}$, $v\in\mathbb{R}^{K}$, and

$F_{ij}(\lambda)=\int
q_{\lambda}(x)\left(\mathinner{\tfrac{\partial{}}{\partial{\lambda_{i}}}}\log
q_{\lambda}(x)\right)\left(\mathinner{\tfrac{\partial{}}{\partial{\lambda_{j}}}}\log
q_{\lambda}(x)\right)\operatorname{d\\!}x$ (10)

is the Fisher information matrix of the parametric family, also known as the
Fisher metric. Thus all f-divergences agree up to a constant factor on the
divergence between two nearby distributions, and they are all just scaled
versions of the Fisher distance in this regime. Alternatively this may be
stated in the non-parametric form

$D_{f}\left(q,q+\varepsilon
v\right)=\tfrac{1}{2}\varepsilon^{2}f^{\prime\prime}(1)\int\frac{\left(v(x)\right)^{2}}{q(x)}\operatorname{d\\!}x+O(\varepsilon^{3})$
(11)

where $v\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{K}\to\mathbb{R}$
satisfies $\int v(x)\operatorname{d\\!}x=0$. Informally we may state this as

$D_{f}(p,q)\approx\tfrac{1}{2}f^{\prime\prime}(1)\int\frac{\left(p(x)-q(x)\right)^{2}}{p(x)}\operatorname{d\\!}x$
(12)

Thus all f-divergences agree up to a constant factor on the divergence between
nearby distributions.

The fact that $f^{\prime\prime}$ has a simpler algebraic form than $f$ for
many common f-divergences will be seen when we consider specific f-divergences
below. Considering $f^{\prime\prime}$ is natural for a number of reasons.
Adding an affine-linear term to $f$ does not change $f^{\prime\prime}$. Thus
even without the constraints $f(1)=f^{\prime}(1)=0$, $f^{\prime\prime}$ would
not have the two unnecessary degrees of freedom mentioned above and $D_{f}$
would uniquely determine $f^{\prime\prime}$. Strict convexity of $f$
corresponds to the simple condition $f^{\prime\prime}(u)>0$ for all $u$. We
will also see below that various gradients of $D_{f}$ and of its lower bound
$E_{f}$ depend on $f$ only through $f^{\prime\prime}$. Considering
$f^{\prime\prime}$ also provides a simple view of how various f-divergences
are related. For example, three common symmetric divergences are the Jensen-
Shannon, squared Hellinger and Jeffreys divergences. In the $f^{\prime\prime}$
domain, these may all be viewed as forms of average of KL and reverse KL,
specifically as the harmonic mean, geometric mean and arithmetic mean
respectively.

##  2 Variational divergence estimation

f-GANs are based on an elegant way to estimate the f-divergence between two
distributions given only samples from the two distributions (Nguyen et al.,
2010). In this section we review this approach to _variational divergence
estimation_. We provide a simple and easy to understand derivation involving
elementary facts about convex functions. At the end of this section we discuss
how our derivation and notation relates to that of the original f-GAN paper
(Nowozin et al., 2016).

###  2.1 Variational lower bound

$0.5$$1$$1.5$$2$$-0.5$$0.5$$1$$1.5$$u$$f(u)$Tangent line Figure 1: A convex
function $f\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}_{>0}\to\mathbb{R}$
and a tangent line. The variational bound used by f-GANs is based on the fact
that a convex function $f$ lies at or above its tangent lines.

We first derive a variational lower bound on the f-divergence $D_{f}$. Since
$f$ is strictly convex, its graph lies at or above any of its tangent lines
and only touches in one place. That is, for $k,u>0$,

$f(k)\geq
f(u)+(k-u)f^{\prime}(u)=kf^{\prime}(u)-\left[uf^{\prime}(u)-f(u)\right]$ (13)

with equality iff $k=u$. This inequality is illustrated in Figure 1.
Substituting $p(x)/q(x)$ for $k$ and $u(x)$ for $u$, for any continuously
differentiable function $u\mathrel{\mathop{\mathchar
58\relax}}\mathbb{R}^{K}\to\mathbb{R}_{>0}$ we obtain

$D_{f}(p,q)\geq\int p(x)f^{\prime}(u(x))\operatorname{d\\!}x-\int
q(x)\left[u(x)f^{\prime}(u(x))-f(u(x))\right]\operatorname{d\\!}x$ (14)

with equality iff $u=u^{*}$, where $u^{*}(x)=p(x)/q(x)$. The function $u$ is
referred to as the _critic_. It will be helpful to have a concise notation for
this bound. Writing $u(x)=\exp(d(x))$ without loss of generality, for any
continuously differentiable function $d\mathrel{\mathop{\mathchar
58\relax}}\mathbb{R}^{K}\to\mathbb{R}$, we have

$D_{f}(p,q)\geq E_{f}(p,q,d)$ (15)

with equality iff $d=d^{*}$, where

$\displaystyle E_{f}(p,q,d)$ $\displaystyle=\int
p(x)a_{f}(d(x))\operatorname{d\\!}x-\int q(x)b_{f}(d(x))\operatorname{d\\!}x$
(16) $\displaystyle a_{f}(d)$ $\displaystyle=f^{\prime}(\exp(d))$ (17)
$\displaystyle b_{f}(d)$ $\displaystyle=\exp(d)f^{\prime}(\exp(d))-f(\exp(d))$
(18) $\displaystyle d^{*}(x)$ $\displaystyle=\log p(x)-\log q(x)$ (19)

Note that both $a_{f}$ and $b_{f}$ are linear in $f$. Their derivatives
$a_{f}^{\prime}(\log u)=uf^{\prime\prime}(u)$ and $b_{f}^{\prime}(\log
u)=u^{2}f^{\prime\prime}(u)$ depend on $f$ only through $f^{\prime\prime}$.
Note that $b_{f}^{\prime}(d)=a_{f}^{\prime}(d)\exp(d)$.

###  2.2 Formulation of variational divergence estimation

The bound (15) leads naturally to _variational divergence estimation_. The
$f$-divergence between $p$ and $q$ can be estimated by maximizing $E_{f}$ with
respect to $d$ (Nguyen et al., 2010). Conveniently $E_{f}$ is expressed in
terms of expectations and may be approximately computed and maximized with
respect to $d$ using only samples from $p$ and $q$. Ultimately this property
derives from the fact that the tangent lines of $f(u)$ are affine-linear in
$u$, and the constant term leads to expectations with respect to $q$ and the
linear term leads to expectations with respect to $p$. If we parameterize $d$
as a neural net $d_{\nu}$ with parameters $\nu$ then we can approximate the
divergence by maximizing $E_{f}(p,q,d_{\nu})$ with respect to $\nu$. This does
not compute the exact divergence for several reasons: there is no guarantee
that $d^{*}$ lies in the family $\\{d_{\nu}\mathrel{\mathop{\mathchar
58\relax}}\nu\\}$ of functions representable by the neural net; gradient-based
optimization may find a local but not global minimum; and we have to be
careful not to overfit given a finite set of samples from $p$ (typically $q$
is a model and so we have access to arbitrarily many samples). However we hope
for sufficiently flexible neural nets and careful optimization that the
approximation will be close.

###  2.3 Expressions for common f-divergences

In this section we express several common divergences in terms of (1) and the
variational lower bound (16). For each f-divergence, we give explicit
expressions for $f$, $f^{\prime\prime}$, $D_{f}$, $E_{f}$, $a_{f}$, $b_{f}$,
$a_{f}^{\prime}$ and $b_{f}^{\prime}$. We also list the _tail weights_ , which
are related to the asymptotic behavior of $f^{\prime\prime}(u)$ as $u\to 0$
and $u\to\infty$ and which determine the key qualitative properties of an
f-divergence (Shannon et al., 2020). We mention below how some of the
divergences are related to each other by _softening_ (Shannon et al., 2020).
The _p-softening_ of a divergence $D$ is the divergence $\tilde{D}$ given by
$\tilde{D}(p,q)=4D(\tfrac{1}{2}p+\tfrac{1}{2}q,q)$. Similarly the _q-
softening_ is given by $\tilde{D}(p,q)=4D(p,\tfrac{1}{2}p+\tfrac{1}{2}q)$. The
factor of $4$ here ensures that $\tilde{D}$ remains canonical (Shannon et al.,
2020).

The _Kullback-Leibler (KL) divergence_ (or _I divergence_ or _relative
entropy_) satisfies:

$\displaystyle D_{f}(p,q)$ $\displaystyle=\operatorname{KL}(p\,\|\,q)=\int
p(x)\log\frac{p(x)}{q(x)}\operatorname{d\\!}x$ (20) $\displaystyle
E_{f}(p,q,d)$ $\displaystyle=1+\int p(x)d(x)\operatorname{d\\!}x-\int
q(x)\exp(d(x))\operatorname{d\\!}x$ (21) $\displaystyle f(u)$
$\displaystyle=u\log u-u+1$ (22) $\displaystyle f^{\prime\prime}(u)$
$\displaystyle=u^{-1}$ (23) $\displaystyle a_{f}(d)$ $\displaystyle=d$ (24)
$\displaystyle b_{f}(d)$ $\displaystyle=\exp(d)-1$ (25) $\displaystyle
a_{f}^{\prime}(d)$ $\displaystyle=1$ (26) $\displaystyle b_{f}^{\prime}(d)$
$\displaystyle=\exp(d)$ (27)

The KL divergence has $(1,2)$ tail weights. The KL divergence defining
function is sometimes given as $f(u)=u\log u$. The additional affine-linear
term in our expression for $f$ is due to the constraints
$f(1)=f^{\prime}(1)=0$. We note in passing that this precisely corresponds to
what is sometimes referred to as the _generalized KL divergence_ $D(p,q)=\int
p(x)\log\frac{p(x)}{q(x)}\operatorname{d\\!}x-\int
p(x)\operatorname{d\\!}x+\int q(x)\operatorname{d\\!}x$. This has the property
that $D(p,q)\geq 0$ with equality iff $p=q$ even if we remove the constraint
that $p$ and $q$ be valid densities that integrate to one.

The _reverse KL divergence_ satisfies:

$\displaystyle D_{f}(p,q)$ $\displaystyle=\operatorname{KL}(q\,\|\,p)=\int
q(x)\log\frac{q(x)}{p(x)}\operatorname{d\\!}x$ (28) $\displaystyle
E_{f}(p,q,d)$ $\displaystyle=1-\int p(x)\exp(-d(x))\operatorname{d\\!}x-\int
q(x)d(x)\operatorname{d\\!}x$ (29) $\displaystyle f(u)$ $\displaystyle=-\log
u+u-1$ (30) $\displaystyle f^{\prime\prime}(u)$ $\displaystyle=u^{-2}$ (31)
$\displaystyle a_{f}(d)$ $\displaystyle=1-\exp(-d)$ (32) $\displaystyle
b_{f}(d)$ $\displaystyle=d$ (33) $\displaystyle a_{f}^{\prime}(d)$
$\displaystyle=\exp(-d)$ (34) $\displaystyle b_{f}^{\prime}(d)$
$\displaystyle=1$ (35)

The reverse KL divergence has $(2,1)$ tail weights. Note the explicit symmetry
between the representations of KL and reverse KL in terms of $a_{f}$ and
$b_{f}$. Their symmetric relationship is less apparent from $f$ and
$f^{\prime\prime}$. In general if $D_{g}(p,q)=D_{f}(q,p)$ then
$a_{g}(d)=b_{f}(-d)$ and $b_{g}(d)=a_{f}(-d)$.

The canonicalized _Jensen-Shannon divergence_ (Lin, 1991) (or _Jensen
difference_ (Burbea and Rao, 1982) or _capacitory discrimination_ (Topsoe,
2000)) satisfies:

$\displaystyle D_{f}(p,q)$ $\displaystyle=4\operatorname{JS}(p,q)$ (36)
$\displaystyle=2\operatorname{KL}(p\,\|\,\tfrac{1}{2}p+\tfrac{1}{2}q)+2\operatorname{KL}(q\,\|\,\tfrac{1}{2}p+\tfrac{1}{2}q)$
(37) $\displaystyle=4\log 2+2\int
p(x)\log\frac{p(x)}{p(x)+q(x)}\operatorname{d\\!}x+2\int
q(x)\log\frac{q(x)}{p(x)+q(x)}\operatorname{d\\!}x$ (38) $\displaystyle
E_{f}(p,q,d)$ $\displaystyle=4\log 2+2\int
p(x)\log\sigma(d(x))\operatorname{d\\!}x+2\int
q(x)\log\sigma(-d(x))\operatorname{d\\!}x$ (39) $\displaystyle f(u)$
$\displaystyle=2u\log u-2(u+1)\log(u+1)+2u\log 2+2\log 2$ (40) $\displaystyle
f^{\prime\prime}(u)$ $\displaystyle=\frac{2}{u(u+1)}$ (41) $\displaystyle
a_{f}(d)$ $\displaystyle=2\log\sigma(d)+2\log 2$ (42) $\displaystyle b_{f}(d)$
$\displaystyle=-2\log\sigma(-d)-2\log 2$ (43) $\displaystyle
a_{f}^{\prime}(d)$ $\displaystyle=2\sigma(-d)$ (44) $\displaystyle
b_{f}^{\prime}(d)$ $\displaystyle=2\sigma(d)$ (45)

Here $\operatorname{JS}$ is the conventional, non-canonical Jensen-Shannon
divergence with $f^{\prime\prime}(1)=1/4$. The Jensen-Shannon divergence has
$(1,1)$ tail weights. Its $f^{\prime\prime}(u)$ is the harmonic mean of
$f^{\prime\prime}(u)$ for KL and $f^{\prime\prime}(u)$ for reverse KL. The
square root of the Jensen-Shannon divergence defines a metric on the space of
probability distributions (Endres and Schindelin, 2003; Österreicher and
Vajda, 2003).

The canonicalized _squared Hellinger distance_ (closely related to the
_Freeman-Tukey statistic_ for hypothesis testing) satisfies:

$\displaystyle D_{f}(p,q)$
$\displaystyle=2\int\left(\sqrt{p(x)}-\sqrt{q(x)}\right)^{2}\operatorname{d\\!}x$
(46) $\displaystyle=4-4\int\sqrt{p(x)q(x)}\operatorname{d\\!}x$ (47)
$\displaystyle E_{f}(p,q,d)$ $\displaystyle=4-2\int
p(x)\exp\left(-\tfrac{1}{2}d(x)\right)\operatorname{d\\!}x-2\int
q(x)\exp\left(\tfrac{1}{2}d(x)\right)\operatorname{d\\!}x$ (48) $\displaystyle
f(u)$ $\displaystyle=2(1-\sqrt{u})^{2}$ (49) $\displaystyle
f^{\prime\prime}(u)$ $\displaystyle=u^{-\frac{3}{2}}$ (50) $\displaystyle
a_{f}(d)$ $\displaystyle=2-2\exp\left(-\tfrac{1}{2}d\right)$ (51)
$\displaystyle b_{f}(d)$ $\displaystyle=2\exp\left(\tfrac{1}{2}d\right)-2$
(52) $\displaystyle a_{f}^{\prime}(d)$
$\displaystyle=\exp\left(-\tfrac{1}{2}d\right)$ (53) $\displaystyle
b_{f}^{\prime}(d)$ $\displaystyle=\exp\left(\tfrac{1}{2}d\right)$ (54)

Here $\int\sqrt{p(x)q(x)}\operatorname{d\\!}x$ is known as the _Bhattacharyya
coefficient_. The squared Hellinger distance has $(\frac{3}{2},\frac{3}{2})$
tail weights. Its $f^{\prime\prime}(u)$ is the geometric mean of
$f^{\prime\prime}(u)$ for KL and $f^{\prime\prime}(u)$ for reverse KL. The
Hellinger distance defines a metric on the space of probability distributions
(Vajda, 2009).

The _Jeffreys divergence_ (or _J divergence_)
$(p,q)\mapsto\tfrac{1}{2}\operatorname{KL}(p,q)+\tfrac{1}{2}\operatorname{KL}(q,p)$
is the arithmetic mean of the KL divergence and reverse KL divergence. Since
$f$, $f^{\prime\prime}$, $D_{f}$, $E_{f}$, $a_{f}$, $b_{f}$, $a_{f}^{\prime}$
and $b_{f}^{\prime}$ are all linear in $f$, they are also all just the
arithmetic mean of the corresponding quantities for KL and reverse KL and so
we do not list them separately. The Jeffreys divergence has $(2,2)$ tail
weights.

The canonicalized _squared Le Cam distance_ (or _squared Puri-Vincze distance_
or _triangular discrimination_) satisfies:

$\displaystyle D_{f}(p,q)$ $\displaystyle=2\Delta(p,q)$ (55)
$\displaystyle=\chi^{2}(p,\tfrac{1}{2}p+\tfrac{1}{2}q)$ (56)
$\displaystyle=\chi^{2}(q,\tfrac{1}{2}p+\tfrac{1}{2}q)$ (57)
$\displaystyle=\int\frac{\left(p(x)-q(x)\right)^{2}}{p(x)+q(x)}\operatorname{d\\!}x$
(58) $\displaystyle E_{f}(p,q,d)$ $\displaystyle=2-4\int
p(x)\left(\sigma(-d(x))\right)^{2}\operatorname{d\\!}x-4\int
q(x)\left(\sigma(d(x))\right)^{2}\operatorname{d\\!}x$ (59) $\displaystyle
f(u)$ $\displaystyle=\frac{(u-1)^{2}}{1+u}$ (60) $\displaystyle
f^{\prime\prime}(u)$ $\displaystyle=\frac{8}{(1+u)^{3}}$ (61) $\displaystyle
a_{f}(d)$ $\displaystyle=1-4\left(\sigma(-d)\right)^{2}$ (62) $\displaystyle
b_{f}(d)$ $\displaystyle=4\left(\sigma(d)\right)^{2}-1$ (63) $\displaystyle
a_{f}^{\prime}(d)$ $\displaystyle=8\sigma(d)\left(\sigma(-d)\right)^{2}$ (64)
$\displaystyle b_{f}^{\prime}(d)$
$\displaystyle=8\left(\sigma(d)\right)^{2}\sigma(-d)$ (65)

Here $\Delta$ is the conventional, non-canonical squared Le Cam distance with
$f^{\prime\prime}(1)=\tfrac{1}{2}$, and $\chi^{2}$ is the Pearson $\chi^{2}$
divergence defined below. The squared Le Cam distance has $(0,0)$ tail
weights. It is symmetric with respect to $p$ and $q$. The canonicalized
squared Le Cam distance may be obtained by q-softening the canonicalized
Pearson $\chi^{2}$ divergence or p-softening the canonicalized Neymann
divergence. The Le Cam distance defines a metric on the space of probability
distributions (Vajda, 2009).

The canonicalized _Pearson $\chi^{2}$ divergence_ (or _Kagan divergence_)
satisfies:

$\displaystyle D_{f}(p,q)$ $\displaystyle=\tfrac{1}{2}\chi^{2}(p,q)$ (66)
$\displaystyle=\tfrac{1}{2}\int\frac{\left(p(x)-q(x)\right)^{2}}{q(x)}\operatorname{d\\!}x$
(67) $\displaystyle E_{f}(p,q,d)$ $\displaystyle=-\tfrac{1}{2}+\int
p(x)\exp\left(d(x)\right)\operatorname{d\\!}x-\tfrac{1}{2}\int
q(x)\exp\left(2d(x)\right)\operatorname{d\\!}x$ (68) $\displaystyle f(u)$
$\displaystyle=\tfrac{1}{2}(u-1)^{2}$ (69) $\displaystyle f^{\prime\prime}(u)$
$\displaystyle=1$ (70) $\displaystyle a_{f}(d)$ $\displaystyle=\exp(d)-1$ (71)
$\displaystyle b_{f}(d)$ $\displaystyle=\tfrac{1}{2}\exp(2d)-\tfrac{1}{2}$
(72) $\displaystyle a_{f}^{\prime}(d)$ $\displaystyle=\exp(d)$ (73)
$\displaystyle b_{f}^{\prime}(d)$ $\displaystyle=\exp(2d)$ (74)

Here $\chi^{2}$ is the conventional, non-canonical Pearson $\chi^{2}$
divergence with $f^{\prime\prime}(1)=2$. The Pearson $\chi^{2}$ divergence has
$(0,3)$ tail weights. The p-softened canonicalized Pearson $\chi^{2}$
divergence is itself. The q-softened canonicalized Pearson $\chi^{2}$
divergence is the canonicalized squared Le Cam distance.

The canonicalized _Neymann divergence_ satisfies:

$\displaystyle D_{f}(p,q)$ $\displaystyle=\tfrac{1}{2}\chi^{2}(q,p)$ (75)
$\displaystyle=\tfrac{1}{2}\int\frac{\left(p(x)-q(x)\right)^{2}}{p(x)}\operatorname{d\\!}x$
(76) $\displaystyle E_{f}(p,q,d)$
$\displaystyle=-\tfrac{1}{2}-\tfrac{1}{2}\int
p(x)\exp\left(-2d(x)\right)\operatorname{d\\!}x+\int
q(x)\exp\left(-d(x)\right)\operatorname{d\\!}x$ (77) $\displaystyle f(u)$
$\displaystyle=\frac{(u-1)^{2}}{2u}$ (78) $\displaystyle f^{\prime\prime}(u)$
$\displaystyle=u^{-3}$ (79) $\displaystyle a_{f}(d)$
$\displaystyle=\tfrac{1}{2}-\tfrac{1}{2}\exp(-2d)$ (80) $\displaystyle
b_{f}(d)$ $\displaystyle=1-\exp(-d)$ (81) $\displaystyle a_{f}^{\prime}(d)$
$\displaystyle=\exp(-2d)$ (82) $\displaystyle b_{f}^{\prime}(d)$
$\displaystyle=\exp(-d)$ (83)

The Neymann divergence has $(3,0)$ tail weights. It is the reverse of the
Pearson $\chi^{2}$ divergence. The p-softened canonicalized Neymann divergence
is the canonicalized squared Le Cam distance. The q-softened canonicalized
Neymann divergence is itself.

The _softened reverse KL divergence_ (Shannon et al., 2020) satisfies:

$\displaystyle D_{f}(p,q)$
$\displaystyle=4\operatorname{KL}(\tfrac{1}{2}p+\tfrac{1}{2}q\,\|\,p)$ (84)
$\displaystyle\begin{split}E_{f}(p,q,d)&=2-4\log 2+2\int
p(x)\left[-\exp\left(-d(x)\right)-\log\sigma\left(d(x)\right)\right]\operatorname{d\\!}x\\\
&\quad-2\int q(x)\log\sigma\left(d(x)\right)\operatorname{d\\!}x\end{split}$
(85) $\displaystyle f(u)$ $\displaystyle=2(u+1)\log\frac{u+1}{u}-4\log 2$ (86)
$\displaystyle f^{\prime\prime}(u)$ $\displaystyle=\frac{2}{u^{2}(u+1)}$ (87)
$\displaystyle a_{f}(d)$ $\displaystyle=-2\exp(-d)-2\log\sigma(d)-2-2\log 2$
(88) $\displaystyle b_{f}(d)$ $\displaystyle=2\log\sigma(d)+2\log 2$ (89)
$\displaystyle a_{f}^{\prime}(d)$ $\displaystyle=2\exp(-d)\sigma(-d)$ (90)
$\displaystyle b_{f}^{\prime}(d)$ $\displaystyle=2\sigma(-d)$ (91)

The softened reverse KL divergence has $(2,0)$ tail weights. It is obtained by
q-softening the reverse KL divergence. This is the divergence approximately
minimized by conventional non-saturating GAN training (Shannon et al., 2020).

divergence | $f^{\prime\prime}(u)$ | (left, right) |
---|---|---|---
tail weights |
KL | $u^{-1}$ | $(1,2)$ |
reverse KL | $u^{-2}$ | $(2,1)$ |
Jensen-Shannon | $\frac{2}{u(1+u)}$ | $(1,1)$ | (harmonic mean of KL and RKL)
squared Hellinger | $u^{-\frac{3}{2}}$ | $(\frac{3}{2},\frac{3}{2})$ | (geometric mean of KL and RKL)
Jeffreys | $\frac{1+u}{2u^{2}}$ | $(2,2)$ | (arithmetic mean of KL and RKL)
squared Le Cam | $\frac{8}{(1+u)^{3}}$ | $(0,0)$ |
Pearson $\chi^{2}$ | $1$ | $(0,3)$ |
Neymann | $u^{-3}$ | $(3,0)$ |
softened reverse KL | $\frac{2}{u^{2}(1+u)}$ | $(2,0)$ |
Table 1: Concise specification of various f-divergences in terms of
$f^{\prime\prime}$. The divergences are scaled to make them canonical
($f^{\prime\prime}(1)=1$). All are low-degree rational functions of $u$ (or
$\sqrt{u})$. Tail weights, which determine the most important qualitative
properties of an f-divergence, are also shown (Shannon et al., 2020).

The $f^{\prime\prime}$ for various f-divergences is summarized in Table 1. We
see that $f^{\prime\prime}$ provides a particular simple and concise way to
define many common f-divergences. These are all rational functions of
$\sqrt{u}$.

###  2.4 Relationship to original f-GAN formulation

The original f-GAN paper (Nowozin et al., 2016) phrases the results presented
in §2 in terms of the _Legendre transform_ or _Fenchel conjugate_ $f^{*}$ of
$f$. The two descriptions are equivalent333Assuming $f$ is differentiable.
This is also assumed in practice in the original f-GAN paper. If this was not
the case then it would not be possible to train the critic using gradient-
based optimization. , as can be seen by setting $T(x)=f^{\prime}(u(x))$ and
using the result $f^{*}(f^{\prime}(u))=uf^{\prime}(u)-f(u)$. We find our
description helpful since it avoids having to explicitly match the domain of
$f^{*}$, ensures the optimal $d$ is the same for all $f$-divergences, and
because the Legendre transform is complicated for one of the divergences we
consider. An “output activation” was used in the original f-GAN paper to adapt
the output $d$ of the neural net to the domain of $f^{*}$. This is equal to
$f^{\prime}(\exp(d))$, up to irrelevant additive constants, for all the
divergences we consider, and so our description also matches the original
description in this respect.

##  3 Variational divergence minimization

f-GANs (Nowozin et al., 2016) generalize classic GANs to allow approximately
minimizing any f-divergence. In this section we review and discuss the f-GAN
formulation.

Consider the task of estimating a probabilistic model from data using an
f-divergence. Here $p$ is the true distribution and the goal is to minimize
$l(\lambda)=D_{f}(p,q_{\lambda})$ with respect to $\lambda$, where
$\lambda\mapsto q_{\lambda}$ is a parametric family of densities over
$\mathbb{R}^{K}$. We refer to $q_{\lambda}$ as the _generator_. For implicit
generative models such as typical GAN generators, the distribution
$q_{\lambda}$ is the result of a deterministic transform $g_{\lambda}(z)$ of a
stochastic latent variable $z$. However we do not need to assume this specific
form for most of our discussion.

###  3.1 Gradient matching property

We first note that the variational divergence bound $E_{f}$ satisfies a
convenient gradient matching property. This is not made explicit in the
original f-GAN paper. Denote the optimal $d$ given $p$ and $q_{\lambda}$ by
$d^{*}_{\lambda}$. We saw above that $D_{f}(p,q_{\lambda})$ and
$E_{f}(p,q_{\lambda},d)$ match values at $d=d^{*}_{\lambda}$. They also match
gradients with respect to the generator parameters $\lambda$:

$\mathinner{\dfrac{\partial{}}{\partial{\lambda}}}D_{f}(p,q_{\lambda})=\mathinner{\dfrac{\partial{}}{\partial{\lambda}}}E_{f}(p,q_{\lambda},d)\mathinner{\Biggr{\rvert}}_{d=d^{*}_{\lambda}}=-\int\left[\mathinner{\dfrac{\partial{}}{\partial{\lambda}}}q_{\lambda}(x)\right]b_{f}(d^{*}_{\lambda}(x))\operatorname{d\\!}x$
(92)

This follows from the fact that $E_{f}$ is a tight lower bound on $D_{f}$,
similarly to the one-dimensional result that any differentiable function
$f\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}\to\mathbb{R}$ with $f(x)\geq
0$ for all $x$ and $f(0)=0$ has $f^{\prime}(0)=0$. We can also verify this
property directly from the definitions of $D_{f}$ and $E_{f}$.

###  3.2 Formulation of variational divergence minimization

We can minimize $D_{f}(p,q_{\lambda})$ using _variational divergence
minimization_ , maximizing $E_{f}(p,q_{\lambda},d_{\nu})$ with respect to
$\nu$ while minimizing it with respect to $\lambda$. Adversarial optimization
such as this lies at the heart of all flavors of GAN training. Define
$\overline{\lambda}$ and $\overline{\nu}$ as

$\displaystyle\overline{\lambda}$
$\displaystyle=-\mathinner{\dfrac{\partial{}}{\partial{\lambda}}}E_{f}(p,q_{\lambda},d_{\nu})=\int\left[\mathinner{\dfrac{\partial{}}{\partial{\lambda}}}q_{\lambda}(x)\right]b_{f}(d_{\nu}(x))\operatorname{d\\!}x$
(93) $\displaystyle\overline{\nu}$
$\displaystyle=\mathinner{\dfrac{\partial{}}{\partial{\nu}}}E_{f}(p,q_{\lambda},d_{\nu})$
(94)

To perform the adversarial optimization, we can feed $\overline{\lambda}$ and
$\overline{\nu}$ (or in practice, stochastic approximations to them) as the
gradients into any gradient-based optimizer designed for minimization, e.g.
stochastic gradient descent or ADAM.

The gradient matching property shows that performing very many critic updates
followed by a single generator update is a sensible learning strategy which,
assuming the critic is sufficiently flexible and amenable to optimization,
essentially performs very slow gradient-based optimization on the true
divergence $D_{f}$ with respect to $\lambda$. However in practice performing a
few critic updates for each generator update, or simultaneous generator and
critic updates, performs well, and it is easy to see that these approaches at
least have the correct fixed points in terms of Nash equilibria of $E_{f}$ and
optima of $D_{f}$, subject as always to the assumption that the critic is
sufficiently richly parameterized. Convergence properties of these schemes are
investigated much more thoroughly elsewhere, for example (Nagarajan and
Kolter, 2017; Gulrajani et al., 2017; Mescheder et al., 2017, 2018; Balduzzi
et al., 2018; Peng et al., 2019), and are not the main focus here.

###  3.3 Hybrid training schemes

There is a simple generalization of the above training procedure, which is to
base the generator gradients on $E_{f}$ but the critic gradients on $E_{h}$
for a possibly different function $h$ (Poole et al., 2016, Section 2.3). We
refer to this as using _hybrid $(f,h)$_ gradients. This also approximately
minimizes $D_{f}$. Subject as always to the assumption of a richly
parameterized critic, if we perform very many critic updates for each
generator update, then the $d$ used to compute the generator gradient will
still be close to $d^{*}$, and so the generator gradient will be close to the
gradient of $D_{f}$, even though the path $d$ took to approach $d^{*}$ was
governed by $g$ rather than $f$. The fixed points of the two gradients are
also still correct, and so it seems reasonable to again use more general
update schemes and we might hope for similar convergence results (not analyzed
here).

Hybrid schemes may potentially be useful for stabilizing training. For example
the reverse KL generator gradient depends on $f$ only through
$b_{f}^{\prime}(d)=1$, so is likely to be stable with respect to minor
inaccuracies in the critic, but the reverse KL critic gradient involves
$a_{f}^{\prime}(d)=\exp(-d)$ which may lead to very large updates if $d(x)$ is
ever large and negative for real $x$. A hybrid (reverse KL, Jensen-Shannon)
scheme uses a stable update for both the generator and critic while still
approximately minimizing reverse KL.

###  3.4 Low-dimensional generator support

Many GAN generators used in practice have low-dimensional support and do not
satisfy the condition $q(x)>0$ for all $x$ which we assumed for simplicity. In
this section we briefly discuss the implications of this for GAN training. We
argue that using generators with $q(x)>0$ everywhere has both theoretical and
practical benefits, and so it is not unreasonable to restrict attention to
this case.

The vast majority of GAN generators consist of a deterministic neural net
applied to a fixed source of noise. Often the noise is far lower-dimensional
than the output space, meaning that the set of possible generator outputs for
a given trained generator (its _support_ as a probability distribution) is a
low-dimensional manifold in output space. For example, the progressive GAN
generator (Karras et al., 2018) used a $512$-dimensional noise source and an
output space with roughly $3$ million dimensions. The natural data is often
assumed to also lie on a low-dimensional manifold in output space, but we
would argue that this is almost never exactly the case in practice. Due to
sensor noise and dithering if nothing else, it is difficult to say that any
image, say, is literally _impossible_ in natural data. If desired, we can even
guarantee that this is the case by adding a perceptually insignificant amount
of white noise to the data. It seems more accurate to say that the natural
data lies _close to_ a low-dimensional manifold, but that no output is
impossible, i.e. $p(x)>0$ everywhere. The low-dimensional generator support
combined with high-dimensional data distribution support leads to several
pathologies:

  * •

The set of all possible generator outputs has probability zero under the data
distribution.

  * •

With probability $1$, the generator assigns a natural image a probability
density of zero.

  * •

The KL divergence between the data distribution and the generator is infinite.

  * •

The true log likelihood of natural data under the model is $-\infty$ (despite
approaches based on Parzen windows which produce finite estimates for the log
likelihood).

  * •

Essentially all f-divergences are either undefined or completely saturate with
gradient precisely zero.

  * •

The optimal critic $d^{*}(x)=\log p(x)-\log q(x)$ is $\infty$ almost
everywhere.

  * •

A sufficiently powerful critic can learn to distinguish generator output
essentially perfectly by outputting “fake” for a tiny sliver around the low-
dimensional generator support (a region which has vanishingly small
probability under the data distribution) without learning anything about the
natural data distribution. In general the critic is incentivized to focus on
tiny details relevant to detecting the current generator support but
potentially imperceptible to a human, and the generator is incentivized to
change these tiny details slightly to move the current support.

These pathologies seem highly undesirable for a probabilistic model. Much
attention has been devoted to this issue, especially under the assumption that
the data support is also low-dimensional (Arjovsky et al., 2017; Mescheder et
al., 2018, for example), and it is one of the motivating scenarios for
Wasserstein GANs.

This issue is extremely easy to fix by injecting noise at all levels of the
generator network, including the output (with a learned variance parameter).
If a given injection of noise is not useful then it easy enough for the
generator to learn to ignore it. This injected noise (indeed, even just the
output noise) is enough to formally make the generator support cover all of
output space and eliminate all of the above theoretical pathologies. The
dimensionality argument above no longer applies since the dimensionality of
all noise sources is now greater than the output dimensionality (the output
noise alone ensures this). The injected noise can also have advantages in
practice (Karras et al., 2019).

## References

  * Ali and Silvey (1966) S. M. Ali and S. D. Silvey.  A general class of coefficients of divergence of one distribution from another.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 28(1):131–142, 1966.
  * Arjovsky et al. (2017) M. Arjovsky, S. Chintala, and L. Bottou.  Wasserstein generative adversarial networks.  In _Proc. ICML_ , pages 214–223, 2017.
  * Balduzzi et al. (2018) D. Balduzzi, S. Racaniere, J. Martens, J. Foerster, K. Tuyls, and T. Graepel.  The mechanics of n-player differentiable games.  In _Proc. ICML_ , 2018.
  * Burbea and Rao (1982) J. Burbea and C. Rao.  On the convexity of some divergence measures based on entropy functions.  _IEEE Transactions on Information Theory_ , 28(3):489–495, 1982.
  * Csiszár (1967) I. Csiszár.  Information-type measures of difference of probability distributions and indirect observation.  _Studia Scientiarum Mathematicarum Hungarica_ , 2:229–318, 1967.  Available at http://real-j.mtak.hu/id/eprint/5453.
  * Endres and Schindelin (2003) D. M. Endres and J. E. Schindelin.  A new metric for probability distributions.  _IEEE Transactions on Information theory_ , 49(7):1858–1860, 2003.
  * Gulrajani et al. (2017) I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville.  Improved training of Wasserstein GANs.  In _Advances in Neural Information Processing Systems_ , pages 5767–5777, 2017.
  * Karras et al. (2018) T. Karras, T. Aila, S. Laine, and J. Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In _Proc. ICLR_ , 2018.
  * Karras et al. (2019) T. Karras, S. Laine, and T. Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4401–4410, 2019.
  * Lin (1991) J. Lin.  Divergence measures based on the Shannon entropy.  _IEEE Transactions on Information theory_ , 37(1):145–151, 1991.
  * Mescheder et al. (2017) L. Mescheder, S. Nowozin, and A. Geiger.  The numerics of GANs.  In _Advances in Neural Information Processing Systems_ , pages 1825–1835, 2017.
  * Mescheder et al. (2018) L. Mescheder, A. Geiger, and S. Nowozin.  Which training methods for GANs do actually converge?  In _Proc. ICML_ , 2018.
  * Nagarajan and Kolter (2017) V. Nagarajan and J. Z. Kolter.  Gradient descent GAN optimization is locally stable.  In _Advances in Neural Information Processing Systems_ , pages 5585–5595, 2017.
  * Nguyen et al. (2010) X. Nguyen, M. J. Wainwright, and M. I. Jordan.  Estimating divergence functionals and the likelihood ratio by convex risk minimization.  _IEEE Transactions on Information Theory_ , 56(11):5847–5861, 2010.
  * Nowozin et al. (2016) S. Nowozin, B. Cseke, and R. Tomioka.  f-GAN: Training generative neural samplers using variational divergence minimization.  In _Advances in Neural Information Processing Systems_ , pages 271–279, 2016.
  * Österreicher and Vajda (2003) F. Österreicher and I. Vajda.  A new class of metric divergences on probability spaces and its applicability in statistics.  _Annals of the Institute of Statistical Mathematics_ , 55(3):639–653, 2003.
  * Peng et al. (2019) W. Peng, Y. Dai, H. Zhang, and L. Cheng.  Training GANs with centripetal acceleration.  _arXiv preprint arXiv:1902.08949_ , 2019.
  * Poole et al. (2016) B. Poole, A. A. Alemi, J. Sohl-Dickstein, and A. Angelova.  Improved generator objectives for GANs.  In _Proc. NIPS Workshop on Adversarial Training_ , 2016.
  * Shannon et al. (2020) M. Shannon, S. Mariooryad, B. Poole, T. Bagby, E. Battenberg, D. Kao, D. Stanton, and R. Skerry-Ryan.  The divergences minimized by non-saturating GAN training.  _arXiv preprint arXiv:_ , 2020.
  * Topsoe (2000) F. Topsoe.  Some inequalities for information divergence and related measures of discrimination.  _IEEE Transactions on Information Theory_ , 46(4):1602–1609, 2000.
  * Vajda (2009) I. Vajda.  On metric divergences of probability measures.  _Kybernetika_ , 45(6):885–900, 2009.