# Conformally Natural Families of Probability Distributions on Hyperbolic Disc
with a View on Geometric Deep Learning

Vladimir Jaćimović  Faculty of Sciences and MathematicsUniversity of
MontenegroDžordža Vašingtona bb81000 PodgoricaMontenegro<EMAIL_ADDRESS>and  Marijan Marković  Faculty of Sciences and MathematicsUniversity of
MontenegroDžordža Vašingtona bb81000 PodgoricaMontenegro
<EMAIL_ADDRESS>
###### Abstract.

We introduce the novel family of probability distributions on hyperbolic disc.
The distinctive property of the proposed family is invariance under the
actions of the group of disc-preserving conformal mappings. The group-
invariance property renders it a convenient and tractable model for encoding
uncertainties in hyperbolic data. Potential applications in Geometric Deep
Learning and bioinformatics are numerous, some of them are briefly discussed.
We also emphasize analogies with hyperbolic coherent states in quantum
physics.

###### Key words and phrases:

hyperbolic data embedding; wrapped Cauchy; homogeneous spaces; Möbius
transformations; directional statistics; geometric deep learning

###### 2020 Mathematics Subject Classification:

Primary 65C20; Secondary 60-08

##  1\. Introduction

Mathematically natural probability models are those that are invariant under
the actions of a certain transformation group. This principle which underlies
the field of integral geometry [1] appears to be particularly relevant for
machine learning (ML). Indeed, stochastic policies parametrized by group-
invariant families have many advantages. By implementing such policies one
gains in efficiency and transparency of algorithms.

A notable example supporting the above points is the family of Gaussian
distributions. This family provides by far the most popular model in inference
problems, probabilistic modeling and stochastic search over continuous spaces.
Being group-invariant, the Gaussian family has very modest representative
power. Those setups in which this limitation is essential, can be addressed by
involving mixtures. The dominance of Gaussian distributions has a strong
justification, as they posses a unique combination of desirable properties.
One could present information-theoretic (Gaussians are maximum entropy
distributions), convex-analytic (Gaussians belong to the exponential family)
and information-geometric (there is a pretty simple explicit expression for
the Fisher information) arguments in favor of this family. Still, one might
argue that the most important virtue is invariance under the group of linear-
affine transformations of the vector space. The last property is (often
implicitly) exploited in many algorithms where it plays a crucial role.

In conclusion, the Gaussian family justifiably holds a superior role in
statistical ML over continuous spaces. The last assertion is true as long as
these spaces are equipped with some flat metric. However, many data sets
coming from various fields of science, engineering and everyday life are
characterized by non-zero curvatures. This means that such data are naturally
embedded into non-Euclidean spaces. It is easy to support this point by many
examples. For instance, data representing rotations, or orientations in space
obviously have inherent spherical geometry. On the other side, hierarchical
data (such as complex networks [2], or semantic hierarchies [3, 4]) are
naturally embedded into hyperbolic geometries. Yet another (immensely broad)
class of non-Euclidean data sets are those representing beliefs or
uncertainties. The field of information geometry [5] sprang from the
observation that the natural metric on families of probability measures is
generated by the Kullback-Leibler divergence, thus turning them into
statistical (curved) manifolds.

Significance and ubiquity of non-Euclidean data has been widely recognized in
data science and ML only in the last two decades. This motivated enormous
research efforts, thus paving the way for the emerging field of geometric deep
learning [6].

Modeling uncertainty in non-Euclidean data requires statistical models
(families of probability measures) over curved spaces. In such setups the
Gaussian family becomes inadequate. The mathematical framework for
probabilistic ML over Riemannian manifolds is partially provided by
directional statistics. Directional statistics is a subdiscipline within the
broad field of probability and statistics which deals with probability
distributions over spheres and rotation groups [7, 8]. Although it provides a
solid mathematical apparatus, its significance for ML is recognized only
recently and applications are still pretty sparse. Moreover, statistical
models over manifolds with negative curvature are beyond the scope of
directional statistics and are poorly investigated. This is to be changed in
the near future, due to advances in ML over hyperbolic geometries [9, 10, 11].
Recent trends entail the necessity for the corresponding statistical models in
order to encode uncertainties in hyperbolic spaces.

The main goal of the present study is to introduce the family of probability
distributions on hyperbolic disc which has desirable properties for
probabilistic ML. The group-invariance property facilitates implementation of
algorithms and improves their transparency. For the completeness of
exposition, in Section 2 we revisit the previously known family on the circle
with the group-invariance property. In Section 3 we introduce the novel family
on hyperbolic disc and derive the most essential formulae. Relations with
mathematical physics are exposed in Section 4. Potential applications in ML
are briefly pointed out in Section 5. Finally, Section 6 contains some
concluding remarks and an outlook for the future research.

##  2\. The wrapped Cauchy family on the circle

One of the most important families of probability measures on the unit circle
is obtained by ”wrapping” the Cauchy (Lorentzian) distributions from the real
line to the circle. The ”wrapping” is realized by the Cayley transform and
yields so-called wrapped Cauchy distributions which are described by the
density functions [12, 13]

(2.1)
$p_{wC}(\varphi)=\frac{1}{2\pi}\frac{1-r^{2}}{1-2r\cos(\varphi-\Phi)+r^{2}},\;\varphi\in{\mathbb{S}}^{1}.$

Denote this family by $wC(a)$, where $a=re^{i\Phi}$ is a point in the unit
disc. The Fisher information metric for this family coincides with the
hyperbolic metric in the unit disc [14]. Hence, the statistical manifold
$wC(a)$ is isomorphic to the hyperbolic disc ${\mathbb{B}}^{2}$.

It is easy to verify that the mean value (center of mass) of the density (2.1)
is equal to $a$.

The densities (2.1) can be written in another parametrization using the
complex variable $z=e^{i\varphi}$ as follows

$p_{wC}(z)=\frac{1}{2\pi}\frac{1-|a|^{2}}{|z-a|^{2}}.$

Although $wC(a)$ is one of the central families in directional statistics, it
has not been, to our best knowledge, used in ML so far. We argue that $wC(a)$
provides the most convenient statistical model on the circle (and tori) with
important favorable properties. Obviously, the family (2.1) contains the
uniform density on ${\mathbb{S}}^{1}$ for $r=0$. Moreover, the Dirac delta
distributions on the circle are limit cases of (2.1) when $r\to 1$.

In order to clarify the group-invariance property of $wC(a)$ denote by
${\mathbb{G}}$ the group of Möbius transformations which leave the unit disk
$\mathbb{B}^{2}$ invariant, i.e. the group of conformal automorphisms of
$\mathbb{B}^{2}$:

$\mathrm{Aut}(\mathbb{B}^{2})=\left\\{z\to
e^{i\theta}\frac{a-z}{1-\overline{a}z}:a\in\mathbb{B}^{2},\,\theta\in[0,2\pi)\right\\}.$

Recall that ${\mathbb{G}}$ is isomorphic to the matrix group
$PSU(1,1)=SU(1,1)/\pm I$.

For $a\in\mathbb{B}^{2}$, denote by

(2.2) $g_{a}(z)=\frac{a-z}{1-\overline{a}z}$

the involutive Möbius transformation of $\mathbb{B}^{2}$ (in the sense that
$g_{a}\circ g_{a}$ is the identity transformation, and therefore
$g_{a}^{-1}=g_{a}$). Note that $g_{a}(0)=a$ and $g_{a}(a)=0$.

######  Proposition 2.1.

The family $wC(a)$ is invariant under the actions of the Möbius group
${\mathbb{G}}$. All wrapped Cauchy distributions are obtained as Möbius
transformations of the uniform distribution on the circle.

In other words, $wC(a)$ is the two-dimensional orbit of the three-dimensional
Lie group ${\mathbb{G}}$ (where the dimension is understood in the sense of
real numbers). Moreover, ${\mathbb{G}}$ acts transitively on this family,
meaning that for any pair of measures $\mu_{1}$ and $\mu_{2}$ from $wC(a)$,
there exists a transformation $g\in{\mathbb{G}}$, such that
$\mu_{2}=g_{*}\mu_{1}$. Here, the notation $g_{*}$ stands for the pull-back
measure, defined by

$g_{*}\nu(A)=\nu(g^{-1}(A))\mbox{ for any Borel set
}A\subseteq{\mathbb{S}}^{1}.$

Due to the low dimension of statistical manifold $wC(a)$ the representative
power of this model is very limited. Indeed, all densities (2.1) are unimodal
and symmetric. Just as for the Gaussian family, the representative power can
be improved by considering mixtures of the densities (2.1).

######  Remark 2.2.

The family of von Mises distributions [7] is essentially the only circular
statistical model that has been actively exploited in ML and bioinformatics so
far. Notice that von Mises distributions are special cases of the von Mises-
Fisher distributions over spheres. The von Mises family has some favorable
information-theoretic properties (maximum entropy; they naturally appear as
stationary distributions for diffusion processes on the circle). However,
unlike wrapped Cauchy, this family does not have group-invariance properties.
It is invariant only under the group of planar rotations.

We refer to [15] for a brief overview of probability distributions on circles,
tori, spheres with an emphasis on their generation and (potential)
applications in ML.

######  Remark 2.3.

Another family of probability distributions which is invariant under the
Möbius group is so-called Kato-Jones (K-J) family [16]. This family gives rise
to the $4$-dimensional statistical manifold, which consists of those
probability distributions that are obtained by actions of the Möbius group on
von Mises distributions. For fixed concentration parameter of the von Mises
distribution, one obtains orbits of the Möbius group consisting of
$3$-dimensional K-J sub-families. The K-J family is well investigated, with
closed form expression for the density and a number of useful formulae
available [16]. This family is more demanding to work with than the wrapped
Cauchy, but has higher representative power. Wrapped Cauchy distributions
constitute the sub-manifold within the K-J family, obtained as a special case
when the concentration parameter equals zero (meaning that the initial von
Mises distribution is in fact uniform). The whole $4$-dimensional K-J family
contains asymmetric and bimodal densities, along with symmetric and unimodal
ones.

##  3\. Conformally natural family of probability distributions over the
hyperbolic disc

Consider analytic functions on the unit disk generated by the family
$\\{g_{a}:a\in\mathbb{B}^{2}\\}$ of involutive Möbius transformations

$\begin{split}\mathcal{F}_{\alpha,a}&=\left\\{p_{a}(z)=c_{\alpha}(1-|g_{a}(z)|^{2})^{\alpha}:a\in\mathbb{B}^{2}\right\\}\\\
&=\left\\{c_{\alpha}\left(1-\left|\frac{a-z}{1-\overline{a}z}\right|^{2}\right)^{\alpha}\right\\}\\\
&=\left\\{c_{\alpha}\left(\frac{(1-|a|^{2})(1-|z|^{2})}{|1-\overline{a}z|^{2}}\right)^{\alpha}\right\\}\\\
&=\left\\{c_{\alpha}(1-|a|^{2})^{\alpha}\left(\frac{(1-|z|^{2})}{|1-\overline{a}z|^{2}}\right)^{\alpha}\right\\}.\end{split}$

The point $a=|a|e^{i\varphi}\in{\mathbb{B}}^{2}$ and real number $\alpha>1$
are parameters of the family, while $c_{\alpha}$ are normalizing constants,
chosen in such a way to make the above functions probability densities over
${\mathbb{B}}^{2}$:

(3.1) $\int_{\mathbb{B}^{2}}p_{a}(z)d\tau(z)=1,$

where $d\tau(z)$ denotes the element of area in hyperbolic metric. Recall that
the area-hyperbolic density on the unit hyperbolic disc reads

$\tau(z)=\frac{1}{(1-|z|^{2})^{2}}$

The area-hyperbolic measure of a Lebesgue set $U\subseteq\mathbb{B}^{2}$ is
given by

$\mathrm{hyp-area}(U)=\int_{U}\tau(z)dA(z),$

where $dA(z)$ is the standard (Euclidean) Lebesgue measure on
$\mathbb{C}=\mathbb{R}^{2}$.

In order to calculate $c_{\alpha}$ we need to evaluate the integral (3.1). To
that end we will need the following

######  Proposition 3.1.

For any automorphism $g$ of the unit disk onto itself, the following
invariance formula holds:

$\int_{\mathbb{B}^{2}}h(g(z))\tau(z)dA(z)=\int_{\mathbb{B}^{2}}h(w)\tau(w)dA(w),$

where $h$ is any Lebesgue measurable mapping.

For the proof we refer to [17, p.2].

Using the above Proposition, we find that

$\begin{split}c_{\alpha}^{-1}&=\int_{\mathbb{B}^{2}}(1-|T_{a}(z)|^{2})^{\alpha}\tau(z)dA(z)=\int_{\mathbb{B}^{2}}(1-|w|^{2})^{\alpha}\tau(w)dA(w)\\\
&=\int_{\mathbb{B}^{2}}(1-|w|^{2})^{\alpha-2}dA(w)=\int_{0}^{1}\int_{0}^{2\pi}(1-r^{2})^{\alpha-2}rdrd\theta\\\
&=2\pi\int_{0}^{1}(1-r^{2})^{\alpha-2}rdr=\frac{\pi}{\alpha-1}.\end{split}$

In conclusion, we consider the family of probability measures over
${\mathbb{B}}^{2}$ defined by the following densities

(3.2)
$p(z;\alpha,a)=\frac{\alpha-1}{\pi}(1-|a|^{2})^{\alpha}\left(\frac{(1-|z|^{2})}{|1-\overline{a}z|^{2}}\right)^{\alpha}$

depending on parameters $\alpha>1$ and $a\in{\mathbb{B}}^{2}$.

######  Remark 3.2.

We note that

$\lim_{a\to
e^{i\varphi}}c_{\alpha}^{-1}(1-|a|^{2})^{-\alpha}p_{a}(z)=\mathrm{P}(|z|,\theta-\varphi),\quad
z=|z|e^{i\theta},$

where

$\mathrm{P}(r,\Theta)=\frac{1-r^{2}}{1-2r\cos\Theta+r^{2}}.$

In the absence of better term we (following Douady and Earle [18]) refer to
densities (3.2) as conformally natural family of probability measures on the
hyperbolic disc ${\mathbb{B}}^{2}$. We will denote this family by
${\mathcal{F}}_{\alpha,a}$.

The family (3.2) is invariant under the Möbius group. Indeed, if
$\nu\in\mathcal{F}_{\alpha,a}$ and $g\in{\mathbb{G}}$, then obviously
$g_{*}\nu\in\mathcal{F}_{\alpha,g(a)}$.

The space $\mathcal{F}_{\alpha,a}$ is three-dimensional, parametrized by
$\alpha>1$ and a complex number $a,\,|a|<1$. By fixing $\alpha$, we obtain
two-dimensional statistical sub-manifold.

######  Proposition 3.3.

For fixed $\alpha=\alpha_{0}$ the Möbius group acts transitively on
$\mathcal{F}_{\alpha_{0},a}$. In other words, for any two measures $\mu_{v}$
and $\mu_{w}$ from $\mathcal{F}_{\alpha_{0},a}$, there exists a Möbius
transformation $g$, such that $\mu_{w}=g_{*}\mu_{v}$.

###### Proof.

Let $\mu_{w}$ and $\mu_{v}$ be two measures defined by the density (3.2)
parametrized by points $w$ and $v$ in ${\mathbb{B}}^{2}$, respectively.
Consider the Möbius transformation $g=g_{w}\circ g_{v}^{-1}=g_{w}\circ g_{v}$,
where $g_{v}$ and $g_{w}$ are defined by (2.2). Then, it is evident that
$\mu_{w}=g_{*}\mu_{v}=(g_{w}\circ g_{v})_{*}\mu_{v}$. ∎

######  Proposition 3.4.

Let $Z$ be the random variable with the density (3.2). The probability that
$Z$ belongs to the disk $D_{b}=\\{z:|z|<b\\}$ for $0<b<1$ is given by the
following formula

(3.3)
${\mathbb{P}}\\{|Z|<b\\}=(\alpha-1)(1-|a|^{2})^{\alpha}\int_{0}^{\sqrt{b}}(1-s)^{\alpha-2}{{}_{2}}F_{1}(\alpha,\alpha,1,|a|^{2}s)ds.$

###### Proof.

In the proof we will refer to the following formula (see [20, Lemma 2.1]):

(3.4)
$\frac{1}{2\pi}\int_{0}^{2\pi}\frac{dt}{|x-e^{it}|^{2\alpha}}={{}_{2}}F_{1}(\alpha,\alpha,1,|x|^{2}),\mbox{
where }x\in{\mathbb{B}}^{2},$

where the notion ${{}_{2}}F_{1}(\cdot,\cdot,\cdot,\cdot)$ stands for the
Gauss’ hypergeometric series.

Passing to polar coordinates $a=|a|e^{i\varphi}$, and $z=re^{i\theta}$, we
obtain the following chain of equalities:

$\begin{split}{\mathbb{P}}\\{z\in
D_{b}\\}&=\int_{D_{b}}p_{a}(z)d\tau(z)=\frac{\alpha-1}{\pi}\int_{D_{b}}\left(\frac{(1-|a|^{2})(1-|z|^{2})}{|1-\overline{a}z|^{2}}\right)^{\alpha}d\tau(z)\\\
&=\frac{\alpha-1}{\pi}(1-|a|^{2})^{\alpha}\int_{D_{b}}\frac{(1-|z|^{2})^{\alpha-2}}{|1-\overline{a}z|^{2\alpha}}dA(z)\\\
&=\frac{\alpha-1}{\pi}(1-|a|^{2})^{\alpha}\int_{0}^{b}\int_{0}^{2\pi}\frac{(1-r^{2})^{\alpha-2}}{|1-|a|re^{i(\theta-\varphi)}|^{2\alpha}}rdrd\theta\\\
&=\frac{\alpha-1}{\pi}(1-|a|^{2})^{\alpha}\int_{0}^{b}\int_{0}^{2\pi}\frac{(1-r^{2})^{\alpha-2}}{|1-|a|re^{it}|^{2\alpha}}rdrdt\\\
&=\frac{\alpha-1}{\pi}(1-|a|^{2})^{\alpha}\int_{0}^{b}r(1-r^{2})^{\alpha-2}\int_{0}^{2\pi}\frac{dt}{|e^{it}-|a|r|^{2\alpha}}\\\
&=2(\alpha-1)(1-|a|^{2})^{\alpha}\int_{0}^{b}r(1-r^{2})^{\alpha-2}{{}_{2}}F_{1}(\alpha,\alpha,1,|a|^{2}r^{2})dr\\\
&=(\alpha-1)(1-|a|^{2})^{\alpha}\int_{0}^{b}(1-r^{2})^{\alpha-2}{{}_{2}}F_{1}(\alpha,\alpha,1,|a|^{2}r^{2})d(r^{2})\\\
&=(\alpha-1)(1-|a|^{2})^{\alpha}\int_{0}^{\sqrt{b}}(1-s)^{\alpha-2}{{}_{2}}F_{1}(\alpha,\alpha,1,|a|^{2}s)ds.\end{split}$

∎

The formula (3.3) is quite complicated in general case, involving the integral
over Gauss’ hypergeometric series. Simple expressions can be obtained in some
special cases, most notably for $\alpha=2$, which will be exposed below.

The integral (3.3) can also be easily evaluated when $a=0$. This is
substantiated in the following

######  Corollary 3.5.

Let $Z$ be the random variable with the density

$p(z;\alpha,0)=\frac{\alpha-1}{\pi}(1-|z|^{2})^{\alpha}.$

Then the probability that $Z$ belongs to the disc with radius $b$ equals

(3.5) ${\mathbb{P}}\\{|Z|<b\\}=1-(1-\sqrt{b})^{\alpha-1}.$

From the formula (3.5) it is obvious that the distributions (3.2) are more
concentrated for greater values of $\alpha$, with the Dirac delta
distributions arising in the limit $\alpha\to\infty$.

When dealing with probability measures on Riemannian manifolds, the meaningful
analogue of mathematical expectation is the Riemannian center of mass (also
referred to as Karcher mean). In our setup, this is a point in
${\mathbb{B}}^{2}$ defined as a unique minimizer of the following functional
[19]:

$J(z)=\int_{{\mathbb{B}}^{2}}d_{hyp}^{2}(z,p(z;\alpha,a))d\tau(z),$

where $p(z;\alpha,a)$ is defined by (3.2) and $d_{hyp}$ denotes the hyperbolic
distance in the disc.

######  Proposition 3.6.

The Riemannian center of mass of densities (3.2) is $a$.

In general, computation of the Riemannian center of mass may be demanding and
requires the gradient descent algorithm [19]. However, thanks to the group-
invariance of the family ${\mathcal{F}}_{a,\alpha}$ it suffices to apply the
simple geometric argument in order to verify the above Proposition. Indeed,
for $a=0$ all densities (3.2) are rotationally symmetric. Hence, their mean
must be zero. Furthermore, the densities with the same parameter $\alpha$ are
related by isometries (2.2) which map zero to point $a$. Then, the mean
(center of mass) is mapped by by the same isometry, i.e. it equals $g(0)=a$.

Figure 1. The plot of $100$ randomly generated points from
$Z\sim{\mathcal{F}}_{a,\alpha}$ for $\alpha=2$ and $a=0$ (left), $a=1/2$
(right).

Figure 2. The plot of $100$ randomly generated points from
$Z\sim{\mathcal{F}}_{a,\alpha}$ for $\alpha=10$ and $a=0$ (left), $a=1/2$
(right).

###  3.1. Random variate generation in the hyperbolic disc

One could think of several approaches to the generation of random points from
distributions ${\mathcal{F}}_{\alpha,a}$. The simplest method is based on
Corollary 3.4.

Recall that the Möbius group acts transitively on sub-families
${\mathcal{F}}_{2,a}$ for fixed $\alpha=\alpha_{0}$. Therefore, it suffices to
generate the random sample from the distribution with parameters $\alpha_{0}$
and $0$. Then, we can act on these random points by the Möbius transformation
in order to obtain the random sample from the distribution with parameter
values $\alpha_{0}$ and $a$.

In order to generate random points from ${\mathcal{F}}_{a,\alpha}$ with $a=0$
and arbitrary $\alpha$ notice that densities for $a=0$ are rotationally
symmetric for all values $\alpha>1$. Therefore, the angles are uniformly
distributed on $[0,2\pi]$, while the distribution of their radii is defined by
the function (3.5).

This yields the following algorithm for generation of random variate from the
distribution with parameters $\alpha$ and $a$.

1\. Sample two random numbers $U_{1}$ and $U_{2}$ from the uniform
distribution on $[0,1]$.

2\. Set $\psi=2\pi U_{1}$.

3\. Obtain the random number $\rho$ by inverting the distribution function
(3.5), that is - by solving the equation

$1-(1-\sqrt{\rho})^{\alpha-1}=U_{2}$

which yields

$\rho=\left(1-\sqrt[\alpha-1]{1-U_{2}}\right)^{2}.$

4\. Set $z=\rho e^{i\psi}$. The random point $z\in{\mathbb{B}}^{2}$ is
generated from the distribution ${\mathcal{F}}_{\alpha,0}$.

5\. Denote $\zeta=g_{a}(z)$, where $g_{a}$ is defined by (2.2). The random
point $\zeta\in{\mathbb{B}}^{2}$ is generated from the distribution
${\mathcal{F}}_{\alpha,a}$.

In figures 1 and 2 we depict the sets of random points from
${\mathcal{F}}_{\alpha,a}$ for $\alpha=2$ and $\alpha=10$, respectively, and
three values of $a$. It is evident that distributions for $\alpha=10$ are more
concentrated around their mean values, than those obtained for $\alpha=2$.

###  3.2. The special case $\alpha=2$

As emphasized above, the formula (3.2) defines a class of families of
probability measures, parametrized by $\alpha>1$. This class is invariant
under the actions of the Möbius group. However, the Möbius group does not act
transitively on the whole class, but only on sub-families obtained by fixing
$\alpha$. Algorithms over these two-dimensional statistical sub-manifolds can
be realized by learning Möbius transformations (which boils down to learning
generators of the Möbius group).

The most transparent sub-manifold is $\mathcal{F}_{2,a}$ which is parametrized
by a single point $a\in{\mathbb{B}}^{2}$.

Gauss’ hypergeometric series admit a simple expression for $\alpha=2$

${{}_{2}}F_{1}(2,2,1,|a|^{2}s)=\frac{1+|a|^{2}s}{(1-|a|^{2}s)^{3}},$

Therefore, setting $\alpha=2$ in (3.3) yields

(3.6) $\begin{split}P\\{z\in
D_{b}\\}&=(1-|a|^{2})^{2}\int_{0}^{\sqrt{b}}\frac{1+|a|^{2}s}{(1-|a|^{2}s)^{3}}ds=\frac{(1-|a|^{2})^{2}\sqrt{b}}{(1-|a|^{2}\sqrt{b})^{2}}.\end{split}$

In conclusion, the two-dimensional statistical manifold ${\mathcal{F}}_{2,a}$
provides the convenient statistical model for a wide range of problems of
inference, black-box optimization and probabilistic modeling in hyperbolic
disc. This particular family allows for simple and tractable formulae that can
be used for sampling, parameter estimation, etc.

The family ${\mathcal{F}}_{2,a}$ is the orbit of the Möbius group obtained by
actions of this group on the probability measure defined by the density:

(3.7) $p(z;2,0)=\frac{1}{\pi}(1-|z|^{2})^{2}.$

Obviously, this family has very limited representative power. Again, this
limitation can be addressed by considering mixtures of probability measures
from ${\mathcal{F}}_{2,a}$. Notice that the Dirac delta distributions inside
the disc can not be obtained as limiting case of densities belonging to the
family $\mathcal{F}_{2,a}$. This means that those stochastic policies that are
supposed to converge towards an optimal point should gradually increase the
value of $\alpha$ in order to increase the concentration and thus ensure
convergence towards the Dirac delta distribution.

##  4\. Relations with mathematical physics

Wrapped Cauchy distributions were introduced to statistics in 1990’s, [12, 13]
but the expression (2.1) for their densities is familiar to mathematicians for
longer than one century. If we consider (2.1) as analytic functions in the
hyperbolic disc (depending on the variable $a$), they appear to be well known
Poisson kernels. Poisson kernels are harmonic functions for the hyperbolic
Laplace-Beltrami operator on the disc and play a central role in complex
analysis and potential theory. They appear in many important formulae in
mathematical physics.

In the same way, densities (3.2) appear as eigenfunctions of the hyperbolic
Laplace-Beltrami operator in the unit disc corresponding to eigenvalues
$\alpha(\alpha-1)$. This fact underpins their relevance in quantum optics
[21].

In general quantum theory functions of the form (3.2) arise as
$SU(1,1)$-coherent states (or hyperbolic coherent states) in analytic
representations, see [22, 23, 24].

Adopting the abstract framework of coherent states in quantum mechanics [22],
the distribution with $a=0$ corresponds to the ground state. Indeed, densities
(3.2) for $a=0$ are rotationally symmetric, that is - they are invariant under
the maximal compact subgroup $SO(2)\sim{\mathbb{S}}^{1}$ of $SU(1,1)$. Recall
that the hyperbolic disc is a homogeneous space, obtained by taking the
quotient of $SU(1,1)$ over its subgroup $SO(2)$, i.e.
$SU(1,1)/SO(2)={\mathbb{B}}^{2}$. Hence, in accordance with the general group-
theoretic framework of coherent quantum states [22], functions (3.2) are
obtained by actions of the group $SU(1,1)$ on the ground state and, hence, is
parametrized by points in ${\mathbb{B}}^{2}$. In our setup, the ground state
corresponds to the density (3.7).

The correspondence between the family (3.2) and $SU(1,1)$-coherent states
further implies an analogy between learning over this family and quantum
evolution on invariant manifold of coherent states. Therefore, an algorithms
over ${\mathcal{F}}_{a,\alpha}$ can be implemented by controlling the
Hamiltonian of a quantum system. Since the Hamiltonian of the system is a
linear combination of generators of $SU(1,1)$, its control is realized by
updating the coefficients of this linear combination.

##  5\. Significance for ML and bioinformatics

Recent trends in Geometric DL and data science indicate that the significance
of hyperbolic data embeddings and optimization methods over hyperbolic spaces
will rapidly grow both in theoretical and practical ML. This entails the
necessity for tractable statistical models over hyperbolic manifolds. The
family (3.2) provides a framework for many ML setups, such as parametrization
of stochastic policies in reinforcement learning, or for the evolutionary
optimization over hyperbolic (multi)discs.

There are many tasks where family (3.2) can be used for the design of
efficient and robust algorithms. A wide range of learning algorithms dealing
with uncertainties in hyperbolic data can be implemented using (mixtures or
generalizations of) the family (3.2).

One important research direction in data science investigates and exploits
embeddings of complex networks into hyperbolic manifolds. Networks are
represented by so-called geometric graphs (sets of points), where the
hyperbolic distance between two points is proportional to the probability that
the corresponding two vertices are adjacent (i.e. that there exists an edge
between them). It has been demonstrated that statistical power law (also known
as Pareto-Zipf law) for the degrees of vertices implies hyperbolic geometries
of complex networks and vice versa [2]. On the other hand, it is well known
that the vast majority of large networks arising in real life (such as
internet, social networks, citations networks, biological networks) obey the
general power law [25]. This indicates that the family (3.2) might have many
applications in statistical network science and in ML over hierarchical data
of various kinds.

Since Möbius transformations are isometries in ${\mathbb{B}}^{2}$, two sets of
points which are related by a Möbius transformation represent the same
network. Therefore, one complex network corresponds to the whole sub-family
${\mathcal{F}}_{a,\alpha}$ for arbitrary $a\in{\mathbb{B}}^{2}$ and fixed
$\alpha$. Mixtures of densities (3.2) can be used for probabilistic modeling
of complex networks, and applied in many problems, such as comparison of the
two networks, quantification of the difference/distance between them, or
matching sub-networks (i.e. finding similar patterns within different
networks).

We are aware of only one study [26] which experimented with probability
distributions on the hyperbolic disc for purposes of ML. The authors proposed
the family which they named wrapped normal distributions on hyperbolic disc.
They used this family for stochastic policies in various ML problems and
architectures, including stochastic policies in RL (problem of playing Atari),
designing hyperbolic variational auto-encoders (hyperbolic VAE’s) and
probabilistic word embeddings. However, we find the model proposed by the
authors inconvenient and difficult to learn over. Another particular
probability distribution (rather than the whole family) on hyperbolic disc is
briefly mentioned in the paper [2] on hyperbolic geometry of complex networks.
This distribution referred to as uniform on hyperbolic disc by the authors is
rotationally symmetric, while distribution of the radius is defined by the
density function which is proportional to $\sinh$ (hyperbolic sine).

Experiments with densities (3.2) in supervised/unsupervised learning, as well
as in RL with hyperbolic space of states (and, possibly, actions) are still to
be conducted.

##  6\. Conclusion

With recent geometric insights in data science, there is an apparent need for
geometry informed ML architectures and models. Inferring symmetries and
inherent curvatures of data appears as one of essential conceptual issues in
this broad field. Recent developments entail a growing significance of
classical mathematical fields of Riemannian geometry and Lie group theory for
ML. This further implies the need for statistical models on Riemannian
manifolds for encoding uncertainties in curved data.

Statistical models of data with positive curvature are provided by directional
statistics and fairly well understood (although still not properly recognized
and exploited by the ML research community).

On the other hand, data sets with inherently negative curvature seem to be
even more ubiquitous in science, engineering and everyday life. The present
study addresses the necessity for statistical models on hyperbolic manifolds.
As the first step towards this goal, we propose the novel family of
probability distributions on hyperbolic discs. This family is invariant with
respect to actions of the group of hyperbolic isometries, which entails many
favorable properties. Learning over group-invariant families boils down to
learning orbits and infinitesimal generators of the corresponding group, thus
facilitating implementation of stochastic policies and improving transparency
of algorithms.

In practice, large hierarchical data sets are embedded into higher-dimensional
manifolds such as hyperbolic balls or hyperbolic multidiscs. Therefore,
extensions of the model proposed here to higher dimensions could potentially
be very significant for Geometric DL.

Notice that the wrapped Cauchy family has been extended to higher-dimensional
spheres [27, 28], thus providing a suitable statistical model (along with the
von Mises-Fisher family) for spherical data. However, balls in even-
dimensional (with the dimension greater than two) spaces can be equipped with
two different kinds of hyperbolic metrics. Accordingly, there are two groups
of hyperbolic isometries preserving the spheres [29], thus giving rise to non-
equivalent statistical models. We aim to continue the present study with
investigations of group-invariant families on spheres and hyperbolic balls.

## References

  * [1] L. A. Santaló, Integral Geometry and Geometric Probability, Cambridge University Press, 2004.
  * [2] D. Kriukov, F. Papadopoulos, M. Kitsak, A. Vahdat and M. Boguña, Hyperbolic geometry of complex networks, Phys. Rev. E 82 (2010) 036106
  * [3] A. Tifrea, G. Bécigneul and O.-E. Ganea, Poincaré GloVe: Hyperbolic Word Embeddings, arXiv:1810.06546 (2018).
  * [4] M. Leimeister and B. J. Wilson, Skip-Gram word embeddings in hyperbolic space, arXiv:1809.01498 (2018).
  * [5] N. Ay, J. Jost, H. V. Lê and L. Schwachhöfer, Information Geometry, A Series of Modern Surveys in Mathematics, vol. 64, Springer, 2017.
  * [6] M. M. Bronstein, J. Bruna, T. Cohen and P. Velicković,´ Geometric Deep Learning: Grids, Groups, Graphs, Geodesics and Gauges, arXiv:2014.13478 (2021).
  * [7] K. V. Mardia and P. E. Jupp, Directional Statistics, Wiley Series in Probability and Statistics, 1st Edition, 1999.
  * [8] A. Pewsey and E. García-Portugués, Recent advances in directional statistics, TEST 30 (2021), 1-58, https://doi.org/10.1007/s11749-021-00759-x
  * [9] O. E. Ganea, G. Bécigneul and T. Hofmann, Hyperbolic Neural Networks, in: Advances of Neural Information Processing Systems 31, NeurlPS18 (2018).
  * [10] M. Nickel and D. Kiela, Poincaré Embeddings for Learning Hierarchical Representations, in: Advances of Neural Information Processing Systems 30, NeurlPS17 (2017).
  * [11] M. T. Law, R. Liao, J. Snell and R. S. Zemel, Lorentzian Distance Learning for Hyperbolic Representations, in: International Conference on Machine Learning (2019), 3672-3681.
  * [12] P. McCullagh, Möbius transformation and Cauchy parameter estimation, Annals of Statistics, vol. 24 (1996), no. 2, 787-808.
  * [13] P. McCullagh, Conditional Inference and Cauchy Models, Biometrika, vol. 79 (1992), no. 2, 247-259.
  * [14] A. Aleksandrov and A. Gorsky, Information geometry and synchronization phase transition in the Kuramoto model, Phys. Rev. E, 107 (2023) 044211.
  * [15] V. Jaćimović, Swarms on manifolds and Kuramoto oscillators for geometry informed machine learning, arXiv:2405.09453 (2024).
  * [16] S. Kato and M. C. Jones, A family of distributions on the circle with links to, and applications arising from, Möbius transformation, J. Am. Stat. Assoc. 105 (489) (2010), 249-262.
  * [17] M. Pavlović, Function Classes on the Unit Disc. De Gruyter Studies in Mathematics, Berlin/Boston, 2019
  * [18] A. Douady and C. J. Earle, Conformally natural extensions of homeomorphisms on the circle, Acta Math. 157 (1986), 23-47, DOI: 10.1007/BF02392590
  * [19] M. Arnaudon, F. Barbaresco, L. Yang (2013) Medians and Means in Riemannian Geometry: Existence, Uniqueness and Computation, In: Nielsen, F., Bhatia, R. (eds) Matrix Information Geometry. Springer, Berlin, Heidelberg.
  * [20] C. Liu, L. Peng, Boundary Regularity in the Dirichlet Problem for the Invariant Laplacians $\Delta_{\gamma}$ on the Unit Real Ball, Proc. Amer. Math. Soc. 132 (2004), 3259–3268
  * [21] E. De Michelli, I. Scorza and G. A. Viano, Hyperbolic geometrical optics: Hyperbolic glass, J. Math. Phys. 47 (2006) 023503, https://doi.org/10.1063/1.2165796
  * [22] A. M. Perelomov, Coherent States and Symmetric Spaces, Commun. Math. Phys. 44 (1975) 197-210.
  * [23] A. Wünsche, A class of phase-like states, J. Opt. B: Quantum Semiclass. Opt. 3 206 (2001), no. 4 DOI 10.1088/1464-4266/3/4/303
  * [24] A. Vourdas, Analytic representations in quantum mechanics, J. Phys. A: Math. Gen. 39 R65 (2006) DOI 10.1088/0305-4470/39/7/R01
  * [25] M. Newman Power laws, Pareto distributions and Zipf’s law, Contemp. Phys, 46 (2005), no. 5, 323-351, https://doi.org/10.1080/00107510500052444
  * [26] Y. Nagano, S. Yamaguchi, Y. Fujita and M. Koyama, A Wrapped Normal Distribution on Hyperbolic Space for Gradient-Based Learning, Proceedings of the 36th International Conference on Machine Learning, PMLR 97 (2019) 4693-4702.
  * [27] J.-L. Dunau and H. Sénateur, Une caracterisation du type de la loi de Cauchy-conforme sur ${\mathbb{R}}^{n}$, Probab. Th. Rel. Fields, 77 (1988) 129-135.
  * [28] S. Kato and P. McCullagh, Some properties of a Cauchy family on the sphere derived from the Möbius transformations, Bernoulli 26 (2020), no. 4, 3224-3248, DOI: 10.3150/20-BEJ1222
  * [29] M. Stoll, Harmonic and Subharmonic Function Theory on the Hyperbolic Ball. London Mathematical Society Lecture Notes Series Number 341, 1st Edition, Cambridge University Press, 2016.


# GANetic Loss for Generative Adversarial Networks with a Focus on Medical
Applications

Shakhnaz Akhmedova
Centre for Artificial Intelligence in
Public Health Research
Robert Koch Institute
Berlin, Germany
<EMAIL_ADDRESS>
&Nils Körber
Centre for Artificial Intelligence in
Public Health Research
Robert Koch Institute
Berlin, Germany
<EMAIL_ADDRESS>

###### Abstract

Generative adversarial networks (GANs) are machine learning models that are
used to estimate the underlying statistical structure of a given dataset and
as a result can be used for a variety of tasks such as image generation or
anomaly detection. Despite their initial simplicity, designing an effective
loss function for training GANs remains challenging, and various loss
functions have been proposed aiming to improve the performance and stability
of the generative models. In this study, loss function design for GANs is
presented as an optimization problem solved using the genetic programming (GP)
approach. Initial experiments were carried out using small Deep Convolutional
GAN (DCGAN) model and the MNIST dataset, in order to search experimentally for
an improved loss function. The functions found were evaluated on CIFAR10, with
the best function, named GANetic loss, showing exceptionally better
performance and stability compared to the losses commonly used for GAN
training. To further evalute its general applicability on more challenging
problems, GANetic loss was applied for two medical applications: image
generation and anomaly detection. Experiments were performed with
histopathological, gastrointestinal or glaucoma images to evaluate the GANetic
loss in medical image generation, resulting in improved image quality compared
to the baseline models. The GANetic Loss used for polyp and glaucoma images
showed a strong improvement in the detection of anomalies. In summary, the
GANetic loss function was evaluated on multiple datasets and applications
where it consistently outperforms alternative loss functions. Moreover,
GANetic loss leads to stable training and reproducible results, a known weak
spot of GANs. ††Code: https://github.com/ZKI-PH-ImageAnalysis/GANetic-Loss.

##  1 Introduction

Figure 1: Implemented GP search process. In the first row an example of the
generated trees is shown, in this example $x$ and $y$ are $y_{real}$ and
$y_{pred}$.

Generative Adversarial Networks (GANs) were first introduced in Goodfellow et
al. (2014) as a new framework for estimating generative models via an
adversarial process. Specifically, the GAN model is designed as two competing
neural networks that are trained simultaneously, with the first network
referred to as generator $G$ and the second as discriminator $D$. The goal of
the generator is to capture the data distribution in such a way that it is
able to create data patterns that are as similar as possible to those in the
training data. On the other hand, the discriminator tries to estimate the
probability that a sample comes from the training data and was not generated
by $G$.

GANs have been utilized in the generation of images Bousmalis et al. (2016);
Ledig et al. (2016), video Tulyakov et al. (2017); Walker et al. (2017), audio
Lee et al. (2017); Guimaraes et al. (2017) and text Guo et al. (2017); de
Masson d’Autume et al. (2019). Moreover, GANs can be applied for different
problems in the machine learning field such as anomaly detection Xia et al.
(2022), semi-supervised learning Dai et al. (2017); Gao et al. (2021),
representation learning Yuan & Peng (2020); Tran et al. (2017), which attempts
to address the lack of labeled data and learn representations of the data to
facilitate the extraction of useful information related to the task.

Originally it was proposed to train GANs by using the adversarial loss
function Goodfellow et al. (2014). However, several studies have shown that
these GAN models are prone to mode collapse, show unstable results as they
fail to converge in some cases, and consequently do not produce realistic
images Che et al. (2016); Richardson & Weiss (2018); Metz et al. (2016).

There are various variants of GAN models, which differ from each other either
by the layers, overall structure of the generator and/or discriminator
networks, or by the incorporated loss function used for training. However, the
training process of GANs is very complex and often unstable, so an effective
loss function can significantly improve the efficiency of GAN models. Thus,
the focus of this study is to find a loss function that is generally good
regardless of the specific GAN architecture and dataset.

The loss function search can be represented as an optimization problem, that
can be solved by using Genetic Programming (GP), which is a well-known
population-based evolutionary algorithm Augusto & Barbosa (2000). The GP
approach is frequently used to design previously unknown heuristics in order
to solve complex computational search problems K.Burke et al. (2009). GP was
applied to determine the loss function for a relatively simple Deep
Convolutional GAN model (DCGAN) Radford et al. (2016) using MNIST dataset
LeCun et al. (2010). The GP experiments resulted in several loss function
candidates potentially suitable for GAN training.

All loss functions found were used to train a slightly larger DCGAN model with
the CIFAR10 dataset and were compared to commonly used loss functions. The
best function found, GANetic loss, was used for medical applications, namely
image generation and anomaly detection, using state-of-the-art models. A
comparison with other commonly used loss functions for GANs showed the
efficiency of the GANetic loss, as it was able to outperform known
representative baselines in performance and stability.

Thus, the main contributions in this paper can be summarized as follows:

  * •

It is proposed to use the GP approach to design the loss function for GANs.
The loss function found, GANetic loss, showed more stable training and better
results compared to commonly used loss functions.

  * •

The GANetic loss was evaluated by using two different GAN models for medical
imaging tasks such as image generation and anomaly detection. Experimental
results show that GANetic loss is superior to other loss functions resulting
in better image generation and anomaly detection.

  * •

Furthermore, the comparison with different known losses commonly used for
training GANs is shown to prove that the new loss function is able to avoid
mode collapses, leading to reproducible results.

  * •

Self-regularization naturally implemented in the GANetic loss indicates a
direction for loss function design in general.

##  2 Related Work

Generative adversarial networks have gained interest and popularity among
researchers due to the relative simplicity of the idea of two competing
networks, where one tries to trick another, and their capability to represent
complex and high-dimensional data. The GAN architecture, originally proposed
in Goodfellow et al. (2014), consists of two neural networks, generator $G$
and discriminator $D$, competing adversarially against each other. The
architecture of these networks can have fully-connected Barua et al. (2019),
convolutional Radford et al. (2016) or recurrent layers Esteban et al. (2017)
depending on the data domain and application. The generator $G$ is trained to
produce samples which capture the underlying distribution of the training
data, while the discriminator $D$ is fed a sample of mixed data in order to
estimate the probability of whether the sample is real, i.e. originates from
the training dataset, or fake, i.e. was generated by $G$. Thus, GANs minimize
the Jensen-Shannon divergence between a target data distribution and the model
distribution given by the generator Goodfellow et al. (2014):

$\min_{G}\max_{D}V(D,G)=\mathbb{E}_{x\sim
P_{data}(x)}[log(D(x)]+\mathbb{E}_{z\sim P_{z}(z)}[1-log(D(G(z))].$ (1)

The following notations are used in Equation 1: $G(z)$ is the data generated
by the generator model, $P_{data}(x)$ is the probability distribution of the
real samples presented in the training dataset, $P_{z}(z)$ is the probability
distribution of input noise variables, $D(x)$ and $D(G(z))$ are the
probabilities that $x$ and $G(z)$ are from the training dataset. It should be
noted that one of the most commonly used implementations of GANs consists in
using the binary cross entropy loss function for discriminator $D$ to update
both generator and discriminator after each training step. However, in Mao et
al. (2017) authors showed that the sigmoid cross entropy loss function leads
to the problem of vanishing gradients, which can cause the model to fail to
converge and give poor results. Therefore, it is crucial to properly design
the loss function for a chosen GAN model to ensure its stability and
efficiency.

In recent years various GAN models have been proposed, which differ from each
other either by the architecture of the implemented neural networks Liu et al.
(2021); Sauer et al. (2021), by applying augmentations Karras et al. (2020) or
by the designed loss functions used during the training Arjovsky et al.
(2017); Mao et al. (2017). The most well-known studies among them, related to
the GAN loss function design, are about the Wasserstein loss, that was
introduced Arjovsky et al. (2017) for WGANs, and the least squares loss
function, which was implemented for LSGANs Mao et al. (2017). Another
alternative, commonly used to train GAN models, is the hinge loss function
Mustafa et al. (2020). However, the choice of loss function usually depends on
the experience of the researchers, the model, and the specific problem.

##  3 Genetic Programming for Loss Function Design

Genetic Programming (GP) is a population-based evolutionary algorithm, which
can be applied to design different heuristics, depending on the problem at
hand, and, additionally, it is considered as a machine learning tool as it can
be used to discover a functional relationship between features in data. GP was
initially inspired by the biological evolution, including natural processes
such as mutation and selection. In GP, each solution is represented by a tree
structure, and each of these trees are evaluated recursively to produce the
resulting multivariate expression. There are two types of nodes used for tree-
based GP: the terminal node, also called leaf, which is randomly chosen from
the set of variables, and the tree node, which can be chosen from a predefined
set of operators.

The GP search process starts with random initialization of a set of potential
solutions, which is also called a population of individuals, in the functional
search space. The number of individuals is the predefined parameter $n$, which
does not change through the whole search process. The overall GP search
process used here is summarized in Figure 1. Thus, for this study a set of
loss functions was randomly generated, and each loss function was represented
as a tree, where the terminals were randomly chosen from the set
$\\{y_{pred},y_{real},\mathbb{R}\\}$, while the operators were chosen from the
set $\\{+,-,\times,\div,\sqrt{,}\log,\exp,\sin,\cos\\}$. To utilize these
functions and terminals for loss function generation several modifications,
described in Appendix A, were applied.

After initialization, the main search loop starts, which consists of the
iteratively repeated steps crossover, mutation, fitness function evaluation
and selection (the number of GP steps or generations is denoted as $T$). The
fitness function $F$ determines the quality of the individual and is the
crucial part of the search process, as our goal was to experimentally find a
robust loss function that can be generalized to a variety of datasets.
Specifically, each individual loss function was used to train both generator
and discriminator of a simple DCGAN model $5$ times on MNIST dataset LeCun et
al. (2010), and the mean FID value and standard deviation calculated over the
$5$ program runs was used as the fitness value.

The crossover operator is used to exchange the subtrees between two
individuals (Figure 1). Mutation can be applied in various ways, but in this
study two variants were used: a random subtree in the tree is chosen and
replaced with another randomly generated subtree; a random node in the tree is
chosen and replaced with another randomly generated node.

Both mutation operators as well as crossover have their own parameters, such
as mutation $M_{ST}$, $M_{N}$ and crossover $Cr$ rates, which determine how
often individuals will be changed during the main search loop. Finally, all
individuals and the ones generated after crossover and mutation steps are
combined into one population, and then the selection operator is applied.
Here, two selection schemes were considered: the tournament selection with
$k_{t}$ as tournament size and selection of the most fit individuals from the
combined population.

Moreover, the success history-based adaptation strategy was used to improve
the efficiency of the GP approach. To be more specific, at the beginning of
the main search loop an empty external archive $A$ was created. The maximum
size of this archive was set equal to the population size, $n_{A}=n$. All
individuals not passing selection could have been saved in the archive with a
given probability $p_{A}$. Individuals saved in the external set were used
during crossover step with some probability $Cr_{A}$.

In summary, loss function search was performed by the GP algorithm and
essentially can be described as an optimization process. A set of individuals,
where each individual is a mathematical formula representing a loss function,
is generated. These individuals are modified by crossover and mutation, and
the most fit individuals are selected for the next generation. These actions
are repeated a certain number of times in order to find the best loss
function.

##  4 Experiments

###  4.1 Genetic Programming

As mentioned, a small GAN model was trained on the MNIST dataset for each
individual during GP. Each GAN (its generator and discriminator) was trained 5
times with its corresponding generated loss function to calculate the mean and
deviation of the FID value. The discriminator of this GAN model consists of
two convolutional layers with LeakyReLU activations and no pooling layers, the
last convolutional layer was flattened and connected with a dropout Srivastava
et al. (2014) of $0.4$ to a dense layer comprising a single output node used
to discriminate between real and fake images.

The architecture of the generator was slightly more complex. A sample of
random numbers from the standard normal distribution were given as input to
the generator. These values were transformed by reshaping a fully-connected
layer, followed by a convolutional layer with LeakyReLU activation into
matrices of the size of the training images. Finally, the generated 2D
matrices were used as input to an autoencoder whose encoder had the same
layers as the discriminator but without flattening or dropout, and a decoder
built accordingly in reverse order (a transposed convolutional layer for each
convolutional layer of the encoder with a final hyperbolic tangent output
activation). The included autoencoder was chosen since most anomaly detection
models use similar architecture which was one of the tasks considered in the
study.

Genetic programming algorithm has several parameters, which can affect speed
and quality of the search process. The following parameters were the same for
all GP experiments: population size $n=10$, number of generations $T=50$,
crossover rate $Cr=0.7$, the minimum tree height was set to $2$, while the
maximum tree size was $100$. To create leaf nodes real valued terminals were
sampled from a uniform distribution in the range $[-5,5]$. In total, we tested
$8$ different configurations of the GP algorithm (see Table B for details),
resulting in $8$ individual experiments, each yielding a sub-optimal solution
for the search process. The equations for found loss functions are shown in
Equations C1-C8 (Appendix C), where $N=2$ is the number of classes.

In order to limit the total GAN training time of the GP search, a batch size
of $512$ and $70$ epochs of training were chosen. Each GAN was trained using
the Adam optimizer Kingma & Ba (2015) for both networks with learning rate and
$\beta_{1}$ of $0.0002$ and $0.5$, respectively, and the latent variable
dimension of $100$. To further reduce the evaluation, the FID value was
calculated by generating $256$ images from the trained model and randomly
selecting the same number from the data set. Each GP experiment took about
$100$ hours to complete on an internal server node equipped with $8$ Nvidia
A100. The results obtained for the MNIST data set using the best loss
functions found in the experiments compared to the binary cross entropy (BCE)
function are shown in Table 1.

Table 1: Numerical results from MNIST dataset. Mean FID values, standard deviation (STD) and mean discriminator accuracy ($D\textsubscript{Acc}$) over 5 runs. Loss | BCE | $f_{1}$ | $f_{2}$ | $f_{3}$ | $f_{4}$ | $f_{5}$ | $f_{6}$ | $f_{7}$ | $f_{8}$
---|---|---|---|---|---|---|---|---|---
Mean | 11.59 | 9.86 | 15.00 | 6.54 | 5.96 | 7.17 | 6.18 | 7.44 | 9.70
STD | 7.77 | 2.29 | 8.55 | 1.25 | 1.86 | 3.68 | 2.56 | 2.90 | 1.43
$D\textsubscript{Acc}$ | 98.1 | 2.8 | 4.8 | 17.4 | 94.1 | 49.9 | 96.4 | 49.8 | 46.6

To evaluate the functions found by GP, not only the quality of the generated
images was taken into account, but also the accuracy of the discriminator and
the variation of individual training runs (explanation for that choice is
given in Appendix D). Quality of generated images, measured by FID, compared
to the BCE loss, could be improved by all but one found functions. To find a
function resulting in stable training, we used the standard deviation for
evaluation during GP to take reproducibility into account. As a result the
functions found show, except for one, better stability compared to BCE loss.

From the $8$ functions resulting from GP, $f_{4}$ and $f_{6}$ showed the best
results, both found using the success history-based adaptation strategy.
However, the function $f_{4}$ showed better results than $f_{6}$ in terms of
FID value and standard deviation. Consequently, only $f_{4}$, denoted as
GANetic loss, was further evaluated and compared to popular loss functions
commonly used to train GANs. GANetic loss is defined by the following formula:

$GANetic=\frac{1}{N}\sum_{i=1}^{N}{\left(\left(y_{pred}^{(i)}\right)^{3}+\sqrt{\left|\alpha\times\frac{y_{real}^{(i)}}{y_{pred}^{(i)}+\varepsilon}\right|+\varepsilon}\right)},$
(2)

where $\alpha=3.985$.

###  4.2 Evaluation of the GANetic Loss

The first set of experiments was conducted to evaluate the performance of the
GANetic loss function and to compare it to other well-known loss functions
commonly used to train GANs. For these experiments we used a slightly bigger
GAN and trained it on the CIFAR10 dataset Krizhevsky et al. (2009).
Specifically, the model from the previous section was extended to include two
additional convolutional layers for the discriminator and the encoder of the
generator, and consequently two additional transposed convolutional layers for
the decoder.

Each GAN was trained with the batch size of $128$ for $200$ epochs. The
results of $25$ independent program runs were compared to the identical model
using binary cross entropy, hinge, adversarial, Wasserstein, and least square
loss functions. Corresponding FID values (the best, the worst, the mean) and
their standard deviation are presented in Table 2. Examples of the generated
images are demonstrated in Figure E (Appendix E)

Table 2: FID values on CIFAR10 dataset. Loss | Adversarial | BCE | Hinge | Wasserstein | LeastSquare | GANetic
---|---|---|---|---|---|---
Best | 9.45 | 2.06 | 7.11 | 3.04 | 4.60 | 1.52
Worst | 72.58 | 45.30 | 62.47 | 31.74 | 80.87 | 2.76
Mean | 18.52 | 8.34 | 12.41 | 12.93 | 17.52 | 2.20
STD | 16.12 | 8.92 | 10.59 | 4.85 | 15.81 | 0.39

From the tested functions GANetic loss showed the most consistent results
outperforming all other functions by a large margin. Figure 2 shows the
variation of FID values for the tested loss functions over $25$ runs. GANetic
loss leads to very reproducible results avoiding mode collapses, which can be
observed for all other functions during the $25$ trials.

Figure 2: Variation of results on CIFAR10 for different losses over 25 runs.

Finally, two additional configurations of the GANetic loss were tested in an
ablation study. For that purpose the same DCGAN model and CIFAR10 dataset were
used. We tested configurations in which the generator, the discriminator or
both replaced the BCE function with GANetic loss during training. Obtained
results are demonstrated in Table E.

While most of the performance gains were achieved by using GANetic loss for
the discriminator, the overall best performance was achieved when the
generator and discriminator were trained with GANetic loss. Results from Table
E show that DCGAN with both networks, generator and discriminator, trained by
using GANetic loss achieves better results in terms of the FID metric value.
Unless stated otherwise, all other experiments were conducted using GANetic
loss for generator and discriminator.

###  4.3 Medical Applications

####  4.3.1 Image Generation

To test the performance of GANetic loss in a more complex scenario, we used
the state-of-the-art projected GAN (prGAN) Sauer et al. (2021) and
StyleGAN2-ADA Karras et al. (2020) architectures and three medical datasets,
namely BrecaHAD Aksac et al. (2019), Hyper-Kvasir Borgli et al. (2020) and LAG
Li et al. (2019). The GANetic loss was applied to train the generator and
discriminator networks implemented for projected GANs (prGANs) and only
generator network for StyleGAN2-ADA model.

The BreCaHAD dataset consist of only 162 breast cancer histopathology images
with an image size of $1360\times 1024$ that were reorganized into 1944
partially overlapping crops of size $512\times 512$, as implemented in Karras
et al. (2020). The Hyper-Kvasir dataset contains 110,079 gastrointestinal
images and 374 videos where it captures anatomical landmarks and pathological
and normal findings. Following Tian et al. (2021), we take 2100 images from
cecum, ileum, bbps-2–3 and 1000 images from polyp using the same subset as
used for anomly detection as shown later. The images were combined and
reorganized into 10,417 partially overlapping crops, that were resized to
$256\times 256$. The LAG dataset used for glaucoma detection, contains 4854
images with 1711 positive glaucoma and 3143 negative glaucoma scans. For image
generation experiments, the images from LAG datasets were center cropped and
resized to $256\times 256$.

The evaluation protocol of Sauer et al. (2021) (including FastGAN generator)
was followed to allow fair comparisons for prGANs. Therefore, the prGANs
trained with all the loss functions were given the same fixed number of frames
on all datasets, namely $10^{7}$. It was found that for the BreCaHAD dataset
regardless of the loss function, the prGAN model began to overfit after
$3\times 10^{6}$ or $4\times 10^{6}$ images and consequently showed poor
results, so experiments were repeated with the same parameters but given only
$4\times 10^{6}$ images. For the Hyper-Kvasir and LAG datasets experiments
were carried out with $10^{7}$ images. For StyleGAN2-ADA the number of frames
was set to $10\times 10^{7}$ for all datasets.

Figure 3: Examples of images generated by prGANs using different losses for BreCaHAD (left), Hyper-Kvasir (middle) and LAG (right) datasets. From top to bottom: ground truth images, original loss and GANetic loss. Table 3: FID and KID values for different losses trained on BreCaHAD, Hyper-Kvasir and LAG datasets. | BreCaHAD | Kvasir | LAG
---|---|---|---
Model (Loss) | FID | KID | FID | KID | FID | KID
prGAN (original) | 15.50 | 3.49 | 4.01 | 1.77 | 3.73 | 0.93
prGAN (GANetic) | 14.57 | 3.38 | 3.81 | 1.16 | 3.53 | 0.90
StyleGAN2-ADA (original) | 15.77 | 2.88 | 5.55 | 2.19 | 6.38 | 3.00
StyleGAN2-ADA (GANetic) | 16.2 | 2.85 | 5.80 | 2.07 | 6.09 | 2.79

The obtained results, FID and KID values between $50k$ generated and all real
images, are presented in Table 3. The results show that GANetic loss leads to
an improved performance for prGANs on all datasets and for StyleGAN2-ADA
improves at least one metric. In addition, it should be noted that prGANs
trained with GANetic loss showed the fastest convergence after approximately
$1.8\times 10^{6}$ iterations, while the original configuration required
$3.4\times 10^{6}$, and $2.8\times 10^{6}$, respectively, which is
demonstrated in Figure F (see Appendix F). The convergence was similar on the
Hyper-Kvasir and LAG datasets regardless of the loss function. Examples of the
generated images for both models and all the mentioned datasets are
demonstrated in Figures 3 and G (Appendix G). Additional evaluation using the
same setup on the AFHQ-Cats,AFHQ-Dogs and AFHQ-Wild datasets Choi et al.
(2020) can be found in Appendix 7.

####  4.3.2 Anomaly Detection

The detection of imaging biomarkers correlating with disease status is
important for initial diagnosis, assessment of treatment response and follow-
up examinations. However, training deep learning models for the identification
of imaging biomarkers is diffult as it requires expert annotated data. Even
with annotated training data available, supervised learning is limited to
already known markers. Thus, in Schlegl et al. (2019) authors proposed a GAN
based unsupervised learning approach, called f-AnoGAN, to identify anomalous
images and image segments, that can serve as imaging biomarker candidates.

In Schlegl et al. (2019) authors described an anomaly detection framework,
which consists of two training steps on normal ”healthy” images: GAN training
and encoder training, where the latter is based on the trained GAN model.
After training, inference yields an anomaly score for a new image utilizing
these trained components.

Experiments were carried out on two datasets: Hyper-Kvasir and LAG Li et al.
(2019). Following Tian et al. (2021), we took 2100 images from cecum, ileum
and bbps-2–3 cases as normal and 1000 abnormal images from polyp as abnormal.
We used 1600 images for the training set and 500 images for the test set
similarly to what was done in Rafiee et al. (2023). For the LAG dataset 2343
normal (negative glaucoma) images were used for training and 800 normal images
and 1711 abnormal (positive glaucoma) images were used for testing the trained
model.

All images were resized to $64\times 64$, and augmentation included horizontal
flip and normalization. The f-AnoGAN approach uses an improved WGAN
architecture, namely, it is based on the WGAN model with gradient penalty
(WGAN-GP) Gulrajani et al. (2017). In this study, the number of epochs for
both WGAN-GP and encoder was set to 1000.

Table 4: AUROC results for Hyper-Kvasir and LAG datasets. Model | Kvasir | LAG
---|---|---
DAE Masci et al. (2011) | 0.705 | 0.651
OCGAN Perera et al. (2019) | 0.813 | 0.737
f-anoGAN Schlegl et al. (2019) | 0.907 | 0.778
ADGAN Liu et al. (2019) | 0.913 | 0.752
CAVGA-Ru Venkataramanan et al. (2019) | 0.928 | 0.819
IGD Chen et al. (2021) | 0.937 | 0.857
CCD Tian et al. (2021) | 0.972 | 0.874
f-anoGAN (GANetic) | 0.987 | 0.782

Experimental results for both datasets of the f-AnoGAN model, which generator
and discriminator (but not encoder) were trained using GANetic loss, are
demonstrated in Table 4. Overall, applying GANetic loss to f-AnoGAN improved
model’s performance for both datasets. While it only leads to a mediocre
improvement for the LAG dataset, it results in a massive performance boost
when applied to Hyper-Kvasir dataset outperforming all other methods.

Results for the remaining configurations of the trained f-AnoGAN model are
shown in Table I (Appendix I). Additionally, the discrete distribution of
anomaly scores and the AUROC curve for Hyper-Kvasir datasets are shown in
Figure I (Appendix I)).

##  5 Discussion

The training of GANs can be viewed as a binary classification problem, where
the generator is optimized by making the discriminator predict true labels and
the discriminator distinguishes between true and false images. Thus, the
GANetic loss function in equation 2 can be simplified as:

$GANetic=\left(y_{pred}^{(0)}\right)^{3}+\left(y_{pred}^{(1)}\right)^{3}+\sqrt{\left|\frac{\alpha}{y_{pred}^{(1)}+\varepsilon}\right|+\varepsilon},$
(3)

where $\alpha=3.985$, $\varepsilon=10^{-8}$ and $y_{pred}^{\left(i\right)}$,
$i\in\\{0,1\\}$, denotes the model output for $y_{real}\in\\{0,1\\}$.

It should be noted that $y_{pred}$ was in the range $[0,1]$, being passed
through a sigmoid function. Several conclusions can be drawn from the
experimental derived equation. First, predictions for $y_{real}=0$ are treated
differently compared to $y_{real}=1$. The former showing a cubic,
monotonically increasing function, while the latter exhibits a minimum near
$0.75$. Morevover, discriminator’s penalty for misclassifying the fake images
is limited by $1$, while it is much larger for misclassifying real images.
This could indicate that it is advantageous to focus on the real images during
training and could explain the faster convergence of GANetic loss, as the
generator normally requires a considerable number of images to produce
reasonably realistic images.

Figure 4: GANetic and BCE loss functions (left) and their gradients (right).

The case where $y_{real}=1$ is plotted in Figure 4 for the BCE and GANetic
losses. The cross-entropy loss is shown to be monotonically decreasing, while
the GANetic loss function shows an increase in the loss value as $y_{pred}$
approaches $y_{true}$. This increase of the loss value allows the GANetic loss
function to prevent the model from becoming too confident in its output
predictions and may provide an important advantage as it lowers the
probability of overfitting. Thus, this loss function provides an implicit form
of regularization, enabling better generalization.

An implicit form of regularization naturally implemented in the GANetic loss
as well as a prioritization of correctly classifying real images point
directions of future work, as they can be used as prior knowledge introduced
into GP or any other search paradigm to reduce the evaluation overhead. More
comments can be found in Appendix J.

##  6 Conclusions

In this study Genetic Programming was utilized for GAN loss function search.
The final function was evaluated by using various GAN models and datasets. The
obtained results of GANetic loss were compared to the ones achieved by the
same models but trained by using other well-known loss functions such as
binary cross entropy, hinge loss and Wasserstein loss. The GANetic loss was
shown to convincingly outperform existing commonly used loss functions and to
be efficient for network training regardless of model architecture or dataset.
Moreover, it is applicable for medical problems such as image generation and
anomaly detection.

However, while functions found by GP were achieved with limited populations
size in this study, it is clear that experiments with GP with additional
computing resources could lead to even better performing functions. In
addition, separating generator and discriminator loss during the search
process increases the search space and could lead to an even better, more
specialized combined loss. Further analysis suggested that the improvements
provided by the new loss result from implicit regularization that reduces
overfitting to the data.

##  7 Broader Impact

Generative models learn the underlying representation of the training data,
addressing a fundamental machine learning problem. Especially, in the medical
sector those models pose a huge potential, for example allowing identification
of novel disease phenotypes. In this study, we present an experimentally found
loss function that generally improves the performance of GANs and additionally
stabilizes the training procedure. Furthermore, we evaluated the loss for
medical applications improving the performance of existing models.

Despite the potential of generative models, they also pose a significant risk,
particularly the generation of fake information. Moreover, there are risks in
the medical field not only from misuse, but also from improper use, which can
impair patient care. Nevertheless, we are convinced that the benefits outweigh
the risks and may improve patient healthcare in the long run. Development of
robust models and the detection or watermarking of fake images requires a
community effort to further shift the balance towards the positive side. Our
work does not provide fundamentally new capabilities, but rather improves the
performance of existing models leading to more reproducible results using GAN-
based networks. Our work may hopefully contribute in future work learning the
representation of complex real-world phenomena.

## References

  * Aksac et al. (2019) Aksac, A., Demetrick, D. J., Özyer, T., and Alhajj, R.  Brecahad: A dataset for breast cancer histopathological annotation and diagnosis.  1 2019.  doi: 10.6084/m9.figshare.7379186.v3.  URL https://figshare.com/articles/dataset/BreCaHAD_A_Dataset_for_Breast_Cancer_Histopathological_Annotation_and_Diagnosis/7379186.
  * Arjovsky et al. (2017) Arjovsky, M., Chintala, S., and Bottou, L.  Wasserstein generative adversarial networks.  In Precup, D. and Teh, Y. W. (eds.), _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 214–223. PMLR, 06–11 Aug 2017.
  * Augusto & Barbosa (2000) Augusto, D. and Barbosa, H.  Symbolic regression via genetic programming.  In _Proceedings. Vol.1. Sixth Brazilian Symposium on Neural Networks_ , pp. 173–178, 2000.
  * Barua et al. (2019) Barua, S., Erfani, S. M., and Bailey, J.  FCC-GAN: A fully connected and convolutional net architecture for gans.  _CoRR_ , abs/1905.02417, 2019.  URL http://arxiv.org/abs/1905.02417.
  * Borgli et al. (2020) Borgli, H., Thambawita, V., Smedsrud, P. H., Hicks, S., Jha, D., Eskeland, S. L., Randel, K. R., Pogorelov, K., Lux, M., Nguyen, D. T. D., Johansen, D., Griwodz, C., Stensland, H. K., Garcia-Ceja, E., Schmidt, P. T., Hammer, H. L., Riegler, M. A., Halvorsen, P., and de Lange, T.  HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy.  _Scientific Data_ , 7(1):283, 2020.  ISSN 2052-4463.  doi: 10.1038/s41597-020-00622-y.  URL https://doi.org/10.1038/s41597-020-00622-y.
  * Bousmalis et al. (2016) Bousmalis, K., Silberman, N., Dohan, D., Erhan, D., and Krishnan, D.  Unsupervised pixel-level domain adaptation with generative adversarial networks.  _CoRR_ , abs/1612.05424, 2016.  URL http://arxiv.org/abs/1612.05424.
  * Che et al. (2016) Che, T., Li, Y., Jacob, A. P., Bengio, Y., and Li, W.  Mode regularized generative adversarial networks.  _CoRR_ , abs/1612.02136, 2016.  URL http://arxiv.org/abs/1612.02136.
  * Chen et al. (2021) Chen, Y., Tian, Y., Pang, G., and Carneiro, G.  Unsupervised anomaly detection and localisation with multi-scale interpolated gaussian descriptors.  _ArXiv_ , abs/2101.10043, 2021.  URL https://api.semanticscholar.org/CorpusID:231699267.
  * Choi et al. (2020) Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W.  Stargan v2: Diverse image synthesis for multiple domains.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8188–8197, 2020.
  * Dai et al. (2017) Dai, Z., Yang, Z., Yang, F., Cohen, W. W., and Salakhutdinov, R.  Good semi-supervised learning that requires a bad gan.  _ArXiv_ , abs/1705.09783, 2017.
  * de Masson d’Autume et al. (2019) de Masson d’Autume, C., Rosca, M., Rae, J. W., and Mohamed, S.  Training language gans from scratch.  _CoRR_ , abs/1905.09922, 2019.  URL http://arxiv.org/abs/1905.09922.
  * Esteban et al. (2017) Esteban, C., Hyland, S. L., and Rätsch, G.  Real-valued (medical) time series generation with recurrent conditional gans, 2017.  URL https://arxiv.org/abs/1706.02633.
  * Gao et al. (2021) Gao, Y., Zhai, P., and Mosalam, K. M.  Balanced semisupervised generative adversarial network for damage assessment from low-data imbalanced-class regime.  _Computer-Aided Civil and Infrastructure Engineering_ , 36:1094–1113, 2021.
  * Gonzalez & Miikkulainen (2019) Gonzalez, S. and Miikkulainen, R.  Improved training speed, accuracy, and data utilization through loss function optimization.  _2020 IEEE Congress on Evolutionary Computation (CEC)_ , pp. 1–8, 2019.  URL https://api.semanticscholar.org/CorpusID:167217832.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial nets.  In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K. (eds.), _Advances in Neural Information Processing Systems_ , volume 27, pp. 2672–2680. Curran Associates, Inc., 2014.
  * Guimaraes et al. (2017) Guimaraes, G. L., Sanchez-Lengeling, B., Outeiral, C., Farias, P. L. C., and Aspuru-Guzik, A.  Objective-reinforced generative adversarial networks (organ) for sequence generation models, 2017.  URL https://arxiv.org/abs/1705.10843.
  * Gulrajani et al. (2017) Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C.  Improved training of wasserstein gans.  In _Neural Information Processing Systems_ , 2017.  URL https://api.semanticscholar.org/CorpusID:10894094.
  * Guo et al. (2017) Guo, J., Lu, S., Cai, H., Zhang, W., Yu, Y., and Wang, J.  Long text generation via adversarial training with leaked information.  _CoRR_ , abs/1709.08624, 2017.  URL http://arxiv.org/abs/1709.08624.
  * Karras et al. (2020) Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T.  Training generative adversarial networks with limited data, 2020.  URL https://arxiv.org/abs/2006.06676.
  * K.Burke et al. (2009) K.Burke, E., Hyde, M. R., Kendall, G., Ochoa, G., Ozcan, E., and Woodward, J. R.  _Exploring Hyper-heuristic Methodologies with Genetic Programming_ , pp. 177–201.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2009.
  * Kingma & Ba (2015) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  In Bengio, Y. and LeCun, Y. (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al.  Learning multiple layers of features from tiny images.  2009\.
  * LeCun et al. (2010) LeCun, Y., Cortes, C., and Burges, C.  Mnist handwritten digit database.  _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ , 2, 2010.
  * Ledig et al. (2016) Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A. P., Tejani, A., Totz, J., Wang, Z., and Shi, W.  Photo-realistic single image super-resolution using a generative adversarial network.  _CoRR_ , abs/1609.04802, 2016.  URL http://arxiv.org/abs/1609.04802.
  * Lee et al. (2017) Lee, S., Hwang, U., Min, S., and Yoon, S.  A seqgan for polyphonic music generation.  _CoRR_ , abs/1710.11418, 2017.  URL http://arxiv.org/abs/1710.11418.
  * Li et al. (2019) Li, L., Xu, M., Wang, X., Jiang, L., and Liu, H.  Attention based glaucoma detection: A large-scale database and cnn model.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Liu et al. (2021) Liu, B., Zhu, Y., Song, K., and Elgammal, A.  Towards faster and stabilized {gan} training for high-fidelity few-shot image synthesis.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=1Fqg133qRaI.
  * Liu et al. (2019) Liu, Y., Tian, Y., Maicas, G., Pu, L. Z. C. T., Singh, R., Verjans, J. W., and Carneiro, G.  Photoshopping colonoscopy video frames.  _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_ , pp. 1–5, 2019.  URL https://api.semanticscholar.org/CorpusID:204837911.
  * Mao et al. (2017) Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., and Smolley, S. P.  Least squares generative adversarial networks.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * Masci et al. (2011) Masci, J., Meier, U., Ciresan, D. C., and Schmidhuber, J.  Stacked convolutional auto-encoders for hierarchical feature extraction.  In _International Conference on Artificial Neural Networks_ , 2011\.  URL https://api.semanticscholar.org/CorpusID:12640199.
  * Metz et al. (2016) Metz, L., Poole, B., Pfau, D., and Sohl-Dickstein, J.  Unrolled generative adversarial networks.  _CoRR_ , abs/1611.02163, 2016.  URL http://arxiv.org/abs/1611.02163.
  * Mustafa et al. (2020) Mustafa, A., Pia, N., and Fuchs, G.  Stylemelgan: An efficient high-fidelity adversarial vocoder with temporal adaptive normalization, 2020.  URL https://arxiv.org/abs/2011.01557.
  * Perera et al. (2019) Perera, P., Nallapati, R., and Xiang, B.  Ocgan: One-class novelty detection using gans with constrained latent representations.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2893–2901, 2019.  URL https://api.semanticscholar.org/CorpusID:84186723.
  * Radford et al. (2016) Radford, A., Metz, L., and Chintala, S.  Unsupervised representation learning with deep convolutional generative adversarial networks.  In Bengio, Y. and LeCun, Y. (eds.), _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.  URL http://arxiv.org/abs/1511.06434.
  * Rafiee et al. (2023) Rafiee, N., Gholamipoor, R., and Kollmann, M.  Abnormality detection for medical images using self-supervision and negative samples.  _bioRxiv_ , 2023.  URL https://api.semanticscholar.org/CorpusID:259047480.
  * Richardson & Weiss (2018) Richardson, E. and Weiss, Y.  On gans and gmms.  In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * Sauer et al. (2021) Sauer, A., Chitta, K., Müller, J., and Geiger, A.  Projected gans converge faster.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Schlegl et al. (2019) Schlegl, T., Seeböck, P., Waldstein, S. M., Langs, G., and Schmidt-Erfurth, U. M.  f-anogan: Fast unsupervised anomaly detection with generative adversarial networks.  _Medical Image Analysis_ , 54:30–44, 2019.  URL https://api.semanticscholar.org/CorpusID:73516151.
  * Srivastava et al. (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.
  * Tian et al. (2021) Tian, Y., Pang, G., Liu, F., Chen, Y., Shin, S. H., Verjans, J. W., Singh, R., and Carneiro, G.  Constrained contrastive distribution learning for unsupervised anomaly detection and localisation in medical images.  _ArXiv_ , abs/2103.03423, 2021.  URL https://api.semanticscholar.org/CorpusID:232135090.
  * Tran et al. (2017) Tran, L., Yin, X., and Liu, X.  Disentangled representation learning gan for pose-invariant face recognition.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1283–1292, 2017.
  * Tulyakov et al. (2017) Tulyakov, S., Liu, M., Yang, X., and Kautz, J.  Mocogan: Decomposing motion and content for video generation.  _CoRR_ , abs/1707.04993, 2017.  URL http://arxiv.org/abs/1707.04993.
  * Venkataramanan et al. (2019) Venkataramanan, S., Peng, K.-C., Singh, R. V., and Mahalanobis, A.  Attention guided anomaly localization in images.  In _European Conference on Computer Vision_ , 2019.  URL https://api.semanticscholar.org/CorpusID:215870627.
  * Walker et al. (2017) Walker, J., Marino, K., Gupta, A., and Hebert, M.  The pose knows: Video forecasting by generating pose futures.  _CoRR_ , abs/1705.00053, 2017.  URL http://arxiv.org/abs/1705.00053.
  * Xia et al. (2022) Xia, X., Pan, X., Li, N., He, X., Ma, L., Zhang, X., and Ding, N.  Gan-based anomaly detection: A review.  _Neurocomputing_ , 493:49–535, 2022.  ISSN 0925-2312.  doi: https://doi.org/10.1016/j.neucom.2021.12.093.  URL https://www.sciencedirect.com/science/article/pii/S0925231221019482.
  * Yuan & Peng (2020) Yuan, M. and Peng, Y.  Bridge-gan: Interpretable representation learning for text-to-image synthesis.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 30:4258–4268, 2020.

##  Appendix A Appendix: Genetic Programming

The GP search process starts with random initialization of a set of potential
solutions, which is also called a population of individuals, in the functional
search space. The number of individuals is the predefined parameter $n$, which
does not change through the whole search process. For this study a set of loss
functions was randomly generated, and each loss function was an individual
represented as a tree, where the terminals were randomly chosen from the set
$\\{y_{pred},y_{real},\mathbb{R}\\}$, while the operators were chosen from the
set $\\{+,-,\times,\div,\sqrt{,}\log,\exp,\sin,\cos\\}$. To utilize these
functions and terminals for loss function generation, the following
modifications were applied:

  * •

$x\div y$ was changed to $x\div\left(y+\varepsilon\right)$, with
$\varepsilon=10^{-8}$;

  * •

$\sqrt{x}$ was changed to $\sqrt{\left|x\right|+\varepsilon}$, with
$\varepsilon=10^{-8}$;

  * •

$\log(x)$ was changed to $\log\left(\left|x\right|+\varepsilon\right)$, with
$\varepsilon=10^{-8}$;

  * •

It was implemented as a rule that both $y_{pred}$ and $y_{real}$ should be
present in the formula;

  * •

Regardless of the trained GAN’s architecture, the sigmoid function was applied
on the last layer so that $y_{pred}$ was always in the range $[0,1]$.

##  Appendix B Appendix: Parameter setting for GP

Table B: Parameter setting for GP. | $M_{ST}$ | $M_{N}$ | $p_{A}$ | $Cr_{A}$ | Selection | $k_{t}$
---|---|---|---|---|---|---
1 | $0.3$ | $0$ | $0$ | $0$ | tournament | $3$
2 | $0.3$ | $0$ | $0.5$ | $0.5$ | tournament | $3$
3 | $0.3$ | $0$ | $0$ | $0$ | select $n$ best | $-$
4 | $0.3$ | $0$ | $0.5$ | $0.5$ | select $n$ best | $-$
5 | $0.2$ | $0.1$ | $0$ | $0$ | tournament | $3$
6 | $0.2$ | $0.1$ | $0.5$ | $0.5$ | tournament | $3$
7 | $0.2$ | $0.1$ | $0$ | $0$ | select $n$ best | $-$
8 | $0.2$ | $0.1$ | $0.5$ | $0.5$ | select $n$ best | $-$

The population size, $n$, is one of the most important hyperparameters. The
population size, which was set at 10, determines the number of individuals in
each generation, which evolve from generation to generation. The more
individuals, the bigger is the pool of the potential solutions. However, GP is
highly efficient even with the small populations, as the number of
generations, $T$, which is another important hyper-parameter, highly affects
the final results, as it provides more time for the initially randomly
generated individuals to evolve to better fit the objective function. So
although we set the population size to $10$, this was still sufficient for GP,
as the other more important parameters were generally set very carefully.
Additionaly, the computational cost scales with the number of individuals, so
our choice was determined by the attempt to limit the computational workload
of the conducted experiments.

##  Appendix C Appendix: Loss functions found for all GP experiments

$f_{1}=\frac{1}{N}\sum_{i=1}^{N}{\left(e^{2.2061}+\sin{(1.7577)}+(y_{real}^{(i)}-4.092)^{3}-\log{(|y_{real}^{(i)}-y_{pred}^{(i)}|+\varepsilon)}\right)},\\\
$ (C1)

$f_{2}=\frac{1}{N}\sum_{i=1}^{N}{e^{\cos{\left(\sqrt{|y_{real}^{(i)}-y_{pred}^{(i)}|+\varepsilon}\right)}}},$
(C2)

$f_{3}=\frac{1}{N}\sum_{i=1}^{N}{\left(\sqrt{\left|y_{real}^{(i)}+\log{\left(\left|y_{pred}^{(i)}\right|+\varepsilon\right)}\right|+\varepsilon}\right)^{3}},$
(C3)

$f_{4}=\frac{1}{N}\sum_{i=1}^{N}{\left(\left(y_{pred}^{(i)}\right)^{3}+\sqrt{\left|3.985\times\frac{y_{real}^{(i)}}{y_{pred}^{(i)}+\varepsilon}\right|+\varepsilon}\right)},$
(C4)

$f_{5}=\frac{1}{N}\sum_{i=1}^{N}{\left[\left(\sqrt{\left|\log{\left(\left|y_{pred}^{(i)}\right|+\varepsilon\right)}\right|+\varepsilon}\right)^{3}+\frac{\log{\left(\left|\left(y_{pred}^{(i)}\right)^{3}\right|+\epsilon\right)}}{3.6278\times
y_{real}^{(i)}\times\left(y_{pred}^{(i)}\right)^{2}+\varepsilon}\right]},$
(C5)

$f_{6}=\frac{1}{N}\sum_{i=1}^{N}{\left(e^{\cos{\left(y_{pred}^{(i)}\right)}-\left(y_{pred}^{(i)}\right)^{2}}\times\left(y_{pred}^{(i)}-y_{real}^{(i)}\right)^{4}\right)},$
(C6)

$f_{7}=\frac{1}{N}\sum_{i=1}^{N}{\left[\sin{\left(1.0657\times
y_{real}^{(i)}+\frac{0.4129}{y_{pred}^{(i)}+\varepsilon}\right)}+\cos{\left(y_{real}^{(i)}+y_{pred}^{(i)}\right)^{2}}\right]},$
(C7)

$f_{8}=\frac{1}{N}\sum_{i=1}^{N}{\left[y_{pred}^{(i)}\times\cos{\left(y_{real}^{(i)}\right)}^{2}\times\log{\left(\left|y_{real}^{(i)}\times
y_{pred}^{(i)}\right|+\varepsilon\right)}-\sqrt{\left(\left|\log{\left(\left|y_{pred}^{(i)}\right|+\varepsilon\right)}\right|+\varepsilon\right)}\right]}.$
(C8)

##  Appendix D Appendix: Discriminator Accuracy for GAN Performance Evaluation

The discriminator values should only be considered in conjunction with the FID
values of the generated images. First of all, the discriminator should have a
score between 50% and 100%, and would otherwise indicate a corrupt
discriminator. Secondly, given the same FID values a higher discriminator
accuracy points to a better generative performance, as having a discriminator
that is still able to discriminate fake and generated samples has the
potential to further improve the generator. A discriminator being unable to
discriminate generated from real samples cannot point the direction for the
generator to produce better results and thus may not further improve
generative results.

The CIFAR10 results (Section 4.2) showed a discriminator accuracy of about 50%
for GANetic loss indicating that training finished properly using the full
discriminator potential. As a side note, we did train GANetic loss on
different classification tasks (CIFAR10, ImageNet) where it showed similar
performance in classification accuracy compared to cross entropy loss.

##  Appendix E Appendix: Evaluation of the GANetic Loss on CIFAR10 Dataset

In Figure E, images generated with GANetic loss and BCE (as it was the second
best on CIFAR10 dataset) are compared.

Figure E: Examples of images generated by DCGANs for CIFAR10 dataset. GANs trained by BCE (top) or trained by GANetic loss (bottom). Table E: Results of ablation study on CIFAR10 for GANetic loss. Generator | Discriminator | Mean FID | STD
---|---|---|---
✓ |  | 9.67 | 3.60
| ✓ | 2.51 | 0.57
✓ | ✓ | 2.20 | 0.39

To be more specific, in Figure E the first column of images in the series was
generated during the best run of the DCGAN model used for BCE and GANetic
loss, the second column of images was generated from the worst DCGAN, and the
third column of images was generated from the model randomly selected from the
remaining program runs. DCGANs trained with the BCE loss function produced the
best results compared to other regular losses, but were still unable to
produce images indistinguishable to those of the original dataset, while
DCGANs trained with GANetic loss produced convincing images in all
experiments.

##  Appendix F Appendix: Convergence of Projected GANs

Figure F: Convergence of prGANs trained by GANetic, BCE and Hinge losses on
the BreCaHAD dataset.

##  Appendix G Appendix: Image Generation for Medical Applications

Figure G: Examples of images generated by StyleGAN2-ADA using GANetic loss for
BreCaHAD (left), Hyper-Kvasir (middle) and LAG (right) datasets.

##  Appendix H Appendix: Additional Evaluation of the GANetic Loss.

Table 7: FID and KID values for different losses trained on AFHQ-Cats,AFHQ-Dogs and AFHQ-Wild datasets Choi et al. (2020). Model (Loss) | AFHQ-Cats | AFHQ-Dogs | AFHQ-Wild
---|---|---|---
| FID | KID | FID | KID | FID | KID
prGAN (original) | 3.25 | 0.29 | 7.08 | 2.12 | 4.21 | 0.20
prGAN (GANetic) | 2.64 | 0.19 | 6.69 | 1.86 | 4.86 | 0.35
StyleGAN2-ADA (original) | 3.55 | 0.66 | 7.40 | 1.16 | 3.05 | 0.45
StyleGAN2-ADA (GANetic) | 3.22 | 0.61 | 9.12 | 2.33 | 2.38 | 0.31

##  Appendix I Appendix: Anomaly Detection

Table I: Ablation study of f-AnoGAN configuration (D: Discriminator, G: Generator, E: Encoder), values are mean AUROC $\pm$ STD. $D$ | $G$ | $E$ | Kvasir | LAG
---|---|---|---|---
✓ |  |  | $99.01\pm 0.06$ | $78.03\pm 1.19$
| ✓ |  | $93.10\pm 5.24$ | $75.39\pm 1.32$
|  | ✓ | $76.06\pm 0.69$ | $58.39\pm 1.62$
✓ | ✓ |  | $98.97\pm 0.14$ | $78.22\pm 0.88$
| ✓ | ✓ | $76.58\pm 0.28$ | $59.29\pm 0.80$
✓ |  | ✓ | $76.92\pm 0.11$ | $57.75\pm 0.47$
✓ | ✓ | ✓ | $76.99\pm 0.03$ | $58.50\pm 1.91$
Figure I: Discrete distributions of anomaly scores for the Hyper-Kvasir
dataset obtained by the f-AnoGAN model, trained using GANetic loss.

##  Appendix J Appendix: Discussion

Generally, the cross entropy loss has a similar shape as GANetic loss, but BCE
is monotonically decreasing. Additionally, the $f_{6}$ loss (Equation C6),
that is the second best in Table 1, has a very similar shape compared to the
GANetic loss when $y_{pred}=1$: it also has a minimum and after reaching it
the loss value increases even though the model’s confidence in the prediction
is high. The minimum of the $f_{6}$ loss is at $1.01$, making it monotonically
decreasing in the range $[0,1]$. Thus, its behaviour is the same as the one of
the BCE loss in the range $[0,1]$. However, it wasn’t able to train DCGAN on
CIFAR10 dataset, which led to its exclusion from the further experiments.

To evaluate the implicit regularization more comprehensively a more systematic
approach would be required with an in-depth analysis that would be out of
scope of the current work. However, the slope of the GANetic function could be
the starting point for such an analysis in future work and may lead to further
theoretical insights about implicit regularization and optimization of
adversarial networks.