Tunneling Effect in Gapped Phosphorene through Double Barriers

Jilali Seffadia, Ilham Redouania, Youness Zahidi<EMAIL_ADDRESS>and
Ahmed Jellal†††a.jellal@ucd.ac.maa,c

aLaboratory of Theoretical Physics, Faculty of Sciences, Chouaïb Doukkali
University,

PO Box 20, 24000 El Jadida, Morocco

bMRI Labortory, Polydisciplinary Faculty, Sultan Moulay Selimane University,
PO Box 145, 25000 Khouribga, Morocco

cCanadian Quantum Research Center, 204-3002 32 Ave Vernon,
BC V1T 2L7, Canada

We study the transport properties of charge carriers in phosphorene with a
mass term through double barriers. The solutions of the energy spectrum are
obtained and the dependence of the eigenvalues on the barrier potentials and
wave vectors in the $x$-direction is numerically computed. Using the boundary
conditions together with the matrix transfer method, we determine transmission
and the conductance of our system. These two quantities are analyzed by
studying their main characteristics as a function of the physical parameters
along the armchair direction. Our results show the highly anisotropic
character of phosphorene and the no signature of Klein tunneling at normal
incidence contrary to graphene. Moreover, it is found that the transmission
and conductance display oscillatory behaviors in terms of the barrier width
under suitable conditions.

PACS numbers: 73.22.-f; 73.63.Bd; 72.10.Bg; 72.90.+y

Keywords: Phosphorene, double barriers, energy gap, transmission, conductance.

##  1 Introduction

Two dimensional (2D) crystals consisting of single or a few atomic layers has
been the focus of instance research currently, due to their fundamental
properties and possible applications. Since the first experimental fabrication
of graphene[1], a 2D single sheet of carbon honeycomb, the researchers were
focused on it because of its unique electronic [2, 4, 3, 5], optical [6] and
mechanical properties [7, 8]. From then, there is a growing interest in the
realization of other (2D) materials. The investigation of analogs of graphene
has resulted in the discovery of various single-layer crystals of different
elements. Among them, silicon (silicene) [9], germanium (germanene)[10] and a
class of materials known as transition-metal dichalcogenides[11]. In recent
years, it is showed that such materials serve as test bed for Dirac physics,
which arises from the fact that their low-energy quasiparticles obey an
effective Dirac-like equation. These quasiparticles lead to a host of
unconventional transport properties and distinguishes them from conventional
materials obeying the Schrödinger equation.

The black phosphorus is among the most promising 2D materials, which is an
allotrope of phosphorus[12, 13, 14, 15] and well-known as phosphorene. It has
triggered tremendous attention since its first discovery in 2014 [14, 15].
Similarly to graphene, phosphorene can be mechanically exfoliated to obtain
samples with a single or few layers. It features an orthorhombic crystal
structure, whose individual layers feature a puckered, honeycomb lattice
structure and each atom is covalently bonded to three of its neighbor.
Phosphorene is a semiconductor and has a high electronic mobility in the range
of $1000$ cm2 V-1 s-1 [14], which makes it as a possible candidate for device
applications[16, 17]. Its electronic structure [18] shows, in contrast to
graphene, an intrinsic band-gap and strong anisotropy, which causes the
electrons to behave in one direction like massive Dirac fermions and in the
orthogonal direction like non-relativistic Schrödinger electrons [19].
Phosphorene shows a robust direct band gap for all thickness varying from
$1.8$ eV for single layer to $0.4$ eV for bulk samples [14], which depends on
the number of layers.

The quantum transport in phosphorene was analyzed under different
circumstances. For instance, it was highlighted the anisotropic transport
signatures of phosphorene across a NBN junction with different orientations
[20]. Additionally, it was shown that for a barrier along the $x$-direction,
the dominant contribution to the transport comes from the quasiparticles,
which impinge on the barrier at near-normal incidence provided the applied
voltage is close to the bottom of the conduction band, leading to collimated
transport of electrons. Recently, it was demonstrated that phosphorene PNP
junctions constitute perfect electron waveguides, without any leakage current
through the side- walls [21] and at the interfaces of the junction, the omni-
directional total reflection takes place, which is not due to an energetically
forbidden region but due to pseudo-spin blocking.

Motivated by the results developed in [20, 21], we consider a monolayer
phosphorene and study the scattering of charge carriers through double barrier
structure with an energy gap in the central region. We solve the eigenvalue
equation of the continuum model and then determine the solutions of energy
spectrum in different regions. By requiring the continuity of the wave
functions at interfaces and using the transfer matrix method, we determine the
transmission and the conductance as a function of the physical parameters.
Numerically we show that the transmission displays sharp pics inside the
transmission gap around some energy values, which are resulted from the
quasibound states formed in the double barrier structure. Additionally, our
analysis tells us there is no signature of the Klein tunneling at normal
incidence. Moreover, we find that the conductance displays oscillatory
behavior as a function of the barrier width.

The present paper is organized as follows. In section 2, we describe our
system by setting the appropriate Hamiltonian, which will be used to obtain
the eigenvalues and eigenspinors. By applying the boundary conditions at
interfaces, we determine the transmission probability and the associated
conductance. In section 4, we numerically analyze our results under suitable
conditions of the physical parameters. Finally, we conclude our work.

##  2 Hamiltonian formalism

Recall that phosphorene is a 2D hexagonal lattice that is buckled (or
”puckered”) due to the hybridization $sp^{3}$. The phosphorus atoms at
different sublayers are represented in Figure 1 with sublattices $(A,B)$ at
bottom sublayer (pink symbol) and ($C,D$) at top one (green symbol). This
geometry results in two types of bands as shown as in Figure 1(a) where atoms
$P$, connected by bonds parallel to the plane, form upper and lower sublayers,
while the bonds connecting atoms $P$ between these two sublayers are oriented
by an angle $71.7^{\circ}$ out of plane. Figures 1(b,c) present the top and
side-view, along the armchair direction, of the lattice structure of monolayer
phosphorene. The shaded rectangle indicates the unit cell and the side view
highlights the inter-atomic coupling. The hopping parameter $t_{1}$ is for the
connection along a zigzag direction in the upper or lower sublayer and $t_{2}$
is for that between a pair of zigzag chains in the upper and lower sublayers.
The parameters $t_{3}$ and $t_{4}$ are between the nearest-neighbor and next
nearest neighbor sites of a pair of zigzag chains in the upper or lower
sublayer. The parameter $t_{5}$ is between two atoms on upper and lower zigzag
chains that are farthest from each other. Figure 1(d) shows the lattice
structure of phosphorene, emphasizing the bond lengths and bond angles between
the phosphorus atomic sites, with $a_{1}=2.22\ \AA$ is the distance between
nearest neighbour sites in sublattices ($A,B$) or ($C,D$) and $a_{2}=2.24\
\AA$ is that between nearest neighbor sites in sublattices ($A,C$) or ($B,D$).
The bond angles are $\alpha_{1}=96.5^{\circ}$, $\alpha_{2}=101.9^{\circ}$ and
$\beta=72^{\circ}$.

Figure 1:  (a): Bird eye’s view of the lattice structure of monolayer
phosphorene, where the phosphorus atoms at different sublayers are represented
by different colors in each sublayer. (b): 2D lattice of monolayer projected
onto the xy-plane. Shaded rectangle indicates the unit cell. (c): The ”zigzag”
and ”armchair” edges with inter-atomic coupling. (d): Lattice structure of
monolayer phosphorene systems with the different parameters.

It has been recently shown within the five-hopping parameter approach [22, 23]
that the continuum approximation is very suited for describing the physics of
large phosphorene, yielding very accurate results within its limit of
validation. A unitary transformation can be performed to rewrite the monolayer
Hamiltonian in a simpler block form. In our case, we consider the block
Hamiltonian [22]

$H^{\pm}(k)=\left(\begin{array}[]{cc}t_{AA}(k)\pm t_{AD}(k)&t_{AB}(k)\pm
t_{AC}(k)\\\ t_{AB}^{*}(k)\pm t_{AC}^{*}(k)&t_{AA}(k)\pm t_{AD}(k)\\\
\end{array}\right)$ (1)

where under expansion around the long-wavelength $k=0$ ($\Gamma$ point) up to
second order, the structure factors become

$t_{AA}(k)=\delta_{AA}+\eta_{AA}k_{y}^{2}+\gamma_{AA}k_{x}^{2}\qquad\qquad\quad$
(2)
$t_{AB}(k)=\delta_{AB}+\eta_{AB}k_{y}^{2}+\gamma_{AB}k_{x}^{2}+i\chi_{AB}k_{x}$
(3)
$t_{AC}(k)=\delta_{AC}+\eta_{AC}k_{y}^{2}+\gamma_{AC}k_{x}^{2}+i\chi_{AC}k_{x}$
(4) $t_{AD}(k)=\delta_{AD}+\eta_{AD}k_{y}^{2}+\gamma_{AD}\qquad\qquad\quad$
(5)

giving rise to the following Hamiltonian

$H^{+}(k)=\left(\begin{array}[]{cc}u_{0}+\eta_{x}k_{x}^{2}+\eta_{y}k_{y}^{2}&\delta+\gamma_{x}k_{x}^{2}+\gamma_{y}k_{y}^{2}+i\chi
k_{x}\\\ \delta+\gamma_{x}k_{x}^{2}+\gamma_{y}k_{y}^{2}-i\chi
k_{x}&u_{0}+\eta_{x}k_{x}^{2}+\eta_{y}k_{y}^{2}\\\ \end{array}\right)$ (6)

with the setting

$\displaystyle
u_{0}=\delta_{AA}+\delta_{AD},\qquad\delta=\delta_{AB}+\delta_{AC}$ (7)
$\displaystyle\eta_{x}=\eta_{AA}+\eta_{AD},\qquad\eta_{y}=\gamma_{AA}+\gamma_{AD}$
(8)
$\displaystyle\gamma_{x}=\eta_{AB}+\eta_{AC},\qquad\gamma_{y}=\gamma_{AB}+\gamma_{AC}$
(9) $\displaystyle\chi=\chi_{AB}+\chi_{AC}$ (10)

$\vec{k}=(k_{x},k_{y})$ is the wave vector and the coefficient values of the
expanded structure factors for both five-hopping models are summarized in
Table 1.

| 5-hopping |  | 5-hopping | Units
---|---|---|---|---
$\delta_{AA}$ | 0.00 | $\delta_{AB}$ | -2.85 | eV
$\delta_{AC}$ | 3.61 | $\delta_{AD}$ | -0.42 | eV
$\eta_{AA}$ | 0.00 | $\eta_{AB}$ | 3.91 | eV ${\AA}^{2}$
$\eta_{AC}$ | -0.53 | $\eta_{AD}$ | 0.58 | eV ${\AA}^{2}$
$\gamma_{AA}$ | 0.00 | $\gamma_{AB}$ | 4.41 | eV ${\AA}^{2}$
$\gamma_{AC}$ | 0.00 | $\gamma_{AD}$ | 1.01 | eV ${\AA}^{2}$
$\chi_{AB}$ | 2.41 | $\chi_{AC}$ | 2.84 | eV ${\AA}$
Table 1: Structure factor coefficients.

In order to study the scattering of charge carriers in phosphorene, we
consider the following double barrier

$V_{j}(x)=\left\\{\begin{array}[]{llll}{V_{1,5}=0}&\mbox{if}&{|x|>d_{2}},\\\
{V_{2}}&\mbox{if}&{-d_{2}<x<-d_{1}},\\\ {V_{3}}&\mbox{if}&{|x|<d_{1}},\\\
{V_{4}}&\mbox{if}&{d_{1}<x<d_{2}}.\\\ \end{array}\right.$ (11)

where $V_{1,2,3}$ and $d_{1,2,3}$ are positive potential parameters
characterizing the double barrier structure shown in Figure 2 with $j$
labeling the five regions.

Figure 2:  Schematic profile of the barrier potentials $V(x)$ composed of five
regions with three amplitudes $(V_{2},V_{3},V_{4})$ and a gap placed in region
$3$.

According to the possible profile, the Hamiltonian describing the four regions
$1,2,4,5$ can be written as

$H_{j}(k)=H^{+}(k)+V_{j}(x)\mathbb{I}_{2}$ (12)

and as for region $3$ we have

$H_{3}(k)=H^{+}(k)+V_{3}(x)\mathbb{I}_{2}+\Delta\sigma_{z}$ (13)

where $\sigma_{z}$ is the Pauli matrix and $\Delta$ is the involved gap. Then,
the dispersion relations can be calculated to end up with the energies in
regions $j=(1,2,4,5)$

$\displaystyle
E_{j}=V_{j}+u_{0}+\eta_{x}k_{jx}^{2}+\eta_{y}k_{y}^{2}+s_{j}\sqrt{\left(\delta+\gamma_{x}k_{jx}^{2}+\gamma_{y}k_{y}^{2}\right)^{2}+\left(\chi
k_{jx}\right)^{2}}$ (14)

as well as in region 3

$\displaystyle
E_{3}=V_{3}+u_{0}+\eta_{x}k_{jx}^{2}+\eta_{y}k_{y}^{2}+s_{3}\sqrt{\left(\delta+\gamma_{x}k_{jx}^{2}+\gamma_{y}k_{y}^{2}\right)^{2}+\left(\chi
k_{jx}\right)^{2}+\Delta^{2}}$ (15)

with
$s_{j}=\text{sign}\left(E-V_{j}-u_{0}-\eta_{x}k_{jx}^{2}-\eta_{y}k_{y}^{2}\right)$
and
$s_{3}=\text{sign}\left(E-V_{3}-u_{0}-\eta_{x}k_{jx}^{2}-\eta_{y}k_{y}^{2}\right)$.
Note that, the term proportional to $k_{jx}^{2}$ in the energy spectrum can be
ignored because at low momenta we have $u_{0},\delta,k_{jx}\gg k_{jx}^{2}$ and
$k_{jx}^{2}<1$ [20]. The longitudinal wave vectors $k_{jx}$ corresponding to
(14) and (15) are given by

$\displaystyle
k_{jx}=\chi^{-1}\sqrt{\left(E_{j}-V_{j}-u_{0}-\eta_{y}k_{y}^{2}\right)^{2}-\left(\delta+\gamma_{y}k_{y}^{2}\right)^{2}}$
(16) $\displaystyle
k_{3x}=\chi^{-1}\sqrt{\left(E_{3}-V_{3}-u_{0}-\eta_{y}k_{y}^{2}\right)^{2}-\left(\delta+\gamma_{y}k_{y}^{2}\right)^{2}-\left(\Delta\right)^{2}}$
(17)

We note that the energy spectrum is linear (Dirac like) in $k_{x}$ but
parabolic (Schrödinger like) in $k_{y}$. Consequently, the nature of the
quasiparticle transport and the effect of a potential barrier depends
crucially on the its orientation in the $xy$-plane. In addition, by increasing
the potential height $V_{3}$, the energy bands are shifted upwards also by
$V_{3}$, as shown in Figure 3(a). Moreover, we can clearly see the effect of
the energy gap, as shown in Figure 3(b), that increase the gap between the
valence and conduction bands.

Figure 3: The energy eigenvalues outside and inside the barrier regions as a
function of the wave vector $k_{y}$ and $k_{x}$. (a): $V_{3}=0$ eV and
$\Delta=0$ eV (green lines), $V_{3}=1$ eV and $\Delta=0$ eV (blue lines). (b):
$V_{3}=0$ eV and $\Delta=1$ eV (green lines), $V_{3}=0$ eV and $\Delta=0$ eV
(blue lines).

The associated eigenspinors to (14) and (15) can be determined by solving the
eigenvalue equation $H_{j}\Phi_{j}=E_{j}\Phi_{j}$. This process yields the
solutions in five regions

$\displaystyle\Phi_{\sf 1}=\left(\begin{array}[]{c}{1}\\\ {z_{1}}\\\
\end{array}\right)e^{i(k_{1x}x)}+r\left(\begin{array}[]{c}{1}\\\
{z_{1}^{-1}}\\\ \end{array}\right)e^{i(-k_{1x}x)}$ (22)
$\displaystyle\Phi_{\sf 2}=a\left(\begin{array}[]{c}{1}\\\ {z_{2}}\\\
\end{array}\right)e^{i(k_{2x}x)}+b\left(\begin{array}[]{c}{1}\\\
{z_{2}^{-1}}\\\ \end{array}\right)e^{i(-k_{2x}x)}$ (27)
$\displaystyle\Phi_{\sf 3}=c\left(\begin{array}[]{c}{\sigma}\\\ {\tau
z_{3}}\\\
\end{array}\right)e^{i(k_{3x}x)}+d\left(\begin{array}[]{c}{\sigma}\\\ {\tau
z_{3}^{-1}}\\\ \end{array}\right)e^{i(-k_{3x}x)}$ (32) $\displaystyle\Phi_{\sf
4}=e\left(\begin{array}[]{c}{1}\\\ {z_{4}}\\\
\end{array}\right)e^{i(k_{4x}x)}+f\left(\begin{array}[]{c}{1}\\\
{z_{4}^{-1}}\\\ \end{array}\right)e^{i(-k_{4x}x)}$ (37)
$\displaystyle\Phi_{\sf 5}=t\left(\begin{array}[]{c}{1}\\\ {z_{5}}\\\
\end{array}\right)e^{i(k_{5x}x)}$ (40)

where we have set $z_{j}=s_{j}e^{-i\theta_{j}}$ with
$s_{j}={\mbox{sign}}{\left(E-V_{j}-u_{0}-\eta_{y}k_{y}^{2}\right)}$ and the
involved parameters are given by

$\displaystyle\theta_{j}=\arctan\left(\frac{\chi
k_{jx}}{\delta+\gamma_{y}k_{y}^{2}}\right)$ (41)
$\displaystyle\sigma=\sqrt{1+\frac{s_{3}\Delta}{\sqrt{\Delta^{2}+(k_{3x})^{2}}}}$
(42)
$\displaystyle\tau=\sqrt{1-\frac{s_{3}\Delta}{\sqrt{\Delta^{2}+(k_{3x})^{2}}}}$
(43)

##  3 Transmission and conductance

Next we will calculate the transmission probability of electrons across the
potential barriers in our phosphorene system. In doing so, it is convenient to
use the matrix formalism. Using the boundary conditions at interfaces
$x=-d_{2}$, $-$$d_{1}$, $d_{1}$, $d_{2}$ demanding the eigenspenors
continuities

$\displaystyle\Phi_{\sf 1}(-d_{2})=\Phi_{\sf 2}(-d_{2}),\qquad\Phi_{\sf
2}(-d_{1})=\Phi_{\sf 3}(-d_{1}),\qquad\Phi_{\sf 3}(d_{1})=\Phi_{\sf
4}(d_{1}),\qquad\Phi_{\sf 4}(d_{2})=\Phi_{\sf 5}(d_{2})$ (44)

which can be mapped as

$\displaystyle M_{1}[-d_{2}]\left(\begin{array}[]{c}{1}\\\ {r}\\\
\end{array}\right)=M_{2}[-d_{2}]\left(\begin{array}[]{c}{a}\\\ {b}\\\
\end{array}\right)$ (49) $\displaystyle
M_{2}[-d_{1}]\left(\begin{array}[]{c}{a}\\\ {b}\\\
\end{array}\right)=M_{3}[-d_{1}]\left(\begin{array}[]{c}{c}\\\ {d}\\\
\end{array}\right)$ (54) $\displaystyle
M_{3}[d_{1}]\left(\begin{array}[]{c}{c}\\\ {d}\\\
\end{array}\right)=M_{4}[d_{1}]\left(\begin{array}[]{c}{e}\\\ {f}\\\
\end{array}\right)$ (59) $\displaystyle
M_{4}[d_{2}]\left(\begin{array}[]{c}{e}\\\ {f}\\\
\end{array}\right)=M_{5}[d_{2}]\left(\begin{array}[]{c}{t}\\\ {0}\\\
\end{array}\right)$ (64)

and then after a simple manipulation we get

$\left(\begin{array}[]{c}{1}\\\ {r}\\\
\end{array}\right)=M\left(\begin{array}[]{c}{t}\\\ {0}\\\ \end{array}\right)$
(65)

where the introduced transfer matrix is given by

$\displaystyle M$ $\displaystyle=M_{1}^{-1}[-d_{2}]\ M_{2}[-d_{2}]\
M_{2}^{-1}[-d_{1}]\ M_{3}[-d_{1}]\ M_{3}^{-1}[d_{1}]\ M_{4}[d_{1}]\
M_{4}^{-1}[d_{2}]\ M_{5}[d_{2}]$ (66)
$\displaystyle=\begin{pmatrix}M_{11}&M_{12}\\\ M_{21}&M_{22}\end{pmatrix}$
(67)

Since the wave vector incident and transmitted waves is the same, therefore,
the transmission probability can be obtained as

$T=|t|^{2}=\frac{1}{|M_{11}|^{2}}$ (68)

where $t$ is given by

$\displaystyle t=\frac{16\tau\sigma
s_{1}s_{2}s_{3}s_{4}\sin\theta_{1}\sin\theta_{2}\sin\theta_{3}\sin\theta_{4}e^{i\left(d_{2}(-2k_{1x}+k_{2x}+k_{4x})+d_{1}(k_{2x}+2k_{3x}+k_{4x})\right)}}{e^{4id_{1}k_{3x}}A_{1}A_{2}+B_{1}B_{2}}$
(69)

and the parameters are

$\displaystyle A_{1}=(z_{1}-z_{4}^{-1})(z_{4}\sigma-
z_{3}\tau)e^{2id_{1}k_{4x}}+(z_{1}-z_{4})(z_{4}^{-1}\sigma-
z_{3}\tau)e^{2id_{2}k_{4x}}$ (70) $\displaystyle
A_{2}=(z_{1}^{-1}-z_{2}^{-1})(z_{2}\sigma-
z_{3}^{-1}\tau)e^{2id_{2}k_{2x}}+(z_{1}^{-1}-z_{2})(z_{2}^{-1}\sigma-
z_{3}^{-1}\tau)e^{2id_{1}k_{2x}}$ (71) $\displaystyle
B_{1}=(z_{1}-z_{4})(z_{4}^{-1}\sigma-
z_{3}^{-1}\tau)e^{2id_{2}k_{4x}}+(z_{1}-z_{4}^{-1})(z_{4}\sigma-
z_{3}\tau)e^{2id_{1}k_{4x}}$ (72) $\displaystyle
B_{2}=(z_{1}^{-1}-z_{2}^{-1})(z_{2}\sigma-
z_{3}\tau)e^{2id_{2}k_{2x}}+(z_{1}^{-1}-z_{2})(z_{2}^{-1}\sigma-
z_{3}\tau)e^{2id_{1}k_{2x}}.$ (73)

To complete our study, we investigate an important physical quantity relevant
to double barrier structure in phosphorene that is the conductance. Then,
using the Landauer-Buttiker formula [20] and obtained transmission probability
we get

$G=G_{0}\int_{-k_{y}^{\text{max}}}^{k_{y}^{\text{max}}}\frac{dk_{y}}{2\pi}T$
(74)

where $G_{0}=\frac{e^{2}L_{y}}{\hbar}$ is the unit conductance, $L_{y}$ is the
width of the sample in the $y$-direction and $k_{y}^{\text{max}}$ denotes the
maximum transverse momenta. In next section, we will study numerically these
two quantities. This will help us understand the effect of various physical
parameters on the transmission and the conductance related to our double
barrier structure.

##  4 Results and discussions

Figure 4 shows the transmission probability as a function of the wave vector
$k_{y}$ for different values of barrier width $d$ and incident energy $E$. The
barrier heights are fixed at $V_{2}=V_{4}=12$ eV, $V_{3}=4$ eV, and the gap
$\Delta=0$ eV. $T$ is plotted for $d=3{\text{\AA}}$ (red line), 15Å (blue
line), 20 Å (green line), 25 Å (black line) in Figure 4(a) for $E=6$ (red
line), 7 (blue line) and $8$ eV (green line) in Figure 4(b). We observe that
the transmission is bilaterally symmetrical with respect to the normal
incidence, i.e. $k_{y}=0$, and exhibits a behavior similar to that observed
for Dirac particle in graphene [24, 25, 26]. It is important to note that in
the case of graphene in the absence of a magnetic field, the perfect
transmission situation appears very clearly in the case $k_{y}=0$ (signature
of Klein tunneling), which is not the case in our system made of phosphorene.
As shown in Figure 4(a), the transmission through thin barriers is negligible
compared to that for large $d$, such behavior is similar to that seen for the
transmission thought a single barrier in monolayer phosphorene [20]. From
Figure 4(b), we notice that the transmission vanishes for specific values that
increase by increasing $E$. Note that when $k_{y}$ exceed a critical value the
transmission probability go to zero (the transmission is blocked by the
barrier) due to the evanescent nature of the states in the barrier. There
exist several line-shaped peaks in the forbidden transmission region that are
consequences of resonant transmission through the double barrier. The number
of resonant peaks changes with the change of the energy value as can be seen
in Figure 4(b).

Figure 4: Transmission probability $T$ as a function of $k_{y}$, with
$V_{2}=V_{4}=12$ eV , $V_{3}=4$ eV , $\Delta=0$ eV and $d_{2}=2d_{1}=d$ for
different values of $d$ and $E$. (a) : $d=3\text{ \AA}$ (red line),
$d=15\text{ \AA}$ (blue line), $d=20\text{ \AA}$ (green line), $d=25\text{
\AA}$ (black line) and $E=6$ eV. (b) : $E=6$ eV (red line), $E=7$ eV (blue
line), $E=8$ eV (green line) and $d=25\text{ \AA}$.

In Figure 5, we present the transmission probability as a function of the
transverse wave vector $k_{y}$ and the energy $E$ for four configurations of
the double barrier. It is clearly seen that there are different energy zones
characterizing the transmission. Indeed, Figure 5(a) corresponds to the case
$V_{2}<V_{3}<V_{4}$ and shows seven energy zones. At normal incidence, we
observe that the first, third and fifth zones are bounded by the energy
intervals $0<E<V_{2}+u_{0}-\delta$, $V_{2}+u_{0}+\delta<E<V_{3}+u_{0}-\delta$
and $V_{3}+u_{0}+\delta<E<V_{4}+u_{0}-\delta$, respectively, which contains
oscillations (resonances) in the transmission. While the transmission is zero
in the second, fourth and sixth zones
$V_{i}+u_{0}-\delta<E<V_{i}+u_{0}+\delta$ with $i=2,3$ and 4, respectively. It
is important to note that the transmission displays sharp peaks inside the
transmission gap around the point $E=V_{2}+u_{0}$ and $E=V_{4}+u_{0}$, but
they are absent around the energy point $E=V_{3}+u_{0}$. Finally, the seventh
zone $E>V_{4}+u_{0}+\delta$ contains the usual high energy barrier
oscillations and asymptotically goes to unity at high energy. It is important
to note that, the transmission exhibit the same behavior for both cases
$V_{2}<V_{4}<V_{3}$ and $V_{3}<V_{2}<V_{4}$ as shown in Figure 5(b,c). Form
Figure 5(d) for $V_{2}=V_{4}\neq V_{3}$, we observe that there are only five
energy zones where in the first ($0<E<V_{2}+u_{0}-\delta$) and third
($V_{2}+u_{0}-\delta<E<V_{4}+u_{0}+\delta$) zones, the transmission exhibits
an oscillatory behavior as a function of the incident energy. Moreover, the
transmission displays sharp peaks inside the transmission gap around the point
$E=V_{2}+u_{0}=V_{4}+u_{0}$, that are absent around $E=V_{3}+u_{0}$. These
peaks can be attributed to the quasi-bound states formed in the double barrier
structure in similar way to that obtained for graphene [27].

Figure 5: Density plot of transmission probability $T$ as a function of the
transverse wave vector $k_{y}$ and energy $E$, with $d_{1}=20\ \text{\AA}$,
$d_{2}=30.3\ \text{\AA}$ and $\Delta=0$ eV, for different barrier structures.
(a): $V_{2}=4$ eV, $V_{3}=8$ eV and $V_{4}=12$ eV. (b): $V_{2}=V_{4}=12$ eV
and $V_{3}=4$ eV. (c): $V_{2}=4$ eV, $V_{3}=12$ eV and $V_{4}=8$ eV. (d):
$V_{2}=8$ eV, $V_{3}=4$ eV and $V_{4}=12$ eV.

We now investigate the effect of the introduced gap $\Delta$ on the
transmission probability $T$ in Figure 6, which has been performed by fixing
the parameters $d_{1}=20\ \text{\AA}$, $d_{2}=30.3\ \text{\AA}$ and $k_{y}=0$
for different barrier structures as in Figure 5. We observe that for
$\Delta=0$, $T$ exhibits sharp peaks around $E=V_{2}+u_{0}$ and
$E=V_{4}+u_{0}$, but they are absent around $E=V_{3}+u_{0}$, which confirm
what we found in Figure 5. It is clearly seen that as long as $\Delta$ the
transmission gap around $E=V_{3}+u_{0}$ increases in all cases and for large
$\Delta$, the transmission oscillation starts to decrease until vanishing.

It is worth while to investigate the conductance as a function of the incident
energy together with transmission in Figure 7. To analyze the conductance
behavior we distinguish five zones such that in the first zone
($E<V_{3}+u_{0}-\delta$), the resonances in the transmission correspond to
peaks in the conductance. The second zone
($V_{3}+u_{0}-\delta<E<V_{3}+u_{0}+\delta$) shows a window where $T$ and
$G/G_{0}$ are zero. In the third zone
($V_{3}+u_{0}+\delta<E<V_{4}+u_{0}-\delta$), the peaks in conductance have
shoulders due to the presence of resonances in the transmission. We notice
that the number of peaks is the same for the transmission and the conductance.
The fourth ($V_{4}+u_{0}-\delta<E<V_{4}+u_{0}+\delta$) presents a bowl where
$T$ and $G/G_{0}$ are mostly zero but contains resonances peaks. The fifth
zone ($V_{1}<E_{1}$) contains oscillations, the transmission converges to
unity at high energies and conductance researches a maximum value.

Figure 6: Density plot of transmission probability $T$ as a function of energy
gap $\Delta$ and incident energy $E$ with $d_{1}=20\ \text{\AA}$, $d_{2}=30.3\
\text{\AA}$ and $k_{y}=0\ \text{\AA}^{-1}$ for different barrier structures.
(a): $V_{2}=4$ eV, $V_{3}=8$ eV and $V_{4}=12$ eV. (b): $V_{2}=V_{4}=12$ eV
and $V_{3}=4$ eV. (c): $V_{2}=4$ eV, $V_{3}=12$ eV and $V_{4}=8$ eV. (d):
$V_{2}=8$ eV, $V_{3}=4$ eV and $V_{4}=12$ eV.

Figure 7: Transmission probability and conductance as a function of the
incident energy $E$ for $d_{2}=2d_{1}=d=12\ \text{\AA}$, $\Delta=0$,
$V_{2}=V_{4}=8$ eV, $V_{3}=4$ eV and $k_{y}=0$.

Figure (8) is intended to see the influence of the barrier width $d$ for
different values of the central potential $V_{3}$. It is clearly seen that the
transmission $T$ and conductance $G/G_{0}$ are either a decaying or an
oscillatory depending on the value of $V_{3}$. For the range
$E<V_{4}+u_{0}<V_{3}+u_{0}$, $T$ and $G/G_{0}$ display oscillatory behavior
with the increase of $d$ in similar way to that of massless Dirac like
electrons as seen in graphene [27]. However in contrast to graphene, $T$ and
$G/G_{0}$ show a sharp decay as a function of $d$ when
$E<V_{3}+u_{0}<V_{4}+u_{0}$. For $V_{3}+u_{0}<E<V_{4}+u_{0}$, $T$ and
$G/G_{0}$ show an oscillatory behavior similar to single and multiple barrier
structure in monolayer phosphorene [20].

Figure 8: The transmission probability $T$ and conductance $G/G_{0}$ as a
function of the barrier width $d$ with $d_{1}=d/2$, $d_{2}=d$ for $\Delta=0$
eV, $E=7.5$ eV and $k_{y}=0$. $V_{2}=V_{4}=8$ eV, $V_{3}=5$ eV (red solid
line), $V=8$ eV (blue solid line), $V=10eV$ (green solid line).

##  5 Conclusion

We have investigated the transport properties of charge carriers transmitted
through monolayer phosphorene with double barriers. We have started by
formulating the Hamiltonian model describing our system and getting the
associated energy bands. We have shown that the anisotropic properties arise
from the difference between $\Gamma-X$ and $\Gamma-Y$ dispersions
corresponding to the armchair and zigzag directions, respectively.
Subsequently, by using the transfer matrix method, we have calculated the
transmission as well as the conductance in terms of the physical parameters.

Our numerical analysis showed that the transmission displays sharp pics inside
the transmission gap around $E=V_{4}+u_{0}$ and $E=V_{2}+u_{0}$, which are
absent around $E=V_{3}+u_{0}$. These peaks can be attributed to the quasi-
bound states formed in the double barrier structure. This behavior was
observed in monolayer graphene with double barrier structure. Moreover, we
have found that the transmission is bilaterally symmetrical with respect to
the normal incidence $k_{y}$ and there is no signature of the Klein tunneling
contrary to graphene. It was argued with the increase of $k_{y}$ the
transmission vanishes for some specific values depending on the incident
energy. We have observed that the transmission through thin barriers is
negligible compared to that in single barrier for large width $d$. It was
shown that when $k_{y}$ exceed a critical value the transmission is blocked by
the barrier, which is due to the evanescent nature of the states in the
barrier. We have seen that many line-shaped peaks appeared in the forbidden
transmission region whose number changes with the change of energy that are
the consequences of resonant transmission through the double barrier.

Finally, we have shown that the transmission and the conductance display
oscillatory behavior as a function of the barrier width $d$ for both cases
$E<V_{4}+u_{0}<V_{3}+u_{0}$ and $V_{3}+u_{0}<E<V_{4}+u_{0}$, but sharp decay
for $E<V_{3}+u_{0}<V_{4}+u_{0}$ similar to that obtained for single barrier in
phosphorene. It was found that the conductance as a function of the incident
energy presents a behavior similar to the transmission one. It was argued that
for $E<V_{3}+u_{0}-\delta$ the resonances in transmission correspond to peaks
in conductance. Moreover, around $E=V_{3}+u_{0}$ and $E=V_{4}+u_{0}$, we have
found that the conductance is zero, while the peaks in the conductance around
$E=V_{4}+u_{0}$ are due to the presence of resonances in the transmission
probability.

## References

  * [1] K. S. Novoselov, A. K. Geim, S. V. Morozov, D.-e. Jiang, Y. Zhang, S. V. Dubonos, I. V. Grigorieva, A. A. Firsov, Science 306 (2004) 666.
  * [2] A. C. Neto, F. Guinea, N. M. Peres, K. S. Novoselov, A. K. Geim, Reviews of Modern Physics 81 (2009) 109.
  * [3] L. E. F. Torres, S. Roche, J.-C. Charlier, Introduction to graphene-based nanomaterials: from electronic structure to quantum transport (Cambridge University Press, 2014).
  * [4] M. Katsnelson, K. Novoselov, A. Geim, Nature physics 2 (2006) 620.
  * [5] K. Novoselov, S. Morozov, T. Mohinddin, L. Ponomarenko, D. Elias, R. Yang, I. Barbolina, P. Blake, T. Booth, D. Jiang, J. Giesbers, E. W. Hill, A. K. Geim, Physica Status Solidi (b) 244 (2007) 4106.
  * [6] J. Qiao, X. Kong, Z.-X. Hu, F. Yang, W. Ji, Nature Communications 5 (2014) 1.
  * [7] D. G. Papageorgiou, I. A. Kinloch, R. J. Young, Progress in Materials Science 90 (2017) 75.
  * [8] I. Ovid’Ko, Reviews on Advanced Materials Science 34 (2013) 1.
  * [9] P. De Padova, C. Ottaviani, C. Quaresima, B. Olivieri, P. Imperatori, E. Salomon, T. Angot, L. Quagliano, C. Romano, A. Vona, 2D Materials 1 (2014) 021003\.
  * [10] M. Dávila, L. Xian, S. Cahangirov, A. Rubio, G. Le Lay, New Journal of Physics 16 (2014) 095002.
  * [11] B. Radisavljevic, A. Radenovic, J. Brivio, V. Giacometti, A. Kis, Nature Nanotechnology 6 (2011) 147.
  * [12] L. Li, Y. Yu, G. J. Ye, Q. Ge, X. Ou, H. Wu, D. Feng, X. H. Chen, Y. Zhang, Nature Nanotechnology 9 (2014) 372.
  * [13] A. Rodin, A. Carvalho, A. C. Neto, Physical Review Letters 112 (2014) 176801.
  * [14] S. P. Koenig, R. A. Doganov, H. Schmidt, A. Castro Neto, B. Özyilmaz, Applied Physics Letters 104 (2014) 103106.
  * [15] H. Liu, A. T. Neal, Z. Zhu, Z. Luo, X. Xu, D. Tománek, P. D. Ye, ACS nano 8 (2014) 4033.
  * [16] D. K. Sang, H.Wang, Z. Guo, N. Xie, H. Zhang, Advanced Functional Materials 29 (2019) 1903419\.
  * [17] L. Li, Y. Yu, G. J. Ye, Q. Ge, X. Ou, H. Wu, D. Feng, X. H. Chen, Y. Zhang, Nature Nanotechnology 9 (2014) 372.
  * [18] S. Fukuoka, T. Taen, T. Osada, Journal of the Physical Society of Japan 84 (2015) 121004.
  * [19] L. Li, B. Partoens, F. Peeters, Physical Review B 97 (2018) 155424.
  * [20] S. De Sarkar, A. Agarwal, K. Sengupta, Journal of Physics: Condensed Matter 29 (2017) 285601.
  * [21] Y. Betancur-Ocampo, E. Paredes-Rocha, T.s Stegmann, Journal of Applied Physics 128 (2020) 114303.
  * [22] J. Pereira Jr, M. Katsnelson, Physical Review B 92 (2015) 075437.
  * [23] D. De Sousa, L. De Castro, D. Da Costa, J. M. Pereira Jr, Physical Review B 94 (2016) 235415.
  * [24] L. Liu, Y.-X. Li, J.-J. Liu, Physics Letters A 376 (2012) 3342.
  * [25] J.-H. Yuan, J.-J. Zhang, Q.-J. Zeng, J.-P. Zhang, Z. Cheng, , Physica B 406 (2011) 4214.
  * [26] Y. Zahidi, I. Redouani, A. Jellal, Physica E 81 (2016) 259.
  * [27] A. Jellal, I. Redouani, Y. Zahidi, H. Bahlouli, Physica E 58 (2014) 30.


# Intriguing Properties of Modern GANs

Roy Friedman<EMAIL_ADDRESS>
Department of Computer Science
Hebrew University of Jerusalem Yair Weiss
Department of Computer Science
Hebrew University of Jerusalem

###### Abstract

Modern GANs achieve remarkable performance in terms of generating realistic
and diverse samples. This has led many to believe that “GANs capture the
training data manifold”. In this work we show that this interpretation is
wrong. We empirically show that the manifold learned by modern GANs does not
fit the training distribution: specifically the manifold does not pass through
the training examples and passes closer to out-of-distribution images than to
in-distribution images. We also investigate the distribution over images
implied by the prior over the latent codes and study whether modern GANs learn
a density that approximates the training distribution. Surprisingly, we find
that the learned density is very far from the data distribution and that GANs
tend to assign higher density to out-of-distribution images. Finally, we
demonstrate that the set of images used to train modern GANs are often not
part of the typical set described by the GANs’ distribution.

##  1 Introduction

Starting with their original formulation, Generative Adversarial Networks
(GANs, Goodfellow et al. 2014) quickly gained enormous popularity. As their
popularity increased, so did the approaches to evaluating the quality of the
images created by GANs and their fidelity to the true distribution (e.g.
Heusel et al. 2017; Richardson & Weiss 2018; Arora et al. 2018; Kynkäänniemi
et al. 2019). These methods exposed many problems in the distributions of
images generated by GANs, such as mode collapse (Goodfellow 2016 and artifacts
within generated images (Karras et al. 2021). Such developments were carried
out with the understanding that “the GAN’s training criteria has a global
minimum at the true distribution” (from Theorem 1 of Goodfellow et al.).

Following these evaluations, there was a refinement of GANs and such failures
have become much less prominent. For example, FID (Heusel et al. 2017) and
recall (Kynkäänniemi et al. 2019) both check whether the full diversity of the
training distribution is captured by the generative model. GANs which have
both low FID and high recall are deemed not to suffer from mode collapse, as
they supposedly capture the full support of the training data. Due to their
impressive performance in such metrics, many applications have progressed with
a working assumption that GANs learn the “training data manifold” (Schlegl et
al. 2017) This assumption has even carried through to domains where an
incorrect model of the true distribution can incur heavy risks, such as super-
resolving MRI scans (Bendel et al. 2022), simulations for self-driving cars
(Zhang et al. 2018a), detecting cancer through GAN inversions (Schlegl et al.
2017), recovering frequencies in astrophysical images ( Schawinski et al.
2017) and more.

In this work we study GANs with excellent performance and question whether
these GANs have actually learned the true data manifold. By critically
analyzing GANs through this lens, we find that they behave in ways counter to
popular belief.

As mentioned above, modern GANs are frequently thought of as manifold methods,
wherein the GANs’ generator captures the image manifold. Surprisingly, we find
that training examples are not part of the learned manifold. We go on to
demonstrate that the GAN manifold passes almost the same distance from the
class of images the generator was trained on as other classes, even when
measured with a perceptual distance (Zhang et al. 2018b). This failure is
intriguing as a GAN trained on ships, for instance, is only able to generate
images of ships and not other object classes. Furthermore, we find that the
GAN manifold passes unusually close to images outside the training domain. In
particular, we show that the GAN manifold passes closer to SVHN images than to
the images it was trained on.

While GANs are most commonly viewed as manifold methods, we demonstrate how
analyzing them only in these terms might not capture the full breadth of their
capabilities. In particular, the manifold view ignores the density the
generator assigns to different parts of space and whether or not the manifold
is more abundant in regions around the true distribution. To account for this,
we analyze the density the GAN’s generator assigns to samples in terms of the
average test log-likelihood. While mostly overlooked in the GAN literature,
the log-likelihood is widely used to evaluate other families of generative
models, such as score based models (Song et al. 2020) and normalizing flows
(Kingma & Dhariwal 2018). Moreover, the log-likelihood was used in the past to
show that large families of generative models assign higher likelihood to
images outside the training distribution than those used in training
(Nalisnick et al. 2018) and that many conditional generative models are weak
classifiers (Fetaya et al. 2019). Works such as those by Nalisnick et al.
focused on models where the likelihood is tractable, which have much worse FID
scores than modern GANs. There is scarce literature of this vein concerning
generative models that show excellent results on metrics other than average
test log-likelihood.

By evaluating GANs with low FID as density estimators, we show that modern
GANs under-perform in terms of average test log-likelihood when compared to
models with much worse FID. We go on to show that GANs are biased towards
images with larger “flat” areas. Indeed, the log-likelihood assigned by the
GAN is anti-correlated with local variance in the image. Finally, we question
whether the images used to train the GAN are part of the typical set of
images, a test previously used to determine whether a set of images are
outliers according to a generative model (Nalisnick et al. 2019).

To summarize, in this paper we discuss four intriguing properties of modern
GANs:

  * •

The learned manifold does not pass through the training examples.

  * •

The learned manifold is closer to out-of-distribution images than to in-
distribution images.

  * •

The density model learned by by the GAN assigns higher density to out-of-
distribution images.

  * •

The training images are not in the typical set of the GAN.

Figure 1: Modern GANs work amazingly well, to the point that it is expected
that they capture the data manifold (left). It is typically assumed that
because GANs can generate realistic images (right) that they capture the true
data manifold. In this paper we show that this assumption is false.

##  2 Background

Figure 2:  Left: $\ell_{2}$ distance (lower is better) of reconstructions of
generated and train images under StyleGAN-XL (Sauer et al. 2022) trained on
ImageNet (when only $z$ is optimized). The distribution of reconstruction
errors for training images is distinct from the same for images generated by
the GAN, implying that the training images are not part of the GAN manifold.
Right: examples of projections to the GAN. Generated images are part of the
manifold (and can be reconstructed), but training images are not. For training
images, the reconstructed images are realistic birds but they are different
birds.

###  2.1 Evaluation of Generative Models

State of the art GANs (Sauer et al. 2022; Karras et al. 2020b; Kang et al.
2023) are typically evaluated by methods that analyze the statistics of the
generated samples (Heusel et al. 2017; Kynkäänniemi et al. 2019; Richardson &
Weiss 2018). Almost all of these approaches define a statistic that is zero
only if the GAN-generated and true samples are from the same distribution.
Methods such as these are extremely effective in determining whether the
generative model has learned the correct statistic but are somewhat limited.
One such limitation is that for data in very high dimensions, many samples
might be needed to determine if the true and generated distribution are
distinct. Another is that the difference between the distributions is only
measured on particular statistics, ignoring the others.

Many generative models other than GANs utilize the average test log-likelihood
(LL) as an additional evaluation metric, since high test LL is equivalent to
low KL divergence between the generative distribution and the true data
distribution. For example, score-based models (Song et al. 2020) and
normalizing flows (Kingma & Dhariwal 2018; Nalisnick et al. 2018; Fetaya et
al. 2019) report the test LL or the negative test LL as a way to compare the
quality of the model. Since the LL is never explicitly used in GANs, it was
largely ignored when comparing such models and calculated in only relatively
few studies for small scale models on simple images, such as MNIST (Wu et al.
2016; Theis et al. 2015).

In the past, generative models were also evaluated through their performance
in inference tasks, such as generative classification (e.g. Fetaya et al.
2019; Hinton et al. 1995). In generative classification, a generative model
$p_{\theta_{c}}(x|c)$ is separately trained for each class $c$ in the data. An
unseen sample is then classified by determining which class-conditional LL is
the largest:

$\hat{c}(x)=\arg\max_{c}\log p_{\theta_{c}}(x|c)$ (1)

where $\hat{c}(x)$ is the predicted class for sample $x$. If
$p_{\theta_{c}}(x|c)\approx p_{\text{data}}(x|c)$ for all classes $c$, then
the above classification scheme is guaranteed to be optimal (Duda et al.
1973). On the flip side, if the classification performance is sub-optimal,
then in no uncertain terms $p_{\theta_{c}}(x|c)\neq p_{\text{data}}(x|c)$.
Using inference tasks to evaluate generative models can expose biases in the
generative distribution, and is potentially a sample-efficient way to find out
whether a generative model hasn’t learned the true data distribution.

###  2.2 Annealed Importance Sampling

Calculating the LL of any decoder-based model involves calculating the
following integral:

$p_{\theta}(x)=\intop p_{\theta}(x|z)p(z)dz$ (2)

where $p(z)$ is the prior over the latent dimension and $p_{\theta}(x|z)$ is
the probability of generating $x$, using the model, with the latent code $z$.
Unfortunately, analytically solving this integral is intractable for complex
decoder-based models. Instead, approximate methods such as Markov Chain Monte
Carlo (MCMC) must be used in order to calculate the LL. In particular, Wu et
al. showed that annealed importance sampling (AIS, Neal 2001) can be used to
accurately approximate the LL of GANs.

AIS is an MCMC approach that uses multiple intermediate distributions in order
to estimate the integral in equation 2. Wu et al. showed that an accurate
estimate can be achieved, using many intermediate distributions (i.e. long
MCMC chains) and multiple chains. When used to calculate the _log_ of equation
2, the estimate is only a stochastic lower-bound (Grosse et al. 2015) of the
LL that gets tighter as more intermediate distributions are introduced. For a
more detailed explanation, see appendix A or Wu et al..

##  3 Are GANs Good Manifold Methods?

GANs transform samples from a low dimensional latent space, $\mathcal{Z}$,
into the larger dimension of the data $\mathcal{X}$, effectively describing a
manifold. In much of the literature, it is believed that the GAN captures the
true low dimensional behavior of the training data (e.g. Menon et al.; Schlegl
et al.; Xiao et al.; Bendel et al.; Schawinski et al.). A simple schematic of
this assumption can be seen in figure 1, where it is usually expected that the
images from the training distribution are very close to, or exactly on, the
learned manifold. Furthermore, points that are not part of the training
distribution are expected to be far from the learned manifold, whether they
are natural images or not.

If GANs truly behave as believed and shown in figure 1, it should be possible
to use them for classification and outlier detection (Schlegl et al. (2017)).
For classification, an image is separately projected onto GANs trained on each
of the classes and is classified according to the manifold whose projection
was closest to the original image. Outlier detection can be carried out
analogously, where the point is projected to the GAN manifold and if it’s
distance from the original point is too large it is deemed an outlier.

###  3.1 Inference with the GAN Manifold

Figure 3: Performance of different GANs at the task of outlier detection (top)
and classification (bottom). In all cases, both the $\ell_{2}$ (plain) and the
LPIPS (hatched, Zhang et al. 2018b) distances are used. The GANs are compared
to a 1-nearest neighbor (1NN) baseline. For CIFAR10, all methods are also
compared with ViT-H (Dosovitskiy et al. 2020). The GANs always underperform,
compared to the 1NN baseline. Figure 4: Left: $\ell_{2}$ reconstruction errors
(lower is better) for the reconstruction of different image groups by
StyleGAN-XL trained on birds. The reconstruction error for birds or cars is
similar, while there exist images that the GAN can reconstruct much better
than those it was trained on. Right: examples for StyleGAN-XL reconstructions
of car images and rescaled SVHN images. In all cases, the GANs capture the
overall image structure, but don’t retain the identity of the main object in
the image, leading to projections which are far from the image whether in the
training domain or not.

We test whether modern GANs can be considered as manifold methods using a
range of GANs and various datasets. The GANs we use are: StyleGAN-XL (Sauer et
al. 2022) with an FID of 1.94 on ImageNet and 1.88 on CIFAR10, ReACGAN (Kang
et al. 2021; 2022) with an FID of 8.19 on ImageNet and 3.87 on CIFAR10,
BigGAN-DiffAug (Zhao et al. 2020) with an FID of 8.7 on CIFAR10, BigGAN (Kang
et al. 2022; Brock et al. 2018) with an FID of 8.54 on ImageNet, and
StyleGAN2-ADA (Karras et al. 2020a) with an FID of 3.55 on AFHQCat and 7.4 on
AFHQDog. To project the images on the GAN, we optimize the latent $z$ code
from multiple restarts and choose the optimized code with the smallest
reconstruction error. Further details can be found in appendix B. All of the
GANs chosen are able to generate highly realistic images and seemingly capture
the full diversity of the datasets they were trained on.

In the task of classification, we analyze the performance of the GANs on a
subset ( 1000 test samples) of the CIFAR10 dataset, a binary problem from
ImageNet (“sports car” class versus the “blue heron” class) and a subset (
1000) of AFHQ images, where the classes are AFHQDog, AFHQCat and AFHQWild. For
outlier detection in all datasets, we consider images of a single color and
rescaled versions of SVHN images as outliers and report the area under the ROC
curve (OD AUC).

The first assumption we question is whether the GAN manifold actually passes
through the training examples. For a given input $x$ we can determine if it is
on the GAN manifold by using gradient descent to approximately solve the
optimization problem:

$z^{*}=\arg\min_{z}\|G(z)-x\|$ (3)

Surprisingly, it turns out that most of the training points are _not part of
the GAN manifold_. Figure 2 (right) shows that for samples generated by the
GAN, the optimization algorithm succeeds in finding a latent $z$ so that the
generated images is almost identical to the input image $x$. But for images
from the training set, the best $z$ found by the optimization algorithm
generates an image that looks very different, although realistic. The figure
also shows histograms of the reconstruction error of training samples found by
inverting them from the $\mathcal{Z}$ space for StyleGAN-XL on ImageNet (for
results on more GANs, see appendix C). As a baseline, samples generated by the
GAN are also inverted and the histogram of their reconstruction error is shown
as well. In general, the reconstruction error of training images is an order
of magnitude worse than the reconstruction error of a generated image.

We then test the classification accuracy and outlier detection capabilities of
the GANs. If the GANs have learned something close to the true data manifold,
then they should act as good classifiers and outlier detectors. As a baseline,
we compare the GANs to a simple 1-nearest-neighbor (1NN) estimator. Under the
1NN estimator, a sample is classified according to the class of its nearest
neighbor. In outlier detection, a sample is considered an outlier if it is
further than a pre-defined threshold from its nearest neighbor in the training
distribution. This baseline is analogous to a GAN that has completely
memorized the training examples and can’t generate anything outside the
training examples.

We again find that the learned manifolds do not behave as expected. Using the
GAN manifolds for classification results in poor accuracy, as shown in figure
3 (bottom, in orange). The GANs are usually worse than the simple 1NN
classifier (in blue), even when using a more perceptual distance such as LPIPS
(Zhang et al. 2018b). The performance of GANs as outlier detectors is not much
better, as shown in figure 3 (top) - the performance of most GANs is worse
than random, even when using the perceptual distance. On the other hand, the
1NN outlier detector is always better than chance when using LPIPS.

We can better understand this poor performance by explicitly looking at the
projections to the GAN manifold. Figure 4 (left) shows the reconstruction
errors that different image groups have under a StyleGAN-XL trained on
ImageNet blue heron (bird) images. Many images of sports cars have the same
reconstruction error as images of birds, explaining why GANs are so bad at
classifying between different images. Moreover, the GAN is able to reconstruct
images of a single color and sometimes those from SVHN better than images from
the distribution they were trained on. Figure 4 (right) shows projections of
images of sports cars (top) and SVHN (bottom) onto the bird GAN. The
projection of any image keeps the overall image structure, even for images
from the wrong class. Because of this, the projection’s distance from the
original image is about the same whether the original was from the training
distribution or not.

Such behavior is surprising. Images of sports cars and SVHN digits are
frequently very different than those of blue herons, but the GAN is still able
to reconstruct much of the context of the images. In images of cars, the GAN
reconstruct the background, removing the car. For SVHN, the dominant colors
and positions of the digits are kept. This is the same reconstruction behavior
for images from the true distribution, an example of which can be seen in
figure 2 (right) for training images.

##  4 Are GANs Good Density Estimators?

Figure 5: An example of a case where simply calculating the distance from the
manifold might not tell the whole story. In this case, both $x_{1}$ and
$x_{2}$ are the same distance from the manifold but the manifold is more
abundant near $x_{1}$, making images in its region more plausible (left). By
adding an observation model (right), queries to the model are transformed to
calculating the log-likelihood under this observation model, essentially
integrating over all possible areas of the manifold. With the observation
model (right), $x_{1}$ is assigned a higher log-likelihood than $x_{2}$, even
though they are equally distant from the manifold.

Model | Dataset | LL (bits/dim.) $\uparrow$ | FID $\downarrow$
---|---|---|---
BigGAN-DiffAug | CIFAR10 | -6.68 | 8.7
StyleGAN-XL | CIFAR10 | -7.27 | 1.88
Glow (Kingma & Dhariwal, 2018) | CIFAR10 | -3.35 | 3.35
DDPM++ (Song et al., 2020) | CIFAR10 | -2.99 | 2.92
BigGAN | ImageNet | -7.59 | 8.54
StyleGAN-XL | ImageNet | -7.29 | 1.94
Glow (Kingma & Dhariwal, 2018) | ImageNet | -3.81 | -
VDM (Kingma et al., 2021) | ImageNet | -3.4 | -
StyleGAN2-ADA | AFHQ | -7.75 | 3.55

Table 1: GAN performance as a density estimator next to other generative
models. Although GANs sometimes come ahead in terms of FID, their average test
log-likelihood is always lower.

Viewing GANs purely as manifold methods potentially ignores two important
aspects. First, the prior over the latent codes is completely ignored in all
of the tests conducted above. Furthermore, while the GAN is equally far from
points in or outside of the distribution, a larger portion of it might pass
closer to the training samples than to other samples. An example of this can
be seen in figure 5 (left): while both points are equally far from the
manifold, more of the manifold passes close to the point in the middle of the
spiral ($x_{1}$), and much less near the other point ($x_{2}$). Such behavior
would explain why GANs are able to generate realistic images from the correct
class while still being equally distant from points inside and out of the
training distribution. To account for both of these aspects, we evaluate the
GANs as density estimators, implicitly taking both the distribution over the
latent space and the curvature of the manifold into account.

###  4.1 Adding an Observation Model

Figure 6: Performance of GANs as generative classifiers (left) and outlier
detectors (right). The GANs are outperformed by simple baselines, as well as
other generative models.

When distributions are defined on a low-rank space, it is common to assume an
added “observation noise” (e.g. Wu et al. 2016; Kingma & Welling 2013; Tipping
& Bishop 1999). In practice, this amounts to the generative model
$x=G_{\theta}(z)+\text{noise}$ where the noise is often assumed to be Gaussian
with a fixed variance, $\sigma^{2}$. The density (or likelihood) of a sample
under the noisy model is then given by:

$p_{\theta}(x)=\intop p_{\sigma}\left(x|\ G_{\theta}(z)\right)p(z)dz$ (4)

As mentioned by Wu et al., while the Gaussian observation model is possibly
too simplistic, it can used to better understand the distributions learned by
GANs. However, as long as this observation is used to compare GANs to each
other, then we believe that it does give an indication of which GAN better
captures the distribution.

We use the above described observation model in order to estimate the density
GANs assign to previously unseen samples, conducting the same experiments as
those for the manifold.

In all of our experiments, we use AIS with 500 steps, 4 chains, a Hamiltonian
Monte Carlo (HMC) kernel and 10 leap-frogs to estimate a sample’s LL. These
values were chosen to mitigate the high computational load of running AIS
while still extracting accurate estimates of the LL. More experimental details
can be found in appendix B.1. The variance of the observation model was chosen
to be the mean variance of the reconstruction error of training samples, which
is the maximum likelihood estimator for the observation noise. In all cases,
the PSNR of the images was high; that is, the variance of the noise assumed
was much smaller than the strength of the signal. For more details regarding
the variances used, see appendix B.3.

###  4.2 Evaluation as Density Models

Using the added observation model described above, we calculate the LL of the
different GANs on unseen data and compare them to other generative models.
This comparison can be found in table 1. GANs assign much lower LL to test
samples than other methods, even when their FID is better than the other
models.

Figure 6 shows the performance of the GANs as generative classifiers (left)
and outlier detectors (right), next to the LPIPS 1NN baseline and other
generative models. The classification accuracy of Glow is as reported by
Fetaya et al. and that of DDPM++ as reported by Zimmermann et al.. Once again,
the GANs are completely outperformed by the relatively simple 1NN estimator,
indicating that the density GANs assign to samples is different than that of
the true distribution.

Moreover, GANs tend to assign higher LL to images completely outside their
training distribution, as can be seen in figure 7 (left): the StyleGAN-XL bird
GAN gave higher LL to SVHN images than to test images.

Figure 7: Left: log-likelihoods (higher is better) that different image groups
attain under a StyleGAN-XL trained only on birds. Images of single color and
SVHN images have higher log-likelihoods than images from the domain the GAN
was trained on. Middle: the top and bottom images according to the log-
likelihood. GANs tend to have a bias towards images with larger flat region.
Right: a scatter plot of the images’ log-likelihood against the average
coefficient of variation (CV) of 8x8 patches. The log-likelihood assigned by
the GAN is anti-correlated with the average variance of the patches in the
image.

To further understand the behavior of the GANs as density estimators, we look
at which images have the highest and lowest LLs. The images with highest and
lowest LLs from the “blue heron” class of ImageNet can be seen in figure 7
(center). Visually it seems that GANs prefer images with larger flat areas and
low contrast. To be specific, figure 7 shows how the average coefficient of
variance in small 8x8 patches of the image is anti-correlated with the LL (on
the “sports car” and “blue heron” classes of ImageNet).

###  4.3 Typicality of Training Examples

While the samples from the true distribution might have lower LL than other
sets of images, this alone is not an indication of a failure of the GAN. A
stronger test is that of _typicality_ of a set of images. For the distribution
$p_{\theta}(x)$, we follow Nalisnick et al. and define the
$\left(\epsilon,N\right)$-typical set $\mathcal{A}_{\epsilon}^{N}(p_{\theta})$
of the distribution as all sets of $N$ samples that satisfy:

$\left|\frac{1}{N}\sum_{i=1}^{N}\log
p_{\theta}(x_{i})+H\left(p_{\theta}\right)\right|\leq\epsilon$ (5)

where $H\left(p_{\theta}\right)$ is the entropy of the distribution
$p_{\theta}(x)$. When $N$ is large, then (Cover 1999):

$P\left(\mathcal{A}_{\epsilon}^{N}(p_{\theta})\right)>1-\epsilon$ (6)

That is, when $N$ is large enough, this set of samples captures most of the
distribution according to $p_{\theta}(x)$.

We begin by drawing i.i.d. samples from this typical set, i.e. we generate
images from the GAN. We then calculate their LL using AIS, in exactly the same
procedure as that for the training and SVHN images. The average LL of images
generated by the GAN gives us an estimate of the entropy of $p_{\theta}(x)$:
$\hat{H}\left(p_{\theta}\right)=-\frac{1}{N}\sum_{i=1}^{N}\log p_{\theta}(x)$
where $x_{i}\sim p_{\theta}(x)$. $\epsilon$ is chosen such that in 95% of the
time, 50 samples drawn from the model are part of the typical set (i.e. the
bootstrap confidence interval suggested by Nalisnick et al.).

Figure 8: Histograms of log-likelihoods assigned to images generated by the
GAN (black), train images (blue) and SVHN images (red). Each plot is for a
different GAN-dataset pair. The dashed vertical lines are the means of the
log-likelihood of the corresponding colored histogram, and the shaded area
around the mean of the generated images are values of mean log-likelihoods
associated with the typical set of the GAN. In all cases, the training images
are outside this shaded area, meaning the set of training images _is not_ part
of the typical set of any of the GANs.

Using the typicality test suggested by Nalisnick et al., we find that for the
GANs we studied the set of _training samples is not part of the typical set_
of the distribution defined by the GAN. Sometimes images from SVHN are even
closer to being part of the typical set of the GAN than training images. This
can be seen in figure 8 where histograms of the LLs for images generated by
the GAN are shown next to histograms for other images. The dashed vertical
lines are the means of the histograms for each image type and the shaded
region is the $\epsilon$-ball around the estimate of the entropy containing
the typical sets $\mathcal{A}_{\epsilon}^{N}\left(p_{\theta}\right)$,
according to a bootstrap estimator of $\epsilon$. The mean LL for training
images is always outside the shaded area, indicating that training images are
not part of the typical sets of the GANs.

##  5 Related Works

This work follows a line of works (e.g. Nalisnick et al. 2018; Fetaya et al.
2019; Kirichenko et al. 2020) that show that many modern generative models do
not truly capture the underlying data distribution. As far as we know, GANs
have not been analyzed through the LL, despite their ability to generate
images perceptually similar to natural images, as there is no direct access to
the model likelihood.

There is a vast literature on evaluating whether GANs have learned the correct
distribution (e.g. Heusel et al. 2017; Sajjadi et al. 2018; Ravuri & Vinyals
2019; Webster et al. 2019; Naeem et al. 2020; Borji 2022; Ravuri et al. 2023).
Previous works predominantly use samples from the GAN in order to evaluate
them, which raises some uncertainties because the space of all images is huge.
In this work, we used the assumptions made when defining GANs in order to
evaluate them, thereby side-stepping the issue of determining whether the
learned distribution is correct through use of samples.

A key aspect of our investigation is the use of likelihood estimation and
inference with GANs. There have been many works in the past that have
suggested _extensions_ to GANs in order to allow them to work as classifiers
or outlier detectors (e.g. Schlegl et al. 2017; Donahue & Simonyan 2019; Kang
et al. 2021; Nitzan et al. 2022). Other works change the definition of GANs in
order to train them as density estimators together with their adversarial loss
(Lucas et al. 2019; Abbasnejad et al. 2019). All such methods use an
additional component besides the GAN generator, such as an encoder, which is
not part of the generative process of the GAN. In this work, we utilize the
above mentioned tasks as a way to better analyze how well the GAN has captured
the true data distribution. We argue that if GANs accurately capture the true
data distribution, their generator _alone_ should be enough to achieve near-
optimal performance in classification/outlier detection. Additionally, even
with the added observation model, we should expect that the training images
would be a part of the typical set of images supported by the GAN.

As mentioned in the introduction, GANs have been shown to not fully capture
the training distribution in the past, especially due to “mode collapse”
(Goodfellow 2016; Richardson & Weiss 2018). Arora et al. have quantified mode
collapse by a “birthday paradox” test, in which they found that the support of
many GANs is smaller than the size of the training distribution. To a large
part, mode collapse seems not to be a problem of modern GANs which capture the
full diversity of the training distribution, as evidenced by excellent FID
scores. Instead, our analysis points to a deeper problem than previously
investigated. Instead of fitting only small parts of the distribution, our
results indicate that GANs misrepresent almost all parts of the distribution.

##  6 Discussion

Modern GANs are able to generate incredibly realistic images, that seem to
arise from the same distribution as the images used to train the GAN.
Moreover, they have excellent performance under modern evaluation metrics.
These facts have led to a common belief that GANs capture the true data
manifold and the true data distribution.

When analyzing GANs as manifold methods, we find that their manifolds do not
fit the training samples. Quantitatively, this failure can be summarized by
the classification and outlier detection performance of the GANs when defining
the distance from the manifold as an estimate for whether the sample is from
the GAN’s distribution or not. Even when relaxing the requirement that samples
should be part of the GAN manifold does not improve the situation, as is
evidenced by the poor generative classification behavior. Worse, we showed how
the training examples are not part of the typical set of the GAN.

Even though this work has presented an overall negative view of GANs, they are
still extremely powerful data samplers. Indeed, this fact was taken advantage
of in domains such as image manipulation, style transfer and data
augmentation, to great effect. However, our work shows that a more cautious
stance should be taken when attempting to use GANs as priors for the true data
distribution.

### Acknowledgements

We thank the Gatsby Foundation for financial support.

## References

  * Abbasnejad et al. (2019) M Ehsan Abbasnejad, Qinfeng Shi, Anton van den Hengel, and Lingqiao Liu.  A generative adversarial density estimator.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10782–10791, 2019.
  * Arora et al. (2018) Sanjeev Arora, Andrej Risteski, and Yi Zhang.  Do gans learn the distribution? some theory and empirics.  In _International Conference on Learning Representations_ , 2018.
  * Bendel et al. (2022) Matthew Bendel, Rizwan Ahmad, and Philip Schniter.  A regularized conditional gan for posterior sampling in inverse problems.  _arXiv preprint arXiv:2210.13389_ , 2022.
  * Borji (2022) Ali Borji.  Pros and cons of gan evaluation measures: New developments.  _Computer Vision and Image Understanding_ , 215:103329, 2022\.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Cover (1999) Thomas M Cover.  _Elements of information theory_.  John Wiley & Sons, 1999.
  * Donahue & Simonyan (2019) Jeff Donahue and Karen Simonyan.  Large scale adversarial representation learning.  _Advances in neural information processing systems_ , 32, 2019.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Duda et al. (1973) Richard O Duda, Peter E Hart, and David G Stork.  _Pattern classification and scene analysis_ , volume 3.  Wiley New York, 1973.
  * Fetaya et al. (2019) Ethan Fetaya, Jörn-Henrik Jacobsen, Will Grathwohl, and Richard Zemel.  Understanding the limitations of conditional generative models.  _arXiv preprint arXiv:1906.01171_ , 2019.
  * Goodfellow (2016) Ian Goodfellow.  Nips 2016 tutorial: Generative adversarial networks.  _arXiv preprint arXiv:1701.00160_ , 2016.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Grosse et al. (2015) Roger B Grosse, Zoubin Ghahramani, and Ryan P Adams.  Sandwiching the marginal likelihood using bidirectional monte carlo.  _arXiv preprint arXiv:1511.02543_ , 2015.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Hinton et al. (1995) Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal.  The" wake-sleep" algorithm for unsupervised neural networks.  _Science_ , 268(5214):1158–1161, 1995.
  * Kang et al. (2021) Minguk Kang, Woohyeon Shim, Minsu Cho, and Jaesik Park.  Rebooting acgan: Auxiliary classifier gans with stable training.  _Advances in neural information processing systems_ , 34:23505–23518, 2021.
  * Kang et al. (2022) Minguk Kang, Joonghyuk Shin, and Jaesik Park.  Studiogan: a taxonomy and benchmark of gans for image synthesis.  _arXiv preprint arXiv:2206.09479_ , 2022.
  * Kang et al. (2023) Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10124–10134, 2023.
  * Karras et al. (2020a) Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  _Advances in neural information processing systems_ , 33:12104–12114, 2020a.
  * Karras et al. (2020b) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8110–8119, 2020b.
  * Karras et al. (2021) Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  _Advances in Neural Information Processing Systems_ , 34:852–863, 2021.
  * Kingma et al. (2021) Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  _Advances in neural information processing systems_ , 34:21696–21707, 2021.
  * Kingma & Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kingma & Dhariwal (2018) Durk P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  _Advances in neural information processing systems_ , 31, 2018.
  * Kirichenko et al. (2020) Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson.  Why normalizing flows fail to detect out-of-distribution data.  _Advances in neural information processing systems_ , 33:20578–20589, 2020.
  * Kynkäänniemi et al. (2019) Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved precision and recall metric for assessing generative models.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Lucas et al. (2019) Thomas Lucas, Konstantin Shmelkov, Karteek Alahari, Cordelia Schmid, and Jakob Verbeek.  Adaptive density estimation for generative models.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Menon et al. (2020) Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin.  Pulse: Self-supervised photo upsampling via latent space exploration of generative models.  In _Proceedings of the ieee/cvf conference on computer vision and pattern recognition_ , pp. 2437–2445, 2020.
  * Naeem et al. (2020) Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo.  Reliable fidelity and diversity metrics for generative models.  In _International Conference on Machine Learning_ , pp. 7176–7185. PMLR, 2020.
  * Nalisnick et al. (2018) Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan.  Do deep generative models know what they don’t know?  _arXiv preprint arXiv:1810.09136_ , 2018.
  * Nalisnick et al. (2019) Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan.  Detecting out-of-distribution inputs to deep generative models using typicality.  _arXiv preprint arXiv:1906.02994_ , 2019.
  * Neal (2001) Radford M Neal.  Annealed importance sampling.  _Statistics and computing_ , 11:125–139, 2001.
  * Nitzan et al. (2022) Yotam Nitzan, Rinon Gal, Ofir Brenner, and Daniel Cohen-Or.  Large: Latent-based regression through gan semantics.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 19239–19249, 2022.
  * Ravuri & Vinyals (2019) Suman Ravuri and Oriol Vinyals.  Classification accuracy score for conditional generative models.  _Advances in neural information processing systems_ , 32, 2019.
  * Ravuri et al. (2023) Suman Ravuri, Mélanie Rey, Shakir Mohamed, and Marc Peter Deisenroth.  Understanding deep generative models with generalized empirical likelihoods.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 24395–24405, 2023.
  * Richardson & Weiss (2018) Eitan Richardson and Yair Weiss.  On gans and gmms.  _Advances in neural information processing systems_ , 31, 2018.
  * Sajjadi et al. (2018) Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly.  Assessing generative models via precision and recall.  _Advances in neural information processing systems_ , 31, 2018.
  * Sauer et al. (2022) Axel Sauer, Katja Schwarz, and Andreas Geiger.  Stylegan-xl: Scaling stylegan to large diverse datasets.  In _ACM SIGGRAPH 2022 conference proceedings_ , pp. 1–10, 2022\.
  * Schawinski et al. (2017) Kevin Schawinski, Ce Zhang, Hantian Zhang, Lucas Fowler, and Gokula Krishnan Santhanam.  Generative adversarial networks recover features in astrophysical images of galaxies beyond the deconvolution limit.  _Monthly Notices of the Royal Astronomical Society: Letters_ , 467(1):L110–L114, 2017.
  * Schlegl et al. (2017) Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs.  Unsupervised anomaly detection with generative adversarial networks to guide marker discovery.  In _International conference on information processing in medical imaging_ , pp. 146–157. Springer, 2017.
  * Song et al. (2020) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Theis et al. (2015) Lucas Theis, Aäron van den Oord, and Matthias Bethge.  A note on the evaluation of generative models.  _arXiv preprint arXiv:1511.01844_ , 2015.
  * Tipping & Bishop (1999) Michael E Tipping and Christopher M Bishop.  Principal component analysis.  1999\.
  * Webster et al. (2019) Ryan Webster, Julien Rabin, Loic Simon, and Frédéric Jurie.  Detecting overfitting of deep generative networks via latent recovery.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11273–11282, 2019.
  * Wu et al. (2016) Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, and Roger Grosse.  On the quantitative analysis of decoder-based generative models.  _arXiv preprint arXiv:1611.04273_ , 2016.
  * Xiao et al. (2021) Zhisheng Xiao, Karsten Kreis, and Arash Vahdat.  Tackling the generative learning trilemma with denoising diffusion gans.  _arXiv preprint arXiv:2112.07804_ , 2021.
  * Zhang et al. (2018a) Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid.  Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems.  In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ , pp. 132–142, 2018a.
  * Zhang et al. (2018b) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 586–595, 2018b.
  * Zhao et al. (2020) Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.  Differentiable augmentation for data-efficient gan training.  _Advances in Neural Information Processing Systems_ , 33:7559–7570, 2020.
  * Zimmermann et al. (2021) Roland S Zimmermann, Lukas Schott, Yang Song, Benjamin A Dunn, and David A Klindt.  Score-based generative classifiers.  _arXiv preprint arXiv:2110.00473_ , 2021.

##  Appendix A Brief Explanation of the AIS Algorithm

Let $f(z)$ be a target un-normalized distribution. An AIS chain is defined by
an initial distribution $Q_{0}(z)=q_{0}(z)/Z_{0}$ whose normalization
coefficient is known, together with $T$ intermediate distributions
$Q_{1}(z),\cdots,Q_{T}(Z)$ such that $Q_{T}(z)=q_{T}(z)/Z_{T}=f(z)/Z_{T}$.
Each step of the chain further requires an MCMC transition operator
$\mathcal{T}_{t}$ which keeps $Q_{t}(z)$ invariant, such as Hamiltonian Monte-
Carlo (HMC). Beginning with a sample from the initial distribution $z_{0}\sim
Q_{0}(z)$ and setting $w_{0}=1$, AIS iteratively carries out the following
steps:

$w_{t}=w_{t-1}\cdot\frac{q_{t}(z_{t-1})}{q_{t-1}(z_{t-1})}\qquad\qquad
z_{t}\sim\mathcal{T}_{t}\left(z|z_{t-1}\right)$ (7)

The importance weights $w_{T}$ aggregated during the sampling procedure are an
unbiased estimate of the ratio of normalizing coefficients, such that
$\mathbb{E}\left[w_{T}\right]=Z_{T}/Z_{0}$.

Given a sample $x$, the likelihood can be calculated through AIS by setting
$f(z)=p(z)p_{\theta}(x|z)$ as the target distribution, such that
$p_{\theta}(x)$ is the corresponding normalization constant $\mathcal{Z}_{T}$.
In practice, the likelihood is estimated in log-space to avoid numerical
difficulties such as underflows. Calculating the log of the importance weights
as described above is straightforward, however Grosse et al. have shown that
doing so results in a stochastic lower bound of the log-likelihood. As the
number of intermediate steps $T$ increases, this stochastic lower bound
becomes tighter and converges to the true log-likelihood.

##  Appendix B Implementation Details

###  B.1 AIS Details

Figure 9: The estimated log-likelihood as a function of AIS steps, for
different GANs on CIFAR10 on random test samples. The log-likelihood of most
samples converges after 500 steps.

We follow the implementation of AIS from Wu et al. (publicly available in
GitHub), reimplemented in PyTorch. When possible, we used the same settings as
Wu et al.:

  * •

The transition operator we used was HMC with 10 leapfrog steps and a
Metropolis-Hastings (MH) adjustment. During sampling, the learning rate is
initialized to $5\cdot 10^{-2}$ and adjusted according to a moving average of
the MH rejection rate

  * •

During sampling, the intermediate distributions we used were:

$Q_{t}(z)\propto p(z)\cdot p_{\gamma}\left(x|\
G_{\theta}(z)\right)^{\beta_{t}}$ (8)

$\beta_{t}$ was annealed according to a sigmoidal schedule

#### Choice of Number of Steps

The bound on the log-likelihood approximated by AIS becomes tight and accurate
only as the number of intermediate distribution and number of chains grows,
respectively. However, AIS with many chains and intermediate distributions is
incredibly computationally costly. Due to these considerations we use a
relatively small number of intermediate distributions, while still ensuring
accurate enough results.

Each chain used 500 intermediate steps. This number is in stark contrast to
the 10,000 iterations used by Wu et al.. We chose this number by running
multiple AIS chains with a differing number of intermediate steps, plotting
the estimated log-likelihood as a function of AIS steps, as shown in Figure 9.
The estimated log-likelihood typically stabilizes very close to the value
reached after 500 iterations. The difference between the converged value and
the one after 500 steps is much smaller than the resolution of log-likelihoods
we are looking at, so this is a compromise between accuracy and computational
cost.

Further justification for this is due to the comparison between AIS and GAN
inversion in terms of gradient steps. Because of the leapfrog steps, a single
iteration of AIS is similar to 10 gradient steps in GAN inversion. In all of
our experiments, $\sim$750 iterations were enough to converge during
inversion, well below the 5000 gradient steps used during the AIS procedure.

###  B.2 Reconstruction through GAN Inversion

There is a vast literature on the best way to reconstruct test images using
GANs, also called _GAN inversion_. In this work we used a simple, albeit
rather costly, approach in order to find the best possible reconstruction.

We used an optimization approach towards GAN inversion, using ADAM as the
optimizer and a cosine schedule (similar to the scheme used by Sauer et al. in
their implementation) for $\sim 500-750$ iterations (which we found to usually
be many more iterations than required). To find better reconstructions, we
sampled $\sim 500$ images from the GAN and initialized the optimizer from the
latent code of the image closest to the input image in $\ell_{2}$ distance.
Furthermore, this process was repeated $\sim 4$ times for each image. Using
this GAN inversion scheme, we were typically able to invert images generated
by the GAN (and frequently images of a single color as well).

Finally, note that for all experiments with the StyleGAN variants, the
inversion took place in $\mathcal{Z}$ space, as the generative model is
defined in terms of this latent space and not the $\mathcal{W}/\mathcal{W+}$
spaces.

###  B.3 Observation Model

To calculate the GANs’ density, we assumed an additive Gaussian noise
observation model, with the same variance $\sigma^{2}$ for all samples. This
observation model was used by Wu et al. among others. The values of
$\sigma^{2}$ and their respective PSNR can be found in Table 2. As mentioned
in the main body of text, the value of $\sigma^{2}$ used was the variance of
the GAN’s reconstruction error. This choice of $\sigma^{2}$ maximizes the
likelihood the model gives to training data.

GAN | Dataset | Value of $\sigma^{2}$ | PSNR
---|---|---|---
BigGAN | CIFAR10 | 0.008 | 21.2
StyleGAN-XL | CIFAR10 | 0.018 | 17.4
BigGAN | ImageNet10 | 0.012 | 19.2
StyleGAN-XL | ImageNet10 | 0.021 | 16.8
StyleGAN2-ADA | AFHQ | 0.019 | 17.2
Table 2: Table of variances used for $\sigma^{2}$ throughout the paper and
their respective PSNR.

##  Appendix C More Results

Figure 10: Log reconstruction errors of training vs generated examples for
more GANs. In all cases, the distribution of the training examples is
different than of the generated examples. The plot for StyleGAN-XL on ImageNet
is on a bigger subset of ImageNet classes than in figure 2, showing that this
result is broader than the 2 classes discussed in the main body of text.
Figure 11: More GAN inversions. Figure 12: Histograms of LLs for more GANs
than shown in the main text.

Figure 10 shows more histograms of the reconstruction errors for training
examples vs. images generated by GANs. In all cases, the distribution of the
reconstruction errors is different for train and generated examples,
indicating that for all GANs examined the training samples are not on the GAN
manifold.

Figure 11 shows more examples of reconstructions.

Figure 12 shows LL histograms for different image groups under different GANs.