# Formation of metastable phases by spinodal decomposition

Ricard Alert<EMAIL_ADDRESS>Departament de Física de la Matèria
Condensada, Universitat de Barcelona, Av. Diagonal 647, 08028 Barcelona, Spain
Universitat de Barcelona Institute of Complex Systems (UBICS), Universitat de
Barcelona, Barcelona, Spain  Pietro Tierno  Departament de Física de la
Matèria Condensada, Universitat de Barcelona, Av. Diagonal 647, 08028
Barcelona, Spain  Universitat de Barcelona Institute of Complex Systems
(UBICS), Universitat de Barcelona, Barcelona, Spain  Institut de Nanociència i
Nanotecnologia, Universitat de Barcelona, Barcelona, Spain  Jaume Casademunt
<EMAIL_ADDRESS>Departament de Física de la Matèria Condensada,
Universitat de Barcelona, Av. Diagonal 647, 08028 Barcelona, Spain
Universitat de Barcelona Institute of Complex Systems (UBICS), Universitat de
Barcelona, Barcelona, Spain

###### Abstract

Metastable phases may be spontaneously formed from other metastable phases
through nucleation. Here, we demonstrate the spontaneous formation of a
metastable phase from an unstable equilibrium by spinodal decomposition, which
leads to a transient coexistence of stable and metastable phases. This
phenomenon is generic within the recently introduced scenario of the
Landscape-Inversion Phase Transitions, that we experimentally realize as a
structural transition in a colloidal crystal. This transition exhibits a rich
repertoire of new phase-ordering phenomena, including the coexistence of two
equilibrium phases connected by two physically different interfaces. In
addition, this scenario enables the control of sizes and lifetimes of
metastable domains. Our findings open a new setting that broadens the
fundamental understanding of phase-ordering kinetics, and yield new prospects
of applications in materials science.

###### pacs:

64.75.Gh, 05.70.Fh, 64.70.K-

Upon a quench, namely a sudden change of the external conditions, a system is
initially in a nonequilibrium state. The processes of relaxation to the new
equilibrium state may be complex, specially if a phase transition boundary is
crossed and more than one phase is locally stable. The corresponding phase-
ordering processes and their kinetics have been intensively studied for
decades and are now a classical topic of nonequilibrium physics. Indeed,
phase-ordering kinetics is central to understand and control domain formation
in a wide range of materials, ranging from liquid mixtures or metal alloys to
structural or magnetic domains in solids, through liquid crystals, polymers,
and many soft-matter systems 1.

The dynamics of phase transitions often involves metastable phases, namely
states that are only transiently stable, since they relax to the actual
equilibrium by a finite-size perturbation. Examples are ubiquitous and include
diamond or supercooled liquid water, which are metastable respectively to
graphite and ice at room pressure. A given phase may become metastable upon a
change of thermodynamic variables, such as temperature, pressure, or magnetic
field, that modifies its relative stability, such as when liquid water is
supercooled. Subsequently, a transition to the stable equilibrium phase occurs
typically via nucleation, which requires overcoming an energy barrier to form
a growing nucleus of the final phase. In contrast, if the quench is such that
the initial phase is in unstable equilibrium, new equilibrium phases are
spontaneously generated by the relaxation dynamics. By this process, known as
spinodal decomposition, infinitesimal fluctuations directly grow to give rise
to domains of the final coexisting phases 2; 3; 4; 5.

Metastable phases may be generated de novo through nucleation from other
preexisting metastable phases, such as supercooled water giving rise to
metastable structures of ice. According to Ostwald’s step rule 6, this will
occur when the nucleation kinetics of the stable phase is slower than that of
an intermediate metastable phase. On the other hand, nonequilibrium metastable
states such as gels may form via a dynamic arrest of a spinodal decomposition
process 7; 8. However, to our knowledge, metastable equilibrium phases, i.e.
metastable states of possible equilibrium phases of the system, have never
been observed to appear spontaneously by spinodal decomposition, this is from
an unstable equilibrium phase. Here, we predict and experimentally verify the
direct, spontaneous formation of a metastable equilibrium phase upon a quench
into an unstable equilibrium.

This phenomenon is observed in a solid-solid transition of a two-dimensional
(2D) colloidal crystal made of paramagnetic particles. Colloidal systems have
proven to be very useful experimental models for studying the kinetics of
phase transitions 9. Specifically, several aspects of the dynamics of solid-
solid transitions were revealed by studies on colloidal crystals 10, including
the appearance of long-lived metastable structures 11; 12; 13, usually through
displacive transformations. Indeed, diffusive nucleation was only recently
observed in colloidal crystals, in the form of a two-stage, liquid-mediated
process 14; 15 that was later found in a metal 16.

From a fundamental point of view, the universal features of phase-ordering
processes are usually captured by time-dependent continuum models based on
coarse-grained free energy functionals 17; 2; 3; 4; 5. Recently, these
classical field models are being revisited to formulate a theory for phase
separation in active systems 18; 19. However, the fundamental theory of phase
separation in traditional, non-active systems has remained essentially
unchanged for decades. Herein, we report a battery of new phase-ordering
phenomena that have no counterpart within the classical theory. Our results
stem from a recently introduced, nonstandard scenario of phase transitions,
the so-called landscape-inversion phase transitions (LIPT)20, where the
periodic energy landscape of the system can be inverted by changing a single
parameter.

Figure 1: Landscape-inversion phase transition in dipolar colloids on a
periodic magnetic substrate. a-b, Experimental image and sketch of the dipolar
colloidal crystal at external magnetic fields $H$ lower and higher than the
substrate field $H_{\text{s}}=13$ kA m-1. Particles assemble into lines, on
top of the walls between oppositely magnetized domains (white and grey) of the
substrate, with spatial period $\lambda=2.6$ $\mu$m. The structural order is
described by the lattice angle $\alpha$. Scale bars, $20$ $\mu$m. c, The
energy landscape of the system completely inverts for $H>H_{\text{s}}$. This
induces a transition from the $\alpha_{\text{a}}$ to the $\alpha_{\text{b}}$
structure in the crystal 20. A metastable equilibrium structure,
$\alpha_{\text{m}}$, exists for $H<H_{\text{s}}$. Dashed lines illustrate the
identification $\alpha_{\text{a}}\leftrightarrow-\alpha_{\text{a}}$, since
these angles correspond to the same structure (see text). The energy scale is
$u_{0}\equiv\mu_{0}\chi^{2}a^{3}H_{\text{s}}^{2}$, with $\chi\sim 1$ the
magnetic susceptibility of the particles of radius $a=0.5$ $\mu$m.

Among the new phenomena, we highlight the existence of an asymmetric spinodal
decomposition whereby the system phase separates into two coexisting
equilibrium phases of different relative stability. This process leads to the
aforementioned formation of the metastable domains. We also predict that these
domains are subsequently eliminated by a front propagation mechanism, thus
differing from the self-similar domain coarsening that usually follows
spinodal decomposition. Moreover, we also show that the range of sizes and
lifetimes of the metastable domains, and thus the overall phase transition
kinetics, can be externally controlled by a magnetic field. Finally, we
further reveal the possibility that two coexisting stable phases are
simultaneously connected by two distinct interfaces, with different physical
properties such as interfacial tension.

##  I Results

###  I.1 The landscape-inversion phase transition

A phase transition scenario based on a complete inversion of the energy
landscape was recently discovered in a 2D crystal of paramagnetic colloidal
particles on top of a periodic magnetic substrate 20, and later found in a
suspension of magnetic and nonmagnetic particles within a ferrofluid of
tunable susceptibility 21. In the former realization, particles arrange along
parallel lines following the domain walls of a striped substrate. At these
lines, the substrate generates a magnetic field $\pm H_{\text{s}}$ that
magnetizes particles on consecutive lines in opposite directions (Fig. 1a).
Then, when a uniform external magnetic field $H$ is applied, particles on
consecutive lines acquire magnetic dipoles $m_{i}\propto H+H_{\text{s}}$ and
$m_{j}\propto H-H_{\text{s}}$. Thus, the dipolar interaction between these
particles yields a contribution $U_{ij}\propto m_{i}m_{j}\propto
H^{2}_{\text{s}}-H^{2}$ to the energy. Consequently, the application of an
external field $H>H_{\text{s}}$ causes the energy landscape to globally
invert. This induces a structural phase transition in the crystal, with the
lattice angle $\alpha$ as (nonconserved) order parameter, which is accompanied
by a magnetic transition from an antiferromagnetic to a ferromagnetic order
(Fig. 1). Moreover, without crossing the phase transition boundary at
$H=H_{\text{s}}$, the external field tunes the relative stability of the
different structures, and hence their dynamics, without modifying their
crystalline order $\alpha$, which can be independently tuned by changing the
density of particles 20.

Figure 2: Formation of a metastable phase. a, Radial distribution function
$g\left(r\right)$ of the system before ($t=4$ s, blue), during ($t=6.57$ s,
green), and after ($t=11$ s, red) the phase-ordering process induced by a
quench of the magnetic field from $H=3H_{\text{s}}/2$ to $H=0$ at $t=6$ s. The
theoretical positions of the main peaks of the initial and final equilibrium
structures are indicated by blue and red arrows, respectively, with black
arrows standing for peaks common to both lattices. The distance is divided by
the spatial semiperiod of the substrate, $\lambda/2=1.3$ $\mu$m. b, Time
evolution of the height of the secondary peak of the $g\left(r\right)$. Its
transient increase right after the quench is due to the spontaneous formation
of a metastable rectangular structure with $\alpha_{\text{m}}=0$, which
disappears afterwards. Colour lines indicate the average height of the peak at
equilibrium, both before and after the quench. Colour stars correspond to the
three radial distribution functions in a. c, Time evolution of the width of
the first peak of the $g\left(r\right)$ at a height $g=1.3$. After the quench,
this peak widens due to the appearance of a second peak of the final lattice
very close to the primary peak, which is not resolved experimentally. The
widening, this is the formation of the final equilibrium phase, occurs on the
same characteristic time as the disappearance of the metastable phase shown in
b. Therefore, the metastable domains are eliminated in favour of the stable
crystalline structure. All quantities are averaged over 15 realizations.

We remark that, for $H<H_{\text{s}}$, the equilibrium angle
$\alpha_{\text{a}}$ is equivalent to $-\alpha_{\text{a}}$ (Fig. 1c). This is
because, in the corresponding structure, particles on one line are equidistant
from the four neighbouring particles on the two nearest lines (sketch in Fig.
1a). Therefore, the structures with lattice angles $\alpha_{\text{a}}$ and
$-\alpha_{\text{a}}$ are exactly the same, and hence the identification
$\alpha_{\text{a}}\leftrightarrow-\alpha_{\text{a}}$. In contrast, for
$H>H_{\text{s}}$, the two equilibrium structures, with opposed lattice angles
$\alpha_{\text{b}}$ and $-\alpha_{\text{b}}$ (Fig. 1c), correspond to
different particle arrangements (sketch in Fig. 1b).

###  I.2 Formation of metastable domains by spinodal decomposition

We first focus on the phase-ordering kinetics associated to the LIPT by
suddenly switching off an external magnetic field $H=3H_{\text{s}}/2$. This
quench forces the crystal to transit from the rhomboidal structure with
$\alpha_{\text{b}}\approx 7^{\circ}$ (Fig. 1b) to that with
$\alpha_{\text{a}}\approx 25^{\circ}$ (Fig. 1a). We monitor the dynamics of
the structural rearrangement via the time evolution of the radial distribution
function $g\left(r\right)$ of the crystal, as shown in Fig. 2. Different
crystalline lattices are distinguished by their corresponding peaks in the
$g\left(r\right)$. In Fig. 2a, the initial $\alpha_{\text{b}}$ crystal (Fig.
1b) is distinguished by the presence of secondary peaks at $r\sim
1.4\,\lambda/2$ and $r\sim 2.25\,\lambda/2$ (blue). In turn, the final
$\alpha_{\text{a}}$ crystal (Fig. 1a) features a wider first peak and no
secondary peaks (red). The theoretical positions of the main peaks of both
structures are indicated by arrows in Fig. 2a. From them, we infer that the
widening of the first peak in the $\alpha_{\text{a}}$ structure indeed results
from the appearance of a second peak at $r\sim 1.1\,\lambda/2$, which can not
be resolved from the primary one within the experimental resolution.

However, Fig. 2a also reveals that the structural transition does not occur
via a homogeneous relaxation but that it rather involves a nontrivial phase-
ordering process. This follows from the fact that the secondary peaks of the
initial structure first increase in height (green) before disappearing (see
also Fig. 2b). This corresponds to the transient formation of the metastable
rectangular structure with $\alpha_{\text{m}}=0$ (Fig. 1c), which contributes
peaks at $r=\sqrt{2}\,\lambda/2$ and $r=\sqrt{5}\,\lambda/2$. However, the
stable $\alpha_{\text{a}}$ structure also starts forming right after the
quench, as indicated by the widening of the first peak of the
$g\left(r\right)$ (Fig. 2c). Therefore, both phases transiently coexist for
$\sim 2$ s, after which the system reaches the final homogeneous
$\alpha_{\text{a}}$ phase.

Figure 3: Formation of a metastable phase by spinodal decomposition, and front
propagation. a, Simulation snapshots of the phase-ordering process resulting
from a quench from $H>H_{\text{s}}$ to $H<H_{\text{s}}$ (see Supplementary
Movie 1). Color indicates the order parameter field
$\varphi\left(\mathbf{r},t\right)$ of the model. The system, initially at the
unstable state $\varphi=30^{\circ}$ (dark blue), undergoes an asymmetric
spinodal decomposition that spontaneously generates both stable
$\varphi=90^{\circ}$ (red) and metastable $\varphi=0^{\circ}$ (light blue)
domains. The latter are subsequently invaded by the former via propagating
fronts. $\Gamma^{-1}$ and $\sqrt{\kappa}$ define the time and length units in
the simulations, respectively. Scale bar, $50\sqrt{\kappa}$. b, Model energy
landscape of the LIPT, Eq. 2. The dashed part illustrates the periodicity of
the potential. c, Dispersion relation of the unstable state, Eq. 4, for
different values of the magnetic field. The field controls the region of
unstable modes $q<q_{\text{c}}$ and thus the range of sizes of the forming
domains. d, Velocity of planar fronts as a function of the landscape-inversion
parameter $h\equiv 1-H^{2}/H_{\text{s}}^{2}$ from simulations, with a fit of
the predicted scaling $v\propto h^{1/2}.$ e, Time evolution of the area
fraction covered by the metastable state for different values of the magnetic
field. This graph shows the formation and subsequent elimination of the
metastable state, with the overall phase transition dynamics controlled by the
external magnetic field.

The theoretical model of the LIPT (Fig. 1c) allows to predict the phase-
ordering processes associated to the inversion of the energy landscape. A
simple quench from $H>H_{\text{s}}$ to $H<H_{\text{s}}$ leaves the system at
an unstable equilibrium state, from which it phase separates into domains of
two locally stable phases. However, in contrast to standard spinodal
decomposition 2; 3; 4; 5, the two coexisting phases have different relative
stability. Hence, we term this phase separation process “asymmetric spinodal
decomposition”. Thereby, half of the system initially evolves towards a
metastable phase that was not present in the initial condition. Therefore, in
the light of the model, the experimental data conclusively demonstrate the
direct spontaneous formation of a metastable phase by spinodal decomposition.

Hitherto, metastable phases could only form de novo from other metastable
phases, either through diffusive nucleation or displacive transformations.
Indeed, a mechanism for the formation of metastable domains upon the
nucleation of the stable phase was proposed based on a dynamical instability
of the fronts propagating from stable into unstable states, both for
nonconserved 22; 23; 24 and conserved 25; 26 order parameters. In any case,
the formation of the metastable phases was not generic but depended on the
depth or rate of the quench. In contrast, within the LIPT scenario, metastable
domains are spontaneously formed by spinodal decomposition, thus directly from
the unstable state, simultaneously to the formation of stable domains. In
other words, metastable domains are naturally formed at the initial stages of
the phase separation process, as a direct consequence of the energy landscape
inversion. In addition, their appearance is completely generic, independent of
the quench depth.

To further investigate the novel phase-ordering processes of the LIPT, we
formulate a time-dependent Ginzburg-Landau-like model for its dynamics (see
Methods). Stochastic simulations of such a model clearly illustrate the
spontaneous formation of metastable domains upon the inversion of the energy
landscape (Fig. 3a-b). The dispersion relation of the unstable state, plotted
in Fig. 3c, gives the minimal size of the forming domains, $\sim
2\pi/q_{\text{c}}$, which is controlled by the magnetic field at which the
quench is performed.

###  I.3 Front propagation

The generated metastable domains coexist with globally stable ones, and
therefore they are subsequently invaded and eliminated by fronts of the stable
state. Consequently, the late stages of the phase-ordering process do not
proceed by a self-similar coarsening as in usual spinodal decomposition 3; 4;
5. Thus, domain dynamics is not governed by interfacial curvature but rather
by front propagation, and hence by the free energy difference between the
stable and metastable phases 27; 28; 29. This enables the external control of
the domain dynamics by means of the magnetic field.

The dependence of the speed of the fronts on the value of the magnetic field
upon the quench can be deduced from the dynamics of the order parameter field
28; 29. Neglecting curvature corrections, the interface speed, width, and
tension are thereby predicted to scale as $v\propto h^{1/2}$, $\delta\propto
h^{-1/2}$, and $\sigma\propto h^{1/2}$, respectively (see Methods). Here we
have defined the landscape-inversion parameter $h\equiv
1-H^{2}/H_{\text{s}}^{2}$, which changes sign at the transition point. Then,
we measure the speed of planar fronts in simulations for several values of
$h$. The numerical results agree with the predicted scaling, as shown in Fig.
3d.

We note that the scaling $v\propto h^{1/2}$ contrasts with the prediction of a
linear scaling $v\propto\epsilon$ of the front speed with the distance from
the transition point,
$\epsilon\equiv\left(T-T_{\text{c}}\right)/T_{\text{c}}$, for first-order
phase transitions 30. For the LIPT, $h$ directly measures the distance from
the transition point at $h=0$. In fact, a square-root scaling
$v\propto|\epsilon|^{1/2}$ like the one we find was predicted for fronts
associated to second-order transitions 30. Thus, despite being discontinuous,
the LIPT shares some dynamical properties with second-order phase transitions.

Finally, since the external magnetic field controls both the range of sizes of
the generated domains and their rate of disappearance, it actually tunes the
overall phase transition dynamics. Higher fields, closer to the transition
threshold $H_{\text{s}}$, imply a wider distribution of domain sizes (Fig. 3c)
and slower propagating fronts, so that more metastable phase is generated and
it lives longer. This is shown in Fig. 3e, which reports the fraction of area
covered by the metastable state in simulations at different magnetic fields.

Last, it is also worth mentioning that, in principle, the inversion of the
energy landscape could be useful as a generic probe for front propagation
problems (see Supplementary Note).

###  I.4 Phase coexistence with two physically different interfaces

We next consider the phase-ordering process upon the opposite quench, from
$H<H_{\text{s}}$ to $H>H_{\text{s}}$. Again, as illustrated in Fig. 4a, the
inversion of the energy landscape from the equilibrium state
$\varphi=90^{\circ}$ at $H<H_{\text{s}}$ leaves the system at an unstable
state, from which it undergoes spinodal decomposition. However, this process
is symmetric for the present situation, leading to two equilibrium phases,
$\varphi=\pm 30^{\circ}$, with equal energy. As a consequence, interfaces
between these two phases can not propagate as fronts, and they only move under
curvature. Their speed is locally proportional to their curvature, following
the Allen-Cahn law 31. This behaviour leads to a usual curvature-driven
coarsening process characterized by a scaling regime of the typical domain
size 31; 32; 2; 3; 4; 5 $R\left(t\right)\propto t^{1/2}$.

Figure 4: Two physically different interfaces connect the two equilibrium
phases. a, Model energy landscape of the LIPT, Eq. 2. The dashed part
illustrates the periodicity of the potential. b, Simulation snapshots of the
phase-ordering process that results from a quench from $H<H_{\text{s}}$ to
$H>H_{\text{s}}$ (see Supplementary Movie 2). Color indicates the order
parameter field $\varphi\left(\mathbf{r},t\right)$ of the model. From an
initial coexistence of a stable $\varphi=90^{\circ}$ (red) and a metastable
$\varphi=0^{\circ}$ (light blue) domains, the system undergoes a double
spinodal decomposition. This leads to the formation of two physically
different interfaces (A and B) connecting the two equilibrium phases
$\varphi=\pm 30^{\circ}$ (dark blue and green, respectively). $\Gamma^{-1}$
and $\sqrt{\kappa}$ define the time and length units in the simulations,
respectively. Scale bar, $50\sqrt{\kappa}$.

Now, because of the periodicity of the free energy, the two equilibrium phases
are indeed connected through two distinct interfaces, with different profiles
and interfacial tensions (Fig. 4a). Remarkably, the system forms only the most
energetic of both interfaces when quenched from the equilibrium state
$\varphi=90^{\circ}$ at $H<H_{\text{s}}$. In contrast, if the system presents
stable-metastable coexistence when quenched to $H>H_{\text{s}}$, it naturally
undergoes a double spinodal decomposition leading to the formation of both
interfaces, as illustrated in Fig. 4b. Simulations also show that the most
energetic interface disappears in favour of the least energetic one when both
get into contact (Supplementary Movie 3).

##  II Discussion

Our findings open a new scenario of phase-ordering kinetics, associated to the
landscape-inversion phase transition (LIPT), in which several unexpected
phenomena take place. In particular, these include a new mechanism to form
metastable phases, namely by spinodal decomposition. By this mechanism,
metastable phase formation is robust within the LIPT scenario, occurring
regardless of the amplitude or the rate of the quench. Remarkably, the LIPT
never leads to nucleation despite the existence of a metastable state in the
free energy. A summary of the phase-ordering processes allowed by this
nonstandard scenario is provided in Fig. 5a. There, the case of a conserved
order parameter has also been included for completeness (see Supplementary
Discussion for details). Additionally, a concise summary of the phase-ordering
processes associated to the classical A and B models of phase transition
dynamics 5 is given in Fig. 5b for comparison (see Supplementary Discussion).

Figure 5: Comparison of the phase-ordering kinetics of the LIPT and classical
scenarios. Initial stages of phase separation (domain formation) are indicated
in red, and late stages (domain growth) are indicated in blue. Key properties
of some processes are specified (shaded). Details are given in the
Supplementary Discussion. Only the results of quenches to temperatures below
critical, $T<T_{\text{c}}$, are shown for the classical models, since quenches
to $T>T_{\text{c}}$ lead to a homogeneous relaxation without phase separation.
In the LIPT scenario, phase separation processes occur at both sides of the
transition.

While we experimentally demonstrate the unstable-to-metastable phase change,
the rest of our predictions remain to be experimentally verified due to
limitations of our setup, such as the relatively small system size, the
presence of vacancies in the crystals, and the lack of precise control of the
in-line particle density. However, the LIPT scenario has been already realized
in another physical system 21. Thus, searching for other systems that broaden
the experimental possibilities or finding a LIPT with conserved order
parameter remain appealing open challenges. In this sense, our results may
open new research avenues in the field of phase transition dynamics and may
foster, for instance, the exploration of nonstandard routes to phase
separation.

With regard to colloidal materials, in addition to enabling novel routes for
solid-solid transitions, our system also provides external magnetic control
over the phase-ordering dynamics of 2D crystals 10. Understanding and
controlling the ordering and relative stability of different crystalline
structures could indeed be relevant for applications of colloidal crystals
themselves 33; 34, such as photonic band-gap materials, but also of atomic
alloys in the nanotechnological domain 35; 36. In this respect, our work could
provide a basis for the search of new self-assembly strategies, in particular
related to the possibility of tuning the spatial organization of crystalline
structures in 2D materials.

##  III Methods

###  III.1 Experiments

The periodic magnetic substrate is a uniaxial ferrite garnet film grown by
liquid phase epitaxy. An aqueous suspension of paramagnetic colloidal
particles (Dynabeads Myone) is deposited on top. The external magnetic field
is generated by custom-made coils. Particle positions are tracked by a custom-
made software from video microscopy recordings at $60$ Hz over an area of
$140\times 105$ $\mu$m2.

###  III.2 Dynamical field model of the LIPT

We build a Ginzburg-Landau-like model for the dynamics of the LIPT by
formulating a coarse-grained free energy functional of a scalar order
parameter $\varphi\left(\mathbf{r},t\right)$ 37:

$F\left[\varphi\right]=u\int_{\Omega}\left(f_{0}\left[\varphi\right]+\frac{\kappa}{2}\left(\mathbf{\nabla}\varphi\right)^{2}\right)d^{d}\mathbf{r}.$
(1)

where $u$ and $\kappa$ are phenomenological parameters governing the energy
scale and the spatial coupling respectively.

The dimensionless local free energy density $f_{0}$ must capture the essential
features of the actual potential of the LIPT (Fig. 1c). Therefore, $f_{0}$
must feature, at least, a stable and a metastable state, and it must allow for
a complete inversion under change of a control parameter. The order parameter
must have the topology of an angle, hence identifying the two extremum values
of its range ($\alpha_{\text{a}}\leftrightarrow-\alpha_{\text{a}}$ in Fig. 1c,
and $90^{\circ}\leftrightarrow-90^{\circ}$ in this model, Fig. 3b). A model
free energy including all these ingredients is

$f_{0}\left[\varphi\right]=h\sin^{2}\varphi\left(1-A\sin^{2}\varphi\right),$
(2)

where $h$ and $A$ are two control parameters. Here, we have defined the
landscape-inversion parameter $h\equiv 1-H^{2}/H_{\text{s}}^{2}$, while the
role of the particle density is played by $A$, which we take to be $A=2$.
Indeed, $0<h\leq 1$ corresponds to the $0\leq H<H_{\text{s}}$, and $h<0$
corresponds to $H>H_{\text{s}}$, so that $h$ changes sign to invert the energy
landscape. In turn, $A>1$ controls the relative stability of the stable
$\varphi=90^{\circ}$ and metastable $\varphi=0$ states for $h>0$, and the
energy barrier between the two degenerated states at
$\varphi=\pm\arcsin\sqrt{1/\left(2A\right)}=30^{\circ}$ for $h<0$.

Then, the dynamics of the nonconserved order parameter is specified by the
time-dependent Ginzburg-Landau equation

$\frac{\partial\varphi}{\partial t}=-\frac{\Gamma}{u}\frac{\delta
F\left[\varphi\right]}{\delta\varphi}+\xi=\Gamma\left(-\frac{\partial
f_{0}}{\partial\varphi}+\kappa\nabla^{2}\varphi\right)+\xi,$ (3)

where $\Gamma$ is a kinetic coefficient and $\xi\left(\mathbf{r},t\right)$ is
a Gaussian white noise field with
$\left\langle\xi\left(\mathbf{r},t\right)\xi\left(\mathbf{r}^{\prime},t^{\prime}\right)\right\rangle=2D\delta\left(\mathbf{r}-\mathbf{r}^{\prime}\right)\delta\left(t-t^{\prime}\right)$.
A linear stability analysis of the unstable states $\varphi=\pm 30^{\circ}$ at
$H<H_{\text{s}}$, leads to their dispersion relation:

$\omega\left(q\right)=\Gamma\left(3h-\kappa q^{2}\right),$ (4)

where $q$ is the wave vector.

Finally, an extended model including the possibility of liquid interfaces
between crystalline regions is introduced in the Supplementary Method
(Supplementary Fig. 1), which is closer to the experimental situations of Fig.
1 or liquid-mediated transitions 14.

###  III.3 Front dynamics

The properties and motion of the interfaces follow from the order parameter
dynamics 28; 29. Particularly, a closed equation for the profile of a steadily
moving flat interface can be derived from Eq. 3. In the comoving frame of
reference of a planar front advancing at a constant velocity $\mathbf{v}$, the
evolution of the order parameter can be written as
$\partial_{t}\varphi=-\mathbf{v}\cdot\mathbf{\nabla}\varphi$, so that the
time-dependent Ginzburg-Landau equation Eq. 3 becomes

$-v\frac{d\varphi}{dz}=\Gamma\left(-\frac{\partial
f_{0}}{\partial\varphi}+\kappa\frac{d^{2}\varphi}{dz^{2}}\right),$ (5)

where we have taken $\mathbf{v}=v\hat{\mathbf{z}}$ and disregarded
fluctuations. This equation can not be solved analytically for the model free
energy functional of the LIPT ($f_{0}\left[\varphi\right]$ in Eq. 2), but a
dimensional analysis of Eq. 5 predicts the interface width and speed to scale
as $\delta\propto\sqrt{\kappa/h}$ and $v\propto\Gamma\sqrt{\kappa h}$,
respectively. In addition, the projection of Eq. 5 onto the Goldstone mode
$d\varphi/dz$ 29 also predicts a scaling
$\sigma\propto\left(A-1\right)\sqrt{h}$ for the interfacial tension.

###  III.4 Simulation details

In all cases, numerical results are obtained from simulations of Eq. 3 under
periodic boundary conditions, following usual stochastic algorithms for the
noise field 38, and with a rescaled time step $\Gamma\Delta t=5\cdot 10^{-3}$.

Results in Fig. 3a,d,e are obtained on a $200\times 200$ grid in simulation
units $\sqrt{\kappa}$, and for $D=0.02\,\kappa\Gamma$. A quench from $h=-2$ to
$h=0.25$ is applied at time $t=0$. In Fig. 3d, the fit of the predicted
scaling $v=b\Gamma\sqrt{\kappa h}$ to the simulation results gives the
prefactor $b=1.36\pm 0.01$. In Fig. 3e, the simulation grid points
contributing to the area covered by the metastable state are those featuring a
value of the order parameter within its free energy basin. The limits of this
basin are set by the inflection points of the local free energy (see Fig. 3b),
which define the corresponding region of local thermodynamic stability, giving
$\varphi\approx\pm 20^{\circ}$.

In turn, simulations in Fig. 4b are performed on a $100\times 100$ grid, for
the same value of $D$, and the quench is applied from $h=1$ to $h=-2$.

###  III.5 Data availability

The data that support the findings of this study are available from the
corresponding authors upon request.

## References

  * (1) Onuki, A.  _Phase transition dynamics_ (Cambridge University Press, 2002).
  * (2) Binder, K.  Theory of first-order phase transitions.  _Reports Prog. Phys._ 50, 783–859 (1987).
  * (3) Langer, J.  An introduction to the kinetics of first-order phase transition.  In Godrèche, C. (ed.) _Solids far from equilibrium_ , chap. 3 (Cambridge University Press, 1992).
  * (4) Bray, A.  Theory of phase-ordering kinetics.  _Adv. Phys._ 43, 357–459 (1994).
  * (5) Chaikin, P. M. & Lubensky, T. C.  _Principles of condensed matter physics_ (Cambridge University Press, 1995).
  * (6) Ostwald, W.  Studien über die Bildung und Umwandlung fester Körper.  _Z. Phys. Chem._ 22, 289–330 (1897).
  * (7) Zaccarelli, E.  Colloidal gels: equilibrium and non-equilibrium routes.  _J. Phys. Condens. Matter_ 19, 323101 (2007).
  * (8) Lu, P. J. _et al._ Gelation of particles with short-range attraction.  _Nature_ 453, 499–503 (2008).
  * (9) Anderson, V. J. & Lekkerkerker, H. N. W.  Insigths into phase transition kinetics from colloid science.  _Nature_ 416, 811–815 (2002).
  * (10) Li, B., Zhou, D. & Han, Y.  Assembly and phase transitions of colloidal crystals.  _Nat. Rev. Mater._ 1, 15011 (2016).
  * (11) Yethiraj, A., Wouterse, A., Groh, B. & van Blaaderen, A.  Nature of an Electric-Field-Induced Colloidal Martensitic Transition.  _Phys. Rev. Lett._ 92, 058301 (2004).
  * (12) Assoud, L. _et al._ Ultrafast Quenching of Binary Colloidal Suspensions in an External Magnetic Field.  _Phys. Rev. Lett._ 102, 238301 (2009).
  * (13) Mohanty, P. S., Bagheri, P., Nöjd, S., Yethiraj, A. & Schurtenberger, P.  Multiple Path-Dependent Routes for Phase-Transition Kinetics in Thermoresponsive and Field-Responsive Ultrasoft Colloids.  _Phys. Rev. X_ 5, 011030 (2015).
  * (14) Peng, Y. _et al._ Two-step nucleation mechanism in solid-solid phase transitions.  _Nat. Mater._ 14, 101–108 (2015).
  * (15) Qi, W., Peng, Y., Han, Y., Bowles, R. K. & Dijkstra, M.  Nonclassical Nucleation in a Solid-Solid Transition of Confined Hard Spheres.  _Phys. Rev. Lett._ 115, 185701 (2015).
  * (16) Pogatscher, S., Leutenegger, D., Schawe, J. E. K., Uggowitzer, P. J. & Löffler, J. F.  Solid-solid phase transitions via melting in metals.  _Nat. Commun._ 7, 11113 (2016).
  * (17) Hohenberg, P. & Halperin, B.  Theory of dynamic critical phenomena.  _Rev. Mod. Phys._ 49, 435–479 (1977).
  * (18) Wittkowski, R. _et al._ Scalar $\phi$(4) field theory for active-particle phase separation.  _Nat. Commun._ 5, 4351 (2014).
  * (19) Speck, T., Bialké, J., Menzel, A. M. & Löwen, H.  Effective Cahn-Hilliard Equation for the Phase Separation of Active Brownian Particles.  _Phys. Rev. Lett._ 112, 218304 (2014).
  * (20) Alert, R., Casademunt, J. & Tierno, P.  Landscape-Inversion Phase Transition in Dipolar Colloids: Tuning the Structure and Dynamics of 2D Crystals.  _Phys. Rev. Lett._ 113, 198301 (2014).
  * (21) Carstensen, H., Kapaklis, V. & Wolff, M.  Phase formation in colloidal systems with tunable interaction.  _Phys. Rev. E_ 92, 012303 (2015).
  * (22) Bechhoefer, J., Löwen, H. & Tuckerman, L.  Dynamical Mechanism for the Formation of Metastable Phases.  _Phys. Rev. Lett._ 67, 1266–1269 (1991).
  * (23) Celestini, F. & ten Bosch, A.  Importance of interfacial coupling on the formation and growth of metastable phases.  _Phys. Rev. E_ 50, 1836–1842 (1994).
  * (24) Iwamatsu, M.  Cell dynamics modeling of phase transformation and metastable phase formation.  _J. Alloys Compd._ 504, S538–S542 (2010).
  * (25) Evans, R. M. L., Poon, W. C. K. & Cates, M. E.  Role of metastable states in phase ordering dynamics.  _Europhys. Lett._ 38, 595–600 (1997).
  * (26) Gránásy, L. & Oxtoby, D. W.  Cahn–Hilliard theory with triple-parabolic free energy. II. Nucleation and growth in the presence of a metastable crystalline phase.  _J. Chem. Phys._ 112, 2410 (2000).
  * (27) Chan, S.-K.  Steady-state kinetics of diffusionless first order phase transformations.  _J. Chem. Phys._ 67, 5755 (1977).
  * (28) Desai, R. C. & Kapral, R.  _Dynamics of self-organized and self-assembled structures_ (Cambridge University Press, 2009).
  * (29) Pismen, L. M.  _Patterns and interfaces in dissipative dynamics_ (Springer, 2006).
  * (30) Cladis, P. _et al._ Dynamical test of phase transition order.  _Phys. Rev. Lett._ 62, 1764–1767 (1989).
  * (31) Allen, S. M. & Cahn, J. W.  A microscopic theory for antiphase boundary motion and its application to antiphase domain coarsening.  _Acta Metall._ 27, 1085–1095 (1979).
  * (32) Safran, S.  Domain Growth of Degenerate Phases.  _Phys. Rev. Lett._ 46, 1581–1584 (1981).
  * (33) Khalil, K. S. _et al._ Binary colloidal structures assembled through Ising interactions.  _Nat. Commun._ 3, 794 (2012).
  * (34) Mahynski, N. A., Panagiotopoulos, A. Z., Meng, D. & Kumar, S. K.  Stabilizing colloidal crystals by leveraging void distributions.  _Nat. Commun._ 5, 4472 (2014).
  * (35) Lu, J. _et al._ Order-disorder transition in a two-dimensional boron-carbon-nitride alloy.  _Nat. Commun._ 4, 2681 (2013).
  * (36) Duerloo, K.-A. N., Li, Y. & Reed, E. J.  Structural phase transitions in two-dimensional Mo- and W-dichalcogenide monolayers.  _Nat. Commun._ 5, 4214 (2014).
  * (37) Cahn, J. W. & Hilliard, J. E.  Free Energy of a Nonuniform System. I. Interfacial Free Energy.  _J. Chem. Phys._ 28, 258 (1958).
  * (38) See$\beta$elberg, M. & Petruccione, F.  Numerical integration of stochastic partial differential equations.  _Comput. Phys. Commun._ 74, 303–315 (1993).

##  IV Acknowledgements

R.A. acknowledges support from Fundació “La Caixa”, and from the University of
Cambridge under the grant ITN-COMPLOIDS 234810 of the 7th Framework Programme
of the EU. P.T. acknowledges support from the European Research Council under
project No. 335040 and MINECO under project RYC-2011-07605. The authors
acknowledge support from MINECO under project FIS2013-41144-P and Generalitat
de Catalunya under project 2014-SGR-878.

##  V Author contributions

All authors conceived and planned the research. R.A. developed the theory and
performed the simulations. P.T. performed the experiments. R.A. and P.T.
analyzed the experimental data. J.C. supervised the work. All authors
discussed and interpreted results, and wrote the paper.

##  VI Additional information

Supplementary information is available in the online version of the paper.
Correspondence and requests should be addressed to R.A.

##  VII Competing financial interests

The authors declare no competing financial interests.


# SWAGAN: A Style-based WAvelet-driven Generative Model

Rinon Gal
Tel-Aviv University  Dana Cohen
Tel-Aviv University  Amit Bermano
Tel-Aviv University  Daniel Cohen-Or
Tel-Aviv University

###### Abstract

In recent years, considerable progress has been made in the visual quality of
Generative Adversarial Networks (GANs). Even so, these networks still suffer
from degradation in quality for high-frequency content, stemming from a
spectrally biased architecture, and similarly unfavorable loss functions. To
address this issue, we present a novel general-purpose Style and WAvelet based
GAN (SWAGAN) that implements progressive generation in the frequency domain.
SWAGAN incorporates wavelets throughout its generator and discriminator
architectures, enforcing a frequency-aware latent representation at every step
of the way. This approach yields enhancements in the visual quality of the
generated images, and considerably increases computational performance. We
demonstrate the advantage of our method by integrating it into the SyleGAN2
framework, and verifying that content generation in the wavelet domain leads
to higher quality images with more realistic high-frequency content.
Furthermore, we verify that our model’s latent space retains the qualities
that allow StyleGAN to serve as a basis for a multitude of editing tasks, and
show that our frequency-aware approach also induces improved downstream visual
quality.

##  1 Introduction

Image synthesis is a cornerstone of modern deep learning research, owing to
the applicability of deep generative networks to a multitude of image related
tasks. Such tasks range from content generation to novel view synthesis,
rendering, image in-painting, super resolution or modifications of specific
image properties [31, 18, 16, 25]. The performance of such downstream tasks
has seen rapid improvements in recent years, due in part to the significant
improvement in the visual quality of generative models, and in particular
Generative Adversarial Networks (GANs).

Figure 1: By working directly in the wavelet-domain, rather than image or
feature space, we can alleviate the spectral bias of neural networks and
successfully generate high frequency data where other models fail. Our model
can produce patterns that elude state-of-the-art models such as StyleGAN2 even
in an over-fitting setup, where the training set includes only a single image.
We show (from left to right): the original image and the outputs of StyleGAN2
and SWAGAN (our model), respectively, after being trained for 24 hours on each
image. Zooming in is encouraged.

However, both generative models and their associated downstream tasks
consistently suffer from a common flaw - their performance degrades rapidly
when handling high frequency content [6, 7, 3]. This demeaned performance
often manifests as blurriness, or the lack of sharp edges around finer image
content. Prior works have attributed these shortcomings to the nature of the
network architecture and common cost functions. The oft-used mean squared
reconstruction error, for example, is much less sensitive to small
perturbations, thus leading networks that utilize this loss to favor the
accuracy of low frequency and large-scale content over finer features [10]. In
addition, recent works have pointed to an inherent spectral bias in neural
networks that manifests as a frequency-dependent learning speed [27].

A straightforward approach to combat this shortcoming is to push resolutions
to new heights. By doing so, previously minute details become more dominant
and thus are better captured by the traditional losses. This approach,
however, does little to overcome the frequency-dependent learning rate.

The current state-of-the-art results for high resolution generation are
achieved by style-based models [16, 17]. These models employ a new approach to
content generation - rather than progressively growing a set of features and
converting them to an image representation at the very end, they first
generate a low resolution image and then grow it in size in a hierarchical
manner [15, 14]. This approach affords additional regularization and stability
to the trained model and has been shown to achieve improved visual results.
While these models lead to remarkable strides in visual quality, they rely on
networks which are larger and notoriously expensive to train, with the
development of recent state-of-the-art networks requiring upwards of 50 years
of computation time using a top of the line GPU [17].

An alternative approach to addressing the high-frequency gap is to directly
employ frequency-aware training objectives, for example by empowering a
discriminator to consider not only an image, but also a spectral decomposition
of it [6, 7, 3]. Others opt to generate high-frequency content directly by
predicting a full set of wavelet coefficients. Such an approach has been used
successfully on a range of tasks, from image super resolution [10, 36] to
compressed video enhancement [32]. However, to the best of our knowledge,
these prior works focus on training a constrained generator for a specific
task (e.g., face super resolution), and none of them are adapted to the modern
approach of hierarchical image growth, rather than purely feature-based
growth.

We argue that frequency-based representations hold another advantage that is
often overlooked or squandered due to misuse. By using an appropriate
representation, the network can be biased to affect high-frequency
modifications in some target domain through low frequency modifications of the
representation. Such bias can help bypass the need to learn any high-frequency
functions and thus avoid the spectral bias of neural networks.

In our work, we propose to marry the core ideas from these different
approaches. Our goal is to train a general purpose image generator. However,
rather than creating an image by predicting deep features or the color content
of each pixel in a progressively growing series of resolutions, we instead
propose to work directly in frequency space by predicting coefficients of
basis functions of increasing frequency. Similarly, we propose to make use of
a discriminator that compares not only the pixel content of generated images
and real samples, but also their frequency space decomposition. These
frequency domain considerations are incorporated not only as a training goal,
but are baked directly into the architecture of our model at each resolution
step.

We demonstrate the benefits of our approach using the popular StyleGAN2
framework [17] (see Figure 3). We show that by adapting its architecture to
work directly in a wavelet-based space, we achieve improved visual quality and
more realistic content in the high-frequency range. By working directly in the
wavelet-domain, rather than image or feature space, we alleviate the spectral
bias of neural networks, converge with fewer epochs, and successfully generate
high frequency data where other models fail (see Figure 1)

Figure 2:  Our style-based generator architecture predicts wavelet-
coefficients at increasing resolution scales.

Another surprising benefit of our approach is a significant reduction in
training iteration and inference times. Our experiments show that the wavelet-
based model not only converges with fewer epochs, but also that the tradeoff
between spatial dimensions and channel depth inherent to wavelet
decomposition, results in a significant speedup of network computations. This
behavior can be dependent on the specific details of the convolution
operations implemented by each framework, but popular implementations such as
GEMM or Winograd variants show only a mild increase in computation time when
increasing the number of filters [12].

Finally, we show that our model’s latent space can support the editing
operations that are commonplace in StyleGAN-based works. We show that our
high-frequency performance can be successfully transmitted downstream, and
that by using our model one can avoid many of the blurry artifacts present in
prior works. Our experiments indicate that shifting content generation to the
wavelet domain offers a significant set of advantages, with no apparent
drawbacks.
In summary, our key contributions are:

  * •

A hierarchical, wavelet-based approach to image generation that achieves
improved visual fidelity and more realistic spectra.

  * •

An approach to network design that reduces the natural spectral-bias
inclination of networks, resulting in a significant reduction of the
computational budget required to train high resolution generative models.

Our code and trained models will be made public.

##  2 Background and related work

###  2.1 Generative Adversarial Networks

Generative Adversarial Networks (GANs), first proposed by Goodfellow et al.
[9], have been successfully used in a wide range of image synthesis tasks,
ranging from general content creation to more specific tasks such as image
super resolution and in-painting. Due to their ubiquity, considerable effort
has gone into improving the quality of images generated by such models.
Earlier models [26, 2, 24] performed content generation in feature space,
growing a feature representation in a hierarchical manner before converting
the final feature set into an image.

More recently, progressive [15] and style-based [16, 17] generative networks
have appeared. These models achieved unprecedented resolutions and visual
fidelity and solidified themselves as leading paradigms in the content
generation field. One of the core differences between these models and
previous generative works is that they generate hierarchical content directly
in image space, without resorting to a conversion from learned features at the
very end. This is achieved through direct supervision in the progressive case,
or merely by applying naive bilinear-upsampling between different scales and
teaching the network to augment them with high-resolution detail as in Karras
et al. [17]. These additional constraints serve to regularize the network,
preventing any learned features from deviating too widely and eventually
improving the quality of results.

Our work aims to improve upon this idea by progressively generating image
decompositions in a frequency domain. Rather than training the different
layers to implicitly add higher-frequency data to an image by modifying its
color content, we train them to do so explicitly by modifying the coefficients
of a wavelet band decomposition. Similarly, our discriminator extracts
features not only from the image space representations but also from wavelet
domain sub-bands. The network is thus empowered to better identify content
that lacks high-frequency data, and can more directly restore it.

###  2.2 Frequency-based Approaches

The use of frequency based information, and in particular wavelet transforms,
has been well studied in the deep learning literature. Convolutional neural
networks have been augmented with wavelet-based features, inputs or pooling
operations [8, 20, 33] in order to improve performance on style transfer [34],
denoising [19] and even medical analysis tasks [13]. Others [30, 23] have
proposed a decomposition of implicit representation input coordinates into a
set of Fourier-based features. They showed, both empirically and
theoretically, that explicitly providing the network with high frequency
inputs allows it to overcome spectral bias and better reconstruct high-
frequency outputs. In the generative domain, several works [11, 32, 35] have
proposed empowering networks to create not only images, but also full wavelet
based decompositions. Others [21, 32] have augmented discriminators by
providing them with not only an image, but also frequency sub-bands derived
from a wavelet packet transform.

None of these works, however, are general purpose generative models. Instead,
they utilize adversarial losses in order to improve performance on specific
tasks such as image super resolution or denoising. As such, none of them can
generate a novel image from a latent code but rather only add or improve upon
existing images.

Our work, meanwhile, seeks to train a fully generative model which can
synthesize novel images from noise samples. Such a generator can later be
adapted to a multitude of tasks by projecting images into its richer latent
space, either through direct optimization [1] or through the use of a trained
encoder [28]. This methodology has shown considerable success since the
inception of powerful generative models such as Karras et al. [16].

More recently, a significant number of studies [3, 6] have highlighted the
failures of general-purpose generative networks in the high-frequency domain.
All of them, however, address the problem by introducing frequency related
terms to the loss function. In contrast, we modify not only the loss term, but
also introduce a frequency-based inductive bias into the generator itself.
This bias is shown to further bolster the visual quality of synthesized
images, while also leading to significant reductions in training and inference
times.

Lastly, our work is further distinguished from prior wavelet-based approaches
in that we adopt the modern viewpoint of progressive generation in the
frequency domain. Whereas previous works opted to utilize a progressive
feature representation and generate wavelet content only at the final layer,
or alternatively provide the discriminator with wavelets at the onset but
merely encode them into features - our model generates content directly in
wavelet space throughout the entire process.

##  3 Method

###  3.1 Wavelet Transform

At the core of our method are wavelet transforms, which decompose an image
into a sequence of channels, each of which represents a different range of
frequency content. We employ Haar Wavelets as the basis function for our
transformations, owing to their simple nature coupled with their well
documented ability to represent multi-frequency information [4, 5]. Our model
works with the first-level wavelet decomposition, where each image is broken
into four sub-bands: LL, LH, HL and HH. These bands are attained by passing
the image through a series of low-pass and high-pass wavelet filters and serve
as the wavelet coefficients. The first sub-band, LL, corresponds to low
frequency information, and indeed it is visually similar to a blurred version
of the input image. The remaining sub-bands, LH, HL and HH, correspond to high
frequency content in the horizontal, vertical and diagonal directions,
respectively.

###  3.2 Network Architecture

Figure 3: Our SWAGAN generator (left) and discriminator (right) architectures.
Each ConvBlock is equivalent to a feature-resolution increasing block of the
StyleGAN2 architecture, which is itself composed of two style blocks.
tWavelets and fWavelets correspond to the tRGB and fRGB layers of StyleGAN2
and their purpose is to learn a mapping between wavelet decompositions and
high dimensional features. Inverse wavelet transforms are denoted by IWT,
while Up and Down are non-learning layers responsible for converting an image
to an initial wavelet-decomposition of a higher or lower resolution,
respectively.

Our wavelet-aware architecture is based on the original implementation of
StyleGAN2 [17]. While the original implementation generates content directly
in image-space, our proposed architecture operates in the frequency domain.
Similarly, our discriminator is designed to consider not only the RGB space of
images, but their entire wavelet decomposition.

By allowing the generator to generate content directly in the wavelet domain,
we gain on two fronts: First, as shown by [27], neural networks prioritize
learning in the low frequency domain. By converting our representation to a
frequency based one, we empower the network to affect high-frequency changes
in the image domain by learning low-frequency modifications of the
representation. This differs from a simple loss-based modification, as the
later can provide an incentive for the network to learn high-frequencies, but
it does not make the learning task easier. Second, wavelet decompositions are
more spatially compact. In a first-level wavelet decomposition, each $2N\times
2N$ image is fully represented by four channels of $N\times N$ coefficients
each. This allows us to employ convolutions on lower resolution
representations throughout the entire generative process, at the cost of
requiring additional filters. However, this tradeoff can be advantageous when
utilizing the popular deep learning frameworks [12].

Similarly, by providing frequency information to the discriminator, the
network is able to better identify the high frequency content that is often
missing in generated images. As a result, the generator is better motivated to
re-create plausible high-frequency data.

In the proposed generator, each resolution block receives a full wavelet
decomposition as an input. In a similar fashion to StyleGAN2, the wavelet
coefficients are refined using a set of high dimensional features mapped back
to the frequency domain via skip connections. In StyleGAN2, the image is
resized between blocks using simple bilinear up-sampling. Performing this up-
sampling in the frequency domain would not carry the same meaning. Instead, we
choose the natural alternative and perform up-sampling by converting the
wavelet representation back to the image domain by applying the inverse
wavelet transform (IWT), resizing the image as usual, and then predicting the
next set of wavelet coefficients from the higher-resolution image. The
network’s output is formed by applying an IWT to the wavelet decomposition
provided by the output of the last layer. The generator’s architecture is
illustrated in Figure 3 (left).

In the discriminator, we similarly provide the wavelet coefficients of the
corresponding resolution to each block. At each resolution step, a skip-
connection based network is used to extract features from the wavelet
decomposition and merge them into the feature representations derived from
higher resolution blocks. In order to downscale the image between blocks, the
wavelet coefficients are combined back into a full image via IWT, the image is
bilinearly down-sampled, and the lower-resolution image is decomposed back
into the wavelet coefficients, which are fed into the next block. The input to
the first block of the discriminator is simply the DWT of any image (real or
fake). The discriminator’s architecture is illustrated in Figure 3 (right).

In addition to the architecture described above, we explored a series of
variations, including different up-sampling steps and mixing image and
wavelet-domain generation steps in the same network. We describe and analyze
these variations in Section 4.

###  3.3 Downstream Tasks

Any adaptation of the style-based network would be incomplete if it could not
be used as a backbone for the downstream tasks built around the original. We
demonstrate that our wavelet-based network can still support the same
applications, and in some cases even achieve better results.

We analyze our model using the optimization-based approach to inversion, where
we use gradient descent to find the latent space representation for which the
generator outputs the best approximation of a given source image. We use the
latent space projector of Karras et al. [17], which finds a latent
representation in the so-called W space of a style-based network, using a
target of minimizing a perceptual based loss (LPIPS). The later is crucial for
our needs, as any L2 based reconstruction target will inevitably discard high-
frequency information in the optimization process.

###  3.4 Implementation and Training Details

We build upon the official TensorFlow implementation of StyleGAN2 [17].
Specifically, we modify the architecture of the skip-based generator and
discriminators, but otherwise inherit all of the corresponding training
details and parameters from the configuration F setup. Our models were trained
on FFHQ at resolutions of $256\times 256$ and $1024\times 1024$ and on LSUN
Churches at a resolution of $256\times 256$. For FFHQ at $1024\times 1024$, we
utilized a batch size of 8 for the wavelet-based models and 2 for StyleGAN2
comparisons. For FFHQ at $256\times 256$ resolution, we used batch sizes of 16
and 8, respectively. All models were trained on a single NVIDIA GeForce RTX
2080 GPU, i.e., using commercially available, non-corporate grade
computational power. For quantitative evaluations of downstream tasks, we
utilized the publicly available CelebA dataset [22].

##  4 Experiments

We have conducted a series of experiments in order to evaluate the increased
computational and visual performance of our generator, and to evaluate the
importance of high frequency data to downstream tasks. We additionally conduct
an ablation study to highlight the importance of generation in the wavelet-
domain, the importance of a wavelet-based discriminator and to investigate
natural alternatives to some of our ad-hoc architectural choices. In all of
the experiments below, we dubbed the bilinear up-sampling variation of our
generator (presented in Section 3) as SWAGAN-Bi.

###  4.1 Computational Performance and Visual Quality

We first demonstrate our ability to capture high-frequency data on a set of
toy problems. We generated three datasets, each containing only a single image
of a pattern with high-frequency detail. In addition, we created a fourth
dataset composed of a single real face with considerable high-frequency data.
We trained our model and the original StyleGAN2 model on these single image
datasets for a period of 24 hours. Figure 1 shows the outputs of the trained
models. When dealing with repeated patterns, our model can cleanly capture and
recreate them with high accuracy. The original StyleGAN2 model, meanwhile,
fails to recreate the data in a meaningful manner, showing that purely high-
frequency content is beyond its scope. The face image is well reconstructed by
both models. However, our model is better able to capture details such as the
patterns on the shirt or fine wrinkles.

Figure 4: Uncurated face samples generated by our SWAGAN-Bi model, using
truncation with $\psi=0.5$. Figure 5: Uncurated samples generated by our
SWAGAN-Bi model trained on the LSUN Church dataset at a resolution of 256x256,
using truncation with $\psi=0.5$.

We compared the time required to train our model and the visual quality of
generated results throughout the process to those achieved by StyleGAN2.
Figure 4 shows an uncurated sample of faces drawn from SWAGAN-Bi model trained
on FFHQ at 1024x1024. Figure 5 shows uncurated samples drawn from a SWAGAN-Bi
model trained on LSUN Church at a resolution of 256x256. As can be seen, our
model can produce high quality results with significant detail around regions
of high frequency-content, such as wrinkles or hair.

In Table 1, we report the time required for the model to process 1000 real
images on a single GeForce RTX 2080. This metric is equivalent to time per
epoch (up to a constant, multiplicative factor), and is the standard unit of
work used in [16] and its derivatives. The wavelet-based variants require a
significantly reduced training time across both resolutions.

Table 2 reports the final Fréchet Inception Distance (FID) of our models at
various resolutions and on multiple datasets. As the computational overhead
for training a model for the number of iterations reported by StyleGAN2
remains high, we elected to reduce the resolution and the number of training
steps used for training the models. In all such cases, we provide results on a
StyleGAN2 model trained for the same number of iterations and resolution,
utilizing the official implementation of Karras et al. [17]. Note that this
comparison favors the StyleGAN2 models by allotting them roughly double the
computational budget, due to increased training times and higher memory
requirements. Even under this biased comparison, the wavelet-based approach
can capture more realistic details and achieve better visual quality.

Figure 6 shows a comparison of FID metrics over the duration of training
between a wavelet based model (SWAGAN-Bi) and the original StyleGAN2 model,
trained on FFHQ at 1024x1024 resolution. We show the progress, both as a
function of the number of images viewed by the discriminator and as a function
of wall-time. Our model, not only trains faster, but also converges with fewer
epochs on the same data set. This hints that the inductive bias built into our
model is indeed capable of alleviating the spectral bias of neural network
learning.

Model | Resolution | Seconds / 1k images $\downarrow$
---|---|---
StyleGAN2 | 1024 | 184.97
SWAGAN-Bi | 1024 | 95.06
StyleGAN2 | 256 | 143.04
SWAGAN-Bi | 256 | 80.07
SWAGAN-Bi w/ NWD | 256 | 129.65
SWAGAN-NU | 256 | 70.06
Wavelet at final layer | 256 | 116.06

Table 1: Average time (in seconds) for the network to process 1000 real images during training. Wavelet-based generators consistently outperform the alternatives. The reported times are based on the calculation provided in StyleGAN2’s original implementation and do not include any of the (significant) time spent on benchmark calculations or sample generation. Model | Dataset | Images Viewed | FID $\downarrow$
---|---|---|---
StyleGAN2 | FFHQ 1024 | 9.76M | 7.62
SWAGAN-Bi | FFHQ 1024 | 9.76M | 4.68
SWAGAN-Bi | FFHQ 1024 | 17.1M | 4.06
StyleGAN2 | FFHQ 256 | 8.5M | 5.82
SWAGAN-Bi w/ NWD | FFHQ 256 | 8.5M | 12.27
SWAGAN-Bi | FFHQ 256 | 8.5M | 5.22
Wavelet at final layer | FFHQ 256 | 8.5M | 7.96
SWAGAN-NU | FFHQ 256 | 8.5M | 9.20
StyleGAN2 | Church 256 | 6.5M | 5.46
SWAGAN-Bi | Church 256 | 6.5M | 4.97

Table 2: The best FID achieved by all tested setups until the discriminator
observed the indicated number of images. Our SWAGAN-Bi model achieves the best
results for different resolutions and datasets. Figure 6: Model FID as a
function of: (a) wall-clock time, (b) number of images viewed by the
discriminator. Our model outperforms StyleGAN2 in both scenarios and displays
increased stability during training.

In Figure 7, we plot the distance between the power spectrum of real images
and those generated by a subset of our trained models, averaged over 4050
images per model and normalized by the spectra of real images (dubbed
”spectrum gap”). For convenience, we additionally plot the spectrum gap for
real images blurred via Gaussian kernels with filter sizes of $3\times 3$ and
$5\times 5$. Our frequency-domain model consistently generates more realistic
spectra, particularly in the medium and high frequency range.

Figure 7: The power spectrum distance between real images and those produced
by the listed models, normalized by the spectra of real images. To assist
comparisons, we also provide the spectrum gap observed in real images blurred
using Gaussian kernels of size 3x3 and 5x5. Our full model (orange) achieves
more realistic results in the middle and high frequency range. Note that a
model without a wavelet-based discriminator, or one where the generative
process is not entirely wavelet, based both fail to capture the same degree of
high frequency data.

###  4.2 Ablation Study

In order to measure the importance of each aspect of our suggested solution,
we asked ourselves the following questions, and conducted a series of
experiments to answer each in turn. The quantitative results of these
experiments are provided in tables 1 and 2 and in Figure 7.

#### Is it important to progressively generate content in the frequency
domain?

We investigate this question with two experiments. In the first, we observe
the behaviour of the network when all layers, but the last utilize the
original tRGB mapping. In such a setup, content is progressively generated in
the RGB domain and any wavelet information is only added at the final layer.
The model which only predicts wavelet information in the final layer has shown
inferior performance, both in time-based metrics and in visual quality,
highlighting the importance of wavelet-domain generation throughout the entire
process. Note that this model performs worse than even the original StyleGAN2.
This may be due to the network’s need to conserve both frequency and image
based information in the hidden feature representation.

Figure 8: The upscaling blocks for SWAGAN-NU (top) and SWAGAN-Bi (bottom)
architectures. SWAGAN-NU architecture includes $3\times 3$ convolutions in
order to predict the new wavelet channels; The SWAGAN-Bi architecture is
comprised of bilinear up sampling, followed by discrete wavelet transform
(DWT).

Next, we investigated a variation on the model’s up-scaling layer. In this
setup, rather than resizing the image through bilinear up-sampling, we utilize
a network to predict an initial set of wavelet coefficients for the next
layer, effectively generating content at the next resolution step. This
upscaling layer is visualized in Figure 8. We dubbed this neural upscaling
variation as SWAGAN-NU.

A reasonable expectation would be for the network to perform better in this
scenario, as it has sufficient degrees of freedom to learn a better upsampling
step. However, although it sounds promising, such a change also breaks the
inductive bias built into the network. In the bilinear up-sampling approach,
the network must truly generate wavelet based content at each scale because it
has no means through which to convert the hidden representation back to the
wavelet domain before the output layer. The added neural mapping of the
SWAGAN-NU architecture, however, could also be used to learn a mapping from
any feature representation back to wavelet space, thus allowing the network to
ignore our intent and merely work in feature space throughout. The loss of
this bias adversely affects the visual quality of generated results. We
further examined this hypothesis by investigating the images observed in the
intermediate scales of the network. A comparison of such images is provided in
Figure 9, where we plot the four wavelet maps predicted at increasing
resolution scales, resized to $256\times 256$. As can be seen, intermediate
representations in the SWAGAN-Bi generator portray a gradual growth in wavelet
space, while those of the SWAGAN-NU setup do not.

Figure 9: Intermediate wavelet representations in the generative path of our
SWAGAN-Bi and SWAGAN-NU models. At each row, we plot the four wavelet
coefficient maps obtained at a given resolution step, prior to the IWT
operation. All outputs were resized to $256\times 256$ using bicubic
interpolation. The SWAGAN-Bi outputs portray an increasing level of detail in
the wavelet space, while the SWAGAN-NU outputs portray a deep feature
representation until the final resolution. Zooming in is encouraged

As our ablation experiments show, progressive generation in the frequency
domain is key to improving results.

#### Does the network benefit from modifying both the generator and the
discriminator?

To answer this question, we trained a model utilizing the SWAGAN-Bi generator,
along with the original residual based discriminator of StyleGAN2 (dubbed as
NWD for non-wavelet-discriminator). The trained model offers significantly
degraded performance when compared to the full SWAGAN-Bi model. This demeaned
performance can be observed in both the FID metrics of Table 2 and in the
power spectrum comparisons of Fig 7, where the average spectra of images
generated by the full model more closely resemble that of real images. Indeed,
the results of such a model are worse than those of the original StyleGAN2
across the board. In addition to the degraded results, we also observe a
significant increase in training instability when using a wavelet-based
generator without a wavelet-based discriminator. When the loss term is
insensitive to high-frequencies, the corresponding high-frequency bands in the
wavelet decomposition receive diminished gradients, and the network fails to
learn how to modify them in a meaningful manner. These results indicate that a
wavelet-based generator should be accompanied by an appropriate frequency
aware loss term (and the original StyleGAN2 discriminator is insufficient in
that regard, as shown in Chen et al. [3]).

##  5 GAN Inversion and Applications

One of the key benefits of the style-based generative framework is its well
behaved latent space, a property which can be exploited for editing tasks or
for interpolating images in the generator’s latent space. As we deem this to
be a crucial property, we conducted a series of experiments to ensure that our
latent space still supports these operations.

In Figure 10, we encode pairs of images into the latent spaces of a SWAGAN-Bi
generator and StyleGAN2’s original generator using the projector provided in
their original implementation. We perform linear interpolation between each
pair of images and compare the results of both models. As can be seen, both
latent spaces seem to behave comparably well, although our model generates
sharp images throughout the interpolation process, without any of the blurring
often observed during the transition (see for example the hair in the the
central images of the figure). In Figure 11, we show the results of semantic
editing in our generator’s latent space. As in the case of linear
interpolation, our results do not exhibit blur around high-frequency areas
such as the hair.

Figure 10: Latent space interpolation. In each cluster of images, the left-
most and right-most images are the ground truth, taken from the FFHQ dataset.
We project them into the generator’s latent W-space using the StyleGAN
projector, and interpolate between both ends. The top row of each cluster
contains images obtained using a StyleGAN2 configuration F model, while the
bottom row contains images obtained using the SWAGAN-Bi model. Interpolated
StyleGAN2 images show considerable blur around regions of high frequency such
as the hair, while the SWAGAN images do not. Figure 11: Semantic face editing
using the latent projection method of [29]. We modify (from top to bottom):
smile, age and hair length. Figure 12: The power spectrum gap between real
images and those generated by our downstream experiments. Though performance
varies greatly between tasks, our high-frequency advantage is maintained for
all of them.

We further analyzed the spectrum gap of images generated in all our downstream
experiments. For the reconstruction experiment, we compare the spectra of
$500$ reconstructed images to those of their corresponding projection targets.
For the editing and interpolation experiments, we sampled random images and
compared their spectra to 5000 real images from the FFHQ dataset. The results
are shown in Figure 12. In all scenarios, our model is able to generate more
realistic spectra, indicating that our improvements can be transmitted to
downstream tasks.

Our experiments show that our frequency based model preserves the versatility
of StyleGAN2’s latent space, allowing us to support similar operations while
generating sharper results.

Figure 13: Curated face samples generated by our SWAGAN-Bi model without
truncation.

##  6 Conclusions

We have outlined an alternative approach to the style-based generative
framework, operating on the frequency domain rather than directly in RGB-
space. This approach was shown to lead to more realistic visual results,
particularly in the middle-to-high frequency range. By operating directly in
frequency space, the model can affect high frequency changes in the output
through low frequency changes in the representation, i.e. the wavelet
coefficients. This directly tackles the spectral bias of neural networks and
prompts a significant increase in rate of convergence. As it turns out,
adopting the wavelet-based generative approach allows for training a style-
based generative model with similar performance at roughly a fourth of the
computational budget. We have further shown that our quality benefits can be
extended to downstream tasks, allowing for more realistic image
reconstructions or editing operations.

The landscape of frequency-domain generation leaves many venues for future
research. Our exploration focused on a single wavelet function (the Haar
Wavelet) which may not necessarily be optimal. Indeed, it may be possible to
learn a better wavelet, or to utilize a set of different wavelets, while
allowing the network to mix and match between them using appropriate weights.
Another interesting direction for further research may be an adaptation of
frequency-domain generation to the realm of videos, for example by building
upon existing spatio-temporal wavelet schemes.

Where downstream tasks are concerned, we have limited ourselves to those that
are tackled through direct optimization methods. Recently, however, encoder-
based methods [28] have been gaining popularity, offering a faster alternative
to optimization, with increasing quality for various image-to-image
translation tasks. These encoders suffer from acute high-frequency
shortcomings, in part due to their use of L2 based losses. It may therefore be
beneficial to explore a similar frequency-based treatment for such encoders.
Additionally, some disentanglement tasks probably lend themselves better to a
spectral representation, especially those that are frequency related. This
avenue is also one that should probably be explored.

In summary, we hope our work inspires others to consider alternative
representation for content generation, since these can be, as demonstrated
here, a more natural fit for the network to learn, yielding broad advantages
in quality, runtime, and training convergence.

## References

  * Abdal et al. [2019] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4432–4441, 2019.
  * Chen et al. [2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel.  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  In _Advances in neural information processing systems_ , pages 2172–2180, 2016.
  * Chen et al. [2020] Yuanqi Chen, Ge Li, Cece Jin, Shan Liu, and Thomas Li.  Ssd-gan: Measuring the realness in the spatial and spectral domains.  _arXiv preprint arXiv:2012.05535_ , 2020.
  * Daubechies [1990] Ingrid Daubechies.  The wavelet transform, time-frequency localization and signal analysis.  _IEEE transactions on information theory_ , 36(5):961–1005, 1990.
  * Daubechies [1992] Ingrid Daubechies.  _Ten lectures on wavelets_.  SIAM, 1992.
  * Durall et al. [2020] Ricard Durall, Margret Keuper, and Janis Keuper.  Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7890–7899, 2020.
  * Dzanic et al. [2019] Tarik Dzanic, Karan Shah, and Freddie Witherden.  Fourier spectrum discrepancies in deep network generated images.  _arXiv preprint arXiv:1911.06465_ , 2019.
  * Gao and Xiong [2016] Xing Gao and Hongkai Xiong.  A hybrid wavelet convolution network with sparse-coding for image super-resolution.  In _2016 IEEE International Conference on Image Processing (ICIP)_ , pages 1439–1443. IEEE, 2016.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative Adversarial Nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_ , pages 2672–2680. Curran Associates, Inc., 2014.  URL http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf.
  * Huang et al. [2017] Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan.  Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1689–1697, 2017.
  * Huang et al. [2019] Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan.  Wavelet domain generative adversarial network for multi-scale face hallucination.  _International Journal of Computer Vision_ , 127(6-7):763–784, 2019.
  * Jordà et al. [2019] Marc Jordà, Pedro Valero-Lara, and Antonio J Peña.  Performance evaluation of cudnn convolution algorithms on nvidia volta gpus.  _IEEE Access_ , 7:70461–70473, 2019.
  * Kang et al. [2017] Eunhee Kang, Junhong Min, and Jong Chul Ye.  A deep convolutional neural network using directional wavelets for low-dose x-ray ct reconstruction.  _Medical physics_ , 44(10):e360–e375, 2017.
  * Karnewar and Wang [2019] Animesh Karnewar and Oliver Wang.  Msg-gan: multi-scale gradient gan for stable image synthesis.  _arXiv preprint arXiv:1903.06048_ , 2019.
  * Karras et al. [2017] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_ , 2017.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4401–4410, 2019.
  * Karras et al. [2020] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8110–8119, 2020.
  * Ledig et al. [2017] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4681–4690, 2017.
  * Liu et al. [2020] Lin Liu, Jianzhuang Liu, Shanxin Yuan, Gregory Slabaugh, Ales Leonardis, Wengang Zhou, and Qi Tian.  Wavelet-based dual-branch network for image demoiréing.  _arXiv preprint arXiv:2007.07173_ , 2020.
  * Liu et al. [2019a] Pengju Liu, Hongzhi Zhang, Wei Lian, and Wangmeng Zuo.  Multi-level wavelet convolutional neural networks.  _IEEE Access_ , 7:74973–74985, 2019a.
  * Liu et al. [2019b] Yunfan Liu, Qi Li, and Zhenan Sun.  Attribute-aware face aging with wavelet-based generative adversarial networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 11877–11886, 2019b.
  * Liu et al. [2018] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Large-scale celebfaces attributes (celeba) dataset.  2018\.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _arXiv preprint arXiv:2003.08934_ , 2020.
  * Mirza and Osindero [2014] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_ , 2014.
  * Nguyen-Phuoc et al. [2019] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7588–7597, 2019.
  * Radford et al. [2015] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _arXiv preprint arXiv:1511.06434_ , 2015.
  * Rahaman et al. [2019] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville.  On the spectral bias of neural networks.  In _International Conference on Machine Learning_ , pages 5301–5310. PMLR, 2019.
  * Richardson et al. [2020] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  _arXiv preprint arXiv:2008.00951_ , 2020.
  * Shen et al. [2020] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In _CVPR_ , 2020.
  * Tancik et al. [2020] Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  _arXiv preprint arXiv:2006.10739_ , 2020.
  * Thies et al. [2019] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  _ACM Transactions on Graphics (TOG)_ , 38(4):1–12, 2019.
  * Wang et al. [2020] Jianyi Wang, Xin Deng, Mai Xu, Congyong Chen, and Yuhang Song.  Multi-level wavelet-based generative adversarial network for perceptual quality enhancement of compressed video.  _arXiv preprint arXiv:2008.00499_ , 2020.
  * Williams and Li [2018] Travis Williams and Robert Li.  Wavelet pooling for convolutional neural networks.  In _International Conference on Learning Representations_ , 2018.
  * Yoo et al. [2019] Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung-Woo Ha.  Photorealistic style transfer via wavelet transforms.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 9036–9045, 2019.
  * Zhang et al. [2019a] Qi Zhang, Huafeng Wang, Tao Du, Sichen Yang, Yuehai Wang, Zhiqiang Xing, Wenle Bai, and Yang Yi.  Super-resolution reconstruction algorithms based on fusion of deep learning mechanism and wavelet.  In _Proceedings of the 2nd International Conference on Artificial Intelligence and Pattern Recognition_ , pages 102–107, 2019a.
  * Zhang et al. [2019b] Qi Zhang, Huafeng Wang, and Sichen Yang.  Image super-resolution using a wavelet-based generative adversarial network.  _arXiv preprint arXiv:1907.10213_ , 2019b.