# Thermodynamic optimization equalities in weak processes

Pierre Nazé<EMAIL_ADDRESS>Universidade Estadual Paulista, 14800-090,
Araraquara, São Paulo, Brazil

###### Abstract

Equalities are always a better option to be verified in an experiment than
inequalities. In this work, for classical and isothermal processes, I provide
thermodynamic equalities valid when the optimizations of macroscopic
thermodynamic quantities are achieved in weak drivings. I illustrate their
applicability by using them as a convergence criterion in the global
optimization technique of genetic programming.

##  I Introduction

Due to inherent errors in experiments, observing inequalities between physical
quantities is always a task one should avoid. In particular, optimization
problems, where one should verify for instance the least possible value of a
quantity, face such a difficulty, and strategies to overcome it are very
welcome.

In recent times, many efforts have been made in strategies of optimization
problems in non-equilibrium thermodynamics, in particular, to find the optimal
protocols that minimize the thermodynamic work Deffner and Bonança (2020);
Blaber and Sivak (2023). For classical and weak processes, an Euler-Lagrange
equation has been proposed as a way to find such an optimal protocol Nazé  _et
al._ (2022). Even though such an equation has not been explored in its
physical meaning, and important information may be lacking.

In this work, based on the First Law of Thermodynamics, I present the physical
meaning behind the Euler-Lagrange equation for the optimization of work, heat,
and energy. Basically, when optimization is achieved, the rates of work, heat,
and energy become particularly proportional to each other, for all instants of
the process. Such relations are called in this work thermodynamic optimization
equalities. They offer therefore a resolution to the problem of optimization
since no more comparisons to verify inequalities will be made with them.

As an application, I use them as a convergence criterion for the global
optimization technique of genetic programming Duriez  _et al._ (2017), by
comparing the difference of the values of an equality in a prescribed error.
This seems more appropriate than using the naive criterion of waiting until
the system does not evolve anymore Nazé (2023). In particular, the application
is illustrated in the optimization problems of work, heat, and energy of the
overdamped Brownian motion subjected to a moving laser or stiffening traps and
white noise.

##  II Linear response theory

Consider a classical system, with a Hamiltonian
$\mathcal{H}_{S}(z(t),\lambda(t))$, where $z(t)$ is a point in the phase space
of the coordinates of the system and $\lambda(t)$ a control parameter. The
system is weak coupled to a thermal reservoir of Hamiltonian
$\mathcal{H}_{\mathcal{B}}(z(t),Z(t))$, where $Z(t)$ is a point in the phase
space of the coordinates exclusively associated to the thermal reservoir.
Starting in a thermal state of temperature $T$, the system performs a driving
process $\lambda(t)=\lambda_{0}+g(t)\delta\lambda$, during a switching time
$\tau$, where $g(t)$ is a protocol that obeys $g(0)=0$ and $g(\tau)=1$. The
statistical distribution $\rho(t)$ of the total system evolves accordingly to
the Liouville equation $\partial_{t}\rho=\mathcal{L}\rho$, where $\mathcal{L}$
is the Liouville operator. Suppose in particular that the sum of the
Hamiltonian of the system and bath is given by
$\mathcal{H}=\mathcal{H}_{0}+\lambda(t)\mathcal{H}_{1}$, where
$\mathcal{H}=\mathcal{H}_{\mathcal{S}}+\mathcal{H}_{\mathcal{B}}$. In this
case, the decomposition of the probability distribution $\rho(t)$ in terms of
the first-order of $\delta\lambda$ is

$\rho(t)=\rho_{0}+\rho_{1}(t)\delta\lambda,$ (1)

where $\rho_{0}$ is the associated canonical ensemble and

$\rho_{1}(t)=\int_{0}^{t}e^{(t-s)\mathcal{L}_{0}}\mathcal{L}_{1}\rho_{0}g(s)ds,$
(2)

with $\mathcal{L}_{0}(\cdot)=-\\{\cdot,\mathcal{H}_{0}\\}$ and
$\mathcal{L}_{1}(\cdot)=-\\{\cdot,\mathcal{H}_{1}\\}$. In particular, we call
$\phi(t)=\langle\\{\mathcal{H}_{1},\mathcal{H}_{1}(t)\\}\rangle_{0}$ response
function Kubo  _et al._ (2012), where $\\{\cdot,\cdot\\}$ is the Poisson
bracket and $\langle\cdot\rangle_{0}$ the average along the initial
equilibrium state. Also, considering the relation $\phi(t)=-\dot{\Psi}(t)$, we
call $\Psi(t)$ relaxation function Kubo  _et al._ (2012). These quantities are
going to be useful to express the macroscopic thermodynamic quantities as
internal energy, work and heat in a simpler way.

##  III First Law of Thermodynamics

Consider the system plus the thermal reservoir. As the whole system is
thermally isolated, the First Law of Thermodynamics is

$U_{\mathcal{S+B}}(\tau)=W(\tau),$ (3)

where $U_{\mathcal{S+B}}(\tau)$ and $W(\tau)$ are respectively the variation
of the total energy of the whole system and the work performed along the
process. These quantities are given by

$U_{\mathcal{S+B}}(\tau)=\left[\int\mathcal{H}(\Gamma,t)\rho(\Gamma,t)d\Gamma\right]\bigg{|}_{0}^{\tau}$
(4)
$W(\tau)=\int_{0}^{\tau}\int\partial_{\lambda}\mathcal{H}_{\mathcal{S}}(\Gamma,t)\dot{\lambda}(t)\rho(\Gamma,t)d\Gamma
dt,$ (5)

where $d\Gamma=dzdZ$. Rewriting it in terms of the variation of the internal
energy $U_{\mathcal{S}}$ of the system along the process, one has

$U_{\mathcal{S}}(\tau)=W(\tau)-U_{\mathcal{B}}(\tau),$ (6)

where

$U_{\mathcal{S}}(\tau)=\left[\int\mathcal{H}_{\mathcal{S}}(\Gamma,t)\rho(\Gamma,t)d\Gamma\right]\bigg{|}_{0}^{\tau},$
(7)
$U_{\mathcal{B}}(\tau)=\left[\int\mathcal{H}_{\mathcal{B}}(\Gamma,t)\rho(\Gamma,t)d\Gamma\right]\bigg{|}_{0}^{\tau},$
(8)

where the second term on the right-hand side of the equation can be identified
as the heat spent along the process. Another way to identify this last
quantity is by expressing the variation of the internal energy of the system
along the process as

$U_{\mathcal{S}}(\tau)=\int\int_{0}^{\tau}\frac{d}{dt}\left[\mathcal{H}_{\mathcal{S}}(\Gamma,t)\right]\rho(\Gamma,t)dtd\Gamma,$
(9)

where we have used the Liouville theorem. Using now
$d/dt=-\mathcal{L}+\partial_{t}$, one has

$U_{\mathcal{S}}(\tau)=W(\tau)+Q(\tau),$ (10)

with

$Q(\tau)=\int_{0}^{\tau}\int[\mathcal{L}\mathcal{H}_{\mathcal{S}}(\Gamma,t)]\rho(\Gamma,t)d\Gamma
dt.$ (11)

By comparison with Eq. (6), $Q(\tau)$ is the heat spent along the process.

##  IV Thermodynamic optimization equalities

Applying linear-response theory to express up to the second-order perturbation
in the parameter $\delta\lambda$ the variation of the internal energy of the
system, the work performed, and the heat spent, one has

$U_{\mathcal{S}}(\tau)=\delta\lambda\langle\mathcal{H}_{1}\rangle_{0}+\delta\lambda^{2}\int_{0}^{\tau}\dot{\Psi}(\tau-t)g(t)dt,$
(12)
$\begin{split}W(\tau)=\delta\lambda\langle\mathcal{H}_{1}\rangle_{0}+\delta\lambda^{2}\int_{0}^{\tau}\dot{\Psi}(\tau-t)g(t)dt\\\
-\frac{\delta\lambda^{2}}{2}\int_{0}^{\tau}\int_{0}^{\tau}\ddot{\Psi}(t-t^{\prime})g(t)g(t^{\prime})dtdt^{\prime},\end{split}$
(13)
$Q(\tau)=\frac{\delta\lambda^{2}}{2}\int_{0}^{\tau}\int_{0}^{\tau}\ddot{\Psi}(t-t^{\prime})g(t)g(t^{\prime})dtdt^{\prime}.$
(14)

###  IV.1 Optimization of work

As it was presented in Ref. Nazé  _et al._ (2022), the total optimal protocol
is a combination of a continuous protocol with jumps at the beginning and end
of the process. In this manner, applying calculus of variations in the work in
order to minimize it, one has the following Euler-Lagrange equation

$\int_{0}^{\tau}\ddot{\Psi}(t-t^{\prime})g^{*}_{W}(t^{\prime})dt^{\prime}=\dot{\Psi}(\tau-t),$
(15)

where $g^{*}_{W}(t)$ is the optimal protocol that will minimize the work. I
remark that such optimal protocol is the continuous part of the total optimal
protocol. In this manner, using the Euler-Lagrange equation and the First Law
of Thermodynamics, the thermodynamic relations that define optimality are

$\dot{U}_{\mathcal{S}}(t)=2\dot{W}^{*}(t)=2\dot{Q}(t),$ (16)

where $\dot{U}_{\mathcal{S}}(t),\dot{W}^{*}(t)$, and $\dot{Q}(t)$ are
respectively the rates of the internal energy of the system, work performed,
and heat spent along the process. Indeed

$\dot{U}_{\mathcal{S}}(t)=\delta\lambda^{2}\dot{\Psi}(\tau-t)g^{*}_{W}(t),$
(17)
$\dot{W}^{*}(t)=\delta\lambda^{2}g^{*}_{W}(t)\left(\dot{\Psi}(\tau-t)-\frac{1}{2}\int_{0}^{\tau}\ddot{\Psi}(t-t^{\prime})g^{*}_{W}(t^{\prime})dt^{\prime}\right),$
(18)
$\dot{Q}(t)=\frac{\delta\lambda^{2}}{2}g^{*}_{W}(t)\int_{0}^{\tau}\ddot{\Psi}(t-t^{\prime})g^{*}_{W}(t^{\prime})dt^{\prime}.$
(19)

Observe that such thermodynamic optimization equalities must be verified for
all instants in the process of a particular switching time. One may argue
that, for each one of those quantities, an integration in the variable $t$ in
the duration of the process will furnish just a single quantity to evaluate in
the laboratory, making the procedure easier. However, this statement holds
only in one sense. To make valid in the reverse one, one may verify these
integration identities for all switching times below the original one. In the
end, the effort will be the same. The price to verify an equality for
optimization is to verify it in a whole interval.

###  IV.2 Optimization of heat

For its turn, to calculate the protocol that will minimize the heat, one must
have to calculate the total optimal protocol obeying the boundary conditions.
I assume that such optimal protocol is the combination of a continuous
protocol with jumps, as it was in the case of the optimization of the work.
Applying the calculus of variation, one has

$\int_{0}^{\tau}\ddot{\Psi}(t-t^{\prime})g^{*}_{Q}(t^{\prime})dt^{\prime}=0,$
(20)

or

$\dot{Q}^{*}(t)=\frac{\delta\lambda^{2}}{2}g^{*}_{Q}(t)\int_{0}^{\tau}\ddot{\Psi}(t-t^{\prime})g^{*}_{Q}(t^{\prime})dt^{\prime}=0,$
(21)

where $g^{*}_{Q}(t)$ is the continuous part of the optimal protocol that will
minimize the heat. This result implies that the optimal heat must be null

$Q^{*}(\tau)=0,$ (22)

which means that the process is adiabatic. A trivial total optimal protocol is
$g^{*}_{Q}(t)=0$ with a jump at the end of the process. Is it possible that
Eq. (20) has a non-trivial solution for any $\tau$? The answer is negative,
since $\ddot{\Psi}(t)$ is a negative kernel, and the double integral of
$Q(\tau)$ is only zero if and only if $g^{*}_{Q}(t)=0$. Therefore, the only
situation where the heat is minimized – more than that, null –, occurs in
sudden processes. This is reasonable since the system has no time to interact
with the environment.

###  IV.3 Optimization of internal energy

Finally, I apply the same reasoning used in the previous sections in the
optimization of the variation of the internal energy of the system. The total
protocol is considered to be composed of a continuous protocol and jumps. By
applying the calculus of variations to the variation of the internal energy,
one has

$\dot{\Psi}(\tau-t)=0.$ (23)

This problem has a solution in two situations: $\Psi(t)=0$ or $\tau\rightarrow
0^{+}$ (it is expected that $\dot{\Psi}(t\rightarrow 0^{+})=0$). The first
case implies that the Hamiltonian of the system does not depend on the
variation of the parameter, and, therefore, the energy is constant. This is
out of our scope. For its turn, the second case one has an effective driving,
where the optimal protocol will become a jump in the sudden process performed.
In this case, one has

$\dot{U}_{\mathcal{S}}^{*}(t)=\delta\lambda^{2}\dot{\Psi}(\tau-t)g^{*}_{U}(t)=0,$
(24)

where $g^{*}_{U}(t)$ is the continuous part of the optimal protocol, which can
be any continuous function that connects $g(0)=0$ to $g(\tau)=1$. This result
implies that the optimal variation of the energy of the system must be equal
to a constant

$U_{\mathcal{S}}^{*}(\tau\rightarrow
0^{+})=\delta\lambda\langle\mathcal{H}_{1}\rangle_{0}.$ (25)

Let us see what are the effects of applying the optimal protocols of one
quantity in the other ones.

###  IV.4 Effects of optimal protocols in other quantities

First, the optimal protocol of the work $g^{*}_{W}(t)$, when applied in the
expression of heat or internal energy, will not lead to their optimal values,
since the protocol is not null at all times. Indeed

$U_{\mathcal{S}}(\tau)=\delta\lambda\langle\mathcal{H}_{1}\rangle_{0}+\delta\lambda^{2}\int_{0}^{\tau}\dot{\Psi}(\tau-t)g^{*}_{W}(t)dt,$
(26)
$W^{*}(\tau)=\delta\lambda\langle\mathcal{H}_{1}\rangle_{0}+\frac{\delta\lambda^{2}}{2}\int_{0}^{\tau}\dot{\Psi}(\tau-t)g^{*}_{W}(t)dt$
(27)
$Q(\tau)=\frac{\delta\lambda^{2}}{2}\int_{0}^{\tau}\dot{\Psi}(\tau-t)g^{*}_{W}(t)dt.$
(28)

On the other hand, using the optimal protocol that will minimize the heat,
$g^{*}_{Q}(t)=0$, for any switching time, the work will not go to its optimal
value, except for $\tau\rightarrow 0^{+}$. Indeed

$U_{\mathcal{S}}(\tau)=\delta\lambda\langle\mathcal{H}_{1}\rangle_{0},$ (29)
$W(\tau)=\delta\lambda\langle\mathcal{H}_{1}\rangle_{0},$ (30)
$Q^{*}(\tau)=0.$ (31)

Observe that, in this case, the thermodynamic optimization equalities
trivially hold

$\dot{U}_{\mathcal{S}}(t)=2\dot{W}(t)=2\dot{Q}^{*}(t)=0.$ (32)

Finally, to optimize the variation of energy of the system, one has to use any
continuous protocol $g^{*}_{U}(t)$ that connects $g(0)=0$ to $g(\tau)=1$ in a
sudden process. In this case

$U_{\mathcal{S}}^{*}(\tau\rightarrow
0^{+})=\delta\lambda\langle\mathcal{H}_{1}\rangle_{0},$ (33)
$W^{*}(\tau\rightarrow 0^{+})=\delta\lambda\langle\mathcal{H}_{1}\rangle_{0},$
(34) $Q^{*}(\tau\rightarrow 0^{+})=0,$ (35)

where all macroscopic quantities achieve their optimality. Observe that the
thermodynamic quantities trivially still hold, as it was in the optimization
of heat.

##  V Convergence criterion

An application for thermodynamic optimization equalities is using them as a
convergence criterion for the global optimization technique of genetic
programming. This approach searches via evolutionary procedures the best
protocol to minimize a cost functional Duriez  _et al._ (2017). The
convergence criteria to stop the algorithm mostly naively used would be
waiting for the non-evolution in the result for a significant period Nazé
(2023). However, this is not a guarantee that, indeed, the search process has
ended. Instead, the criterion proposed in this work stops the code since some
thermodynamic equality is achieved with a previously established error.

The basic idea is to consider the error function

$\epsilon_{N}=\frac{1}{N}\sum_{n=0}^{N-1}\frac{|\dot{U}_{\mathcal{S}}(n\tau/N)-2\dot{Q}(n\tau/N)|}{\delta\lambda^{2}\Psi_{0}/\tau_{R}},$
(36)

which will measure the average discrepancy between the quantities in the
thermodynamic equality $\dot{U}_{\mathcal{S}}=2\dot{Q}$, and compare it with a
previously established value $\epsilon_{0}$, at each interaction of the
algorithm. If $\epsilon_{N}<\epsilon_{0}$, the best protocol at that point is
chosen and the search algorithm is stopped. The error function can be
redefined for other differences of the quantities in some thermodynamic
equality and can be used in any optimization process of work, heat, or
variation of energy of the system.

###  V.1 Example: Overdamped Brownian motion

Consider an overdamped Brownian motion subjected to a moving laser or
stiffening traps and white noise Nazé  _et al._ (2022). The relaxation
function associated with this system is

$\Psi(t)=\Psi_{0}e^{-\frac{|t|}{\tau_{R}}},$ (37)

where $\Psi_{0}$ is a constant and $\tau_{R}$ is its relaxation time. The
optimal protocol of the work is Nazé  _et al._ (2022)

$g^{*}_{W}(t)=\frac{t+\tau_{R}}{\tau+2\tau_{R}},$ (38)

with jumps at the beginning and final point of the process. Let us apply the
global optimization technique of genetic programming with the convergence
criterion proposed to optimize the work, heat and variation of energy of the
system.

Figure 1: Graphic (a): Comparison between numerical optimal protocols for work
and exact result for switching times $\tau/\tau_{R}=0.1,1,10$. Graphics
(b)-(d): Comparison between $\dot{U}_{\mathcal{S}}$ and $2\dot{Q}$ for
different switching times at the moment where the code stopped according to
our convergence criterion. They are measured in unit
$\Psi_{0}\delta\lambda^{2}/\tau_{R}$.

For the optimization of work, considering $N=50$ and $\epsilon_{0}=10^{-3}$,
graphic (a) of Fig. 1 depicts the comparison of the outcomes of the code with
the exact result for switching times $\tau=0.1\tau_{R},\tau_{R},10\tau_{R}$.
The agreement is satisfactory, and, in the case of the work, the running time
of the code reduces sensibly in comparison with the numerical experiments of
my previous work Nazé (2023). Graphics (b)-(d) of Fig. 1 show the comparison
between $\dot{U}_{\mathcal{S}}$ and $2\dot{Q}$ at the moment when the code
stopped for different switching times. The value of the error $\epsilon_{50}$
is presented for each graphic as well.

For the optimization of heat, considering $N=50$ and $\epsilon_{0}=10^{-3}$,
graphic (a) of Fig. 2 depicts the comparison of the outcomes of the code with
the exact result for switching times $\tau=0.1\tau_{R},\tau_{R},10\tau_{R}$.
The agreement is satisfactory. Graphics (b)-(d) of Fig. 2 show the comparison
between $\dot{U}_{\mathcal{S}}$ and $2\dot{Q}$ at the moment when the code
stopped for different switching times. The value of the error $\epsilon_{50}$
is presented for each graphic as well.

Figure 2: Graphic (a): Comparison between numerical optimal protocols for heat
and exact result for switching times $\tau/\tau_{R}=0.1,1,10$. Graphics
(b)-(d): Comparison between $\dot{U}_{\mathcal{S}}$ and $2\dot{Q}$ for
different switching times at the moment where the code stopped according to
our convergence criterion. They are measured in unit
$\Psi_{0}\delta\lambda^{2}/\tau_{R}$.

On the other hand, for the optimization of the variation of energy of the
system, performed always in sudden process, the outcomes show that there is no
optimal protocol for this system. Indeed, the code runs indefinitely, always
reducing the variation of energy of the system. This occurs because the
derivative of the relaxation function is discontinuous at $t=0$, achieving the
limit

$\dot{\Psi}(t\rightarrow 0^{+})=-\frac{1}{\tau_{R}}\neq 0,$ (39)

excluding therefore the necessary hypothesis $\dot{\Psi}(t\rightarrow
0^{+})=0$.

##  VI Final remarks

In this work, I present the physical meaning of the Euler-Lagrange equations
for the optimization of work, heat, and energy, calling them thermodynamic
optimization equalities. Basically, the rates of the work, heat, and energy
are particularly proportional between themselves when optimization is
achieved. As an application, I proposed as a convergence criterion the
achievement of the thermodynamic optimization equalities with prescribed error
in the global optimization technique of genetic programming. I illustrate it
with the overdamped Brownian motion, observing agreement for work and heat.
Surprisingly, the process of the optimization of energy shows the non-
existence of an optimal protocol. For future studies, important questions are:
Do the thermodynamic optimization equalities break down for higher orders? How
do they behave in the slowly-varying regime and arbitrarily strong processes?

###### Acknowledgements.

I thank Maurice de Koning for the question made during my doctorate defense
that motivates the applicability of this work, and Marcus V. S. Bonança for
enlightening discussions.

## References

  * Deffner and Bonança (2020) S. Deffner and M. V. Bonança, Europhysics Letters 131, 20001 (2020).
  * Blaber and Sivak (2023) S. Blaber and D. A. Sivak, Journal of Physics Communications 7, 033001 (2023).
  * Nazé  _et al._ (2022) P. Nazé, S. Deffner, and M. V. Bonança, Journal of Physics Communications 6, 083001 (2022).
  * Duriez  _et al._ (2017) T. Duriez, S. L. Brunton, and B. R. Noack,  _Machine learning control-taming nonlinear dynamics and turbulence_, Vol. 116 (Springer, 2017).
  * Nazé (2023) P. Nazé, Physical Review E 107, 024120 (2023).
  * Kubo  _et al._ (2012) R. Kubo, M. Toda, and N. Hashitsume,  _Statistical physics II: nonequilibrium statistical mechanics_, Vol. 31 (Springer Science & Business Media, 2012).


#  Self-Conditioned Generative Adversarial Networks for Image Editing

Yunzhe Liu1 Rinon Gal2 Amit H. Bermano2 Baoquan Chen1 Daniel Cohen-Or2
CFCS, Peking University1
Tel Aviv University2

###### Abstract

Generative Adversarial Networks (GANs) are susceptible to bias, learned from
either the unbalanced data, or through mode collapse. The networks focus on
the core of the data distribution, leaving the tails — or the edges of the
distribution — behind. We argue that this bias is responsible not only for
fairness concerns, but that it plays a key role in the collapse of latent-
traversal editing methods when deviating away from the distribution’s core.
Building on this observation, we outline a method for mitigating generative
bias through a self-conditioning process, where distances in the latent-space
of a pre-trained generator are used to provide initial labels for the data. By
fine-tuning the generator on a re-sampled distribution drawn from these self-
labeled data, we force the generator to better contend with rare semantic
attributes and enable more realistic generation of these properties. We
compare our models to a wide range of latent editing methods, and show that by
alleviating the bias they achieve finer semantic control and better identity
preservation through a wider range of transformations. Our code and models
will be available at https://github.com/yzliu567/sc-gan.

##  1 Introduction

StyleFlow |  |  |  |  |
---|---|---|---|---|---
Ours |  |  |  |  |
StyleFlow |  |  |  |  |
Ours |  |  |  |  |
|  $\mathrel{\hbox{\char
40\relax}\mkern-4.0mu\hbox{\rule[-0.2pt]{85.35826pt}{0.4pt}}}$ Age
$\mathrel{\hbox{\rule[-0.2pt]{85.35826pt}{0.4pt}}\mkern-4.0mu\hbox{\char
41\relax}}$
Figure 1: Our method reduces the Generative Bias of neural networks, which
stems from the training data and the training scheme. Our self-conditioned
training scheme induces robust editing in regions where data is scarce. For
example, identity is better preserved during age manipulations, even for
ethnicities underrepresented in the training data.

Generative Adversarial Networks [14] (GANs) have shown remarkable performance
on a wide range of synthesis-related tasks. Recently, the structure of their
latent space has been thoroughly explored, giving birth to a wide range of
methods designed to manipulate the generated images in semantically meaningful
ways. However, GANs tend to suffer from two kinds of biases. The first, and
most intuitive type of bias, is the biased learned from the data. If two
attributes are highly correlated in the data (_e.g_., glasses and age) the
network learns to tie them together, leading to entanglement in the latent
space. The second kind of bias is an inherent Generative Bias, so called
because it is derived from the training process rather than merely the data
[44]. This bias can be found near the edges of the distribution, where data
exists - but in insufficient quantities. As these data points appear rarely,
the GAN naturally prefers to pay the cost of their miss-classification,
freeing it to devote more resources to denser regions.

We argue that this Generative Bias plays a key role in the poor performance of
GANs when tasked with synthesizing such rare instances. We focus on latent-
based image manipulations, and postulate that the failures of classical GAN-
based editing techniques can be traced in part to this same root. We propose
to tackle this bias by converting a pre-trained generator into a self-
conditioned model [21], where continuous conditioning labels are derived from
the latent structure of the generator itself. In doing so, we build on
existing linear editing methods and achieve superior quality, control and
identity preservation.

Our method consists of four steps: First, we find a linear editing direction
responsible for a faulty attribute which we wish to enhance. Such directions
can be found with weak supervision [32], in a zero-shot manner [24] or even in
an unsupervised fashion [33, 16]. Second, we build on prior observations that
latent space distances are linearly correlated with semantic attribute
strengths [22]. We can thus label the original training set by inverting all
images into the latent space of the network and calculating their latent
projections on the editing direction. Third, we convert the generator and the
discriminator into conditional variants, where we condition the generation on
the latent-space distance labels. Finally, we fine-tune the network with
samples that are drawn uniformly according to their latent-distance labels.

By following this approach, the conditional model is penalized for ignoring
the edges of the distribution, drawing it towards a more uniform output
distribution. Moreover, the editing directions are baked into the model’s
architecture, allowing for better semantic control. Our method additionally
enables multi-attribute editing, even for those cases where latent traversal
often fails. We validate our approach through a set of experiments and
demonstrate that it leads to more robust editing and better identity
preservation.

In summary, our key contributions are:

  * •

A framework for enhancing existing editing linear methods through self-
conditional learning.

  * •

An approach to editing that tackles shortcomings in the existing latent space
by unfreezing the generator and fine-tuning it towards a fairer
representation.

Figure 2: Overview of our training pipeline. Our model takes as an input a set
of linear editing directions and an unlabeled training set. We label the
training set by inverting each image into the latent space of the GAN, and
finding its projection on each of the linear editing directions. These labels
are then used to fine-tune a pre-trained StyleGAN model into a conditional
version.

##  2 Related Work

#### Bias in Generative Networks

Generative models have been extensively studied in the context of bias
evaluation and mitigation. Early works focused on fairer generation with the
aim of improving the performance of a downstream classifier [42, 30, 15, 10].
They employ importance re-weighting schemes [15], leverage a reference data
set [10], or train a generator conditioned on the biased attributes [42, 30].
Others analyze the role of inductive biases in the generative process [45] or
mitigate biases without training through better sampling of latent codes [34].
Yu _et al_. [44] studied Generative Bias in greater depth. They demonstrated
that when training GANs, minority modes are more likely to suffer from
collapse.

In contrast to these works, we investigate the bias as a root cause for poor
performance in editing. We employ a self-conditioned model and a re-sampling
scheme to mitigate this bias, stave off mode collapse, and achieve greater
editability for minority attributes.

#### Latent-Space Editing

The unprecedented ability of StyleGAN to encode semantic properties within its
latent space has spawned an impressive array of image manipulation methods
[32, 24, 33, 16, 40]. These methods typically aim to find linear directions in
the latent space of the GAN, such that modifying a latent code along these
directions will produce a change in a single semantic property of the
generated image. These methods range widely in the level of supervision they
require, from weak supervision in the form of binary attribute classification
[32] through detailed 3D morphable face models [35]. Others have proposed ways
to identify such semantic directions in an entirely unsupervised manner [33,
16] or in a zero-shot manner by leveraging models [26] that jointly encode
image and text [24]. Linear editing directions, however, typically suffer from
entanglement or rapidly deteriorating performance when applying large changes.
Recently, it has been suggested that these shortcomings can be tackled by
discovering non-linear paths in the latent space [3]. Typically these methods
train a network to perform local manipulations on a given latent code [2, 43,
20]. Others suggest to model the warped manifold of the GAN [38] or traverse
this manifold by finding a new local-basis at every step [9]. While such
methods have enjoyed relative success, we argue that they aim at solving the
problem by attacking the symptom rather than its cause. Instead of employing
complex methods to find non-linear directions in the space of a pre-trained
GAN, we propose to re-train the model and alleviate the bias which gives rise
to these flaws.

#### Generator Unfreezing

Typically, frameworks which utilize a pre-trained GAN for downstream tasks
elect to keep the GAN frozen. Doing so is expected to bring stability and
ensure higher quality. A recent line of works, however, challenges these
assumption and demonstrates that some tasks can be better handled by modifying
the generator itself [7].

The intuition is that brief fine-tuning sessions, whether adversarial [41] or
non adversarial [13], tend to keep the models well aligned and preserve most
of the structure of the latent space. While this property is typically used
for tasks that cannot be accomplished with the original generator, such as
out-of-domain editing [25, 13], it has been shown that modifying the generator
can also dramatically improve inversions [23, 28, 5], create new editing
directions [8], improve temporal consistency [37] or enable a user to re-write
the synthesis rules of the network [6].

In contrast to these works, we propose that generator fine-tuning can also be
used to improve on existing editing approaches. By converting implicitly
learned latent directions into explicit conditioning codes, we force the
generator to contend with minority attributes, reduce the Generative Bias, and
enable better control of the modified attributes under more significant
modifications.

##  3 Method

To tackle the Generative Bias and enable more robust editing, we propose a
scheme for converting a pre-trained generator and an existing linear editing
direction into a self-conditioned model. We do so by tackling three aspects of
the training flow: First, we generate self-labeled data using distances in the
latent-space of the generator. Second, we modify the generator’s architecture
in order to explicitly represent a multitude of rare modalities. Third, we
employ a re-sampled fine-tuning session with broadened discriminator
supervision. These modifications are outlined below. An overview of the
different steps is provided in Fig. 2.

###  3.1 Latent-Distances as Self-Conditional Labels

In a recent work, Nitzan  _et al_. [22] demonstrated that the latent space
distance of an image from a semantic editing hyperplane is linearly correlated
with the magnitude of the semantic attribute in the image. Drawing on their
insight, we propose to use these distances as labels for the original dataset.
These labels could then be used to fine-tune the generator into an unbiased
conditional model.

To self-label the data, we begin by extracting an editing hyperplane for each
property that we want to de-bias. The typical method for identifying such
planes is through the use of InterfaceGAN [32]. However, any method which
extracts linear editing directions in any of the GAN’s multiple latent spaces
is equally suitable. As we shall later demonstrate, our method can work
equally well with directions extracted by a wide array of methods in multiple
latent spaces, including: StyleCLIP [24] directions in $\mathcal{S}$ [40],
InterFaceGAN [32] directions in $\mathcal{W}$, and SeFA [33] directions in
$\mathcal{W}+$.

Armed with these editing directions, we next turn to labeling our training
data. We invert all training set images into the latent space of the network
using a pre-trained e4e [36] model. We then calculate, for each image, the
latent space distance between its inverted code and the editing hyperplane. We
find the minimal and maximal distances in the set and re-scale all distance
labels such that the range of possible values is in $[-1,1]$. We use these
distances as a set of continuous labels for each image.

In the case of methods which produce editing directions without an intercept,
we arbitrarily set the intercept to $0$ (_i.e_. we use the projection of the
latent code on the editing direction). As distances are then normalized, this
choice bears no effect on the results.

###  3.2 Conditional Multi-Constants

---
Figure 3: Overview of our multi-constant conditional generator. We treat the
controlled attributes as different data modalities, introduced through
modulations of the learned constant. Each modality has two additive constants,
denoting a positive and negative attribute change with respect to the mean
image. The direction and strength of the attributes is controlled through a
vector $\vec{\alpha}$, where each entry $\alpha_{i}$ controls a different
image property.

Using the self-labeled data, we turn to converting our generator into a
conditional model. Our goal is to enhance control and improve representation
of rare dataset modalities. We thus adopt the multi-constant approach proposed
by Sendik _et al_. [31].

Multi-constant models expand StyleGAN’s initial learned constant to a set of
constants, each of which is expected to control a different modality present
in the data. At inference time, a constant can be chosen either conditional on
a label, or by augmenting StyleGAN’s mapping network to output a weight vector
which denotes an importance-weighting score for each constant. One can then
choose the most dominant constant, or simply mix them together in proportion
to their weights. Sendik _et al_. [31] demonstrated that such a setup allows
the model to better encode the unique attributes relevant to each modality,
while allowing the rest of the network to share information from all
modalities, leading to higher quality synthesis. In our case, the information
sharing aspect is crucial as we are interested in better treatment of rare
modalities, which may not contain sufficient samples to train a high quality
GAN.

In practice, we modify the multi-constant approach in the following way: In
addition to StyleGAN’s original constant $C$, we add two new constants for
each attribute we wish to control: $c_{i}^{+},c_{i}^{-}$. The intuition here
is that many attributes may not be symmetric around the average image (_e.g_.
young faces differ significantly from old faces) and by introducing a constant
for each editing direction, we enable the network to model this discrepancy.
When synthesizing a new image, we provide the network with a score for each
attribute $\alpha_{i}\in[-1,1]$. We then use a ’mixed’ constant that contains
the information about our desired attribute strengths and directions:

$C^{\prime}=C+\Sigma_{i}|\alpha_{i}|*c_{i}^{sign\left(\alpha_{i}\right)}\leavevmode\nobreak\
.$ (1)

In contrast to Sendik _et al_. [31], our method uses additive constants for
each attribute. This allows us to learn a model which can simultaneously
control multiple attributes, without having to learn a constant for each
attribute combination. Moreover, this motivates the constants to focus only
the information which differs between the modalities, a simpler task which can
be tackled with fewer data.

Typically, conditional StyleGAN models inject the conditioning code through
the mapping network [17]. Adding such conditioning to a pre-trained model
through fine-tuning is non trivial, owing to the increased dimensionality of
the latent codes. In such a scenario, the mapping network has to be entirely
replaced, or part of the latent code has to be re-purposed for the
conditioning. Additive constants, meanwhile, can be used to modulate the
existing network without any re-learning of the core generative path. In Sec.
4 we investigate the option of replacing the mapping network with a
conditional one and find that this loss of information leads to poor editing
control and demeaned performance.

On the discriminator front, we employ the original StyleGAN2 [19]
discriminator and augment it with a new prediction head tasked with regressing
the mixture of modalities, $\alpha_{i}$, used in the image synthesis. In order
to help the network encode differences between asymmetric modalities such as
adding hats or glasses (where the ideal negative direction constant may simply
be a tensor of zeros), we once again separate positive and negative editing
directions. We do so by defining a 3-entry score vector for each attribute we
want to control: $[|\alpha_{i}|,1-|\alpha_{i}|,0]$ for $\alpha_{i}<0$ and
$[0,1-|\alpha_{i}|,|\alpha_{i}|]$ for $\alpha_{i}>0$. We employ a soft cross-
entropy loss between the discriminator’s predictions and these soft labels.
For real images, we substitute $\alpha_{i}$ with the normalized distance-
labels for each attribute in the image.

By combining these modifications, we create a conditional GAN architecture
where conditioning codes, $\alpha_{i}$, are injected through linear
interpolation of learned constants. Our discriminator is further incentivized
to pay attention to rarer modalities, making mode collapse less likely. The
full network architecture is outlined in Fig. 3.

###  3.3 Re-sampled Tuning

With the image labels and architecture modifications at hand, we turn to fine-
tuning the GAN. We continue training using the original, now labeled, dataset.
However, rather than sampling a random image at every iteration, we uniformly
sample a random score for each attribute and draw the nearest-neighbor image
in attribute space. By doing so, we further ensure that the discriminator sees
an unbiased distribution.

###  3.4 Image Inversion

An essential requirement for latent space editing methods is the ability to
modify real images, inverted into the latent space of the GAN [1, 4, 27]. As
our method extracts particular semantics out of the latent space and into a
set of explicit conditional labels, we find that codes in the original model
are no longer tied to the same identities in the new model. We are thus unable
to employ off-the-shelf encoders for use with our model. However, such models
can be easily adapted to our conditional setting. We do so by fine-tuning a
pre-trained e4e [36] model. The encoder’s function is unchanged. It receives
an image, and produces a latent code in $\mathcal{W}+$. This code is then fed
into our conditional generator, along with a set of latent-space distance
labels derived from the same input image in the original source model. These
distances can be efficiently approximated by simply passing the image through
the discriminator’s mode prediction head. The encoder is fine-tuned using the
same optimization goal and loss terms as in the original e4e model. A similar
approach can be used to integrate our model with PTI [28], enabling accurate
and highly editable reconstructions of real images.

###  3.5 Training Details

We train our models using the StyleGAN2-Pytorch implementation. We use the
official FFHQ [18] 1024x1024 and AFHQ-Cat [11] 512x512 checkpoints. FFHQ
models were fine-tuned using $30k$ iterations and a batch size of $4$. The
AFHQ model was fine tuned with $30k$ iterations and a batch size of $8$. e4e
models were fine-tuned using $50k$ iterations and a batch size of $2$. When
integrating with PTI, we use the latent provided by our fine-tuned e4e model
as a pivot. The generator optimization is performed over $350$ iterations.
Learning rates, relative loss weights and all other parameters were left
unmodified from the original models.

##  4 Experiments

###  4.1 Qualitative Comparison

|  |  |
---|---|---|---
Original |  $+$Age |  $+$Glasses |  $+$Pose
|  |  |
Original |  $+$Smile |  $+$Age |  $+$Glasses
|  |  |
Original |  $+$Smile |  $+$Glasses |  $+$Age
|  |  |
Original |  $+$Pose |  $+$Age |  $+$Glasses
|  |  |
Original |  $+$Age |  $+$Hair Color |  $+$Hair Length
Figure 4: Our framework enables sequential editing of multiple attributes, even when conditioned on latent distances derived from methods which do not. The self-conditioned model is insensitive to the order of editing operations, and can successfully synthesize rare combinations such as young people with glasses and a large pose. InterFaceGAN |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
Ours |  |  |  |  |  |  |
SeFa |  |  |  |  |  |  |
Ours |  |  |  |  |  |  |
| Input | Inversion |  $\mathrel{\hbox{\char 40\relax}\mkern-4.0mu\hbox{\rule[-0.2pt]{85.35826pt}{0.4pt}}}$ Pose $\mathrel{\hbox{\rule[-0.2pt]{85.35826pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$
StyleCLIP |  |  |  |  |  |  |
Ours |  |  |  |  |  |  |
SeFa |  |  |  |  |  |  |
Ours |  |  |  |  |  |  |
| Input | Inversion |  $\mathrel{\hbox{\char 40\relax}\mkern-4.0mu\hbox{\rule[-0.2pt]{85.35826pt}{0.4pt}}}$ Age $\mathrel{\hbox{\rule[-0.2pt]{85.35826pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$
StyleCLIP |  |  |  |  |  |  |
Ours |  |  |  |  |  |  |
| Input | Inversion |  $\mathrel{\hbox{\char 40\relax}\mkern-4.0mu\hbox{\rule[-0.2pt]{85.35826pt}{0.4pt}}}$ Glasses $\mathrel{\hbox{\rule[-0.2pt]{85.35826pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$
Figure 5: Linear editing comparisons on real images. In each pair of rows, we compare our method against the initial latent direction which was used to extract our self-conditioning labels. In all cases, our method better preserves the identity and allows for more significant manipulations before image quality suffers drastically. In the glasses example (bottom), continued movement in the negative direction on an image without glasses leads to an increase in age. Our model avoids this problem and produces the same identity. |  Local Basis[9] |  |  |  |  |
---|---|---|---|---|---
Ours |  |  |  |  |
| Age
|  Local Basis[9] |  |  |  |  |
---|---|---|---|---|---
Ours |  |  |  |  |
| Pose
|  StyleFlow |  |  |  |  |
---|---|---|---|---|---
Ours |  |  |  |  |
| Age
|  StyleFlow |  |  |  |  |
---|---|---|---|---|---
Ours |  |  |  |  |
| Pose
Figure 6: Comparisons to non-linear editing methods. Our model maintains
better identity when adjusting age, particularly in the case of racial
minorities. When modifying the pose, our model shows reduced corruption for
large changes.

We begin with a qualitative comparison of our method to existing editing
techniques. In Fig. 4 we show sequential editing of synthesized images using
our method. Our model is capable of successfully controlling multiple
attributes, across large modifications, without significantly compromising the
identity.

In Fig. 5 we compare our editing capabilities on real images to a range of
linear editing methods. Real images were inverted into our model and into the
official FFHQ 1024x1024 model, using PTI [28]. For each identity, we compare
the editing performance of our model to the baseline linear editing direction
which was used to extract the conditioning labels. Our method consistently
outperforms these baselines. The performance gap widens as we move closer to
the edges of the distribution, such as when considering large poses or old
faces. When considering asymmetric attributes such as glasses, our model
maintains the same image even as we keep moving in the no-glasses direction.
The same operation with the StyleCLIP baseline, meanwhile, leads to an
increase in age. These results demonstrate that editing performance can be
improved, with no additional supervision, simply by addressing the Generative
Bias and allowing the generator to be fine-tuned.

Finally, in Fig. 6 we compare our performance to non-linear alternatives:
Local Basis [9] and StyleFlow [2]. Recall that our method does not maintain
the identities of the original model. To facilitate comparisons in spite of
this limitation, we use the alternatives to edit images synthesized by the
original model, and project the same images into our models using PTI.

Despite the more challenging (inverted image) setup, our method still displays
more robust editing performance, maintaining better identity through age
changes and reducing deterioration for large poses. Importantly, unlike the
supervised StyleFlow, our method relies only on the weak (left / right pose
labels) or CLIP-based supervision used to find the initial editing directions
(InterFaceGAN for pose, StyleCLIP for age).

As our experiments demonstrate, self-conditioned GANs are capable of
consistent manipulations over larger spans than existing methods, even when
compared to complex, non-linear approaches.

###  4.2 Quantitative Comparisons

We quantitatively evaluate the performance of our method by considering
identity preservation for large modifications.

We follow [28] and measure the cosine similarity between the embeddings of a
pre-trained identity recognition network (ArcFace [12]) using pairs of pre-
and post-editing images. When considering continuous attributes, we ensure a
similar magnitude of change by employing pre-trained pose [29] and age
detection [39] networks. Images are edited until they reach a desired level of
change, _e.g_. $+20$ years. For binary attributes, we observe that for any
fixed step size, there exist a portion of images where the manipulation fails
(_e.g_. glasses are not added), and a portion of images where the manipulation
is too strong, and identity is lost. Increasing the step size leads to an
increase in both successful manipulations, and in identity loss. To facilitate
fairer comparisons, we thus test each method along a wide range of step sizes
and report the identity preservation scores as a function of the percent of
images that were successfully manipulated. A manipulation is considered
successful if it causes an off-the-shelf classifier [18] to change its result.

The results are shown in Fig. 7. Our model maintains a higher degree of
identity similarity for rare attributes (glasses) or in regions where data is
sparse and the baseline generator tends towards mode-collapse (large age,
pose), and performs on par with non-linear methods where data is abundant
(smiles).

|
---|---
|
Figure 7: Identity preservation graphs. Our model maintains better identity
for meaningful changes, and shows more moderate decline as we edit towards
regions where training data was scarce. For non-minority attributes like
smile, our model achieves comparable performance to non-linear editing
methods, and outperforms the linear editing directions from which our self-
labels were drawn.

###  4.3 Ablation study

|  Ours |  |  |
---|---|---|---
Conditional Mapping |  |  |
Ours |  |  |
Classifier Labels |  |  |
| Age $\mathrel{\hbox{\rule[-0.2pt]{85.35826pt}{0.4pt}}\mkern-4.0mu\hbox{\char
41\relax}}$
|  Ours |  |  |
---|---|---|---
Frozen Generator |  |  |
Ours |  |  |
Non-Uniform Sampling |  |  |
| Pose
$\mathrel{\hbox{\rule[-0.2pt]{85.35826pt}{0.4pt}}\mkern-4.0mu\hbox{\char
41\relax}}$
Figure 8: Qualitative ablation results. Using a conditional mapping network or
binary classifier labels leads to reduced range of control. Non-uniform
sampling leads to worsened performance for rare modalities. Not fine-tuning
the generator leads to lack of control or severe image corruptions when
attempting to modify the constant.

We evaluate different aspects of our proposed method by conducting a
qualitative ablation study. Specifically, we investigate our choice of a
multi-constant setup, the use of latent-space labels, the importance of
uniform sampling, and the benefits of unfreezing the generator.

In the first scenario, rather than spreading our modalities across constants,
we build on the conditional setup of [17] and extend the latent code with a
conditioning code that consists of one entry per property, with values
$\alpha_{i}\in\left[-1,1\right]$. For a model with control over $n$
attributes, the latent code therefore takes the form
$z^{\prime}=z\oplus\alpha_{1}\oplus\alpha_{2}\oplus...\oplus\alpha_{n}$. The
discriminator is similarly tasked with predicting the set
$\\{\alpha_{i}\\}_{i=1}^{n}$. For real images, the $\alpha$ values are given
by the self-labeling scores described in Sec. 3.1.

In the second scenario, we replace the latent-space distance labels with
confidence scores derived from a binary attribute classifier - the same
classifier used to generate InterFaceGAN directions.

In the third scenario, we fine-tune the model with images sampled randomly
from the FFHQ set, with no regard to their labels.

Finally, in the fourth scenario, we learn new constants without modifying any
of the pre-trained generator’s weights.

In Fig. 8 we show the results of each experiment. When replacing the constants
with a conditional mapping network, we observe a severely decreased range of
control. We hypothesize that, as the mapping network needs to be trained from
scratch, the optimization prefers to devote most efforts to re-aligning the
generated distribution rather than to the attribute control. If this is the
case, a change of hyper-parameters might enable successful conditional
editing, but we were unable to find such parameters.

Using classifier scores similarly leads to a reduced editing range. This is
likely a result of the quick saturation in classifier scores.

When we do not perform uniform sampling, editing range is similar to that of
our full model, but the quality of results near the edges of the distribution
deteriorates, demonstrating that the data bias contributes directly to the
performance in these regions.

Lastly, if the generator remains frozen, it fails to adapt to the changes in
constants. Attempts to manipulate the image through constant interpolations
lead to minor changes at best, and more commonly to severe quality
deterioration.

##  5 Discussion

We presented a self-conditioned generative model, designed to tackle the
inherent Generative Bias of GANs and improve image editing. Our method
leverages existing linear latent traversal methods, and empowers them to
properly deal with minority modalities. In doing so, it achieves better
identity preservation and direct control over image attributes.

Our results demonstrate the benefits of keeping fairness considerations in
mind when dealing with generative tasks. More importantly, they provide
further proof that there exist venues through which these, often painful,
biases can be mitigated — without having to collect additional, unbiased data.

While our network demonstrated improved performance in regions that suffer
from the Generative Bias – _i.e_., the GAN’s tendency towards mode collapse
around minority modalities – we observe that in some cases it can increase the
network’s susceptibility to the other kind of bias which affects the network -
entanglement of attributes in the dataset. However, in many cases this effect
can be alleviated by training a multi-attribute model to further condition on
the entangled attribute.

In the future, we would like to further develop mechanisms that better exploit
the existing data in order to promote better control over specific attributes
of interest. We hope that our work serves as additional motivation for further
explorations into fairer generative models, as these can impact both ethical
and practical concerns.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432–4441, 2019.
  * [2] Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka.  Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows, 2020.
  * [3] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or.  Only a matter of style: Age transformation using a style-based regression model.  ACM Trans. Graph., 40(4), 2021.
  * [4] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or.  Restyle: A residual-based stylegan encoder via iterative refinement.  arXiv preprint arXiv:2104.02699, 2021.
  * [5] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit H. Bermano.  Hyperstyle: Stylegan inversion with hypernetworks for real image editing, 2021.
  * [6] David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba.  Rewriting a deep generative model.  In European Conference on Computer Vision, pages 351–369. Springer, 2020.
  * [7] Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam.  Optimizing the latent space of generative networks.  In International Conference on Machine Learning, pages 600–609. PMLR, 2018.
  * [8] Anton Cherepkov, Andrey Voynov, and Artem Babenko.  Navigating the gan parameter space for semantic image editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3671–3680, 2021.
  * [9] Jaewoong Choi, Junho Lee, Changyeon Yoon, Jung Ho Park, Geonho Hwang, and Myungjoo Kang.  Do not escape from the manifold: Discovering the local coordinates on the latent space of gans, 2021.
  * [10] Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon.  Fair generative modeling via weak supervision.  In International Conference on Machine Learning, pages 1887–1898. PMLR, 2020.
  * [11] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [12] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [13] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators, 2021\.
  * [14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [15] Aditya Grover, Jiaming Song, Alekh Agarwal, Kenneth Tran, Ashish Kapoor, Eric Horvitz, and Stefano Ermon.  Bias correction of learned generative models using likelihood-free importance weighting.  In Advances in Neural Information Processing Systems, 2019.
  * [16] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  arXiv preprint arXiv:2004.02546, 2020.
  * [17] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  In Proc. NeurIPS, 2020.
  * [18] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [19] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  * [20] Bingchuan Li, Shaofei Cai, Wei Liu, Peng Zhang, Miao Hua, Qian He, and Zili Yi.  Dystyle: Dynamic neural network for multi-attribute-conditioned style editing, 2021.
  * [21] Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, and Antonio Torralba.  Diverse image generation via self-conditioned gans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14286–14295, 2020.
  * [22] Yotam Nitzan, Rinon Gal, Ofir Brenner, and Daniel Cohen-Or.  Large: Latent-based regression through gan semantics, 2021.
  * [23] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo.  Exploiting deep generative prior for versatile image restoration and manipulation.  In European Conference on Computer Vision, pages 262–277. Springer, 2020.
  * [24] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery, 2021.
  * [25] Justin NM Pinkney and Doron Adler.  Resolution dependent gan interpolation for controllable image synthesis between domains.  arXiv preprint arXiv:2010.05334, 2020.
  * [26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [27] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  arXiv preprint arXiv:2008.00951, 2020.
  * [28] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  arXiv preprint arXiv:2106.05744, 2021.
  * [29] Nataniel Ruiz, Eunji Chong, and James M. Rehg.  Fine-grained head pose estimation without keypoints.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
  * [30] Prasanna Sattigeri, Samuel C Hoffman, Vijil Chenthamarakshan, and Kush R Varshney.  Fairness gan: Generating datasets with fairness properties using a generative adversarial network.  IBM Journal of Research and Development, 63(4/5):3–1, 2019.
  * [31] Omry Sendik, Dani Lischinski, and Daniel Cohen-Or.  Unsupervised k-modal styled content generation.  ACM Transactions on Graphics (TOG), 39(4):100–1, 2020.
  * [32] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9243–9252, 2020.
  * [33] Yujun Shen and Bolei Zhou.  Closed-form factorization of latent semantics in gans.  arXiv preprint arXiv:2007.06600, 2020.
  * [34] Shuhan Tan, Yujun Shen, and Bolei Zhou.  Improving the fairness of deep generative models without retraining.  arXiv preprint arXiv:2012.04842, 2020.
  * [35] Ayush Tewari, Mohamed A. Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt.  Stylerig: Rigging stylegan for 3d control over portrait images.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6141–6150, 2020.
  * [36] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  arXiv preprint arXiv:2102.02766, 2021.
  * [37] Rotem Tzaban, Ron Mokady, Rinon Gal, Amit H. Bermano, and Daniel Cohen-Or.  Stitch it in time: Gan-based facial editing of real videos, 2022.
  * [38] Christos Tzelepis, Georgios Tzimiropoulos, and Ioannis Patras.  WarpedGANSpace: Finding non-linear rbf paths in GAN latent space.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6393–6402, October 2021.
  * [39] Yusuke Uchida.  Age estimation - pytorch, 2018.  https://github.com/yu4u/age-estimation-pytorch.
  * [40] Zongze Wu, Dani Lischinski, and Eli Shechtman.  Stylespace analysis: Disentangled controls for stylegan image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12863–12872, 2021.
  * [41] Zongze Wu, Yotam Nitzan, Eli Shechtman, and Dani Lischinski.  Stylealign: Analysis and applications of aligned stylegan models, 2021\.
  * [42] Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu.  Fairgan: Fairness-aware generative adversarial networks.  In 2018 IEEE International Conference on Big Data (Big Data), pages 570–575. IEEE, 2018.
  * [43] Xu Yao, Alasdair Newson, Yann Gousseau, and Pierre Hellier.  A latent transformer for disentangled face editing in images and videos.  2021 International Conference on Computer Vision, 2021.
  * [44] Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz.  Inclusive gan: Improving data and minority coverage in generative models.  In European Conference on Computer Vision, pages 377–393. Springer, 2020.
  * [45] Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon.  Bias and generalization in deep generative models: An empirical study.  arXiv preprint arXiv:1811.03259, 2018.


# Better call Surrogates: A hybrid Evolutionary Algorithm for Hyperparameter
optimization

Subhodip Biswas
Computer Science, Virginia Tech
Arlington, VA 22203
<EMAIL_ADDRESS>
&Adam D. Cobb
US Army Research Laboratory
Adelphi, MD 20783
<EMAIL_ADDRESS>
&Andreea Sistrunk
US Army, ERDC-GRL
Alexandria, VA 22315
<EMAIL_ADDRESS>
&Naren Ramakrishnan
Computer Science, Virginia Tech
Arlington, VA 22203
<EMAIL_ADDRESS>
&Brian Jalaian
US Army Research Laboratory
Adelphi, MD 20783
<EMAIL_ADDRESS>

###### Abstract

In this paper, we propose a surrogate-assisted evolutionary algorithm (EA) for
hyperparameter optimization of machine learning (ML) models. The proposed
SteaDE model initially estimates the objective function landscape using Radial
Basis Function interpolation, and then transfers the knowledge to an EA
technique called Differential Evolution that is used to evolve new solutions
guided by a Bayesian optimization framework. We empirically evaluate our model
on the hyperparameter optimization problems as a part of the black box
optimization challenge at NeurIPS 2020, and demonstrate the improvement
brought about by SteaDE over the vanilla EA.

##  1 Introduction

The empirical performance of ML models can be enhanced by configuring its
hyperparameters. However, manually searching for a good configuration of
hyperparameters is nearly impossible due to the exponentially large size of
the parameter space. This has led to an increased interest from the research
community of recent in devising automated ways of searching for optimal
hyperparameter configuration, popularly known as hyperparameter optimization
(HPO) [10]. HPO is a blackbox optimization problem, where the objective
function $f$ is usually expensive to evaluate, and no other information like
gradient, presence of special structures like linearity/concavity is
available. The objective in HPO is to seek the optimum of blackbox function
$f$, which is usually some error metric quantifying the performance of a ML
model, by expending as little computational budget as possible.

EAs belong to a class of stochastic derivative-free optimization algorithms
[8]. In spite of being fast, EAs employ random search moves that result in low
sample efficiency, i.e., the expected gain in the objective function per unit
of function evaluation (FE). Hence EAs are preferred when the FEs are cheap to
compute and a high evaluation budget is available. On the other hand,
surrogate-based optimization algorithms have high data efficiency as they
approximate the objective function [12], but incur high cost of suggesting new
points and hence they are used in solving computationally expensive
optimization problems [16, 3]. Radial Basis Functions (RBFs) [4] and Gaussian
Processes (GPs) [19] are popular choices when it comes to surrogate models.
Motivated by this, we devise a hybrid search algorithm for adapting EAs to low
budget optimization problems, like HPO, by balancing the randomness of EA
search moves with strategically generated search points via informative
surrogates. As detailed next, the proposed SteaDE uses a mix of surrogate
models (RBF and GP) and mutation mechanism (based on EA) for solving HPO
problems.

##  2 The SteaDE algorithm

Our algorithm Surrogate-assisted Bayesian Differential Evolution (SteaDE)
starts with a stochastic RBF method [14, 15], which is parallel surrogate
optimization algorithm, to estimate the functional landscape and then switches
to an EA-based scheme guided by Bayesian Optimization (BO) [3]. The design
rationale is to perform initial exploration of the parameter space using the
RBF model, and then transfer the knowledge to a GP-based Bayesian optimization
framework, which is further augmented by a Differential Evolution (DE)
algorithm [18]. DE is a population-based real-parameter global optimization
technique that has gained widespread popularity as it is fast and reliable,
easy to implement, highly flexible, and robust to simple transformations [7].
Recent work has shown the advantage of information sharing in DE framework
specially for solving multimodal optimization [2].

SteaDE begins by instantiating an experimental design like symmetric Latin
hypercube design (SLHD)111We use SLHD as it allows an arbitrary number of
design points/trial solutions and works well in practice [6] to generate the
design points (trial solutions) and evaluate them. The RBF-based surrogate
model $\mathcal{R}$ is then used to approximate the objective function [14].
After building the model, we solve an auxiliary problem by optimizing the
acquisition function to generate trial solution(s) for evaluation. In an
iterative manner, we evaluate the trial solution(s) and update the surrogate
model till $\lambda$ iterations. Then we switch to DE-based search for
generating trial solution(s) till a termination criterion is met.

DE maintains a population $\mathbf{X}$ of $Np$ randomly generated
$D$-dimensional real-parameter vectors representing trial solutions to a
problem. We can represent the $i$th solution (also called target vector),
$i\in\left[1,2,\ldots,Np\right]$, of the population at the current generation
$G$, $G=0,1,\ldots,G_{\max}$, as

$\vec{X}{\vphantom{g}}_{i,G}=\left[x{\vphantom{g}}_{1,i,G},x{\vphantom{g}}_{2,i,G},x{\vphantom{g}}_{3,i,G},\ldots,x{\vphantom{g}}_{D,i,G}\right].$

Following the initialization, the solutions are improved iteratively through a
series of steps$-$mutation, crossover or recombination, and selection$-$till a
termination criterion is met. For an in-depth understanding of the DE
algorithm, kindly refer to [7]. In our approach, we propose a Bayesian
mutation mechanism to evolve new solutions (also called donor vectors) as
follows.

$\vec{V}{\vphantom{g}}_{i,G}=\vec{B}{\vphantom{g}}_{r^{i}_{1},G}+F\cdot\left(\vec{R}{\vphantom{g}}_{r^{i}_{2},G}-\vec{R}{\vphantom{g}}_{r^{i}_{3},G}\right)+\frac{1}{G}\cdot
rand\left(1,D\right)\cdot\left(\vec{X}{\vphantom{g}}_{r^{i}_{2},G}-\vec{X}{\vphantom{g}}_{r^{i}_{3},G}\right),$
(1)

where the indices $r^{i}_{1}$, $r^{i}_{2}$, and $r^{i}_{3}$ are mutually
exclusive integers randomly chosen from the range $\left[1,Np\right]$ such
that $r^{i}_{1}\neq r^{i}_{2}\neq r^{i}_{3}\neq i$, $rand\left(1,D\right)$ is
a $D$-dimensional vector of uniformly distributed random numbers lying between
$0$ and $1$, $F$ is a scaling factor that typically lies in the interval
$\left[0.4,1\right]$, $\vec{B}$ and $\vec{R}$ are the populations of trial
solution(s) generated by a BO-based model and the RBF-based method,
respectively. The Bayesian mutation leverages the sample-efficiency of BO to
search for solutions at promising regions of the parameter-space. The second
component in Eq. (1) acts as a global search (exploratory) component while the
third component gradually transitions from a global to local search as $G$
increases (and $1/G$ shrinks accordingly).

Next, the trial vectors are generated via the crossover operation in which
every donor vector exchanges its components/features with its corresponding
target vector. DE can use two types of crossover$-$ binomial and exponential.
Due to its flexibility, we adopt the binomial crossover as

$u_{j,i,G}=\left\\{\begin{matrix}v_{j,i,G}&\text{if}\;(rand_{i,j}[0,1]\leq Cr\
or\ j==j_{rand})\\\ x_{j,i,G}&\text{otherwise}\hfill\end{matrix}\right.,$ (2)

where the crossover rate $Cr$ approximates the probability of recombination,
$rand{\vphantom{g}}_{i,j}[0,1]$ is an uniform random number generated for
every $(i,j)$ pair, $j{\vphantom{g}}_{rand}\in[1,\ldots,D]$ is a randomly
chosen index to ensure that $\vec{U}{\vphantom{g}}_{i,G}$ has at least one
component from $\vec{V}{\vphantom{g}}_{i,G}$. Binomial crossover ensures that
the number of components inherited by the trial vector from the donor vector
roughly follows a binomial distribution. Finally, a fitness-based selection
mechanism takes place between the trial vector and the mutant vector to select
the fitter one. The pseudocode of the SteaDE is outlined in Algorithm 1.

We use the DE/rand/2/bin model where two scaled difference vectors are used to
mutate the target vector (generated by BO). We set the scaling factor $F$ and
crossover rate $Cr$ to 0.7, $\lambda$ to 10 for activating the Bayesian
mutation, and use population size $Np$ of 16 as per the competition
guidelines.

Input : Objective function $f$, Parameter space $\mathbf{\Lambda}$, Population
size $Np$, Evaluation budget $G{\vphantom{g}}_{\max}$

Result: Best possible solution to $f$

begin

Select an experimental design to instantiate trial solutions
$\mathbf{X}{\vphantom{g}}_{0}\in\mathbf{\Lambda}$ and evaluate them
$f(\mathbf{X}{\vphantom{g}}_{0})$

Use $f(\mathbf{X}{\vphantom{g}}_{0})$ to build a surrogate model based on RBF

// Iterative improvement of the solutions till a termination criterion is met

for  _$G=1,2,\ldots,G{\vphantom{g}}_{\max}$_ do

if  _$G <\lambda$_ then

Generate trial solutions
$\vec{U}{\vphantom{g}}_{i,G},\,i\in\left[1,\ldots,Np\right]$ using the RBF-
based method

else

Generate solutions $\vec{\mathcal{B}}$ and $\vec{\mathcal{R}}$ from the RBF-
based and BO-based model respectively

Perform Bayesian mutation (1) to generate donor vectors/solutions

Simulate binomial crossover (2) to produce trial solutions
$\vec{U}{\vphantom{g}}_{i,G}$

Perform fitness-based selection to update the trial solutions as
$\vec{X}{\vphantom{g}}_{i,G+1}=\left\\{\begin{matrix}\vec{U}{\vphantom{g}}_{i,G}&\text{if}\;f(\vec{U}{\vphantom{g}}_{i,G})\leq
f(\vec{X}{\vphantom{g}}_{i,G})\\\
\vec{X}{\vphantom{g}}_{i,G}&\text{otherwise}\hfill\end{matrix}\right.,$ Use
the trial solutions $\mathbf{X}{\vphantom{g}}_{G+1}$ to update the surrogate
models.

Output the best solution
$\vec{X}{\vphantom{g}}_{*}\in\mathbf{X}{\vphantom{g}}_{G_{\max}}$

Algorithm 1 SteaDE algorithm

###  2.1 Implementation details

Our SteaDE is built on top of the Bayesmark222Available in GitHub at
https://github.com/uber/bayesmark. package, and uses its in-built space
functionality to translate the parameter-space into a continuous search space
using the idea of warping [17].

The python Surrogate Optimization Toolbox (pySOT) is a library of surrogate
optimization techniques leveraging the Plumbing for Optimization with
Asynchronous Parallelism (POAP) framework. For more details about these
frameworks, kindly refer to [9]. Our code is based on pySOT v0.3.3 and POAP
v0.1.26. We use a SLHD experimental design, a stochastic RBF surrogate with a
cubic kernel and linear tail [14], and the DYCORS strategy [15] for generating
trial solutions, i.e., $\vec{\mathcal{R}}$.

The other surrogate model is GP-based and is implemented using BoTorch, a
recent programming framework that includes advanced Bayesian optimization
techniques [1]. Since GPs involve expensive computation, we use batch BO to
reduce the computation time. In batch BO, the design points are selected in
parallel by doing joint optimization over the multiple design points. To do
that we use the parallel counterpart of the sequential acquisition functions
(EI, PI, UCB) [20]. In our implementation, we use qEI acquisition function
[11] with randomized quasi-Monte Carlo (RQMC) sampling based on scrambled
Sobol sequences for sample average approximation [5, 13]. The code of SteaDE
is made available at https://github.com/subhodipbiswas/BayesianEvolution.

##  3 Experimentation

###  3.1 Methodology

The SteaDE model was developed for the blackbox optimization challenge333The
starter kit and implementation details are available at
github.com/rdturnermtl/bbo_challenge_starter_kit. at NeurIPS 2020 under the
team name Better call Bayes. The experimental setup consists of ML
hyperparameter tuning problems. The competition guidelines allowed each
optimizer to run for $16$ iterations using a batch size of $8$ for making
suggestions on each benchmark problem. There was a strict time limit of 640
$s$ (or 40 $s$/iteration) which was not to be exceeded. Optimizers exceeding
the time limits were cut off from making further suggestions and the best
optima found till then was used. We simulated $15$ runs444Each run consists of
testing $9$ ML models on $6$ datasets using $2$ metric, thereby resulting in
$108$ studies. of SteaDE on the benchmark resulting in $1620$ independent
studies. The Bayesmark library was used to compute the performance of the
optimizer on different benchmark problems.

Since SteaDE is an composed of three components (RBF, BO and DE), we do
ablation testing by selectively simulating each component under identical
experimental conditions. The visible_to_opt, generalization and
leaderboard555This score is based on the simulations performed in the local
machine, and is different from our actual leaderboard score of $89.846$.
scores are reported in Table 1. The visible_to_opt is the score seen by an
optimizer (e.g, in a ML hyper-parameter tuning problem this can be the cross-
validation error), whereas generalization is a related metric the optimizer
does not get to see (e.g, error on held-out test set in a ML hyper-parameter
tuning context). For more details on how the mean, median and normalized mean
are calculated, kindly refer to the documentation666See
bayesmark.readthedocs.io/en/latest/scoring.html#analyze-and-summarize-results
for scoring details.. We also plot the convergence profile of the different
optimizers in Figure 1 based on the visible_to_opt score.

Table 1: Comparative analysis of SteaDE with its component algorithms. Models | visible_to_opt | generalization | Leaderboard
---|---|---|---
Median | Mean | Normalized Mean | Median | Mean | Normalized Mean
BO | 0.10906 | 0.03151 | 0.37376 | 2.36782 | 0.29934 | 2.58823 | 96.8487
DE | 0.21536 | 0.02817 | 0.33408 | 2.5 | 0.29478 | 2.54886 | 97.1833
RBF | 0.02002 | 0.01998 | 0.23693 | 2.62130 | 0.30189 | 2.61027 | 98.0024
STEADE | 0.10301 | 0.01304 | 0.15463 | 2.51204 | 0.297115 | 2.56901 | 98.6963

Figure 1: The median and mean visible_to_opt score obtained by the different
algorithms.

###  3.2 Results and Discussions

In Figure 1, we notice that the random nature of DE search move results in low
search efficiency during the initial stages of run while the surrogate-based
models perform informed search due to their functional estimation capability.
The search efficiency of the surrogate-based models gradually fall and after a
certain point, there seems to be a saturation in a model’s knowledge about the
functional landscape. In such scenarios, one plausible way is to switch to a
different model for carrying out further search. In SteaDE, we transfer the
knowledge from an RBF interpolation to a GP-based model. The solutions
generated by the GP are further perturbed using a DE-based mutation. The
efficacy of our approach is evident from Table 1, wherein SteaDE is able to
achieve better normalized mean score and leaderboard score in comparison to
the baselines.

Even though DE has been extensively applied to solve cheap optimization
problems, interestingly, very few works have explored its applicability to
low-budget (computationally expensive) optimization problems. In this work, we
take a step towards this direction and demonstrate how EAs can benefit from
surrogate models in such scenarios. However, the increased cost of surrogates
somewhat limits the applicability of such hybrid models to cheap/moderate
budget optimization problems. In order to make that possible, we need to
design careful learning mechanisms that will transition from the expensive
surrogate-based model to a cheap EA model depending on the perceived gain in
functional value with respect to computational time. This is a possible
research direction that we wish to undertake in near future.

## References

  * [1] Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy.  Botorch: A framework for efficient monte-carlo bayesian optimization.  Advances in Neural Information Processing Systems, 33, 2020.
  * [2] Subhodip Biswas, Souvik Kundu, and Swagatam Das.  Inducing niching behavior in differential evolution through local information sharing.  IEEE Transactions on Evolutionary Computation, 19(2):246–263, 2014\.
  * [3] Eric Brochu, Vlad M Cora, and Nando De Freitas.  A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning.  arXiv preprint arXiv:1012.2599, 2010.
  * [4] Martin D Buhmann.  Radial basis functions: Theory and implementations, volume 12.  Cambridge university press, 2003.
  * [5] Russel E Caflisch et al.  Monte carlo and quasi-monte carlo methods.  Acta numerica, 1998:1–49, 1998.
  * [6] Alberto Costa and Giacomo Nannicini.  RBFOpt: an open-source library for black-box optimization with costly function evaluations.  Mathematical Programming Computation, 10(4):597–629, 2018.
  * [7] Swagatam Das and Ponnuthurai Nagaratnam Suganthan.  Differential Evolution: A Survey of the State-of-the-Art.  IEEE Transactions on Evolutionary Computation, 15(1):4–31, 2011\.
  * [8] Agoston E Eiben and James E Smith.  Introduction to Evolutionary Computing.  Springer, 2015.
  * [9] David Eriksson, David Bindel, and Christine A Shoemaker.  pysot and poap: An event-driven asynchronous framework for surrogate optimization.  arXiv preprint arXiv:1908.00420, 2019.
  * [10] Matthias Feurer and Frank Hutter.  Hyperparameter optimization.  In Automated Machine Learning, pages 3–33. Springer, Cham, 2019\.
  * [11] Donald R Jones, Matthias Schonlau, and William J Welch.  Efficient global optimization of expensive black-box functions.  Journal of Global optimization, 13(4):455–492, 1998.
  * [12] Slawomir Koziel, David Echeverría Ciaurri, and Leifur Leifsson.  Surrogate-Based Methods, pages 33–59.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2011.
  * [13] Art B Owen.  Quasi-monte carlo sampling.  Monte Carlo Ray Tracing: SIGGRAPH, 1:69–88, 2003.
  * [14] Rommel G Regis and Christine A Shoemaker.  A stochastic radial basis function method for the global optimization of expensive functions.  INFORMS Journal on Computing, 19(4):497–509, 2007.
  * [15] Rommel G Regis and Christine A Shoemaker.  Combining radial basis function surrogates and dynamic coordinate search in high-dimensional expensive black-box optimization.  Engineering Optimization, 45(5):529–555, 2013.
  * [16] Jasper Snoek, Hugo Larochelle, and Ryan P Adams.  Practical bayesian optimization of machine learning algorithms.  In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, pages 2951–2959. Curran Associates, Inc., 2012.
  * [17] Jasper Snoek, Kevin Swersky, Rich Zemel, and Ryan Adams.  Input warping for bayesian optimization of non-stationary functions.  In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1674–1682, Bejing, China, 22–24 Jun 2014. PMLR.
  * [18] Rainer M. Storn and Kenneth V. Price.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces.  Journal of global optimization, 11(4):341–359, 1997.
  * [19] Christopher KI Williams and Carl Edward Rasmussen.  Gaussian processes for machine learning, volume 2.  MIT press Cambridge, MA, 2006.
  * [20] James Wilson, Frank Hutter, and Marc Deisenroth.  Maximizing acquisition functions for bayesian optimization.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 9884–9895. Curran Associates, Inc., 2018.