# eVAE: Evolutionary Variational Autoencoder

Zhangkai Wu,1 Longbing Cao, 1 Lei Qi 2

###### Abstract

The surrogate loss of variational autoencoders (VAEs) poses various challenges
to their training, inducing the imbalance between task fitting and
representation inference. To avert this, the existing strategies for VAEs
focus on adjusting the tradeoff by introducing hyperparameters, deriving a
tighter bound under some mild assumptions, or decomposing the loss components
per certain neural settings. VAEs still suffer from uncertain tradeoff
learning.We propose a novel evolutionary variational autoencoder (eVAE)
building on the variational information bottleneck (VIB) theory and
integrative evolutionary neural learning. eVAE integrates a variational
genetic algorithm into VAE with variational evolutionary operators including
variational mutation, crossover, and evolution. Its inner-outer-joint training
mechanism synergistically and dynamically generates and updates the uncertain
tradeoff learning in the evidence lower bound (ELBO) without additional
constraints. Apart from learning a lossy compression and representation of
data under the VIB assumption, eVAE presents an evolutionary paradigm to tune
critical factors of VAEs and deep neural networks and addresses the premature
convergence and random search problem by integrating evolutionary optimization
into deep learning. Experiments show that eVAE addresses the KL-vanishing
problem for text generation with low reconstruction loss, generates all
disentangled factors with sharp images, and improves the image generation
quality,respectively. eVAE achieves better reconstruction loss,
disentanglement, and generation-inference balance than its competitors.

## Introduction

Variational autoencoders (Kingma and Welling 2013) (VAEs) have attracted
significant interest for their capability of learning continuous and smooth
distributions from observations by integrating probabilistic and deep neural
learning principles. However, VAEs still face some significant issues,
including dynamic uncertainty learning during variational inference and the
tradeoff between representation compression and task fitting, despite various
recent VAE mechanisms and variants. Below, we briefly analyze these issues and
the gaps of existing solutions. A novel evolutionary VAE (eVAE) framework is
then introduced to address some of these issues and gaps.

#### VAEs and Gap Analysis

VAEs have demonstrated significant advantages of incorporating prior
knowledge, mapping inputs to probabilistic representations, and approximating
the likelihood of outputs. The integration of stochastic gradient variational
Bayes (SGVB) estimator (Kingma and Welling 2013) with neural settings learns a
narrow probabilistic latent space to infer more representative attributes in
the hidden space. VAEs have been applied in various domains, including time
series forecasting (Fortuin et al. 2019), out-of-domain detection in images
(Nalisnick et al. 2019; Ran et al. 2022; Havtorn et al. 2021; Xiao, Yan, and
Amit 2020), generating images with spiking signals (Kamata, Mukuta, and Harada
2022), and generating text by language modeling (Zhang et al. 2019).

Theoretically, the bound optimization of variational inference was introduced
to substitute the log-likelihood function with a surrogate function to make it
optimized by gradient descent. In practice, the evidence lower bound (ELBO)
cannot approach the maximum of conditional likelihood with a close gap between
posterior and prior. It causes dynamic uncertainty during inference and failed
tradeoff between representation robustness and reconstruction effectiveness.
Weak KL divergence could incur KL vanishing while strong KL divergence could
result in bad likelihood. The tradeoff is sensitive to posterior distribution
disjoint, data characteristics, and network architectures (Bozkurt et al.
2021).

Recently, various techniques have been proposed to address these issues. The
first set aims to adjust the balance in objective functions. Examples are
$\beta$-VAE incorporating a hyperparameter $\beta$ (Higgins et al. 2017),
InfoVAE adding a scaling parameter to the KL term (Zhao, Song, and Ermon
2019), SA-VAE having a cyclical annealing schedule to progressively increase
$\beta$ for reducing KL vanishing (Fu et al. 2019), and ControlVAE introducing
the proportional-integral-derivative (PID) control to tune the hyperparameter
(Shao et al. 2020). They are partial solutions only adjusting one part of the
objectives, failing to weigh and resolve the balance issue in dynamic
settings. Another set is to tune network architectures or VAE settings, e.g.,
VQ-VAE and their variants (Razavi, Van den Oord, and Vinyals 2019; Berliner et
al. 2022), spiking VAE (Kamata, Mukuta, and Harada 2022), multimodal VAE
(Sutter, Daunhawer, and Vogt 2021), mixture-of-experts multimodal VAE (Shi et
al. 2019) and tuning of their ELBO (Kviman et al. 2022; Jankowiak and Phan
2022) building on the product of experts structure. However, these models are
subject to specific tasks or data settings.

eVAE From the information bottleneck perspective, VAEs can be treated as a
lossy information compression process. Adjusting the KL within a range
controls the information bottleneck flowing from representing latent variables
to reconstructing samples, thus benefiting the trade-off between compression
and reconstruction (Alemi et al. 2017; Burgess et al. 2018). This variational
information bottleneck inspires us to improve the VAE mechanism and objective
function. To directly address the generation-inference balance in an evolving
dynamic setting without constraints on networks and inputs, we introduce a
novel evolutionary VAE (eVAE). eVAE incorporates and integrates variational
evolutionary learning and variational information bottleneck into VAE to
achieve better optimum exploration and the tradeoff between representation
compression and generation fitting. However, integrating evolutionary learning
into VAE is an open topic. We propose the variational genetic algorithm into
eVAE to optimize VAE objectives and its modeling exploration in an evolving
manner.

Consequently, eVAE dynamically optimizes the VAE inference (the KL terms) in
an evolutionary and probabilistic manner, which further tunes the VAE
generation toward balancing generation and inference. To avert premature
convergence, eVAE introduces probabilistic chromosome selection for smooth
search. To avoid exhaustive random search, we apply the simulated binary
crossover (Deb and Beyer 2001) and Cauchy distributional mutation to guide the
training toward a stable convergence. The main contributions of this work are
as follows:

  * •

We propose the eVAE model to balance the tradeoff between generation and
inference. Specifically, a Cauchy-based variational mutation operator and an
SBX-based variational crossover operator are proposed to train a model with an
evolving inner-outer-joint training algorithm, tackling the premature
convergence and random search problem when optimizing deep models assisted
with a variational genetic algorithm.

  * •

We illustrate eVAE by deriving the lower bound on $\beta$-VAE related models
under the information bottleneck theory. We analyze the information flow in
the corresponding lower bound per the rate-distortion theory to show that only
the iteration-specific lower bound in eVAE can capture the optimization trend
to train effectively.

  * •

We evaluate eVAE on three tasks: disentangled representation learning on
dSprites, image generation on CelebA, and text generation on PTB corpus,
respectively.

eVAE forms the first evolutionary VAE framework without empowering constraints
on VAE architectures, input settings, and objective function (ELBO). It
integrates variational encoding and decoding, information bottleneck, and
evolutionary learning into the deep neural framework. The evaluation against
the state-of-the-art VAEs shows that eVAE achieves better disentanglement,
reconstruction loss, and solve the KL vanishing effectively.

## VAE Background and Issues

Here, we briefly introduce the background of VAEs by focusing on their
objectives and approaches to addressing the inference-generation balance since
they are mostly relevant to this work. Other developments on VAEs such as
adjusting VAE network structures, input settings, and learning constraints are
excluded due to their irrelevance to this work.

#### VAE

VAEs mitigate autoencoder issues like sparse representation (Tolstikhin et al.
2017) by learning continuous and smooth representation distribution $p(x)$,
$x\in\mathcal{X}$ from observations $\mathcal{X}$ over latent variables
$\boldsymbol{z}$. After learning an encoding distribution
$q_{\phi}(\boldsymbol{z}\mid\boldsymbol{x})$ in encoding neural networks, VAEs
apply variational inference to approximate the posterior distribution
$p_{\theta}(\boldsymbol{x}\mid\boldsymbol{z})$. Learning tasks such as
reconstructed and generated outputs can then be sampled from this learned
distribution in a generative process. With the SGVB estimator and
reparameterization trick, the gradients become tractable, and the generative
parameters $\theta$ and inference parameters $\phi$ are learnable. The
objectives of VAEs can be converted to ELBO with the expectation over
empirical distribution $p_{\text{data}}$ of the data towards both
reconstruction $\mathcal{L}_{\mathrm{R}}$ and inference
$\mathcal{L}_{\mathrm{I}}$ (Esmaeili et al. 2019; Ghosh et al. 2019).

$\displaystyle\mathcal{L}_{ELBO}$ $\displaystyle=E_{x\sim p_{\text{data
}}}\left[E_{q_{\phi}(\boldsymbol{z}\mid\boldsymbol{x})}\left[\log
p_{\theta}(\boldsymbol{x}\mid\boldsymbol{z})\right]\right.$ (1)
$\displaystyle\left.-\mathrm{KL}\left(q_{\phi}(\boldsymbol{z}\mid\boldsymbol{x})\|p(\boldsymbol{z})\right)\right]$
$\displaystyle=E_{x\sim p_{\text{data
}}}\mathcal{L}_{\mathrm{R}}+\mathcal{L}_{\mathrm{I}}$

#### VAE Issues

To solve ELBO, the inference model $q_{\phi}(z|x)$ can be trained jointly by
maximizing the ELBO to acquire reasonable compression for task fitting.
However, a weak capacity of the decoder and the variety of data could make the
expressive posterior favor task fitting rather than optimal inference (Zhao,
Song, and Ermon 2019). For example, in variational language generation, the
decoder built on autoregressive models such as LSTM and PixelCNN can generate
language samples by the autoregressive property rather than the posterior-
based latent variables (Wu, Wang, and Zhang 2019). The VAE degenerates to an
autoregressive model where the KL divergence between posterior and prior
reaches zero quickly during training. This results in KL vanishing and poor
generalization in test for the lack of diversity. The approaches of learning
orthogonal transformation of priors with the same distribution by the decoder
(Khemakhem et al. 2020; Mita, Filippone, and Michiardi 2021) may sacrifice
accurate inference in optimal representation and generalization of fitting the
data. Other research attributes the training conflict to the inherent property
of bound optimization. For example, under a solid factorial assumption about
the posterior distribution (Locatello et al. 2019), i.e.,

$p(\mathbf{x},\mathbf{z})=p(\mathbf{x}\mid\mathbf{z})\prod_{i}p\left(z_{i}\right),$
(2)

the ELBO constraining the variational samples favors the data fitting (Burda,
Grosse, and Salakhutdinov 2016) but fails to maximize the probability mass on
log-likelihood. In addition, the vanilla VAE optimizer strengthens the
disjointness between $q_{\phi}(z|x_{i})$, i.e.,
$\mu_{i}\rightarrow\infty,\sigma_{i}\rightarrow 0^{+}$, to separate the log-
likelihood concentrated on each sample, resulting in maximizing the mass of
joint distribution (Zhao, Song, and Ermon 2019).

#### VAE Enhancement

One way to address the above issues is to tighten the log-likelihood lower
bound for correct variational approximation in posterior (Alemi et al. 2018;
Wang and Wang 2019; Domke and Sheldon 2019; Jankowiak and Phan 2022; Ruiz et
al. 2021; Kviman et al. 2022), prior (Tomczak and Welling 2018; Klushyn et al.
2019) and decomposition of ELBO (Zhao, Song, and Ermon 2019; Esmaeili et al.
2019) under some mild assumptions. For example, $\beta$-VAE (Higgins et al.
2017) adds the hyper-parameter $\beta$ to weigh the $\mathcal{L}_{KL}$ term.
Then, ELBO minimizes $\mathcal{L}_{R}$ to the convergence of data fitting
collectively with the regularization $\mathcal{L}_{I}$ by varying $\beta$:

$\displaystyle\mathcal{L}_{ELBO}=$ $\displaystyle E_{x\sim
p_{data}}\left[E_{q_{\phi}(\mathbf{z}\mid\mathbf{x})}\left[\log
p_{\theta}(\mathbf{x}\mid\mathbf{z})\right]\right.$ (3)
$\displaystyle\left.-\beta
D_{KL}\left(q_{\phi}(\mathbf{z}\mid\mathbf{x})\|p(\mathbf{z})\right)\right]$
$\displaystyle=$ $\displaystyle E_{x\sim p_{\text{data
}}}\left[\mathcal{L}_{\mathrm{R}}+\beta\mathcal{L}_{\mathrm{I}}\right]$

$\beta$-VAE introduces some fundamental limitations, which trigger various
follow-up research. InfoVAE introduces a scaling parameter $\lambda$ on the
KL-term and converts the objective to (Zhao, Song, and Ermon 2019):

$\displaystyle\mathcal{L}_{ELBO}=$ $\displaystyle\alpha
I_{q}(\mathbf{x};\mathbf{z})-D_{KL}\left(q_{\phi}(\mathbf{z}\mid\mathbf{x})\|p(\mathbf{z})\right)$
(4) $\displaystyle-
E_{q(z)}\left[D_{KL}\left(q_{\phi}(\mathbf{z}\mid\mathbf{x})\|p_{\theta}(\mathbf{x}\|\mathbf{z})\right)\right]$

where $I_{q}$ is the mutual information with weight $\alpha$ and
$\alpha+\lambda-1=0$. This linear tuning on the KL shows limitation to dynamic
uncertainty.

Further, conditional VAE (CVAE) (Sohn, Lee, and Yan 2015) introduces an
initial guess as a conditional variable into the objective function for
multimodal data. The SA-VAE involves a cyclical annealing schedule to split
the training to multiple cycles starting at $\beta=0$ and progressively
increases $\beta$ until $\beta=1$ to reduce the KL vanishing (Fu et al. 2019).
In ControlVAE, the PID control compares the KL divergence with a set point,
with their difference as feedback to the controller to tune the hyperparameter
$\beta(t)$ (Shao et al. 2020). ControlVAE thus optimizes KL dynamically but is
constrained by the PID controller which follows a separate tuning mechanism
from the VAE itself.

The existing work leaves gaps for building an approximate weight allocation
between reconstruction and inference, tuning external hypberparameters within
the VAE working mechanism and handling these issues in a dynamic manner over
an evolutionary learning process. Our eVAE addresses these gaps by
incorporating the variational genetic learning into balancing inference and
generation and evolutionarily involving their effect into adjusting the VAE
learning behaviors toward better uncertain tradeoff learning.

## The eVAE Model

eVAE also tunes the representation-generation balance in Eq. (1) by jointly
addressing the following issues in Eqs. (3) and (4): tuning hyperparameter in
an outer circle irrelevant to VAE behaviors, non-dynamic optimization, and
constrained settings on data and networks. Figure 1 illustrates the eVAE
framework of variational evolutionary learning to improve the VAE balance.

Figure 1: The framework of eVAE. The VAE results inform the chromosome
sampling. The genes are then updated by variational V-crossover and
V-mutation. The evolved results are checked per fitness for $t+1$ retraining,
giving up, or converging.

### eVAE - Evolving Inner-outer-joint Training

eVAE implements an evolving learning process of optimizing both VAE and its
evolutionary parameters through an inner-outer-joint iterative training
process, as shown in Figure 2. At time $t$, given the input
$x_{t}\in\mathcal{X}$, a VAE inner training process trains a VAE model
$\mathcal{V}$ (we do not constrain the VAE framework, for generality, we use
$\beta$-VAE for case study in this paper) with some initialization $\beta_{0}$
to learn the encoder $q_{\phi}(\mathbf{z}_{t}\mid\mathbf{x}_{t})$, the
posterior $p_{\theta}(\mathbf{x}_{t}\mid\mathbf{z}_{t})$ with prior
$p(\mathbf{z})$. We obtain the VAE inner objective function with initial
weight $\beta_{0}$:

$\begin{array}[]{r}\mathcal{L}_{ELBO_{t}}=E_{x_{t}\sim
p_{\mathcal{X}}}\left[E_{q_{\phi}\left(\mathbf{z}_{t}\mid\mathbf{x}_{t}\right)}\left[\log
p_{\theta}\left(\mathbf{x}_{t}\mid\mathbf{z}_{t}\right)\right]-\right.\\\
\left.\beta_{0}D_{KL}\left(q_{\phi}\left(\mathbf{z}_{t}\mid\mathbf{x}_{t}\right)\|p(\mathbf{z})\right)\right]\end{array}$
(5)

By applying SGVB and reparameterization trick, the generative process of
$\mathcal{V}$ estimates the posterior as

$p_{\theta}(\mathbf{x}\mid\mathbf{z})=\prod_{t}p_{\theta}(\mathbf{x}_{t}\mid\mathbf{z}_{t})$
(6)

where
$\mathbf{x}_{t}\mid\mathbf{z}_{t}\sim\mathcal{N}(\mu(\mathbf{z}_{t};\theta),\Sigma(\mathbf{z}_{t};\theta))$
with $\mathbf{z}_{t}=\mu_{x_{t}}+\mathbf{\epsilon}\Sigma_{x_{t}}$ and
$\mathbf{\epsilon}\sim\mathcal{N}(0,I)$.

Then, we introduce an outer variational evolutionary learner $\mathcal{E}$
(illustrated by a variational genetic algorithm introduced below on
Variational Evolution) to optimize the objective in Eq. (5) in an outer
process. $\mathcal{E}$ with the fitness function $f$ samples a chromosome
$\beta_{t}$ from an evolving distribution $\beta_{t}\sim\mathcal{R}$ and
evolves it by taking variational crossover and mutation to generate and
improve parameters $\beta_{t+1}$.

The evolved parameter is then incorporated into $\mathcal{V}_{t}$ for the next
iteration $t+1$ training. With the relevant ELBO information back-propagated
to tune the VAE network and optimize the relevant parameters, we obtain the
updated reconstruction and inference performance. Consequently, Eq. (5) is
evolved to:

$\begin{array}[]{r}\mathcal{L}_{ELBO_{t+1}}=E_{x_{t+1}\sim
p_{\mathcal{X}}}\left[E_{q_{\phi}(\mathbf{z}_{t+1}|\mathbf{x}_{t+1})}\left[\operatorname{log}p_{\theta}\left(\mathbf{x}_{t+1}\right.\right.\right.\\\
\left.\left.\left.\mid\mathbf{z}_{t+1}\right)\right]-\beta_{t+1}D_{KL}\left(q_{\phi}\left(\mathbf{z}_{t+1}\mid\mathbf{x}_{t+1}\right)\|p(\mathbf{z})\right)\right]\end{array}$
(7)

If the fitness Eq. (17) is satisfied, we update $\beta$ by $\beta_{t+1}$. This
then results in updated $\mathcal{V}_{t+1}$ and its parameters $\phi_{t+1}$,
$\theta_{t+1}$ and $\beta_{t+1}$ for iteration $t+1$. We further repeat the
above VGA $\mathcal{E}$ to obtain another set of parameters and retrain model
$\mathcal{V}_{t+1}$ until it converges.

The above inner-outer iterative training progresses over time and samples in
the generative process to iteratively optimize the balance between
reconstruction (i.e., minimizing $\mathcal{L}_{R}=\|x_{t}-\hat{x}_{t}\|$ with
the reconstructed $\hat{x}_{t}$) and inference (i.e., seeking appropriate
$\mathcal{L}_{I}$) and the VAE learning behaviors. The VAE $\mathcal{V}$ is
then optimized until converges.

Accordingly, the eVAE generative process optimizes the following objective
function:

$\begin{array}[]{r}\mathcal{L}_{eVAE}=\min_{\boldsymbol{\theta}_{t},\boldsymbol{\phi}_{t}}\sum_{t=1}^{N}\mathcal{L}_{ELBO_{t+1}}\left(\boldsymbol{\theta}_{t},\boldsymbol{\phi}_{t}\right.;\\\
\left.f\left(\mathcal{E}\left(\phi_{t},\theta_{t},\\{\beta\\},\mathcal{L}_{ELBO_{t}}\right)\right),\mathbf{x}_{t}\right)\end{array}$
(8)

to approach $\beta^{*}$ for the optimal balance between representation and
construction:

$\beta^{*}\sim
f\left(\mathcal{E}\left(\\{\beta\\}\mid\phi_{t},\theta_{t},\mathcal{L}_{ELBO_{t}}\right)\right)$
(9)

The overall ELBO of eVAE can then be represented by:

$\begin{array}[]{r}\mathcal{L}_{eVAE}=E_{\mathbf{z}\sim
q_{\phi}\left(\mathbf{z}_{t}\mid\mathbf{x}_{t}\right)}\log
p_{\theta}(\mathbf{x}_{t}\mid\mathbf{z}_{t})+\\\ f_{\beta_{t}\sim
R}(\beta_{t},\mathcal{E})D_{KL}(q_{\phi}\left(\mathbf{z}_{t}\mid\mathbf{x}_{t}\right)\|p(\mathbf{z}))\\\
=E_{x_{t+1}\sim
p_{\mathcal{X}}}[\mathcal{L}_{R}+f(\beta)\mathcal{L}_{I}+\mathcal{E}(\Delta\mathcal{L}_{ELBO})]\end{array}$
(10)

### Variational Evolution in eVAE

Here, we introduce the variational evolutionary learner $\mathcal{E}$
optimizing the parameters. We instantiate $\mathcal{E}$ by a variational
genetic algorithm (VGA) with its variational crossover and mutation
operations. This avoids typical issues including premature convergence and
random search during discordant exploitation and exploration.

#### Variational Genetic Algorithm

eVAE integrates the VGA-based outer parameter optimization combined with the
VAE-internal gradient descent-based optimization. Accordingly, VGA consists of
a few variational steps and operations, initialization, variational crossover
(V-crossover), variational mutation (V-mutation), and variational evaluation
(V-evaluation). As shown in the bottom part of Figure 2, below, we introduce
them respectively.

Chromosome selection: To enable a larger and more smooth GA search space
compatible with VAEs, chromosomes are embedded by a continuous variable
$\beta$, sampled from an evolving distribution $\mathcal{R}$ (e.g., in the
crossover). Assume $L$ individual chromosomes forming a candidate group
$\\{\beta_{l}\\}=\\{\beta_{1},\dots,\beta_{L}\\}$, a chromosome is chosen from
this candidate group to train the VAE, which is associated with a fitness
value $f$ after the VGA operations and VAE retraining. We thus have the
chromosome-fitness pairs embedded for the VGA evolution and offspring
selection over generations, such as:

$\\{\beta,f\\}=((\beta_{1},f_{1}),\dots,(\beta_{L},f_{L}))$ (11)

for optimizing the inner-outer joint training.

V-crossover: Paired chromosomes at time $t-1$ and $t$ are crossovered to
generate new genes for improving the genetic variety of offsprings. Following
the simulated binary crossover (SBX) (Deb and Beyer 2001) method, we implement
the following crossover operations $\mathcal{C}$ to induce a large search
space evolving over chromosome $\beta_{t-1}$ at time $t-1$ and chromosome
$\beta_{t}$ selected from the chromosome candidate group $\\{\beta_{l}\\}$ at
time $t$ to obtain the offspring candidate chromosome $\beta_{t+1}$ for the
next time $t+1$ in $\mathcal{C}$ strategy:

$\mathcal{C}:\left\\{\begin{array}[]{l}\beta_{t+1}=\frac{1}{2}[(1+r_{c})\beta_{l,t}+(1-r_{c})\beta_{t-1}],or\\\
\beta_{t+1}=\frac{1}{2}[(1-r_{c})\beta_{l,t}+(1+r_{c})\beta_{t-1}]\end{array}\right.$
(12)

where $r_{c}$ is the crossover rate, which is drawn from a probability density
function $P_{c}(r_{c})$:

$P_{c}\left(r_{c}\right)=\begin{cases}0.5(\eta+1)r_{c}^{n},&\text{ if
}r_{c}\leq 1\\\ 0.5(\eta+1)\frac{1}{r_{c}^{\eta+2}},&\text{ otherwise
}\end{cases}$ (13)

by concentrating their corresponding parents $\beta_{t-1}$ and $\beta_{l,t}$.
By sampling $r_{c}$ from the following function:

$r_{c}=\begin{cases}(2u)^{\frac{1}{\eta+1}},&\text{ if }u\leq 0.5\\\
\left(\frac{1}{2(1-u)}\right)^{\frac{1}{\eta+1}},&\text{ otherwise
}\end{cases}$ (14)

where $u$ is a random variable, and $\eta$ is a hyper-parameter for scaling,
the offsprings $\beta_{t+1}$ can then be created by selecting the better
$\mathcal{C}$ strategy (i.e., better satisfying Eq. (17)).

Figure 2: eVAE - inner-outer joint evolutionary training process. The upper
part shows the VAE training at time $t$, the objectives are then incorporated
into the lower part - the outer training by variational genetic algorithm,
whose fitness-based optimized results are fed to the VAE for further training.

V-mutation: The crossover-generated offsprings can be further mutated to
improve their genomic diversity and evolutionary capacity. Aligning with the
V-crossover, a variational mutation strategy $\mathcal{M}$ is taken to
diversify the offspring to update $\beta_{t+1}$ inherited from crossover or
$\beta_{t}$ from the current moment. With a relatively small probability
$p_{m}$, a chromosome $\beta_{t+1}$ selected from group is mutated as:

$\mathcal{M}:\beta_{t+1}=\beta_{t+1}+r_{m}$ (15)

where $r_{m}$ is a random variable sampled from the Cauchy distribution
$p_{m}$:

$P_{m}=\frac{1}{\pi}(\frac{1}{1+\beta_{t+1}^{2}})$ (16)

#### V-evaluation & VGA fitness function

The chromosomes updated by the V-crossover and V-mutation operations are
evaluated in alignment with the VAE objectives to determine whether they are
transferable to the next generation. Within the VAE framework, we take the
following heuristic fitness function $f$ to guide the evolution of chromesomes
and their fitness to VAE objectives. The fitness function integrates the
direction of VAE-oriented stochastic gradient descent and the distance to the
optimal KL divergence to achieve multi-objectives.

$f_{t+1}=\Delta\mathcal{L}_{ELBO_{t+1}}+||KL_{t+1}(\beta_{t+1})-c||$ (17)

where $\Delta\mathcal{L}_{ELBO_{t+1}}$ is:

$\Delta\mathcal{L}_{ELBO_{t+1}}=\mathcal{L}_{ELBO_{t+1}}(\beta_{t+1})-\mathcal{L}_{ELBO_{t}}(\beta_{t})$
(18)

denoting the VAE’s ELBO difference after applying the evolved $\beta$. $c$ is
the task-specific information bottleneck, which ensures the bound optimization
of eVAE over the evolutionary optimization.

This V-evaluation and the fitness $f$ guide eVAE to converge and balance the
reconstruction and inference with enhanced evolutionary parameterization. The
larger the fitness value, the stronger the chromosome in its candidate group,
which forms a strong pair $\\{\beta^{*},f^{*}\\}$.

The whole training processes of the eVAE model is illustrated in Algorithm 1
in the Supplementary Material A.

## Theoretical Analysis

In this paper, eVAE takes $\beta$-VAE as a case study of evolutionary VAE.
Hence, we analyze the effects of eVAE in adjusting $\beta$-VAE parameters,
tradeoff between reconstruction and regularization, and training performance
below.

$\beta$-VAE directly tunes hyperparameter $\beta$ setting a task-relevant
ratio between the inference loss $\mathcal{L}_{\mathrm{I}}$ and the generation
loss $\mathcal{L}_{\mathrm{R}}$. In essence, incrementally regularizing
$\mathcal{L}_{\mathrm{I}}$ by $\beta$ is equivalent to maximizing the
likelihood on an experiment-specific threshold, i.e., a constant $C$, (Higgins
et al. 2017):

$\displaystyle\max_{\phi,\theta}E_{x\sim p_{\text{data
}}}\mathcal{L}_{R}\quad\text{ s.t. }\beta\mathcal{L}_{I}<C$ (19)

This learning process can be interpreted in information bottleneck (IB) theory
(Alemi et al. 2018). The $\beta\mathcal{L}_{I}$ constraint operates as the
bottleneck, compressing the capacity of latent variable $Z$ organized from
input $X$ to expressively represent the target $\hat{X}$ (Alemi et al. 2017).
Accordingly, IB maximizes $I(Z,\hat{X})$ to acquire a concise representation:

$\max I(Z,\hat{X})\quad\text{ s.t. }I(X,Z)\leq I_{c}$ (20)

Further, the variational information bottleneck (VIB) introduces an
experiment-specific lower bound for optimization:

$\mathcal{L}_{\beta-VAE}(\theta,\phi)=-D-\beta_{C}R\leq-\beta_{C}I-D$ (21)

where $R$ refers to the compression rate, and $D$ refers to the distortion
measuring the representation relevance to a task, and $\beta_{C}$ is an
experiment-specific constant. VAEs can thus be viewed as a process of
achieving the rate-distortion tradeoff.

ControlVAE is a variant of $\beta$-VAE, tuning the KL-specific $\beta_{KL}$
with a PID control to a desired KL point:

$\mathcal{L}_{ControlVAE}(\theta,\phi)=-D-\beta_{KL}R\leq-\beta_{KL}I-D$ (22)

eVAE extends $\beta$-VAE tuning the information bottleneck $\beta I(X,Z)$ by a
variational evolutionary learner $\mathcal{E}$. After iteration $t$ of outer
training, $\beta_{t}$ evolves to fit the task and training phase, guided by
the fitness function $f$. eVAE generates a lower bound to optimize iteration
$t+1$ of the inner training phase:

$\displaystyle\mathcal{L}_{eVAE}$
$\displaystyle\leq-\mathcal{E}(\beta_{t})I_{t}-\mathcal{E}(\beta_{t})D_{t}$
(23)

where an evolving iteration-specific lower bound can strike a compromise
between task fitting in $-\mathcal{E}(\beta_{t})D_{t}$ and inference quality
in $\mathcal{E}(\beta_{t})I_{t}$. The derivations of Eq. (21) and Eq. (23) can
be found in the Supplementary Materials B and C.

We further compare the VIB effects of eVAE against VAE, $\beta$-VAE and
ControlVAE by the rate-distortion (R-D) curve in Figure 3. VAE (yellow) tends
to sacrifice empirical error minimization for representation learning in the
early stage, where the rate is optimized to decrease in a quarter of the
iterations. As for $\beta$-VAE (green), due to the experiment-specific lower
bound ($-\beta_{C}I-D$), the distortion cannot be minimized. ControlVAE (pink)
concentrates on optimizing the distortion initially to fit data and optimizing
the rate eventually to acquire smooth representation, resorting to the KL-
specific lower bound ($-\beta_{KL}I-D$). However, directly modifying $\beta$
by the given PID controller leads to fluctuations between inference capacity
and reconstruction quality during the iterations. Rather than monitoring the
optimization process, eVAE generates an iteration-specific lower bound
($-\mathcal{E}(\beta_{t})I_{t}-\mathcal{E}(\beta_{t})D_{t}$) to achieve a
balance between minimizing distortion and controlling rate.

Figure 3: The information plane with the $R-D$ curves of VAE, $\beta$-VAE,
ControlVAE and eVAE on dSprites.

## Experiments

We evaluate eVAE in three typical tasks: disentangled representation, image
generation, and language modeling.

### Dataset and Baselines

Three different learning tasks and their datasets are listed below to evaluate
eVAE against the VAE baselines.

Disentangled representation learns independent latent variables to generate an
image. We validate eVAE on dSprites, a 2D-shape dataset with 737,280 binary
images generated by six ground-truth factors: shape (square, ellipse, heart),
scale, orientation, position $x$, and position $y$. To examine the
disentanglement performance and image generation equally and comprehensively,
$\beta$-VAE and ControlVAE are compared.

Image generation reconstructs an imagery sample based on given datapoints. The
CelebA dataset (cropped version) (Liu et al. 2015) is used, which is a real-
world celebrity face dataset with 202,599 RGB images. The baselines are
$\beta$-VAE and ControlVAE.

Language modeling conducts the word-level text generation. The Penn Treebank
dataset (PTB) with English corpus (Marcinkiewicz 1994) is used. To reveal the
performance of restraining KL vanishing, we compare eVAE with three heuristic
methods: cost annealing (noted as Cost-10k) (Bowman et al. 2016), cyclical
annealing (Cyc-8) (Fu et al. 2019), and PID control (PID-3) (Shao et al.
2020).

For a fair comparison, we adopt the same embedding network as $\beta$-VAE and
tune eVAE to achieve the same KL set point as ControlVAE. The VAE
architectures and hyperparameter settings are given in Supplementary Material
D.

### Performance Evaluation

Performance of disentangled representation learning Here, we validate the
performance of reconstructing a sharp image with disentangled features. Figure
2(a) shows that eVAE achieves a lower reconstruction loss of 9.2, compared
with ControlVAE under the same expected KL divergence (KL = 19). Moreover,
Figure 2(a) and (b) show that eVAE presents a more stable training curve than
other VAEs because of its dynamic weighting guided by the VGA fitness
function, rather than a direct modification in $\beta$-VAE and ControlVAE. The
variation of each disentangled factor is illustrated in Figure 2(c). Figure 6
further demonstrates the reliable disentanglement of all factors and sharp
reconstruction quality of eVAE under a set point KL = 19. In contrast,
ControlVAE captures the generative factors of scale, $x$ and $y$ while
$\beta$-VAE entangles other four factors except scale.

Performance of image generation We evaluate eVAE on CelebA for image
generation. For a fair and concise comparison, the KL set point is set to 200
achieving the best reconstruction quality in (Shao et al. 2020). eVAE is tuned
to achieve the desired KL point similar to ControlVAE. Figure 4(a) shows that
eVAE has the lowest reconstruction error compared with vanilla VAE and
ControlVAE. Guided by the evolving $\beta$, the KL divergence in eVAE can
achieve a desired point dynamically, as illustrated in Figure 4(b).

Figure 4: Performance comparison of different VAEs for image generation on
CelebA.

Performance of language modeling To show the validity of eVAE in preventing
the variational language model from degenerating to a standard language model,
we draw the learning curves of KL, reconstruction loss, and weight of KL over
iterations. The evolution of $\beta$ is shown in Figure 7(a), demonstrating
that eVAE can learn to tune $\beta$ over iterations automatically rather than
by a heuristic setting. Figure 7(b) shows that the Cost annealing (Cost-10k)
model still suffers from KL vanishing, with KL approaching 0 at the end of
iterations. By contrast, the Cyclical annealing (Cyc-8), PID, and GA can
converge to a nonzero divergence, while eVAE achieves the lowest
reconstruction loss to 70 in word generation, illustrated in Figure 7(c).
Further, compared to the PID control, VGA can find a path toward a more stable
stage, without undergoing reconstruction fluctuations during training even if
the non-zero learned $\beta$ is fixed.

## Conclusion

We propose evolutionary VAE (eVAE) to dynamically learn the tradeoff between
reconstruction effectiveness and inference robustness in VAEs. eVAE involves
variational evolving operators to solve the premature convergence and random
search problem. SGD optimization and genetic algorithm synergistically follow
an inner-outer-joint training mechanism in eVAE. Following the variational
information bottleneck theory, eVAE derives an iteration-specific lower bound
balancing the capacity of compression and decompression over iterations.
Multiple task experiments show that eVAE outperforms SOTA VAEs in traversing
samples with more disentangled features, generating sharp images with lower
reconstruction loss, and resolving the KL vanishing automatically in
variational language modeling. We will extend eVAE to other VAE models and
tasks.

Figure 5: Learning curves on dSprites222dSprites: Disentanglement testing
Sprites dataset. https://github.com/deepmind/dsprites-dataset/. (a,b) indicate
that eVAE has the lowest reconstruction loss compared with VAE ($\beta=1$),
$\beta$-VAE ($\beta=4$), and ControlVAE (KL = 19) under a fixed KL point
KL=19. (c) is the element-wise KL divergence as a function of iterations in
eVAE. We can observe that eVAE retains a stable and reasonable KL divergence
of each generator dimension (factor): position-$y$ (z2), scale (z3), shape
(z4), position-$x$ (z6), orientation (z7). More comparisons in terms of
generated KL divergence can be found in Supplementary Material E.More
comparisons in terms of generated KL divergence can be found in Supplementary
Material E. Figure 6: Latent traversal on dSprites. To distinguish the latent
factors visually, we take the ellipse for illustration. Each row represents a
latent factor in the traversal, while keeping others fixed. The first column
refers to the seed image for initialization, and we then manipulate the latent
dimension $z$ across the range [-3, 3].  Figure 7: Learning curves on PTB for
language modeling. The suffix of PID and VGA refer to a KL set point i.e., KL
= 3; Cyc-8 indicates 8 iteration cycles where the weight increases linearly
from 0 to 1; Cost-10k refers to 10k iterations during which the weight
increases from 0 to 1 by a sigmoid function iteratively.

## References

  * Alemi et al. (2018) Alemi, A.; Poole, B.; Fischer, I.; Dillon, J.; Saurous, R. A.; and Murphy, K. 2018\.  Fixing a broken ELBO.  In _ICML_.
  * Alemi et al. (2017) Alemi, A. A.; Fischer, I.; Dillon, J. V.; and Murphy, K. 2017.  Deep variational information bottleneck.  In _ICLR_.
  * Berliner et al. (2022) Berliner, A.; Rotman, G.; Adi, Y.; Reichart, R.; and Hazan, T. 2022.  Learning discrete structured variational auto-encoder using natural evolution strategies.  In _ICLR_.
  * Bowman et al. (2016) Bowman, S. R.; Vilnis, L.; Vinyals, O.; Dai, A. M.; Jozefowicz, R.; and Bengio, S. 2016.  Generating sentences from a continuous space.  In _CONLL_.
  * Bozkurt et al. (2021) Bozkurt, A.; Esmaeili, B.; Tristan, J.-B.; Brooks, D. H.; Dy, J. G.; and van de Meent, J.-W. 2021.  Rate-regularization and generalization in VAEs.  _AISTATS_.
  * Burda, Grosse, and Salakhutdinov (2016) Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2016.  Importance weighted autoencoders.  In _ICLR_.
  * Burgess et al. (2018) Burgess, C. P.; Higgins, I.; Pal, A.; Matthey, L.; Watters, N.; Desjardins, G.; and Lerchner, A. 2018.  Understanding disentangling in $\beta$-VAE.  In _NeurIPS_.
  * Deb and Beyer (2001) Deb, K.; and Beyer, H.-G. 2001.  Self-adaptive genetic algorithms with simulated binary crossover.  _Evolutionary Computation_ , 9(2): 197–221.
  * Domke and Sheldon (2019) Domke, J.; and Sheldon, D. R. 2019.  Divide and couple: Using monte carlo variational objectives for posterior approximation.  In _NeurIPS_.
  * Esmaeili et al. (2019) Esmaeili, B.; Wu, H.; Jain, S.; Bozkurt, A.; Siddharth, N.; Paige, B.; Brooks, D. H.; Dy, J.; and Meent, J.-W. 2019.  Structured disentangled representations.  In _AISTATS_.
  * Fortuin et al. (2019) Fortuin, V.; Hüser, M.; Locatello, F.; Strathmann, H.; and Rätsch, G. 2019\.  Som-vae: Interpretable discrete representation learning on time series.  In _ICLR_.
  * Fu et al. (2019) Fu, H.; Li, C.; Liu, X.; Gao, J.; Celikyilmaz, A.; and Carin, L. 2019.  Cyclical annealing schedule: A simple approach to mitigating KL vanishing.  In _NAACL_.
  * Ghosh et al. (2019) Ghosh, P.; Sajjadi, M. S.; Vergari, A.; Black, M.; and Schölkopf, B. 2019.  From variational to deterministic autoencoders.  In _ICLR_.
  * Havtorn et al. (2021) Havtorn, J. D. D.; Frellsen, J.; Hauberg, S.; and Maaløe, L. 2021.  Hierarchical VAEs know what they don’t know.  In _ICML_.
  * Higgins et al. (2017) Higgins, I.; Matthey, L.; Pal, A.; Burgess, C.; Glorot, X.; Botvinick, M.; Mohamed, S.; and Lerchner, A. 2017.  $\beta$-VAE: Learning basic visual concepts with a constrained variational framework.  In _ICLR_.
  * Jankowiak and Phan (2022) Jankowiak, M.; and Phan, D. 2022.  Surrogate likelihoods for variational annealed importance sampling.  In _ICML_.
  * Kamata, Mukuta, and Harada (2022) Kamata, H.; Mukuta, Y.; and Harada, T. 2022.  Fully spiking variational autoencoder.  _AAAI_.
  * Khemakhem et al. (2020) Khemakhem, I.; Kingma, D.; Monti, R.; and Hyvarinen, A. 2020.  Variational autoencoders and nonlinear ICA: A unifying framework.  In _AISTATS_.
  * Kingma and Welling (2013) Kingma, D. P.; and Welling, M. 2013.  Auto-encoding variational bayes.  _arXiv_.
  * Klushyn et al. (2019) Klushyn, A.; Chen, N.; Kurle, R.; Cseke, B.; and van der Smagt, P. 2019.  Learning hierarchical priors in VAEs.  In _NeurIPS_.
  * Kviman et al. (2022) Kviman, O.; Melin, H.; Koptagel, H.; Elvira, V.; and Lagergren, J. 2022.  Multiple importance sampling ELBO and deep ensembles of variational approximations.  In _AISTATS_.
  * Liu et al. (2015) Liu, Z.; Luo, P.; Wang, X.; and Tang, X. 2015.  Deep learning face attributes in the wild.  In _ICCV_.
  * Locatello et al. (2019) Locatello, F.; Abbati, G.; Rainforth, T.; Bauer, S.; Schölkopf, B.; and Bachem, O. 2019.  On the fairness of disentangled representations.  In _NeurIPS_.
  * Marcinkiewicz (1994) Marcinkiewicz, M. A. 1994.  Building a large annotated corpus of English: The penn treebank.  _Using Large Corpora_.
  * Mita, Filippone, and Michiardi (2021) Mita, G.; Filippone, M.; and Michiardi, P. 2021.  An identifiable double VAE for disentangled representations.  In _ICML_.
  * Nalisnick et al. (2019) Nalisnick, E.; Matsukawa, A.; Teh, Y. W.; Gorur, D.; and Lakshminarayanan, B. 2019\.  Do deep generative models know what they don’t know?  In _ICLR_.
  * Ran et al. (2022) Ran, X.; Xu, M.; Mei, L.; Xu, Q.; and Liu, Q. 2022.  Detecting out-of-distribution samples via variational autoencoder with reliable uncertainty estimation.  _Neural Networks_.
  * Razavi, Van den Oord, and Vinyals (2019) Razavi, A.; Van den Oord, A.; and Vinyals, O. 2019.  Generating diverse high-fidelity images with VQ-VAE-2.  In _NeurIPS_.
  * Ruiz et al. (2021) Ruiz, F. J.; Titsias, M. K.; Cemgil, T.; and Doucet, A. 2021.  Unbiased gradient estimation for variational autoencoders using coupled markov chains.  In _UAI_.
  * Shao et al. (2020) Shao, H.; Yao, S.; Sun, D.; Zhang, A.; Liu, S.; Liu, D.; Wang, J.; and Abdelzaher, T. 2020.  Controlvae: Controllable variational autoencoder.  In _ICML_.
  * Shi et al. (2019) Shi, Y.; Paige, B.; Torr, P.; et al. 2019.  Variational mixture-of-experts autoencoders for multi-modal deep generative models.  In _NeurIPS_.
  * Sohn, Lee, and Yan (2015) Sohn, K.; Lee, H.; and Yan, X. 2015.  Learning structured output representation using deep conditional generative models.  In _NeurIPS_.
  * Sutter, Daunhawer, and Vogt (2021) Sutter, T. M.; Daunhawer, I.; and Vogt, J. E. 2021.  Generalized multimodal ELBO.  _arXiv_.
  * Tolstikhin et al. (2017) Tolstikhin, I.; Bousquet, O.; Gelly, S.; and Schoelkopf, B. 2017.  Wasserstein autoencoders.  _arXiv_.
  * Tomczak and Welling (2018) Tomczak, J.; and Welling, M. 2018.  VAE with a vampprior.  In _AISTATS_.
  * Wang and Wang (2019) Wang, P. Z.; and Wang, W. Y. 2019.  Neural gaussian copula for variational autoencoder.  In _EMNLP_.
  * Wu, Wang, and Zhang (2019) Wu, D.; Wang, L.; and Zhang, P. 2019.  Solving statistical mechanics using variational autoregressive networks.  _Physical Review Letters_ , 122(8): 080602.
  * Xiao, Yan, and Amit (2020) Xiao, Z.; Yan, Q.; and Amit, Y. 2020.  Likelihood regret: An out-of-distribution detection score for variational auto-encoder.  _NeurIPS_.
  * Zhang et al. (2019) Zhang, Y.; Wang, Y.; Zhang, L.; Zhang, Z.; and Gai, K. 2019.  Improve diverse text generation by self labeling conditional variational auto encoder.  In _ICASSP_.
  * Zhao, Song, and Ermon (2019) Zhao, S.; Song, J.; and Ermon, S. 2019.  InfoVAE: Balancing learning and inference in variational autoencoders.  In _AAAI_.

## References

  * Alemi et al. (2018) Alemi, A.; Poole, B.; Fischer, I.; Dillon, J.; Saurous, R. A.; and Murphy, K. 2018\.  Fixing a broken ELBO.  In _ICML_.
  * Alemi et al. (2017) Alemi, A. A.; Fischer, I.; Dillon, J. V.; and Murphy, K. 2017.  Deep variational information bottleneck.  In _ICLR_.
  * Berliner et al. (2022) Berliner, A.; Rotman, G.; Adi, Y.; Reichart, R.; and Hazan, T. 2022.  Learning discrete structured variational auto-encoder using natural evolution strategies.  In _ICLR_.
  * Bowman et al. (2016) Bowman, S. R.; Vilnis, L.; Vinyals, O.; Dai, A. M.; Jozefowicz, R.; and Bengio, S. 2016.  Generating sentences from a continuous space.  In _CONLL_.
  * Bozkurt et al. (2021) Bozkurt, A.; Esmaeili, B.; Tristan, J.-B.; Brooks, D. H.; Dy, J. G.; and van de Meent, J.-W. 2021.  Rate-regularization and generalization in VAEs.  _AISTATS_.
  * Burda, Grosse, and Salakhutdinov (2016) Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2016.  Importance weighted autoencoders.  In _ICLR_.
  * Burgess et al. (2018) Burgess, C. P.; Higgins, I.; Pal, A.; Matthey, L.; Watters, N.; Desjardins, G.; and Lerchner, A. 2018.  Understanding disentangling in $\beta$-VAE.  In _NeurIPS_.
  * Deb and Beyer (2001) Deb, K.; and Beyer, H.-G. 2001.  Self-adaptive genetic algorithms with simulated binary crossover.  _Evolutionary Computation_ , 9(2): 197–221.
  * Domke and Sheldon (2019) Domke, J.; and Sheldon, D. R. 2019.  Divide and couple: Using monte carlo variational objectives for posterior approximation.  In _NeurIPS_.
  * Esmaeili et al. (2019) Esmaeili, B.; Wu, H.; Jain, S.; Bozkurt, A.; Siddharth, N.; Paige, B.; Brooks, D. H.; Dy, J.; and Meent, J.-W. 2019.  Structured disentangled representations.  In _AISTATS_.
  * Fortuin et al. (2019) Fortuin, V.; Hüser, M.; Locatello, F.; Strathmann, H.; and Rätsch, G. 2019\.  Som-vae: Interpretable discrete representation learning on time series.  In _ICLR_.
  * Fu et al. (2019) Fu, H.; Li, C.; Liu, X.; Gao, J.; Celikyilmaz, A.; and Carin, L. 2019.  Cyclical annealing schedule: A simple approach to mitigating KL vanishing.  In _NAACL_.
  * Ghosh et al. (2019) Ghosh, P.; Sajjadi, M. S.; Vergari, A.; Black, M.; and Schölkopf, B. 2019.  From variational to deterministic autoencoders.  In _ICLR_.
  * Havtorn et al. (2021) Havtorn, J. D. D.; Frellsen, J.; Hauberg, S.; and Maaløe, L. 2021.  Hierarchical VAEs know what they don’t know.  In _ICML_.
  * Higgins et al. (2017) Higgins, I.; Matthey, L.; Pal, A.; Burgess, C.; Glorot, X.; Botvinick, M.; Mohamed, S.; and Lerchner, A. 2017.  $\beta$-VAE: Learning basic visual concepts with a constrained variational framework.  In _ICLR_.
  * Jankowiak and Phan (2022) Jankowiak, M.; and Phan, D. 2022.  Surrogate likelihoods for variational annealed importance sampling.  In _ICML_.
  * Kamata, Mukuta, and Harada (2022) Kamata, H.; Mukuta, Y.; and Harada, T. 2022.  Fully spiking variational autoencoder.  _AAAI_.
  * Khemakhem et al. (2020) Khemakhem, I.; Kingma, D.; Monti, R.; and Hyvarinen, A. 2020.  Variational autoencoders and nonlinear ICA: A unifying framework.  In _AISTATS_.
  * Kingma and Welling (2013) Kingma, D. P.; and Welling, M. 2013.  Auto-encoding variational bayes.  _arXiv_.
  * Klushyn et al. (2019) Klushyn, A.; Chen, N.; Kurle, R.; Cseke, B.; and van der Smagt, P. 2019.  Learning hierarchical priors in VAEs.  In _NeurIPS_.
  * Kviman et al. (2022) Kviman, O.; Melin, H.; Koptagel, H.; Elvira, V.; and Lagergren, J. 2022.  Multiple importance sampling ELBO and deep ensembles of variational approximations.  In _AISTATS_.
  * Liu et al. (2015) Liu, Z.; Luo, P.; Wang, X.; and Tang, X. 2015.  Deep learning face attributes in the wild.  In _ICCV_.
  * Locatello et al. (2019) Locatello, F.; Abbati, G.; Rainforth, T.; Bauer, S.; Schölkopf, B.; and Bachem, O. 2019.  On the fairness of disentangled representations.  In _NeurIPS_.
  * Marcinkiewicz (1994) Marcinkiewicz, M. A. 1994.  Building a large annotated corpus of English: The penn treebank.  _Using Large Corpora_.
  * Mita, Filippone, and Michiardi (2021) Mita, G.; Filippone, M.; and Michiardi, P. 2021.  An identifiable double VAE for disentangled representations.  In _ICML_.
  * Nalisnick et al. (2019) Nalisnick, E.; Matsukawa, A.; Teh, Y. W.; Gorur, D.; and Lakshminarayanan, B. 2019\.  Do deep generative models know what they don’t know?  In _ICLR_.
  * Ran et al. (2022) Ran, X.; Xu, M.; Mei, L.; Xu, Q.; and Liu, Q. 2022.  Detecting out-of-distribution samples via variational autoencoder with reliable uncertainty estimation.  _Neural Networks_.
  * Razavi, Van den Oord, and Vinyals (2019) Razavi, A.; Van den Oord, A.; and Vinyals, O. 2019.  Generating diverse high-fidelity images with VQ-VAE-2.  In _NeurIPS_.
  * Ruiz et al. (2021) Ruiz, F. J.; Titsias, M. K.; Cemgil, T.; and Doucet, A. 2021.  Unbiased gradient estimation for variational autoencoders using coupled markov chains.  In _UAI_.
  * Shao et al. (2020) Shao, H.; Yao, S.; Sun, D.; Zhang, A.; Liu, S.; Liu, D.; Wang, J.; and Abdelzaher, T. 2020.  Controlvae: Controllable variational autoencoder.  In _ICML_.
  * Shi et al. (2019) Shi, Y.; Paige, B.; Torr, P.; et al. 2019.  Variational mixture-of-experts autoencoders for multi-modal deep generative models.  In _NeurIPS_.
  * Sohn, Lee, and Yan (2015) Sohn, K.; Lee, H.; and Yan, X. 2015.  Learning structured output representation using deep conditional generative models.  In _NeurIPS_.
  * Sutter, Daunhawer, and Vogt (2021) Sutter, T. M.; Daunhawer, I.; and Vogt, J. E. 2021.  Generalized multimodal ELBO.  _arXiv_.
  * Tolstikhin et al. (2017) Tolstikhin, I.; Bousquet, O.; Gelly, S.; and Schoelkopf, B. 2017.  Wasserstein autoencoders.  _arXiv_.
  * Tomczak and Welling (2018) Tomczak, J.; and Welling, M. 2018.  VAE with a vampprior.  In _AISTATS_.
  * Wang and Wang (2019) Wang, P. Z.; and Wang, W. Y. 2019.  Neural gaussian copula for variational autoencoder.  In _EMNLP_.
  * Wu, Wang, and Zhang (2019) Wu, D.; Wang, L.; and Zhang, P. 2019.  Solving statistical mechanics using variational autoregressive networks.  _Physical Review Letters_ , 122(8): 080602.
  * Xiao, Yan, and Amit (2020) Xiao, Z.; Yan, Q.; and Amit, Y. 2020.  Likelihood regret: An out-of-distribution detection score for variational auto-encoder.  _NeurIPS_.
  * Zhang et al. (2019) Zhang, Y.; Wang, Y.; Zhang, L.; Zhang, Z.; and Gai, K. 2019.  Improve diverse text generation by self labeling conditional variational auto encoder.  In _ICASSP_.
  * Zhao, Song, and Ermon (2019) Zhao, S.; Song, J.; and Ermon, S. 2019.  InfoVAE: Balancing learning and inference in variational autoencoders.  In _AAAI_.

## Supplementary Materials -
eVAE: Evolutionary Variational Autoencoder

Here, we present some supplemental materials to the main content, including
the eVAE algorithm, the experimental settings, the latent traversal on square
and heart, and derivation of the variational lower and upper bounds.

### A The eVAE algorithm

Below, Algorithm 1 illustrates the modeling process of eVAE, which enables an
inner-outer-joint training of VAE and evolutionary learning iteratively.

Algorithm 1 Evolving Inner-outer-joint Training

1: Input: crossover rate $Pr_{c}$, mutation rate $Pr_{m}$, batch data
$\mathbf{x}$.

2: Output: $\\{\beta^{*},f^{*}\\}$

3: Initialization: parameters of decoder $\theta$, parameters of encoder
$\phi$, $\\{\beta_{l}\\}=\\{\beta_{1},\dots,\beta_{L}\\}$ //initialize group

4: while $t<T$ do

5: $Pr_{t}$ $\leftarrow$ sample from $N\left(0,1\right)$ //probability to
evolve

6: if $Pr_{t}<Pr_{m}$ then

7: $\theta_{t}$, $\phi_{t}$ //save current variational parameters

8: generate $\beta_{t+1}$ by V-crossover operation
$\mathcal{C}(\beta_{t-1},\beta_{t})$ by Eqs. (12), (13) and (14)

9: evaluate $\beta_{t+1}$ by
$f(\mathcal{E}(\phi_{t},\theta_{t},\\{\beta\\},\mathcal{L}_{ELBO_{t}}))$ by
Eqs. (17) and (18)

10: else if $Pr_{t}<Pr_{c}$ then

11: $\theta_{t}$, $\phi_{t}$ //save current variational parameters

12: generate $\beta_{t+1}$ by V-mutation operation $\mathcal{M}(\beta_{t})$ by
Eqs. (15) and (16)

13: evaluate $\beta_{t+1}$ by fitness function
$f(\mathcal{E}(\phi_{t},\theta_{t},\\{\beta\\},\mathcal{L}_{ELBO_{t}}))$ by
Eqs. (17) and (18)

14: else

15: $\\{\beta^{*},f^{*}\\}\leftarrow\beta_{l}$ //select the strongest pair

16:
$\theta_{t+1},\phi_{t+1}\leftarrow\mathcal{L}_{eVAE}(\theta_{t},\phi_{t},\beta^{*}|\mathbf{x}_{t})$
// update variational parameters

17: end if

18: Return: $\\{\beta^{*},f^{*}\\}$

19: end while

### B Derivation of $\beta$-VAE lower bound

According to the VIB theory, elbo can be bounded by distortion
$D=-E_{q_{\phi}(x,z)}[\log p_{\theta}(x\mid z)]$ and rate
$R=E_{q_{\phi}(x,z)}[D_{KL}(q_{\phi}(z|x)||p(z))]$, where $I(X,Z)$ has the
upper bound $I\leq R$:

$\displaystyle R$ $\displaystyle=\int q(x)\int q_{\theta}(z\mid
x)\log\frac{q_{\theta}(z\mid x)}{p(z)}$ (24) $\displaystyle=I(X,Z)+TC(Z)$
$\displaystyle\geq I(X,Z)$

and the lower bound $I\geq H-D$:

$\displaystyle I$ $\displaystyle=\int dxdzp(x,z)\log\frac{p(x\mid z)}{p(x)}$
(25) $\displaystyle\geq\int dxdzp(x,z)\log\frac{q(x\mid z)}{p(x)}$
$\displaystyle=\int dxdzp(x,z)\log q(x\mid z)$ $\displaystyle-\int dxp(x)\log
p(x)$ $\displaystyle=\int dxdzp(x,z)\log q(x\mid z)$ $\displaystyle+H(X)$
$\displaystyle=H-D$

All in all, the variational information bottleneck can be regarded as:

$H-D\leq I\leq R$ (26)

where $H$ is the entropy of the data.

In $\beta$-VAE framework, $\beta_{C}$ impose a experiment-specific constraint
on compressor and decompressor, giving the variational lower bound:

$\displaystyle H-D$ $\displaystyle\leq I\leq R$ (27) $\displaystyle H-D$
$\displaystyle\leq\beta I\leq\beta R$ $\displaystyle H$
$\displaystyle\leq\beta I+D\leq\beta R+D$

In summary, the lower bound can be denoted as:

$\mathcal{L}_{\beta-VAE}(\theta,\phi)=-D-\beta_{C}R\leq-\beta_{C}I-D$ (28)

### C Derivation of eVAE lower bound

According to Eq. (10) and Eq. (17) , we can obtain the loss function of eVAE
in iteration $t$ as follows:

$\displaystyle\mathcal{L}_{eVAE}$ $\displaystyle=E_{\mathbf{z}\sim
q_{\phi}(\mathbf{z}_{t}\mid\mathbf{x}_{t})}\log
p_{\theta}(\mathbf{x}_{t}\mid\mathbf{z}_{t})+$ (29) $\displaystyle
f_{\beta_{t}\sim
R}(\beta_{t},\mathcal{E})D_{KL}(q_{\phi}(\mathbf{z}_{t}\mid\mathbf{x}_{t})\|p(\mathbf{z}))$
$\displaystyle=E_{x_{t+1}\sim
p_{\mathcal{X}}}[\mathcal{L}_{R}+f(\beta_{t})\mathcal{L}_{I}+\mathcal{E}(\Delta\mathcal{L}_{ELBO})]$
$\displaystyle=E_{x_{t+1}\sim
p_{\mathcal{X}}}[\mathcal{L}_{R}+f(\beta_{t})\mathcal{L}_{I}$
$\displaystyle+\mathcal{E}(\Delta\mathcal{L}_{ELBO_{t+1}}+||KL_{t+1}(\beta_{t+1})-c||)]$

According to the VIB theory, $R_{t}$ sets the upper bound on information
bottleneck $I_{t}(X_{t},Z_{t})$ in $t$-th iteration :

$\displaystyle R_{t}$ $\displaystyle=\int q(x_{t})\int q_{\theta}(z_{t}\mid
x_{t})\log\frac{q_{\theta}(z_{t}\mid x_{t})}{p(z_{t})}$ (30)
$\displaystyle=I_{t}(X_{t},Z_{t})+TC(Z_{t})$ $\displaystyle\geq
I_{t}(X_{t},Z_{t})$

and $D_{t}$ sets the lower bound:

$\displaystyle I_{t}$ $\displaystyle=\int
dxdzp(x_{t},z_{t})\log\frac{p(x_{t}\mid z_{t})}{p(x_{t})}$ (31)
$\displaystyle\geq\int dxdzp(x_{t},z_{t})\log\frac{q(x_{t}\mid
z_{t})}{p(x_{t})}$ $\displaystyle=\int dxdzp(x_{t},z_{t})\log q(x_{t}\mid
z_{t})$ $\displaystyle-\int dxp(x_{t})\log p(x_{t})$ $\displaystyle=\int
dxdzp(x_{t},z_{t})\log q(x_{t}\mid z_{t})$ $\displaystyle+H(X)$
$\displaystyle=H-D_{t}$

After optimized by a variational evolutionary learner $\mathcal{E}$, the loss
function of eVAE constrained by a dynamic lower bound is as follows:

$\displaystyle\mathcal{L}_{eVAE}$ $\displaystyle=E_{\mathbf{z}\sim
q_{\phi}\left(\mathbf{z}_{t}\mid\mathbf{x}_{t}\right)}\log
p_{\theta}(\mathbf{x}_{t}\mid\mathbf{z}_{t})+$ (32) $\displaystyle
f_{\beta_{t}\sim
R}\left(\beta_{t},\mathcal{E}\right)D_{KL}(q_{\phi}(\mathbf{z}_{t}\mid\mathbf{x}_{t})\|p(\mathbf{z}))$
$\displaystyle=E_{x_{t+1}\sim
p_{\mathcal{X}}}[\mathcal{L}_{R}+f(\beta_{t})\mathcal{L}_{I}+\mathcal{E}(\Delta\mathcal{L}_{ELBO})]$
$\displaystyle=E_{x_{t+1}\sim
p_{\mathcal{X}}}[\mathcal{L}_{R}+f(\beta_{t})\mathcal{L}_{I}$
$\displaystyle+\mathcal{E}(\Delta\mathcal{L}_{ELBO_{t+1}}+||KL_{t+1}(\beta_{t+1})-c||)]$
$\displaystyle=-D_{t}-\beta_{t}R_{t}-\mathcal{E}(-D_{t+1}-\beta_{t+1}R_{t+1}+D_{t}+c)$
$\displaystyle\leq-\mathcal{E}(\beta_{t})D_{t}-\mathcal{E}(\beta_{t})I_{t}$

where $c$ is the desired KL point and $\beta_{t}$ is a chromosome sample in
$t$-th iteration.

### D Experimental settings

Table 1 shows the experimental settings of the three learning tasks for
evaluating eVAE. We give the settings of the encoder-decoder network
architectures and the parameters for the evolutionary learning on three
datasets dSprites, CelebA, and PTB and their corresponding optimizers.

Dataset | Optimiser |  | Architecture | eVAE initialization |
---|---|---|---|---|---
dSprites | Adam(1e-4) |  | Input
---
Encoder
Latent
Decoder
| (64,64,1)
---
Conv(32, 32, 32), ReLu, Conv(32,16,16),
ReLu, Conv(32,8,8), ReLu,Conv(32,4,4),
Relu, Linear(256), Relu, Linear(256),
Relu, Linear(20)
10
Linear(256), Relu, Linear(256),
Relu, Linear(512), Relu,
Conv(32,4,4), Relu, Conv(32,8,8),
Relu, Conv(32,16,16), Relu,
Conv(32,32,32), Relu, (1, 64,64)
| Mutation Rate
---
Crossover Rate
Individual Number
| 0.001
---
0.04
20
| CelebA
---
(KL = 200)
Adam(1e-4) |  | Input
---
Encoder
Latent
Decoder
| (128,128,3)
---
Conv(32, 32, 32), ReLu,
Conv(32,16,16), ReLu,
Conv(32,8,8),ReLu,
Conv(32,4,4), Relu, Linear(256),
Relu, Linear(256), Relu, Linear(20)
10
Linear(256), Relu, Linear(256),
Relu, Linear(512), Relu,
Conv(32,4,4), Relu,
Conv(32,8,8), Relu,
Conv(32,16,16), Relu, Conv(32,32,32),
Relu, (3, 64,64)
| Mutation Rate
---
Crossover Rate
Individual Number
| 0.001
---
0.03
20
| PTB
---
(KL = 3)
| init_lr: 0.001
---
threshold:100
decay_factor: 0.5
max_decay: 5
| Input
---
Encoder
Latent
Decoder
|  | Mutation Rate
---
Crossover Rate
Individual Number
| 0.006
---
0.07
10

Table 1: Experimental settings for disentangled representation learning on
dSprites, image generation on CelebA, and language generation on PTB.

### E Analysis of element-wise KL divergence in disentangled representation
learning

Figure 8 further shows the element-wise KL divergence evolving over iterations
of three VAE models: $\beta$-VAE and ControlAVE in comparison with our eVAE on
the dataset dSprites.

Figure 8: The element-wise KL divergence over iterations for $\beta$-VAE,
ControlAVE, and eVAE on dSprites. It shows that each KL divergence of eVAE is
larger than that of $\beta$-VAE, providing an enough space for generating
disentangled representation. In addition, eVAE generates more stable KL
divergence compared to ControlVAE.