Naturally, a number of works have proposed other convolution-based flows that permit larger kernel sizes. A number of these apply variations on causal convolutions [153], including emerging convolutions [80] whose inverse is entirely sequential, MaCow [125] which uses smaller conditional fields allowing more efficient sampling, and MintNet [181] which approximates the inverse using fixed-point iteration. Alternative approaches to causal masking involve imposing repeated (periodic) structure  [95], however in general this is not a good assumption for image modelling, as well as representing convolutions as exponential matrix-vector products, $\exp(\vec{M})\vec{x}$, which can be approximated implicitly with a power series, allowing otherwise unconstrained kernels [81].

% exp(M) \cdot \vec{x} can be represented as a power series: exp(M) \cdot \vec{x} = \sum_{i=0}^\inf \frac{M^i \cdot \vec{x}}{i!}. So therefore exp(M) can be calculated purely in terms of convolutions, never needing to store the whole of M in memory which would be ridiculous. Note that exp(M)^{-1} = exp(-M) so the inverse is almost the same. Also \ln \det exp(M) = Trace(M).

% Unlike i-ResNet, MintNet's Jacobian determinants can be calculated exactly.

\subsection{Residual Flows}
Residual networks [70] are a popular technique to build deep neural networks, avoiding vanishing gradients by stacking blocks of the form
\begin{equation}
\vec{y} = \vec{x} + f_\theta(\vec{x}).
\end{equation}
By restricting $f_\theta$ it is possible to enforce invertibility.

\subsubsection{Matrix Determinant Lemma}
If a function has a certain residual form, then its Jacobian determinant can be computed with the matrix determinant lemma [166]. A simple example is planar flow [166] which is equivalent to a 3 layer MLP with a single neuron bottleneck:
\begin{equation}
\vec{y} = \vec{x} + \vec{u} h(\vec{w}^T \vec{x} + b),
\end{equation}
where $\vec{u}, \vec{w} \in \reals^d$, $b \in \reals$, and $h$ is a differentiable non-linearity function. Planar flows are invertible provided some simple conditions are satisfied, however its inverse is difficult to compute making it only practical for density estimation tasks. A higher rank generalisation of the matrix determinant lemma has been applied to planar flows, known as Sylvester flows, removing the severe bottleneck thus allowing greater representation ability [68, 11].

% Radial flows are another one [166].

\subsubsection{Lipschitz Constrained}
By making the Lipschitz constant of $f_\theta$ less than 1, $\norm{f_\theta}_L < 1$, then this block is invertible [6]. The inverse, however, has no closed form definition but can be found through fixed-point iteration which by the Banach fixed-point theorem converges to a fixed unique solution at an exponential rate dependant on $\text{Lip}(f_\theta)$. The authors originally proposed a biased approximation of the log determinant of the Jacobian as a power series where the Jacobian trace is approximated using Hutchkinson's trace estimator (see Table \ref{tab:normalizing-flows}), but an unbiased approximator known as a Russian roulette estimator has also been proposed [23]. Unlike coupling layers, residual flows have dense Jacobians, allowing all inputs to affect all outputs. Enforcing strong Lipschitz constraints has been achieved with convolutional networks [136, 119, 54] as well as self-attention [102].

Making strong Lipschitz assumptions severely restricts the class of functions learned; an $N$ layer residual flow network is at most $2^N$-Lipshitz. Implicit flows [122], a proposed solution to this, are based around solving implicit equations of the form
\begin{equation}
F(\vec{x}, \vec{y}) = f_\theta(\vec{x}) - f_\phi(\vec{y}) + \vec{x} - \vec{y} = \vec{0},
\end{equation}
where both $f_\theta$ and $f_\phi$ both have Lipschitz constants less than 1. Both the forwards (solve for $\vec{y}$ given $\vec{x}$) and backwards (solve for $\vec{x}$ given $\vec{y}$) directions require solving a root finding problem similar to the inverse process of residual flows; indeed, an implicit flow is equivalent to the composition of a residual flow and the inverse of a residual flow. This allows them to model arbitrary Lipschitz transformations.

% Make sure all Jacobians are capitalised
% All Continuous time flows have to be Lipschitz continuous, but not as strongly, this can be obtained just with smooth Lipschitz activations.

\subsection{Surjective and Stochastic Layers}
Restricting the class of functions available to those that are invertible introduces a number of practical problems related to the topology-preserving property of diffeomorphisms. For example, mapping a uni-modal distribution to a multi-modal distribution is extremely challenging, requiring a highly varying Jacobian [39]. By composing bijections with surjective or stochastic layers these topological constraints can be bypassed [143]. While the log-likelihood of stochastic layers can only be bounded by their ELBO, functions surjective in the inference direction permit exact likelihood evaluation even with altered dimensionality. Surjective transformations have the following likelihood contributions:
\begin{equation}
\mathbb{E}_{q(\vec{y}|\vec{x})} \bigg[ \ln \frac{p(\vec{x}|\vec{y})}{q(\vec{y}|\vec{x})} \bigg],
\end{equation}
where $p(\vec{x}|\vec{y})$ is deterministic for generative surjections, and $q(\vec{y}|\vec{x})$ is deterministic for inference surjections.

One approach to build a surjective layer is to augment the input space with additional dimensions allowing smoother transformation to be learned [43, 82, 22]; the inverse process, where some dimensions are factored out, is equivalent to a multi-scale architecture [38].
% This approach loses asymptotically unbiased sampling.
Another approach known as RAD [39] learns a partitioning of the data space into disjoint subsets $\{\mathcal{Y}_i\}_{i=1}^K$, and applies piece-wise bijections to each region $g_i \colon \mathcal{X} \to \mathcal{Y}_i, \forall i \in \{1,\dots,K\}$. The generative direction learns a classifier on $\mathcal{X}$, $i \sim p(i|\vec{x})$, allowing the inverse to be calculated as $\vec{y} = g_i(\vec{x})$. Similar to both of these approaches are CIFs [33] which consider a continuous partitioning of the data space via augmentation, which is equivalent to an infinite mixture of normalizing flows. Other approaches include modelling finite mixtures of flows [42].
% DeepGMMs \cite{vandenoord2015locallyconnected} - special case of CIF where bijections are linear transformations.

Some powerful stochastic layers have already been discussed in this survey, namely VAEs [105] and DDPMs [76]. Stochastic layers have been incorporated into normalizing flows by interleaving small energy models, sampled with MCMC, between bijectors. % Read more on this to explain it more/better.

% Flow++ with its dequantization [75].

% The term Normalizing Flow was coined in "density estimation by dual ascent of the log-likelihood" and 2A family of nonparametric density estimation algorithms"

% Rather than assuming the data follows a simple, e.g. gaussian distribution, can be arbitrarily complex.

% \paragraph{Implementations} NICE [37]. RealNVP [38]. GLOW [106]. IAF [107]. MAF [158]. NODE [24]  and Ffjord [55]. i-RevNet?. More recent flows e.g. Neural Spline Flows, Block Neural Autoregressive Flow

% $f$ - functions. $h$ - non-linearities. $x$ - input. $y$ - output. All other varaibles are free parameters

% ------------> INSERT DOUBLE WIDTH TABLE HERE

% \subsection{Universality}
% Would be nice to have bit of a proof that normalizing flows can express any distribution e.g. \url{https://arxiv.org/pdf/1912.02762.pdf} . Formal proof \cite{Bogachev2005TriangularTransformationsMeasures}

% For several autoregressive flows the universality property has been proven [Huang et al., 2018; Jaini et al., 2019]

% Also more formal formulation? Data can be understood as a sample from a measured ``data'' space $(\mathcal{Y},\Sigma_{\mathcal{Y}},\upsilon)$, which we want to learn. To do that one can introduce a simpler measured space $(\mathcal{Z},\Sigma_{\mathcal{Z}},\mu)$ and find a function $g \colon \mathcal{Z} \to \mathcal{Y}$, such that $\upsilon=g \ast \mu$. This function $g$ can be interpreted as a ``generator'', and $\mathcal{Z}$ as a latent space. This view puts generative models in the context of transportation theory [207].

\subsection{Continuous Time Flows}
It is possible to consider a normalizing flow with an infinite number of steps that is defined instead by an ordinary differential equation specified by a Lipschitz continuous neural network $f$ with parameters $\theta$, that describes the transformation of a hidden state $\vec{z}(t) \in \mathbb{R}^D$ [24],
\begin{equation}
\frac{\partial\vec{z}(t)}{\partial t} = f(\vec{z}(t), t, \theta).
\end{equation}
Starting from input noise $\vec{z}(0)$, an ODE solver can solve an initial value problem for some time T, at which data is defined, $\vec{z}(T)$. Modelling a transformation in this form has a number of advantages such as inherent invertibility by running the ODE solver backwards, parameter efficiency, and adaptive computation. However, it is not immediately clear how to train such a model through backpropagation. While it is possible to backpropagate directly through an ODE solver, this limits the choice of solvers to differentiable ones as well as requiring large amounts of memory. Instead, the authors apply the adjoint sensitivity method which instead solves a second, augmented ODE backwards in time and allows the use of a black box ODE solver. That is, to optimise a loss dependent on an ODE solver: % IMPORTANT: invertible given that it is Lipschitz continuous!
\begin{equation}
\begin{split}
\mathcal{L}(\vec{z}(t_1)) &= \mathcal{L}\bigg( \vec{z}(t_0) + \int_{t_0}^{t_1} f(\vec{z}(t),t,\theta)dt \bigg), \\
&= \mathcal{L}(\text{ODESolve}(\vec{z}(t_0), f, t_0, t_1, \theta)),
\end{split}
\end{equation}
the adjoint $\vec{a}(t)=\frac{\partial \mathcal{L}}{\partial \vec{z}(t)}$ can be used to calculate the derivative of loss with respect to the parameters in the form of another initial value problem [160],
\begin{equation}
\frac{\partial\mathcal{L}}{\partial\theta} = \int_{t_1}^{t_0} \Big(\frac{\partial \mathcal{L}}{\partial \vec{z}(t)}\Big)^T \frac{\partial f(\vec{z}(t), t, \theta)}{\partial \theta} dt,
\end{equation}
which can be efficiently evaluated by automatic differentiation at a time cost similar to evaluating $f$ itself.

Despite the complexity of this transformation, the continuous variant of the change of variables rule is remarkably simple:
\begin{equation}
\frac{\partial \ln p(\vec{z}(t))}{\partial t} = - tr \bigg( \frac{\partial}{\partial\vec{z}(t)} f(\vec{z}(t),t,\theta) \bigg),
\end{equation}
and can be computed using an ODE solver as well. The resulting continuous-time flow is known as FFJORD [55]. Since the length of the flow tends to infinity (an infinitesimal flow), the true posterior distribution can be recovered [166].

% Consider the Langevin Flow; in this case, the stationary solution is given by the Boltzmann distribution $q_\infty(\vec{z})\propto e^{-\loss(\vec{z})}$. Transforming an intial density $q_0(\vec{z})$ though the Langevin SDE, the resulting points $z_\infty$ are distributed according to $q_\infty(\vec{z})$, the true posterior [166].
% NOTE: this is about SDEs so how trye is it, probably better to just say it is the case and cite someone else saying it.
% Then again, maybe this should be added, see Augmented normalizing flows appendix about deterministic langevin diffusion equation.

% Continuous time flows offer significantly better parameter usage compared to typical normalizing flows but they are much slower due to the use of an ODE solver. Additionally, their continuous nature means that there are functions they are unable to represent.
% Not sure how true this is any more so removed until checked

As previously mentioned, invertible functions suffer from topological problems; this is especially true for Neural ODEs since their continuous nature prevents trajectories from crossing. Similar to augmented normalizing flows [82], this can be solved by providing additional dimensions for the flow to traverse [43]. Specifically, a $p$-dimensional Euclidean space can be approximated by a Neural ODE in a $(2p+1)$-dimensional space [222].

% Accelerated flow for probability distributions \cite{taghvaei2019accelerated} - show that solving a family of Hamiltonian ODEs with an infinite time horizon gives a transport map from the initial data distribution to an arbitrary target distribution with a log-concave density function.

% Neural ODEs have also been applied to stochastic differential equations (SDE) resulting in a generative diffusion process [200] as well as stochastic normalizing flows \cite{wu2020stochastic}.

% Neural Stochastic Differential Equations use neural nets to model the coefficients of a stochastic differential equation. To train Neural SDE one needs an analog of backpropagation, Tzen and Raginsky [2019] proposed the use of Kunita’s theory of stochastic flows. They manifest nice mathematical properties but have not found practical applications.

% `When the length of the flow tends to infinity, known as an infinitesimal flow, the flow is described by a partial differential equation (i.e. Neural ODE). One important flow is the Langevin Flow. In this case the stationary solution is given by the Boltzmann distribution $q_\infty(\vec{z})\propto e^{-\mathcal{L}(\vec{z})}$ i.e. if we start from an initial density $q_0(\vec{z})$ and evolve its samples through the Langevin SDE, the resulting points $z_\infty$ will be distributed according to $q_\infty(\vec{z})$, the true posterior, overcoming one oft-quoted limitation of variational inference.' - [166].

% Much much better parameter usage. But much slower due to use of ODE solver. Continuity is bad in some cases - Augmented NODE

% In `Approximation capabilities of neural ordinary differential equations' - Zhang 2019 they note that while vanilla neural ODEs are not universal approximators, depth-invariant neural ODEs are.

% This section needs work. Need to understand it properly.
\subsubsection{Regularising Trajectories}
ODE solvers can require large numbers of network evaluations, notably when the ODE is stiff or the dynamics change quickly in time. By introducing regularisation, a simpler ODE can be learned, reducing the number of evaluations required. Specifically, all works here are inspired by optimal transport theory to encourage straight trajectories. Monge-Amp\`ere Flow [225] and Potential Flow Generators [218] parameterise a potential function satisfying the Monge-Amp\`ere equation [19, 207] with a neural network. RNODE [49] applies transport costs to FFJORD as well as regularising the Frobenius norm of the Jacobian, encouraging straight trajectories. OT-Flow [151] combines these approaches, parameterising a potential as well as applying transport costs, additionally, utilising the optimal transport derivation they derive an exact trace definition with similar complexity to using Hutchkinson's trace estimator.

% \subsection{Variants}
% Although the continuous nature of normalizing flows is beneficial for many tasks, discretisation can be essential, e.g. for text generation. A recurrent neural network could be trained first then subsequently train a flow on the latent space [233]. However, applying the straight-through gradient estimator [8] permits a fully invertible architecture with faster runtime than RNNs \cite{Tran2019DiscreteFlowsInvertible, Hoogeboom2019IntegerDiscreteFlows}. Additionally, a number of data modalities such as images are discrete but must be dequanitised to avoid arbitrarily high likelihoods; the choice of dequantisation procedure has been found to significantly affect performance \cite{Hoogeboom2020LearningDiscreteDistributions}. Videos have been synthesised with flow models by autoregressively estimating the latents for a flow model in an invertible manner, achieving significant speedup over standard autoregressive approaches [111]. To improve the quality of samples generated from flow models, adversarial learning has been applied by jointly training through max-likelihood and adversarial loss by using an invertible network as the generator in a GAN \cite{Grover2018FlowGANCombiningMaximum}. This also allows alternative divergences to be used such as Wasserstein distance [157].

% Boltzmann Generators: one shot sampling of energy functions \cite{Noe2019BoltzmannGeneratorsSampling}

% TO POTENTIALLY ADD: Convex Potential Flows \cite{huang2021convex}

% Normalizing flows have been applied to a number of probabilistic tasks such as importance sampling [137], one shot sampling of energy functions \cite{Noe2019BoltzmannGeneratorsSampling}, supporting mixing in energy-based models [144], and jointly training energy-based and flow models [50].

% `Besides  difficulties  that could arise due to optimisation (in both MLE and adversarial learning), the optimality of MLE holds only when there is no model misspecification for the generator i.e., the true data distribution $P_{data}$ is a member of the parametric family of distributions under consideration (White 1982). This is generally not the case for high-dimensional distributions, and hence the choice of the learning objective becomes largely an  empirical  question.' - FlowGAN
% `a naive Gaussianmixture model can outperform adversarially learned modelson both log-likelihood estimates' - FlowGAN

% [157] - They also discuss generalisation to non-euclidean spaces and Riemannian manifolds

% A trade-off between the complexity of the generative transformation and the form of base measure was explored in [90].

% What inductive bias does the architecture impose? For instance, Spantini et al. [2017] investigate the relation between the sparsity of the triangular flow and Markov property of the target distribution.

% Cons
% \begin{itemize}
%     \item Hard to get expressive powerful invertible functions
%     \item Trade offs between density estimation speed/sampling speed and expressivity
%     \item The problems that come with likelihood based methods i.e. having to output $-\infty$ to have $p(x)=0$
%     \item Tend to be parameter inefficient
%     \item Input dimensionality has to be equal to output dimensionality
% \end{itemize}

% Pros
% \begin{itemize}
%     \item No chance of mode collapse
%     \item Potential for must faster sampling than autoregressive likelihood methods
%     \item No need for quantization unlike autoregressive likelihood methods
% \end{itemize}

% \subsection{Notes}
% Variational inference requires that intractable posterior distributions be approximated by a class of known probability distributions, over which we search for the best approximation to the true posterior [166]

% \section{Representation Learning}

% \section{Score Matching/Markov Chain}
% Paper on generative modelling through gradients [178]. Doesn't actually need to sample from a Markov Chain

% The diffusion model [76]

% GSNs: generative stochastic networks (Alain 2016) and Generalized denoising auto-encoders as generative models (Bengio 2013)

% An example of a floating figure using the graphicx package.
% Note that \label must occur AFTER (or within) \caption.
% For figures, \caption should occur after the \includegraphics.
% Note that IEEEtran v1.7 and later has special internal code that
% is designed to preserve the operation of \label within \caption
% even when the captionsoff option is in effect. However, because
% of issues like this, it may be the safest practice to put all your
% \label just after \caption rather than within \caption{}.
% Reminder: the "draftcls" or "draftclsnofoot", not "draft", class
% option should be used if it is desired that the figures are to be
% displayed while in draft mode.
% where an .eps filename suffix will be assumed under latex,
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{Simulation results for the network.}

% Note that the IEEE typically puts floats only at the top, even when this
% results in a large percentage of a column being occupied by floats.
% However, the Computer Society has been known to put floats at the bottom.

% An example of a double column floating figure using two subfigures.
% (The subfig.sty package must be loaded for this to work.)
% The subfigure \label commands are set within each subfloat command,
% and the \label for the overall figure must come after \caption.
% \hfil is used as a separator to get equal spacing.
% Watch out that the combined width of all the subfigures on a
% line do not exceed the text width or a line break will occur.
%\subfloat[Case I]{\includegraphics[width=2.5in]{box}%
%\subfloat[Case II]{\includegraphics[width=2.5in]{box}%
%\caption{Simulation results for the network.}
% Note that often IEEE papers with subfigures do not employ subfigure
% captions (using the optional argument to \subfloat[]), but instead will
% reference/describe all of them (a), (b), etc., within the main caption.
% Be aware that for subfig.sty to generate the (a), (b), etc., subfigure
% labels, the optional argument to \subfloat must be present. If a
% subcaption is not desired, just leave its contents blank,
% e.g., \subfloat[].

% An example of a floating table. Note that, for IEEE style tables, the
% \caption command should come BEFORE the table and, given that table
% captions serve much like titles, are usually capitalized except for words
% such as a, an, and, as, at, but, by, for, in, nor, of, on, or, the, to
% and up, which are usually not capitalized unless they are the first or
% last word of the caption. Table text will default to \footnotesize as
% the IEEE normally uses this smaller font for tables.
% The \label must come after \caption as always.
%% increase table row spacing, adjust to taste
% if using array.sty, it might be a good idea to tweak the value of
% \extrarowheight as needed to properly center the text within the cells
%\caption{An Example of a Table}
%% Some packages, such as MDW tools, offer better commands for making tables
%% than the plain LaTeX2e tabular which is used here.
%One & Two\\
%Three & Four\\

% Note that the IEEE does not put floats in the very first column
% - or typically anywhere on the first page for that matter. Also,
% in-text middle ("here") positioning is typically not used, but it
% is allowed and encouraged for Computer Society conferences (but
% not Computer Society journals). Most IEEE journals/conferences use
% top floats exclusively.
% Note that, LaTeX2e, unlike IEEE journals/conferences, places
% footnotes above bottom floats. This can be corrected via the
% \fnbelowfloat command of the stfloats package.

% compendium
\section{Connections between Generative Models}
While deep generative models all maximise the likelihood of the training data, they achieve this in various ways, making different trade offs such as diversity for quality, sampling for training speed, and capacity for complexity. There are also a plethora of hybrid approaches, attempting to get the best of both worlds, or at least find saddle points, balancing trade-offs. The varied connections between these systems mean that advances in one field inevitably benefit others, for instance, improved variational bounds [124, 18] are beneficial for not only VAEs [105] but also diffusion models [75] and surjective flows [143].

An area of particular overlap is the augmentation of training data, acting as regularisation while increasing the quantity of available training data; augmentation can be applied almost freely to GANs (see Section \ref{sec:gan-augmentation}), however, direct application to other architectures would lead to augmentations leaking into samples. By conditioning these models on the augmentation type, effective training is not only possible but has been found to substantially improve sample quality, even on small models [94].

% Significant overlap
% - All maximise likelihood of data in different ways, making different trade offs: diversity for quality, sample/training speed, capacity and complexity, etc.
% - Hybrid approaches are popping up, e.g. combining variational inference with energy models, trying to get the best of both worlds. These aren't independent approaches e.g. energy model inside a flow.
% - Data augmentation is talked about in detail for GANs and has experienced great success; application to likelihood models is more difficult but possible by conditioning on the type of augmentation used [94]
% - Similarly, better variational bound approximations can potentially help, this is discussed in VAEs but other architectures especially denoising diffusion, hybrids and surjective flows have variational bounds which can potentially be improved [124, 18].

% [94]
% Distribution Augmentation for Generative Modeling (Jun et al 2020 ICML) – Augment data and condition the generative model on the type of transformation allowing more aggressive transformations to be used. i.e. for transformation t instead of p(t(x)) use p(t(x)|t). This is applied to autoregressive and flow models.

% [124]
% Unbiased estimation of log marginal probability for latent variable models (Luo et al. 2020 ICLR) – improvements to lower bounding likelihood (ELBO) can be made e.g. this paper, also using importance sampling (Burda et al. 2015 Importance weighted autoencoders) – relevant to everything using ELBO, so especially VAEs.

% An introduction to sampling via measure transport Marzouk 2016 – ‘the process of fitting a transport map by variational inference can be interpreted in terms of the inverse map.’

% \subsection{Applications}
% Just a short-ish paragraph on video, text, etc? Already have the hybrids section in flows which feels out of place compared to the rest of the paper.

\subsection{Assessing Quality}
A huge problem when developing generative models is how to effectively evaluate and compare them. Qualitative comparison of random samples plays a large role in the majority of state-of-the-art works, however, it is subjective and time-consuming to compare many works. Calculating the log-likelihood on a separate validation set is popular for tractable likelihood models but comparison with implicit likelihood models is difficult and while likelihood is a good measure of diversity, it has been shown to not correlate well with quality [190].

One approach to quantify sample quality is Inception Score (IS) [168] which takes a trained classifier and determines whether a sample has low label entropy, indicating that a meaningful class is likely, and whether the distribution of classes over a large number of samples has high entropy, indicating that a diverse range of images can be sampled. A perfect IS can be scored by a model that creates only one image per class [123] leading to the creation of Fr\'echet Inception Distance (FID) [71] which models the activations of a particular layer of a classifier as multivariate Gaussians for real data and generated data, measuring the Fr\'echet distance between the two.

% A note on the inception score (Barratt \& Sharma) – potential problems reported for IS on Cifar-10

These approaches are trivially solved by memorising the dataset and are less applicable to non-natural image-related data. Kernel Inception Distance (KID) [12] mitigates this somewhat, instead calculating the squared maximum mean discrepancy in feature space, however, pretrained features may not be sufficient to detect overfitting. Another approach is to train a neural network to distinguish between real and generated samples similar to the discriminator from a GAN; while this detects overfitting well, it increases the complexity and time required to evaluate a model and is biased towards adversarially trained models [63].

% LPIPS \cite{zhang2018unreasonable}.

% Neural Network Divergences NNDs, Arora et al, Liu et al, Huang et al. all 2017

% Towards GAN Benchmarks which require generalization [63] - CNN Divergence i.e. a DCGAN discriminator trained to classify real/sampled images - requires training a network and adds complexity, prefers GANs over other models - But detects overfitting well, but not diversity.

% One approach is Inception Score [168] which feeds generated samples through an inception model trained on ImageNet and measures their entropy. A downside of this approach is that real data points are not compared to generated ones. An alternative approach is Fr\'echet Inception Distance (FID) [71] which measures the Wasserstein-2 (Fr\'echet) distance between the inception distributions of synthetic and real data samples.

% Estimating likelihood from ELBO could also be here, or just in VAEs. Also relevant to some flows though.

\subsection{Applications}
In general, the definition of a generative model means that any technique can be used on any modality/task, however, some models are more suited for certain tasks. Standard autoregressive networks are popular for text/audio generation [29, 17, 153]; VAEs have been applied but posterior collapse is difficult to mitigate [15, 5]; GANs are more parameter efficient but struggle to model discrete data [142] and suffer from mode collapse [110]; some normalizing flows offer parallel synthesis, providing substantial speedup [233, 195, 161]. Video synthesis is more challenging due its exceptionally high dimensionality, typically approaches combine a latent-based implicit generative model to generate individual frames, with an autoregressive network used to predict future latents [114, 5, 111]. This approach is similar to the one used in reinforcement learning to construct world models [65, 66].

% Text: Autoregressive is natural [29, 17], but also GANs [142], VAEs suffer from posterior collapse [15], and Flows [233, 195] - More accurate with MAF, or parallel synthesis with IAF. Changing the temperature of MLE is better than GAN language models - GANs are continuous, have to approximate discrete with gumbel softmax which isn't great.
% Video: Flow [111], VAE [5], VAE-GAN [114] - most combine autoregressive generation with implicit generation for faster sampling.
% Audio: Autoregressive WaveNet [153], Flow: Parallel Wavenet [152] which needs a teacher and WaveGlow is also parallel without a teacher [161], GANs are more parameter efficient and train faster but mode collapse [110]
% Graph: Autoregressive GraphRNN, Flow GraphNVP, GAN MolGAN
% Reinforcement Learning: RNN Based world models - VAE to encode images to latent vector, autoregressive RNN predicts future latents [65, 66]. Normalizing Flows for density estimating state values \cite{touati2020randomized}.

% \subsection{Variants}
% Although the continuous nature of normalizing flows is beneficial for many tasks, discretisation can be essential, e.g. for text generation. A recurrent neural network could be trained first then subsequently train a flow on the latent space [233]. However, applying the straight-through gradient estimator [8] permits a fully invertible architecture with faster runtime than RNNs \cite{Tran2019DiscreteFlowsInvertible, Hoogeboom2019IntegerDiscreteFlows}. Additionally, a number of data modalities such as images are discrete but must be dequanitised to avoid arbitrarily high likelihoods; the choice of dequantisation procedure has been found to significantly affect performance \cite{Hoogeboom2020LearningDiscreteDistributions}. Videos have been synthesised with flow models by autoregressively estimating the latents for a flow model in an invertible manner, achieving significant speedup over standard autoregressive approaches [111]. To improve the quality of samples generated from flow models, adversarial learning has been applied by jointly training through max-likelihood and adversarial loss by using an invertible network as the generator in a GAN \cite{Grover2018FlowGANCombiningMaximum}. This also allows alternative divergences to be used such as Wasserstein distance [157].

\subsection{Implicit Representation}
Typically deep architectures discussed in this survey are built with data represented as discrete arrays thus using discrete components such as convolutions and self-attention. Implicit representation on the other hand treats data as continuous signals, mapping coordinates to data values [174, 188]. Implicit Gradient Origin Networks (GONs; Fig. \ref{fig:implicit-gon}) [14] form a latent variable model by concatenating latent vectors with coordinates which are passed through an implicit network; here latent vectors are calculated as the gradient of a reconstruction loss with respect to the origin. By sampling using a finer grid of coordinates, super-resolution beyond resolutions seen during training is possible. Another approach to learn an implicit generative model as a GAN is to map latents to the weights of an implicit function using a hyper-network, the output of which is classified by a discriminator [44] (Fig. \ref{fig:implicit-gan}).

% [44] - GAN - Sample z, pass through hypernetwork to get an implicit network, reconstruct at points. Discriminator is a PointConv architecture, a continuous generalisation of convolutions parameterised by MLPs found by nearest neighbour search. Neighbourhood search for the PointConv can be slow.

% [14] - VAE - decoder with no need for any additional networks by using gradients as encodings.

\begin{figure}[t]
\centering
\begin{subfigure}{0.49\linewidth}
\centering
\begin{tikzpicture}[shorten >=1pt,->]
\tikzstyle{cir}=[circle,fill=black!30,minimum size=17pt,inner sep=0pt]
\tikzstyle{dia}=[diamond,fill=black!30,minimum size=19pt,inner sep=0pt]
\tikzstyle{box}=[rounded corners=6pt,fill=black!25,minimum size=17pt,inner xsep=4pt, inner ysep=0pt]
\tikzstyle{sqr}=[fill=black!25,minimum size=17pt, inner ysep=0pt, inner xsep=4pt]

% draw GON
\draw (0,1) -- (0,0);

\node[dia,fill=black!12]                (GON-0)    at (0,1) {$\vec{0}$};
\node[cir,fill=black!12]  (GON-1)    at (0,0) {$\vec{c}$};
\node[cir]                (GON-2)    at (1,0.5) {$\vec{\hat{x}}$};
\draw (0,0.5) -> (GON-2);
\draw[densely dashed] (GON-2.north) to [out=90,in=0] (GON-0.east);
\node[align=center] at (0.42,0.7) {$F$};
% \node[align=center] at (0.42,1.3) {$\nabla$};

\end{tikzpicture}
\caption{Implicit GON [14].}
\label{fig:implicit-gon}
\end{subfigure}
\begin{subfigure}{0.49\linewidth}
\centering
\begin{tikzpicture}[shorten >=1pt,->]
\tikzstyle{cir}=[circle,fill=black!30,minimum size=17pt,inner sep=0pt]
\tikzstyle{dia}=[diamond,fill=black!30,minimum size=19pt,inner sep=0pt]
\tikzstyle{box}=[rounded corners=6pt,fill=black!25,minimum size=17pt,inner xsep=4pt, inner ysep=0pt]
\tikzstyle{sqr}=[fill=black!25,minimum size=17pt, inner ysep=0pt, inner xsep=4pt]

\node[cir]                (GAN-xh)   at (2,1) {$\vec{\hat{x}}$};
\node[box]                (GAN-F)    at (1,1) {$F$};
\node[cir,fill=black!12]  (GAN-z)    at (1,0) {$\vec{z}$};
\node[cir,fill=black!12]  (GAN-x)    at (2,0) {$\vec{x}$};
\node[cir,fill=black!12]  (GAN-c)    at (0,1) {$\mathbf{c}$};
\node[box]                (GAN-rf)   at (3.3,0.5) {0,1};
\draw[>=, decorate, decoration={brace, amplitude=16pt, mirror}] (2.3,0) -- coordinate [left=10pt] (B) (2.3,1) node {};

\draw(GAN-z) -- (GAN-F);
\draw(GAN-F) -- (GAN-xh);
\draw(GAN-c) -- (GAN-F);

\node[align=center] at (2.85,1) {$D$};
\node[align=center] at (1.25,0.5) {$H$};
\end{tikzpicture}
\caption{Implicit GAN [44].}
\label{fig:implicit-gan}
\end{subfigure}
\caption{Implicit networks model data continuously permitting arbitrarily high resolutions. Dashed lines represent gradients, $F$ is an implicit network, and $H$ is a hypernetwork.}
\label{fig:implicit-networks}
\end{figure}

% \section{Discussion}

\section{Conclusion}
% In this paper, we cover recent research in deep generative models, demonstrating how each approach attempts to mimic the data distribution, with particular focus on sampling procedures and the statistical divergences used. Considerable progress has been made but no single framework covers all bases with all making trade-offs in sample quality, diversity, and sampling speed.

% key findings, surprises, and insights/indicators to the future (that people can cite) "Bond-Taylor et al., review... and show/indicate that... X" - the X needs to be obvious from the conclusion

% While GANs have led the way in terms of sample quality for some time now, the gap between other approaches is shrinking; the diminished mode collapse and simpler training objectives make these models more enticing than ever, however, the number of parameters required in addition to slow run-times pose a substantial handicap. Despite this, recent work notably in hybrid models offers a balance between extremes at the expense of extra complexity. A clear stand out is the innovative application of data augmentation within generative models, offering benefits across all techniques, impressively, without necessitating more powerful architectures. When it comes to scaling models to high dimensional data, attention is a common theme, allowing long-range dependencies to be learned; recent advances in linear attention will aid scaling to even higher resolutions. Implicit networks are another promising direction, allowing efficient synthesis of arbitrarily high resolution and irregular data. Similar unified generative models capable of modelling continuous, irregular, and arbitrary length data, from different domains will be key for the future of generalisation.

% made a few changes below, sorry was on different editor
While GANs have led the way in terms of sample quality for some time now, the gap between other approaches is shrinking; the diminished mode collapse and simpler training objectives make these models more enticing than ever, however, the number of parameters required in addition to slow run-times pose a substantial handicap. Despite this, recent work notably in hybrid models offers a balance between extremes at the expense of extra model complexity that hinders broader adoption. A clear stand out is the application of innovative data augmentation strategies within generative models, offering benefits impressively without necessitating more powerful architectures. When it comes to scaling models to high-dimensional data, attention is a common theme, allowing long-range dependencies to be learned; recent advances in linear attention will aid scaling to even higher resolutions. Implicit networks are another promising direction, allowing efficient synthesis of arbitrarily high resolution and irregular data. Similar unified generative models capable of modelling continuous, irregular, and arbitrary length data, over different scales and domains will be key for the future of generalisation.

% Modelling approaches that handle such data types; continuous, irregular, arbitrarily length, and from different domains, under a single unified generative model will be key for the future of generalisation.

% Combining the various techniques discussed in this survey will be key to unsupervised learning of cross-modal data

% Another interesting future direction is unsupervised learning of cross-modal data

% Looking forward, the lack of mode collapse and single training objectives in non-adversarial architectures offers clear advantages. While the gap in sample quality is shrinking, the number of parameters required in addition to slow run-times pose a substantial handicap. Despite this, recent work notably in hybrid models offer a balance between extremes. When it comes to scaling model to high dimensional data, attention is a common theme, allowing long range dependencies to be learned; recent advances in linear attention will aid scaling to even higher resolutions. Implicit networks are another promising direction, allowing efficient synthesis of arbitrarily high resolution data.

% While GANs still lead the way in terms of sample quality, this gap is shrinking

% if have a single appendix:
%\appendix[Proof of the Zonklar Equations]
% or
%\appendix  % for no appendix heading
% do not use \section anymore after \appendix, only \section*
% is possibly needed

% use appendices with more than one appendix
% then use \section to start each appendix
% you must declare a \section before using any
% \subsection or using \label (\appendices by itself
% starts a section numbered zero.)

% \appendices
% \section{Proof of the First Zonklar Equation}
% Appendix one text goes here.

% % you can choose not to have a title for an appendix
% % if you want by leaving the argument blank
% \section{}
% Appendix two text goes here.

% use section* for acknowledgment
% \ifCLASSOPTIONcompsoc
%   % The Computer Society usually uses the plural form
%   \section*{Acknowledgments}
% \else
%   % regular IEEE prefers the singular form
%   \section*{Acknowledgment}
% \fi

% The authors would like to thank...

% Can use something like this to put references on a page
% by themselves when using endfloat and the captionsoff option.
\ifCLASSOPTIONcaptionsoff
\newpage
\fi

% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
% The "triggered" command can be changed if desired:

% references section

% can use a bibliography generated by BibTeX as a .bbl file
% BibTeX documentation can be easily obtained at:
% http://mirror.ctan.org/biblio/bibtex/contrib/doc/
% The IEEEtran BibTeX style support page is at:
% http://www.michaelshell.org/tex/ieeetran/bibtex/
% argument is your BibTeX string definitions and bibliography database(s)
% <OR> manually copy in the resultant .bbl file
% set second argument of \begin to the number of references
% (used to reserve space for the reference number labels box)
\bibliographystyle{plain}
\begin{thebibliography}{100}

[1]
Guillaume Alain, Yoshua Bengio, Li~Yao, Jason Yosinski, \'eric
Thibodeau-Laufer, Saizheng Zhang, and Pascal Vincent.
\newblock {GSNs}: generative stochastic networks.
\newblock {\em Information and Inference: A Journal of the IMA}, 5(2):210--249,

[2]
Michael Arbel, Liang Zhou, and Arthur Gretton.
\newblock Generalized {Energy} {Based} {Models}.
\newblock In {\em ICLR}, 2021.

[3]
Martin Arjovsky and Leon Bottou.
\newblock Towards {Principled} {Methods} for {Training} {Generative}
{Adversarial} {Networks}.
\newblock In {\em ICLR}, 2017.

[4]
Martin Arjovsky, Soumith Chintala, and L\'eon Bottou.
\newblock Wasserstein {GAN}.
\newblock {\em arXiv:1701.07875}, 2017.

[5]
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy~H. Campbell, and Sergey
\newblock Stochastic {Variational} {Video} {Prediction}.
\newblock In {\em ICLR}, 2018.

[6]
Jens Behrmann, Will Grathwohl, Ricky T.~Q. Chen, David Duvenaud, and
Jörn-Henrik Jacobsen.
\newblock Invertible {Residual} {Networks}.
\newblock In {\em ICML}, 2019.

[7]
Yoshua Bengio, R\'ejean Ducharme, Pascal Vincent, and Christian Jauvin.
\newblock A {Neural} {Probabilistic} {Language} {Model}.
\newblock {\em JMLR}, 3(Feb):1137--1155, 2003.

[8]
Yoshua Bengio, Nicholas L\'eonard, and Aaron Courville.
\newblock Estimating or {Propagating} {Gradients} {Through} {Stochastic}
{Neurons} for {Conditional} {Computation}.
\newblock {\em arXiv:1308.3432}, 2013.

[9]
Yoshua Bengio, Li~Yao, Guillaume Alain, and Pascal Vincent.
\newblock Generalized {Denoising} {Auto}-{Encoders} as {Generative} {Models}.
\newblock In C.~J.~C. Burges, L.~Bottou, M.~Welling, Z.~Ghahramani, and K.~Q.
Weinberger, editors, {\em NeurIPS 26}, pages 899--907. Curran Associates,
Inc., 2013.

[10]
Yoshua Bengio, Li~Yao, Guillaume Alain, and Pascal Vincent.
\newblock Generalized {Denoising} {Auto}-{Encoders} as {Generative} {Models}.
\newblock {\em NeurIPS}, 26, 2013.

[11]
Rianne van~den Berg, Leonard Hasenclever, Jakub~M. Tomczak, and Max Welling.
\newblock Sylvester {Normalizing} {Flows} for {Variational} {Inference}.
\newblock In {\em UAI}, 2018.

[12]
Mikołaj Bińkowski, Dougal~J. Sutherland, Michael Arbel, and Arthur Gretton.
\newblock Demystifying {MMD} {GANs}.
\newblock In {\em ICLR}, 2018.

[13]
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam.
\newblock Optimizing the {Latent} {Space} of {Generative} {Networks}.
\newblock {\em arXiv:1707.05776}, 2019.

[14]
Sam Bond-Taylor and Chris~G. Willcocks.
\newblock Gradient {Origin} {Networks}.
\newblock In {\em ICLR}, 2021.

[15]
Samuel~R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew~M. Dai, Rafal Jozefowicz,
and Samy Bengio.
\newblock Generating {Sentences} from a {Continuous} {Space}.
\newblock {\em arXiv:1511.06349}, 2016.

[16]
Andrew Brock, Jeff Donahue, and Karen Simonyan.
\newblock Large {Scale} {GAN} {Training} for {High} {Fidelity} {Natural}
{Image} {Synthesis}.
\newblock {\em arXiv:1809.11096}, 2019.

[17]
Tom~B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan,
Rewon Child, Aditya Ramesh, Daniel~M. Ziegler, Jeffrey Wu, Clemens Winter,
Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray,
Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford,
Ilya Sutskever, and Dario Amodei.
\newblock Language {Models} are {Few}-{Shot} {Learners}.
\newblock {\em arXiv:2005.14165}, 2020.

[18]
Yuri Burda, Roger~B. Grosse, and Ruslan Salakhutdinov.
\newblock Importance {Weighted} {Autoencoders}.
\newblock In {\em ICLR}, 2016.

[19]
Luis~A. Caffarelli and Mario Milman.
\newblock {Monge} {Ampère} {Equation}: {Applications} to {Geometry},
\newblock American Mathematical Soc., 1999.

[20]
Miguel~A Carreira-Perpinan and Geoffrey~E Hinton.
\newblock On {Contrastive} {Divergence} {Learning}.
\newblock In {\em AISTATS}, volume~10, pages 33--40, 2005.

[21]
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull,
Yuan Cao, and Yoshua Bengio.
\newblock Your {GAN} is {Secretly} an {Energy}-based {Model} and {You} {Should}
{Use} {Discriminator} {Driven} {Latent} {Sampling}.
\newblock {\em NeurIPS}, 33, 2020.

[22]
Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian.
\newblock {VFlow}: {More} {Expressive} {Generative} {Flows} with {Variational}
{Data} {Augmentation}.
\newblock In {\em ICML}, pages 1660--1669. PMLR, 2020.

[23]
Ricky T.~Q. Chen, Jens Behrmann, David~K. Duvenaud, and Joern-Henrik Jacobsen.
\newblock Residual {Flows} for {Invertible} {Generative} {Modeling}.
\newblock In {\em NeurIPS}, pages 9916--9926, 2019.

[24]
Ricky T.~Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David~K Duvenaud.
\newblock Neural {Ordinary} {Differential} {Equations}.
\newblock In S.~Bengio, H.~Wallach, H.~Larochelle, K.~Grauman, N.~Cesa-Bianchi,
and R.~Garnett, editors, {\em NeurIPS 31}, pages 6571--6583. Curran
Associates, Inc., 2018.

[25]
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby.
\newblock Self-{Supervised} {GANs} via {Auxiliary} {Rotation} {Loss}.
\newblock In {\em IEEE CVPR}, pages 12154--12163, 2019.

[26]
Xi~Chen, Diederik~P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John
Schulman, Ilya Sutskever, and Pieter Abbeel.
\newblock Variational {Lossy} {Autoencoder}.
\newblock In {\em ICLR}, 2017.

[27]
Xi~Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel.
\newblock {PixelSNAIL}: {An} {Improved} {Autoregressive} {Generative} {Model}.
\newblock {\em ICML}, 2:1364--1372, 2017.

[28]
Rewon Child.
\newblock Very {Deep} {VAEs} {Generalize} {Autoregressive} {Models} and {Can}
{Outperform} {Them} on {Images}.
\newblock In {\em ICLR}, 2021.

[29]
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
\newblock Generating {Long} {Sequences} with {Sparse} {Transformers}.
\newblock {\em arXiv:1904.10509}, 2019.

[30]
Francois Chollet.
\newblock Xception: {Deep} {Learning} {With} {Depthwise} {Separable}
\newblock In {\em IEEE CVPR}, pages 1251--1258, 2017.

[31]
Krzysztof~Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song,
Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared~Quincy Davis, Afroz
Mohiuddin, Lukasz Kaiser, David~Benjamin Belanger, Lucy~J. Colwell, and
Adrian Weller.
\newblock Rethinking {Attention} with {Performers}.
\newblock In {\em ICLR}, 2021.

[32]
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.
\newblock Empirical {Evaluation} of {Gated} {Recurrent} {Neural} {Networks} on
{Sequence} {Modeling}.
\newblock {\em arXiv:1412.3555}, 2014.

[33]
Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet.
\newblock Relaxing {Bijectivity} {Constraints} with {Continuously} {Indexed}
{Normalising} {Flows}.
\newblock In {\em ICML}, pages 2133--2143. PMLR, 2020.

[34]
Bin Dai and David Wipf.
\newblock Diagnosing and {Enhancing} {VAE} {Models}.
\newblock {\em arXiv:1903.05789}, 2019.

[35]
Nicola De~Cao, Ivan Titov, and Wilker Aziz.
\newblock Block {Neural} {Autoregressive} {Flow}.
\newblock In {\em UAI}, 2019.

[36]
Emily~L. Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus.
\newblock Deep {Generative} {Image} {Models} using a {Laplacian} {Pyramid} of
{Adversarial} {Networks}.
\newblock {\em NeurIPS}, 28:1486--1494, 2015.

[37]
Laurent Dinh, David Krueger, and Yoshua Bengio.
\newblock {NICE}: {Non}-linear independent components estimation.
\newblock In {\em 3rd {International} {Conference} on {Learning}
{Representations}, {ICLR} 2015 - {Workshop} {Track} {Proceedings}}.
International Conference on Learning Representations, ICLR, 2015.

[38]
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.
\newblock Density estimation using {Real} {NVP}.
\newblock {\em 5th International Conference on Learning Representations, ICLR
2017 - Conference Track Proceedings}, 2016.

[39]
Laurent Dinh, Jascha Sohl-Dickstein, Razvan Pascanu, and Hugo Larochelle.
\newblock A {RAD} approach to deep mixture models.
\newblock In {\em {ICLR} {Workshop}}, 2019.

[40]
Alexey Dosovitskiy and Thomas Brox.
\newblock Generating {Images} with {Perceptual} {Similarity} {Metrics} based on
{Deep} {Networks}.
\newblock {\em NeurIPS}, pages 658--666, 2016.

[41]
Yilun Du and Igor Mordatch.
\newblock Implicit {Generation} and {Generalization} in {Energy}-{Based}
\newblock {\em NeurIPS}, 33, 2019.

[42]
Leo~L. Duan.
\newblock Transport {Monte} {Carlo}.
\newblock {\em arXiv:1907.10448}, 2020.

[43]
Emilien Dupont, Arnaud Doucet, and Yee~Whye Teh.
\newblock Augmented {Neural} {ODEs}.
\newblock In H.~Wallach, H.~Larochelle, A.~Beygelzimer,
F.~d{\textbackslash}textquotesingle Alch\'e-Buc, E.~Fox, and R.~Garnett,
editors, {\em NeurIPS 32}, pages 3140--3150. Curran Associates, Inc., 2019.

[44]
Emilien Dupont, Yee~Whye Teh, and Arnaud Doucet.
\newblock Generative {Models} as {Distributions} of {Functions}.
\newblock {\em arXiv:2102.04776}, 2021.

[45]
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios.
\newblock Cubic-{Spline} {Flows}.
\newblock In {\em Workshop on {InvertibleNeural} {Nets} and {Normalizing}
{Flows}, {ICML}}, 2019.

[46]
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios.
\newblock Neural {Spline} {Flows}.
\newblock In H.~Wallach, H.~Larochelle, A.~Beygelzimer,
F.~d{\textbackslash}textquotesingle Alch\'e-Buc, E.~Fox, and R.~Garnett,
editors, {\em NeurIPS 32}, pages 7511--7522. Curran Associates, Inc., 2019.

[47]
Conor Durkan, Iain Murray, and George Papamakarios.
\newblock On {Contrastive} {Learning} for {Likelihood}-free {Inference}.
\newblock In {\em ICML}, pages 2771--2781. PMLR, 2020.

[48]
Patrick Esser, Robin Rombach, and Björn Ommer.
\newblock Taming {Transformers} for {High}-{Resolution} {Image} {Synthesis}.
\newblock {\em arXiv:2012.09841}, 2021.

[49]
Chris Finlay, Joern-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman.
\newblock How to {Train} {Your} {Neural} {ODE}: the {World} of {Jacobian} and
{Kinetic} {Regularization}.
\newblock In {\em ICML}, pages 3154--3164. PMLR, 2020.

[50]
Ruiqi Gao, Erik Nijkamp, Diederik~P. Kingma, Zhen Xu, Andrew~M. Dai, and
Ying~Nian Wu.
\newblock Flow {Contrastive} {Estimation} of {Energy}-{Based} {Models}.
\newblock In {\em IEEE CVPR}, pages 7518--7528, 2020.

[51]
Ruiqi Gao, Yang Song, Ben Poole, Ying~Nian Wu, and Diederik~P. Kingma.
\newblock Learning {Energy}-{Based} {Models} by {Diffusion} {Recovery}
\newblock In {\em ICLR}, 2021.

[52]
Partha Ghosh, Mehdi S.~M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard
\newblock From {Variational} to {Deterministic} {Autoencoders}.
\newblock In {\em ICLR}, 2020.

[53]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
\newblock Generative {Adversarial} {Nets}.
\newblock In Z.~Ghahramani, M.~Welling, C.~Cortes, N.~D. Lawrence, and K.~Q.
Weinberger, editors, {\em NeurIPS 27}, pages 2672--2680. Curran Associates,
Inc., 2014.

[54]
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael~J. Cree.
\newblock Regularisation of neural networks by enforcing {Lipschitz}
\newblock {\em Machine Learning}, 110(2):393--416, 2021.

[55]
Will Grathwohl, Ricky T.~Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David
\newblock {FFJORD}: {Free}-form {Continuous} {Dynamics} for {Scalable}
{Reversible} {Generative} {Models}.
\newblock In {\em ICLR}, 2019.

[56]
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, and
Richard Zemel.
\newblock Learning the {Stein} {Discrepancy} for {Training} and {Evaluating}
{Energy}-{Based} {Models} without {Sampling}.
\newblock In {\em ICML}, pages 3732--3747. PMLR, 2020.

[57]
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud,
Mohammad Norouzi, and Kevin Swersky.
\newblock Your {Classifier} is {Secretly} an {Energy} {Based} {Model} and {You}
{Should} {Treat} it {Like} {One}.
\newblock In {\em ICLR}, 2020.

[58]
Will~Sussman Grathwohl, Jacob~Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin
Swersky, and David Duvenaud.
\newblock No {MCMC} for me: {Amortized} sampling for fast and stable training
of energy-based models.
\newblock In {\em ICLR}, 2021.

[59]
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo~Jimenez Rezende, and Daan
\newblock {DRAW}: {A} {Recurrent} {Neural} {Network} {For} {Image}
\newblock In {\em ICML}, 2015.

[60]
Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye.
\newblock A {Review} on {Generative} {Adversarial} {Networks}: {Algorithms},
{Theory}, and {Applications}.
\newblock {\em arXiv:2001.06937}, 2020.

[61]
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron~C
\newblock Improved {Training} of {Wasserstein} {GANs}.
\newblock In I.~Guyon, U.~V. Luxburg, S.~Bengio, H.~Wallach, R.~Fergus,
S.~Vishwanathan, and R.~Garnett, editors, {\em NeurIPS 30}, pages 5767--5777.
Curran Associates, Inc., 2017.

[62]
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien~Ali Taiga, Francesco Visin,
David Vazquez, and Aaron Courville.
\newblock {PixelVAE}: {A} {Latent} {Variable} {Model} for {Natural} {Images}.
\newblock In {\em ICLR}, 2017.

[63]
Ishaan Gulrajani, Colin Raffel, and Luke Metz.
\newblock Towards {GAN} {Benchmarks} {Which} {Require} {Generalization}.
\newblock In {\em ICLR}, 2019.

[64]
Michael Gutmann and Aapo Hyv\"arinen.
\newblock Noise-{Contrastive} {Estimation}: {A} {New} {Estimation} {Principle}
for {Unnormalized} {Statistical} {Models}.
\newblock In {\em AISTATS}, pages 297--304. JMLR Workshop and Conference
Proceedings, 2010.

[65]
David Ha and Jürgen Schmidhuber.
\newblock World {Models}.
\newblock {\em arXiv:1803.10122}, 2018.

[66]
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.
\newblock Dream to {Control}: {Learning} {Behaviors} by {Latent} {Imagination}.
\newblock In {\em ICLR}, 2020.

[67]
Tian Han, Yang Lu, Song-Chun Zhu, and Ying~Nian Wu.
\newblock Alternating {Back}-{Propagation} for {Generator} {Network}.
\newblock {\em AAAI}, 31(1), 2017.

[68]
Leonard Hasenclever, Jakub~M Tomczak, and Max Welling.
\newblock Variational {Inference} with {Orthogonal} {Normalizing} {Flows}.
\newblock In {\em Workshop on {Bayesian} {Deep} {Learning}, {NIPS}}, 2017.

[69]
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick.
\newblock Lagging {Inference} {Networks} and {Posterior} {Collapse} in
{Variational} {Autoencoders}.
\newblock In {\em ICLR}, 2019.

[70]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep {Residual} {Learning} for {Image} {Recognition}.
\newblock In {\em IEEE CVPR}, pages 770--778, 2016.

[71]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
\newblock {GANs} {Trained} by a {Two} {Time}-{Scale} {Update} {Rule} {Converge}
to a {Local} {Nash} {Equilibrium}.
\newblock In I.~Guyon, U.~V. Luxburg, S.~Bengio, H.~Wallach, R.~Fergus,
S.~Vishwanathan, and R.~Garnett, editors, {\em NeurIPS 30}, pages 6626--6637.
Curran Associates, Inc., 2017.

[72]
G~E Hinton and T~J Sejnowski.
\newblock Optimal {Perceptual} {Inference}.
\newblock In {\em IEEE CVPR}, pages 448--453, 1983.

[73]
Geoffrey~E. Hinton.
\newblock Training {Products} of {Experts} by {Minimizing} {Contrastive}
\newblock {\em Neural Computation}, 14(8):1771--1800, 2002.

[74]
Geoffrey~E. Hinton, Simon Osindero, and Yee-Whye Teh.
\newblock A {Fast} {Learning} {Algorithm} for {Deep} {Belief} {Nets}.
\newblock {\em Neural Computation}, 18(7):1527--1554, 2006.

[75]
Jonathan Ho, Xi~Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel.
\newblock Flow++: {Improving} {Flow}-{Based} {Generative} {Models} with
{Variational} {Dequantization} and {Architecture} {Design}.
\newblock In {\em ICML}, 2019.

[76]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
\newblock Denoising {Diffusion} {Probabilistic} {Models}.
\newblock {\em arXiv:2006.11239}, 2020.

[77]
Sepp Hochreiter and Jürgen Schmidhuber.
\newblock Long {Short}-{Term} {Memory}.
\newblock {\em Neural Computation}, 9(8):1735--1780, 1997.

[78]
Matthew Hoffman, Pavel Sountsov, Joshua~V. Dillon, Ian Langmore, Dustin Tran,
and Srinivas Vasudevan.
\newblock {NeuTra}-lizing {Bad} {Geometry} in {Hamiltonian} {Monte} {Carlo}
{Using} {Neural} {Transport}.
\newblock {\em arXiv:1903.03704}, 2019.

[79]
Matthew~D Hoffman and Matthew~J Johnson.
\newblock {ELBO} surgery: yet another way to carve up the variational evidence
lower bound.
\newblock In {\em Workshop in {Advances} in {Approximate} {Bayesian}
{Inference}, {NIPS}}, page~4, 2016.

[80]
Emiel Hoogeboom, Rianne van~den Berg, and Max Welling.
\newblock Emerging {Convolutions} for {Generative} {Normalizing} {Flows}.
\newblock In {\em ICML}, pages 2771--2780, 2019.

[81]
Emiel Hoogeboom, Victor Garcia~Satorras, Jakub Tomczak, and Max Welling.
\newblock The {Convolution} {Exponential} and {Generalized} {Sylvester}
\newblock {\em NeurIPS}, 33, 2020.

[82]
Chin-Wei Huang, Laurent Dinh, and Aaron Courville.
\newblock Augmented {Normalizing} {Flows}: {Bridging} the {Gap} {Between}
{Generative} {Flows} and {Latent} {Variable} {Models}.
\newblock {\em arXiv:2002.07101}, 2020.

[83]
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville.
\newblock Neural {Autoregressive} {Flows}.
\newblock In {\em ICML}, 2018.

[84]
Chin-Wei Huang, Ahmed Touati, Laurent Dinh, Michal Drozdzal, Mohammad Havaei,
Laurent Charlin, and Aaron Courville.
\newblock Learnable {Explicit} {Density} for {Continuous} {Latent} {Space} and
{Variational} {Inference}.
\newblock {\em arXiv:1710.02248}, 2017.

[85]
Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, and Serge Belongie.
\newblock Stacked {Generative} {Adversarial} {Networks}.
\newblock In {\em IEEE CVPR}, pages 5077--5086, 2017.

[86]
Peter~J. Huber.
\newblock Robust {Estimation} of a {Location} {Parameter}.
\newblock In Samuel Kotz and Norman~L. Johnson, editors, {\em Breakthroughs in
{Statistics}: {Methodology} and {Distribution}}, Springer {Series} in
{Statistics}, pages 492--518. Springer, New York, NY, 1992.

[87]
Ferenc Huszár.
\newblock How (not) to {Train} your {Generative} {Model}: {Scheduled}
{Sampling}, {Likelihood}, {Adversary}?
\newblock {\em arXiv:1511.05101}, 2015.

[88]
Aapo Hyv\"arinen.
\newblock Estimation of {Non}-{Normalized} {Statistical} {Models} by {Score}
\newblock {\em JMLR}, 6:695--709, 2005.

[89]
Sergey Ioffe and Christian Szegedy.
\newblock Batch {Normalization}: {Accelerating} {Deep} {Network} {Training} by
{Reducing} {Internal} {Covariate} {Shift}.
\newblock {\em arXiv:1502.03167}, 2015.

[90]
Priyank Jaini, Kira~A. Selby, and Yaoliang Yu.
\newblock Sum-of-{Squares} {Polynomial} {Flow}.
\newblock In {\em ICML}, 2019.

[91]
Yifan Jiang, Shiyu Chang, and Zhangyang Wang.
\newblock {TransGAN}: {Two} {Transformers} {Can} {Make} {One} {Strong} {GAN}.
\newblock {\em arXiv:2102.07074}, 2021.

[92]
Alexia Jolicoeur-Martineau.
\newblock The relativistic discriminator: a key element missing from standard
\newblock {\em ICLR}, 2018.

[93]
Michael~I. Jordan, Zoubin Ghahramani, Tommi~S. Jaakkola, and Lawrence~K. Saul.
\newblock Introduction to variational methods for graphical models.
\newblock {\em Machine Learning}, 37(2):183--233, 1999.

[94]
Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford,
and Ilya Sutskever.
\newblock Distribution {Augmentation} for {Generative} {Modeling}.
\newblock In {\em ICML}, pages 5006--5019. PMLR, 2020.

[95]
Mahdi Karami, Dale Schuurmans, Jascha Sohl-Dickstein, Laurent Dinh, and Daniel
\newblock Invertible {Convolutional} {Flow}.
\newblock In {\em NeurIPS}, page~11, 2019.

[96]
Animesh Karnewar and Oliver Wang.
\newblock {MSG}-{GAN}: {Multi}-{Scale} {Gradients} for {Generative}
{Adversarial} {Networks}.
\newblock In {\em IEEE CVPR}, pages 7799--7808, 2020.

[97]
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
\newblock Progressive {Growing} of {GANs} for {Improved} {Quality},
{Stability}, and {Variation}.
\newblock In {\em ICLR}, 2018.

[98]
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and
Timo Aila.
\newblock Training {Generative} {Adversarial} {Networks} with {Limited} {Data}.
\newblock {\em NeurIPS}, 33, 2020.

[99]
Tero Karras, Samuli Laine, and Timo Aila.
\newblock A {Style}-{Based} {Generator} {Architecture} for {Generative}
{Adversarial} {Networks}.
\newblock In {\em IEEE CVPR}, pages 4401--4410, 2019.

[100]
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and
Timo Aila.
\newblock Analyzing and {Improving} the {Image} {Quality} of {StyleGAN}.
\newblock In {\em IEEE CVPR}, pages 8110--8119, 2020.

[101]
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.
\newblock Transformers are {RNNs}: {Fast} {Autoregressive} {Transformers} with
{Linear} {Attention}.
\newblock In {\em ICML}, pages 5156--5165. PMLR, 2020.

[102]
Hyunjik Kim, George Papamakarios, and Andriy Mnih.
\newblock The {Lipschitz} {Constant} of {Self}-{Attention}.
\newblock {\em arXiv:2006.04710}, 2020.

[103]
Taesup Kim and Yoshua Bengio.
\newblock Deep {Directed} {Generative} {Models} with {Energy}-{Based}
{Probability} {Estimation}.
\newblock {\em arXiv:1606.03439}, 2016.

[104]
Yoon Kim, Sam Wiseman, Andrew Miller, David Sontag, and Alexander Rush.
\newblock Semi-{Amortized} {Variational} {Autoencoders}.
\newblock In {\em ICML}, pages 2678--2687. PMLR, 2018.

[105]
Diederik~P. Kingma and Max Welling.
\newblock Auto-{Encoding} {Variational} {Bayes}.
\newblock {\em ICLR}, 2014.

[106]
Durk~P Kingma and Prafulla Dhariwal.
\newblock Glow: {Generative} {Flow} with {Invertible} 1x1 {Convolutions}.
\newblock In S.~Bengio, H.~Wallach, H.~Larochelle, K.~Grauman, N.~Cesa-Bianchi,
and R.~Garnett, editors, {\em NeurIPS 31}, pages 10215--10224. Curran
Associates, Inc., 2018.

[107]
Durk~P Kingma, Tim Salimans, Rafal Jozefowicz, Xi~Chen, Ilya Sutskever, and Max
\newblock Improved {Variational} {Inference} with {Inverse} {Autoregressive}
\newblock In D.~D. Lee, M.~Sugiyama, U.~V. Luxburg, I.~Guyon, and R.~Garnett,
editors, {\em NeurIPS 29}, pages 4743--4751. Curran Associates, Inc., 2016.

[108]
I.~Kobyzev, S.~Prince, and M.~Brubaker.
\newblock Normalizing {Flows}: {An} {Introduction} and {Review} of {Current}
\newblock {\em IEEE TPAMI}, pages 2008--2026, 2020.

[109]
Alex Krizhevsky.
\newblock Learning {Multiple} {Layers} of {Features} from {Tiny} {Images}.
\newblock 2009.

[110]
Kundan Kumar, Rithesh Kumar, Thibault de~Boissiere, Lucas Gestin, Wei~Zhen
Teoh, Jose Sotelo, Alexandre de~Br\'ebisson, Yoshua Bengio, and Aaron~C.
\newblock {MelGAN}: {Generative} {Adversarial} {Networks} for {Conditional}
{Waveform} {Synthesis}.
\newblock {\em NeurIPS}, 32, 2019.

[111]
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine,
Laurent Dinh, and Durk Kingma.
\newblock {VideoFlow}: {A} {Flow}-{Based} {Generative} {Model} for {Video}.
\newblock In {\em Workshop on {InvertibleNeural} {Nets} and {Normalizing}
{Flows}, {ICML}}, page~14, 2019.

[112]
Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua
\newblock Maximum {Entropy} {Generators} for {Energy}-{Based} {Models}.
\newblock {\em arXiv:1901.08508}, 2019.

[113]
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc’Aurelio Ranzato, and Fu~Jie
\newblock A {Tutorial} on {Energy}-{Based} {Learning}.
\newblock {\em A Tutorial on Energy-Based Learning}, 1(0):59, 2006.

[114]
Alex~X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and
Sergey Levine.
\newblock Stochastic {Adversarial} {Video} {Prediction}.
\newblock {\em arXiv:1804.01523}, 2018.

[115]
Zengyi Li, Yubei Chen, and Friedrich~T. Sommer.
\newblock Learning {Energy}-{Based} {Models} in {High}-{Dimensional} {Spaces}
with {Multi}-scale {Denoising} {Score} {Matching}.
\newblock {\em arXiv:1910.07762}, 2019.

[116]
Zengyi Li, Yubei Chen, and Friedrich~T. Sommer.
\newblock A {Neural} {Network} {MCMC} sampler that maximizes {Proposal}
\newblock {\em arXiv:2010.03587}, 2020.

[117]
Jae~Hyun Lim and Jong~Chul Ye.
\newblock Geometric {GAN}.
\newblock {\em arXiv:1705.02894}, 2017.

[118]
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal.
\newblock Towards {Faster} and {Stabilized} {GAN} {Training} for
{High}-fidelity {Few}-shot {Image} {Synthesis}.
\newblock In {\em ICLR}, 2021.

[119]
Kanglin Liu, Guoping Qiu, Wenming Tang, and Fei Zhou.
\newblock Spectral {Regularization} for {Combating} {Mode} {Collapse} in
\newblock In {\em IEEE ICCV}, pages 6381--6389, Seoul, Korea (South), 2019.

[120]
Xuanqing Liu and Cho-Jui Hsieh.
\newblock Rob-{GAN}: {Generator}, {Discriminator}, and {Adversarial}
\newblock In {\em IEEE CVPR}, pages 11234--11243, 2019.

[121]
Gabriel Loaiza-Ganem and John~P. Cunningham.
\newblock The continuous {Bernoulli}: fixing a pervasive error in variational
\newblock {\em NeurIPS}, 32:13287--13297, 2019.

[122]
Cheng Lu, Jianfei Chen, Chongxuan Li, Qiuhao Wang, and Jun Zhu.
\newblock Implicit {Normalizing} {Flows}.
\newblock In {\em ICLR}, 2021.

[123]
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier
\newblock Are {GANs} {Created} {Equal}? {A} {Large}-{Scale} {Study}.
\newblock In S.~Bengio, H.~Wallach, H.~Larochelle, K.~Grauman, N.~Cesa-Bianchi,
and R.~Garnett, editors, {\em NeurIPS 31}, pages 700--709. Curran Associates,
Inc., 2018.

[124]
Yucen Luo, Alex Beatson, Mohammad Norouzi, Jun Zhu, David Duvenaud, Ryan~P.
Adams, and Ricky T.~Q. Chen.
\newblock {SUMO}: {Unbiased} {Estimation} of {Log} {Marginal} {Probability} for
{Latent} {Variable} {Models}.
\newblock In {\em ICLR}, 2020.

[125]
Xuezhe Ma, Xiang Kong, Shanghang Zhang, and Eduard Hovy.
\newblock {MaCow}: {Masked} {Convolutional} {Generative} {Flow}.
\newblock In {\em NeurIPS}, page~10, 2019.

[126]
Xuezhe Ma, Chunting Zhou, and Eduard Hovy.
\newblock {MAE}: {Mutual} {Posterior}-{Divergence} {Regularization} for
{Variational} {AutoEncoders}.
\newblock In {\em ICLR}, 2019.

[127]
Lars Maaløe, Marco Fraccaro, Valentin Li\'evin, and Ole Winther.
\newblock {BIVA}: {A} {Very} {Deep} {Hierarchy} of {Latent} {Variables} for
{Generative} {Modeling}.
\newblock In {\em NeurIPS}, volume~33, page~12, 2019.

[128]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and
Adrian Vladu.
\newblock Towards {Deep} {Learning} {Models} {Resistant} to {Adversarial}
\newblock {\em arXiv:1706.06083}, 2019.

[129]
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan
\newblock Adversarial {Autoencoders}.
\newblock {\em arXiv:1511.05644}, 2016.

[130]
Xudong Mao, Qing Li, Haoran Xie, Raymond Y.~K. Lau, Zhen Wang, and Stephen
\newblock Least {Squares} {Generative} {Adversarial} {Networks}.
\newblock In {\em IEEE CVPR}, pages 2794--2802, 2017.

[131]
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain,
Jose Sotelo, Aaron Courville, and Yoshua Bengio.
\newblock {SampleRNN}: {An} {Unconditional} {End}-to-{End} {Neural} {Audio}
{Generation} {Model}.
\newblock In {\em ICLR}, 2017.

[132]
Chenlin Meng, Jiaming Song, Yang Song, Shengjia Zhao, and Stefano Ermon.
\newblock Improved {Autoregressive} {Modeling} with {Distribution} {Smoothing}.
\newblock In {\em ICLR}, 2021.

[133]
Chenlin Meng, Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon.
\newblock Autoregressive {Score} {Matching}.
\newblock In {\em NeurIPS}, volume~34, page~11, 2020.

[134]
Jacob Menick and Nal Kalchbrenner.
\newblock Generating {High} {Fidelity} {Images} with {Subscale} {Pixel}
{Networks} and {Multidimensional} {Upscaling}.
\newblock In {\em ICLR}, 2019.

[135]
Mehdi Mirza and Simon Osindero.
\newblock Conditional {Generative} {Adversarial} {Nets}.
\newblock {\em arXiv:1411.1784}, 2014.

[136]
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.
\newblock Spectral {Normalization} for {Generative} {Adversarial} {Networks}.
\newblock {\em ICLR}, 2018.

[137]
Thomas Müller, Brian Mcwilliams, Fabrice Rousselle, Markus Gross, and Jan
\newblock Neural {Importance} {Sampling}.
\newblock {\em ACM TOG}, 38(5):145:1--145:19, 2019.

[138]
Charlie Nash and Conor Durkan.
\newblock Autoregressive {Energy} {Machines}.
\newblock In {\em ICML}, pages 1735--1744. PMLR, 2019.

[139]
Kirill Neklyudov, Evgenii Egorov, and Dmitry~P. Vetrov.
\newblock The {Implicit} {Metropolis}-{Hastings} {Algorithm}.
\newblock {\em NeurIPS}, 32:13954--13964, 2019.

[140]
M.~Ngxande, J.~Tapamo, and M.~Burke.
\newblock {DepthwiseGANs}: {Fast} {Training} {Generative} {Adversarial}
{Networks} for {Realistic} {Image} {Synthesis}.
\newblock In {\em {SAUPEC}/{RobMech}/{PRASA}}, pages 111--116, 2019.

[141]
Alex Nichol and Prafulla Dhariwal.
\newblock Improved {Denoising} {Diffusion} {Probabilistic} {Models}.
\newblock {\em arXiv:2102.09672}, 2021.

[142]
Weili Nie, Nina Narodytska, and Ankit Patel.
\newblock {RelGAN}: {Relational} {Generative} {Adversarial} {Networks} for
{Text} {Generation}.
\newblock In {\em ICLR}, 2019.

[143]
Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling.
\newblock {SurVAE} {Flows}: {Surjections} to {Bridge} the {Gap} between {VAEs}
and {Flows}.
\newblock {\em NeurIPS}, 33, 2020.

[144]
Erik Nijkamp, Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo~Pang, Song-Chun
Zhu, and Ying~Nian Wu.
\newblock Learning {Energy}-based {Model} with {Flow}-based {Backbone} by
{Neural} {Transport} {MCMC}.
\newblock {\em arXiv:2006.06897}, 2020.

[145]
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying~Nian Wu.
\newblock On the {Anatomy} of {MCMC}-{Based} {Maximum} {Likelihood} {Learning}
of {Energy}-{Based} {Models}.
\newblock In {\em {arXiv}:1903.12370}, 2020.

[146]
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying~Nian Wu.
\newblock Learning {Non}-{Convergent} {Non}-{Persistent} {Short}-{Run} {MCMC}
{Toward} {Energy}-{Based} {Model}.
\newblock In H.~Wallach, H.~Larochelle, A.~Beygelzimer,
F.~d{\textbackslash}textquotesingle Alch\'e-Buc, E.~Fox, and R.~Garnett,
editors, {\em NeurIPS 32}, pages 5232--5242. Curran Associates, Inc., 2019.

[147]
Erik Nijkamp, Bo~Pang, Tian Han, Linqi Zhou, Song-Chun Zhu, and Ying~Nian Wu.
\newblock Learning {Multi}-layer {Latent} {Variable} {Model} via {Variational}
{Optimization} of {Short} {Run} {MCMC} for {Approximate} {Inference}.
\newblock In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm,
editors, {\em ECCV}, Lecture {Notes} in {Computer} {Science}, pages 361--378,
Cham, 2020. Springer International Publishing.

[148]
Sajad Norouzi, David~J. Fleet, and Mohammad Norouzi.
\newblock Exemplar {VAE}: {Linking} {Generative} {Models}, {Nearest} {Neighbor}
{Retrieval}, and {Data} {Augmentation}.
\newblock {\em NeurIPS}, 33, 2020.

[149]
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka.
\newblock f-{GAN}: {Training} {Generative} {Neural} {Samplers} using
{Variational} {Divergence} {Minimization}.
\newblock In D.~D. Lee, M.~Sugiyama, U.~V. Luxburg, I.~Guyon, and R.~Garnett,
editors, {\em NeurIPS 29}, pages 271--279. Curran Associates, Inc., 2016.

[150]
Augustus Odena, Christopher Olah, and Jonathon Shlens.
\newblock Conditional {Image} {Synthesis} with {Auxiliary} {Classifier} {GANs}.
\newblock In {\em ICML}, pages 2642--2651. PMLR, 2017.

[151]
Derek Onken, Samy~Wu Fung, Xingjian Li, and Lars Ruthotto.
\newblock {OT}-{Flow}: {Fast} and {Accurate} {Continuous} {Normalizing} {Flows}
via {Optimal} {Transport}.
\newblock In {\em AAAI}, 2021.

[152]
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray
Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg,
Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen,
Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov,
and Demis Hassabis.
\newblock Parallel {WaveNet}: {Fast} {High}-{Fidelity} {Speech} {Synthesis}.
\newblock In {\em ICML}, pages 3918--3926, 2018.

[153]
Aaron van~den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals,
Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.
\newblock {WaveNet}: {A} {Generative} {Model} for {Raw} {Audio}.
\newblock {\em arXiv:1609.03499}, 2016.

[154]
Georg Ostrovski, Will Dabney, and R\'emi Munos.
\newblock Autoregressive {Quantile} {Networks} for {Generative} {Modeling}.
\newblock {\em ICML}, 9:6293--6308, 2018.

[155]
A.~Oussidi and A.~Elhassouny.
\newblock Deep {Generative} {Models}: {Survey}.
\newblock In {\em IEEE ISCV}, pages 1--8, 2018.

[156]
Bo~Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying~Nian Wu.
\newblock Learning {Latent} {Space} {Energy}-{Based} {Prior} {Model}.
\newblock In {\em NeurIPS}, volume~34, page~15, 2020.

[157]
George Papamakarios, Eric Nalisnick, Danilo~Jimenez Rezende, Shakir Mohamed,
and Balaji Lakshminarayanan.
\newblock Normalizing {Flows} for {Probabilistic} {Modeling} and {Inference}.
\newblock {\em arXiv:1912.02762}, 2019.

[158]
George Papamakarios, Theo Pavlakou, and Iain Murray.
\newblock Masked {Autoregressive} {Flow} for {Density} {Estimation}.
\newblock In I.~Guyon, U.~V. Luxburg, S.~Bengio, H.~Wallach, R.~Fergus,
S.~Vishwanathan, and R.~Garnett, editors, {\em NeurIPS 30}, pages 2338--2347.
Curran Associates, Inc., 2017.

[159]
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer,
Alexander Ku, and Dustin Tran.
\newblock Image {Transformer}.
\newblock In {\em ICML}, pages 4055--4064. PMLR, 2018.

[160]
L.~S. Pontryagin.
\newblock {\em Mathematical {Theory} of {Optimal} {Processes}}.
\newblock Routledge, 2018.

[161]
R.~Prenger, R.~Valle, and B.~Catanzaro.
\newblock {WaveGlow}: {A} {Flow}-based {Generative} {Network} for {Speech}
\newblock In {\em IEEE ICASSP}, pages 3617--3621, 2019.

[162]
Alec Radford, Luke Metz, and Soumith Chintala.
\newblock Unsupervised {Representation} {Learning} with {Deep} {Convolutional}
{Generative} {Adversarial} {Networks}.
\newblock In {\em ICLR}, 2016.

[163]
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec
Radford, Mark Chen, and Ilya Sutskever.
\newblock Zero-{Shot} {Text}-to-{Image} {Generation}.
\newblock {\em arXiv:2102.12092}, 2021.

[164]
Ali Razavi, Aaron van~den Oord, and Oriol Vinyals.
\newblock Generating {Diverse} {High}-{Fidelity} {Images} with {VQ}-{VAE}-2.
\newblock In H.~Wallach, H.~Larochelle, A.~Beygelzimer,
F.~d{\textbackslash}textquotesingle Alch\'e-Buc, E.~Fox, and R.~Garnett,
editors, {\em NeurIPS 32}, pages 14866--14876. Curran Associates, Inc., 2019.

[165]
Scott Reed, A\"aron Oord, Nal Kalchbrenner, Sergio~Gómez Colmenarejo, Ziyu
Wang, Yutian Chen, Dan Belov, and Nando Freitas.
\newblock Parallel {Multiscale} {Autoregressive} {Density} {Estimation}.
\newblock In {\em ICML}, pages 2912--2921. PMLR, 2017.

[166]
Danilo~Jimenez Rezende and Shakir Mohamed.
\newblock Variational {Inference} with {Normalizing} {Flows}.
\newblock {\em ICML}, 2:1530--1538, 2015.

[167]
Gareth~O. Roberts and Richard~L. Tweedie.
\newblock Exponential convergence of {Langevin} distributions and their
discrete approximations.
\newblock {\em Bernoulli}, 2(4):341--363, 1996.

[168]
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and
\newblock Improved {Techniques} for {Training} {GANs}.
\newblock In {\em NeurIPS}, pages 2234--2242, 2016.

[169]
Tim Salimans, Andrej Karpathy, Xi~Chen, and Diederik~P. Kingma.
\newblock {PixelCNN}++: {Improving} the {PixelCNN} with {Discretized}
{Logistic} {Mixture} {Likelihood} and {Other} {Modifications}.
\newblock {\em 5th International Conference on Learning Representations, ICLR
2017 - Conference Track Proceedings}, 2017.

[170]
Tim Salimans, Diederik Kingma, and Max Welling.
\newblock Markov {Chain} {Monte} {Carlo} and {Variational} {Inference}:
{Bridging} the {Gap}.
\newblock In {\em ICML}, pages 1218--1226. PMLR, 2015.

[171]
Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyv\"arinen.
\newblock Deep {Energy} {Estimator} {Networks}.
\newblock {\em arXiv:1805.08306}, 2018.

[172]
Jiaxin Shi, Shengyang Sun, and Jun Zhu.
\newblock A {Spectral} {Approach} to {Gradient} {Estimation} for {Implicit}
\newblock In {\em ICML}, pages 4644--4653. PMLR, 2018.

[173]
Samarth Sinha, Han Zhang, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and
Augustus Odena.
\newblock Small-{GAN}: {Speeding} up {GAN} {Training} using {Core}-{Sets}.
\newblock In {\em ICML}, pages 9005--9015. PMLR, 2020.

[174]
Vincent Sitzmann, Julien N.~P. Martel, Alexander~W. Bergman, David~B. Lindell,
and Gordon Wetzstein.
\newblock Implicit {Neural} {Representations} with {Periodic} {Activation}
\newblock {\em arXiv:2006.09661}, 2020.

[175]
Jascha Sohl-Dickstein, Eric~A. Weiss, Niru Maheswaranathan, and Surya Ganguli.
\newblock Deep {Unsupervised} {Learning} using {Nonequilibrium}
\newblock In {\em nternational {Conference} on {Machine} {Learning}}, pages
2256--2265, 2015.

[176]
Jiaming Song, Chenlin Meng, and Stefano Ermon.
\newblock Denoising {Diffusion} {Implicit} {Models}.
\newblock In {\em ICLR}, 2021.

[177]
Jiaming Song, Shengjia Zhao, and Stefano Ermon.
\newblock A-{NICE}-{MC}: {Adversarial} {Training} for {MCMC}.
\newblock In {\em NeurIPS}, volume~30, pages 5140--5150, 2017.

[178]
Yang Song and Stefano Ermon.
\newblock Generative {Modeling} by {Estimating} {Gradients} of the {Data}
\newblock In H.~Wallach, H.~Larochelle, A.~Beygelzimer,
F.~d{\textbackslash}textquotesingle Alch\'e-Buc, E.~Fox, and R.~Garnett,
editors, {\em NeurIPS 32}, pages 11918--11930. Curran Associates, Inc., 2019.

[179]
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon.
\newblock Sliced {Score} {Matching}: {A} {Scalable} {Approach} to {Density} and
{Score} {Estimation}.
\newblock In {\em UAI}, pages 574--584. PMLR, 2020.

[180]
Yang Song and Diederik~P. Kingma.
\newblock How to {Train} {Your} {Energy}-{Based} {Models}.
\newblock {\em arXiv:2101.03288}, 2021.

[181]
Yang Song, Chenlin Meng, and Stefano Ermon.
\newblock {MintNet}: {Building} {Invertible} {Neural} {Networks} with {Masked}
\newblock {\em NeurIPS}, 32:11004--11014, 2019.

[182]
Yang Song, Jascha Sohl-Dickstein, Diederik~P. Kingma, Abhishek Kumar, Stefano
Ermon, and Ben Poole.
\newblock Score-{Based} {Generative} {Modeling} through {Stochastic}
{Differential} {Equations}.
\newblock In {\em ICLR}, 2021.

[183]
Yuxuan Song, Qiwei Ye, Minkai Xu, and Tie-Yan Liu.
\newblock Discriminator {Contrastive} {Divergence}: {Semi}-{Amortized}
{Generative} {Modeling} by {Exploring} {Energy} of the {Discriminator}.
\newblock {\em arXiv:2004.01704}, 2020.

[184]
Ilya Sutskever and Tijmen Tieleman.
\newblock On the {Convergence} {Properties} of {Contrastive} {Divergence}.
\newblock In {\em AISTATS}, pages 789--795, 2010.

[185]
Kevin Swersky, Marc’Aurelio Ranzato, David Buchman, Benjamin~M Marlin, and
Nando de~Freitas.
\newblock On {Autoencoders} and {Score} {Matching} for {Energy} {Based}
\newblock In {\em ICML}, 2011.

[186]
Casper~Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc
\newblock Amortised {MAP} {Inference} for {Image} {Super}-resolution.
\newblock In {\em ICLR}, 2017.

[187]
Casper~Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren~Kaae Sønderby, and
Ole Winther.
\newblock Ladder {Variational} {Autoencoders}.
\newblock {\em NeurIPS}, 29:3738--3746, 2016.

[188]
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin
Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.
\newblock Fourier {Features} {Let} {Networks} {Learn} {High} {Frequency}
{Functions} in {Low} {Dimensional} {Domains}.
\newblock {\em NeurIPS}, 33:7537--7547, 2020.

[189]
H.~Thanh-Tung and T.~Tran.
\newblock Catastrophic forgetting and mode collapse in {GANs}.
\newblock In {\em IJCNN}, pages 1--10, 2020.

[190]
Lucas Theis, A\"aron van~den Oord, and Matthias Bethge.
\newblock A note on the evaluation of generative models.
\newblock {\em arXiv:1511.01844}, 2016.

[191]
Michalis Titsias and Petros Dellaportas.
\newblock Gradient-based {Adaptive} {Markov} {Chain} {Monte} {Carlo}.
\newblock In {\em NeurIPS}, volume~32, pages 15730--15739, 2019.

[192]
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf.
\newblock Wasserstein {Auto}-{Encoders}.
\newblock {\em arXiv:1711.01558}, 2019.

[193]
Jakub Tomczak and Max Welling.
\newblock {VAE} with a {VampPrior}.
\newblock In {\em AISTATS}, pages 1214--1223. PMLR, 2018.

[194]
Dustin Tran, Rajesh Ranganath, and David~M. Blei.
\newblock Variational {Gaussian} {Process}.
\newblock In {\em ICLR}, 2016.

[195]
Dustin Tran, Keyon Vafa, Kumar Agrawal, Laurent Dinh, and Ben Poole.
\newblock Discrete {Flows}: {Invertible} {Generative} {Models} of {Discrete}
\newblock In H.~Wallach, H.~Larochelle, A.~Beygelzimer,
F.~d{\textbackslash}textquotesingle Alch\'e-Buc, E.~Fox, and R.~Garnett,
editors, {\em NeurIPS 32}, pages 14719--14728. Curran Associates, Inc., 2019.

[196]
N.-T. Tran, V.-H. Tran, N.-B. Nguyen, T.-K. Nguyen, and N.-M. Cheung.
\newblock On {Data} {Augmentation} for {GAN} {Training}.
\newblock {\em IEEE TIP}, 30:1882--1897, 2021.

[197]
Ngoc-Trung Tran, Viet-Hung Tran, Bao-Ngoc Nguyen, Linxiao Yang, and
Ngai-Man~(Man) Cheung.
\newblock Self-supervised {GAN}: {Analysis} and {Improvement} with
{Multi}-class {Minimax} {Game}.
\newblock {\em NeurIPS}, 32:13253--13264, 2019.

[198]
Richard~Eric Turner and Maneesh Sahani.
\newblock Two problems with variational expectation maximisation for time
series models.
\newblock In David Barber, A.~Taylan Cemgil, and Silvia Chiappa, editors, {\em
Bayesian {Time} {Series} {Models}}, pages 104--124. Cambridge University
Press, Cambridge, 2011.

[199]
Ryan Turner, Jane Hung, Eric Frank, Yunus Saatchi, and Jason Yosinski.
\newblock Metropolis-{Hastings} {Generative} {Adversarial} {Networks}.
\newblock In {\em ICML}, pages 6345--6353. PMLR, 2019.

[200]
Belinda Tzen and Maxim Raginsky.
\newblock Neural {Stochastic} {Differential} {Equations}: {Deep} {Latent}
{Gaussian} {Models} in the {Diffusion} {Limit}.
\newblock {\em arXiv:1905.09883}, 2019.

[201]
Belinda Tzen and Maxim Raginsky.
\newblock Theoretical guarantees for sampling and inference in generative
models with latent diffusions.
\newblock In {\em Conference on {Learning} {Theory}}, pages 3084--3114. PMLR,

[202]
Arash Vahdat and Jan Kautz.
\newblock {NVAE}: {A} {Deep} {Hierarchical} {Variational} {Autoencoder}.
\newblock {\em NeurIPS}, 33, 2020.

[203]
Aaron van~den Oord, Nal Kalchbrenner, Lasse Espeholt, koray kavukcuoglu, Oriol
Vinyals, and Alex Graves.
\newblock Conditional {Image} {Generation} with {PixelCNN} {Decoders}.
\newblock In D.~D. Lee, M.~Sugiyama, U.~V. Luxburg, I.~Guyon, and R.~Garnett,
editors, {\em NeurIPS 29}, pages 4790--4798. Curran Associates, Inc., 2016.

[204]
A\"aron Van Den~Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.
\newblock Pixel {Recurrent} {Neural} {Networks}.
\newblock In {\em ICML}, volume~4, pages 2611--2620. International Machine
Learning Society (IMLS), 2016.

[205]
Aaron van~den Oord, Oriol Vinyals, and koray kavukcuoglu.
\newblock Neural {Discrete} {Representation} {Learning}.
\newblock In I.~Guyon, U.~V. Luxburg, S.~Bengio, H.~Wallach, R.~Fergus,
S.~Vishwanathan, and R.~Garnett, editors, {\em NeurIPS 30}, pages 6306--6315.
Curran Associates, Inc., 2017.

[206]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, Łukasz Kaiser, and Illia Polosukhin.
\newblock Attention is {All} you {Need}.
\newblock In I.~Guyon, U.~V. Luxburg, S.~Bengio, H.~Wallach, R.~Fergus,
S.~Vishwanathan, and R.~Garnett, editors, {\em NeurIPS 30}, pages 5998--6008.
Curran Associates, Inc., 2017.

[207]
C\'edric Villani.
\newblock {\em Topics in {Optimal} {Transportation}}.
\newblock Number~58. American Mathematical Soc., 2003.

[208]
C\'edric Villani.
\newblock {\em Optimal {Transport}: {Old} and {New}}, volume 338.
\newblock Springer Science \& Business Media, 2008.

[209]
Pascal Vincent.
\newblock A {Connection} {Between} {Score} {Matching} and {Denoising}
\newblock {\em Neural Computation}, 23(7):1661--1674, 2011.

[210]
Sinong Wang, Belinda~Z. Li, Madian Khabsa, Han Fang, and Hao Ma.
\newblock Linformer: {Self}-{Attention} with {Linear} {Complexity}.
\newblock {\em arXiv:2006.04768}, 2020.

[211]
Yuanhao Wang, Guodong Zhang, and Jimmy Ba.
\newblock On {Solving} {Minimax} {Optimization} {Locally}: {A}
{Follow}-the-{Ridge} {Approach}.
\newblock In {\em ICLR}, 2020.

[212]
Antoine Wehenkel and Gilles Louppe.
\newblock Unconstrained {Monotonic} {Neural} {Networks}.
\newblock {\em NeurIPS}, 32:1545--1555, 2019.

[213]
Max Welling and Geoffrey~E. Hinton.
\newblock A {New} {Learning} {Algorithm} for {Mean} {Field} {Boltzmann}
\newblock In Gerhard Goos, Juris Hartmanis, Jan van Leeuwen, and Jos\'e~R.
Dorronsoro, editors, {\em ICANN}, volume 2415, pages 351--357. Springer
Berlin Heidelberg, Berlin, Heidelberg, 2002.

[214]
Max Welling and Yee~Whye Teh.
\newblock Bayesian {Learning} via {Stochastic} {Gradient} {Langevin}
\newblock In {\em ICML}, pages 681--688, 2011.

[215]
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat.
\newblock {VAEBM}: {A} {Symbiosis} between {Variational} {Autoencoders} and
{Energy}-based {Models}.
\newblock In {\em ICLR}, 2021.

[216]
J.~Xie, Y.~Lu, R.~Gao, S.~Zhu, and Y.~N. Wu.
\newblock Cooperative {Training} of {Descriptor} and {Generator} {Networks}.
\newblock {\em IEEE TPAMI}, 42(1):27--45, 2020.

[217]
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Ying~Nian Wu.
\newblock A {Theory} of {Generative} {ConvNet}.
\newblock In {\em ICML}, pages 2635--2644, 2016.

[218]
L.~Yang and G.~E. Karniadakis.
\newblock Potential {Flow} {Generator} {With} {L2} {Optimal} {Transport}
{Regularity} for {Generative} {Models}.
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems},
pages 1--11, 2020.

[219]
Xin Yi, Ekta Walia, and Paul Babyn.
\newblock Generative {Adversarial} {Network} in {Medical} {Imaging}: {A}
\newblock {\em Medical Image Analysis}, 58:101552, 2019.

[220]
Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon.
\newblock Training {Deep} {Energy}-{Based} {Models} with f-{Divergence}
\newblock In {\em ICML}, pages 10957--10967. PMLR, 2020.

[221]
Sergey Zagoruyko and Nikos Komodakis.
\newblock Wide {Residual} {Networks}.
\newblock {\em arXiv:1605.07146}, 2017.

[222]
Han Zhang, Xi~Gao, Jacob Unterman, and Tom Arodz.
\newblock Approximation {Capabilities} of {Neural} {Ordinary} {Diﬀerential}
\newblock {\em arXiv:1907.12998}, page~11, 2019.

[223]
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.
\newblock Self-{Attention} {Generative} {Adversarial} {Networks}.
\newblock {\em arXiv:1805.08318}, 2019.

[224]
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang,
and Dimitris~N. Metaxas.
\newblock {StackGAN}: {Text} to {Photo}-{Realistic} {Image} {Synthesis} {With}
{Stacked} {Generative} {Adversarial} {Networks}.
\newblock In {\em IEEE CVPR}, pages 5907--5915, 2017.

[225]
Linfeng Zhang, Weinan E, and Lei Wang.
\newblock Monge-{Amp\`ere} {Flow} for {Generative} {Modeling}.
\newblock {\em arXiv:1809.10188}, 2018.

[226]
Junbo Zhao, Michael Mathieu, and Yann LeCun.
\newblock Energy-based {Generative} {Adversarial} {Network}.
\newblock In {\em ICLR}, 2017.

[227]
Shengjia Zhao, Jiaming Song, and Stefano Ermon.
\newblock Learning {Hierarchical} {Features} from {Deep} {Generative} {Models}.
\newblock In {\em ICML}, 2017.

[228]
Shengjia Zhao, Jiaming Song, and Stefano Ermon.
\newblock Towards {Deeper} {Understanding} of {Variational} {Autoencoding}
\newblock {\em arXiv:1702.08658}, 2017.

[229]
Shengjia Zhao, Jiaming Song, and Stefano Ermon.
\newblock {InfoVAE}: {Balancing} {Learning} and {Inference} in {Variational}
\newblock {\em AAAI}, 33(01):5885--5892, 2019.

[230]
Shengyu Zhao, Zhijian Liu, Ji~Lin, Jun-Yan Zhu, and Song Han.
\newblock Differentiable {Augmentation} for {Data}-{Efficient} {GAN}
\newblock {\em NeurIPS}, 33, 2020.

[231]
Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang.
\newblock Image {Augmentations} for {GAN} {Training}.
\newblock {\em arXiv:2006.02595}, 2020.

[232]
Jiachen Zhong, Xuanqing Liu, and Cho-Jui Hsieh.
\newblock Improving the {Speed} and {Quality} of {GAN} by {Adversarial}
\newblock {\em arXiv:2008.03364}, 2020.

[233]
Zachary~M. Ziegler and Alexander~M. Rush.
\newblock Latent {Normalizing} {Flows} for {Discrete} {Sequences}.
\newblock In {\em ICML}, pages 7673--7682, 2019.

\end{thebibliography}

% biography section
% If you have an EPS/PDF photo (graphicx package needed) extra braces are
% needed around the contents of the optional argument to biography to prevent
% the LaTeX parser from getting confused when it sees the complicated
% \includegraphics command within an optional argument. (You could create
% your own custom macro containing the \includegraphics command to make things
% simpler here.)
%\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{mshell}}]{Michael Shell}
% or if you just want to reserve a space for a photo:

% \begin{IEEEbiography}{Sam Bond-Taylor}
% Biography text here.
% \end{IEEEbiography}

% if you will not have a photo at all:
%\begin{IEEEbiographynophoto}{Yang Long}
%Biography text here.

% insert where needed to balance the two columns on the last page with
% biographies

% \begin{IEEEbiography}{Adam Leach}
% Biography text here.
% \end{IEEEbiography}

% \begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{imgs/bios/Long_Yang.jpg}}]{Yang Long}
% is an assistant professor in the department of computer science, Durham university. He is also an MRC innovation fellow aiming to design scalable AI solutions for large-scale healthcare applications. His research background is in the highly interdisciplinary field of computer vision and machine learning.
% % While he is passionate about unveiling the black-box of AI brain and transferring the knowledge to seek Scalable, Interactable, Interpretable, and sustainable solutions for other disciplinary researches, e.g. physical activity, mental health, design, education, security, and geoengineering.
% He has authored/co-authored 20+ top-tier papers in refereed journals/conferences such as IEEE TPAMI, TIP, CVPR, AAAI, and ACM MM.
% \end{IEEEbiography}
% %\vfill

% \begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{imgs/bios/Willcocks_Christopher.jpg}}]{Chris G. Willcocks}
% is an assistant professor in computer science at Durham university, where his interdisciplinary research focuses on generative models, medical image computing, computational biophysics and machine reasoning. He teaches deep learning, reinforcement learning and cyber security, and regularly publishes journals/conferences in venues such as ICLR, PRX, IEEE TPAMI, TMI and TIFS.
% \end{IEEEbiography}

% You can push biographies down or up by placing
% a \vfill before or after them. The appropriate
% use of \vfill depends on what kind of text is
% on the last page and whether or not the columns
% are being equalized.

% Can be used to pull up biographies so that the bottom of the last one
% is flush with the other column.

% that's all folks
\end{document}