# Concurrent Density Estimation with Wasserstein Autoencoders: Some
Statistical Insights

Anish Chakrabarty  Arkaprabha Basu  Swagatam Das

###### Abstract

Variational Autoencoders (VAEs) have been a pioneering force in the realm of
deep generative models. Amongst its legions of progenies, Wasserstein
Autoencoders (WAEs) stand out in particular due to the dual offering of
heightened generative quality and a strong theoretical backbone. WAEs consist
of an encoding and a decoding network— forming a bottleneck— with the prime
objective of generating new samples resembling the ones it was catered to. In
the process, they aim to achieve a target latent representation of the encoded
data. Our work is an attempt to offer a theoretical understanding of the
machinery behind WAEs. From a statistical viewpoint, we pose the problem as
concurrent density estimation tasks based on neural network-induced
transformations. This allows us to establish deterministic upper bounds on the
realized errors WAEs commit. We also analyze the propagation of these
stochastic errors in the presence of adversaries. As a result, both the large
sample properties of the reconstructed distribution and the resilience of WAE
models are explored.

††journal: Nuclear Physics B

[inst1]organization=Statistics and Mathematics Unit,addressline=Indian
Statistical Institute, Kolkata

[inst2]organization=Electronics and Communication Sciences
Unit,addressline=Indian Statistical Institute, Kolkata

##  1 Introduction

Variational Autoencoder (VAE) [1] is one of the earlier agents of the modern-
day revolution we call deep generative modeling. Vanilla autoencoders (AE), a
precursor to VAEs, being used primarily for representation learning, were
incapable of adding variation to the reconstructed signal. As such, they could
not ‘generate’ new observations resembling the target. VAEs came into being
with the promise of overcoming this limitation, inspiring numerous variants in
the process [2]. Perhaps the one that stirs up a statistician’s intrigue the
most is the Wasserstein Autoencoder (WAE) [3]. Approaching the problem from an
optimal transport (OT) point of view, it achieved significant improvement in
generated image quality.

The discussion regarding deep generative models starts with an unknown target
probability distribution $\mu$. A model is devised to simulate new
observations from the same by learning it gradually based on samples. While it
is demanding to imagine a data set consisting of images following such a
distribution, they can be readily deemed as residents of a high-dimensional
non-Euclidean space, perhaps manifolds. However, in our discussion, we surmise
that $\mu$ is defined on a Borel subset $\mathcal{X}$ of $\mathbb{R}^{d}$.
This becomes a reasonable starting point for our discussion based on the well-
known fact that the information necessary to ‘represent’ an image typically
possesses a low-dimensional structure compared to its ambient dimension $d$
[4]. There lie two constituents in a typical WAE model: an ‘encoder’ ($E$),
and a ‘decoder’ ($D$). It is the encoder that explores the prospect of
achieving a low-dimensional representation of the data. Sampled observations
from $\mu$ are fed into the encoder, which is tasked with producing replicates
of such a reduced dimension. As such, it may be viewed as a parametric class
of Borel functions from $\mathcal{X}$ to the ‘latent space’
$\mathcal{Z}\subseteq\mathbb{R}^{k}$, $d>k$. In practice, both encoders and
decoders are parameterized by neural networks (NNs). The goal of the encoding
exercise is to reach a desired distribution $\rho$ defined on this space,
fittingly called the ‘latent law’. Evidently, there must remain some
discrepancy between the encoded and the desired latent distributions.
Tolstikhin et al. [3] prescribe the usage of Jensen-Shannon divergence (JS)
and Maximum Mean Discrepancy (MMD) to encapsulate this ‘latent loss’. This
quantity makes a major contribution toward the overall objective that drives
WAEs. It is also the target latent law that inspires smooth interpolation
between modes of $\mu$ while generating new observations.

Once the encoding is over, reconstruction must take place. Decoders can be
similarly described as the class of functions (from
$\mathcal{Z}\rightarrow\mathcal{X}$) that aim at inducing inverse maps to
those brought in by the encoders. Encoded observations go through such a
transformation in an attempt to get back to where they originally came from,
$\mu$. The deviation of the regenerated distribution from the input law makes
for the reconstruction error. Needless to say, in a WAE model, this loss is
represented by the Wasserstein distance (WD).

Before laying the groundwork for a comprehensive statistical analysis of WAEs,
one must acknowledge the accruing wisdom that has led us to this point.

###  1.1 On Related Literature

Chasing after the remarkable empirical success, theoretical explanations
corresponding to deep generative models have come a long way. Riding the late
surge are Generative Adversarial Networks (GAN) [5, 6, 7, 8] and their
immediate descendants (e.g. WGAN [9], Bidirectional GAN [10] etc.). Compared
to such a sensation, VAEs seem underappreciated when it comes to statistical
scrutiny of their machinery. Though philosophically distinctive, some effort
has been put into establishing an equivalence between GANs and autoencoder-
based models. Borrowing from GANs the adversarial behavior of partaking
network components, Adversarial AE (AAE) [11] was conceived. The introduction
of adversarial training into VAEs saw them expressing arbitrary class of
latent laws by posing the posterior maximum likelihood estimation as a two-
player game [12]. In spirit, this made VAEs at par with GANs. The resemblance
between the two architectures became even more cogent when seen through the
lens of variational inference [13]. However, the first evidence of a VAE-
variant with comparable generative performance to that of GANs came in the
form of WAE [3]. Husain et al. [14] supported this empirical similitude
theoretically by showing a primal-dual relationship between the two
objectives. This fact motivates us to transport the cumulative knowledge from
statistical inquiries regarding GANs to WAEs.

The scarcity of rigorous statistical studies corresponding to WAEs makes the
existing ones even more precious. It is well known by now that a VAE model
with Gaussian decoders behaves similarly to Robust PCA [15] under mild
assumptions. As a result, such VAEs are capable of recovering uncorrupted
observations hailing from input data manifolds, fending off outliers [16].
However, an agency of WAE architectures towards robust reconstruction lies
unchecked. The Gaussian assumptions on both the encoder and decoder networks
also have a profound impression on the VAE’s capability to reconstruct the
input law. Dai and Wipf [17] show that in case the data manifold has the full
ambient dimension, reaching the global minima of the VAE loss is equivalent to
ensuring a successful recovery. However, for image data, where the
observations typically have a lower-dimensional true representation, non-
unique solutions may exist. Similar avenues for WAEs still await to be
explored. In an earlier work [18], we set out to answer some of such
questions. The paper reformulated the WAE objective as simultaneous density
estimation tasks, a viewpoint adopted previously by statistical analyses of
GANs. In this work, we intend to build on top of the groundwork.

####  1.1.1 A Note on the Latent Dimension

Practitioners and theorists are often divided based on their perceptions.
However, the problem that unites them in shared discomfort is the precise
prescription of the latent dimension $k$. The question remains simple: ‘Given
a set of samples from a distribution, what should be the extent of
dimensionality reduction (DR) such that they can be reconstructed?’. In
generative exercises, however, the data distribution lies unknown, unlike the
ambient dimension of its support. Clearly, the answer should ideally be
multifaceted. There lie several aspects, heavily intertwined, that contribute
to the complexity.

The first hint comes from the input data dimension itself. Signals from
naturally occurring events are mere instantiations of underlying random
processes. Variability in a set of observations is rooted in this very idea.
The explanatory attributes and their corresponding directions encapsulate this
variation, giving rise to the notion of ‘dimensionality’ of the data. As
characterized by [19], this quantity is formally known as the embedding
(ambient) dimension [20]. However, dealing with high-dimensional real datasets
(e.g. images) we have come to observe that such a space tends to have a lower-
dimensional structure (typically submanifolds $\mathcal{M}$) where most of the
variation lies, with a high probability [21]. The smaller set of directions
the ‘Manifold Hypothesis’ points at is called the intrinsic dimension (ID).
While there is significant disagreement between authors regarding the exact
definition of the same (e.g. Minkowski dimension), we recognize ID as the
topological dimension of $\mathcal{M}$. Several attempts have been made to
estimate its ID given the data distribution [22, 23, 24]. We emphasize the
importance of such an intrinsic pattern of the signal to reflect on the
encoded law as well. If one goes by the notion of ID being the set of
independent dimensions that capture most of the variation in the dataset, $k$
should be a near-estimate of it.

Since WAEs are restricted to reconstructing input observations, they must
preserve as much information as possible while encoding. In our density
estimation regime, the notion of ‘information’ is somewhat different from that
offered by geometry. While the responsibility to preserve local and broader
geometric signatures (based on topologies) lies with the encoder transform,
our impression of the statistical information being conserved is that ‘the
estimates perform with comparable accuracy even after being pushed forward’.
Section 3.1 elaborates on the same. Observe that the necessity to learn a
latent representation puts an upper bound to the encoded dimension. At the
same time, the need to preserve information hints at the existence of a lower
bound. As such, the dimensions in between these two extremities invoke a
trade-off between the accuracy of achieved representation and the amount of
information lost.

Along this discussion comes the call to clarify what we mean by a good
representation. Though WAEs prioritize the task of regeneration, one must not
forget the roots of its predecessors in learning disentangled representation.
Without a robust definition, the idea of disentanglement is marred by
subjectivity. Most, however, deem it as the process of compartmentalizing
information into groups of independent ‘semantic’ attributes [4, 25]. The
underlying assumption being $\mathcal{M}=\bigcup_{j=1}^{l}\mathcal{M}_{j}$,
where $l$ is the number of such groups and $\mathcal{M}_{j}$ are the support
submanifolds. The notion of independence may be softened to ‘uncorrelatedness’
in case group actions are linear [25]. Yu et al. [26] argue that additionally,
such representations should be between-class heterogeneous and within-class
homogeneous to the greatest extent. However, based on human perception, no
measurement of this extent can summarize the whole picture [27]. This
discussion finds great motivation in [28]’s experiments showing WAEs as
efficient representation learners. With further regularization on the latent
space, we may expect to enhance the efficiency in both static [29] and dynamic
[30] data regimes. From a statistical viewpoint, we understand disentanglement
as the process of attaining a distribution with a block diagonal (axis-aligned
as a special case) dispersion matrix. This should ideally be the
characterization of the latent distribution. In other words, a disentangled
law will be our key to the latent dimension. However, finding such a law,
devoid of inductive biases, in an unsupervised setting is theoretically
impossible [31].

It is evident that a typical WAE model, during encoding, performs a non-linear
dimensionality reduction. The standard convolutional architecture carries out
a feature aggregation in the process that is intractable and is not expected
to attain a disentangled law without additional regularization [32, 33]. Thus,
instead of pursuing the optimal value of $k$ directly, we turn our focus on
the transformation induced by the encoder. The resilience of such functions
against distortion along with their regularity becomes paramount in the
following discussion.

###  1.2 Our Contributions

Key highlights of our upcoming discussion are as follows:

  1. (i)

We introduce a probabilistic characterization of the notion information
preservation, which becomes the cornerstone of our depiction of ideal encoders
in a WAE model [Section 3.1]. We explore the measures of discrepancies over
classes of probability measures that allow information preservation based on
naive estimators. Along the line, this enables us to prescribe suitable model
architectures that foster consistency of estimators in the latent space.

  2. (ii)

We establish deterministic upper bounds on the latent loss incurred by WAE
models in a non-parametric regime [Theorem 3.15]. Our approach turns out to be
versatile in the sense that given the input data distribution ($\mu$)
possesses intrinsic structures, they can be readily made free of the input
dimension ($d$). The bounds can be further improved towards greater generality
if the target latent law ($\rho$) is invariant to group actions. In the
process, we explore the desirable properties of underlying kernels in a WAE-
MMD setup that allows latent space consistency.

  3. (iii)

In Section 4, following a density estimation approach, we derive non-
asymptotic sharp upper bounds on the realized reconstruction loss in WAEs. The
regeneration guarantees come with accompanying prescriptions of ideal decoder
networks. The bounds hint at the extent to which optimization errors incurred
in the latent space propagate to reconstruction losses. All of our theoretical
results are empirically substantiated by numerical experiments based on real
and simulated data sets [Section 3.2, 4.1].

  4. (iv)

We additionally examine the effects of contamination in input data on the
performance of WAEs in reconstructions [Section 5]. The discussion includes
desirable properties of kernel estimates that limit the corruption in
translated data. Derived upper bounds to regeneration errors under
distribution shift test WAEs’ inherent capability to offer robustness against
such adversaries.

To maintain lucidity in our elaborate discussion, all proofs of theorems and
additional lemmas are deferred to Appendix Appendix: Technical Proofs.

##  2 Preliminaries

This section is devoted to laying the groundwork in terms of basic definitions
and the statistical formulation of the WAE problem thereafter. The input data
space $\mathcal{X}$, equipped with the metric $c_{x}$ is taken to be Polish.
For most real scenarios, a typical characterization of the same is
$\mathbb{R}^{d}$, $d\geq 1$. We refer to the space of probability measures
defined on $\mathcal{X}$ as $\mathcal{P}(\mathcal{X})$. The same conventions
follow for the latent space $\mathcal{Z}\subseteq\mathbb{R}^{k}$ ($k<d$),
equipped with the metric $c_{z}$, as well. The class of measurable functions
mapping $\mathcal{X}$ to $\mathcal{Z}$ is denoted by
$\mathscr{F}(\mathcal{X},\mathcal{Z})$. For ease of understanding, we
abbreviate the ‘encoder’ and ‘decoder’ transforms as $E$ and $D$ respectively.
Given non-negative real sequences $\\{a_{n}\\}_{n\in\mathbb{N}}$ and
$\\{b_{n}\\}_{n\in\mathbb{N}}$, the suppression of the universal constant
$C>0$, such that $\limsup_{n\rightarrow\infty}\frac{a_{n}}{b_{n}}\leq C$, is
represented as $a_{n}\lesssim b_{n}$, or $a_{n}=\mathcal{O}(b_{n})$. We also
denote $\max\\{x,y\\}\mathrel{\mathop{\ordinarycolon}}=x\vee y$ and
$\min\\{x,y\\}\mathrel{\mathop{\ordinarycolon}}=x\wedge y$.

######  Definition 2.1 (Push-forward).

Given $f\in\mathscr{F}(\mathcal{X},\mathcal{Z})$, the push-forward of
$\mu\in\mathcal{P}(\mathcal{X})$ is defined as
$f_{\\#}\mu(\omega)=\mu(f^{-1}(\omega))$, where
$\omega\in\sigma(\mathcal{Z})$.

######  Definition 2.2 (Integral Probability Metric [34]).

For a class of bounded, measurable evaluation functions
$\mathcal{F}=\left\\{f\mathrel{\mathop{\ordinarycolon}}\mathcal{X}\rightarrow\mathbb{R}\right\\}$,
the integral probability metric (IPM) measuring the discrepancy between
$\mu,\nu\in\mathcal{P}(\mathcal{X})$ is given by

$d_{\mathcal{F}}(\mu,\nu)=\sup_{f\in\mathcal{F}}\Big{\\{}\int_{\mathcal{X}}f(x)d\mu(x)-\int_{\mathcal{X}}f(x)d\nu(x)\Big{\\}}.$

######  Remark 1.

A particular variant of this measure we frequent in our discussion is the
Maximum Mean Discrepancy (MMD). It is obtained by taking $\mathcal{F}$ as the
unit ball in a reproducing kernel Hilbert space (RKHS) $\mathcal{H}$, i.e.
$\mathcal{F}=\left\\{f\mathrel{\mathop{\ordinarycolon}}\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{H}}\leq 1\right\\}$. Moreover, a continuous kernel
$\kappa(\cdot,\cdot)$ based on a compact metric space that results in
$\mathcal{H}$— dense in the space of bounded continuous functions— will compel
the associated MMD to be a metric [35]. The $1$-Wasserstein metric also boils
down to an IPM, given that the underlying class of critics
$\mathcal{F}\mathrel{\mathop{\ordinarycolon}}=\mathcal{L}_{c_{x}}^{1}$, i.e.
$1$-Lipschitz functions with respect to $c_{x}$ (Villani [36], Remark 6.5).
This duality may not hold in general, which is evident from the definition of
the $r^{th}$ Wasserstein distance:

$W^{r}_{c_{x}}(\mu,\nu)=\inf_{\gamma\in\Gamma(\mu,\nu)}\Big{\\{}\int_{\mathcal{X}\times\mathcal{X}}[c_{x}(x,y)]^{r}d\gamma(x,y)\Big{\\}}^{\frac{1}{r}},$

where
$\Gamma(\mu,\nu)=\big{\\{}\gamma\in\mathcal{P}({\mathcal{X}\times\mathcal{X}})\mathrel{\mathop{\ordinarycolon}}\int_{\mathcal{X}}\gamma(x,y)dy=\mu,\;\int_{\mathcal{X}}\gamma(x,y)dx=\nu\big{\\}}$
is the set of all measure couples between $\mu$ and $\nu$; $r\in[1,\infty)$.

######  Definition 2.3 (Probability space automorphism).

Let us denote $X=(\mathcal{X},\sigma(\mathcal{X}),\mu)$, where
$\mu\in\mathcal{P}(\mathcal{X})$. We call
$f\mathrel{\mathop{\ordinarycolon}}X\rightarrow X$ an automorphism if it
admits a measure preserving, essential inverse $f^{{}^{\prime}}$ such that
$f\circ f^{{}^{\prime}}=f^{{}^{\prime}}\circ f=\textrm{id}_{X}$, $\mu$ almost
everywhere.

###  2.1 Problem Setup and Background

Throughout the forthcoming discussion, we denote the input data distribution
by $\mu$, and that corresponding to the latent space by $\rho$. In an earlier
work, Chakrabarty and Das [18] attest to the theoretical superiority of the
constrained formulation of the WAE loss compared to its Lagrangian
counterpart:

$\mathcal{L}_{c_{x},\lambda}(\mu,\rho,D)=\inf_{E\in\mathscr{F}(\mathcal{X},\mathcal{P}(\mathcal{Z}))}\left\\{W_{c_{x}}^{1}(\mu,(D\circ
E)_{\\#}\mu)+\lambda.\Omega(E_{\\#}\mu,\rho)\right\\},$ (1)

where $\lambda>0$. Moreover, to establish consistency of plug-in estimates
under the empirical WAE-GAN loss [3], it is sufficient to consider
$\Omega(\cdot,\cdot)$ as the total variation (TV) metric [18]. This is based
on the fact that TV acts as an upper bound to the Jensen-Shannon divergence
(JS), classically deployed as a regularizer. This modified framework has
attracted theoretical intrigue due to its equivalence with the original one
under the invertibility of decoders. It is often called the $f$-WAE [14].
Building on this density-matching regime, in the current article we also
analyze the WAE-MMD architecture, i.e. when $\Omega\equiv d_{\mathcal{H}}$.

We begin our discussion by focusing on the set of solutions that bring about
zero loss. This is crucial since during training, practitioners frequently
achieve such near-perfect results. However, the solution maps thus obtained
may result in noisy reconstructions. It stems from the fact that WAEs
essentially try to solve an ‘inverse’ problem. Our first result suggests that
in case the latent space admits nontrivial automorphisms, there may exist non-
unique solutions that achieve zero loss.

######  Lemma 2.4 (Invariance of zero solutions [37]).

Let $Z=(\mathcal{Z},\sigma(\mathcal{Z}),\rho)$. Also, the encoder-decoder pair
$(E,D)$ satisfies $\mathcal{L}_{c_{x},\lambda}=0$ for a proability divergence
$\Omega(\cdot,\cdot)$ that metrize $\mathcal{P}(\mathcal{Z})$. Then, given a
non-trivial probability space automorphism
$\varphi\mathrel{\mathop{\ordinarycolon}}Z\rightarrow Z$, the pair
$(\varphi^{-1}\circ E,D\circ\varphi)$ also brings about zero loss.

Observe that one may obtain a zero solution by morphing an existing one.
Rooted in the intractability of $\varphi$, this leads to confusing
prescriptions for practitioners solving a WAE problem based on neural network-
based maps. Moreover, the disentangled representation is sensitive to
rotations of the latent embedding [38]. Thus, one needs to do more than just
point out solutions that achieve zero loss. Also, when seen from an OT point
of view, the existence and consequently approximation of such non-unique
target maps become questionable. We elaborate on the same in Section 3.1. This
brings us to adopting the constrained formulation with heightened conviction:

$\inf_{E\in\mathscr{F}(\mathcal{X},\mathcal{P}(\mathcal{Z}))}\left\\{W_{c_{x}}^{1}(\mu,(D\circ
E)_{\\#}\mu)\right\\}\enspace\textrm{subject
to}\enspace\Omega(E_{\\#}\mu,\rho)\leq t,$ (2)

where $t\geq 0$ signifies the tolerable error margin. Building on earlier
foundation [18], we search for realistic model architectures that promote
consistency of estimators, a stronger notion compared to non-asymptotic
nullity of losses.

###  2.2 Data Distributions

Typically WAE-based architectures are used to deal with image data. The
statistical construct we follow favors such cases without being restricted to
them only. For example, pixel values of raw image data tend to lie in bounded
intervals. As such, considering the support of the probability distribution—
from which they may originate— to be bounded seems plausible. Moreover,
feature-extracted image data attest to this assumption. A key aspect of input
distributions that we are interested in particular is their regularity.
Chakrabarty and Das [18] tested WAEs’ ability to reconstruct Hölder densities,
based on compact supports. We extend our setup to cater to more diverse
distributions. Let us describe some classes of functions that characterize the
same.

Let us denote the space of $p$-fold Lebesgue-integrable functions by
$L_{p}(\mathbb{R}^{d})$, endowed with the norm
$\mathinner{\\!\left\lVert\>\cdot\>\right\rVert}_{p}$, $p\in[1,\infty)$. The
uniform norm is denoted by
$\mathinner{\\!\left\lVert\>\cdot\>\right\rVert}_{\infty}$.

######  Definition 2.5 (Sobolev Space).

Given $\alpha=(\alpha_{1},\alpha_{2},...\>,\alpha_{d})$,
$\alpha_{i}\in\mathbb{N}^{+}\cup\\{0\\}$, a multi-index such that
$|\alpha|=\sum_{i=1}^{d}\alpha_{i}$ stands for the length, the mixed partial
weak differential operator of order $|\alpha|$ is given by
$D^{\alpha}=\frac{\partial^{|\alpha|}}{\partial
x^{\alpha_{1}}_{1}...\>\partial x^{\alpha_{d}}_{d}}$. Here,
$x^{\alpha}=x^{\alpha_{1}}_{1}...\>x^{\alpha_{d}}_{d}$ whenever
$x\in\mathbb{R}^{d}$. Under this setup, the $L^{p}$-Sobolev Space of order $m$
with radius $L\in\mathbb{R}_{\geq 0}$ is defined as

$\mathcal{W}^{m,p}_{L}(\mathbb{R}^{d})=\left\\{f\in
L_{p}(\mathbb{R}^{d})\mathrel{\mathop{\ordinarycolon}}D^{\alpha}f\in
L_{p}(\mathbb{R}^{d})\>\forall\>|\alpha|\leq
m\mathrel{\mathop{\ordinarycolon}}\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{W}^{m,p}}\equiv\>\mathinner{\\!\left\lVert
f\right\rVert}_{p}+\sum_{\mathinner{\\!\left\lvert\alpha\right\rvert}=m}\mathinner{\\!\left\lVert
D^{\alpha}f\right\rVert}_{p}<L\right\\}.$

######  Remark 2.

In case
$f\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{d}\rightarrow\mathbb{R}$ is
differentiable at $x$, set $D^{\alpha}f=f^{(\alpha)}$, i.e., the classical
mixed partial derivative. Also, denote by $C_{u}(\mathbb{R}^{d})$ the space of
uniformly continuous functions. This allows us to extend the earlier class for
$p=\infty$, namely

$\mathcal{W}^{m,\infty}_{L}(\mathbb{R}^{d})=\left\\{f\in
C_{u}(\mathbb{R}^{d})\mathrel{\mathop{\ordinarycolon}}f^{(\alpha)}\in
C_{u}(\mathbb{R}^{d})\>\forall\>|\alpha|\leq
m\mathrel{\mathop{\ordinarycolon}}\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{W}^{m}}\equiv\>\mathinner{\\!\left\lVert
f\right\rVert}_{\infty}+\sum_{|\alpha|=m}\mathinner{\\!\left\lVert
f^{(\alpha)}\right\rVert}_{\infty}<L\right\\}.$

We find the extension of this class for non-integer $s\in\mathbb{R}_{>0}$,
with its integer part $\lfloor s\rfloor$, particularly helpful in the
analysis. The following definition formalizes the same.

######  Definition 2.6 (Hölder-Zygmund Space).

Under the setup described in Remark (2),

$\mathcal{C}^{s}_{L}(\mathbb{R}^{d})=\left\\{f\in
C_{u}(\mathbb{R}^{d})\mathrel{\mathop{\ordinarycolon}}\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{C}^{s}}\equiv\>\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{W}^{\lfloor s\rfloor}}+\sum_{|\alpha|=\lfloor
s\rfloor}\sup_{\begin{subarray}{c}{x\neq y}\\\
{x,y\in\mathbb{R}^{d}}\end{subarray}}\frac{|D^{\alpha}f(x)-D^{\alpha}f(y)|}{|x-y|^{s-\lfloor
s\rfloor}}<L\right\\}.$

All the above functions can be shown to be inhabitants of Besov spaces with
parallel definitions based on wavelets. For further elaboration, one may turn
to Giné and Nickl [39], Chapter 4. Now, let us denote the input density
corresponding to $\mu$ by $p_{\mu}$, with respect to the Lebesgue measure.

######  Assumption 1 (Regularity of Input Law).

There exists $m_{x}\in\mathbb{N}^{+}$ such that
$p_{\mu}\in\mathcal{W}^{m_{x},p}_{L}(\Omega_{x})$, where the support
$\Omega_{x}\subseteq\mathcal{X}$ is compact, $p\in[1,\infty)$.

This assumption is put in place to give coherence to the discussion so far. We
will focus on the case of unbounded domains under varying regularity as
generalizations of the initial results. The more challenging of tasks is
perhaps characterizing the latent distribution. In our density matching
paradigm, it should ideally be a distribution that embodies the
dimensionality-reduced representation of $p_{\mu}$. Let us similarly assume
that $\rho$ also has the corresponding density $p_{\rho}$.

######  Assumption 2.

$p_{\rho}$ is based on a compact and convex support
$\Omega_{z}\subseteq\mathcal{Z}$, such that there exists a positive constant
$c$ satisfying $\inf_{z}p_{\mu}(z)\geq c$.

The generative aspect of WAEs comes from their capability to simulate novel
samples that resemble input observations. The generated set includes smooth
interpolations between modal values of $\mu$. As such, the latent law—
encapsulating the input information into local geometric signatures— must
distribute positive mass between encoded modes. Tolstikhin et al. [3]
demonstrate the same fact with facial image data. This stems from the idea
that the meld between two faces in the Wasserstein geodesic might result in
another one, even if ‘unrealistic’. The convexity of the support of
$p_{\rho}$, coupled with its departure from nullity, is a mathematical
representation of the same philosophy. Asatryan et al. [40] argue that an
explicit lower bound to the density can always be found, for a slightly
modified measure (Remark 3.3). As such, we assume $\rho$ to have a strong
density. Also, to conform to disentanglement, $\rho$ should ideally have a
diagonal or block-diagonal dispersion matrix. In our non-parametric depiction,
we keep ample room for such specifications.

##  3 Latent Space Consistency

With the foundations laid, let us concentrate on the encoding. In an empirical
WAE problem, we only have access to a set of samples $\\{X_{i}\\}^{n}_{i=1}$
drawn i.i.d. from $\mu$. Thus, the sample version of the optimization task (2)
needs to satisfy the corresponding constraint:
$\Omega(E_{\\#}\hat{\mu}_{n},\rho)\leq t$, given $t\geq 0$. Here,
$\hat{\mu}_{n}=\frac{1}{n}\sum^{n}_{i=1}\delta_{X_{i}}$ is the classical plug-
in estimate. We use the same notation to signify the empirical distribution
throughout the article. Observe that, the resultant set of encoder transforms
that fulfill this criterion are indeed functions of $n$, i.e. $E=E(n)$. In the
absence of uniqueness, our suggestions of a capable encoder begin with a
definition of its chassis: neural networks.

######  Definition 3.1 (Feed-Forward Neural Networks).

Given $L\in\mathbb{N}^{+}$, a Neural Network (NN) with $L$ hidden layers is
defined as the collection of maps
$\phi\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{N_{0}}\longrightarrow\mathbb{R}^{N_{L+1}}$,
$\\{N_{i}\\}^{L+1}_{i=0}\in\mathbb{N}^{+}$ given by

$\phi(x)\mathrel{\mathop{\ordinarycolon}}=A_{L}\circ\sigma\circ
A_{L-1}\circ...\circ\sigma\circ A_{0}(x),$

where $A_{i}(y)=M_{i}y+b_{i}$; $M_{i}\in\mathbb{R}^{N_{i+1}\times N_{i}}$ and
$b_{i}\in\mathbb{R}^{N_{i+1}}$ is an affinity, $i=0,...,L$. Here, $\sigma$
signifies the activation function, applied componentwise. Under this setup, we
call $W=\vee^{L}_{i=1}N_{i}$ the width of the network and $L$ its depth.
Denote this collection by $\Phi(W,L)^{N_{L+1}}_{N_{0}}$. Additionally, the
quantity $S=\sum^{L}_{i=1}N_{i}$ is said to be the size. A reparameterized
version of the same, given as $N(\Phi)=d+S$, denotes the number of neurons in
the network.

######  Remark 3.

Based on its simplicity and resilience to vanishing gradients, ReLU
($\sigma(x)=x\vee 0$, also called ramp or first order spline) has emerged as
the most desired activation to practitioners. However, what we find intriguing
is the remarkable capability of ReLU-based NNs to approximate regular
functions [41, 42, 43]. Another activation function that is critical to our
analysis is GroupSort [44]. Preserving all the goodness offered by ReLU, it
additionally provides adversarial robustness [45]. We also stress the fact
that GroupSort (equivalently OPLU, when grouping size is 2 [46]) NNs are
better suited at universally approximating non-linear Lipschitz maps.

###  3.1 Encoder maps

Encoders are transformations that can be said to enforce dimensionality
reduction preserving key properties of $\mu$. Though not obvious, typically,
such maps enforce a non-linear reduction due to the non-linearities
(activations, e.g. tanh) present in the underlying NN. The process it
undergoes is significantly different from classically known DR techniques.
However, in case the maps are assumed to be linear embeddings (decoder as
well), latent factors obtained by a VAE tend to align along the Principal
Component (PC) directions [38]. Regularised VAEs can also be related to the DR
carried out by non-linear ICA [47] under a parametric regime. The similarity
stems from the achievement of identifiability of the parameters characterizing
the latent law in both methods [48]. This departure from traditional
techniques forces us to change our viewpoint on DR as we know it. Besides, the
encoding in the posterior density-matching setup of WAEs differs even further.
Instead of looking at the encoder’s capacity to conserve local and broader
geometry of the spaces in terms of distances, we focus on its trait to limit
distortions of estimators. Let us provide a probabilistic definition of the
same.

######  Definition 3.2 (Information Preserving Transform [18]).

Given an estimate $\hat{\mu}_{n}$ based on $n$ observations from the
distribution $\mu$ and $\epsilon>0$, a map
$I\in\mathscr{F}\left(\mathcal{X},\mathcal{P}(\mathcal{Z})\right)$ is said to
be Information Preserving of degree $r$ under the distance metric
$d(\cdot,\cdot)$ if there exist constants $k_{1},k_{2}>0$, such that

$\mathbb{P}\left(d\left(I_{\\#}\hat{\mu}_{n},\widehat{(I_{\\#}\mu)}_{m}\right)\leq\epsilon\right)\geq
1-k_{1}\exp{\left\\{-k_{2}(n\wedge m)^{r}\epsilon^{2}\right\\}}.$

Here, $\widehat{(I_{\\#}\mu)}_{m}$ denotes an estimate of the translated
distribution based on $m\in\mathbb{N}^{+}$ i.i.d. samples and $r\geq 1$.

The immediate question coming to mind may be what are the transformations that
behave as IPTs. Precisely, the answer lies in the regularity of the functions.
Though not apparent, the notion of IPTs is intrinsically related to Bourgain’s
discretization theorem and Lipschitz embeddings. To that end, we first explore
the capability of Lipschitz continuous maps— between the input and latent
spaces— to pose as IPTs. Let us denote by
$\mathscr{F}_{L}(\mathcal{X},\mathcal{Z})$ the class of $L$-Lipschitz
functions mapping $(\mathcal{X},c_{x})$ to $(\mathcal{Z},c_{z})$. So far we
have not imposed any regularity on the class of latent distributions. In such
a general setting, the role of the underlying divergence metrizing the measure
space becomes crucial. In this context, we recall the caution sounded by
Sriperumbudur et al. [49] that the total variation metric
($d_{\mathcal{F}}\equiv d_{\textrm{TV}}$, where
$\mathcal{F}=\left\\{f\mathrel{\mathop{\ordinarycolon}}\mathinner{\\!\left\lVert
f\right\rVert}_{\infty}\leq 1\right\\}$) is often unable to ensure strong
consistency of estimators under them. The issue is rooted in the class of
critics $\mathcal{F}$ being ‘too large’. The first method to circumvent this
problem is to look at IPMs employing more precise critics.

######  Theorem 3.3 (Information Preservation of Lipschitz Encoders).

Let $\mathcal{H}$ denote a class of bounded real-valued functions on
$\mathcal{Z}$, such that the associated entropy has at most polynomial
discrimination. In other words, there exists $q,A\in\mathbb{R}_{>0}$ such that
$\forall\epsilon>0,\;\log\mathcal{N}\left(\mathcal{H},\mathinner{\\!\left\lVert\cdot\right\rVert}_{\infty},\epsilon\right)\leq
A\epsilon^{-q}$. Then for any $g\in\mathscr{F}_{L}(\mathcal{X},\mathcal{Z})$
there exists constants $l,E_{1},E_{2}$ and $E_{3}>0$ such that given
$0<t\leq\frac{2}{3}$,

$d_{\mathcal{H}}\left(g_{\\#}\tilde{\mu}_{n},\widehat{(g_{\\#}\mu)}_{m}\right)\leq
t+\frac{lLh^{m_{x}}}{2}+\mathcal{O}(m^{-\frac{1}{q\vee 2}})$

holds with probability $\geq
1-\left(E_{1}+E_{2}(\frac{\sqrt{d}LB_{x}}{h^{d+1}t})^{d}\right)\exp\left\\{-E_{3}(m\wedge
nh^{d})t^{2}\right\\}$, where $\tilde{\mu}_{n}$ is a density estimate of $\mu$
based on the Regularly invariant kernel $\kappa$, satisfying
$\sup_{\kappa}\sup_{x\in\Omega_{x}}\kappa(\cdot,\cdot)\leq 1$, and with
bandwidth $h\equiv h_{n}\searrow 0$.

The theorem implies that Lipschitz transforms can approximately pose as IPTs
of order $1$. By choosing $h$ judiciously, one may show that the approximation
error turns $o(1)$ in the asymptotic regime. We deliberately use the smoother
kernel density estimate instead of the plug-in to appreciate Assumption 1. The
choice of the kernel function— as regularly invariant— is of high
significance, which the proof (see Appendix Appendix: Technical Proofs)
demonstrates. We elaborate on the same while discussing MMDs (Definition 3.7).
Now, the classes $\mathcal{H}$ whose entropy increase polynomially lie in
abundance [50]. A particular case that we emphasize on is
$\mathcal{L}_{c_{z}}^{1}$, i.e. $1$-Lipschitz functions with respect to
$c_{z}$. It is known that
$\log\mathcal{N}\left(\mathcal{L}_{c_{z}}^{1},\mathinner{\\!\left\lVert\cdot\right\rVert}_{\infty},\epsilon\right)\lesssim\lambda(\mathcal{Z}^{1})\epsilon^{-k}$,
where $\lambda(\mathcal{Z}^{1})$ is the Lebesgue measure of the set
$\left\\{z\mathrel{\mathop{\ordinarycolon}}c_{z}(z,\mathcal{Z})<1\right\\}$
(Van Der Vaart and Wellner [51], Theorem 2.7.1). The choice of critics as
Lipschitz also provides a generalization over most Besov functions.

######  corollary 3.4.

Given $g\in\mathscr{F}_{L}(\mathcal{X},\mathcal{Z})$ and the empirical
distribution $\hat{\mu}_{n}$, there exists a positive constant
$E_{1}^{{}^{\prime}}$, such that

$d_{\mathcal{L}_{c_{z}}^{1}}\left(g_{\\#}\hat{\mu}_{n},\widehat{(g_{\\#}\mu)}_{m}\right)\leq
t+\mathcal{O}(m^{-\frac{1}{k\vee 2}})+\mathcal{O}(n^{-\frac{1}{d}})$

holds with probability at least $1-\exp\left\\{-E_{1}^{{}^{\prime}}(n\wedge
m)t^{2}\right\\}$.

######  Remark 4 (Extension for $b$-Lipschitz critics).

A similar result to that of the previous theorem in case of the divergence
$d_{\mathcal{L}_{c_{z}}^{b}}(\cdot,\cdot)$ can be established based on the
fact that
$\log\mathcal{N}\left(\mathcal{L}_{c_{z}}^{b},\mathinner{\\!\left\lVert\cdot\right\rVert}_{\infty},\epsilon\right)\leq\mathcal{N}\left(\mathcal{Z},c_{z},\frac{\epsilon}{8b}\right)\log\big{(}\frac{8}{\epsilon}\big{)}$
(Gottlieb et al. [52], Lemma 6), given $b>0$. Observe that, it also enables
one to remove the boundedness of the support, latent class of measures lie on.
Instead, we may impose milder restrictions such as having sub-exponential
tails (essentially bounded). Specifically, if
$\mathbb{E}_{p}\\{\mathinner{\\!\left\lVert
Z\right\rVert}\mathbb{1}_{\\{\mathinner{\\!\left\lVert
Z\right\rVert}>\log(m)\\}}\\}=\mathcal{O}(m^{-\frac{{(\log{m})^{\delta}}}{k}})$,
where $p\in\mathcal{P}(\mathcal{Z})$ and $\delta>0$; we may recover Corollary
(3.4) with only an altered approximation error
$\mathcal{O}(m^{-\frac{1}{k}}(\log m)^{1+\frac{1}{k}})$.

Now let us focus on the second remedy, that being more regulated classes of
translated laws. This is crucial since otherwise the convergence of empirical
measures under TV might become arbitrarily slow [53]. To that end, let us
first recall the notion of Yatracos classes (YC) (Devroye and Lugosi [54],
Chapter 6). Given
$\mathcal{F}\mathrel{\mathop{\ordinarycolon}}\mathcal{Z}\rightarrow\mathbb{R}$,
the Yatracos class associated to it is the set system
$\\{z\in\mathcal{Z}\mathrel{\mathop{\ordinarycolon}}f(z)\geq
g(z);\;f,g\in\mathcal{F}\\}$, denoted by $\mathcal{Y}(\mathcal{F})$. In other
words, it characterizes the domain in terms of candidates in a Scheffé
tournament. Our next result suggests that if the Vapnik-Chervonenkis (VC)
dimension of the YC corresponding to the family of latent distributions is
finite, we can recover Theorem (3.3). The proposition becomes quite intuitive
following the maximal packing argument of Van Handel [55], Theorem 7.16.

######  corollary 3.5.

Let the VC-dim$[\mathcal{Y}(\mathcal{P}(\mathcal{Z}))]=v_{z}<\infty$. Then for
any $g\in\mathscr{F}_{L}(\mathcal{X},\mathcal{Z})$ and $0<t\leq\frac{2}{3}$
the constants $E_{1},E_{2}$ and $E_{3}>0$ are retained such that

$d_{\textrm{TV}}\left(g_{\\#}\tilde{\mu}_{n},\widehat{(g_{\\#}\mu)}_{m}\right)\leq
t+\frac{lLh^{m_{x}}}{2}+\mathcal{O}(\sqrt{v_{z}}m^{-\frac{1}{2}})$

holds with probability $\geq
1-\left(E_{1}+E_{2}(\frac{\sqrt{d}LB_{x}}{h^{d+1}t})^{d}\right)\exp\left\\{-E_{3}(m\wedge
nh^{d})t^{2}\right\\}$, where $\tilde{\mu}_{n}$ is a Regularly Invariant
kernel (RIK) density estimate of $\mu$, defined similarly as in Theorem 3.3.

There are two key highlights of the latest result that turn out to be
indispensable in the upcoming discussion on latent consistency. The first
aspect we emphasize is the tail condition of the target law. Sub-exponential
is a fairly general notion in the sense that all bounded and sub-Gaussian
distributions fall under its umbrella. Moreover, all results obtained under
such a characterization can be directly extended to sub-Weibull distributions
[56]. In practice, WAEs are mostly trained with multivariate Gaussian as
conjugate prior (and hence, posterior) latent laws [3], which also conforms to
our argument. The second facet— arguably the cornerstone of the analysis by
Chakrabarty and Das [18], and responsible for controlling the complexity of
the underlying space— is the quantity VC-dim$[\mathcal{Y}(\cdot)]$. The
finiteness assumption on the same is frequented in density estimation [57, 58]
solely based on its plausibility. It is known that the class of
$k$-dimensional Gaussian distributions have VC-
dim$[\mathcal{Y}(\cdot)]=\mathcal{O}(k^{2})$. The same corresponding to axis-
aligned densities hailing from $k$-variate exponential families turn out to be
$\mathcal{O}(k)$ (Devroye and Lugosi [54], Theorem 8.1).

Before moving on to further examples of IPTs, let us examine the worth of NN-
based maps in the same context. Observe that, the transformations
$A_{i}(\cdot)$ (see Definition 3.1) can be easily shown to be Lipschitz
continuous by limiting $\mathinner{\\!\left\lVert
M_{i}\right\rVert}_{p}=\sup_{\>\mathinner{\\!\left\lVert
y\right\rVert}_{p}=1}\mathinner{\\!\left\lVert M_{i}y\right\rVert}_{p}\leq t$,
given $t>0$. Anil et al. [44] gave simple recipes to preserve such norms in
case $p=2$ and $\infty$. This fact, coupled with the Lipschitz continuity of
activation functions (e.g. ReLU, leaky ReLU, GroupSort, tanh, sigmoid, etc.)
typically applied, it is not difficult to imagine NN-transforms to be exactly
so. However, not all such $\sigma(\cdot)$ preserve gradient norms under
composition without additional regularization (e.g. ReLU). Furthermore,
recovering the exact Lipschitz constant, and hence the map, often turns out to
be NP-hard [59]. So instead, we harness the approximation capabilities of deep
NNs to our aid. ReLU has attracted the most attention in this regard [60, 61,
62, 63, 64]. To motivate our next result, we present a simple observation:

######  Lemma 3.6.

Given $\mu_{1},\mu_{2}\in\mathcal{P}(\mathcal{X})$ and
$\phi\in\Phi(W,L)^{k}_{d}$, under arbitrary IPM
$d_{\mathcal{F}}(\cdot,\cdot)$, such that
$\mathcal{F}=\left\\{f\mathrel{\mathop{\ordinarycolon}}\mathcal{Z}\rightarrow\mathbb{R}\right\\}$
is symmetric, we obtain

$d_{\mathcal{F}}(\phi_{\\#}\mu_{1},\phi_{\\#}\mu_{2})\leq
2\left[\inf_{g\in\mathscr{F}(\mathcal{X},\mathcal{P}(\mathcal{Z}))}\mathinner{\\!\left\lVert\phi-g\right\rVert}_{\infty}+\mathcal{E}(\mathcal{F},\mathcal{L}_{c_{z}}^{1})\right]+d_{\mathcal{L}_{c_{z}}^{1}}(g_{\\#}\mu_{1},g_{\\#}\mu_{2}),$

where
$\mathcal{E}(\mathcal{F},\mathcal{L}_{c_{z}}^{1})=\sup_{f\in\mathcal{F}}\inf_{l\in\mathcal{L}_{c_{z}}^{1}}\mathinner{\\!\left\lVert
f-l\right\rVert}_{\infty}$ denotes the essential disagreement between classes
of critics and $c_{z}\equiv L_{1}$.

Observe that, the statement holds true under arbitrary choices of the second
class of evaluation functions. We mention $\mathcal{L}_{c_{z}}^{1}$, in
particular, to continue our discussion in the light of Corollary 3.4. The
result suggests that it is sufficient for a feed-forward NN-induced function
to approximate Lipschitz maps (between the input and latent space) to behave
like an IPT, approximately. Under the TV metric, the proof becomes much
simpler based on the fact that
$d_{\textrm{TV}}(\phi_{\\#}\mu_{1},\phi_{\\#}\mu_{2})\leq
d_{\textrm{TV}}(\mu_{1},\mu_{2})$, for
$\phi\in\mathscr{F}(\mathcal{X},\mathcal{P}(\mathcal{Z}))$. However, there may
arise difficulties in case the underlying distance measure is MMD. Let us
first go through some rudiments of kernel methods to facilitate our
investigation on the same.

### MMD and Kernel Mean Embedding

The well-known Riesz representation theorem ensures the existence of a unique
representer $K(x)\in\mathcal{H}$, such that $\forall
f\in\mathcal{H},f(x)=\langle f,K(x)\rangle$ for all $x\in\mathcal{X}$. In this
setup, the function $\kappa(x,y)=\langle K(x),K(y)\rangle$ is called the
reproducing kernel of $\mathcal{H}$. The opposite characterization also holds
true. By Aronszajn’s theorem, given a symmetric, positive definite $\kappa$ on
$\mathcal{X}\times\mathcal{X}$, there exists a unique RKHS
$\mathcal{H}_{\kappa}$. This inspires us to meaningfully narrow down our focus
on the distributions
$\mathcal{P}_{\kappa}=\\{\pi\in\mathcal{P}\mathrel{\mathop{\ordinarycolon}}\int\sqrt{\kappa(x,x)}\mathinner{\\!\left\lvert\pi\right\rvert}(dx)<\infty\\}$.
The MMD between two of such laws $\mu_{1},\mu_{2}$ can be rewritten as
$d_{\mathcal{H}_{\kappa}}(\mu_{1},\mu_{2})=\mathinner{\\!\left\lVert
K(\mu_{1})-K(\mu_{2})\right\rVert}_{\mathcal{H}_{\kappa}}$, i.e. the Hilbert
space distance between the respective kernel mean embeddings (KME), given by
$K(\pi)(x)=\int\kappa(x,y)\pi(dy)$. For a detailed exposition on the same, one
may seek refuge to Sriperumbudur et al. [65]. Since it is the kernel function
that determines the nature of the RKHS, we introduce some regularities which
in turn aid our cause.

######  Definition 3.7 (Regularly Invariant Kernels).

A measurable function
$\kappa(x,y)\mathrel{\mathop{\ordinarycolon}}\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$
is said to be regular, if for $N\in\mathbb{N}$ it satisfies

  1. (i)

$\int_{\mathcal{X}}\sup_{v\in\mathcal{X}}\mathinner{\\!\left\lvert\kappa(v,v-u)\right\rvert}{\mathinner{\\!\left\lvert
u\right\rvert}}^{N}du<\infty$, and

  2. (ii)

$\int_{\mathcal{X}}\kappa(v,v+u)du=1;\int_{\mathcal{X}}\kappa(v,v+u)u^{\alpha}du=0$
for every $v\in\mathcal{X}$ and
$\mathinner{\\!\left\lvert\alpha\right\rvert}=1,...,N-1$.

If such a kernel additionally satisfies the weaker invariance property:
$\left\\{\int\mathinner{\\!\left\lvert\kappa(w,v)-\kappa(w,u)\right\rvert}^{r}dw\right\\}^{\frac{1}{r}}=\mathcal{O}\left(\mathinner{\\!\left\lVert
v-u\right\rVert}\right)$, given $r\geq 1$; we say it is regularly invariant.

Observe that, most kernel functions based on standard probability
distributions with finite and centered moments will tend to be regular.
Moreover, an immediate example of our version of invariant would be Energy
kernels: $\kappa_{\alpha}(u,v)=\mathinner{\\!\left\lVert
u\right\rVert}^{2\alpha}+\mathinner{\\!\left\lVert
v\right\rVert}^{2\alpha}-\mathinner{\\!\left\lVert
u-v\right\rVert}^{2\alpha}$, $u,v\in\mathcal{X}$ and $\alpha\in(0,1)$ [66].
For further coherence, we introduce the notion of strong invariance. A kernel
function is called strongly invariant if the following holds:
$\mathinner{\\!\left\lVert
K(u)-K(v)\right\rVert}=\mathcal{O}\left(\mathinner{\\!\left\lVert
u-v\right\rVert}\right)$. This in turn implies weak invariance based on the
fact that

$\displaystyle\left\\{\int\mathinner{\\!\left\lvert\kappa(w,v)-\kappa(w,u)\right\rvert}^{r}dw\right\\}^{\frac{1}{r}}\leq\mathinner{\\!\left\lVert
K(v)-K(u)\right\rVert}\left[\int\mathinner{\\!\left\lVert
K(w)\right\rVert}^{r}dw\right]^{\frac{1}{r}}.$

For example, in case of Energy kernels, $\mathinner{\\!\left\lVert
K(u)-K(v)\right\rVert}=2\mathinner{\\!\left\lVert u-v\right\rVert}^{\alpha}$.
Based on such functions, MMD indeed promotes exponential decay in information
dissipated while encoding. To set the stage for a supporting mathematical
argument, let us first notice that given
$\mu\in\mathcal{P}_{\kappa}(\mathcal{X})$,

$d^{2}_{\mathcal{H}_{\kappa}}(\hat{\mu}_{n},\mu)=\langle
K(\hat{\mu}_{n}-\mu),K(\hat{\mu}_{n}-\mu)\rangle_{\mathcal{H}_{\kappa}}=\int_{\mathcal{X}\times\mathcal{X}}\kappa(x,y)(\hat{\mu}_{n}-\mu)\otimes(\hat{\mu}_{n}-\mu)(dxdy).$

Also,
$\mathbb{E}\left[d_{\mathcal{H}_{\kappa}}(\hat{\mu}_{n},\mu)\right]\leq\left[\mathbb{E}\;d^{2}_{\mathcal{H}_{\kappa}}(\hat{\mu}_{n},\mu)\right]^{\frac{1}{2}}\leq\sqrt{\frac{2}{n}}\sup_{x\in\Omega_{x}}\kappa(x,x)^{\frac{1}{2}}$.

######  Theorem 3.8 (Information preservation under MMD).

Let $\mu\in\mathcal{P}_{\kappa}(\mathcal{X})$, where $\kappa(\cdot,\cdot)$ is
a strongly invariant kernel satisfying $\sup_{z\in\Omega_{z}}\kappa(z,z)\leq
C_{\kappa}$, such that $C_{\kappa}>0$. Given
$g\in\mathscr{F}_{L}(\mathcal{X},\mathcal{Z})$, there exists
$\phi\in\Phi(W,L)^{k}_{d}$ which implies that

$\displaystyle d_{\mathcal{H}_{\kappa}}$
$\displaystyle\left(\phi_{\\#}\hat{\mu}_{n},\widehat{(\phi_{\\#}\mu)}_{m}\right)\leq(m\wedge
n)^{-\frac{1}{2}}\sqrt{\frac{B\ln{(\frac{2}{\delta})}}{2}}+\sqrt{\frac{2C_{\kappa}}{m}}$
$\displaystyle\qquad+\sqrt{D_{n}}\underbrace{\mathinner{\\!\left\lVert\phi-g\right\rVert}_{\infty}^{\frac{1}{2}}}_{\text{$(\ast)$}}+\sqrt{\mathcal{O}(c_{g,n}{(d^{2}n)}^{-\frac{1}{d}})+L(m\wedge
n)^{-\frac{1}{2}}c_{g,n}\sqrt{\frac{B\ln{(\frac{2}{\delta})}}{2}}}$

holds with probability at least $1-\delta$, $\delta>0$. Here, $B$ is a
positive constant dependent on $C_{\kappa}$, and both $D_{n}$ and $c_{g,n}$
are sequences based on $n$ that $\searrow 0$ almost surely as
$n\rightarrow\infty$.

This result in turn enables to showcase the information-preserving capability
of NNs deploying several activation functions. Observe that, the quantity
$(\ast)$ in Theorem 3.8 acts as an upper bound to the departure of an NN-
induced map from its exemplar $g$. The following corollaries look for the
sharp values of $(\ast)$ under different circumstances.

######  corollary 3.9.

  1. (i)

(Information Preservation of ReLU Encoders [67])
There exists $\phi\in\Phi(W,L)^{k}_{d}$ based on ReLU activations with
$W=\mathcal{O}(d\lfloor N_{1}^{\frac{1}{d}}\rfloor\vee N_{1}+1)$ and
$L=\mathcal{O}(N_{2})$, that satisfy Theorem 3.8 for any
$N_{1},N_{2}\in\mathbb{N}^{+}$, such that
$(\ast)=\mathcal{O}(\sqrt{d}LB_{x}N_{1}^{-\frac{2}{d}}N_{2}^{-\frac{2}{d}})$.
Here, $B_{x}\mathrel{\mathop{\ordinarycolon}}=$ diameter of $\Omega_{x}$ with
respect to the metric $c_{x}$.

  2. (ii)

(Information Preservation of GroupSort Encoders [68])
Given $\varepsilon>0$, there exists a GroupSort NN induced map
$\phi\in\Phi(W,L)^{k}_{d}$ of depth
$L=\mathcal{O}(d^{2}\log_{2}(\frac{2\sqrt{d}}{\varepsilon}))$ and size
$S=(\frac{2\sqrt{d}}{\varepsilon})^{d^{2}}$, also satisfying
$\mathinner{\\!\left\lVert
M_{0}\right\rVert}_{2,\infty}=\sup_{\>\mathinner{\\!\left\lVert
x\right\rVert}=1}\mathinner{\\!\left\lVert M_{0}x\right\rVert}_{\infty}\leq
1$, $\max\\{\>\mathinner{\\!\left\lVert
M_{i}\right\rVert}_{\infty};i=1,\cdots,L\\}\leq 1$ and
$\max\\{\>\mathinner{\\!\left\lVert
b_{j}\right\rVert}_{\infty};j=0,\cdots,L\\}\leq\infty$, such that
$(\ast)=\mathcal{O}(\varepsilon)$.

######  Remark 5 (Barron Functions as IPT).

Based on our previous discussion it becomes evident that being Lipschitz
continuous is a desirable property for encoder transforms to behave as IPTs,
approximately at the least. This very fact accentuates the importance of the
class of functions known as Barron functions. While there exist numerous
characterizations of the same [69], we keep to the following definition.

######  Definition 3.10 (Caragea et al. [70]).

A function
$f\mathrel{\mathop{\ordinarycolon}}\Omega_{x}(\subset\mathbb{R}^{d})\rightarrow\mathbb{R}$
is said to belong to Barron class with constant $C>0$ (say,
$\mathcal{B}_{C}(\Omega_{x})$), if the exists $x^{{}^{\prime}}\in\Omega_{x}$
and a measurable function
$g\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{d}\rightarrow\mathbb{C}$ such
that $\forall x\in\Omega_{x}$ both the conditions

$\int_{\mathbb{R}^{d}}\sup_{x\in\Omega_{x}}\mathinner{\\!\left\lvert\langle\eta,x-x^{{}^{\prime}}\rangle\right\rvert}\mathinner{\\!\left\lvert
g(\eta)\right\rvert}d\eta\leq C\quad\textrm{and}\quad
f(x)-c=\int_{\mathbb{R}^{d}}(e^{i\langle x,\eta\rangle}-e^{i\langle
x^{{}^{\prime}},\eta\rangle})g(\eta)d\eta$

are satisfied, where $\mathinner{\\!\left\lvert c\right\rvert}\leq C$.

For vector-valued functions
$f\mathrel{\mathop{\ordinarycolon}}\Omega_{x}\rightarrow\mathbb{R}^{k}$, which
we are mostly interested in, the same criteria need to be satisfied
componentwise. Lee et al. [71] provide an equivalent definition (also based on
Fourier inversion) of Barron class as well. It is intriguing to observe that
$\mathcal{B}_{C}$ embeds continuously into the class of real-valued Lipschitz
maps, under the $L_{1}$ metric (Wojtowytsch et al. [69], Theorem 3.3). As
such, Barron functions are naturally prone to preserving information while
serving as encoders (Theorem 3.3). Now, observe that the real architectures
suggested in the context of IPT so far might suffer from the curse of
dimensionality. Also, they both tend to be deep, scaling exponentially with
the input data dimension. While this serves our purpose in the asymptotic
regime, shallow networks ($L=1$) might be of greater priority to
practitioners. Meanwhile, the Barron class can be shown to accommodate all
finite norm-bounded neural networks and their limits [69]. This is crucial
since it allows one to demonstrate shallow networks’ capability to act as an
IPT approximately. Barron [72], in his seminal paper, first proved that a
function $f\in\mathcal{B}_{C}$ can be approximated up to arbitrary accuracy by
a shallow NN deploying sigmoidal activations111A bounded measurable function
$h\mathrel{\mathop{\ordinarycolon}}\mathbb{R}\rightarrow\mathbb{R}$ is said to
be sigmoidal if $\lim_{x\rightarrow-\infty}h(x)=0$ and
$\lim_{x\rightarrow\infty}h(x)=1$. Common examples of such activation
functions include logistic, hyperbolic tangent, and
$h(x)=\textrm{ReLU}(x)-\textrm{ReLU}(x-1)$.. In other words, there exists
$\phi\in\Phi(W,1)^{1}_{d}$ with $N(\Phi)=m$, such that
$\\{\int_{\Omega_{x}}(f(x)-\phi(x))^{2}\mu(dx)\\}^{\frac{1}{2}}=\mathinner{\\!\left\lVert
f-\phi\right\rVert}\lesssim m^{-\frac{1}{2}}$. In the asymptotic regime,
however, to achieve an infinitesimally small error, the map $\phi$ approaches
an infinitely wide NN. Wojtowytsch et al. [69] draw the same conclusion based
on input distributions $\mu$ having finite second moment (Theorem 3.8). This
approximation can be further improved in case $\mu$ has bounded support,
conforming to Assumption 1. Klusowski and Barron [73] show the existence of
such a $\phi$, based on general Lipschitz activations, that ensures
$\mathinner{\\!\left\lVert
f-\phi\right\rVert}_{\infty}\lesssim\sqrt{d+\log{m}}\>m^{-\frac{1}{2}-\frac{1}{d}}$.
Despite being the highlight, our discussion on the approximation of Barron
functions is not limited to shallow networks. The information-preserving
behavior of Barron maps stays intact under compositions as well. This is again
rooted in them being Lipschitz continuous. As an immediate consequence, we
find an alternative avenue to show that sigmoid-activated deep encoders too
act as IPTs.

######  corollary 3.11 (Information Preservation of Sigmoidal Encoders [71]).

Let $\\{N_{i}\\}^{L+1}_{i=0}\in\mathbb{N}^{+}$ be a sequence of intermediate
dimensions as given in Definition 3.1, where $N_{0}=d$ and $N_{L+1}=k$. Also
let
$f_{i}\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{N_{i-1}}\rightarrow\mathbb{R}^{N_{i}}$
such that $f_{1}\in\mathcal{B}_{C_{0}}(\Omega_{0})$ and for $2\leq i\leq L+1$
and a given parameter $s>0$,
$f_{i}\in\mathcal{B}_{C_{i-1}}(\Omega^{s,N_{i-1}}_{i-1})$. Here
$\\{C_{i}\\}^{L}_{i=0}>0$ and
$\Omega^{s,N_{i-1}}_{i-1}\mathrel{\mathop{\ordinarycolon}}=\\{y=y_{1}+y_{2}\mathrel{\mathop{\ordinarycolon}}y_{1}\in\Omega_{i-1},y_{2}\in
B^{N_{i-1}}(s)\\}$, $B^{N_{i-1}}(s)$ being the $L_{2}$ ball of radius $s$ in
$\mathbb{R}^{N_{i-1}}$, that satisfy $f_{i}(\Omega_{i-1})\subseteq\Omega_{i}$,
$1\leq i\leq L+1$. In particular, define $\Omega_{0}\equiv\Omega_{x}$ and
$\Omega_{L+1}\equiv\Omega_{z}$. Under this setup, given
$f_{1\mathrel{\mathop{\ordinarycolon}}L+1}\mathrel{\mathop{\ordinarycolon}}=f_{1}\circ
f_{2}\circ\cdots\circ f_{L+1}$ and $\varepsilon>0$, there exists an $L$-deep
sigmoid-activated $\phi$, with $\mathcal{O}(N_{i}\varepsilon^{-2})$ nodes in
the $i$th layer, such that $(\ast)=\mathcal{O}(\varepsilon)$.

With the desired regularity of an ideal encoder specified, we move closer to
testing whether the constraint, as in (2), is met. We reiterate that the
problem at hand is rather the sample equivalent of the same. While our
probabilistic notion based on estimators provides a view into the solution,
encoding might also be seen in the light of local geometry. This not only
provides further clarity to the definition of representation but also opens up
a pathway to understanding decoding in greater detail. Observe that, the
process of encoding seeks to learn embeddings onto the latent space. A
meaningful characterization of the same might be a map that aims to preserve
pairwise distances between discrete points (the sample set) residing in the
input space [74]. It also turns out to be the cornerstone of the techniques
relying on the latent space to form clusters [75]. Such a map
$E\mathrel{\mathop{\ordinarycolon}}\Omega_{x}\rightarrow\Omega_{z}$ satisfying
$ac_{x}(x,y)\leq c_{z}(E(x),E(y))\leq Ac_{x}(x,y)$ is said to be bi-Lipschitz
(BL) with distortion $D(E)=\frac{A}{a}<\infty$. These immediately fit the bill
of IPTs. Moreover, the inverse of such embeddings turns out to be Lipschitz as
well. We will later see that this fact can be exploited to aid in efficient
decoding. Another feature that stands out is that $E$ restricts the encoded
law from being degenerated at a point. Now, the immediate question that arises
is whether such embeddings exist. The first affirmative evidence was presented
by Johnson and Lindenstrauss [76], taking both $c_{x}$ and $c_{z}$ to be
$L_{2}$ in their respective spaces.

######  Lemma 3.12 (Johnson-Lindenstrauss Embedding).

Given a set of size $n$ from $\Omega_{x}$ and $0<\varepsilon<\frac{1}{2}$,
there exists a Bi-Lipschitz map
$E\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{d}\rightarrow\mathbb{R}^{k}$
with distortion $(1+\varepsilon)$, such that
$k=\mathcal{O}(\frac{\log(n)}{\varepsilon^{2}})$.

This result additionally ties the extent of distortion to the latent
dimension, shedding new light on our previous discussion on the optimal value
of $k$. In fact, the bound in the lemma turns out to be optimal up to constant
factors [77]. Later, Bourgain [78] also showed the existence of BL transforms
that achieve encoding onto $k=\mathcal{O}(\log^{2}n)$ with distortion
$\lesssim\log(n)$. For now, the existence of Lipschitz encoders, acting as
benchmarks to NN-based maps, will be sufficient. To that end, we turn to the
following result.

######  Lemma 3.13 (Bartal et al. [79]).

Given $X\subseteq\Omega_{x}$, for every $\varepsilon>0$, there exists a finite
$1$-Lipschitz embedding
$E\mathrel{\mathop{\ordinarycolon}}X\rightarrow\mathbb{R}^{k}$ such that
$k=\mathcal{O}\left(\varepsilon^{-2}d^{*}(X)(\frac{\log(d^{*}(X))}{\varepsilon})\right)$,
where $d^{*}(X)$ is the doubling dimension222The doubling dimension is defined
as $d^{*}(X)=\log_{2}\lambda$, where $\lambda\geq 1$ (doubling constant) is
the smallest number such that at most $\lambda$ balls of half radius are
needed to cover every ball in $X$. of $X$.

This is an exact deterministic answer to our search for an ideal encoder. Such
a map can immediately be extended to the whole input space using Kirzbraun’s
theorem. To show latent space consistency, first observe that given a metric
$\Omega$ and encoder $E$, the realized latent loss turns out to be
$\Omega(E_{\\#}\hat{\mu}_{n},\rho)$. It can be fragmented into the following
parts based on the independent sources of variation:

$\Omega(E_{\\#}\hat{\mu}_{n},\rho)\leq\underbrace{\Omega(E_{\\#}\hat{\mu}_{n},\widehat{(E_{\\#}\mu)}_{m})}_{\text{Information
dissipated}}+\underbrace{\Omega(\widehat{(E_{\\#}\mu)}_{m},\rho)}_{\text{Estimation
error}}.$ (3)

While a suitably chosen IPT takes care of the first part, the latter embodies
the error committed trying to estimate $\rho$ using samples from the encoded
law. In a lossless encoding ($E_{\\#}\mu\>{=}_{d}\>\rho$), it will boil down
to the usual estimation error. Otherwise, one might be left with a surplus
error due to the discrepancy between $\widehat{(E_{\\#}\mu)}_{m}$ and a
certain $\hat{\rho}_{m}$. Chakrabarty and Das [18] give a non-asymptotic upper
bound to the same based on the empirical Yatracos minimizer of
$\hat{\rho}_{m}$ (Theorem 1). In the process, they assume the finiteness of
VC-dim$[\mathcal{Y}(\cdot)]$ corresponding to $\mathcal{P}(\mathcal{Z})$.
Before stating such results, let us take a quick look at the ‘lossless’
setting itself.

######  Remark 6 (Lossless encoding in WAEs).

This occurs only when $E$ is an exact measure preserving map. Since we allow
the distributions to have densities, the idea translates to having a change of
variables given as
$\int_{\Omega_{z}}p_{\rho}(z)dz=\int_{\Omega_{x}}(p_{\rho}\circ
E)(x)\>[J_{E}(x)]dx$, where in general, $J_{E}(x)$ is the Clarke differential
or generalized Jacobian at $x$ [80] 333See the exact form the transported
density achieves under the co-area formula in McCann and Pass [81], Section 2.
In general cases concerning transformations between variables, the surplus
multiplicand is rather
$\textrm{vol}[J_{E}(x)]\mathrel{\mathop{\ordinarycolon}}=$ product of singular
values of the $k\times d$ Jacobian matrix $J_{E}$ [82].. This boils down to
the more familiar $\mathinner{\\!\left\lvert\textrm{det}(\nabla
E(x))\right\rvert}$, given
$E\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ is
injective on its domain and continuously differentiable. Consequently, our
desired distribution $p_{\rho}$ turns out to be the density corresponding to a
$k$-marginal of $E_{\\#}\mu$ 444Such transformations can merely be of the
Rosenblatt type. Given that $p_{\rho}\in\mathcal{C}^{m_{z}}_{L}(\Omega_{z})$,
for some $m_{z}\in\mathbb{R}_{>0}$; $E$ can be shown to be smooth in the sense
of Hölder-Zygmund [40].. The existence, let alone the regularity of such a map
is not automatically guaranteed. In case both $\mu$ and $\rho$ are nonatomic,
such that $\mu$ vanishes on all Lipschitz $(d-1)$-surfaces, there exists a
unique $E$ ($\mu$ a.e.), that offers a lossless encoding [83]. Brenier theorem
[84] argues the same under the additional assumption that the distributions
have finite variance. Though obtained under restrictive scenarios, a Brenier
map pushing forward the standard Gaussian measure to a uniformly log-concave
target distribution would be locally Lipschitz [85, 86]. While this belongs to
the class of possible encoders under a special case, it is rather challenging
to verify that minimizing the WAE loss attains such a solution. Moreover,
given a sample (semi-discrete) problem like ours, the optimal map is likely to
be discontinuous. Even if they do not, the injectivity will be sacrificed when
the supports are unbounded since they cannot be continuously embedded at the
same time.

Hence, we focus on finding the tolerable margins of losses incurred during
encoding instead. (3) becomes a platform on which the exploration of
deterministic upper bounds rests. To obtain an upper bound in the case of WAE-
GANs, first notice that given $\rho_{1},\rho_{2}\in\mathcal{P}(\mathcal{Z})$,
equipped with corresponding densities such that $\rho_{1}\ll\rho_{2}$,

$\displaystyle\textrm{JS}(\rho_{1},\rho_{2})\leq\left[\pi\ln\left(\frac{1}{\pi}\right)+(1-\pi)\ln\left(\frac{1}{1-\pi}\right)\right]d_{\textrm{TV}}(\rho_{1},\rho_{2})\leq\ln(2)\>d_{\textrm{TV}}(\rho_{1},\rho_{2})$

(this form is also called the Information Transmission Rate [87]),
$0\leq\pi\leq 1$. While easier to calculate, it is often technically
challenging to deal with JS from a density estimation perspective. By
convention, it is assigned value $+\infty$ in case the underlying
distributions do not have densities, and as a result, does not metrize
$\mathcal{P}(\mathcal{Z})$ in general. However, when taken under square root,
JS follows the triangle inequality [88]. Now, given a Lipschitz encoder $E$,
let us consider the realized latent loss under JS-divergence due to the RIK
estimator $\tilde{\mu}_{n}$ (as discussed in Theorem 3.3), defined as
$\frac{d\tilde{\mu}_{n}}{dx}=\frac{1}{nh^{d}}\sum^{n}_{i=1}K\left(\frac{x}{h},\frac{x_{i}}{h}\right)$,
$x\in\Omega_{x}$ where $h_{n}\rightarrow 0$. Fragmenting the same based on
unique sources of variation yields,

$\displaystyle
f(\pi)^{-1}\>\textrm{JS}(E_{\\#}\tilde{\mu}_{n},\rho)-\Delta_{E,n}\leq
d_{\textrm{TV}}\left(E_{\\#}\tilde{\mu}_{n},\widehat{(E_{\\#}\mu)}_{n}\right)+d_{\textrm{TV}}\left(\widehat{\rho}_{n},\rho\right),$
(4)

given that
$\Delta_{E,n}=d_{\textrm{TV}}(\widehat{(E_{\\#}\mu)}_{n},\widehat{\rho}_{n})$,
which essentially (in asymptotic regime) determines how much latent
regularization can be tolerated. As such, the choice of the optimal encoder
must be the one that minimizes $\Delta_{E,n}$. It coincides with the minimum
distance estimator (Devroye and Lugosi [54], Theorem 6.4) of $\rho$ amongst
encoded candidates
($E_{n}^{*}=\operatorname*{argmin}d_{\textrm{TV}}(\widehat{(E_{\\#}\mu)}_{n},\widehat{\rho}_{n})$).
Since the other error terms shrink arbitrarily asymptotically
($n\rightarrow\infty$ with fixed $d,k$), given $t\in\mathbb{R}_{>0}$ (as in
2), we only need to ensure that $\Delta_{E_{n}^{*},n}<t$.

######  Remark 7 (Cost to the Scheffé Tournament).

Substituting the plug-in estimates $\widehat{(E_{\\#}\mu)}_{n}$ and
$\widehat{\rho}_{n}$ with smoother alternatives might be computationally
beneficial. A successful search for the Scheffé tournament-winning encoder
takes quadratic time [89]. Instead, let us consider estimators
$\widetilde{(E_{\\#}\mu)}$ and $\tilde{\rho}$ respectively, such that
$\frac{d\widetilde{(E_{\\#}\mu)}}{dz},\frac{d\tilde{\rho}}{dz}\in
L_{1}(\mathbb{R}^{k})$. The benefit is rooted in viewing the problem from an
OT standpoint. Let us write,

$\displaystyle
2d_{\textrm{TV}}\left(\widetilde{(E_{\\#}\mu)}_{n},\tilde{\rho}_{n}\right)=\mathinner{\\!\left\lVert\widetilde{(E_{\\#}\mu)}_{n}-\tilde{\rho}_{n}\right\rVert}_{1}$
$\displaystyle\leq\underbrace{\mathinner{\\!\left\lVert\widetilde{(E_{\\#}\mu)}_{n}-K_{h}(\widetilde{(E_{\\#}\mu)}_{n})\right\rVert}_{1}}_{\text{(i)}}+\underbrace{\mathinner{\\!\left\lVert
K_{h}(\widetilde{(E_{\\#}\mu)}_{n})-K_{h}(\tilde{\rho}_{n})\right\rVert}_{1}}_{\text{(ii)}}+\underbrace{\mathinner{\\!\left\lVert
K_{h}(\tilde{\rho}_{n})-\tilde{\rho}_{n}\right\rVert}_{1}}_{\text{(iii)}},$
(5)

where given $\rho_{1}\in\mathcal{P}(\mathcal{Z})$,
$K_{h}(\rho_{1})=\int_{\Omega_{z}}K_{h}(.,y)\rho_{1}(dy)=\frac{1}{h^{k}}\int_{\Omega_{z}}K(\frac{.}{h},\frac{y}{h})\rho_{1}(dy)$
defines the convolution with RI kernel $K$. Also,
$\frac{y}{h}=(\frac{y_{1}}{h},...,\frac{y_{k}}{h})^{{}^{\prime}}$, for $h>0$.
The terms (i) and (iii) both $\rightarrow 0$ as $h\rightarrow 0$ (Giné and
Nickl [39], Proposition 4.3.31). On the other hand,

$\displaystyle\mathinner{\\!\left\lVert
K_{h}(\widetilde{(E_{\\#}\mu)}_{n})-K_{h}(\tilde{\rho}_{n})\right\rVert}_{1}$
$\displaystyle\leq\int\Big{\\{}\frac{1}{h^{k}}\int\mathinner{\\!\left\lvert
K(\frac{z}{h},\frac{y}{h})-K(\frac{z}{h},\frac{y^{{}^{\prime}}}{h})\right\rvert}dz\Big{\\}}d\Pi(y,y^{{}^{\prime}})$
(6) $\displaystyle=\int\Bigg{\\{}\frac{\int\mathinner{\\!\left\lvert
K(z^{{}^{\prime}},\frac{y}{h})-K(z^{{}^{\prime}},\frac{y^{{}^{\prime}}}{h})\right\rvert}dz^{{}^{\prime}}}{\mathinner{\\!\left\lVert
y-y^{{}^{\prime}}\right\rVert}}\Bigg{\\}}\>\mathinner{\\!\left\lVert
y-y^{{}^{\prime}}\right\rVert}d\Pi(y,y^{{}^{\prime}})$
$\displaystyle\lesssim\frac{1}{h}\int\mathinner{\\!\left\lVert
y-y^{{}^{\prime}}\right\rVert}d\Pi(y,y^{{}^{\prime}}),$ (7)

where (6) is due to Jensen’s inequality and $\Pi$ denotes a coupling between
$\widetilde{(E_{\\#}\mu)}_{n}$ and $\tilde{\rho}_{n}$. The invariance of $K$
implies the inequality (7), which holds for all such measure couples. Hence,
given $c_{z}\equiv L_{2}$, the quantity (ii)
$\lesssim\frac{1}{h}d_{\mathcal{L}_{c_{z}}^{1}}(\widetilde{(E_{\\#}\mu)}_{n},\tilde{\rho}_{n})$.
As such, to obtain an optimal encoder $E_{n}^{*}$— achieving latent
consistency— it is sufficient to compute
$\Delta^{{}^{\prime}}_{E_{n}^{*},n}=\inf_{E}d_{\mathcal{L}_{c_{z}}^{1}}(\widetilde{(E_{\\#}\mu)}_{n},\tilde{\rho}_{n})$
instead. This is highly maneuverable computationally due to the sheer
attention the problem has received in recent years. One can achieve a
complexity of $\tilde{O}(\frac{n^{\frac{9}{4}}}{t}\wedge\frac{n^{2}}{t^{2}})$
[90], even beyond what Sinkhorn’s algorithm offers.

Inequality (4) provides a clear pathway to a non-asymptotic upper bound to the
realized latent loss in a WAE-GAN setup. Given Assumptions 1 and 2, we begin
with $\tilde{\mu}_{n}$, an RIK density estimate (strongly invariant) of $\mu$
based on bandwidth $h\equiv h_{n}\rightarrow 0$ as $n\rightarrow\infty$, such
that $\frac{nh^{d}_{n}}{\mathinner{\\!\left\lvert\log
h^{d}_{n}\right\rvert}}\rightarrow\infty$. Corollary 3.5 implies the existence
of a positive constant $E_{1}^{{}^{\prime\prime}}$ such that

$d_{\textrm{TV}}\left(g_{\\#}\tilde{\mu}_{n},\widehat{(g_{\\#}\mu)}_{n}\right)\geq
t+\mathcal{O}(h^{m_{x}}\vee\sqrt{v_{z}}n^{-\frac{1}{2}})$ (8)

holds with probability $\leq
E_{1}^{{}^{\prime\prime}}\exp\left\\{-E_{3}(1\wedge h^{d})nt^{2}\right\\}$,
whenever for $t\geq\sqrt{\frac{\mathinner{\\!\left\lvert\log
h_{n}\right\rvert}}{nh^{d}_{n}}}$ [91] and
$g\in\mathscr{F}_{L}(\mathcal{X},\mathcal{Z})$. This eventually determines the
rate associated with the probabilistic statement

$\textrm{JS}(g_{\\#}\tilde{\mu}_{n},\rho)-f(\pi)\sup_{\rho^{\otimes
n}}\Delta_{L,n}=o_{\mathbb{P}}(1),$

obtained as a consequence of (4) and assuming VC-
dim$[\mathcal{Y}(\mathcal{P}(\mathcal{Z}))]=v_{z}<\infty$. Here,
$\Delta_{L,n}=\inf_{g\in\mathscr{F}_{L}(\mathcal{X},\mathcal{Z})}d_{\textrm{TV}}(\widehat{(g_{\\#}\mu)}_{n},\widehat{\rho}_{n})$
and the supremum is taken over naive estimators constructed based on $n$
replicates from $\rho$. If one employs instead a NN-based encoder
$\phi\in\Phi(W,L)^{k}_{d}$ according to our previous prescriptions— for
example Corollary 3.9 (i) or (ii)— an additional estimation error is duly
incurred. This, along with the realized $\Delta_{\Phi,n}$ contributes to the
extent of tolerable latent loss.

######  Remark 8.

There are some interesting implications in The WAE-GAN regime if along with
Assumption 2, there exists $m_{z}\in\mathbb{R}_{>0}$, such that
$p_{\rho}\in\mathcal{C}^{m_{z}}_{L^{{}^{\prime}}}(\Omega_{z})$,
$L^{{}^{\prime}}>0$. Observe that, due to its definition

$\displaystyle\textrm{JS}(E_{\\#}\tilde{\mu}_{n},\rho)$
$\displaystyle=\pi\textrm{KL}\left(E_{\\#}\tilde{\mu}_{n}\Big{|}\mathcal{M}_{E_{\\#}\tilde{\mu}_{n},\rho}(\pi)\right)+(1-\pi)\textrm{KL}\left(\rho\Big{|}\mathcal{M}_{E_{\\#}\tilde{\mu}_{n},\rho}(\pi)\right)$
$\displaystyle\geq\frac{1}{2}\left[\pi
d^{2}_{\textrm{TV}}\left(E_{\\#}\tilde{\mu}_{n},\mathcal{M}_{E_{\\#}\tilde{\mu}_{n},\rho}(\pi)\right)+(1-\pi)d^{2}_{\textrm{TV}}\left(\rho,\mathcal{M}_{E_{\\#}\tilde{\mu}_{n},\rho}(\pi)\right)\right]$
(9)
$\displaystyle=\frac{1}{2}\pi(1-\pi)d^{2}_{\textrm{TV}}\left(E_{\\#}\tilde{\mu}_{n},\rho\right)\geq\frac{1}{8}\pi(1-\pi)\mathinner{\\!\left\lVert
E_{\\#}\tilde{\mu}_{n}-\rho\right\rVert}^{2},$ (10)

where we define the the mixture as
$\mathcal{M}_{E_{\\#}\tilde{\mu}_{n},\rho}(\pi)=\pi
E_{\\#}\tilde{\mu}_{n}+(1-\pi)\rho$. The step (9) is due to Pinsker’s
inequality. We reach (10) using

$\displaystyle
d_{\textrm{TV}}\left(E_{\\#}\tilde{\mu}_{n},\mathcal{M}_{E_{\\#}\tilde{\mu}_{n},\rho}(\pi)\right)$
$\displaystyle=\sup_{\omega\in\sigma(\mathcal{Z})}\mathinner{\\!\left\lvert
E_{\\#}\tilde{\mu}_{n}(\omega)-\pi
E_{\\#}\tilde{\mu}_{n}(\omega)-(1-\pi)\rho(\omega)\right\rvert}$
$\displaystyle=(1-\pi)\sup_{\omega\in\sigma(\mathcal{Z})}\mathinner{\\!\left\lvert
E_{\\#}\tilde{\mu}_{n}(\omega)-\rho(\omega)\right\rvert}=(1-\pi)d_{\textrm{TV}}\left(E_{\\#}\tilde{\mu}_{n},\rho\right).$

Typically, the value of $\pi$ is taken to be $1/2$. Now,

$\mathinner{\\!\left\lVert
E_{\\#}\tilde{\mu}_{n}-\rho\right\rVert}^{2}\geq\inf_{\tilde{\rho}_{n}}\mathinner{\\!\left\lVert\tilde{\rho}_{n}-\rho\right\rVert}^{2},$

where the infimum is taken over the class of RIK density estimates based on
$n$ i.i.d. samples from $p_{\rho}$. Such estimators, under the $L_{2}$ loss
tend to have the optimal convergence rate, i.e.
$\inf_{\tilde{\rho}_{n}}\mathbb{E}\mathinner{\\!\left\lVert\tilde{\rho}_{n}-\rho\right\rVert}^{2}\gtrsim
n^{-\frac{2m_{z}}{2m_{z}+k}}$ (Van der Vaart [92], Theorem 24.4). As such,
this gives us a sharp lower bound for latent performance.

Since the latent distribution embodies the hidden representation in input
images, it must also remain invariant to certain deformations. For example,
latent codes corresponding to an image of a lesion and its rotated counterpart
($\textrm{SO}(k)$) should ideally appear equiprobably. Generative models
achieve this by ensuring group symmetry in the target space [93]. Now, given a
group $\Sigma$ (an ordered pair of a nonempty set and a binary operation,
satisfying the group axioms), a group action $\varphi$ on $\mathcal{Z}$ is an
automorphism defined as
$\varphi_{\sigma}=\varphi(\sigma,\cdot)\mathrel{\mathop{\ordinarycolon}}\mathcal{Z}\rightarrow\mathcal{Z}$,
for all $\sigma\in\Sigma$, also satisfying
$\varphi_{\sigma_{1}}\circ\varphi_{\sigma_{2}}=\varphi_{\sigma_{1}\cdot\sigma_{2}}$,
$\forall\sigma_{1},\sigma_{2}\in\Sigma$. The following definition gives such a
specific characterization to $\rho$.

######  Definition 3.14 (Invariant Distributions).

Given a group $\Sigma$, the class of $\Sigma$-invariant probability
distributions on $\mathcal{Z}$ is defined as

$\mathcal{P}_{\Sigma}(\mathcal{Z})=\\{\mathbb{P}\in\mathcal{P}(\mathcal{Z})\mathrel{\mathop{\ordinarycolon}}\mathbb{P}=(\varphi_{\sigma})_{\\#}\mathbb{P},\forall\sigma\in\Sigma\\}.$

Throughout the paper, we only consider finite groups, i.e.
$\mathinner{\\!\left\lvert\Sigma\right\rvert}<\infty$. This makes the
representation of the underlying space under group actions much easier. To
that end, let us introduce the fundamental domain
$\mathcal{Z}_{0}\subset\mathcal{Z}$, which is defined such that the subsets
$\sigma\mathcal{Z}_{0}$, $\sigma\in\Sigma$ form a locally finite cover of
$\mathcal{Z}$ without sharing common interior points. This translates to
saying that there exists a unique $z_{0}\in\mathcal{Z}_{0}$ corresponding to
each $z\in\mathcal{Z}$ such that $z=\sigma z_{0}$ [94]. In order to adapt to
the estimation of $\Sigma$-invariant latent distributions, we make additional
assumptions for the underlying kernels in MMDs.

######  Assumption 3 (Group Invariant Kernels).

The kernel $\kappa(\cdot,\cdot)$ satisfies
$\forall\sigma(\neq\textrm{id})\in\Sigma$

  1. (i)

Given $z,z^{{}^{\prime}}\in\mathcal{Z}$, $\kappa(\sigma z,\sigma
z^{{}^{\prime}})=\kappa(z,z^{{}^{\prime}})$, and

  2. (ii)

There exists $0<\varsigma_{\kappa,\Sigma}<1$ such that $\kappa(\sigma
z,z)\leq\varsigma_{\kappa,\Sigma}C_{\kappa}$, where $z\in\mathcal{Z}_{0}$.

######  Theorem 3.15 (Latent Space Consistency in WAE-MMD under Invariance).

Let, $\rho\in\mathcal{P}_{\Sigma}(\Omega_{z})$ such that
$\mathinner{\\!\left\lvert\Sigma\right\rvert}<\infty$. Also, let
$\kappa(\cdot,\cdot)$ be strongly invariant satisfying Assumption 3 such that
$\sup_{z\in\Omega_{z}}\kappa(z,z)\leq C_{\kappa}$, for $C_{\kappa}>0$. Then,
there exists a probabilistic encoder $\phi\in\Phi(W,L)^{k}_{d}$, based on ReLU
activations with $W=\mathcal{O}(d\lfloor N_{1}^{\frac{1}{d}}\rfloor\vee
N_{1}+1)$ and $L=\mathcal{O}(N_{2})$ such that given $\delta>0$, we have with
probability $1-\delta$

$\displaystyle d_{\mathcal{H}_{\kappa}}$
$\displaystyle\left(\phi_{\\#}\hat{\mu}_{n},\rho\right)-\sqrt{C_{\kappa}}\sup_{\rho^{\otimes
n}}\Delta_{\Phi,n}\leq\sqrt{c_{g,n}L}\left(\frac{\max\\{B^{2}_{x},4C_{\kappa}[1+\varsigma_{\kappa,\Sigma}\left(\mathinner{\\!\left\lvert\Sigma\right\rvert}-1\right)]\\}}{2n}\ln{(\frac{2}{\delta})}\right)^{\frac{1}{4}}$
$\displaystyle+\mathcal{O}(\sqrt{c_{g,n}}(d^{2}n)^{-\frac{1}{2d}})+\sqrt{\frac{2C_{\kappa}}{n}}\left[1+\sqrt{\frac{1+\varsigma_{\kappa,\Sigma}\left(\mathinner{\\!\left\lvert\Sigma\right\rvert}-1\right)}{\mathinner{\\!\left\lvert\Sigma\right\rvert}}}\right]+\mathcal{O}(\sqrt{dD_{n}}N_{1}^{-\frac{2}{d}}N_{2}^{-\frac{2}{d}}),$

where both $D_{n}$ and $c_{g,n}$ are sequences based on $n$ that $\searrow 0$
almost surely as $n\rightarrow\infty$.

######  Remark 9 (Mitigating Curse of Dimensionality).

The term contributing to a slower convergence rate (second on the RHS) due to
its dependence on $d$ is rooted in the estimation error under the Wasserstein
metric (see proof). While we do not allow the input dimension $d$ to grow as a
function of $n$, it being inherently large degrades the sharpness of the non-
asymptotic bound. In search of a remedy, we recall the solution Chakrabarty
and Das [18] resorted to, namely the 1-upper Wasserstein dimension
($d^{*}_{1}$). It is typically smaller compared to the Minkowski–Bouligand
dimension. However, in case $\mu$ is essentially supported on a ‘latent’
regular space, e.g. a compact $d^{{}^{\prime}}$-dimensional differentiable
manifold ($d^{{}^{\prime}}<d$), we have $d^{*}_{1}=d^{{}^{\prime}}$ [95]. This
suits our discussion since such a phenomenon regularly occurs in image
datasets [24]. The definition of $d^{*}_{1}$ goes as follows

$\displaystyle
d^{*}_{1}(\mu)=\inf\\{s\in(2,\infty)\mathrel{\mathop{\ordinarycolon}}\limsup_{\varepsilon\rightarrow
0}\frac{\mathcal{N}_{\varepsilon}(\mu,\varepsilon^{\frac{s}{s-2}})}{-\log(\varepsilon)}\leq
s\\},$

where $\mathcal{N}_{\varepsilon}$ denotes the covering number. Now, if
$s>d^{*}_{1}(\mu)$, the second term on the right hand of the inequality in
Theorem 3.15 can be replaced by
$\mathcal{O}(\sqrt{c_{g,n}}n^{-\frac{1}{2s}})$. We also point out that the
other term carrying $d$ in the exponent does not contribute to asymptotic
rates since the encoders constructed are usually of fixed proportions. Since
the upper bound becomes $o(1)$, using the Borel-Cantelli lemma, the latent
WAE-MMD error deviated from $\sqrt{C_{\kappa}}\sup_{\rho^{\otimes
n}}\Delta_{\Phi,n}$ vanishes almost surely.

###  3.2 Simulations

To validate our findings empirically, we carry out experiments on both real
and synthetic data [Fig 1]. The existing data set we work on is MNIST,
consisting of 70,000 2D images of hand-written digits. The other data set,
‘Five-Gaussian’ is a collection of 50,000 observations, drawn independently at
random out of five trivariate Gaussians with unit dispersion and mean at five
vertices of a unit cube. We run both WAE-GAN and WAE-MMD to reconstruct
observations from the two data sets. All the experiments were carried out on
an RTX 3090 GPU.

Figure 1: The (a) MNIST and (b) Five-Gaussian data sets.

Five-Gaussian: The encoders we use in the case of Five-Gaussian data, map the
points to a two-dimensional latent space. The first kind we deploy is $4$-deep
and is based on ReLU activation. The last layer uses an additional rescaling
to span the target support and mitigate zero-inflation. To suit our
theoretical specifications, we experiment with diverse latent distributions.
Namely, Bivariate standard Gaussian, and the classes of bivariate
distributions having Beta and Exponential marginals respectively. We call them
Beta and Exponential copulas by somewhat abusing the terminology. This way we
encompass unbounded supports, multimodality, and skewed densities. Firstly, we
train the model based on the entire sample ($n=50,000$) to obtain the nearest
estimate of the population loss. Our goal now is to observe the propagation of
the losses as we gradually increase the sample size
$n=(1000,3000,5000,\cdots,50,000)$, drawn uniformly at random. Our findings
from 20 runs corresponding to each $n$ are given as follows.

(a) Gaussian

(b) Beta copula

(c) Exponential copula

(d) Mean and Variance of latent loss

Figure 2: Latent loss under Jensen-Shannon divergence and ReLU encoders. The
Lagrangian weight assigned to the latent space, as given in 1, is taken to be
$\lambda=0.2$. We consider both Gaussian and Exponential marginal densities
standard. The parameters for Beta marginals are taken as $(0.5,0.8)$.

The illustrations show that for all three latent laws, the sample losses
approach their population counterpart with diminishing variance. In search of
the sharpest asymptotic rate associated— even beyond the theoretically
achievable $\mathcal{O}(n^{-\frac{1}{2}})$— we observe the movement of the
loss multiplied by $n$. In our experiments [Fig 3], such a sequence also tends
to converge to a small constant, namely the population error margin. This is
an empirical guarantee to the impression Asatryan et al. [40] [Remark 4.6] had
($\hat{d}_{\textrm{JS}}\sim n^{-1}$) in a parametric GAN generation. However,
using a GroupSort activated encoder (grouping size $2$) one may observe the
approximate rate of $\mathcal{O}(n^{-\frac{1}{2}})$ in diminishing MMD losses
[Fig 16 (b), (c)]. The choice of regularizing parameter $\lambda$ is chosen
based on a trade-off between quality reconstruction and latent performance.

(a) Gaussian

(b) Beta

(c) Exponential

Figure 3: Propagation of sample corrected ($\times n$) latent JS loss under
ReLU encoders.

We follow the same experimental protocol for WAE-MMD [Fig 4], taking the
latent distribution as bivariate standard Gaussian. Here also, a similar trait
is noticed. The observed MMD losses gradually decrease to their population
counterpart with diminishing variability. The rate of convergence however,
becomes comparable to $\mathcal{O}(n^{-\frac{1}{2}})$, attesting the
theoretical result. While the latent loss— asymptotically at the least— moves
close to nullity, the bin estimates corresponding to the target and the
encoded law must differ. This discrepancy, as we have already discussed, is
rooted in the information preserved.

(a)

(b)

Figure 4: Propagation of (a) sample MMD losses and (b) sample corrected
($\times n^{\frac{1}{2}}$) MMD losses, trained with the Lagrangian parameter
$\lambda=0.2$ using ReLU encoders.

We study the concentration of encoded bins in contrast with the latent ones
over regular intervals of 200 training epochs. It is fascinating to observe
the rearrangement of density as the losses slowly diminish. We present a
detailed commentary on the same in the Appendix [Fig 13, 14]. Here instead, we
show the encoded estimates after the completion of 2000 epochs [Fig 5]. The
information retention can be readily identified from the high-density areas
representing the distinct clusters. The quantile-quantile (QQ) plots [Fig
15(a)] between the empirical encoded distribution and the targets also tell
the same story. To check the extent of approximation, we also perform
multivariate goodness-of-fit tests (see Appendix Appendix: Experiments and
Simulations).

(a)

(b)

(c)

Figure 5: Bin estimates of ReLU-encoded (yellow) vs latent distribution (blue)
in case of the Five-Gaussian data. Under the JS loss, we observe (a) Beta
$(0.5,0.8)$ and (b) standard Exponential copula, (c) shows standard Gaussian
under MMD loss. (Effective range of values scaled to aid visualization)

For checking the efficacy of GroupSort activations in encoding, we repeat the
experiment in a WAE-GAN setup [Fig 6]. The regularization remains at
$\lambda=0.2$ and grouping size is taken as $2$ (OPLU). Quite similar to
previous observations, the losses tend to decrease at a familiar rate.

(a) Gaussian

(b) Beta

(c) Exponential

Figure 6: Latent JS loss under Groupsort encoders of grouping size 2.

MNIST: The individual images in MNIST are of size $(28\times 28)$. In
vectorized form, the input observations are reduced to $d=512$. Here also we
deploy a $4$-deep ReLU encoder with layers of width 512-256-128-64$=k$. During
decoding, the output tensor is reshaped to have a size $(\textrm{batch
size},1,28,28)$, which in turn enables us to calculate the reconstruction
loss. Keeping in mind the high dimensionality of the latent space, we only
consider a multivariate standard Gaussian target. The findings from training
runs on both WAE-GAN and WAE-MMD with regularization $\lambda=0.2$ are
obtained as in Fig 7.

(a)

(b)

Figure 7: Latent (a) JS and (b) MMD loss for MNIST data set with Gaussian
targets.

##  4 Reconstruction Consistency

With a clearer understanding of WAEs’ performance towards meeting the
constraint it was formulated under, we move on to its main objective. If we
position ourselves along the flow of information— first through the encoder
and now at the footsteps of the decoder— we have ourselves a density
estimation task in higher dimensions. This is typical of inverse models and
the underlying goal is to utilize the encoded information at one’s disposal to
reach as close as to $\hat{\mu}_{n}$. Needless to say, it comes as a
consequence of the error $W_{c_{x}}^{1}(\hat{\mu}_{n},(D\circ
E^{*}_{n}(t))_{\\#}\hat{\mu}_{n})$ being minimized, given the optimal encoder
$E^{*}_{n}(t)$ incurring latent loss $\leq t$.

In spirit, the role of the decoder is somewhat similar to a generative map.
The aspects in which they differ from those in a GAN architecture are mainly
twofold. Firstly there is no dynamic critic in the form of a discriminator to
guide its learning. The role is taken up by $\mathcal{L}_{c_{x}}^{1}$ only.
However, on the upside, while the latent distribution in GANs is non-
informative, to begin with, WAEs have latent laws with input information
‘preserved’. During decoding, this very information needs to be used with
utmost efficacy. Since the resemblance between spaces of different dimensions
in a sample problem lies in the local geometry based on pairwise distances
between samples, Chakrabarty and Das [18] identify quasi-isometries as ideal
decoders. The definition provided in their article however indicates a bi-
Lipschitz characterization. In general, $\mathbb{R}^{k}$ is not quasi-
isometric to $\mathbb{R}^{d}$, $d>k$. However, there may exist bi-Lipschitz
maps from $\Omega_{z}\rightarrow\mathbb{R}^{l}$, $l\leq d$, which can thus be
used to form outer extensions mapping
$\mathbb{R}^{k}\rightarrow\mathbb{R}^{d}$ [96]. Such extensions typically
preserve distortions up to constant factors and as a result, do not depreciate
the asymptotic behavior of estimates post-translation. Another technique of
ensuring bi-Lipschitzness is to search for regular555A Lipschitz map
$f\mathrel{\mathop{\ordinarycolon}}(\mathcal{Z},c_{z})\rightarrow(\mathcal{X},c_{x})$
is said to be regular if there exists a constant $C>0$ such that given any
ball $B$ in $\mathcal{Z}$, $f^{-1}(B)$ can be covered by at most $C$ balls,
each of radius $C.\textrm{rad}(B)$ [97]. maps
$f\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{k}\rightarrow\mathbb{R}^{d}$.
These in turn make the restriction of $f$ to $\overline{\Omega}_{z}$ (closure)
to be BL. While bi-Lipschitz transforms acting as decoders automatically
enforce non-degeneracy in reconstructed signals, to obtain upper bounds of
associated errors it is sufficient to have Lipschitz decoders. Most
theoretical studies on GANs tend to impose this restriction on generators. The
existence of such a benchmark Lipschitz transform, however, is readily
guaranteed if encoders are considered according to Lemma 3.12. This also
supports the practical convention of building the decoder as exactly the
inverse of $E^{*}(t)$.

We, on the contrary, prescribe constructing a decoder map that utilizes the
information encapsulated in the latent law wholly and efficiently. To
demonstrate the notion, let us fragment the reconstruction loss, given an
encoder $E$, as follows

$W_{c_{x}}^{1}(\mu,(D\circ E)_{\\#}\hat{\mu}_{n})\leq W_{c_{x}}^{1}((D\circ
E)_{\\#}\hat{\mu}_{n},D_{\\#}\rho)+\underbrace{W_{c_{x}}^{1}(D_{\\#}\rho,\hat{\mu}_{n})}_{\text{Decoder
Translation Error}}+W_{c_{x}}^{1}(\hat{\mu}_{n},\mu).$ (11)

Observe that the first term on the right is essentially the propagated
disagreement at the latent space. If the metric, measuring the discrepancy
follows a data-processing inequality, we can expect a non-asymptotic upper
bound of the same order as that obtained on the latent error. As such, $D$
must be constructed keeping in mind the sole aim of minimizing the semi-
discrete translation error. The following lemma provides the backbone of the
construction.

######  Lemma 4.1 (Yang et al. [98]).

Let $\nu$ be an univariate absolutely continuous distribution and
$\hat{\mu}_{n}\in\mu^{\otimes n}$. Given $W\geq 7d+1$ and $L\geq 2$, there
exists a NN transform based on ReLU activation
$\phi^{{}^{\prime}}\in\Phi(W,L)_{1}^{d}$ such that $\forall\varepsilon>0$

$W_{c_{x}\equiv
L_{1}}^{1}(\hat{\mu}_{n},\phi^{{}^{\prime}}_{\\#}\nu)\leq\varepsilon,$

whenever
$n\leq\frac{W-d-1}{2}\lfloor\frac{W-d-1}{6d}\rfloor\lfloor\frac{L}{2}\rfloor+2$.

The transport hinted in the lemma is essentially a piecewise linear map,
Lipschitz continuous on bounded balls. The restriction on $n$ indicates the
number of breakpoints. Since the result holds for all probability measures
$\nu$ having densities, the only modification required in our case is
projecting $\rho$ onto $\mathbb{R}$, using linear maps beforehand. Given such
a linear map
$D_{0}\mathrel{\mathop{\ordinarycolon}}\Omega_{z}\rightarrow\mathbb{R}$, and
$\phi^{{}^{\prime}}$ according to lemma 4.1, the desired decoder is given by
$D=\phi^{{}^{\prime}}\circ D_{0}$. This operation also preserves the Lipschitz
continuity in the resultant decoder. Since there is no unique way of selecting
the linear transform, one may use instead a pooled distribution. As such,
$D=\phi^{{}^{\prime}}\circ\sum_{i=1}^{N_{3}}D_{i}\equiv\sum_{i=1}^{N_{3}}\phi^{{}^{\prime}}\circ
D_{i}$, where $D_{1},\cdots,D_{N_{3}}$ are individual linear maps aimed at
preserving different aspects of $\rho$. This is especially useful when
$\mathinner{\\!\left\lvert\Sigma\right\rvert}>1$. In this case also $D$ turns
out to be Lipschitz (since the property itself stems from individual
components and summation— that too without scaling— only changes the
associated constant). The following result formalizes our discussion.

######  Theorem 4.2 (Reconstruction Consistency in a Latent-Consistent WAE).

Given a margin of latent error $t>0$, let $E^{*}_{n}(t)$ be an optimal encoder
satisfying latent consistency under the metric $d_{\textrm{TV}}$. Then, there
exists a decoder $D\in\Phi(W,L)_{k}^{d}$, $d\geq 3$ based on ReLU activations,
with $W\geq 7d+1$ and $L\geq 3$ such that

$\displaystyle\mathbb{E}\left[W_{c_{x}}^{1}(\mu,(D\circ
E^{*}_{n}(t))_{\\#}\hat{\mu}_{n})\right]-\mathcal{O}(t)\lesssim
n^{-\frac{1}{d}},$

where $n=\mathcal{O}(\frac{W^{2}L}{d})$.

The theorem reveals the extent to which the realized latent loss can
potentially amplify during reconstruction. The corresponding excess error
always stays $\mathcal{O}(n^{-\frac{1}{d\vee 2}})$, with high probability
(using McDiarmid’s inequality). The result also allows for the formulation of
WAEs to be made general by replacing $W_{c_{x}}^{1}$ with $W_{c_{x}}^{p}$,
$1\leq p$. In such a case, the observation: $W^{p}(\mu_{1},\mu_{2})\leq
B^{\frac{p-1}{p}}_{x}W^{1}(\mu_{1},\mu_{2})^{\frac{1}{p}}$,
$\mu_{1},\mu_{2}\in\mathcal{P}(\mathcal{X})$ coupled with Theorem 4.2 provides
the regeneration guarantee.

######  Remark 10 (Reconstruction as a Consequence of Latent Consistency in
WAE-MMD).

Considering a latent error
$d_{\textrm{TV}}\left(E^{*}_{n}(t)_{\\#}\hat{\mu}_{n},\rho\right)\leq t$
unites both WAE models (WAE-GAN and WAE-MMD) in showing reconstruction
consistency based on the fact that TV is a natural upper bound to both JS and
MMD. However, if the bounded kernel $\kappa$ applied in a latent space (in
WAE-MMDs) is integrally strictly positive definite and follows strong
invariance, we have the partial inequality— given Assumption 1 and 2— as
follows

$\displaystyle W_{c_{x}}^{1}((D\circ E)_{\\#}\hat{\mu}_{n},D_{\\#}\rho)$
$\displaystyle\lesssim W_{c_{z}}^{1}(E_{\\#}\hat{\mu}_{n},\rho)$
$\displaystyle\leq
C_{\varepsilon}\>d_{\mathcal{H}_{\kappa}}\left(E_{\\#}\hat{\mu}_{n},\rho\right)+\varepsilon,$

for some $C_{\varepsilon}>0$ and $\forall\varepsilon>0$ (Modeste and Dombry
[66], Proposition 3.9). The first inequality is due to the Lipschitz
continuity of $D$. Now, if MMD, equipped with the same $\kappa$ is applied on
the input space as well, the same $D$ constructed so far satisfies
$d_{\mathcal{H}_{\kappa}}\left(D_{\\#}\rho,\hat{\mu}_{n}\right)<\varepsilon$,
$\forall\varepsilon>0$ (Yang et al. [98], Lemma 3.3). As a result, the right-
hand side of inequality (11) can be written entirely in terms of MMD. Hence,
Theorem 3.15 can be readily plugged in to obtain a deterministic upper bound
to the realized reconstruction error.

###  4.1 Simulations

We continue with the earlier experimental setup to provide empirical
validation. In fact, reconstruction outputs are obtained simultaneously with
latent results. Conforming to our previous prescription, we employ $4$-deep
decoders for the Five Gaussian data set. On the other hand, to reconstruct
observations corresponding to MNIST, we take a pragmatic approach while
choosing network widths ($k=$64-128-256-512). The final output tensor is
suitably reshaped to have the size of an image (batch size, 1, 28, 28).

(a) Gaussian

(b) Beta copula

(c) Exponential copula

Figure 8: Wasserstein reconstruction loss for the three latent distributions
under MMD using ReLU encoders. The penalization handed to the latent loss is
kept at $\lambda=0.2$.

The diminishing trait of error values with shrinking variance is evident in
Fig 8. The limiting margin of error where the sequence converges (for all
latent distributions under consideration) remains well below the tolerable
latent loss. This further attests to our theoretical bound. The optimizations
not only make the error eventually vanish, but also produce perceptually alike
samples [Fig 10]. Reconstruction errors corresponding to Five Gaussian
replicates in a WAE-MMD setup also tend to follow a convergence rate
$\mathcal{O}(n^{-\frac{1}{2}})$ [Fig 16]. This is a validation to the Remark
10, even under the deployment of GroupSort encoders. Reconstructions of MNIST
also result in photo-realistic copies of the input law [Fig 9]. The
corresponding errors exhibit sharply decaying behavior under both WAE
architectures.

(a)

(b)

(c) Reconstruction under latent JS loss

Figure 9: MNIST reconstruction error given Gaussian latent laws under (a) JS
and (b) MMD latent loss, using ReLU encoders. In (c), the odd rows hold the
input digits and the even ones are their reconstructed counterparts.

(a) Input data

(b) Gaussian

(c) Beta copula

Figure 10: Reconstructed samples ($n=10,000$) from Five Gaussian dataset under
JS latent loss for given latent distributions, using ReLU encoders after
$1500$ epochs.

##  5 Robustness to Distribution Shift

The quality of deep generative model outputs is often marred by contamination
in data. Images, and consequently WAEs are very much susceptible to such
adversarial corruption. In our recommendation, we have prioritized the free
flow of information through the model networks in a WAE. As such, a corrupted
set of input samples runs the risk of spoiling all the downstream tasks. Thus,
to get a comprehensive look at the machinery of WAE, one must test its innate
capability to preserve regeneration quality under the influence of
adversaries. Some of the well-recognized models in robust statistics include
Adaptive, Oblivious and the Huber contamination model [99, 100]. In Huber
contamination, instead of assuming independent replicates from the input
density $p_{\mu}$, it is assumed that there lies a probability $\epsilon>0$
that the sample comes from a contamination $p_{c}\in\mathcal{P}(\mathcal{X})$.
The density $p_{c}$ is independent of the input law and remains unknown. As
such, input observations

$X_{1},\cdots,X_{n}\sim\tilde{p}\mathrel{\mathop{\ordinarycolon}}=(1-\epsilon)p_{\mu}+\epsilon
p_{c}$ (12)

are what we have at hand. Under such a setup, Liu and Gao [101] showed that
given $p_{\mu}\in\mathcal{C}^{s}_{L}(\Omega_{x})$, kernel density estimates
incur euclidean losses
$\mathcal{O}(n^{-\frac{2s}{2s+1}}\vee\epsilon^{\frac{2s}{s+1}})$. The
underlying kernels are considered to be square-integrable and bounded, with
centered moments. This result is particularly motivating since given that
$D\circ E\approx\textrm{id}$ a.e., it gives us a bound on the regeneration
error for VAEs.

The regime we consider in our following discussion is closer to oblivious
contamination. We assume that the contaminated input distribution $\tilde{p}$
is such that $\mathbb{E}_{X\sim\tilde{p},Y\sim
p_{\mu}}c_{x}(X,Y)\leq\epsilon$, a more general notion compared to Huber.
Observe that, such a criterion automatically implies $1$-Wasserstein
contamination under metric $c_{x}$ [102]. No additional assumption on the
regularity of $\tilde{p}$ is assumed. The goal lies the same as before:
reconstructing $p_{\mu}$ based on an input estimator. In other words, in the
absence of additional regularization, we check for the extent of inherent
distributional robustness WAEs possess.

We have already seen that both the encoder and decoder, under careful
construction can follow Lipschitz continuity. As a result, their composition
behaves similarly. To generalize such composite maps, in this section, we
consider maps
$\mathcal{G}\mathrel{\mathop{\ordinarycolon}}\mathcal{X}\rightarrow\mathcal{X}$
which induce group actions. Now, if $G\in\mathcal{G}$ satisfies information
preservation, it is approximately equivalent to constructing an estimator
based on translated observations rather than translating a pre-constructed
estimator to approach $p_{\mu}$. As such, given replicates
$X_{1},\cdots,X_{n}\sim\tilde{p}$, we essentially need to look for upper
bounds to the loss
$W_{c_{x}}^{1}(\hat{p}_{n},p_{\mu})\lesssim\mathinner{\\!\left\lVert\hat{p}_{n}-p_{\mu}\right\rVert}_{1}$,
where $\hat{p}_{n}\in L_{1}(\mathbb{R}^{d})$ is based on
$\\{G(X_{i})\\}^{n}_{i=1}$.

To cope with the adversary, first, we modify the properties of the regularly
invariant kernels we utilized earlier. We call a kernel
$\kappa(x,y)\mathrel{\mathop{\ordinarycolon}}\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$
transformation invariant if given action $G\in\mathcal{G}$,
$\kappa(G(x),y)=\kappa(x,G^{-1}(y))$ [103]. The following theorem provides the
extent of WAEs’ resilience based on such kernel estimates.

######  Theorem 5.1 (Reconstruction Consistency under Contamination).

Let the contaminated distribution $\tilde{p}$ be such that
$\sup_{\Omega_{x}}\tilde{p}(x)<\infty$. Also, let the kernel $\kappa$ be
regular, translation invariant with respect to $L_{1}$ (3.7) and
transformation invariant which satisfies
$\int_{\mathcal{X}}\kappa^{2}(v,v-u)du<\infty$. Then, given any
$G\in\mathcal{G}$, a kernel density estimate $\hat{p}_{h}\equiv\hat{p}_{h,n}$
based on $\kappa$ satisfies

$\mathbb{E}\mathinner{\\!\left\lvert\hat{p}_{h}(0)-p_{\mu}(0)\right\rvert}\lesssim
n^{-\frac{m_{x}}{d+2m_{x}}}\vee\epsilon^{\frac{m_{x}}{2d+m_{x}}}.$

###  5.1 Simulations

During empirical validation, based on the diverse natures of their support,
the thickness of tails, and central tendencies, we select three potential
contaminating distributions: Standard Gaussian, Cauchy, and Dirichlet. While
the Five Gaussian data set is corrupted with the latter two, we apply all
three on MNIST. For utmost rigor in our experiments, we adopt an elaborate
contaminating regime. We not only vary the proportion of observations getting
corrupted but also regulate the extent of it. For example, there may be a set
of input observations $\\{X_{i}\\}_{i=1}^{n}$, out of whom
$\lfloor\frac{n}{2}\rfloor$ are replaced by $(0.8)X_{i}+(1-0.8)Y_{i}$, where
$\\{Y_{i}\\}^{i\in\mathscr{C}}$ are replicates from a contaminating law.
$\mathscr{C}$ denotes the indexes receiving the corruption,
$\mathinner{\\!\left\lvert\mathscr{C}\right\rvert}=\lfloor\frac{n}{2}\rfloor$.
We refer to the mixing proportion as level ($\alpha$). This regime generalizes
the entire contamination landscape in statistics. The specific choice of
parameters for Dirichlet is taken as $(5,3,5)$. Perhaps the most interesting
observation from our huge body of experiments is some of the near-accurate
reconstructions.

(a) Actual Data

(b) Contaminated Data

(c) Reconstructed Data

Figure 11: Reconstructed samples ($n=10,000$) from Five Gaussian dataset with
half the observations contaminated at level $0.2$, under JS latent loss. The
corrupting distribution is taken to be Dirichlet($5,3,5$).

In the case of the MNIST data set, even at a significant level of
contamination, the reconstruction errors continue to converge to a near-zero
value. The corresponding reconstructed samples are of remarkably sound
resolution, given the regenerative capability of WAEs in general [Fig 12]. We
also study the effect of varying $\alpha$ on reconstructed image quality [Fig
18], which hints at tolerable levels of contamination.

(a) Gaussian

(b) Cauchy

(c) Dirichlet

(d) Reconstruction under Cauchy contamination

Figure 12: Reconstruction errors incurred by a ReLU-induced WAE-MMD for MNIST,
under different contaminating distributions at level $0.2$. In all the
experiments, the latent distribution is kept standard Gaussian. In (d), the
first row represents contaminated samples (standard Cauchy at level $0.2$) and
the second row contains their reconstructed counterparts.

##  6 Discussion and Future Work

In this paper, we provide statistical guarantees regarding the concurrent
tasks Wasserstein autoencoders carry out, i.e. achieving both the latent and
reconstruction benchmark laws. Under probabilistic characterization of the
input data, we establish deterministic upper bounds to both losses. Our non-
parametric estimation approach caters to both WAE architectures, namely WAE-
MMD and WAE-GAN. In the process, we find out sufficient properties an encoder
must possess to become information preserving. This particular notion further
enables us to prescribe to a practitioner building an encoder, the
architectural specifications of an ideal neural network. Deployment of such a
network, in turn, aids the latter process of reconstruction. In a WAE-MMD
framework, we explore the sufficient conditions the deployed kernel needs to
satisfy to ensure latent consistency. We put to test our theoretical findings
in simulations based on real and synthetic data sets. The phenomenon of
information preservation in the latent space is fascinating to witness in the
flesh. The encoded distributions tend to maximize their alignment with the
latent target without losing the local geometric properties of the input.
Similarly, we recommend decoder frameworks that achieve near-perfect
regenerations. Such results are also substantiated by accompanying numerical
experiments that show sharply decaying losses over varying sample sizes.
Finally, in a density estimation setup, we test the degree of robustness WAEs
hold naturally, against distribution shifts without additional regularization.

We dedicate the rest of the section to pointing out possibilities our
theoretical framework spawns. The first question arises regarding the modes of
the distributions involved. Regenerated samples using a WAE are not known to
be plagued by ‘mode collapse’ as severely as a vanilla GAN output. However, it
is not uncommon for real data distributions ($\mu$) to have non-convex
support. In such a case, the OT (due to Brenier) map between the latent
distribution and $\mu$ will mostly be discontinuous. Moreover, NN-based
transforms often fail to universally approximate such discontinuous functions.
This may lead to significant mismatches between the supports of $\mu$ and
$D_{\\#}\rho$, however good the representative samples may be. As such, there
always lies an innate possibility of missing out on modes of $\mu$, even if
the effect is benign to the eye. The question also involves the role of
underlying divergences. In generative modeling, they are typically judged
based on their capacity of mode covering and mode seeking [104]. A mode-
covering divergence tends to prioritize the spread of masses to all target
modes and as a result, generates out-of-sample observations. While mode-
seeking distances avoid doing so, their conservative mass assignment leads to
the model missing out on one or several modes. Unfortunately, both TV (weakly
mode-seeking) and JS (uniformly mode-seeking) lean towards the later
characterization. As such, in case $\rho$ is multi-modal and there exists a
mismatch between the number of modes of $\mu$ (unknown) and $\rho$, WAEs
operate under a heightened risk of losing information on some modes. In our
non-parametric setup, we do not specify the modality of input and latent
distributions. This creates an interesting prospect to study the effect of
varying numbers of modes in $\rho$ on information preservation and
reconstruction. Future work may also look into the tolerable modality of input
laws in a WAE-MMD before mode collapse transcends benignity. This is
particularly intriguing since the mode-seeking properties of MMD remain
unexplored.

Another question that remains closely related to that regarding modes, is the
uniformity of samples over them. The model may also fail to recognize a mode
of $\mu$ if observations from it are vastly outnumbered. We have already seen
the denoising capability of WAEs. If the input data suffer such imbalance, it
is not improbable for the model to treat them as outliers, especially if the
modes are well-separated. As such, along with the modality, formulating
appropriate estimators that ensure proportionate participation and accurate
reconstruction under imbalance must be taken up further.

## Appendix: Technical Proofs

###### Proof of Lemma 2.4.

Let $E\mathrel{\mathop{\ordinarycolon}}\mathcal{X}\rightarrow\mathcal{Z}$ and
$D\mathrel{\mathop{\ordinarycolon}}\mathcal{Z}\rightarrow\mathcal{X}$ be
measurable maps— namely a encoder-decoder pair— that satisfy

$W_{c_{x}}^{1}(\mu,(D\circ E)_{\\#}\mu)+\lambda.\Omega(E_{\\#}\mu,\rho)=0,$
(13)

given $\lambda>0$. Since $\Omega$ is a divergence metric metrizing the
underlying class of probability distributions, it is implied that
$E_{\\#}\mu=\rho$. As such, we observe a lossless encoding in the population
sense. Now,

$\displaystyle E_{\\#}\mu=\rho\implies(D\circ
E)_{\\#}\mu=D_{\\#}\rho\overset{(1)}{\implies}\mu=D_{\\#}\rho,$

where ($1$) is due to (13). This hints towards an absolute information
preservation in reconstruction based on the fact that $D\circ
E=\textrm{id}_{X}$ a.e.

Given a probability space automorphism $\varphi$, $\forall A\in\mathcal{Z}$

$\rho(\varphi(A))=\rho(\varphi^{-1}(\varphi(A)))=\rho(A),$

since $\varphi^{-1}$ also becomes an automorphism. Hence,

$\displaystyle W_{c_{x}}^{1}(\mu,(D\circ\varphi)\circ(\varphi^{-1}\circ
E)_{\\#}\mu)+\lambda.\Omega((\varphi^{-1}\circ E)_{\\#}\mu,\rho)$
$\displaystyle=$
$\displaystyle\;W_{c_{x}}^{1}(\mu,(D\circ(\varphi\circ\varphi^{-1})\circ
E)_{\\#}\mu)+\lambda.\Omega(\varphi^{-1}_{\\#}(E_{\\#}\mu),\rho)$
$\displaystyle=$ $\displaystyle\;W_{c_{x}}^{1}(\mu,(D\circ
E)_{\\#}\mu)+\lambda.\Omega(\varphi^{-1}_{\\#}\rho,\rho)=0.$

As such, the encoder-decoder pair $(\varphi^{-1}\circ E,D\circ\varphi)$ is
also a zero-solution of the population loss function. Also, in case
$\varphi\neq\textrm{id}$, the pair clearly differs from $(E,D)$ a.e. ∎

###### Proof of Theorem 3.3.

Given a transform $g\in\mathscr{F}_{L}(\mathcal{X},\mathcal{Z})$, the
information dissipated can be decomposed using the triangle inequality as
follows

$\displaystyle
d_{\mathcal{H}}\left(g_{\\#}\tilde{\mu}_{n},\widehat{(g_{\\#}\mu)}_{m}\right)\leq
d_{\mathcal{H}}\left(g_{\\#}\tilde{\mu}_{n},g_{\\#}\mu\right)+d_{\mathcal{H}}\left(g_{\\#}\mu,\widehat{(g_{\\#}\mu)}_{m}\right).$
(14)

Here, $\tilde{\mu}_{n}$ denotes the RIK estimator (see Definition 3.7),
defined as
$\frac{d\tilde{\mu}_{n}}{dx}=\frac{1}{nh^{d}}\sum^{n}_{i=1}\kappa\left(\frac{x}{h},\frac{x_{i}}{h}\right)=\hat{p}_{h}(x)$,
$x\in\Omega_{x}$ where $h$ is the bandwidth. Also, since the kernels are
bounded, there exists $B>0$ such that
$\sup_{\Omega_{x}}\kappa(\cdot,\cdot)=B\leq 1$. In most cases, with choices of
kernels being distributions themselves, the modal values tend to satisfy this
criterion. Since members of $\mathcal{H}$ are bounded, total variation turns
out to be a natural upper bound for the associated loss. In particular,

$\displaystyle d_{\mathcal{H}}\left(g_{\\#}\tilde{\mu}_{n},g_{\\#}\mu\right)$
$\displaystyle=\sup_{h\in\mathcal{H}}\int
h(z)\>d(g_{\\#}\tilde{\mu}_{n}-g_{\\#}\mu)$
$\displaystyle=L\sup_{h^{\prime}\in\frac{1}{L}\mathcal{H}\circ g}\int
h^{{}^{\prime}}(x)\>d(\tilde{\mu}_{n}-\mu)$
$\displaystyle\leq\frac{L}{2}\int\mathinner{\\!\left\lvert\hat{p}_{h}(x)-p_{\mu}(x)\right\rvert}dx$
(15)
$\displaystyle\leq\frac{L}{2}\left\\{\int\mathinner{\\!\left\lvert\hat{p}_{h}(x)-\mathbb{E}[\hat{p}_{h}(x)]\right\rvert}dx+\mathinner{\\!\left\lVert\mathbb{E}[\hat{p}_{h}(x)]-p_{\mu}\right\rVert}_{1}\right\\},$
(16)

where $\mathcal{H}\circ g=\\{h\circ
g\mathrel{\mathop{\ordinarycolon}}h\in\mathcal{H}\\}$ such that
$\mathinner{\\!\left\lVert h\circ
g\right\rVert}_{\infty}=\max_{x}\mathinner{\\!\left\lvert
h(g(x))\right\rvert}<\infty$. The first inequality is obtained by taking the
supremum over all bounded real-valued functions on $\mathcal{X}$. As such, it
is sufficient to find the concentration of $\hat{p}_{h}$ around $p_{\mu}$
under the essential supremum norm.

The bias term under the norm satisfies

$\displaystyle\mathbb{E}[\hat{p}_{h}(x)]-p_{\mu}(x)$
$\displaystyle=\frac{1}{h^{d}}\int\kappa\left(\frac{x}{h},\frac{y}{h}\right)p_{\mu}(y)dy-
p_{\mu}(x)$
$\displaystyle=\int\kappa\left(\frac{x}{h},\frac{x}{h}-u\right)[p_{\mu}(x-hu)-p_{\mu}(x)]du$
(17)
$\displaystyle=\int\kappa\left(\frac{x}{h},\frac{x}{h}-u\right)\sum_{\mathinner{\\!\left\lvert\alpha\right\rvert}\leq
m_{x}-1}\frac{D^{\alpha}p_{\mu}(x)}{\alpha!}(-uh)^{\alpha}du$ (18)
$\displaystyle\quad+m_{x}\int\kappa\left(\frac{x}{h},\frac{x}{h}-u\right)\sum_{\mathinner{\\!\left\lvert\alpha\right\rvert}=m_{x}}\frac{(-uh)^{\alpha}}{\alpha!}\int^{1}_{0}(1-t)^{m_{x}-1}D^{\alpha}p_{\mu}(x-tuh)dt$
$\displaystyle=\int\int^{1}_{0}\kappa\left(\frac{x}{h},\frac{x}{h}-u\right)(-u)^{m_{x}}h^{m_{x}}\frac{(1-t)^{m_{x}-1}}{(m_{x}-1)!}D^{m_{x}}p_{\mu}(x-tuh)dt\>du,$
(19)

where (17) is obtained using the change in variables
$\frac{y}{h}=\frac{x}{h}-u$. In (18), we use Taylor’s expansion for
multivariate functions, in particular,
$p_{\mu}\in\mathcal{W}^{m_{x},p}_{L}(\Omega_{x})$. The first part of the sum
vanishes due to the regularity of underlying kernels (see Definition 3.7).
Now, using Minkowski’s inequality for integrals, given $p=1$ we get

$\displaystyle\mathinner{\\!\left\lVert\mathbb{E}[\hat{p}_{h}(x)]-p_{\mu}\right\rVert}_{1}$
$\displaystyle\leq h^{m_{x}}\mathinner{\\!\left\lVert
D^{m_{x}}p_{\mu}\right\rVert}_{1}\int\sup_{v}\mathinner{\\!\left\lvert\kappa(v,v-u)\right\rvert}{\mathinner{\\!\left\lvert
u\right\rvert}}^{m_{x}}du\int^{1}_{0}\frac{(1-t)^{m_{x}-1}}{(m_{x}-1)!}dt$
$\displaystyle\lesssim h^{m_{x}},$ (20)

again due to the regularity of $\kappa$ and Assumption 1.

Let us define
$\mathscr{K}=\left\\{f(x,\cdot)=\frac{1}{h^{d}}\kappa\left(\frac{x}{h},\frac{\cdot}{h}\right)\mathrel{\mathop{\ordinarycolon}}x\in\Omega_{x}\right\\}$.
Now, given the invariance of the underlying kernels $\kappa$, the bracketing
number of $\mathscr{K}$ turns out to satisfy

$\mathcal{N}_{[\>]}(\mathscr{K},L_{1}(\mu),\epsilon)\leq
E_{\kappa}\left(\frac{L\sqrt{d}B_{x}}{h^{d+1}\epsilon}\right)^{d},$

where $E_{\kappa}>0$ is an universal constant depending on $d$ (Van der Vaart
[92], Example 19.7). Also, observe that

$\displaystyle\textrm{Var}(f)\leq\int f^{2}d\mu\leq\mathinner{\\!\left\lVert
f\right\rVert}_{\infty}\int\mathinner{\\!\left\lvert
f\right\rvert}d\mu\leq\frac{B}{h^{d}},$

where the last inequality utilizes the fact
$\sup_{f}\mathbb{E}\mathinner{\\!\left\lvert
f\right\rvert}\leq\mathbb{E}[\sup_{f}\mathinner{\\!\left\lvert
f\right\rvert}]$. Hence, using Bernstein’s inequality (see, Yukich [105] for
the detailed bracketing argument) we infer that

$\mathbb{P}^{n}\left(\sup_{x}\mathinner{\\!\left\lvert\hat{p}_{h}(x)-\mathbb{E}[\hat{p}_{h}(x)]\right\rvert}>\epsilon\right)\leq
4E_{\kappa}\left(\frac{L\sqrt{d}B_{x}}{h^{d+1}\epsilon}\right)^{d}\exp\left\\{-\frac{En\epsilon^{2}h^{d}}{B}\right\\},$
(21)

where $E>0$ is an universal constant666Lafferty et al. [106] derive the sharp
value of $E$ under Lipschitz continuous kernels. and
$0<\epsilon\leq\frac{2}{3}$. This enables us to state a concentration bound
for $d_{\mathcal{H}}\left(g_{\\#}\tilde{\mu}_{n},g_{\\#}\mu\right)$ readily
using (16) and (20).

Bounding the expected estimation error based on the translated data (second
term in 14) is comparatively straightforward. We present the refined Dudley’s
entropy integral, which becomes the cornerstone of the rest of the proof.

######  Lemma 6.1.

Given a symmetric class of functions $\mathcal{H}$, satisfying
$\sup_{h\in\mathcal{H}}\mathinner{\\!\left\lVert h\right\rVert}_{\infty}\leq
M$, $M>0$ we have

$\mathbb{E}[d_{\mathcal{H}}\left(\hat{\rho}_{m},\rho\right)]\leq
2\inf_{\delta\in(0,M)}\left(2\delta+\frac{12}{\sqrt{m}}\int^{M}_{\delta}\sqrt{\log\mathcal{N}\left(\mathcal{H},\mathinner{\\!\left\lVert\cdot\right\rVert}_{\infty},\epsilon\right)}d\epsilon\right),$

where $\rho\in\mathcal{P}(\mathcal{Z})$.

As such, given that the entropy corresponding to the underlying critic
functions satisfy polynomial discrimination, we obtain an upper bound
corresponding to the infimum as follows

$\mathbb{E}\left[d_{\mathcal{H}}\left(g_{\\#}\mu,\widehat{(g_{\\#}\mu)}_{m}\right)\right]\lesssim
m^{-\frac{1}{q\vee 2}}.$ (22)

To obtain a probabilistic concentration inequality corresponding to the same
error observe that

$d_{\mathcal{H}}\left(g_{\\#}\mu,\widehat{(g_{\\#}\mu)}_{m}\right)=\sup_{h\in\mathcal{H}}\Big{\\{}\frac{1}{m}\sum^{m}_{i=1}h(Y_{i})-\mathbb{E}_{g_{\\#}\mu}h\Big{\\}}\mathrel{\mathop{\ordinarycolon}}=W(Y_{1},\cdots,Y_{m}),$

given $Y_{1},\cdots,Y_{m}\sim g_{\\#}\mu$. As such, for
$y_{1},\cdots,y_{m},y^{{}^{\prime}}_{m}$

$\displaystyle\mathinner{\\!\left\lvert
W(y_{1},\cdots,y_{m})-W(y_{1},\cdots,y^{{}^{\prime}}_{m})\right\rvert}$
$\displaystyle=$
$\displaystyle\frac{1}{m}\mathinner{\\!\left\lvert\sup_{h\in\mathcal{H}}\sum^{m}_{i=1}\left(h(y_{i})-\mathbb{E}_{g_{\\#}\mu}h\right)-\sup_{h^{{}^{\prime}}\in\mathcal{H}}\sum^{m-1}_{i=1}\left(h^{{}^{\prime}}(y_{i})-\mathbb{E}_{g_{\\#}\mu}h^{{}^{\prime}}\right)+h^{{}^{\prime}}(y^{{}^{\prime}}_{m})-\mathbb{E}_{g_{\\#}\mu}h^{{}^{\prime}}\right\rvert}$
$\displaystyle\leq$
$\displaystyle\frac{1}{m}\mathinner{\\!\left\lvert\sup_{h\in\mathcal{H}}h(y_{m})-h(y_{m}^{{}^{\prime}})\right\rvert}\leq\frac{2M}{m}.$

Applying McDiarmid’s inequality,

$d_{\mathcal{H}}\left(g_{\\#}\mu,\widehat{(g_{\\#}\mu)}_{m}\right)\leq\epsilon+\mathcal{O}(m^{-\frac{1}{q\vee
2}})$

holds with probability $\geq
1-\exp\left\\{-\frac{n\epsilon^{2}}{2M^{2}}\right\\}$, where $\epsilon>0$.
This bound, along with (21) ensure the existence of constants $l,E_{1},E_{2}$
and $E_{3}>0$ that proof the theorem. Observe that, the bound satisfies for
arbitrary choices of $h$. To ensure that the realized error is indeed $o(1)$
with high probability, we specify
$h\mathrel{\mathop{\ordinarycolon}}=h_{n}=(\frac{1}{n})^{\xi}$ such that
$\xi\geq\frac{1}{d}$. ∎

###### Proof of Lemma 3.6.

Given any $\phi\in\Phi(W,L)^{k}_{d}$ and
$\mu_{1},\mu_{2}\in\mathcal{P}(\mathcal{X})$, by the definition of IPMs

$\displaystyle
d_{\mathcal{L}_{c_{z}}^{1}}(\phi_{\\#}\mu_{1},\phi_{\\#}\mu_{2})$
$\displaystyle=\sup_{l\in\mathcal{L}_{c_{z}}^{1}}\mathbb{E}_{\mu_{1}}[l\circ\phi]-\mathbb{E}_{\mu_{2}}[l\circ\phi]$
$\displaystyle=\sup_{l\in\mathcal{L}_{c_{z}}^{1}}\left\\{\mathbb{E}_{\mu_{1}}[l\circ\phi]-\mathbb{E}_{\mu_{1}}[l\circ
g]+\mathbb{E}_{\mu_{2}}[l\circ
g]-\mathbb{E}_{\mu_{2}}[l\circ\phi]+\mathbb{E}_{\mu_{1}}[l\circ
g]-\mathbb{E}_{\mu_{2}}[l\circ g]\right\\}$
$\displaystyle\leq\sup_{l\in\mathcal{L}_{c_{z}}^{1}}\left\\{\mathbb{E}_{\mu_{1}}\mathinner{\\!\left\lvert
l\circ\phi-l\circ g\right\rvert}+\mathbb{E}_{\mu_{2}}\mathinner{\\!\left\lvert
l\circ g-l\circ\phi\right\rvert}+\mathbb{E}_{\mu_{1}}[l\circ
g]-\mathbb{E}_{\mu_{2}}[l\circ g]\right\\}$ $\displaystyle\leq
2\mathinner{\\!\left\lVert\phi-g\right\rVert}_{\infty}+\sup_{l\in\mathcal{L}_{c_{z}}^{1}}\mathbb{E}_{\mu_{1}}[l\circ
g]-\mathbb{E}_{\mu_{2}}[l\circ g],$

where the first inequality is due to Jensen’s inequality and in the second one
we use the fact that $l\in\mathcal{L}_{c_{z}}^{1}$, given $c_{z}\equiv L_{1}$.
The arbitrary choice of
$g\in\mathscr{F}(\mathcal{X},\mathcal{P}(\mathcal{Z}))$ makes the result hold
for the infimum as well. As such,

$d_{\mathcal{L}_{c_{z}}^{1}}(\phi_{\\#}\mu_{1},\phi_{\\#}\mu_{2})\leq
2\inf_{g\in\mathscr{F}(\mathcal{X},\mathcal{P}(\mathcal{Z}))}\mathinner{\\!\left\lVert\phi-g\right\rVert}_{\infty}+d_{\mathcal{L}_{c_{z}}^{1}}(g_{\\#}\mu_{1},g_{\\#}\mu_{2}).$
(23)

Now, under the critic $\mathcal{F}$, $\forall\varepsilon>0$ there exists
$f_{\varepsilon}\in\mathcal{F}$ such that

$\displaystyle d_{\mathcal{F}}(\phi_{\\#}\mu_{1},\phi_{\\#}\mu_{2})$
$\displaystyle\leq\mathbb{E}_{\phi_{\\#}\mu_{1}}[f_{\varepsilon}]-\mathbb{E}_{\phi_{\\#}\mu_{2}}[f_{\varepsilon}]+\varepsilon$
$\displaystyle=\mathbb{E}_{\phi_{\\#}\mu_{1}}[f_{\varepsilon}-l]+\mathbb{E}_{\phi_{\\#}\mu_{2}}[l-f_{\varepsilon}]+\mathbb{E}_{\phi_{\\#}\mu_{1}}[l]-\mathbb{E}_{\phi_{\\#}\mu_{2}}[l]+\varepsilon$
$\displaystyle\leq 2\mathinner{\\!\left\lVert
f_{\varepsilon}-l\right\rVert}_{\infty}+\sup_{l\in\mathcal{L}_{c_{z}}^{1}}\mathbb{E}_{\phi_{\\#}\mu_{1}}[l]-\mathbb{E}_{\phi_{\\#}\mu_{2}}[l]+\varepsilon.$

Similarly, taking infimum over all such choices of $l$

$d_{\mathcal{F}}(\phi_{\\#}\mu_{1},\phi_{\\#}\mu_{2})\leq
2\mathcal{E}(\mathcal{F},\mathcal{L}_{c_{z}}^{1})+d_{\mathcal{L}_{c_{z}}^{1}}(\phi_{\\#}\mu_{1},\phi_{\\#}\mu_{2})+\varepsilon.$
(24)

The inequalities (23) and (24) together proof the lemma. ∎

###### Proof of Theorem 3.8.

Given any NN-induced map $\phi$, using the triangle inequality on MMDs one can
write

$d_{\mathcal{H}_{\kappa}}\left(\phi_{\\#}\hat{\mu}_{n},\widehat{(\phi_{\\#}\mu)}_{m}\right)\leq\underbrace{d_{\mathcal{H}_{\kappa}}\left(\phi_{\\#}\hat{\mu}_{n},\phi_{\\#}\mu\right)}_{\text{(i)}}+\underbrace{d_{\mathcal{H}_{\kappa}}\left(\phi_{\\#}\mu,\widehat{(\phi_{\\#}\mu)}_{m}\right)}_{\text{(ii)}}.$
(25)

Since the underlying kernels are bounded to begin with, an immediate upper
bound for (ii) might be:
$d_{\mathcal{H}_{\kappa}}(\phi_{\\#}\mu,\widehat{(\phi_{\\#}\mu)}_{m})\leq\sqrt{\sup_{z\in\Omega_{z}}\kappa(z,z)}\>d_{\textrm{TV}}(\phi_{\\#}\mu,\widehat{(\phi_{\\#}\mu)}_{m})$
(Sriperumbudur et al. [49], Theorem 14 (ii)). Being larger in general, TV may
enforce information preservation onto MMDs. However, the very property of
boundedness of the kernels enables us to show the concentration of empirical
measures under MMD as well. Observe that, for bounded kernels, MMD satisfies
the bounded difference inequality with the universal upper bound
$2m^{-1}\sqrt{\sup_{z\in\Omega_{z}}\kappa(z,z)}$. As such, using McDiarmid’s
inequality

$\mathbb{P}\left(d_{\mathcal{H}_{\kappa}}(\phi_{\\#}\mu,\widehat{(\phi_{\\#}\mu)}_{m})\leq\mathbb{E}[d_{\mathcal{H}_{\kappa}}(\phi_{\\#}\mu,\widehat{(\phi_{\\#}\mu)}_{m})]+t\right)\geq
1-e^{-\frac{mt^{2}}{2C_{\kappa}}},$ (26)

where $C_{\kappa}$ is a positive constant such that
$\sup_{z\in\Omega_{z}}\kappa(z,z)\leq C_{\kappa}$. Furthermore, we observe
that
$\mathbb{E}\left[d_{\mathcal{H}_{\kappa}}(\phi_{\\#}\mu,\widehat{(\phi_{\\#}\mu)}_{m})\right]\leq\left[\mathbb{E}\>d^{2}_{\mathcal{H}_{\kappa}}(\phi_{\\#}\mu,\widehat{(\phi_{\\#}\mu)}_{m})\right]^{\frac{1}{2}}\leq\sqrt{\frac{2C_{k}}{m}}$
(Briol et al. [107], lemma 2). In pursuit of establishing an upper bound to
(i), one needs additional enforcement. The first of which is presented as the
following lemma.

######  Lemma 6.2.

Given arbitrary $\phi,g\in\mathscr{F}(\mathcal{X},\mathcal{Z})$ and
$\mu_{1},\mu_{2}\in\mathcal{P}_{\kappa}(\mathcal{X})$ such that the underlying
kernel function $\kappa$ is strongly invariant, there exists a constant $D>0$
(dependant on $\kappa$ and the latent dimension) for which

$d^{2}_{\mathcal{H}_{\kappa}}(\phi_{\\#}\mu_{1},\phi_{\\#}\mu_{2})\leq
D\mathinner{\\!\left\lVert\phi-g\right\rVert}_{\infty}+d^{2}_{\mathcal{H}_{\kappa}}(g_{\\#}\mu_{1},g_{\\#}\mu_{2}).$

###### Proof of lemma 6.2.

Observe that

$\displaystyle\mathinner{\\!\left\lvert
d^{2}_{\mathcal{H}_{\kappa}}(\phi_{\\#}\mu_{1},\phi_{\\#}\mu_{2})-d^{2}_{\mathcal{H}_{\kappa}}(g_{\\#}\mu_{1},g_{\\#}\mu_{2})\right\rvert}$
$\displaystyle=$
$\displaystyle\mathinner{\\!\left\lvert\int_{\Omega_{x}\times\Omega_{x}}\left[\kappa(\phi(x),\phi(y))-\kappa(g(x),g(y))\right](\mu_{1}-\mu_{2})\otimes(\mu_{1}-\mu_{2})(dxdy)\right\rvert}$
$\displaystyle=$
$\displaystyle\Bigg{|}\int_{\Omega_{x}\times\Omega_{x}}\left[\kappa(\phi(x),\phi(y))-\kappa(g(x),\phi(y))+\kappa(g(x),\phi(y))-\kappa(g(x),g(y))\right]$
$\displaystyle\mspace{250.0mu}(\mu_{1}-\mu_{2})\otimes(\mu_{1}-\mu_{2})(dxdy)\Bigg{|}$
$\displaystyle\overset{(1)}{=}$
$\displaystyle\Bigg{|}\int_{\Omega_{x}}\left[K(\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2})(\phi(x))-K(\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2})(g(x))\right](\mu_{1}-\mu_{2})(dx)$
$\displaystyle\mspace{50.0mu}+\int_{\Omega_{x}}\left[K(g_{\\#}\mu_{1}-g_{\\#}\mu_{2})(\phi(y))-K(g_{\\#}\mu_{1}-g_{\\#}\mu_{2})(g(y))\right](\mu_{1}-\mu_{2})(dy)\Bigg{|}$
$\displaystyle\leq$ $\displaystyle\int_{\Omega_{x}}\mathinner{\\!\left\lvert
K(\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2})(\phi(x))-K(\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2})(g(x))\right\rvert}\mathinner{\\!\left\lvert\mu_{1}-\mu_{2}\right\rvert}(dx)$
$\displaystyle\mspace{50.0mu}+\int_{\Omega_{x}}\mathinner{\\!\left\lvert
K(g_{\\#}\mu_{1}-g_{\\#}\mu_{2})(\phi(y))-K(g_{\\#}\mu_{1}-g_{\\#}\mu_{2})(g(y))\right\rvert}\mathinner{\\!\left\lvert\mu_{1}-\mu_{2}\right\rvert}(dy),$

where ($1$) is due to the fact

$\displaystyle\int\kappa(\phi(x),\phi(y))(\mu_{1}-\mu_{2})\otimes(\mu_{1}-\mu_{2})(dxdy)$
$\displaystyle=$
$\displaystyle\int\kappa(\phi(x),y)(\mu_{1}-\mu_{2})\otimes(\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2})(dxdy)=\int
K(\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2})(\phi(x))(\mu_{1}-\mu_{2})(dx).$

Now,

$\displaystyle\mathinner{\\!\left\lvert
K(\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2})(\phi(x))-K(\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2})(g(x))\right\rvert}$
$\displaystyle\leq\int_{\Omega_{z}}\mathinner{\\!\left\lvert\kappa(\phi(x),y)-\kappa(g(x),y)\right\rvert}\mathinner{\\!\left\lvert\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2}\right\rvert}(dy)$
$\displaystyle\overset{(2)}{\leq}\int_{\Omega_{z}}\mathinner{\\!\left\lVert
K(\phi(x))-K(g(x))\right\rVert}\sqrt{\kappa(y,y)}\mathinner{\\!\left\lvert\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2}\right\rvert}(dy)$
$\displaystyle\overset{(3)}{\lesssim}\mathinner{\\!\left\lVert\phi(x)-g(x)\right\rVert}\int_{\Omega_{z}}\sqrt{\kappa(y,y)}\mathinner{\\!\left\lvert\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2}\right\rvert}(dy),$

where ($2$) is due to the reproducing kernel property, coupled with the
Cauchy-Schwartz inequality. The strong invariance of the underlying kernel
inspires ($3$). Noticing the quantity under the integral to be finite, we
obtain

$\displaystyle\int_{\Omega_{x}}\mathinner{\\!\left\lvert
K(\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2})(\phi(x))-K(\phi_{\\#}\mu_{1}-\phi_{\\#}\mu_{2})(g(x))\right\rvert}\mathinner{\\!\left\lvert\mu_{1}-\mu_{2}\right\rvert}(dx)$
$\displaystyle\lesssim\int_{\Omega_{x}}\mathinner{\\!\left\lVert\phi(x)-g(x)\right\rVert}\mathinner{\\!\left\lvert\mu_{1}-\mu_{2}\right\rvert}(dx)\lesssim\mathinner{\\!\left\lVert\phi-g\right\rVert}_{\infty},$

since the dominating measure is sigma-finite. The suppressed constant is
namely $k$ (the latent dimension). Similarly, observing
$\int_{\Omega_{z}}\sqrt{\kappa(y,y)}\mathinner{\\!\left\lvert
g_{\\#}\mu_{1}-g_{\\#}\mu_{2}\right\rvert}(dy)<\infty$ in addition, we
conclude

$\displaystyle\mathinner{\\!\left\lvert
d^{2}_{\mathcal{H}_{\kappa}}(\phi_{\\#}\mu_{1},\phi_{\\#}\mu_{2})-d^{2}_{\mathcal{H}_{\kappa}}(g_{\\#}\mu_{1},g_{\\#}\mu_{2})\right\rvert}\lesssim\mathinner{\\!\left\lVert\phi-g\right\rVert}_{\infty}.$

As such, there indeed exists a constant that satisfies the lemma. ∎

Now, let us choose in particular
$g\in\mathscr{F}_{L}(\mathcal{X},\mathcal{Z})$. For ease of understanding, we
continue with the distributions
$\mu_{1},\mu_{2}\in\mathcal{P}_{\kappa}(\mathcal{X})$. Using the reproducing
property again, we get

$\displaystyle d^{2}_{\mathcal{H}_{\kappa}}(g_{\\#}\mu_{1},g_{\\#}\mu_{2})$
$\displaystyle=\int_{\Omega_{x}\times\Omega_{x}}\kappa(g(x),g(y))(\mu_{1}-\mu_{2})\otimes(\mu_{1}-\mu_{2})(dxdy)$
$\displaystyle=\int_{\Omega_{x}}K(g_{\\#}\mu_{1}-g_{\\#}\mu_{2})(g(x))(\mu_{1}-\mu_{2})(dx).$

While proving Lemma 6.2, we have observed that the function
$K(g_{\\#}\mu_{1}-g_{\\#}\mu_{2})$ is Lipschitz continuous with accompanying
constant
$c_{g}^{(\mu_{1},\mu_{2})}=\int_{\Omega_{z}}\sqrt{\kappa(y,y)}\mathinner{\\!\left\lvert
g_{\\#}\mu_{1}-g_{\\#}\mu_{2}\right\rvert}(dy)$. We mention that for Energy
kernels, the same function rather turns out to be Hölder continuous. As such,

$\displaystyle d^{2}_{\mathcal{H}_{\kappa}}(g_{\\#}\mu_{1},g_{\\#}\mu_{2})$
$\displaystyle\leq c_{g}^{(\mu_{1},\mu_{2})}\sup_{f\in\mathcal{L}_{c_{z}\equiv
L_{2}}^{1}}\left[\int_{\Omega_{x}}f(g(x))(\mu_{1}-\mu_{2})(dx)\right]$
$\displaystyle\overset{(4)}{=}c_{g}^{(\mu_{1},\mu_{2})}d_{\mathcal{L}_{c_{z}}^{1}}(g_{\\#}\mu_{1},g_{\\#}\mu_{2})\leq
c_{g}^{(\mu_{1},\mu_{2})}Ld_{\mathcal{L}_{c_{x}}^{1}}(\mu_{1},\mu_{2}),$

where ($4$) is due to the Kantorovitch-Rubinstein duality. Hence, we may write

$\displaystyle
d^{2}_{\mathcal{H}_{\kappa}}\left(\phi_{\\#}\hat{\mu}_{n},\phi_{\\#}\mu\right)\leq
D_{n}\mathinner{\\!\left\lVert\phi-g\right\rVert}_{\infty}+c_{g}^{(\hat{\mu}_{n},\mu)}Ld_{\mathcal{L}_{c_{x}}^{1}}(\hat{\mu}_{n},\mu),$
(27)

where $D_{n}$ and $c_{g}^{(\hat{\mu}_{n},\mu)}$ are no longer constants, but
sequences based on $n$ that converge to $0$ almost surely as
$n\rightarrow\infty$ (by dominated convergence theorem). In particular,

$c_{g}^{(\hat{\mu}_{n},\mu)}=c_{g,n}=\int_{\Omega_{z}}\sqrt{\kappa(y,y)}\mathinner{\\!\left\lvert
g_{\\#}\hat{\mu}_{n}-g_{\\#}\mu\right\rvert}(dy)=\int_{\Omega_{x}}\sqrt{\kappa(g(y),g(y))}\mathinner{\\!\left\lvert\hat{\mu}_{n}-\mu\right\rvert}(dy)$

and $D_{n}=o(c_{g,n}\vee c_{\phi,n})$. Applying the concentration of
$\hat{\mu}_{n}$ around $\mu$ under the metric $d_{\mathcal{L}_{c_{x}}^{1}}$,
along with the inequalities 25 and 26 we infer that given $t>0$

$\mathbb{P}\left(d_{\mathcal{H}_{\kappa}}\left(\phi_{\\#}\hat{\mu}_{n},\widehat{(\phi_{\\#}\mu)}_{m}\right)\leq
t+\sqrt{\frac{2C_{\kappa}}{m}}+\sqrt{D_{n}\mathinner{\\!\left\lVert\phi-g\right\rVert}_{\infty}}+\sqrt{\mathcal{O}(c_{g}^{(\hat{\mu}_{n},\mu)}{(d^{2}n)}^{-\frac{1}{d}})+c_{g}^{(\hat{\mu}_{n},\mu)}Lt}\right)$

remains at least $1-2\exp{\left(-\frac{2(m\wedge n)t^{2}}{B}\right)}$, where
$B=\max\\{B^{2}_{x},4C_{\kappa}\\}$. The quantity $B_{x}$ symbolises the
diameter of $\Omega_{x}$ under the metric $c_{x}$. ∎

###### Proof of Corollary 3.11.

The proof takes inspiration from Theorem 3.5 of Lee et al. [71]. First, let us
construct
$\phi_{1\mathrel{\mathop{\ordinarycolon}}L+1}=\phi_{1}\circ\cdots\circ\phi_{L+1}$
where the individual functions are defined as

$\phi_{i}\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{N_{i-1}}\rightarrow\mathbb{R}^{N_{i}}\;\textrm{such
that}\;\left(\phi_{i}(x)\right)_{j}=c_{ij0}+\sum_{k=1}^{r_{i}}c_{ijk}\sigma\left(\langle
m_{ijk},x\rangle+b_{ijk}\right),$

where $c_{ijk},b_{ijk}\in\mathbb{R}$ and $m_{ijk}\in\mathbb{R}^{N_{i-1}}$ are
model parameters in accordance with Definition 3.1. Since these are shallow
networks, the quantity $r_{i},1\leq i\leq L+1$ denotes the number of nodes
they have. Observe that, following Barron’s argument,
$f_{1}\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{d}\rightarrow\mathbb{R}^{N_{1}}$
can be approximated using $\phi_{1}$ (under the $L_{2}$ norm, with respect to
$\mu$). The same result holds for all intermediate individual pieces, with
respect to measures at their respective domains. Our goal is to approximate
the composition of them all. To invoke an induction argument, let us first
consider a sequence of nested sets
$\\{S_{i}\\}_{i=1}^{L+1}\subseteq\mathbb{R}^{d}$ such that
$S_{i}=S_{i-1}\cap\\{x\mathrel{\mathop{\ordinarycolon}}\phi_{1\mathrel{\mathop{\ordinarycolon}}i-1}(x)\in\Omega^{s,N_{i-1}}_{i-1}\\}$.
The measure $\mu$, restricted onto $S_{i}$, when pushed-forward by the map
$\phi_{1\mathrel{\mathop{\ordinarycolon}}i-1}$ yields
$\mu^{i}=(\phi_{1\mathrel{\mathop{\ordinarycolon}}i-1})_{\\#}(\mathbb{1}_{S_{i}}\mu)$