(need not be a probability measure). The support of $\mu^{i}$ is thus obtained
to be
$\phi_{1\mathrel{\mathop{\ordinarycolon}}i-1}(S_{i})\subseteq\Omega^{s,N_{i-1}}_{i-1}$.
Let us denote the Lipschitz constant corresponding to $f_{i}$ by
$\mathinner{\\!\left\lVert f_{i}\right\rVert}_{\mathcal{B}}$ [69]. Now, given
any $\varepsilon>0$, define
$r_{i}=\lceil{\frac{4C^{2}_{i}N_{i}}{\varepsilon^{2}}}\rceil$. Now,

$\displaystyle\left(\int_{\mathbb{R}^{d}}\mathbb{1}_{S_{i}}\mathinner{\\!\left\lVert
f_{1\mathrel{\mathop{\ordinarycolon}}i}-\phi_{1\mathrel{\mathop{\ordinarycolon}}i}\right\rVert}^{2}d\mu\right)^{\frac{1}{2}}$
$\displaystyle\overset{(1)}{\leq}$
$\displaystyle\left(\int_{\mathbb{R}^{d}}\mathbb{1}_{S_{i}}\mathinner{\\!\left\lVert
f_{i}\circ
f_{1\mathrel{\mathop{\ordinarycolon}}i-1}-f_{i}\circ\phi_{1\mathrel{\mathop{\ordinarycolon}}i-1}\right\rVert}^{2}d\mu\right)^{\frac{1}{2}}+\left(\int_{\mathbb{R}^{N_{i-1}}}\mathinner{\\!\left\lVert
f_{i}-\phi_{i}\right\rVert}^{2}d(\phi_{1\mathrel{\mathop{\ordinarycolon}}i-1})_{\\#}(\mathbbm{1}_{S_{i}}\mu)\right)^{\frac{1}{2}}$
$\displaystyle\leq$ $\displaystyle\>\mathinner{\\!\left\lVert
f_{i}\right\rVert}_{\mathcal{B}}\left(\int_{\mathbb{R}^{d}}\mathbb{1}_{S_{i}}\mathinner{\\!\left\lVert
f_{1\mathrel{\mathop{\ordinarycolon}}i-1}-\phi_{1\mathrel{\mathop{\ordinarycolon}}i-1}\right\rVert}^{2}d\mu\right)^{\frac{1}{2}}+\varepsilon$
$\displaystyle\overset{(2)}{\leq}$ $\displaystyle\>\mathinner{\\!\left\lVert
f_{i}\right\rVert}_{\mathcal{B}}\left(\int_{\mathbb{R}^{d}}\mathbb{1}_{S_{i-1}}\mathinner{\\!\left\lVert
f_{1\mathrel{\mathop{\ordinarycolon}}i-1}-\phi_{1\mathrel{\mathop{\ordinarycolon}}i-1}\right\rVert}^{2}d\mu\right)^{\frac{1}{2}}+\varepsilon$
$\displaystyle\leq$ $\displaystyle\left[1+\mathinner{\\!\left\lVert
f_{i}\right\rVert}_{\mathcal{B}}+\mathinner{\\!\left\lVert
f_{i}\right\rVert}_{\mathcal{B}}\mathinner{\\!\left\lVert
f_{i-1}\right\rVert}_{\mathcal{B}}+\cdots+\prod_{j=1}^{i}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}\right]\varepsilon\leq\frac{\left\\{\bigvee_{1}^{i}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}\right\\}^{i+1}-1}{\bigvee_{1}^{i}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}-1}\varepsilon,$

where $(1)$ is due to the triangle inequality. The second term in the same
stage turns out to be $\leq\varepsilon$ using Barron’s theorem [72, 71] given
our specific choice of $r_{i}$. The inequality $(2)$ is based on the
filtration offered by $S_{i}$. Observe that, in case the maximum of the
Lipschitz constants is exactly $1$, the upper bound becomes $i\varepsilon$.
This implies that

$\displaystyle\mu\left(S_{i-1}\cup\\{x\mathrel{\mathop{\ordinarycolon}}\mathinner{\\!\left\lVert
f_{1\mathrel{\mathop{\ordinarycolon}}i-1}(x)-\phi_{1\mathrel{\mathop{\ordinarycolon}}i-1}(x)\right\rVert}\geq
s\\}\right)\leq\left[\frac{\left\\{\bigvee_{1}^{i-1}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}\right\\}^{\overline{i-1}+1}-1}{\bigvee_{1}^{i-1}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}-1}\right]^{2}\cdot\frac{\varepsilon^{2}}{s^{2}}.$

As such,

$\displaystyle\mu(S_{i})$
$\displaystyle=\mu(S_{i-1})-\mu(S_{i-1}\cup\\{x\mathrel{\mathop{\ordinarycolon}}\mathinner{\\!\left\lVert
f_{1\mathrel{\mathop{\ordinarycolon}}i-1}(x)-\phi_{1\mathrel{\mathop{\ordinarycolon}}i-1}(x)\right\rVert}\geq
s\\})$
$\displaystyle\geq\mu(S_{i-1})-\left[\frac{\left\\{\bigvee_{1}^{i-1}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}\right\\}^{i}-1}{\bigvee_{1}^{i-1}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}-1}\right]^{2}\cdot\frac{\varepsilon^{2}}{s^{2}}\geq
1-\frac{\varepsilon^{2}}{s^{2}}\sum_{l=1}^{i-1}\left[\frac{\left\\{\bigvee_{1}^{l}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}\right\\}^{l+1}-1}{\bigvee_{1}^{l}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}-1}\right]^{2}$ (28)

In other words, there exists
$\phi=\phi_{1\mathrel{\mathop{\ordinarycolon}}L+1}$ such that
$\left(\int_{\mathbb{R}^{d}}\mathbb{1}_{S}\mathinner{\\!\left\lVert
f_{1\mathrel{\mathop{\ordinarycolon}}L+1}-\phi\right\rVert}^{2}d\mu\right)^{\frac{1}{2}}\leq\mathcal{O}(\varepsilon)$
for a set $S\subset\mathbb{R}^{d}$ satisfying inequality 28. Now, owing to the
compactness of the base domain and its regularity, the range of
$\phi_{i}=(\phi_{i1},\phi_{i2},\cdots,\phi_{iN_{i}})$ is always contained in a
space of finite diameter. Hence,

$\displaystyle\int_{\Omega_{x}}\mathinner{\\!\left\lVert
f_{1\mathrel{\mathop{\ordinarycolon}}i}-\phi_{1\mathrel{\mathop{\ordinarycolon}}i}\right\rVert}^{2}d\mu\leq$
$\displaystyle\int_{S_{i}}\mathinner{\\!\left\lVert
f_{1\mathrel{\mathop{\ordinarycolon}}i}-\phi_{1\mathrel{\mathop{\ordinarycolon}}i}\right\rVert}^{2}d\mu+\int_{S^{c}_{i}}\mathinner{\\!\left\lVert
f_{1\mathrel{\mathop{\ordinarycolon}}i}-\phi_{1\mathrel{\mathop{\ordinarycolon}}i}\right\rVert}^{2}d\mu$
$\displaystyle\leq$
$\displaystyle\left[\frac{\left\\{\bigvee_{1}^{i}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}\right\\}^{i+1}-1}{\bigvee_{1}^{i}\mathinner{\\!\left\lVert
f_{j}\right\rVert}_{\mathcal{B}}-1}\right]^{2}\cdot{\varepsilon}^{2}+\mathcal{O}\left(\frac{\varepsilon^{2}}{s^{2}}\right).$

Taking the square root and choosing $i$ corresponding to the ultimate layer we
prove the result. ∎

###### Proof of Theorem 3.15.

Given an encoder $\phi\in\Phi(W,L)^{k}_{d}$ and an empirical distribution
corresponding to $\mu$ (based on $n$ i.i.d. replicates), let us fragment the
realized latent WAE-MMD loss as usual,

$\displaystyle
d_{\mathcal{H}_{\kappa}}\left(\phi_{\\#}\hat{\mu}_{n},\rho\right)\leq
d_{\mathcal{H}_{\kappa}}\left(\phi_{\\#}\hat{\mu}_{n},\widehat{(\phi_{\\#}\mu)}_{n}\right)+d_{\mathcal{H}_{\kappa}}\left(\widehat{(\phi_{\\#}\mu)}_{n},\hat{\rho}_{n}\right)+d_{\mathcal{H}_{\kappa}}\left(\hat{\rho}_{n},\rho\right)$
(29)

which holds for all empirical $\hat{\rho}_{n}$, based on $n$ i.i.d. samples
from $\rho$. Observe that, the first quantity is the information dissipated
during encoding and can be put under a deterministic upper bound using Theorem
3.8. The second quantity, due to the observation
$\mathinner{\\!\left\lVert\kappa(\cdot,z)-\kappa(\cdot,z^{{}^{\prime}})\right\rVert}_{\mathcal{H}_{\kappa}}\leq
2\sqrt{C_{\kappa}}\mathbb{1}_{z\neq z^{{}^{\prime}}}$, satisfies

$\displaystyle
d_{\mathcal{H}_{\kappa}}\left(\widehat{(\phi_{\\#}\mu)}_{n},\hat{\rho}_{n}\right)\leq\sqrt{C_{\kappa}}d_{\textrm{TV}}\left(\widehat{(\phi_{\\#}\mu)}_{n},\hat{\rho}_{n}\right).$
(30)

Now, recall that $\rho\in\mathcal{P}_{\Sigma}(\Omega_{z})$. As such, the
observations from $\rho$, forming its empirical counterpart, are first
projected onto the subset of $\Sigma$-invariant distributions. This is termed
symmetrization and the corresponding operator enabling it,
$S^{\Sigma}\mathrel{\mathop{\ordinarycolon}}\mathcal{P}(\mathcal{Z})\rightarrow\mathcal{P}(\mathcal{Z})$
is defined as

$\displaystyle\mathbb{E}_{S^{\Sigma}[\rho]}f=\int_{\mathcal{Z}}\left[\int_{\Sigma}f(\varphi_{\sigma}(z))\mu_{\Sigma}(d\sigma)\right]d\rho(z)=\mathbb{E}_{\rho}\mathbb{E}_{\mu_{\Sigma}}[f\circ\varphi_{\sigma}],$

where $\mu_{\Sigma}$ is the Haar measure on the compact group $\Sigma$ and $f$
denotes any bounded measurable function. In other words, the recipe to obtain
samples from $S^{\Sigma}[\rho]$ in general is to draw i.i.d.
$\\{z_{i}\\}^{n}_{1}\sim\rho$ and $\\{\sigma_{i}\\}^{n}_{1}\sim\mu_{\Sigma}$
independently, followed by the operation $\varphi_{\sigma_{j}}(z_{i})$ [93].
This in turn enables us to narrow down the class of critic functions under MMD
to its $\Sigma$-invariant subset $\mathcal{H}^{\Sigma}_{\kappa}$. We use the
same idea to obtain an upper bound to the remaining estimation error in (29)
as follows. Let,

$\displaystyle\gamma(z_{1},z_{2},\cdots,z_{n})$
$\displaystyle=d_{\mathcal{H}_{\kappa}}(\hat{\rho}_{n},\rho)=d_{\mathcal{H}^{\Sigma}_{\kappa}}(\hat{\rho}_{n},\rho)$
$\displaystyle=\sup_{\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{H}_{\kappa}}\leq
1}\big{\\{}\mathbb{E}_{S^{\Sigma}[\hat{\rho}_{n}]}f-\mathbb{E}_{\rho}f\big{\\}}$
$\displaystyle=\sup_{\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{H}_{\kappa}}\leq
1}\big{\\{}\frac{1}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}\sum_{i=1}^{n}\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}f(\sigma_{j}z_{i})-\mathbb{E}_{\rho}f\big{\\}}$
$\displaystyle\leq\sup_{\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{H}_{\kappa}}\leq
1}\mathinner{\\!\left\lvert\frac{1}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}\sum_{i=1}^{n}\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}f(\sigma_{j}z_{i})-\mathbb{E}_{\rho}f\right\rvert}.$
(31)

To look for the specific constant satisfying the bounded difference property,
observe that

$\displaystyle\mathinner{\\!\left\lvert\gamma(z_{1},\cdots,z_{i},\cdots,z_{n})-\gamma(z_{1},\cdots,z^{{}^{\prime}}_{i},\cdots,z_{n})\right\rvert}$
$\displaystyle\leq\sup_{\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{H}_{\kappa}}\leq
1}\mathinner{\\!\left\lvert\frac{1}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}f(\sigma_{j}z_{i})-f(\sigma_{j}z^{{}^{\prime}}_{i})\right\rvert}$
$\displaystyle\leq\frac{1}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}\Big{\\{}\mathinner{\\!\left\lVert\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}K(\sigma_{j}z_{i})\right\rVert}_{\mathcal{H}_{\kappa}}+\mathinner{\\!\left\lVert\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}K(\sigma_{j}z^{{}^{\prime}}_{i})\right\rVert}_{\mathcal{H}_{\kappa}}\Big{\\}},$
(32)

where

$\displaystyle\mathinner{\\!\left\lVert\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}K(\sigma_{j}z_{i})\right\rVert}_{\mathcal{H}_{\kappa}}$
$\displaystyle=\left[\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}\kappa(\sigma_{j}z_{i},\sigma_{j}z_{i})+\sum_{j\neq
l}\kappa(\sigma_{j}z_{i},\sigma_{l}z_{i})\right]^{\frac{1}{2}}$
$\displaystyle=\left[\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}\kappa(\sigma_{j}z_{i},\sigma_{j}z_{i})+\sum_{\sigma_{j}\neq\textrm{id}}\kappa(\sigma_{j}z_{i},z_{i})\right]^{\frac{1}{2}}$
(33)
$\displaystyle\leq\sqrt{C_{\kappa}\mathinner{\\!\left\lvert\Sigma\right\rvert}}\left[1+\varsigma_{\kappa,\Sigma}\left(\mathinner{\\!\left\lvert\Sigma\right\rvert}-1\right)\right]^{\frac{1}{2}}.$
(34)

As such, using only the one-sided McDiarmid’s inequality, for every $t>0$ we
have with probability $\geq
1-\exp\Big{\\{}-\frac{n\mathinner{\\!\left\lvert\Sigma\right\rvert}t^{2}}{2C_{\kappa}\left[1+\varsigma_{\kappa,\Sigma}\left(\mathinner{\\!\left\lvert\Sigma\right\rvert}-1\right)\right]}\Big{\\}}$

$\displaystyle
d_{\mathcal{H}_{\kappa}}(\hat{\rho}_{n},\rho)-\mathbb{E}[d_{\mathcal{H}_{\kappa}}(\hat{\rho}_{n},\rho)]\leq
t.$ (35)

In order to upper bound the expectation, we use the symmetrization trick as
follows

$\displaystyle\mathbb{E}_{\\{Z_{i}\\}_{1}^{n}\sim\rho}\sup_{\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{H}_{\kappa}}\leq
1}\mathinner{\\!\left\lvert\frac{1}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}\sum_{i=1}^{n}\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}f(\sigma_{j}z_{i})-\mathbb{E}_{\rho}f\right\rvert}$
$\displaystyle=\mathbb{E}_{Z}\sup_{\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{H}_{\kappa}}\leq
1}\mathinner{\\!\left\lvert\frac{1}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}\sum_{i=1}^{n}\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}f(\sigma_{j}z_{i})-\mathbb{E}_{\\{Z^{{}^{\prime}}_{i}\\}_{1}^{n}\sim\rho}\left(\frac{1}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}\sum_{i=1}^{n}\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}f(\sigma_{j}z^{{}^{\prime}}_{i})\right)\right\rvert}$
$\displaystyle\leq\mathbb{E}_{Z,Z^{{}^{\prime}}}\sup_{\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{H}_{\kappa}}\leq
1}\mathinner{\\!\left\lvert\frac{1}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}\sum_{i=1}^{n}\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}f(\sigma_{j}z_{i})-f(\sigma_{j}z^{{}^{\prime}}_{i})\right\rvert}$
$\displaystyle=\mathbb{E}_{Z,Z^{{}^{\prime}},\xi}\sup_{\mathinner{\\!\left\lVert
f\right\rVert}_{\mathcal{H}_{\kappa}}\leq
1}\mathinner{\\!\left\lvert\frac{1}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}\sum_{i=1}^{n}\xi_{i}\sum_{j=1}^{\mathinner{\\!\left\lvert\Sigma\right\rvert}}f(\sigma_{j}z_{i})-f(\sigma_{j}z^{{}^{\prime}}_{i})\right\rvert}$
(36) $\displaystyle\leq
2\sqrt{\frac{C_{\kappa}\left[1+\varsigma_{\kappa,\Sigma}\left(\mathinner{\\!\left\lvert\Sigma\right\rvert}-1\right)\right]}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}},$
(37)

where $(\xi_{1},\xi_{2},\cdots,\xi_{n})\sim$ i.i.d. standard Rademacher and
(37) is due to (Chen et al. [94], Lemma A.14). As such, 35 implies that

$\displaystyle d_{\mathcal{H}_{\kappa}}(\hat{\rho}_{n},\rho)\leq
2\sqrt{\frac{C_{\kappa}\left[1+\varsigma_{\kappa,\Sigma}\left(\mathinner{\\!\left\lvert\Sigma\right\rvert}-1\right)\right]}{n\mathinner{\\!\left\lvert\Sigma\right\rvert}}}\left(1+\sqrt{\frac{1}{2}\ln{\frac{1}{\delta}}}\right)$

holds with probability $1-\delta$ for $\delta>0$. Observe that, the proof of
this concentration bound acts as a generalization to the usual MMD, which
arises in case $\mathinner{\\!\left\lvert\Sigma\right\rvert}=1$.

Going back to 29, we write

$d_{\mathcal{H}_{\kappa}}\left(\phi_{\\#}\hat{\mu}_{n},\rho\right)-\sqrt{C_{\kappa}}\sup_{\mathcal{P}_{n}(\rho)}\Delta_{\Phi,n}\leq
d_{\mathcal{H}_{\kappa}}\left(\phi_{\\#}\hat{\mu}_{n},\widehat{(\phi_{\\#}\mu)}_{n}\right)+d_{\mathcal{H}_{\kappa}}\left(\hat{\rho}_{n},\rho\right),$

where the supremum is taken over possible estimators based on replicates from
$\rho$ of size $n$. It becomes evident that the quantity on the left has
finite expectation and is essentially bounded in probability. Based on the
concentration found in Theorem 3.8, along with the bound as in Corollary 3.9
we conclude that given $t>0$

$\displaystyle
d_{\mathcal{H}_{\kappa}}\left(\phi_{\\#}\hat{\mu}_{n},\rho\right)$
$\displaystyle-\sqrt{C_{\kappa}}\sup_{\mathcal{P}_{n}(\rho)}\Delta_{\Phi,n}\leq\sqrt{t}(\sqrt{c_{g,n}L}+2\sqrt{t})+\mathcal{O}(\sqrt{c_{g,n}}(d^{2}n)^{-\frac{1}{2d}})$
(38)
$\displaystyle+\sqrt{\frac{2C_{\kappa}}{n}}\left[1+\sqrt{\frac{1+\varsigma_{\kappa,\Sigma}\left(\mathinner{\\!\left\lvert\Sigma\right\rvert}-1\right)}{\mathinner{\\!\left\lvert\Sigma\right\rvert}}}\right]+\mathcal{O}(\sqrt{dD_{n}}N_{1}^{-\frac{2}{d}}N_{2}^{-\frac{2}{d}})$

holds with probability at least
$1-2\exp\Big{(}-\frac{2nt^{2}}{\max\left\\{B^{2}_{x},4C_{\kappa}\left[1+\varsigma_{\kappa,\Sigma}\left(\mathinner{\\!\left\lvert\Sigma\right\rvert}-1\right)\right]\right\\}}\Big{)}$.
Observe that, here $c_{g,n}$ is as described in Theorem 3.8 and typically
behave as $\mathcal{O}(n^{-\frac{1}{2}})$ due to the strong law. $N_{1}$ and
$N_{2}$ specify the width $W=\mathcal{O}(d\lfloor
N_{1}^{\frac{1}{d}}\rfloor\vee N_{1}+1)$ and length $L=\mathcal{O}(N_{2})$ of
the encoder. Applying a change in variables on (38) we prove the theorem. ∎

###### Proof of Theorem 4.2.

Given an optimal encoder $E^{*}_{n}(t)$ that incurs latent loss
$d_{\textrm{TV}}\left(E^{*}_{n}(t)_{\\#}\hat{\mu}_{n},\rho\right)\leq t$,
fragmenting the reconstruction error according to (11) yields

$W_{c_{x}}^{1}(\mu,(D\circ E^{*}_{n}(t))_{\\#}\hat{\mu}_{n})\leq
W_{c_{x}}^{1}((D\circ
E^{*}_{n}(t))_{\\#}\hat{\mu}_{n},D_{\\#}\rho)+W_{c_{x}}^{1}(D_{\\#}\rho,\hat{\mu}_{n})+W_{c_{x}}^{1}(\hat{\mu}_{n},\mu).$

The decoder transform is constructed as $D=\phi^{{}^{\prime}}\circ D_{0}$,
where $\phi^{{}^{\prime}}$ is according to lemma 4.1 and
$D_{0}\mathrel{\mathop{\ordinarycolon}}\Omega_{z}\rightarrow\mathbb{R}$ is a
linear map (or an ensemble of several). $D$ can be equivalently written as
$D=\phi^{{}^{\prime}}\circ\sigma_{I}\circ D_{0}$, where $\sigma_{I}$ is the
identity activation, applied componentwise. As such, based on our definition,
$D$ indeed belongs to $\Phi(W,L)_{k}^{d}$ with depth $\geq 3$. As discussed
earlier, the resultant $D$ thus turns out to be Lipschitz continuous. If
$c_{x}$ is taken as $L_{1}$, then observe

$\displaystyle W_{c_{x}}^{1}\left((D\circ
E^{*}_{n}(t))_{\\#}\hat{\mu}_{n},D_{\\#}\rho\right)$ $\displaystyle\leq
B_{x}d_{\textrm{TV}}\left((D\circ
E^{*}_{n}(t))_{\\#}\hat{\mu}_{n},D_{\\#}\rho\right)$
$\displaystyle=B_{x}\sup_{\omega\in\sigma(\mathcal{X})}\mathinner{\\!\left\lvert
E^{*}_{n}(t)_{\\#}\hat{\mu}_{n}(D^{-1}(\omega)),\rho(D^{-1}(\omega))\right\rvert}$
$\displaystyle\leq
B_{x}\sup_{\omega^{{}^{\prime}}\in\sigma(\mathcal{Z})}\mathinner{\\!\left\lvert
E^{*}_{n}(t)_{\\#}\hat{\mu}_{n}(\omega^{{}^{\prime}}),\rho(\omega^{{}^{\prime}})\right\rvert}$
$\displaystyle=B_{x}d_{\textrm{TV}}\left(E^{*}_{n}(t)_{\\#}\hat{\mu}_{n},\rho\right)\leq
tB_{x},$

where the latter inequality is reached by taking supremum over all measurable
sets belonging to the Borel $\sigma$-algebra on $\mathcal{Z}$ instead of the
particular path directed by $D^{-1}$. Also, the definition of $D$ implies that
given arvitrary $\varepsilon>0$,
$W_{c_{x}}^{1}(D_{\\#}\rho,\hat{\mu}_{n})<\varepsilon$ (lemma 4.1).

As such, the concentration of the reconstruction error is essentially
determined by the statistical estimation error in the input space. Taking
expectation over the samples we write

$\displaystyle\mathbb{E}\left[W_{c_{x}}^{1}(\mu,(D\circ
E^{*}_{n}(t))_{\\#}\hat{\mu}_{n})\right]-tB_{x}\leq\mathcal{O}(n^{-\frac{1}{d}}),$

whenever $d\geq 3$. Using the naive estimator, this is the sharpest rate one
can achieve. However, to remove the influence of the dimensionality $d$, here
also [95]’s device can be applied (see Remark 9).

The bound, based on the empirical estimator does not appreciate the smoothness
of the input density $p_{\mu}$. On the other hand, given $m>0$, if $m_{x}>m$
we have
$\mathcal{W}^{m_{x},p}_{L}\subset\mathscr{B}^{m}_{pq}(L^{{}^{\prime}})$, for
some $L^{{}^{\prime}}>0$, where $1\leq q\leq\infty$ and $1\leq p<\infty$ i.e.
belonging to the general Besov space (Giné and Nickl [39], Section 4.3). Thus,
if wavelet estimates (as in Weed and Berthet [108]) are deployed instead, the
rate can be expected to be $\mathcal{O}(n^{-\frac{1+m}{d+2m}})$ given $d\geq
3$. ∎

###### Proof of Theorem 5.1.

Let us consider the kernel $\kappa(x,y)$ to be of the form $\kappa(x-y)$,
without loss of generality. Given replicates
$X_{1},\cdots,X_{n}\sim\tilde{p}$, the density estimate based on transformed
(due to $G\in\mathcal{G}$) samples is given as

$\displaystyle\hat{p}_{h}(x)=\frac{1}{nh^{d}}\sum^{n}_{i=1}\kappa\left(\frac{G(X_{i})-x}{h}\right),$

where $h$ is the bandwidth. Decomposing the $L_{1}$ reconstruction error
yields

$\displaystyle\mathinner{\\!\left\lvert\hat{p}_{h}(x)-p_{\mu}(x)\right\rvert}\leq\mathinner{\\!\left\lvert\hat{p}_{h}(x)-\mathbb{E}[\hat{p}_{h}(x)]\right\rvert}+\mathinner{\\!\left\lvert\mathbb{E}[\hat{p}_{h}(x)]-p_{\mu}(G^{-1}(x))\right\rvert}+\mathinner{\\!\left\lvert
p_{\mu}(x)-p_{\mu}(G^{-1}(x))\right\rvert}.$ (39)

Now, observe that

$\displaystyle\mathbb{E}\mathinner{\\!\left\lvert\hat{p}_{h}(x)-\mathbb{E}[\hat{p}_{h}(x)]\right\rvert}$
$\displaystyle\leq\sqrt{\mathbb{E}\left(\hat{p}_{h}(x)-\mathbb{E}[\hat{p}_{h}(x)]\right)^{2}}$
$\displaystyle=\sqrt{\frac{1}{n}\textrm{Var}_{\tilde{p}}\left[\frac{1}{h^{d}}\kappa\left(\frac{G(X)-x}{h}\right)\right]}$
$\displaystyle\overset{(1)}{=}\sqrt{\frac{1}{n}\textrm{Var}_{\tilde{p}}\left[\frac{1}{h^{d}}\kappa\left(\frac{X-G^{-1}(x)}{h}\right)\right]}$
$\displaystyle\overset{(2)}{\lesssim}\sqrt{\frac{1}{nh^{d}}},$

where ($1$) is due to the transformation invariance of the kernel. The
inequality ($2$) is obtained as a result of $\int\kappa^{2}(u)du<\infty$. The
suppressed constant in the process is the upper bound to the density
$\tilde{p}$. For the bias term, we write

$\displaystyle\mathinner{\\!\left\lvert\mathbb{E}[\hat{p}_{h}(x)]-p_{\mu}(G^{-1}(x))\right\rvert}$
$\displaystyle=\mathinner{\\!\left\lvert\frac{1}{h^{d}}\mathbb{E}_{\tilde{p}}\left[\kappa\left(\frac{G(X_{i})-x}{h}\right)\right]-p_{\mu}(G^{-1}(x))\right\rvert}$
$\displaystyle\leq\mathinner{\\!\left\lvert\frac{1}{h^{d}}\mathbb{E}_{\tilde{p}}\left[\kappa\left(\frac{X-G^{-1}(x)}{h}\right)\right]-\frac{1}{h^{d}}\mathbb{E}_{p_{\mu}}\left[\kappa\left(\frac{Y-G^{-1}(x)}{h}\right)\right]\right\rvert}$
$\displaystyle\quad+\mathinner{\\!\left\lvert\frac{1}{h^{d}}\mathbb{E}_{p_{\mu}}\left[\kappa\left(\frac{Y-G^{-1}(x)}{h}\right)\right]-p_{\mu}(G^{-1}(x))\right\rvert}$
$\displaystyle\overset{(3)}{\lesssim}\frac{1}{h^{d}}\mathbb{E}_{\tilde{p},p_{\mu}}\mathinner{\\!\left\lvert\frac{X-Y}{h}\right\rvert}+h^{m_{x}}\lesssim\frac{\epsilon}{h^{2d}}\vee
h^{m_{x}},$

where in ($3$) we use the invariance of $\kappa$ for the first term, and the
second bound is obtained using Giné and Nickl [39], Proposition 4.3.33. We
emphasize the fact that kernels satisfying Lipschitz continuity are commonly
taken in practice, which are readily invariant to translations. The latter
inequality is due to the contamination model. As such, assuming without loss
of generality that the transform $G$ preserves the origin, we get

$\mathbb{E}\mathinner{\\!\left\lvert\hat{p}_{h}(0)-p_{\mu}(0)\right\rvert}\lesssim\sqrt{\frac{1}{nh^{d}}}\vee\frac{\epsilon}{h^{2d}}\vee
h^{m_{x}}.$

Taking $h=n^{-\frac{1}{d+2m_{x}}}\vee\epsilon^{\frac{1}{2d+m_{x}}}$, we prove
the theorem. ∎

## Appendix: Experiments and Simulations

### Encoded vs. Latent Distributions

To check the efficacy of a WAE-encoding statistically, we perform two-sample
non-parametric tests of equality on target latent and encoded observations.
Peacock [109] suggested a multi-dimensional generalization to the well-known
Kolmogorov–Smirnov test, which, however, has high computational complexity. In
our study, we identify the test suggested by Fasano and Franceschini [110]
(FF) as a suitable alternative based on its manageable complexity without
sacrificing the power and consistency777We follow the fasano.franceschini.test
implementation [111] on R.. Many suggestions have been made to improve the
reliability of two-sample tests in higher dimensions since. However, given the
ease of implementation, we use FF’s version of the KS test. To ascertain our
findings, we additionally carry out a referral test of equality of
distributions, based on kernelized distances between pairs of observations,
namely the Cramér test [112]. Unlike FF, the test statistic corresponding to
Cramér888We implement the cramer package [113] in R with underlying kernel
specified to $\kappa(z)=\sqrt{z}/2$. is not distribution-free, and requires
bootstrapping to obtain the $p$-value. We utilize two of such methods, namely
the usual Monte-Carlo and calculating the approximate eigenvalues as the
weights to the limiting distribution (of the statistic). Test results on the
Five Gaussian data at $5\%$ level of significance, given $\lambda=0.8$, are as
follows:

Table 1: Two-sample tests of equality on latent and encoded distributions.

Architecture | Latent Distribution | KS test | Cramér test
---|---|---|---
|  |  | Monte-Carlo | Eigenvalue
WAE-GAN | Gaussian | ✗ | ✗ | ✗
Exponential | ✗ | ✗ | ✗
WAE-WAE | Gaussian | ✗ | ✗ | ✗
Exponential | ✗ | ✗ | ✗

  * •

∗The decisions ‘Accept’ and ‘Reject’ against the null hypothesis that the two
distributions are equal are denoted by the symbols (✓) and (✗) respectively.

The test results corroborate our theoretical findings. Though we only
establish an upper bound to the latent loss, it is apparent that there lies an
optimization error due to the minimum distance estimate. As such, under a
metrizing measure of discrepancy, the target latent law and the encoded
estimate must be distinct in distribution. The rejection of the null
hypothesis reiterates the same observation.

Figure 13: Concentration of bin estimates under ReLU encoders (yellow) for
latent Beta$(0.5,0.8)$ copula (blue), over epochs
$(200,400,600,800,1000,1200,1400,2000)$ (left to right) in a WAE-GAN setup.

It becomes even more clear looking at the histograms corresponding to samples
from the two, overlaid. The interesting observation from Fig 13, 14 is the
visual manifestation of information preservation. Semantic information, in the
form of cluster structures originally present in the data set, remains intact
in encoded distributions while trying to maximize similarity with their target
counterparts. The evolution of this ‘maximization’ is clear from the
histograms obtained over epochs. Another viewpoint that attests to this
finding is the quantile-quantile plot [Fig 15] of the marginals.

Figure 14: Concentration of bin estimates under ReLU encoders (yellow) for
latent bivariate Gaussian distribution (blue), over epochs
$(200,400,600,800,1200,1400,1600,1800)$ (left to right) in a WAE-MMD setup
with regularization $\lambda=0.1$.

(a) Gaussian

(b) Beta Copula

Figure 15: Propagation of quantile-quantile (QQ) plots of marginals
corresponding to encoded vs latent distribution under ReLU encoders, over
epochs $(0,200,800,1400,1800)$ in a WAE-MMD setup with regularization
$\lambda=0.1$.

(a) Gaussian

(b) Beta

(c) Exponential

Figure 16: Reconstruction error of Five Gaussian data under (a) JS and (b),
(c) sample corrected ($\times n^{\frac{1}{2}}$) MMD latent loss, using
GroupSort encoders (grouping 2).

(a) Actual Data

(b) Contaminated Data

(c) Reconstructed Data

Figure 17: Reconstructed samples ($n=10,000$) from Five Gaussian dataset with
$10\%$ observations contaminated at level $0.2$, under MMD latent loss. The
corrupting distribution is taken to be standard tri-variate Cauchy.

(a) Gaussian

(b) Cauchy

Figure 18: Reconstructed samples ($n=10,000$) from Five Gaussian dataset with
$10\%$ observations contaminated at level $0.2$, under MMD latent loss. The
corrupting distribution is taken to be standard tri-variate Cauchy.

## References

  * Kingma and Welling [2014] D. P. Kingma, M. Welling,  Auto-encoding variational bayes,  in: 2nd International Conference on Learning Representations, ICLR 2014, 2014.
  * Wei et al. [2020] R. Wei, C. Garcia, A. El-Sayed, V. Peterson, A. Mahmood,  Variations in variational autoencoders - a comparative evaluation,  IEEE Access 8 (2020).
  * Tolstikhin et al. [2018] I. Tolstikhin, O. Bousquet, S. Gelly, B. Schoelkopf,  Wasserstein auto-encoders,  in: International Conference on Learning Representations, 2018.
  * Bengio et al. [2013] Y. Bengio, A. Courville, P. Vincent,  Representation learning: A review and new perspectives,  IEEE transactions on pattern analysis and machine intelligence 35 (2013) 1798–1828.
  * Arora et al. [2017] S. Arora, R. Ge, Y. Liang, T. Ma, Y. Zhang,  Generalization and equilibrium in generative adversarial nets (gans),  in: International Conference on Machine Learning, PMLR, 2017, pp. 224–232.
  * Liu et al. [2017] S. Liu, O. Bousquet, K. Chaudhuri,  Approximation and convergence properties of generative adversarial learning,  in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, 2017, p. 5551–5559.
  * Biau et al. [2020] G. Biau, B. Cadre, M. Sangnier, U. Tanielian,  Some theoretical properties of GANS,  The Annals of Statistics 48 (2020) 1539 – 1566.
  * Belomestny et al. [2021] D. Belomestny, E. Moulines, A. Naumov, N. Puchkin, S. Samsonov,  Rates of convergence for density estimation with gans,  arXiv preprint arXiv:2102.00199 (2021).
  * Biau et al. [2021] G. Biau, M. Sangnier, U. Tanielian,  Some theoretical insights into wasserstein gans,  Journal of Machine Learning Research 22 (2021) 1–45.
  * Liu et al. [2021] S. Liu, Y. Yang, J. Huang, Y. Jiao, Y. Wang,  Non-asymptotic error bounds for bidirectional GANs,  in: A. Beygelzimer, Y. Dauphin, P. Liang, J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems, 2021.
  * Makhzani et al. [2015] A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, B. Frey,  Adversarial autoencoders,  arXiv preprint arXiv:1511.05644 (2015).
  * Mescheder et al. [2017] L. Mescheder, S. Nowozin, A. Geiger,  Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks,  in: Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, 2017, p. 2391–2400.
  * Hu et al. [2018] Z. Hu, Z. Yang, R. Salakhutdinov, E. P. Xing,  On unifying deep generative models,  in: International Conference on Learning Representations, 2018.
  * Husain et al. [2019] H. Husain, R. Nock, R. C. Williamson,  A primal-dual link between gans and autoencoders,  Advances in Neural Information Processing Systems 32 (2019).
  * Candès et al. [2011] E. J. Candès, X. Li, Y. Ma, J. Wright,  Robust principal component analysis?,  Journal of the ACM (JACM) 58 (2011) 1–37.
  * Dai et al. [2018] B. Dai, Y. Wang, J. Aston, G. Hua, D. Wipf,  Connections with robust pca and the role of emergent sparsity in variational autoencoder models,  The Journal of Machine Learning Research 19 (2018) 1573–1614.
  * Dai and Wipf [2019] B. Dai, D. Wipf,  Diagnosing and enhancing VAE models,  in: International Conference on Learning Representations, 2019.
  * Chakrabarty and Das [2021] A. Chakrabarty, S. Das,  Statistical regeneration guarantees of the wasserstein autoencoder with latent space consistency,  Advances in Neural Information Processing Systems 34 (2021) 17098–17110.
  * Bennett [1969] R. Bennett,  The intrinsic dimensionality of signal collections,  IEEE Transactions on Information Theory 15 (1969) 517–525.
  * Eneva et al. [2002] E. Eneva, K. Kumaraswami, M. Matteucci,  Wekkem: A study in fractal dimension and dimensionality reduction,  in: Workshop on Fractals and Self-similarity in Data Mining: Issues and Approaches, 2002.
  * Fefferman et al. [2016] C. Fefferman, S. Mitter, H. Narayanan,  Testing the manifold hypothesis,  Journal of the American Mathematical Society 29 (2016) 983–1049.
  * Levina and Bickel [2004] E. Levina, P. Bickel,  Maximum likelihood estimation of intrinsic dimension,  Advances in neural information processing systems 17 (2004).
  * Facco et al. [2017] E. Facco, M. d’Errico, A. Rodriguez, A. Laio,  Estimating the intrinsic dimension of datasets by a minimal neighborhood information,  Scientific reports 7 (2017) 1–8.
  * Pope et al. [2021] P. Pope, C. Zhu, A. Abdelkader, M. Goldblum, T. Goldstein,  The intrinsic dimension of images and its impact on learning,  in: International Conference on Learning Representations, 2021.
  * Higgins et al. [2018] I. Higgins, D. Amos, D. Pfau, S. Racaniere, L. Matthey, D. Rezende, A. Lerchner,  Towards a definition of disentangled representations,  arXiv preprint arXiv:1812.02230 (2018).
  * Yu et al. [2020] Y. Yu, K. H. R. Chan, C. You, C. Song, Y. Ma,  Learning diverse and discriminative representations via the principle of maximal coding rate reduction,  Advances in Neural Information Processing Systems 33 (2020) 9422–9434.
  * Do and Tran [2020] K. Do, T. Tran,  Theory and evaluation metrics for learning disentangled representations,  in: International Conference on Learning Representations, 2020.
  * Rubenstein et al. [2018] P. K. Rubenstein, B. Schoelkopf, I. Tolstikhin, Learning disentangled representations with wasserstein auto-encoders, 2018.
  * Gaujac et al. [2021] B. Gaujac, I. Feige, D. Barber,  Learning disentangled representations with the wasserstein autoencoder,  in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, 2021, pp. 69–84.
  * Han et al. [2021] J. Han, M. R. Min, L. Han, L. E. Li, X. Zhang,  Disentangled recurrent wasserstein autoencoder,  in: International Conference on Learning Representations, 2021.
  * Locatello et al. [2019] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, O. Bachem,  Challenging common assumptions in the unsupervised learning of disentangled representations,  in: international conference on machine learning, PMLR, 2019, pp. 4114–4124.
  * Kim and Mnih [2018] H. Kim, A. Mnih,  Disentangling by factorising,  in: International Conference on Machine Learning, PMLR, 2018, pp. 2649–2658.
  * Mathieu et al. [2019] E. Mathieu, T. Rainforth, N. Siddharth, Y. W. Teh,  Disentangling disentanglement in variational autoencoders,  in: International Conference on Machine Learning, PMLR, 2019, pp. 4402–4412.
  * Müller [1997] A. Müller,  Integral probability metrics and their generating classes of functions,  Advances in Applied Probability 29 (1997) 429–443.
  * Gretton et al. [2012] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, A. Smola,  A kernel two-sample test,  The Journal of Machine Learning Research 13 (2012) 723–773.
  * Villani [2009] C. Villani, Optimal transport: old and new, volume 338, Springer, 2009.
  * Moriakov et al. [2020] N. Moriakov, J. Adler, J. Teuwen,  Kernel of cyclegan as a principal homogeneous space,  in: International Conference on Learning Representations, 2020.
  * Rolinek et al. [2019] M. Rolinek, D. Zietlow, G. Martius,  Variational autoencoders pursue pca directions (by accident),  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 12406–12415.
  * Giné and Nickl [2021] E. Giné, R. Nickl, Mathematical foundations of infinite-dimensional statistical models, Cambridge university press, 2021.
  * Asatryan et al. [2023] H. Asatryan, H. Gottschalk, M. Lippert, M. Rottmann,  A convenient infinite dimensional framework for generative adversarial learning,  Electronic Journal of Statistics 17 (2023) 391–428.
  * Yarotsky [2017] D. Yarotsky,  Error bounds for approximations with deep relu networks,  Neural Networks 94 (2017) 103–114.
  * Yarotsky [2018] D. Yarotsky,  Optimal approximation of continuous functions by very deep relu networks,  in: Conference on learning theory, PMLR, 2018, pp. 639–649.
  * Petersen and Voigtlaender [2018] P. Petersen, F. Voigtlaender,  Optimal approximation of piecewise smooth functions using deep relu neural networks,  Neural Networks 108 (2018) 296–330.
  * Anil et al. [2019] C. Anil, J. Lucas, R. Grosse,  Sorting out lipschitz function approximation,  in: International Conference on Machine Learning, PMLR, 2019, pp. 291–301.
  * Huster et al. [2019] T. Huster, C.-Y. J. Chiang, R. Chadha,  Limitations of the lipschitz constant as a defense against adversarial examples,  in: ECML PKDD 2018 Workshops: Nemesis 2018, UrbReas 2018, SoGood 2018, IWAISe 2018, and Green Data Mining 2018, Dublin, Ireland, September 10-14, 2018, Proceedings 18, Springer, 2019, pp. 16–29.
  * Chernodub and Nowicki [2016] A. Chernodub, D. Nowicki,  Norm-preserving orthogonal permutation linear unit activation functions (oplu),  arXiv preprint arXiv:1604.02313 (2016).
  * Hyvärinen and Pajunen [1999] A. Hyvärinen, P. Pajunen,  Nonlinear independent component analysis: Existence and uniqueness results,  Neural networks 12 (1999) 429–439.
  * Khemakhem et al. [2020] I. Khemakhem, D. Kingma, R. Monti, A. Hyvarinen,  Variational autoencoders and nonlinear ica: A unifying framework,  in: International Conference on Artificial Intelligence and Statistics, PMLR, 2020, pp. 2207–2217.
  * Sriperumbudur et al. [2009] B. K. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf, G. R. Lanckriet,  On integral probability metrics,$\backslash$phi-divergences and binary classification,  arXiv preprint arXiv:0901.2698 (2009).
  * Nickl and Pötscher [2007] R. Nickl, B. M. Pötscher,  Bracketing metric entropy rates and empirical central limit theorems for function classes of besov-and sobolev-type,  Journal of Theoretical Probability 20 (2007) 177–199.
  * Van Der Vaart and Wellner [1996] A. W. Van Der Vaart, J. A. Wellner, Weak convergence, Springer, 1996.
  * Gottlieb et al. [2017] L.-A. Gottlieb, A. Kontorovich, R. Krauthgamer,  Efficient regression in metric spaces via approximate lipschitz extension,  IEEE Transactions on Information Theory 63 (2017) 4838–4849.
  * Devroye and Gyorfi [1990] L. Devroye, L. Gyorfi,  No empirical probability measure can converge in the total variation sense for all distributions,  The Annals of Statistics (1990) 1496–1499.
  * Devroye and Lugosi [2001] L. Devroye, G. Lugosi, Combinatorial methods in density estimation, Springer Science & Business Media, 2001.
  * Van Handel [2014] R. Van Handel, Probability in high dimension, Technical Report, PRINCETON UNIV NJ, 2014.
  * Vladimirova et al. [2020] M. Vladimirova, S. Girard, H. Nguyen, J. Arbel,  Sub-weibull distributions: Generalizing sub-gaussian and sub-exponential properties to heavier tailed distributions,  Stat 9 (2020) e318.
  * Ashtiani et al. [2018] H. Ashtiani, S. Ben-David, A. Mehrabian,  Sample-efficient learning of mixtures,  in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * Jain and Orlitsky [2020] A. Jain, A. Orlitsky,  A general method for robust learning from batches,  Advances in Neural Information Processing Systems 33 (2020) 21775–21785.
  * Virmaux and Scaman [2018] A. Virmaux, K. Scaman,  Lipschitz regularity of deep neural networks: analysis and efficient estimation,  Advances in Neural Information Processing Systems 31 (2018).
  * Suzuki [2019] T. Suzuki,  Adaptivity of deep reLU network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality,  in: International Conference on Learning Representations, 2019.
  * Chen et al. [2019] M. Chen, H. Jiang, W. Liao, T. Zhao,  Efficient approximation of deep relu networks for functions on low dimensional manifolds,  Advances in neural information processing systems 32 (2019).
  * Montanelli and Du [2019] H. Montanelli, Q. Du,  New error bounds for deep relu networks using sparse grids,  SIAM Journal on Mathematics of Data Science 1 (2019) 78–92.
  * Daubechies et al. [2022] I. Daubechies, R. DeVore, S. Foucart, B. Hanin, G. Petrova,  Nonlinear approximation and (deep) relu networks,  Constructive Approximation 55 (2022) 127–172.
  * Gribonval et al. [2022] R. Gribonval, G. Kutyniok, M. Nielsen, F. Voigtlaender,  Approximation spaces of deep neural networks,  Constructive approximation 55 (2022) 259–367.
  * Sriperumbudur et al. [2010] B. K. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf, G. R. Lanckriet,  Hilbert space embeddings and metrics on probability measures,  The Journal of Machine Learning Research 11 (2010) 1517–1561.
  * Modeste and Dombry [2022] T. Modeste, C. Dombry,  Characterization of translation invariant mmd on r d and connections with wasserstein distances (2022).
  * Shen et al. [2019] Z. Shen, H. Yang, S. Zhang,  Deep network approximation characterized by number of neurons,  arXiv preprint arXiv:1906.05497 (2019).
  * Tanielian and Biau [2021] U. Tanielian, G. Biau,  Approximating lipschitz continuous functions with groupsort neural networks,  in: International Conference on Artificial Intelligence and Statistics, PMLR, 2021, pp. 442–450.
  * Wojtowytsch et al. [2022] S. Wojtowytsch, et al.,  Representation formulas and pointwise properties for barron functions,  Calculus of Variations and Partial Differential Equations 61 (2022) 1–37.
  * Caragea et al. [2020] A. Caragea, P. Petersen, F. Voigtlaender,  Neural network approximation and estimation of classifiers with classification boundary in a barron class,  arXiv preprint arXiv:2011.09363 (2020).
  * Lee et al. [2017] H. Lee, R. Ge, T. Ma, A. Risteski, S. Arora,  On the ability of neural nets to express distributions,  in: Conference on Learning Theory, PMLR, 2017, pp. 1271–1296.
  * Barron [1993] A. R. Barron,  Universal approximation bounds for superpositions of a sigmoidal function,  IEEE Transactions on Information theory 39 (1993) 930–945.
  * Klusowski and Barron [2018] J. M. Klusowski, A. R. Barron,  Approximation by combinations of relu and squared relu ridge functions with $l^{1}$ and $l^{0}$ controls,  IEEE Transactions on Information Theory 64 (2018) 7649–7656.
  * Courty et al. [2018] N. Courty, R. Flamary, M. Ducoffe,  Learning wasserstein embeddings,  in: International Conference on Learning Representations, 2018.
  * Jiang et al. [2017] Z. Jiang, Y. Zheng, H. Tan, B. Tang, H. Zhou,  Variational deep embedding: An unsupervised and generative approach to clustering,  in: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, 2017, pp. 1965–1972.
  * Johnson and Lindenstrauss [1984] W. B. Johnson, J. Lindenstrauss,  Extensions of Lipschitz mappings into a Hilbert space,  Conference in modern analysis and probability 26 (1984) 189–206.
  * Larsen and Nelson [2017] K. G. Larsen, J. Nelson,  Optimality of the johnson-lindenstrauss lemma,  in: 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), IEEE, 2017, pp. 633–638.
  * Bourgain [1985] J. Bourgain,  On lipschitz embedding of finite metric spaces in hilbert space,  Israel Journal of Mathematics 52 (1985) 46–52.
  * Bartal et al. [2011] Y. Bartal, B. Recht, L. J. Schulman,  Dimensionality reduction: beyond the johnson-lindenstrauss bound,  in: Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms, SIAM, 2011, pp. 868–887.
  * Chiappori et al. [2017] P.-A. Chiappori, R. J. McCann, B. Pass,  Multi-to one-dimensional optimal transport,  Communications on Pure and Applied Mathematics 70 (2017) 2405–2444.
  * McCann and Pass [2020] R. J. McCann, B. Pass,  Optimal transportation between unequal dimensions,  Archive for Rational Mechanics and Analysis 238 (2020) 1475–1520.
  * Ben-Israel [1999] A. Ben-Israel,  The change-of-variables formula using matrix volume,  SIAM Journal on Matrix Analysis and Applications 21 (1999) 300–312.
  * McCann [1995] R. J. McCann,  Existence and uniqueness of monotone measure-preserving maps,  Duke Mathematical Journal 80 (1995) 309 – 323.
  * Brenier [1991] Y. Brenier,  Polar factorization and monotone rearrangement of vector-valued functions,  Communications on pure and applied mathematics 44 (1991) 375–417.
  * Caffarelli [2000] L. A. Caffarelli,  Monotonicity properties of optimal transportation and the fkg and related inequalities,  Communications in Mathematical Physics 214 (2000) 547–563.
  * Colombo and Fathi [2021] M. Colombo, M. Fathi,  Bounds on optimal transport maps onto log-concave measures,  Journal of Differential Equations 271 (2021) 1007–1022.
  * Topsoe [2000] F. Topsoe,  Some inequalities for information divergence and related measures of discrimination,  IEEE Transactions on information theory 46 (2000) 1602–1609.
  * Endres and Schindelin [2003] D. M. Endres, J. E. Schindelin,  A new metric for probability distributions,  IEEE Transactions on Information theory 49 (2003) 1858–1860.
  * Acharya et al. [2014] J. Acharya, A. Jafarpour, A. Orlitsky, A. T. Suresh,  Sorting with adversarial comparators and application to density estimation,  in: 2014 IEEE International Symposium on Information Theory, IEEE, 2014, pp. 1682–1686.
  * Dvurechensky et al. [2018] P. Dvurechensky, A. Gasnikov, A. Kroshnin,  Computational optimal transport: Complexity by accelerated gradient descent is better than by sinkhorn’s algorithm,  in: Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, PMLR, 2018, pp. 1367–1376.
  * Giné and Guillou [2002] E. Giné, A. Guillou,  Rates of strong uniform consistency for multivariate kernel density estimators,  in: Annales de l’Institut Henri Poincare (B) Probability and Statistics, volume 38, Elsevier, 2002, pp. 907–921.
  * Van der Vaart [2000] A. W. Van der Vaart, Asymptotic statistics, volume 3, Cambridge university press, 2000.
  * Birrell et al. [2022] J. Birrell, M. A. Katsoulakis, L. Rey-Bellet, W. Zhu,  Structure-preserving gans,  in: International Conference on Machine Learning, 2022.
  * Chen et al. [2023] Z. Chen, M. Katsoulakis, L. Rey-Bellet, W. Zhu,  Sample complexity of probability divergences under group symmetry,  in: International Conference on Machine Learning, PMLR, 2023, pp. 4713–4734.
  * Weed and Bach [2017] J. Weed, F. R. Bach,  Sharp asymptotic and finite-sample rates of convergence of empirical measures in wasserstein distance,  Bernoulli (2017).
  * Mahabadi et al. [2018] S. Mahabadi, K. Makarychev, Y. Makarychev, I. Razenshteyn,  Nonlinear dimension reduction via outer bi-lipschitz extensions,  in: Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, 2018, pp. 1088–1101.
  * David and Semmes [2000] G. David, S. Semmes,  Regular mappings between dimensions,  Publicacions Matematiques (2000) 369–417.
  * Yang et al. [2022] Y. Yang, Z. Li, Y. Wang,  On the capacity of deep generative networks for approximating distributions,  Neural networks 145 (2022) 144–154.
  * Chen et al. [2016] M. Chen, C. Gao, Z. Ren,  A general decision theory for Huber’s $\epsilon$-contamination model,  Electronic Journal of Statistics 10 (2016) 3752 – 3774.
  * Zhu et al. [2022] B. Zhu, J. Jiao, J. Steinhardt,  Generalized resilience and robust statistics,  The Annals of Statistics 50 (2022) 2256–2283.
  * Liu and Gao [2019] H. Liu, C. Gao,  Density estimation with contamination: minimax rates and theory of adaptation,  Electronic Journal of Statistics 13 (2019) 3613 – 3653.
  * Liu and Loh [2022] Z. Liu, P.-L. Loh,  Robust W-GAN-based estimation under Wasserstein contamination,  Information and Inference: A Journal of the IMA 12 (2022) 312–362.
  * Liu et al. [2022] T. Liu, P. Kumar, R. Zhou, X. Liu,  Learning from few samples: Transformation-invariant svms with composition and locality at multiple scales,  Advances in Neural Information Processing Systems 35 (2022) 9151–9163.
  * Li and Farnia [2023] C. T. Li, F. Farnia,  Mode-seeking divergences: Theory and applications to gans,  in: Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, PMLR, 2023, pp. 8321–8350.
  * Yukich [1985] J. Yukich,  Laws of large numbers for classes of functions,  Journal of Multivariate Analysis 17 (1985) 245–260.
  * Lafferty et al. [2008] J. Lafferty, H. Liu, L. Wasserman,  Concentration of measure,  On-line]. Available: http://www. stat. cmu. edu/ larry/= sml/Concentration. pdf (2008).
  * Briol et al. [2019] F.-X. Briol, A. Barp, A. B. Duncan, M. Girolami,  Statistical inference for generative models with maximum mean discrepancy,  arXiv preprint arXiv:1906.05944 (2019).
  * Weed and Berthet [2019] J. Weed, Q. Berthet,  Estimation of smooth densities in wasserstein distance,  in: conference on Learning Theory, PMLR, 2019, pp. 3118–3119.
  * Peacock [1983] J. A. Peacock,  Two-dimensional goodness-of-fit testing in astronomy,  Monthly Notices of the Royal Astronomical Society 202 (1983) 615–627.
  * Fasano and Franceschini [1987] G. Fasano, A. Franceschini,  A multidimensional version of the kolmogorov–smirnov test,  Monthly Notices of the Royal Astronomical Society 225 (1987) 155–170.
  * Puritz et al. [2021] C. Puritz, E. Ness-Cohn, R. Braun,  fasano. franceschini. test: An implementation of a multidimensional ks test in r,  arXiv preprint arXiv:2106.10539 (2021).
  * Baringhaus and Franz [2004] L. Baringhaus, C. Franz,  On a new multivariate two-sample test,  Journal of multivariate analysis 88 (2004) 190–206.
  * Franz [2006] C. Franz,  cramer: multivariate nonparametric cramer-test for the two-sample-problem,  R package version 0.8-1 (2006).