* Raissi & Karniadakis (2018) Raissi, M. & Karniadakis, G. E. (2018), ‘Hidden physics models: Machine learning of nonlinear partial differential equations’, Journal of Computational Physics 357, 125–141.
  * Raissi et al. (2019) Raissi, M., Perdikaris, P. & Karniadakis, G. E. (2019), ‘Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations’, Journal of Computational Physics 378, 686–707.
  * Raissi et al. (2018) Raissi, M., Yazdani, A. & Karniadakis, G. E. (2018), ‘Hidden fluid mechanics: A Navier-Stokes informed deep learning framework for assimilating flow visualization data’, arXiv preprint arXiv:1808.04327 .
  * Schwab & Zech (2019) Schwab, C. & Zech, J. (2019), ‘Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in uq’, Analysis and Applications 17(01), 19–55.
  * Stuart (2010) Stuart, A. M. (2010), ‘Inverse problems: a Bayesian perspective’, Acta numerica 19, 451–559.
  * Tang & Yang (2016) Tang, T. & Yang, J. (2016), ‘Implicit-explicit scheme for the allen-cahn equation preserves the maximum principle’, Journal of Computational Mathematics 34(5), 451–461.
  * Tianping Chen & Liu (1990) Tianping Chen, H. C. & Liu, R.-W. (1990), A constructive proof of Cybenko’s approximation theorem and its extensions, in ‘Computing Science and Statistics (edited by LePage and Page) Proc. of the 22nd Symposium on Interface’, Springer-Verlag, pp. 163–168.
  * Welti (2020) Welti, T. (2020), High-dimensional stochastic approximation: algorithms and convergence rates, ETH Dissertation N. 26805.
  * Yang et al. (2018) Yang, J., Du, Q. & Zhang, W. (2018), ‘Uniform l p-bound of the allen-cahn equation and its numerical discretization’, International Journal of Numerical Analysis & Modeling 15.
  * Yarotsky (2017) Yarotsky, D. (2017), ‘Error bounds for approximations with deep ReLU networks’, Neural Networks 94, 103–114.
  * Yarotsky (2018) Yarotsky, D. (2018), Optimal approximation of continuous functions by very deep relu networks, in ‘Conference on Learning Theory’, PMLR, PMLR, pp. 639–649.
  * You et al. (2021) You, H., Yu, Y., Trask, N., Gulian, M. & D’Elia, M. (2021), ‘Data-driven learning of nonlocal physics from high-fidelity synthetic data’, Computer Methods in Applied Mechanics and Engineering 374, 113553.
https://www.sciencedirect.com/science/article/pii/S0045782520307386

  * Zhu & Zabaras (2018) Zhu, Y. & Zabaras, N. (2018), ‘Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification’, Journal of Computational Physics 366, 415–447.
https://www.sciencedirect.com/science/article/pii/S0021999118302341

##  Appendix A Notation for Standard Fourier basis

In several instances in this paper, we employ the following “standard” real
Fourier basis $\\{\bm{\mathrm{e}}_{{k}}\\}_{{k}\in\mathbb{Z}^{d}}$ in $d$
dimensions: For ${k}=(k_{1},\dots,k_{d})\in\mathbb{Z}^{d}$, we define

$\displaystyle\bm{\mathrm{e}}_{{k}}:=C_{{k}}\begin{cases}\cos({k}\cdot{x}),&(k_{1}\geq
0),\\\ \sin({k}\cdot{x}),&(k_{1}<0),\end{cases}$ (A.1)

where the factor $C_{k}>0$ ensures that $\bm{\mathrm{e}}_{{k}}$ is properly
normalized, i.e. that $\|\bm{\mathrm{e}}_{{k}}\|_{L^{2}(\mathbb{T}^{d})}=1$,
or explicitly,

$C_{{k}}=\frac{1}{(2\pi)^{d}}\begin{cases}2,&({k}\neq 0),\\\
1,&({k}=0).\end{cases}$

We note that the basis $\\{\bm{\mathrm{e}}_{{k}}\\}_{{k}\in\mathbb{Z}^{d}}$
simply consists of the real and imaginary parts of the complex Fourier basis
$\\{e^{i{k}\cdot{x}}\\}_{{k}\in\mathbb{Z}^{d}}$.

On occasion, it will also be convenient to write the standard Fourier basis in
the form $\\{\bm{\mathrm{e}}_{j}\\}_{j\in\mathbb{N}}$ (indexed by integer
$j\in\mathbb{N}$, rather than ${k}\in\mathbb{Z}^{d}$). In this case, we
identify

$\bm{\mathrm{e}}_{j}({x}):=\bm{\mathrm{e}}_{{\upkappa}(j)}({x}),\quad(j\in\mathbb{N}),$

where ${\upkappa}:\mathbb{N}\to\mathbb{Z}^{d}$ is a fixed enumeration of
$\mathbb{Z}^{d}$, with the property that $j\mapsto|{\upkappa}(j)|_{\infty}$ is
monotonically increasing, i.e. such that $j\leq j^{\prime}$ implies that
$|{\upkappa}(j)|_{\infty}\leq|{\upkappa}(j^{\prime})|_{\infty}$, where

$\displaystyle|{k}|_{\infty}:=\max_{\ell=1,\dots,d}|k_{\ell}|,\quad{k}=(k_{1},\dots,k_{d})\in\mathbb{Z}^{d}.$
(A.2)

##  Appendix B On the definition of Error (2.11)

We need the following lemma in order to conclude that the error (2.11) is
well-defined on $L^{2}(D)$, even if the encoder $\mathcal{E}$ is only well-
defined on continuous functions.

######  Lemma B.1.

Let $\mathcal{E}:C(D)\to\mathbb{R}^{m}$ denote the point-wise encoder
$u\mapsto\mathcal{E}(u)=(u(x_{1}),\dots,u(x_{m}))$, for some
$x_{1},\dots,x_{m}\in D$. There exists a Borel measurable extension
$\overline{\mathcal{E}}:L^{2}(D)\to\mathbb{R}^{m}$, such that
$\overline{\mathcal{E}}(u)=\mathcal{E}(u)$ for any $u\in C(D)\cap L^{2}(D)$.

###### Proof.

It suffices to consider the case $m=1$. In this case, we note that
$\mathcal{E}(u)=\limsup_{k\to\infty}\mathcal{E}_{k}(u)$ for any $u\in C(D)$,
where

$\mathcal{E}_{k}:L^{2}(D)\to\mathbb{R},\quad\mathcal{E}_{k}(u)=\fint_{B_{1/k}(x_{1})}u(y)\,dy,$

is continuous for any $k\in\mathbb{N}$. In particular, it follows that for
$u\in L^{2}(D)$, the functional

$\widetilde{\mathcal{E}}:L^{2}(D)\to[-\infty,\infty],\quad\widetilde{\mathcal{E}}(u)=\limsup_{k\to\infty}\mathcal{E}_{k}(u),$

is Borel measurable. We can now define a measurable extension
$\overline{\mathcal{E}}$ of $\mathcal{E}$ by e.g. setting

$\overline{\mathcal{E}}(u)=\begin{cases}\widetilde{\mathcal{E}}(u),&(\widetilde{\mathcal{E}}(u)\in\mathbb{R}),\\\
0,&(\widetilde{\mathcal{E}}(u)=\pm\infty).\end{cases}$

∎

##  Appendix C Proofs of Results in Section 3

###  C.1. Proof of Theorem 3.1

The proof of the universal approximation theorem will be based on an
application of the following well-known version of Lusin’s theorem (see e.g.
Bogachev (2007), Thm. 7.1.13), which we will state for the special case of
probability measures on Polish spaces, below:

######  Theorem C.1 (Lusin’s theorem).

Let $X,Y$ be separable and complete metric spaces. Let $\mu\in\mathcal{P}(X)$
be a probability measure, and let $\mathcal{G}:X\to Y$ be a Borel measurable
mapping. Then for any $\epsilon>0$, there exists a compact set $K\subset X$,
such that $\mu(X\setminus K)<\epsilon$, and such that the restriction
$\mathcal{G}|_{K}:K\to Y$ is continuous.

In addition to Lusin’s theorem, the following clipping lemma will be used in
the proof of Theorem 3.1:

######  Lemma C.2 (Clipping lemma).

Let $\epsilon>0$, and fix $0<R_{1}<R_{2}$. There exists a ReLU neural network
$\gamma:\mathbb{R}^{p}\to\mathbb{R}^{p}$, such that

$\displaystyle\begin{cases}\|\gamma(x)-x\|_{\ell^{2}}<\epsilon,&\text{if
}\|x\|_{\ell^{2}}\leq R_{1},\\\ \|\gamma(x)\|_{\ell^{2}}\leq
R_{2},&\forall\,x\in\mathbb{R}^{p}.\end{cases}$

###### Proof.

Without loss of generality, we may assume that $\epsilon\leq R_{2}-R_{1}$. We
first note that $x\mapsto\sigma_{R_{1}}(x):=\min(\max(x,-R_{1}),R_{1})$ maps
$\mathbb{R}^{p}\to[-R_{1},R_{1}]^{p}$, and $\sigma_{R_{1}}$ can be represented
exactly by a (two-layer) ReLU neural network. Furthermore, for any
$x\in[-R_{1},R_{1}]^{p}$, we have $\sigma_{R_{1}}(x)=x$. Define a continuous
function $\phi:\mathbb{R}^{p}\to\mathbb{R}^{p}$ by

$\phi(x):=\begin{cases}x,&(\|x\|_{\ell^{2}}\leq R_{1}),\\\
R_{1}\dfrac{x}{\|x\|_{\ell^{2}}},&(\|x\|_{\ell^{2}}>R_{1}).\end{cases}$

By the universal approximation theorem, there exists a ReLU network
$\widetilde{\gamma}:[-R_{1},R_{1}]^{p}\to\mathbb{R}^{p}$, such that

$\|\widetilde{\gamma}(x)-\phi(x)\|_{\ell^{2}}<\epsilon,\quad\forall\,x\in[-R_{1},R_{1}]^{p}.$

Define now $\gamma:\mathbb{R}^{p}\to\mathbb{R}^{p}$, by
$\gamma(x):=\widetilde{\gamma}\circ\sigma_{R_{1}}(x)$. Then, for
$\|x\|_{\ell^{2}}\leq R_{1}$, we have

$\|{\gamma}(x)-x\|_{\ell^{2}}=\|\widetilde{\gamma}(x)-\phi(x)\|_{\ell^{2}}<\epsilon,$

and

$\displaystyle\sup_{x\in\mathbb{R}^{p}}\|{\gamma}(x)\|_{\ell^{2}}$
$\displaystyle=\sup_{\xi\in[-R_{1},R_{1}]^{p}}\|\widetilde{\gamma}(\xi)\|_{\ell^{2}}$
$\displaystyle\leq\sup_{\xi\in[-R_{1},R_{1}]^{p}}\left\\{\|\widetilde{\gamma}(\xi)-\phi(\xi)\|_{\ell^{2}}+\|\phi(\xi)\|_{\ell^{2}}\right\\}$
$\displaystyle\leq\epsilon+R_{1}\leq R_{2}.$

∎

Using the above clipping lemma, we can now prove the universal approximation
theorem.

###### Proof of Theorem 3.1.

Let $\epsilon>0$ be given. By assumption $\mathcal{G}:C(D)\to L^{2}(U)$ is a
measurable mapping, $\mathcal{G}\in L^{2}(\mu)$, where
$\mu\in\mathcal{P}(C(D))$. We have to show that there exists a DeepONet
$\mathcal{N}=\mathcal{R}\circ\mathcal{A}\circ\mathcal{E}$, such that

$\|\mathcal{G}-\mathcal{N}\|_{L^{2}(\mu)}<\epsilon.$

Given $M>0$, define $\mathcal{G}_{M}$ by clipping $\mathcal{G}$ at size $M>0$,
i.e. define

$\mathcal{G}_{M}(u):=\begin{cases}\mathcal{G}(u),&(\|\mathcal{G}(u)\|_{L^{2}(U)}\leq
M),\\\
M\dfrac{\,G(u)}{\|\mathcal{G}(u)\|_{L^{2}(U)}},&(\|\mathcal{G}(u)\|_{L^{2}(U)}>M).\end{cases}$

so that $\|\mathcal{G}_{M}(u)\|_{L^{2}(U)}\leq M$ for all $u\in C(D)$. Then

$\displaystyle\|\mathcal{G}-\mathcal{N}\|_{L^{2}(\mu)}\leq\|\mathcal{G}-\mathcal{G}_{M}\|_{L^{2}(\mu)}+\|\mathcal{G}_{M}-\mathcal{N}\|_{L^{2}(\mu)},$
(C.1)

and the first term goes to $0$ by the dominated convergence theorem and the
fact that $\mathcal{G}_{M}(u)\to\mathcal{G}(u)$ pointwise, as $M\to\infty$. We
can thus choose $M>\epsilon$, such that

$\displaystyle\|\mathcal{G}-\mathcal{G}_{M}\|_{L^{2}(\mu)}<\epsilon/3.$ (C.2)

It now remains to approximate $\mathcal{G}_{M}$ by a suitable DeepONet
$\mathcal{N}$, where $\mathcal{G}_{M}$ is bounded.

Next, we note that $C(D)$ and $L^{2}(U)$ are Polish spaces (separable,
complete metric spaces). This allows us to invoke Lusin’s theorem, Theorem
C.1, which shows that there exists a compact set $K\subset C(D)$, such that
the restriction $\mathcal{G}_{M}|_{K}$ of $\mathcal{G}_{M}$ to $K$,

$\mathcal{G}_{M}|_{K}:K\to L^{2}(U),$

is continuous, and such that $\mu(C(D)\setminus K)<(\epsilon/9M)^{2}$.

Next, fix an orthonormal basis $\phi_{1},\phi_{2},\phi_{3},\dots\subset
L^{2}(U)$ consisting of continuous functions. For $\kappa\in\mathbb{N}$, let
$P_{\kappa}:L^{2}(U)\to C(U)$ denote the projection onto
$\phi_{1},\dots,\phi_{\kappa}$, i.e.

$P_{\kappa}(v)=\sum_{k=1}^{\kappa}\langle v,\phi_{k}\rangle\phi_{k}.$

We note that $P_{\kappa}:L^{2}(U)\to C(U)$ is continuous for any fixed
$\kappa$. Let now $K^{\prime}:=\mathcal{G}_{M}(K)$ denote the image of the
compact set $K$ under $\mathcal{G}_{M}$. Since $\mathcal{G}_{M}|_{K}:K\to
L^{2}(U)$ is continuous, $K^{\prime}$ is compact as a subset of $L^{2}(U)$. By
the compactness of $K^{\prime}$, there exists $\kappa\in\mathbb{N}$, such that
$\max_{v\in K^{\prime}}\|v-P_{\kappa}(v)\|<\epsilon/6$. We conclude that the
composition $\widetilde{\mathcal{G}}:=P_{\kappa}\circ\mathcal{G}_{M}:K\to
C(U)$, with $K\subset C(D)$ a compact subset, is continuous, and

$\displaystyle\begin{aligned} \max_{u\in
K}\left\|\mathcal{G}_{M}(u)-\widetilde{\mathcal{G}}(u)\right\|_{L^{2}(U)}&=\max_{u\in
K}\left\|\mathcal{G}_{M}(u)-P_{\kappa}\circ\mathcal{G}_{M}(u)\right\|_{L^{2}(U)}^{2}\\\
&=\max_{v\in
K^{\prime}}\left\|v-P_{\kappa}(v)\right\|_{L^{2}(U)}<\epsilon/6.\end{aligned}$
(C.3)

We can apply the universal approximation theorem of continuous operators on
compact subsets of Chen & Chen (1995) to the continuous mapping
$\widetilde{\mathcal{G}}:K\subset C(D)\to C(U)$ to conclude that there exists
an operator network $\widetilde{\mathcal{N}}$ with a single hidden layer in
the approximator network and a single hidden layer in the trunk network (and
with $\tau_{0}\equiv 0$), such that

$\displaystyle\sup_{u\in
K}\|\widetilde{\mathcal{G}}(u)-\widetilde{\mathcal{N}}(u)\|_{L^{2}(U)}<\epsilon/12.$

Note that this implies in particular that

$\displaystyle\|\widetilde{\mathcal{N}}(u)\|_{L^{2}(U)}$
$\displaystyle\leq\|\mathcal{G}_{M}(u)\|_{L^{2}(U)}+\|\mathcal{G}_{M}(u)-\widetilde{\mathcal{G}}(u)\|_{L^{2}(U)}+\|\widetilde{\mathcal{G}}(u)-\widetilde{\mathcal{N}}(u)\|_{L^{2}(U)}$
$\displaystyle\leq M+\epsilon/6+\epsilon/6<2M,$

for all $u\in K$. We note that after suitably modifying222If
$A\in\mathbb{R}^{p\times p}$ is an invertible matrix, then the transformed
trunk and branch nets $\widehat{\tau}:=A\cdot\tau$ and
$\widehat{\beta}:=A^{-T}\cdot\beta$ represent the same DeepONet, i.e. we have
$\sum_{k=1}^{p}\widehat{\beta}_{k}(u)\widehat{\tau}_{k}(y)=\sum_{k=1}^{p}{\beta}_{k}(u){\tau}_{k}(y)$
for all $u,y$. the (linear) output layers of the branch and trunk nets of
$\widetilde{\mathcal{N}}$, we can write $\widetilde{\mathcal{N}}$ in the form

$\widetilde{\mathcal{N}}(u)(y)=\sum_{k=1}^{p}\beta_{k}(u)\tau_{k}(y),$

with _orthonormal_ trunk net functions $\\{\tau_{1},\dots,\tau_{p}\\}\subset
L^{2}(U)$. In particular, we then have

$\|\widetilde{\mathcal{N}}(u)\|_{L^{2}}=\|\beta(u)\|_{\ell^{2}},\quad\forall\,u\in
C(D),$

and

$\|\beta(u)\|_{\ell^{2}}\leq M+\epsilon/3<2M,\quad\forall\,u\in K.$

Applying Lemma C.2 with $R_{1}=M+\epsilon/3$, $R_{2}=2M$, we conclude that
there exists a ReLU neural network $\gamma:\mathbb{R}^{p}\to\mathbb{R}^{p}$,
such that

$\begin{cases}\|\gamma(x)-x\|_{\ell^{2}}<\epsilon/12,&\text{if
}\|x\|_{\ell^{2}}\leq R_{1},\\\ \|\gamma(x)\|_{\ell^{2}}\leq
2M,&\forall\,x\in\mathbb{R}^{p}.\end{cases}$

We now define a “clipped” DeepONet $\mathcal{N}:C(D)\to L^{2}(U)$, by

$\mathcal{N}(u):=\sum_{k=1}^{p}\gamma_{k}(\beta(u))\tau_{k}(y).$

Then $\mathcal{N}(u)$ satisfies

$\displaystyle\begin{aligned} \max_{u\in
K}\|\mathcal{N}(u)-\widetilde{\mathcal{G}}(u)\|_{L^{2}}&\leq\max_{u\in
K}\|\mathcal{N}(u)-\widetilde{\mathcal{N}}(u)\|_{L^{2}}+\max_{u\in
K}\|\widetilde{\mathcal{N}}(u)-\widetilde{\mathcal{G}}(u)\|_{L^{2}}\\\
&=\max_{u\in K}\|\gamma(\beta(u))-\beta(u)\|_{\ell^{2}}+\max_{u\in
K}\|\widetilde{\mathcal{N}}(u)-\widetilde{\mathcal{G}}(u)\|_{L^{2}}\\\
&\leq\max_{\|x\|_{\ell^{2}}\leq R_{1}}\|\gamma(x)-x\|_{\ell^{2}}+\max_{u\in
K}\|\widetilde{\mathcal{N}}(u)-\widetilde{\mathcal{G}}(u)\|_{L^{2}}\\\
&\leq\epsilon/12+\epsilon/12=\epsilon/6,\end{aligned}$ (C.4)

and $\mathcal{N}(u)$ is bounded from above by

$\|\mathcal{N}(u)\|_{L^{2}}=\|\gamma(\beta(u))\|_{\ell^{2}}\leq
2M,\quad\forall\,u\in C(D).$

It follows that for this clipped DeepONet $\mathcal{N}$, we have

$\displaystyle\|\mathcal{G}_{M}-\mathcal{N}\|_{L^{2}(\mu)}$
$\displaystyle\leq\|\mathcal{G}_{M}-\mathcal{N}\|_{L^{2}(\mu;K)}+\|\mathcal{G}_{M}\|_{L^{2}(\mu;X\setminus
K)}+\|\mathcal{N}\|_{L^{2}(\mu;X\setminus K)}$
$\displaystyle\leq\|\mathcal{G}_{M}-\mathcal{N}\|_{L^{2}(\mu;K)}+3M\mu(X\setminus
K)^{1/2}$
$\displaystyle\leq\|\mathcal{G}_{M}-\mathcal{N}\|_{L^{2}(\mu;K)}+\epsilon/3,$
(C.5)

and, using (C.3) and (C.4), we find

$\displaystyle\begin{aligned}
\|\mathcal{G}_{M}-\mathcal{N}\|_{L^{2}(\mu;K)}&\leq\max_{u\in
K}\left\\{\|\mathcal{G}_{M}(u)-\widetilde{\mathcal{G}}(u)\|_{L^{2}(U)}+\|\widetilde{\mathcal{G}}(u)-\mathcal{N}(u)\|_{L^{2}(U)}\right\\}\\\
&<\epsilon/3.\end{aligned}$ (C.6)

Hence, by (C.5) and (C.6), we have

$\displaystyle\|\mathcal{G}_{M}-\mathcal{N}\|_{L^{2}(\mu)}<2\epsilon/3.$ (C.7)

Combining (C.7) and (C.2), we conclude that the DeepONet $\mathcal{N}$
satisfies

$\|\mathcal{G}-\mathcal{N}\|_{L^{2}(\mu)}<2\epsilon/3+\epsilon/3=\epsilon.$

∎

###  C.2. Proof of Theorem 3.3

###### Proof.

The upper estimate follows by a suitable decomposition of the difference. We
write

$\displaystyle\mathcal{N}-\mathcal{G}=\mathcal{R}\circ\mathcal{A}\circ\mathcal{E}-\mathcal{G}$
$\displaystyle=\left[\mathcal{R}\circ\mathcal{A}\circ\mathcal{E}-\mathcal{R}\circ\mathcal{P}\circ\mathcal{G}\right]+\left[\mathcal{R}\circ\mathcal{P}\circ\mathcal{G}-\mathcal{G}\right]$
$\displaystyle=\left[\mathcal{R}\circ\mathcal{A}\circ\mathcal{E}-\mathcal{R}\circ\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}\circ\mathcal{E}\right]$
$\displaystyle\quad+\left[\mathcal{R}\circ\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}\circ\mathcal{E}-\mathcal{R}\circ\mathcal{P}\circ\mathcal{G}\right]$
$\displaystyle\quad+\left[\mathcal{R}\circ\mathcal{P}\circ\mathcal{G}-\mathcal{G}\right]$
$\displaystyle=:T_{1}+T_{2}+T_{3}.$

We can now estimate the norm of each of the three terms, as follows:

$\displaystyle\|T_{1}\|_{L^{2}(\mu)}$
$\displaystyle=\left(\int_{X}\|\mathcal{R}\circ\mathcal{A}\circ\mathcal{E}-\mathcal{R}\circ\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}\circ\mathcal{E}\|^{2}_{L^{2}(U)}\,d\mu\right)^{1/2}$
$\displaystyle\leq\mathrm{Lip}(\mathcal{R})\left(\int_{X}\|\mathcal{A}\circ\mathcal{E}-\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}\circ\mathcal{E}\|^{2}_{\ell^{2}(\mathbb{R}^{p})}\,d\mu\right)^{1/2}$
$\displaystyle=\mathrm{Lip}(\mathcal{R})\left(\int_{\mathbb{R}^{m}}\|\mathcal{A}-\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}\|^{2}_{\ell(\mathbb{R}^{p})}\,d(\mathcal{E}_{\\#}\mu)\right)^{1/2}$
$\displaystyle=\mathrm{Lip}(\mathcal{R})\|\mathcal{A}-\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}\|_{L^{2}(\mathcal{E}_{\\#}\mu)}.$

In the second line above, we denote

$\mathrm{Lip}(\mathcal{R})=\mathrm{Lip}\left(\mathcal{R}:(\mathbb{R}^{p},\|{\,\cdot\,}\|_{\ell^{2}(\mathbb{R}^{p})})\to(L^{2}(U),\|{\,\cdot\,}\|_{L^{2}(U)})\right).$

For the second term, we obtain

$\displaystyle\|T_{2}\|_{L^{2}(\mu)}$
$\displaystyle=\left(\int_{X}\|\mathcal{R}\circ\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}\circ\mathcal{E}-\mathcal{R}\circ\mathcal{P}\circ\mathcal{G}\|_{L^{2}(U)}^{2}\,d\mu\right)^{1/2}$
$\displaystyle\leq\mathrm{Lip}(\mathcal{R}\circ\mathcal{P})\left(\int_{X}\|\mathcal{G}\circ\mathcal{D}\circ\mathcal{E}-\mathcal{G}\|_{L^{2}(U)}^{2}\,d\mu\right)^{1/2}$
$\displaystyle\leq\mathrm{Lip}(\mathcal{R}\circ\mathcal{P})\mathrm{Lip}_{\alpha}(\mathcal{G})\left(\int_{X}\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\|_{X}^{2\alpha}\,d\mu\right)^{1/2},$

where

$\displaystyle\mathrm{Lip}(\mathcal{R}\circ\mathcal{P})$
$\displaystyle=\mathrm{Lip}(\mathcal{R}\circ\mathcal{P}:(L^{2}(U),\|{\,\cdot\,}\|_{L^{2}(U)})\to(L^{2}(U),\|{\,\cdot\,}\|_{L^{2}(U)})),$
$\displaystyle\mathrm{Lip}_{\alpha}(\mathcal{G})$
$\displaystyle=\mathrm{Lip}_{\alpha}(\mathcal{G}:(A\subset
X,\|{\,\cdot\,}\|)\to(L^{2}(U),\|{\,\cdot\,}\|_{L^{2}(U)})),$

and $\alpha\in(0,1]$. Since $\alpha\in(0,1]$, we can estimate the last term by
Jensen’s inequality to obtain:

$\displaystyle\|T_{2}\|_{L^{2}(\mu)}$
$\displaystyle\leq\mathrm{Lip}(\mathcal{R}\circ\mathcal{P})\mathrm{Lip}_{\alpha}(\mathcal{G})\left(\int_{X}\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\|_{X}^{2\alpha}\,d\mu\right)^{1/2}$
$\displaystyle\leq\mathrm{Lip}(\mathcal{R}\circ\mathcal{P})\mathrm{Lip}_{\alpha}(\mathcal{G})\left(\int_{X}\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\|_{X}^{2}\,d\mu\right)^{\alpha/2}$
$\displaystyle=\mathrm{Lip}(\mathcal{R}\circ\mathcal{P})\mathrm{Lip}_{\alpha}(\mathcal{G})\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\|_{L^{2}(\mu)}^{\alpha}.$

Finally for the third term, we have (by the definition of the push-forward)

$\displaystyle\|T_{3}\|_{L^{2}(\mu)}$
$\displaystyle=\left(\int_{X}\|\mathcal{R}\circ\mathcal{P}\circ\mathcal{G}-\mathcal{G}\|^{2}_{L^{2}(U)}\,d\mu\right)^{1/2}$
$\displaystyle=\left(\int_{L^{2}(U)}\|\mathcal{R}\circ\mathcal{P}-\mathrm{Id}\|^{2}_{L^{2}(U)}\,d(\mathcal{G}_{\\#}\mu)\right)^{1/2}$
$\displaystyle=\|\mathcal{R}\circ\mathcal{P}-\mathrm{Id}\|_{L^{2}(\mathcal{G}_{\\#}\mu)}.$

∎

###  C.3. Proof of Theorem 3.10

The proof of Theorem 3.10 is a consequence of the following series of lemmas.

######  Lemma C.3.

Given a separable Hilbert space $H$ and a $p$-dimensional subspace
$\widehat{V}\subset H$, we have

$\displaystyle\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V})=\int_{H}\|v\|^{2}\,d\mu(v)-\sum_{i=1}^{p}\int_{H}\left|\langle
v,\widehat{v}_{i}\rangle\right|^{2}\,d\mu(v),$ (C.8)

where $\widehat{v}_{i}$, $i=1,\dots,p$ is any orthonormal basis of
$\widehat{V}$. In particular, minimizing
$\widehat{\mathscr{E}}_{\mathrm{Proj}}$ is equivalent to maximizing

$\displaystyle\sum_{i=1}^{p}\int_{X}\left|\langle
v,\widehat{v}_{i}\rangle\right|^{2}\,d\mu(v)=\sum_{i=1}^{p}\left[\left|\langle\mathbb{E}[v],\widehat{v}_{i}\rangle\right|^{2}+\langle\widehat{v}_{i},\Gamma\widehat{v}_{i}\rangle\right],$
(C.9)

where

$\Gamma=\int_{X}(v-\mathbb{E}[v])\otimes(v-\mathbb{E}[v])\,d\mu(v).$

###### Proof.

Let $\widehat{v}_{i}$, $i=1,\dots,p$ be any orthonormal basis of
$\widehat{V}$. Then

$\displaystyle\inf_{\widehat{v}\in\widehat{V}}\|v-\widehat{v}\|^{2}$
$\displaystyle=\left\|v-\sum_{k=1}^{p}\langle
v,\widehat{v}_{i}\rangle\,\widehat{v}_{i}\right\|^{2},$

and since $v-\sum_{k}\langle v,\widehat{v}_{i}\rangle\,\widehat{v}_{i}$ is
perpendicular to all $\widehat{v}_{i}$, we have

$\displaystyle\|v\|^{2}$ $\displaystyle=\left\|v-\sum_{k=1}^{p}\langle
v,\widehat{v}_{i}\rangle\,\widehat{v}_{i}\right\|^{2}+\left\|\sum_{k=1}^{p}\langle
v,\widehat{v}_{i}\rangle\,\widehat{v}_{i}\right\|^{2}$
$\displaystyle=\inf_{\widehat{v}\in\widehat{V}}\|v-\widehat{v}\|^{2}+\sum_{k=1}^{p}\left|\langle
v,\widehat{v}_{i}\rangle\right|^{2}.$

This is equivalent to (C.8). Furthermore, for any $i=1,\dots,p$, we have

$\displaystyle\int_{H}|\langle v,\widehat{v}_{i}\rangle|^{2}\,d\mu(v)$
$\displaystyle=\int_{H}|\langle
v-\mathbb{E}[v],\widehat{v}_{i}\rangle+\langle\mathbb{E}[v],\widehat{v}_{i}\rangle|^{2}\,d\mu(v)$
$\displaystyle=\int_{H}\left\\{|\langle
v-\mathbb{E}[v],\widehat{v}_{i}\rangle|^{2}+2\langle
v-\mathbb{E}[v],\widehat{v}_{i}\rangle\langle\mathbb{E}[v],\widehat{v}_{i}\rangle+|\langle\mathbb{E}[v],\widehat{v}_{i}\rangle|^{2}\right\\}\,d\mu(v)$
$\displaystyle=\int_{H}\left\\{|\langle
v-\mathbb{E}[v],\widehat{v}_{i}\rangle|^{2}+|\langle\mathbb{E}[v],\widehat{v}_{i}\rangle|^{2}\right\\}\,d\mu(v)$
$\displaystyle=\langle\widehat{v}_{i},\Gamma\widehat{v}_{i}\rangle+|\langle\mathbb{E}[v],\widehat{v}_{i}\rangle|^{2}.$

∎

######  Lemma C.4.

The covariance operator $\Gamma$ of a probability measure
$\nu\in\mathcal{P}_{2}(H)$ is a compact, self-adjoint operator. In particular,
there exists a discrete set of (unique) eigenvalues
$\overline{\lambda}_{1}>\overline{\lambda}_{2}>\dots$, and orthogonal
subspaces $E_{1},E_{2},\dots$, such that

$H=\bigoplus_{j=1}^{\infty}E_{j},\quad\text{and}\quad\Gamma=\sum_{j=1}^{\infty}\overline{\lambda}_{j}P_{E_{j}}.$

Here $P_{E}:H\to E$ denotes the orthogonal projection onto the subspace
$E\subset H$.

###### Proof.

The proof can e.g. be found in (Pinelis & Molzon 2016, Appendix E,F). ∎

As an immediate corollary, we also obtain

######  Lemma C.5.

For any $\nu\in\mathcal{P}_{2}(H)$, the operator

$\overline{\Gamma}=\int_{H}(v\otimes v)\,d\nu(v),$

is a self-adjoint, compact operator, and hence it also possesses an
eigendecomposition as in Lemma C.4.

###### Proof.

We can write $\overline{\Gamma}=\Gamma+\mathbb{E}[v]\otimes\mathbb{E}[v]$,
where $\mathbb{E}[v]:=\int_{H}v\,d\nu(v)$ is the expected value under $\nu$.
It is now immediate that $\overline{\Gamma}$ is self-adjoint, since both
$\Gamma$ and $\mathbb{E}[v]\otimes\mathbb{E}[v]$ are self-adjoint.
Furthermore, $\overline{\Gamma}$ is also compact, since
$\overline{\Gamma}=\Gamma+\mathbb{E}[v]\otimes\mathbb{E}[v]$ is the sum of a
compact operator $\Gamma$ and the finite-rank operator
$\mathbb{E}[v]\otimes\mathbb{E}[v]$. ∎

The next lemma shows that the quantity (C.9) to be maximized can be written in
an equivalent form, which only involves the orthogonal projection onto
$V_{p}$.

######  Lemma C.6.

Let $\overline{\Gamma}=\Gamma+\mathbb{E}[v]\otimes\mathbb{E}[v]$. Let
$\phi_{k}$, $(k\in\mathbb{N})$, be an orthonormal basis of eigenvectors of
$\overline{\Gamma}$ with corresponding eigenvalues $\lambda_{k}$. Let
$\widehat{v}_{1},\dots,\widehat{v}_{p}$ be any orthonormal basis of $V_{p}$.
Then

$\displaystyle\sum_{j=1}^{p}\langle\widehat{v}_{i},\overline{\Gamma}\widehat{v}_{i}\rangle=\sum_{k=1}^{\infty}\lambda_{k}\|P_{V_{p}}\phi_{k}\|^{2}.$
(C.10)

Here $P_{V_{p}}:H\to V_{p}$ denotes the orthogonal projection onto $V_{p}$.

###### Proof.

Let $u_{1},\dots,u_{p}\in V_{p}$ be an orthonormal basis $V_{p}$. Let
$\lambda_{1}\geq\lambda_{2}\geq\dots$ denote the eigenvalues of
$\overline{\Gamma}$ with corresponding orthonormal eigenbasis
$\phi_{1},\phi_{2},\dots$. Then

$u_{j}=\sum_{k=1}^{\infty}\langle u_{j},\phi_{k}\rangle\phi_{k},$

and hence

$\displaystyle\sum_{j=1}^{p}\left\langle
u_{j},\overline{\Gamma}u_{j}\right\rangle$
$\displaystyle=\sum_{j=1}^{p}\left\langle
u_{j},\overline{\Gamma}\sum_{k=1}^{\infty}\langle
u_{j},\phi_{k}\rangle\phi_{k}\right\rangle$
$\displaystyle=\sum_{j=1}^{p}\sum_{k=1}^{\infty}\langle
u_{j},\phi_{k}\rangle\left\langle
u_{j},\overline{\Gamma}\phi_{k}\right\rangle$
$\displaystyle=\sum_{j=1}^{p}\sum_{k=1}^{\infty}\lambda_{k}\left|\langle
u_{j},\phi_{k}\rangle\right|^{2}$
$\displaystyle=\sum_{k=1}^{\infty}\lambda_{k}\left(\sum_{j=1}^{p}\left|\langle
u_{j},\phi_{k}\rangle\right|^{2}\right)$
$\displaystyle=\sum_{k=1}^{\infty}\lambda_{k}\left\|P_{V_{p}}\phi_{k}\right\|^{2}.$

∎

######  Lemma C.7.

If $V_{p}$ is a minimizing subspace, i.e. a subspace which minimizes
$\widehat{\mathscr{E}}_{\mathrm{Proj}}$, then

$\displaystyle\bigoplus_{j=1}^{n-1}E_{j}\subset
V_{p}\subset\bigoplus_{j=1}^{n}E_{j},$ (C.11)

where $n$ is chosen such that

$\dim\Big{(}\bigoplus_{j=1}^{n-1}E_{j}\Big{)}=\sum_{j=1}^{n-1}\dim(E_{j})<p\leq\sum_{j=1}^{n}\dim(E_{j})=\dim\Big{(}\bigoplus_{j=1}^{n}E_{j}\Big{)}.$

Here, the $E_{j}$ denote the eigenspaces of

$\overline{\Gamma}=\Gamma+\mathbb{E}[v]\otimes\mathbb{E}[v]=\sum_{j=1}^{\infty}\overline{\lambda}_{j}P_{E_{j}},$

corresponding to the (distinct) eigenvalues
$\overline{\lambda}_{1}>\overline{\lambda}_{2}>\overline{\lambda}_{3}>\dots$
of $\overline{\Gamma}$.

###### Proof.

Let $V_{p}\subset H$ be a minimizer of $\widehat{\mathscr{E}}_{\mathrm{Proj}}$
with respect to $\nu$ of dimension $\dim(V_{p})=p$. For each eigenspace
$E_{j}$ of $\overline{\Gamma}$, denote

$p_{j}=\dim(E_{j}),$

and define $p_{0}:=0$. Now, we choose an orthonormal eigenbasis
$\phi_{1},\phi_{2},\dots$ of $\overline{\Gamma}$, with corresponding
eigenvalues $\lambda_{1}\geq\lambda_{2}\geq\dots$, such that

$E_{j}=\mathrm{span}(\phi_{p_{j-1}+1},\dots,\phi_{p_{j}}),\quad(j\in\mathbb{N}).$

Note that the eigenvalues thus satisfy

$\underbrace{\lambda_{1}=\dots=\lambda_{p_{1}}}_{=\overline{\lambda}_{1}}>\underbrace{\lambda_{p_{1}+1}=\dots=\lambda_{p_{2}}}_{=\overline{\lambda}_{2}}>\dots$

By Lemma C.3 and C.6, $V_{p}$ is a maximizer of the mapping

$\displaystyle\widehat{V}\mapsto\sum_{k=1}^{\infty}\lambda_{k}\left\|P_{\widehat{V}}\phi_{k}\right\|^{2},$

among all $p$-dimensional subspaces $\widehat{V}\subset X$. Given a
$p$-dimensional subspace $\widehat{V}\subset X$, let $u_{1},\dots,u_{p}$ be an
orthonormal basis of $\widehat{V}$. We note that

$\displaystyle\sum_{k=1}^{\infty}\left\|P_{\widehat{V}}\phi_{k}\right\|^{2}=\sum_{k=1}^{\infty}\sum_{j=1}^{p}\left|\langle
u_{j},\phi_{k}\rangle\right|^{2}=\sum_{j=1}^{p}\|u_{j}\|^{2}=p,$

And for all $k\in\mathbb{N}$, we have

$0\leq\left\|P_{\widehat{V}}\phi_{k}\right\|^{2}\leq 1.$

Thus, for any $p$-dimensional subspace, the coefficients

$\alpha_{k}=\alpha_{k}(\widehat{V}):=\|P_{\widehat{V}}\phi_{k}\|^{2},\quad(k\in\mathbb{N}),$

belong to the set

$\mathcal{A}_{p}:={\left\\{\alpha=(\alpha_{k})_{k\in\mathbb{N}}\,\middle|\,\alpha_{k}\in[0,1]\,\forall\,k\in\mathbb{N},\;\sum_{k=1}^{\infty}\alpha_{k}=p\right\\}}.$

And we are interested in the maximizer $\widehat{V}=V_{p}$ of

$(\alpha_{k}(\widehat{V}))_{k\in\mathbb{N}}\mapsto\sum_{k=1}^{\infty}\lambda_{k}\alpha_{k}(\widehat{V}).$

We now make the following claim:

######  Claim.

For any $(\alpha_{k})_{k\in\mathbb{N}}\in\mathcal{A}_{p}$, we have

$\displaystyle\sum_{k=1}^{\infty}\lambda_{k}\alpha_{k}\leq\sum_{k=1}^{p}\lambda_{k},$
(C.12)

with equality, if and only if

$\alpha_{k}=\begin{cases}1,&\left(\text{if
}k\leq\sum_{j=1}^{n-1}p_{j}\right),\\\ 0,&\left(\text{if
}k>\sum_{j=1}^{n}p_{j}\right),\end{cases}$

where $n$ is chosen such that

$\sum_{j=1}^{n-1}p_{j}\leq p<\sum_{j=1}^{n}p_{j}.$

Before proving the above claim, we show that it implies (C.11). Indeed, for
the subspace $\widehat{V}=\mathrm{span}(\phi_{1},\dots,\phi_{p})$, we clearly
have

$\alpha_{k}(\widehat{V})=\|P_{\widehat{V}}\phi_{k}\|^{2}=\begin{cases}1,&(k\leq
p),\\\ 0,&(k>p),\end{cases}$

and hence

$\sum_{k=1}^{\infty}\lambda_{k}\alpha_{k}(\widehat{V})=\sum_{k=1}^{p}\lambda_{k},$

achieves the upper bound in (C.12). If $V_{p}$ is another optimizer, then we
must thus also have

$\sum_{k=1}^{\infty}\lambda_{k}\alpha_{k}(V_{p})=\sum_{k=1}^{p}\lambda_{k}.$

The claim then implies that

$\|P_{V_{p}}\phi_{k}\|^{2}=\begin{cases}1,&\left(\text{if
}k\leq\sum_{j=1}^{n-1}p_{j}\right),\\\ 0,&\left(\text{if
}k>\sum_{j=1}^{n}p_{j}\right).\end{cases}$

The latter is equivalent to the statement that

$\phi_{k}\in V_{p},\;\text{for
}k\leq\sum_{j=1}^{n-1}p_{j},\quad\text{and}\quad\phi_{k}\perp
V_{p},\;\text{for }k>\sum_{j=1}^{n}p_{j},$

i.e. that

$\bigoplus_{j=1}^{n-1}E_{j}\subset
V_{p},\quad\text{and}\quad\bigoplus_{j=n+1}^{\infty}E_{j}\perp V_{p}.$

Thus, assuming the claim (C.12), it follows that for any optimal subspace
$V_{p}$, we must have

$\bigoplus_{j=1}^{n-1}E_{j}\subset V_{p}\subset\bigoplus_{j=1}^{n}E_{j}.$

We finally need to prove the claim: To prove the inequality (C.12), we simply
note that

$\displaystyle\sum_{k=1}^{\infty}\lambda_{k}\alpha_{k}-\sum_{k=1}^{p}\lambda_{k}$
$\displaystyle=\sum_{k=1}^{\infty}\lambda_{k}\alpha_{k}-\sum_{k=1}^{p}\lambda_{k}\alpha_{k}-\sum_{k=1}^{p}\lambda_{k}(1-\alpha_{k})$
$\displaystyle=\sum_{k>p}\lambda_{k}\alpha_{k}-\sum_{k=1}^{p}\lambda_{k}(1-\alpha_{k}).$

Since the sequence of eigenvalues $\lambda_{k}$ is monotonically decreasing
and $\alpha_{k}\geq 0$, $1-\alpha_{k}\geq 0$, we have

$\sum_{k>p}\lambda_{k}\alpha_{k}\leq\sum_{k>p}\lambda_{p}\alpha_{k},\quad\text{and}\quad\sum_{k=1}^{p}\lambda_{k}(1-\alpha_{k})\geq\sum_{k=1}^{p}\lambda_{p}(1-\alpha_{k}),$

and hence

$\displaystyle\sum_{k>p}\lambda_{k}\alpha_{k}-\sum_{k=1}^{p}\lambda_{k}(1-\alpha_{k})$
$\displaystyle\leq\sum_{k>p}\lambda_{p}\alpha_{k}-\sum_{k=1}^{p}\lambda_{p}(1-\alpha_{k})$
$\displaystyle=\sum_{k>p}\lambda_{p}\alpha_{k}-p\lambda_{p}+\sum_{k=1}^{p}\lambda_{p}\alpha_{k}$
$\displaystyle=\lambda_{p}\left(\sum_{k=1}^{\infty}\alpha_{k}-p\right)$
$\displaystyle=0.$

The last line follows from the fact that $\sum_{k=1}^{\infty}\alpha_{k}=p$,
for any $(\alpha_{k})_{k\in\mathbb{N}}\in\mathcal{A}_{p}$. We thus conclude
that

$\sum_{k=1}^{\infty}\lambda_{k}\alpha_{k}\leq\sum_{k=1}^{p}\lambda_{k},$

for all $(\alpha_{k})_{k\in\mathbb{N}}\in\mathcal{A}_{p}$. Furthermore, if
$\sum_{k=1}^{\infty}\lambda_{k}\alpha_{k}=\sum_{k=1}^{p}\lambda_{k}$, then in
all the above estimates, we must have equality. In particular, we must have

$\sum_{k>p}\lambda_{k}\alpha_{k}=\sum_{k>p}\lambda_{p}\alpha_{k},\quad\sum_{k=1}^{p}\lambda_{k}(1-\alpha_{k})=\sum_{k=1}^{p}\lambda_{p}(1-\alpha_{k}).$

The first equality is only possible, if $\alpha_{k}=0$, for any
$k\in\mathbb{N}$, such that $\lambda_{k}<\lambda_{p}$, i.e. we must have

$\alpha_{k}=0,\quad\text{if }k>\sum_{j=1}^{n}p_{j}.$

The second equality is only possible, if $\alpha_{k}=1$, for any
$k\in\mathbb{N}$, such that $\lambda_{k}>\lambda_{p}$, i.e. we must have

$\alpha_{k}=1,\quad\text{if }k\leq\sum_{j=1}^{n-1}p_{j}.$

This concludes the proof. ∎

######  Remark C.8.

It is also straight-forward to check that if $\widehat{V}\subset H$ is a
$p$-dimensional subspace, such that

$\bigoplus_{k=1}^{n-1}E_{k}\subset\widehat{V}\subset\bigoplus_{k=1}^{n}E_{k},$

where

$\sum_{k=1}^{n-1}\dim(E_{k})<p\leq\sum_{k=1}^{n}\dim(E_{k}),$

then

$\sum_{k=1}^{\infty}\lambda_{k}\|P_{\widehat{V}}\phi_{k}\|^{2}=\sum_{k=1}^{p}\lambda_{k},$

i.e. $\widehat{V}$ is optimal for
$\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V};\nu)$, in this case.

From the last lemma, we immediately have the following corollary.

######  Corollary C.9.

For any $n\in\mathbb{N}$, the minimizing subspace $V_{p}$ of
$\widehat{\mathscr{E}}_{\mathrm{Proj}}(V_{p};\nu)$ of dimension

$p=\sum_{k=1}^{n}\dim(E_{k}),$

is unique.

###### Proof.

By Lemma C.7, if $V_{p}$ is any minimizer of
$\widehat{\mathscr{E}}_{\mathrm{Proj}}$, then

$V_{p}\subset\bigoplus_{k=1}^{n}E_{k}.$

Since $\dim(V_{p})=p=\sum_{k=1}^{n}\dim(E_{k})$, by assumption, it follows
that $V_{p}=\bigoplus_{k=1}^{n}E_{k}$ is uniquely determined by the
eigenspaces $E_{1},\dots,E_{n}$. ∎

####  C.3.1. Proof of Theorem 3.10

###### Proof.

The existence and characterization of optimal subspaces $\widehat{V}$ is a
consequence of Lemma C.7 and Remark C.8. Uniqueness of the optimal subspace
$V_{p}\subset H$ for $p_{n}=\sum_{k=1}^{n}\dim(E_{k})$ is proved in Corollary
C.9. Finally, the identity for the projection error
$\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V})=\sum_{k>p}\lambda_{k}$ for
the optimal subspace $V_{p}=\mathrm{span}(\phi_{1},\dots,\phi_{p})$, with
$\phi_{k}$ denote the eigenfunctions of the uncentered covariance operator
$\overline{\Gamma}$ corresponding to decreasing eigenvalues
$\lambda_{1}\geq\lambda_{2}\geq\dots$, follows from

$\displaystyle\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V})$
$\displaystyle\overset{\mathclap{\underset{\downarrow}{\eqref{eq:linear}}}}{=}\int_{H}\|v\|^{2}\,d\mu(u)-\sum_{i=1}^{p}\int_{H}|\langle
v,\phi_{i}\rangle|^{2}\,d\mu(v)$
$\displaystyle={\mathrm{Tr}}(\overline{\Gamma})-\sum_{i=1}^{p}\langle\phi_{i},\overline{\Gamma}\phi_{i}\rangle$
$\displaystyle\overset{\mathclap{\underset{\downarrow}{\eqref{eq:Eproj-
equiv2}}}}{=}\sum_{k=1}^{\infty}\lambda_{k}-\sum_{k=1}^{\infty}\lambda_{k}\|P_{V_{p}}\phi_{k}\|^{2}$
$\displaystyle=\sum_{k=1}^{\infty}\lambda_{k}-\sum_{k=1}^{p}\lambda_{k}=\sum_{k>p}\lambda_{k}.$

∎

####  C.3.2. Proof of Theorem 3.12

###### Proof.

Since $V_{0}$ is a finite-dimensional affine subspace, the existence and
uniqueness of

$\widehat{v}_{0}\in\operatorname*{argmin}_{\widehat{v}\in\widehat{V}_{0}}\|\mathbb{E}[v]-\widehat{v}\|^{2},$

is straight-forward. We also note that
$\mathbb{E}[v]-\widehat{v}_{0}\perp\widehat{V}$. We can now write

$\displaystyle\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V}_{0})$
$\displaystyle=\int_{H}\inf_{\widehat{v}\in\widehat{V}_{0}}\|{v}-\widehat{v}\|^{2}\,d\nu(v)$
$\displaystyle=\int_{H}\inf_{\widehat{v}\in\widehat{V}}\|{v}-\widehat{v}_{0}-\widehat{v}\|^{2}\,d\nu(v)$
$\displaystyle=\int_{H}\inf_{\widehat{v}\in\widehat{V}}\|[{v}-\mathbb{E}[v]-\widehat{v}]-[\mathbb{E}[v]-\widehat{v}_{0}]\|^{2}\,d\nu(v)$
$\displaystyle=\int_{H}\inf_{\widehat{v}\in\widehat{V}}\left\\{\|\mathbb{E}[v]-\widehat{v}_{0}\|^{2}-2\langle{v}-\mathbb{E}[v]-\widehat{v},\mathbb{E}[v]-\widehat{v}_{0}\rangle\right.$
$\displaystyle\hskip
56.9055pt\left.+\|{v}-\mathbb{E}[v]-\widehat{v}\|^{2}\right\\}\,d\nu(v)$

Since $\mathbb{E}[v]-\widehat{v}_{0}\perp\widehat{v}$ for all
$\widehat{v}\in\widehat{V}$, we have

$2\langle{v}-\mathbb{E}[v]-\widehat{v},\mathbb{E}[v]-\widehat{v}_{0}\rangle=2\langle{v}-\mathbb{E}[v],\mathbb{E}[v]-\widehat{v}_{0}\rangle,$

and thus

$\displaystyle\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V}_{0})$
$\displaystyle=\int_{H}\inf_{\widehat{v}\in\widehat{V}}\left\\{\|\mathbb{E}[v]-\widehat{v}_{0}\|^{2}-2\langle{v}-\mathbb{E}[v],\mathbb{E}[v]-\widehat{v}_{0}\rangle\right.$
$\displaystyle\hskip
56.9055pt\left.+\|{v}-\mathbb{E}[v]-\widehat{v}\|^{2}\right\\}\,d\nu(v)$
$\displaystyle=\int_{H}\|\mathbb{E}[v]-\widehat{v}_{0}\|^{2}\,d\nu(v)$
$\displaystyle\quad-2\int_{H}\langle{v}-\mathbb{E}[v],\mathbb{E}[v]-\widehat{v}_{0}\rangle\,d\nu(v)$
$\displaystyle\quad+\int_{H}\inf_{\widehat{v}\in\widehat{V}}\|{v}-\mathbb{E}[v]-\widehat{v}\|^{2}\,d\nu(v).$

The first term is independent of $v$, the second term averages to zero. Hence,
we finally obtain

$\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V}_{0})=\|\mathbb{E}[v]-\widehat{v}_{0}\|^{2}+\int_{H}\inf_{\widehat{v}\in\widehat{V}}\|{v}-\mathbb{E}[v]-\widehat{v}\|^{2}\,d\nu(v).$

Note now if $\widehat{V}_{0}$ is an affine subspace with associated vector
space $\widehat{V}$ for which the first term is not equal to $0$, then the
affine subspace $\widehat{V}_{0}^{\prime}:=\mathbb{E}[v]+\widehat{V}$
satisfies

$\displaystyle\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V}_{0}^{\prime})$
$\displaystyle=\int_{H}\inf_{\widehat{v}\in\widehat{V}}\|v-\mathbb{E}[v]-\widehat{v}\|^{2}\,d\nu(v)$
$\displaystyle<\|\mathbb{E}[v]-\widehat{v}_{0}\|^{2}+\int_{H}\inf_{\widehat{v}\in\widehat{V}}\|v-\mathbb{E}[v]-\widehat{v}\|^{2}\,d\nu(v)$
$\displaystyle=\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V}_{0}).$

Thus, if $\widehat{V}_{0}$ is a minimizer among affine subspaces, then we must
have $\widehat{v}_{0}=\mathbb{E}[v]$. Next, define a measure
$\overline{\nu}\in\mathcal{P}(H)$ by

$\int_{H}\Phi(v)\,d\overline{\nu}(v):=\int_{H}\Phi(v-\mathbb{E}[v])\,d\nu(v),\quad\forall\,\Phi\in
L^{\infty}(H).$

Then $\int_{H}v\,d\overline{\nu}(v)=0$, and

$\int_{H}\inf_{\widehat{v}\in\widehat{V}}\|v-\mathbb{E}[v]-\widehat{v}\|^{2}\,d\nu(v)=\int_{H}\inf_{\widehat{v}\in\widehat{V}}\|v-\widehat{v}\|^{2}\,d\overline{\nu}(v),$

is the projection error
$\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V};\overline{\nu})$ of the
$p$-dimensional vector space $\widehat{V}$ with respect to the measure
$\overline{\nu}$. Since $\widehat{V}_{0}$ is a minimizer, we must have that
$\widehat{V}$ is a minimizer of
$\widehat{\mathscr{E}}_{\mathrm{Proj}}(\widehat{V})$ among $p$-dimensional
subspaces. By Lemma C.7, it follows that

$\bigoplus_{j=1}^{n-1}E_{j}\subset\widehat{V}\subset\bigoplus_{j=1}^{n}E_{j},$

where $E_{j}$ denote the eigenspaces of the covariance operator of
$\overline{\nu}$, given by

$\int_{H}v\otimes
v\,d\overline{\nu}(v)=\int_{H}(v-\mathbb{E}[v])\otimes(v-\mathbb{E}[v])\,d\nu(v)=\Gamma,$

and corresponding to distinct eigenvalues
$\overline{\lambda}_{1}>\overline{\lambda}_{2}>\dots$ of ${\Gamma}$, as
claimed. This concludes the proof of this theorem. ∎

###  C.4. Proof of Lemma 3.13

###### Proof.

Let $V=\mathrm{span}(\tau_{1},\dots,\tau_{p})$. We first note that since we
have a direct sum $L^{2}(U)=V^{\perp}\oplus V$, we can decompose any $u\in
L^{2}(U)$ uniquely as

$u=u^{\perp}+\sum_{k=1}^{p}\alpha_{k}\tau_{k},$

for coefficients $\alpha_{1},\dots,\alpha_{p}\in\mathbb{R}$. Let
$\tau^{\ast}_{1},\dots,\tau^{\ast}_{p}\in\mathrm{span}(\tau_{1},\dots,\tau_{p})$
denote the dual basis of $\tau_{1},\dots,\tau_{p}$. Taking the inner product
of the last identity with $\tau_{\ell}^{\ast}$, it follows that

$\langle\tau_{\ell}^{\ast},u\rangle=\langle\tau_{\ell}^{\ast},u^{\perp}\rangle+\sum_{k=1}^{p}\alpha_{k}\langle\tau_{\ell}^{\ast},\tau_{k}\rangle=\alpha_{\ell},$

where we have used that $u^{\perp}\perp V\ni\tau_{\ell}^{\ast}$, and
$\langle\tau_{\ell}^{\ast},\tau_{k}\rangle=\delta_{k\ell}$. Applying this
identity, we now note that

$\displaystyle\mathcal{R}\circ\mathcal{P}(u)$
$\displaystyle=\tau_{0}+\sum_{k=1}^{p}[\mathcal{P}(u)]_{k}\tau_{k}=\tau_{0}^{\perp}+\sum_{k=1}^{p}\big{(}[\mathcal{P}(u)]_{k}+\langle\tau_{k}^{\ast},\tau_{0}\rangle\big{)}\tau_{k},$

and

$\displaystyle
u=u^{\perp}+\sum_{k=1}^{p}\langle\tau_{k}^{\ast},u\rangle\tau_{k}.$

Hence

$\displaystyle\mathcal{R}\circ\mathcal{P}(u)-u$
$\displaystyle=\tau_{0}^{\perp}-u^{\perp}+\sum_{k=1}^{p}\big{(}[\mathcal{P}(u)]_{k}+\langle\tau_{k}^{\ast},\tau_{0}-u\rangle\big{)}\tau_{k}.$

The norm of the last term is clearly minimized if the sum over $k=1,\dots,p$
vanishes. This is the case, provided that

$[\mathcal{P}(u)]_{k}=\langle\tau_{k}^{\ast},u-\tau_{0}\rangle,\quad(k=1,\dots,p).$

The claim follows. ∎

###  C.5. Proof of Proposition 3.15

###### Proof.

Denote $\widehat{V}_{0}:=\mathrm{Im}(\mathcal{R})$ the affine image of
$\mathcal{R}=\mathcal{R}_{\bm{\tau}}$. Taking into account Theorem 3.12, we
obtain

$\displaystyle(\widehat{\mathscr{E}}_{\mathcal{R}})^{2}$
$\displaystyle=\int_{Y}\|\mathcal{R}\circ\mathcal{P}-\mathrm{Id}\|^{2}_{L^{2}_{y}}\,d\nu$
$\displaystyle=\inf_{\widehat{v}\in\widehat{V}_{0}}\|\mathbb{E}_{\nu}[v]-\widehat{v}\|_{L^{2}_{y}}^{2}+\sum_{k=1}^{\infty}\lambda_{k}\|\mathcal{R}\circ\mathcal{P}\phi_{k}-\phi_{k}\|^{2}_{L^{2}_{y}}$
$\displaystyle\leq\|\mathbb{E}_{\nu}[v]-\tau_{0}\|_{L^{2}_{y}}^{2}+\sum_{k\leq
p}\lambda_{k}\|\tau_{k}-\phi_{k}\|^{2}_{L^{2}_{y}}+\sum_{k>p}\lambda_{k}\|\mathcal{R}\circ\mathcal{P}\phi_{k}-\phi_{k}\|^{2}_{L^{2}_{y}}$
$\displaystyle=\|\widehat{\tau}_{0}-\tau_{0}\|_{L^{2}_{y}}^{2}+\sum_{k=1}^{p}\lambda_{k}\|\tau_{k}-\widehat{\tau}_{k}\|^{2}_{L^{2}_{y}}+\sum_{k>p}\lambda_{k}\|\mathcal{R}\circ\mathcal{P}\phi_{k}-\phi_{k}\|^{2}_{L^{2}_{y}}$
$\displaystyle\leq\|\widehat{\tau}_{0}-\tau_{0}\|_{L^{2}_{y}}^{2}+\left(\sum_{k=1}^{\infty}\lambda_{k}\right)\sup_{k=1,\dots,p}\|\tau_{k}-\widehat{\tau}_{k}\|^{2}_{L^{2}_{y}}+\sum_{k>p}\lambda_{k}$
$\displaystyle\leq\left[1+{\mathrm{Tr}}(\Gamma_{\nu})\right]\sup_{k=0,1,\dots,p}\|\tau_{k}-\widehat{\tau}_{k}\|^{2}_{L^{2}_{y}}+\sum_{k>p}\lambda_{k}.$

Taking square roots of both sides, the claimed estimate (3.27) now follows
from the trivial inequality $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$ for $a,b\geq 0$.
∎

###  C.6. Proof of Lemma 3.16

###### Proof.

By the triangle inequality, we have

$\displaystyle\widehat{\mathscr{E}}_{\mathcal{R}}$
$\displaystyle\leq\widehat{\mathscr{E}}_{\widetilde{\mathcal{R}}}+\left(\int_{L^{2}}\|\mathcal{R}\circ\mathcal{P}(u)-\widetilde{\mathcal{R}}\circ\widetilde{\mathcal{P}}(u)\|^{2}_{L^{2}}\,d\nu(u)\right)^{1/2}.$

Note that by the assumed orthonormality of the $\widetilde{\tau}_{k}$, the
projection can be written as follows (cp. (3.22))

$\widetilde{\mathcal{R}}\circ\widetilde{\mathcal{P}}(u)=\sum_{k=1}^{p}\langle\widetilde{\tau}_{k},u\rangle\widetilde{\tau}_{k}.$

Furthermore, in terms of the dual basis
$\tau_{1}^{\ast},\dots,\tau_{p}^{\ast}\in Y$, we have

$\mathcal{R}\circ\mathcal{P}(u)=\sum_{k=1}^{p}\langle\tau_{k}^{\ast},u\rangle{\tau}_{k}.$

In terms of this expansion we can again use the triangle inequality to obtain

$\displaystyle\begin{aligned}
\left(\int_{L^{2}}\right.&\|\mathcal{R}\circ\mathcal{P}(u)-\widetilde{\mathcal{R}}\circ\widetilde{\mathcal{P}}(u)\|^{2}_{L^{2}}\left.\,d\nu(u)\right)^{1/2}\\\
&\leq\sum_{k=1}^{p}\left(\int_{L^{2}}\|\langle\tau_{k}^{\ast},u\rangle\tau_{k}-\langle\widetilde{\tau}_{k},u\rangle\widetilde{\tau}_{k}\|^{2}_{L^{2}}\,d\nu(u)\right)^{1/2}\end{aligned}$
(C.13)

Since

$\displaystyle\begin{aligned}
\|\langle\tau_{k}^{\ast},u\rangle\tau_{k}-\langle\widetilde{\tau}_{k},u\rangle\widetilde{\tau}_{k}\|_{L^{2}}&\leq\|\langle\tau_{k}^{\ast},u\rangle(\tau_{k}-\widetilde{\tau}_{k})\|_{L^{2}}+\|\langle\tau_{k}^{\ast}-\widetilde{\tau}_{k},u\rangle\widetilde{\tau}_{k}\|_{L^{2}}\\\
&\leq\|\tau_{k}^{\ast}\|_{L^{2}}\|u\|_{L^{2}}\|\tau_{k}-\widetilde{\tau}_{k}\|_{L^{2}}+\|\tau_{k}^{\ast}-\widetilde{\tau}_{k}\|_{L^{2}}\|u\|_{L^{2}}\|\widetilde{\tau}_{k}\|_{L^{2}},\end{aligned}$
(C.14)

we next wish to establish that under the assumptions of this lemma, we can
bound

$\|\tau_{k}^{\ast}-\widetilde{\tau}_{k}\|_{L^{2}}\leq
C\sqrt{p}\max_{j=1,\dots,p}\|\tau_{j}-\widetilde{\tau}_{j}\|_{L^{2}},$

for some absolute constant $C>0$, independent of $k$ and $p$. To see this, we
note that for any $k,j=1,\dots,p$, we have
$\langle\tau_{k}^{\ast},\tau_{j}\rangle=\delta_{kj}$, and hence

$\langle(\tau_{k}^{\ast}-\widetilde{\tau}_{k}),\widetilde{\tau}_{j}\rangle=\langle\tau_{k}^{\ast},\widetilde{\tau}_{j}\rangle-\delta_{kj}=\langle\tau_{k}^{\ast},\widetilde{\tau}_{j}\rangle-\langle\tau_{k}^{\ast},\tau_{j}\rangle=\langle\tau_{k}^{\ast},(\widetilde{\tau}_{j}-\tau_{j})\rangle.$

If $u\in Y$ is arbitrary, then we obtain from the above identity

$\displaystyle|\langle(\tau_{k}^{\ast}-\widetilde{\tau}_{k}),u\rangle|$
$\displaystyle=\left|\sum_{j=1}^{p}\langle\widetilde{\tau}_{j},u\rangle\langle(\tau_{k}^{\ast}-\widetilde{\tau}_{k}),\widetilde{\tau}_{j}\rangle\right|$
$\displaystyle=\left|\sum_{j=1}^{p}\langle\widetilde{\tau}_{j},u\rangle\langle\tau_{k}^{\ast},\widetilde{\tau}_{j}-\tau_{j}\rangle\right|$
$\displaystyle\leq\left(\sum_{j=1}^{p}|\langle\widetilde{\tau}_{j},u\rangle|^{2}\right)^{1/2}\left(\sum_{j=1}^{p}|\langle\tau_{k}^{\ast},\widetilde{\tau}_{j}-\tau_{j}\rangle|^{2}\right)^{1/2}$
$\displaystyle\leq\|u\|_{L^{2}}\,\|\tau_{k}^{\ast}\|_{L^{2}}\,\sqrt{p}\max_{j=1,\dots,p}\|\tau_{j}-\widetilde{\tau}_{j}\|_{L^{2}}.$
(C.15)

By assumption, we have that

$\sqrt{p}\max_{j=1,\dots,p}\|\tau_{j}-\widetilde{\tau}_{j}\|_{L^{2}}\leq\epsilon\leq\frac{1}{2}.$

It follows that for any $k=1,\dots,p$, we have

$\|\tau_{k}^{\ast}-\widetilde{\tau}_{k}\|_{L^{2}}=\sup_{\|u\|\leq
1}|\langle\tau_{k}^{\ast}-\widetilde{\tau}_{k},u\rangle|\leq\epsilon\,\|\tau_{k}^{\ast}\|_{L^{2}}\leq\epsilon\,\|\tau_{k}^{\ast}-\widetilde{\tau}_{k}\|_{L^{2}}+\epsilon\,\|\widetilde{\tau}_{k}\|_{L^{2}},$

and hence

$\|\tau_{k}^{\ast}-\widetilde{\tau}_{k}\|_{L^{2}}\leq\frac{\epsilon}{1-\epsilon}\|\widetilde{\tau}_{k}\|_{L^{2}}=\frac{\epsilon}{1-\epsilon}\leq
2\epsilon.$

Note also that this implies

$\displaystyle\|\tau_{k}^{\ast}\|_{L^{2}}\leq\|\tau_{k}^{\ast}-\widetilde{\tau}_{k}\|_{L^{2}}+\|\widetilde{\tau}_{k}\|_{L^{2}}\leq
2\epsilon+1\leq 2.$ (C.16)

It now follows from (C.14), that

$\displaystyle\|\langle\tau_{k}^{\ast},u\rangle\tau_{k}-\langle\widetilde{\tau}_{k},u\rangle\widetilde{\tau}_{k}\|_{L^{2}}$
$\displaystyle\leq
2\|u\|_{L^{2}}\left((\sqrt{p}+1)\max_{j=1,\dots,p}\|\tau_{j}-\widetilde{\tau}_{j}\|_{L^{2}}.\right)$
$\displaystyle\leq
4\|u\|_{L^{2}}\sqrt{p}\,\max_{j=1,\dots,p}\|\tau_{j}-\widetilde{\tau}_{j}\|_{L^{2}}.$

Substitution in (C.13), finally yields

$\widehat{\mathscr{E}}_{\mathcal{R}}\leq\widehat{\mathscr{E}}_{\widetilde{\mathcal{R}}}+4\left(\int_{L^{2}}\|u\|^{2}_{L^{2}}\,d\nu(u)\right)^{1/2}\,p^{3/2}\,\max_{j=1,\dots,p}\|\tau_{j}-\widetilde{\tau}_{j}\|_{L^{2}}.$

Letting

$C:=4\left(\int_{L^{2}}\|u\|^{2}_{L^{2}}\,d\nu(u)\right)^{1/2},$

we conclude that

$\widehat{\mathscr{E}}_{\mathcal{R}}\leq\widehat{\mathscr{E}}_{\widetilde{\mathcal{R}}}+Cp^{3/2}\,\max_{j=1,\dots,p}\|\tau_{j}-\widetilde{\tau}_{j}\|_{L^{2}}\leq\widehat{\mathscr{E}}_{\widetilde{\mathcal{R}}}+C\epsilon,$

as claimed.

To prove the Lipschitz bound on $\mathcal{P}:L^{2}(U)\to\mathbb{R}^{p}$, let
$u,u^{\prime}\in L^{2}(U)$ be given, and denote $w=u-u^{\prime}$. Then

$\displaystyle\|\mathcal{P}(u)-\mathcal{P}(u^{\prime})\|_{\ell^{2}}$
$\displaystyle=\left(\sum_{k=1}^{p}|\langle\tau_{k}^{\ast},w\rangle|^{2}\right)^{1/2}$
$\displaystyle\leq\left(\sum_{k=1}^{p}|\langle\tau_{k}^{\ast}-\widetilde{\tau}_{k},w\rangle|^{2}\right)^{1/2}+\left(\sum_{k=1}^{p}|\langle\widetilde{\tau}_{k},w\rangle|^{2}\right)^{1/2}$

By (C.15) and (C.16), we can bound each term in the first sum by

$|\langle\tau_{k}^{\ast}-\widetilde{\tau}_{k},w\rangle|\leq
2\sqrt{p}\max_{j=1,\dots,p}\|\tau_{j}-\widetilde{\tau}_{j}\|_{L^{2}}\,\|u\|_{L^{2}}.$

Hence,

$\left(\sum_{k=1}^{p}|\langle\tau_{k}^{\ast}-\widetilde{\tau}_{k},w\rangle|^{2}\right)^{1/2}\leq
2p\max_{j=1,\dots,p}\|\tau_{j}-\widetilde{\tau}_{j}\|_{L^{2}}\|u\|_{L^{2}}.$

As the $\widetilde{\tau}_{k}$ are orthonormal by assumption, the second sum
can be estimated simply by

$\left(\sum_{k=1}^{p}|\langle\widetilde{\tau}_{k},w\rangle|^{2}\right)^{1/2}\leq\|w\|_{L^{2}}.$

Recalling that $w=u-u^{\prime}$, it follows that

$\displaystyle\|\mathcal{P}(u)-\mathcal{P}(u^{\prime})\|_{\ell^{2}}$
$\displaystyle\leq\left(1+2p\max_{j=1,\dots,p}\|\tau_{j}-\widetilde{\tau}_{j}\|_{L^{2}}\right)\,\|u-u^{\prime}\|_{L^{2}}.$
$\displaystyle\leq\left(1+2\epsilon\right)\,\|u-u^{\prime}\|_{L^{2}}\leq
2\|u-u^{\prime}\|_{L^{2}}.$

As $u,u^{\prime}\in L^{2}(U)$ were arbitrary, the claimed estimate for
$\mathrm{Lip}(\mathcal{P})$ follows. Similarly, we can estimate for the
reconstruction:

$\displaystyle\|\mathcal{R}(\alpha)-\mathcal{R}(\alpha^{\prime})\|_{L^{2}}$
$\displaystyle=\left\|\sum_{k=1}^{p}(\alpha_{k}-\alpha_{k}^{\prime})\tau_{k}\right\|_{L^{2}}$
$\displaystyle\leq\left\|\sum_{k=1}^{p}(\alpha_{k}-\alpha_{k}^{\prime})\widetilde{\tau}_{k}\right\|_{L^{2}}+\left\|\sum_{k=1}^{p}(\alpha_{k}-\alpha_{k}^{\prime})(\tau_{k}-\widetilde{\tau}_{k})\right\|_{L^{2}}$
$\displaystyle\leq\|\alpha-\alpha^{\prime}\|_{\ell^{2}}+\sum_{k=1}^{p}|\alpha_{k}-\alpha_{k}^{\prime}|\|\tau_{k}-\widetilde{\tau_{k}}\|_{L^{2}}$
$\displaystyle\leq\left(1+p^{1/2}\max_{k=1,\dots,p}\|\tau_{k}-\widetilde{\tau_{k}}\|_{L^{2}}\right)\|\alpha-\alpha^{\prime}\|_{\ell^{2}}$
$\displaystyle\leq\left(1+\epsilon\right)\|\alpha-\alpha^{\prime}\|_{\ell^{2}}\leq
2\|\alpha-\alpha^{\prime}\|_{\ell^{2}}.$

Thus, $\mathrm{Lip}(\mathcal{R})\leq 2$. ∎

###  C.7. Proof of Lemma 3.17

###### Proof.

We note that each element in the (real) trigonometric basis
$\bm{\mathrm{e}}_{1},\dots,\bm{\mathrm{e}}_{p}$ can be expressed in the form

$\bm{\mathrm{e}}_{j}({x})=\cos(\kappa\cdot{x}),\quad\text{or}\quad\bm{\mathrm{e}}_{j}({x})=\sin(\kappa\cdot{x}),$

for $\kappa=\kappa(j)\in\mathbb{Z}^{n}$ with $|{k}|_{\infty}\leq N$, where $N$
is chosen as the smallest natural number such that $p\leq(2N+1)^{n}$. It
follows from

$|\kappa\cdot{x}|\leq n|\kappa|_{\infty}|{x}|_{\infty}\leq 2\pi nN,$

that if
$\mathcal{C}_{\epsilon},\mathcal{S}_{\epsilon}:\mathbb{R}\to\mathbb{R}$ are
neural networks, such that

$\displaystyle\sup_{\xi\in[0,2\pi
nN]}|\mathcal{C}_{\epsilon}(\xi)-\cos(\xi)|,\;|\mathcal{S}_{\epsilon}(\xi)-\sin(\xi)|\leq\epsilon,$
(C.17)

then the map

${x}\mapsto((\kappa(j)\cdot{x}))_{j=1,\dots,p}\mapsto(\mathcal{C}_{\epsilon}(\kappa(j)\cdot{x}),\mathcal{S}_{\epsilon}(\kappa(j)\cdot{x}))_{j=1,\dots,p}$

can be represented by a neural network $\mathcal{T}$ with a size bounded by

$\begin{gathered}\mathrm{size}(\mathcal{T})=\mathcal{O}(p\,\mathrm{size}(\mathcal{C}_{\epsilon})+p\,\mathrm{size}(\mathcal{S}_{\epsilon})),\\\
0pt(\mathcal{N})=\mathcal{O}\left(\max\left(0pt(\mathcal{C}_{\epsilon}),0pt(\mathcal{S}_{\epsilon})\right)\right).\end{gathered}$

By (Elbrächter et al. 2021, Theorem III.9), there exist
$\mathcal{C}_{\epsilon}$, $\mathcal{S}_{\epsilon}$, satisfying (C.17), with

$\begin{gathered}\mathrm{size}(\mathcal{C}_{\epsilon}),\mathrm{size}(\mathcal{S}_{\epsilon})=\mathcal{O}(\log(\epsilon^{-1})^{2}+\log(N)),\\\
0pt(\mathcal{C}_{\epsilon}),0pt(\mathcal{S}_{\epsilon})=\mathcal{O}(\log(\epsilon^{-1})^{2}+\log(N)).\end{gathered}$

Finally, we note that $\log(N)\sim 1/n\log(p)=\mathcal{O}(\log(p))$. We thus
conclude that for any $\epsilon>0$, there exists a neural network
$\mathcal{T}_{\epsilon}=(\mathcal{T}_{\epsilon,1},\dots,\mathcal{T}_{\epsilon,p})$,
such that

$\|\mathcal{T}_{\epsilon,j}-\bm{\mathrm{e}}_{j}\|_{L^{\infty}([0,2\pi]^{n})}\leq\epsilon,$

for all $j=1,\dots,p$, and

$\begin{gathered}\mathrm{size}(\mathcal{T}_{\epsilon})=\mathcal{O}(p\,(\log(p)+\log(\epsilon^{-1})^{2}),\\\
0pt(\mathcal{T}_{\epsilon})=\mathcal{O}(\log(p)+\log(\epsilon^{-1})^{2}).\end{gathered}$

To satisfy the estimate (3.32), we set
$\bm{\tau}=\mathcal{T}_{\epsilon/p^{3/2}}$, for which we have

$\begin{gathered}\mathrm{size}(\bm{\tau})=\mathcal{O}(p\,\log(\epsilon^{-1}p)^{2}),\\\
0pt(\bm{\tau})=\mathcal{O}(\log(\epsilon^{-1}p)^{2}).\end{gathered}$

and

$\max_{j=1,\dots,p}\|\tau_{j}-\bm{\mathrm{e}}_{j}\|_{L^{\infty}([0,2\pi]^{n})}\leq\epsilon.$

∎

###  C.8. Proof of Proposition 3.19

###### Proof.

Proposition 3.19 follows form the following two claims:

######  Claim.

If $\mu$ is a non-degenerate Gaussian, then there exists a Lipschitz
continuous mapping $P:X\to[0,1]$, such that $P_{\\#}\mu=dx$ is the uniform
measure on $[0,1]$.

######  Claim.

If $Y$ is an infinite-dimensional Hilbert space with orthonormal basis
$\\{e_{k}\\}_{k\in\mathbb{N}}$, and $\gamma_{k}\geq 0$, $k\in\mathbb{N}$, a
sequence, such that

$\sum_{k=1}^{\infty}k^{2}\gamma_{k}<\infty,$

then there exists a Lipschitz continuous mapping $G:[0,1]\to Y$, such that the
covariance operator of $G_{\\#}dx$ is given by

$\Gamma_{G_{\\#}dx}=\sum_{k=1}^{\infty}\gamma_{k}(e_{k}\otimes e_{k}).$

The sought-after map $\mathcal{G}$ can then be defined as $\mathcal{G}=G\circ
P:X\to Y$, where $P:X\to[0,1]$ and $G:[0,1]\to Y$ are defined as in the above
claims.

To prove the first claim, we simply note that since $\mu$ is a non-degenerate
Gaussian measure, there exists a one-dimensional projection
$L:X\to\mathbb{R}$, such that $L_{\\#}\mu=\mathcal{N}(m,\sigma^{2})$ is a non-
degenerate Gaussian with mean $m\in\mathbb{R}$, and variance $\sigma^{2}>0$.
Upon performing a translation by $\mu$ and scaling by $1/\sigma$, we obtain an
affine mapping $\widetilde{L}:X\to\mathbb{R}$, such that
$\widetilde{L}_{\\#}\mu=\mathcal{N}(0,1)$. We finally note that the error
function

$\mathrm{erf}(x):=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{x}e^{-u^{2}/2}\,du,$

maps the standard Gaussian distribution to the uniform distribution on
$[0,1]$. And hence, we have $(\mathrm{erf}\circ\widetilde{L})_{\\#}\mu=dx$ on
$[0,1]$.

To prove the second claim, we show that the function

$G:[0,1]\to Y,\quad G(x):=\sum_{k=1}^{\infty}\sqrt{2\gamma_{k}}\cos(2\pi
kx)e_{k},$

possesses the desired properties: We first note that

$\|\partial_{x}G(x)\|_{Y}^{2}=(2\pi)^{2}\sum_{k=1}^{\infty}2\gamma_{k}k^{2}|\sin(2\pi
kx)|^{2}\leq(2\pi)^{2}\sum_{k=1}^{\infty}2\gamma_{k}k^{2}<+\infty,$

is uniformly bounded in $x$ by assumption on $\gamma_{k}$. In particular, it
follows that $x\mapsto G(x)$ is Lipschitz continuous. Furthermore, we have

$\displaystyle\int_{Y}u\otimes u\,d(G_{\\#}dx)$
$\displaystyle=\sum_{k,k^{\prime}=1}^{\infty}2\sqrt{\gamma_{k}\gamma_{k^{\prime}}}\int_{0}^{1}\cos(2\pi
kx)\cos(2\pi k^{\prime}x)(e_{k}\otimes e_{k^{\prime}})\,dx$
$\displaystyle=\sum_{k=1}^{\infty}\gamma_{k}(e_{k}\otimes e_{k}).$

∎

###  C.9. Proof of Proposition 3.20

###### Proof.

Let $v_{1},\dots,v_{p}\in X$ denote pairwise orthogonal eigenvectors
corresponding to the first $p$ eigenvalues $\lambda_{1},\dots,\lambda_{p}$ of
the covariance operator $\Gamma_{\mu}$. It follows from Theorem 3.12, that the
$p$-dimensional affine subspace $V_{0}$, given by $V_{0}=v_{0}+V$, where
$v_{0}=\mathbb{E}_{\mu}[u]$ and $V=\mathrm{span}(v_{1},\dots,v_{p})$,
satisfies

$\int_{X}\inf_{v\in V_{0}}\|v-u\|^{2}\,d\mu(u)=\sum_{j>p}\lambda_{j}.$

Let $W_{0}:=\mathcal{G}(V_{0})$. Since $\mathcal{G}:X\to Y$ is linear, $W_{0}$
is an affine subspace of dimension at most $p$. If $\dim(W_{0})<p$, we extend
$W_{0}$ to an affine subspace $W\subset Y$, such that $W_{0}\subset W$ and
$\dim(W)=p$, else set $W:=W_{0}$. Then, irrespective of the choice of the
extension, we have

$\displaystyle\int_{Y}\inf_{w\in
W}\|w-u\|_{Y}^{2}\,d(\mathcal{G}_{\\#}\mu(u))$
$\displaystyle\leq\int_{Y}\inf_{w\in
W_{0}}\|w-u\|_{Y}^{2}\,d(\mathcal{G}_{\\#}\mu(u))$
$\displaystyle=\int_{X}\inf_{w\in W_{0}}\|w-\mathcal{G}(u)\|_{Y}^{2}\,d\mu(u)$
$\displaystyle=\int_{X}\inf_{v\in
V_{0}}\|\mathcal{G}(v)-\mathcal{G}(u)\|_{Y}^{2}\,d\mu(u)$
$\displaystyle\leq\int_{X}\inf_{v\in
V_{0}}\|\mathcal{G}\|^{2}\|v-u\|_{Y}^{2}\,d\mu(u)$
$\displaystyle=\|\mathcal{G}\|^{2}\,\sum_{j>p}\lambda_{j},$

as claimed. Finally, we note that if $(\mathcal{R},\mathcal{P})$ are chosen
such that $\mathcal{R}\circ\mathcal{P}:Y\to Y$ is the orthogonal projection
onto $W$, then

$(\widehat{\mathscr{E}}_{\mathcal{R}})^{2}=\int_{Y}\|\mathcal{R}\circ\mathcal{P}(u)-u\|^{2}\,d(\mathcal{G}_{\\#}\mu(u))=\int_{Y}\inf_{w\in
W}\|w-u\|^{2}\,d(\mathcal{G}_{\\#}\mu(u)).$

∎

###  C.10. Proof of Theorem 3.8

###### Proof.

Since $\mathcal{D}:\mathbb{R}^{m}\to X$ is assumed to be linear, there exist
$\psi_{1},\dots,\psi_{m}\in X$, such that

$\mathcal{D}(u_{1},\dots,u_{m})=\sum_{k=1}^{m}u_{k}\psi_{k}.$

Let $U:=\mathrm{span}(\psi_{1},\dots,\psi_{m})\subset X$. Since $U$ is a
subspace of $X$ of dimension at most $m$, we have

$\displaystyle\int_{X}\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\|_{L^{2}_{x}}^{2}\,d\mu$
$\displaystyle\geq\int_{X}\inf_{\widehat{u}\in
U}\|u-\widehat{u}\|_{L^{2}_{x}}^{2}\,d\mu(u)$
$\displaystyle\geq\inf_{{\begin{subarray}{c}\widehat{U}\subset X;\\\
\dim(U)=m\end{subarray}}}\int_{X}\inf_{\widehat{u}\in\widehat{U}}\|u-\widehat{u}\|_{L^{2}_{x}}^{2}\,d\mu(u).$

By Theorem 3.10, the infimum on the last line is attained for the subspace
$\widehat{U}$ spanned by the orthonormal eigenfunctions
$\phi_{1},\dots,\phi_{m}$ of the uncentered covariance operator $\Gamma_{\mu}$
of $\mu$. For this space, we have

$\displaystyle\int_{X}\inf_{\widehat{u}\in\widehat{U}}\|u-\widehat{u}\|_{L^{2}_{x}}^{2}\,d\mu(u)$
$\displaystyle=\int_{X}\|u\|^{2}\,d\mu(u)-\sum_{k=1}^{m}\int_{X}|\langle
u,\phi_{k}\rangle|^{2}\,d\mu(u)$
$\displaystyle={\mathrm{Tr}}(\Gamma)-\sum_{k=1}^{m}\langle\phi_{k},\Gamma\phi_{k}\rangle$
$\displaystyle=\sum_{k=1}^{\infty}\lambda_{k}-\sum_{k=1}^{m}\lambda_{k}$
$\displaystyle=\sum_{k>m}\lambda_{k}.$

Thus, we conclude that

$\int_{X}\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\|_{L^{2}_{x}}^{2}\,d\mu\geq\sum_{k>m}\lambda_{k}.$

∎

###  C.11. Proof of Lemma 3.22

###### Proof.

Since $\mathcal{D}\circ\mathcal{E}$ is the identity on
$\mathrm{span}(\phi_{1},\dots,\phi_{m})$, we have

$\mathcal{D}\circ\mathcal{E}-\mathrm{Id}=\mathcal{D}\circ\mathcal{E}\circ
P_{m}^{\perp}-P_{m}^{\perp}.$

Furthermore, since
$\mathrm{Im}(\mathcal{D})\subset\mathrm{span}(\phi_{1},\dots,\phi_{m})$, we
also have $\mathcal{D}\circ\mathcal{E}(P_{m}^{\perp}u)\perp P_{m}^{\perp}u$,
for all $u\in L^{2}_{x}$. Hence

$\displaystyle(\widehat{\mathscr{E}}_{\mathcal{E}})^{2}$
$\displaystyle=\int_{X}\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\|_{L^{2}_{x}}^{2}\,d\mu(u)$
$\displaystyle=\int_{X}\|\mathcal{D}\circ\mathcal{E}(P_{m}^{\perp}u)-P_{m}^{\perp}u\|_{L^{2}_{x}}^{2}\,d\mu(u)$
$\displaystyle=\underbrace{\int_{X}\|\mathcal{D}\circ\mathcal{E}(P_{m}^{\perp}u)\|^{2}_{L^{2}_{x}}\,d\mu(u)}_{\displaystyle=(\widehat{\mathscr{E}}_{\mathrm{aliasing}})^{2}}+\underbrace{\int_{X}\|P_{m}^{\perp}u\|_{L^{2}_{x}}^{2}\,d\mu(u)}_{\displaystyle=(\widehat{\mathscr{E}}_{\perp})^{2}}.$

The aliasing error

$(\widehat{\mathscr{E}}_{\mathrm{aliasing}})^{2}=\int_{X}\|\mathcal{D}\circ\mathcal{E}(P^{\perp}_{m}u)\|_{L^{2}_{x}}^{2}\,d\mu(u),$

can be re-written as follows333we assume $\int_{X}u\,d\mu(u)=0$, but this is
not essential here: Let $\phi_{1},\phi_{2},\dots$ denote the eigenfunctions of
the covariance operator of $\mu$, $\Gamma\phi_{k}=\lambda_{k}\phi_{k}$, where
$\lambda_{1}\geq\lambda_{2}\geq\dots$. First, we note that there exist random
variables $Z_{\ell}$ (not necessarily iid), with

$\mathbb{E}[Z_{\ell}Z_{\ell^{\prime}}]=\delta_{\ell\ell^{\prime}},$

such that the random variable

$\sum_{\ell=1}^{\infty}\sqrt{\lambda_{k}}\,Z_{k}\phi_{k}\sim\mu$

is distributed according to $\mu$. Then, we have

$\displaystyle(\widehat{\mathscr{E}}_{\perp})^{2}$
$\displaystyle=\mathbb{E}\left[\left\|\sum_{\ell=1}^{\infty}\sqrt{\lambda_{\ell}}Z_{\ell}P^{\perp}_{m}\phi_{\ell}\right\|^{2}\right]=\mathbb{E}\left[\left\|\sum_{\ell>m}\sqrt{\lambda_{\ell}}Z_{\ell}\phi_{\ell}\right\|^{2}\right]$
$\displaystyle=\sum_{\ell,\ell^{\prime}>m}\sqrt{\lambda_{\ell}\lambda_{\ell^{\prime}}}\mathbb{E}\left[Z_{\ell}Z_{\ell^{\prime}}\right]\langle\phi_{\ell},\phi_{\ell^{\prime}}\rangle=\sum_{\ell,\ell^{\prime}>m}\sqrt{\lambda_{\ell}\lambda_{\ell^{\prime}}}\,\delta_{\ell,\ell^{\prime}}$
$\displaystyle=\sum_{\ell>m}\lambda_{\ell}.$

With $\Phi_{M}:=(\phi_{i}(X_{j}))\in\mathbb{R}^{m\times M}$ a matrix such that
$\det(\Phi_{M}\Phi_{M}^{T})\neq 0$, the aliasing error is given by

$\displaystyle\int_{X}\|\mathcal{D}\circ\mathcal{E}(P^{\perp}_{m}u)\|_{L^{2}_{x}}^{2}\,d\mu(u)$
$\displaystyle=\mathbb{E}\left[\left\|\sum_{k=1}^{m}\phi_{k}\sum_{j=1}^{m}[\Phi^{\dagger}]_{kj}\sum_{\ell>m}\sqrt{\lambda_{\ell}}\,Z_{\ell}\phi_{\ell}(X_{j})\right\|^{2}\right]$
$\displaystyle=\sum_{k=1}^{m}\|\phi_{k}\|^{2}\sum_{j,j^{\prime}=1}^{m}[\Phi^{\dagger}]_{kj}[\Phi^{\dagger}]_{kj^{\prime}}\quad\times$
$\displaystyle\qquad\sum_{\ell,\ell^{\prime}>m}\sqrt{\lambda_{\ell}\lambda_{\ell^{\prime}}}\,\mathbb{E}\left[Z_{\ell}Z_{\ell^{\prime}}\right]\phi_{\ell}(X_{j})\phi_{\ell^{\prime}}(X_{j})$
$\displaystyle=\sum_{k=1}^{m}\|\phi_{k}\|^{2}\sum_{j,j^{\prime}=1}^{m}[\Phi^{\dagger}]_{kj}[\Phi^{\dagger}]_{kj^{\prime}}\sum_{\ell>m}\lambda_{\ell}\phi_{\ell}(X_{j})\phi_{\ell}(X_{j})$
$\displaystyle=\sum_{\ell>m}\lambda_{\ell}\sum_{k=1}^{m}\Bigg{(}\sum_{j=1}^{m}[\Phi^{\dagger}]_{kj}\phi_{\ell}(X_{j})\Bigg{)}^{2}$

Denote $\phi_{\ell}(\bm{X}):=(\phi_{\ell}(X_{1}),\dots,\phi_{\ell}(X_{m}))$.
Then we can write the last line equivalently in the form

$(\widehat{\mathscr{E}}_{\mathrm{aliasing}})^{2}=\int_{X}\|\mathcal{D}\circ\mathcal{E}(P^{\perp}_{m}u)\|_{L^{2}_{x}}^{2}\,d\mu(u)=\sum_{\ell>m}\lambda_{\ell}\|\Phi^{\dagger}\phi_{\ell}(\bm{X})\|^{2}_{\ell^{2}},$

as claimed. ∎

###  C.12. Proof of Lemma 3.23

###### Proof.

The minimal singular value of $A_{M}=\frac{|D|}{M}\Phi_{M}\Phi_{M}^{T}$ is
given by

$\displaystyle\sigma_{\mathrm{min}}\left(A_{M}\right)$
$\displaystyle=\inf_{\|v\|_{\ell^{2}}=1}\langle v,A_{M}v\rangle$
$\displaystyle=\inf_{\|v\|_{\ell^{2}}=1}\left[\langle v,\bm{1}v\rangle-\langle
v,(\bm{1}-A_{M})v\rangle\right]$
$\displaystyle=1-\sup_{\|v\|_{\ell^{2}}=1}\langle v,(\bm{1}-A_{M})v\rangle$
$\displaystyle=1-\sigma_{\mathrm{max}}\left(\bm{1}-A_{M}\right).$

The maximal singular value (=spectral norm) of
$\bm{1}-A_{M}\in\mathbb{R}^{m\times m}$ can be estimated from above by

$\sigma_{\mathrm{max}}\left(\bm{1}-A_{M}\right)\leq\|\bm{1}-A_{M}\|_{\mathrm{F}},$

where we denote, for any matrix $A=(a_{ij})$, by $\|A\|_{\mathrm{F}}$ the
Frobenius norm,

$\|A\|_{\mathrm{F}}=\left(\sum_{i,j=1}^{m}|a_{ij}|^{2}\right)^{1/2}.$

Note that for all $i,j\in\\{1,\dots,m\\}$, we have that the $(i,j)$ entry of
$\bm{1}-A_{M}$ is given by

$\delta_{ij}-\frac{|D|}{M}\sum_{k=1}^{m}\phi_{i}(X_{k})\phi_{j}(X_{k}).$

With $Y_{k}=-|D|\phi_{i}(X_{k})\phi_{j}(X_{k})$, this can be written in the
form

$\frac{1}{M}\sum_{k=1}^{m}Y_{k}-\mathbb{E}[Y_{k}],$

where the $Y_{k}$ are iid random variables bounded by $|D|\omega_{m}^{2}$. It
follows from Hoeffding’s inequality that for any $\delta>0$, we have

$\mathrm{Prob}\left[\left|\frac{1}{M}\sum_{k=1}^{m}Y_{k}-\mathbb{E}[Y_{k}]\right|\geq\delta\right]\leq
2\exp\left(-\frac{2M^{2}\delta^{2}}{|D|^{2}\omega_{m}^{4}}\right).$

Equivalently, for any $i,j\in\\{1,\dots,m\\}$ we have:

$\mathrm{Prob}\left[|\left[\bm{1}-A_{M}\right]_{ij}|\geq\delta\right]\leq
2\exp\left(-\frac{2M^{2}\delta^{2}}{|D|^{2}\omega_{m}^{4}}\right).$

It now follows that

$\displaystyle\mathrm{Prob}\left[\|\bm{1}-A_{M}\|_{F}\geq\delta\right]$
$\displaystyle=\mathrm{Prob}\left[\sum_{i,j=1}^{m}|(\bm{1}-A_{M})_{ij}|^{2}\geq\delta^{2}\right]$
$\displaystyle\leq\mathrm{Prob}\left[\max_{i,j=1,\dots,m}|(\bm{1}-A_{M})_{ij}|\geq\frac{\delta}{m}\right]$
$\displaystyle\leq
m^{2}\max_{i,j=1,\dots,m}\mathrm{Prob}\left[|(\bm{1}-A_{M})_{ij}|\geq\frac{\delta}{m}\right]$
$\displaystyle\leq
2m^{2}\exp\left(-2\left(\frac{M\delta}{|D|\omega_{m}^{2}m}\right)^{2}\right)$

Choosing $\delta=1/\sqrt{2}$, we have

$\mathrm{Prob}\left[\sigma_{\mathrm{max}}\left(\bm{1}-A_{M}\right)\geq
1/\sqrt{2}\right]\leq
2m^{2}\exp\left(-\left(\frac{M}{|D|\omega_{m}^{2}m}\right)^{2}\right),$

and thus, from
$\sigma_{\mathrm{min}}(A_{M})=1-\sigma_{\mathrm{max}}(\bm{1}-A_{M})$, also

$\mathrm{Prob}\left[\sigma_{\mathrm{min}}\left(A_{M}\right)<1-1/\sqrt{2}\right]\leq
2m^{2}\exp\left(-\left(\frac{M}{|D|\omega_{m}^{2}m}\right)^{2}\right).$

∎

###  C.13. Proof of Theorem 3.9

###### Proof.

Let

$C=\frac{\sqrt{2}\,\max(|D|,1)}{\sqrt{2}-1}\sup_{\ell\in\mathbb{N}}(1+\|\phi_{\ell}\|_{L^{\infty}}^{2}).$

By assumption, we have $C<\infty$, and we also note that $C\geq 1$. Let
$(\Omega,\mathcal{P})$ be the probability space
$\Omega=\prod_{\ell=1}^{\infty}D$, with probability measure
$\mathcal{P}=\prod_{\ell=1}^{\infty}\mathrm{Unif}(D)$, such that the iid
random variables $X_{1},X_{2},\dots$ are given by projection onto the
corresponding factor

$X_{\ell}:\Omega\to D,\quad X_{\ell}(\omega)=\omega_{\ell},$

where $\omega=(\omega_{1},\omega_{2},\dots,)\in\Omega$.

By Lemma 3.24, if we choose $M(m)=\lceil C\kappa m\log(m)\rceil$, with
$\kappa=4$, then we have

$\displaystyle\mathrm{Prob}\left[{\left\\{\omega\,\middle|\,\widehat{\mathscr{E}}_{\mathcal{E}}(X_{1}(\omega),\dots,X_{M}(\omega))\leq
C\textstyle\sum_{\ell>m}\lambda_{\ell}\right\\}}\right]\geq 1-2m^{-2}.$ (C.18)

where
$\widehat{\mathscr{E}}_{\mathcal{E}}=\widehat{\mathscr{E}}_{\mathcal{E}}(X_{1},\dots,X_{M})$
is the random encoding error based on the sensors $X_{1},\dots,X_{M}$. We note
that asymptotically as $m\to\infty$, we have

$M\sim C\kappa m\log(m)\gtrsim C\kappa m,$

and hence $\log(M)\gtrsim\log(C\kappa)+\log(m)\geq\log(m)$, where we used that
$C>1$ and $\kappa=4$, so that $\log(C\kappa)\geq 0$ in the last estimate. In
particular, this implies that

$M/\log(M)\lesssim\left[C\kappa m\log(m)\right]/\log(m)=C\kappa m.$

It follows that, by possibly enlarging the constant $C>1$, we have

$\sum_{\ell>M/C\log(M)}\lambda_{\ell}\geq\sum_{\ell>m}\lambda_{\ell},$

and using also (C.18), we conclude that for sufficiently large $C$, we have

$\mathrm{Prob}\left[{\left\\{\omega\,\middle|\,\widehat{\mathscr{E}}_{\mathcal{E}}(X_{1},\dots,X_{M})\leq
C\textstyle\sum_{\ell>M/C\log(M)}\lambda_{\ell}\right\\}}\right]\geq
1-C\frac{\log(M)^{2}}{M^{2}},$

for all $M\in\mathbb{N}$. In particular, the probability that

$\widehat{\mathscr{E}}_{\mathcal{E}}(X_{1},\dots,X_{M})>C\sum_{\ell>M/C\log(M)}\lambda_{\ell},$

for infinitely many $M$, can be bounded by

$\displaystyle\mathrm{Prob}$
$\displaystyle\Big{[}\widehat{\mathscr{E}}_{\mathcal{E}}(X_{1},\dots,X_{M})>C\textstyle\sum_{\ell>M/C\log(M)}\lambda_{\ell}\text{
infinitely often}\Big{]}$
$\displaystyle\leq\limsup_{M_{0}\to\infty}\mathrm{Prob}\left[\bigcup_{M>M_{0}}{\left\\{\omega\,\middle|\,\widehat{\mathscr{E}}_{\mathcal{E}}(X_{1}(\omega),\dots,X_{M}(\omega))>C\sum_{\ell>M/C\log(M)}\lambda_{\ell}\right\\}}\right]$
$\displaystyle\leq\limsup_{M_{0}\to\infty}\sum_{M>M_{0}}\mathrm{Prob}\left[{\left\\{\omega\,\middle|\,\widehat{\mathscr{E}}_{\mathcal{E}}(X_{1}(\omega),\dots,X_{M}(\omega))>C\sum_{\ell>M/C\log(M)}\lambda_{\ell}\right\\}}\right]$
$\displaystyle\leq
C\limsup_{M_{0}\to\infty}\sum_{M>M_{0}}\frac{\log(M)^{2}}{M^{2}}$
$\displaystyle=0.$

Thus, for almost all $\omega\in\Omega$, we have

$\widehat{\mathscr{E}}_{\mathcal{E}}(X_{1}(\omega),\dots,X_{M}(\omega))\leq
C\sum_{\ell>M/C\log(M)}\lambda_{\ell},$

for all sufficiently large $M>0$. ∎

###  C.14. Proof of Lemma 3.25

###### Proof.

The starting point is the claim that if the covariance operator $\Gamma$ be
given by (3.47), the eigenfunctions and eigenvalues $(\phi_{k},\lambda_{k})$
of $\Gamma$ are given by

$\displaystyle\phi_{k}(x)=e^{-ikx},\quad\lambda_{k}=\sqrt{2\pi}\ell\,e^{-(\ell
k)^{2}/2},\quad(k\in\mathbb{Z}).$ (C.19)

To see this, we note that

$\displaystyle\int_{0}^{2\pi}k_{p}(x,x^{\prime})\phi_{k}(x^{\prime})\,dx^{\prime}$
$\displaystyle=\sum_{h\in
2\pi\mathbb{Z}}\int_{0}^{2\pi}e^{-(x-x^{\prime}-h)^{2}/(2\ell^{2})}e^{-ikx^{\prime}}\,dx^{\prime}$
$\displaystyle=\int_{-\infty}^{\infty}e^{-(x-x^{\prime})^{2}/(2\ell^{2})}e^{-ikx^{\prime}}\,dx^{\prime}$
$\displaystyle=\sqrt{2\pi}\ell\,e^{-(\ell k)^{2}}\,e^{-ikx},$

where we used that the Fourier transform on the real line $\mathbb{R}$

$\mathcal{F}[u](k)=\int_{-\infty}^{\infty}u(x^{\prime})e^{-ikx^{\prime}}\,dx^{\prime},$

satisfies

$\mathcal{F}[u({\,\cdot\,}-x)][k]=\mathcal{F}[u](k)\,e^{-ikx},$

and we recall that for $\alpha>0$, the Fourier transform of a Gaussian is

$\mathcal{F}[\exp(-\alpha
x^{2})](k)=\sqrt{\frac{\pi}{\alpha}}\exp(-k^{2}/4\alpha).$

For simplicity, assume $m=2K+1$ for $K\in\mathbb{N}$. Recall that the decoder
$\mathcal{D}$ for encoder $\mathcal{E}$ is the discrete Fourier transform
(3.48), it is straightforward to check that
$\mathcal{E}\circ\mathcal{D}=\mathrm{Id}$, so $(\mathcal{E},\mathcal{D})$ is
an admissible encoder/decoder pair, and by the definition (3.1) of
$\widehat{\mathscr{E}}_{\mathcal{E}}$:

$(\widehat{\mathscr{E}}_{\mathcal{E}})^{2}\leq\int_{X}\left\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\right\|_{L^{2}_{x}}^{2}\,d\mu(u).$

Let $P_{K}:L^{2}_{x}\to L^{2}_{x}$ denote the orthogonal projection onto
$\mathrm{span}(e^{ikx};\,|k|\leq K)$, and denote by
$P_{K}^{\perp}:L^{2}_{x}\to L^{2}_{x}$ the orthogonal projection onto the
orthogonal complement, so that $\mathrm{Id}=P_{K}+P_{K}^{\perp}$. We note that
$\mathcal{D}\circ\mathcal{E}$ is _linear_ and we have
$(\mathcal{D}\circ\mathcal{E})(e^{ikx})=e^{ikx}$ for all $|k|\leq K$ (where
$m=2K+1$). From this, it follows that

$\displaystyle\mathcal{D}\circ\mathcal{E}-\mathrm{Id}$
$\displaystyle=\left[\mathcal{D}\circ\mathcal{E}\circ
P_{K}-P_{K}\right]+\left[\mathcal{D}\circ\mathcal{E}\circ
P_{K}^{\perp}-P_{K}^{\perp}\right]$
$\displaystyle=\mathcal{D}\circ\mathcal{E}\circ P_{K}^{\perp}-P_{K}^{\perp}$
$\displaystyle=P_{K}\circ\mathcal{D}\circ\mathcal{E}\circ
P_{K}^{\perp}-P_{K}^{\perp},$

where we used that $\mathcal{D}=P_{K}\circ\mathcal{D}$. The two terms on the
last line are obviously perpendicular to each other. Hence

$\displaystyle\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\|_{L^{2}_{x}}^{2}=\|\mathcal{D}\circ\mathcal{E}\circ
P_{K}^{\perp}\|_{L^{2}_{x}}^{2}+\|P_{K}^{\perp}\|_{L^{2}_{x}}^{2}.$ (C.20)

We note that the non-vanishing of the first term in C.20 is due to _aliasing_
, i.e. the fact that for any point on the grid $x_{j}$, $j=1,\dots,m$, we have

$e^{ikx_{j}}=e^{i(k+qm)x_{j}},\quad\text{for all }q\in\mathbb{Z}.$

Therefore, the higher-order modes $e^{ikx}$ for $|k|>K$ map under
$\mathcal{E}$ to

$\mathcal{E}(e^{ikx})=\mathcal{E}(e^{ik_{0}x}),$

where $k_{0}\in\\{-K,\dots,K\\}$ is the unique value such that there exists
$q\in\mathbb{Z}$ with $k_{0}=k+qm$.

As the functions $x\mapsto e^{ikx}$, $k\in\mathbb{Z}$, are the eigenfunctions
of the covariance operator of $\mu$. Let $\lambda_{k}$, $k\in\mathbb{Z}$,
denote the corresponding eigenvalues. By the Karhunen-Loeve expansion for the
Gaussian measure $\mu$, we can now write

$\displaystyle\int_{X}\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\|_{L^{2}_{x}}^{2}\,d\mu$
$\displaystyle=\mathbb{E}\left[\|\mathcal{D}\circ\mathcal{E}(X)-X\|_{L^{2}_{x}}^{2}\right]$
$\displaystyle=\mathbb{E}\left[\|\mathcal{D}\circ\mathcal{E}\left(P_{K}^{\perp}X\right)\|_{L^{2}_{x}}^{2}\right]+\mathbb{E}\left[\|P_{K}^{\perp}(X)\|_{L^{2}_{x}}^{2}\right],$

where

$X=\sum_{k\in\mathbb{Z}}\sqrt{\lambda_{k}}X_{k}e^{ikx},$

and the $X_{k}\sim\mathcal{N}(0,1)$ are iid Gaussian random variables with
unit variance. Due to aliasing, we have

$\mathcal{D}\circ\mathcal{E}\left(P_{K}^{\perp}X\right)=\sum_{k_{0}=-K}^{K}\left(\sum_{q\in\mathbb{Z}\setminus\\{0\\}}\sqrt{\lambda_{k_{0}+qm}}X_{k_{0}+qm}\right)e^{ik_{0}x}.$

And

$\displaystyle\mathbb{E}\left[\|\mathcal{D}\circ\mathcal{E}\left(P_{K}^{\perp}X\right)\|_{L^{2}_{x}}^{2}\right]$
$\displaystyle=\mathbb{E}\left[\sum_{k_{0}=-K}^{K}\left(\sum_{q\in\mathbb{Z}\setminus\\{0\\}}\sqrt{\lambda_{k_{0}+qm}}X_{k_{0}+qm}\right)^{2}\right]$
$\displaystyle=\sum_{k_{0}=-K}^{K}\mathbb{E}\left[\sum_{q,q^{\prime}\in\mathbb{Z}\setminus\\{0\\}}\sqrt{\lambda_{k_{0}+qm}}\sqrt{\lambda_{k_{0}+q^{\prime}m}}X_{k_{0}+qm}X_{k_{0}+q^{\prime}m}\right]$
$\displaystyle=\sum_{k_{0}=-K}^{K}\sum_{q,q^{\prime}\in\mathbb{Z}\setminus\\{0\\}}\sqrt{\lambda_{k_{0}+qm}}\sqrt{\lambda_{k_{0}+q^{\prime}m}}\mathbb{E}\left[X_{k_{0}+qm}X_{k_{0}+q^{\prime}m}\right].$

As the $X_{k}$ are iid with zero mean and unit variance, we have

$\mathbb{E}[X_{k}X_{k^{\prime}}]=\delta_{k,k^{\prime}}=\begin{cases}1,&k=k^{\prime}\\\
0,&k\neq k^{\prime}\end{cases}.$

Thus,

$\displaystyle\mathbb{E}\left[\|\mathcal{D}\circ\mathcal{E}\left(P_{K}^{\perp}X\right)\|_{L^{2}_{x}}^{2}\right]$
$\displaystyle=\sum_{k_{0}=-K}^{K}\sum_{q,q^{\prime}\in\mathbb{Z}\setminus\\{0\\}}\sqrt{\lambda_{k_{0}+qm}}\sqrt{\lambda_{k_{0}+q^{\prime}m}}\delta_{q,q^{\prime}}$
$\displaystyle=\sum_{k_{0}=-K}^{K}\sum_{q\in\mathbb{Z}\setminus\\{0\\}}\lambda_{k_{0}+qm}$
$\displaystyle=\sum_{|k|>K}\lambda_{k}.$

On the other hand, it is easy to see that

$\mathbb{E}\left[\|P_{K}^{\perp}X\|_{L^{2}_{x}}^{2}\right]=\sum_{|k|>K}\lambda_{k}.$

We thus conclude that

$\displaystyle\int_{X}\|\mathcal{D}\circ\mathcal{E}-\mathrm{Id}\|_{L^{2}_{x}}^{2}\,d\mu$
$\displaystyle=\mathbb{E}\left[\|\mathcal{D}\circ\mathcal{E}\left(P_{K}^{\perp}X\right)\|_{L^{2}_{x}}^{2}\right]+\mathbb{E}\left[\|P_{K}^{\perp}(X)\|_{L^{2}_{x}}^{2}\right]$
$\displaystyle=2\sum_{|k|>K}\lambda_{k}.$

Recalling that by (C.19), we have

$\lambda_{k}=\sqrt{2\pi}\ell\,e^{-(\ell k)^{2}/2},$

we finally obtain

$\displaystyle 2\sum_{|k|>K}\lambda_{k}$
$\displaystyle=2\sqrt{2\pi}\sum_{|k|>K}\ell\,e^{-(\ell k)^{2}/2}$
$\displaystyle\leq 2\pi\sqrt{2}\frac{2}{\sqrt{\pi}}\int_{K}^{\infty}e^{-(\ell
x)^{2}/2}\,\ell\,dx$
$\displaystyle=4\pi\left[\frac{2}{\sqrt{\pi}}\int_{K}^{\infty}e^{-(\ell
x/\sqrt{2})^{2}}\,d\left(\ell x/\sqrt{2}\right)\right]$
$\displaystyle=4\pi\,\mathrm{erfc}\left(\frac{\ell K}{\sqrt{2}}\right).$

∎

###  C.15. Proof of Lemma 3.26

###### Proof.

By Lemma 3.25, the eigenfunctions of the covariance operator are
$\phi_{k}=\bm{\mathrm{e}}_{k}$, the standard Fourier basis. By Theorem 3.9,
there exists a constant $C\geq 1$, depending on $|D|$ and
$\sup_{k\in\mathbb{N}}\|\phi_{k}\|_{L^{\infty}}\leq 2(2\pi)^{-d}$, such that
with probability $1$ in the random sensors $X_{1},X_{2},X_{3},\dots\in D$, we
have for almost all $M\in\mathbb{N}$:

$\widehat{\mathscr{E}}_{\mathcal{E}}\leq
C\sqrt{\sum_{k>M/C\log(M)}\lambda_{k}}.$

By Lemma 3.25 (cp. (3.49), (3.50)), we have

$\sqrt{\sum_{k>M/C\log(M)}\lambda_{k}}\lesssim\exp\left(-\widetilde{\gamma}\frac{M^{2}\ell^{2}}{C^{2}\log(M)^{2}}\right),\quad\forall\widetilde{\gamma}<\frac{1}{16}.$

The implied constant here only depends on the value of $\widetilde{\gamma}$
and on $|D|$. Thus, choosing e.g. $\widetilde{\gamma}=1/20$ and noting that
$|D|=2\pi$ is a fixed constant, we conclude that

$\widehat{\mathscr{E}}_{\mathcal{E}}\leq
C\exp\left(-\gamma\frac{M^{2}\ell^{2}}{\log(M)^{2}}\right),$

for $\gamma=1/(20C^{2})$, where the constant $C>0$ is independent of $\ell$
and $M$. ∎

###  C.16. Proof of Proposition 3.27

###### Proof.

Fix $u=u({\,\cdot\,};Y)$ for some $Y\in[-1,1]^{\mathcal{J}}$. Let
$\widehat{Y}=\widehat{Y}(\mathcal{E}(u))$ be given by (3.61). Let
$\widetilde{Y}$ be given by (3.59), such that

$\widehat{Y}_{j}=\mathrm{shrink}(\widetilde{Y}_{j}),\quad\forall\,j\in\mathcal{J}=\mathbb{Z}^{d}.$

We note that since $Y_{j}\in[-1,1]$ for all $j\in\mathcal{J}$, we have

$\displaystyle\left|Y_{j}-\widehat{Y}_{j}\right|=\left|Y_{j}-\mathrm{shrink}(\widetilde{Y}_{j})\right|\leq\left|Y_{j}-\widetilde{Y}_{j}\right|,\quad\forall\,j\in\mathcal{J}=\mathbb{Z}^{d}.$

But, by definition of $\widetilde{Y}_{j}$, we have that

$u\mapsto
u({\,\cdot\,};\widetilde{Y}(\mathcal{E}(u)))=\sum_{j\in\mathbb{Z}^{d}}\widetilde{Y}_{j}(\mathcal{E}(u))\alpha_{j}\bm{\mathrm{e}}_{j}=\sum_{j\in\mathbb{Z}^{d}}\widehat{u}_{j}\bm{\mathrm{e}}_{j},$

is the pseudo-spectral Fourier projection of $u$ onto
$\\{\bm{\mathrm{e}}_{j}\\}_{j\in\mathcal{K}_{N}}$. In particular, it follows
from standard estimates for the pseudo-spectral projection that for $u\in
H^{s}(\mathbb{T}^{d})$, with $s>n/2$, we have

$\Big{\|}u({\,\cdot\,};Y)-u({\,\cdot\,};\widetilde{Y}(\mathcal{E}(u))\Big{\|}_{L^{2}}\leq
C\|u\|_{H^{s}}\,N^{-s},$

for some $C=C(s)>0$, and hence also

$\displaystyle\left\|u({\,\cdot\,};Y)-u({\,\cdot\,};\widehat{Y})\right\|_{L^{2}}^{2}$
$\displaystyle=\sum_{j\in\mathbb{Z}^{d}}\left|Y_{j}-\widehat{Y}_{j}\right|^{2}\alpha_{j}^{2}$
$\displaystyle\leq\sum_{j\in\mathbb{Z}^{d}}\left|Y_{j}-\widetilde{Y}_{j}\right|^{2}\alpha_{j}^{2}$
$\displaystyle=\Big{\|}u({\,\cdot\,};Y)-u({\,\cdot\,};\widetilde{Y}\Big{\|}_{L^{2}}^{2}$
$\displaystyle\leq C\|u\|_{H^{s}}\,N^{-s}.$

∎

###  C.17. Proof of Theorem 3.28

###### Proof.

We note that since $m=(2N+1)^{d}$, we have $N\gtrsim m^{1/d}$. Thus, it
suffices to show that $\widehat{\mathscr{E}}_{\mathcal{E}}\leq C\exp(-cN)$ for
some constants $c,C$ independent of $N$. We now note that if
$Y=(Y_{j})_{j\in\mathcal{J}}$ are distributed according to the probability
measure $\rho\in\mathcal{P}([-1,1]^{\mathcal{J}})$, and if
$\widehat{Y}=\widehat{Y}(\mathcal{E}(u)$ and
$\widetilde{Y}=\widetilde{Y}(\mathcal{E}(u))$ are given by (3.61), (3.59),
respectively, then

$\displaystyle(\widehat{\mathscr{E}}_{\mathcal{E}})^{2}$
$\displaystyle=\int_{L^{2}(D)}\|\mathcal{D}\circ\mathcal{E}(u)-u\|_{L^{2}}^{2}\,d\mu(u)$
$\displaystyle=\int_{L^{2}(D)}\|u({\,\cdot\,};\widehat{Y})-u({\,\cdot\,};Y)\|_{L^{2}}^{2}\,d\rho(Y)$

As in the proof of Proposition 3.27 in appendix C.16, we see that – due to the
exponential decay $\lesssim\exp(-|k|\ell)$ of the Fourier coefficients of
$u({\,\cdot\,};Y)$ – there exist $C,c>0$, independent of $N$ and
$Y\in[-1,1]^{\mathcal{J}}$, such that the last term can be estimated by

$\displaystyle\leq\int_{L^{2}(D)}\left[C\exp(-c\ell N)\right]^{2}\,d\rho(Y)$
$\displaystyle=\left[C\exp(-c\ell N)\right]^{2}.$

This shows that $\widehat{\mathscr{E}}_{\mathcal{E}}\leq C\exp(-c\ell N)$. And
we recall that $N\lesssim m^{1/d}$, by definition of $m=(2N+1)^{d}$.

Finally, if

$\mathcal{F}(\bm{y})=\mathcal{G}(u({\,\cdot\,};\bm{y})),$

then for any $\bm{u}=(u_{i})_{i\in\mathcal{I}_{N}}$, we have by the definition
of $\mathcal{D}(\bm{u})=u({\,\cdot\,};\widehat{Y}(\bm{u}))$ (cp. (3.60))

$\mathcal{G}\circ\mathcal{D}(\bm{u})=\mathcal{G}\left(u({\,\cdot\,},\widehat{Y}(\bm{u}))\right)=\mathcal{F}(\widehat{Y}(\bm{u})).$

∎

###  C.18. Proof of Corollary 3.34

###### Proof.

Let $\widehat{\mathcal{N}}$ be a neural network satisfying the estimates of
Proposition 3.33, with $N:=m$. Then

$\mathcal{N}:=\mathcal{P}\circ\widehat{\mathcal{N}}=\sum_{\bm{\nu}\in\Lambda_{N}}\underbrace{(\mathcal{P}c_{\bm{\nu}})}_{\in\mathbb{R}^{p}}\widehat{\mathcal{N}}_{\bm{\nu}}(y_{\kappa(1)},\dots,y_{\kappa(N)})$

is a linear combination of the neural network output of
$\widehat{\mathcal{N}}$, with coefficients in $\mathbb{R}^{p}$. In particular,
by adding a linear output layer of size $\mathcal{O}(pN)$ to the network
$\widehat{\mathcal{N}}$, we can represent $\mathcal{N}$ as a neural network
with

$\mathrm{size}(\mathcal{N})\leq\mathrm{size}(\widehat{\mathcal{N}})+pN,\quad
0pt(\mathcal{N})\leq 0pt(\widehat{\mathcal{N}})+1.$

The claimed bounds on the size of $\mathcal{N}$ thus readily follow from the
corresponding bounds for $\widehat{\mathcal{N}}$. Furthermore, we have for any
$\bm{y}\in[-1,1]^{\mathcal{J}}$:

$\displaystyle\left\|\mathcal{P}\circ\mathcal{F}(\bm{y})-\mathcal{N}(y_{\kappa(1)},\dots,y_{\kappa(N)}\right\|_{\ell^{2}}$
$\displaystyle=\left\|\mathcal{P}\circ\mathcal{F}(\bm{y})-\mathcal{P}\circ\widehat{\mathcal{N}}(y_{\kappa(1)},\dots,y_{\kappa(N)}\right\|_{\ell^{2}}$
$\displaystyle\leq\|\mathcal{P}\|\left\|\mathcal{F}(\bm{y})-\widehat{\mathcal{N}}(y_{\kappa(1)},\dots,y_{\kappa(N)}\right\|_{V}.$

The claimed error estimate thus follows from the error estimate in Proposition
3.33. ∎

###  C.19. Proof of Theorem 3.35

###### Proof.

The proof of this theorem requires the following simple Lemma on Neural
Network Calculus,

######  Lemma C.10.

Let $\mathcal{N}_{1}:\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{1}}$,
$\mathcal{N}_{2}:\mathbb{R}^{n_{1}}\to\mathbb{R}^{n_{2}}$ be two neural
networks. Then the composition
$\mathcal{N}=\mathcal{N}_{2}\circ\mathcal{N}_{1}:\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{2}}$
can be represented by a neural network $\mathcal{N}$ with

$\mathrm{size}(\mathcal{N})=\mathrm{size}(\mathcal{N}_{1})+\mathrm{size}(\mathcal{N}_{2}),\quad
0pt(\mathcal{N})=0pt(\mathcal{N}_{1})+0pt(\mathcal{N}_{2}).$

####  C.19.1. Proof of the Theorem 3.35

From the network size bounds of Corollary 3.34, $\mathcal{N}$ is a neural
network of size

$\begin{gathered}\mathrm{size}(\mathcal{N})\leq C(1+mp\log(m)\log\log(m))),\\\
0pt(\mathcal{N})\leq C(1+\log(m)\log\log(m))).\end{gathered}$

with a constant $C>0$, independent of $m$, $p$. By Lemma 3.29, the map
$\bm{u}\mapsto\widehat{Y}_{\kappa}(\bm{u})$ can be represented by a neural
network of size

$\mathrm{size}(\widehat{Y}_{\kappa})\leq C(1+m\log(m)),\quad
0pt(\widehat{Y}_{\kappa})\leq C(1+\log(m)),$

for a constant $C>0$, independent of $m$. Since the definition of
$\widehat{Y}_{\kappa}$ does not involve the projection $\mathcal{P}$ at all,
the constant is also independent of $p$. By the composition Lemma C.10 it
follows that
$\bm{u}\mapsto\mathcal{A}(\bm{u})=(\mathcal{N}\circ\widehat{Y}_{\kappa})(\bm{u})$
can be represented by a neural network with

$\mathrm{size}(\mathcal{A})=\mathrm{size}(\mathcal{N})+\mathrm{size}(\widehat{Y}_{\kappa}),\,0pt(\mathcal{A})=0pt(\mathcal{N})+0pt(\widehat{Y}_{\kappa}).$

We also note that the following estimate for the approximation error
$\widehat{\mathscr{E}}_{\mathcal{A}}$:

$\displaystyle(\widehat{\mathscr{E}}_{\mathcal{A}})^{2}$
$\displaystyle=\int_{L^{2}(D)}\|\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}(\bm{u})-\mathcal{A}(\bm{u})\|^{2}_{\ell^{2}}\,d(\mathcal{E}_{\\#}\mu)(\bm{u})$
$\displaystyle\leq\sup_{\bm{u}\in\mathrm{supp}(\mu)}\|\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}(\bm{u})-\mathcal{A}(\bm{u})\|^{2}_{\ell^{2}}$
$\displaystyle=\sup_{\bm{u}\in\mathrm{supp}(\mu)}\|\mathcal{P}\circ\mathcal{F}(\widehat{Y}(\bm{u}))-\mathcal{N}(\widehat{Y}(\bm{u}))\|^{2}_{\ell^{2}}$

By Corollary 3.34 we can further estimate the last term by

$\displaystyle\leq\sup_{\bm{y}\in[-1,1]^{\mathcal{J}}}\|\mathcal{P}\circ\mathcal{F}(\bm{y})-\mathcal{N}(y_{\kappa(1)},\dots,y_{\kappa(m)})\|_{\ell^{2}}^{2}$
$\displaystyle\leq C\|\mathcal{P}\|\,m^{-2s},\quad\text{where
}s:=\frac{1}{q}-1>0,$

provided that $\alpha_{k}\in\ell^{q}(\mathbb{Z}^{d})$ with $q\in(0,1)$. But by
the exponential decay assumption (3.54), we have
$\alpha_{k}\in\ell^{q}(\mathbb{Z}^{d})$ for any $q\in(0,1)$. The claim
follows. ∎

##  Appendix D DeepONet approximation of linear operators

In this section, we will illustrate the ability of a DeepONet to approximate
bounded linear operators efficienty. A simple numerical example of the
DeepONet approximation of a linear functional
$\mathcal{G}:C([0,1]^{2})\to\mathbb{R}$ is presented in Figure 2, below.

(a)

(b)

(c)

(d)

Figure 2. Illustration of a DeepONet ((1.1)-(1.3), with $p=1$ and
$\sigma(x)=x$) approximating the operator
$\mathcal{G}:C([0,1]^{2})\to\mathbb{R}$, defined as
$\mathcal{G}(u):=\int_{\Delta}u(x)\,dx$, with integration domain $\Delta$
(shown in red in (A)), with underlying measure being the law of a Gaussian
random field with covariance kernel $k(x,y)=\exp(-|x-y|^{2}/2\ell^{2})$, with
$\ell=0.1$. $m$ sensor points $x_{1},\dots,x_{m}$ are drawn at random from the
uniform distribution on $[0,1]^{2}$ and the DeepONet is trained by minimizing
the mean square loss function, with respect to $N_{u}$ samples, drawn from the
underlying measure. (A) Illustration of a typical sample drawn from the
Gaussian random field. (B) Convergence of the test mean squared error
(computed with respect to $10^{\prime}240$ test samples) as $m\to\infty$, for
different numbers of training samples $N_{u}$. We observe a clear exponential
decay of error wrt $m$ as well as “resonances” at $m=N_{u}$ (double descent).
(C) Convergence of the test MSE as $N_{u}\to\infty$, for different numbers of
random sensor points $x_{1},\dots,x_{m}$. (D) semilog plot of the MSE loss as
$N_{u}\to\infty$ where one observes that the test error decays as
$\exp(-\sqrt{N_{u}})$ when $N_{u}\leq m$. Note that similar behavior of the
error is seen for more complicated operators in Lu et al. (2019).

Of particular interest is the observed exponential decay in the DeepONet
approximation error as the number of sensors $m\to\infty$, which has also been
observed for other (linear and even non-linear) problems in Lu et al. (2019).
Can the theoretical framework developed in the present work explain such
behaviour? To answer this question, a general error estimate for the DeepONet
approximation of linear operators is derived in Theorem D.2. The estimate is
then applied to a prototypical elliptic PDE (cf. example D.4). In the
following, we consider the following setup:

######  Setup D.1.

We consider data for the DeepONet approximation problem $\mu$, $\mathcal{G}$
(cp. Definition 2.1), where

  * •

$\mathcal{G}:L^{2}(D)\to L^{2}(U)$ is a bounded linear mapping,

  * •

$\mu\in\mathcal{P}_{2}(L^{2}(D))$ is a probability measure with mean $0$, and
with uniformly bounded eigenfunctions of the covariance operator
$\Gamma_{\mu}$,

  * •

for $m\in\mathbb{N}$, the sensors $x_{1},\dots,x_{m}\sim\mathrm{Unif}(D)$ are
drawn iid random,

  * •

for $p\in\mathbb{N}$, we denote by $\widehat{\tau}_{k}$, $k=0,\dots,p$, the
optimal choice for an affine reconstruction
$\mathcal{R}_{\widehat{\bm{\tau}}}=\mathcal{R}_{\mathrm{opt}}$ for the push-
forward measure $\mathcal{G}_{\\#}\mu$, as in Theorem 3.14; i.e., in the
present case, $\widehat{\tau}_{0}\equiv 0$, and $\widehat{\tau}_{k}$,
$k=1,\dots,p$, are the first $p$ eigenfunctions of the covariance operator
$\Gamma_{\mathcal{G}_{\\#}\mu}$.

Under the above assumptions, we then have

######  Theorem D.2.

Consider the setup D.1. Let $m,p\in\mathbb{N}$ denote the number of sensors
and the output dimension of the branch/trunk nets $(\bm{\beta},\bm{\tau})$,
respectively. Let $\bm{\tau}$ be a trunk-net approximation of
$\widehat{\bm{\tau}}$, such that the associated reconstruction
$\mathcal{R}=\mathcal{R}_{\bm{\tau}}$ and projection $\mathcal{P}$ satisfy
$\mathrm{Lip}(\mathcal{R}),\,\mathrm{Lip}(\mathcal{R}\circ\mathcal{P})\leq 2$.
For $m\in\mathbb{N}$, we define the (random) encoder
$\mathcal{E}:\,u\mapsto(u(x_{1}),\dots,u(x_{m}))$. Then, with probability $1$
in the choice of the random sensor points, there exists a constant $C>0$,
depending only on $\mu$ and the measure of the domain $|D|$, such that for any
$m,p\in\mathbb{N}$ there exists a shallow ReLU approximator net
$\mathcal{A}:\mathbb{R}^{m}\to\mathbb{R}^{p}$ with

$\mathrm{size}({\mathcal{A}})\leq 2(2+m)p,\quad 0pt(\mathcal{A})\leq 1,$

and such that the DeepONet
$\mathcal{N}=\mathcal{R}_{\bm{\tau}}\circ\mathcal{A}\circ\mathcal{E}$, with
branch net $\bm{\beta}=\mathcal{A}\circ\mathcal{E}$ and trunk net $\bm{\tau}$
satisfies the following asymptotic DeepONet approximation error estimate

$\displaystyle\widehat{\mathscr{E}}\leq
C\sqrt{1+\|\mathcal{G}\|^{2}}\left\\{\max_{k=0,\dots,p}\|\tau_{k}-\widehat{\tau}_{k}\|_{L^{2}(U)}+\sqrt{\sum_{\ell>p}\lambda_{\ell}}+\sqrt{\sum_{\ell>\frac{m}{C\log(m)}}\lambda_{\ell}}\right\\},$
(D.1)

for almost all $m$, as $m\to\infty$.

###### Proof.

By the DeepONet error decomposition of Theorem 3.3 and the assumed Lipschitz
bounds
$\mathrm{Lip}(\mathcal{R}),\,\mathrm{Lip}(\mathcal{R}\circ\mathcal{P})\leq 2$,
we have

$\widehat{\mathscr{E}}\leq
2\|\mathcal{G}\|\widehat{\mathscr{E}}_{\mathcal{E}}+2\widehat{\mathscr{E}}_{\mathcal{A}}+\widehat{\mathscr{E}}_{\mathcal{R}}.$

We first observe that for any choice of the (affine) encoder/decoder and
reconstruction/projection pairs $(\mathcal{E},\mathcal{D})$,
$(\mathcal{R},\mathcal{P})$, and for a linear mapping $\mathcal{G}$ there
exists an _exact, affine approximator_
$\mathcal{A}:\mathbb{R}^{m}\to\mathbb{R}^{p}$, such that
$\mathcal{A}(\bm{u})=\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}(\bm{u})$ for
all $\bm{u}\in\mathbb{R}^{m}$. Furthermore, $\mathcal{A}$ can be represented
by a shallow ReLU neural net of the claimed size, on account of the fact that
$Ax+b=\sigma(Ax+b)-\sigma(-(Ax+b))$ has an exact representation for the ReLU
activation function $\sigma(x)=\max(x,0)$. Thus, the approximation error
$\widehat{\mathscr{E}}_{\mathcal{A}}$ can be made to vanish in this case,
$\widehat{\mathscr{E}}_{\mathcal{A}}=0$.

Under the assumptions of this theorem, the random encoding error has been
estimated in Theorem 3.9, where it is shown that there exists a constant
$C\geq 1$, depending only on the uniform upper bound of the eigenfunctions of
$\Gamma_{\mu}$, and on $|D|$, such that (for almost all $m\in\mathbb{N}$):

$\displaystyle\widehat{\mathscr{E}}_{\mathcal{E}}\leq
C\sqrt{\sum_{\ell>m/C\log(m)}\lambda_{\ell}},$ (D.2)

as $m\to\infty$, with probability $1$ in the iid random sensors
$x_{1},x_{2},\dots\sim\mathrm{Unif}(D)$. Here, $C=C(|D|,\mu)$ is a constant.

Finally, we estimate the reconstruction error
$\widehat{\mathscr{E}}_{\mathcal{R}}$: By Proposition 3.15, we have

$\widehat{\mathscr{E}}_{\mathcal{R}}\leq\sqrt{1+{\mathrm{Tr}}(\Gamma_{\mathcal{G}_{\\#}\mu})}\max_{k=0,\dots,p}\|\widehat{\tau}_{k}-\tau_{k}\|_{L^{2}(U)}+\sqrt{\sum_{k>p}\lambda_{k}^{\mathcal{G}_{\\#}\mu}},$

where
$\lambda^{\mathcal{G}_{\\#}\mu}_{1}\geq\lambda^{\mathcal{G}_{\\#}\mu}_{2}\geq\dots$
denote the eigenvalues of the covariance operator
$\Gamma_{\mathcal{G}_{\\#}\mu}$ of the push-forward measure
$\mathcal{G}_{\\#}\mu$. By Proposition 3.20, we can estimate

$\sum_{k>p}\lambda_{k}^{\mathcal{G}_{\\#}\mu}\leq\|\mathcal{G}\|^{2}\sum_{k>p}\lambda_{k},$

in terms of the eigenvalues $\lambda_{1}\geq\lambda_{2}\geq\dots$, of the
covariance operator $\Gamma_{\mu}$ of $\mu$, and in particular
${\mathrm{Tr}}(\Gamma_{\mathcal{G}_{\\#}\mu})\leq\|\mathcal{G}\|^{2}{\mathrm{Tr}}(\Gamma_{\mu})\leq
C\|\mathcal{G}\|^{2}$, where $C=C(\mu)$ depends only on $\mu$. It thus,
follows that there exists a constant $C=C(\mu)$, such that

$\displaystyle\widehat{\mathscr{E}}_{\mathcal{R}}\leq
C\sqrt{1+\|\mathcal{G}\|^{2}}\left\\{\max_{k=0,\dots,p}\|\widehat{\tau}_{k}-\tau_{k}\|_{L^{2}(U)}+\sqrt{\sum_{\ell>p}\lambda_{\ell}}\right\\}.$
(D.3)

Combining (D.2) and (D.3) with the error decomposition, and taking into
account that $\widehat{\mathscr{E}}_{\mathcal{A}}=0$, yields the desired
result. ∎

######  Remark D.3.

From the proof of Theorem D.2, it is clear that to achieve the error bound
(D.1), we can replace the shallow ReLU approximator net $\mathcal{A}$, by a
single-layer affine approximator
$\mathcal{A}:\mathbb{R}^{m}\to\mathbb{R}^{p}$,
$\mathcal{A}(\bm{u}):=A\cdot\bm{u}+b$, where $A\in\mathbb{R}^{p\times m}$,
$b\in\mathbb{R}^{m}$. This clearly corresponds to the use of the linear
activation function $\sigma(x)=x$ in the branch net. In contrast, the
activation function in the trunk net must be kept non-linear to be able to
approximate the optimal trunk net $\widehat{\bm{\tau}}$.

######  Example D.4.

To illustrate Theorem D.2, we consider the following example of a linear
operator $\mathcal{G}:L^{2}(\mathbb{T}^{d})\to L^{2}(\mathbb{T}^{d})$,
$f\mapsto v$, mapping the source term $f$ to the solution of the PDE $\Delta
v=f-\fint_{\mathbb{T}^{d}}f\,dx$, with periodic boundary conditions and
imposing $\fint_{\mathbb{T}^{d}}v\,dx=0$. It is well-known that this operator
is bounded; in fact, we have $\|\mathcal{G}\|\leq 1$. We fix the initial
measure $\mu\in\mathcal{P}(L^{2}(\mathbb{T}^{d}))$ as a Gaussian random field,
with Karhunen-Loeve expansion

$f=\sum_{k\in\mathbb{Z}^{d}}\alpha_{k}X_{k}\bm{\mathrm{e}}_{k},$

where $|\alpha_{k}|\leq\exp(-\ell|k|)$ have exponential decay with typical
length scale $\ell>0$, $\\{\bm{\mathrm{e}}_{k}\\}_{k\in\mathbb{Z}^{d}}$ is the
standard Fourier basis on $\mathbb{T}^{d}$, and the
$X_{k}\sim\mathcal{N}(0,1)$ are iid Gaussian random variables. In this case,
the eigenfunctions of the associated covariance operator $\Gamma_{\mu}$ of
$\mu$ are given by the $\bm{\mathrm{e}}_{k}$, with corresponding eigenvalues
$\alpha_{k}$, $k\in\mathbb{Z}^{d}$. In particular, it follows that there
exists constants $C,c>0$, depending only on $d$ and $\ell$, such that the last
two terms in (D.1) can be bounded from above by

$\leq
C\exp\left(-c\,p^{1/d}\right)+C\exp\left(-\frac{c\,m^{1/d}}{\log(m)^{1/d}}\right).$

In particular, Theorem D.2 implies that for any fixed $\sigma>0$, and with
$p\sim\log(\epsilon^{-1})^{d}$, $m\sim\log(\epsilon^{-1})^{d(1+\sigma)}$, we
can achieve an overall DeepONet approximation error

$\widehat{\mathscr{E}}\lesssim\epsilon+\max_{j=0,\dots,p}\|\bm{\mathrm{e}}_{j}-\tau_{j}\|_{L^{2}(U)},$

where we have taken into account that the Fourier basis $\bm{\mathrm{e}}_{j}$
is an eigenbasis also for the push-forward measure $\mathcal{G}_{\\#}\mu$.
Furthermore, it follows from Lemma 3.17, that for any fixed $\sigma>0$, there
exists a trunk net $\bm{\tau}$ with asymptotic size (as $\epsilon\to 0$)

$\displaystyle\mathrm{size}(\bm{\tau})\lesssim\log(\epsilon^{-1})^{d+2+\sigma},\quad
0pt(\bm{\tau})\lesssim\log(\epsilon^{-1})^{2+\sigma},$ (D.4)

and such that
$\max_{k=0,\dots,p}\|\bm{\mathrm{e}}_{j}-\tau_{j}\|_{L^{2}(U)}<\epsilon$.
Thus, for the present example, an overall DeepONet error
$\widehat{\mathscr{E}}\lesssim\epsilon$ can be achieved with a DeepONet
$(\bm{\beta},\bm{\tau})$ with trunk net $\bm{\tau}$ satisfying the size bounds
(D.4), and a branch net $\bm{\beta}$ of size

$\displaystyle\mathrm{size}(\bm{\beta})\lesssim\log(\epsilon^{-1})^{2d+\sigma},\quad
0pt(\bm{\beta})\lesssim 1,$ (D.5)

for any fixed $\sigma>0$. The implied constants here depend on the length
scale $\ell>0$, the dimension $d$ and the additional parameter $\sigma>0$,
which was introduced to avoid the appearance of multiple logarithms.

##  Appendix E Proofs of Results in Section 4

###  E.1. Proof of Lemma 4.1

###### Proof.

Let $v,v^{\prime}$ solve (4.2) with forcing $u,u^{\prime}$, respectively. Then
we can write

$\displaystyle\frac{d(v-v^{\prime})}{dt}=G(v,v^{\prime})(v-v^{\prime})+(u-u^{\prime}),$
(E.1)

where

$G(v,v^{\prime})=\int_{0}^{1}g^{(1)}(sv+(1-s)v^{\prime})\,ds,$

so that $|G(v,v^{\prime})|\leq\|g^{(1)}\|_{L^{\infty}}$ for all
$v,v^{\prime}$. It follows readily from (E.1) that

$\frac{d}{dt}|v-v^{\prime}|^{2}\leq C|v-v^{\prime}|^{2}+|u-u^{\prime}|^{2},$

for some $C>0$ depending only on $\|g^{(1)}\|_{L^{\infty}}$. Gronwall’s
inequality then implies that

$|v-v^{\prime}|^{2}(t)\leq\int_{0}^{t}|u-u^{\prime}|^{2}\,ds\,e^{Ct}\leq\|u-u^{\prime}\|^{2}_{L^{2}([0,T])}\,e^{CT}.$

The claim follows by integration over $t\in[0,T]$. ∎

###  E.2. Proof of Lemma 4.3

###### Proof.

We first show that there exists a constant $C>0$, depending only on the final
time $T$, such that

$\|v\|_{L^{\infty}}\leq C\|u\|_{L^{2}}.$

To this end, we simply note that integrating (4.2) from $0$ to $T$ and taking
into account that $v(0)=0$, we have

$\displaystyle|v(t)|$
$\displaystyle\leq\int_{0}^{t}|g(v(s))|\,ds+\int_{0}^{t}|u(s)|\,ds$
$\displaystyle\leq\|g^{(1)}\|_{L^{\infty}}\int_{0}^{t}|v(s)|\,ds+\sqrt{T}\|u\|_{L^{2}([0,T])}$
$\displaystyle\leq C_{1}\int_{0}^{t}|v(s)|\,ds+\sqrt{T}\|u\|_{L^{2}([0,T])}.$

Gronwall’s inequality implies that

$|v(t)|\leq\sqrt{T}\|u\|_{L^{2}([0,T])}e^{C_{1}T},\quad\forall\,t\in[0,T].$

Thus, $\|v\|_{L^{\infty}([0,T])}\leq C\|u\|_{L^{2}([0,T]}$, where
$C=\sqrt{T}\exp(C_{1}T)$ depends only on $T$. For general $k\in\mathbb{N}$, we
take $k$ derivatives of (4.2) to find:

$\displaystyle\frac{d}{dt}v^{(k)}=g^{(k)}(v)\,v^{(k)}+P_{k}\left(\\{g^{(\ell)}\\}_{\ell=1}^{k-1},\\{v^{\ell}\\}_{\ell=1}^{k-1}\right)+u^{(k)}.$
(E.2)

Here,
$P_{k}=P_{k}\left(\\{g^{(\ell)}\\}_{\ell=1}^{k-1},\\{v^{\ell}\\}_{\ell=1}^{k-1}\right)$
is a polynomial of the following form

$P_{k}\left(\\{g^{(\ell)}\\}_{\ell=1}^{k-1},\\{v^{\ell}\\}_{\ell=1}^{k-1}\right)=\sum_{\ell=1}^{k-1}g^{(\ell)}(v)\left(\sum_{\bm{\gamma}}C_{\ell,\bm{\gamma}}\prod_{j=1}^{N_{\bm{\gamma}}}v^{(\gamma_{j})}\right).$

The sum in the parentheses is over the (finite) set of
$\bm{\gamma}=(\gamma_{1},\dots,\gamma_{N_{\bm{\gamma}}})$,
$\gamma_{j}\in\mathbb{N}$, satisfying

$\sum_{j=1}^{N_{\bm{\gamma}}}\gamma_{j}=k,\quad
1\leq\gamma_{j}<k,\quad\forall\,j=1,\dots,N_{\bm{\gamma}}.$

The coefficients $C_{\ell,\bm{\gamma}}$ are combinatorial coefficients that
depend only on $k$, and which can in principle be determined for any given
$k$. We will prove the claimed estimate on $\|v^{(k)}\|_{L^{\infty}}$ by
induction on $k=1,2,\dots$. We will first estimate the size of the polynomial
$P_{k}$: To this end, note that for $k=1$, the sum defining $P_{k}$ is
necessarily empty and thus, we have $P_{k}\equiv 0$. For $k>1$, we assume that
the claimed inequality for $\|v^{(k)}\|_{L^{\infty}}$ has already been proven
to hold for derivatives $v^{(\gamma_{j})}$ of order $\gamma_{j}\leq k-1$. In
this case, we can estimate

$|P_{k}|\leq\sum_{\ell=1}^{k-1}\|g^{(\ell)}\|_{L^{\infty}}\left(\sum_{\bm{\gamma}}|C_{\ell,\bm{\gamma}}|\prod_{j=1}^{N_{\bm{\gamma}}}\|v^{(\gamma_{j})}\|_{L^{\infty}}\right),$

and

$\displaystyle\prod_{j=1}^{N_{\bm{\gamma}}}\|v^{(\gamma_{j})}\|_{L^{\infty}}$
$\displaystyle\leq\prod_{j=1}^{N_{\bm{\gamma}}}A_{\gamma_{j}}\left(1+\|u\|_{H^{\gamma_{j}}}\right)^{\gamma_{j}}$
$\displaystyle\leq\prod_{j=1}^{N_{\bm{\gamma}}}A_{\gamma_{j}}\left(1+\|u\|_{H^{k}}\right)^{\gamma_{j}}$
$\displaystyle=\left(\prod_{j=1}^{N_{\bm{\gamma}}}A_{\gamma_{j}}\right)\left(1+\|u\|_{H^{k}}\right)^{k}.$

Thus, taking into account that the derivatives $g^{(\ell)}(v)$ of order
$\ell\leq k-1$ are assumed to be uniformly bounded, $|g^{(\ell)}(v)|\leq
C_{k}$, we can now estimate

$|P_{k}|\leq C(1+\|u\|_{H^{k}}^{k}),\quad(k>1),$

where $C=C(k,g)>0$ is a constant depending only on $k$ and $g$. As pointed out
above, this inequality for $P_{k}$ holds trivially also for $k=1$, since
$P_{k}\equiv 0$, in this case. Inserting the above estimate in (E.2), and
integrating over $[0,t]$, we obtain

$\displaystyle|v^{(k)}(t)|$
$\displaystyle\leq\|g^{(k)}\|_{L^{\infty}}\int_{0}^{t}|v^{(k)}(s)|\,ds+C(1+\|u\|_{H^{k}})^{k}+\int_{0}^{T}|u^{(k)}(s)|\,ds$
$\displaystyle\leq
C_{k}\int_{0}^{t}|v^{(k)}(s)|\,ds+C(1+\|u\|_{H^{k}})^{k}+\sqrt{T}\|u\|_{H^{k}}.$

Increasing the constant $C$ if necessary, we can absorb the last term in the
second term and conclude that there exists a constant $C=C(k,g,T)>0$, such
that

$|v^{(k)}(t)|\leq C_{k}\int_{0}^{t}|v^{(k)}(s)|\,ds+C(1+\|u\|_{H^{k}})^{k}.$

Gronwall’s inequality now yields

$|v^{(k)}(t)|\leq Ce^{C_{k}T}(1+\|u\|_{H^{k}})^{k},$

for all $t\in[0,T]$. Hence, for $A_{k}=Ce^{C_{k}T}$, we have

$\|v^{(k)}\|_{L^{\infty}([0,T])}\leq A_{k}(1+\|u\|_{H^{k}})^{k},$

where $A_{k}$ is independent of $u$. ∎

###  E.3. Proof of Lemma 4.4

###### Proof.

The Legendre polynomials form an orthonormal basis of $L^{2}([0,T])$. For
$p\in\mathbb{N}$, let $P_{p}:L^{2}([0,T])\to L^{2}([0,T])$ denote the
orthogonal projection onto the span of the first $p$ Legendre polynomials,
$\mathrm{span}(\widetilde{\tau}_{1},\dots,\widetilde{\tau}_{p})$. By (Canuto &
Quarteroni 1982, Theorem 2.3), for any $k\in\mathbb{N}$, there exists a
constant $C=C(T,k)>0$, such that

$\|u-P_{p}u\|_{L^{2}([0,T])}\leq Cp^{-k}\|u\|_{H^{k}([0,T])}.$

Thus, if $\mathcal{R}=\mathcal{R}_{\widetilde{\bm{\tau}}}$ is the
reconstruction with trunk net
$\widetilde{\bm{\tau}}=(0,\widetilde{\tau}_{1},\dots,\widetilde{\tau}_{p})$,
and if $\mathcal{P}$ denotes the corresponding optimal projection (3.21), then
$P_{p}=\mathcal{R}\circ\mathcal{P}$, and

$\displaystyle\widehat{\mathscr{E}}_{\mathcal{R}}$
$\displaystyle=\left(\int_{X}\|\mathcal{R}\circ\mathcal{P}(v)-v\|_{L^{2}([0,T])}^{2}\,d(\mathcal{G}_{\\#}\mu)(v)\right)^{1/2}$
$\displaystyle\leq\frac{C}{p^{k}}\left(\int_{X}\|v\|_{H^{k}([0,T])}^{2}\,d(\mathcal{G}_{\\#}\mu)(v)\right)^{1/2}$
$\displaystyle=\frac{C}{p^{k}}\left(\int_{X}\|\mathcal{G}(u)\|_{H^{k}([0,T])}^{2}\,d\mu(u)\right)^{1/2}$

By Lemma 4.3, we can furthermore estimate $\|v\|_{H^{k}}\leq
C(1+\|u\|_{H^{k}})^{k}$, for some constant $C>0$, depending on $T$ and $k$.
Hence, we find

$\displaystyle\widehat{\mathscr{E}}_{\mathcal{R}}$
$\displaystyle\leq\frac{C}{p^{k}}\left(\int_{X}\left(1+\|u\|_{H^{k}([0,T])}\right)^{2k}\,d\mu(u)\right)^{1/2},$

where $C=C(k,T)>0$ is a constant independent of $p$. ∎

###  E.4. Proof of Lemma 4.6

###### Proof.

We have by (4.12)

$|v_{i}(t)|\leq\alpha+\int_{0}^{t}|v_{i}(s)|\beta(s)\,ds,$

where $\alpha=\|\mathrm{Im}(u)\|_{L^{\infty}}T$, and
$\beta(s)=\gamma(1+\exp(|v_{i}(s)|))$. By Gronwall’s inequality, it follows
that

$\displaystyle|v_{i}(t)|$
$\displaystyle\leq\alpha\exp\left(\int_{0}^{t}\beta(s)\,ds\right)$
$\displaystyle=\|\mathrm{Im}(u)\|_{L^{\infty}}T\exp\left(\int_{0}^{t}\gamma\left(1+e^{|v_{i}(s)|}\right)\,ds\right).$

Let

$\delta:=\frac{1}{2T\exp(\gamma(1+e)T)}.$

We claim that if $\|\mathrm{Im}(u)\|_{L^{\infty}}\leq\delta$, then
$|v_{i}(t)|<1$ for all $t\in[0,T]$. Suppose this was not the case. If there
exists $t\in[0,T]$ such that $|v_{i}(t)|\geq 1$, then the set

$B:={\left\\{t\in[0,T]\,\middle|\,|v_{i}(t)|\geq 1\right\\}}$

is nonempty. Let $t_{0}$ be given by

$t_{0}=\inf B.$

By the continuity of $t\mapsto v_{i}(t)$, $B$ is a closed set. In particular,
this implies that $t_{0}\in B$. Since $v_{i}(0)=0$, we must have $t_{0}>0$,
and $|v_{i}(t)|<1$ for all $t\in[0,t_{0})$. But then,

$\displaystyle|v_{i}(t_{0})|$
$\displaystyle\leq\|\mathrm{Im}(u)\|_{L^{\infty}}T\exp\left(\int_{0}^{t_{0}}\gamma\left(1+e^{|v_{i}(s)|}\right)\,ds\right)$
$\displaystyle\leq\|\mathrm{Im}(u)\|_{L^{\infty}}\,T\exp(\gamma(1+e)T)$
$\displaystyle\leq\delta\,T\exp(\gamma(1+e)T)$
$\displaystyle\leq\frac{1}{2}<1,$

leads to a contradiction to the assumption that $|v_{i}(t_{0})|\geq 1$. Thus,
we conclude that the set $B$ must in fact be empty, i.e. that

$|v_{i}(s)|<1,\quad\text{for all }s\in[0,T].$

∎

###  E.5. Proof of Lemma 4.7

###### Proof.

We argue by contradiction. Suppose the claim is not true. Then there exists
$u:[0,T]\to\mathbb{C}$, $u\in L^{\infty}([0,T])$, and $0<T_{1}<T$ such that
the solution444We note that short-time existence follows from the fact that
the right-hand side is locally Lipschitz continuous in $v$, so that $T_{1}>0$.
of (4.11) is defined on $[0,T_{1})$, but $\lim_{t\nearrow
T_{1}}|v(t)|=\infty$. Since the right-hand side of (4.11) is uniformly
Lipschitz continuous in $v_{r}$, this can only be the case, if
$\lim_{t\nearrow T_{1}}|v_{i}(t)|=\infty$. In particular, there exists
$T_{0}<T_{1}$, such that $|v_{i}(T_{0})|>1$. But then, this would imply the
existence of a solution of (4.11), which is defined on $[0,T_{0}]$, for which
$\sup_{s\in[0,T]}|\mathrm{Im}(u(s))|\leq\delta$, and such that we have
$\sup_{s\in[0,T_{0}]}|v_{i}(s)|\geq|v_{i}(T_{0})|>1$. This is clearly in
contradiction with Lemma 4.6. By contradiction, it thus follows that we must
have $T_{0}=T$, and hence the solution of (4.11) exists on $[0,T]$ for any
$u\in L^{\infty}([0,T])$, with
$\|\mathrm{Im}(u)\|_{L^{\infty}([0,T])}\leq\delta$. Furthermore, it follows
from Lemma 4.6 that $\sup_{s\in[0,T]}|v_{i}(s)|\leq 1$ in this case. ∎

###  E.6. Proof of Lemma 4.8

###### Proof.

Fix $z_{k}\in E_{\rho_{k}}$ for $k\neq j$. For $z_{j}\in E_{\rho_{j}}$, and by
slight abuse of notation, let us denote the forcing by
$u(t,z_{j})=u(t,\bm{z})$, and the corresponding solution of the pendulum
equations (4.11) by $v(t,z_{j})=\mathcal{F}(\bm{z})(t)$, where
$\bm{z}=(z_{\ell})_{\ell\in\mathbb{N}}$. Then the claim is that $z_{j}\mapsto
v(t,z_{j})$ is complex-differentiable in $z_{j}\in E_{\rho_{j}}$. We note that
$v(t,z_{j})$ is the unique fixed point of

$v(t,z_{j})=\int_{0}^{t}G(s,z_{j},v(s,z_{j}))\,ds,$

where $G(t,z_{j},v):=g(v)+u(t,z_{j})$. To show boundedness, we note that by
(4.13), the assumed $(\bm{b},\delta)$-admissibility of $\bm{\rho}$ and Lemma
4.7, the solution $t\mapsto v(t,z_{j})$ exists for any $z_{j}\in
E_{\rho_{j}}$, and that $\sup_{t\in[0,T]}|\mathrm{Im}(v(t,z_{j}))|\leq 1$. But
then, also the real part $v_{r}(t):=\mathrm{Re}(v(t,z_{j}))$ is bounded,
because from (4.11), we conclude that

$\displaystyle\left|\frac{dv_{r}(t)}{dt}\right|$
$\displaystyle\leq|g_{r}(v_{r}(t),v_{i}(t))|+|\mathrm{Re}(U(t))|$
$\displaystyle\leq|v_{r}(t)|+k\cosh(|v_{i}(t)|)+|\mathrm{Re}(U(t))|$
$\displaystyle\leq|v_{r}(t)|+k\cosh(1)+\|\mathrm{Re}(U)\|_{L^{\infty}([0,T])},$

which implies by Gronwall’s inequality that

$|v_{r}(t)|\leq\left(k\cosh(1)+\|\mathrm{Re}(U)\|_{L^{\infty}([0,T])}\right)Te^{T},$

for all $t\in[0,T]$. We furthermore note that

$\displaystyle\|\mathrm{Re}(U)\|_{L^{\infty}([0,T])}$
$\displaystyle\leq\sum_{k=1}^{\infty}|\mathrm{Re}(z_{k})|\alpha_{k}\|\psi_{k}\|_{L^{\infty}}=\sum_{k=1}^{\infty}|\mathrm{Re}(z_{k})|b_{k}$
$\displaystyle\leq\sum_{k=1}^{\infty}\rho_{k}b_{k}=\sum_{k=1}^{\infty}b_{k}+\sum_{k=1}^{\infty}(\rho_{k}-1)b_{k}$
$\displaystyle\leq\|\bm{b}\|_{\ell^{1}(\mathbb{N})}+\delta<\infty,$

is uniformly bounded for all $\bm{z}\in E_{\bm{\rho}}$ for any admissible
$\bm{\rho}$. This shows that

$\displaystyle\sup_{\bm{z}\in E_{\bm{\rho}}}\|v\|_{L^{2}_{\mathbb{C}}([0,T])}$
$\displaystyle\leq\sup_{\bm{z}\in
E_{\bm{\rho}}}\sqrt{T}\|v\|_{L^{\infty}([0,T])}$
$\displaystyle\leq\sup_{\bm{z}\in
E_{\bm{\rho}}}\sqrt{T}\|v_{r}\|_{L^{\infty}([0,T])}+\sup_{\bm{z}\in
E_{\bm{\rho}}}\sqrt{T}\|v_{i}\|_{L^{\infty}([0,T])}$ $\displaystyle\leq
T\sqrt{T}e^{T}\left(\gamma\cosh(1)+\|\bm{b}\|_{\ell^{1}(\mathbb{N})}+\delta\right)+\sqrt{T}$
$\displaystyle=:C,$

is uniformly bounded for all $(\bm{b},\epsilon)$-admissible $\bm{\rho}$.

Finally, we prove the holomorphy of $z_{j}\mapsto v(t,z_{j})$: By definition,
$z_{j}\mapsto u(t,z_{j})$ is an affine function. Hence $z_{j}\mapsto
G(t,z_{j},v)$ is also an affine function of $z_{j}$, and in particular
differentiable in $z_{j}$. Since $v(t,z_{j})$ exists and is bounded for all
$z_{j}\in E_{\rho_{j}}$, it then follows from the general theory of parametric
ODEs that $\partial_{z_{j}}v(t,z_{j})$ exists and that

$\partial_{z_{j}}v(t,z_{j})=\int_{0}^{t}\left\\{\partial_{z_{j}}G(s,z_{j},v(s,z_{j}))+\partial_{v}G(s,z_{j},v(s,z_{j}))\partial_{z_{j}}v(s,z_{j})\right\\}\,ds.$

This implies that $z_{j}\mapsto v(t,z_{j})$ is a holomorphic mapping. ∎

###  E.7. Proof of Lemma 4.14

###### Proof.

The difference $w=u-u^{\prime}$ is a solution of the equation

$\displaystyle\nabla\cdot\left(a\nabla w\right)$
$\displaystyle=\nabla\cdot\left(a\nabla u\right)-\nabla\cdot\left(a\nabla
u^{\prime}\right)$ $\displaystyle=f-\nabla\cdot\left(a\nabla
u^{\prime}\right)$ $\displaystyle=\nabla\cdot\left(a^{\prime}\nabla
u^{\prime}\right)-\nabla\cdot\left(a\nabla u^{\prime}\right)$
$\displaystyle=\nabla\cdot\left((a^{\prime}-a)\nabla u^{\prime}\right).$

By elliptic theory, we thus have

$\displaystyle\|u-u^{\prime}\|_{L^{2}(D)}$
$\displaystyle=\|w\|_{L^{2}(D)}\leq\|\nabla\cdot\left((a^{\prime}-a)\nabla
u^{\prime}\right)\|_{H^{-1}(D)}$ $\displaystyle\leq\|(a^{\prime}-a)\nabla
u^{\prime}\|_{L^{2}(D)}\leq\|a^{\prime}-a\|_{L^{\infty}(D)}\|\nabla
u^{\prime}\|_{L^{2}(D)}$
$\displaystyle\leq\|a^{\prime}-a\|_{L^{\infty}(D)}\|u^{\prime}\|_{H^{1}_{0}(D)}\leq\|a^{\prime}-a\|_{L^{\infty}(D)}C\|f\|_{L^{2}(D)}.$

∎

###  E.8. Proof of Lemma 4.16

###### Proof.

It is well-known that if $u$ is a solution of (4.18), with smooth coefficient
$a(x)$ and right-hand side $f\in H^{k}$, then $u\in H^{k+1}$. The main point
of this lemma is the explicit dependence on the norm of $a$, which will be
required to estimate the reconstruction error. Let
$\bm{k}=(k_{1},\dots,k_{m})\in\mathbb{N}_{0}^{m}$ denote any multi-index. Then
by differentiation of (4.18), we find

$\displaystyle\nabla\cdot\left(a(x)\nabla\partial_{x}^{\bm{k}}u\right)=-\nabla\cdot\left(\sum_{\bm{\ell}<\bm{k}}{\bm{k}\choose\bm{\ell}}\partial_{x}^{\bm{k-\ell}}a(x)\nabla\partial_{x}^{\bm{\ell}}u\right)+\partial_{x}^{\bm{k}}f.$
(E.3)

where $\ell$ runs over all indices
$\bm{\ell}=(\ell_{1},\dots,\ell_{m})\in\mathbb{N}_{0}$, such that
$\ell_{j}\leq k_{j}$ for all $j=1,\dots,m$, with a strict inequality for at
least one $j$. We also write

${\bm{k}\choose\bm{\ell}}:=\prod_{j=1}^{m}{k_{j}\choose\ell_{j}}.$

Integrating (E.3) against $\partial_{x}^{\bm{k}}u$, with
$k:=|\bm{k}|=k_{1}+\dots+k_{m}$, it follows that

$\displaystyle\lambda\|\nabla\partial_{x}^{\bm{k}}u\|^{2}_{L^{2}_{x}}$
$\displaystyle\leq\int_{\mathbb{T}^{n}}a(x)|\nabla\partial_{x}^{\bm{k}}(x)|^{2}\,dx$
$\displaystyle\leq
C_{k}\sum_{\bm{\ell}<\bm{k}}\int_{\mathbb{T}^{n}}\left|\partial_{x}^{\bm{k-\ell}}a(x)\right|\left|\nabla\partial_{x}^{\bm{\ell}}u(x)\right|\left|\nabla\partial_{x}^{\bm{k}}u(x)\right|\,dx$
$\displaystyle\qquad+\int_{\mathbb{T}^{n}}|\partial_{x}^{\bm{k}}f||\partial_{x}^{\bm{k}}u|\,dx$
$\displaystyle\leq
C_{k}\|a\|_{C^{k}}\|u\|_{H^{k}}\|\nabla\partial_{x}^{\bm{k}}u\|_{L^{2}}+\|\partial_{x}^{\bm{k}}f\|_{L^{2}}\|u\|_{H^{k}}.$

Using the inequality $ab\leq 2^{-1}\epsilon a^{2}+(2\epsilon)^{-1}b^{2}$ for
$a,b,\epsilon>0$, we find

$\displaystyle\lambda\|\nabla\partial_{x}^{\bm{k}}u\|^{2}_{L^{2}_{x}}$
$\displaystyle\leq\frac{C_{k}^{2}\|a\|_{C^{k}}^{2}}{2\lambda}\|u\|_{H^{k}}^{2}+\frac{\lambda}{2}\|\nabla\partial_{x}^{\bm{k}}u\|_{L^{2}}^{2}+\frac{\lambda}{2}\|\partial_{x}^{\bm{k}}f\|_{L^{2}}^{2}+\frac{1}{2\lambda}\|u\|_{H^{k}}^{2},$

and hence

$\|\nabla\partial_{x}^{\bm{k}}u\|^{2}_{L^{2}_{x}}\leq(C_{k}^{2}\|a\|_{C^{k}}^{2}+1)\frac{\|u\|_{H^{k}}^{2}}{\lambda^{2}}+\|\partial_{x}^{\bm{k}}f\|_{L^{2}}^{2}.$

Summing the last estimate over all $|\bm{k}|=k$, and increasing the constant
$C_{k}$, if necessary, we obtain

$\|u\|^{2}_{H^{k+1}}\leq
C_{k}(\|a\|_{C^{k}}^{2}+1)\|u\|_{H^{k}}^{2}+\|f\|_{H^{k}}^{2},$

where the new constant $C_{k}=C_{k}(k,\lambda)$ now depends on both $k$ and
$\lambda$. Repeating the same argument for any $\ell\leq k$, we also find

$\displaystyle\|u\|^{2}_{H^{\ell+1}}\leq
C_{\ell}(\|a\|_{C^{\ell}}^{2}+1)\|u\|_{H^{\ell}}^{2}+\|f\|_{H^{\ell}}^{2}$
$\displaystyle\leq
C^{\prime}(\|a\|_{C^{k}}^{2}+1)\|u\|_{H^{\ell}}^{2}+\|f\|_{H^{k}}^{2},$

with $C^{\prime}:=\max_{\ell\leq k}C_{\ell}$ a fixed constant depending only
on $k$ and $\lambda$. Writing the last inequality in the form

$\|u\|^{2}_{H^{\ell+1}}\leq A\|u\|^{2}_{H^{\ell}}+\|f\|_{L^{2}}^{2},$

it follows by induction on $\ell=0,\dots,k$, that

$\|u\|^{2}_{H^{k+1}}\leq
A^{k}\|u\|_{H^{1}}^{2}+\|f\|_{L^{2}}^{2}\sum_{\ell=0}^{k-1}A^{\ell}.$

For $k=0$, we have the well-known bound $\|u\|_{H^{1}}^{2}\leq
C^{\prime\prime}\|f\|_{L^{2}}^{2}$ with
$C^{\prime\prime}=C^{\prime\prime}(\lambda)$. We may wlog assume
$C^{\prime\prime}\geq 1$. In particular, we then conclude that

$\|u\|^{2}_{H^{k+1}}\leq
C^{\prime\prime}\|f\|_{L^{2}}^{2}\sum_{\ell=0}^{k}A^{\ell},$

where $A:=C^{\prime}(\|a\|_{C^{k}}+1)$. Finally, for fixed $k$, we note that
since $C^{\prime}=C^{\prime}(k,\lambda)$,
$C^{\prime\prime}=C^{\prime\prime}(\lambda)$, there exists a constant $C$
depending only on $k$ and $\lambda$, but independent of $\|a\|_{C^{k}}$, such
that

$C^{\prime\prime}\sum_{\ell=0}^{k}A^{\ell}=C^{\prime\prime}\,C^{\prime}\sum_{\ell=0}^{k}(1+\|a\|_{C^{k}}^{2})^{\ell}\leq
C(1+\|a\|^{2k}_{C^{k}}).$

For such $C=C(k,\lambda)>0$, we conclude that

$\|u\|^{2}_{H^{k+1}}\leq C\|f\|_{H^{k}}^{2}(1+\|a\|_{C^{k}}^{2k}),$

as claimed. ∎

###  E.9. Proof of Theorem 4.22

To prove Theorem 4.22, we recall some notions and results from the theory of
second order parabolic equations (cp. Lieberman (1996)). First, let
$U=[0,T]\times\mathbb{T}^{d}$ denote the domain of the solution of the
parabolic equation (4.33). We denote by (cp. (Lieberman 1996, p. 46))

$\mathcal{C}^{k,\alpha}(U):={\left\\{v\in
C(U)\,\middle|\,|v|_{(k,\alpha)}<\infty\right\\}},\quad
k\in\mathbb{N},\;\alpha\in(0,1),$

the parabolic Hölder space on $U$, where

$|v|_{(k,\alpha)}:=\sum_{\beta+2j\leq
k}\sup_{U}|\partial_{x}^{\beta}\partial^{j}_{t}v|+[v]_{(k,\alpha)},$

and

$[v]_{(k,\alpha)}:=\sum_{\beta+2j=k}[\partial^{\beta}_{x}\partial^{j}_{t}v]_{\alpha},$

and where the parabolic Hölder semi-norm $[w]_{\alpha}$ of a function $w$ is
defined by

$[w]_{\alpha}:=\sup_{(t,x)\in
U}\sup_{(t^{\prime},x^{\prime})\neq(t,x)}\frac{|w(t,x)-w(t^{\prime},x^{\prime})|}{\left(|x-x^{\prime}|+|t-t^{\prime}|^{1/2}\right)^{\alpha}}.$

We then have the following Schauder estimate:

######  Lemma E.1 (Schauder estimate).

Let $\alpha\in(0,1)$. For any $k\in\mathbb{N}$, there exists a constant $C>0$,
such that if $v$ is a solution to

$\left\\{\begin{aligned} \partial_{t}v-\Delta v&=f,\\\
v(t=0)&=0,\end{aligned}\right.$

with $f\in\mathcal{C}^{(k,\alpha)}(U)$, then

$|v|_{(k+2,\alpha)}\leq
C|f|_{(k,\alpha)}+\|u\|_{C^{k+2,\alpha}(\mathbb{T}^{d})}.$

###### Proof.

For $k=0$, this follows e.g. from (Lieberman 1996, Theorem 4.28). For $k>0$,
we note that the coefficients in equation $\partial_{t}v-\Delta v=f$ are
constant, and hence we can apply the base-case to partial derivatives of the
equation (more precisely, finite-difference approximations thereof, and take
the limit). ∎

We also note the following strong $L^{p}$ estimate for parabolic equations:

######  Lemma E.2.

Let $p\in[2,\infty)$. There exists a constant $C>0$, such that if $v\in
L^{\infty}([0,T]\times\mathbb{T}^{d})$ is a weak solution of
$\partial_{t}v-\Delta v=f$, for $f\in L^{\infty}(\mathbb{T}^{d})$, and with
initial data $v(t=0)=u$, then

$\|v\|_{W^{1,p}([0,T]\times\mathbb{T}^{d})}\leq
C\left(\|f\|_{L^{p}}+\|u\|_{W^{2,p}(\mathbb{T}^{d})}\right).$

###### Proof.

Theorem 7.32 of Lieberman (1996) provides a sharper estimate, from which the
claim readily follows. ∎

######  Corollary E.3.

Let $d\in\\{2,3\\}$. There exists a constant $C>0$, and $\alpha\in(0,1)$, such
that if $f\in L^{\infty}$, and $u\in C^{1}([0,T]\times\mathbb{T}^{d})$, and if
$v$ solves $\partial_{t}v-\Delta v=f$, with initial data $v(t=0)=u$, then

$\|v\|_{(0,\alpha)}\leq
C\left(\|f\|_{L^{\infty}}+\|u\|_{C^{2}(\mathbb{T}^{d})}\right).$

###### Proof.

This follows directly from Lemma E.2 and the fact that by Sobolev embedding,
we have $W^{1,p}{\hookrightarrow}C^{\alpha}$, for $\alpha\leq 1-(d+1)/p$ for
$p>d+1$. ∎

###### Proof of Theorem 4.22.

The claim follows from a bootstrap argument: From the a priori estimate of
Theorem 4.21 for the Allen-Cahn equation, we know that for initial data
$\|u\|_{L^{\infty}}\leq 1$, we have $\|v(t)\|_{L^{\infty}}\leq 1$ for all
$t\in[0,T]$. In particular, it follows that $|f|=|f(v)|\leq C$ is bounded in
$L^{\infty}$. Thus, $v$ solves

$\displaystyle\left\\{\begin{aligned} \partial_{t}v-\Delta v&=f,\\\
v(t=0)&=u,\end{aligned}\right.$ (E.4)

with source term $f\in L^{\infty}$. By the strong $L^{p}$ estimate (cp.
Corollary E.3), and the assumed smoothness of $u\in C^{4,\alpha}$, it follows
that $v\in\mathcal{C}^{\alpha}([0,T]\times\mathbb{T}^{d})$ for some
$\alpha\in(0,1)$. In turn this implies the following chain of improved
regularity based on the Schauder estimate of Lemma E.1, and using also the
fact that $u\in C^{k,\alpha}(\mathbb{T}^{d})$ for all $k\leq 4$:

$\displaystyle v\in\mathcal{C}^{\alpha}(U)$ $\displaystyle\implies
f(v)\in\mathcal{C}^{\alpha}(U)\implies v\in\mathcal{C}^{2,\alpha}(U)$
$\displaystyle\implies f(v)\in\mathcal{C}^{2,\alpha}(U)\implies
v\in\mathcal{C}^{4,\alpha}(U).$

Thus, we conclude that if $u\in C^{4,\alpha}(\mathbb{T}^{d})$, then we must
have $v\in\mathcal{C}^{4,\alpha}(U)$. Furthermore, it follows from the
estimates of Corollary E.3, Lemma E.1, that there in fact exists a constant
$\sigma=\sigma(\|u\|_{C^{4,\alpha}})>0$, depending on the $C^{4,\alpha}$-norm
of the initial data $u$, such that

$|v|_{(4,\alpha)}\leq\sigma(\|u\|_{C^{4,\alpha}(\mathbb{T}^{d})}).$

Clearly, we have $\|v\|_{C^{(4,2)}(U)}\leq|v|_{(4,\alpha)}$ for any
$\alpha>0$. The claimed estimate thus follows. ∎

###  E.10. Proof of Corollary 4.23

###### Proof.

Since we have
$\|v(t)\|_{L^{\infty}(\mathbb{T}^{d})},\,\|v^{\prime}(t)\|_{L^{\infty}(\mathbb{T}^{d})}$
for all $t\in[0,T]$, the difference $w=v-v^{\prime}$, solves the following
equation

$\partial_{t}w=\Delta w+F(v,v^{\prime})w,$

where $F(a,b):=1-a^{2}-ab-b^{2}$. It follows from the uniform boundedness of
$v,v^{\prime}$ that $|F(v,v^{\prime})|\leq 4$ is uniformly bounded.
Multiplying the equation by $w$ and integrating over $x$, we obtain

$\frac{d}{dt}\int_{\mathbb{T}^{d}}w^{2}\,dx\leq
8\int_{\mathbb{T}^{d}}w^{2}\,dx.$

And hence, by Gronwall’s inequality, we must have
$\|w(t)\|_{L^{2}}^{2}\leq\|w(0)\|_{L^{2}}^{2}e^{8t}$ for all $t\in[0,T]$. We
conclude that

$\|v(T)-v^{\prime}(T)\|_{L^{2}(\mathbb{T}^{d})}\leq
e^{4T}\|u-u^{\prime}\|_{L^{2}(\mathbb{T}^{d})}.$

∎

###  E.11. Proof of Theorem 4.27

###### Proof.

Our main observation is that the local truncation error for the scheme (4.41),
given by

$T^{n}_{j}:=(I-\Delta tD_{\Delta x})v(t_{n},x_{j})-(v(t_{n},x_{j})+\Delta
t\left(v(t_{n-1},x_{j})-v(t_{n-1},x_{j})^{3}\right),$

has a Taylor expansion

$\displaystyle T^{n}_{j}$ $\displaystyle:=\Delta t\left(\frac{\partial
v(t_{n-1},x_{j})}{\partial t}-\Delta
v(t_{n-1},x_{j})-(v(t_{n-1},x_{j})-v(t_{n-1},x_{j})^{3})\right)$
$\displaystyle\qquad+\Delta t\,R(\Delta t,\Delta x),$

where similar to Tang & Yang (2016), the remainder term $R(\Delta t,\Delta x)$
can be estimated by

$\displaystyle|R(\Delta t,\Delta x)|$ $\displaystyle\leq C\left(\Delta
t\,\left\|\frac{\partial v}{\partial t}\right\|_{L^{\infty}}+\Delta
t\,\left\|\frac{\partial^{2}v}{\partial t^{2}}\right\|_{L^{\infty}}+\Delta
x^{2}\,\sum_{k=1}^{d}\left\|\frac{\partial^{4}v}{\partial
x_{k}^{4}}\right\|_{L^{\infty}}\right)$ $\displaystyle\leq C\left(\Delta
t+\Delta x^{2}\right)\|v\|_{C^{(2,4)}([0,T]\times\mathbb{T}^{d})},$

where $\|v\|_{C^{(2,4)}([0,T]\times\mathbb{T}^{d})}$ is defined by (4.34). We
note that in contrast to our estimate, the remainder term was bounded in Tang
& Yang (2016) by the larger norm $\|v\|_{C^{2}([0,T];C^{4}(\mathbb{T}^{d}))}$.
In view of the available a priori estimates for parabolic equations, the
parabolic norm $\|v\|_{C^{(2,4)}([0,T]\times\mathbb{T}^{d})}$ appears better
adapted to the problem, and provides a less restrictive convergence result for
the scheme. The remainder of the proof is the same as in (Tang & Yang 2016,
Theorem 4.1). ∎

###  E.12. Proof of Lemma 4.28

###### Proof.

By a result of Yarotsky (Yarotsky 2017, Prop. 2, 3), there exists a constant
$C^{\prime}>0$, such that for any $\epsilon>0$, there exists a ReLU network
$\widetilde{\times}:[-2,2]\to\mathbb{R}$, such that
$\mathrm{size}(\widetilde{\times})\leq C^{\prime}(|\log(\epsilon)|+1)$, and

$\sup_{\xi,\eta\in[-2,2]}|\widetilde{\times}(\xi,\eta)-\xi\eta|<\epsilon/2.$

In fact, the mapping constructed in Yarotsky (2017) is based on the identity
$\xi\eta=\frac{1}{2}\left((\xi+\eta)^{2}-\xi^{2}-\eta^{2}\right)$, and finding
a suitable neural network approximation $f_{m}(x)\approx x^{2}$, of the form
(Yarotsky 2017, paragraph above eq. (3))
$f_{m}(x)=x-\sum_{s=1}^{m}2^{-2s}g_{s}(x)$, with
$m=\mathcal{O}(|\log(\epsilon)|)$ and with $g_{s}$ a $s$-fold iteration of the
following sawtooch function $g(x)$:

$g(x)=\begin{cases}2x,&x<\frac{1}{2},\\\
2(1-x),&x\geq\frac{1}{2},\end{cases}\quad
g_{s}(x)=\underbrace{(g\circ\dots\circ g)}_{s\text{ times}}(x).$

It is then immediate that $\mathrm{Lip}(g_{s})\leq\mathrm{Lip}(g)^{s}\leq
2^{s}$, and hence $\mathrm{Lip}(f_{m})\leq
1+\sum_{s=1}^{m}2^{-2s}\mathrm{Lip}(g_{s})\leq 1+\sum_{s=1}^{m}2^{-s}\leq 2$.
It is then readily seen that there exists a constant $M>0$, independent of
$\epsilon$, such that

$\mathrm{Lip}(\widetilde{\times}:[-2,2]^{2}\to)\leq M.$

in addition to the approximation property

$\sup_{\xi,\eta\in[-1,1]^{2}}|\widetilde{\times}(\xi,\eta)-\xi\eta|<\epsilon/2.$

In particular, with the mapping constructed above, we then have
$\sup_{\eta\in[-1,1]}|\widetilde{\times}(\eta,\eta)-\eta^{2}|<\epsilon$, and
from the assumption that $\epsilon<1$, it follows that
$\widetilde{\times}(\eta,\eta)-1\in[-2,2]$ for all $\eta\in[-1,1]$. Observing
that $\xi^{3}-\xi=\xi(\xi^{2}-1)$, we find for any $\eta\in[-1,1]$

$\displaystyle\left|\widetilde{\times}\left(\eta,(\widetilde{\times}(\eta,\eta)-1)\right)-(\eta^{3}-\eta)\right|$
$\displaystyle\leq\left|\widetilde{\times}\left(\eta,(\widetilde{\times}(\eta,\eta)-1)\right)-\eta(\widetilde{\times}(\eta,\eta)-1)\right|$
$\displaystyle\quad+\left|\eta(\widetilde{\times}(\eta,\eta)-1)-\eta(\eta^{2}-1)\right|$
$\displaystyle\leq\sup_{\xi\in[-2,2]}\left|\widetilde{\times}\left(\eta,\xi\right)-\eta\xi\right|$
$\displaystyle\quad+|\eta|\left|\widetilde{\times}(\eta,\eta)-\eta^{2}\right|$
$\displaystyle\leq
2\sup_{\xi,\eta\in[-1,1]}\left|\widetilde{\times}\left(\eta,\xi\right)-\eta\xi\right|$
$\displaystyle<\epsilon.$

To finish the proof, we note that if
$(\xi,\eta)\mapsto\widetilde{\times}(\xi,\eta)$ is represented by a ReLU
neural network of size $\leq C^{\prime}(|\log(\epsilon)|+1)$, then the
function

$g_{\epsilon}(\eta):=\max\left(-1,\min\left(1,\widetilde{\times}\left(\eta,(\widetilde{\times}(\eta,\eta)-1)\right)\right)\right)$

can be represented by a ReLU neural network of size $\leq
C(|\log(\epsilon)|+1)$, where $C$ is a constant multiple of $C^{\prime}$, and
we have $g_{\epsilon}(\eta)\in[-1,1]$ for all $\eta\in\mathbb{R}$.
Furthermore, since $\eta^{3}-\eta\in[-1,1]$ for all $\eta\in[-1,1]$, it also
follows from the above estimate that

$\sup_{\eta\in[-1,1]}|g_{\epsilon}(\eta)-(\eta^{3}-\eta)|\leq\epsilon.$

∎

######  Remark E.4.

If we consider neural networks $g$ with _any smooth_ (e.g. $\sigma\in
C^{3}(\mathbb{R})$) non-linear activation function
$\sigma:\mathbb{R}\to\mathbb{R}$, then the previous approximation result can
be considerably improved (Pinkus 1999, Prop. 3.4): Indeed, by assumption on
$\sigma$ there exists a point $x_{0}\in\mathbb{R}$, such that
$\sigma^{\prime\prime}(x_{0})\neq 0$. Then for $h>0$, $\eta\in[-1,1]$, we have
by Taylor expansion

$\frac{\sigma(x_{0}+\eta h)-2\sigma(x_{0})+\sigma(x_{0}-\eta
h)}{\sigma^{\prime\prime}(x_{0})h^{2}}=\eta^{2}+\mathcal{O}(h),$

as $h\to 0$, uniformly in $\eta\in[-1,1]$. In particular, it follows that for
smooth $\sigma$ there exists a neural network architecture _of fixed size_ ,
such that and for any $h>0$, there is neural network $g:[-1,1]\to\mathbb{R}$,
$\eta\mapsto g(\eta)$, such that $|g(\eta)-\eta^{2}|\leq Ch$ for all
$\eta\in[-1,1]$. As a consequence, using the representation Yarotsky (2017)

$xy=\frac{1}{2}((x+y)^{2}-x^{2}-y^{2}),$

it follows that there exists a constant $C>0$, such that for any $\epsilon>0$,
there exists a neural network $\times_{\epsilon}:[-1,1]^{2}\to\mathbb{R}$ of
size $\mathrm{size}(\times_{\epsilon})\leq C$, such that

$\sup_{x,y\in[-1,1]}|\times_{\epsilon}(x,y)-xy|<\epsilon.$

But then, arguing as in the proof of Lemma 4.28, we find that for any
$\epsilon>0$, the function
$g_{\epsilon}(x)=\times_{\epsilon}(\times_{\epsilon}(x,x),x)-x$ can be
represented by a neural network, with a neural network size,
$\mathrm{size}(g_{\epsilon})\leq C^{\prime}$, which is uniformly bounded in
$\epsilon>0$, and such that
$\sup_{\eta\in[-1,1]}|g_{\epsilon}(\eta)-(\eta^{3}-\eta)|<\epsilon$.

###  E.13. Proof of Lemma 4.29

###### Proof.

By Lemma 4.28, the non-linearity in (4.42) can be represented by a neural
network with size bounded by

$\begin{gathered}\mathrm{size}(g_{\epsilon})\leq C(1+|\log(\epsilon)|),\quad
0pt(g_{\epsilon})\leq C(1+|\log(\epsilon)|).\end{gathered}$

It follows that the mapping

$\left(\widetilde{U}^{k}_{1},\dots,\widetilde{U}^{k}_{m}\right)\mapsto\left(g_{\epsilon}\left(\widetilde{U}^{k}_{1}\right),\dots,g_{\epsilon}\left(\widetilde{U}^{k}_{m}\right)\right)=:G_{\epsilon}(\widetilde{U}^{k}),$

can be represented by a neural network
$G_{\epsilon}:\mathbb{R}^{m}\to\mathbb{R}^{m}$, with

$\begin{gathered}\mathrm{size}(G_{\epsilon})=\mathcal{O}(m|\log(\epsilon)|),\quad
0pt(G_{\epsilon})=\mathcal{O}(|\log(\epsilon)|).\end{gathered}$

Since the identity mapping $\widetilde{U}^{k}\mapsto\widetilde{U}^{k}$ can be
represented by a ReLU network with $\mathrm{size}=\mathcal{O}(m)$,
$0pt=\mathcal{O}(1)$, there exists a neural network with
$\mathrm{size}=\mathcal{O}(m|\log(\epsilon)|)$,
$0pt=\mathcal{O}(|\log\epsilon|)$, which represents

$\widetilde{U}^{k}\mapsto\widetilde{U}^{k}+\Delta
tG_{\epsilon}(\widetilde{U}^{k}).$

Finally, we note that

$\widetilde{U}^{k}+\Delta tG_{\epsilon}(\widetilde{U}^{k})\mapsto R_{\Delta
x,\Delta t}(\widetilde{U}^{k}+\Delta tG_{\epsilon}(\widetilde{U}^{k})),$

is simply a matrix-vector multiplication of a _fixed_ matrix $R_{\Delta
x,\Delta t}=(I-\Delta t\,D_{\Delta x})^{-1}\in\mathbb{R}^{m\times m}$
(independent of $\widetilde{U}^{k}$), with a vector in $\mathbb{R}^{m}$. This
operation can be represented by a ReLU neural network layer with
$\mathrm{size}=\mathcal{O}(m^{2})$, $0pt=\mathcal{O}(1)$. We conclude that the
composition

$\widetilde{U}^{k}\mapsto\widetilde{U}^{k}+\Delta
t\,G_{\epsilon}(\widetilde{U}^{k})\mapsto R_{\Delta x,\Delta
t}\left(\widetilde{U}^{k}+\Delta
t\,G_{\epsilon}(\widetilde{U}^{k})\right)=\widetilde{U}^{k+1},$

can be represented by a neural network
$\widehat{\mathcal{N}}:\mathbb{R}^{m}\to\mathbb{R}^{m}$ with

$\displaystyle\mathrm{size}(\widehat{\mathcal{N}})=\mathcal{O}(m^{2}+m|\log\epsilon|),\quad
0pt(\widehat{\mathcal{N}})=\mathcal{O}(|\log\epsilon|).$

Since
$\widetilde{U}^{0}\mapsto\widetilde{U}^{1}\mapsto\dots\mapsto\widetilde{U}^{n}$
involves the composition of $n$ such steps, we conclude that
$\widetilde{U}^{0}\mapsto\widetilde{U}^{n}$ can be represented by a neural
network
$\mathcal{N}=\widehat{\mathcal{N}}\circ\widehat{\mathcal{N}}\circ\dots\circ\widehat{\mathcal{N}}:\mathbb{R}^{m}\to\mathbb{R}^{m}$
($n$ iterations) with

$\displaystyle\mathrm{size}(\mathcal{N})=\mathcal{O}(n(m^{2}+m|\log\epsilon|)),\quad
0pt(\mathcal{N})=\mathcal{O}(n|\log(\epsilon)|).$

∎

###  E.14. Proof of Lemma 4.30

###### Proof.

We note that

$\displaystyle U^{k+1}-\widetilde{U}^{k+1}$ $\displaystyle=R_{\Delta x,\Delta
t}\left(U^{k}-\widetilde{U}^{k}+\Delta
t\left[g_{\epsilon}(U^{k})-g_{\epsilon}(\widetilde{U}^{k})\right]\right)$
$\displaystyle\quad+\Delta tR_{\Delta x,\Delta
t}\left(\left[g_{\epsilon}(U^{k})-f({U}^{k})\right]\right).$

We note that $\|U^{k}\|_{\ell^{\infty}}\leq 1$ for all $k>0$, by Theorem 4.26.
Furthermore, it has been shown in Tang & Yang (2016) that $\|R_{\Delta
x,\Delta t}\|_{\ell^{\infty}\to\ell^{\infty}}\leq 1$. It follows that

$\left\|R_{\Delta x,\Delta
t}\left(\left[g_{\epsilon}(U^{k})-f({U}^{k})\right]\right)\right\|_{\ell^{\infty}}\leq\sup_{\eta\in[-1,1]}|g_{\epsilon}(\eta)-f(\eta)|\leq\epsilon.$

Denote now $E^{k}=\|U^{k}-\widetilde{U}^{k}\|_{\ell^{\infty}}$, for
$k=0,\dots,n$. Then

$\displaystyle E^{k+1}$ $\displaystyle\leq\|R_{\Delta t,\Delta
x}\|_{\ell^{\infty}\to\ell^{\infty}}\left(E^{k}+\Delta
t\,\mathrm{Lip}(g_{\epsilon})E^{k}\right)+\Delta t\,\epsilon$
$\displaystyle\leq\left(1+\Delta
t\,\mathrm{Lip}(g_{\epsilon})\right)E^{k}+\Delta t\,\epsilon.$

Summing over $k=1,\dots,\ell$, and taking into account that $E^{0}=0$, we find
for any $\ell\in\\{1,\dots,n\\}$ that

$\displaystyle E^{\ell}$
$\displaystyle=E^{0}+\sum_{k=0}^{\ell-1}[E^{k+1}-E^{k}]$ $\displaystyle\leq
T\epsilon+\Delta t\,\sum_{k=1}^{\ell-1}\mathrm{Lip}(g_{\epsilon})E^{k}.$

By the Gronwall inequality, it follows that

$E^{n}\leq\epsilon\,T\exp(\mathrm{Lip}(g_{\epsilon})T).$

The result follows from $\mathrm{Lip}(g_{\epsilon})\leq M$. ∎

###  E.15. Proof of Proposition 4.31

###### Proof.

By definition, the approximator neural network $\mathcal{A}$ should provide an
approximation $\mathcal{A}\approx\mathcal{P}\circ\mathcal{G}\circ\mathcal{D}$.
Our goal is to construct $\mathcal{A}$ based on the neural network
approximation (4.42) of the convergent numerical scheme (4.41). To this end,
we first note that by Lemmas 4.29, 4.30, there exists a constant
$C=C(\sup_{u\in\mathrm{supp}(\mu)}\|u\|_{C^{(2,4)}},T)>0$, independent of $m$,
$n$ and $\epsilon$, and a neural network $\mathcal{N}$ with

$\mathrm{size}(\mathcal{N})\leq C(1+n(m^{2}+m|\log(\epsilon)|)),\quad
0pt(\mathcal{N})\leq C(1+n|\log(\epsilon)|),$

such that for any $u\in\mathrm{supp}(\mu)$, we have

$\max_{j=1,\dots,m}|\mathcal{N}_{j}(\bm{u})-v(x_{j},T)|\leq C(\epsilon+\Delta
x^{2}+T/n),$

where $\bm{u}=(u(x_{1}),\dots,u(x_{m}))=\mathcal{E}(u)$. We note that for the
present choice of the $x_{j}$ as nodes on an equidistant grid, we have $\Delta
x\sim m^{-2/d}$. Thus, we choose $n=\lceil Tm^{2/d}\rceil$,
$\epsilon=m^{-2/d}$ to conclude that there exists a neural network
$\mathcal{N}:\mathbb{R}^{m}\to\mathbb{R}^{m}$, such that

$\displaystyle\mathrm{size}(\mathcal{N})\leq C(1+m^{2+2/d}),\quad
0pt(\mathcal{N})\leq C(1+m^{2/d}\log(m)),$ (E.5)

and

$\displaystyle\max_{j=1,\dots,m}|\mathcal{N}_{j}(\bm{u})-v(x_{j},T)|\leq
Cm^{-2/d}.$ (E.6)

Given the grid points $x_{j}$, we note that we can define a linear mapping

$\mathcal{L}:\mathbb{R}^{m}\to
C(D),\quad\bm{v}=(v_{1},\dots,v_{m})\mapsto\mathcal{L}(\bm{v})(x),$

by employing local linear interpolation of the values $v_{j}$ on the mesh
$x_{j}$, i.e. such that $\mathcal{L}(\bm{v})(x_{j})=v_{j}$. Given the
projection operator $\mathcal{P}$, we define a linear mapping
$\widehat{\mathcal{L}}:\mathbb{R}^{m}\to\mathbb{R}^{p}$ by
$\widehat{\mathcal{L}}(\bm{v}):=\mathcal{P}\circ\mathcal{L}(\bm{v})$. With
these definitions, we now set