# Wide stable neural networks: Sample regularity, functional convergence and
Bayesian inverse problems

Tomás Soto  School of Engineering Sciences, LUT University<EMAIL_ADDRESS>
###### Abstract.

We study the large-width asymptotics of random fully connected neural networks
with weights drawn from $\alpha$-stable distributions, a family of heavy-
tailed distributions arising as the limiting distributions in the Gnedenko-
Kolmogorov heavy-tailed central limit theorem. We show that in an arbitrary
bounded Euclidean domain $\mathcal{U}$ with smooth boundary, the random field
at the infinite-width limit, characterized in previous literature in terms of
finite-dimensional distributions, has sample functions in the fractional
Sobolev-Slobodeckij-type quasi-Banach function space $W^{s,p}(\mathcal{U})$
for integrability indices $p<\alpha$ and suitable smoothness indices $s$
depending on the activation function of the neural network, and establish the
functional convergence of the processes in
$\mathcal{P}(W^{s,p}(\mathcal{U}))$. This convergence result is leveraged in
the study of functional posteriors for edge-preserving Bayesian inverse
problems with stable neural network priors.

###### Key words and phrases:

Neural networks, $\alpha$-stable distributions, sample regularity, fractional
Sobolev spaces, Bayesian inverse problems

###### 2020 Mathematics Subject Classification:

Primary: 68T07, 62F15, 60G52, 60G17; Secondary: 46E35

##  1\. Introduction

The study of scaling limits of wide random neural networks was initiated by R.
Neal in [32], in the context of priors for Bayesian learning. Neal used the
classical central limit theorem to show, roughly speaking, that a perceptron
with one hidden layer of width $H\in\mathbb{N}$ and suitable assumptions (such
as that of finite variance) on the weight distributions and the activation
function, converges with respect to finite-dimensional marginal distributions
to a Gaussian process as $H\to\infty$.

The Gaussian process behaviour at the infinite-width limit has since been
studied for a variety of neural network architectures, and different modes of
convergence, in an expansive collection of papers by numerous authors.
Motivations, apart from the context of Bayesian learning proposed by Neal,
include the understanding of different initialization schemes for training,
the formal study of gradient descent based training dynamics via the so-called
neural tangent kernel at the infinite-width limit, and quantitative estimation
of the convergence rates. As examples we mention the papers [31, 21, 29, 41,
18, 10, 2, 6, 13], although this list is by necessity very incomplete.

The focus of this paper is in the non-Gaussian regime, where we consider
heavy-tailed distributions for the weight distributions, resulting with
suitable assumptions and scaling in an _$\alpha$ -stable process_ at the
infinite-width limit. These neural networks have been considered as priors for
edge-preserving Bayesian inversion in [30], and recently it has been observed
in [17] that heavy-tail like behavior for the network weights can also arise
naturally in deep learning via gradient descent training.

In Sections 1.1–1.2 below, we recall the relevant concepts and definitions,
and present a brief overview of literature pertaining to the heavy-tailed
regime. The main results are presented in Section 1.3. Section 1.4 contains an
overview of the paper’s structure.

###  1.1. Stable distributions

Most random variables considered in this paper will have a _Lévy
$\alpha$-stable_ distribution. We recall here their basic properties, and
mention [35] as the standard reference for both univariate and multivariate
stable distributions.

We are in particular interested in symmetric $\alpha$-stable distributions,
which are characterized in terms of a _stability index_ $\alpha\in(0,2]$ and a
_scale parameter_ $\sigma>0$, in the sense that if the random variable $u$ has
an $\alpha$-stable distribution, its characteristic function is given by

$\mathbb{E}\left[\exp\left(i\theta
u\right)\right]=\exp\left(-\left(\sigma|\theta|\right)^{\alpha}\right)$ (1)

for all $\theta\in\mathbb{R}$. In this case, we write

$u\sim\mathcal{S}_{\alpha}(\sigma).$

From (1) it is routinely seen that $2$-stable distributions are zero-mean
Gaussians, and we will largely ignore this edge case in the sequel. Besides
the case $\alpha=2$, a closed-form expression for the density function for a
random variable $u$ distributed as (1) is known only for $\alpha=1$, in which
case $u$ has a standard Cauchy distribution. However, for all
$\alpha\in(0,2)$, $u$ is a heavy-tailed distribution, and denoting by $p$ and
$F$ its density and cumulative density functions respectively, we have

$p(x)\sim c_{\alpha,\sigma}x^{-\alpha-1}\quad\text{and}\quad 1-F(x)\sim
c^{\prime}_{\alpha,\sigma}x^{-\alpha}$

as $x\to\infty$ for some constants $c_{\alpha,\sigma}$,
$c^{\prime}_{\alpha,\sigma}>0$. This obviously means that for $p>0$ we have
$\mathbb{E}[|u|^{p}]<\infty$ if and only if $p<\alpha$.

By the generalized heavy-tailed central limit theorem (see e.g. [15, §35,
Theorem 2]), the symmetric $\alpha$-stable distributions are precisely the
heavy-tailed distributions arising as the limiting distributions of suitably
scaled sums of i.i.d symmetric random variables with heavy power-law-like
tails.

An immediate consequence of (1) (and in fact an alternative definition for
symmetric $\alpha$-stable distributions) is that if $u_{1}$, $u_{2}$,
$\cdots$, $u_{n}$ and $u$ are independently distributed as
$\mathcal{S}_{\alpha}(\sigma)$, then

$\sum_{i=1}^{n}\theta_{i}u_{i}\stackrel{{\scriptstyle
d}}{{=}}\left(\sum_{i=1}^{n}|\theta_{i}|^{\alpha}\right)^{1/\alpha}u=\mathrel{\mathop{:}}\|\boldsymbol{\theta}\|_{l^{\alpha}}u$

for all
$\boldsymbol{\theta}\mathrel{\mathop{:}}=(\theta_{1},\cdots,\theta_{n})\in\mathbb{R}^{n}$,
generalizing the familiar property from Gaussian distributions.

Multivariate symmetric $\alpha$-stable distributions can be defined in a
similar manner, with a _spectral measure_ $\Lambda$ instead of the scale
parameter $\sigma$. Since we will not need to directly access the spectral
measure of any multivariate stable distribution in this paper, let us simply
recall [35, Theorem 2.1.5] that a random vector in $\mathbb{R}^{d}$ is
symmetric $\alpha$-stable if and only if all linear combinations of its
components have a univariate symmetric $\alpha$-stable distribution.

Finally, a stochastic process $(f(x))_{x\in\mathcal{X}}$ with any (possibly
uncountable) indexing set $\mathcal{X}$ is said to be symmetric
$\alpha$-stable if $(f(x_{1}),\cdots,f(x_{n}))\in\mathbb{R}^{n}$ is symmetric
$\alpha$-stable for all finite $(x_{1},\cdots,x_{n})\subset\mathcal{X}$.

###  1.2. The model and previous literature

Write $\alpha\in(0,2)$ for the stability index, fixed from now. For the sake
of notational convenience, we will for the most part consider shallow neural
networks with a single hidden layer (extensions of the main results to deeper
architectures are presented in Section 4) and real output. We thus define the
network $f^{H}$ of width $H\in\mathbb{N}$ as

$f^{H}(x)\mathrel{\mathop{:}}=\frac{1}{H^{1/\alpha}}\sum_{i=1}^{H}v_{i}\varphi(u_{i}^{\intercal}x+a_{i})+b$

for $x\in\mathbb{R}^{d}$, where the $v_{i}\in\mathbb{R}$, $b\in\mathbb{R}$,
the coordinates of $u_{i}\in\mathbb{R}^{d}$ and $a\in\mathbb{R}$ are all
indepedently distributed as $\mathcal{S}_{\alpha}(\sigma_{v})$,
$\mathcal{S}_{\alpha}(\sigma_{b})$, $\mathcal{S}_{\alpha}(\sigma_{u})$ and
$\mathcal{S}_{\alpha}(\sigma_{a})$ respectively, and
$\varphi\colon\mathbb{R}\to\mathbb{R}$ is a continuous activation function.

The scaling factor $H^{-1/\alpha}$ above is chosen in spirit of the heavy-
tailed central limit theorem [15, §35, Theorem 2], as in it is in essence what
one would expect to give rise to a scaling limit as $H\to\infty$ in case
$\varphi$ were a bounded function.

The basic question of large-width asymptotics then concerns the existence of a
limiting process as $H\to\infty$, e.g. whether there exists an $\alpha$-stable
process $(f^{\infty}(x))_{x\in\mathbb{R}^{d}}$ such that

$\bigl{(}f^{H}(x_{1}),f^{H}(x_{2}),\cdots,f^{H}(x_{n})\bigr{)}\stackrel{{\scriptstyle
d}}{{\longrightarrow}}\bigl{(}f^{\infty}(x_{1}),f^{\infty}(x_{2}),\cdots,f^{\infty}(x_{n})\bigr{)}$
(2)

as $H\to\infty$ for all $\\{x_{1},x_{2},\cdots,x_{n}\\}\subset\mathbb{R}^{d}$.

This convergence was already heuristically explored by Neal in [32], and
formally it was established for shallow neural networks in [9] with certain
assumptions (such as essentially sub-linear growth) on the activation function
$\varphi$. Further results for deep neural networks, convolutional
architectures, more general heavy-tailed network weights, and e.g. ReLU-
activations have recently been obtained in [12, 22, 5, 11, 7, 28].

Due to the heavy-tailed nature of the weights, the sample functions of $f^{H}$
are known [30] to have discontinuity-like properties with high probability at
moderate to large widths, even with a bounded and smooth activation function
such as $\tanh$ and even with a single hidden layer as in our model, making
them an attractive choice of a functional prior for Bayesian inverse problems
where the goal is to model rough features or edges [38].

In Figures 1 and 2 we have plotted simulated sample functions for input
dimensions $d\in\\{1,2\\}$ with different stability indices $\alpha$, chosen
such that the characteristic jump-like behaviour (or lack of thereof) can be
observed in $[-1,1]$ and $[-1,1]^{2}$ respectively. The simulations were done
with $(\sigma_{v},\sigma_{b},\sigma_{u},\sigma_{a})=(1,0,5,2)$, $H=10^{5}$ and
$\varphi=x\mapsto x/\max(|x|,1).$ As a side remark and without going into
detail, the expected locations of the “jumps” of the sample functions of
shallow heavy-tailed neural networks is an interesting question as noted in
[30, Section 2.2], and it can be controlled essentially by modifying the
distribution of the input biases $a_{i}$. The main results of this paper
continue to hold with a variety of such modifications.

Figure 1. Sample paths of wide neural networks with one hidden layer on
$[-1,1]$ Figure 2. Sample functions of wide neural networks with one hidden
layer on $[-1,1]^{2}$

###  1.3. Main results and discussion

As our first main result, we establish an infinite-dimensional variant of the
convergence (2). This involves determining a suitable function space
containing the sample functions of both $f^{H}$ and $f^{\infty}$ with full
probability. This function space will then act as an ambient space for the
Bayesian posterior analysis in our second main result.

The analysis is purely theoretical in nature, and we refer to the recent paper
[30] for numerical examples of Bayesian inversion with this kind of priors.
Our two main theorems together can be seen as a sort of _discretization
invariance_ result (see e.g. [33, Section 3] and the references therein) for
wide heavy-tailed neural network priors, in the sense that widening a properly
scaled finite-width neural network will bring both the prior and the resulting
posterior closer to well-defined limit processes. Our results are also
tangentially related to the literature concerned with the construction of
continuous-parameter random fields for Bayesian inversion. We refer to e.g.
[27, 24, 1, 38, 8, 25] for related examples of Besov, Laplace, $\alpha$-stable
and non-standard Gaussian priors.

Our first main result is Theorem 1.1 below. The second main result of this
paper, concerning Bayesian posterior convergence, requires some technical
preparations to be stated in its full generality, so this will be postponed
until Section 3, but we present a preliminary version of it in Remark 1.2
right after Theorem 1.1. This will be followed by Remarks 1.3–1.4 offering
some commentary on the assumptions, parameter ranges and the choice of the
function space in Theorem 1.1. An extension of the main results to deep neural
networks is presented in Section 4.

In order to state the main results, we first introduce a relevant class of
function spaces. Let $\mathcal{U}\subset\mathbb{R}^{d}$ be a domain. The
fractional Sobolev-Slobodeckij space $W^{s,p}(\mathcal{U})$ for $s\in(0,1)$
and $p>d/(d+s)$ is defined as the collection of Lebesgue-measurable functions
$f\colon\mathcal{U}\to\mathbb{R}$ such that

$\|f\|_{W^{s,p}(\mathcal{U})}\mathrel{\mathop{:}}=\|f\|_{L^{p}(\mathcal{U})}+\left(\int_{\mathcal{U}\times\mathcal{U}}\frac{|f(x)-f(y)|^{p}}{|x-y|^{sp+d}}\mathrm{d}x\mathrm{d}y\right)^{1/p}$
(3)

is finite.

######  Theorem 1.1.

Let $\mathcal{U}$ be a bounded domain in $\mathbb{R}^{d}$ with
$C^{\infty}$-smooth boundary (open interval in case $d=1$). Assume that the
activation function $\varphi$ is Hölder-continuous and uniformly sublinearly
increasing, i.e. that there exist $\lambda\in(0,1]$ and $\beta\in[0,1)$ such
that

$|\varphi(x)-\varphi(y)|\leq\begin{cases}c_{\varphi}|x-y|^{\lambda}\qquad&\text{if
}|x-y|<1,\\\ c^{\prime}_{\varphi}|x-y|^{\beta}&\text{if }|x-y|\geq
1\end{cases}$

for some constants $c_{\varphi}$, $c^{\prime}_{\varphi}<\infty$. Assume that
$\alpha\in(d/(d+\lambda),2)$.

Then

  1. (i)

For any integrability exponent $p\in(d/(d+\lambda),\alpha)$ and smoothness
index $s\in(0,\lambda)$ such that $p>d/(d+s)$, there exists a version of the
limit process $(f^{\infty}(x))_{x\in\mathcal{U}}$ in (2) with sample functions
in the fractional Sobolev-Slobodeckij space $W^{s,p}(\mathcal{U})$;

  2. (ii)

Denoting by $\mathcal{L}(f^{H})$, $H\in\mathbb{N}\cup\\{\infty\\}$, the law of
the process $f^{H}$ in part (i), we have the convergence

$\mathcal{L}(f^{H})\stackrel{{\scriptstyle
w}}{{\longrightarrow}}\mathcal{L}(f^{\infty})$

in $\mathcal{P}(W^{s,p}(\mathcal{U}))$ as $H\to\infty$.

We refer to Remark 2.4 (ii) in Section 2 for the precise mathematical meaning
of “process with sample functions in a function space”, as in part (i) above.

######  Remark 1.2.

From the above Theorem we immediately have the following functional result on
Bayesian posterior convergence, in the spirit of [20, Proposition 1]. Let
$\mathcal{G}\colon W^{s,p}(\mathcal{U})\to\mathbb{R}^{M}$ be a continuous
forward operator. This could be a collection of local averages, or more
generally integrations against test functions in $L^{q}(\mathcal{U})$ where

$q=\frac{dp}{(d+s)p-d}>1$

in case $sp<d$, or a vector-valued function constructed from the afore-
mentioned operations and continuous functions on $\mathbb{R}$. Then we
consider the Bayesian inverse problem

$\boldsymbol{u}=\mathcal{G}(f)+\boldsymbol{\varepsilon}_{M},$

where $\boldsymbol{\varepsilon}_{M}$ is an independent $\mathbb{R}^{M}$-valued
noise term (for example multivariate Gaussian). For given observations
$\boldsymbol{u}$, the likelihood $\pi(\boldsymbol{u}\;\big{|}\;\cdot\,)$ is
under suitable mild assumptions a bounded and continuous function on
$W^{s,p}(\mathcal{U})$, and so the posterior with prior $\mathcal{L}(f^{H})$,
$H\in\mathbb{N}\cup\\{\infty\\}$, can be characterized as

$\pi^{H}(F\;\big{|}\;\boldsymbol{u})=\frac{\gamma^{H}(F\;\big{|}\;\boldsymbol{u})}{\gamma^{H}(1\;\big{|}\;\boldsymbol{u})}$

for bounded and continuous test functions $F\colon
W^{s,p}(\mathcal{U})\to\mathbb{R}$, where the unnormalized posterior
$\gamma^{H}(\,\cdot\;\big{|}\;\boldsymbol{u})$ is given by

$\gamma^{H}(F\;\big{|}\;\boldsymbol{u})=\int_{W^{s,p}(\mathcal{U})}F(f)\pi(\boldsymbol{u}\;\big{|}\;f)\mathcal{L}(f^{H})(\mathrm{d}f).$

The integrand above is again under suitable assumptions a bounded and
continuous function on $W^{s,p}(\mathcal{U})$, so we have the convergence

$\pi^{H}(\,\cdot\;\big{|}\;\boldsymbol{u})\stackrel{{\scriptstyle
w}}{{\longrightarrow}}\pi^{\infty}(\,\cdot\;\big{|}\;\boldsymbol{u})$

in $\mathcal{P}(W^{s,p}(\mathcal{U}))$ as $H\to\infty$. We refer to Section 3
for details, and a posterior convergence result (Theorem 3.2) for more general
forward operators.

######  Remark 1.3.

  1. (i)

The Sobolev-Slobodeckij spaces $W^{s,p}(\mathcal{U})$ are Besov-type function
spaces of fractional-order smoothness that arise e.g. in the contexts of trace
theorems for standard integer-order Sobolev spaces and real interpolation
between Sobolev and Lebesgue spaces. We refer to Appendix A for a short survey
of properties of functions with this sort of regularity.

  2. (ii)

The space $W^{s,p}(\mathcal{U})$ is a quasi-Banach space when $p<1$, in the
sense that the triangle inequality holds only up to a multiplicative constant.
The applicability of quasi-Banach function spaces for edge preserving Bayesian
inversion has previously been observed in [37].

  3. (iii)

The spaces $W^{s,p}(\mathcal{U})$ were chosen as the framework for our
analysis because they are reasonably well-known and their definition is easy
to state and understand. However, because the parameter range for $(p,s)$ in
Theorem 1.1 is open-ended, we can replace $W^{s,p}(\mathcal{U})$ in the
statement of the Theorem with a variety of function spaces with similar
smoothness and integrability properties.

For instance, we have [39, Theorem 3.3.1]

$W^{s+\epsilon,p}(\mathcal{U})\subset
B^{s}_{p,q}(\mathcal{U})\quad\text{and}\quad
W^{s+\epsilon,p}(\mathcal{U})\subset F^{s}_{p,q}(\mathcal{U})$

for all $q>d/(d+s)$ and (small enough) $\epsilon>0$ with continuous
embeddings, so the Theorem continues to hold for the Besov space
$B^{s}_{p,q}(\mathcal{U})$ and the Triebel-Lizorkin space
$F^{s}_{p,q}(\mathcal{U})$ with admissible incides. As an example, denoting by
$\mathcal{F}$ the standard Fourier transform on $\mathbb{R}^{d}$, the
fractional Hardy-Sobolev space of functions $f\in
L^{1}_{\mathrm{loc}}(\mathbb{R}^{d})$ satisfying

$\mathcal{F}^{-1}\bigl{[}\xi\mapsto(1+|\xi|^{2})^{s/2}(\mathcal{F}f)(\xi)\bigr{]}\in\begin{cases}L^{p}(\mathbb{R}^{d}),\quad&p>1,\\\
h^{p}(\mathbb{R}^{d}),\quad&p\leq 1\end{cases}$

restricted to $\mathcal{U}$ coincides with the space
$F^{s}_{p,2}(\mathcal{U})$ [39, Theorems 2.3.8 and 2.5.8/1]. For $p\leq 1$,
$h^{p}(\mathbb{R}^{d})$ above stands for the real-variable local Hardy space.

  4. (iv)

The restriction $\beta<1$ precludes the use of the popular ReLU activation
function in the Theorem. This is not a deficiency of the result, but a
reflection of the fact that the question of $\alpha$-stable network
convergence is genuinely different for linearly increasing activation
functions, and it may hold (if at all) under a different scaling regime. In
the context of ReLU activation in particular, this has recently been
investigated in [11].

  5. (v)

In case $d=1$ and $\alpha\lambda>1$, the sample paths of $f^{\infty}$ are in
fact Hölder-continuous and a stronger mode of convergence holds; see Remark
2.2 (iv).

######  Remark 1.4.

A couple further comments on the parameter range in Theorem 1.1.

  1. (i)

The upper bound $\lambda$ for the smoothness index $s$ is essentially sharp.
Starting from the definition of the space $W^{\lambda,p}(\mathcal{U})$, it is
not hard to construct an example of an admissible $\lambda$-Hölder activation
function $\varphi$ that does not belong to $W^{\lambda,p}(\mathbb{R})$
locally, and the sample functions of any $f^{H}$ do not belong to
$W^{s,p}(\mathcal{U})$ with positive probability.

  2. (ii)

The upper bound $\alpha$ for $p$ comes essentially from the application of
Fubini’s theorem, and the fact that $\alpha$-stable random variables are
$p$-integrable precisely for $p<\alpha$. We do not know whether this is the
best possible integrability range for the $s$-smoothness for the sample
functions of $f^{\infty}$ for given $s$. However, in general we do not expect
the permissible range of $ps$ to extend beyond $\alpha\lambda$. For if this
were the case for $(d,\alpha,\lambda)=(1,1,1)$, then the infinite-width limit
of Cauchy neural networks on the real line would have Hölder-continuous sample
paths (see Remark 1.3 (v) above), which empirically seems doubtful.

  3. (iii)

The lower bound $d/(d+\lambda)$ for $p$ (and consequently for $\alpha$)
becomes increasingly restrictive for large $d$, but it always admits the case
of Cauchy distributions with $\alpha=1$.

It is a consequence of the fact that, in order to work with quasi-Banach
function spaces with integrability and smoothness $(p,s)$ measured in terms of
local integrals, we basically need the lower bound $p>d/(d+s)$ to quarantee
local integrability, and this condition also directly shows up in our proofs
in a natural manner.

For $p<d/(d+s)$, it is possible to define Besov-type spaces such as
$W^{s,p}(\mathcal{U})$ in a variety of ways, but the wide unity of different
characterizations in the literature breaks down. For example, the Fourier-
analytical definition of these spaces [39] will in this case contain tempered
distributions that do not admit representations as functions. A more direct
definition like that in Appendix A will result in a space of functions that
does not embed in the space of tempered distributions, complicating the
analysis of smoothness properties considerably [39, Section 2.2.3].

Recently [19] approximation results based on local medians instead of local
integral averages have been obtained for Besov-type spaces for the full range
$p>0$, but this approach does not appear to be easily applicable in our
setting, where Fubini’s theorem for local integrals plays a key role in many
proofs. More advanced results, such as fractional Rellich-Kondrachov-type
compactness theorems, are also not known in this context as far as we are
aware.

###  1.4. Structure of the paper and notation

The paper is structured as follows. In Section 2 we dig into the sample
regularity of the processes $f^{H}$, $H\in\mathbb{N}$, verifying certain
measurability properties (implicit in the statement of Theorem 1.1) and
approximation results in the process, culminating in the proof of Theorem 1.1.

In Section 3 we review some technicalities concerning functional Bayesian
inverse problems, already alluded to in Remark 1.2 above, and state and prove
the second main theorem of this paper, Theorem 3.2.

In Section 4 we extend Theorems 1.1 and 3.2 for arbtirarily deep neural
networks in the infinite-width limit, under the additional assumption that the
activation function $\varphi$ is Lipschitz-continuous (i.e. $\lambda=1$).

In Appendix A we review standard properties of the function spaces
$W^{s,p}(\mathcal{U})$, which are extensively used in many of our proofs.
Appendix B contains the proofs of some auxiliary technical results used in
Sections 2–3.

We end this section by introducing some notation conventions. Denote by
$\mathbb{N}\mathrel{\mathop{:}}=\\{1,2,3,\cdots\\}$ the set of positive
integers, and by $a{:}b$ the discrete interval $\\{a,a+1,\cdots,b-1,b\\}$ for
any $a$, $b\in\mathbb{Z}$ with $a<b$. The standard Lebesgue measure on
$\mathbb{R}^{d}$ is denoted by $m_{d}$, and we use the shorthand $\mathrm{d}x$
for Lebesgue integration with respect to $m_{d}$.

Probabilities and expectations (with respect to probability measures obvious
from context) are denoted by $\mathbb{P}$ and $\mathbb{E}$ respectively. A
random process indexed by a set $\mathcal{X}$, living in a probability space
$\Omega$, will be denoted by both $(f(x))_{x\in\mathcal{X}}$ and
$(f(\omega,x))_{(\omega,x)\in(\Omega,\mathcal{X})}$ depending on the context,
with no room for confusion.

The space of Borel probability measures on a metric space $Z$ is denoted by
$\mathcal{P}(Z)$. The notations

$\stackrel{{\scriptstyle
w}}{{\longrightarrow}}\quad\text{and}\quad\stackrel{{\scriptstyle
d}}{{\longrightarrow}}$

are used to signify weak convergence of measures in $\mathcal{P}(Z)$ and
convergence in distribution of random elements in $Z$ respectively.

We will use the symbols $\lesssim$, $\gtrsim$ and $\approx$ when dealing with
unimportant multiplicative constants. That is, when $f$ and $g$ are non-
negative functions with the same domain, the notations $f\lesssim g$ or
$g\gtrsim f$ mean that there exists a positive constant $c$, independent of
some parameters, so that $f\leq cg$. We will emphasize the parameters with
respect to which $f\lesssim g$ holds if they are not obvious from context. The
notation $f\approx g$ means that $f\lesssim g$ and $g\lesssim f$.

##  2\. Sample regularity and functional convergence

The goal here is to establish regularity results (and conditions for them) for
the sample functions of the limit process in (2). Although we have talked
about “the” limit process, we only have access to the process in terms of
finite-dimensional distributions, and it is a general fact of probability
theory that the fine properties of the sample functions of a continuous-
parameter process are not uniquely determined by its finite-dimensional
distributions.

In order to talk about regularity properties measured in terms of local
integrals, we will reconcile with a version of the limit process that admits
measurable sample functions.

######  Proposition 2.1.

Suppose that the activation function $\varphi$ satisfies the assumptions of
Theorem 1.1 with $\lambda\in(0,1]$ and $\beta\in[0,1)$, and that
$\alpha\in(0,2)$.

  1. (i)

For any $p\in(0,\alpha)$, we have the following uniform modulus of continuity
in probability:

$\sup_{H\in\mathbb{N}\cup\\{\infty\\}}\mathbb{E}\left[|f^{H}(x)-f^{H}(y)|^{p}\right]\leq\begin{cases}C_{p}|x-y|^{\lambda
p}\quad&\text{if }\lambda<1\\\
C_{p}|x-y|^{p}\left(|\log|x-y||+1\right)^{p/\alpha}&\text{if
}\lambda=1\end{cases}$

for all $x$, $y\in\mathbb{R}^{d}$ with $|x-y|\leq 1$, with a constant $C_{p}$
independent of $x$ and $y$.

  2. (ii)

For any $H\in\mathbb{N}\cup\\{\infty\\}$, there exists a version of the
process $f^{H}$ defined on a complete probability space
$(\Omega^{H},\mathcal{S}^{H},\mathbb{P}^{H})$ such that the mapping

$\Omega^{H}\times\mathbb{R}^{d}\owns(\omega,x)\mapsto
f^{H}(\omega,x)\in\mathbb{R}\cup\\{-\infty,\infty\\}$

is jointly measurable with respect to the product $\sigma$-field.

######  Remark 2.2.

  1. (i)

The condition $|x-y|\leq 1$ in part (i) above can be replaced with $|x-y|\leq
R$ for any $R>0$, affecting only the constant $C_{p}$. This obvious from the
proof below.

  2. (ii)

Part (ii) is of course obvious for $H<\infty$, in which case $f^{H}(x)$ is a
continuous function of the input $x$ and a finite number of continuously
distributed parameters.

  3. (iii)

We a priori allow the version of $f^{\infty}$ in part (ii) to take values in
the two-point compactification $\mathbb{R}\cup\\{-\infty,\infty\\}$ of
$\mathbb{R}$ in order to appeal to general results concerning the existence of
measurable versions. This will be refined later in Proposition 2.3.

  4. (iv)

In case $d=1$ and $\lambda\alpha>1$, part (i) together with a general version
of the Kolmogorov-Chentsov continuity theorem (see e.g. [23, Theorem 23.7])
implies that the limit process $f^{\infty}$ is almost surely Hölder continuous
with exponent $r$ for any $r\in(0,\lambda-1/\alpha)$, and the $d$-convergence
$f^{H}\longrightarrow f^{\infty}$ takes place in the space
$C_{\mathbb{R}}(\mathbb{R})$ of continuous functions with the topology
generated by local $\sup$-norms. When $d\geq 2$ we are far from this regime
however.

###### Proof.

(i) We begin by noting that the case $H=\infty$ will follow from the uniform
bound for the finite-width networks. For $j\in\mathbb{N}$, write $\Lambda^{j}$
for the function $x\mapsto\min(|x|,j)$. Then
$(z,z^{\prime})\mapsto\Lambda^{j}(z-z^{\prime})^{p}$ is a bounded and
continuous function on $(\mathbb{R}^{d})^{2}$, so by monotonicity and the
convergence (2),

$\displaystyle\mathbb{E}\left[|f^{\infty}(x)-f^{\infty}(y)|^{p}\right]$
$\displaystyle=\lim_{j\to\infty}\lim_{H\to\infty}\mathbb{E}\left[\Lambda^{j}(f^{H}(x)-f^{H}(y))^{p}\right]$
$\displaystyle\leq\liminf_{H\to\infty}\mathbb{E}\left[|f^{H}(x)-f^{H}(y)|^{p}\right].$

Let then $H<\infty$. In the rest of this proof we write
$\varphi_{i}\mathrel{\mathop{:}}=\varphi(u_{i}^{\intercal}\cdot+a_{i})$ for
$i\in 1{:}H$. Using the freezing lemma in conjunction with the tower property
of conditional expectations and the $\alpha$-stability property of the weights
$v_{i}$, we have

$\displaystyle\mathbb{E}\left[|f^{H}(x)-f^{H}(y)|^{p}\right]$
$\displaystyle=\mathbb{E}\left[\mathbb{E}\left[\left|\sum_{i=1}^{H}\frac{\varphi_{i}(x)-\varphi_{i}(y)}{H^{1/\alpha}}v_{i}\right|^{p}\;\big{|}\;\left\\{\varphi_{i}(x),\,\varphi_{i}(y)\right\\}_{i\in
1{:}H}\right]\right]$
$\displaystyle=\mathbb{E}\left[|v_{1}|^{p}\right]\mathbb{E}\left[\left(\sum_{i=1}^{H}\frac{|\varphi_{i}(x)-\varphi_{i}(y)|^{\alpha}}{H}\right)^{p/\alpha}\right].$

Noting that $\mathbb{E}[|v_{1}|^{p}]\approx\sigma_{v}^{p}\approx 1$ and using
Hölder’s inequality (with exponent $\alpha/p>1$) in the latter expectation, we
get

$\mathbb{E}\left[|f^{H}(x)-f^{H}(y)|^{p}\right]\lesssim\mathbb{E}\left[|\varphi_{1}(x)-\varphi_{1}(y)|^{\alpha}\right]^{p/\alpha},$
(4)

where the right-hand side and the implicit multiplicative constant are
independent of $H$.

Now since
$\varphi_{1}(x)-\varphi_{1}(y)=\varphi(u_{1}^{\intercal}x+a_{1})-\varphi(u_{1}^{\intercal}y+a_{1})$,
the assumptions on the activation $\varphi$ imply

$|\varphi_{1}(x)-\varphi_{1}(y)|\leq\begin{cases}c_{\varphi}|u_{1}^{\intercal}(x-y)|^{\lambda}\qquad&\text{if
}|u_{1}^{\intercal}(x-y)|<1,\\\
c^{\prime}_{\varphi}|u_{1}^{\intercal}(x-y)|^{\beta}&\text{if
}|u_{1}^{\intercal}(x-y)|\geq 1,\end{cases}$

and the right-hand side estimates above are independent of the input bias
$a_{1}$. For fixed $x$, $y\in\mathbb{R}^{d}$, the random variable
$\|x-y\|_{l^{\alpha}}^{-1}u_{1}^{\intercal}(x-y)$ is by assumption symmetric
and $\alpha$-stable with scale parameter $\sigma_{u}$. Write $q$ for its
density function. We have $q(u)\lesssim 1\land|u|^{-\alpha-1}$ for
$u\in\mathbb{R}\setminus\\{0\\}$, with a multiplicative constant depending
only on $\alpha$ and $\sigma_{u}$.

Thus,

$\displaystyle\mathbb{E}\left[|\varphi_{1}(x)-\varphi_{1}(y)|^{\alpha}\boldsymbol{1}\left(\|x-y\|_{l^{\alpha}}^{-1}|u_{1}^{\intercal}(x-y)|\geq\|x-y\|_{l^{\alpha}}^{-1}\right)\right]$
$\displaystyle\qquad\lesssim\|x-y\|_{l^{\alpha}}^{\alpha\beta}\int_{|u|\geq\|x-y\|_{l^{\alpha}}^{-1}}u^{\alpha\beta}u^{-\alpha-1}\mathrm{d}u\lesssim\|x-y\|_{\ell^{\alpha}}^{\alpha}\approx|x-y|^{\alpha}$

with multiplicative constants independent of $x$, $y\in\mathbb{R}^{d}$. On the
other hand,

$\displaystyle\mathbb{E}\left[|\varphi_{1}(x)-\varphi_{1}(y)|^{\alpha}\boldsymbol{1}\left(\|x-y\|_{l^{\alpha}}^{-1}|u_{1}^{\intercal}(x-y)|<\|x-y\|_{l^{\alpha}}^{-1}\right)\right]$
$\displaystyle\qquad\lesssim\|x-y\|_{\ell^{\alpha}}^{\alpha\lambda}\int_{|u|<\|x-y\|_{l^{\alpha}}^{-1}}u^{\alpha\lambda}q(u)\mathrm{d}u\lesssim\begin{cases}|x-y|^{\alpha\lambda}\quad&\text{if
}\lambda<1,\\\ |x-y|^{\alpha}\left(|\log|x-y||+1\right)&\text{if
}\lambda=1,\end{cases}$

again with constants independent of $x$ and $y$.

Overall,

$\mathbb{E}\left[|\varphi_{1}(x)-\varphi_{1}(y)|^{\alpha}\right]\lesssim|x-y|^{\lambda\alpha}\left(|\log|x-y||+1\right),$

where the logarithmic term can be discarded in case $\lambda<1$. Applying this
in (4) yields the desired estimate.

(ii) Part (i) readily implies that the process $f^{H}$ is for every
$H\in\mathbb{N}\cup\\{\infty\\}$ continuous in probability in the usual sense:

$\mathbb{P}\left(|f^{H}(x)-f^{H}(y)|>\varepsilon\right)\leq\frac{C_{p}}{\varepsilon^{p}}\mathbb{E}[|f^{H}(x)-f^{H}(y)|^{p}]\longrightarrow
0$

as $|x-y|\to 0$ for all $\varepsilon>0$. This is a well-known criterion for
the existence of a measurable version of $f^{H}$, possibly taking values in
the two-point compactification of $\mathbb{R}$; see e.g. [14, Chapter IV,
Section 3, Theorem 1 and Remarks 1–2]. ∎

In the sequel, we write $\mathcal{U}$ for a fixed bounded domain in
$\mathbb{R}^{d}$ with $C^{\infty}$-smooth boundary (open interval in case
$d=1$). This boundary regularity condition is imposed mostly for the sake of
technical convenience – we believe that a Lipschitz domain such as the unit
cube would also work, but were unable to find suitable references for the
analysis of quasi-Banach function spaces in this context.

Recall the fractional Sobolev-Slobodeckij space $W^{s,p}(\mathcal{U})$ for
$s\in(0,1)$ and $p>d/(d+s)$, defined as the collection of Lebesgue-measurable
functions $f\colon\mathcal{U}\to\mathbb{R}$ such that the quasinorm (3) in
Section 1.3 is finite (more precisely, $W^{s,p}(\mathcal{U})$ becomes a quasi-
normed space after identifying the functions that agree $m_{d}$-almost
everywhere on $\mathcal{U}$).

Although $\|\cdot\|_{W^{s,p}(\mathcal{U})}$ is not a norm for $p<1$,
$W^{s,p}(\mathcal{U})$ is a separable and complete metric space when endowed
with the metric $(f,g)\mapsto\|f-g\|_{W^{s,p}(\mathcal{U})}^{p\land 1}$. A
Sobolev-type embedding theorem implies that in the range $p>d/(d+s)$,
functions in $W^{s,p}(\mathcal{U})$ are in fact locally integrable, and that
evaluations of local averages are continuous functionals in
$W^{s,p}(\mathcal{U})$. For these and other results concerning these function
spaces, see Proposition A.1 in Appendix A.

We then have following result concerning the expected smoothness energy of the
sample functions of $f^{H}$, including the limiting process.

######  Proposition 2.3.

Suppose that the activation function $\varphi$ satisfies the assumptions of
Theorem 1.1 with $\lambda\in(0,1]$ and $\beta\in[0,1)$, and that
$\alpha\in(d/(d+\lambda),2)$.

  1. (i)

For any $p\in(d/(d+\lambda),\alpha)$ and $s\in(0,\lambda)$ such that
$p>d/(d+s)$,

$\sup_{H\in\mathbb{N}\cup\\{\infty\\}}\mathbb{E}\left[\|f^{H}\|_{W^{s,p}(\mathcal{U})}^{p}\right]<\infty.$

  2. (ii)

For every $H\in\mathbb{N}\cup\\{\infty\\}$, and $p$ and $s$ like in part (i),
an indistinguishable version of the process $f^{H}$ can be realized as a
process with sample functions in $W^{s,p}(\mathcal{U})$, in the sense that
$f^{H}(\omega,\cdot)\in W^{s,p}(\mathcal{U})$ for all $\omega\in\Omega^{H}$,
and $\omega\mapsto f^{H}(\omega,\cdot)$ is an $\mathcal{S}^{H}$-measurable
random element in the complete and separable metric space
$W^{s,p}(\mathcal{U})$.

######  Remark 2.4.

  1. (i)

In part (i) above, the process $f^{H}$ is to be understood as a measurable
version as in part (ii) of Proposition 2.1, restricted to $x\in\mathcal{U}$.
This version could in principle take values in $\\{-\infty,\infty\\}$,
presenting a problem for the definition of the
$W^{s,p}(\mathcal{U})$-quasinorm defined in terms of first-order differences.
However, the first part of the proof below shows that $|f^{H}(x)|<\infty$ for
almost all $x\in\mathcal{U}$ almost surely. With this and the measurability of
the process $f^{H}$ in mind, the statement of part (i) is well-defined, and
Fubini’s theorem implies that the exchanges of the order of integration in the
proof are justified.

  2. (ii)

Regarding part (ii), we stress that from this point on, it is understood that
$f^{H}$ exists both as an $\mathcal{S}^{H}$-measurable pointwise defined
process $(f^{H}(x))_{x\in\mathcal{U}}$, and as an $\mathcal{S}^{H}$-measurable
random function as in the statement of the result. These are a priori two
distinct (but not mutually exclusive) ways for a process to exist, in the
sense that pointwise evaluation of functions is not a strictly well-defined
operation in the function space $W^{s,p}(\mathcal{U})$.

###### Proof.

(i) We first consider the expected $L^{p}$-energy of $f^{H}$ for $H<\infty$
(the uniform estimate for $H<\infty$ can be extended to $H=\infty$ like in the
proof of Proposition 2.1). By Fubini’s theorem and the freezing lemma, we have

$\displaystyle\mathbb{E}\left[\|f^{H}\|_{L^{p}(\mathcal{U})}^{p}\right]$
$\displaystyle=\mathbb{E}\left[\int_{\mathcal{U}}|f^{H}(x)|^{p}\mathrm{d}x\right]=\int_{\mathcal{U}}\mathbb{E}\left[|f^{H}(x)|^{p}\right]\mathrm{d}x$
$\displaystyle=\int_{\mathcal{U}}\mathbb{E}[|v_{1}|^{p}]\mathbb{E}\left[\left(\sum_{i=1}^{H}\frac{|\varphi(u^{\intercal}_{i}x+a_{i})|^{\alpha}}{H}+\left(\frac{\sigma_{b}}{\sigma_{v}}\right)^{\alpha}\right)^{p/\alpha}\right]\mathrm{d}x$
$\displaystyle\lesssim\int_{\mathcal{U}}\mathbb{E}\left[|\varphi(u_{1}^{\intercal}x+a_{1})|^{\alpha}\right]^{p/\alpha}\mathrm{d}x+1$

as in the proof of Proposition 2.1. The upper bound above is independent of
$H$, and the integral over $\mathcal{U}$ is finite because of the sublinearity
condition on $\varphi$. Namely, $u_{1}^{\intercal}x+a_{1}$ has by assumption a
symmetric $\alpha$-stable distribution with scale parameter
$(\sigma_{u}^{\alpha}\|x\|_{\ell^{\alpha}}^{\alpha}+\sigma_{a}^{\alpha})^{1/\alpha}$,
and $|\varphi(y)|\leq c^{\prime\prime}_{\varphi}(1+|y|^{\beta})$ for some
constant $c^{\prime\prime}_{\varphi}$. Thus,

$\mathbb{E}\left[|\varphi(u_{1}^{\intercal}x+a_{1})|^{\alpha}\right]\lesssim\mathbb{E}\left[1+|u_{1}^{\intercal}x+a_{1}|^{\alpha\beta}\right]\lesssim
1+(\sigma_{u}^{\alpha}\|x\|_{\ell^{\alpha}}^{\alpha}+\sigma_{a}^{\alpha})^{\beta}$

for some implicit multiplicative constants independent of $x$. In fact, the
latter quantity can be bounded independently of $x$ since the domain
$\mathcal{U}$ is bounded by assumption.

Since $f^{H}\in L^{p}(\mathcal{U})$ almost surely, we obviously have
$|f^{H}(x)|<\infty$ for almost all $x\in\mathcal{U}$ almost surely. Thus we
can consider homogeneous smoothness part of (3), which we denote by
$\|f^{H}(x)\|_{\dot{W}^{s,p}(\mathcal{U})}$. We have, again by Fubini’s
theorem,

$\displaystyle\mathbb{E}\left[\|f^{H}(x)\|_{\dot{W}^{s,p}(\mathcal{U})}^{p}\right]$
$\displaystyle=\mathbb{E}\left[\int_{\mathcal{U}\times\mathcal{U}}\frac{|f^{H}(x)-f^{H}(y)|^{p}}{|x-y|^{sp+d}}\mathrm{d}x\mathrm{d}y\right]$
$\displaystyle=\int_{\mathcal{U}\times\mathcal{U}}|x-y|^{-sp-d}\mathbb{E}\left[|f^{H}(x)-f^{H}(y)|^{p}\right]\mathrm{d}x\mathrm{d}y.$

Applying the modulus of continuity estimate in Proposition 2.1 (i) here yields

$\mathbb{E}\left[\|f^{H}(x)\|_{\dot{W}^{s,p}(\mathcal{U})}^{p}\right]\lesssim\int_{\mathcal{U}\times\mathcal{U}}|x-y|^{(\lambda-s)p-d}\left|\log|x-y|\right|^{p/\alpha}\mathrm{d}x\mathrm{d}y,$

where the upper bound is independent of $H$ (and finite because
$\lambda-s>0$).

(ii) By part (i) and the completeness of the probability space
$(\Omega^{H},\mathcal{S}^{H},\mathbb{P}^{H})$ (see part (ii) in Proposition
2.1), we can redefine $f^{H}(\omega,\cdot)\equiv 0$ for the events $\omega$
such that $\|f(\omega,\cdot)\|_{W^{s,p}(\mathcal{U})}$ is $\infty$ or
undefined (in the sense of $|f^{H}(\omega,x)-f^{H}(\omega,y)|$ being undefined
in case $f^{H}(\omega,z)\in\\{-\infty,\infty\\}$ for $z$ in a positive-measure
subset of $\mathcal{U}$), the totality of which by part (i) has zero
$\mathbb{P}^{H}$-measure.

It then remains to show that under the mapping $\omega\mapsto
f^{H}(\omega,\cdot)$, the preimages of closed balls in $W^{s,p}(\mathcal{U})$
belong to $\mathcal{S}^{H}$. Under the properties of the process $f^{H}$
accumulated thus far, this result is not particular to the neural networks
under consideration, and is most easily proven using additional results on the
topology of $W^{s,p}(\mathcal{U})$, so we will postpone the proof until
Appendix B. ∎

With the above result in mind, we can introduce the next major ingredient for
the proof of Theorem 1.1.

######  Proposition 2.5.

Let $p$ and $s$ be like in Theorem 1.1. Write $\mathcal{L}(f^{H})$ for the law
of the process $f^{H}$ on $W^{s,p}(\mathcal{U})$ (see part (ii) of Proposition
2.3). Then the set

$\left\\{\mathcal{L}(f^{H})\,:\,H\in\mathbb{N}\right\\}$

is relatively compact in the space $\mathcal{P}(W^{s,p}(\mathcal{U}))$.

###### Proof.

We can obviously find $p^{\prime}<p$ and $s^{\prime}\in(s,\lambda)$ so that
$p^{\prime}>d/(d+s^{\prime})$ and $s^{\prime}-d/p^{\prime}<s-d/p$. Then a
Rellich-Kondrachov-type embedding theorem (see e.g. Proposition A.1 (iv) in
Appendix A, where the roles of $(s,p)$ and $(s^{\prime},p^{\prime})$ are
reversed) implies that we have

$W^{s^{\prime},p^{\prime}}(\mathcal{U})\subset W^{s,p}(\mathcal{U})$

with a compact embedding. In particular

$K_{C}\mathrel{\mathop{:}}=\\{f\in
W^{s,p}(\mathcal{U})\,:\,\|f\|_{W^{s^{\prime},p^{\prime}}(\mathcal{U})}\leq
C\\}$

is a compact set in $W^{s,p}(\mathcal{U})$ for all $C>0$, and by Proposition
2.3 (i), we have

$\sup_{H\in\mathbb{N}}\mathbb{P}\left(f^{H}\in W^{s,p}(\mathcal{U})\setminus
K_{C}\right)\leq
C^{-p^{\prime}}\sup_{H\in\mathbb{N}}\mathbb{E}\left[\|f^{H}\|_{W^{s^{\prime},p^{\prime}}(\mathcal{U})}^{p^{\prime}}\right]\longrightarrow
0.$

as $C\to\infty$. Prohorov’s theorem [3, Theorem 5.1] thus implies the stated
compactness. ∎

For the sake of stating the next results, we introduce the following notation
for local averages. Write

$f_{A}\mathrel{\mathop{:}}=\frac{1}{m_{\mathcal{U}}(A)}\int_{A}f(x)\mathrm{d}x$

for all $f\colon\mathcal{U}\to\mathbb{R}$ and $A\subset\mathcal{U}$ such that
the right-hand side above is well-defined (in particular, it is for $f\in
L^{1}(\mathcal{U})$ and measurable $A\subset\mathcal{U}$ with $m_{d}(A)>0$).

######  Proposition 2.6.

Under the hypotheses of Theorem 1.1, the family of processes
$(f^{H})_{H\in\mathbb{N}}$ converges to $f^{\infty}$ in terms of local
averages, in the sense that

$\left((f^{H})_{B_{1}},\cdots,(f^{H})_{B_{N}}\right)\stackrel{{\scriptstyle
d}}{{\longrightarrow}}\left((f^{\infty})_{B_{1}},\cdots,(f^{\infty})_{B_{N}}\right)$
(5)

as $H\to\infty$ for all finite collections $(B_{i})_{1\leq i\leq N}$ of balls
centered in $\mathcal{U}$.

For the proof, we need the following uniform convergence result for discrete
approximation of local integrals. Its proof is given in Appendix B.

######  Lemma 2.7.

Let $\alpha$, $\beta$, $\lambda$ and $p$ be as in Proposition 2.3. Then for
any measurable subset $A$ of $\mathcal{U}$ with nonzero measure, and any
$n\in\mathbb{N}$, there exist points
$(x^{n}_{j})_{j=1}^{\kappa(n)}\subset\mathcal{U}$ and scalars
$(\theta^{n}_{j})_{j=1}^{\kappa(n)}\subset\mathbb{R}_{+}$ independent of $H$
such that

$\lim_{n\to\infty}\mathbb{E}\left[\Biggl{|}(f^{H})_{A}-\sum_{j=1}^{\kappa(n)}\theta^{n}_{j}f^{H}(x^{n}_{j})\Biggr{|}^{p}\right]=0$

uniformly with respect to $H$.

###### Proof of Proposition 2.6.

Write $\mathcal{B}$ for the collection of balls in (5), and
$(f^{H})_{\mathcal{B}}$ for the vector on the left-hand side of (5). It
suffices to show that for arbitrary scalars
$\nu\mathrel{\mathop{:}}=(\nu_{i})_{i=1}^{N}\subset\mathbb{R}^{N}$ and an
arbitrary bounded Lipschitz-continuous function
$g\colon\mathbb{R}\to\mathbb{R}$, it holds that

$\lim_{H\to\infty}\mathbb{E}\left[g\left(\nu^{\intercal}(f^{H})_{\mathcal{B}}\right)\right]=\mathbb{E}\left[g\left(\nu^{\intercal}(f^{\infty})_{\mathcal{B}}\right)\right].$
(6)

For all $i\in\\{1,2,\cdots,N\\}$, write
$X^{n}_{i}\mathrel{\mathop{:}}=(x^{n}_{j,i})_{j=1}^{\kappa(n,i)}$ for the
collection of points given by Proposition B.4 for
$A\mathrel{\mathop{:}}=B_{i}$, and $\theta^{n}_{i}$ for the collection
(vector) of scalars given by the same Proposition. Here and in the rest of
this proof, we have abused notation by writing $f^{H}(X^{n}_{i})$ for the
vector obtained by applying $f^{H}$ pointwise in $X^{n}_{i}$.

We can then estimate

$\displaystyle\left|\mathbb{E}\left[g\left(\nu^{\intercal}(f^{H})_{\mathcal{B}}\right)\right]-\mathbb{E}\left[g\left(\nu^{\intercal}(f^{\infty})_{\mathcal{B}}\right)\right]\right|$
$\displaystyle\leq\;$
$\displaystyle\mathbb{E}\left[\left|g\left(\nu^{\intercal}(f^{H})_{\mathcal{B}}\right)-g\left(\sum_{i=1}^{N}\nu_{i}(\theta^{n}_{i})^{\intercal}f^{H}(X^{n}_{i})\right)\right|\right]$
(7)
$\displaystyle\qquad+\mathbb{E}\left[\left|g\left(\nu^{\intercal}(f^{\infty})_{\mathcal{B}}\right)-g\left(\sum_{i=1}^{N}\nu_{i}(\theta^{n}_{i})^{\intercal}f^{\infty}(X^{n}_{i})\right)\right|\right]$
(8)
$\displaystyle\qquad+\left|\mathbb{E}\left[g\left(\sum_{i=1}^{N}\nu_{i}(\theta^{n}_{i})^{\intercal}f^{H}(X^{n}_{i})\right)\right]-\mathbb{E}\left[g\left(\sum_{i=1}^{N}\nu_{i}(\theta^{n}_{i})^{\intercal}f^{\infty}(X^{n}_{i})\right)\right]\right|.$
(9)

By the assumptions on $g$, we have $|g(x)-g(y)|\leq c_{g}|x-y|^{p\land 1}$ for
all $x$, $y\in\mathbb{R}$ and some constant $c_{g}>0$. Thus for any given
$\epsilon>0$, we can use subadditivity and Proposition B.4 to find an
approximation level $n$ such that the terms (7) and (8) are $<\epsilon/2$ for
all $H\in\mathbb{N}\cup\\{\infty\\}$. For this $n$, the term (9) converges to
zero as $H\to\infty$ by the convergence (2). This establishes (6). ∎

We are now ready to present the proof of the main result of this paper.

###### Proof of Theorem 1.1.

Part (i) of the result is contained in Proposition 2.3 (see also Remark 2.4
(ii)).

Concerning part (ii), we recall from Proposition 2.6 that the local averages
of $f^{H}$ convergence in distribution to those of $f^{\infty}$ as
$H\to\infty$. By Proposition B.4 in Appendix B, the collection of local
averages is a separating family for $\mathcal{P}(W^{s,p}(\mathcal{U}))$, so it
follows that any $d$-convergent subsequence of
$(\mathcal{L}(f^{H}))_{H\in\mathbb{N}}$ must converge to
$\mathcal{L}(f^{\infty})$. This together with Proposition 2.5 then implies
that the sequence $(\mathcal{L}(f^{H}))_{H\in\mathbb{N}}$ must itself converge
to $\mathcal{L}(f^{\infty})$. ∎

##  3\. Bayesian posterior analysis

We return to the question of Bayesian posterior convergence, which we briefly
discussed in Remark 1.2 in the context of forward operators that are
continuous on $W^{s,p}(\mathcal{U})$. In this section we will refine this
result so as to accommodate forward operators based on pointwise evaluations,
which are not continuous and not even strictly well-defined operators in
$W^{s,p}(\mathcal{U})$.

Recall the basic setting, i.e. the Bayesian inverse problem

$\boldsymbol{u}=\mathcal{G}(f)+\boldsymbol{\varepsilon}_{M},$ (10)

where $\mathcal{G}\colon W^{s,p}(\mathcal{U})\to\mathbb{R}^{M}$ is the forward
operator and $\boldsymbol{\varepsilon}_{M}$ is an independent
$\mathbb{R}^{M}$-valued noise term (such as component-wise i.i.d Gaussian).
Let $\pi\in\mathcal{P}(W^{s,p}(\mathcal{U}))$ be a prior distribution for $f$.
Under fairly general conditions (see e.g. [36, Theorem 6.31] or [26, Theorem
3.3]), the posterior distribution
$\pi(\,\cdot\;\big{|}\;\boldsymbol{u})\in\mathcal{P}(W^{s,p}(\mathcal{U}))$
for given observations $\boldsymbol{u}\in\mathbb{R}^{M}$ exists as an
absolutely continuous measure with respect to the prior $\pi$, with Radon-
Nikodym derivative

$\frac{\mathrm{d}\pi(\,\cdot\;\big{|}\;\boldsymbol{u})}{\mathrm{d}\pi}(f)\propto\pi(\boldsymbol{u}\;\big{|}\;f)\mathrel{\mathop{:}}=\rho_{M}\bigl{(}\boldsymbol{u}-\mathcal{G}(f)\bigr{)},$
(11)

where $\rho_{M}$ is the density function of $\boldsymbol{\varepsilon}_{M}$,
and the implicit proportionality constant is independent of $f$.

As the observation $\boldsymbol{u}$ has a continuous distribution on
$\mathbb{R}^{M}$ in our model, the posterior distribution as a conditional
probability measure is strictly speaking well-defined only for almost all
$\boldsymbol{u}\in\mathbb{R}^{M}$. In our use case (Theorem 3.2 below), the
density on the right-hand side of (11) is a bounded and continuous function of
$\boldsymbol{u}$, so we may view $\pi(\,\cdot\;\big{|}\;\boldsymbol{u})$
defined by (11) as the canonical posterior for _all_
$\boldsymbol{u}\in\mathbb{R}^{M}$. We thus have
$\pi(\,\cdot\;\big{|}\;\boldsymbol{u^{\prime}})\longrightarrow\pi(\,\cdot\;\big{|}\;\boldsymbol{u})$
in distribution (and e.g. setwise and in the total variation metric etc.) as
$\boldsymbol{u^{\prime}}\to\boldsymbol{u}$. We refer to [26, Section 1] for a
thorough discussion and literature review concerning the question of existence
and uniqueness of posterior distributions in the functional setting.

We will consider forward operators of the form

$\mathcal{G}(f)\mathrel{\mathop{:}}=G\left(f(x_{1}),f(x_{2}),\cdots,f(x_{N}),f\right)$
(12)

for some fixed $(x_{i})_{i=1}^{N}\subset\mathcal{U}$, where
$G\colon\mathbb{R}^{N}\times W^{s,p}(\mathcal{U})\to\mathbb{R}^{M}$ is
continuous. In order to make sense of the Bayesian posterior with this type of
forward operation, $\mathcal{G}$ should be a measurable function on
$W^{s,p}(\mathcal{U})$, so strictly speaking we should modify (12) by
replacing the evaluations $f(x_{i})$ with e.g.

$\liminf_{n\to\infty}f_{B_{\mathcal{U}}(x_{i},1/n)}$

(or zero in case this quantity is not finite), which as a function of $f$ is a
pointwise limes inferior of continuous functions on $W^{s,p}(\mathcal{U})$ and
thus Borel-measurable. This modification is immaterial for the processes
$f^{H}$, $H\in\mathbb{N}\cup\\{\infty\\}$, as demonstrated by the following
Lemma.

######  Lemma 3.1.

Let the assumptions on the activation function $\varphi$ be as in Theorem 1.1,
and $p\in(d/(d+\lambda),\alpha)$. Then for all
$H\in\mathbb{N}\cup\\{\infty\\}$ and $x\in\mathcal{U}$,

$\lim_{r\to 0^{+}}(f^{H})_{B_{\mathcal{U}}(x,r)}=f^{H}(x)$

with full probability, and

$\lim_{r\to
0^{+}}\mathbb{E}\left[\left(|f^{H}-f^{H}(x)|_{B_{\mathcal{U}}(x,r)}\right)^{p}\right]=0$

uniformly in $H\in\mathbb{N}\cup\\{\infty\\}$.

The proof shares similarities with that of the “converse” result Lemma 2.7,
and is presented in Appendix B.

######  Theorem 3.2.

Let $\varphi$ and $(p,s)$ be as in Theorem 1.1, and let
$\boldsymbol{u}\in\mathbb{R}^{N}$. Assume that the density function $\rho_{M}$
of the noise term $\boldsymbol{\varepsilon}_{M}$ in (10) is bounded and
Hölder-continuous with some exponent $\lambda_{\rho}\in(0,1]$, and that its
support is the entirety of $\mathbb{R}^{M}$. Assume that the function
$G\colon\mathbb{R}^{N}\times W^{s,p}(\mathcal{U})\to\mathbb{R}^{M}$ in (12)
continuous, and uniformly Hölder-continuous with some exponent
$\lambda_{G}\in(0,1]$ in the $\mathbb{R}^{N}$-component.

Then, for the inverse problem (10) with forward operator (12) and prior
$\pi^{H}\mathrel{\mathop{:}}=\mathcal{L}(f^{H})$, we have the posterior
convergence

$\pi^{H}(\,\cdot\;\big{|}\;\boldsymbol{u})\stackrel{{\scriptstyle
w}}{{\longrightarrow}}\pi^{\infty}(\,\cdot\;\big{|}\;\boldsymbol{u})$

in $\mathcal{P}(W^{s,p}(\mathcal{U}))$ as $H\to\infty$.

The assumptions on the noise density $\rho_{M}$ are fairly general, and they
admit e.g. Gaussian and component-wise i.i.d symmetric $\alpha$-stable
vectors.

###### Proof.

We first briefly note that the posterior distribution, given observations
$\boldsymbol{u}\in\mathbb{R}^{M}$, exists in the sense of (11), and that this
is where the assumption about $\rho_{M}$ being bounded and supported on the
entirety of $\mathbb{R}^{M}$ is needed. Namely, these assumptions quarantee
that the posterior distribution is well-defined in the sense that

$0<\int_{W^{s,p}(\mathcal{U})}\rho_{M}\bigl{(}\boldsymbol{u}-\mathcal{G}(f)\bigr{)}\,\pi^{H}(\mathrm{d}f)<\infty$
(13)

for all $\boldsymbol{u}\in\mathbb{R}^{M}$ and
$H\in\mathbb{N}\cup\\{\infty\\}$. This existence result can be found in [26,
Section 5.1] under the condition that the function

$W^{s,p}(\mathcal{U})\times\mathbb{R}^{M}\owns(f,\boldsymbol{u})\mapsto\rho_{M}\bigl{(}\boldsymbol{u}-\mathcal{G}(f)\bigr{)}\in[0,\infty)$

is jointly Borel-measurable, as is the case under our assumptions.

We introduce the auxiliary notation

$\pi_{y}(\boldsymbol{u}\;\big{|}\;f)\mathrel{\mathop{:}}=\rho_{M}\bigl{(}\boldsymbol{u}-G(y,f)\bigr{)}$

for $y\in\mathbb{R}^{N}$ and $f\in W^{s,p}(\mathcal{U})$. It follows from the
assumptions that the function

$\mathbb{R}^{N}\owns y\mapsto\pi_{y}(\boldsymbol{u}\;\big{|}\;f)\in[0,\infty)$
(14)

is bounded and Hölder-continuous with exponent
$\lambda^{\prime}\mathrel{\mathop{:}}=\lambda_{\rho}\lambda_{G}\in(0,1]$
uniformly in $f$.

Recall from (11) that the unnormalized posterior
$\gamma^{H}(\,\cdot\;\big{|}\;\boldsymbol{u})$ can for
$H\in\mathbb{N}\cup\\{\infty\\}$ and bounded and continuous test functions
$F\colon W^{s,p}(\mathcal{U})\to\mathbb{R}$ be expressed by

$\gamma^{H}(F\;\big{|}\;\boldsymbol{u})=\int_{W^{s,p}(\mathcal{U})}F(f)\pi(\boldsymbol{u}\;\big{|}\;f)\mathcal{L}(f^{H})(\mathrm{d}f),$

where

$\pi(\boldsymbol{u}\;\big{|}\;f)=\pi_{(f(x_{1}),\cdots,f(x_{N}))}(\boldsymbol{u}\;\big{|}\;f)$

is the likelihood associated with the forward operator (12). For $r>0$, we
write

$\pi^{r}(\boldsymbol{u}\;\big{|}\;f)\mathrel{\mathop{:}}=\pi_{(f_{B_{\mathcal{U}}(x_{1},r)},\cdots,f_{B_{\mathcal{U}}(x_{N},r)})}(\boldsymbol{u}\;\big{|}\;f).$

With fixed $r$, this is by assumption a bounded and continuous function of
$f\in W^{s,p}(\mathcal{U})$.

By the continuity properties of the function (14), we have

$\displaystyle\left|\gamma^{H}(F\;\big{|}\;\boldsymbol{u})-\gamma^{\infty}(F\;\big{|}\;\boldsymbol{u})\right|$
$\displaystyle\leq$
$\displaystyle\left|\int_{W^{s,p}(\mathcal{U})}F(f)\left(\pi(\boldsymbol{u}\;\big{|}\;f)-\pi^{r}(\boldsymbol{u}\;\big{|}\;f)\right)\mathcal{L}(f^{H})(\mathrm{d}f)\right|$
$\displaystyle\qquad+\left|\int_{W^{s,p}(\mathcal{U})}F(f)\left(\pi(\boldsymbol{u}\;\big{|}\;f)-\pi^{r}(\boldsymbol{u}\;\big{|}\;f)\right)\mathcal{L}(f^{\infty})(\mathrm{d}f)\right|$
$\displaystyle\qquad+\left|\int_{W^{s,p}(\mathcal{U})}F(f)\pi^{r}(\boldsymbol{u}\;\big{|}\;f)\left\\{\mathcal{L}(f^{H})(\mathrm{d}f)-\mathcal{L}(f^{\infty})(\mathrm{d}f)\right\\}\right|$
$\displaystyle\lesssim$
$\displaystyle\sum_{i=1}^{N}\mathbb{E}\left[\left(|f^{H}-f^{H}(x_{i})|_{B_{\mathcal{U}}(x_{i},r)}\right)^{p\land\lambda^{\prime}}\right]$
(15)
$\displaystyle\qquad+\sum_{i=1}^{N}\mathbb{E}\left[\left(|f^{\infty}-f^{\infty}(x_{i})|_{B_{\mathcal{U}}(x_{i},r)}\right)^{p\land\lambda^{\prime}}\right]$
(16)
$\displaystyle\qquad+\left|\int_{W^{s,p}(\mathcal{U})}F(f)\pi^{r}(\boldsymbol{u}\;\big{|}\;f)\left\\{\mathcal{L}(f^{H})(\mathrm{d}f)-\mathcal{L}(f^{\infty})(\mathrm{d}f)\right\\}\right|.$
(17)

with some multiplicative constant independent of $H$ and $r>0$. Given
$\epsilon>0$, we can by Lemma 3.1 take $r$ so that the terms (15) and (16) are
$<\epsilon/2$ for all $H$. For this $r$, the term (17) converges to zero as
$H\to\infty$ by Theorem 1.1.

This shows that

$\lim_{H\to\infty}\gamma^{H}(F\;\big{|}\;\boldsymbol{u})=\gamma^{\infty}(F\;\big{|}\;\boldsymbol{u})$

for all bounded and continuous $F\colon W^{s,p}(\mathcal{U})\to\mathbb{R}$,
and by (13), the same convergence therefore holds for the normalized
posteriors $\pi^{H}(\,\cdot\;\big{|}\;\boldsymbol{u})$. ∎

##  4\. Deeper architectures

A very natural question is whether the results in the previous sections,
particularly the ones concerning functional convergence and Bayesian posterior
consistency (Theorems 1.1 and 3.2), generalize to perceptrons with more than
one hidden layer. Here we establish this generalization for the case of
Lipschitz-continuous activation functions, which allows for a convenient
recursion of the sample regularity analysis through the hidden layers. Similar
results could perhaps be attained for more general activations through a more
sophisticated analysis of the distributions of the neurons at each hidden
layer, but this is beyond the scope of this paper.

To set up the result, denote by $L\in\\{2,3,4,\cdots\\}$ the fixed number of
hidden layers in our neural network, by
$H_{0}\mathrel{\mathop{:}}=d\in\mathbb{N}$ the input dimension, by
$H_{L+1}\mathrel{\mathop{:}}=1$ the output dimension and by
$\boldsymbol{H}\mathrel{\mathop{:}}=(H_{1},H_{2},\cdots,H_{L})\in\mathbb{N}^{L}$
the widths of the hidden layers. Let $\boldsymbol{U}\in\mathbb{R}^{H_{1}\times
H_{0}}$ be a random matrix with i.i.d components distributed as
$\mathcal{S}_{\alpha}(\sigma_{u})$, for $\ell\in
1{:}L\mathrel{\mathop{:}}=\\{1,2,\cdots,L\\}$ let
$\boldsymbol{V}^{(\ell)}\in\mathbb{R}^{H_{\ell+1}\times H_{\ell}}$ be a random
matrix with i.i.d components distributed as
$\mathcal{S}_{\alpha}(\sigma_{v})$, and for $\ell\in 1{:}(L+1)$ let
$\boldsymbol{a}^{(\ell)}\in\mathbb{R}^{H_{\ell}}$ be a random vector with
i.i.d components distributed as $\mathcal{S}_{\alpha}(\sigma_{a})$.

Our neural network is then defined sequentially through the layers as follows.
Let

$f^{(1)}(x)\mathrel{\mathop{:}}=\left(f_{1}^{(1)}(x),\cdots,f_{H_{1}}^{(1)}(x)\right)\mathrel{\mathop{:}}=\boldsymbol{U}x+\boldsymbol{a}^{(1)}\in\mathbb{R}^{H_{1}},$

and for $\ell\in 1{:}L$, let

$f^{(\ell+1)}(x)\mathrel{\mathop{:}}=\left(f_{1}^{(\ell+1)}(x),\cdots,f_{H_{\ell+1}}^{(\ell+1)}(x)\right)\mathrel{\mathop{:}}=H_{\ell}^{-1/\alpha}\boldsymbol{V}^{(\ell)}\varphi\left(f^{(\ell)}(x)\right)+\boldsymbol{a}^{(\ell+1)}\in\mathbb{R}^{H_{\ell+1}},$

where $\varphi(\cdot)\in\mathbb{R}^{H_{\ell}}$ is interpreted as an
elementwise action on a vector. We denote the ultimate scalar output as

$f^{\boldsymbol{H}}(x)\mathrel{\mathop{:}}=f^{(L+1)}(x).$

The finite-dimensional convergence of these neural networks under some mild
assumptions (satisfied by the conditions of Theorem 4.1 below) has been
obtained in [12, Theorem 2] and [22, Theorem 3.1 and Section 4], in the sense
that there exists an $\alpha$-stable process
$(f^{\infty}(x))_{x\in\mathbb{R}^{d}}$ such that

$\bigl{(}f^{\boldsymbol{H}}(x_{1}),f^{\boldsymbol{H}}(x_{2}),\cdots,f^{\boldsymbol{H}}(x_{n})\bigr{)}\stackrel{{\scriptstyle
d}}{{\longrightarrow}}\bigl{(}f^{\infty}(x_{1}),f^{\infty}(x_{2}),\cdots,f^{\infty}(x_{n})\bigr{)}$
(18)

as $\min(H_{1},H_{2},\cdots,H_{L})\to\infty$ for all
$\\{x_{1},x_{2},\cdots,x_{n}\\}\subset\mathbb{R}^{d}$. The cited results also
describe a recursive formula for the finite-dimensional spectral measures of
$f^{\infty}$, which we omit here.

Let us note that the convergence $\eqref{eq:deep-convergence-finite-dims}$
also holds under some additional hypotheses if the stability index $\alpha$
were to vary between the hidden layers. This however complicates both the
analysis and the permissible parameter range for $\beta$ (see the discussion
in [22, Section 4]), so we will only consider uniformly $\alpha$-stably
weighted networks in the following result.

######  Theorem 4.1.

Let $\mathcal{U}$ and $\varphi$ be as in Theorem 1.1, with $\varphi$ Hölder-
continuous (i.e. $\lambda=1$), and let $\alpha\in(d/(d+1),2)$. Then the
statements of Theorems 1.1 and 3.2 continue to hold with $f^{\boldsymbol{H}}$
in place of $f^{H}$, and limiting process $f^{\infty}$ for
$\min(H_{1},\cdots,H_{L})\longrightarrow\infty$ described by (18).

###### Proof.

Key to much of our analysis in the shallow regime is the uniform modulus of
continuity estimate in Proposition 2.1 (i). We will establish a weaker version
of this estimate for deep networks with Lipschitz activation, which
nevertheless is qualitatively close enough in a certain sense.

We begin by noting that for each $i\in 1{:}H_{2}$ and all $x$,
$y\in\mathcal{U}$,

$f^{(2)}_{i}(x)-f^{(2)}_{i}(y)$

is distributed as $f^{H}(x)-f^{H}(y)$, where $f^{H}$ is the shallow network
considered in Section 2. Thus, by Proposition 2.1 (i), we have

$\mathbb{E}\left[|f^{(2)}_{i}(x)-f^{(2)}_{i}(y)|^{p}\right]\lesssim|x-y|^{p-\epsilon}$
(19)

for all $x$, $y\in\mathcal{U}$, $p\in(0,\alpha)$ and $\epsilon\in(0,p)$ close
to $0$ (to be specified below). The implicit constant here depends (among
other things) on $\alpha$, $p$, $\epsilon$ and $\mathcal{U}$, but not on
$(x,y)$ nor the widths $\boldsymbol{H}$.

Let then $c\in(\beta,1)$ be close to $1$ (again, to be specified below). By
the continuity and growth criteria on $\varphi$, it is immediate that

$|\varphi(x)-\varphi(y)|\lesssim|x-y|^{c}$ (20)

for all $x$, $y\in\mathbb{R}$, with an implicit constant that does not depend
on $(x,y)$. With this estimate, we can for any $i\in 1{:}H_{3}$ carry out the
following estimates using the freezing lemma, as in the proof of Proposition
2.1:

$\displaystyle\mathbb{E}\left[|f^{(3)}_{i}(x)-f^{(3)}_{i}(y)|^{p}\right]$
$\displaystyle=\,$
$\displaystyle\mathbb{E}\left[\left|\sum_{j=1}^{H_{2}}\frac{\varphi\bigl{(}f^{(2)}_{j}(x)\bigr{)}-\varphi\bigl{(}f^{(2)}_{j}(y)\bigr{)}}{H_{2}^{1/\alpha}}\boldsymbol{V}^{(2)}_{i,j}\right|^{p}\right]$
$\displaystyle=\,$
$\displaystyle\mathbb{E}\left[|\boldsymbol{V}^{(2)}_{1,1}|^{p}\right]\mathbb{E}\left[\left(\frac{1}{H_{2}}\sum_{j=1}^{H_{2}}\bigl{|}\varphi\bigl{(}f^{(2)}_{j}(x)\bigr{)}-\varphi\bigl{(}f^{(2)}_{j}(y)\bigr{)}\bigr{|}^{\alpha}\right)^{p/\alpha}\right]$
$\displaystyle\leq\,$
$\displaystyle\mathbb{E}\left[|\boldsymbol{V}^{(2)}_{1,1}|^{p}\right]\left(\frac{1}{H_{2}}\sum_{j=1}^{H_{2}}\mathbb{E}\left[\bigl{|}\varphi\bigl{(}f^{(2)}_{j}(x)\bigr{)}-\varphi\bigl{(}f^{(2)}_{j}(y)\bigr{)}\bigr{|}^{\alpha}\right]\right)^{p/\alpha}$
$\displaystyle\lesssim\,$
$\displaystyle\left(\frac{1}{H_{2}}\sum_{j=1}^{H_{2}}\mathbb{E}\left[|f^{(2)}_{j}(x)-f^{(2)}_{j}(y)|^{c\alpha}\right]\right)^{p/\alpha}\lesssim|x-y|^{cp-\epsilon(p/\alpha)},$

where in the second-to-last inequality we used (20), and in the last one we
used (19) with $c\alpha<\alpha$ in place of $p$ – here $\epsilon$ has to be
small enough so that $\epsilon<c\alpha$. Since $c\in(\beta,1)$ was arbitrary,
and $p/\alpha<1$, we can take $c$ so that $cp-\epsilon(p/\alpha)>p-\epsilon$,
yielding

$\mathbb{E}\left[|f^{(3)}_{i}(x)-f^{(3)}_{i}(y)|^{p}\right]\lesssim|x-y|^{p-\epsilon}$

for all $i\in 1{:}H_{3}$, which is of the same form as (19) with $f^{(3)}$ in
place of $f^{(2)}$.

We can induct on this idea in an obvious manner, culminating in

$\mathbb{E}\left[|f^{\boldsymbol{H}}(x)-f^{\boldsymbol{H}}(y)|^{p}\right]\lesssim|x-y|^{p-\epsilon}$
(21)

for all $x$, $y\in\mathcal{U}$ and any sufficiently small $\epsilon>0$. To be
clear, the implicit multiplicative constant depends among other things on $L$,
$\alpha$, $p$, $\epsilon$ and $\mathcal{U}$, but _not_ on $(x,y)$ nor the
widths $\boldsymbol{H}$. In particular, the uniformity with respect to
$\boldsymbol{H}$ allows us to extend (21) to the limit process $f^{\infty}$ as
in the proof of Proposition 2.1.

The idea is then to use (21) instead of Proposition 2.1 (i) with a
sufficiently small $\epsilon>0$ whenever needed – this will always be possible
due to the open-ended nature of the parameter ranges appearing in our results.
In fact, we have done this kind of parameter nudging even in the shallow
regime in the proofs of Lemmas 2.7 and 3.1 in the Lipschitz case, in order to
avoid dealing with the logarithmic term in Proposition 2.1 (i).

Let us sketch a proof of the counterpart of Proposition 2.3 (i) as an example.

First, a measurable version of each $f^{\boldsymbol{H}}$,
$\boldsymbol{H}\in\mathbb{N}^{L}\cup\\{\infty\\}$, exists as in Proposition
2.1 (ii), which justifies the use of Fubini’s theorem in what follows. Let
$s\in(0,1)$ and $p\in(d/(d+s),\alpha)$. That

$\sup_{\boldsymbol{H}\in\mathbb{N}^{L}}\mathbb{E}\left[\left\|f^{\boldsymbol{H}}\right\|^{p}_{L^{p}(\mathcal{U})}\right]=\sup_{\boldsymbol{H}\in\mathbb{N}^{L}}\int_{\mathcal{U}}\mathbb{E}\left[\left|f^{\boldsymbol{H}}(x)\right|^{p}\right]\mathrm{d}x<\infty$

can be proven by inductively estimating the integrand through the hidden
layers as above, using the sub-polynomial growth of $\varphi$, and this
uniform estimate extends to $f^{\infty}$ in the same way as (21).

Then, for $\boldsymbol{H}\in\mathbb{N}^{L}\cup\\{\infty\\}$, we can use (21)
to estimate

$\displaystyle\mathbb{E}\left[\left\|f^{\boldsymbol{H}}\right\|^{p}_{\dot{W}^{s,p}(\mathcal{U})}\right]$
$\displaystyle=\int_{\mathcal{U}\times\mathcal{U}}|x-y|^{-sp-d}\mathbb{E}\left[|f^{\boldsymbol{H}}(x)-f^{\boldsymbol{H}}(y)|^{p}\right]\mathrm{d}x\mathrm{d}y$
$\displaystyle\lesssim\int_{\mathcal{U}\times\mathcal{U}}|x-y|^{-sp-d+p-\epsilon}\mathrm{d}x\mathrm{d}y,$

with an implicit constant independent of $\boldsymbol{H}$. Since $s<1$, the
parameter $\epsilon>0$ can be taken smaller than $p-sp$, so that the latter
double integral is finite. ∎

##  Appendix A Properties of Sobolev-Slobodeckij spaces

Here we collect some useful properties of the Sobolev-Slobodeckij spaces
$W^{s,p}$, along with references.

Let $\mathcal{U}\subset\mathbb{R}^{d}$ be a bounded domain in the Euclidean
space with $C^{\infty}$-smooth boundary. It is not hard to see that the domain
$\mathcal{U}$ is then also _Ahlfors $d$-regular_, in the sense that

$m_{d}\left(B_{\mathcal{U}}(x,r)\right)\approx r^{d}$ (22)

for all $x\in\mathcal{U}$ and $r\in(0,\mathop{\mathrm{diam}}(\mathcal{U})]$
with some multiplicative constants indepedent of $(x,r)$, where $m_{d}$ and
$B_{\mathcal{U}}$ stand for the Lebesgue $d$-measure and the Euclidean ball
restricted to $\mathcal{U}$.

For $s\in(0,1)$ and $p>d/(d+s)$, we then recall the homogeneous function space
$\dot{W}^{s,p}(\mathcal{U})$, defined in terms of the quasinorm (modulo
additive constants)

$\|f\|_{\dot{W}^{s,p}(\mathcal{U})}\mathrel{\mathop{:}}=\left(\int_{\mathcal{U}\times\mathcal{U}}\frac{|f(x)-f(y)|^{p}}{|x-y|^{sp+d}}\mathrm{d}x\mathrm{d}y\right)^{1/p}.$

The space $W^{s,p}(\mathcal{U})$ is then defined as
$L^{p}(\mathcal{U})\cap\dot{W}^{s,p}(\mathcal{U})$, with proper quasinorm

$\|f\|_{W^{s,p}(\mathcal{U})}\mathrel{\mathop{:}}=\|f\|_{L^{p}(\mathcal{U})}+\|f\|_{\dot{W}^{s,p}(\mathcal{U})}.$
(23)

######  Proposition A.1.

Let $s\in(0,1)$ and $p>d/(d+s)$. We have the following structural properties
for the space $W^{s,p}(\mathcal{U})$.

  1. (i)

The space $W^{s,p}(\mathcal{U})$ coincides with the space of distributions
$B^{s}_{p,p}(\mathcal{U})$ considered in [39, Chapter 3] and the space
$\mathcal{F}^{s}_{p,p}(\mathcal{U})$ of locally integrable functions
considered in [34, Definition 5.1], all with equivalent quasinorms.

  2. (ii)

For $d/(d+s)<p<1$, the the functions in $W^{s,p}(\mathcal{U})$ are integrable,
and the term $\|f\|_{L^{p}(\mathcal{U})}$ in (23) can be replaced with
$\|f\|_{L^{1}(\mathcal{U})}$, resulting in the same space with equivalent
quasinorms.

  3. (iii)

For $p^{\prime}>p$ and $s^{\prime}\in(0,s)$ such that
$s^{\prime}-d/p^{\prime}=s-d/p$, we have

$W^{s,p}(\mathcal{U})\subset W^{s^{\prime},p^{\prime}}(\mathcal{U})$ (24)

with a continuous embedding (it automatically holds that
$p^{\prime}>d/(s^{\prime}+d)$).

  4. (iv)

For $p^{\prime}>p$ and $s^{\prime}\in(0,s)$ such that
$s^{\prime}-d/p^{\prime}>s-d/p$, the embedding (24) is compact (it
automatically holds that $p^{\prime}>d/(s^{\prime}+d)$).

  5. (v)

Endowed with

$(f,g)\mapsto\|f-g\|_{W^{s,p}(\mathcal{U})}^{p\land 1},$

$W^{s,p}(\mathcal{U})$ is a complete and separable metric space.

The classic monograph [39, Chapter 3] contains many of the results mentioned
above. In the interest of accessibility, we present here a more precise list
of sources for each of the claims in the above Proposition.

  1. (i)

That $W^{s,p}(\mathcal{U})$ coincides with the distribution space
$B^{s}_{p,p}(\mathcal{U})$ is contained in [39, Proposition 3.4.2]. Concerning
the space $\mathcal{F}^{s}_{p,p}(\mathcal{U})$, we first note that
$W^{s,p}(\mathcal{U})$ by definition agrees with the space
$\dot{B}^{s}_{p,p}(\mathcal{U})\cap L^{p}(\mathcal{U})$, where the homogeneous
space $\dot{B}^{s}_{p,p}(\mathcal{U})$ is as defined in [16, Definition 1.1].
Then by [16, Theorem 1.2 and Definition 3.1], $\dot{B}^{s}_{p,p}(\mathcal{U})$
coincides with a certain function space $\dot{M}^{s}_{p,p}(\mathcal{U})$
defined in terms of fractional gradients. By [4, Proposition 3.1], this space
in turn coincides with the homogeneous space
$\dot{\mathcal{F}}^{s}_{p,p}(\mathcal{U})$, and by [34, Proposition 5.2] we
have $\dot{\mathcal{F}}^{s}_{p,p}(\mathcal{U})\cap
L^{p}(\mathcal{U})=\mathcal{F}^{s}_{p,p}(\mathcal{U})$ with equivalent
quasinorms.

  2. (ii)

This result for the distribution space $B^{s}_{p,p}(\mathcal{U})$ is contained
in [40, Theorem 1.118 (ii)] (see also the discussion in [40, Remark 1.117]).

  3. (iii–iv)

These embeddings can be found in [40, Remark 1.96 and Theorem 1.97] for the
distribution spaces $B^{s}_{p,p}(\mathcal{U})$ and
$B^{s^{\prime}}_{p^{\prime},p^{\prime}}(\mathcal{U})$.

  4. (v)

That the function of $(f,g)$ in question is in fact a metric is an easy
consequence of the sub-additivity of the function $x\mapsto|x|^{r\land 1}$ for
any $r>0$. Completeness and separability are more or less straightforward
consequences of the corresponding properties of the distribution space
$B^{s}_{p,p}(\mathbb{R}^{d})$ on the Euclidean space, which are inherited by
the localized space $B^{s}_{p,p}(\mathcal{U})$; see [40, Remark 1.96].

##  Appendix B Auxiliary results

Recall from Section 2 the notation

$f_{A}\mathrel{\mathop{:}}=\frac{1}{m_{d}(A)}\int_{A}f(x)\mathrm{d}x$

for all $f\colon\mathcal{U}\to\mathbb{R}$ and $A\subset\mathcal{U}$ such that
the right-hand side above is well-defined (in particular, it is for $f\in
L^{1}(\mathcal{U})$ and measurable $A\subset\mathcal{U}$ with $m_{d}(A)>0$).

Before stating the first results of this section, we recall a family of
discrete convolution operators on $L^{1}(\mathcal{U})$ which have been studied
in the context of first-order Besov spaces in e.g. [4, 34, 19], and which we
will apply extensively below. For this purpose, we fix for every
$n\in\mathbb{N}$ a maximal set of points
$(x^{n}_{i})_{i=1}^{\kappa(n)}\subset\mathcal{U}$ such that
$|x^{n}_{i}-x^{n}_{j}|\geq 2^{-n-1}$ for all $i\neq j$ (it can be shown that
$\kappa(n)\approx 2^{dn}$ with implicit constants independent of $n$). Let

$\mathcal{B}^{n}\mathrel{\mathop{:}}=\left(B^{n}_{1},\cdots,B^{n}_{\kappa(n)}\right)\mathrel{\mathop{:}}=\left(B_{\mathcal{U}}(x^{n}_{1},2^{-n}),\cdots,B_{\mathcal{U}}(x^{n}_{\kappa(n)},2^{-n})\right),$

and for $f\in L^{1}(\mathcal{U})$

$\pi_{\mathcal{B}^{n}}(f)\mathrel{\mathop{:}}=\bigl{(}f_{B^{n}_{1}},\cdots,f_{B^{n}_{\kappa(n)}}\bigr{)}\in\mathbb{R}^{\kappa(n)}.$

It is easy to see that the balls in $\mathcal{B}^{n}$ have uniformly (with
respect to $n$) bounded overlap, and that there exists a family of Lipschitz
continuous functions $(\psi^{n}_{i})_{i=1}^{\kappa(n)}$ on $\mathcal{U}$ such
that $\mathop{\mathrm{supp}}\psi^{n}_{i}\subset B^{n}_{i}$, $\psi^{n}_{i}\geq
0$,

$\sum_{i=1}^{\kappa(n)}\psi^{n}_{i}\equiv 1$

and that the Lipschitz constants of the $\psi^{n}_{i}$ are bounded by a
constant (indepdent of $n$) times $2^{n}$. We refer to e.g. [34, Definition
3.4] and the discussion therein.

######  Definition B.1.

  1. (i)

For $n\in\mathbb{N}$ and $v\in\mathbb{R}^{\kappa(n)}$, define the function
$T^{n}v\colon\mathbb{R}^{\kappa(n)}\to L^{1}(\mathcal{U})$ by

$T^{n}v=\sum_{i=1}^{\kappa(n)}v(i)\psi^{n}_{i}.$

  2. (ii)

For $n\in\mathbb{N}$ and $f\in L^{1}(\mathcal{U})$, define

$T^{n}f\mathrel{\mathop{:}}=T^{n}\bigl{(}\pi_{\mathcal{B}^{n}}(f)\bigr{)}.$

######  Proposition B.2.

Let $s\in(0,1)$ and $p>d/(d+s)$.

  1. (i)

For each $n$, the operator $T^{n}$ is continuous from $\mathbb{R}^{\kappa(n)}$
to $W^{s,p}(\mathcal{U})$.

  2. (ii)

The convergence

$\lim_{n\to\infty}T^{n}f=f\quad$

holds in the metric of $W^{s,p}(\mathcal{U})$ for all $f\in
W^{s,p}(\mathcal{U})$.

###### Proof.

Part (i) is a simple computation using the Lipschitz continuity and
boundedness properties of the $\psi^{n}_{i}$’s. By linearity, it suffices to
check that

$\|T^{n}v\|_{W^{s,p}(\mathcal{U})}^{p\land 1}\lesssim\kappa(n)2^{sn/(p\lor
1)}\max_{1\leq i\leq\kappa(n)}|v(i)|^{p\land 1}$

with an implicit multiplicative constant independent of $v$ and $n$. Part (ii)
can be found in [34, Proposition 5.4]. ∎

For the crucial measurability property in Proposition 2.3 (ii), we need the
following description of closed balls in $W^{s,p}(\mathcal{U})$, which gives
an almost complete description of the topology of $W^{s,p}(\mathcal{U})$ in
terms of the operators $\pi_{\mathcal{B}^{n}}$ introduced above. Note that by
Proposition A.1 (ii) in Appendix A, each $\pi_{\mathcal{B}^{n}}$ is continuous
from $W^{s,p}(\mathcal{U})$ to $\mathbb{R}^{\kappa(n)}$.

######  Lemma B.3.

Let $s\in(0,1)$ and $p>d/(d+s)$. For $f\in W^{s,p}(\mathcal{U})$,
$\varepsilon>0$ and $n\in\mathbb{N}$, write $V^{n}(f,\varepsilon)$ for the
closed set

$\left\\{v\in\mathbb{R}^{\kappa(n)}\,:\,\|T^{n}v-T^{n}f\|_{W^{s,p}(\mathcal{U})}^{p\land
1}\leq\varepsilon\right\\}\subset\mathbb{R}^{\kappa(n)}.$

Then the closed $W^{s,p}(\mathcal{U})$-ball centered at $f\in
W^{s,p}(\mathcal{U})$ with radius $\varepsilon$ can be expressed as

$\overline{B}_{W^{s,p}(\mathcal{U})}(f,\varepsilon)=\bigcap_{m=1}^{\infty}\bigcup_{n=m}^{\infty}\pi_{\mathcal{B}^{n}}^{-1}\left(V^{n}(f,\varepsilon_{m})\right),$
(25)

where $(\varepsilon_{m})_{m\in\mathbb{N}}$ is any strictly decreasing sequence
converging to $\varepsilon$.

That the sets $V^{n}(f,\varepsilon)$ in the statement are closed follows from
Proposition B.2 (i) above.

###### Proof.

We have by Proposition B.2 (ii) that

$\displaystyle\|g-f\|_{W^{s,p}(\mathcal{U})}^{p\land 1}$
$\displaystyle=\lim_{n\to\infty}\|T^{n}g-T^{n}f\|_{W^{s,p}(\mathcal{U})}^{p\land
1}$

for all $g$, $f\in W^{s,p}(\mathcal{U})$. The inclusions between the left-hand
and right-hand sides of (25) in both directions follow straightforwardly from
this. ∎

###### Proof of Proposition 2.3 (ii), continuation.

Write $F\colon\Omega^{H}\to W^{s,p}(\mathcal{U})$ for the mapping
$\omega\mapsto f^{H}(\omega,\cdot)$. By Lemma B.3, the closed
$W^{s,p}(\mathcal{U})$-ball at $f$ with radius $\varepsilon$ can be written as
(25), where the $V^{n}(f,\varepsilon_{m})$ are closed subsets as in the
statement of the Lemma. Thus,

$\displaystyle F^{-1}\overline{B}_{W^{s,p}(\mathcal{U})}(f,\varepsilon)$
$\displaystyle=\bigcap_{m=1}^{\infty}\bigcup_{n=m}^{\infty}F^{-1}\pi_{\mathcal{B}^{n}}^{-1}\left(V^{n}(f,\varepsilon_{m})\right)$
$\displaystyle=\bigcap_{m=1}^{\infty}\bigcup_{n=m}^{\infty}\left\\{\omega\in\Omega^{H}\,:\,\pi_{\mathcal{B}^{n}}\left(f^{H}(\omega,\cdot)\right)\in
V^{n}(f,\varepsilon_{m})\right\\}.$

Recall the measurability of the process $f^{H}$, established in Proposition
2.1 (ii). The measurability part of Fubini’s theorem, i.e. that integrals with
respect to one coordinate are measurable with respect to the other, can be
localized to integrals over the balls in $\mathcal{B}^{n}$, meaning
$\omega\mapsto\pi_{\mathcal{B}^{n}}(f^{H}(\omega,\cdot))\in\mathbb{R}^{\kappa(n)}$
is an $\mathcal{S}^{H}$-measurable random vector for all $n$. Each set in the
inner union above is therefore in $\mathcal{S}^{H}$, and hence so is
$F^{-1}\overline{B}_{W^{s,p}(\mathcal{U})}(f,\varepsilon)$. ∎

A key ingredient of the proof of the main result, similar in flavor to the
measurability proof above, is the following observation, which essentially
implies that after establishing the relative compactness of the laws of the
processes $f^{H}$, it suffices to check the convergence of local averages to
those of $f^{\infty}$.

######  Proposition B.4.

Let $s\in(0,1)$ and $p>d/(d+s)$. Then the collection $\Pi$ of local averages,
i.e. the mappings from $W^{s,p}(\mathcal{U})$ to finite-dimensional Euclidean
spaces of the form

$\pi_{B_{1},\cdots,B_{n}}\mathrel{\mathop{:}}=f\mapsto\left(f_{B_{1}},\cdots,f_{B_{n}}\right)\in\mathbb{R}^{n}$

for some ordered collection $(B_{i})_{1\leq i\leq n}$ of balls in
$\mathcal{U}$, separates $\mathcal{P}(W^{s,p}(\mathcal{U}))$, in the sense
that if $P$ and $Q$ are Borel probability measures on $W^{s,p}(\mathcal{U})$
such that $P\circ\pi^{-1}=Q\circ\pi^{-1}$ for all $\pi\in\Pi$, then $P=Q$.

###### Proof.

It suffices to show (see [3, Example 1.3] and the discussion therein) that the
class of sets

$\left\\{\pi^{-1}_{\mathcal{B}}(V)\,:\,\mathcal{B}\text{ finite collection of
balls},\,V\subset\mathbb{R}^{\\#\mathcal{B}}\text{ closed}\right\\}$ (26)

is a $\pi$-system, and that the $\sigma$-field generated by this class
coincides with the Borel $\sigma$-field of $W^{s,p}(\mathcal{U})$.

The $\pi$-system property is immediate from the definition. The mappings in
$\Pi$ are continuous on $W^{s,p}(\mathcal{U})$ by Proposition A.1 (ii), so it
suffices to show that every closed ball in $W^{s,p}(\mathcal{U})$ is contained
in the $\sigma$-field generated by (26). This follows directly from Lemma B.3.
∎

We now turn to the proofs of Lemmas 2.7 and 3.1. For this purpose, we
introduce another approximation scheme, this time a probabilistic one for the
processes $f^{H}$.

For all $n\in\mathbb{N}$, write $Q^{n}_{j}$, $1\leq j\leq\kappa(n)$, for the
collection of the standard semi-open dyadic cubes in $\mathbb{R}^{d}$ with
side length $2^{-n}$ that intersect $\mathcal{U}$, and pick arbitrary
$x^{n}_{j}\in Q^{n}_{j}\cap\mathcal{U}$. Note that $\kappa(n)$ is not
necessarily the same number as the corresponding one we used in the definition
of the operators $T^{n}$ earlier in this appendix, but we still have
$\kappa(n)\approx 2^{dn}$ with multiplicative constants independent of $n$.
For $H\in\mathbb{N}\cup\\{\infty\\}$ and $n\in\mathbb{N}$, write $f^{H,n}$ for
the process on $\mathcal{U}$ defined as

$f^{H,n}(x)=\sum_{j=1}^{\kappa(n)}f^{H}(x^{n}_{j})\boldsymbol{1}\left(x\in
Q^{n}_{j}\right).$

We recall (Proposition 2.1 (ii)) that each process $f^{H}$ was realized as a
measurable version on the product space $\Omega^{H}\times\mathcal{U}$. Write
$\mu^{H}$ for the product measure on this product space. For any given
$p\in(0,\alpha)$ we have by Proposition 2.1 (i) and Fubini’s theorem

$\displaystyle\int_{\Omega^{H}\times\mathcal{U}}\sum_{n\in\mathbb{N}}\left|f^{H}(\omega,x)-f^{H,n}(\omega,x)\right|^{p}\mu^{H}\left(\mathrm{d}(\omega,x)\right)$
$\displaystyle=$
$\displaystyle\sum_{n\in\mathbb{N}}\int_{\mathcal{U}}\mathbb{E}\left[\left|f^{H}(x)-f^{H,n}(x)\right|^{p}\right]\mathrm{d}x<\infty,$

which means that the series inside the left-hand side integral converges
$\mu^{H}$-almost surely, and so

$\lim_{n\to\infty}f^{H,n}(\omega,x)=f^{H}(\omega,x)$ (27)

for $\mu^{H}$-almost every $(\omega,x)$. It is then not hard to see that for
$\mathbb{P}^{H}$-almost every $\omega$, (27) holds for $m_{d}$-almost every
$x\in\mathcal{U}$.

###### Proof of Lemma 2.7.

We will show that

$\lim_{n\to\infty}\mathbb{E}\left[\left(\int_{\mathcal{U}}\left|f^{H}(x)-f^{H,n}(x)\right|\mathrm{d}x\right)^{p}\right]=0$
(28)

for all $p\in(d/(d+\lambda),\alpha)$, uniformly in
$H\in\mathbb{N}\cup\\{\infty\\}$, so the statement of the lemma follows by
taking $x^{n}_{j}$ as in the definition of the process $f^{H,n}$ above and
$\theta^{n}_{j}=m_{d}(Q^{n}_{j}\cap A)/m_{d}(A)$ (possibly discarding the
point-scalar pairs $(x^{n}_{j},\theta^{n}_{j})$ where $\theta^{n}_{j}=0)$.

For $p\geq 1$, (28) is an immediate consequence of Hölder’s inequality,
Fubini’s theorem and Proposition 2.1 (i). Below we will consider the case
$p\in(d/(d+\lambda),1\land\alpha)$, where instead of Hölder’s inequality we
approximate the integral in (28) with a countable sum and use the
subadditivity of the function $x\mapsto|x|^{p}$. For this purpose, we write
$\lambda^{\prime}\mathrel{\mathop{:}}=\lambda$ in case $\lambda<1$, and if
$\lambda=1$, pick $\lambda^{\prime}\in(0,1)$ so that
$p>d/(d+\lambda^{\prime})$.

We begin by estimating

$\displaystyle\int_{\mathcal{U}}|f^{H,n+1}(x)-f^{H,n}(x)|\mathrm{d}x$
$\displaystyle=\sum_{j=1}^{\kappa(n)}\sum_{Q^{n+1}_{k}\subset
Q^{n}_{j}}|f^{H}(x^{n+1}_{k})-f^{H}(x^{n}_{j})|m_{d}(Q^{n+1}_{k}\cap\mathcal{U})$
$\displaystyle\leq 2^{-nd}\sum_{j=1}^{\kappa(n)}\sum_{Q^{n+1}_{k}\subset
Q^{n}_{j}}|f^{H}(x^{n+1}_{k})-f^{H}(x^{n}_{j})|,$

and since there is an uniformly bounded number of cubes $Q^{n+1}_{k}$
contained in each $Q^{n}_{j}$, we can use subadditivity and the modulus of
continuity estimate in Proposition 2.1 (i) we get

$\mathbb{E}\left[\left(\int_{\mathcal{U}}\left|f^{H,n+1}(x)-f^{H,n}(x)\right|\mathrm{d}x\right)^{p}\right]\lesssim
2^{n(d-(d+\lambda^{\prime})p)}$

with a multiplicative constant independent of $n$ and $H$. Since
$n(d-(d+\lambda^{\prime})p)<0$ by assumption, $n+1$ in the integral above can
by a telescoping argument be replaced by any $m>n$, and in particular

$\sup_{m>n}\mathbb{E}\left[\left(\int_{\mathcal{U}}\left|f^{H,m}(x)-f^{H,n}(x)\right|\mathrm{d}x\right)^{p}\right]\lesssim
2^{n(d-(d+\lambda^{\prime})p)}$ (29)

with again a multiplicative constant independent of $n$ and $H$.

We now recall that the pointwise convergence (27) holds
$\mathbb{P}^{H}$-almost surely for $m_{d}$-almost every $x\in\mathcal{U}$.
Thus, by Fatou’s lemma,

$\displaystyle\mathbb{E}\left[\left(\int_{\mathcal{U}}\left|f^{H}(x)-f^{H,n}(x)\right|\mathrm{d}x\right)^{p}\right]$
$\displaystyle\leq\liminf_{m\to\infty}\mathbb{E}\left[\left(\int_{\mathcal{U}}\left|f^{H,m}(x)-f^{H,n}(x)\right|\mathrm{d}x\right)^{p}\right]$
$\displaystyle\leq\sup_{m>n}\mathbb{E}\left[\left(\int_{\mathcal{U}}\left|f^{H,m}(x)-f^{H,n}(x)\right|\mathrm{d}x\right)^{p}\right],$

and the latter quantity converges to zero as $n\to\infty$ uniformly in $H$, as
noted in (29). ∎

###### Proof of Lemma 3.1.

Let $\lambda^{\prime}\mathrel{\mathop{:}}=\lambda$ in case $\lambda<1$, and if
$\lambda=1$, pick $\lambda^{\prime}\in(0,1)$ so that
$p>d/(d+\lambda^{\prime})$. We will start by establishing

$\mathbb{E}\left[\left(|f^{H}-f^{H}(x)|_{B_{\mathcal{U}}(x,r)}\right)^{p}\right]\lesssim
r^{\lambda^{\prime}p}$ (30)

for $r\in(0,1/2)$, with a multiplicative constant independent of $r$ and $H$.
As in the proof of Lemma 2.7 above, this is for $p\geq 1$ an immediate
consequence of Hölder’s inequality and Proposition 2.1 (i). Below we will
establish this estimate for $p\in(d/(d+\lambda^{\prime}),1\land\alpha)$.

Recall the discrete approximation processes $f^{H,n}$, $n\in\mathbb{N}$,
introduced before the proof of Lemma 2.7. We begin by considering $n$ such
that $2^{-n-1}\leq r<2^{-n}$. It is not hard to see that there are finitely
many cubes $Q^{n}_{j}$ that intersect $B_{\mathcal{U}}(x,r)$, and the number
of such cubes has an upper bound independent of $x$ and $r$. Thus, by
Proposition 2.1 (i),

$\displaystyle\mathbb{E}\left[\left(|f^{H,n}-f^{H}(x)|_{B_{\mathcal{U}}(x,r)}\right)^{p}\right]\lesssim\mathbb{E}\left[\left(\sum_{Q^{n}_{j}\cap
B_{\mathcal{U}}(x,r)\neq\emptyset}|f^{H}(x^{n}_{j})-f^{H}(x)|\right)^{p}\right]\lesssim
2^{-n\lambda^{\prime}p}.$

Next, for $m\geq n$, we note that there are $\lesssim 2^{(m-n)d}$ cubes
$Q^{m}_{j}$ intersecting $B_{\mathcal{U}}(x,r)$ with a multiplicative constant
independent of $x$ and $r$, so

$\displaystyle\mathbb{E}\left[\left(|f^{H,m+1}-f^{H,m}(x)|_{B_{\mathcal{U}}(x,r)}\right)^{p}\right]$
$\displaystyle\qquad\lesssim\mathbb{E}\left[\left(2^{nd}\sum_{Q^{m}_{j}\cap
B_{\mathcal{U}}(x,r)\neq\emptyset}2^{-md}\sum_{Q^{m+1}_{i}\subset
Q^{m}_{j}}|f^{H}(x^{m}_{j})-f^{H}(x^{m+1}_{i})|\right)^{p}\right]$
$\displaystyle\qquad\lesssim
2^{(n-m)dp}2^{(m-n)d}2^{-m\lambda^{\prime}p}=2^{n(dp-d)}2^{m(d-(d+\lambda^{\prime})p)},$

where $d-(d+\lambda^{\prime})p<0$ by assumption.

Recall that $f^{H,n}(\omega,x)\to f^{H}(\omega,x)$ as $n\to\infty$ almost
everywhere in the product space $\Omega^{H}\times\mathcal{U}$ with respect to
the product measure (see the discussion preceding the proof of Lemma 2.7).
Combining the estimates established above we have

$\displaystyle\mathbb{E}\left[\left(|f^{H}-f^{H}(x)|_{B_{\mathcal{U}}(x,r)}\right)^{p}\right]$
$\displaystyle\qquad\leq\mathbb{E}\left[\left(|f^{H,n}-f^{H}(x)|_{B_{\mathcal{U}}(x,r)}\right)^{p}\right]+\sum_{m\geq
n}\mathbb{E}\left[\left(|f^{H,m+1}-f^{H,m}(x)|_{B_{\mathcal{U}}(x,r)}\right)^{p}\right]$
$\displaystyle\qquad\lesssim 2^{-n\lambda^{\prime}p}+2^{n(dp-d)}\sum_{m\geq
n}2^{m(d-(d+\lambda^{\prime})p)}\approx 2^{-n\lambda^{\prime}p}\approx
r^{\lambda^{\prime}p},$

which is the estimate (30).

The first claim of the Lemma will follow if we can show

$\lim_{r\to 0^{+}}|f^{H}-f^{H}(x)|_{B_{\mathcal{U}}(x,r)}=0$ (31)

almost surely. By the Ahlfors regularity condition (22), for any $r>0$ we have

$|f^{H}-f^{H}(x)|_{B_{\mathcal{U}}(x,2^{-n-1})}\lesssim|f^{H}-f^{H}(x)|_{B_{\mathcal{U}}(x,r)}\lesssim|f^{H}-f^{H}(x)|_{B_{\mathcal{U}}(x,2^{-n})}$

for $n\in\mathbb{Z}$ such that $2^{-n-1}\leq r<2^{-n}$, so it suffices to show
(31) for $r=2^{-n}\to 0$. By (30), we have

$\mathbb{E}\left[\left(\sum_{n\geq
1}|f^{H}-f^{H}(x)|_{B_{\mathcal{U}}(x,2^{-n})}\right)^{p\land
1}\right]\lesssim\sum_{n\geq 1}2^{-n\lambda^{\prime}(p\land 1)}<\infty,$

which means that the series inside the left-hand side expectation has to
converge almost surely. ∎

## Acknowledgments

The author would like to thank Lassi Roininen and Petteri Piiroinen for
helpful discussions concerning the subject. The work was supported by the
Research Council of Finland through the Flagship of Advanced Mathematics for
Sensing, Imaging and Modelling, and the Centre of Excellence of Inverse
Modelling and Imaging (decision numbers 359183 and 353095).

## References

  * [1] S. Agapiou and S. Wang,  Laplace priors and spatial inhomogeneity in Bayesian inverse problems,  _Bernoulli_ , 30 (2024), 878-910, DOI:10.3150/22-BEJ1563.
  * [2] A. Basteri and D. Trevisan,  Quantitative Gaussian approximation of randomly initialized deep neural networks,  2023, arXiv:2203.07379.
  * [3] P. Billingsley,  _Convergence of probability measures_ ,  2nd edition, John Wiley & Sons, Inc., New York, 1999.
  * [4] M. Bonk, E. Saksman and T. Soto,  Triebel-Lizorkin spaces on metric spaces via hyperbolic fillings,  _Indiana Univ. Math. J._ , 67 (2018), 1625-1663, DOI:10.1512/iumj.2018.67.7282.
  * [5] A. Bordino, S. Favaro and S. Fortini,  Infinitely wide limits for deep stable neural networks: sub-linear, linear and super-linear activation functions,  _Transactions on Machine Learning Research_ , 2022, https://openreview.net/forum?id=A5tIluhDW6.
  * [6] D. Bracale, S. Favaro, S. Fortini and S. Peluchetti,  Large-width functional asymptotics for deep Gaussian neural networks,  _International Conference on Learning Representation_ , 2021, https://openreview.net/forum?id=0aW6lYOYB7d.
  * [7] D. Bracale, S. Favaro, S. Fortini and S. Peluchetti,  Infinite-channel deep stable convolutional neural networks,  2022, arXiv:2102.03739.
  * [8] N. K. Chada, S. Lasanen and L. Roininen,  Posterior convergence analysis of $\alpha$-stable sheets,  2019, arXiv:1907.03086.
  * [9] R. Der and D. Lee,  Beyond Gaussian processes: On the distributions of infinite networks,  _Advances in Neural Information Processing Systems_ , volume 18, MIT Press, 2005, 275-282.
  * [10] R. Eldan, D. Mikulincer and T. Schramm,  Non-asymptotic approximations of neural networks by Gaussian processes,  _Proceedings of 34th Conference on Learning Theory_ , volume 134 of _Proceedings of Machine Learning Research_ , 2021, 1754-1775, arXiv:2102.08668.
  * [11] S. Favaro, S. Fortini and S. Peluchetti,  Large-width asymptotics for ReLU neural networks with $\alpha$-stable initializations,  2022, arXiv:2206.08065.
  * [12] S. Favaro, S. Fortini and S. Peluchetti,  Deep stable neural networks: Large-width asymptotics and convergence rates,  _Bernoulli_ , 29 (2023), 2574-2597, DOI:10.3150/22-BEJ1553.
  * [13] S. Favaro, B. Hanin, D. Marinucci, I. Nourdin and G. Peccati,  Quantitative CLTs in Deep Neural Networks,  2023, arXiv:2307.06092.
  * [14] I. I. Gikhman and A. V. Skorokhod,  _Introduction to the theory of random processes_ ,  Saunders, Philadelphia, 1969.
  * [15] A. V. Gnedenko and A. N. Kolmogorov,  _Limit distributions for sums of independent random variables_ ,  Addison-Wesley, Boston, 1968.
  * [16] A. Gogatishvili, P. Koskela and Y. Zhou,  Characterizations of Besov and Triebel–Lizorkin spaces on metric measure spaces,  _Forum Mathematicum_ , 25 (2013), 787-819, DOI:10.1515/form.2011.135.
  * [17] M. Gurbuzbalaban, U. Simsekli, and L. Zhu,  The heavy-tail phenomenon in SGD,  _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , 2021, 3964-3975, arXiv:2006.04740.
  * [18] B. Hanin,  Random neural networks in the infinite width limit as Gaussian processes,  _The Annals of Applied Probability_ , 33 (2023), 4798-4819, DOI:10.1214/23-AAP1933.
  * [19] T. Heikkinen, P. Koskela and H. Tuominen,  Approximation and quasicontinuity of Besov and Triebel–Lizorkin functions,  _Transactions of the American Mathematical Society_ , 369 (2017), 3547-3573, DOI:10.1090/tran/6886.
  * [20] J. Hron, Y. Bahri, R. Novak, J. Pennington and J. Sohl-Dickstein,  Exact posterior distributions of wide Bayesian neural networks,  2020, arXiv:2006.10541.
  * [21] A. Jacot, F. Gabriel and C. Hongler,  Neural tangent kernel: Convergence and generalization in neural networks,  _Advances in Neural Information Processing Systems_ , volume 31, 2018.
  * [22] P. Jung, H. Lee, J. Lee and H. Yang,  $\alpha$-stable convergence of heavy-/light-tailed infinitely wide neural networks,  _Advances in Applied Probability_ , 55 (2023), 1415-1441, DOI:10.1017/apr.2023.3.
  * [23] O. Kallenberg,  _Foundations of modern probability_ ,  Springer, New York, 2002.
  * [24] H. Kekkonen, M. Lassas, E. Saksman and S. Siltanen,  Random tree Besov priors - Towards fractal imaging,  _Inverse Problems and Imaging_ , 17 (2023), 507-531, DOI:10.3934/ipi.2022059.
  * [25] Y. Korolev, J. Latz and C.-B. Schönlieb,  Gaussian random fields on non-separable Banach spaces,  2022, arXiv:2203.04650.
  * [26] S. Lasanen,  Non-Gaussian statistical inverse problems. Part II: Posterior convergence for approximated unknowns,  _Inverse Problems and Imaging_ , 6 (2012), 267–287, DOI:10.3934/ipi.2012.6.267.
  * [27] M. Lassas, E. Saksman and S. Siltanen,  Discretization-invariant Bayesian inversion and Besov space priors,  _Inverse Problems and Imaging_ , 3 (2009), 87-122, DOI:10.3934/ipi.2009.3.87.
  * [28] H. Lee, F. Ayed, P. Jung, J. Lee, H. Yang and F. Caron,  Deep neural networks with dependent weights: Gaussian process mixture limit, heavy tails, sparsity and compressibility,  _Journal of Machine Learning Research_ , 24 (2023), 1-78.
  * [29] J. Lee, L. Xiao, S. S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington,  Wide neural networks of any depth evolve as linear models under gradient descent,  _Advances in Neural Information Processing Systems_ , volume 32, MIT Press, 2019.
  * [30] C. Li, M. Dunlop and G. Stadler,  Bayesian neural network priors for edge-preserving inversion,  _Inverse Problems and Imaging_ , 16 (2022), 1229–1254, DOI:10.3934/ipi.2022022.
  * [31] A. G. D. G. Matthews, J. Hron, M. Rowland, R. E. Turner and Z. Ghahramani,  Large-width functional asymptotics for deep Gaussian neural networks,  _International Conference on Learning Representation_ , 2018, https://openreview.net/forum?id=H1-nGgWC-.
  * [32] R. M. Neal,  _Bayesian learning for neural networks_ ,  Springer, New York, 1996.
  * [33] L. Roininen, P. Piiroinen and M. Lehtinen,  Constructing continuous stationary covariances as limits of the second-order stochastic difference equations,  _Inverse Problems and Imaging_ , 7 (2013), 611–647, DOI:10.3934/ipi.2013.7.611.
  * [34] E. Saksman and T. Soto,  Traces of Besov, Triebel-Lizorkin and Sobolev spaces on metric spaces,  _Analysis and Geometry in Metric Spaces_ , 5 (2017), 98-115, DOI:10.1515/agms-2017-0006.
  * [35] G. Samorodnitsky and M. S. Taqqu,  _Stable non-Gaussian random processes: Stochastic models with infinite variance_ ,  Chapman & Hall, New York, 1994.
  * [36] A. M. Stuart,  Inverse problems: A Bayesian perspective,  _Acta Numerica_ , 19 (2010), 451-559, DOI:10.1017/S0962492910000061.
  * [37] T. J. Sullivan,  Well-posed Bayesian inverse problems and heavy-tailed stable quasi-Banach space priors,  _Inverse Problems and Imaging_ , 11 (2017), 857-874, DOI:10.3934/ipi.2017040.
  * [38] J. Suuronen, N. K. Chada and L. Roininen,  Cauchy Markov random field priors for Bayesian inversion,  _Statistics and Computing_ , 32 (2022), DOI:10.1007/s11222-022-10089-z.
  * [39] H. Triebel,  _Theory of Function Spaces_ ,  Birkhäuser Verlag, Basel, 1983.
  * [40] H. Triebel,  _Theory of Function Spaces III_ ,  Birkhäuser Verlag, Basel, 2006.
  * [41] G. Yang,  Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation,  2020, arXiv:1902.04760.