[datatype=bibtex] [fieldsource=title, match=
$, replace=$] [fieldsource=title, match={
textasciicircum}, replace=^] [fieldsource=title, match=
_, replace=_] [fieldsource=title, match={
textbackslash}, replace=
] [fieldsource=title, match=
{{R}
}, replace={R}]

# Space-Time Approximation with Shallow Neural Networks in Fourier Lebesgue
spaces

Ahmed Abdeljawad∗ and Thomas Dittrich
Johann Radon Institute for Computational and Applied Mathematics (RICAM),
Austrian Academy of Sciences,
Altenbergerstr. 69, 4040 Linz, Austria
{ahmed.abdeljawad<EMAIL_ADDRESS>Both authors contributed
equally.

###### Abstract

Approximation capabilities of shallow neural networks (SNNs) form an integral
part in understanding the properties of deep neural networks (DNNs). In the
study of these approximation capabilities some very popular classes of target
functions are the so-called spectral Barron spaces. This spaces are of special
interest when it comes to the approximation of partial differential equation
(PDE) solutions. It has been shown that the solution of certain static PDEs
will lie in some spectral Barron space. In order to alleviate the limitation
to static PDEs and include a time-domain that might have a different
regularity than the space domain, we extend the notion of spectral Barron
spaces to anisotropic weighted Fourier-Lebesgue spaces. In doing so, we
consider target functions that have two blocks of variables, among which each
block is allowed to have different decay- and integrability-properties. For
these target functions we first study the inclusion of anisotropic weighted
Fourier-Lebesgue spaces in the Bochner-Sobolev spaces. With that we can now
also measure the approximation error in terms of an anisotropic Sobolev norm,
namely the Bochner-Sobolev norm. We use this observation in a second step
where we establish a bound on the approximation rate for functions from the
anisotropic weighted Fourier-Lebesgue spaces and approximation via SNNs in the
Bochner-Sobolev norm.

Keywords Function Space$\,\cdot\,$Anisotropic Space$\,\cdot\,$Neural
Networks$\,\cdot\,$Approximation Theory

Mathematics Subject Classification
41A25,$\,\cdot\,$41A46,$\,\cdot\,$41A30,$\,\cdot\,$46E35,$\,\cdot\,$62M45,$\,\cdot\,$68T05

##  1 Introduction

In recent years DNNs have gained a huge amount of attention not only in
practical applications, but also from a theoretical perspective [22, 15]. Up
to now the development of a rigorous theory that explains the empirical
success of DNNs is an active field. In this context the study of networks with
a single hidden layer is especially important in order do build a foundation
for the understanding of more complex DNNs by understanding the properties of
the individual layers. The study of these SNNs has recently (re-)gained a
surge of attention from several different perspectives such as the
characterization of representable functions and the associated representation
cost [1, 12, 48, 42, 41], asymptotic approximation properties [3, 9, 17, 29,
30, 50, 51, 52, 38, 59], and the application to PDEs [10, 11, 20, 37, 36, 40].

In the context of approximation based on SNNs as hypothesis class, a very
popular family of target functions are the spectral Barron spaces [4, 8, 9,
11, 16, 30, 38, 50, 59]. For the present work, these classes of functions are
important out of two major reasons: First, learning the parameters of a SNN is
a special case of function-representation by means of a dictionary. This means
that the theory of dictionary learning (see [14, 32, 5]) can be applied to
SNNs. Thereby, one can develop bounds on the approximation error based on the
width $N$ of the network. For functions in the spectral Barron spaces, these
bounds come without curse of dimensionality, i.e., the scaling in $N$ is
independent of the input dimension $d$. Such an approximation result for
spectral Barron spaces was first shown by [4] in [4]. Second, the spectral
Barron spaces are especially interesting for application to PDEs This is
because for different types of PDEs it has been shown that under certain
conditions the solution will be in a spectral Barron space (see for example
[20, 11]).

As already mentioned, the notion of spectral Barron spaces dates back to the
1990s where [4] [4] identified a class of functions that can be approximated
without curse of dimensionality by a shallow network with sigmoidal activation
function. This class of functions $f$ is characterized by the $L^{1}$
integrability condition for the first Fourier-moment, i.e.,

$\displaystyle\int_{\mathbf{R}^{d}}|\xi||\hat{f}(\xi)|d\xi<\infty,$ (1.1)

where $\hat{f}$ denotes the Fourier transform of $f$. Throughout the
literature, this weighted spectral $L^{1}$ integrability condition has been
modified in several different ways with regard to the weighting. As seen
above, in the seminal work of [4] [4] the choice for the weight function was
$\omega(\xi)=|\xi|$. Other choices for polynomially bounded weights that have
been proposed in the literature are $\omega(\xi)=|\xi|^{2}$ in [8],
$\omega(\xi)=|\xi|_{1}^{s}$ with $s\in\\{2,3\\}$ in [29, 30],
$\max\\{1,|\xi|^{2}\\}$ in [20], $\omega(\xi)=(1+|\xi|)^{s}$ with any $s>0$ in
[50, 51, 52, 53, 59, 60], $\omega(\xi)=(1+|\xi|^{2})^{\frac{s}{2}}$ with $s>0$
in [11] and with $s=2$ in [17], and
$\omega(\xi)=\sup_{x\in\Omega}|\left<\xi,x-x_{0}\right>|$ for some $x_{0}$ in
the domain $\Omega$ of $f$ in [9]. The exponentially bounded weight
$\omega(\xi)=e^{c|\xi|^{\beta}}$ with $0<\beta<1$ and $0<c$ has been treated
in [51].

A common rationale in the definition of spectral Barron spaces is to allow
functions that are only defined on some bounded domain $\Omega$. This has the
consequence that different Fourier representations lead to the same function
when restricting to the given domain. In order for a function to be in the
spectral Barron space, there needs to be at least one representation for which
the integrbility condition is finite. Thus, it is desirable to consider the
infimum over all representation. A unified formulation with a general weight
function that also allows the possibility of bounded domains can be written as
follows:

######  Definition 1.1 (Spectral Barron Space with General Weight).

Let $d\in\mathbf{N}$, $\Omega\subseteq\mathbf{R}^{d}$,
$\omega:\mathbf{R}^{d}\to(0,\infty)$ such that $\Omega$ and $\omega$ are a
measurable set and function, respectively. For a function $f\in L^{1}(\Omega)$
the spectral Barron (semi-)norm with general weight is given by

$\displaystyle\|f\|_{\mathscr{B}_{\omega}(\Omega)}:=\inf_{\begin{subarray}{c}f_{e}\in
L^{1}(\mathbf{R}^{d})\\\
f_{e}|_{\Omega}=f\end{subarray}}\int_{\mathbf{R}^{d}}\omega(\xi)|\hat{f}_{e}(\xi)|d\xi$

and the spectral Barron space with general weight is given by

$\displaystyle\mathscr{B}_{\omega}(\Omega):=\\{f\in
L^{1}(\Omega):\|f\|_{\mathscr{B}_{\omega}(\Omega)}<\infty\\}.$

Note that in general $\|\cdot\|_{\mathscr{B}_{|\cdot|}(\Omega)}$ is only a
semi-norm, whereas $\|\cdot\|_{\mathscr{B}_{(1+|\cdot|)^{s}}}$ is indeed a
norm [52].

We want to note that the term _Barron space_ (without the prefix _spectral_)
is also used for a second, different, class of functions. These _infinite-
width Barron spaces_ are defined via representations in terms of infinite-
width shallow networks [9, 34, 16]. For SNNs with heaviside activation and
ReLU activation it can be shown that the spectral Barron space with some
specific choices of weights is a subset of the infinite-width Barron space
(cf. [9, Lemma 7.1]). However, in the present work we deal with
generalizations of the spectral Barron space and extensions for these
embeddings into the infinite-width Barron space are left for future work.

One of the central elements of approximation theory for SNNs is $N$-term
approximation from dictionaries. According to [14] [14], this was first
introduced by [49] [49] in 1907. A major contribution to the study of upper
bounds on the approximation error of $N$-term approximations is attributed to
some non published results of Maurey by [46] [46]. This result is as follows:
Functions in the closure of the convex hull of some subset $\mathcal{G}$ of a
type-$2$ Banach space $\mathcal{B}$ can be approximated well by linear
combinations of $N$ elements of $\mathcal{G}$ in the sense that the error in
the $\mathcal{B}$-norm is bounded by $cN^{-1/2}$ with some constant $c>0$.
This statement can be found in [4, Lemma 1] stated in terms of Hilbert spaces
and in [52, Theorem 1] for general type-$2$ Banach spaces. [4] [4] used this
result to prove that for every $N\in\mathbf{N}$ and every $f$ with
$\|f\|_{\mathscr{B}_{|\cdot|}(\mathbf{R}^{d})}<\infty$ there is a SNN $f_{N}$
with $N$ neurons and sigmoidal activation function, such that

$\displaystyle\|f-f_{N}\|_{L^{2}(\mu,\mathbf{R}^{d})}\lesssim
N^{-\frac{1}{2}}$

for any probability measure $\mu$. Recently, the class of functions to which
the approximation result of Maurey applies has been extended to the variation
space with respect to dictionaries [3, 32, 38, 52, 53]. This allowed the
development of bounds on the $L^{2}$ norm [8, 30] and the $H^{m}$ norm (cf.
[37, 11] for $m=1$ and [50, 51] for general $m\in\mathbf{N}$) in terms of the
spectral Barron norm. All these bounds apply to shallow networks with either
ReLUk activation with $k=1$ [8] or with an arbitrary $k\in\mathbf{N}$ [51],
cosine activation [51, 11], or some general polynomially decaying activation
functions [50].

For compact or smooth dictionaries and certain choices of parameters, the
approximation rates can be further improved from $O(n^{-1/2})$ up to
$O(n^{-1})$ for sigmoidal activation function with $L^{2}$ error [39] and to
$O(n^{-(k+1)})$ for ReLUk activation function of order $k$ with $L^{2}$ [51]
and $L^{\infty}$ error [38]. However, in our setting we do not require
compactness or smoothness of the dictionary and, thus, we restrict to
approximations rates of order $O(n^{-1/2})$. In our theory, we unify all the
bounds with finite integrability-exponent in the error measure by considering
the Sobolev norm $W^{m,p}$ with $m\geq 0$ and $2\leq p<\infty$ and by allowing
arbitrary weight functions.

In the present literature on the application of SNNs to PDEs, spectral Barron
spaces have been used to characterize the solutions of the static Schrödinger
equation [37, 11], the Poisson equation [37], nonlinear variational PDEs [40],
Black-Scholes Type PDEs [20]. This natural application of spectral Barron
spaces comes from the fact that the weight in the integrability condition can
be seen as the symbol of a certain PDE. However, it can be seen that in the
current applications, the variables are constrained to be of the same order of
differentiability. Nevertheless, for example for evolution equations it is
beneficial to consider different orders of differentiability for the time- and
the space-variable (see [18, Chapter 7]). In our work we build the foundation
to extend the theory and application of spectral Barron spaces to more general
classes of PDEs by allowing a general weight $\omega$ that fulfills some mild
conditions and by allowing two blocks of variables (each of arbitrary size)
that come with different orders of differentiability.

###  1.1 Contribution and Outline

Inspired by the successfull application of spectral Barron spaces to the
approximation of solutions of PDEs, we have established a novel perspective in
this field which is aimed at allowing non-isotropic differentiability and
integrability of these solutions. The motivation for this is (as hinted
already above) that for time-dependent PDEs, it is often advantageous to
consider the time variable separately and allow different properties for the
resulting two blocks of variables. We achieve this goal by introducing the so-
called _anisotropic weighted Fourier-Lebesgue spaces_ to the field of SNNs as
the natural extension of the spectral Barron spaces. The main focus of our
investigation is on the properties and the approximation capability of target
functions in the anisotropic weighted Fourier-Lebesgue spaces. In what
follows, we will cover the different branches of our contribution:

From the perspective of harmonic analysis, we extended the existing theory for
isotropic weighted Fourier-Lebesgue spaces
$\mathscr{F}L^{p}(\omega;\mathbf{R}^{d})$ (see [45]) to the case of two blocks
$\mathscr{F}L^{p,q}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})$ in
Definition 2.1 (the multi blocks case is then a straightforward extension of
this). We build the cornerstone for these spaces by combining ideas from
anistropic Lebesgue spaces and weighted Banach spaces. The first result in
this regard is an inclusion between Bochner-Sobolev spaces and Fourier-
Lebesgue spaces in different situations. Specifically for solutions to time-
dependent PDEs, we showed the following:

######  Lemma 1.2.

Let $1\leq s_{i},q_{i}\leq 2\leq p_{i}\leq\infty$ with $p_{1}\leq p_{2}$ such
that $\frac{1}{s_{i}}+\frac{1}{q_{i}}+\frac{1}{p_{i}}=2$, for $i\in\\{1,2\\}$.
Let $\omega(x,y)$ be a weight function defined on
$\mathbf{R}\times\mathbf{R}^{d}$, elliptic with respect to $\langle
t\rangle^{n_{1}}\langle x\rangle^{n_{2}}$, $\Omega\subseteq\mathbf{R}^{d}$ be
a bounded space domain, and $I\subset\mathbf{R}$ be a bounded time domain. Let
$u\in\mathscr{F}L^{q_{1},q_{2}}(\omega;\mathbf{R},\mathbf{R}^{d})$ then we
have

$\|u\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(I,\Omega)}\lesssim\|{\chi_{I}}\|_{\mathscr{F}L^{s_{1}}(\mathbf{R})}\|{\chi_{\Omega}}\|_{\mathscr{F}L^{s_{2}}(\mathbf{R}^{d})}\|u\|_{\mathscr{F}L^{q_{1},q_{2}}(\omega;\mathbf{R},\mathbf{R}^{d})},$
(1.2)

where the hidden constant in Eq. 1.2 depends only on the regularity and the
integrability of the solution $u$ with respect to time and space variables.

We refer the reader to Lemma 3.1 for the general statement of Lemma 1.2. It
can be seen that the right side of Eq. 1.2 depends on the integrability of the
characteristic function of both domains. We discuss the importance of this and
the dependency on the geometry for more general domains in details in Section
3.1.1. For the particular choice $I=[0,T]$ with $T>0$ for the time domain,
$\Omega=[-1,1]^{d}$ for the space domain, and $s_{1},s_{2}>1$ for the degree
of integrability, the inclusion inequality Eq. 1.2 can be reduced to

$\displaystyle\|u\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}([0,T],[-1,1]^{d})}\lesssim\left(\frac{4s_{1}}{T(s_{1}-1)}\right)^{\frac{s_{1}+1}{s_{1}}}\left(\frac{2s_{2}}{s_{2}-1}\right)^{d\frac{s_{2}+1}{s_{2}}}\|u\|_{\mathscr{F}L^{q_{1},q_{2}}(\omega;\mathbf{R},\mathbf{R}^{d})}.$
(1.3)

Our approach has the potential for further expansion, in the sense that our
framework can be applied to various cases such as Bochner-Besov spaces and
more generally to Bochner-Banach spaces. However, we chose to restrict our
focus to the Bochner-Sobolev case in order to streamline our computations and
maintain a clear focus. We leave these extensions for future research.

Furthermore, we contributed to the field of learning theory with SNNs by
providing a generalization of spectral Barron spaces and studying the
associated approximation properties. As seen in Definition 1.1 and the
preceeding discussion, the existing work deals solely with a single block of
variables (i.e.,
$\mathscr{B}_{\omega}(\mathbf{R}^{d})=\mathscr{F}L^{1}(\omega,\mathbf{R}^{d})$).
That is, either the PDE of interest is static (i.e., there is no time
variable) or the time- and space variables are stacked into a single joint
variable. The latter limits the analysis of the approximation error in the
Sobolev norm to the minimum degree of differentiability of the two blocks and
it enforces that both variables have to be integrable in the same $L^{p}$
norm. In the present work we introduce a space-time version of spectral Barron
spaces and thereby, we allow for much more generality and precision when
addressing time dependent PDEs. More generally, this also applies to the
approximation multi-variable functions with different integrability-, growth-
and regularity properties in each block of variables. Additionally, the
extension to Fourier-Lebesgue spaces allows us to treat different types of
integrability, namely, $L^{q}$ with $q\in[1,2]$. By doing so, we obtain
existing (single-block) approximation results for the spectral Barron space
(i.e., $q=1$), as well as for the Hilbert-Sobolev space (i.e., $q=2$) as
special (extreme) cases (see [4, 50]). To the best of our knowledge, our paper
is the first work that treats the space-time generalization and the different
types of integrability. Our main result (again with simplifications to address
the setting of time-dependent PDEs; the full generality can be found in
Theorem 3.9) can be stated as follows:

######  Theorem 1.3.

For $i\in\\{1,2\\}$ let $n_{i}\in\mathbf{N}$, and $1\leq q_{i}\leq 2\leq
p_{i}<\infty$ with $p_{1}\leq p_{2}$. Let
$\vartheta(t_{1},t_{2})\gtrsim\langle t_{1}\rangle^{\gamma_{1}}\langle
t_{2}\rangle^{\gamma_{2}}$ for some $\gamma_{1},\gamma_{2}>1$ and any
$(t_{1},t_{2})\in\mathbf{R}^{2}$, and $\omega(t,x)\gtrsim\langle
t\rangle^{n_{1}}\langle x\rangle^{n_{2}}$ for any $(t,x)\in\mathbf{R}^{d+1}$
moreover let

$\displaystyle\tilde{\omega}(t,x):=\omega(t,x)\langle
t\rangle^{2(1-\frac{1}{q_{1}})+1}\langle
x\rangle^{(d+1)(1-\frac{1}{q_{2}})+1}.$

For an activation function $\sigma\in
W_{n_{1},\infty}^{n_{2},\infty}(\vartheta;\mathbf{R},\mathbf{R}){\setminus}\\{0\\}$
and $u\in\mathscr{F}L^{q_{1},q_{2}}(\tilde{\omega};\mathbf{R},\mathbf{R}^{d})$
there exists a constant $C>0$ such that

$\displaystyle\inf_{u_{N}\in\Sigma_{1,d}^{N}(\sigma)}\|u-u_{N}\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}([0,T],[-1,1]^{d})}$
$\displaystyle\leq
CN^{-\frac{1}{2}}T^{\frac{1}{p_{1}}}2^{\frac{d}{p_{2}}}\|u\|_{\mathscr{F}L^{q_{1},q_{2}}(\tilde{\omega};\mathbf{R},\mathbf{R}^{d})}.$
(1.4)

It is worth to mention that our target class includes the setting of [43],
where the authors measured the accuracy of approximating a given function in
the weighted Feichtinger’s Segal algebra (i.e., the weighted modulation space
with $p=q=1$ where the weight $\omega$ is the Bessel potential). The relation
between anisotropic weighted Fourier-Lebesgue spaces and weighted modulation
spaces is presented in Remark 2.2.

Note that our analysis also covers the case of a single block. To see this, we
consider the case with $u(t,x)=\chi_{[0,1]}(t)u(x)$, $T=1$, and
$p_{1}=q_{1}=2$. Then, every derivative with respect to $t$ vanishes, thereby
reducing the Sobolev norm on the left side to the second variable. On the
right side, we see with the product structure of $u$, the norm of the product
can be split into the product of the norms. For $p_{1}=2$ we can apply
Parseval’s theorem and see that every factor that is related to $t$ is
trivially equal to $1$. For the application to time-dependent PDEs, this means
that we define a new space variable
$X:=(t,x)=(t,x_{1},\cdots,x_{d})\in\mathbf{R}^{d+1}$ and instead of
considering different integrability- and differentiability degrees we have to
choose $p_{1}=p_{2}=p$, $q_{1}=q_{2}=q$ and $n=\min\\{n_{1},n_{2}\\}$. The
single-block version of Theorem 1.3 is given by

######  Theorem 1.4.

Let $d,m,n\in\mathbf{N}$, such that $m\geq n$, and $1\leq q,\leq 2\leq
p<\infty$. Let $\vartheta(t)\gtrsim\langle t\rangle^{\gamma}$ for some
$\gamma>1$ and any $t\in\mathbf{R}$, and $\omega(x)\gtrsim\langle
x\rangle^{n}$ for any $x\in\mathbf{R}^{d}$ moreover let

$\displaystyle\tilde{\omega}(x):=\omega(x)\langle
x\rangle^{(d+1)(1-\frac{1}{q})+1}.$

For an activation function $\sigma\in
W^{m,\infty}(\vartheta;\mathbf{R}){\setminus}\\{0\\}$ and
$u\in\mathscr{F}L^{q}(\tilde{\omega})$, there exists a constant $C>0$ such
that we have

$\displaystyle\inf_{u_{N}\in\Sigma_{\sigma}^{N}}\|u-u_{N}\|_{W^{n,p}([0,T]\times[-1,1]^{d})}$
$\displaystyle\leq
CN^{-\frac{1}{2}}T^{\frac{1}{p}}2^{\frac{d}{p}}\|u\|_{\mathscr{F}L^{q}(\tilde{\omega};\mathbf{R}^{d+1})}.$
(1.5)

(See Section 2 for more details regarding the notations.)

We have seen that both versions (single-block and two-block) of our
approximation bound depends on some constant $C$. In both cases the constant
depends on the supremum of the domain, the polynomial-decay exponent of the
activation function, the integral degree of the approximation error, and the
number of derivatives in the approximation error (thereby implicitely on the
input dimension). Unlike the recent literature (cf. [38]) we also perform a
theoretical analysis of the constants embedded within the approximation
inequality. With that we find scenarios, in which SNNs successfully overcome
the curse of dimensionality. The details on that can be found in Proposition
3.10.

At the present moment our analysis is limited to measuring the error in the
Bochner-Sobolev norm with $p\geq 2$. This is because the currently available
techniques and methods rely on the fact that the error is measured in a
type-$2$ Banache space. For $p<2$, the Bochner-Sobolev space will be a
type-$p$ Banach space and therefore, the existing theory does not apply
anymore. We leave the investigation of this line of research for future
projects.

The paper is organised as follows: In Section 2 we briefly recall some
essential tools from harmonic and functional analysis. Moreover we define the
Bochner-Sobolev spaces considered in out setting. Then we review the variation
space and its connection to conclude an approximation rate. While our results
regarding the inclusion of Fourier-Lebesgue spaces in Bochner-Sobolev Spaces
for the high-order and low-order cases developed in Sections 3.1 and 3.2,
respectively. Our main contribution concerning the efficiency of SNNs in
approximating Fourier-Lebesgue functions (in particular Barron functions) with
respect to Bochner-Sobolev norm can be found in Section 3.3 Finally in Section
4 we present examples and experiments studies to demonstrate the practical
relevance our finding.

###  1.2 Notation

Throughout this work, we denote the Schwartz space of rapidly decreasing
functions on $\mathbf{R}^{d}$ by $\mathscr{S}(\mathbf{R}^{d})$. The Lebesgue
measure of a set $E$ is denoted by $|E|$ and its characteristic function by
$\chi_{E}(x)$. Binary relations on multi-indices act element-wise, i.e., for
$\alpha=(\alpha_{1},\dots,\alpha_{d}),\beta=(\beta_{1},\dots,\beta_{d})\in\mathbf{Z}_{+}^{d}$
we say $\alpha\leq\beta$ if and only if $\alpha_{i}\leq\beta_{i}$ for all
$i\in\\{1,\dots,d\\}$ and for $\beta\leq\alpha$ we define the difference
$\alpha-\beta$ as the multi-index
$\alpha_{1}-\beta_{1},\dots,\alpha_{d}-\beta_{d}$. The magnitude of a
multindex $\alpha\in\mathbf{Z}_{+}^{d}$ is given by
$|\alpha|=\alpha_{1}+\dots+\alpha_{d}$, and for any $x\in\mathbf{R}^{d}$ the
element-wise exponent is $x^{\alpha}=x_{1}^{\alpha_{1}}x_{2}^{\alpha_{2}}\dots
x_{d}^{\alpha_{d}}$ and similarly, the partial derivatives are given by
$\partial^{\alpha}=\partial_{x_{1}}^{\alpha_{1}}\partial_{x_{2}}^{\alpha_{2}}\dots\partial_{x_{d}}^{\alpha_{d}}$.
For a scalar $x$ we denote the ReLU function as $(x)_{+}:=\max\\{0,x\\}$ for
$x\in\mathbf{R}^{d}$ we denote the Bessel potential via the bracket
$\langle\cdot\rangle:=(1+|x|^{2})^{\frac{1}{2}}$.

We stress that the constants that appear in inequalities may differ from line
to line; i.e., $C$ is a placeholder for constants whose dependence is listed
in the subscript or in the accompanying text.

##  2 Preliminaries

As a first step towards our approximation results for SNNs, we recall some
basic concepts and results.

###  2.1 Anisotropic Weighted Fourier-Lebesgue Spaces

As a first step, we will cover the basics of harmonic analysis and introduce
our concept of anisotropic weighted Fourier-Lebesgue spaces.

For the Fourier transform $\mathscr{F}$ and its inverse $\mathscr{F}^{-1}$ we
use the convention with symmetric normalization, i.e.,

$\displaystyle(\mathscr{F}f)(\xi)$
$\displaystyle\equiv\frac{1}{(2\pi)^{\frac{d}{2}}}\int_{\mathbf{R}^{d}}f(x)e^{-i\langle
x,\xi\rangle}\,dx\qquad\text{and}\qquad(\mathscr{F}^{-1}\hat{f})(x)\equiv\frac{1}{(2\pi)^{\frac{d}{2}}}\int_{\mathbf{R}^{d}}\hat{f}(\xi)e^{i\langle
x,\xi\rangle}\,d\xi,$

and we will write $\hat{f}$ as a short form of $\mathscr{F}f$. Both integrals
are well defined when $f\in L^{1}(\mathbf{R}^{d})$ and $\hat{f}\in
L^{1}(\mathbf{R}^{d})$, respectively.

For the Fourier transform on multiple blocks, we first let $\mathscr{F}_{1}$
and $\mathscr{F}_{2}$ denote the (partial) Fourier transform with respect to
the first and second block of variables, respectively. That is, for a function
$f\in L^{1}(\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}})$ we have

$\displaystyle(\mathscr{F}_{1}f)(\xi,y)$
$\displaystyle\equiv\frac{1}{(2\pi)^{\frac{d_{1}}{2}}}\int_{\mathbf{R}^{d_{1}}}f(x,y)e^{-i\langle
x,\xi\rangle}\,dx,\qquad\text{and}$ $\displaystyle(\mathscr{F}_{2}f)(x,\eta)$
$\displaystyle\equiv\frac{1}{(2\pi)^{\frac{d_{2}}{2}}}\int_{\mathbf{R}^{d_{2}}}f(x,y)e^{-i\langle
y,\eta\rangle}\,dy.$

By combining the partial Fourier transforms, we get the two-block Fourier
transform

$\mathscr{F}f(\xi,\eta)=(\mathscr{F}_{2}(\mathscr{F}_{1}f))(\xi,\eta)=(\mathscr{F}_{1}(\mathscr{F}_{2}f))(\xi,\eta)$
(2.1)

for $f\in L^{1}(\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}})$. A similar notion
to this is the so-called _spacetime Fourier transform_ which is commonly used
in the analysis of dispersive partial differential equations [57]. It is
formally defined for $u:\mathbf{R}\times\mathbf{R}^{d}\rightarrow\mathbf{C}$
as

$\displaystyle(\mathscr{F}{u})(\tau,\xi):=\int_{\mathbf{R}}\int_{\mathbf{R}^{d}}u(t,x)e^{-i(t\tau+x\cdot\xi)}dtdx$

and can thereby be seen as a two-block Fourier transform with $d_{1}=1$.

For the weights in our norm, we will consider weight functions
$\omega:\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}\rightarrow(0,\infty)$ which
are measurable and such that $\omega(x,y),1/\omega(x,y)>0$ for any
$(x,y)\in\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$. Later in our embedding
result we require that the growth of the weight is bounded in some way. For
that we adopt the notion of moderateness from [45], i.e., for two weight
functions $\omega$ and $v$ on $\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$, we
say $\omega$ is $v$-moderate if

$\omega(x_{1}+x_{2},y_{1}+y_{2})\leq C\omega(x_{1},y_{1})v(x_{2},y_{2})$ (2.2)

for some uniform constant $C$. If $v$ in (2.2) can be chosen as a polynomial
(or exponential), then $\omega$ is called polynomially (or exponentially)
moderated. As an example, the weight $\omega(x,y)=\langle x\rangle^{s}\langle
y\rangle^{\sigma}$, with $s,t\in\mathbf{R}$, is polynomially moderated and for
$r,\rho,s,t>0$ the weight $\omega(x,y)=e^{r|x|^{s}+\rho|y|^{t}}$ is
exponentially moderated. In the usual convention of Harmonic analysis (see
[24, Page 4]), we will also assume that weights are submultiplicative, i.e., a
weight $\omega$ is $v$-moderate with $v\equiv\omega$. Contrary to the upper
bound provided by moderatedness, we additionally require that the weight is
lower bounded in the following way: For two weight functions
$\omega,\vartheta$ we say that $\omega$ is elliptic with respect to
$\vartheta$ if

$0<\vartheta(x,y)\leq c\,\omega(x,y)\text{ for any
}(x,y)\in\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$ (2.3)

with a uniform constant $c>0$.

In [6] Benedek and Panzone introduced the mixed (anisotropic) Lebesgue spaces
$L^{\overrightarrow{p}}(\mathbf{R}^{d})$ with
$\overrightarrow{p}=(p_{1},\dots,p_{d})\in[1,\infty]^{d}$. A function $u$ is
said to belong to $L^{\overrightarrow{p}}(\mathbf{R}^{d})$ if

$\displaystyle\|u\|_{L^{\overrightarrow{p}}(\mathbf{R}^{d})}\equiv\left(\int_{\mathbf{R}}\dots\left(\int_{\mathbf{R}}|u(x_{1},x_{2},\dots,x_{d})|^{p_{1}}dx_{1}\right)^{{p_{2}}/{p_{1}}}\dots
dx_{d}\right)^{{1}/{p_{d}}}<\infty$

with obvious interpretation when $p_{i}=\infty$. The anisotropic Lebesgue
spaces $L^{\overrightarrow{p}}(\mathbf{R}^{d})$ are a generalization of
Lebesgue spaces $L^{p}(\mathbf{R}^{d})$, such that the integrability exponent
is different for each variable. In our case, we consider that case that the
exponent is constant within each block of variables, but it is allowed to be
different between the two blocks. For that we let $d_{1},d_{2}\in\mathbf{N}$,
$p,q\in[1,\infty]$, and $U\subseteq\mathbf{R}^{d_{1}}$,
$V\subseteq\mathbf{R}^{d_{2}}$ and define the norm

$\|u\|_{L^{p,q}(U,V)}\equiv\left(\int_{U}\left(\int_{V}|u(x,y)|^{q}dy\right)^{p/q}dx\right)^{1/p}.$
(2.4)

If $p=q$ this simplifies to $L^{p}(U\times V)$ and in case
$U=\mathbf{R}^{d_{1}}$ and $V=\mathbf{R}^{d_{2}}$ we use the short notation
$\|\cdot\|_{L^{p,q}(\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})}=\|\cdot\|_{L^{p,q}}$
(not to be confused with the Lorentz spaces) and further
$\|\cdot\|_{L^{p,p}}=\|\cdot\|_{L^{p}}$.

A similar definition for the mixed Lebesgue norms is used in the analysis of
dispersive PDEs [57] (see also [55]) where $U=I$ is an interval and
$V=\mathbf{R}^{d}$. We stress that the mixed Lebesgue norms are the right
spaces to deliver Strichartz estimates which are essential to the well-
posedness of certain PDEs, e.g., non-linear Schrödinger equations or linear
Schrödinger equations with time-dependent potentials [28, 19].

Combining the concepts of Fourier-Lebesgue spaces, two-block Fourier
transform, weight functions, and anisotropic Lebesgue spaces leads to the
definition of anisotropic weighted Fourier-Lebesgue spaces:

######  Definition 2.1 (Anisotropic Weighted Fourier-Lebesgue Spaces).

Let $p,q\in[1,\infty]$ and $\omega$ be a weight defined over
$\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$. The anisotropic (weighted)
Fourier-Lebesgue space
$\mathscr{F}L^{p,q}(\omega,\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})$ consists of
all $f\in L^{1}(\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}})$ such that

$\displaystyle\|f\|_{\mathscr{F}L^{p,q}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})}$
$\displaystyle\equiv\|\omega\mathscr{F}f\|_{L^{p,q}(\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})}$
(2.5)

is finite.

Here and in what follows we use the notation $\mathscr{F}L^{p,q}(\omega)$
instead of $\mathscr{F}L^{p,q}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})$.
If $\omega=1$, then the notation $\mathscr{F}L^{p,q}$ is used instead of
$\mathscr{F}L^{p,q}(1)$. We note that if $d_{1}=d_{2}=d$,
$\omega(\xi,\eta)=\langle(\xi,\eta)\rangle^{s}$, then
$\mathscr{F}L^{p,p}(\omega)$ is the Fourier image of the Bessel potential
space $H^{p}(\omega;\mathbf{R}^{2d})$ (cf. [7]). Furthermore, if $p=q$ we
write $\mathscr{F}L^{p}(\omega)$ instead of $\mathscr{F}L^{p,p}(\omega)$.

######  Remark 2.2.

We note that if $d_{1}=d_{2}=d$, $\omega$ is a weight function on
$\mathbf{R}^{d}\times\mathbf{R}^{d}$ and
$u,\varphi\in\mathscr{S}(\mathbf{R}^{d})$, then for functions of the form

$f(x,y)=\mathscr{F}_{1}^{-1}(u(y)\varphi(y-\,\cdot)),$

we get

$\mathscr{F}f(\xi,\eta)=\mathscr{F}_{2}(u(\cdot)\varphi(\cdot-\xi)(\eta)=(2\pi)^{-{d}/2}\int_{\mathbf{R}^{d}}u(y)\varphi(y-\xi)e^{-i\langle
y,\eta\rangle}\,dy\equiv V_{\varphi}u(\xi,\eta),$

where $V_{\varphi}u$ is the _short-time Fourier transform_ of the signal $u$
with respect to a window function $\varphi$. Thus, when restricting to
functions of this type, then our definition of Fourier-Lebesgue spaces
$\mathscr{F}L^{p,q}(\omega)$ agrees with the definition of modulation spaces.
To see that, we recall that for $p,q\in[1,\infty]$,
$\varphi\in\mathscr{S}(\mathbf{R}^{d})$, the weighted modulation space
$M^{p,q}(\omega)\left(\mathbf{R}^{d}\right)$ consists of all
$u\in\mathscr{S}^{\prime}\left(\mathbf{R}^{d}\right)$ such that

$\left(\int_{\mathbf{R}^{d}}\left(\int_{\mathbf{R}^{d}}\left|\omega(x,\xi)V_{\varphi}u(x,\xi)\right|^{q}dx\right)^{p/q}d\xi\right)^{1/p}<\infty,$

(with obvious modification when $p=\infty$ or $q=\infty$ ). More details on
modulation spaces can be found in [21].

###  2.2 Bochner-Sobolev space

In our main approximation result, we will measure the approximation error in
terms of a mixed-degree Sobolev space. In order to provide a profound
definition for that, we now review some properties of _Bochner-Sobolev
spaces_. We start by recalling the definition of Sobolev space.

######  Definition 2.3 (Sobolev space).

Assume that $\Omega$ is an open subset of $\mathbf{R}^{d}$, and let
$n\in\mathbf{Z}_{+}$, $1\leq q\leq\infty$. The Sobolev space $W^{n,q}(\Omega)$
consists of functions $u\in L^{q}(\Omega)$ such that for every multi-index
$\alpha$ with $|\alpha|\leqslant n$ the partial derivative
$\partial^{\alpha}u$ exists and $\partial^{\alpha}u\in L^{q}(\Omega)$. Thus

$\displaystyle W^{n,q}(\Omega):=\left\\{f\in
L^{q}(\Omega):\partial^{\alpha}f\in L^{q}(\Omega)\text{ for all
}\alpha\in\mathbf{Z}_{+}^{d}\text{ with }|\alpha|\leq n\right\\}.$

Furthermore, for $f\in W^{n,q}(\Omega)$ and $1\leq q<\infty,$ we define the
norm

$\displaystyle\|f\|_{W^{n,q}(\Omega)}:=\left(\sum_{0\leq|\alpha|\leq
n}\|\partial^{\alpha}f\|_{L^{q}(\Omega)}^{q}\right)^{1/q}$

and

$\displaystyle\|f\|_{W^{n,\infty}(\Omega)}:=\max_{0\leq|\alpha|\leq
n}\|\partial^{\alpha}f\|_{L^{\infty}(\Omega)}.$

We consider the so-called _Bochner space_ which is the natural generalisation
of Lebesgue integral to the case that the function has values in an arbitrary
Banach space. More details about Bochner spaces can be found in e.g., [27,
Section 3], [35, Chapter 10]. In this work, we consider a much simpler
situation that we fix the Banach space to be the Sobolev space in order to
avoid technicalities and focus more on the targeted result. Mainly, we deal
with functions which belong to some Sobolev space on $\mathbf{R}^{d_{1}}$
(i.e., $W^{m,p}(\mathbf{R}^{d_{1}})$) with values in another Sobolev space on
$\mathbf{R}^{d_{2}}$ (i.e., $W^{n,q}(\mathbf{R}^{d_{2}})$).

Note that the extension of our results to the general case of Bochner space
can be thought as a future work.

######  Definition 2.4 (Bochner-Sobolev space).

Let $1\leq p,q\leq\infty$, $m,n\in\mathbf{Z}_{+}$,
$U\subseteq\mathbf{R}^{d_{1}}$, and $V\subseteq\mathbf{R}^{d_{2}}$. Let
$W_{m,p}^{n,q}(U,V)$ be defined as follows

$W_{m,p}^{n,q}(U,V):=\left\\{f\in
L^{p}\left(U,W^{n,q}(V)\right):\partial_{x}^{\alpha}f\in
L^{p}\left(U,W^{n,q}(V)\right)\text{ for all }|\alpha|\leq m\right\\}$

such that

$\|f\|_{W_{m,p}^{n,q}(U,V)}:=\bigg{(}\sum_{|\alpha|\leq
m}\|\partial_{x}^{\alpha}f\|_{L^{p}\left(U,W^{n,q}(V)\right)}^{p}\bigg{)}^{1/p}<\infty,$
(2.6)

when $1\leq p,q<\infty$, with the obvious modifications when $p=\infty$ and/or
$q=\infty$.

More precisely, (2.6) is the same as

$\|f\|_{W_{m,p}^{n,q}(U,V)}:=\bigg{(}\sum_{|\alpha|\leq
m}\bigg{\|}\bigg{(}\sum_{|\beta|\leq
n}\|\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{L^{q}\left(V\right)}^{q}\bigg{)}^{1/q}\bigg{\|}_{L^{p}\left(U\right)}^{p}\bigg{)}^{1/p}.$
(2.7)

Note that if $n=m=0$, then $W_{0,p}^{0,q}(U,V)=L^{p}(U,L^{q}(V))$. Hence, we
shall write $L^{p,q}(U,V):=W_{0,p}^{0,q}(U,V)$, where $L^{p,q}(U,V)$ stands
for the Lebesgue integral with respect to $L^{q}(V)$ then $L^{p}(U)$.

###  2.3 Smoothing by convolution

In the proof of Lemma 3.1 we require $L^{1}$ integrability of the Fourier
transform characteristic functions, which is not given a-priory in general. In
order to get around this limitation, we first approximate the function by a
smoothed version via convolution and then we take the limit such that the
convolution is with the dirac-delta distribution. For that we consider the
following theory:

Following [58, Lemma 2.4 and Definition 3.1] for $\epsilon>0$ we define the
smoothing sequence

$\displaystyle\rho_{\epsilon}(x):=\frac{1}{\epsilon^{d}\|\phi\|_{L^{1}}}\phi\left(\frac{x}{\epsilon}\right)\quad\text{with}\quad\phi(x)=\exp\left(-\frac{1}{1-|x|^{2}}\right)\chi_{B_{1}(0)}(x),$

where $B_{1}(0)$ is the closed unit ball. Thus, $\rho_{\epsilon}\in
C_{c}^{\infty}$ with

$\displaystyle\|\rho_{\epsilon}\|_{L^{1}}=1\quad\text{and}\quad\|\rho_{\epsilon}\|_{L^{2}}=\left\|\frac{1}{\epsilon^{d}}\rho_{1}\left(\frac{\cdot}{\epsilon}\right)\right\|_{L^{2}}=\frac{1}{\epsilon^{d/2}}\|\rho_{1}^{2}\|_{L^{1}}^{1/2}\leq\frac{1}{\epsilon^{d/2}}\|\rho_{1}\|_{L^{1}}^{1/2}=\frac{1}{\epsilon^{d/2}}$
(2.8)

by substitution in multiple variables and using $\rho_{1}(x)<1$ for all
$x\in\mathbf{R}^{d}$. For a domain $\Omega\subset\mathbf{R}^{d}$ we define the
smoothed characteristic function of $\Omega$ as

$\displaystyle\chi_{\Omega}^{\epsilon}:=\chi_{\Omega}*\rho_{\epsilon}.$

With $\Omega_{\epsilon}:=\left\\{x\in\mathbf{R}^{d}\mid
B_{\epsilon}(x)\cap\Omega\neq\emptyset\right\\}$ being the domain extended by
a margin of width $\epsilon$ and
$\Omega_{-\epsilon}:=\left\\{x\in\mathbf{R}^{d}\mid
B_{\epsilon}(x)\subset\Omega\right\\}$ being the domain shrinked by a margin
of width $\epsilon$ we see that the smoothed characteristic funciton is
smoothly decaying over the margin
$\Omega_{\epsilon}\setminus\Omega_{-\epsilon}$ of width $2\epsilon$.

For bounded $\Omega$, we see $\chi_{\Omega}^{\epsilon}\in C^{\infty}$ by [58,
Lemma 2.3], it has bounded support by [58, Equation (2.2)] as $\Omega$ is
bounded and the support of $\rho_{\epsilon}$ is bounded, and
$\chi_{\Omega}^{\epsilon}\to_{\epsilon\to 0}\chi_{\Omega}$ by [58, Lemma 3.2].
Thus, $\chi_{\Omega}^{\epsilon}\in L^{1}$ and

$\displaystyle\mathscr{F}\left(\chi_{\Omega}^{\epsilon}\right)=\mathscr{F}\left(\chi_{\Omega}*\rho_{\epsilon}\right)=\mathscr{F}\left(\chi_{\Omega}\right)\mathscr{F}\left(\rho_{\epsilon}\right).$

This yields the upper bound on the $\mathscr{F}L^{1}$-norm

$\displaystyle\begin{split}\|\mathscr{F}\left(\chi_{\Omega}^{\epsilon}\right)\|_{L^{1}(\mathbf{R}^{d})}&\leq\|\mathscr{F}\left(\chi_{\Omega}\right)\|_{L^{2}(\mathbf{R}^{d})}\|\mathscr{F}\left(\rho_{\epsilon}\right)\|_{L^{2}(\mathbf{R}^{d})}=\|\chi_{\Omega}\|_{L^{2}(\mathbf{R}^{d}}\|\rho_{\epsilon}\|_{L^{2}(\mathbf{R}^{d})}\\\
&=|\Omega|^{1/2}\|\rho_{\epsilon}\|_{L^{2}(\mathbf{R}^{d})}<\infty\end{split}$
(2.9)

by Hölder’s inequality. For $1\leq s\leq\infty$ we further get the upper bound
on the $\mathscr{F}L^{s}$-norm

$\displaystyle\begin{split}\|\mathscr{F}\left(\chi_{\Omega}^{\epsilon}\right)\|_{L^{s}(\mathbf{R}^{d})}&\leq\|\mathscr{F}\left(\chi_{\Omega}\right)\|_{L^{s}(\mathbf{R}^{d})}\|\mathscr{F}\left(\rho_{\epsilon}\right)\|_{L^{\infty}(\mathbf{R}^{d})}\\\
&\leq\|\mathscr{F}\left(\chi_{\Omega}\right)\|_{L^{s}(\mathbf{R}^{d})}\|\rho_{\epsilon}\|_{L^{1}(\mathbf{R}^{d})}=\|\mathscr{F}\left(\chi_{\Omega}\right)\|_{L^{s}(\mathbf{R}^{d})}\end{split}$
(2.10)

by taking the supremum over $\mathscr{F}\left(\rho_{\epsilon}\right)$ and
using the $L^{1}$-norm as upper bound on the Fourier Transform (see [26, page
294]). Note, however, that $\mathscr{F}\left(\chi_{\Omega}\right)$ is not
necessarily in $L^{s}(\mathbf{R}^{d})$ for $s<2$, which will be discussed in
Section 3.1.1. Nevertheless, Eq. 2.9 shows that
$\mathscr{F}\left(\chi_{\Omega}^{\epsilon}\right)\in L^{1}$ for all
$\epsilon>0$.

###  2.4 Variation Space and Approximation Rate

In this section we provide a brief overview over tools that are necessary to
establish our approximation bound in Section 3.3. Namely, we will review
variation spaces and the resulting approximation rates for these spaces, all
of that already with the focus on shallow neural networks with two blocks of
variables. The extension to multiple blocks is straightforward, furthermore,
the one block of variables is a particular special case of our analysis. For a
general overview over variation spaces we refer the interested reader to [14,
32, 52, 53].

The definition of a variation space is based on some dictionary that is a
subset of some function-space. In our work, we mostly rely on the theory of
[52], which means that we define a dictionary as
$\mathbb{D}\subseteq\mathcal{B}$ with some Banach space $\mathcal{B}$ (other
works such as [53] considers the stronger assumption that the dictionary has
to be in some Hilbert space). In order to approximate some target functions
via the dictionary, we will consider the linear combinations of
$N\in\mathbf{N}$ elements of this dictionary and bound the weight of the
combination with respect to $\ell_{1}$ by some constant $M>0$. That is, we
consider the set

$\displaystyle\Sigma_{N,M}(\mathbb{D}):=\left\\{\sum_{j=1}^{N}a_{j}h_{j}:h_{j}\in\mathbb{D},\sum_{j=1}^{N}|a_{j}|\leq
M\right\\}.$

The variation space $\mathcal{K}(\mathbb{D})$ associated with this dictionary
can roughly be seen as the set of functions that can be realized via an
infinite linear combination. It is defined via the variation norm as follows:

######  Definition 2.5.

Let $\mathcal{B}$ be a Banach space and $\mathbb{D}\subseteq\mathcal{B}$ be a
dictionary. Then for $f\in\mathcal{B}$, the variation norm of $\mathbb{D}$ is
defined as

$\displaystyle\|f\|_{\mathcal{K}(\mathbb{D})}$
$\displaystyle:=\inf\\{c>0:f/c\in\overline{\operatorname{conv}(\pm\mathbb{D})}\\}$
were, $\overline{\operatorname{conv}(\pm\mathbb{D})}$ is the closure of the
convex hull of $\mathbb{D}\cup(-\mathbb{D})$. The corresponding variation
space is then the set of functions with finite variation norm
$\displaystyle\mathcal{K}(\mathbb{D})$
$\displaystyle:=\\{f\in\mathcal{B}:\|f\|_{\mathcal{K}(\mathbb{D})}<\infty\\}.$

The main focus of this work is on shallow networks with two blocks of
variables. That is, we consider an activation function
$\sigma:\mathbf{R}^{2}\to\mathbf{R}$ and the parameter spaces
$\Lambda=\Lambda_{\xi}\times\Lambda_{b}$, where $\Lambda_{\xi}$ is the set of
admissible input weights and $\Lambda_{b}$ is the set of admissible input
biases. Every single neuron in the neural network is then composed of a
parametrized affine function

$\displaystyle
T(\cdot,\cdot\cdot;\xi,b):\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}\to\mathbf{R}^{2}\qquad\text{with}\qquad
T(x,y;\xi,b):=(\xi_{1}\cdot x+b_{1},\xi_{2}\cdot y+b_{2})$

with the weight $\xi=(\xi_{1},\xi_{2})\in\Lambda_{\xi}$ and bias
$b=(b_{1},b_{2})\in\mathbf{R}^{2}$ and the activation function
$\sigma:\mathbf{R}^{2}\to\mathbf{R}$. In this setting, we consider the
dictionary

$\displaystyle\mathbb{D}_{\sigma}^{d_{1},d_{2}}:=\\{\sigma(T(\cdot,\cdot\cdot;\xi,b))\in\mathcal{B}:(\xi_{1},\xi_{2})\in\Lambda_{\xi},(b_{1},b_{2})\in\Lambda_{b}\\}.$

Throughout our work, we also consider the dictionary with a scaled activation
function. That is, for some strictly positive function
$\phi:\Lambda_{\xi}\times\Lambda_{b}\to(0,\infty)$ we consider

$\displaystyle\mathbb{D}_{\phi,\sigma}^{d_{1},d_{2}}:=\\{\phi(\xi,b)\sigma(T(\cdot,\cdot\cdot,\xi,b))\in\mathcal{B}:(\xi_{1},\xi_{2})\in\Lambda_{\xi},(b_{1},b_{2})\in\Lambda_{b}\\}.$

A central element of our main approximation result is an upper bound on the
approximation rate for Banach spaces of Rademacher type $2$, which is
typically attributed to Maurey’s (see [46, 4, 52]. Before stating Maurey’s
result, we introduce the definition for the general Rademacher type $p$:

######  Definition 2.6 (Rademacher Type $p$ [2, Definition 6.2.10]).

A Banach Space $\mathcal{B}$ is said to have Rademacher type $p$ for some
$1\leq p\leq 2$ if there is a constant $C_{p,\mathcal{B}}$ such that for every
finite set of vectors $\\{x_{i}\\}_{i=1}^{n}$ in $\mathcal{B}$,

$\displaystyle\left(\mathbb{E}\left\\{\left\|\sum_{i=1}^{n}\varepsilon_{i}x_{i}\right\|_{\mathcal{B}}^{p}\right\\}\right)^{1/p}\leq
C_{p,\mathcal{B}}\left(\sum_{i=1}^{n}\left\|x_{i}\right\|_{\mathcal{B}}^{p}\right)^{1/p}.$

In short, we say that $\mathcal{B}$ is a type-$p$ Banach space.

A version of Maurey’s approximation result that is already tailored to
functions in the variation space of a dictionary is stated in [52] as follows:

######  Proposition 2.7 (Approximation Rate in Type-2 Banach Spaces).

Let $\mathcal{B}$ be a type-2 Banach space and $\mathbb{D}\subset\mathcal{B}$
be a dictionary with
$K_{\mathbb{D}}:=\sup_{d\in\mathbb{D}}\|d\|_{\mathcal{B}}<\infty$. Then for
$f\in\mathcal{K}(\mathbb{D})$, we have

$\displaystyle\inf_{f_{N}\in\Sigma_{N,M_{f}}(\mathbb{D})}\left\|f-f_{N}\right\|_{\mathcal{B}}\leq
4C_{2,\mathcal{B}}K_{\mathbb{D}}\|f\|_{\mathcal{K}(\mathbb{D})}N^{-\frac{1}{2}}$

with $M_{f}=\|f\|_{\mathcal{K}(\mathbb{D})}$.

A discussion regarding possible extensions of Maurey’s result can be found in
[14, Section 8].

The prerequisite for the approximation result is that the function $f$ is in
the variation space $\mathcal{K}(\mathbb{D})$ of the dictionary $\mathbb{D}$.
A sufficient condition for this can be obtained by a slight modification of
[53, Lemma 3]. Namely, we ask for boundedness instead of compactness, consider
a Banach space instead of a Hilbert space, and treat only the implication in
one direction. The precise formulation of this modification is as follows:

######  Proposition 2.8 ([53, Lemma 3]).

Let $\mathcal{B}$ be a Banach space and suppose that
$\mathbb{D}\subset\mathcal{B}$ is bounded. Then $f\in\mathcal{K}(\mathbb{D})$
if there exists a Borel measure $\mu$ on $\mathbb{D}$ such that

$\displaystyle f=\int_{\mathbb{D}}i_{\mathbb{D}\to\mathcal{B}}\,d\mu.$

Moreover,

$\displaystyle\|f\|_{\mathcal{K}(\mathbb{D})}=\inf\left\\{\|\mu\|:f=\int_{\mathbb{D}}i_{\mathbb{D}\to\mathcal{B}}\,d\mu\right\\},$

where the infimum is taken over all Borel measures $\mu$ defined on
$\mathbb{D}$, and $\|\mu\|$ is the total variation111The interested reader can
check [13] for more details regarding measure theory. of $\mu$.

##  3 Convergence Rates with Respect to Bochner-Sobolev Norms

In this section, we extend the single-block approximation results for
functions in the Barron space with Hilbert-Sobolev error measure of [50]
towards our setting with functions in the anisotropic weighted Fourier-
Lebesgue spaces and the Bochner-Sobolev norm as error measure. The goal of
these approximation results is to show that functions that lie in the
anisotropic weighted Fourier-Lebesgue spaces can be approximated well by
shallow neural networks. This is done by providing an upper bound on the
approximation error that is only dependent on the number of neurons, the
domain of the function and on the degree of the chosen norms.

###  3.1 Inclusion of Fourier-Lebesgue Spaces in High-Degree Bochner-Sobolev
Spaces

The first step in the approximation result is to show that functions in the
anisotropic weighted Fourier-Lebesgue spaces lie in the Bochner-Sobolev space.

######  Lemma 3.1 (Inclusion in high-degree Bochner-Sobolev Space).

Let $1\leq s_{i},t_{i}\leq 2\leq p_{1}\leq p_{2}\leq\infty$ and
$d_{i},n_{i}\in\mathbf{Z}_{+}$ such that
$\frac{1}{s_{i}}+\frac{1}{t_{i}}+\frac{1}{p_{i}}=2$, for $i\in\\{1,2\\}$. Let
$\omega(x,y)$ be a polynomially moderated weight function defined on
$\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$, elliptic with respect to
$\langle x\rangle^{n_{1}}\langle y\rangle^{n_{2}}$, and
$U\subset\mathbf{R}^{d_{1}}$, $V\subset\mathbf{R}^{d_{2}}$ be bounded and
measurable with non-empty interior. Let
$f\in\mathscr{F}L^{t_{1},t_{2}}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})$
then we have

$\displaystyle\|f\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}\leq
C_{n_{1},n_{2},p_{1},p_{2}}\|{\chi_{U}}\|_{\mathscr{F}L^{s_{1}}(\mathbf{R}^{d_{1}})}\|{\chi_{V}}\|_{\mathscr{F}L^{s_{2}}(\mathbf{R}^{d_{2}})}\|f\|_{\mathscr{F}L^{t_{1},t_{2}}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})},$
(3.1)

where $C_{n_{1},n_{2},p_{1},p_{2}}>0$ depends only on $n_{1},n_{2}$, $p_{1}$,
and $p_{2}$.

Note that the ellipticity assumption on the weight $\omega$ results in
$1/\omega\in L_{1}$ as

$\displaystyle 0\leq\frac{1}{\omega(x,y)}\leq\frac{1}{\langle
x\rangle^{n_{1}}\langle y\rangle^{n_{2}}}\in
L^{1}(\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}})$

whenever $n_{1}>d_{1}$ and $n_{2}>d_{2}$.

In the proof of this Lemma and later in Lemma 3.7, we use the following
relation between the Sobolev-norm and the anisotropic Lebesgue norm, which is
due to the monotonicity of $\ell_{p}$-norms
($\|\cdot\|_{p_{1}},\|\cdot\|_{p_{2}}\leq\|\cdot\|_{1}$ for $p_{1},p_{2}\geq
1$) and the Minkowski inequality:

$\displaystyle\begin{split}\|f\|_{W_{m,p_{1}}^{n,p_{2}}(U,V)}&=\left(\sum_{|\alpha|\leq
m}\|\|\partial_{x}^{\alpha}f(x,\cdot)\|_{W^{n,p_{2}(V)}}\|_{L^{p_{1}}(U)}^{p_{1}}\right)^{\frac{1}{p_{1}}}\\\
&=\left(\sum_{|\alpha|\leq m}\left\|\left(\sum_{|\beta|\leq
n}\|\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{L^{p_{2}}(V)}^{p_{2}}\right)^{\frac{1}{p_{2}}}\right\|_{L^{p_{1}}(U)}^{p_{1}}\right)^{\frac{1}{p_{1}}}\\\
&\leq\sum_{|\alpha|\leq m}\sum_{|\beta|\leq
n}\|\|\partial_{y}^{\beta}\partial_{x}^{\alpha}f(x,y)\|_{L^{p_{2}}(V)}\|_{L^{p_{1}}(U)}\\\
&=\sum_{|\alpha|\leq m}\sum_{|\beta|\leq
n}\|\chi_{U}\chi_{V}\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{L^{p_{1},p_{2}}}.\end{split}$
(3.2)

###### Proof.

For a polynomially moderated weight $\omega$ we limit to the Schwartz class
i.e., let $f\in\mathscr{S}(\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}})$. Then
we have by Eq. 3.2 and Proposition A.2

$\displaystyle\begin{split}\|f\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}&\leq\sum_{|\alpha|\leq
n_{1}}\sum_{|\beta|\leq
n_{2}}\|\chi_{U}\chi_{V}\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{L^{p_{1},p_{2}}}\\\
&=\lim_{\epsilon_{1}\to 0}\lim_{\epsilon_{2}\to 0}\sum_{|\alpha|\leq
n_{1}}\sum_{|\beta|\leq
n_{2}}\|\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{L^{p_{1},p_{2}}}.\end{split}$
(3.3)

With $f\in\mathscr{S}(\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}})$ we have
$\partial_{x}^{\alpha}\partial_{y}^{\beta}f\in L^{1}$ and
$\partial_{x}^{\alpha}\partial_{y}^{\beta}f\in\mathscr{F}L^{1}$ for any
$\alpha\in\mathbf{Z}_{+}^{d_{1}},\beta\in\mathbf{Z}_{+}^{d_{2}}$. Further, for
any $\epsilon_{1},\epsilon_{2}>0$ we have
$\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}\in
L^{1}(\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}})$ and
$\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}\in\mathscr{F}L^{1}(\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})$
by construction of the smoothed characteristic function. Thus, by [26, Section
13.B, Page 316],
$\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}\partial_{y}^{\beta}\partial_{x}^{\alpha}f\in
L^{1}(\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}})$ and

$\displaystyle\mathscr{F}(\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}\partial_{y}^{\beta}\partial_{x}^{\alpha}f)=\mathscr{F}(\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}})*\mathscr{F}(\partial_{y}^{\beta}\partial_{x}^{\alpha}f).$
(3.4)

Young’s convolution inequality then shows

$\displaystyle\|\mathscr{F}(\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}\partial_{y}^{\beta}\partial_{x}^{\alpha}f)\|_{L^{1}}\leq\|\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}\|_{\mathscr{F}L^{1}}\|\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{\mathscr{F}L^{1}}<\infty.$

Hence, the inverse Fourier transform $\mathscr{F}^{-1}$ exists and

$\displaystyle\|\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{L^{p_{1},p_{2}}}=\|\mathscr{F}^{-1}\left[\mathscr{F}\left(\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}{\partial_{y}^{\beta}\partial_{x}^{\alpha}f}\right)\right]\|_{L{p_{1},p_{2}}}.$

Therefore, using the Hausdorff-Young inequality for mixed Lebesgue spaces [6,
Section 12] for $2\leq p_{1}\leq p_{2}\leq\infty$, where
$\frac{1}{p_{i}}+\frac{1}{q_{i}}=1$ (thus, $1\leq q_{2}\leq q_{1}\leq 2$),
there exists a constant $C_{p_{1},p_{2}}>0$ depending on $p_{1}$ and $p_{2}$
such that

$\displaystyle\|\mathscr{F}^{-1}\left[\mathscr{F}\left(\
\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}{\partial_{y}^{\beta}\partial_{x}^{\alpha}f}\right)\right]\|_{L^{p_{1},p_{2}}}\leq
C_{p_{1},p_{2}}\|\mathscr{F}\left(\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}{\partial_{y}^{\beta}\partial_{x}^{\alpha}f}\right)\|_{L^{q_{1},q_{2}}}.$
(3.5)

We have seen in Eq. 3.4 that the convolution theorem applies and with Young’s
convolution inequality we get

$\displaystyle\|\mathscr{F}\left(\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}{\partial_{y}^{\beta}\partial_{x}^{\alpha}f}\right)\|_{L^{q_{1},q_{2}}}$
$\displaystyle=\|\mathscr{F}\left(\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}\right)*\mathscr{F}\left({\partial_{y}^{\beta}\partial_{x}^{\alpha}f}\right)\|_{L^{q_{1},q_{2}}}$
$\displaystyle\leq\|\mathscr{F}\left({\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}}\right)\|_{L^{s_{1},s_{2}}}\|\mathscr{F}{\left(\partial_{y}^{\beta}\partial_{x}^{\alpha}f\right)}\|_{L^{t_{1},t_{2}}}.$

Note that the assumption $\frac{1}{s_{i}}+\frac{1}{t_{i}}+\frac{1}{p_{i}}=2$
implies $\frac{1}{s_{i}}+\frac{1}{t_{i}}=1+\frac{1}{q_{i}}$ with $1\leq
q_{i},s_{i},t_{i}\leq\infty$, which fulfills the condition of Young’s
convolution inequality.

We can now split the two-block Fourier transform of the characteristic
functions into the product of the two partial Fourier transforms and
subsequently into a product of norms. With the bound Eq. 2.10 on the smoothed
characteristic function we get

$\displaystyle\|\mathscr{F}\left(\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}{\partial_{y}^{\beta}\partial_{x}^{\alpha}f}\right)\|_{L^{q_{1},q_{2}}}$
$\displaystyle\leq\|\mathscr{F}_{1}({\chi_{U}})\|_{L^{s_{1}}(\mathbf{R}^{d_{1}})}\|\mathscr{F}_{2}({\chi_{V}})\|_{L^{s_{2}}(\mathbf{R}^{d_{2}})}\|\mathscr{F}{\left(\partial_{y}^{\beta}\partial_{x}^{\alpha}f\right)}\|_{L^{t_{1},t_{2}}}.$

Combining this with the Hausdorff-Young inequality Eq. 3.5 leads to an upper
bound on Eq. 3.3 which is independent of $\epsilon_{1}$ and $\epsilon_{2}$ and
therefore holds in the limit $\epsilon_{1},\epsilon_{2}\to 0$. All together,
this is

$\displaystyle\|f\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}$ $\displaystyle\leq
C_{p_{1},p_{2}}\|{\chi_{U}}\|_{\mathscr{F}L^{s_{1}}(\mathbf{R}^{d_{1}})}\|{\chi_{V}}\|_{\mathscr{F}L^{s_{2}}(\mathbf{R}^{d_{2}})}\sum_{\begin{subarray}{c}|\alpha|\leq
n_{1}\\\ |\beta|\leq
n_{2}\end{subarray}}\|\mathscr{F}{\left(\partial_{y}^{\beta}\partial_{x}^{\alpha}f\right)}\|_{L^{t_{1},t_{2}}}$
$\displaystyle\leq
C_{p_{1},p_{2}}\|{\chi_{U}}\|_{\mathscr{F}L^{s_{1}}(\mathbf{R}^{d_{1}})}\|{\chi_{V}}\|_{\mathscr{F}L^{s_{2}}(\mathbf{R}^{d_{2}})}\sum_{\begin{subarray}{c}|\alpha|\leq
n_{1}\\\ |\beta|\leq
n_{2}\end{subarray}}\||\cdot\cdot|^{|\beta|}|\cdot|^{|\alpha|}\widehat{f}(\,\cdot\,,\,\cdot\cdot)\|_{L^{t_{1},t_{2}}}$
$\displaystyle\leq
C_{n_{1},n_{2},p_{1},p_{2}}\|{\chi_{U}}\|_{\mathscr{F}L^{s_{1}}(\mathbf{R}^{d_{1}})}\|{\chi_{V}}\|_{\mathscr{F}L^{s_{2}}(\mathbf{R}^{d_{2}})}\||\cdot\cdot|^{n_{1}}|\cdot|^{n_{2}}\widehat{f}(\,\cdot\,,\,\cdot\cdot)\|_{L^{t_{1},t_{2}}},$

where $C_{n_{1},n_{2},p_{1},p_{2}}$ is a non-negative constant depends only on
$n_{1}$, $n_{2}$, $p_{1}$, and $p_{2}$ which comes from the sum over all the
multi-indices $\alpha$ and $\beta$.

Finally,

$\displaystyle\|f\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}$ $\displaystyle\leq
C_{n_{1},n_{2},p_{1},p_{2}}\|{\chi_{U}}\|_{\mathscr{F}L^{s_{1}}(\mathbf{R}^{d_{1}})}\|{\chi_{V}}\|_{\mathscr{F}L^{s_{2}}(\mathbf{R}^{d_{2}})}\||\cdot\cdot|^{n_{1}}|\cdot|^{n_{2}}\widehat{f}(\,\cdot\,,\,\cdot\cdot)\|_{L^{t_{1},t_{2}}}$
$\displaystyle\leq
C_{n_{1},n_{2},p_{1},p_{2}}\|{\chi_{U}}\|_{\mathscr{F}L^{s_{1}}(\mathbf{R}^{d_{1}})}\|{\chi_{V}}\|_{\mathscr{F}L^{s_{2}}(\mathbf{R}^{d_{2}})}\|\omega\widehat{f}\|_{L^{t_{1},t_{2}}}$
$\displaystyle=C_{n_{1},n_{2},p_{1},p_{2}}\|{\chi_{U}}\|_{\mathscr{F}L^{s_{1}}(\mathbf{R}^{d_{1}})}\|{\chi_{V}}\|_{\mathscr{F}L^{s_{2}}(\mathbf{R}^{d_{2}})}\|f\|_{\mathscr{F}L^{t_{1},t_{2}}(\omega)}.$

For polynomially controlled weight $\omega$ the Schwartz class is dense in
$\mathscr{F}L^{t_{1},t_{2}}(\omega)$ (see [24, Theorem 10.1.7]), therefore,
the statement holds in the limit, i.e, for any
$f\in\mathscr{F}L^{t_{1},t_{2}}(\omega)$. ∎

The proof of Lemma 3.1 uses a density argument with the Schwartz class; in
order to avoid this type of argument, it is sufficient to consider the initial
restriction that $f$ belongs to
$W_{n_{1},\infty}^{n_{2},\infty}(\omega;U,V))\cap\mathscr{F}L^{1}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}}))$
with $n_{1}>d_{1},n_{2}>d_{2}$ where
$W_{n_{1},\infty}^{n_{2},\infty}(\omega;U,V))$ stands for the weighted
Bochner-Sobolev space 222The weighted Bochner-Sobolev space
$W_{n_{1},p_{1}}^{n_{2},p_{2}}(\omega,U,V)$ is the class of functions $f$ such
that $\omega\partial_{1}^{\alpha}\partial_{2}^{\beta}f\in
L^{p_{1},p_{2}}(U,V)$ for all $\alpha\in\mathbf{Z}_{+}^{d_{1}}$ with
$|\alpha|\leq n_{1}$ and $\beta\in\mathbf{Z}_{+}^{d_{2}}$ with $|\beta|\leq
n_{2}$. or $u\in
W_{n_{1},1}^{n_{2},1}(U,V))\cap\mathscr{F}L^{1}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}}))$
for any $n_{i}\in\mathbf{Z}_{+}$.

We immediately obtain the results for the spectral Barron space and Hilbert-
Sobolev error from [50, Lemma 2] as special case with a single block,
$p_{1}=s_{1}=2$, and $t_{1}=1$. The following corollary first shows the
analogous result for two blocks, then the result for general parameters on a
single block, and finally the existing result for a single block.

######  Corollary 3.2.

  * •

Under the assumptions of Lemma 3.1 with $p_{i}=s_{i}=2$ and $t_{i}=1$
Plancherel’s theorem results in

$\|f\|_{W_{n_{1},2}^{n_{2},2}(U,V)}\leq
C_{n_{1},n_{2},2,2}|U|^{1/2}|V|^{1/2}\|f\|_{\mathscr{F}L^{1}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})},$

where $C_{n_{1},n_{2},2,2}>0$ depends only on $n_{1}$ and $n_{2}$.

  * •

For functions $f\in\mathscr{F}L^{t_{2}}(\omega;\mathbf{R}^{d_{1}})$ with only
one block of variables, we define $\tilde{f}(x,y):=f(y)$,
$\tilde{\omega}(x,y):=\omega(y)$, and $U=(0,1)$. Then
$\tilde{f}\in\mathscr{F}L^{2,t_{2}}(\tilde{\omega})$,

$\displaystyle\|f\|_{W^{n_{2},p_{2}}(U)}=\|\tilde{f}\|_{W_{0,2}^{n_{2},p_{2}}(U,V)},\quad\text{and}\quad\|f\|_{\mathscr{F}L^{t_{2}}(\omega;\mathbf{R}^{d_{2}})}=c\|\tilde{f}\|_{\mathscr{F}L^{\frac{4}{3},t_{2}}(\tilde{\omega};\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})}$

for some $c>0$. The inequality then reads (with $s_{1}=t_{1}=\frac{4}{3}$)

$\displaystyle\|f\|_{W^{n_{2},p_{2}}(V)}\leq
C_{n_{2},p_{2}}\|\chi_{V}\|_{\mathscr{F}L^{s_{2}}(\mathbf{R}^{d_{2}})}\|f\|_{\mathscr{F}L^{t_{2}}(\omega;\mathbf{R}^{d_{2}})},$

where $C_{n_{2},p_{2}}>0$ depends only on $n_{1}$ and $p_{1}$.

  * •

Similar to the first case, we can specify this further to $p_{2}=s_{2}=2$,
$t_{2}=1$, and $\omega(\xi_{2})=\langle\xi_{2}\rangle^{n_{2}}$, in which case
we obtain the already known result

$\displaystyle\|f\|_{W^{n_{2},2}(V)}\leq
C_{n_{2},2}|V|^{1/2}\|f\|_{\mathscr{F}L^{1}(\langle\cdot\rangle^{n_{2}};\mathbf{R}^{d_{1}})},$

for the spectral Barron space.

####  3.1.1 Structure of the Domain and Admissible Degrees

In the proof of Lemma 3.1, we construct an upper bound on the smoothed
characteristic functions. At a first glance, it might seem that this is an
unnecessary detour; one could simply add the assumption that
$\chi_{\Omega_{i}}\in\mathscr{F}L^{1}(\mathbf{R}^{d_{i}})$ for $\Omega_{1}=U$
and $\Omega_{2}=V$ and thereby trivially get
$\chi_{\Omega_{i}}\in\mathscr{F}L^{s_{i}}(\mathbf{R}^{d_{i}})$ as
$|\widehat{\chi_{\Omega_{i}}}|\leq\|\chi_{\Omega_{i}}\|_{L^{1}}=|\Omega_{i}|$
and

$\displaystyle\|\chi_{\Omega_{i}}\|_{\mathscr{F}L^{s_{i}}}^{s_{i}}=\|\widehat{\chi_{\Omega_{i}}}^{s_{i}}\|_{L^{1}}=|\Omega_{i}|^{s_{i}}\left\|\frac{\widehat{\chi_{\Omega_{i}}}^{s_{i}}}{|\Omega_{i}|^{s_{i}}}\right\|_{L^{1}}\leq|\Omega_{i}|^{s_{i}}\left\|\frac{\widehat{\chi_{\Omega_{i}}}}{|\Omega_{i}|}\right\|_{L^{1}}=|\Omega_{i}|^{s_{i}-1}\|\chi_{\Omega_{i}}\|_{\mathscr{F}L^{1}}.$

However, doing so, would add an unnecessary strong assumption. In order to
understand this, we consider the following very simple example:

######  Example 3.3.

Let $\Omega=[-1/2,1/2]$. Then,

$\displaystyle\mathscr{F}(\chi_{\Omega})(\xi)=\frac{1}{\sqrt{2\pi}}\operatorname{sinc}(\xi/2),$

which is known to be non-integrable and therefore,
$\chi_{\Omega}\notin\mathscr{F}L^{1}$. However, the $\operatorname{sinc}$ is
integrable with respect to the $L^{s}$ norm for any $s>1$. Thus, for $s>1$,
$\chi_{\Omega}\in\mathscr{F}L^{s}$.

This example directly extends to bounded hyperrectangles and a finite number
of disjoint unions thereof. It shows that requiring
$\chi_{\Omega}\in\mathscr{F}L^{1}$ would definitely exclude this simple case
from the use-cases of Lemma 3.1 while it is admissible with the provided proof
for any $s>1$. However, in general there is no guarantee that $s>1$ is
sufficient or whether it is even necessary for
$\chi_{\Omega}\in\mathscr{F}L^{s}$.

For some general measurable and bounded domain $\Omega\subset\mathbf{R}^{d}$
the integrability with respect to the $L^{2}$ norm is trivially provided by
Plancherel’s theorem as

$\displaystyle\|\chi_{\Omega}\|_{\mathscr{F}L^{2}}=\|\chi_{\Omega}\|_{L^{2}}=|\Omega|^{\frac{1}{2}}$

and for $s>2$ [31] suggests to verify the itegrability via the Hausdorff-Young
inequality. For the case $1\leq s<2$, the integrability of
$\widehat{\chi_{\Omega}}$ is its own field of research (see [33, 31] and
references therein) and the results of [33, 31] show that the integrability
depends on the geometric properties of $\Omega$. More specifically, they show
that it depends on the geometry of the boundary of the domain.

[33] mentions that for a ball $\mathcal{B}$ in $\mathbf{R}^{d}$ the condition

$\displaystyle s>\frac{2d}{d+1}$ (3.7)

is necessary and sufficient so that $\chi_{B}\in\mathscr{F}L^{s}$ [33]. That
is, despite having a smooth surface in $C^{\infty}$, the constraint becomes
stronger for increasing $d$, whereas the constraint for a hyperrectangle does
not change, even though its surface is not differentiable everywhere. In [33]
Lebedev further elaborates on that and shows that Eq. 3.7 is sufficient for
domains with a $C^{1}$ boundary and on top of that becomes necessary when the
boundary is $C^{2}$.

Note, that the results up to now did not include any case, where $s=1$ is
allowed. This case is treated by [31] with the following proposition:

######  Proposition 3.4 (Proposition 1.2 in [31]).

Let $1\leq s\leq 2$ and $\Omega$ be a bounded domain in $\mathbf{R}^{d}$.
Then,

$\|\chi_{\Omega}\|_{\mathscr{F}L^{s}}=\|\widehat{\chi_{\Omega}}\|_{L^{s}}\lesssim|\Omega|+\left(\int_{0}^{1}\delta^{-d\left(1-\frac{s}{2}\right)}\left|(\partial\Omega)_{\delta}\right|^{\frac{s}{2}}\frac{d\delta}{\delta}\right)^{1/s},$

with

$(\partial\Omega)_{\delta}=\\{x:\operatorname{dist}(x,\partial\Omega)<\delta\\}.$
(3.8)

Here, $\operatorname{dist}(x,\partial\Omega)$ is defined as
$\inf_{\eta\in\partial\Omega}\|\eta-x\|$ and $\|\cdot\|$ is the Euclidean
norm.

Combining this theory with Lemma 3.1 yields the following proposition.

######  Proposition 3.5.

Under the assumptions of Lemma 3.1 let

$\displaystyle\tau_{i}:=\left(\int_{0}^{1}\delta^{-d_{i}\left(1-\frac{s_{i}}{2}\right)}\left|(\partial
E_{i})_{\delta}\right|^{\frac{s_{i}}{2}}\frac{d\delta}{\delta}\right)^{1/s_{i}},$

with $E_{1}=U$ and $E_{2}=V$. Then, for any
$f\in\mathscr{F}L^{t_{1},t_{2}}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})$
we have

$\|f\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}\leq
C_{n_{1},n_{2},p_{1},p_{2}}\left(|U|+\tau_{1}\right)\left(|V|+\tau_{2}\right)\|f\|_{\mathscr{F}L^{t_{1},t_{2}}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})},$
(3.9)

where $C_{n_{1},n_{2},p_{1},p_{2}}>0$ depends only on $n_{1},n_{2}$, $p_{1}$,
and $p_{2}$.

###### Proof.

This proposition is a direct consequence of Lemma 3.1 and Proposition 3.4 ∎

The lower bounds on the possible choices for the integrability exponents
$s_{i}$ arising from the structure of the domain translate into an upper bound
on the choice of the integrability exponents $p_{i}$ of the error measure.
This is due to the condition $s_{i}$, $t_{i}$, and $p_{i}$ in Lemma 3.1; the
following remark sheds light on this observation.

######  Remark 3.6 (Admissible Degrees).

Let $s_{i}$ be lower bounded by $\bar{s}_{i}$ (i.e., $1\leq\bar{s}<s\leq 2$)
and $1\leq t_{i}\leq 2\leq p_{i}$ such that
$\frac{1}{s_{i}}+\frac{1}{t_{i}}=2-\frac{1}{q_{i}}$. Then there is a hard
upper bound on the choice of $p_{i}$ given by

$\displaystyle
p_{i}\leq\frac{1}{2-\frac{1}{s_{i}}-\frac{1}{t_{i}}}<\frac{1}{2-\frac{1}{\bar{s}_{i}}-\frac{1}{t_{i}}}\leq\frac{1}{1-\frac{1}{\bar{s}_{i}}}=\bar{s}_{i}^{\prime},$

where the last inequality can be attaind with equality when choosing
$t_{i}=1$.

###  3.2 Inclusion of Fourier-Lebesgue Spaces in Low-Degree Bochner-Sobolev
Spaces

In Lemma 3.1, we covered the inclusion of anisotropic weighted Fourier-
Lebesgue spaces in Bochner-Sobolev spaces with high-degree. This is especially
interesting because of these spaces are type-2 Banach spaces. For sake of
completeness, we also study the conjugate case of low-degree Bochner-Sobolev
spaces in the following lemma:

######  Lemma 3.7 (Inclusion in low-degree Bochner-Sobolev Space).

Let $1\leq p_{i},t_{i}\leq 2$, $d_{i},n_{i}\in\mathbf{N}$ for $i\in\\{1,2\\}$
with $t_{1}\geq t_{2}$. Let $\omega(x,y)$ be a polynomially moderated weight
function defined on $\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$, elliptic
with respect to $\langle x\rangle^{n_{1}}\langle y\rangle^{n_{2}}$, and
$U\subset\mathbf{R}^{d_{1}}$, $V\subset\mathbf{R}^{d_{2}}$ are bounded
domains. Let
$f\in\mathscr{F}L^{t_{1},t_{2}}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})$,
then

$\displaystyle\|f\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}\leq
C_{n_{1},n_{2},p_{1},p_{2}}|U|^{\frac{1}{p_{1}}+\frac{1}{t_{1}}-1}|V|^{\frac{1}{p_{2}}+\frac{1}{t_{2}}-1}\|f\|_{\mathscr{F}L^{t_{1},t_{2}}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})},$
(3.10)

where $C_{n_{1},n_{2},p_{1},p_{2}}>0$ depends only on $n_{1},n_{2}$, $p_{1}$
and $p_{2}$.

###### Proof.

Similar to the proof of Lemma 3.1 we consider a polynomially moderated weight
$\omega$ and limit to the Schwartz class i.e., let
$f\in\mathscr{S}(\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}})$. By Eq. 3.2 we
have

$\displaystyle\|f\|_{W_{m,p_{1}}^{n,p_{2}}(U,V)}$
$\displaystyle\leq\sum_{|\alpha|\leq m}\sum_{|\beta|\leq
n}\|\chi_{U}\chi_{V}\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{L^{p_{1},p_{2}}}.$

and use the monotonicity of $L^{p}$ norms over bounded domains for $p_{i}\leq
r_{i}$ ($i\in\\{1,2\\}$) (see Proposition A.1). That is, for $i\in\\{1,2\\}$
choose $r_{i}\geq p_{i}$ and set $s_{i}:=\frac{r_{i}}{r_{i}-p_{i}}$
($s_{i}=\infty$ if $r_{i}=p_{i}$). Then, Proposition A.1 guarantees that

$\displaystyle\|\chi_{U}\chi_{V}\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{L^{p_{1},p_{2}}}$
$\displaystyle\leq|U|^{\frac{1}{p_{1}s_{1}}}|V|^{\frac{1}{p_{2}s_{2}}}\|\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{L^{r_{1},r_{2}}}.$

With $f\in\mathscr{S}$, we also have
$\partial_{y}^{\beta}\partial_{x}^{\alpha}f\in L^{1}$ for any
$\alpha\in\mathbf{Z}_{+}^{d_{1}}$ and $\beta\in\mathbf{Z}_{+}^{d_{2}}$, and
$f\in\mathscr{F}L^{1}(\omega)$. It follows by straightforward computations
that $\mathscr{F}(\partial_{y}^{\beta}\partial_{x}^{\alpha}f)\in L^{1}$ for
any $|\alpha|\leq m$ and $|\beta|\leq n$. Consequently, for any
$(x,y)\in\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$,
$\alpha\in\mathbf{Z}_{+}^{d_{1}}$ and $\beta\in\mathbf{Z}_{+}^{d_{2}}$ such
that $|\alpha|\leq m$ and $|\beta|\leq n$, we can write

$\partial_{y}^{\beta}\partial_{x}^{\alpha}f(x,y)\equiv\mathscr{F}^{-1}\left(\mathscr{F}(\partial_{y}^{\beta}\partial_{x}^{\alpha}f)(\xi,\eta)\right)(x,y).$

The Hausdorff-Young inequality for mixed Lebesgue spaces [6, Section 12] for
$2\geq t_{1}\geq t_{2}\geq 1$, then yields

$\displaystyle\|\partial_{y}^{\beta}\partial_{x}^{\alpha}f\|_{L^{r_{1},r_{2}}}$
$\displaystyle=\|\mathscr{F}^{-1}\left(\mathscr{F}\left(\partial_{y}^{\beta}\partial_{x}^{\alpha}f\right)\right)\|_{L^{r_{1},r_{2}}}\leq
C\|\mathscr{F}\left(\partial_{y}^{\beta}\partial_{x}^{\alpha}f\right)\|_{L^{t_{1},t_{2}}},$

where $C_{r_{1},r_{2}}>0$ is a constant and $r_{i}=\frac{t_{i}}{t_{i}-1}\geq
2\geq p_{i}$ (i.e., the above assumption on $r_{i}$ is still true). All
together, we have

$\displaystyle\|f\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}^{p}$
$\displaystyle\leq
C|U|^{\frac{1}{p_{1}s_{1}}}|V|^{\frac{1}{p_{2}s_{2}}}\sum_{|\alpha|\leq
m}\sum_{|\beta|\leq
n}\|\mathscr{F}\left(\partial_{y}^{\beta}\partial_{x}^{\alpha}f\right)\|_{L^{t_{1},t_{2}}}.$

Expressing $s_{i}$ in terms of $t_{i}$ yields
$\frac{1}{s_{i}}=1+\frac{p_{i}}{t_{i}}-p_{i}$ and continuing in the same
fashion as in the proof of Lemma 3.1, this is

$\displaystyle\|f\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}\leq
C_{m,n,p}|U|^{\frac{1}{p_{1}}+\frac{1}{t_{1}}-1}|V|^{\frac{1}{p_{2}}+\frac{1}{t_{2}}-1}\|f\|_{\mathscr{F}L^{t_{1},t_{2}}(\omega)}$

with some constant $C_{n_{1},n_{2},p_{1},p_{2}}>0$ that is only dependent on
$n_{1}$, $n_{2}$, $p_{1}$ and $p_{2}$.

We now use the same density argument as in the proof of Lemma 3.1, to show
that the result extends to all $f\in\mathscr{F}L^{t_{1},t_{2}}(\omega)$. ∎

An important thing to note here is that the degrees $p$ and $t$ can be chosen
independently of one another.

######  Corollary 3.8.

With $p_{1}=p_{2}=2$ and $t_{1}=t_{2}=1$ the assumption of Lemmas 3.1 and 3.7
overlap and we get

$\|f\|_{W_{n_{1},2}^{n_{2},2}(U,V)}\leq
C_{n_{1},n_{2},2}|U|^{1/2}|V|^{1/2}\|f\|_{\mathcal{F}L^{1}(\omega;\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})},$

where $C_{n_{1},n_{2},2}>0$ depends only on $n_{1}$ and $n_{2}$. This result
follows directly from Lemma 3.7 and by applying Plancherel’s theorem in Lemma
3.1.

###  3.3 Approximation of Fourier Lebesgue Space

In Lemma 3.1 we showed (for a proper choice of parameters) the inclusion

$\displaystyle\mathscr{F}L^{t_{1},t_{2}}(\omega)$ $\displaystyle\subseteq
W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V).$

These spaces inherit the Rademacher type from the underlying Lebesgue space
[44, 25], which implies that $W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)$ is a type-2
Banach Space for $2\leq p_{1},p_{2}<\infty$ (note the strict inequality on the
right side). For more general information on the Rademacher type of Bochner
spaces we refer the interested reader to [25].

We now use this information in order to show that any
$f\in\mathscr{F}L^{q_{1},q_{2}}(\omega)$ is in the variation space of the
dictionary of activation functions with some certain minimal decay. With this
in mind, we can extend the proof ideas of [4] and [50] from a single-block
$L^{2}$\- and Hilbert-Sobolev-Error, respectively, to the two-block Bochner-
Sobolev-Error.

######  Theorem 3.9.

For $i\in\\{1,2\\}$ let $d_{i},m_{i},n_{i}\in\mathbf{N}$, such that $m_{i}\geq
n_{i}$, and $1\leq q_{i}\leq 2\leq p_{i}<\infty$, with $p_{1}\leq p_{2}$. Let
$U\subset\mathbf{R}^{d_{1}}$ and $V\subset\mathbf{R}^{d_{2}}$ be measurable
and bounded domains with non-empty interior such that
$\|\chi_{U}\|_{\mathscr{F}L^{p_{1}^{\prime}}}<\infty$ and
$\|\chi_{V}\|_{\mathscr{F}L^{p_{2}^{\prime}}}<\infty$. Let
$\vartheta(t_{1},t_{2})$ be a weight over $\mathbf{R}^{2}$ which is elliptic
with respect to $\langle t_{1}\rangle^{\gamma_{1}}\langle
t_{2}\rangle^{\gamma_{2}}$ for some $\gamma_{1},\gamma_{2}>1$,
$\omega(\xi_{1},\xi_{2})$ be a weight over
$\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$ elliptic with respect to
$\langle\xi_{1}\rangle^{n_{1}}\langle\xi_{2}\rangle^{n_{2}}$, and

$\displaystyle\tilde{\omega}(\xi_{1},\xi_{2}):=\omega(\xi_{1},\xi_{2})\langle\xi_{1}\rangle^{(d_{1}+1)(1-\frac{1}{q_{1}})+1}\langle\xi_{2}\rangle^{(d_{2}+1)(1-\frac{1}{q_{2}})+1}.$

For an activation function $\sigma\in
W_{m_{1},\infty}^{m_{2},\infty}(\vartheta;\mathbf{R}^{2}){\setminus}\\{0\\}$,
$f\in\mathscr{F}L^{q_{1},q_{2}}(\tilde{\omega})$, and sufficiently large
$M>0$, there exits a constant $C>0$ such that

$\displaystyle\inf_{f_{N}\in\Sigma_{N,M}(\mathbb{D}_{\sigma}^{d_{1},d_{2}})}\|f-f_{N}\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}$
$\displaystyle\leq
CN^{-\frac{1}{2}}|U|^{1/p_{1}}|V|^{1/p_{2}}\|f\|_{\mathscr{F}L^{q_{1},q_{2}}(\tilde{\omega})}$
(3.11)

for all $N\in\mathbf{N}$.

###### Proof.

We split the proof in the following 5 steps:

  1. 1.

develop a representation of the phase term;

  2. 2.

represent the target function as an infinite-width shallow network;

  3. 3.

provide an upper bound on the variation norm of the target function;

  4. 4.

provide an upper bound on the supremum-norm of all functions in the
dictionary;

  5. 5.

combine the upper bounds in Maurey’s sampling argument.

Phase Construction: For $\sigma\in W_{m_{1},\infty}^{m_{2},\infty}(\vartheta)$
we get $\sigma\in L^{1}$ due to the ellipticity of $\vartheta$, thus,

$\displaystyle\hat{\sigma}(\tau_{1},\tau_{2})$
$\displaystyle=\frac{1}{2\pi}\int_{\mathbf{R}}\int_{\mathbf{R}}\sigma(t_{1},t_{2})e^{-i(\tau_{1}t_{1}+\tau_{2}t_{2})}dt_{2}dt_{1}$
$\displaystyle=\frac{1}{2\pi}\int_{\mathbf{R}}\int_{\mathbf{R}}\sigma(w_{1}\cdot
x_{1}+b_{1},w_{2}\cdot x_{2}+b_{2})e^{-i(\tau_{1}(w_{1}\cdot
x_{1}+b_{1})+\tau_{2}(w_{2}\cdot x_{2}+b_{2}))}db_{2}db_{1}.$

By substituting the linear shift $t_{i}=w_{i}\cdot x_{i}+b_{i}$ with some
arbitrary constant $w_{i},x_{i}\in\mathbf{R}^{d_{i}}$ ($i\in\\{1,2\\}$). With
the assumptions on $\sigma$, there exists a tuple $(\tau_{1},\tau_{2})$ such
that $\tau_{1},\tau_{2}\neq 0$ and $\hat{\sigma}(\tau_{1},\tau_{2})\neq 0$,
and so,

$\displaystyle e^{i(\tau_{1}w_{1}\cdot x_{1}+\tau_{2}w_{2}\cdot
x_{2})}=\frac{1}{2\pi\hat{\sigma}(\tau_{1},\tau_{2})}\int_{\mathbf{R}}\int_{\mathbf{R}}\sigma(w_{1}\cdot
x_{1}+b_{1},w_{2}\cdot
x_{2}+b_{2})e^{-i(\tau_{1}b_{1}+\tau_{2}b_{2})}db_{2}db_{1}.$

Shallow-Net representation: We insert the representation of $e^{iw\cdot x}$ in
the inverse Fourier transform of $\hat{f}$ and consider the short notation
$\xi:=(\xi_{1},\xi_{2})$ and $b:=(b_{1},b_{2})$ with
$\Lambda_{\xi}=\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$ and
$\Lambda_{b}:=\mathbf{R}\times\mathbf{R}$. That is,

$\displaystyle f(x_{1},x_{2})$
$\displaystyle=\int_{\Lambda_{\xi}}e^{i(\xi_{1}\cdot x_{1}+\xi_{2}\cdot
x_{2})}\hat{f}(\xi_{1},\xi_{2})d(\xi_{1},\xi_{2})$
$\displaystyle=\int_{\Lambda_{\xi}}\int_{\Lambda_{b}}\sigma\left(\frac{\xi_{1}\cdot
x_{1}}{\tau_{1}}+b_{1},\frac{\xi_{2}\cdot
x_{2}}{\tau_{2}}+b_{2}\right)\frac{e^{-i(\tau_{1}b_{1}+\tau_{2}b_{2})}\hat{f}(\xi_{1},\xi_{2})}{2\pi\hat{\sigma}(\tau_{1},\tau_{2})}d(b_{1},b_{2})d(\xi_{1},\xi_{2})$

and with the parametrized affine function333Note that we consider $\tau_{1}$
and $\tau_{2}$ to be constants and, thus, do not include them in the notation
of $T$.

$\displaystyle T\left(x_{1},x_{2};\xi,b\right)$
$\displaystyle:=\left(\frac{\xi_{1}\cdot
x_{1}}{\tau_{1}}+b_{1},\frac{\xi_{2}\cdot x_{2}}{\tau_{2}}+b_{2}\right),$
(3.12)

and the following constant and phase-term

$\displaystyle C_{\sigma}$
$\displaystyle:=|2\pi\hat{\sigma}(\tau_{1},\tau_{2})|,\qquad\theta(\xi,b):=\arg\left(\frac{\hat{f}(\xi_{1},\xi_{2})}{\hat{\sigma}(\tau_{1},\tau_{2})}\right)-\tau_{1}b_{1}-\tau_{2}b_{2}$

we have the integral representation

$\displaystyle f(x_{1},x_{2})$
$\displaystyle=\frac{1}{C_{\sigma}}\int_{\Lambda_{b}}\int_{\Lambda_{\xi}}\sigma\left(T\left(x_{1},x_{2};\xi,b\right)\right)\cos\left(\theta(\xi,b)\right)|\hat{f}(\xi)|d\xi
db,$

where we replaced the complex exponential by the cosine as we know that $f$ is
real valued.

However, this does not guarantee that $f$ is in the variation space of the
dictionary $\mathbb{D}_{\sigma}^{d_{1},d_{2}}$ as the bound
$\|f\|_{\mathcal{K}(\mathbb{D}_{\sigma}^{d_{1},d_{2}})}\leq\|\cos(\theta(\cdot,\cdot\cdot))\hat{f}(\cdot)\|_{L^{1}(\Lambda_{\xi}\times\Lambda_{b})}$
on its variation norm (see Proposition 2.8) does not converge over $b$. We
therefore modify the dictionary by introducing the weight

$\displaystyle\tilde{\vartheta}(\xi,b):=\vartheta\left(\left(|b_{1}|-R_{U}|\xi_{1}/\tau_{1}|\right)_{+},\left(|b_{2}|-R_{V}|\xi_{2}/\tau_{2}|\right)_{+}\right),$

where

$\displaystyle R_{U}=\sup_{x_{1}\in U}|x_{1}|\quad\text{and}\quad
R_{V}=\sup_{x_{2}\in V}|x_{2}|.$ (3.13)

Based on this weight, we now consider the dictionary
$\mathbb{D}_{\tilde{\sigma}}^{d_{1},d_{2}}$ with

$\displaystyle\tilde{\sigma}(x_{1},x_{2};\xi,b):=\frac{\tilde{\vartheta}(\xi,b)}{\omega(\xi)}\sigma(T\left(x_{1},x_{2};\xi,b\right)),$
(3.14)

where the activation function is scaled based on the parameters of the affine
function. This leads to the representation

$\displaystyle f(x_{1},x_{2})$
$\displaystyle=\frac{1}{C_{\sigma}}\int_{\Lambda_{\xi}}\int_{\Lambda_{b}}\tilde{\sigma}(x_{1},x_{2};\xi,b)\frac{\omega(\xi)}{\tilde{\vartheta}(\xi,b)}\cos\left(\theta(\xi,b)\right)|\hat{f}(\xi)|dbd\xi$

for all $(x_{1},x_{2})\in\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$.

Bound for the Variation Norm: The variation norm is now bounded by the
$L^{1}$-norm

$\displaystyle\|f\|_{\mathcal{K}(\mathbb{D}_{\tilde{\sigma}}^{d_{1},d_{2}})}\leq\frac{1}{C_{\sigma}}\|(\omega/\tilde{\vartheta})\cos(\theta)\hat{f}\|_{L^{1}(\Lambda_{\xi}\times\Lambda_{b})}.$

To obtain a bound, we first calculate the integral in direction of $b$.

$\displaystyle
I(\xi):=\int_{\mathbf{R}^{2}}\frac{1}{\tilde{\vartheta}\left(\xi,b\right)}db$
$\displaystyle=\int_{\mathbf{R}^{2}}\frac{1}{\vartheta\left(\left(|b_{1}|-R_{U}|\xi_{1}/\tau_{1}|\right)_{+},\left(|b_{2}|-R_{V}|\xi_{2}/\tau_{2}|\right)_{+}\right)}d(b_{1},b_{2})$
$\displaystyle=4\int_{\mathbf{R}_{+}^{2}}\frac{1}{\vartheta\left(\left(b_{1}-R_{U}|\xi_{1}/\tau_{1}|\right)_{+},\left(b_{2}-R_{V}|\xi_{2}/\tau_{2}|\right)_{+}\right)}d(b_{1},b_{2})$

Note that we integrate over a function that is constant along $b_{1}$ for
$b_{1}<|\frac{R_{U}\xi_{1}}{\tau_{1}}|$ and constant along $b_{2}$ for
$b_{2}<|\frac{R_{V}\xi_{2}}{\tau_{2}}|$. In all other cases, it is a shifted
version of $\vartheta$. Reverting the shift leads to

$\displaystyle I(\xi)$
$\displaystyle=4\left|\frac{R_{U}\xi_{1}}{\tau_{1}}\right|\left|\frac{R_{V}\xi_{2}}{\tau_{2}}\right|\frac{1}{\vartheta\left(0,0\right)}+4\left|\frac{R_{V}\xi_{2}}{\tau_{2}}\right|\int_{\mathbf{R}_{+}}\frac{1}{\vartheta\left(b_{1},0\right)}db_{1}$
$\displaystyle\qquad+4\left|\frac{R_{U}\xi_{1}}{\tau_{1}}\right|\int_{\mathbf{R}_{+}}\frac{1}{\vartheta\left(0,b_{2}\right)}db_{2}+4\int_{\mathbf{R}_{+}\times\mathbf{R}_{+}}\frac{1}{\vartheta\left(b_{1},b_{2}\right)}d(b_{1},b_{2}).$

Furthermore, the ellipticity of $\vartheta$ (i.e., $\vartheta(b_{1},b_{2})\geq
c\langle b_{1}\rangle^{\gamma_{1}}\langle
b_{2}\rangle^{\gamma_{2}}\geq\frac{c}{2}(1+|b_{1}|)^{\gamma_{1}}(1+|b_{2}|)^{\gamma_{2}}$
for some $c>0$) finally results in the upper bound on the integrals

$\displaystyle I(\xi)$ $\displaystyle\leq
8c\left(\left|\frac{R_{U}\xi_{1}}{\tau_{1}}\right|+\frac{1}{\gamma_{1}-1}\right)\left(\left|\frac{R_{V}\xi_{2}}{\tau_{2}}\right|+\frac{1}{\gamma_{2}-1}\right)$
$\displaystyle\leq
C_{U,V,m_{1},m_{2}}\langle\xi_{1}\rangle\langle\xi_{2}\rangle,$

where the constant in the last inequality is given by

$\displaystyle
C_{U,V,m_{1},m_{2}}=8c\cdot\max\left\\{\left|\frac{R_{U}}{\tau_{1}}\right|,\frac{1}{\gamma_{1}-1}\right\\}\max\left\\{\left|\frac{R_{V}}{\tau_{2}}\right|,\frac{1}{\gamma_{2}-1}\right\\}.$

Using this in the variation norm leads to

$\displaystyle\|f\|_{\mathcal{K}(\mathbb{D}_{\tilde{\sigma}})}$
$\displaystyle\leq\frac{1}{C_{\sigma}}\|\omega\hat{f}/\tilde{\vartheta}\|_{L^{1}(\Lambda_{\xi}\times\Lambda_{b})}=\frac{1}{C_{\sigma}}\int_{\Lambda_{\xi}}\int_{\Lambda_{b}}\frac{\omega(\xi)|\hat{f}(\xi)|}{\tilde{\vartheta}(\xi,b)}dbd\xi$
$\displaystyle=\frac{1}{C_{\sigma}}\int_{\Lambda_{\xi}}\omega(\xi)|\hat{f}(\xi)|\int_{\Lambda_{b}}\frac{1}{\tilde{\vartheta}\left(\xi,b\right)}dbd\xi=\frac{1}{C_{\sigma}}\int_{\Lambda_{\xi}}\omega(\xi)|\hat{f}(\xi)|I(\xi)d(\xi)$
$\displaystyle\leq
C_{U,V,\sigma,m_{1},m_{2}}\int_{\Lambda_{\xi}}\omega(\xi_{1},\xi_{2})\langle\xi_{1}\rangle\langle\xi_{2}\rangle|\hat{f}(\xi_{1},\xi_{2})|d(\xi_{1},\xi_{2}).$

We now define the probability measure

$\displaystyle\nu(\xi_{1},\xi_{2})$
$\displaystyle:=\frac{c_{1}c_{2}}{\langle\xi_{1}\rangle^{d_{1}+1}\langle\xi_{2}\rangle^{d_{2}+1}}\qquad\text{with}\qquad
c_{i}^{-1}:=\left\|\langle\cdot\rangle^{-(d_{i}+1)}\right\|_{L^{1}(\mathbf{R}^{d_{i}})}$

and the marginal probability measures

$\displaystyle\nu_{1}(\xi_{1})=\frac{c_{1}}{\langle\xi_{1}\rangle^{d_{1}+1}}\qquad\text{and}\qquad\nu_{2}(\xi_{2})=\frac{c_{2}}{\langle\xi_{2}\rangle^{d_{2}+1}}.$

Consequently, we can continue the bound on the variation norm with

$\displaystyle\psi(\xi_{1},\xi_{2}):=\omega(\xi_{1},\xi_{2})\langle\xi_{1}\rangle^{d_{1}+2}\langle\xi_{2}\rangle^{d_{2}+2}$

as

$\displaystyle\begin{split}\|f\|_{\mathcal{K}(\mathbb{D}_{\tilde{\sigma}})}&\leq
C_{U,V,\sigma,m_{1},m_{2}}\int_{\Lambda_{\xi}}\left|\omega(\xi_{1},\xi_{2})\langle\xi_{1}\rangle\langle\xi_{2}\rangle\hat{f}(\xi_{1},\xi_{2})\right|d(\xi_{1},\xi_{2})\\\
&=\frac{C_{U,V,\sigma,m_{1},m_{2}}}{c_{1}c_{2}}\int_{\Lambda_{\xi}}\left|\omega(\xi_{1},\xi_{2})\langle\xi_{1}\rangle^{d_{1}+2}\langle\xi_{2}\rangle^{d_{2}+2}\hat{f}(\xi_{1},\xi_{2})\right|d\nu(\xi_{1},\xi_{2})\\\
&=\frac{C_{U,V,\sigma,m_{1},m_{2}}}{c_{1}c_{2}}\left(\int_{\mathbf{R}^{d_{1}}}\left(\int_{\mathbf{R}^{d_{2}}}\left|\psi(\xi_{1},\xi_{2})\hat{f}(\xi_{1},\xi_{2})\right|d\nu_{2}(\xi_{2})\right)^{\frac{q_{2}}{q_{2}}}d\nu_{1}(\xi_{1})\right)^{\frac{q_{1}}{q_{1}}}\\\
&\leq\frac{C_{U,V,\sigma,m_{1},m_{2}}}{c_{1}c_{2}}\left(\int_{\mathbf{R}^{d_{1}}}\left(\int_{\mathbf{R}^{d_{2}}}\left|\psi(\xi_{1},\xi_{2})\hat{f}(\xi_{1},\xi_{2})\right|^{q_{2}}d\nu_{2}(\xi_{2})\right)^{\frac{q_{1}}{q_{2}}}d\nu_{1}(\xi_{1})\right)^{\frac{1}{q_{1}}}\end{split}$
(3.15)

by using Jensen’s inequality (see [47]) on both integrals. We now proceed to
express the right side in terms of the $\mathscr{F}L$-norm.

$\displaystyle\begin{split}\|f\|_{\mathcal{K}(\mathbb{D}_{\tilde{\sigma}})}\kern-18.00003pt&\kern
18.00003pt\leq\frac{C_{U,V,\sigma,m_{1},m_{2}}}{c_{1}c_{2}}\left(\int_{\mathbf{R}^{d_{1}}}\left(\int_{\mathbf{R}^{d_{2}}}\left|\psi(\xi_{1},\xi_{2})\hat{f}(\xi_{1},\xi_{2})\right|^{q_{2}}d\nu_{2}(\xi_{2})\right)^{\frac{q_{1}}{q_{2}}}d\nu_{1}(\xi_{1})\right)^{\frac{1}{q_{1}}}\\\
&=\frac{C_{U,V,\sigma,m_{1},m_{2}}}{c_{1}c_{2}}\left(\int_{\mathbf{R}^{d_{1}}}\left(\int_{\mathbf{R}^{d_{2}}}\left|\psi(\xi_{1},\xi_{2})\hat{f}(\xi_{1},\xi_{2})\right|^{q_{2}}\frac{c_{2}}{\langle\xi_{2}\rangle^{d_{2}+1}}d\xi_{2}\right)^{\frac{q_{1}}{q_{2}}}\frac{c_{1}}{\langle\xi_{1}\rangle^{d_{1}+1}}d\xi_{1}\right)^{\frac{1}{q_{1}}}\\\
&=\frac{C_{U,V,\sigma,m_{1},m_{2}}}{c_{1}^{1-\frac{1}{q_{1}}}c_{2}^{1-\frac{1}{q_{2}}}}\left(\int_{\mathbf{R}^{d_{1}}}\left(\int_{\mathbf{R}^{d_{2}}}\left|\tilde{\omega}(\xi_{1},\xi_{2})\hat{f}(\xi_{1},\xi_{2})\right|^{q_{2}}d\xi_{2}\right)^{\frac{q_{1}}{q_{2}}}d\xi_{1}\right)^{\frac{1}{q_{1}}}\\\
&=\frac{C_{U,V,\sigma,m_{1},m_{2}}}{c_{1}^{1-\frac{1}{q_{1}}}c_{2}^{1-\frac{1}{q_{2}}}}\|f\|_{\mathscr{F}L^{q_{1},q_{2}}(\tilde{\omega})}\end{split}$
(3.16)

with

$\displaystyle\tilde{\omega}(\xi_{1},\xi_{2})$
$\displaystyle:=\psi(\xi_{1},\xi_{2})\langle\xi_{1}\rangle^{-(d_{1}+1)\frac{1}{q_{1}}}\langle\xi_{2}\rangle^{-(d_{2}+1)\frac{1}{q_{2}}}$
$\displaystyle\phantom{:}=\omega(\xi_{1},\xi_{2})\langle\xi_{1}\rangle^{(d_{1}+1)(1-\frac{1}{q_{1}})+1}\langle\xi_{2}\rangle^{(d_{2}+1)(1-\frac{1}{q_{2}})+1}.$

Bound for the Dictionary: Let $\tilde{\sigma}$ from (3.14) and
$(\xi,b)\in\Lambda_{\xi}\times\Lambda_{b}$; to see that
$\|\tilde{\sigma}(\cdot,\cdot\cdot;\xi,b)\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}}$
exists and to obtain a bound we first write
$T\left(x_{1},x_{2};\xi,b\right)=(T_{1}\left(x_{1},x_{2};\xi,b\right),T_{2}\left(x_{1},x_{2};\xi,b\right))\in\mathbf{R}^{2}$
for $T$ from Eq. 3.12 and verify for $i\in\\{1,2\\}$ that

$\displaystyle
T_{i}\left(x_{1},x_{2};\xi,b\right)=|x_{i}\cdot\xi_{i}/\tau_{i}+b_{i}|$
$\displaystyle\geq\left(|b_{i}|-|x_{i}\cdot\xi_{i}/\tau_{i}|\right)_{+}\geq\left(|b_{i}|-R_{\Omega_{i}}|\xi_{i}/\tau_{i}|\right)_{+},$
(3.17)

where $\Omega_{1}=U$, $\Omega_{2}=V$, and $R_{\Omega_{i}}$ as in Eq. 3.13.
Together with the assumption $\sigma\in
W_{m_{1},\infty}^{m_{2},\infty}(\vartheta)$ we get

$\displaystyle\left(\partial_{1}^{i_{1}}\partial_{2}^{i_{2}}\sigma\right)(T\left(x_{1},x_{2};\xi,b\right))\leq\frac{C_{\sigma,\vartheta}}{\vartheta(T\left(x_{1},x_{2};\xi,b\right))}\leq\frac{C_{\sigma,\vartheta}}{\tilde{\vartheta}(\xi,b)}$

for all $(x_{1},x_{2})\in U\times V$ and
$i_{1}\in\mathbf{Z}_{+},i_{2}\in\mathbf{Z}_{+}$ such that $i_{1}\leq
m_{1},i_{2}\leq m_{2}$ where $\partial_{i}$ refers to the derivative with
respect to the $i$-th variable. The constant $C_{\sigma,\vartheta}$ in this
inequality is given by

$\displaystyle C_{\sigma,\vartheta}=\max_{\begin{subarray}{c}0\leq i_{1}\leq
m_{1}\\\ 0\leq i_{2}\leq
m_{2}\end{subarray}}\sup_{t\in\mathbf{R}^{2}}\vartheta(t)\partial_{t_{1}}^{i_{1}}\partial_{t_{2}}^{i_{2}}\sigma(t).$

This leads to

$\displaystyle\|\partial_{x_{1}}^{\alpha}\partial_{x_{2}}^{\beta}\tilde{\sigma}(T\left(\cdot,\cdot\cdot;\xi,b\right))\|_{L^{p_{1},p_{2}}(U,V)}\kern-90.00015pt$
$\displaystyle\kern
90.00015pt=\frac{\tilde{\vartheta}(\xi,b)}{\omega(\xi)}\|\partial_{x_{1}}^{\alpha}\partial_{x_{2}}^{\beta}\sigma(T\left(\cdot,\cdot\cdot;\xi,b\right))\|_{L^{p_{1},p_{2}}(U,V)}$
$\displaystyle=\frac{\tilde{\vartheta}(\xi,b)}{\omega(\xi)}\left|\frac{\xi_{1}^{\alpha}}{\tau_{1}^{|\alpha|}}\frac{\xi_{2}^{\beta}}{\tau_{2}^{|\beta|}}\right|\left\|\left(\partial_{1}^{|\alpha|}\partial_{2}^{|\beta|}\sigma\right)(T\left(\cdot,\cdot\cdot;\xi,b\right))\right\|_{L^{p_{1},p_{2}}(U,V)}$
$\displaystyle\leq\tilde{\vartheta}(\xi,b)|\tau_{1}|^{-|\alpha|}|\tau_{2}|^{-|\beta|}\left\|\left(\partial_{1}^{|\alpha|}\partial_{2}^{|\beta|}\sigma\right)(T\left(\cdot,\cdot\cdot;\xi,b\right))\right\|_{L^{p_{1},p_{2}}(U,V)}$
$\displaystyle\leq
C_{\sigma,\vartheta}\tilde{\vartheta}(\xi,b)|\tau_{1}|^{-|\alpha|}|\tau_{2}|^{-|\beta|}\left\|\frac{1}{\tilde{\vartheta}(\xi,b)}\right\|_{L^{p_{1},p_{2}}(U,V)}$
$\displaystyle\leq
C_{\sigma,\vartheta}|U|^{1/p_{1}}|V|^{1/p_{2}}|\tau_{1}|^{-|\alpha|}|\tau_{2}|^{-|\beta|}$

and finally to the bound on the Sobolev norm

$\displaystyle\|\tilde{\sigma}(\cdot,\cdot\cdot;\xi,b)\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}\kern-45.00008pt$
$\displaystyle\kern 45.00008pt=\Bigg{(}\sum_{|\alpha|\leq
n_{1}}\Bigg{(}\sum_{|\beta|\leq
n_{2}}\|\partial_{x_{1}}^{\alpha}\partial_{x_{2}}^{\beta}\tilde{\sigma}(\cdot,\cdot\cdot;\xi,b)\|_{L^{p_{1},p_{2}}(U,V)}^{p_{2}}\Bigg{)}^{\frac{p_{1}}{p_{2}}}\Bigg{)}^{\frac{1}{p_{1}}}$
$\displaystyle\leq C_{\sigma,\vartheta}\Bigg{(}\sum_{|\alpha|\leq
n_{1}}\Bigg{(}\sum_{|\beta|\leq
n_{2}}|U|^{p_{2}/p_{1}}|V|^{p_{2}/p_{2}}|\tau_{1}|^{-|\alpha|p_{2}}|\tau_{2}|^{-|\beta|p_{2}}\Bigg{)}^{\frac{p_{1}}{p_{2}}}\Bigg{)}^{\frac{1}{p_{1}}}$
$\displaystyle=C_{\sigma,\vartheta}|U|^{1/p_{1}}|V|^{1/p_{2}}\Bigg{(}\sum_{|\alpha|\leq
n_{1}}|\tau_{1}|^{-p_{1}|\alpha|}\Bigg{)}^{\frac{1}{p_{1}}}\Bigg{(}\sum_{|\beta|\leq
n_{2}}|\tau_{2}|^{-p_{2}|\beta|}\Bigg{)}^{\frac{1}{p_{2}}}.$

The final bound: With the assumption
$f\in\mathscr{F}L^{q_{1},q_{2}}(\tilde{\omega})$ and Eq. 3.15 and Eq. 3.16, we
get that
$f\in\mathscr{F}L^{1}(\omega(\cdot,\cdot\cdot)\langle\cdot\rangle\langle\cdot\cdot\rangle)\subseteq\mathscr{F}L^{1}(\omega)$.
With the additional assumption that $p_{1}\leq p_{2}$, we can therefore apply
Lemma 3.1 with $t_{i}=1$ and $s_{i}=p_{i}^{\prime}$ to get $f\in
W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)$. This is a type-2 Banach space for $2\leq
p_{1},p_{2}<\infty$ and thus, with Maurey’s approximation bound (see
Proposition 2.7) and
$M:=\|f\|_{\mathcal{K}(\mathbb{D}_{\tilde{\sigma}}^{d_{1},d_{2}})}$, the final
bound is

$\displaystyle\kern-18.00003pt\inf_{f_{N}\in\Sigma_{N,M}(\mathbb{D}_{\tilde{\sigma}}^{d_{1},d_{2}})}\|f-f_{N}\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}$
$\displaystyle\lesssim
N^{-1/2}\sup_{\xi,b}\|\tilde{\sigma}(\cdot,\cdot\cdot;\xi,b)\|_{W_{n_{1},p_{1}}^{n_{2},p_{2}}(U,V)}\|f\|_{\mathcal{K}(\mathbb{D}_{\tilde{\sigma}}^{d_{1},d_{2}})}$
$\displaystyle=N^{-1/2}C|U|^{1/p_{1}}|V|^{1/p_{2}}\|f\|_{\mathscr{F}L^{q_{1},q_{2}}(\tilde{\omega})}.$

∎

The constant $C$ in Theorem 3.9 can be written as (note that
$\frac{1}{q_{i}^{\prime}}=1-\frac{1}{q_{i}}$)

$\displaystyle C$
$\displaystyle=C_{U,V,\sigma,m_{1},m_{2}}c_{1}^{-\frac{1}{q_{1}^{\prime}}}c_{2}^{-\frac{1}{q_{2}^{\prime}}}C_{\sigma,\vartheta}\kappa_{1}^{\frac{1}{p_{1}}}\kappa_{2}^{\frac{1}{p_{2}}},$

where

$\displaystyle C_{U,V,\sigma,m_{1},m_{2}}$
$\displaystyle=\frac{8}{|2\pi\hat{\sigma}(\tau_{1},\tau_{2})|}\max\left\\{\left|\frac{R_{U}}{\tau_{1}}\right|,\frac{1}{\gamma_{1}-1}\right\\}\max\left\\{\left|\frac{R_{V}}{\tau_{2}}\right|,\frac{1}{\gamma_{2}-1}\right\\},$
$\displaystyle C_{\sigma,\vartheta}$
$\displaystyle=\max_{\begin{subarray}{c}0\leq i_{1}\leq m_{1}\\\ 0\leq
i_{2}\leq
m_{2}\end{subarray}}\sup_{t\in\mathbf{R}^{2}}\vartheta(t)\partial_{t_{1}}^{i_{1}}\partial_{t_{2}}^{i_{2}}\sigma(t),$
$\displaystyle c_{i}$
$\displaystyle=\left\|\langle\cdot\rangle^{-(d_{i}+1)}\right\|_{L^{1}(\mathbf{R}^{d_{i}})}^{-1},\qquad\text{and}\qquad\kappa_{i}=\sum_{\begin{subarray}{c}|\alpha|\leq
n_{i}\\\
\alpha\in\mathbf{Z}_{+}^{d_{i}}\end{subarray}}|\tau_{i}|^{-p_{i}|\alpha|},\qquad
i\in\\{1,2\\}.$

We will now show that
$c_{i}^{-\frac{1}{q_{i}^{\prime}}}\kappa_{i}^{\frac{1}{p_{1}}}$ can be bounded
independent of the dimension, which leaves only the dependence on $R_{U}$ and
$R_{V}$.

######  Proposition 3.10.

If $n_{i}$ grows at most linearly with $d_{i}$, then the constant $C$ in
Theorem 3.9 is dimension-independent apart from the structure of the domains
$U$ and $V$.

###### Proof.

To show this independence, we now consider the individual components of the
constant as listed in Section 3.3. Note that $C_{U,V,\sigma,m_{1},m_{2}}$ is
independent of the dimensions except for the implicit dependence on the
supremum $R_{U}$ and $R_{V}$ of $U$ and $V$. For $\kappa_{i}$ we develop an
upper bound by counting the possible combinations for $|\alpha|=k$ for
$k\in\\{0,\ldots,n_{i}\\}$ and $i\in\\{1,2\\}$

$\displaystyle\kappa_{i}$ $\displaystyle=\sum_{\begin{subarray}{c}|\alpha|\leq
n_{i}\\\
\alpha\in\mathbf{Z}_{+}^{d_{i}}\end{subarray}}|\tau_{i}|^{-p_{i}|\alpha|}=\sum_{k=0}^{n_{i}}|\tau_{i}|^{-p_{i}k}\sum_{\begin{subarray}{c}|\alpha|=k\\\
\alpha\in\mathbf{Z}_{+}^{d_{i}}\end{subarray}}1=\sum_{k=0}^{n_{i}}\binom{k+d_{i}-1}{k}|\tau_{i}|^{-p_{i}k}.$

By taking the upper bound
$\frac{(k+d_{i}-1))!}{(d_{i}-1)!}\leq(k+d_{i}-1)^{k}$ on the binomial
coefficient, bounding $k\leq n_{i}$, and taking the limit of the summation we
get

$\displaystyle\kappa_{i}$
$\displaystyle\leq\sum_{k=0}^{n_{i}}\frac{(k+d_{i}-1)^{k}}{k!}|\tau_{i}|^{-p_{i}k}\leq\sum_{k=0}^{n_{i}}\frac{((n_{i}+d_{i}-1)|\tau_{i}|^{-p_{i}})^{k}}{k!}$
$\displaystyle\leq\sum_{k=0}^{\infty}\frac{((n_{i}+d_{i}-1)|\tau_{i}|^{-p_{i}})^{k}}{k!}=e^{(n_{i}+d_{i}-1)|\tau_{i}|^{-p_{i}}}.$

Further, via a transformation to spherical coordinates (see [56, (6.131)
Examples (c)], [54, Section 3]) we can calculate

$\displaystyle c_{i}^{-1}$
$\displaystyle=\left\|\langle\cdot\rangle^{-(d_{i}+1)}\right\|_{L^{1}(\mathbf{R}^{d_{i}})}=\int_{\mathbf{R}^{d_{i}}}(1+|x|^{2})^{-\frac{d_{i}+1}{2}}dx$
$\displaystyle=\frac{2\pi^{\frac{d_{i}}{2}}}{\Gamma(\frac{d_{i}}{2})}\int_{\mathbf{R}_{+}}\frac{r^{d_{1}-1}}{(1+r^{2})^{\frac{d_{i}+1}{2}}}dr=\frac{\pi^{\frac{d_{i}}{2}}}{\Gamma(\frac{d_{i}}{2})}\int_{\mathbf{R}_{+}}\frac{t^{\frac{d_{1}}{2}-1}}{(1+t)^{\frac{d_{i}+1}{2}}}dt$
$\displaystyle=\frac{\pi^{\frac{d_{i}}{2}}}{\Gamma(\frac{d_{i}}{2})}B\left(\frac{d_{i}}{2},\frac{1}{2}\right)=\frac{\pi^{\frac{d_{i}+1}{2}}}{\Gamma(\frac{d_{i}+1}{2})}$

where $B(\cdot,\cdot\cdot)$ is the Euler beta-function. Combining $c_{i}$ and
$\kappa_{i}$ leads to

$\displaystyle\kern-9.00002pt\left(c_{i}^{-\frac{1}{q_{i}^{\prime}}}\kappa_{i}^{\frac{1}{p_{1}}}\right)^{q_{i}^{\prime}}\leq\frac{2\pi^{\frac{d_{i}+1}{2}}}{\Gamma(\frac{d_{i}+1}{2})}\exp\left({\frac{q_{i}^{\prime}}{p_{i}}(n_{i}+d_{i}-1)|\tau_{i}|^{-p_{i}}}\right),$

for which we can find a uniform bound, independend of $d_{i}$. To do so, we
first assume that the order of differentiability grows at most linearly with
the number of dimensions (i.e., $n_{i}\leq\delta_{i}d_{i}$ for some
$\delta_{i}>0$), thereby

$\displaystyle\frac{q_{i}^{\prime}}{p_{i}}(n_{i}+d_{i}-1)|\tau_{i}|^{-p_{i}}$
$\displaystyle\leq\frac{q_{i}^{\prime}}{p_{i}}(\delta_{i}d_{i}+d_{i}-1)|\tau_{i}|^{-p_{i}}\leq\frac{q_{i}^{\prime}}{p_{i}}(\delta_{i}+1)d_{i}|\tau_{i}|^{-p_{i}}$
(3.18) $\displaystyle\leq
2\frac{q_{i}^{\prime}}{p_{i}}(\delta_{i}+1)\frac{d_{i}+1}{2}|\tau_{i}|^{-p_{i}}=:\gamma_{i}\frac{d_{i}+1}{2}.$
(3.19)

Second, we observe from the lower bound in the Stirling formula that
$x\Gamma(x)\geq\left(\frac{x}{e}\right)^{x}$. This means that

$\displaystyle\frac{2\pi^{\frac{d_{i}+1}{2}}}{\Gamma(\frac{d_{i}+1}{2})}$
$\displaystyle\exp\left({\frac{q_{i}^{\prime}}{p_{i}}(n_{i}+d_{i}-1)|\tau_{i}|^{-p_{i}}}\right)\leq\frac{2\pi^{\frac{d_{i}+1}{2}}}{\Gamma(\frac{d_{i}+1}{2})}\exp\left(\gamma_{i}\frac{d_{i}+1}{2}\right)$
$\displaystyle\leq\frac{(d_{i}+1)\pi^{\frac{d_{i}+1}{2}}}{\left(\frac{d_{i}+1}{2}\right)^{\frac{d_{i}+1}{2}}}\exp\left(\gamma_{i}\frac{d_{i}+1}{2}+\frac{d_{i}+1}{2}\right)$
$\displaystyle=\left(\frac{2\pi\exp\left(\gamma_{i}+1\right)}{d_{i}+1}\right)^{\frac{d_{i}+1}{2}}(d_{i}+1).$

Note that the numerator in the fraction is constant with respect to $d_{i}$
for any choice of parameters. Therefore, for sufficiently large number
$d_{i}$, it can be upper bounded by a geometric decay, which decays faster
than the remaining linear term. Thus, the constant $C$ can be uniformly
bounded for all $d_{i}\in\mathbf{N}$ by taking the maximum over $d_{i}$. ∎

A very interesting observation for the the spectral Barron space is that it
has a similar structure to the (fractional) Hilbert-Sobolev space. To see
this, consider the domain $\Omega=\mathbf{R}^{d}$, then the infimum is over a
set with one element, i.e.,

$\displaystyle\|f\|_{\mathscr{B}_{\omega}(\mathbf{R}^{d})}=\int_{\mathbf{R}^{d}}\omega(\xi)|\hat{f}(\xi)|d\xi.$

For the special case $\omega(\xi)=(1+|\xi|^{2})^{\frac{s}{2}}$ this is then
the $L^{1}$-equivalent of the Sobolev space $H^{s}$ [23, Definition 7.9.1].
With this in mind, we can view the weighted Fourier-Lebesgue spaces as an
interpolatn between the spectral Barron space and the fractional Sobolev
space. For a bounded domain $\Omega\subseteq\mathbf{R}^{d}$, [4] showed that
$H^{\lfloor\frac{d}{2}\rfloor+2}(\mathbf{R}^{d})\subseteq\mathscr{B}_{|\cdot|}(\mathbf{R}^{d})$
cf., [4, Property 15], which allows to measure the approximation error in
$L^{2}$. Based on Theorem 3.9 we can extend this result such that we allow the
error measure to an arbitrary Hilbert-Sobolev norm, while simply asking for a
linear increase in the regularity of the target function.

######  Corollary 3.11 (Approximation of Hilbert-Sobolev Spaces).

Let $f\in W_{m_{1},2}^{m_{2},2}(\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})$ with
$m_{i}=n_{i}+\lfloor\frac{d_{i}}{2}\rfloor+2$ $U\subset\mathbf{R}^{d_{1}}$,
$V\subset\mathbf{R}^{d_{2}}$ be bounded domains, and $M>0$ be sufficiently
large. Let $\sigma$ be the activation function from Theorem 3.9. Then there
exits a constant $C>0$ such that

$\displaystyle\inf_{f_{N}\in\Sigma_{N,M}(\mathbb{D}_{\sigma}^{d_{1},d_{2}})}\|f-f_{N}\|_{W_{n_{1},2}^{n_{2},2}(U,V)}$
$\displaystyle\leq
CN^{-\frac{1}{2}}|U|^{1/2}|V|^{1/2}\|f\|_{W_{m_{1},2}^{m_{2},2}(\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})}$
(3.20)

for all $N\in\mathbf{N}$.

###### Proof.

With $\omega(\xi)=\langle\xi_{1}\rangle^{n_{1}}\langle\xi_{2}\rangle^{n_{2}}$
and $\tilde{\omega}$ defined as in Theorem 3.9 we observe

$\displaystyle\tilde{\omega}(\xi_{1},\xi_{1})$
$\displaystyle=\langle\xi_{1}\rangle^{n_{1}+d_{1}/2+3/2}\langle\xi_{2}\rangle^{n_{2}+d_{2}/2+3/2}$
$\displaystyle\leq\langle\xi_{1}\rangle^{n_{1}+\lfloor
d_{1}/2\rfloor+2}\langle\xi_{2}\rangle^{n_{2}+\lfloor d_{2}/2\rfloor+2}$
$\displaystyle=\langle\xi_{1}\rangle^{m_{1}}\langle\xi_{2}\rangle^{m_{2}}$

and therefore

$\displaystyle\|f\|_{\mathscr{F}L^{2}(\tilde{\omega})}\leq\|f\|_{\mathscr{F}L^{2}(\langle\cdot\rangle^{m_{1}}\langle\cdot\cdot\rangle^{m_{2}})}=\|f\|_{W_{m_{1},2}^{m_{2},2}(\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})}.$

For $f\in W_{m_{1},2}^{m_{2},2}(\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})$ we
thus get $f\in\mathscr{F}L^{2}(\tilde{\omega})$ and with Theorem 3.9 there is
a $C>0$ such that

$\displaystyle\inf_{f_{N}\in\Sigma_{N,M}(\mathbb{D}_{\sigma}^{d_{1},d_{2}})}\|f-f_{N}\|_{W_{n_{1},2}^{n_{2},2}(U,V)}$
$\displaystyle\leq
CN^{-\frac{1}{2}}|U|^{\frac{1}{2}}|V|^{\frac{1}{2}}\|f\|_{\mathscr{F}L^{2}(\tilde{\omega})}$
$\displaystyle\leq
CN^{-\frac{1}{2}}|U|^{\frac{1}{2}}|V|^{\frac{1}{2}}\|f\|_{W_{m_{1},2}^{m_{2},2}(\mathbf{R}^{d_{1}},\mathbf{R}^{d_{2}})}.$

∎

##  4 Experiments: Approximating Functions with Anisotropic Differentiability

In our initial assumption on the activation function (see Section 2.4), we
assumed that it is a function of two variables. This assumption deviates from
the typical convention in the machine learning community, where the activation
function acts on a one-dimensional input space. To illustrate the gain in
approximation accuracy that is obtained by our two-block structure, we
consider the following simple case: Let $f$ be a function with two input
variables and assume that for the training we use the Hilbert-Bochner-Sobolev
norm with $n_{1}<n_{2}$ as loss function.

The example that we consider will be the function

$\displaystyle f(t,x)=e^{-|t|-|x|^{3}}$ (4.1)

which is a solution of the two-phase heat equation

$\displaystyle(\partial_{t}-\partial_{x}^{2}+\operatorname{sign}(t)+9|x|^{4}-6|x|)f(t,x)=0\quad\text{with}\quad
f(0,x)=e^{-|x|^{3}}$ (4.2)

over the domain $U=V=[-1,1]$. In order to approximate this function, we
consider two models for the activation function, namely, a conventional
single-block model and a novel two-block model. More precisely, for width
$N\in\mathbf{N}$, the single-block model is given by the ReLUm-network

$\displaystyle\Phi_{1}(t,x)=w\cdot(w_{t}t+w_{x}x+b)_{+}^{m}+c,$ (4.3)

with the trainable parameters $w,w_{t},w_{x},b\in\mathbf{R}^{N}$ and
$c\in\mathbf{R}$ and the two-block model is given by the mixed network

$\displaystyle\Phi_{2}(t,x)=w\cdot\left[(w_{t}t+b_{t})_{+}^{m_{1}}(w_{x}x+b_{x})_{+}^{m_{2}}\right]+c,$
(4.4)

with the trainable parameters $w,w_{t},w_{x},b_{t},b_{x}\in\mathbf{R}^{N}$ and
$c\in\mathbf{R}$. We see that $\Phi_{1}$ and $\Phi_{2}$ have $4N+1$ and $5N+1$
trainable parameters, respectively. For a fair comparison between the two
models, we thus fix the number of trainable parameters instead of the width of
the network.

For the loss function we consider $n_{1}=0$ derivatives for $t$ and $n_{2}=1$
derivatives for $x$. In order to allow optimization via gradient decent while
having the first derivative inside the objective, we fix $m=m_{2}=2$ and
$m_{1}=1$.

With this setting, we train $10$ random initializations of both models for 201
and 401 parameters, respectively. The resulting average temporal evolution of
the of the logarithmic loss function is displayed in Figure 1. For each set of
hyperparameters, we plot the mean value along with an error band of one
standard deviation in both directions. The temporal evolution of the loss was
smoothed (i.e., remove ripples which are due to the optimization algorithms)
by applying a cumulative minimum before calculating the average and the
standard deviation. From this evolution it is clear, that the two-block model
outperforms the single block model for each of the two settings for the number
of parameters.

Figure 1: Temporal evolution of the logarithmic loss for the single-block
model and the two-block model with 201 and 401 parameters.

To further illustrate this difference, we present a contour plot of the
resulting approximation with 401 parameters in Figure 2. In this figure we see
that the contour lines of the target function show sharp edges which can be
approximated very well by the ReLU-part of the two-block activation function.
Contrary to that, the ReLU2-structure of the single block network is not
capable of fitting to this structure of the target function.

Figure 2: Countour plot of the target function $f$ (left), the single-block
model (middle), and the two-block model (right). The first row shows the
resulting function and the second row shows the partial derivative with
respect to $x$.

## References

  * [1] Ahmed Abdeljawad and Philipp Grohs “Integral Representations of Shallow Neural Network with Rectified Power Unit Activation Function” In _Neural Networks_ 155, 2022, pp. 536–550 DOI: 10.1016/j.neunet.2022.09.005
  * [2] Fernando Albiac and Nigel J. Kalton “Topics in Banach Space Theory”, Graduate texts in mathematics 233 New York: Springer, 2006
  * [3] Francis Bach “Breaking the Curse of Dimensionality With Convex Neural Networks” In _Journal of Machine Learning Research_ 18.19, 2017, pp. 1–53 URL: http://jmlr.org/papers/v18/14-546.html
  * [4] Andrew R. Barron “Universal Approximation Bounds for Superpositions of a Sigmoidal Function” In _IEEE Transactions on Information Theory_ 39.3, 1993, pp. 930–945 DOI: 10.1109/18.256500
  * [5] Andrew R. Barron, Albert Cohen, Wolfgang Dahmen and Ronald A. DeVore “Approximation and Learning by Greedy Algorithms” In _The Annals of Statistics_ 36.1, 2008 DOI: 10.1214/009053607000000631
  * [6] A. Benedek and R. Panzone “The Space ${L}^{p}$, with Mixed Norm” In _Duke Mathematical Journal_ 28.3, 1961 DOI: 10.1215/S0012-7094-61-02828-9
  * [7] Jöran Bergh and Jörgen Löfström “Interpolation Spaces: An Introduction” 223, Grundlehren der mathematischen Wissenschaften Berlin, Heidelberg: Springer Berlin Heidelberg, 1976 DOI: 10.1007/978-3-642-66451-9
  * [8] L. Breiman “Hinging Hyperplanes for Regression, Classification, and Function Approximation” In _IEEE Transactions on Information Theory_ 39.3, 1993, pp. 999–1013 DOI: 10.1109/18.256506
  * [9] Andrei Caragea, Philipp Petersen and Felix Voigtlaender “Neural Network Approximation and Estimation of Classifiers with Classification Boundary in a Barron Class” In _The Annals of Applied Probability_ 33.4, 2023 DOI: 10.1214/22-AAP1884
  * [10] Ziang Chen, Jianfeng Lu and Yulong Lu “On the Representation of Solutions to Elliptic PDEs in Barron Spaces” In _Advances in neural information processing systems_ 34 Curran Associates, Inc., 2021, pp. 6454–6465 URL: https://proceedings.neurips.cc/paper_files/paper/2021/file/32cfdce9631d8c7906e8e9d6e68b514b-Paper.pdf
  * [11] Ziang Chen, Jianfeng Lu, Yulong Lu and Shengxuan Zhou “A Regularity Theory for Static Schrödinger Equations on $\mathbb{R}^{d}$ in Spectral Barron Spaces” In _SIAM Journal on Mathematical Analysis_ 55.1, 2023, pp. 557–570 DOI: 10.1137/22M1478719
  * [12] Lénaïc Chizat and Francis Bach “Implicit Bias of Gradient Descent for Wide Two-Layer Neural Networks Trained with the Logistic Loss” In _Proceedings of thirty third conference on learning theory_ 125, Proceedings of machine learning research PMLR, 2020, pp. 1305–1338 URL: https://proceedings.mlr.press/v125/chizat20a.html
  * [13] Donald L. Cohn “Measure Theory: Second Edition”, Birkhäuser Advanced Texts Basler Lehrbücher New York, NY: Springer New York, 2013 DOI: 10.1007/978-1-4614-6956-8
  * [14] Ronald Alvin DeVore “Nonlinear Approximation” In _Acta Numerica_ 7, 1998, pp. 51–150 DOI: 10.1017/S0962492900002816
  * [15] Ronald Alvin DeVore, Boris Hanin and Guergana Petrova “Neural Network Approximation” In _Acta Numerica_ 30, 2021, pp. 327–444 DOI: 10.1017/S0962492921000052
  * [16] Weinan E, Chao Ma and Lei Wu “The Barron Space and the Flow-Induced Function Spaces for Neural Network Models” In _Constructive Approximation_ 55.1, 2022, pp. 369–406 DOI: 10.1007/s00365-021-09549-y
  * [17] Weinan E and Stephan Wojtowytsch “Representation Formulas and Pointwise Properties for Barron Functions” In _Calculus of Variations and Partial Differential Equations_ 61.2, 2022, pp. 46 DOI: 10.1007/s00526-021-02156-6
  * [18] Lawrence C. Evans “Partial Differential Equations”, Graduate studies in mathematics 19 Providence, Rhode Island: American Mathematical Society, 2022
  * [19] J. Ginibre and G. Velo “Generalized Strichartz Inequalities for the Wave Equation” In _Journal of Functional Analysis_ 133.1, 1995, pp. 50–68 DOI: 10.1006/jfan.1995.1119
  * [20] Lukas Gonon “Random Feature Neural Networks Learn Black-Scholes Type PDEs Without Curse of Dimensionality” In _Journal of Machine Learning Research_ 24.189, 2023, pp. 1–51 URL: http://jmlr.org/papers/v24/21-0987.html
  * [21] Karlheinz Gröchenig “Foundations of Time-Frequency Analysis”, Applied and Numerical Harmonic Analysis Boston, MA: Birkhäuser Boston, 2001 DOI: 10.1007/978-1-4612-0003-1
  * [22] “Mathematical Aspects of Deep Learning” Cambridge University Press, 2022 DOI: 10.1017/9781009025096
  * [23] Lars Hörmander “The Analysis of Linear Partial Differential Operators I” 256, Grundlehren der mathematischen Wissenschaften Berlin, Heidelberg: Springer Berlin Heidelberg, 1998 DOI: 10.1007/978-3-642-96750-4
  * [24] Lars Hörmander “The Analysis of Linear Partial Differential Operators II”, Classics in Mathematics Berlin, Heidelberg: Springer Berlin Heidelberg, 2005 DOI: 10.1007/b138375
  * [25] Tuomas Hytönen, Jan Neerven, Mark Veraar and Lutz Weis “Analysis in Banach Spaces” Cham: Springer International Publishing, 2017 DOI: 10.1007/978-3-319-69808-3
  * [26] Frank Jones “Lebesgue Integration on Euclidean Space” OCLC: ocm45703642, Jones and Bartlett books in mathematics Sudbury, Mass: JonesBartlett, 2001
  * [27] Paul C. Kainen and Andrew Vogt “Bochner Integrals and Neural Networks” Series Title: Intelligent Systems Reference Library In _Handbook on Neural Information Processing_ 49 Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 183–214 DOI: 10.1007/978-3-642-36657-4˙6
  * [28] Markus Aloysius Keel and Terence Tao “Endpoint Strichartz Estimates” In _American Journal of Mathematics_ 120.5, 1998, pp. 955–980 DOI: 10.1353/ajm.1998.0039
  * [29] Jason M. Klusowski and Andrew R. Barron “Risk Bounds for High-Dimensional Ridge Function Combinations Including Neural Networks” Publisher: arXiv Version Number: 4, 2016 DOI: 10.48550/ARXIV.1607.01434
  * [30] Jason M. Klusowski and Andrew R. Barron “Approximation by Combinations of ReLU and Squared ReLU Ridge Functions With $\ell^{1}$ and $\ell^{0}$ Controls” In _IEEE Transactions on Information Theory_ 64.12, 2018, pp. 7649–7656 DOI: 10.1109/TIT.2018.2874447
  * [31] Hyerim Ko and Sanghyuk Lee “Fourier Transform and Regularity of Characteristic Functions” In _Proceedings of the American Mathematical Society_ 145.3, 2016, pp. 1097–1107 DOI: 10.1090/proc/13435
  * [32] V. Kurkova and M. Sanguineti “Bounds on Rates of Variable-Basis and Neural-Network Approximation” In _IEEE Transactions on Information Theory_ 47.6, 2001, pp. 2659–2665 DOI: 10.1109/18.945285
  * [33] V.. Lebedev “On the Fourier Transform of the Characteristic Functions of Domains With ${C}^{1}$ Boundary” In _Functional Analysis and Its Applications_ 47.1, 2013, pp. 27–37 DOI: 10.1007/s10688-013-0004-1
  * [34] Zhong Li, Chao Ma and Lei Wu “Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms” Publisher: arXiv Version Number: 1, 2020 DOI: 10.48550/ARXIV.2009.06132
  * [35] William Allan Light and Elliott Ward Cheney “Approximation Theory in Tensor Product Spaces” 1169, Lecture Notes in Mathematics Berlin, Heidelberg: Springer Berlin Heidelberg, 1985 DOI: 10.1007/BFb0075391
  * [36] Jianfeng Lu and Yulong Lu “A Priori Generalization Error Analysis of Two-Layer Neural Networks for Solving High Dimensional Schrödinger Eigenvalue Problems” In _Communications of the American Mathematical Society_ 2.1, 2022, pp. 1–21 DOI: https://doi.org/10.1090/cams/5
  * [37] Yulong Lu, Jianfeng Lu and Min Wang “A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Partial Differential Equations” In _Proceedings of thirty fourth conference on learning theory_ 134, Proceedings of machine learning research PMLR, 2021, pp. 3196–3241 URL: https://proceedings.mlr.press/v134/lu21a.html
  * [38] Limin Ma, Jonathan W. Siegel and Jinchao Xu “Uniform Approximation Rates and Metric Entropy of Shallow Neural Networks” In _Research in the Mathematical Sciences_ 9.3, 2022, pp. 46 DOI: 10.1007/s40687-022-00346-y
  * [39] Y. Makovoz “Random Approximants and Neural Networks” In _Journal of Approximation Theory_ 85.1, 1996, pp. 98–109 DOI: 10.1006/jath.1996.0031
  * [40] Tanya Marwah, Zachary Chase Lipton, Jianfeng Lu and Andrej Risteski “Neural Network Approximations of PDEs Beyond Linearity: A Representational Perspective” In _Proceedings of the 40th international conference on machine learning_ 202, Proceedings of machine learning research PMLR, 2023, pp. 24139–24172 URL: https://proceedings.mlr.press/v202/marwah23a.html
  * [41] Greg Ongie, Rebecca Willett, Daniel Soudry and Nathan Srebro “A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case”, 2020 URL: https://openreview.net/forum?id=H1lNPxHKDH
  * [42] Rahul Parhi and Robert D. Nowak “Banach Space Representer Theorems for Neural Networks and Ridge Splines” Number of pages: 40 Publisher: JMLR.org tex.articleno: 43 tex.issue_date: January 2021 In _Journal of Machine Learning Research_ 22.1, 2021
  * [43] Rahul Parhi and Michael Unser “Modulation Spaces and the Curse of Dimensionality” In _Fourteenth international conference on sampling theory and applications_ , 2023 URL: https://openreview.net/forum?id=0c-grlqX9Sc
  * [44] A. Pełczyński and K. Senator “On Isomorphisms of Anisotropic Sobolev Spaces with ”Classical Banach Spaces” and a Sobolev Type Embedding Theorem” In _Studia Mathematica_ 84.2, 1986, pp. 169–215 DOI: 10.4064/sm-84-2-169-215
  * [45] Stevan Pilipović, Nenad Teofanov and Joachim Toft “Micro-Local Analysis with Fourier Lebesgue Spaces. Part I” In _Journal of Fourier Analysis and Applications_ 17.3, 2011, pp. 374–407 DOI: 10.1007/s00041-010-9138-1
  * [46] G. Pisier “Remarques Sur Un Résultat Non Publié de B. Maurey” In _Séminaire d’Analyse fonctionnelle (dit” Maurey-Schwartz”)_ , 1980, pp. 1–12
  * [47] Walter Rudin “Real and Complex Analysis”, McGraw-Hill international editions Mathematics series New York, NY: McGraw-Hill, 2013
  * [48] Pedro Savarese, Itay Evron, Daniel Soudry and Nathan Srebro “How Do Infinite Width Bounded Norm Networks Look in Function Space?” In _Proceedings of the Thirty-Second Conference on Learning Theory_ PMLR, 2019, pp. 2667–2690 URL: https://proceedings.mlr.press/v99/savarese19a.html
  * [49] Erhard Schmidt “Zur Theorie der linearen und nichtlinearen Integralgleichungen: I. Teil: Entwicklung willkürlicher Funktionen nach Systemen vorgeschriebener” In _Mathematische Annalen_ 63.4, 1907, pp. 433–476 DOI: 10.1007/BF01449770
  * [50] Jonathan W. Siegel and Jinchao Xu “Approximation Rates for Neural Networks With General Activation Functions” In _Neural Networks_ 128, 2020, pp. 313–321 DOI: 10.1016/j.neunet.2020.05.019
  * [51] Jonathan W. Siegel and Jinchao Xu “High-Order Approximation Rates for Shallow Neural Networks With Cosine and ReLU Activation Functions” In _Applied and Computational Harmonic Analysis_ 58, 2022, pp. 1–26 DOI: 10.1016/j.acha.2021.12.005
  * [52] Jonathan W. Siegel and Jinchao Xu “Sharp Bounds on the Approximation Rates, Metric Entropy, and n-Widths of Shallow Neural Networks” In _Foundations of Computational Mathematics_ , 2022 DOI: 10.1007/s10208-022-09595-3
  * [53] Jonathan W. Siegel and Jinchao Xu “Characterization of the Variation Spaces Corresponding to Shallow Neural Networks” In _Constructive Approximation_ , 2023 DOI: 10.1007/s00365-023-09626-4
  * [54] David J. Smith and Mavina K. Vamanamurthy “How Small Is a Unit Ball?” In _Mathematics Magazine_ 62.2, 1989, pp. 101–107 DOI: 10.1080/0025570X.1989.11977419
  * [55] Christopher D. Sogge “Lectures on Non-linear Wave Equations” Somerville, Massachusetts, U.S.A: International Press, 2013
  * [56] Karl R. Stromberg “An Introduction to Classical Real Analysis” Providence, Rhode Island: American Mathematical Society : AMS Chelsea Publishing, 2015
  * [57] Terence Tao “Nonlinear Dispersive Equations: Local and Global Analysis” OCLC: ocm65165502, Conference Board of the Mathematical Sciences regional conference series in mathematics no. 106 Providence, R.I: American Mathematical Society, 2006
  * [58] Luc Tartar “An Introduction to Sobolev Spaces and Interpolation Spaces” 3, Lecture Notes of the Unione Matematica Italiana Berlin, Heidelberg: Springer Berlin Heidelberg, 2007 DOI: 10.1007/978-3-540-71483-5
  * [59] Felix Voigtlaender “${L}^{p}$ Sampling Numbers for the Fourier-Analytic Barron Space” Publisher: arXiv Version Number: 1, 2022 DOI: 10.48550/ARXIV.2208.07605
  * [60] Jinchao Xu “Finite Neuron Method and Convergence Analysis” In _Communications in Computational Physics_ 28.5, 2020, pp. 1707–1745 DOI: 10.4208/cicp.OA-2020-0191

##  Appendix A Appendix

######  Proposition A.1 ($L^{p_{1},p_{2}}(U_{1},U_{2})\subseteq
L^{q_{1},q_{2}}(U_{1},U_{2})$).

For $i\in\\{1,2\\}$ let $d_{i}\in\mathbf{N}$, $U_{i}\subset\mathbf{R}^{d_{i}}$
bounded, $q_{i}\geq p_{i}$, and $s_{i}:=\frac{q_{i}}{q_{i}-p_{i}}$
($s_{i}=\infty$ if $q_{i}=p_{i}$), then for $f\in L^{q_{1},q_{2}}$

$\displaystyle\|(\chi_{U_{1}}\otimes\chi_{U_{2}})f\|_{L^{p_{1},p_{2}}}\leq|U_{1}|^{\frac{1}{p_{1}s_{1}}}|U_{2}|^{\frac{1}{p_{2}s_{2}}}\|f\|_{L^{q_{1},q_{2}}}.$

The proof of this statement is essentially, the same as for a single block.
The argument is simply applied twice.

###### Proof.

The choice of $q_{i}$ and $s_{i}$ guarantees that $\frac{q_{i}}{p_{i}}\geq 1$,
$s_{i}\geq 1$, and $\frac{p_{i}}{q_{i}}+\frac{1}{s_{i}}=1$. Therefore, by
applying Hölder’s inequality for a single block twice,

$\displaystyle\|(\chi_{U_{1}}\otimes\chi_{U_{2}})f\|_{L^{p_{1},p_{2}}}$
$\displaystyle=\left\|\chi_{U_{1}}\cdot\|\chi_{U_{2}}f^{p_{2}}\|_{L^{1}}^{\frac{p_{1}}{p_{2}}}\right\|_{L^{1}}^{\frac{1}{p_{1}}}\leq\left\|\chi_{U_{1}}\cdot\|\chi_{U_{2}}\|_{L^{s_{2}}}^{\frac{p_{1}}{p_{2}}}\cdot\|f^{p_{2}}\|_{L^{\frac{q_{2}}{p_{2}}}}^{\frac{p_{1}}{p_{2}}}\right\|_{L^{1}}^{\frac{1}{p_{1}}}$
$\displaystyle=\|\chi_{U_{2}}\|_{L^{s_{2}}}^{\frac{1}{p_{2}}}\cdot\|\chi_{U_{1}}\cdot\|f\|_{L^{q_{2}}}^{p_{1}}\|_{L^{1}}^{\frac{1}{p_{1}}}\leq\|\chi_{U_{1}}\|_{L^{s_{1}}}^{\frac{1}{p_{1}}}\cdot\|\chi_{U_{2}}\|_{L^{s_{2}}}^{\frac{1}{p_{2}}}\cdot\left\|\|f\|_{L^{q_{2}}}^{p_{1}}\right\|_{L^{\frac{q_{1}}{p_{1}}}}^{\frac{1}{p_{1}}}$
$\displaystyle=|U_{1}|^{\frac{1}{p_{1}s_{1}}}|U_{2}|^{\frac{1}{p_{2}s_{2}}}\|f\|_{L^{q_{1},q_{2}}}.$

∎

######  Proposition A.2 (Convergence of Smoothing for two Blocks).

Let $d_{1},d_{2}\in\mathbf{N}$, $1\leq p_{1},p_{2}\leq\infty$,
$U\subset\mathbf{R}^{d_{1}}$ and $V\subset\mathbf{R}^{d_{2}}$ be bounded, and
$h:\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}\to\mathbf{R}$ be locally
integrable. Then,

$\displaystyle\lim_{\epsilon_{1}\to 0}$ $\displaystyle\lim_{\epsilon_{2}\to
0}\|\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}h\|_{L^{p_{1},p_{2}}}=\|\chi_{U}\chi_{V}h\|_{L^{p_{1},p_{2}}}.$

###### Proof.

Using the smoothed characteristic functions $\chi_{U}^{\epsilon_{1}}$ and
$\chi_{V}^{\epsilon_{2}}$, the extended sets $U_{1}$ and $U_{2}$, and
$H(x)=\int_{\mathbf{R}^{d_{2}}}g(x,y)dy$ we define444Note, that the
definitions of $f$, $f_{\epsilon_{1}}$, and $\tilde{f}$ are deliberately based
on $g$ and not on its variants.

$\displaystyle g(x,y)$ $\displaystyle:=|\chi_{V}(y)h(x,y)|^{p_{2}},$
$\displaystyle f(x)$
$\displaystyle:=\chi_{U}(x)^{p_{1}}H(x)^{\frac{p_{1}}{p_{2}}},$ $\displaystyle
g_{\epsilon_{2}}(x,y)$
$\displaystyle:=|\chi_{V}^{\epsilon_{2}}(y)h(x,y)|^{p_{2}},$ $\displaystyle
f_{\epsilon_{1}}(x)$
$\displaystyle:=\chi_{U}^{\epsilon_{1}}(x)^{p_{1}}H(x)^{\frac{p_{1}}{p_{2}}},$
$\displaystyle\tilde{g}(x,y)$
$\displaystyle:=|\chi_{V_{1}}(y)h(x,y)|^{p_{2}},$ $\displaystyle\tilde{f}(x)$
$\displaystyle:=\chi_{U_{1}}(x)^{p_{1}}H(x)^{\frac{p_{1}}{p_{2}}}.$

All these functions are integrable. Further,
$\tilde{g}\geq|g_{\epsilon_{2}}|,|g|$ everywhere for all $0<\epsilon_{2}\leq
1$, and $g_{\epsilon_{2}}\to_{\epsilon_{2}\to 0}g$ pointwise by [58, Lemma
3.2]555The lemma guarantees uniform convergence which includes pointwise
convergence, which is the necessary condition for the dominated convergence
theorem.. Analogous relations hold for $\tilde{f}$, $f_{\epsilon_{1}}$, $f$,
and $\epsilon_{1}$, respectively. By the dominated convergence theorem we
obtain the limit

$\displaystyle\lim_{\epsilon_{1}\to
0}\int_{\mathbf{R}^{d_{1}}}f_{\epsilon_{1}}(x)dx$
$\displaystyle=\int_{\mathbf{R}^{d_{1}}}f(x)dx$ and the pointwise limit (for
all $x\in\mathbf{R}^{d_{1}}$) $\displaystyle\lim_{\epsilon_{2}\to
0}\int_{\mathbf{R}^{d_{2}}}g_{\epsilon_{2}}(x,y)dy$
$\displaystyle=\int_{\mathbf{R}^{d_{2}}}g(x,y)dy.$

Therefore,

$\displaystyle\lim_{\epsilon_{1}\to 0}$ $\displaystyle\lim_{\epsilon_{2}\to
0}\|\chi_{U}^{\epsilon_{1}}\chi_{V}^{\epsilon_{2}}h\|_{L^{p_{1},p_{2}}}$
$\displaystyle=\lim_{\epsilon_{1}\to 0}\lim_{\epsilon_{2}\to
0}\left(\int_{\mathbf{R}^{d_{1}}}\left(\int_{\mathbf{R}^{d_{2}}}|\chi_{U}^{\epsilon_{1}}(x)\chi_{V}^{\epsilon_{2}}(y)h(x,y)|^{p_{2}}dy\right)^{\frac{p_{1}}{p_{2}}}dx\right)^{\frac{1}{p_{1}}}$
$\displaystyle=\lim_{\epsilon_{1}\to
0}\left(\int_{\mathbf{R}^{d_{1}}}\chi_{U}^{\epsilon_{1}}(x)^{p_{1}}\lim_{\epsilon_{2}\to
0}\left(\int_{\mathbf{R}^{d_{2}}}|\chi_{V}^{\epsilon_{2}}(y)h(x,y)|^{p_{2}}dy\right)^{\frac{p_{1}}{p_{2}}}dx\right)^{\frac{1}{p_{1}}}$
$\displaystyle=\lim_{\epsilon_{1}\to
0}\left(\int_{\mathbf{R}^{d_{1}}}\chi_{U}^{\epsilon_{1}}(x)^{p_{1}}\lim_{\epsilon_{2}\to
0}\left(\int_{\mathbf{R}^{d_{2}}}g_{\epsilon_{2}}(x,y)dy\right)^{\frac{p_{1}}{p_{2}}}dx\right)^{\frac{1}{p_{1}}}$
$\displaystyle=\lim_{\epsilon_{1}\to
0}\left(\int_{\mathbf{R}^{d_{1}}}\chi_{U}^{\epsilon_{1}}(x)^{p_{1}}\left(\int_{\mathbf{R}^{d_{2}}}g(x,y)dy\right)^{\frac{p_{1}}{p_{2}}}dx\right)^{\frac{1}{p_{1}}}$
$\displaystyle=\lim_{\epsilon_{1}\to
0}\left(\int_{\mathbf{R}^{d_{1}}}f_{\epsilon_{1}}(x)dx\right)^{\frac{1}{p_{1}}}$
$\displaystyle=\left(\int_{\mathbf{R}^{d_{1}}}f(x)dx\right)^{\frac{1}{p_{1}}}$
$\displaystyle=\|\chi_{U}\chi_{V}h\|_{L^{p_{1},p_{2}}}$

∎

The following result is of independent interest, where we study embedding
result in the setting of Fourier Lebesgue spaces.

######  Lemma A.3 (Fourier Lebesgue embedding).

For $i\in\\{1,2\\}$ let $d_{i}\in\mathbf{N}$, $t_{i}\in[1,2]$, and
$\vartheta_{i}$ be a weight function such that $1/\vartheta_{i}\in
L^{1}(\mathbf{R}^{d_{i}})$ and that there exists $k>0$ such that
$\vartheta_{i}(x_{i})>k$ for all $x_{i}\in\mathbf{R}^{d_{i}}$. For any weight
function $\omega(x_{1},x_{2})$ over
$\mathbf{R}^{d_{1}}\times\mathbf{R}^{d_{2}}$ elliptic with respect to
$\vartheta_{1}(x_{1})\vartheta_{2}(x_{2})$, it holds

$\displaystyle L^{1}\cap\mathscr{F}L^{t_{1},t_{2}}(\omega)\subseteq
L^{1}\cap\mathscr{F}L^{1}.$

###### Proof.

For the given weight function we have

$\displaystyle\|{1}/{\omega}\|_{L^{s_{1},s_{2}}}\lesssim\|\frac{1}{\vartheta_{1}(\cdot)\vartheta_{2}(\cdot\cdot)}\|_{L^{s_{1},s_{2}}}=\|{1}/{\vartheta_{1}}\|_{L^{s_{1}}}\|{1}/{\vartheta_{2}}\|_{L^{s_{2}}}.$

Due to the lower bound on $\vartheta_{i}$ we know that $1/\vartheta_{i}$ is
upper bounded, which allows us to provide an upper bound to this expression by
the $L^{1}$-norm (with some multiplicative constant) whenever $s_{1},s_{2}\geq
1$. This is due to the monotonicity of the integral. Thus, by the assumption
on $\vartheta_{1}$ and $\vartheta_{2}$ we get that $1/\omega\in
L^{s_{1},s_{2}}$ for $s_{1},s_{2}\geq 1$.

For $a\in L^{1}\cap\mathscr{F}L^{t_{1},t_{2}}(\omega)$ we can then apply
Hölder’s inequality with $s_{i}=\frac{t_{i}}{t_{i}-1}\geq 2$ and get

$\displaystyle\|a\|_{\mathscr{F}L^{1}}$
$\displaystyle=\|\hat{a}\|_{L^{1}}=\|\frac{1}{\omega}\omega\hat{a}\|_{L^{1}}\leq\|{1}/{\omega}\|_{L^{s_{1},s_{2}}}\|\omega\hat{a}\|_{L^{t_{1},t_{2}}}<\infty.$

∎