# Solving PDEs on Spheres with Physics-Informed Convolutional Neural
Networks†00footnotetext: † The work described in this paper is supported
partially by Shanghai Science and Technology Program (Project No. 21JC1400600)
and NSFC/RGC Joint Research Scheme (Project No. 12061160462 and
N_CityU102/20). The work of Lei Shi is also supported by the National Natural
Science Foundation of China (Grant No.12171039). The work of Ding-Xuan Zhou is
also supported by InnoHK initiative, the Government of the HKSAR, and the
Laboratory for AI-Powered Financial Technologies. The corresponding author is
Lei Shi.

Guanhang Lei  School of Mathematical Sciences, Shanghai Key Laboratory for
Contemporary Applied Mathematics, Fudan University, Shanghai, 200433, P. R.
China<EMAIL_ADDRESS>{zlei, leishi<EMAIL_ADDRESS>Zhen
Lei  School of Mathematical Sciences, Shanghai Key Laboratory for Contemporary
Applied Mathematics, Fudan University, Shanghai, 200433, P. R. China
<EMAIL_ADDRESS>{zlei, leishi<EMAIL_ADDRESS>Lei Shi
School of Mathematical Sciences, Shanghai Key Laboratory for Contemporary
Applied Mathematics, Fudan University, Shanghai, 200433, P. R. China
<EMAIL_ADDRESS>{zlei, leishi<EMAIL_ADDRESS>Chenyu
Zeng  School of Mathematical Sciences, Shanghai Key Laboratory for
Contemporary Applied Mathematics, Fudan University, Shanghai, 200433, P. R.
China<EMAIL_ADDRESS>{zlei, leishi<EMAIL_ADDRESS>Ding-Xuan Zhou  School of Mathematics and Statistics, University of Sydney,
Sydney NSW 2006, Australia<EMAIL_ADDRESS>
###### Abstract

Physics-informed neural networks (PINNs) have been demonstrated to be
efficient in solving partial differential equations (PDEs) from a variety of
experimental perspectives. Some recent studies have also proposed PINN
algorithms for PDEs on surfaces, including spheres. However, theoretical
understanding of the numerical performance of PINNs, especially PINNs on
surfaces or manifolds, is still lacking. In this paper, we establish rigorous
analysis of the physics-informed convolutional neural network (PICNN) for
solving PDEs on the sphere. By using and improving the latest approximation
results of deep convolutional neural networks and spherical harmonic analysis,
we prove an upper bound for the approximation error with respect to the
Sobolev norm. Subsequently, we integrate this with innovative localization
complexity analysis to establish fast convergence rates for PICNN. Our
theoretical results are also confirmed and supplemented by our experiments. In
light of these findings, we explore potential strategies for circumventing the
curse of dimensionality that arises when solving high-dimensional PDEs.

Keywords and phrases: Physics-Informed Neural Network; Convolutional Neural
Network; Solving PDEs on spheres; Convergence analysis; Curse of
dimensionality.

##  1 Introduction

Solving partial differential equations (PDEs) is crucial in many science and
engineering problems. Numerous methods, including finite differences, finite
elements, and some meshless schemes, have been well-developed, particularly
for low-dimensional PDEs. Nonetheless, these classical strategies often become
impractical and time-consuming for high-dimensional PDEs attributed to their
computational inefficiencies. The amalgamation of deep learning methodologies
and data-driven architectures has recently exhibited superiority across
various fields. There have been extensive studies on solving high-dimensional
PDEs with deep neural networks, such as the Deep Ritz methods (DRMs) [12],
Physics-Informed Neural Networks (PINNs) [47, 44], Neural Operators [33, 34,
29], and DeepONets [36]. DRMs and PINNs utilize the powerful approximation
capabilities of neural networks to directly learn solutions of PDEs. In
contrast, Neural Operators and DeepONets specialize in learning the operators
that map initial or boundary conditions to solution functions. One can refer
to [22] for an exhaustive review of deep learning techniques for resolving
PDEs.

In this paper, we focus on the PINN approach. While previous research (e.g.,
[37, 19]) has performed convergence analysis on the generalization bound of
PINNs, it primarily pertains to fully connected neural networks learning
solutions to PDEs on a physical domain of a Euclid space. However, there is
still a gap in employing convolutional neural networks (CNNs) for this
purpose. In addition, solving PDE systems on manifolds holds considerable
relevance for practical applications, situating manifold learning as a vibrant
domain within machine learning. The low intrinsic data dimensions have been
validated as effective in shielding learning algorithms from the curse of
dimensionality [54, 55, 38, 53, 16, 32]. Although there has been some
exploration of the PINN framework for PDEs on manifolds [13, 50, 9, 5, 56],
comprehensive convergence analysis is still absent. This study is the
inaugural endeavor to address this gap. We propose the application of PINNs
with CNN architectures, coined as PICNNs, to solve general PDEs of order $s$
on a unit sphere, while concurrently establishing convergence analysis for
such PDE solvers. We probe into the approximation potential of CNNs by
employing the spherical harmonic analysis, assess the Rademacher complexity of
our model, and derive elegant generalization bounds through various technical
estimates. In this context, our research offers significant insights into the
ability of deep learning to tackle high-dimensional PDEs and leverage the low-
dimensional attributes of physical domains. We summarize the contributions of
this paper as follows.

  * •

We are the first to investigate PICNN PDE solvers on a unit sphere. Previous
work (e.g., [13, 50]) employs fully connected neural networks as solvers on
the sphere. Although we focus on PINNs working with CNNs, some discussions in
this paper are also applicable to the general PINNs. Consequently, we may use
the term PINN when referring to the non-convolutional architecture.

  * •

We present comprehensive analysis of the relationship between PDEs and PINNs,
facilitating rigorous assumptions on the well-posedness and regularity of
PDEs. In contrast to previous studies (e.g., [37, 19]) that exclusively focus
on second-order elliptic PDEs, our approach encompasses a broad class of PDEs.

  * •

We demonstrate fast convergence of approximation using
$\mathrm{ReLU}$-$\mathrm{ReLU}^{k}$ CNNs operating on a unit sphere.
Conversely, earlier studies [37, 19] solely explore the approximation ability
of $\mathrm{ReLU}^{3}$ fully connected networks, which frequently suffer from
gradient explosion during training. Despite the success of the ReLU network in
diverse machine learning tasks attributed to its gradient calculation
simplicity, it proves inadequate for approximating high-order derivatives in
PDE problems due to the saturation phenomena (e.g., see [6]). To tackle this
issue, we propose a hybrid CNN architecture utilizing both $\mathrm{ReLU}$ and
$\mathrm{ReLU}^{k}$. Leveraging the advantages of each activation function, we
achieve an effective approximation of the derivatives of PDE solutions while
maintaining ease of network training. During the process of proof, we develop
solid analysis employing ideas and techniques from spherical approximation
theory and B-spline interpolation. These approximation analyses enable us to
address a broad Sobolev smoothness condition, assuming the PDE solution
$u^{*}$ belongs to $W^{r}_{p}(\mathbb{S}^{d-1})$.

  * •

We develop a novel approach involving localization Rademacher complexity
analysis to establish an oracle inequality that bounds the statistical error.
To the best of our knowledge, we are the first to compute the VC-dimensions of
hypothesis spaces generated by the CNN architecture and high-order
derivatives. Furthermore, unlike [37], our approach does not require a sup-
norm restriction on network parameters, making it more aligned with practical
algorithms. Additionally, our upper bound exhibits a significantly sharper
estimation than that presented in [19] due to the absence of a localization
technique in their work.

  * •

We derive fast convergence upper bounds for the excess risk of PINNs in
diverse scenarios, encompassing varying dimensions $d$, PDE orders $s$,
smoothness orders $r$, and activation degrees $k$ of $\mathrm{ReLU}^{k}$. We
establish a convergence rate in the form of $n^{-a}(\log n)^{2a}$, where

$a=\left\\{\begin{aligned} &\frac{r-s}{(r-s)+(d-1)},&&\mbox{ if
}r<\infty,d>3,\\\ &1-\frac{r+4k+6-3s}{2(r-s)(k-s+1)+r+4k+6-3s},&&\mbox{ if
}r<\infty,d=3,\\\ &1-\frac{1}{2(k-s)+3},&&\mbox{ if }r=\infty,d\geq
3.\end{aligned}\right.$

  * •

We validate our theory through comprehensive numerical experiments, and by
integrating our theoretical analysis with these experiments, we ascertain the
conditions under which a PDE PINN solver can surmount the curse of
dimensionality.

The rest of the paper is organized as follows. In Section 2, we first discuss
the strong convexity of the PINN risk, subsequently formulating reasonable
assumptions on the well-posedness and regularity of PDEs. We then introduce
the structure of the CNN employed in our analysis. After elaborating on the
assumptions, we present our main result in Theorem 1. In Section 3, we prove
an upper bound for the approximation error, as evidenced in Theorem 3,
leveraging techniques from the spherical harmonics analysis and B-spline
interpolation. In Section 4, novel localization analysis related to the
stochastic component of error evaluation is conducted, leading to the
derivation of a pivotal oracle inequality as seen in Theorem 5. Section 5
amalgamates the derived approximation bound and the oracle inequality,
culminating in the derivation of accelerated convergence rates for PICNN when
applied to solving sphere PDEs, which gives a proof of Theorem 1. We present
experimental results in Section 6 to validate our theoretical assertions and
shed light on the conditions circumventing the algorithmic curse of
dimensionality.

##  2 Preliminaries and Main Result

###  2.1 PINN for General PDEs

Consider solving the following PDE with a Dirichlet boundary condition:

$\left\\{\begin{aligned}
(\mathcal{L}u)(x)&=f(x),&&x\in\Omega\subset\mathbb{R}^{d},\\\
u(x)&=g(x),&&x\in\partial\Omega,\end{aligned}\right.$ (2.1)

where $\mathcal{L}$ is a general differential operator and $\Omega$ is a
bounded domain. To construct the approximate solution, PINN converts (2.1) to
a minimization problem with the objective function $\mathcal{R}$ defined as

$\mathcal{R}(u)=\frac{1}{\mathrm{vol}(\Omega)}\int_{\Omega}|(\mathcal{L}u)(x)-f(x)|^{2}dx+\frac{1}{\sigma(\partial\Omega)}\int_{\partial\Omega}|u(x)-g(x)|^{2}d\sigma(x).$

Here, $\mathrm{vol}$ is the Lebesgue measure and $\sigma$ is the surface
measure on $\partial\Omega$. This objective function is the mean squared error
of the residual from (2.1) and the following empirical version is used for
numerical optimization:

$\mathcal{R}_{n,m}(u)=\frac{1}{n}\sum_{i=1}^{n}|(\mathcal{L}u)(X_{i})-f(X_{i})|^{2}+\frac{1}{m}\sum_{i=1}^{m}|u(Y_{i})-g(Y_{i})|^{2},$

where
$\\{X_{i}\\}_{i=1}^{n}\subset\Omega,\\{Y_{i}\\}_{i=1}^{m}\subset\partial\Omega$
are independent and identically distributed (i.i.d.) random samples from the
uniform distribution. $\mathcal{R}(u)$ and $\mathcal{R}_{n,m}(u)$ are referred
to as the population risk and empirical risk respectively. Given a function
class $\mathcal{F}$, PINN solves the following empirical risk minimization
(ERM) problem:

$u_{n,m}=\arg\min_{u\in\mathcal{F}}\mathcal{R}_{n,m}(u).$ (2.2)

If equation (2.1) has a unique classical solution, denoted by $u^{*}$,
obviously $\mathcal{R}(u^{*})=\mathcal{R}_{n,m}(u^{*})=0$. Under the framework
of learning theory, this ideal solution $u^{*}$ is termed the Bayes function
since it minimizes the population risk $\mathcal{R}(u)$. The performance of
the ERM estimator $u_{n,m}$ can be quantified by the excess risk
$\mathcal{R}(u_{n,m})-\mathcal{R}(u^{*})$. This study establishes rapidly
decaying rates of $\mathcal{R}(u_{n,m})-\mathcal{R}(u^{*})$ as the size of the
training data set increases. This decaying rate, often termed the convergence
rate or learning rate, is an important measure of the algorithm’s
generalization performance. Contrastingly, the accuracy of an approximate PDE
solution is traditionally measured by estimating the error
$\|u_{n,m}-u^{*}\|$, where $\|\cdot\|$ represents the norm of a regularity
space, such as the Sobolev space $W^{r}_{p}(\Omega)$. This raises a natural
question: Can the excess risk $\mathcal{R}(u)-\mathcal{R}(u^{*})$ bound
$\|u-u^{*}\|$ in a manner that superior generalization performance corresponds
to enhanced accuracy? This property is indispensable for constraining the
statistical error through localization analysis. Equally significant is the
converse question: Can $\|u-u^{*}\|$ control
$\mathcal{R}(u)-\mathcal{R}(u^{*})$? If so, an upper bound on $\|u-u^{*}\|$,
derived from approximation analysis, could provide a bound for the
approximation error $\mathcal{R}(u_{\mathcal{F}})-\mathcal{R}(u^{*})$ (refer
to Subsection 2.5). Our primary objective is to establish an equivalence
between the excess risk $\mathcal{R}(u)-\mathcal{R}(u^{*})$ and the error
$\|u-u^{*}\|_{W^{r}_{p}(\Omega)}$. This relationship is known as the strong
convexity of the PINN risk with respect to the $W^{r}_{p}(\Omega)$ norm, which
is only determined by the underlying PDE.

For the sake of theoretical simplicity, we consider a liner PDE where the
differential operator $\mathcal{L}$ is linear. Hence we write

$\displaystyle\mathcal{R}(u)-\mathcal{R}(u^{*})$
$\displaystyle=\frac{1}{\mathrm{vol}(\Omega)}\int_{\Omega}|(\mathcal{L}u)(x)-f(x)|^{2}dx+\frac{1}{\sigma(\partial\Omega)}\int_{\partial\Omega}|u(x)-g(x)|^{2}d\sigma(x)$
$\displaystyle=\frac{1}{\mathrm{vol}(\Omega)}\int_{\Omega}|\mathcal{L}(u-u^{*})(x)|^{2}dx+\frac{1}{\sigma(\partial\Omega)}\int_{\partial\Omega}|u(x)-u^{*}(x)|^{2}d\sigma(x)$
$\displaystyle=\frac{1}{\mathrm{vol}(\Omega)}\|\mathcal{L}(u-u^{*})\|^{2}_{L^{2}(\Omega)}+\frac{1}{\sigma(\partial\Omega)}\|u-u^{*}\|^{2}_{L^{2}(\partial\Omega)}.$

Let us first consider controlling $\|u-u^{*}\|_{W^{r}_{p}(\Omega)}$ by
$\mathcal{R}(u)-\mathcal{R}(u^{*})$. This problem is intrinsically connected
to the global $W^{r}_{p}$ estimates for PDEs. For an elliptic operator
$\mathcal{L}$ of even order $2s$ where $s\in\mathbb{N}$, some prior estimates
related to Gårding’s inequality have already been well-established. See Lemma
1 and Lemma 2 as follows. Throughout the subsequent discourse, if $A$ and $B$
are two quantities (typically non-negative), the notation $A\lesssim B$
indicates that $A\leq CB$ for some positive absolute constant $C$. It should
be emphasized that $C$ is independent of both $A$ and $B$, or is universal for
indexed $A$ and $B$ of a certain class. The constant $C$ might depend on
particular parameters, such as dimension, regularity, or exponent, as
determined by the context. In this paper, we will not delve into the specifics
of which parameters $C$ relies on, nor will we discuss the optimal estimation
of $C$. Moreover, we use the notation $A\asymp B$ when $A\lesssim B$ and
$B\lesssim A$. Denote the Sobolev space $W^{s}_{2}(\cdot)$ by $H^{s}(\cdot)$.

######  Lemma 1.

(Theorem 12.8 in [1]) Let $s\in\mathbb{N}$ and $\mathcal{L}$ denote a
uniformly elliptic operator of order $2s$ possessing bounded coefficients,
with its leading coefficients being continuous. Furthermore, suppose
$\mathcal{L}$ is weakly positive semi-definite. For a sufficiently large
positive $\lambda$, the inequality

$\|u\|_{H^{2s}(\Omega)}\lesssim\|(\mathcal{L}+\lambda)u\|_{L^{2}(\Omega)}$

holds for every $u\in H^{2s}(\Omega)\cap H^{s}_{0}(\Omega)$.

######  Lemma 2.

(Theorem 15.2 in [1] and remark therein) Given $s\in\mathbb{N}$ and
$1<p<\infty$, consider an operator $\mathcal{L}$ of even order $2s$ that is
uniformly elliptic. Suppose the coefficients of $\mathcal{L}$ belong to
$C(\bar{\Omega})$ and the boundary $\partial\Omega$ is of class $C^{2s}$. If
$u$ is a solution to the equation (2.1) such that
$\|u\|_{W^{2s}_{p}(\Omega)}$, $\|f\|_{L^{p}(\Omega)}$, and
$\|g\|_{W^{2s-1/p}_{p}(\partial\Omega)}$ are all finite, then it follows that

$\|u\|_{W^{2s}_{p}(\Omega)}\lesssim\|f\|_{L^{p}(\Omega)}+\|g\|_{W^{2s-1/p}_{p}(\partial\Omega)}+\|u\|_{L^{p}(\Omega)}.$

Additionally, given the uniqueness of the solution to (2.1) with derivatives
of order up to $2s$ in $L^{p}$, the term $\|u\|_{L^{p}(\Omega)}$ can be
excluded, resulting in

$\|u\|_{W^{2s}_{p}(\Omega)}\lesssim\|f\|_{L^{p}(\Omega)}+\|g\|_{W^{2s-1/p}_{p}(\partial\Omega)}.$
(2.3)

######  Remark 1.

Some similar conclusions can be generalized to elliptic PDEs on unbounded
domains or Riemannian manifolds. Relevant discussions can be found in the
referenced materials such as [1] (Chapter V) regarding unbounded domains, [51]
(Chapter 5), [52], and [30] (Theorem 5.2. in Chapter III) that delve into
Riemannian manifolds. Additionally, the $W^{r}_{p}$ estimates for other types
of PDEs are also extensively studied. These estimates are integral to
understanding the existence, uniqueness, and regularity of the solution.

Returning to the discussion on PINN, let us consider a $2s$-order elliptic PDE
that satisfies the conditions stated in inequality (2.3). By deriving the
following inequality:

$\|u-u^{*}\|_{H^{2s}(\Omega)}\lesssim\|\mathcal{L}(u-u^{*})\|_{L^{2}(\Omega)}+\|u-u^{*}\|_{H^{2s-1/2}(\partial\Omega)},$

we observe that the norm of the boundary term is inconsistent with
$\mathcal{R}(u)-\mathcal{R}(u^{*})$. This can be a touchy issue and one may
have to impose a hard-constraint for the boundary condition to carry on the
analysis, that is, we assume that

$u(x)=g(x),\quad x\in\partial\Omega,u\in\mathcal{F}.$ (2.4)

Then

$\|u-u^{*}\|_{H^{2s}(\Omega)}\lesssim\|\mathcal{L}(u-u^{*})\|_{L^{2}(\Omega)},\quad\text{
for }u\in\mathcal{F}\cap H^{2s}(\Omega).$

The hard-constraint technique is commonly employed in PINN and other deep
learning-based solvers for PDEs. It can be implemented by multiplying a
distance function. In [37] (see Theorem B.12), the authors directly assume a
zero boundary condition for a neural network function space to avoid this
issue. However, this assumption lacks theoretical rigor. Notably, PDEs
formulated on the whole sphere can address this problem without necessitating
a hard-constraint, as the sphere is a boundaryless manifold. We will continue
these discussions in the next subsection.

Now we consider controlling $\mathcal{R}(u)-\mathcal{R}(u^{*})$ by
$\|u-u^{*}\|_{H^{s}(\Omega)}$, which is much more direct, since

$\displaystyle\mathcal{R}(u)-\mathcal{R}(u^{*})$
$\displaystyle\lesssim\|\mathcal{L}(u-u^{*})\|^{2}_{L^{2}(\Omega)}+\|u-u^{*}\|^{2}_{L^{2}(\partial\Omega)}$
(2.5)
$\displaystyle\lesssim\|u-u^{*}\|^{2}_{H^{s}(\Omega)}+\|u-u^{*}\|^{2}_{L^{2}(\partial\Omega)}.$

where we assume that $\mathcal{L}$ is a linear differential operator of order
$s$ with sup-norm bounded coefficient functions.

We conclude that the strong convexity of the PINN risk is a non-trivial
property, especially concerning the $W^{r}_{p}$ error estimates of $u-u^{*}$.
Another crucial consideration is the existence of a unique solution. If an
exact solution does not exist or its existence remains unclear, one can still
employ PINN to obtain an approximate solution through the minimization of
residuals. Particularly, when the solution lacks uniqueness, PINN is still
workable even though we might encounter challenges in ascertaining which
solution PINN aims to approximate. Many references are available on the well-
posedness of PDEs, and we will not go into further discussion here.

###  2.2 PINN for PDEs on Spheres

Prior to the discussion regarding PDEs on the sphere, let us give a brief
introduction to the Laplace-Beltrami operator and Sobolev spaces on the
sphere. One may refer to [10] for more details. Consider a unit sphere
$\mathbb{S}^{d-1}\subset\mathbb{R}^{d}$ for $d\geq 3$. Let $\Delta_{0}$
symbolize the spherical aspect of the Laplace operator, also known as the
Laplace-Beltrami operator, satisfying the equation

$\Delta=\frac{\partial^{2}}{\partial\rho^{2}}+\frac{d-1}{\rho}\frac{\partial}{\partial\rho}+\frac{1}{\rho^{2}}\Delta_{0}$
(2.6)

where $x=\rho\theta$ represents the spherical-polar coordinates, $\rho>0$
indicates the radius, and $\theta\in\mathbb{S}^{d-1}$. Applying equation (2.6)
and

$\frac{\partial}{\partial\rho}=\frac{1}{\rho}\sum_{i=1}^{d}x_{i}\frac{\partial}{\partial
x_{i}},$

one can directly calculate that

$\Delta_{0}=\sum_{i=1}^{d}\frac{\partial^{2}}{\partial
x_{i}^{2}}-\sum_{i=1}^{d}\sum_{j=1}^{d}x_{i}x_{j}\frac{\partial^{2}}{\partial
x_{i}\partial x_{j}}-(d-1)\sum_{i=1}^{d}x_{i}\frac{\partial}{\partial x_{i}}.$
(2.7)

The operator $\Delta_{0}$ is self-adjoint. Moreover, the spherical harmonics
are eigenfunctions of $\Delta_{0}$, with corresponding eigenvalues
$\lambda_{n}=-n(n+d-2)$, where $n=0,1,\cdots$.

The Sobolev space on a sphere, designated as $W^{r}_{p}(\mathbb{S}^{d-1})$, is
a particular subset of $L^{p}(\mathbb{S}^{d-1})$, given that $1\leq
p\leq\infty$ and $r\in\mathbb{N}$. This space is characterized by a finite
Sobolev norm:

$\|f\|_{W^{r}_{p}(\mathbb{S}^{d-1})}=\|f\|_{p}+\sum_{1\leq i<j\leq
d}\|D^{r}_{i,j}f\|_{p},$

where $\|\cdot\|_{p}$ denotes the $L^{p}(\mathbb{S}^{d-1})$ norm corresponding
to the uniform measure on $\mathbb{S}^{d-1}$. Here, $D_{i,j}$ denotes the
angular derivatives, defined as

$D_{i,j}=x_{i}\frac{\partial}{\partial x_{j}}-x_{j}\frac{\partial}{\partial
x_{i}}.$ (2.8)

Interestingly, the Laplace-Beltrami operator, $\Delta_{0}$, can be expressed
in terms of these angular derivatives:

$\Delta_{0}=\sum_{1\leq i<j\leq d}D^{2}_{i,j}.$

For the case where $p=2$, $W^{r}_{2}(\mathbb{S}^{d-1})$ is recognized as a
Hilbert space, for which a specific notation,
$H^{r}(\mathbb{S}^{d-1})=W^{r}_{2}(\mathbb{S}^{d-1})$, is utilized. One could
also consider a fractional-type Sobolev space, referred to as the Lipschitz
space, denoted as $W^{r,\alpha}_{p}(\mathbb{S}^{d-1})$, with $\alpha\in[0,1)$.
As such, the Sobolev space $W^{r}_{p}(\mathbb{S}^{d-1})$ can be defined where
$r\geq 0$. For an elaborate discussion of the Lipschitz space, the reader may
refer to Chapter 4 of [10].

We now formulate the PINN algorithm on the sphere as follows. We focus on
linear PDEs on $\mathbb{S}^{d-1}$ with $d\geq 3$, thus eliminating the
necessity of imposing a boundary condition:

$(\mathcal{L}u)(x)=f(x),\quad x\in\mathbb{S}^{d-1}.$ (2.9)

Let $\sigma$ be the Lebesgue measure on $\mathbb{S}^{d-1}$ and $\omega_{d}$ be
the surface area of $\mathbb{S}^{d-1}$. The population PINN risk is then
defined as

$\mathcal{R}(u)=\frac{1}{\omega_{d}}\int_{\mathbb{S}^{d-1}}|(\mathcal{L}u)(x)-f(x)|^{2}d\sigma(x).$

After drawing $n$ i.i.d. random variables $\\{X_{i}\\}_{i=1}^{n}$ from
$\mathbb{S}^{d-1}$ according to the uniform distribution, we define the
empirical risk as

$\mathcal{R}_{n}(u)=\frac{1}{n}\sum_{i=1}^{n}|(\mathcal{L}u)(X_{i})-f(X_{i})|^{2}.$

Consequently, the ERM estimator $u_{n}$, which belongs to a predefined
function space $\mathcal{F}$, is determined by the following optimization
problem:

$u_{n}=\arg\min_{u\in\mathcal{F}}\mathcal{R}_{n}(u).$ (2.10)

To enhance the algorithm’s stability for practical applications, it is
advantageous to consider the use of uniform lattices when generating training
sample points, as they can offer a minimal fill distance. An example of such a
lattice is the Fibonacci lattice, which provides near-uniform coverage on the
$2$-D sphere (see [2]). However, generating almost uniform lattices on a
general manifold can pose significant challenges. As an alternative, a uniform
random sampling method may be employed. For the sake of simplicity in our
analysis, we restrict our discussion to uniform random sampling on the sphere
in this paper.

As previously discussed in Subsection 2.1, we can identify approximate
solutions within a function space without the hard constraint (2.4), a
condition facilitated by the absence of a boundary on the sphere. However, it
remains essential to assume that (2.9) possesses a unique solution $u^{*}$.
Furthermore, the strong convexity of the PINN risk is a requirement for both
approximation and statistical analysis. For the approximation of $u^{*}$ using
neural networks, it is imperative to make reasonable assumptions regarding the
excess regularity of $u^{*}$. Take, for instance, $\mathcal{L}$, an elliptic
operator of order $s$. By (2.5), our aim is to approximate $u^{*}$ in
$H^{s}(\mathbb{S}^{d-1})$ norm, necessitating a higher order of smoothness of
$u^{*}$ according to our approximation analysis. We should then presume
$u^{*}\in H^{r}(\mathbb{S}^{d-1})$ for an $r\geq s+1$. Based on the elliptic
regularity theorem (Theorem 6.30 in [52]), we establish:

$u^{*}\in H^{-\infty}(\mathbb{S}^{d-1}),f\in
H^{t}(\mathbb{S}^{d-1})\Longrightarrow u^{*}\in H^{t+s}(\mathbb{S}^{d-1}),$

where $H^{-\infty}(\mathbb{S}^{d-1})$ represents the dual space of
$H^{\infty}(\mathbb{S}^{d-1})$. Consequently, it is sufficient to assume $f\in
H^{r-s}(\mathbb{S}^{d-1})$ in this context.

However, to employ concentration inequalities such as Bernstein and Talagrand
inequalities in the generalization analysis, the introduction of a boundedness
assumption becomes necessary. This requirement must be satisfied by both
$u^{*}$ and the approximate function $u\in\mathcal{F}$ up to derivatives of
order $s$. Our approximation analysis allows us to establish an upper bound
for $\|u\|_{W^{s}_{\infty}(\mathbb{S}^{d-1})}$ only if $u^{*}\in
W^{r}_{\infty}(\mathbb{S}^{d-1})$ where $r\geq s+1$. As a result, it is
essential to assume, at the very least, that $u^{*}\in
W^{r}_{\infty}(\mathbb{S}^{d-1})$ for some $r\geq s+1$. In conclusion, we
state the following assumption.

######  Assumption 1.

Consider the sphere PDE (2.9). Assume that $\mathcal{L}$ is a linear
differential operator of order $s\in\mathbb{N}$, taking the form:

$\mathcal{L}=\sum_{|\alpha|\leq s}a_{\alpha}(x)D^{\alpha},$ (2.11)

where $\alpha=(\alpha_{1},\alpha_{2},\cdots,\alpha_{n})$ denotes a multi-index
of non-negative integers, $|\alpha|=\alpha_{1}+\alpha_{2}+\cdots+\alpha_{n}$,
$a_{\alpha}(\cdot)\in L^{\infty}(\mathbb{S}^{d-1})$, and

$D^{\alpha}=\frac{\partial^{|\alpha|}}{\partial x^{\alpha_{1}}_{1}\partial
x^{\alpha_{2}}_{2}\cdots\partial x^{\alpha_{d}}_{d}}.$

Assume that (2.9) has a unique solution $u^{*}\in
W^{r}_{\infty}(\mathbb{S}^{d-1})$ for some $r\geq s+1$. Furthermore, assume
that the following $H^{s}(\mathbb{S}^{d-1})$ estimate holds for all $u\in
H^{s}(\mathbb{S}^{d-1})$:

$\|u-u^{*}\|_{H^{s}(\mathbb{S}^{d-1})}\lesssim\|\mathcal{L}(u-u^{*})\|_{L^{2}(\mathbb{S}^{d-1})}.$
(2.12)

######  Remark 2.

In Assumption 1, we suppose that the operator $\mathcal{L}$ is linear. The
linearity is assumed primarily to expound upon the strong convexity associated
with the PINN risk. However, this property is barely used in the convergence
analysis. As such, it is feasible to consider a nonlinear operator, denoted as
$\mathcal{T}$, defined as:

$\mathcal{T}=\sum_{|\alpha|\leq s}a_{\alpha}D^{\alpha}\mbox{ such that
}\mathcal{T}u=f,$

where coefficients $a_{\alpha}$ and the function $f$ might be dependent on the
unknown function $u$ and its higher-order derivatives. Given that strong
convexity is maintained with $\mathcal{T}$ as:

$\|\mathcal{T}u-\mathcal{T}u^{*}\|_{L^{2}(\mathbb{S}^{d-1})}\asymp\|u-u^{*}\|_{H^{s}(\mathbb{S}^{d-1})},$

and that $\mathcal{T}$ exhibits Lipschitz behavior for any $u_{1},u_{2}\in
H^{s}(\mathbb{S}^{d-1})$ on a point-wise basis (refer to (4.2)):

$\displaystyle\big{|}(\mathcal{T}u_{1})(x)-(\mathcal{T}u_{2})(x)\big{|}\lesssim\sum_{|\alpha|\leq
s}|D^{\alpha}u_{1}(x)-D^{\alpha}u_{2}(x)|,$

with coefficients $a_{\alpha}$ and $f$ possessing bounded sup-norms, our
analysis presented in this paper remains largely applicable. Nonetheless, it
warrants mention that verifying such conditions is extremely nontrivial for
nonlinear operators.

Consequently, the assumption immediately indicates that $f\in
L^{\infty}(\mathbb{S}^{d-1})$ and (2.5) holds true. We admit that the
assumption (2.12) is non-trivial and has been deliberated under certain
conditions in Subsection 2.1, wherein (2.12) could be met. Subsequently, we
provide two specific examples.

######  Example 1.

Consider the static Schrödinger equation given by

$-\Delta_{0}u(x)+V(x)u(x)=f(x),\quad x\in\mathbb{S}^{d-1}.$ (2.13)

Notice that $\Delta_{0}$ is self-adjoint with eigenvalues $\lambda\leq 0$. If
$V$ is a strictly positive constant function, then (2.13) has a unique
solution $u^{*}$ and exhibits a relation between the smoothness of $f$ and
$u^{*}$ as described by the elliptic regularity theorem:

$u^{*}\in H^{r}(\mathbb{S}^{d-1})\Longleftrightarrow f\in
H^{r-2}(\mathbb{S}^{d-1}).$

Furthermore, the strong convexity of the PINN risk is demonstrated in relation
to the $H^{2}(\mathbb{S}^{d-1})$ norm: for all $u\in H^{2}(\mathbb{S}^{d-1})$,
it can be computed that

$\displaystyle\mathcal{R}(u)-\mathcal{R}(u^{*})$
$\displaystyle=\frac{1}{\omega_{d}}\int_{\mathbb{S}^{d-1}}|\Delta_{0}u(x)-Vu(x)+f(x)|^{2}d\sigma(x)$
$\displaystyle=\frac{1}{\omega_{d}}\int_{\mathbb{S}^{d-1}}|\Delta_{0}u(x)-Vu(x)-\Delta_{0}u^{*}(x)+Vu^{*}(x)|^{2}d\sigma(x)$
$\displaystyle=\frac{1}{\omega_{d}}\int_{\mathbb{S}^{d-1}}(\Delta_{0}u-\Delta_{0}u^{*})^{2}+(Vu-
Vu^{*})^{2}-2(\Delta_{0}u-\Delta_{0}u^{*})\cdot(Vu-Vu^{*})d\sigma$
$\displaystyle=\frac{1}{\omega_{d}}\int_{\mathbb{S}^{d-1}}(\Delta_{0}u-\Delta_{0}u^{*})^{2}+V^{2}(u-u^{*})^{2}+2V\|\nabla_{0}(u-u^{*})\|^{2}d\sigma.$

where $\nabla_{0}$ is the projected gradient operator on $\mathbb{S}^{d-1}$
(see Lemma 1.4.4. in [10]). Hence,

$\frac{\min\\{1,V^{2},2V\\}}{\omega_{d}}\|u-u^{*}\|^{2}_{H^{2}(\mathbb{S}^{d-1})}\leq\mathcal{R}(u)-\mathcal{R}(u^{*})\leq\frac{\max\\{1,V^{2},2V\\}}{\omega_{d}}\|u-u^{*}\|^{2}_{H^{2}(\mathbb{S}^{d-1})}.$

Having noticed that this equation enjoys many useful properties, we use it to
implement our experiments in Section 6.

######  Example 2.

Consider an elliptic equation of order $2s$ given by

$(-\Delta_{0}+\lambda I)^{s}u(x)=f(x),\quad x\in\mathbb{S}^{d-1},$ (2.14)

where $\lambda>0$. Similar to the discussion in Example 1, we can verify the
strong convexity of PINN risk. An alternative definition of the Sobolev space
$W^{r}_{p}(\mathbb{S}^{d-1})$ employs the operator $-\Delta_{0}+I$:

$\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}=\|(-\Delta_{0}+I)^{r/2}u\|_{p}.$

In this context, $r\in\mathbb{R}$ and the fractional power is defined in the
distributional sense via the spherical harmonic expansion. Therefore, this
equation inherently obeys the regularity theorem, such that

$u\in W^{r}_{p}(\mathbb{S}^{d-1})\Longleftrightarrow f\in
W^{r-2s}_{p}(\mathbb{S}^{d-1}).$

Accordingly, Assumption 1 is equivalent to assuming that $f\in
W^{r-2s}_{\infty}(\mathbb{S}^{d-1})$ for some $r\geq 2s+1$.

###  2.3 The CNN Architectures

In this study, we specifically concentrate on the computation of approximate
solutions through the ERM algorithm (2.10) within a designated space
$\mathcal{F}$, which is generated by 1-D Convolutional Neural Networks (CNNs)
induced by 1-D convolutions. We present the following definition of the CNN
architecture in the context of our analysis.

The CNN is specified by a series of convolution kernels,
$\\{w^{(l)}\\}_{l=1}^{L}$, where each $w^{(l)}:\mathbb{Z}\to\mathbb{R}$
represents a vector indexed by $\mathbb{Z}$ and supported on
$\\{0,\ldots,S^{(l)}-1\\}$, given a kernel size $S^{(l)}\geq 3$. We can
iteratively define a 1-D deep CNN with $L$ hidden layers using the following
expressions:

$\displaystyle F^{(0)}:\mathbb{R}^{d}\to\mathbb{R}^{d},\quad F^{(0)}(x)$
$\displaystyle=x;$ (2.15) $\displaystyle
F^{(l)}:\mathbb{R}^{d}\to\mathbb{R}^{d_{l}},\quad F^{(l)}(x)$
$\displaystyle=\sigma^{(l)}\left(\left(\sum_{j=1}^{d_{l-1}}w^{(l)}_{i-j}\big{(}F^{(l-1)}(x)\big{)}_{j}\right)_{i=1}^{d_{l}}-b^{(l)}\right),$
$\displaystyle l=1,\ldots,L.$

In the above formulations, we denote the network widths as
$d_{0}=d,\\{d_{l}=d_{l-1}+S^{(l)}-1\\}_{l=1}^{L}$ and define the bias as
$b^{(l)}\in\mathbb{R}^{d_{l}}$. The element-wise activation function,
$\sigma^{(l)}:\mathbb{R}\to\mathbb{R}$, utilizes the Rectified Linear Unit
(ReLU) function, $\sigma(x)=\max\\{x,0\\}$, which operates on each convolution
layer. The convolution of a sequence $w^{(l)}$ on $F^{(l-1)}$ can be described
through a convolutional matrix multiplication:

$\left(\sum_{j=1}^{d_{l-1}}w^{(l)}_{i-j}\big{(}F^{(l-1)}(x)\big{)}_{j}\right)_{i=1}^{d_{l}}=T^{(l)}F^{(l-1)}(x),$

where $T^{(l)}$ represents a $d_{l}\times d_{l-1}$ matrix, defined as:

$T^{(l)}=\begin{bmatrix}w^{(l)}_{0}&0&0&0&\cdots&0\\\
w^{(l)}_{1}&w^{(l)}_{0}&0&0&\cdots&0\\\
\vdots&\vdots&\vdots&\vdots&\vdots&\vdots\\\
w^{(l)}_{S^{(l)}-1}&\cdots&w^{(l)}_{0}&0&\cdots&0\\\
\vdots&\vdots&\vdots&\vdots&\vdots&\vdots\\\
0&\cdots&0&w^{(l)}_{S^{(l)}-1}&\cdots&w^{(l)}_{0}\\\
\vdots&\vdots&\vdots&\vdots&\vdots&\vdots\\\
0&0&0&\cdots&0&w^{(l)}_{S^{(l)}-1}\\\ \end{bmatrix}.$

Upon the completion of $L$ convolution layers, a pooling operation is
typically employed to decrease the output dimension. In this context, we
consider a downsampling operator
$\mathcal{D}:\mathbb{R}^{d_{L}}\to\mathbb{R}^{\left\lfloor
d_{L}/d\right\rfloor}$ defined as $\mathcal{D}(x)=(x_{id})_{i=1}^{\left\lfloor
d_{L}/d\right\rfloor}$. The convolution layers and pooling operator can be
viewed as a feature extraction model. Finally, $L_{0}$ fully connected layers
are implemented and an affine transformation computes the entire network’s
output $F^{(L+L_{0}+1)}(x)\in\mathbb{R}$ according to the following
formulation:

$\displaystyle F^{(L+1)}$
$\displaystyle:\mathbb{R}^{d}\to\mathbb{R}^{d_{L+1}},$ (2.16) $\displaystyle
F^{(L+1)}(x)=\sigma^{(L+1)}\left(W^{(L+1)}\mathcal{D}\big{(}F^{(L)}(x)\big{)}-b^{(L+1)}\right);$
$\displaystyle F^{(l)}$ $\displaystyle:\mathbb{R}^{d}\to\mathbb{R}^{d_{l}},$
$\displaystyle
F^{(l)}(x)=\sigma^{(l)}\left(W^{(l)}F^{(l-1)}(x)-b^{(l)}\right),\
l=L+2,\ldots,L+L_{0};$ $\displaystyle F^{(L+L_{0}+1)}$
$\displaystyle:\mathbb{R}^{d}\to\mathbb{R},$ $\displaystyle
F^{(L+L_{0}+1)}(x)=W^{(L+L_{0}+1)}\cdot F^{(L+L_{0})}(x)-b^{(L+L_{0}+1)}.$

Here, the terms $W^{(L+1)}\in\mathbb{R}^{d_{L+1}\times\left\lfloor
d_{L}/d\right\rfloor}$, $W^{(l)}\in\mathbb{R}^{d_{l}\times d_{l-1}}$ for
$l=L+2,\ldots,L+L_{0}$, and $W^{(L+L_{0}+1)}\in\mathbb{R}^{d_{L+L_{0}}}$
represent weight matrices. The elements $b^{(l)}\in\mathbb{R}^{d_{l}}$ for
$l=L+1,\ldots,L+L_{0}$, and $b^{(L+L_{0}+1)}\in\mathbb{R}$ are biases. As
stated in Section 1, it is not appropriate to use the ReLU function as the
single activation function in our PICNN model. The composite function of some
ReLU and affine functions becomes a piecewise linear function, which results
in the network output’s second derivative being strictly $0$. This outcome
prevents approximating the true solution with a smoothness order of at least
$s+1\geq 2$, a phenomenon known as saturation in approximation theory. To
address this issue, we utilize alternative activation functions with non-
linear second derivatives in place of the ReLU function in at least one fully
connected layer. Suitable alternatives encompass $\mathrm{ReLU}^{k}$ for
$k\geq s$, the logistic function, and smooth approximations of $\mathrm{ReLU}$
such as Softplus and $\mathrm{GeLU}$ [18]:

$\displaystyle\mathrm{ReLU}^{k}(x)$ $\displaystyle=\max\\{0,x\\}^{k};$
$\displaystyle\mathrm{logistic}(x)$ $\displaystyle=\frac{1}{1+e^{-x}};$
$\displaystyle\mathrm{Softplus}(x)$ $\displaystyle=\log(1+e^{x});$
$\displaystyle\mathrm{GeLU}(x)$ $\displaystyle=x\Phi(x)\approx
0.5x\big{(}1+\tanh[\sqrt{2/\pi}(x+0.044715x^{3})]\big{)}.$

Here, $\Phi$ is the cumulative distribution function of the standard normal
distribution. In comparison to the depth $L$ of convolution layers, the depth
$L_{0}$ of the Fully Connected Neural Network (FCNN) is typically much
smaller. In this study, we focus on the ReLUk case and set $L_{0}=2$.

The function space $\mathcal{F}$ is defined by the output $F^{(L+L_{0}+1)}$.
More specifically, $\mathcal{F}$ is characterized by network parameters
$L,L_{0},\\{S^{(l)}\\}_{l=1}^{L},\\{d_{l}\\}_{l=L+1}^{L+L_{0}}$, activation
functions $\\{\sigma^{(l)}\\}_{l=1}^{L+L_{0}}$, and supremum norm constraints
$B_{i}$ for $i=1,\ldots,4$:

$\displaystyle\max_{l=1,\ldots,L}\|w^{(l)}\|_{\infty}\leq B_{1},$ (2.17)
$\displaystyle\max_{l=1,\ldots,L}\|b^{(l)}\|_{\infty}\leq B_{2},$
$\displaystyle\max_{l=L+1,\ldots,L+L_{0}+1}\|W^{(l)}\|_{\max}\leq B_{3},$
$\displaystyle\max_{l=L+1,\ldots,L+L_{0}+1}\|b^{(l)}\|_{\infty}\leq B_{4}.$

Additionally, the total number of free parameters $\mathcal{S}$ also defines
the function space. Another perspective involves restricting the supremum norm
of the network output and its derivatives up to the order of $s$, which is
vital for ensuring a bounded condition in the contraction inequality.
Therefore, we choose $M>0$ to ensure

$\max_{|\alpha|\leq s}\|D^{\alpha}F^{(L+L_{0}+1)}\|_{\infty}\leq M\mbox{ when
restricting $F^{(L+L_{0}+1)}$ on $\mathbb{S}^{d-1}$.}$ (2.18)

Precisely, we describe the function space $\mathcal{F}$ generated by 1-D CNN,
which is given by

$\displaystyle\mathcal{F}(L,L_{0},S,d_{L+1},\ldots,d_{L+L_{0}},\mathrm{ReLU},\mathrm{ReLU}^{k},M,\mathcal{S})$
(2.19) $\displaystyle={}$
$\displaystyle\left\\{F^{(L+L_{0}+1)}:\mathbb{S}^{d-1}\to\mathbb{R}\left|\begin{array}[]{l}\text{$F^{(L+L_{0}+1)}$
satisfies \eqref{sup-norm constraint}, which}\\\ \text{is defined by
\eqref{CNN layer} and \eqref{FCNN layer} with}\\\
\text{$S^{(l)}=S,\sigma^{(l)}=\mathrm{ReLU}$ for $l\neq L+1$,}\\\
\text{$\sigma^{(l)}=\mathrm{ReLU}^{k}$ for $l=L+1,k\geq s$; }\\\ \text{the
total number of free parameters}\\\ \text{is less than or equal to
$\mathcal{S}$.}\end{array}\right.\right\\}.$

When contextually appropriate, we will continue to use the abbreviated
notation, $\mathcal{F}$, to denote the aforementioned function space. By
leveraging an innovative localization technique developed by a series of works
[28, 25, 4, 26, 27, 59], our function space associated with CNN does not
mandate a sup-norm restriction (2.17) on the network’s trainable parameters,
setting us apart from the previous analysis in [37]. However, for the sake of
comparison, we will provide sup-norm bounds during our discussion of CNN’s
approximation capabilities. In addition to Assumption 1, we propose a further
assumption, as outlined below.

######  Assumption 2.

The function space, denoted by

$\mathcal{F}=\mathcal{F}(L,L_{0},S,d_{L+1},\ldots,d_{L+L_{0}},\mathrm{ReLU},\mathrm{ReLU}^{k},M,\mathcal{S}),$

is defined as (2.19). Herein, $M$ is chosen to be sufficiently large such that
(2.18) is satisfied and

$\max_{|\alpha|\leq s}\|D^{\alpha}u^{*}\|_{\infty}\leq M,$ (2.20)

where $u^{*}\in W^{r}_{\infty}(\mathbb{S}^{d-1})$ with $r\geq s+1$ is the
unique solution of equation (2.9).

###  2.4 Main Result

In this subsection, our primary focus lies in presenting our main result,
where we establish fast convergence rates for PICNN in solving PDEs on
spheres. Recall that, the function space denoted by
$\mathcal{F}(L,L_{0},S,d_{L+1},\ldots,d_{L+L_{0}},\mathrm{ReLU},\mathrm{ReLU}^{k},M,\mathcal{S})$
produced by 1D CNNs is provided in (2.19). This space is parameterized by the
number of convolutional layers ($L$), fully connected layers ($L_{0}$), the
size of convolution kernels ($S$), the neuron count in fully connected layers
($\\{d_{l}\\}_{l=L+1}^{L+L_{0}}$), and the upper limits of output function and
the overall count of free parameters ($M$ and $\mathcal{S}$ respectively). We
highlight that the proposed CNN architecture (2.19) is supplemented with
downsampling layers and utilizes both $\mathrm{ReLU}$ and $\mathrm{ReLU}^{k}$
activation functions. The former activation function is employed on the
convolutional layers, while the latter operates on the fully connected layers.
Our theoretical analysis and numerical experiments necessitate just $L_{0}=2$.
This architecture, described in (2.19), has displayed impressive results in
areas such as natural language processing, speech recognition, and biomedical
data classification, as documented in [24] and the references therein. When
contrasted with 2D CNNs that are only designed for 2D data like images and
videos, 1D CNNs significantly curtail the computational load and are proven to
be effective for handling data generated by low-cost applications, especially
on portable devices. In this study, we apply this CNN architecture to solve
the spherical PDE represented by (2.9). We will demonstrate that under a mild
regularity condition, i.e., when the spherical PDE (2.9) satisfies Assumption
1, both the excess risk and estimation error of PICNN estimators decay at
polynomial rates. For two positive sequences, $\\{A_{n}\\}_{n\geq 1}$ and
$\\{B_{n}\\}_{n\geq 1}$, recall that $A_{n}\lesssim B_{n}$ indicates there
exists a positive constant $C$ independent of $n$, such that $A_{n}\leq
CB_{n},\forall\;n\geq 1$. Moreover, we write $A_{n}\asymp B_{n}$ if and only
if both $A_{n}\lesssim B_{n}$ and $B_{n}\lesssim A_{n}$ hold true. Recall that
$u^{*}$ denotes the solution of the equation (2.9). For simplicity, we say
$M\gtrsim\|u^{*}\|_{W^{r}_{\infty}(\mathbb{S}^{d-1})}$ if there is a universal
constant $C>0$ such that $M\geq
C\|u^{*}\|_{W^{r}_{\infty}(\mathbb{S}^{d-1})}$.

######  Theorem 1.

Suppose that Assumption 1 is satisfied with some $s\geq 1$ and $r\geq s+1$ and
Assumption 2 holds with a sufficiently large $M$ satisfying
$M\gtrsim\|u^{*}\|_{W^{r}_{\infty}(\mathbb{S}^{d-1})}$. Let $d\geq 3$ and
$\mathbf{x}=\\{X_{i}\\}_{i=1}^{n}$ be the i.i.d. sample following the uniform
distribution on $\mathbb{S}^{d-1}$. Choose $3\leq S\leq d$ and $k$ satisfying

$\left\\{\begin{aligned} &k=s+\left\lceil\frac{r+s+2}{d-3}\right\rceil\geq
s+1,&&\mbox{ if }r<\infty,d>3,\\\ &k\geq s,&&\mbox{ if }r<\infty,d=3\mbox{ or
}r=\infty,d\geq 3.\\\ \end{aligned}\right.$

Let $u_{n}$ be the estimator of PICNN solving sphere PDE (2.9), which is
defined by (2.10) in the function space of CNN

$\mathcal{F}=\mathcal{F}(L,L_{0}=2,S,d_{L+1},d_{L+2},\mathrm{ReLU},\mathrm{ReLU}^{k},M,\mathcal{S})$

with

$\displaystyle L$ $\displaystyle\asymp n^{\frac{a(d-1)}{2(r-s)}}(\log
n)^{-\frac{a(d-1)}{r-s}},$ $\displaystyle d_{L+1}$ $\displaystyle\asymp
n^{\frac{a(d+r+s-1)}{2(r-s)(k-s+1)}+\frac{a(d+1)}{2(r-s)}}(\log
n)^{-\frac{a(d+r+s-1)}{(r-s)(k-s+1)}-\frac{a(d+1)}{r-s}},$ $\displaystyle
d_{L+2}$ $\displaystyle\asymp n^{\frac{a(d-1)}{2(r-s)}}(\log
n)^{-\frac{a(d-1)}{r-s}},$

and

$\displaystyle\mathcal{S}\asymp\left\\{\begin{aligned}
&n^{\frac{a(d-1)}{2(r-s)}}(\log n)^{-\frac{a(d-1)}{r-s}},&&\mbox{ if
}r<\infty,d>3,\\\ &n^{\frac{a(d+r+s-1)}{2(r-s)(k-s+1)}+\frac{a}{r-s}}(\log
n)^{-\frac{a(d+r+s-1)}{(r-s)(k-s+1)}-\frac{2a}{r-s}},&&\mbox{ if
}r<\infty,d=3\mbox{ or }r=\infty,d\geq 3.\end{aligned}\right.$

Here, the constant $a$ is given by

$a=\left\\{\begin{aligned} &\frac{r-s}{(r-s)+(d-1)},&&\mbox{ for
}r<\infty,d>3,\\\ &1-\frac{r+4k+6-3s}{2(r-s)(k-s+1)+r+4k+6-3s},&&\mbox{ for
}r<\infty,d=3,\\\ &1-\frac{1}{2(k-s)+3},&&\mbox{ for }r=\infty,d\geq
3.\end{aligned}\right.$

Then for all $n\geq 1$, with probability at least $1-\exp(-n^{1-a}(\log
n)^{2a})$, there hold

$\mathcal{R}(u_{n})-\mathcal{R}(u^{*})\lesssim n^{-a}(\log n)^{2a}$

and

$\|u_{n}-u^{*}\|_{H^{s}(\mathbb{S}^{d-1})}\lesssim n^{-a/2}(\log n)^{a}.$

To the best of our knowledge, Theorem 1 provides the first rigorous analysis
of convergence for the PINN algorithm with a CNN architecture, thereby
demonstrating its practical performance. Recent advancements in the theory of
approximation and complexity for deep ReLU CNNs lay the foundation for our
results. A significant contribution of our proof is the implementation of a
scale-sensitive localization theory with scale-insensitive measures, such as
VC-dimension. This approach is coupled with recent work on CNN approximation
theory [58, 57, 14, 15], enabling us to derive these elegant bounds and fast
rates. The idea of our localization is rooted in the work of [28, 25, 4, 59],
which allows us to examine broader classes of neural networks. This contrasts
with analyses that limit the networks to have bounded parameters for each
unit, a constraint resulting from applying scale-sensitive measures like
metric entropy. In our proof, we have broadened previous approximation results
and established an estimation of VC-dimensions involving the function spaces
generated by CNNs and their derivatives. These results hold significance in
their own right and warrant special attention (refer to Section 3 and Section
4). To our knowledge, our work is the first to establish corresponding
localization analysis for estimating the part of the stochastic error in the
theoretical analysis framework of solving equations with neural networks. Our
results can be generalized to theoretical analysis of other PDE solvers with
CNNs. The sphere is commonly viewed as the quintessential low-dimension
manifold; hence, our work has the potential to be extrapolated to PDE solvers
operating on general manifolds. Exploring further possibilities by extending
this approach to other neural network architectures and different types of
PDEs can certainly be advantageous. We posit, with firm conviction, that for
an $s$-order PDE PINN solver operating on an arbitrary $m$-dimensional
manifold $\mathcal{M}$ embedded in $\mathbb{R}^{d}$, and assuming the true
solution $u^{*}\in C^{r}(\mathcal{M})$ (or within another regularity function
space, such as Sobolev space), it is plausible to demonstrate a convergence
rate of $n^{-\frac{r-s}{r-s+m}}$ (modulated by a logarithmic factor). This
intriguing prospect presents promising avenues for future research.

###  2.5 Error Decomposition

This subsection primarily establishes the proof structure for Theorem 1, which
is derived from an error decomposition of the excess risk that we introduce
below. Error decomposition is a standard paradigm for deducing the
generalization bounds of the ERM algorithm. Research on algorithmic
generalization bounds is a fundamental concern in learning theory.
Specifically, it involves providing a theoretical non-asymptotic bound for the
excess risk $\mathcal{R}(u_{n})-\inf_{u}\mathcal{R}(u)$ with respect the
number of training samples $n$. In our PINN model, given our assumption that a
true solution $u^{*}$ exists, we have
$\inf_{u}\mathcal{R}(u)=\mathcal{R}(u^{*})=0$. However, for the sake of
conventional analysis, we will continue to express the excess risk as
$\mathcal{R}(u_{n})-\mathcal{R}(u^{*})$. We now apply a standard error
decomposition procedure to the excess risk of PINN. If we define
$u_{\mathcal{F}}=\arg\min_{u\in\mathcal{F}}\mathcal{R}(u)$, we obtain the
following decomposition:

$\displaystyle\mathcal{R}(u_{n})-\mathcal{R}(u^{*})\leq$
$\displaystyle\big{(}\mathcal{R}(u_{\mathcal{F}})-\mathcal{R}(u^{*})\big{)}+\big{(}\mathcal{R}_{n}(u_{\mathcal{F}})-\mathcal{R}_{n}(u^{*})-\mathcal{R}(u_{\mathcal{F}})+\mathcal{R}(u^{*})\big{)}$
$\displaystyle+\big{(}\mathcal{R}(u_{n})-\mathcal{R}(u^{*})-\mathcal{R}_{n}(u_{n})+\mathcal{R}_{n}(u^{*})\big{)}.$

The first term is often referred to as the approximation error, while the
third term is known as the estimation error or statistical error. The second
term can be bounded by the following lemma through using Bernstein’s
inequality.

######  Lemma 3.

If Assumption 1 and Assumption 2 hold, then for each
$u\in\mathcal{F}\cup\\{u^{*}\\}$,

$\mathbb{E}_{P}(L(X,u)-L(X,u^{*}))^{2}\leq
C_{0}\mathbb{E}_{P}(L(X,u)-L(X,u^{*}))=C_{0}(\mathcal{R}(u)-\mathcal{R}(u^{*})),$
(2.21)

where $P$ is the uniform distribution on $\mathbb{S}^{d-1}$,
$C_{0}=(M\cdot\sum_{|\alpha|\leq
s}\|a_{\alpha}\|_{\infty}+\|f\|_{\infty})^{2}$ and

$L(x,u)=|(\mathcal{L}u)(x)-f(x)|^{2}.$

Furthermore, for all $t>0$, with probability at least $1-\exp(-t)$,

$\mathcal{R}_{n}(u_{\mathcal{F}})-\mathcal{R}_{n}(u^{*})-\mathcal{R}(u_{\mathcal{F}})+\mathcal{R}(u^{*})\leq\mathcal{R}(u_{\mathcal{F}})-\mathcal{R}(u^{*})+\frac{7C_{0}t}{6n}.$

###### Proof.

Notice that $u^{*}$ is the solution of the equation (2.9), $L(x,u^{*})\equiv
0$ so (2.21) holds for $u=u^{*}$. For $u\in\mathcal{F}$ we have

$\displaystyle\mathbb{E}_{P}(L(X,u)-L(X,u^{*}))^{2}$ $\displaystyle={}$
$\displaystyle\frac{1}{\omega_{d}}\int_{\mathbb{S}^{d-1}}|(\mathcal{L}u)(x)-f(x)|^{4}d\sigma(x)$
$\displaystyle\leq{}$
$\displaystyle\frac{C_{0}}{\omega_{d}}\int_{\mathbb{S}^{d-1}}|(\mathcal{L}u)(x)-f(x)|^{2}d\sigma(x)$
$\displaystyle={}$ $\displaystyle C_{0}\mathbb{E}_{P}(L(X,u)-L(X,u^{*})).$

By Bernstein’s inequality and the above estimate, with probability at least
$1-\exp(-t)$,

$\displaystyle\mathcal{R}_{n}(u_{\mathcal{F}})-\mathcal{R}_{n}(u^{*})-\mathcal{R}(u_{\mathcal{F}})+\mathcal{R}(u^{*})$
$\displaystyle\leq{}$
$\displaystyle\sqrt{\frac{2C_{0}\mathbb{E}_{P}(L(X,u_{\mathcal{F}})-L(X,u^{*}))t}{n}}+\frac{2C_{0}t}{3n}$
$\displaystyle\leq{}$
$\displaystyle\mathbb{E}_{P}(L(X,u_{\mathcal{F}})-L(X,u^{*}))+\frac{7C_{0}t}{6n}$
$\displaystyle={}$
$\displaystyle\mathcal{R}(u_{\mathcal{F}})-\mathcal{R}(u^{*})+\frac{7C_{0}t}{6n}.$

Then we complete the proof. ∎

As observed from Lemma 3, with a probability of at least $1-\exp(-t)$, the
following holds:

$\displaystyle\mathcal{R}(u_{n})-\mathcal{R}(u^{*})\leq{}$ $\displaystyle
2\big{(}\mathcal{R}(u_{\mathcal{F}})-\mathcal{R}(u^{*})\big{)}+\frac{7C_{0}t}{6n}$
$\displaystyle+\big{(}\mathcal{R}(u_{n})-\mathcal{R}(u^{*})-\mathcal{R}_{n}(u_{n})+\mathcal{R}_{n}(u^{*})\big{)}.$

Therefore, we can see that to bound $\mathcal{R}(u_{n})-\mathcal{R}(u^{*})$,
we need only derive the upper bounds for the approximation error and the
statistical error respectively. In Section 3, we will establish the estimation
of the approximation error, and in Section 4, we will develop the estimation
of the statistical error. By combining these two, we can derive the
generalization bound for the excess risk
$\mathcal{R}(u_{n})-\mathcal{R}(u^{*})$, and hence, provide the proof for
Theorem 1. A comprehensive proof of Theorem 1 can be found in Section 5.

##  3 Approximation Error Analysis

This section focuses on the estimation of the approximation error. Recent
progress in the approximation theory of deep ReLU CNN [58, 57, 14, 15] and
spherical harmonic analysis [10] are important building blocks for our
estimates. As referenced in Assumption 1, it is assumed that $u^{*}\in
W^{r}_{\infty}(\mathbb{S}^{d-1})$ for some $r\geq s+1$. To establish the
generalization error analysis for spherical data classification, [15] has
obtained an $L^{p}(\mathbb{S}^{d-1})$ norm approximation rate for functions in
$W^{r}_{p}(\mathbb{S}^{d-1})$ using CNNs. In our PINN setup, the
$L^{p}(\mathbb{S}^{d-1})$ approximation is insufficient as we require an
approximation rate of the $H^{s}(\mathbb{S}^{d-1})$ norm, as outlined by
(2.5). Moreover, their work only considers a ReLU network, which is suitable
for an $L^{p}(\mathbb{S}^{d-1})$ norm approximation but not applicable here
due to the phenomenon of saturation. Hence, we employ and extend their results
to the Sobolev norm approximation and our $\mathrm{ReLU}$-$\mathrm{ReLU}^{k}$
network architectures. The approximator [15] utilizes is known as the near-
best approximation by polynomials. Let $u\in L^{p}(\mathbb{S}^{d-1})$ where
$1\leq p\leq\infty$. For $n_{0}\in\mathbb{N}$, we define the error of the best
approximation to $u$ by polynomials of degree at most $n_{0}$ as

$E_{n_{0}}(u)_{p}=\inf_{v\in\varPi_{n_{0}}(\mathbb{S}^{d-1})}\|u-v\|_{p},$

where $\varPi_{n_{0}}(\mathbb{S}^{d-1})$ denotes the space of polynomials of
degree at most $n_{0}$. Let $\lambda=\frac{d-2}{2}$, and let
$C^{\lambda}_{k}(t)$ represent the Gegenbauer polynomial of degree $k$ with
parameter $\lambda$. Given a function $\eta\in C^{\infty}([0,\infty))$ where
$\eta(t)=1$ for $0\leq t\leq 1$, $0\leq\eta(t)\leq 1$ for $1\leq t\leq 2$, and
$\eta(t)=0$ for $t\geq 2$, we define the kernel $l_{n_{0}}$ by

$l_{n_{0}}(t)=\sum_{k=0}^{2n_{0}}\eta\bigg{(}\frac{k}{n_{0}}\bigg{)}\frac{\lambda+k}{\lambda}C^{\lambda}_{k}(t),\quad
t\in[-1,1],$

and the corresponding linear operator

$L_{n_{0}}(u)(x)=\frac{1}{\omega_{d}}\int_{\mathbb{S}^{d-1}}u(y)l_{n_{0}}(\langle
x,y\rangle)d\sigma(y),\quad x\in\mathbb{S}^{d-1}.$

This linear operator provides a near-best approximation in the $L^{p}$ norm.
Notice that the construction of the function $\eta$ is not unique, but we can
fix a specific $\eta$ beforehand in the following statements, i.e., we can
construct a uniform $\eta$ applied to derive all the estimates.

######  Lemma 4.

(Theorem 2.6.3 in [10]) Let $u\in L^{p}(\mathbb{S}^{d-1})$ for $1\leq
p\leq\infty$. Then there exists a constant $C_{1}$ only depending on $\eta$
and $d$, such that for all $n_{0}>0$,

$\displaystyle\|L_{n_{0}}(u)\|_{p}$ $\displaystyle\leq C_{1}\|u\|_{p},$
$\displaystyle\|u-L_{n_{0}}(u)\|_{p}$
$\displaystyle\leq(1+C_{1})E_{n_{0}}(u)_{p}.$

However, $L_{n_{0}}(u)$ is not the approximator we ultimately use to estimate
the error. We introduce a new kernel, $\widetilde{l}_{n_{0}}$, and a linear
operator, $\widetilde{L}_{n_{0}}$, defined as follows:

$\begin{split}\widetilde{l}_{n_{0}}(t)&=\sum_{k=0}^{2n_{0}}\bigg{[}\eta\bigg{(}\frac{k}{n_{0}}\bigg{)}\bigg{]}^{2}\frac{\lambda+k}{\lambda}C^{\lambda}_{k}(t),\quad
t\in[-1,1],\\\
\widetilde{L}_{n_{0}}(u)(x)&=\frac{1}{\omega_{d}}\int_{\mathbb{S}^{d-1}}u(y)\widetilde{l}_{n_{0}}(\langle
x,y\rangle)d\sigma(y),\quad x\in\mathbb{S}^{d-1}.\end{split}$ (3.1)

It should be noted that the function $\eta^{2}$ still meets all the conditions
required above for the function $\eta$. Additionally,
$\widetilde{L}_{n_{0}}(u)$ exhibits near-best approximation properties. We
select $\widetilde{L}_{n_{0}}(u)$ because the integral (3.1) can be
discretized using a cubature formula, thereby expressing it as an additive
ridge function. According to Lemma 3 in [15], we have the following lemma.

######  Lemma 5.

Let $u\in L^{p}(\mathbb{S}^{d-1})$ for $1\leq p\leq\infty$. Then there exists
a constant $C_{2}$ only depending on $d$ such that, for all $m\geq
C_{2}n_{0}^{d-1}$, there exists a cubature rule
$\\{(\mu_{i},y_{i})\\}_{i=1}^{m}$ of degree $4n_{0}$ with
$\mu_{i}\in\mathbb{R}$, $y_{i}\in\mathbb{S}^{d-1}$ and we have

$\widetilde{L}_{n_{0}}(u)(x)=\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})l_{n_{0}}(\langle
x,y_{i}\rangle).$

As a direct consequence, Lemma 3 in [15] asserts that for all
$r\in\mathbb{N},1\leq p\leq\infty$ and $u\in W^{r}_{p}(\mathbb{S}^{d-1})$, one
can obtain

$\bigg{\|}u-\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})l_{n_{0}}(\langle
x,y_{i}\rangle)\bigg{\|}_{p}\leq
C_{3}n_{0}^{-r}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}$

with a constant $C_{3}$ only depends on $\eta$ and $d$. This approximation
bound can be extended to $W^{s}_{p}(\mathbb{S}^{d-1})$ norm approximation with
$s\leq r-1$, as illustrated by Chapter 4 of [10]. Recall that $D_{i,j}$
denotes the angular derivatives given by (2.8).

######  Lemma 6.

If $u\in W^{r}_{p}(\mathbb{S}^{d-1})$ for $1\leq p\leq\infty$ and
$r\in\mathbb{N}$, then there exists a constant $C_{4}$ only depending on
$\eta$ and $d$, such that

$\displaystyle\|D^{r}_{i,j}(u-L_{n_{0}}(u))\|_{p}$ $\displaystyle\leq
C_{4}E_{n_{0}}(D^{r}_{i,j}u)_{p},\quad 1\leq i<j\leq d,$ (3.2)
$\displaystyle\|D^{r}_{i,j}L_{n_{0}}(u)\|_{p}$ $\displaystyle\leq
C_{4}\|D^{r}_{i,j}u\|_{p},\quad\quad\ 1\leq i<j\leq d,$ (3.3) $\displaystyle
E_{n_{0}}(u)_{p}$ $\displaystyle\leq
C_{4}n_{0}^{-r}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}.$ (3.4)

Consequently, for any positive integer $s\leq r-1$ we have

$\displaystyle\|u-L_{n_{0}}(u)\|_{W^{s}_{p}(\mathbb{S}^{d-1})}$
$\displaystyle\leq C_{4}n_{0}^{s-r}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})},$ (3.5)
$\displaystyle\|L_{n_{0}}(u)\|_{W^{r}_{p}(\mathbb{S}^{d-1})}$
$\displaystyle\leq C_{4}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}.$ (3.6)

All these inequalities are also valid for $\widetilde{L}_{n_{0}}$. Hence, we
have

$\displaystyle\bigg{\|}u-\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})l_{n_{0}}(\langle
x,y_{i}\rangle)\bigg{\|}_{W^{s}_{p}(\mathbb{S}^{d-1})}$ $\displaystyle\leq
C_{4}n_{0}^{s-r}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})},$
$\displaystyle\bigg{\|}\sum_{i=1}^{m}L_{n_{0}}(u)(y_{i})l_{n_{0}}(\langle
x,y_{i}\rangle)\bigg{\|}_{W^{r}_{p}(\mathbb{S}^{d-1})}$ $\displaystyle\leq
C_{4}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}.$

###### Proof.

For the validation of (3.2) and (3.4), one can refer to Theorem 4.5.5 and
Corollary 4.5.6 in [10]. Inequality (3.3) is a direct consequence of Lemma 4
and the equality $L_{n_{0}}D^{r}_{i,j}=D^{r}_{i,j}L_{n_{0}}$. For any positive
integer $s\leq r-1$, we write

$\|u-L_{n_{0}}(u)\|_{W^{s}_{p}(\mathbb{S}^{d-1})}=\|u-L_{n_{0}}(u)\|_{p}+\sum_{1\leq
i<j\leq d}\|D^{s}_{i,j}(u-L_{n_{0}}(u))\|_{p}.$ (3.7)

Observing that $D^{s}_{i,j}u\in W^{r-s}_{p}(\mathbb{S}^{d-1})$, the
combination of (3.2) and (3.4) yields

$\displaystyle\|u-L_{n_{0}}(u)\|_{p}$ $\displaystyle\leq
C_{4}E_{n_{0}}(u)_{p}\leq C_{4}n_{0}^{-r}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})},$
$\displaystyle\|D^{s}_{i,j}(u-L_{n_{0}}(u))\|_{p}$ $\displaystyle\leq
C_{4}E_{n_{0}}(D^{s}_{i,j}u)_{p}\leq
C_{4}n_{0}^{s-r}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}.$

Combining these inequalities with (3.7) results in (3.5), and (3.6) is a
direct inference from (3.3) (with a possible reselection of the constant
$C_{4}$). Hence, the proof is completed. ∎

######  Remark 3.

According to Theorem 4.8.1 in [10], the approximation bound is also applicable
to the Lipschitz space $W^{r,\alpha}_{p}(\mathbb{S}^{d-1})$ for
$r\in\mathbb{N}$ and $\alpha\in{[0,1)}$, as indicated by the following:

$E_{n_{0}}(u)_{p}\leq
C_{4}n_{0}^{-r-\alpha}\|u\|_{W^{r,\alpha}_{p}(\mathbb{S}^{d-1})}.$

Thus, we can further extend the conclusion of Lemma 6 to encompass fractional-
type Sobolev solutions $u\in W^{r}_{p}(\mathbb{S}^{d-1})$ where $r\geq 1$ is a
real number.

Given the cubature rule $\\{(\mu_{i},y_{i})\\}_{i=1}^{m}$ as claimed in Lemma
5, convolution layers equipped with a downsampling operator can extract the
inner product $\langle x,y_{i}\rangle$ for all $i=1,\ldots,m$ as
representative features.

######  Lemma 7.

For $m\in\mathbb{N}$ and a set of cubature samples
$y=\\{y_{1},\ldots,y_{m}\\}\subset\mathbb{S}^{d-1}$, there exists a sequence
of convolution kernels $\\{w^{(l)}\\}_{l=1}^{L}$ supported on
$\\{0,\ldots,S^{(l)}-1\\}$ of equal size $S^{(l)}=S$ and depth
$L\leq\left\lceil\frac{md-1}{S-2}\right\rceil$. Additionally, bias vectors
$b^{(l)}$ exist for $l=1,\ldots,L$ such that

$\mathcal{D}(F^{(L)}(x))=\begin{bmatrix}\langle x,y_{1}\rangle\\\ \vdots\\\
\langle x,y_{m}\rangle\\\ 0\\\ \vdots\\\ 0\end{bmatrix}+B^{(L)}{\bf
1}_{\left\lfloor\frac{d+L(S-1)}{d}\right\rfloor},$

where $B^{(L)}=\prod_{l=1}^{L}\|w^{(l)}\|_{1}$. Further, we have the following
sup-norm bounds:

$\displaystyle\|w^{(l)}\|_{\infty}$ $\displaystyle\leq C_{5},\quad
l=1,\ldots,L,$ $\displaystyle\|b^{(l)}\|_{\infty}$ $\displaystyle\leq
C_{6}^{l},\quad l=1,\ldots,L,$

Here, the constants $C_{5}$ and $C_{6}$ only depend on $S$. The total number
of free parameters contributed by $\\{w^{(l)}\\}_{l=1}^{L}$ and
$\\{b^{(l)}\\}_{l=1}^{L}$ is $3LS-L$.

###### Proof.

The construction of the CNN and the total number of free parameters are given
in the proof of Lemma 3 in [14]. Herein, we will focus on providing proof for
the sup-norm bounds.

Define a sequence $W$ supported in $\\{0,\ldots,md-1\\}$ as
$W_{(j-1)d+(d-i)}=(y_{j})_{i}$, where $j=1,\ldots,m$ and $i=1,\ldots,d$.
Following Theorem 3 in [58], there exists a sequence of convolution kernels
$\\{w^{(l)}\\}_{l=1}^{L}$ supported on $\\{0,\ldots,S-1\\}$ with
$L=\left\lceil\frac{md-1}{S-2}\right\rceil$ such that $W$ admits the
convolutional factorization:

$W=w^{(L)}\ast w^{(L-1)}\ast\cdots\ast w^{(1)}$

with the corresponding convolution matrices satisfying

$T=T^{(L)}T^{(L-1)}\cdots T^{(1)}.$

Given a convolution kernel $w$, one can define a polynomial on $\mathbb{C}$ by
$w(z)=\sum_{i=0}^{\infty}w_{i}z^{i}$. Also by the proof of Theorem 3 in [58],
we obtain the following polynomial factorization:

$W(z)=w^{(L)}(z)w^{(L-1)}(z)\cdots w^{(1)}(z),$

according to the convolutional factorization above.

Considering a rotation of the cubature sample $y=\\{y_{1},\ldots,y_{m}\\}$, we
can assume that $y_{m}=(1,0,\cdots,0)$, which implies that $W_{md-1}=1$.
Consequently, the polynomial $W(z)$ can be entirely factorized as

$W(z)=\prod_{i=1}^{s_{1}}(z^{2}-2x_{i}z+x_{i}^{2}+y_{i}^{2})\prod_{j=2s_{1}+1}^{md-1}(z-x_{j}),$

comprising $2s_{1}$ complex roots $x_{i}\pm iy_{i}$ and $md-2s_{1}-1$ real
roots $x_{j}$, both appearing with multiplicity. We can construct $w^{(l)}$ by
taking some quadratic factors and linear factors from the above factorization
so that $w^{(l)}(z)$ is a polynomial of degree up to $S-1$. Employing Cauchy’s
bound on the magnitudes of all complex roots, we establish that

$|x_{j}|\vee|x_{i}\pm iy_{i}|\leq
1+\max\bigg{\\{}\bigg{|}\frac{W_{md-2}}{W_{md-1}}\bigg{|},\ldots,\bigg{|}\frac{W_{0}}{W_{md-1}}\bigg{|}\bigg{\\}}\leq
2.$

Therefore, the coefficients of $w^{(l)}(z)$ are bounded by a constant $C_{5}$,
which depends only on $S$. This verifies that

$\|w^{(l)}\|_{\infty}\leq C_{5},\quad l=1,\ldots,L.$

Adopting the proof of Lemma 3 in [14], we set
$b^{(1)}=-\|w^{(1)}\|_{1}\textbf{1}_{d_{1}}$ and

$b^{(l)}=\left(\prod_{i=1}^{l-1}\|w^{(i)}\|_{1}\right)T^{(l)}\textbf{1}_{d_{l-1}}-\left(\prod_{i=1}^{l}\|w^{(i)}\|_{1}\right)\textbf{1}_{d_{l}},\mbox{
for }l=2,\ldots,L,$

where $\textbf{1}_{d_{l}}$ is the constant $1$ vector in $\mathbb{R}^{d_{l}}$.
Consequently, we find that $\|b^{(1)}\|_{\infty}=\|w^{(1)}\|_{1}\leq S\cdot
C_{5}$ and

$\displaystyle\|b^{(l)}\|_{\infty}\leq\bigg{(}\prod_{i=1}^{l-1}\|w^{(i)}\|_{1}\bigg{)}\cdot\|w^{(l)}\|_{1}+\prod_{i=1}^{l}\|w^{(i)}\|_{1}\leq
2\bigg{(}\prod_{i=1}^{l}\|w^{(i)}\|_{1}\bigg{)}\leq 2S^{l}C^{l}_{5}.$

Finally setting $C_{6}=2SC_{5}$ suffices to complete the proof. ∎

The additive ridge function
$\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})l_{n_{0}}(\langle x,y_{i}\rangle)$ is
employed to approximate the function $u\in W^{r}_{p}(\mathbb{S}^{d-1})$. This
approximation requires an intermediary approximation of the univariate
function $l_{n_{0}}:[-1,1]\to\mathbb{R}$. For this purpose, the interpolant of
B-spline functions is utilized for the ReLUk-1 FCNN.

Consider $u_{1},\ldots,u_{k}$ as functions defined on an interval
$I\subset\mathbb{R}$, with $t_{1}\leq t_{2}\leq\cdots\leq t_{k}$ as points in
$I$. Suppose that

$t_{1},\ldots,t_{k}=\tau_{1},\ldots,\tau_{1},\ldots,\tau_{l},\ldots,\tau_{l},$

where each $\tau_{i}$ is repeated $k_{i}$ times and $\sum_{i=1}^{l}k_{i}=k$.
The $i$-th derivative of $u$ is denoted by $D^{i}u$, and we define a specific
determinant based on these derivatives and the functions, as follows:

$\displaystyle D\begin{pmatrix}t_{1},\cdots,t_{k}\\\
u_{1},\cdots,u_{k}\end{pmatrix}=\det\begin{bmatrix}u_{1}(\tau_{1})&u_{2}(\tau_{1})&\cdots&u_{k}(\tau_{1})\\\
Du_{1}(\tau_{1})&Du_{2}(\tau_{1})&\cdots&Du_{k}(\tau_{1})\\\
\vdots&\vdots&&\vdots\\\
D^{k_{1}-1}u_{1}(\tau_{1})&D^{k_{1}-1}u_{2}(\tau_{1})&\cdots&D^{k_{1}-1}u_{k}(\tau_{1})\\\
\vdots&\vdots&&\vdots\\\
u_{1}(\tau_{l})&u_{2}(\tau_{l})&\cdots&u_{k}(\tau_{l})\\\
Du_{1}(\tau_{l})&Du_{2}(\tau_{l})&\cdots&Du_{k}(\tau_{l})\\\
\vdots&\vdots&&\vdots\\\
D^{k_{l}-1}u_{1}(\tau_{l})&D^{k_{l}-1}u_{2}(\tau_{l})&\cdots&D^{k_{l}-1}u_{k}(\tau_{l})\\\
\end{bmatrix}.$

Given an integer $k>0$, we define the $k$-th order divided difference of
function $f$ on interval $I$ over points $t_{1},\ldots,t_{k+1}$ as

$[t_{1},\cdots,t_{k+1}]f=\frac{D\begin{pmatrix}t_{1},t_{2},\cdots,t_{k},t_{k+1}\\\
1,x,\cdots,x^{k-1},f\end{pmatrix}}{D\begin{pmatrix}t_{1},t_{2},\cdots,t_{k},t_{k+1}\\\
1,x,\cdots,x^{k-1},x^{k}\end{pmatrix}}.$

Consider a sequence of real numbers $\cdots\leq t_{-1}\leq t_{0}\leq
t_{1}\leq\cdots$, and for integers $i$ and $k>0$, we define the normalized
$k$-th order B-spline associated with $t_{i},\ldots,t_{i+k}$ as

$N_{i}^{k}(x)=(-1)^{k}(t_{i+k}-t_{i})[t_{i},\cdots,t_{i+k}](x-t)^{k-1}_{+}.$

Here we notice that $(x)^{k-1}_{+}$ is exactly the $\mathrm{ReLU}^{k-1}$
function.

For $N>0$, we fix an extended uniform partition of $[-1,1]$:

$t_{1}=\cdots=t_{k}=-1<t_{k+1}<\cdots<t_{2N+k-1}<1=t_{2N+k}=\cdots=t_{2N+2k-1}$

with $t_{k+i}=-1+\frac{i}{N}$ for $i=0,\ldots,2N$. For $f\in C[-1,1]$, the
corresponding interpolation spline is defined as

$Q^{k}_{N}(f)=\sum_{i=1}^{2N+k-1}J_{i}(f)N^{k}_{i}(x),$ (3.8)

where $J_{i}$ are certain fixed linear functionals. By Theorem 6.22 in [46] we
know that

$\|Q^{k}_{N}(f)\|_{\infty}\vee|J_{i}(f)|\leq(2k)^{k}\|f\|_{\infty}.$ (3.9)

Moreover, we have the following lemma.

######  Lemma 8.

(Corollary 6.21 in [46]) There exists a constant $C_{7}$ only depending on $k$
such that, for all $f\in C^{k-1}[-1,1]$ and $l=0,\ldots,k-1$, we have

$\|D^{l}(f-Q^{k}_{N}(f))\|_{\infty}\leq\frac{C_{7}}{N^{k-l-1}}\omega\bigg{(}D^{k-1}f,\frac{1}{N}\bigg{)}.$

Here, $\omega(D^{k-1}f,1/N)$ is the modulus of continuity of the $(k-1)$-th
derivative of $f$:

$\omega\bigg{(}D^{k-1}f,\frac{1}{N}\bigg{)}=\sup_{\begin{subarray}{c}t,t^{\prime}\in[-1,1],\\\
|t-t^{\prime}|\leq 1/N\end{subarray}}|D^{k-1}f(t)-D^{k-1}f(t^{\prime})|.$

Indeed, for a particular $l_{n_{0}}\in C^{\infty}[-1,1]$, given that
$\|C^{\lambda}_{k}\|_{\infty}=C^{\lambda}_{k}(1)=\binom{k+d-3}{k}\leq(k+1)^{d-3}$,
we can establish the following inequality:

$\|l_{n_{0}}\|_{\infty}\leq 5\cdot 3^{d-2}n_{0}^{d-1}.$

It is crucial to note that $l_{n_{0}}$ represents a polynomial of degree at
most $2n_{0}$. Applying the Markov inequality for polynomials, one can derive

$\omega\bigg{(}D^{k-1}l_{n_{0}},\frac{1}{N}\bigg{)}\leq\frac{1}{N}\|D^{k}l_{n_{0}}\|_{\infty}\leq\frac{1}{N}(2n_{0})^{2k}\|l_{n_{0}}\|_{\infty}\leq\frac{5\cdot
2^{2k}\cdot 3^{d-2}n_{0}^{d+2k-1}}{N}.$

As a consequence, we obtain

$\|D^{l}(l_{n_{0}}-Q^{k}_{N}(l_{n_{0}}))\|_{\infty}\leq\frac{C_{8}n_{0}^{d+2k-1}}{N^{k-l}}$
(3.10)

where the constant $C_{8}$ only depends on $k$ and $d$.

To demonstrate that ReLUk-1 is capable of expressing the interpolation spline
function, we rewrite $N^{k}_{i}(x)$ for all $x\in[-1,1]$ explicitly, as stated
by Theorem 4.14 and (4.49) in [46]:

$N^{k}_{i}(x)=\left\\{\begin{aligned}
&\sum_{j=1}^{k-i+1}\alpha_{ij}(x+1)^{k-j}_{+}+\sum_{j=1}^{i}\beta_{ij}(x+t_{k+j})^{k-1}_{+},\quad
1\leq i\leq k-1,\\\
&\frac{N^{k-1}}{(k-1)!}\sum_{j=0}^{k}(-1)^{j}\binom{k}{j}\bigg{(}(x-t_{i})-\frac{j}{N}\bigg{)}^{k-1}_{+},\quad
k\leq i\leq 2N,\\\
&\sum_{j=1}^{2N+k-i}\gamma_{ij}(x-t_{i+j-1})^{k-1}_{+},\quad 2N+1\leq i\leq
2N+k-1,\end{aligned}\right.$ (3.11)

wherein $\alpha_{ij},\beta_{ij},\gamma_{ij}$ are constants depending on $N$
with $\max(|\alpha_{ij}|,|\beta_{ij}|,|\gamma_{ij}|)\leq N^{k-1}$. To prove
$(x+1)^{k-j}_{+}$ appearing in $N^{k}_{i}(x)$ for $i=1,\ldots,k-1$ and $j\geq
2$ can be represented using ReLUk-1, we propose the following lemma.

######  Lemma 9.

Given integers $l,k\in\mathbb{N}\cup\\{0\\}$ with $l<k$, there exist constants
$\zeta_{li}$ and $\xi_{li},i=0,\ldots,k$, such that

$x^{l}=\sum_{i=0}^{k}\xi_{li}(x+\zeta_{li})^{k},\quad x\geq 0.$

###### Proof.

Compare the coefficients on both sides results in a linear system of equations

$\begin{bmatrix}\zeta^{k}_{l,0}&\zeta^{k}_{l,1}&\cdots&\zeta^{k}_{l,k}\\\
\vdots&\vdots&\vdots&\vdots\\\
\binom{k}{k-l}\zeta^{k-l}_{l,0}&\binom{k}{k-l}\zeta^{k-l}_{l,1}&\cdots&\binom{k}{k-l}\zeta^{k-l}_{l,k}\\\
\vdots&\vdots&\vdots&\vdots\\\
1&1&\ldots&1\end{bmatrix}\begin{bmatrix}\xi_{l,0}\\\ \xi_{l,1}\\\ \vdots\\\
\vdots\\\ \xi_{l,k}\end{bmatrix}=\begin{bmatrix}0\\\ \vdots\\\ 1\\\ \vdots\\\
0\end{bmatrix}.$

The solution requires finding $\zeta_{li}$, $i=0,\ldots,k$, such that the
matrix is invertible. It is noteworthy that the determinant of the matrix is

$\prod_{i=1}^{k}\binom{k}{i}\cdot\prod_{0\leq i<j\leq
k}(\zeta_{li}-\zeta_{lj}).$

Thus, we only need to ensure that $\zeta_{li}\neq\zeta_{lj}$ for $i\neq j$.
The proof is then finished. ∎

The function $(x+1)_{+}=x+1$ for $x\in[-1,1]$ allows us to express
$N^{k}_{i}(x)$ as a one hidden layer FCNN using ReLUk-1. Now given the output
of the convolution layer $\mathcal{D}(F^{(L)}(x))$ as presented in Lemma 7, we
build the fully connected layer as follows.

We first consider a simpler FCNN that only accepts one input, denoted as
$\langle x,y_{1}\rangle+B^{(L)}$, which is also an output of the convolution
layer $\mathcal{D}(F^{(L)})(x)$. For $j=1,\ldots,k$, Lemma 9 allows us to
write

$\displaystyle\alpha_{ij}(\langle x,y_{1}\rangle+1)^{k-j}_{+}$
$\displaystyle={}$
$\displaystyle\alpha_{ij}\sum_{s=0}^{k-1}\xi_{k-j,s}(\langle
x,y_{1}\rangle+\zeta_{k-j,s}+1)^{k-1}_{+}$ $\displaystyle={}$
$\displaystyle\sum_{s=0}^{k}\text{sgn}(\alpha_{ij}\xi_{k-j,s})\bigg{(}\sqrt[k-1]{|\alpha_{ij}\xi_{k-j,s}|}\bigg{(}\langle
x,y_{1}\rangle+B^{(L)}\bigg{)}$
$\displaystyle\qquad\qquad-\sqrt[k-1]{|\alpha_{ij}\xi_{k-j,s}|}\bigg{(}B^{(L)}-\zeta_{k-j,s}-1\bigg{)}\bigg{)}^{k-1}_{+}.$

Let $w=\sqrt[k-1]{|\alpha_{ij}\xi_{k-j,s}|}$ and
$b=\sqrt[k-1]{|\alpha_{ij}\xi_{k-j,s}|}(B^{(L)}-\zeta_{k-j,s}-1)$, we see that
$|w|\lesssim N$, $|b|\lesssim NB^{(L)}\leq NC_{6}^{L}$ where $C_{6}$ is the
constant from Lemma 7, and

$\alpha_{ij}(\langle
x,y_{1}\rangle+1)^{k-j}_{+}=\sum_{s=0}^{k}\text{sgn}(\alpha_{ij}\xi_{k-j,s})(w(\mathcal{D}(F^{(L)}(x)))_{1}-b)^{k-1}_{+}.$

Here, $(w(\mathcal{D}(F^{(L)}(x)))_{1}-b)^{k-1}_{+}$ can be represented by the
output of one single hidden node using ReLUk-1. Hence, $\alpha_{ij}(\langle
x,y_{1}\rangle+1)^{k-j}_{+}$ is a linear combination of outputs from $k+1$
hidden nodes. A similar argument also applies to

$\begin{split}&\beta_{ij}(\langle x,y_{1}\rangle+t_{k+j})^{k-1}_{+},\\\
&\frac{N^{k-1}}{(k-1)!}(-1)^{j}\binom{k}{j}\bigg{(}(\langle x,y_{1}\rangle-
t_{i})-\frac{j}{N}\bigg{)}^{k-1}_{+}\mbox{ and }\\\ &\gamma_{ij}(\langle
x,y_{1}\rangle-t_{i+j-1})^{k-1}_{+}.\end{split}$

Each of these terms can be represented as the multiplication of a sign and the
output of a single hidden node. After calculating these terms, note by (3.11)
that $N^{k}_{i}(\langle x,y_{1}\rangle)$ is also a linear combination of them.
The overall number of nodes needed to represent $N^{k}_{i}(\langle
x,y_{1}\rangle)$ is

$\left\\{\begin{array}[]{lcl}(k+1)(k-i+1)+i,&&\mbox{ for }1\leq i\leq k-1,\\\
k+1,&&\mbox{ for }k\leq i\leq 2N,\\\ 2N+k-i,&&\mbox{ for }2N+1\leq i\leq
2N+k-1,\end{array}\right.$

where the maximum number of nodes, $k^{2}+k+1$, is achieved when $i=1$. By
(3.8), $Q_{N}^{k}(l_{n_{0}})(\langle x,y_{1}\rangle)$ can be expressed as a
linear combination of $N^{k}_{i}(\langle x,y_{1}\rangle)$. The total number of
nodes needed to build $Q_{N}^{k}(l_{n_{0}})(\langle x,y_{1}\rangle)$ is

$\sum_{i=1}^{k-1}\bigg{(}(k+1)(k-i+1)+i\bigg{)}+(2N-k+1)(k+1)+\sum_{i=2N+1}^{2N+k-1}\bigg{(}2N+k-i\bigg{)}=\frac{1}{2}(4N+k^{2}-k)(k+1).$

We regard $Q_{N}^{k}(l_{n_{0}})(\langle x,y_{1}\rangle)$ as the output of this
simpler FCNN which only accepts the input $\langle x,y_{1}\rangle+B^{(L)}$ and
uses $(4N+k^{2}-k)(k+1)/2$ hidden nodes.

Employing the same argument, we can construct $m$ simpler FCNNs, each
accepting only a single input, $\langle x,y_{i}\rangle+B^{(L)}$, and utilizing
$(4N+k^{2}-k)(k+1)/2$ hidden nodes. These $m(4N+k^{2}-k)(k+1)/2$ hidden nodes
are concatenated to form the first hidden layer $F^{(L+1)}$ of the FCNN, with
the $m$ output nodes forming the second hidden layer $F^{(L+2)}$. The first
layer employs $\mathrm{ReLU}^{k-1}$, while the second utilizes
$\mathrm{ReLU}$, i.e.,

$\displaystyle F^{(L+1)}(x)$
$\displaystyle=\mathrm{ReLU}^{k-1}\left(W^{(L+1)}\mathcal{D}\big{(}F^{(L)}(x)\big{)}-b^{(L+1)}\right)\in\mathbb{R}^{m(4N+k^{2}-k)(k+1)/2},$
$\displaystyle F^{(L+2)}(x)$
$\displaystyle=\mathrm{ReLU}\left(W^{(L+2)}F^{(L+1)}(x)-b^{(L+2)}\right)\in\mathbb{R}^{m}.$

These FCNNs also employ identical parameters, thus resulting in shared weights
and biases. The parameters can be represented as follows:

$\displaystyle W^{(L+1)}$
$\displaystyle=\begin{bmatrix}\overrightarrow{w_{1}}&\overrightarrow{0}&\cdots&\overrightarrow{0}&\overrightarrow{0}&\cdots&\overrightarrow{0}\\\
\overrightarrow{0}&\overrightarrow{w_{1}}&\cdots&\overrightarrow{0}&\overrightarrow{0}&\cdots&\overrightarrow{0}\\\
\vdots&\vdots&\cdots&\vdots&\vdots&\vdots&\vdots\\\
\overrightarrow{0}&\overrightarrow{0}&\cdots&\overrightarrow{w_{1}}&\overrightarrow{0}&\cdots&\overrightarrow{0}\\\
\overrightarrow{0}&\overrightarrow{0}&\cdots&\overrightarrow{0}&\overrightarrow{w_{1}}&\cdots&\overrightarrow{0}\\\
\end{bmatrix}\in\mathbb{R}^{m(4N+k^{2}-k)(k+1)/2\times\left\lfloor
d_{L}/d\right\rfloor},$ (3.12) $\displaystyle b^{(L+1)}$
$\displaystyle=\begin{bmatrix}\overrightarrow{b_{1}}\\\ \vdots\\\
\overrightarrow{b_{1}}\\\ \end{bmatrix}\in\mathbb{R}^{m(4N+k^{2}-k)(k+1)/2},$
$\displaystyle W^{(L+2)}$
$\displaystyle=\begin{bmatrix}\overrightarrow{w_{2}}^{T}&\overrightarrow{0}&\cdots&\overrightarrow{0}&\overrightarrow{0}\\\
\overrightarrow{0}&\overrightarrow{w_{2}}^{T}&\cdots&\overrightarrow{0}&\overrightarrow{0}\\\
\vdots&\vdots&\cdots&\vdots&\vdots\\\
\overrightarrow{0}&\overrightarrow{0}&\cdots&\overrightarrow{w_{2}}^{T}&\overrightarrow{0}\\\
\overrightarrow{0}&\overrightarrow{0}&\cdots&\overrightarrow{0}&\overrightarrow{w_{2}}^{T}\\\
\end{bmatrix}\in\mathbb{R}^{m\times m(4N+k^{2}-k)(k+1)/2},$ $\displaystyle
b^{(L+2)}$ $\displaystyle=b_{2}\mathbf{1}_{m}\in\mathbb{R}^{m},$

where
$\overrightarrow{w_{1}},\overrightarrow{b_{1}},\overrightarrow{w_{2}}\in\mathbb{R}^{(4N+k^{2}-k)(k+1)/2}$
and $b_{2}=-\|Q_{N}^{k}(l_{n_{0}})\|_{\infty}$. From this construction, we see
that for $1\leq i\leq m$,

$\big{(}F^{(L+2)}(x)\big{)}_{i}=\mathrm{ReLU}\bigg{(}\big{(}W^{(L+2)}F^{(L+1)}(x)-b^{(L+2)}\big{)}_{i}\bigg{)}=Q_{N}^{k}(l_{n_{0}})(\langle
x,y_{i}\rangle)+\|Q_{N}^{k}(l_{n_{0}})\|_{\infty}.$

Then we apply an additional affine transformation to yield the output for our
entire network, e.g.,

$\displaystyle F^{(L+3)}(x)$ $\displaystyle=W^{(L+3)}\cdot
F^{(L+2)}(x)-b^{(L+3)}$
$\displaystyle=\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})\big{(}F^{(L+2)}(x)\big{)}_{i}-\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})\|Q_{N}^{k}(l_{n_{0}})\|_{\infty}$
$\displaystyle=\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})Q_{N}^{k}(l_{n_{0}})(\langle
x,y_{i}\rangle).$

In conclusion, we have constructed an FCNN with two hidden layers of width
$d_{L+1}=m(4N+k^{2}-k)(k+1)/2$ and $d_{L+2}=m$ satisfying the following
boundedness constraints:

$\displaystyle\|W^{(L+1)}\|_{\max}\lesssim
N,\quad\|b^{(L+1)}\|_{\infty}\lesssim NC_{6}^{L},$
$\displaystyle\|W^{(L+2)}\|_{\max}=\max\\{|\lambda_{j}(l_{n_{0}})|:j=1,\ldots,2N+k-1\\}\leq
5\cdot 3^{d-2}\cdot(2k)^{k}n_{0}^{d-1},$
$\displaystyle\|b^{(L+2)}\|_{\infty}=\|Q_{N}^{k}(l_{n_{0}})\|_{\infty}\leq
5\cdot 3^{d-2}\cdot(2k)^{k}n_{0}^{d-1},$
$\displaystyle\|W^{(L+3)}\|_{\max}=\max\\{|\mu_{i}L_{n_{0}}(u)(y_{i})|:i=1,\ldots,m\\}\leq
C_{1}\|u\|_{\infty},$
$\displaystyle|b^{(L+3)}|=\bigg{|}\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})\|Q_{N}^{k}(l_{n_{0}})\|_{\infty}\bigg{|}\leq
5\cdot 3^{d-2}\cdot(2k)^{k}n_{0}^{d-1}C_{1}\|u\|_{\infty},$

where $C_{1}$ is from Lemma 4 and $C_{6}$ is from Lemma 7. The total number of
free parameters contributed by the FCNN is

$\|\overrightarrow{w_{1}}\|_{0}+\|\overrightarrow{b_{1}}\|_{0}+\|\overrightarrow{w_{2}}\|_{0}+1+m+1=\frac{3}{2}\cdot(4N+k^{2}-k)(k+1)+m+2.$

We now present Theorem 2, which asserts the approximation capability of our
CNN architecture with respect to the Sobolev norm. The following theorem
invokes a definition analogous to Definition (2.19) to describe the CNN
function space equipped with hybrid activation functions. This is denoted by
$\mathcal{F}(L,L_{0}=2,S,d_{L+1},d_{L+2},\mathrm{ReLU},\mathrm{ReLU}^{k-1},M=\infty,\mathcal{S})$.
Here, the network in this function space employs activation function
$\sigma^{(l)}=\mathrm{ReLU}$ for $l=1,\ldots,L,L+2$ and
$\sigma^{(L+1)}=\mathrm{ReLU}^{k-1}$. The condition $M=\infty$ signifies there
are no constraints on the supremum norm of the output function or its
derivatives.

######  Theorem 2.

Let $d\geq 3$, $3\leq S\leq d$ and $1\leq p\leq\infty$. Given non-negative
integers $s,k,r,n_{0}$ satisfying $0\leq s<\min\\{s,k\\}$ and $n_{0}\geq 1$,
then for all $u\in W^{r}_{p}(\mathbb{S}^{d-1})$, there exists a network

$F^{(L+3)}\in\mathcal{F}(L,L_{0}=2,S,d_{L+1},d_{L+2},\mathrm{ReLU},\mathrm{ReLU}^{k-1},M=\infty,\mathcal{S})$

with

$\displaystyle L=\left\lceil\frac{C_{2}n_{0}^{d-1}d-1}{S-2}\right\rceil,\quad
d_{L+1}\asymp n_{0}^{\frac{d+r+s-1}{k-s}+d+1},\quad d_{L+2}\asymp
n_{0}^{d-1},$
$\displaystyle\mathcal{S}\asymp\max\big{\\{}n_{0}^{2+\frac{d+r+s-1}{k-s}},n_{0}^{d-1}\big{\\}},$

and a constant $C_{9}$ only depending on $d,p,k$ and $s$, such that

$\|u-F^{(L+3)}\|_{W^{s}_{p}(\mathbb{S}^{d-1})}\leq
C_{9}n_{0}^{s-r}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}.$ (3.13)

Here, $C_{2}$ is the constant from Lemma 5. Furthermore, the network
parameters satisfy the sup-norm constraints (2.17) with

$\displaystyle B_{1}=C_{5},\quad B_{2}=C^{L}_{6},\quad
B_{3}\lesssim\max\big{\\{}n_{0}^{2+\frac{d+r+s-1}{k-s}},3^{d}k^{k}n_{0}^{d-1}\big{\\}},$
$\displaystyle B_{4}\lesssim\max\big{\\{}C_{6}^{L}\cdot
n_{0}^{2+\frac{d+r+s-1}{k-s}},3^{d}k^{k}n_{0}^{d-1}\big{\\}},$

where the constants $C_{5}$ and $C_{6}$ are from Lemma 7.

###### Proof.

We first prove (3.13). We leverage Lemma 7 which provides us with the equation

$\mathcal{D}(F^{(L)}(x))=\begin{bmatrix}\langle x,y_{1}\rangle\\\ \vdots\\\
\langle x,y_{m}\rangle\\\ 0\\\ \vdots\\\ 0\end{bmatrix}+B^{(L)}{\bf
1}_{\left\lfloor\frac{d+L(S-1)}{d}\right\rfloor},$

where $m=\left\lceil C_{2}n_{0}^{d-1}\right\rceil$ and the cubature rule
$\\{(\mu_{i},y_{i})\\}_{i=1}^{m}$ is guaranteed by Lemma 5 for degree
$4n_{0}$. As discussed above, we can construct a neural network with two
hidden layers following the convolution layers. The activation functions of
these hidden layers are $\sigma^{(L+1)}=\mathrm{ReLU}^{k-1}$ and
$\sigma^{(L+2)}=\mathrm{ReLU}$, respectively. Applying the affine
transformation, the network outputs

$F^{(L+3)}(x)=\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})Q_{N}^{k}(l_{n_{0}})\big{(}\langle
x,y_{i}\rangle\big{)}.$

Define $B^{k}_{N}=l_{n_{0}}-Q_{N}^{k}(l_{n_{0}})$. For $1\leq i<j\leq d,1\leq
p<\infty$ and $q=p/(p-1)$, by Hölder’s inequality, we have

$\displaystyle\bigg{\|}D^{s}_{i,j}\bigg{(}F^{(L+3)}-\sum_{t=1}^{m}\mu_{t}L_{n_{0}}(u)(y_{t})l_{n_{0}}(\langle\cdot,y_{t}\rangle)\bigg{)}\bigg{\|}_{p}^{p}$
(3.14) $\displaystyle={}$
$\displaystyle\frac{1}{\omega_{d}}\int_{\mathbb{S}^{d-1}}\bigg{|}\sum_{t=1}^{m}\mu_{t}L_{n_{0}}(u)(y_{t})D^{s}_{i,j}B^{k}_{N}(\langle
x,y_{t}\rangle)\bigg{|}^{p}d\sigma(x)$ $\displaystyle\leq{}$
$\displaystyle\frac{1}{\omega_{d}}\sum_{t=1}^{m}\mu_{t}|L_{n_{0}}(u)(y_{t})|^{p}\cdot\int_{\mathbb{S}^{d-1}}\bigg{(}\sum_{t=1}^{m}\mu_{t}\bigg{|}D^{s}_{i,j}B^{k}_{N}(\langle
x,y_{t}\rangle)\bigg{|}^{q}\bigg{)}^{\frac{p}{q}}d\sigma(x)$
$\displaystyle={}$
$\displaystyle\frac{1}{\omega_{d}}\sum_{t=1}^{m}\mu_{t}|L_{n_{0}}(u)(y_{t})|^{p}$
$\displaystyle\cdot\int_{\mathbb{S}^{d-1}}\bigg{(}\sum_{t=1}^{m}\mu_{t}\bigg{|}\bigg{(}x_{i}\frac{\partial}{\partial
x_{j}}-x_{j}\frac{\partial}{\partial x_{i}}\bigg{)}^{s}B^{k}_{N}(\langle
x,y_{t}\rangle)\bigg{|}^{q}\bigg{)}^{\frac{p}{q}}d\sigma(x).$

It is shown by Theorem 1 from [15] that, there exists a constant
$C^{\prime}_{1}$ that only depends on $d,p$ and $\eta$, such that

$\sum_{t=1}^{m}\mu_{t}|L_{n_{0}}(u)(y_{t})|^{p}\leq C^{\prime
p}_{1}\|u\|^{p}_{W^{r}_{p}(\mathbb{S}^{d-1})}.$

Further, using the chain rule, we can validate the existence of a constant
$C^{\prime}_{2}$ depending only on $s$ which ensures that

$\bigg{|}\bigg{(}x_{i}\frac{\partial}{\partial
x_{j}}-x_{j}\frac{\partial}{\partial x_{i}}\bigg{)}^{s}B^{k}_{N}(\langle
x,y_{t}\rangle)\bigg{|}\leq
C^{\prime}_{2}\sum_{i=1}^{s}\|D^{i}B^{k}_{N}\|_{\infty}.$

By (3.10), there exists a constant $C^{\prime}_{3}$ dependent only on $k,s$
and $d$ such that

$\sum_{i=1}^{s}\|D^{i}B^{k}_{N}\|_{\infty}\leq\frac{C^{\prime}_{3}n_{0}^{d+2k-1}}{N^{k-s}}.$

Therefore, we have

$\displaystyle\frac{1}{\omega_{d}}\int_{\mathbb{S}^{d-1}}\bigg{(}\sum_{t=1}^{m}\mu_{t}\bigg{|}\bigg{(}x_{i}\frac{\partial}{\partial
x_{j}}-x_{j}\frac{\partial}{\partial x_{i}}\bigg{)}^{s}B^{k}_{N}(\langle
x,y_{t}\rangle)\bigg{|}^{q}\bigg{)}^{\frac{p}{q}}d\sigma(x)$
$\displaystyle\leq{}$
$\displaystyle\bigg{(}\frac{C^{\prime}_{2}C^{\prime}_{3}n_{0}^{d+2k-1}}{N^{k-s}}\bigg{)}^{p}\bigg{(}\sum_{t=1}^{m}\mu_{t}\bigg{)}^{\frac{p}{q}}=\bigg{(}\frac{C^{\prime}_{2}C^{\prime}_{3}n_{0}^{d+2k-1}}{N^{k-s}}\bigg{)}^{p}.$

Combining the aforementioned inequalities, we obtain

$\bigg{\|}D^{s}_{i,j}\bigg{(}F^{(L+3)}-\sum_{t=1}^{m}\mu_{t}L_{n_{0}}(u)(y_{t})l_{n_{0}}(\langle\cdot,y_{t}\rangle)\bigg{)}\bigg{\|}_{p}\leq\frac{C^{\prime}_{1}C^{\prime}_{2}C^{\prime}_{3}n_{0}^{d+2k-1}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}}{N^{k-s}}.$

An analogous process substantiates that (with a reselection of the constants
$C^{\prime}_{1},C^{\prime}_{2},$ and $C^{\prime}_{3}$)

$\bigg{\|}F^{(L+3)}-\sum_{t=1}^{m}\mu_{t}L_{n_{0}}(u)(y_{t})l_{n_{0}}(\langle\cdot,y_{t}\rangle)\bigg{\|}_{p}\leq\frac{C^{\prime}_{1}C^{\prime}_{2}C^{\prime}_{3}n_{0}^{d+2k-1}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}}{N^{k}},$

and the above analysis pertains equally when $p=\infty$. Therefore, there
exists a constant $C^{\prime}_{4}$ only depends on $d,p,k,s$ and $\eta$ such
that

$\bigg{\|}F^{(L+3)}-\sum_{t=1}^{m}\mu_{t}L_{n_{0}}(u)(y_{t})l_{n_{0}}(\langle\cdot,y_{t}\rangle)\bigg{\|}_{W^{s}_{p}(\mathbb{S}^{d-1})}\leq\frac{C^{\prime}_{4}n_{0}^{d+2k-1}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}}{N^{k-s}}.$
(3.15)

Combining the above estimates with Lemma 6 leads to

$\displaystyle\|u-F^{(L+3)}\|_{W^{s}_{p}(\mathbb{S}^{d-1})}$
$\displaystyle\leq\bigg{\|}u-\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})l_{n_{0}}(\langle\cdot,y_{i}\rangle)\bigg{\|}_{W^{s}_{p}(\mathbb{S}^{d-1})}+$
$\displaystyle\quad\bigg{\|}F^{(L+3)}-\sum_{i=1}^{m}\mu_{i}L_{n_{0}}(u)(y_{i})l_{n_{0}}(\langle\cdot,y_{i}\rangle)\bigg{\|}_{W^{s}_{p}(\mathbb{S}^{d-1})}$
$\displaystyle\leq
C_{4}n_{0}^{s-r}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}+\frac{C^{\prime}_{4}n_{0}^{d+2k-1}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}}{N^{k-s}}.$

Next, we select $N=\left\lceil
n_{0}^{\frac{d+r+2k-s-1}{k-s}}\right\rceil=\left\lceil
n_{0}^{2+\frac{d+r+s-1}{k-s}}\right\rceil$, and noting that $\eta$ is fixed
beforehand, which yields

$\|u-F^{(L+3)}\|_{W^{s}_{p}(\mathbb{S}^{d-1})}\leq
C_{9}n_{0}^{s-r}\|u\|_{W^{r}_{p}(\mathbb{S}^{d-1})}$

with a constant $C_{9}$ only depends on $d,p,k$ and $s$. By applying the
preceding analysis with $m=\left\lceil C_{2}n_{0}^{d-1}\right\rceil$ and
$N=\left\lceil n_{0}^{2+\frac{d+r+s-1}{k-s}}\right\rceil$, we derive the
parameters of $\mathcal{F}$ and their corresponding sup-norm constraints. Thus
we complete the proof. ∎

######  Remark 4.

As previously emphasized in Remark 3, the bound above is also applicable for
$r\in\mathbb{R}$ in the context of fractional Sobolev spaces.

At the end of this section, we can establish an upper bound for the
approximation error $\mathcal{R}(u_{\mathcal{F}})-\mathcal{R}(u^{*})$
utilizing (2.5). It should be pointed out that we will again employ
$\sigma^{(L+1)}=\mathrm{ReLU}^{k}$ in the rest of this paper, deviating from
the usage of $\sigma^{(L+1)}=\mathrm{ReLU}^{k-1}$ as indicated in Theorem 2.
The application of the $\mathrm{ReLU}^{k-1}$ arises from an analytical
convention typically utilized in discussions related to B-spline functions.
Recall that $u^{*}$ denotes the solution of the equation (2.9).

######  Theorem 3.

Assume that Assumption 1 is satisfied with some $d\geq 3,s\geq 1$ and $r\geq
s+1$. For any
$0<\varepsilon<\|u^{*}\|^{2}_{W^{r}_{\infty}(\mathbb{S}^{d-1})}$, let
$\delta=\varepsilon/\|u^{*}\|^{2}_{W^{r}_{\infty}(\mathbb{S}^{d-1})}$ denote
the relative error. If we take the function space of CNN

$\mathcal{F}=\mathcal{F}(L,L_{0}=2,S,d_{L+1},d_{L+2},\mathrm{ReLU},\mathrm{ReLU}^{k},M,\mathcal{S})$

with

$\displaystyle 3\leq S\leq d,\quad k\geq 2,\quad
L\asymp\delta^{-\frac{d-1}{2(r-s)}},$ $\displaystyle
d_{L+1}\asymp\delta^{-\frac{d+r+s-1}{2(r-s)(k-s+1)}-\frac{d+1}{2(r-s)}},\quad
d_{L+2}\asymp\delta^{-\frac{d-1}{2(r-s)}},$ $\displaystyle M\geq
3C_{9}\|u^{*}\|_{W^{r}_{\infty}(\mathbb{S}^{d-1})},$
$\displaystyle\mathcal{S}\asymp\max\left\\{\delta^{-\frac{d+r+s-1}{2(r-s)(k-s+1)}-\frac{1}{r-s}},\delta^{-\frac{d-1}{2(r-s)}}\right\\},$

where constant $C_{9}$ is from Theorem 2 and only depends on $d,k$ and $s$.
Then

$\mathcal{R}(u_{\mathcal{F}})-\mathcal{R}(u^{*})\lesssim\varepsilon,$ (3.16)

where $u_{\mathcal{F}}=\arg\min_{u\in\mathcal{F}}\mathcal{R}(u)$.

###### Proof.

Let

$n_{0}=\left\lceil\delta^{-\frac{1}{2(r-s)}}\right\rceil.$

Then by Theorem 2, we can take a CNN hypothesis space

$\mathcal{F}=\mathcal{F}(L,L_{0}=2,S,d_{L+1},d_{L+2},\mathrm{ReLU},\mathrm{ReLU}^{k},M=\infty,\mathcal{S})$

where $L,S,k,d_{L+1},d_{L+2},\mathcal{S}$ are specified in Theorem 2.
Consequently, there exists a $u\in\mathcal{F}$ satisfying

$\|u-u^{*}\|^{2}_{W^{s}_{\infty}(\mathbb{S}^{d-1})}\leq
C^{2}_{9}n_{0}^{2(s-r)}\|u^{*}\|^{2}_{W^{r}_{\infty}(\mathbb{S}^{d-1})}\leq
2C^{2}_{9}\varepsilon,$ (3.17)

where $C_{9}$ is from Theorem 2. By the triangle inequality, we can deduce
that

$\|u\|_{W^{s}_{\infty}(\mathbb{S}^{d-1})}\leq
C_{9}\sqrt{2\varepsilon}+\|u^{*}\|_{W^{r}_{\infty}(\mathbb{S}^{d-1})}\leq
3C_{9}\|u^{*}\|_{W^{r}_{\infty}(\mathbb{S}^{d-1})}.$

We then further constrain the function space with the extra sup-norm
constraint (2.18) stipulating that

$M\geq 3C_{9}\|u^{*}\|_{W^{r}_{\infty}(\mathbb{S}^{d-1})}.$

Then we ensure that $u\in\mathcal{F}$ remains valid and that $u$ meets the
approximation bound defined in (3.17). Combining (3.17) with (2.5), we
conclude that

$\mathcal{R}(u_{\mathcal{F}})-\mathcal{R}(u^{*})\leq\mathcal{R}(u)-\mathcal{R}(u^{*})\lesssim\|u-u^{*}\|^{2}_{W^{s}_{\infty}(\mathbb{S}^{d-1})}\leq
2C_{9}^{2}\varepsilon.$

The proof is then finished. ∎

##  4 Statistical Error Analysis

This section is dedicated to the estimation of statistical error, where a
novel localization analysis is developed for the stochastic part of error
analysis. Herein, we always use Assumption 1 with some $s\geq 1$ and $r\geq
s+1$ and Assumption 2 with some $M>0$. As required by Theorem 3, we fix a
sufficiently large $M$ such that $M\geq
3C_{9}\|u^{*}\|_{W^{r}_{\infty}(\mathbb{S}^{d-1})}$, where constant $C_{9}$ is
from Theorem 2. Recall that we have defined
$L:\mathbb{S}^{d-1}\times\mathcal{F}\to\mathbb{R}$ by

$L(x,u)=|(\mathcal{L}u)(x)-f(x)|^{2},$ (4.1)

which measures the residual of equation (2.9). Then the PINN risk is the
expectation of $L$ over a uniform distribution $P$ on $\mathbb{S}^{d-1}$,
given by

$\mathcal{R}(u)=\mathbb{E}_{P}[L(X,u)].$

Whereas the empirical PINN risk

$\mathcal{R}_{n}(u)=\frac{1}{n}\sum_{i=1}^{n}L(X_{i},u)$

is the empirical mean of i.i.d. sample $\\{X_{i}\\}_{i=1}^{n}$ from $P$.

Define the composition of $L$ and $\mathcal{F}$ as

$L\circ\mathcal{F}=\\{g:\mathbb{S}^{d-1}\to\mathbb{R}|g(x)=L(x,u),u\in\mathcal{F}\\}.$

The Rademacher complexity of $L\circ\mathcal{F}$ plays a pivotal role in
bounding the statistical error. For an i.i.d. sequence
$\\{\varepsilon_{i}\\}_{i=1}^{n}$ with $\varepsilon_{i}=\pm 1$ having equal
probability independent of $X$ and sample data
$\mathbf{x}=\\{X_{i}\\}_{i=1}^{n}$, define the empirical Rademacher complexity
of $\mathcal{F}$ as

$\mathrm{Rad}(\mathcal{F},\mathbf{x})=\mathbb{E}_{\varepsilon}\sup_{u\in\mathcal{F}}\bigg{|}\frac{1}{n}\sum_{i=1}^{n}\varepsilon_{i}u(X_{i})\bigg{|}.$

The Rademacher complexity is calculated by taking a secondary expectation over
the sample data:

$\mathrm{Rad}(\mathcal{F}):=\mathbb{E}_{P}\mathbb{E}_{\varepsilon}\sup_{u\in\mathcal{F}}\bigg{|}\frac{1}{n}\sum_{i=1}^{n}\varepsilon_{i}u(X_{i})\bigg{|}.$

Accordingly, the empirical Rademacher complexity and Rademacher complexity of
$L\circ\mathcal{F}$ are defined as

$\mathrm{Rad}(L\circ\mathcal{F},\mathbf{x}):=\mathbb{E}_{\varepsilon}\sup_{u\in\mathcal{F}}\bigg{|}\frac{1}{n}\sum_{i=1}^{n}\varepsilon_{i}L(X_{i},u)\bigg{|}\mbox{
and
}\mathrm{Rad}(L\circ\mathcal{F}):=\mathbb{E}_{P}\mathbb{E}_{\varepsilon}\sup_{u\in\mathcal{F}}\bigg{|}\frac{1}{n}\sum_{i=1}^{n}\varepsilon_{i}L(X_{i},u)\bigg{|}.$

Additionally, we will employ the following notations:

$\displaystyle\widetilde{\mathcal{F}}$
$\displaystyle:=\mathcal{F}-u^{*}=\\{u-u^{*}:u\in\mathcal{F}\\},$
$\displaystyle D^{\alpha}\widetilde{\mathcal{F}}$
$\displaystyle:=\\{D^{\alpha}(u-u^{*}):u\in\mathcal{F}\\},\quad|\alpha|\leq
s.$

as well as their respective Rademacher complexities in our subsequent
analysis.

A frequently applied constraint on $L$ requires it to be Lipschitz with
respect to $u$. This constraint allows us to bound
$\mathrm{Rad}(L\circ\mathcal{F},\mathbf{x})$ by
$\mathrm{Rad}(\mathcal{F},\mathbf{x})$ employing a contraction lemma. Within
our PINN framework, note that we impose sup-norm constraints as defined in
Assumption 1 and Assumption 2. Consequently, for all
$u_{1},u_{2}\in\mathcal{F}\cup\\{u^{*}\\}$, the following is true:

$\displaystyle|L(x,u_{1})-L(x,u_{2})|$ (4.2) $\displaystyle={}$
$\displaystyle\big{|}(\mathcal{L}u_{1})(x)+(\mathcal{L}u_{2})(x)-2f(x)\big{|}\cdot\big{|}(\mathcal{L}u_{1})(x)-(\mathcal{L}u_{2})(x)\big{|}$
$\displaystyle\leq{}$ $\displaystyle 2\bigg{(}M\cdot\sum_{|\alpha|\leq
s}\|a_{\alpha}\|_{\infty}+\|f\|_{\infty}\bigg{)}\cdot\bigg{(}\sum_{|\alpha|\leq
s}\|a_{\alpha}\|_{\infty}\cdot|D^{\alpha}u_{1}(x)-D^{\alpha}u_{2}(x)|\bigg{)}.$

Hence, by a contraction technique, we derive the subsequent lemma.

######  Lemma 10.

Assume that $L(x,u)$ defined by (4.1) is Lipschitz with respect to $u$. Then

$\mathrm{Rad}(L\circ\mathcal{F},\mathbf{x})\leq C_{10}\sum_{|\alpha|\leq
s}\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}},\mathbf{x}),$

where $C_{10}:=4\big{(}M\cdot\sum_{|\alpha|\leq
s}\|a_{\alpha}\|_{\infty}+\|f\|_{\infty}\big{)}\cdot\max_{|\alpha|\leq
s}\|a_{\alpha}\|_{\infty}.$

###### Proof.

By (4.2), for all $x\in\mathbb{S}^{d-1}$ and
$u_{1},u_{2}\in\mathcal{F}\cup\\{u^{*}\\}$ we have

$\displaystyle|L(x,u_{1})-L(x,u_{2})|$ $\displaystyle\leq{}$
$\displaystyle\frac{C_{10}}{2}\cdot\sum_{|\alpha|\leq
s}|D^{\alpha}u_{1}(x)-D^{\alpha}u_{2}(x)|$ $\displaystyle={}$
$\displaystyle\frac{C_{10}}{2}\cdot\sum_{|\alpha|\leq
s}|D^{\alpha}(u_{1}-u^{*})(x)-D^{\alpha}(u_{2}-u^{*})(x)|.$

Using the contraction lemma (Theorem 4.12 in [31]), the Lipschitz property of
$L$ yields a bound of the empirical Rademacher complexity:

$\mathrm{Rad}(L\circ\mathcal{F},\mathbf{x})\leq
C_{10}\mathrm{Rad}\bigg{(}\sum_{|\alpha|\leq
s}D^{\alpha}\widetilde{\mathcal{F}},\mathbf{x}\bigg{)}\leq
C_{10}\sum_{|\alpha|\leq
s}\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}},\mathbf{x}).$

The proof is then finished. ∎

Next, we aim to establish a novel localized complexity analysis to bound the
statistical error by the Rademacher complexity.

###  4.1 Localized Complexity Analysis

To estimate the term
$\mathcal{R}(u_{n})-\mathcal{R}(u^{*})-\mathcal{R}_{n}(u_{n})+\mathcal{R}_{n}(u^{*})$,
we propose a novel localization approach that makes use of standard tools from
empirical process analysis, including peeling, symmetrization, and Dudley’s
chaining. Firstly, we bound the estimated error by considering the supremum
norm of an empirical process:

$\mathcal{R}(u_{n})-\mathcal{R}(u^{*})-\mathcal{R}_{n}(u_{n})+\mathcal{R}_{n}(u^{*})\leq\sup_{u\in\mathcal{F}}\mathcal{R}(u)-\mathcal{R}(u^{*})-\mathcal{R}_{n}(u)+\mathcal{R}_{n}(u^{*}).$
(4.3)

For all $u\in\mathcal{F}$ and $\gamma>0$, we define an auxiliary function
$g_{u}:\mathbb{S}^{d-1}\to\mathbb{R}$ as

$g_{u}(x):=L(x,u)-L(x,u^{*}).$

Furthermore, for all $\gamma>\inf_{u\in\mathcal{F}}\mathbb{E}_{P}(g_{u}(X))$,
we introduce the following localized function classes:

$\displaystyle\mathcal{F}_{\gamma}$
$\displaystyle:=\\{u:\mathbb{S}^{d-1}\to\mathbb{R}\ |\
u\in\mathcal{F},\mathbb{E}_{P}(g_{u}(X))=\mathcal{R}(u)-\mathcal{R}(u^{*})\leq\gamma\\},$
$\displaystyle\widetilde{\mathcal{F}_{\gamma}}$
$\displaystyle:=\\{u-u^{*}:\mathbb{S}^{d-1}\to\mathbb{R}\ |\
u\in\mathcal{F}_{\gamma}\\},$ $\displaystyle\mathcal{G}_{\gamma}$
$\displaystyle:=\\{g_{u}:\mathbb{S}^{d-1}\to\mathbb{R}\ |\
u\in\mathcal{F}_{\gamma}\\},$ $\displaystyle D^{\alpha}\mathcal{F}_{\gamma}$
$\displaystyle:=\\{D^{\alpha}u:\mathbb{S}^{d-1}\to\mathbb{R}\ |\
u\in\mathcal{F}_{\gamma}\\},$ $\displaystyle
D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}$
$\displaystyle:=\\{D^{\alpha}u-D^{\alpha}u^{*}:\mathbb{S}^{d-1}\to\mathbb{R}\
|\ u\in\mathcal{F}_{\gamma}\\}.$

Given a sequence of i.i.d. samples $\mathbf{x}=\\{X_{i}\\}_{i=1}^{n}$ and some
function class $\mathcal{G}$ of measurable functions on $\mathbb{S}^{d-1}$, we
can define the empirical $L^{2}$-norm $\|\cdot\|_{L^{2}(\mathbf{x})}$ of
$g\in\mathcal{G}$ as

$\|g\|^{2}_{L^{2}(\mathbf{x})}:=\frac{1}{n}\sum_{i=1}^{n}g(X_{i})^{2},$

and the sample expectation as

$\mathbb{E}_{\mathbf{x}}g:=\frac{1}{n}\sum_{i=1}^{n}g(X_{i}).$

Next, we introduce the symmetrization and Dudley’s chaining lemmas, which can
be found in [4] and [41].

######  Lemma 11.

Consider an i.i.d. sample sequence $\mathbf{x}=\\{X_{i}\\}_{i=1}^{n}$. For any
function $g\in\mathcal{G}$, if $\|g\|_{\infty}\leq G$ and $\mathrm{Var}(g)\leq
V$, then the following inequalities hold: for all $t>0$, with probability at
least $1-\exp(-t)$,

$\sup_{g\in\mathcal{G}}\mathbb{E}_{\mathbf{x}}g-\mathbb{E}_{P}g\leq
3\mathrm{Rad}(\mathcal{G})+\sqrt{\frac{2Vt}{n}}+\frac{4Gt}{3n};$

and with probability at least $1-2\exp(-t)$,

$\sup_{g\in\mathcal{G}}\mathbb{E}_{\mathbf{x}}g-\mathbb{E}_{P}g\leq
6\mathrm{Rad}(\mathcal{G},\mathbf{x})+\sqrt{\frac{2Vt}{n}}+\frac{23Gt}{3n}.$

The above inequalities are also valid for
$\sup_{g\in\mathcal{G}}\mathbb{E}_{P}g-\mathbb{E}_{\mathbf{x}}g$.

######  Lemma 12.

Let $\mathcal{N}(\delta,\mathcal{G},\|\cdot\|_{L^{2}(\mathbf{x})})$ denote the
covering number of $\mathcal{G}$, taking radius $\delta$ and metric
$\|\cdot\|_{L^{2}(\mathbf{x})}$. There holds

$\mathrm{Rad}(\\{g:g\in\mathcal{G},\|g\|_{L^{2}(\mathbf{x})}\leq\gamma\\},\mathbf{x})\leq\inf_{0<\beta<\gamma}\bigg{\\{}4\beta+\frac{12}{\sqrt{n}}\int_{\beta}^{\gamma}\sqrt{\log\mathcal{N}(\delta,\mathcal{G},\|\cdot\|_{L^{2}(\mathbf{x})})}d\delta\bigg{\\}}.$

The quantity
$\log\mathcal{N}(\delta,\mathcal{G},\|\cdot\|_{L^{2}(\mathbf{x})})$ is
recognized as the metric entropy of $\mathcal{G}$. Its upper bound can be
defined by the pseudo-dimension, hence by the VC-dimension of $\mathcal{G}$,
as outlined in Theorem 12.2 and Theorem 14.1 by [3].

######  Definition 1.

Given a function class $\mathcal{G}:\mathcal{X}\to\mathbb{R}$ and a set
$X=\\{x_{i}\\}_{i=1}^{m}$ of $m$ points in the input space $\mathcal{X}$. Let

$\mathrm{sgn}(\mathcal{G})=\\{\mathrm{sgn}(g):g\in\mathcal{G}\\}$

be the set of binary functions $\mathcal{X}\to\\{0,1\\}$ induced by
$\mathcal{G}$. If $\mathrm{sgn}(\mathcal{G})$ can compute all dichotomies of
$X$, that is,

$\\#\\{\mathrm{sgn}(g)|_{X}\in\\{0,1\\}^{m}:g\in\mathcal{G}\\}=2^{m},$

we say that $\mathcal{G}$ or $\mathrm{sgn}(\mathcal{G})$ shatters $X$. The
Vapnik-Chervonenkis dimension (or VC-dimension) of $\mathcal{G}$ or
$\mathrm{sgn}(\mathcal{G})$ is the size of the largest shattered subset of
$\mathcal{X}$, denoted by $\mathrm{VCDim}(\mathrm{sgn}(\mathcal{G}))$.

Moreover, we say that $X$ is pseudo-shattered by $\mathcal{G}$, if there are
real numbers $r_{1},r_{2},\ldots,r_{m}$, such that for each
$v\in\\{0,1\\}^{m}$, there exists a function $g_{v}\in\mathcal{G}$ with
$\mathrm{sgn}(g_{v}(x_{i})-r_{i})=v_{i}$ for $1\leq i\leq m$. The pseudo-
dimension of $\mathcal{G}$, denoted by $\mathrm{PDim}(\mathcal{G})$, is the
maximum cardinality of a subset $X$ of $\mathcal{X}$ that is pseudo-shattered
by $\mathcal{G}$.

######  Lemma 13.

Suppose that for every function $g\in\mathcal{G}$, $\|g\|_{\infty}\leq M$. For
a set of points $\\{x_{i}\\}_{i=1}^{n}$, define
$\mathcal{G}|_{x_{1},\ldots,x_{n}}:=\\{g|_{x_{1},\ldots,x_{n}}:\\{x_{i}\\}_{i=1}^{n}\to\mathbb{R}|g\in\mathcal{G}\\}$.Then
for all $n\geq\mathrm{PDim}(\mathcal{F})$ and $\delta>0$,

$\mathcal{N}(\delta,\mathcal{G}|_{x_{1},\ldots,x_{n}},\|\cdot\|_{\infty})\leq\left(\frac{2enM}{\delta\mathrm{PDim}(\mathcal{G})}\right)^{\mathrm{PDim}(\mathcal{G})}.$

######  Lemma 14.

For a class of neural networks $\mathcal{G}$ with a fixed architecture and
fixed activation functions. Then,

$\mathrm{PDim}(\mathcal{G})\leq\mathrm{VCDim}(\mathrm{sgn}(\mathcal{G}_{0})),$

where $\mathcal{G}_{0}$ is a set extended from $\mathcal{G}$ by adding one
extra input neuron and one extra computational neuron. The additional
computational neuron is a linear threshold neuron that takes inputs from the
output unit of $\mathcal{G}$ and the new input neuron.

When Lemma 12 is later employed, it is essential to control the empirical norm
$\|\cdot\|_{L^{2}(\mathbf{x})}$ by limiting the excess risk
$\mathbb{E}_{P}(g_{u}(X))$ to no more than $\gamma$. We formalize this idea in
the following lemma. We define $\kappa=\\#\\{\alpha:|\alpha|\leq
s,a_{\alpha}(\cdot)\neq 0\\}$ to represent the number of non-zero terms in
$\mathcal{L}$ satisfying Assumption 1.

######  Lemma 15.

For all $t>0$, if

$\gamma\geq\max_{|\alpha|\leq
s}\bigg{\\{}12M\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}),\frac{8C_{11}M^{2}t}{n},\frac{16M^{2}t}{3n}\bigg{\\}},$

then with probability at least $1-\kappa\exp(-t)$, there holds

$\max_{|\alpha|\leq
s}\sup_{u\in\mathcal{F}_{\gamma}}\|D^{\alpha}u-D^{\alpha}u^{*}\|_{L^{2}(\mathbf{x})}\leq\sqrt{(C_{11}+3)\gamma},$

where $C_{11}>0$ is the constant satisfying the estimate (2.12):

$\|u-u^{*}\|^{2}_{H^{s}(\mathbb{S}^{d-1})}\leq
C_{11}\|\mathcal{L}(u-u^{*})\|^{2}_{L^{2}(\mathbb{S}^{d-1})}.$ (4.4)

###### Proof.

Consider the class
$D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}^{2}:=\\{g^{2}:g\in
D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}\\}=\\{(D^{\alpha}u-D^{\alpha}u^{*})^{2}:u\in\mathcal{F}_{\gamma}\\}$.
Note that for all $u\in\mathcal{F}_{\gamma}$ and $|\alpha|\leq s$, we have
$\|(D^{\alpha}u-D^{\alpha}u^{*})^{2}\|_{\infty}\leq 4M^{2}$ and by (4.4),

$\mathrm{Var}((D^{\alpha}u-D^{\alpha}u^{*})^{2})\leq\mathbb{E}_{P}[(D^{\alpha}u-D^{\alpha}u^{*})^{4}]\leq
4M^{2}C_{11}(\mathcal{R}(u)-\mathcal{R}(u^{*}))\leq 4C_{11}M^{2}\gamma.$

Apply Lemma 11 to $D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}^{2}$. With
probability at least $1-\exp(-t)$, we have

$\sup_{u\in\mathcal{F}_{\gamma}}\|D^{\alpha}u-D^{\alpha}u^{*}\|^{2}_{L^{2}(\mathbf{x})}-\mathbb{E}_{P}[(D^{\alpha}u-D^{\alpha}u^{*})^{2}]\leq
3\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}^{2})+\sqrt{\frac{8C_{11}M^{2}\gamma
t}{n}}+\frac{16M^{2}t}{3n}.$

Note that $\|D^{\alpha}u-D^{\alpha}u^{*}\|_{\infty}\leq 2M$ and the square
function is locally $4M$ Lipschitz on $[-2M,2M]$. By the contraction lemma
(Theorem 4.12 in [31]), we have

$\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}^{2})\leq
4M\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}).$

Hence, if we choose

$\gamma\geq\max_{|\alpha|\leq
s}\bigg{\\{}12M\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}),\frac{8C_{11}M^{2}t}{n},\frac{16M^{2}t}{3n}\bigg{\\}},$

then with probability at least $1-\exp(-t)$,

$\sup_{u\in\mathcal{F}_{\gamma}}\|D^{\alpha}u-D^{\alpha}u^{*}\|^{2}_{L^{2}(\mathbf{x})}\leq\sup_{u\in\mathcal{F}_{\gamma}}\mathbb{E}_{P}[(D^{\alpha}u-D^{\alpha}u^{*})^{2}]+\gamma+\gamma+\gamma\leq(C_{11}+3)\gamma,$

This probability inequality holds for all $|\alpha|\leq s$. By the union
bound, we have finished the proof. ∎

Before continuing our analysis, we perform an estimation of the VC-dimension
bounds of our CNN space and its high-order derivative spaces, which will turn
out to be an upper bound control on the Rademacher complexity of the network.

###  4.2 VC-Dimension Bounds of CNN Space Involving Derivatives

The study by [17] substantiates near-precise VC-dimension bounds for piecewise
linear neural networks, laying emphasis on feedforward neural networks with
ReLU activation functions. Herein, this subsection offers a novel proof aimed
at evaluating the VC-dimension bound for our specific CNN space, which
incorporates derivatives and generalizes a bound for CNNs in [59]. For such an
endeavor, the ensuing lemma is essential.

######  Lemma 16.

(Theorem 8.3 in [3]) Suppose $p_{1},\ldots,p_{m}$ represent polynomials of
degree not exceeding $q$ in $t\leq m$ variables. Let the sign function be
defined as $\mathrm{sgn}(x):=\mathbf{1}_{x>0}$ and consider

$K:=|\\{(\mathrm{sgn}(p_{1}(x)),\ldots,\mathrm{sgn}(p_{m}(x))):x\in\mathbb{R}^{t}\\}|,$

where $K$ denotes the total quantity of all sign vectors furnished by
$p_{1},\ldots,p_{m}$. Then, we have $K\leq 2(2emq/t)^{t}$.

Now we are in the position to state our estimates for VC-dimension bounds. The
subsequent theorem, forming a cardinal contribution of this paper, warrants
individual consideration owing to its central role in the analysis of
stochastic error. For ease of notation, we define the set
$\\{1,2,\ldots,m\\}$, for $m\in\mathbb{N}$, as $[m]$.

######  Theorem 4.

Consider the CNN function space $\mathcal{F}$ defined by (2.19) with hybrid
activation functions, i.e., $\sigma^{(l)}=\mathrm{ReLU}$ for $l\neq
L+1,\sigma^{(L+1)}=\mathrm{ReLU}^{k}$ with $k\geq s$, which is parameterized
by $L,L_{0},S^{(l)}\equiv S$ and the total number of free parameters
$\mathcal{S}$. Then for all $|\alpha|\leq s$, it follows that

$\mathrm{VCDim}(\mathrm{sgn}(D^{\alpha}\mathcal{F}))\lesssim\mathcal{S}(L+L_{0})\log\bigg{(}k(L+L_{0})\cdot\max\\{d_{l}:l\in[L+L_{0}]\\}\bigg{)}.$

######  Remark 5.

Given the recurring usage of this VC-dimension bound in the forthcoming proof
of the oracle inequality (refer to Subsection 4.3), it is convenient to
introduce the notation

$\mathrm{VC}_{\mathcal{F}}:=\mathcal{S}(L+L_{0})\log\bigg{(}k(L+L_{0})\cdot\max\\{d_{l}:l\in[L+L_{0}]\\}\bigg{)}.$
(4.5)

###### Proof.

For $l\in[L]$, the convolution kernel $w^{(l)}$ and bias $b^{(l)}$ are the
parameters of convolutional layer $l$. For $l\geq L+1$, the weight matrices
$W^{(l)}$ and bias $b^{(l)}$ are the parameters of fully connected layer $l$.
Let $\mathcal{S}_{l}$ denote the total number of free parameters up to layer
$l$ and $\theta^{(l)}\in\mathbb{R}^{\mathcal{S}_{l}}$ denote the concatenated
free parameters vector up to layer $l$. We see that
$\mathcal{S}_{1}\leq\mathcal{S}_{2}\leq\cdots\leq\mathcal{S}_{L+L_{0}+1}=\mathcal{S}$.

We first consider the function space $\mathcal{F}$ itself. For simplicity, let
$f(x;\theta)$ denote the output of network with input $x\in\mathbb{R}^{d}$ and
parameters vector $\theta\in\mathbb{R}^{\mathcal{S}}$. Thus, every
$\theta\in\mathbb{R}^{\mathcal{S}}$ corresponds to a unique
$f(\cdot;\theta)\in\mathcal{F}$. Assume that the VC-dimension of the network
is $m$ so we can let $\\{x^{1},\ldots,x^{m}\\}\subset\mathbb{R}^{d}$ be a
shattered set, that is,

$K:=|\\{(\mathrm{sgn}(f(x^{1};\theta)),\ldots,\mathrm{sgn}(f(x^{m};\theta))):\theta\in\mathbb{R}^{\mathcal{S}}\\}|=2^{m}.$

If $m\leq\mathcal{S}$, the conclusion is immediate so we assume that
$m>\mathcal{S}$. We will give a bound for $K$, which will imply a bound for
$m$.

To this end, we find a partition $\mathcal{P}$ of the parameter space
$\mathbb{R}^{\mathcal{S}}$ such that for all $P\in\mathcal{P}$, the functions
$f(x^{1};\cdot),\ldots,f(x^{m};\cdot)$ are polynomials on $P$. Then,

$K\leq\sum_{P\in\mathcal{P}}|\\{(\mathrm{sgn}(f(x^{1};\theta)),\ldots,\mathrm{sgn}(f(x^{m};\theta))):\theta\in
P\\}|,$ (4.6)

and we can bound the summand by Lemma 16. To construct such $\mathcal{P}$, we
follow an inductive step to construct a sequence of partitions
$\mathcal{P}_{1},\mathcal{P}_{2},\ldots,\mathcal{P}_{L+L_{0}}=\mathcal{P},$
where $\mathcal{P}_{i}$ is a refinement of $\mathcal{P}_{i-1}$. For
$l\in[L],i\in[d_{l}]$ and $j\in[m]$, let

$h_{l,i}(x^{j};\theta):=\sum_{s=1}^{d_{l-1}}w^{(l)}_{i-s}\big{(}F^{(l-1)}(x^{j};\theta)\big{)}_{s}-b^{(l)}_{i},$
(4.7)

i.e., $h_{l,i}(x^{j};\theta)$ is the input into the $i$-th neural in the
$l$-th layer which is inactive. Hence

$(F^{(l)}(x^{j};\theta))_{i}=\sigma^{(l)}(h_{l,i}(x^{j};\theta))=\mathrm{ReLU}(h_{l,i}(x^{j};\theta)),$
(4.8)

where we note that we abbreviate the dependency of $\theta$ when defining
$F^{(l)}(x^{j};\theta)$ in Subsection 2.3 but we emphasize it in this proof.

We start with $l=1$. Consider the concatenated vector
$(\mathrm{sgn}(h_{1,i}(x^{j};\cdot)))_{i\in[d_{1}],j\in[m]}$. Since each
$h_{1,i}(x^{j};\cdot)$ is a polynomial of degree 1 in
$\mathcal{S}_{1}\leq\mathcal{S}<md_{1}$ variables, by Lemma 16, we can choose
a partition $\mathcal{P}_{1}$ of $\mathbb{R}^{\mathcal{S}}$ such that
$|\mathcal{P}_{1}|\leq 2(2emd_{1}/\mathcal{S}_{1})^{\mathcal{S}_{1}}$ and the
concatenated vector
$(\mathrm{sgn}(h_{1,i}(x^{j};\cdot)))_{i\in[d_{1}],j\in[m]}$ is constant in
each $P\in\mathcal{P}_{1}$. By (4.8), clearly, in each $P\in\mathcal{P}_{1}$,
each $(F^{(1)}(x^{j};\cdot))_{i}$ is either a polynomial of degree 1 or a zero
function in $\mathcal{S}_{1}$ variables.

In the induction step, suppose that we have constructed
$\mathcal{P}_{1},\ldots,\mathcal{P}_{l-1}$ for an $l\in[L]$, and for
$i\in[d_{l-1}],j\in[m]$, $\big{(}F^{(l-1)}(x^{j};\cdot)\big{)}_{i}$ is a
polynomial of degree at most $l-1$ in $\mathcal{S}_{l-1}$ variables in any
$P\in\mathcal{P}_{l-1}$. By (4.7) we see that for $i\in[d_{l}],j\in[m]$,
$h_{l,i}(x^{j};\cdot)$ is a polynomial of degree at most $l$ in
$\mathcal{S}_{l}<md_{l}$ variables in any $P\in\mathcal{P}_{l-1}$. By Lemma
16, we can take a partition $\mathcal{P}_{P,l}$ of $P$ such that
$|\mathcal{P}_{P,l}|\leq 2(2elmd_{l}/\mathcal{S}_{l})^{\mathcal{S}_{l}}$ and
in each $P^{\prime}\in\mathcal{P}_{P,l}$, the concatenated vector
$(\mathrm{sgn}(h_{l,i}(x^{j};\cdot)))_{i\in[d_{l}],j\in[m]}$ is a fixed value
vector. Define

$\mathcal{P}_{l}:=\bigcup_{P\in\mathcal{P}_{l-1}}\mathcal{P}_{P,l}.$

Then by (4.8), in each $P\in\mathcal{P}_{l}$, each
$(F^{(l)}(x^{j};\cdot))_{i}$ is either a polynomial of degree at most $l$ or a
zero function. Moreover, we have

$|\mathcal{P}_{l}|\leq
2(2elmd_{l}/\mathcal{S}_{l})^{\mathcal{S}_{l}}|\mathcal{P}_{l-1}|.$

The case for $l=L+1$ needs an additional discussion. For $i\in[d_{L+1}]$ and
$j\in[m]$, let

$h_{L+1,i}(x^{j};\theta):=\sum_{s=1}^{\lfloor
d_{L}/d\rfloor}(W^{(L+1)})_{i,s}\big{(}\mathcal{D}\big{(}F^{(L)}(x^{j};\theta)\big{)}\big{)}_{s}-b^{(L+1)}_{i}.$

Clearly, by induction, $h_{L+1,i}(x^{j};\cdot)$ is a polynomial of degree at
most $L+1$ in $\mathcal{S}_{L+1}<md_{L+1}$ variables in each
$P\in\mathcal{P}_{L}$. Similarly, we can take a $\mathcal{P}_{L+1}$ such that

$|\mathcal{P}_{L+1}|\leq
2(2e(L+1)md_{L+1}/\mathcal{S}_{L+1})^{\mathcal{S}_{L+1}}|\mathcal{P}_{L}|.$

In each $P\in\mathcal{P}_{L+1}$, each
$(F^{(L+1)}(x^{j};\cdot))_{i}=\mathrm{ReLU}^{k}(h_{L+1,i}(x^{j};\cdot))$ is
either a polynomial of degree at most $k(L+1)$ or a zero function. Hence we
can carry on our induction to $l\geq L+1$. We conclude that, a partition
$\mathcal{P}=\mathcal{P}_{L+L_{0}}$ of $\mathbb{R}^{\mathcal{S}}$ can be
constructed. In each $P\in\mathcal{P}$,
$\big{(}F^{(L+L_{0})}(x^{j};\cdot)\big{)}_{i}$ is a polynomial of degree at
most $k(L+1)+L_{0}-1$ or zero function in $\mathcal{S}_{L+L_{0}}$ variables.
Now, the $(L+L_{0}+1)$-th layer has a single output neural and

$f(x^{j};\cdot)=\sum_{s=1}^{d_{L+L_{0}}}(W^{(L+L_{0}+1)})_{s}(F^{(L+L_{0})}(x^{j};\cdot))_{s}-b^{(L+L_{0}+1)}$

is a polynomial of degree at most $k(L+1)+L_{0}$ in
$\mathcal{S}_{L+L_{0}+1}=\mathcal{S}<m$ variables in any $P\in\mathcal{P}$. By
Lemma 16 and (4.6), we conclude that

$\displaystyle 2^{m}$ $\displaystyle=K\leq
2\bigg{(}\frac{2e(k(L+1)+L_{0})m}{\mathcal{S}}\bigg{)}^{\mathcal{S}}|\mathcal{P}|$
$\displaystyle\leq
2^{L+L_{0}+1}\left(\frac{2e(k(L+1)+L_{0})m}{\mathcal{S}}\right)^{\mathcal{S}}$
$\displaystyle\quad\cdot\prod_{l=1}^{L+1}\bigg{(}\frac{2elmd_{l}}{\mathcal{S}_{l}}\bigg{)}^{\mathcal{S}_{l}}\prod_{l=L+2}^{L+L_{0}}\bigg{(}\frac{2e(k(L+1)+l-L-1)md_{l}}{\mathcal{S}_{l}}\bigg{)}^{\mathcal{S}_{l}}$
$\displaystyle\leq
2^{2(L+L_{0})}\bigg{(}4emk(L+L_{0})\cdot\max\\{d_{l}:l\in[L+L_{0}]\\}\bigg{)}^{2\mathcal{S}(L+L_{0})}$

where we use crude bounds $L+L_{0}+1\leq
2(L+L_{0}),\mathcal{S}_{l}\leq\mathcal{S},k(L+1)+L_{0}\leq 2k(L+L_{0})$ and
$d_{l}\leq\max\\{d_{l}:l\in[L+L_{0}]\\}$. Taking logarithms we yield that

$\displaystyle m$ $\displaystyle\leq
2(L+L_{0})+2\mathcal{S}(L+L_{0})\log_{2}\bigg{(}4emk(L+L_{0})\cdot\max\\{d_{l}:l\in[L+L_{0}]\\}\bigg{)}$
$\displaystyle\leq
4\mathcal{S}(L+L_{0})\log_{2}\bigg{(}4emk(L+L_{0})\cdot\max\\{d_{l}:l\in[L+L_{0}]\\}\bigg{)},$

which implies

$m\lesssim\mathcal{S}(L+L_{0})\log\bigg{(}k(L+L_{0})\cdot\max\\{d_{l}:l\in[L+L_{0}]\\}\bigg{)}.$

Now we turn to bound the VC-dimension of the derivatives of functions in our
CNN function space. To this end, we first bound the VC-dimension of the
derivative $\mathrm{sgn}(\partial\mathcal{F}/\partial x_{1})$ following a
similar step. For simplicity, let $f^{\prime}(x;\theta):=\partial
f(x;\theta)/\partial x_{1}$. In a slight abuse of notation, assume that the
VC-dimension of $\mathrm{sgn}(\mathcal{F}^{\prime})$ is $m>\mathcal{S}$ and
let $\\{x^{1},\ldots,x^{m}\\}\subset\mathbb{R}^{d}$ be a shattered set. Define

$K:=|\\{(\mathrm{sgn}(f^{\prime}(x^{1};\theta)),\ldots,\mathrm{sgn}(f^{\prime}(x^{m};\theta))):\theta\in\mathbb{R}^{\mathcal{S}}\\}|=2^{m}.$

For $l=1$ and $i\in[d_{1}]$, we have

$h^{\prime}_{1,i}(x^{j};\theta)=w^{(1)}_{i-1}.$ (4.9)

For $l=2,\ldots,L$ and $i\in[d_{l}]$, we have

$h^{\prime}_{l,i}(x;\theta)=\sum_{s=1}^{d_{l-1}}w^{(l)}_{i-s}\big{(}F^{(l-1)^{\prime}}(x;\theta)\big{)}_{s}=\sum_{s=1}^{d_{l-1}}w^{(l)}_{i-s}h^{\prime}_{l-1,s}(x;\theta)\mathrm{sgn}(h_{l-1,s}(x;\theta)).$
(4.10)

For $l=L+1$ and $i\in[d_{L+1}]$, we have

$\displaystyle h^{\prime}_{L+1,i}(x;\theta)$
$\displaystyle=\sum_{s=1}^{\lfloor
d_{L}/d\rfloor}(W^{(L+1)})_{i,s}\big{(}\mathcal{D}\big{(}F^{(L)^{\prime}}(x^{j};\theta)\big{)}\big{)}_{s}$
(4.11) $\displaystyle=\sum_{s=1}^{\lfloor
d_{L}/d\rfloor}(W^{(L+1)})_{i,s}h^{\prime}_{L,sd}(x;\theta)\mathrm{sgn}(h_{L,sd}(x;\theta)).$

For $l=L+2$ and $i\in[d_{L+2}]$, we have

$\displaystyle h^{\prime}_{L+2,i}(x;\theta)$
$\displaystyle=\sum_{s=1}^{d_{L+1}}(W^{(L+2)})_{i,s}\big{(}F^{(L+1)^{\prime}}(x^{j};\theta)\big{)}_{s}$
$\displaystyle=k\sum_{s=1}^{d_{L+1}}(W^{(L+1)})_{i,s}h^{\prime}_{L+1,s}(x;\theta)h^{k-1}_{L+1,s}(x;\theta)\mathrm{sgn}(h_{L+1,s}(x;\theta)).$

For $l\geq L+3$ and $i\in[d_{l}]$, we have

$h^{\prime}_{l,i}(x;\theta)=\sum_{s=1}^{d_{l-1}}(W^{(l)})_{i,s}h^{\prime}_{l-1,s}(x;\theta)\mathrm{sgn}(h_{l-1,s}(x;\theta)).$

Define $[\lfloor d_{L}/d\rfloor]d:=\\{d,2d,\ldots,\lfloor d_{L}/d\rfloor
d\\}\subset\mathbb{Z}$. By induction, for the derivative of the network
output, we can write

$f^{\prime}(x;\theta)=k\sum_{s}\left[\left(\prod_{l=1}^{L+L_{0}+1}\theta_{l,s}\right)\cdot\left(\prod_{l=1}^{L+L_{0}}\mathrm{sgn}(h_{l,s_{l}}(x;\theta))\right)\cdot
h^{k-1}_{L+1,s_{L+1}}(x;\theta)\right],$ (4.12)

where $s$ running all over $\prod_{l=1}^{L-1}[d_{l}]\times[\lfloor
d_{L}/d\rfloor]d\times\prod_{l=L+1}^{L+L_{0}}[d_{l}]\subset\mathbb{Z}^{L+L_{0}}$
and $\theta_{l,s}$ is a specific parameter appears in the $l$-th layer.

Similarly, we can construct an identical partition $\mathcal{P}$ of the
parameter space $\mathbb{R}^{\mathcal{S}}$ such that, in each
$P\in\mathcal{P}$, $\mathrm{sgn}(h_{l,s_{l}}(x^{j};\cdot))$ are constants for
all $s\in\prod_{l=1}^{L-1}[d_{l}]\times[\lfloor
d_{L}/d\rfloor]d\times\prod_{l=L+1}^{L+L_{0}}[d_{l}],l\in[L]$ and $j\in[m]$.
Moreover, $h_{L+1,s_{L+1}}(x^{j};\cdot)$ is a polynomial of degree at most
$L+1$. Hence, in each $P\in\mathcal{P}$, $f^{\prime}(x^{j};\cdot)$ is a
polynomial of degree at most $(L+L_{0}+1)+(k-1)(L+1)=k(L+1)+L_{0}$ in
$\mathcal{S}<m$ variables. By Lemma 16 and (4.6), we can derive

$m\lesssim\mathcal{S}(L+L_{0})\log\bigg{(}k(L+L_{0})\cdot\max\\{d_{l}:l\in[L+L_{0}]\\}\bigg{)}$

once more.

Notice that this bound also holds for other derivatives $\partial
f(x;\theta)/\partial x_{i}$ of order $1$, and they share a similar expression
(4.12) with different $\theta_{l,s}$. With this observation, we can now
consider higher order derivatives $D^{\alpha},|\alpha|\geq 2$. We claim that
for all $|\alpha|\geq 1$, we have

$\displaystyle D^{\alpha}f(x;\theta)$ (4.13) $\displaystyle={}$
$\displaystyle\bigg{(}\prod_{l=0}^{|\alpha|-1}(k-l)\bigg{)}\sum_{s^{1}}\sum_{s^{2}}\cdots\sum_{s^{|\alpha|}}\left[\left(\prod_{l=1}^{L+L_{0}+1}\theta_{l,s^{1}}\right)\left(\prod_{l=1}^{L+1}\theta_{l,s^{2},s^{1}_{L+1}}\right)\right.$
$\displaystyle\left.\cdots\left(\prod_{l=1}^{L+1}\theta_{l,s^{|\alpha|},s^{1}_{L+1}}\right)\left(\prod_{l=1}^{L+L_{0}}\mathrm{sgn}(h_{l,s^{1}_{l}}(x;\theta))\right)\left(\prod_{l=1}^{L}\mathrm{sgn}(h_{l,s^{2}_{l}}(x;\theta))\right)\right.$
$\displaystyle\left.\cdots\left(\prod_{l=1}^{L}\mathrm{sgn}(h_{l,s^{|\alpha|}_{l}}(x;\theta))\right)h^{k-|\alpha|}_{L+1,s^{1}_{L+1}}(x;\theta)\right],$

where

$\displaystyle s^{1}\text{ is ranging all over
}\prod_{l=1}^{L-1}[d_{l}]\times[\lfloor
d_{L}/d\rfloor]d\times\prod_{l=L+1}^{L+L_{0}}[d_{l}]\subset\mathbb{Z}^{L+L_{0}},$
$\displaystyle s^{2},\ldots,s^{|\alpha|}\text{ is ranging all over
}\prod_{l=1}^{L-1}[d_{l}]\times[\lfloor
d_{L}/d\rfloor]d\subset\mathbb{Z}^{L},$

and
$\theta_{l,s^{1}},\theta_{l,s^{2},s^{1}_{L+1}},\ldots\theta_{l,s^{|\alpha|},s^{1}_{L+1}}$
are some parameters appearing in the $l$-th layer and the specific locations
are determined by their subscripts.

We prove this claim by induction on $|\alpha|$. For $|\alpha|=1$, (4.13) is
exactly (4.12). Now assume that the claim holds for all $|\beta|\leq|\alpha|$.
Without loss of generality, assume that $\alpha_{1}\geq 1$ and
$\alpha=\beta+(1,0,0,\cdots,0)$. By induction, we calculate that

$\displaystyle
D^{\alpha}f(x;\theta)=\left(D^{\beta}f(x;\theta)\right)^{\prime}$
$\displaystyle={}$
$\displaystyle\bigg{(}\prod_{l=0}^{|\beta|-1}(k-l)\bigg{)}\sum_{s^{1}}\sum_{s^{2}}\cdots\sum_{s^{|\beta|}}\left[\left(\prod_{l=1}^{L+L_{0}+1}\theta_{l,s^{1}}\right)\left(\prod_{l=1}^{L+1}\theta_{l,s^{2},s^{1}_{L+1}}\right)\right.$
$\displaystyle\left.\cdots\left(\prod_{l=1}^{L+1}\theta_{l,s^{|\beta|},s^{1}_{L+1}}\right)\left(\prod_{l=1}^{L+L_{0}}\mathrm{sgn}(h_{l,s^{1}_{l}}(x;\theta))\right)\left(\prod_{l=1}^{L}\mathrm{sgn}(h_{l,s^{2}_{l}}(x;\theta))\right)\right.$
$\displaystyle\left.\cdots\left(\prod_{l=1}^{L}\mathrm{sgn}(h_{l,s^{|\beta|}_{l}}(x;\theta))\right)\left(h^{k-|\beta|}_{L+1,s^{1}_{L+1}}(x;\theta)\right)^{\prime}\right],$

where

$\left(h^{k-|\beta|}_{L+1,s^{1}_{L+1}}(x;\theta)\right)^{\prime}=(k-|\beta|)h^{k-|\beta|-1}_{L+1,s^{1}_{L+1}}(x;\theta)h^{\prime}_{L+1,s^{1}_{L+1}}(x;\theta).$

Plugging in (4.11)(4.10)(4.9) yield the conclusion immediately.

Now with the expression (4.13), we are ready to bound the VC-dimension of
$\mathrm{sgn}(D^{\alpha}\mathcal{F})$. Again, in a slight abuse of notation,
assume that the VC-dimension of $\mathrm{sgn}(D^{\alpha}\mathcal{F})$ is
$m>\mathcal{S}$ and let $\\{x^{1},\ldots,x^{m}\\}\subset\mathbb{R}^{d}$ be a
shattered set. Define

$K:=|\\{(\mathrm{sgn}(D^{\alpha}f(x^{1};\theta)),\ldots,\mathrm{sgn}(D^{\alpha}f(x^{m};\theta))):\theta\in\mathbb{R}^{\mathcal{S}}\\}|=2^{m}.$

By (4.13), clearly, in each $P\in\mathcal{P}$, for all $j\in[m]$,
$D^{\alpha}f(x^{j};\cdot)$ is a polynomial of degree at most

$(L+L_{0}+1)+(|\alpha|-1)(L+1)+(k-|\alpha|)(L+1)=L(k+1)+L_{0}$

in $\mathcal{S}<m$ variables. Hence, we yield

$m\lesssim\mathcal{S}(L+L_{0})\log\bigg{(}k(L+L_{0})\cdot\max\\{d_{l}:l\in[L+L_{0}]\\}\bigg{)}$

again. The proof is then finished. ∎

###  4.3 An Important Oracle Inequality

In this subsection, we will give an estimate of the statistical error by
establishing an important oracle inequality. To this end, we first prove a
series of lemmas. Recall that $\mathrm{VC}_{\mathcal{F}}$ is given by (4.5),
the function classes $\mathcal{F}_{\gamma}$ and
$\widetilde{\mathcal{F}_{\gamma}}$ are defined at the beginning of Section 4.

######  Lemma 17.

Given a dataset $\mathbf{x}=\\{X_{i}\\}_{i=1}^{n}$. If there exists $C_{12}>0$
and $\gamma>0$ satisfying

$\displaystyle\max_{|\alpha|\leq
s}\sup_{u\in\mathcal{F}_{\gamma}}\|D^{\alpha}u-D^{\alpha}u^{*}\|_{L^{2}(\mathbf{x})}$
$\displaystyle\leq C_{12}\sqrt{\gamma},$ (4.14) $\displaystyle(1/n)^{2}$
$\displaystyle\leq\gamma,$ $\displaystyle\max_{|\alpha|\leq
s}\mathrm{PDim}(D^{\alpha}\mathcal{F})\vee(4eM/C_{12})^{2}$ $\displaystyle\leq
n,$

then we have

$\max_{|\alpha|\leq
s}\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma}},\mathbf{x})\lesssim\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}\gamma\log
n},$

Moreover, for any $t>0$, with a probability exceeding $1-2\exp(-t)$, there
holds

$\sup_{u\in\mathcal{F}_{\gamma}}\mathbb{E}_{P}g_{u}-\mathbb{E}_{\mathbf{x}}g_{u}\lesssim
6C_{10}\kappa\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}\gamma\log
n}+\sqrt{\frac{2C_{10}M\gamma t}{n}}+\frac{23C_{10}Mt}{3n}.$

Here, $M>0$ is the constant required in Assumption 2 and $C_{10}$ is from
Lemma 10.

###### Proof.

Consider the class
$\mathcal{G}_{\gamma}=\\{g_{u}:\mathbb{S}^{d-1}\to\mathbb{R}\ |\
u\in\mathcal{F}_{\gamma}\\}$. For all
$g_{u}\in\mathcal{G}_{\gamma},\|g_{u}\|_{\infty}\leq
C_{10}M,\mathrm{Var}(g_{u})\leq\mathbb{E}_{P}(g_{u}^{2})\leq C_{10}M\gamma$.
We apply Lemma 11 to yield that, with probability at least $1-2\exp(-t)$,

$\sup_{u\in\mathcal{F}_{\gamma}}\mathbb{E}_{P}g_{u}-\mathbb{E}_{\mathbf{x}}g_{u}\leq
6\mathrm{Rad}(\mathcal{G}_{\gamma},\mathbf{x})+\sqrt{\frac{2C_{10}M\gamma
t}{n}}+\frac{23C_{10}Mt}{3n}.$

To bound the empirical Rademacher complexity term, we apply Lemma 10, the
condition (4.14) and Lemma 12 to yield

$\displaystyle\mathrm{Rad}(\mathcal{G}_{\gamma},\mathbf{x})$
$\displaystyle\leq{}$ $\displaystyle C_{10}\sum_{|\alpha|\leq
s}\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma}},\mathbf{x})$
$\displaystyle\leq{}$ $\displaystyle C_{10}\sum_{|\alpha|\leq
s}\mathrm{Rad}(\\{D^{\alpha}u-D^{\alpha}u^{*}:u\in\mathcal{F},\|D^{\alpha}u-D^{\alpha}u^{*}\|_{L^{2}(\mathbf{x})}\leq
C_{12}\sqrt{\gamma}\\},\mathbf{x})$ $\displaystyle\leq{}$ $\displaystyle
C_{10}\sum_{|\alpha|\leq
s}\inf_{0<\beta<C_{12}\sqrt{\gamma}}\bigg{\\{}4\beta+\frac{12}{\sqrt{n}}\int_{\beta}^{C_{12}\sqrt{\gamma}}\sqrt{\log\mathcal{N}(\delta,D^{\alpha}\mathcal{F},\|\cdot\|_{L^{2}(\mathbf{x})})}d\delta\bigg{\\}}$
$\displaystyle\leq{}$ $\displaystyle C_{10}\sum_{|\alpha|\leq
s}\inf_{0<\beta<C_{12}\sqrt{\gamma}}\bigg{\\{}4\beta+\frac{12}{\sqrt{n}}\int_{\beta}^{C_{12}\sqrt{\gamma}}\sqrt{\log\mathcal{N}(\delta,D^{\alpha}\mathcal{F}|_{x_{1},\ldots,x_{n}},\|\cdot\|_{\infty})}d\delta\bigg{\\}}.$

When $n\geq\max_{|\alpha|\leq s}\mathrm{PDim}(D^{\alpha}\mathcal{F})$, by
Lemma 13, with
$\beta=C_{12}\sqrt{\mathrm{PDim}(D^{\alpha}\mathcal{F})\gamma/n}\leq
C_{12}\sqrt{\gamma}$ we have

$\displaystyle\inf_{0<\beta<C_{12}\sqrt{\gamma}}\bigg{\\{}4\beta+\frac{12}{\sqrt{n}}\int_{\beta}^{C_{12}\sqrt{\gamma}}\sqrt{\log\mathcal{N}(\delta,D^{\alpha}\mathcal{F}|_{x_{1},\ldots,x_{n}},\|\cdot\|_{\infty})}d\delta\bigg{\\}}$
$\displaystyle\leq{}$
$\displaystyle\inf_{0<\beta<C_{12}\sqrt{\gamma}}\bigg{\\{}4\beta+\frac{12}{\sqrt{n}}\int_{\beta}^{C_{12}\sqrt{\gamma}}\sqrt{\mathrm{PDim}(D^{\alpha}\mathcal{F})\log\left(\frac{2enM}{\delta\mathrm{PDim}(D^{\alpha}\mathcal{F})}\right)}d\delta\bigg{\\}}$
$\displaystyle\leq{}$ $\displaystyle
16\sqrt{\frac{\mathrm{PDim}(D^{\alpha}\mathcal{F})\gamma}{n}\left(\log\frac{4eM}{C_{12}\sqrt{\gamma}}+\frac{3}{2}\log
n\right)}.$

If we choose $\gamma\geq(1/n)^{2}$ and $n\geq(4eM/C_{12})^{2}$, then

$16\sqrt{\frac{\mathrm{PDim}(D^{\alpha}\mathcal{F})\gamma}{n}\left(\log\frac{4eM}{C_{12}\sqrt{\gamma}}+\frac{3}{2}\log
n\right)}\leq 32\sqrt{\frac{\mathrm{PDim}(D^{\alpha}\mathcal{F})\gamma}{n}\log
n}.$

By Lemma 14 and Theorem 4, note that

$\mathrm{VCDim}(\mathrm{sgn}((D^{\alpha}\mathcal{F})_{0}))\asymp\mathrm{VCDim}(\mathrm{sgn}(D^{\alpha}\mathcal{F}))$

we have

$32\sqrt{\frac{\mathrm{PDim}(D^{\alpha}\mathcal{F})\gamma}{n}\log n}\leq
32\sqrt{\frac{\mathrm{VCDim}(\mathrm{sgn}((D^{\alpha}\mathcal{F})_{0}))}{n}\gamma\log
n}\lesssim\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}\gamma\log n}.$

We conclude that, if $\gamma\geq(1/n)^{2}$ and $n\geq\max_{|\alpha|\leq
s}\mathrm{PDim}(D^{\alpha}\mathcal{F})\vee(4eM/C_{12})^{2}$, then

$\max_{|\alpha|\leq
s}\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma}},\mathbf{x})\lesssim\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}\gamma\log
n},\quad\mathrm{Rad}(\mathcal{G}_{\gamma},\mathbf{x})\lesssim
C_{10}\kappa\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}\gamma\log n}$

and with probability at least $1-2\exp(-t)$,

$\sup_{u\in\mathcal{F}_{\gamma}}\mathbb{E}_{P}g_{u}-\mathbb{E}_{\mathbf{x}}g_{u}\lesssim
6C_{10}\kappa\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}\gamma\log
n}+\sqrt{\frac{2C_{10}M\gamma t}{n}}+\frac{23C_{10}Mt}{3n}.$

The proof is then finished. ∎

The critical radius, denoted as $\gamma_{*}$, is defined as the smallest non-
zero fixed point that satisfies the inequality

$\gamma\geq\max_{|\alpha|\leq
s}\bigg{\\{}12M\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}),\frac{8C_{11}M^{2}t}{n},\frac{16M^{2}t}{3n}\bigg{\\}},$

where $C_{11}>0$ is the constant from (4.4). Therefore, $\gamma_{*}$ can be
expressed as

$\gamma_{*}:=\inf\bigg{\\{}\gamma>0:\gamma^{\prime}\geq\max_{|\alpha|\leq
s}\bigg{\\{}12M\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma^{\prime}}}),\frac{8C_{11}M^{2}t}{n},\frac{16M^{2}t}{3n}\bigg{\\}},\forall\gamma^{\prime}\geq\gamma\bigg{\\}}.$
(4.15)

This $\gamma_{*}$ is a positive value that relies on $t>0$. An upper bound for
the critical radius can be determined, shedding light on the inherent
complexity of the problem to some extent.

######  Lemma 18.

Assume that $t\geq 1$ and

$n\geq\max_{|\alpha|\leq
s}\mathrm{PDim}(D^{\alpha}\mathcal{F})\vee\frac{8e^{2}M^{2}}{C_{11}+3}.$

Here, $C_{11}>0$ is the constant from (4.4). Then

$\gamma_{*}\lesssim 144M^{2}\frac{\mathrm{VC}_{\mathcal{F}}}{n}\log n+48\kappa
M^{2}\exp(-t)+\frac{16C_{11}M^{2}t}{n}+\frac{32M^{2}t}{3n}.$

###### Proof.

By definition of $\gamma_{*}$, for all $\varepsilon>0$, there exists a
$\gamma\in[\gamma_{*}-\varepsilon,\gamma_{*}]$, such that

$\displaystyle\gamma$ $\displaystyle<\max_{|\alpha|\leq
s}\bigg{\\{}12M\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma}}),\frac{8C_{11}M^{2}t}{n},\frac{16M^{2}t}{3n}\bigg{\\}}$
$\displaystyle\leq\max_{|\alpha|\leq
s}\bigg{\\{}12M\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma_{*}}}),\frac{8C_{11}M^{2}t}{n},\frac{16M^{2}t}{3n}\bigg{\\}}.$

Letting $\varepsilon\to 0$ yields that

$\gamma_{*}\leq\max_{|\alpha|\leq
s}\bigg{\\{}12M\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma_{*}}}),\frac{8C_{11}M^{2}t}{n},\frac{16M^{2}t}{3n}\bigg{\\}}.$

Then we define an event

$E=\left\\{\max_{|\alpha|\leq
s}\sup_{u\in\mathcal{F}_{2\gamma_{*}}}\|D^{\alpha}u-D^{\alpha}u^{*}\|_{L^{2}(\mathbf{x})}\leq\sqrt{2(C_{11}+3)\gamma_{*}}\right\\}.$

By Lemma 15, $P(E)\geq 1-\kappa\exp(-t)$ for all $t\geq 1$. Notice that
$\gamma_{*}\geq 16M^{2}t/(3n)\geq(1/n)^{2}$, by Lemma 17,

$\displaystyle\gamma_{*}$ $\displaystyle\leq\max_{|\alpha|\leq
s}\bigg{\\{}12M\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma_{*}}}),\frac{8C_{11}M^{2}t}{n},\frac{16M^{2}t}{3n}\bigg{\\}}$
$\displaystyle\lesssim 12M\max_{|\alpha|\leq
s}\mathbb{E}_{P}(\mathbf{1}_{E}\cdot\mathrm{Rad}(D^{\alpha}\widetilde{\mathcal{F}_{\gamma_{*}}},\mathbf{x}))+24M^{2}(1-P(E))+\frac{8C_{11}M^{2}t}{n}+\frac{16M^{2}t}{3n}$
$\displaystyle\lesssim
12M\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}\gamma_{*}\log n}+24\kappa
M^{2}\exp(-t)+\frac{8C_{11}M^{2}t}{n}+\frac{16M^{2}t}{3n}.$

We solve this quadratic inequality for $\gamma^{*}$ and yield that

$\gamma_{*}\lesssim 144M^{2}\frac{\mathrm{VC}_{\mathcal{F}}}{n}\log n+48\kappa
M^{2}\exp(-t)+\frac{16C_{11}M^{2}t}{n}+\frac{32M^{2}t}{3n}.$

We then complete the proof of this lemma. ∎

In the final part of this subsection, we utilize a peeling technique to
demonstrate an important oracle inequality, which presents an upper bound for
statistical error. Extensively studied within the realm of nonparametric
statistics, oracle-type inequalities provide valuable insights (see [21] and
references therein). In the field of nonparametric statistics, an oracle
inequality outlines bounds of the risk associated with an estimator. This
inequality offers an asymptotic guarantee for the performance of an estimator
by contrasting it with an oracle procedure that possesses knowledge of some
unobservable functions or parameters within the model. Theorem 5 establishes
an oracle inequality for our PINN model, relying on the VC dimension estimates
given in Subsection 4.2.

######  Theorem 5.

For all $t\geq 0$ and

$n\geq\max_{|\alpha|\leq
s}\mathrm{PDim}(D^{\alpha}\mathcal{F})\vee\frac{8e^{2}M^{2}}{C_{11}+3},$

with probability at least $1-\exp(-t)$ we have

$\mathcal{R}(u_{n})-\mathcal{R}(u^{*})\lesssim\mathcal{R}(u_{\mathcal{F}})-\mathcal{R}(u^{*})+\frac{\mathrm{VC}_{\mathcal{F}}}{n}\log
n+\frac{t}{n}+\frac{\exp(-t)}{\log(n/\log n)}.$

Here, constant $C_{11}$ is from (4.4).

###### Proof.

Define the critical radius $\gamma_{*}$ as (4.15) with some $t^{\prime}\geq 1$
to be specified. Considering $\gamma>\gamma_{*}\vee(\log n/n)$, its precise
value will also be specified subsequently.

Recall that we have defined the localized class

$\mathcal{F}_{\gamma}:=\\{u\in\mathcal{F}:\mathbb{E}_{P}(g_{u}(X))=\mathcal{R}(u)-\mathcal{R}(u^{*})\leq\gamma\\}.$

We can partition $\mathcal{F}$ into shells without intersection

$\mathcal{F}=\mathcal{F}_{\gamma}\cup(\mathcal{F}_{2\gamma}\backslash\mathcal{F}_{\gamma})\cup\cdots\cup(\mathcal{F}_{2^{l}\gamma}\backslash\mathcal{F}_{2^{l-1}\gamma}),$

where $l\leq\log_{2}(C_{10}M/\gamma)\leq\log_{2}(C_{10}Mn/\log n)$ and
$C_{10}$ is from Lemma 10.

Assume that for some $j\leq l,u_{n}\in\mathcal{F}_{2^{j}\gamma}$. Notice that
$2^{j}\gamma>\gamma_{*}$ satisfies the condition of Lemma 15, with probability
at least $1-\kappa\exp(-t^{\prime})$, there holds

$\max_{|\alpha|\leq
s}\|D^{\alpha}u_{n}-D^{\alpha}u^{*}\|_{L^{2}(\mathbf{x})}\leq\max_{|\alpha|\leq
s}\sup_{u\in\mathcal{F}_{2^{j}\gamma}}\|D^{\alpha}u-D^{\alpha}u^{*}\|_{L^{2}(\mathbf{x})}\leq\sqrt{2^{j}(C_{11}+3)\gamma},$

where $\kappa:=\\#\\{\alpha:|\alpha|\leq s,a_{\alpha}(\cdot)\neq 0\\}$ to
represent the number of non-zero terms in $\mathcal{L}$ satisfying Assumption
1.

From Lemma 17, with probability at least $1-(\kappa+2)\exp(-t^{\prime})$ we
obtain

$\mathbb{E}_{P}g_{u_{n}}-\mathbb{E}_{\mathbf{x}}g_{u_{n}}\lesssim
6C_{10}\kappa\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}2^{j}\gamma\log
n}+\sqrt{\frac{2C_{10}M2^{j}\gamma
t^{\prime}}{n}}+\frac{23C_{10}Mt^{\prime}}{3n}.$

Using Lemma 3, with probability at least $1-(\kappa+3)\exp(-t^{\prime})$,
there holds

$\displaystyle\mathbb{E}_{P}g_{u_{n}}\lesssim{}$
$\displaystyle\mathbb{E}_{\mathbf{x}}g_{u_{n}}+6C_{10}\kappa\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}2^{j}\gamma\log
n}+\sqrt{\frac{2C_{10}M2^{j}\gamma
t^{\prime}}{n}}+\frac{23C_{10}Mt^{\prime}}{3n}$ $\displaystyle\leq{}$
$\displaystyle\mathbb{E}_{\mathbf{x}}g_{u_{\mathcal{F}}}+6C_{10}\kappa\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}2^{j}\gamma\log
n}+\sqrt{\frac{2C_{10}M2^{j}\gamma
t^{\prime}}{n}}+\frac{23C_{10}Mt^{\prime}}{3n}$ $\displaystyle\leq{}$
$\displaystyle\mathbb{E}_{\mathbf{x}}g_{u_{\mathcal{F}}}-\mathbb{E}_{P}g_{u_{\mathcal{F}}}+\mathbb{E}_{P}g_{u_{\mathcal{F}}}+6C_{10}\kappa\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}2^{j}\gamma\log
n}+\sqrt{\frac{2C_{10}M2^{j}\gamma
t^{\prime}}{n}}+\frac{23C_{10}Mt^{\prime}}{3n}$ $\displaystyle\leq{}$
$\displaystyle
2\mathbb{E}_{P}g_{u_{\mathcal{F}}}+6C_{10}\kappa\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}2^{j}\gamma\log
n}+\sqrt{\frac{2C_{10}M2^{j}\gamma
t^{\prime}}{n}}+\frac{23C_{10}Mt^{\prime}}{3n}+\frac{7C_{0}t^{\prime}}{6n},$

where $C_{0}$ is the constant from Lemma 3.

Assume that we have chosen a suitable $\gamma$ such that

$2\mathbb{E}_{P}g_{u_{\mathcal{F}}}+6C_{10}\kappa\sqrt{\frac{\mathrm{VC}_{\mathcal{F}}}{n}2^{j}\gamma\log
n}+\sqrt{\frac{2C_{10}M2^{j}\gamma
t^{\prime}}{n}}+\frac{23C_{10}Mt^{\prime}}{3n}+\frac{7C_{0}t^{\prime}}{6n}\leq
2^{j-1}\gamma.$ (4.16)