$\left\\{W_{i+1}\sigma_{i}(F):W_{i+1}\in\mathcal{B}_{i+1}\right\\},$ (123)

where $F$ is chosen and fixed. Then, we obtain

$\left|\mathcal{G}_{i+1}(F)\right|\leq\sup_{\forall j\leq
i.W_{j}\in\mathcal{B}_{j}}\mathcal{N}\left(\left\\{W_{i+1}F_{W_{1},\ldots,W_{i}}(Z):W_{i+1}\in\mathcal{B}_{i+1}\right\\},\epsilon_{i+1},\|\cdot\|_{2,2}\right),$
(124)

where we note that the matrices $W_{1}^{i}$ are fixed. Lastly we construct the
cover

$\mathcal{F}_{i+1}:=\bigcup_{F\in\mathcal{F}_{i}}\mathcal{G}_{i+1}(F),$ (125)

whose cardinality satisfies

$\left|\mathcal{F}_{i+1}\right|\leq\left|\mathcal{F}_{i}\right|\cdot
N_{i+1}\leq\prod_{l=1}^{i+1}N_{l}.$ (126)

We consider how to cover the term $\Phi_{i}$ within the PINN. We note that
each $\Phi_{i}=\Phi_{i}(Z_{j})$ for the $j$-th data is a diagonal matrix:

$\text{diag}(\Phi_{i}(Z_{j}))=\sigma^{\prime}(W_{i}\cdots\sigma(W_{1}Z_{j})),$
(127)

where the operator diag means to take the diagonal of the matrix to form a
vector, which is a $i$-layer neural network in particular. Therefore,
$\mathcal{F}_{i}$ can cover $W_{i}\cdots\sigma(W_{1}Z)$ with error not larger
than $\sum_{j\leq i}\epsilon_{j}\prod_{l=j+1}^{i}c_{l}$, and in particular the
function class $\sigma^{\prime}(\mathcal{F}_{i})$ can cover
$\sigma^{\prime}(W_{i}\cdots\sigma(W_{1}Z))$ with error not larger than
$\sum_{j\leq i}\epsilon_{j}\prod_{l=j+1}^{i}c_{l}$, where we use the fact that
$\sigma^{\prime}$ is 1-Lipschitz.

Then, we consider the $\left(\prod_{k=1}^{i}c_{k}\right)\epsilon_{i}$-cover of
the following set of intermediate output of a PINN:

$\left\\{W_{i}G_{\mathcal{W}_{1}^{i-1}}(Z):W_{j}\in\mathcal{B}_{j},\forall
j\leq i\right\\},$ (128)

where

$G_{\mathcal{W}_{1}^{i-1}}(Z)=\left[\Phi_{i-1}(z_{j})W_{i-1}\dots\Phi_{1}(z_{j})W_{1}b(z_{j})\right]_{j=1}^{n}\in\mathbb{R}^{m_{i-1}\times
n}.$ (129)

We stress that the image of PINN should be computed data-wise, as each
$\Phi_{i}(Z_{j})$ is a matrix. More concretely, we shall inductively construct
covers $\mathcal{G}_{1},\ldots,\mathcal{G}_{L}$. Specifically,
$\mathcal{G}_{1}$ is an $c_{1}\epsilon_{1}$-cover of the set of one-layer
neural networks $\left\\{W_{1}b(Z)\in\mathbb{R}^{m_{1}\times
n}:W_{1}\in\mathcal{B}_{1}\right\\}$, and its cardinality satisfies

$\left|\mathcal{G}_{1}\right|\leq\mathcal{N}\left(\left\\{W_{1}b(Z):W_{1}\in\mathcal{B}_{1}\right\\},c_{1}\epsilon_{1},\|\cdot\|_{2,2}\right).$
(130)

For every element $G\in\mathcal{G}_{i}$, and $F\in\mathcal{F}_{i}$, we
construct an $\prod_{k=1}^{i+1}c_{k}\epsilon_{i+1}$-cover of the set

$\mathcal{H}(F,G)=\left\\{\left[W_{i+1}\text{diag}(\sigma^{\prime}\left(F(Z_{j})\right))G\right]_{j=1}^{n}\in\mathbb{R}^{m_{i+1}\times
n}:W_{i+1}\in\mathcal{B}_{i+1}\right\\},$ (131)

where $G$ and $F$ are chosen and fixed. We denote

$\mathcal{G}_{i+1}:=\bigcup_{F\in\mathcal{F}_{i},G\in\mathcal{G}_{i}}\mathcal{H}\left(F,G\right).$
(132)

whose cardinality satisfies

$|\mathcal{H}(F,G)|\leq\sup_{\forall j\leq
i,W_{j}\in\mathcal{B}_{j}}\mathcal{N}\left(\left\\{W_{i+1}G_{W_{1}^{i}}(Z)\right\\},\prod_{k=1}^{i+1}c_{k}\epsilon_{i+1},\|\cdot\|_{2,2}\right):=M_{i+1},$
(133)

where $\forall j\leq i,W_{j}\in\mathcal{B}_{j}$ are fixed. Consequently, its
cardinality can be bounded as follows

$|\mathcal{G}_{i+1}|\leq\prod_{l=1}^{i+1}M_{l}N_{l}.$ (134)

After the construction, we shall show that how the function class can cover
the PINN before the $l$-th layer with an error no larger than
$l\left(\prod_{k=1}^{l}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}$,
which will be shown inductively. For the first layer, we have

$\|G_{1}-{W}_{1}b(Z)\|_{2,2}\leq c_{1}\epsilon_{1},$ (135)

for some $G_{1}\in\mathcal{G}_{1}$, due to its definition. For general cases,
consider the following inequalities:

$\displaystyle\quad\|\hat{\Phi}_{l}\hat{G}_{l}-{\Phi}_{l}\cdots{\Phi}_{1}{W}_{1}b(Z)\|_{2,2}$
(136)
$\displaystyle=\|\left[\hat{\phi}_{l}(Z_{j})\odot\hat{G}_{l}(Z_{j})-{\phi}_{l}(Z_{j})\odot
W_{l}\cdots{\Phi}_{1}(Z_{j}){W}_{1}b(Z_{j})\right]_{j=1}^{n}\|_{2,2}$
$\displaystyle\leq\|\left[(\hat{\phi}_{l}(Z_{j})-{\phi}_{l}(Z_{j}))\odot\hat{G}_{l}(Z_{j})\right]_{j=1}^{n}\|_{2,2}+\|\left[{\phi}_{l}(Z_{j})\odot(\hat{G}_{l}-{W}_{l}\cdots{\Phi}_{1}{W}_{1}b(Z_{j}))\right]_{j=1}^{n}\|_{2,2}$
$\displaystyle\leq\|(\hat{\phi}_{l}(Z)-\phi_{l}(Z))\|_{2,2}\|\hat{G}_{l}(Z)\|_{\infty,\infty}+\|{\phi}_{l}(Z)\|_{\infty,\infty}\|\hat{G}_{l}-{W}_{l}\cdots{\Phi}_{1}{W}_{1}b(Z))\|_{2,2}$
$\displaystyle\leq\left(\prod_{k=1}^{l}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}+l\left(\prod_{k=1}^{l}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}$
$\displaystyle=(l+1)\left(\prod_{k=1}^{l}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k},$

where $\hat{\Phi}_{l}\in\mathcal{F}_{l}$. Furthermore, consider the new
$W_{l+1}$, we attain

$\displaystyle\quad\|\hat{G}_{l+1}-W_{l+1}{\Phi}_{l}\cdots{\Phi}_{1}{W}_{1}b(Z)\|_{2,2}$
(137)
$\displaystyle\leq\|\hat{G}_{l+1}-W_{l+1}\hat{\Phi}_{l}\hat{G}_{l-1}\|_{2,2}+\|W_{l+1}\hat{\Phi}_{l}\hat{G}_{l-1}-W_{l+1}{\Phi}_{l}\cdots{\Phi}_{1}{W}_{1}b(Z)\|_{2,2}$
$\displaystyle\leq\left(\prod_{k=1}^{l+1}c_{k}\right)\epsilon_{l+1}+(l+1)\left(\prod_{k=1}^{l+1}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}$
$\displaystyle\leq(l+1)\left(\prod_{k=1}^{l+1}c_{k}\right)\sum_{j=1}^{l+1}\epsilon_{j}\prod_{k=j+1}^{l+1}c_{k}.$

By the same logic, we have

$\displaystyle\quad\log\mathcal{N}\left(\mathcal{H}_{X},\tau,\|\cdot\|_{2,2}\right)$
(138) $\displaystyle=\sum_{i=1}^{L}\sup_{W_{j}\in\mathcal{B}_{j},\forall
j<i}\log\left(\mathcal{N}\left(\left\\{W_{i}F_{\mathcal{W}_{1}^{i-1}}(Z)\right\\},\epsilon_{i},\|\cdot\|_{2,2}\right)\right)+\sup_{W_{j}\in\mathcal{B}_{j},\forall
j<i}\log\left(\mathcal{N}\left(\left\\{W_{i}G_{\mathcal{W}_{1}^{i-1}}(Z)\right\\},\prod_{k=1}^{i}c_{i}\epsilon_{i},\|\cdot\|_{2,2}\right)\right)$
$\displaystyle\leq\sum_{i=1}^{L}\sup_{W_{j}\in\mathcal{B}_{j},\forall
j<i}\frac{b_{i}^{2}\|F_{\mathcal{W}_{1}^{i-1}}(Z)^{\mathrm{T}}\|_{2,2}^{2}\log(2h^{2})}{\epsilon_{i}^{2}}+\sup_{W_{j}\in\mathcal{B}_{j},\forall
j<i}\frac{b_{i}^{2}\|G_{\mathcal{W}_{1}^{i-1}}(Z)^{\mathrm{T}}\|_{2,2}^{2}\log(2h^{2})}{\left(\prod_{k=1}^{i}c_{k}\right)^{2}\epsilon_{i}^{2}}.$

Set $\mathcal{B}_{i}=\left\\{W_{i}:\left\|W_{i}\right\|_{2}\leq
s_{i},\left\|W_{i}\right\|_{2,1}\leq b_{i}\right\\}$, and set the per-layer
cover resolutions $\left(\epsilon_{1},\ldots,\epsilon_{L}\right)$ as

$\epsilon_{i}:=\frac{\alpha_{i}\epsilon}{L\left(\prod_{k=1}^{L}s_{k}\right)\prod_{j>i}s_{j}}\quad\text{
where
}\quad\alpha_{i}:=\frac{1}{\bar{\alpha}}\left(\frac{b_{i}}{s_{i}}\right)^{2/3}\quad,\quad\bar{\alpha}:=\sum_{j=1}^{L}\left(\frac{b_{j}}{s_{j}}\right)^{2/3}.$
(139)

By this choice, it follows that the final cover resolution $\tau$ provided by
Lemma satisfies

$\displaystyle\tau$ $\displaystyle\leq
L\left(\prod_{k=1}^{L}s_{k}\right)\sum_{j=1}^{L}\epsilon_{j}\prod_{k=j+1}^{L}s_{k}$
(140)
$\displaystyle\leq\sum_{i=1}^{L}\frac{\alpha_{i}\epsilon}{\prod_{j>i}s_{j}}\prod_{l=j+1}^{L}s_{l}$
$\displaystyle\leq\sum_{j\leq L}\alpha_{j}\epsilon$ $\displaystyle=\epsilon.$

Therefore, the covering number bound can be written as:

$\displaystyle\quad\log\mathcal{N}\left(\mathcal{H}_{X},\epsilon,\|\cdot\|_{2}\right)$
(141) $\displaystyle\leq\sum_{i=1}^{L}\sup_{A_{j}\in\mathcal{B}_{j},\forall
j<i}\frac{b_{i}^{2}\|F_{\mathcal{A}_{1}^{i-1}}(Z)^{\mathrm{T}}\|_{2,2}^{2}\log(2h^{2})}{\epsilon_{i}^{2}}+\sup_{A_{j}\in\mathcal{B}_{j},\forall
j<i}\frac{b_{i}^{2}\|G_{\mathcal{A}_{1}^{i-1}}(Z)^{\mathrm{T}}\|_{2,2}^{2}\log(2h^{2})}{\left(\prod_{k=1}^{i}c_{k}\right)^{2}\epsilon_{i}^{2}}$
$\displaystyle\leq\sum_{i=1}^{L}\frac{b_{i}^{2}nd\log(2h^{2})}{\epsilon_{i}^{2}}+\frac{b_{i}^{2}\left(\prod_{k=1}^{i}s_{k}\right)^{2}\|Z\|_{2,2}^{2}\log(2h^{2})}{\left(\prod_{k=1}^{i}s_{k}\right)^{2}\epsilon_{i}^{2}}$
$\displaystyle\leq
2nd\log\left(2h^{2}\right)\sum_{i=1}^{L}\frac{b_{i}^{2}\log(2h^{2})}{\epsilon_{i}^{2}}$
$\displaystyle\leq
2nd\log\left(2h^{2}\right)L^{2}\left(\prod_{k=1}^{L}s_{k}\right)^{2}\sum_{i=1}^{L}\frac{b_{i}^{2}\left(\prod_{j>i}s_{j}\right)^{2}}{\epsilon^{2}}$
$\displaystyle\leq
2nd\log\left(2h^{2}\right)L^{2}\left(\prod_{k=1}^{L}s_{k}\right)^{4}\left(\bar{\alpha}^{3}\right)/\epsilon^{2}.$

Consequently, the Rademacher complexity of the function class constructed by
all first-order derivatives is bounded by

$\frac{4}{n\sqrt{n}}+\frac{18L\sqrt{2d\log(2h^{2})}\log
n}{\sqrt{n}}\left(\prod_{l=1}^{L}M(l)\right)^{2}\left(\sum_{l=1}^{L}N(l)^{2/3}\right)^{3/2}.$
(142)

∎

######  Lemma C.7.

The hypothesis class $\mathcal{F}_{1}$ given by

$\displaystyle\mathcal{F}_{1}$
$\displaystyle=\left\\{\boldsymbol{x}\mapsto\sum_{j=1}^{d}\sum_{l=1}^{L-1}{W}_{L}{\Phi}_{L-1}\cdots{W}_{l+1}\text{diag}({\Psi}_{l}{W}_{l}\cdots{\Psi}_{1}({W}_{1})_{:,j}){W}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,j}(\boldsymbol{x})\right\\},$
(143)

where the weight matrices satisfy

$\|W_{l}\|_{2}\leq M(l),\frac{\|W_{l}\|_{2,1}}{\|W_{l}\|_{2}}\leq N(l),\forall
l,$ (144)

satisfies the Rademacher complexity bound

$\text{Rad}(\mathcal{F}_{1};S)\leq\frac{4}{n\sqrt{n}}+\frac{18(L+1)\sqrt{2d\log(2h^{2})}\log
n}{\sqrt{n}}\left(\prod_{l=1}^{L}M(l)\right)^{3}\left(\sum_{l=1}^{L}N(l)^{2/3}\right)^{3/2}.$
(145)

###### Proof.

We consider the $d(L-1)$ terms one-by-one, and focus on the following term in
particular:

${W}_{L}{\Phi}_{L-1}\cdots{W}_{i+1}\text{diag}({\Psi}_{i}{W}_{i}\cdots{\Psi}_{1}({W}_{1})_{:,j}){W}_{i}\cdots{\Phi}_{1}{W}_{1}A_{:,j}(Z),$
(146)

where we recall that $A_{:j}$ is the $j$-th row of the fixed coefficient
function and $A_{:,j}(Z)\in\mathbb{R}^{d\times n}$. We consider the covering
in Lemma C.4. Specifically, denote $\mathcal{F}_{i}$ as the $\epsilon_{i}$
covering of the following set:

$\left\\{W_{i}F_{\mathcal{W}_{1}^{i-1}}(Z):W_{j}\in\mathcal{B}_{j},\forall
j\leq i\right\\},$ (147)

which is the set of all $i$-layer neural networks, with weight matrices in the
set $\mathcal{B}_{j}$. In Lemma C.4, we have inductively constructed covers
$\mathcal{F}_{1},\ldots,\mathcal{F}_{L}$. Specifically, $\mathcal{F}_{1}$ is
an $\epsilon_{1}$-cover of the set of one-layer neural networks
$\left\\{W_{1}Z:W_{1}\in\mathcal{B}_{1}\right\\}$, and its cardinality
satisfies

$\left|\mathcal{F}_{1}\right|\leq\mathcal{N}\left(\left\\{W_{1}Z:W_{1}\in\mathcal{B}_{1}\right\\},\epsilon_{1},\|\cdot\|_{2,2}\right).$
(148)

For every element $F\in\mathcal{F}_{i}$, which is a covering vector of the set
of image of all $i$-layer neural networks, we construct an
$\epsilon_{i+1}$-cover $\mathcal{G}_{i+1}(F)$ of the set

$\left\\{W_{i+1}\sigma(F):W_{i+1}\in\mathcal{B}_{i+1}\right\\},$ (149)

where $F\in\mathcal{F}_{i}$ is chosen and fixed. Then, we obtain

$\left|\mathcal{G}_{i+1}(F)\right|\leq\sup_{\forall j\leq
i.W_{j}\in\mathcal{B}_{j}}\mathcal{N}\left(\left\\{W_{i+1}F_{W_{1},\ldots,W_{i}}(Z):W_{i+1}\in\mathcal{B}_{i+1}\right\\},\epsilon_{i+1},\|\cdot\|_{2,2}\right).$
(150)

Lastly we construct the cover

$\mathcal{F}_{i+1}:=\bigcup_{F\in\mathcal{F}_{i}}\mathcal{G}_{i+1}(F),$ (151)

whose cardinality satisfies

$\left|\mathcal{F}_{i+1}\right|\leq\left|\mathcal{F}_{i}\right|\cdot
N_{i+1}\leq\prod_{l=1}^{i+1}N_{l}.$ (152)

We consider how to cover the term $\Phi_{i}$ within the PINN. We note that
$\Phi_{i}=\Phi_{i}(Z_{j})\in\mathbb{R}_{m_{i}\times m_{i}}$ is a diagonal
matrix:

$\text{diag}(\Phi_{i}(Z_{j}))=\sigma^{\prime}(W_{i}\cdots\sigma(W_{1}Z_{j})),$
(153)

where the which is a $i$-layer neural network in particular. Therefore,
$\sigma^{\prime}(\mathcal{F}_{i})$ can cover
$\sigma^{\prime}(W_{i}\cdots\sigma(W_{1}Z))$ with error not larger than
$\sum_{j\leq i}\epsilon_{j}\prod_{l=j+1}^{i}c_{l}$. Similarly, since
$\Psi_{i}$ is also a $i$-layer neural net, we can do the same reasoning to it.
In these covering, we use the fact that $\sigma^{\prime}$ and
$\sigma^{\prime\prime}$ are 1-Lipschitz functions.

Then, we consider the $\left(\prod_{k=1}^{i}c_{k}\right)\epsilon_{i}$-cover of
the following set of intermediate output of a PINN:

$\left\\{W_{i}G_{\mathcal{W}_{1}^{i-1}}(Z):W_{j}\in\mathcal{B}_{j},\forall
j\leq i\right\\},$ (154)

where

$G_{\mathcal{W}_{1}^{i-1}}(Z)=\left[\Phi_{i-1}(z_{j})W_{i-1}\dots\Phi_{1}(z_{j})W_{1}A_{:,}(z_{j})\right]_{j=1}^{n}\in\mathbb{R}^{m_{i-1}\times
n}.$ (155)

More concretely, we shall inductively construct covers
$\mathcal{G}_{1},\ldots,\mathcal{G}_{L}$. Specifically, $\mathcal{G}_{1}$ is
an $c_{1}\epsilon_{1}$-cover of the set of one-layer neural networks
$\left\\{W_{1}A_{:,}(Z)\in\mathbb{R}^{m_{1}\times
n}:W_{1}\in\mathcal{B}_{1}\right\\}$, and its cardinality satisfies

$\left|\mathcal{G}_{1}\right|\leq\mathcal{N}\left(\left\\{W_{1}A_{:,}(Z):W_{1}\in\mathcal{B}_{1}\right\\},c_{1}\epsilon_{1},\|\cdot\|_{2,2}\right).$
(156)

For every element $G\in\mathcal{G}_{i}$, and $F\in\mathcal{F}_{i}$, we
construct an $\left(\prod_{k=1}^{i+1}c_{k}\right)\epsilon_{i+1}$-cover of the
set

$\mathcal{H}(F,G)=\left\\{\left[W_{i+1}\text{diag}(F(Z_{j}))G\right]_{j=1}^{n}\in\mathbb{R}^{m_{i+1}\times
n}:W_{i+1}\in\mathcal{B}_{i+1}\right\\},$ (157)

where $G$ and $F$ are chosen and fixed. We stree that
$F(Z_{j})\in\mathbb{R}^{m_{i}}$, and thus
$\text{diag}(F(Z_{j}))\in\mathbb{R}^{m_{i}\times m_{i}}$. Since for each input
data, the matrix $F(Z_{j})$ depends on itself, we cannot write the expression
of the functions in $\mathcal{H}(F,G)$ as a direct matrix multiplication. We
denote

$\mathcal{G}_{i+1}:=\bigcup_{F\in\mathcal{F}_{i},G\in\mathcal{G}_{i}}\mathcal{H}\left(F,G\right).$
(158)

whose cardinality satisfies

$|\mathcal{H}(F,G)|\leq\sup_{\forall j\leq
i,W_{j}\in\mathcal{B}_{j}}\mathcal{N}\left(\left\\{W_{i+1}G_{W_{1}^{i}}(Z)\right\\},\left(\prod_{k=1}^{i+1}c_{k}\right)\epsilon_{i+1},\|\cdot\|_{2,2}\right)=:M_{i+1}.$
(159)

Consequently,

$|\mathcal{G}_{i+1}|\leq\prod_{l=1}^{i+1}M_{l}N_{l}.$ (160)

For the first layer, we have

$\|G_{1}-{W}_{1}A_{:,}(Z)\|_{2,2}\leq c_{1}\epsilon_{1},$ (161)

for some $G_{1}\in\mathcal{G}_{1}$, due to its definition. For general cases,
consider the following inequalities:

$\displaystyle\quad\|\hat{\Phi}_{l}\hat{G}_{l}-{\Phi}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
(162) $\displaystyle=\|\hat{\phi}_{l}\odot\hat{G}_{l}-{\phi}_{l}\odot
W_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
$\displaystyle\leq\|(\hat{\phi}_{l}-{\phi}_{l})\odot\hat{G}_{l}\|_{2,2}+\|{\phi}_{l}\odot(\hat{G}_{l}-{W}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z))\|_{2,2}$
$\displaystyle\leq\|(\hat{\phi}_{l}-\phi_{l})\|_{2,2}\|\hat{G}_{l}\|_{\infty,\infty}+\|{\phi}_{l}\|_{\infty,\infty}\|\hat{G}_{l}-{W}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z))\|_{2,2}$
$\displaystyle\leq\left(\prod_{k=1}^{l}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}+l\left(\prod_{k=1}^{l}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}$
$\displaystyle\leq(l+1)\left(\prod_{k=1}^{l}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k},$

where $\hat{\Phi}_{l}\in\mathcal{F}_{i}$ and $\phi=\text{diag}(\Phi)$. More
concretely, we only each data point one-by-one in the above reasoning, i.e.,

$\displaystyle\quad\|\hat{\Phi}_{l}\hat{G}_{l}-{\Phi}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
(163)
$\displaystyle=\|\left[\hat{\phi}_{l}(Z_{j})\odot\hat{G}_{l}(Z_{j})-{\phi}_{l}(Z_{j})\odot
W_{l}\cdots{\Phi}_{1}(Z_{j}){W}_{1}A_{:,}(Z_{j})\right]_{j=1}^{n}\|_{2,2}$
$\displaystyle\leq\|\left[(\hat{\phi}_{l}(Z_{j})-{\phi}_{l}(Z_{j}))\odot\hat{G}_{l}(Z_{j})\right]_{j=1}^{n}\|_{2,2}+\|\left[{\phi}_{l}(Z_{j})\odot(\hat{G}_{l}-{W}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z_{j}))\right]_{j=1}^{n}\|_{2,2}$
$\displaystyle\leq\|(\hat{\phi}_{l}(Z)-\phi_{l}(Z))\|_{2,2}\|\hat{G}_{l}(Z)\|_{\infty,\infty}+\|{\phi}_{l}(Z)\|_{\infty,\infty}\|\hat{G}_{l}-{W}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z))\|_{2,2}$
$\displaystyle\leq\left(\prod_{k=1}^{l}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}+l\left(\prod_{k=1}^{l}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}$
$\displaystyle\leq(l+1)\left(\prod_{k=1}^{l}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k},$

Furthermore, consider the new $W_{l}$, we attain

$\displaystyle\quad\|\hat{G}_{l+1}-W_{l+1}{\Phi}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
(164)
$\displaystyle\leq\|\hat{G}_{l+1}-W_{l+1}\hat{\Phi}_{l}\hat{G}_{l-1}\|_{2,2}+\|W_{l+1}\hat{\Phi}_{l}\hat{G}_{l-1}-W_{l+1}{\Phi}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
$\displaystyle\leq\left(\prod_{k=1}^{l+1}c_{k}\right)\epsilon_{l+1}+(l+1)\left(\prod_{k=1}^{l+1}c_{k}\right)\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}$
$\displaystyle\leq(l+1)\left(\prod_{k=1}^{l+1}c_{k}\right)\sum_{j=1}^{l+1}\epsilon_{j}\prod_{k=j+1}^{l+1}c_{k}.$

Consider the case when $l=i$, and in particular the term
$\Omega_{i}=\text{diag}({\Psi}_{i}{W}_{i}\cdots{\Psi}_{1}({W}_{1})_{:,j})$,
then the PINN can be rewritten as

${W}_{L}{\Phi}_{L-1}\cdots{W}_{i+1}\Omega_{i}{W}_{i}\cdots{\Phi}_{1}{W}_{1}A_{:,j}(Z),$
(165)

where we note that the only difference between the first-order derivatives and
the second-order ones is in the term $\Omega_{i}$.

$\displaystyle\quad\|\hat{\Omega}_{i}\hat{G}_{i}-{\Omega}_{i}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
(166)
$\displaystyle=\|\left[\hat{\omega}_{i}\odot\hat{G}_{i}-{\omega}_{i}\odot
W_{i}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z_{j})\right]_{j=1}^{n}\|_{2,2}$
$\displaystyle\leq\|\left[(\hat{\omega}_{i}(Z_{j})-{\omega}_{i}(Z_{j}))\odot\hat{G}_{i}(Z_{j})\right]_{j=1}^{n}\|_{2,2}+\|\left[{\omega}_{i}(Z_{j})\odot(\hat{G}_{i}-{W}_{i}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z_{j}))\right]\|_{2,2}$
$\displaystyle\leq\|(\hat{\omega}_{i}-\omega_{i})\|_{2,2}\|\hat{G}_{i}\|_{\infty,\infty}+\|{\omega}_{i}\|_{\infty,\infty}\|\hat{G}_{i}-{W}_{i}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z))\|_{2,2}$
$\displaystyle\leq\left(\prod_{k=1}^{i}c_{k}\right)\text{Approximate Error of
Omega}+i\left(\prod_{k=1}^{i}c_{k}\right)^{2}\sum_{j=1}^{i}\epsilon_{j}\prod_{k=j+1}^{i}c_{k}.$

Based on our discussions in the previous lemma on first-order derivatives, we
know that the constructed function class can cover the PINN before the $i$-th
layer with an error no larger than
$i\left(\prod_{k=1}^{i}c_{k}\right)\sum_{j=1}^{i}\epsilon_{j}\prod_{k=j+1}^{i}c_{k}$,
i.e., the approximation error of $\Omega_{l}$ should be that quantity. Thus,
we proceed with our bound and attain:

$\displaystyle\quad\|\hat{\Omega}_{l}\hat{G}_{l}-{\Omega}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
(167) $\displaystyle\leq\left(\prod_{k=1}^{i}c_{k}\right)\text{Approximate
Error of
Omega}+i\left(\prod_{k=1}^{i}c_{k}\right)^{2}\sum_{j=1}^{i}\epsilon_{j}\prod_{k=j+1}^{i}c_{k}$
$\displaystyle\leq
i\left(\prod_{k=1}^{i}c_{k}\right)^{2}\sum_{j=1}^{i}\epsilon_{j}\prod_{k=j+1}^{i}c_{k}+i\left(\prod_{k=1}^{i}c_{k}\right)^{2}\sum_{j=1}^{i}\epsilon_{j}\prod_{k=j+1}^{i}c_{k}$
$\displaystyle\leq
2i\left(\prod_{k=1}^{i}c_{k}\right)^{2}\sum_{j=1}^{i}\epsilon_{j}\prod_{k=j+1}^{i}c_{k}.$

Then, we guess the approximation error after the $l$-th layer is:

$2l\left(\prod_{k=1}^{l}c_{k}\right)^{2}\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{i}c_{k}.$
(168)

For $l\leq i$, the above bound has already been shown. For the case when
$l=i+1$, consider the new $W_{i+1}$, we attain

$\displaystyle\quad\|\hat{G}_{i+1}-W_{i+1}{\Omega}_{i}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
(169)
$\displaystyle\leq\|\hat{G}_{i+1}-W_{i+1}\hat{\Omega}_{i}\hat{G}_{i}\|_{2,2}+\|W_{i+1}\hat{\Omega}_{i}\hat{G}_{i-1}-W_{i+1}{\Omega}_{i}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
$\displaystyle\leq\left(\prod_{k=1}^{i+1}c_{k}\right)\epsilon_{i+1}+2ic_{i+1}\left(\prod_{k=1}^{i}c_{k}\right)^{2}\sum_{j=1}^{i}\epsilon_{j}\prod_{k=j+1}^{i}c_{k}$
$\displaystyle\leq
2i\left(\prod_{k=1}^{i+1}c_{k}\right)^{2}\sum_{j=1}^{i+1}\epsilon_{j}\prod_{k=j+1}^{i+1}c_{k}.$

For the $l$-th layer after the $i$-th layer, there is no $\Omega$ terms any
more, and we only need to focus on $\Phi$:

$\displaystyle\quad\|\hat{\Phi}_{l}\hat{G}_{l}-{\Phi}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
(170) $\displaystyle=\|\hat{\phi}_{l}\odot\hat{G}_{l}-{\phi}_{l}\odot
W_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
$\displaystyle\leq\|(\hat{\phi}_{l}-{\phi}_{l})\odot\hat{G}_{l}\|_{2,2}+\|{\phi}_{l}\odot(\hat{G}_{l}-{W}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z))\|_{2,2}$
$\displaystyle\leq\|(\hat{\phi}_{l}-\phi_{l})\|_{2,2}\|\hat{G}_{l}\|_{\infty,\infty}+\|{\phi}_{l}\|_{\infty,\infty}\|\hat{G}_{l}-{W}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z))\|_{2,2}$
$\displaystyle\leq\left(\prod_{k=1}^{l}c_{k}\right)^{2}\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}+2l\left(\prod_{k=1}^{l}c_{k}\right)^{2}\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}$
$\displaystyle\leq
2(l+1)\left(\prod_{k=1}^{l}c_{k}\right)^{2}\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}.$

And we consider the multiplication of $W_{l+1}$:

$\displaystyle\quad\|\hat{G}_{l+1}-W_{l+1}{\Phi}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
(171)
$\displaystyle\leq\|\hat{G}_{l+1}-W_{l+1}\hat{\Phi}_{l}\hat{G}_{l-1}\|_{2,2}+\|W_{l+1}\hat{\Phi}_{l}\hat{G}_{l-1}-W_{l+1}{\Phi}_{l}\cdots{\Phi}_{1}{W}_{1}A_{:,}(Z)\|_{2,2}$
$\displaystyle\leq\left(\prod_{k=1}^{l+1}c_{k}\right)\epsilon_{l+1}+2(l+1)\left(\prod_{k=1}^{l+1}c_{k}\right)^{2}\sum_{j=1}^{l}\epsilon_{j}\prod_{k=j+1}^{l}c_{k}$
$\displaystyle\leq
2(l+1)\left(\prod_{k=1}^{l+1}c_{k}\right)^{2}\sum_{j=1}^{l+1}\epsilon_{j}\prod_{k=j+1}^{l+1}c_{k}.$

After covering the class of second-order derivatives in the PINN model, by the
same logic, we have the Rademacher complexity of the function class
constructed by all second-order derivatives is bounded by

$\frac{4}{n\sqrt{n}}+\frac{18(L+1)\sqrt{2d\log(2h^{2})}\log
n}{\sqrt{n}}\left(\prod_{l=1}^{L}M(l)\right)^{3}\left(\sum_{l=1}^{L}N(l)^{2/3}\right)^{3/2}.$
(172)

∎

###  C.2 Tree-Like Function Space

Barron space is an important pat in this paper. The spectral norm based bound
can be connected with the Barron space, and thus provide Rademacher complexity
bound for tree-like functions in the Barron space.

To illustrate the idea, we use the original network,

$\mathcal{NN}^{L}_{M}:=\left\\{\boldsymbol{x}\mapsto
W_{L}\sigma(W_{L-1}\sigma(\cdots\sigma(W_{1}\boldsymbol{x})))\ |\
\|W_{l}\|_{2}\leq M(l),\frac{\|W_{l}\|_{2,1}}{\|W_{l}\|_{2}}\leq N(l),\forall
l\right\\},$ (173)

which satisfies the Rademacher complexity bound

$\text{Rad}(\mathcal{NN}^{L}_{M};S)\leq\prod_{l=1}^{L}M(l)\Big{(}\sum_{l=1}^{L}N(l)^{2/3}\Big{)}^{3/2}\frac{\log(2h^{2})\log
n}{{n}^{1/2}},$ (174)

where $h$ is the maximal width of the neural network, i.e.,

$h=\max(m_{L},\cdots,m_{0}).$ (175)

In other words, the Rademacher complexity is related to the following
quantity:

$\prod_{l=1}^{L}\|W_{l}\|_{2}\Big{(}\sum_{l=1}^{L}\left(\frac{\|W_{l}\|_{2,1}}{\|W_{l}\|_{2}}\right)^{2/3}\Big{)}^{3/2}\frac{\log(2h^{2})\log
n}{{n}^{1/2}},$ (176)

which can be upper bounded by the path norm or $(1,\infty)$ norm related to
the Barron space. Concretely, all matrix norms are equivalent, which means
there exists a constant $C(h)$ that depends on the maximal width $h$, such
that

$\prod_{l=1}^{L}\|W_{l}\|_{2}\Big{(}\sum_{l=1}^{L}\left(\frac{\|W_{l}\|_{2,1}}{\|W_{l}\|_{2}}\right)^{2/3}\Big{)}^{3/2}\leq
C(h)\prod_{l=1}^{L}\ \|W_{l}\|_{1,\infty}.$ (177)

These intuitions are summarized in the following theorem.

######  Lemma C.8.

(Rademacher Complexity of Tree-Like Functions). For every $L$, and every set
of $n$ points $S\subset\overline{\Omega}$, the hypothesis class
$\mathcal{NN}^{L}_{M}$ given by the neural networks

$\mathcal{NN}^{L}_{M}:=\left\\{\boldsymbol{x}\mapsto
W_{L}\sigma(W_{L-1}\sigma(\cdots\sigma(W_{1}\boldsymbol{x})))\ |\
\|W_{l}\|_{1,\infty}\leq M(l),\forall l\right\\},$ (178)

satisfies the Rademacher complexity bound

$\text{Rad}(\mathcal{NN}^{L}_{M};S)\leq\left(\prod_{l=1}^{L}\|W_{l}\|_{1,\infty}\right)\frac{C(h)\log
n}{{n}^{1/2}},$ (179)

where $h$ is the maximal width of the neural network, i.e.,

$h=\max(m_{L},\cdots,m_{0}),$ (180)

and $C(h)$ is a universal constant depending only on $h$.

######  Lemma C.9.

(Rademacher Complexity of Tree-Like Functions). For every $L$, and every set
of $n$ points $S\subset\overline{\Omega}$, the hypothesis class
$\mathcal{PINN}^{L}_{M}$ given by the neural networks

$\mathcal{PINN}^{L}_{M}:=\left\\{\boldsymbol{x}\mapsto\mathcal{L}u_{\boldsymbol{\theta}}(\boldsymbol{x})\
|\ u_{\boldsymbol{\theta}}\in\mathcal{NN}_{M}^{L}\right\\},$ (181)

satisfies the Rademacher complexity bound

$\text{Rad}(\mathcal{NN}^{L}_{M};S)\leq\left(\prod_{l=1}^{L}\|W_{l}\|_{1,\infty}\right)^{3}\frac{C(h,K)\log
n}{{n}^{1/2}},$ (182)

where $h$ is the maximal width of the neural network, i.e.,

$h=\max(m_{L},\cdots,m_{0}),$ (183)

and $C(h,K3)$ is a universal constant depending only on $h,K$.

##  Appendix D Proofs of Main Results

###  D.1 Proof of Theorem 3.1

###### Proof.

(Proof of Theorem 3.1) Let $u_{\hat{\boldsymbol{\theta}}}$ parameterized by
$\hat{\boldsymbol{\theta}}$ satisfy the conditions in Theorem 2.3, i.e.,

$\displaystyle\|u_{\boldsymbol{\theta}}-f\|_{H^{2}(\mathbb{P})}$
$\displaystyle\leq\frac{3L\|f\|_{\mathcal{W}^{L}(\Omega)}}{\sqrt{m}},$ (184)
$\displaystyle\|u_{\boldsymbol{\theta}}-f\|_{L^{2}(\mathbb{Q})}$
$\displaystyle\leq\frac{3C_{\Omega}L\|f\|_{\mathcal{W}^{L}(\Omega)}}{\sqrt{m}},$
$\displaystyle\|\hat{W}_{L}\|_{1,\infty}$
$\displaystyle\leq\|u^{*}\|_{\mathcal{W}^{L}(\Omega)}.$

where $\|\hat{W}_{L}\|_{1,\infty}$ is the $L$-th layer weight parameter matrix
of $\hat{\boldsymbol{\theta}}$, and take the probability measures
$\mathbb{P},\mathbb{Q}$ as

$\displaystyle\mathbb{P}=\frac{1}{n_{r}}\sum_{\boldsymbol{x}\in
S\cap\Omega}\delta_{\boldsymbol{x}},\qquad\mathbb{Q}=\frac{1}{n_{b}}\sum_{\boldsymbol{x}\in
S\cap\partial\Omega}\delta_{\boldsymbol{x}},$ (185)

which means $\mathbb{P}$ contains the empirical distribution of residual
points and $\mathbb{Q}$ contains the empirical distribution of boundary
points. Thus

$\displaystyle R_{S}(\theta)$
$\displaystyle=\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}{\left(u_{\hat{\boldsymbol{\theta}}}(\boldsymbol{x}_{b,i})-g(\boldsymbol{x}_{b,i})\right)}^{2}+\frac{1}{n_{r}}\sum_{i=1}^{n_{r}}{\left(\mathcal{L}u_{\hat{\boldsymbol{\theta}}}(\boldsymbol{x}_{r,i})-f(\boldsymbol{x}_{r,i})\right)}^{2}$
(186)
$\displaystyle\leq\|u_{\boldsymbol{\theta}}-u^{*}\|_{H^{2}(\mathbb{P})}+2K\|u_{\boldsymbol{\theta}}-u^{*}\|_{L^{2}(\mathbb{Q})}$
$\displaystyle\leq\frac{3(2KC_{\Omega}+1)L\|f\|_{\mathcal{W}^{L}(\Omega)}}{\sqrt{m}}$

Then, by the fact that
$\boldsymbol{\theta}^{*}=\arg\min_{\boldsymbol{\theta}}R_{S}(\boldsymbol{\theta})+\lambda\|W_{L}\|^{2}_{1,\infty}$
and $\lambda={3(2KC_{\Omega}+1)L^{2}}/{m}$, we have

$\displaystyle
R_{S}(\boldsymbol{\theta}^{*})+\lambda\|W_{L}^{*}\|_{1,\infty}^{2}$
$\displaystyle\leq
R_{S}(\hat{\boldsymbol{\theta}})+\lambda\|\hat{W}_{L}\|_{1,\infty}^{2}$ (187)
$\displaystyle\leq\frac{3(2KC_{\Omega}+1)L^{2}\|u^{*}\|^{2}_{\mathcal{W}^{L}(\Omega)}}{m}+\lambda\|u^{*}\|^{2}_{\mathcal{W}^{L}(\Omega)}$
$\displaystyle=2\lambda\|u^{*}\|_{\mathcal{W}^{L}(\Omega)}.$

In particular,

$\|u_{\boldsymbol{\theta}^{*}}\|_{\mathcal{W}^{L}}\leq\|W_{L}^{*}\|_{1,\infty}\leq\frac{2\lambda\|u^{*}\|_{\mathcal{W}^{L}(\Omega)}}{\lambda}=2\|u^{*}\|_{\mathcal{W}^{L}(\Omega)}.$
(188)

The Rademacher complexity of the neural network model used for the boundary
points prediction is upper bounded by

$\|W_{L}\|_{1,\infty}\frac{C(h)\log n_{b}}{{\sqrt{n_{b}}}}\leq
2\|u^{*}\|_{\mathcal{W}^{L}(\Omega)}\frac{C(h)\log n_{b}}{{\sqrt{n_{b}}}}.$
(189)

Hence, consider the function class of the composition of the MSE loss function
and the neural network model. Since we have truncated the neural network
function to $[-1,1]$, i.e.,

$l(\boldsymbol{x},\boldsymbol{x}^{\prime})=\frac{1}{2}\|\boldsymbol{x}-\boldsymbol{x}^{\prime}\|_{2}^{2}\leq
2,$ (190)

we know that the loss function is $\overline{c}$-Lipschitz. Therefore, we can
attain the following generalization bound for boundary points prediction:

$R_{D\cap\partial\Omega}(\boldsymbol{\theta}^{*})\leq
R_{S\cap\partial\Omega}(\boldsymbol{\theta}^{*})+16\|u^{*}\|_{\mathcal{W}^{L}(\Omega)}\frac{C(h)\log
n_{b}}{{\sqrt{n_{b}}}}+2\sqrt{\frac{2\log(2/\delta)}{n_{b}}}.$ (191)

Similarly, Rademacher complexity of the PINN used for residual points, i.e.,
the differentiated networks are upper bounded by

$\left(\|W_{L}\|_{1,\infty}\right)^{3}\frac{C(h,K)\log
n_{r}}{{\sqrt{n_{r}}}},$ (192)

This is due to the fact that for the neural networks in the tree-like function
space section, we force $\|W_{l}\|_{1,\infty}\leq 1$, for all $1\leq l\leq
L-1$. Thus, only $\|W_{L}\|_{1,\infty}$ matters for its Rademacher complexity.
Therefore, the generalization bound for the residual loss is:

$R_{D\cap\Omega}(\boldsymbol{\theta}^{*})\leq
R_{S\cap\Omega}(\boldsymbol{\theta}^{*})+16\left(\|u^{*}\|_{\mathcal{W}^{L}(\Omega)}\right)^{3}\frac{C(h,K)\log
n_{r}}{{\sqrt{n_{r}}}}+2\sqrt{\frac{2\log(2/\delta)}{n_{r}}}.$ (193)

In sum, we obtain the two generalization bounds on the boundary and in the
residual, respectively. ∎

###  D.2 Proof of Theorem 3.2

###### Proof.

(Proof of Theorem 3.2). Consider the function class

$\mathcal{H}_{M,N}^{L}=\left\\{\boldsymbol{x}\mapsto
l(u(\boldsymbol{x}),u_{\boldsymbol{\theta}}(\boldsymbol{x}))\ |\
\|\boldsymbol{W}^{l}\|_{2}\leq
M(l),\frac{\|\boldsymbol{W}^{l}\|_{2,1}}{\|\boldsymbol{W}^{l}\|_{2}}\leq
N(l),l=1,...,L\right\\},$ (194)

where $M(1),...,M(L)$ and $N(1),...,N(L)$ are positive integers, $M$ and $N$
are the collection of all $M(1),...,M(L)$ and $N(1),...,N(L)$, respectively.
And $l(\cdot,\cdot)$ is the mean square error (MSE) loss function,
$u(\boldsymbol{x})$ is the PDE solution, and $\boldsymbol{W}^{l}$ is the
$l$-th layer weight matrix of neural network
$u_{\boldsymbol{\theta}}(\boldsymbol{x})$. Then the class of composition of
all $L$ layers neural networks and the loss function is

$\mathcal{H}^{L}=\cup_{M(1)=1}^{\infty}\cdots\cup_{M(L)=1}^{\infty}\cup_{N(1)=1}^{\infty}\cdots\cup_{N(L)=1}^{\infty}\mathcal{H}_{M,N}^{L},$
(195)

where $M=(M(1),\cdots,M(L))$ and $N=(N(1),\cdots,N(L))$. Therefore, we
subdivide $\delta>0$ into

$\delta(M,N)=\frac{\delta}{\left[\prod_{l=1}^{L}M(l)(M(l)+1)\right]\left[\prod_{l=1}^{L}N(l)(N(l)+1)\right]},$
(196)

such that

$\sum_{M(1)=1}^{\infty}\cdots\sum_{M(L)=1}^{\infty}\sum_{N(1)=1}^{\infty}\cdots\sum_{N(L)=1}^{\infty}\delta(M,N)=\delta.$
(197)

By the result of Rademacher complexity of neural networks in Lemmas 3.1 and
3.2, for any given $\delta$ and any positive integers $M(1),...,M(L)$ and
$N(1),...,N(L)$ with probability at least $1-\delta(M,N)$ over $S$, we have

$\displaystyle R_{D\cap\partial\Omega}$
$\displaystyle(\boldsymbol{\theta})-R_{S\cap\partial\Omega}(\boldsymbol{\theta})\leq
8\mathbb{E}_{S}\text{Rad}(\mathcal{NN}_{M,N}^{L};S)+2\sqrt{\frac{\log(2/\delta(M,N))}{2n_{b}}}$
(198)
$\displaystyle\leq\frac{32}{n_{b}\sqrt{n_{b}}}+\frac{144\sqrt{d\log(2h^{2})}\log
n_{b}}{\sqrt{n_{b}}}\prod_{l=1}^{L}M(l)\Big{(}\sum_{l=1}^{L}N(l)^{2/3}\Big{)}^{3/2}+2\sqrt{\frac{\log(2/\delta(M,N))}{2n_{b}}}.$

For any parameter $\boldsymbol{\theta}$ minimizes the empirical loss, choose
the integers $M(1),...,M(L)$ and $N(1),...,N(L)$ such that

$\displaystyle M(l)-1$ $\displaystyle<\|\boldsymbol{W}^{l}\|_{2}\leq M(l),$
(199) $\displaystyle N(l)-1$
$\displaystyle<\frac{\|\boldsymbol{W}^{l}\|_{2,1}}{\|\boldsymbol{W}^{l}\|_{2}}\leq
N(l),$

and the integers are the smallest integers satisfying the above equations.
Then we have

$R_{D\cap\partial\Omega}(\boldsymbol{\theta})\leq
R_{S\cap\partial\Omega}(\boldsymbol{\theta})+\frac{32}{n_{b}\sqrt{n_{b}}}+\frac{144\sqrt{d\log(2h^{2})}\log
n_{b}}{\sqrt{n_{b}}}\prod_{l=1}^{L}M(l)\Big{(}\sum_{l=1}^{L}N(l)^{2/3}\Big{)}^{3/2}+2\sqrt{\frac{\log(2/\delta(M,N))}{2n_{b}}},$
(200)

where we note that $M(l)=\lceil\|\boldsymbol{W}^{l}\|_{2}\rceil$, and
$N(l)=\lceil\|\boldsymbol{W}^{l}\|_{2,1}/\|\boldsymbol{W}^{l}\|_{2}\rceil$, in
which $\lceil a\rceil$ of $a\in\mathbb{R}$ is the smallest integer that is
greater than or equal to $a$.

The above bound just holds with probability $1-\delta(M,N)$ for any pair
$(\boldsymbol{\theta},M,N)$ as long as $\boldsymbol{\theta}$ satisfies
$M(l)=\lceil\|\boldsymbol{W}^{l}\|_{2}\rceil$, and
$N(l)=\lceil\|\boldsymbol{W}^{l}\|_{2,1}/\|\boldsymbol{W}^{l}\|_{2}\rceil$.
Since $\sum_{M,N}\delta(M,N)=\delta$, the bound holds with probability
$1-\delta$.

We have already proved the generalization bound of the boundary loss in PINN.
That for residual loss is similar. Specifically, let

$\mathcal{G}_{M,N}^{L}=\left\\{\boldsymbol{x}\mapsto
l(f(\boldsymbol{x}),\mathcal{L}u_{\boldsymbol{\theta}}(\boldsymbol{x}))\ |\
\|\boldsymbol{W}^{l}\|_{2}\leq
M(l),\frac{\|\boldsymbol{W}^{l}\|_{2,1}}{\|\boldsymbol{W}^{l}\|_{2}}\leq
N(l),l=1,...,L\right\\}.$ (201)

Then the class of composition of the loss function and all $L$ layers
differentiated neural networks becomes

$\mathcal{G}^{L}=\cup_{M(1)=1}^{\infty}\cdots\cup_{M(L)=1}^{\infty}\cup_{N(1)=1}^{\infty}\cdots\cup_{N(L)=1}^{\infty}\mathcal{G}_{M,N}^{L}.$
(202)

Similarly, using our assumption of truncated neural network, we obtain:

$\text{Rad}(\mathcal{G}_{M}^{L};S)\leq 4\text{Rad}(\mathcal{PINN}_{M,N}^{L}).$
(203)

By the result of Rademacher complexity of neural networks in Lemma 3.2, for
any given $\delta$ and any positive integers $M(1),...,M(L)$ and
$N(1),...,N(L)$ with probability at least $1-\delta(M,N)$ over the training
dataset $S$, we have

$\displaystyle\quad
R_{D\cap\Omega}(\boldsymbol{\theta})-R_{S\cap\Omega}(\boldsymbol{\theta})$
(204) $\displaystyle\leq
8\mathbb{E}_{S}\text{Rad}(\mathcal{PINN}_{M,N}^{L};S)+2\sqrt{\frac{\log(2/\delta(M,N))}{2n_{r}}}$
$\displaystyle\leq\frac{64K+32d(L-1)K}{n_{r}\sqrt{n_{r}}}+2\sqrt{\frac{\log(2/\delta(M,N))}{2n_{r}}}+\frac{144K\sqrt{d\log(2h^{2})}\log
n_{r}}{\sqrt{n_{r}}}$
$\displaystyle\quad\prod_{l=1}^{L}M(l)\left(\sum_{l=1}^{L}N(l)^{2/3}\right)^{3/2}\left[1+\sqrt{2}L\prod_{l=1}^{L}M(l)+\sqrt{2}d(L^{2}-1)\left(\prod_{l=1}^{L}M(l)\right)^{2}\right].$

For any parameter $\boldsymbol{\theta}$ minimizes the empirical loss, choose
the integers $M(1),...,M(L)$ and $N(1),...,N(L)$ such that

$\displaystyle M(l)-1$ $\displaystyle<\|\boldsymbol{W}^{l}\|_{2}\leq M(l),$
(205) $\displaystyle N(l)-1$
$\displaystyle<\frac{\|\boldsymbol{W}^{l}\|_{2,1}}{\|\boldsymbol{W}^{l}\|_{2}}\leq
N(l),$

and the integers are the smallest integers satisfying the above equations. The
above bound just holds with probability $1-\delta(M,N)$ for any pair
$(\boldsymbol{\theta},M,N)$ as long as $\boldsymbol{\theta}$ satisfies
$M(l)=\lceil\|\boldsymbol{W}^{l}\|_{2}\rceil$, and
$N(l)=\lceil\|\boldsymbol{W}^{l}\|_{2,1}/\|\boldsymbol{W}^{l}\|_{2}\rceil$.
Since $\sum_{M,N}\delta(M,N)=\delta$, the bound holds with probability
$1-\delta$. ∎

###  D.3 Proof of Theorem 3.3

###### Proof.

We shall use Assumption 3.2 and the fact that

$a+b\leq 2^{\frac{p-1}{p}}\left(a^{p}+b^{p}\right)^{\frac{1}{p}}.$ (206)

Specifically, specify $p=2$, we have

$\displaystyle\|u_{\theta}-u\|_{L_{2}(\Omega)}$ $\displaystyle\leq
C_{1}^{-1}\left(\|\mathcal{L}u_{\theta}-\mathcal{L}u\|_{L_{2}(\Omega)}+\|u_{\theta}-u\|_{L_{2}(\partial\Omega)}\right)$
(207)
$\displaystyle\leq\sqrt{2}C_{1}^{-1}\left(\|\mathcal{L}u_{\theta}-\mathcal{L}u\|_{L_{2}(\Omega)}^{2}+\|u_{\theta}-u\|_{L_{2}(\partial\Omega)}^{2}\right)^{1/2}$
$\displaystyle\leq\sqrt{2}C_{1}^{-1}\left(R_{D\cap\Omega}(\theta)+R_{D\cap\partial\Omega}(\theta)\right)^{1/2}.$

∎

##  Appendix E Related Work

In this section, we summarize related works. We focus on related works on
PINNs, Rademacher complexity of neural networks, and the theory of PINNs.

###  E.1 Physics-Informed Neural Networks

We first introduce some background on physics-informed neural networks
(PINNs), which are the models we have considered throughout this paper.

Due to its success in approximating high-dimensional functions while
generalizing well, deep learning has been used to solve partial differential
equations (PDEs). Among them, PINNs [6] approximate the solutions of PDEs by
neural networks, and then optimize them by stochastic gradient descent for
expectation minimization to let them satisfy the physical rule described by
the PDE. Later, the extended PINNs (XPINNs) [12] which adopt domain
decomposition methods show faster convergence and better generalization
performances than vanilla PINNs, but the underlying reason for this remains
unknown. Prior to XPINN, CPINN [30] is also a domain decomposition-based PDE
solver. However, CPINN is only applicable to conservation laws and does not
allow the general spatio-temporal domain decomposition.

To the best of our knowledge, the present work provides the first proof on
generalization of PINNs and XPINNs, and the first analysis on when and how
XPINNs perform better than PINNs.

###  E.2 Rademacher Complexity of Neural Networks

In this subsection, we review the Rademacher complexity of neural networks,
which plays a key role in our generalization theory on PINNs and XPINNs.

In statistical learning theory, the Rademacher complexity measures the
richness of a class of functions on which the generalization error bound is
based. In the literature, there have been various controls and estimations on
the Rademacher complexity of the class of neural network functions.

There are various ways to bound the Rademacher complexity of the class of
neural networks, namely the norm-based control (adopted in this study), and
sharpness. For norm-based capacity control, [34] bounds Rademacher complexity
by product of Frobenius norms of parameter matrices. However, their bounds
grow exponentially as the depth increases, which contradicts the fact that
deeper networks generalize better. To eliminate the exponential dependency on
network depth, [25] uses a covering number approach to show a bound scaling as
$O({\prod_{l=1}^{L}\|\boldsymbol{W}^{l}\|_{2}(\sum_{l=1}^{L}(\frac{\|\boldsymbol{W}^{l}\|_{2,1}}{\|\boldsymbol{W}^{l}\|_{2}}^{\frac{2}{3}})^{\frac{3}{2}})}/{\sqrt{m}})$.
Although the explicit dependency on network depth $L$ disappears, the bound
still has polynomial dependency ($L^{3}$) on the depth due to the fact that
$\|\boldsymbol{W}^{l}\|_{2,1}\geq\|\boldsymbol{W}^{l}\|$. To derive size-
independent sample complexity for neural networks, [35] further proves several
useful results. Firstly, [35] improves the dependency on depth from $L^{3}$ in
[25] to $\sqrt{L}$. Secondly, [35] uses Shatten $p$-norms of matrices to
derive bounds which totally remove any dependency on the depth. [21]
empirically validates the effectiveness of these norm-based capacity controls
to explain the generalization mystery of deep learning. Another line of work
focuses on sharpness, which adopts robustness of the training error to the
perturbations in the parameters as a complexity measure for neural networks.
[36] combines sharpness measure with PAC-Bayesian approach, providing a
generalization bound scaling at
$O(\frac{\prod_{l=1}^{L}\|\boldsymbol{W}^{l}\|(\sum_{l=1}^{L}(\frac{\|\boldsymbol{W}^{l}\|_{F}}{\|\boldsymbol{W}^{l}\|}}{\sqrt{m}})$,
which is shown to be similar to the bound in [25] when weights are sparse, and
tighter than [25] when the weights are fairly dense and are of uniform
magnitude.

In this paper, we mainly consider [25] to control the Rademacher complexity of
PINNs.

###  E.3 Theory on PINNs

Due to the success of PINNs in approximating high-dimensional complicated
functions such as solutions of PDEs, theoretical evidence accounting for the
outstanding empirical performance has increasingly attracted considerable
attention.

The most related work is [13], where the authors consider Barron space for
two-layer networks for prior and posterior generalization bounds. [13] also
leverages neural tangent kernel to show global convergence of PINNs. [15]
introduces an abstract formalism and the stability properties of the
underlying PDE are leveraged to derive an estimate for the generalization
error in terms of the training error and number of training samples. By
adapting the Schauder approach and the maximum principle, [16] shows that as
number of training samples go to infinity, the minimizer converges to the
solution in $C^{0}$ and $H^{1}$. [14] uses the Barron space for two-layer
neural networks to provide a prior analysis on PINN with softplus activation,
via adopting the similarity between softplus and ReLU.

Our work extends existing results to multi-layer networks, which is more
general and realistic, and considers various kinds of capacity controls for
PINNs, namely the Barron norm and the spectral norm. Extensive experiments and
analytical examples further validate the effectiveness of our theory. Our work
is also the first to analyze when and how XPINN is better than PINN.

##  Appendix F Why Barron Space?

This subsection is devoted to clarify why we choose Barron space theory for
developing our prior bound. Overall, it has the following two advantages.

Firstly, we should choose a theory that can measure complexity of both
networks and target functions, which plays a key role in the prior
generalization bound in Theorem 3.1. In the Barron space, we are able to
measure the complexity of target functions easily via Barron norm, and we can
further show that complexities of trained neural networks are controlled by
that of the target functions. Since the success of deep learning owns to its
data-dependent training, i.e. although the class of networks has huge
complexity, gradient descent does find out a simple network, which is
reflected by the Barron space theory.

Secondly, the Barron space in high dimension neural networks resembles Sobolev
and Besov space which are indispensable building blocks for low dimension
classical theory. A proper function space is essential in analyzing PDEs. The
class of network functions define a natural function space, i.e. the Barron
space. By studying the target function of the PDE problem in the Barron space
by its norm, the generalization error of the trained network can be obtained
in terms of that norm. This reasoning resembles prior error analysis in
classical finite element method where the error is controlled by the Sobolev
norm of the target. Therefore, the Barron space adopted is appropriate for PDE
analysis.

##  Appendix G Additional Comparison

###  G.1 Comparison of Boundary Loss via Theorem 3.1

The comparison will be done via computing their respective theoretical bounds.
In particular, the generalization performance of PINN depends on the upper
bound in Theorem 3.1, which is

$R_{S\cap\partial\Omega}(\boldsymbol{\theta}^{*})+8\|u^{*}\|_{\mathcal{W}^{L}(\Omega)}\frac{C(h)\log
n_{b}}{{\sqrt{n_{b}}}}+2\sqrt{\frac{\log(2/\delta)}{n_{b}}}.$ (208)

where $n_{b}$ is the number of boundary training points.

For XPINN’s generalization, we can apply Theorem 3.1 to each of the subdomains
in XPINN. Specifically, for the $i$-th sub-net in the $i$-th subdomain, i.e.
the $\Omega_{i},i\in\left\\{1,2,...,N_{D}\right\\}$, its generalization
performance is upper bounded by

$R_{S\cap\partial\Omega_{i}}(\boldsymbol{\theta}^{*})+8\|u^{*}\|_{\mathcal{W}^{L}(\Omega_{i})}\frac{C(h)}{{\sqrt{n_{b,i}}}}+2\sqrt{\frac{\log(2/\delta)}{n_{b,i}}}.$
(209)

where $n_{b,i}$ is the number of training boundary points in the $i$-th
subdomain.

Hence, since the $i$-th subdomain has $n_{b,i}$ training boundary points and
is in charge of the prediction of $\frac{n_{b,i}}{n_{b}}$ proportion of
testing data, we weighted average their generalization errors to get the
generalization error of XPINN

$\displaystyle\sum_{i=1}^{N_{D}}\frac{n_{b,i}}{n_{b}}\left(R_{S\cap\partial\Omega_{i}}(\boldsymbol{\theta}^{*})+8\|u^{*}\|_{\mathcal{W}^{L}(\Omega_{i})}\frac{C(h)}{{\sqrt{n_{b,i}}}}+2\sqrt{\frac{\log(2/\delta)}{n_{b,i}}}\right),$
(210)

If we omit the last term and assume the empirical losses of PINN and XPINN are
similar, i.e.

$\displaystyle R_{S\cap\partial\Omega}$
$\displaystyle\approx\sum_{i=1}^{N_{D}}\frac{n_{b,i}}{n_{b}}R_{S\cap\partial\Omega_{i}},\sqrt{\frac{2\log(2/\delta)}{n_{b,i}}}$
$\displaystyle\ll\|u^{*}\|_{\mathcal{W}^{L}(\Omega)},\|u^{*}\|_{\mathcal{W}^{L}(\Omega_{i})},$
(211)

then comparing the generalization ability of PINN and XPINN reduces to the
following comparison:

$\underbrace{\|u^{*}\|_{\mathcal{W}^{L}(\Omega)}}_{\text{PINN}}\
\qquad\text{versus}\qquad\underbrace{\sum_{i=1}^{N_{D}}\sqrt{\frac{n_{b,i}}{n_{b}}}\|u^{*}\|_{\mathcal{W}^{L}(\Omega_{i})}}_{\text{XPINN}},$
(212)

where model having smaller corresponding quantity is more generalizable.

###  G.2 Comparison of Boundary Loss via Theorem 3.2

In this subsection, we compare PINN with XPINN by Theorem 3.2, where we focus
on the boundary losses of PINN and XPINN. we denote the upper bound of PINN
testing loss as $B_{\text{PINN}}$ and those of the sub-net $i$ in XPINN as
$B_{i,\text{XPINN}}$, $i\in\left\\{1,2,...,N_{D}\right\\}$ which are provided
by the right sides of Theorem 3.2, i.e. the bounds are

$B_{\text{PINN}}=R_{S\cap\partial\Omega}(\boldsymbol{\theta})+\frac{32}{n_{b}\sqrt{n_{b}}}+\frac{144\sqrt{d\log(2h^{2})}\log
n_{b}}{\sqrt{n_{b}}}\prod_{l=1}^{L}M(l)\Big{(}\sum_{l=1}^{L}N(l)^{2/3}\Big{)}^{3/2}+2\sqrt{\frac{\log(2/\delta(M,N))}{2n_{b}}}.$
(213)
$B_{i,\text{XPINN}}=R_{S\cap\partial\Omega_{i}}(\boldsymbol{\theta})+\frac{32}{n_{b,i}\sqrt{n_{b,i}}}+\frac{144\sqrt{d\log(2h^{2})}\log
n_{b,i}}{\sqrt{n_{b,i}}}\prod_{l=1}^{L}M_{i}(l)\Big{(}\sum_{l=1}^{L}N_{i}(l)^{2/3}\Big{)}^{3/2}+2\sqrt{\frac{\log(2/\delta(M_{i},N_{i}))}{2n_{b,i}}}.$
(214)

Specifically, we assume that all sub-PINNs as well as the PINN model use
neural networks with depth $L$ and width $h$. In the bound of PINN, $n_{r}$ is
the total number of residual training samples.
$M(l)=\lceil\|\boldsymbol{W}^{l}\|_{2}\rceil$, and
$N(l)=\lceil\|\boldsymbol{W}^{l}\|_{2,1}/\|\boldsymbol{W}^{l}\|_{2}\rceil$,
where $\boldsymbol{W}^{l}$ is the $l$-th layer parameter matrix in the PINN
model. Moreover, in the bound of XPINN, $n_{r,i}$ is the number of residual
training samples in subdomain $i$.
$M_{i}(l)=\lceil\|\boldsymbol{W}^{l}_{i}\|_{2}\rceil$, and
$N_{i}(l)=\lceil\|\boldsymbol{W}^{l}_{i}\|_{2,1}/\|\boldsymbol{W}^{l}_{i}\|_{2}\rceil$,
where $\boldsymbol{W}^{l}_{i}$ is the $l$-th layer parameter matrix of the
$i$-th subnet in the XPINN model. Because the $i$-th sub-net in XPINN is in
charge of the prediction of $\frac{n_{b,i}}{n_{b}}$ proportion of testing
data, we weight-averaged their bounds to get that of XPINN, i.e.,
$B_{\text{XPINN}}=\sum_{i=1}^{N_{D}}({n_{b,i}}/{n_{b}})B_{i,\text{XPINN}}$
where $B_{\text{XPINN}}$ is the bound for XPINN. Thus, we only need to compare
$B_{\text{PINN}}$ with $B_{\text{XPINN}}$, where the model having smaller
corresponding quantity is more generalizable.

## References

  * Jagtap et al. [2020a] Ameya D Jagtap, Kenji Kawaguchi, and George Em Karniadakis.  Adaptive activation functions accelerate convergence in deep and physics-informed neural networks.  _J. Comput. Phys._ , 404:109136, 2020a.
  * Kawaguchi [2016] Kenji Kawaguchi.  Deep learning without poor local minima.  In _Advances in neural information processing systems (NeurIPS)_ , pages 586–594, 2016.
  * Xu et al. [2021] Keyulu Xu, Mozhi Zhang, Stefanie Jegelka, and Kenji Kawaguchi.  Optimization of graph neural networks: Implicit acceleration by skip connections and more depth.  In _International Conference on Machine Learning (ICML)_ , 2021.
  * Kawaguchi [2021] Kenji Kawaguchi.  On the theory of implicit deep learning: Global convergence with implicit layers.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Kawaguchi et al. [2018] Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio.  Generalization in deep learning.  _In Mathematics of Deep Learning, Cambridge University Press, to appear. Prepint available as: MIT-CSAIL-TR-2018-014, Massachusetts Institute of Technology_ , 2018.
  * Raissi et al. [2019] Maziar Raissi, Paris Perdikaris, and George E Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  _J. Comput. Phys._ , 378:686–707, 2019.
  * Cai et al. [2021a] Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, and George Em Karniadakis.  Physics-Informed Neural Networks for Heat Transfer Problems.  _J. Heat Transfer_ , 143(6), 04 2021a.  ISSN 0022-1481.  doi: 10.1115/1.4050542.  URL https://doi.org/10.1115/1.4050542.  060801\.
  * Yin et al. [2021] Minglang Yin, Xiaoning Zheng, Jay D Humphrey, and George Em Karniadakis.  Non-invasive inference of thrombus material properties with physics-informed neural networks.  _Comput. Methods Appl. Mech. Engrg._ , 375:113603, 2021\.
  * Chen et al. [2020] Yuyao Chen, Lu Lu, George Em Karniadakis, and Luca Dal Negro.  Physics-informed neural networks for inverse problems in nano-optics and metamaterials.  _Optics express_ , 28(8):11618–11633, 2020.
  * Cai et al. [2021b] Shengze Cai, Zhicheng Wang, Frederik Fuest, Young Jin Jeon, Callum Gray, and George Em Karniadakis.  Flow over an espresso cup: inferring 3-d velocity and pressure fields from tomographic background oriented schlieren via physics-informed neural networks.  _J. Fluid Mech._ , 915, 2021b.
  * Jin et al. [2021] Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis.  NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations.  _J. Comput. Phys._ , 426:109951, 2021.
  * Jagtap and Karniadakis [2020] Ameya D Jagtap and George Em Karniadakis.  Extended physics-informed neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations.  _Commun. Comput. Phys._ , 28(5):2002–2041, 2020\.
  * Luo and Yang [2020] Tao Luo and H. Yang.  Two-layer neural networks for partial differential equations: Optimization and generalization theory.  _ArXiv_ , abs/2006.15733, 2020.
  * Lu et al. [2021] Jianfeng Lu, Yulong Lu, and Min Wang.  A priori generalization analysis of the deep Ritz method for solving high dimensional elliptic equations.  In _Proceedings of Thirty Fourth Conference on Learning Theory (COLT)_ , 2021.
  * Mishra and Molinaro [2020] Siddhartha Mishra and Roberto Molinaro.  Estimates on the generalization error of physics informed neural networks (PINNs) for approximating pdes.  _arXiv:2006.16144_ , 2020.
  * Shin et al. [2020a] Yeonjong Shin, Jérôme Darbon, and George Em Karniadakis.  On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type pdes.  _Commun. Comput. Phys._ , 28(5):2042–2074, 2020a.  ISSN 1991-7120.
  * De Ryck et al. [2022] Tim De Ryck, Ameya D Jagtap, and Siddhartha Mishra.  Error estimates for physics informed neural networks approximating the navier-stokes equations.  _arXiv preprint arXiv:2203.09346_ , 2022.
  * Weinan and Wojtowytsch [2020] E. Weinan and Stephan Wojtowytsch.  On the banach spaces associated with multi-layer relu networks: Function representation, approximation theory and gradient descent dynamics.  _ArXiv_ , abs/2007.15623, 2020.
  * Weinan et al. [2019] E Weinan, Chao Ma, and Lei Wu.  A priori estimates of the generalization error for two-layer neural networks.  _Commun. Math. Sci._ , 17, 2019.
  * Poggio et al. [2017] Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, and Qianli Liao.  Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review.  _Int. J. Autom. Comput._ , 14(5):503–519, 2017\.
  * Neyshabur et al. [2017a] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro.  Exploring generalization in deep learning.  In _Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)_ , 2017a.
  * Cardaliaguet and Euvrard [1992] Pierre Cardaliaguet and Guillaume Euvrard.  Approximation of a function and its derivative with a neural network.  _Neural networks_ , 5(2):207–220, 1992.
  * Attali and Pagés [1997] Jean-Gabriel Attali and Gilles Pagés.  Approximations of functions by a multilayer perceptron: a new approach.  _Neural networks_ , 10(6):1069–1081, 1997.
  * Bartlett and Mendelson [2002] Peter L Bartlett and Shahar Mendelson.  Rademacher and gaussian complexities: Risk bounds and structural results.  volume 3, pages 463–482, 2002.
  * Bartlett et al. [2017] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky.  Spectrally-normalized margin bounds for neural networks.  In _Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)_ , volume 30, 2017.
  * Bochev and Gunzburger [2016] P Bochev and M Gunzburger.  Least-squares methods for hyperbolic problems.  In _Handbook of Numerical Analysis_ , volume 17, pages 289–317. Elsevier, 2016.
  * Bramble and Schatz [1970] James H Bramble and Alfred H Schatz.  Rayleigh-ritz-galerkin methods for dirichlet’s problem using subspaces without boundary conditions.  _Comm. Pure Appl. Math._ , 23(4):653–675, 1970\.
  * Ryck and Mishra [2021] Tim De Ryck and Siddhartha Mishra.  Error analysis for physics informed neural networks (pinns) approximating kolmogorov pdes.  _ArXiv_ , abs/2106.14473, 2021.
  * Shin et al. [2020b] Yeonjong Shin, Zhongqiang Zhang, and George Em Karniadakis.  Error estimates of residual minimization using neural networks for linear pdes.  _arXiv preprint arXiv:2010.08019_ , 2020b.
  * Jagtap et al. [2020b] Ameya D Jagtap, Ehsan Kharazmi, and George Em Karniadakis.  Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems.  _Comput. Methods Appl. Mech. Engrg_ , 365:113028, 2020b.
  * Kingma and Ba [2015] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2015.
  * Barron [1993] A.R. Barron.  Universal approximation bounds for superpositions of a sigmoidal function.  _IEEE Trans. Inform. Theory_ , 39(3):930–945, 1993.  doi: 10.1109/18.256500.
  * Pisier [1981] Gilles Pisier.  Remarques sur un résultat non publié de b. maurey.  _Séminaire Analyse fonctionnelle (dit_ , pages 1–12, 1981.
  * Neyshabur et al. [2015] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro.  Norm-based capacity control in neural networks.  In _Conference on Learning Theory_ , pages 1376–1401. PMLR, 2015\.
  * Golowich et al. [2018] Noah Golowich, A. Rakhlin, and O. Shamir.  Size-independent sample complexity of neural networks.  In _COLT_ , 2018.
  * Neyshabur et al. [2017b] Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro.  A pac-bayesian approach to spectrally-normalized margin bounds for neural networks.  _arXiv preprint arXiv:1707.09564_ , 2017b.