# Uniform Convergence of Deep Neural Networks with Lipschitz Continuous
Activation Functions and Variable Widths

Yuesheng Xu and Haizhang Zhang Department of Mathematics and Statistics, Old
Dominion University, Norfolk, VA 23529, USA. E-mail address<EMAIL_ADDRESS>Supported in part by US National Science Foundation under grants DMS-1912958
and DMS-2208386, and by the US National Institutes of Health under grant
R21CA263876. Corresponding author. School of Mathematics (Zhuhai), Sun Yat-sen
University, Zhuhai, P.R. China. E-mail address<EMAIL_ADDRESS>Supported in part by National Natural Science Foundation of China under grants
11971490 and 12126610.

###### Abstract

We consider deep neural networks with a Lipschitz continuous activation
function and with weight matrices of variable widths. We establish a uniform
convergence analysis framework in which sufficient conditions on weight
matrices and bias vectors together with the Lipschitz constant are provided to
ensure uniform convergence of the deep neural networks to a meaningful
function as the number of their layers tends to infinity. In the framework,
special results on uniform convergence of deep neural networks with a fixed
width, bounded widths and unbounded widths are presented. In particular, as
convolutional neural networks are special deep neural networks with weight
matrices of increasing widths, we put forward conditions on the mask sequence
which lead to uniform convergence of resulting convolutional neural networks.
The Lipschitz continuity assumption on the activation functions allows us to
include in our theory most of commonly used activation functions in
applications.

Keywords: uniform convergence, deep neural network, convolutional neural
network, Lipschitz continuous activation function, variable widths

##  1 Introduction

The last decade has witnessed the immense success of deep learning [9, 20]. As
we all know, a major part of such successes are due to the powerful
expressiveness of the deep neural networks in representing a function. In
other words, the deep neural network is the engine of deep learning. In order
to explain why deep learning works so well, there is need to understand deep
neural networks from rigorous mathematical viewpoints. As deep neural networks
have advanced in machine learning, they have gained much attention in the
applied mathematics community [5, 6, 7, 22, 23, 27, 28, 29, 33, 34, 39] and
have gone beyond machine learning. Deep neural networks, considered as a
function class to represent or approximate a function, have exhibited
superiority in many aspects to classical approximation functions such as
polynomials, trigonometric polynomials, splines, finite elements, wavelets,
and kernel functions in approximation and numerical analysis. Unlike the
classical function approximation in which a function is approximated by a
linear combination of basis functions, deep neural networks approximate a
given function by compositions of functions with a networks. Specifically, a
neural network is a vector-valued function defined through consecutive
function compositions of a given activation function with parameters
consisting of weight matrices and bias vectors. A deep neural network of a
given function may be determined by finding the parameters that minimize the
difference between it and the given function. Mathematically, one would expect
that as the number of layers of the deep neural network increases, the
difference diminishes and eventually goes to zero as the number tends to
infinity. In a special case when the activation function is the rectified
linear unit (ReLU), this question was investigated in a number of studies [5,
29, 39].

A closely related mathematical question, even more basic, is when a deep
neural network converges to a meaningful function as its layer number tends to
infinity. This question was studied recently for the ReLU activation function
with a fixed width in [35], for the ReLU activation function with a
convolution network in [36] and for contractive activation functions with a
fixed width in [16]. Although the contractivity hypothesis covers interesting
activation functions such as sigmoid, there are many activation functions
frequently used in applications that are not contractive, for example, ReLU,
parametric rectified linear unit (PReLU), exponential linear unit (ELU) and
scaled exponential linear unit (SELU), to name a few. Therefore, there is a
need to understand uniform convergence of deep neural networks defined by a
non-contractive activation function. This paper will study uniform convergence
of deep neural networks of Lipschitz continuous activation functions with
weight matrices of variable widths (including bounded and unbounded widths).

Main difficulty in analyzing convergence of deep neural networks is a result
of the nonlinearity of the activation function. This was overcome in [35] for
the ReLU activation function by re-expressing the functional application of
the activation function in terms of matrix-vector multiplication with
activation matrices. Pointwise convergence of neural networks with the ReLU
activation function was then analyzed by using the matrix-vector
multiplication. When a general Lipschitz continuous activation function is
chosen, one can take the advantage of its Lipschitz continuity to overcome the
difficulty caused by its nonlinearity. We propose a condition that intertwines
the Lipschitz constant of the activation function with the weight matrices to
ensure the uniform convergence of the resulting neural networks. The main
contribution of this work lies in laying out a general framework for uniform
convergence analysis of deep neural networks with general activation functions
and pooling operators, both of which are Lipschitz continuous.

Understanding conditions that ensure convergence of deep neural networks is
not only theoretically interesting, but also practically advantageous in
guiding their training in applications. As we know, a deep learning model
usually possesses a large number of hidden layers and a massive amount of
parameters. For example, Residual Networks (ResNets) can reach over 1,000
layers [13]. A ResNet with only 50 layers has over 23 million parameters, and
the overwhelming ChatGPT (GPT-3) model has approximately 175 billion
parameters. In applications, the parameters of a DNN are determined via a
training process by minimizing a loss function on given training data. It is
desirable to figure out whether or not a DNN system with so many parameters
can eventually converge to a meaningful function in a rigorous mathematical
sense. Mathematical conditions on the parameter that ensure convergence of the
DNNs as the number of their layers increases will be beneficial to the
interpretability of the DNNs. Such conditions can also be helpful in training
a DNN. For instance, if such conditions are available, people can generate
initial parameters of a DNN that satisfy or nearly satisfy the conditions.
With such an initial deployment of parameters, the DNN will be inclined to
converge more quickly. Finally, uniform convergence rates of DNNs will be
applicable to mathematical analysis on the generalization ability of DNN
models [15].

We organize this paper in seven sections. In section 2, we describe the
setting of deep neural networks. Section 3 is devoted to developing a general
framework for uniform convergence analysis of deep neural networks with
pooling. A key ingredient that ensures uniform convergence of the deep neural
networks is a condition that intertwines the Lipschitz constants of the
activation function and the pooling operator with the norm of the weight
matrices. In sections 4 and 5, we present uniform convergence results for deep
neural networks with weight matrices of fixed widths and bounded widths,
respectively. While in section 6, we consider deep neural networks with weight
matrices of unbounded widths. Finally, in section 7 we present uniform
convergence theorems for convolutional neural networks.

##  2 Deep Neural Networks

In this section, we recall the definition of deep neural networks.

We now describe deep neural networks with width $m_{n}\in\mathbb{N}$ at the
$n$-th level, for $n\in\mathbb{N}$, from input space $\mathbb{R}^{s}$ to the
output space $\mathbb{R}^{t}$, where $s,t\in\mathbb{N}$. For a given
univariate activation function $\sigma:\mathbb{R}\to\mathbb{R}$, we define the
vector-valued function

$\sigma({\bf{x}}):=[\sigma(x_{1}),\sigma(x_{2}),\dots,\sigma(x_{s})]^{\top},\
\ \mbox{for}\ \ {\bf{x}}:=[x_{1},x_{2},\dots,x_{s}]^{\top}\in\mathbb{R}^{s}.$
(2.1)

As in [35], for $n$ vector-valued functions $f_{j}$,
$j\in\mathbb{N}_{n}:=\\{1,2,\dots,n\\}$, such that the range of $f_{i}$ is
contained in the domain of $f_{i+1}$, $i\in\mathbb{N}_{n-1}$, the consecutive
composition of $f_{i}$, $i\in\mathbb{N}_{n}$, is denoted by

$\bigodot_{i=1}^{n}f_{i}:=f_{n}\circ f_{n-1}\circ\cdots\circ f_{2}\circ
f_{1},$ (2.2)

whose domain is that of $f_{1}$. For each $n\in\mathbb{N}$, by ${\bf{W}}_{n}$
we denote the weight matrix, and by ${\bf{b}}_{n}$ the bias vector, of the
$n$-th hidden layer. Clearly, ${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times
m_{n-1}}$, for $n\in\mathbb{N}$ with $m_{0}:=s$, and
${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$ for $n\in\mathbb{N}$. The deep neural
network is a function defined by

${\cal{N}}_{n}({\bf{x}}):=\left(\bigodot_{i=1}^{n}\sigma({\bf{W}}_{i}\cdot+{\bf{b}}_{i})\right)({\bf{x}}),\
\ {\bf{x}}\in\mathbb{R}^{s}.$ (2.3)

Given the weight matrix ${\bf{W}}_{o}\in\mathbb{R}^{t\times m_{n}}$ and bias
vector ${\bf{b}}_{o}\in\mathbb{R}^{t}$, of the output layer, the output of the
DNN is then given by

${\bf{y}}={\bf{W}}_{o}{\cal{N}}_{n}({\bf{x}})+{\bf{b}}_{o},\ \
{\bf{x}}\in\mathbb{R}^{s}.$

We are concerned with convergence of the functions ${\cal{N}}_{n}$ determined
by the deep neural network as $n$ increases to infinity. Because the output
layer is a linear function of ${\cal{N}}_{n}$ and thus, it does not affect the
convergence. Hence, we will consider convergence of the function sequence
${\cal{N}}_{n}$, as $n\to\infty$. For a fixed $n\in\mathbb{N}$,
${\cal{N}}_{n}$ is a vector-valued function.

It is convenient to express the neural network ${\cal{N}}_{n}$ in recursion in
$n$. From (2.3) and the definition (2.1), we have the recursion

${\cal{N}}_{1}({\bf{x}}):=\sigma({\bf{W}}_{1}{\bf{x}}+{\bf{b}}_{1})$ (2.4)

and

${\cal{N}}_{n+1}({\bf{x}})=\sigma({\bf{W}}_{n+1}{\cal{N}}_{n}({\bf{x}})+{\bf{b}}_{n+1}),\
\ {\bf{x}}\in\mathbb{R}^{s},\ \ \mbox{for all}\ \ n\in\mathbb{N}.$ (2.5)

The goal of this paper is to understand what conditions are required for the
weight matrices ${\bf{W}}_{n}$ and the bias vectors ${\bf{b}}_{n}$ to ensure
convergence of the deep neural network for a general activation function. For
this purpose, we suppose that $\sigma$ is Lipschitz continuous with the
Lipschitz constant $L$. That is,

$|\sigma(x)-\sigma(y)|\leq L|x-y|,\ \ \mbox{for all}\ \ x,y\in\mathbb{R}.$
(2.6)

When $L<1$, $\sigma$ is contractive and when $L=1$, $\sigma$ is non-expansive.
Convergence of deep neural networks with a non-expansive activation function
and uniform convergence of deep neural networks with a contractive activation
function were established in a recent paper [16]. We are particularly
interested in understanding conditions that ensure uniform convergence of deep
neural networks with a Lipschitz continuous activation function with $L>1$.
Many commonly used activation functions are Lipschitz continuous. Among them,
some have their Lipschitz constants $L>1$. For example, the parametric
rectified linear unit (PReLU) [12]

$\sigma(x):=\begin{cases}\alpha x,\ \ &x<0,\\\ x,\ \ &x\geq 0\end{cases}$

is Lipschitz continuous with the Lipschitz constant $L:=\max\\{\alpha,1\\}$.
When $\alpha>1$, PReLU is expanding. The exponential linear unit (ELU) [2] is
defined by

$\sigma(x):=\begin{cases}\alpha(e^{x}-1),\ \ &x\leq 0,\\\ x,\ \
&x>0.\end{cases}$

It can be verified that ELU is Lipschitz continuous with the Lipschitz
constant $L:=\max\\{\alpha,1\\}$. When $\alpha>1$, ELU is expanding. Moreover,
the scaled exponential linear unit (SELU) was proposed in [18] to construct
self-normalized neural networks and it has the form

$\sigma(x):=\lambda\begin{cases}\alpha(e^{x}-1),\ \ &x<0,\\\ x,\ \ &x\geq
0,\end{cases}$

with parameter $\lambda=1.0507$ and $\alpha=1.67326$. Clearly, SELU is
Lipschitz continuous with the Lipschitz constant $L:=\lambda\alpha$, which is
greater than 1. Hence, SELU is again expanding.

Convergence of deep neural networks is measured by a vector norm. We say that
a vector norm $\|\cdot\|$ on $\mathbb{R}^{l}$ satisfies the extension
invariant condition if

$\|\tilde{{\bf{a}}}\|=\|{\bf{a}}\|,\ \ \mbox{for}\ \
\tilde{{\bf{a}}}:=\left[\begin{array}[]{c}{\bf{a}}\\\ {\bf
0}\end{array}\right],\ \mbox{with}\ {\bf{a}}\in\mathbb{R}^{\nu},\ {\bf
0}\in\mathbb{R}^{l-\nu}\ \mbox{for any}\ \nu\in\mathbb{N}_{l}$ (2.7)

and satisfies the monotonicity condition if

$\|{\bf{a}}\|\leq\|{\bf{b}}\|\mbox{ whenever }|a_{i}|\leq|b_{i}|,\
i\in\mathbb{N}_{l},\ \mbox{for}\
{\bf{a}}:=[a_{1},\dots,a_{l}]^{T},{\bf{b}}:=[b_{1},\dots,b_{l}]^{T}\in\mathbb{R}^{l}.$
(2.8)

For each ${\bf{a}}\in\mathbb{R}^{l}$, we recall the $\ell_{p}$-norms as

$\|{\bf{a}}\|_{p}:=\left(\sum_{j=1}^{l}|a_{j}|^{p}\right)^{\frac{1}{p}},\ \
\mbox{for}\ \ 1\leq p<+\infty$

and

$\|{\bf{a}}\|_{\infty}:=\max\\{|a_{j}|:j\in\mathbb{N}_{l}\\},\ \ \mbox{for}\ \
p=+\infty.$

It can be confirmed that the $\ell_{p}$-norms for all $1\leq p\leq+\infty$
satisfy both of these conditions. We also need a matrix norm $\|\cdot\|$ on
$\mathbb{R}^{k\times l}$, which we require to be induced by a vector norm,
that is,

$\|{\bf{A}}\|=\sup_{{\bf{x}}\in\mathbb{R}^{l},x\neq
0}\frac{\|{\bf{A}}{\bf{x}}\|}{\|{\bf{x}}\|},\ \ \mbox{for}\ \
{\bf{A}}\in\mathbb{R}^{k\times l}.$

Clearly, this matrix norm has the property that

$\|{\bf{A}}{\bf{B}}\|\leq\|{\bf{A}}\|\|{\bf{B}}\|,\ \mbox{ for all}\
{\bf{A}},{\bf{B}}\ \mbox{with}\ {\bf{A}}{\bf{B}}\ \mbox{well-defined}.$ (2.9)

It follows from (2.1) and (2.8) that

$\|\sigma({\bf{x}})-\sigma({\bf{y}})\|\leq L\|{\bf{x}}-{\bf{y}}\|,\ \
\mbox{for all}\ \ {\bf{x}},{\bf{y}}\in\mathbb{R}^{s}.$ (2.10)

When $\sigma$ is Lipschitz continuous with the Lipschitz constant $L$, the
neural network ${\cal{N}}_{n}$ is also Lipschitz continuous with the Lipschitz
constant $L^{n}\prod_{j=1}^{n}\|{\bf{W}}_{j}\|$. In fact, from the recursion
(2.5) and the Lipschitz continuity (2.1), we observe that

$\|{\cal{N}}_{n}({\bf{x}})-{\cal{N}}_{n}({\bf{y}})\|\leq\left(L^{n}\prod_{j=1}^{n}\|{\bf{W}}_{j}\|\right)\|{\bf{x}}-{\bf{y}}\|,\
\ \mbox{for all}\ \ {\bf{x}},{\bf{y}}\in\mathbb{R}^{s}.$

When the weight matrices ${\bf{W}}_{n}$ have variable widths, the resulting
neural networks ${\cal{N}}_{n}({\bf{x}})$ have variable dimensions.
Considering convergence of such a sequence of neural networks requires special
care. We first assume that the weight matrices ${\bf{W}}_{n}$ have a fixed
width $l\in\mathbb{N}$. In this case, we say that the deep neural network
${\cal{N}}_{n}$ defined by (2.3) via ${\bf{W}}_{n}$, ${\bf{b}}_{n}$,
$n\in\mathbb{N}$, and an activation function $\sigma$ converges uniformly in a
bounded set $\mathbb{D}\subseteq\mathbb{R}^{s}$ to a limit function
${\cal{N}}:\mathbb{D}\to\mathbb{R}^{l}$ if for any $\epsilon>0$, there exists
$N\in\mathbb{N}$ such that

$\sup_{{\bf{x}}\in\mathbb{D}}\|{\cal{N}}_{n}({\bf{x}})-{\cal{N}}({\bf{x}})\|<\epsilon,\
\ \mbox{whenever}\ \ n\geq N.$

By $C_{l}(\mathbb{D})$, we denote the space of continuous vector-valued
functions ${\cal{N}}:\mathbb{D}\to\mathbb{R}^{l}$ defined on
$\mathbb{D}\subseteq\mathbb{R}^{s}$ with

$\sup\\{\|{\cal{N}}({\bf{x}})\|:{\bf{x}}\in\mathbb{D}\\}<+\infty,$

where $\|\cdot\|$ is a vector norm on $\mathbb{R}^{l}$. For any vector norm,
$C_{l}(\mathbb{D})$ is complete.

We now consider the case when the matrix widths $m_{n}$, $n\in\mathbb{N}$, of
deep neural networks are variable but bounded. Specifically, we let
$l:=\max\\{m_{n}:n\in\mathbb{N}\\}$. Then, we have that $l<+\infty$, $1\leq
m_{n}\leq l$ for all $n\in\mathbb{N}$, and $m_{n_{0}}=l$ for some
$n_{0}\in\mathbb{N}$. We define the deep neural network
${\cal{N}}_{n}({\bf{x}})$ by (2.3). Clearly, ${\cal{N}}_{n}({\bf{x}})$ is a
vector-valued function in $\mathbb{R}^{m_{n}}$. The size of
${\cal{N}}_{n}({\bf{x}})$ varies according to $n$. Considering convergence of
such a sequence requires us to extend the weight matrices
${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times m_{n-1}}$ and bias vectors
${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$ to
$\tilde{\bf{W}}_{n}\in\mathbb{R}^{l\times l}$ and
$\tilde{\bf{b}}_{n}\in\mathbb{R}^{l}$, respectively.

When the widths $m_{n}$, $n\in\mathbb{N}$, of neural networks are unbounded,
there exists a subsequence $m_{n_{i}}$, $i\in\mathbb{N}$, with
$\lim_{i\to+\infty}m_{n_{i}}=+\infty$. Due to the unboundedness of the widths,
we will extend all vectors and matrices to elements in sequence spaces
$\ell_{p}(\mathbb{N})$ and $\ell_{p}(\mathbb{N}^{2})$, respectively, and
consider uniform convergence of the neural networks that result from the
extension in the sequence spaces. We now review the notion of sequence spaces.
For $1\leq p\leq+\infty$, by $\ell_{p}(\mathbb{N})$ we denote the space of
sequences ${\bf{x}}$ with $\|{\bf{x}}\|_{\ell_{p}(\mathbb{N})}<+\infty$, where

$\|{\bf{x}}\|_{\ell_{p}(\mathbb{N})}:=\left(\sum_{j=1}^{\infty}|x_{j}|^{p}\right)^{\frac{1}{p}},\
\ \mbox{for}\ \ 1\leq p<+\infty$

and

$\|{\bf{x}}\|_{\ell_{\infty}(\mathbb{N})}:=\sup\\{|x_{j}|:j\in\mathbb{N}\\}.$

Throughout this paper, we reserve $\|\cdot\|_{p}$ as the vector norm in
$\mathbb{R}^{\mu}$ for a $\mu<\infty$. We also need the notion of spaces of
semi-infinite matrices. For $1\leq p\leq+\infty$, we let
$\ell_{p}(\mathbb{N}^{2}):=\ell_{p}(\mathbb{N}\times\mathbb{N})$ denote the
spaces of semi-infinite matrices
${\bf{W}}:\ell_{p}(\mathbb{N})\to\ell_{p}(\mathbb{N})$, viewed as operators,
with $\|{\bf{W}}\|_{\ell_{p}(\mathbb{N}^{2})}<+\infty$, where
$\|\cdot\|_{\ell_{p}(\mathbb{N}^{2})}$ are operator norms induced from the
$\ell_{p}(\mathbb{N})$ norms. We extend matrices
${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times m_{n-1}}$ to
$\tilde{{\bf{W}}}_{n}\in\ell_{p}(\mathbb{N}\times\mathbb{N}_{s})$ for $n=1$
and $\tilde{{\bf{W}}}_{n}\in\ell_{p}(\mathbb{N}^{2})$ for $n>1$, and extend
vectors ${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$ to
$\tilde{\bf{b}}_{n}\in\ell_{p}(\mathbb{N})$. For $1\leq p\leq+\infty$, we also
define $C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$ to be the space of continuous
sequence-valued functions ${\cal{N}}:\mathbb{D}\to\ell_{p}(\mathbb{N})$
defined on $\mathbb{D}\subseteq\mathbb{R}^{s}$ with

$\sup\\{\|{\cal{N}}({\bf{x}})\|_{\ell_{p}(\mathbb{N})}:{\bf{x}}\in\mathbb{D}\\}<+\infty.$

When the matrix widths $m_{n}$, $n\in\mathbb{N}$, of neural networks are
variable, either bounded or unbounded, we define
$\tilde{\cal{N}}_{n}({\bf{x}})$ by (2.3) with ${\bf{W}}_{n}$ and
${\bf{b}}_{n}$ being replaced by $\tilde{\bf{W}}_{n}$ and
$\tilde{\bf{b}}_{n}$, respectively. The neural networks
$\tilde{\cal{N}}_{n}({\bf{x}})$ are extensions of ${\cal{N}}_{n}({\bf{x}})$
and satisfy the same recursion as (2.4) and (2.5) with $\tilde{\bf{W}}_{n}$
and $\tilde{\bf{b}}_{n}$. We then consider uniform convergence of the sequence
$\tilde{\cal{N}}_{n}({\bf{x}})$ in either $C_{l}(\mathbb{D})$ or
$C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$. We say the sequence
${\cal{N}}_{n}({\bf{x}})$ converges uniformly in $C_{l}(\mathbb{D})$ (resp.
$C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$) if $\tilde{\cal{N}}_{n}({\bf{x}})$
converges uniformly to ${\cal{N}}\in C_{l}(\mathbb{D})$ (resp. ${\cal{N}}\in
C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$) when the matrix widths are bounded
(resp. unbounded). Note that when the widths are not fixed, the uniform
convergence depends on how the weight matrices and bias vectors are extended.
Unambiguous extensions of the weight matrices will be specified in later
sections.

##  3 Uniform Convergence Analysis Framework

This section is devoted to establishing an analysis framework for uniform
convergence of deep neural networks with pooling in a general setting. We will
apply this framework to various scenarios in later sections to produce
convergence results for deep neural networks of various types.

We now describe the setting. Let $l,\mu\in\mathbb{N}$. By $\hat{\bf{W}}_{n}$,
$n\in\mathbb{N}$, we denote a sequence of weight matrices in
$\mathbb{R}^{(l+\mu)\times l}$ (resp. $\ell_{p}(\mathbb{N}^{2})$) and by
$\hat{\bf{b}}_{n}$, $n\in\mathbb{N}$, a sequence of bias vectors in
$\mathbb{R}^{l}$ (resp. $\ell_{p}(\mathbb{N})$). Let
${\cal{P}}_{\mu}:\mathbb{R}^{l+\mu}\to\mathbb{R}^{l}$ (resp.
$\ell_{p}(\mathbb{N})\to\ell_{p}(\mathbb{N})$) denote a pooling operator
(linear or nonlinear). We assume that the pooling operator ${\cal{P}}_{\mu}$
is Lipschitz continuous with the Lipschitz constant $P$. When
${\cal{P}}_{\mu}$ is linear, its Lipschitz constant is identical to its norm.
Specific pooling operators will be discussed later. Suppose that
${\cal{N}}_{n}\in\mathbb{R}^{l}$, $n\in\mathbb{N}$, satisfy the recursion

$\hat{\cal{N}}_{1}({\bf{x}}):=\sigma({\cal{P}}_{\mu}(\hat{\bf{W}}_{1}{\bf{x}})+\hat{\bf{b}}_{1})$
(3.1)

and

$\hat{\cal{N}}_{n+1}({\bf{x}})=\sigma({\cal{P}}_{\mu}(\hat{\bf{W}}_{n+1}\hat{\cal{N}}_{n}({\bf{x}}))+\hat{\bf{b}}_{n+1}),\
\ {\bf{x}}\in\mathbb{R}^{s},\ \ \mbox{for all}\ \ n\in\mathbb{N},$ (3.2)

associated with the weight matrices $\hat{\bf{W}}_{n}$, the bias vectors
$\hat{\bf{b}}_{n}$, and an activation function $\sigma$. If ${\cal{P}}_{\mu}$
is the identity operator, then $\mu=0$ and there is no pooling. When
$\hat{\bf{W}}_{n}:\mathbb{R}^{l+\mu}\to\mathbb{R}^{l}$,
$\hat{\bf{b}}_{n}\in\mathbb{R}^{l}$, $n>1$, we have that
$\hat{\cal{N}}_{n}({\bf{x}})\in\mathbb{R}^{l}$, and when
$\hat{\bf{W}}_{n}:\ell_{p}(\mathbb{N})\to\ell_{p}(\mathbb{N})$,
$\hat{\bf{b}}_{n}\in\ell_{p}(\mathbb{N})$, $n>1$, we have that
$\hat{\cal{N}}_{n}({\bf{x}})\in\ell_{p}(\mathbb{N})$. The goal of this section
is to establish results on uniform convergence of
$\hat{\cal{N}}_{n}({\bf{x}})$, $n\in\mathbb{N}$.

Throughout this section, we assume that the norms for appropriate
vectors/sequences/matrices involved are well-defined and the neural networks
$\hat{\cal{N}}_{n}({\bf{x}})$, $n\in\mathbb{N}$, satisfy the recursion (3.1)
and (3.2) without further mentioning. We adopt the convention for the notation
of the product of numbers

$\prod_{j=\mu}^{\nu}a_{j}:=1,\ \ \mbox{if}\ \ \mu>\nu.$

The next lemma prepares us for convergence analysis of neural networks. To
this end, for $m,k\in\mathbb{N}$, we define
$e_{m,k}:=\|\hat{\bf{b}}_{m+k}-\hat{\bf{b}}_{k}\|$ and
$E_{m,k}:=\|\hat{\bf{W}}_{m+k}-\hat{\bf{W}}_{k}\|$.

###### Lemma 3.1

If the activation function $\sigma$ and the pooling operator ${\cal{P}}_{\mu}$
are Lipschitz continuous with the Lipschitz constants $L$ and $P$,
respectively, then for all $n,m\in\mathbb{N}$,

$\displaystyle\|\hat{\cal{N}}_{n+m}({\bf{x}})-\hat{\cal{N}}_{n}({\bf{x}})\|\leq$
$\displaystyle
L\sum_{i=0}^{n-1}\Lambda_{n+m}^{i-1}e_{m,n-i}+LP\sum_{i=0}^{n-2}\Lambda_{n+m}^{i-1}\|\hat{\cal{N}}_{n-1-i}({\bf{x}})\|E_{m,n-i}$
$\displaystyle+LP\Lambda_{n+m}^{n-2}\|\hat{\bf{W}}_{m+1}\hat{\cal{N}}_{m}({\bf{x}})-\hat{\bf{W}}_{1}{\bf{x}}\|,\
\ {\bf{x}}\in\mathbb{R}^{s},$ (3.3)

where

$\Lambda_{n}^{i}:=\prod_{j=0}^{i}LP\|\hat{\bf{W}}_{n-j}\|,\ \ n\in\mathbb{N},\
i<n.$ (3.4)

Proof: We prove inequality (3.1) by induction on $n$. We first consider the
case $n=1$. Since both the activation function $\sigma$ and the pooling
operator are Lipschitz continuous with the Lipschitz constants $L$ and $P$,
respectively, by recursion (3.1) and (3.2), and the Lipschitz conditions, we
obtain for all $m\in\mathbb{N}$ that

$\|\hat{\cal{N}}_{m+1}({\bf{x}})-\hat{\cal{N}}_{1}({\bf{x}})\|\leq
Le_{m,1}+LP\|\hat{\bf{W}}_{m+1}\hat{\cal{N}}_{m}({\bf{x}})-\hat{\bf{W}}_{1}{\bf{x}}\|.$

That is, inequality (3.1) holds for $n=1$. We now assume that inequality (3.1)
holds for $n=k$. We next proceed for the case $n=k+1$. Again, by the Lipschitz
continuity of the activation function $\sigma$ and the pooling operator
${\cal{P}}_{\mu}$ with the Lipschitz constants $L$ and $P$, respectively, and
by recursion (3.2), for all $m\in\mathbb{N}$, we obtain that

$\displaystyle\|\hat{\cal{N}}_{k+1+m}({\bf{x}})-\hat{\cal{N}}_{k+1}({\bf{x}})\|\leq$
$\displaystyle
Le_{m,k+1}+LP\|\hat{\cal{N}}_{k}(x)\|E_{m,k+1}+LP\|\hat{\bf{W}}_{k+1+m}\|\|\hat{\cal{N}}_{k+m}({\bf{x}})-\hat{\cal{N}}_{k}({\bf{x}})\|.$
(3.5)

Substituting the induction hypothesis into the third term on the right-hand-
side of inequality (3.5) and noting for all $i,k,m\in\mathbb{N}$ that
$\Lambda_{k+m}^{i-1}=\prod_{j=1}^{i}\|\hat{\bf{W}}_{k+1+m-j}\|,$ we find for
all $m\in\mathbb{N}$ that

$\displaystyle\|\hat{\cal{N}}_{k+1+m}({\bf{x}})-\hat{\cal{N}}_{k+1}({\bf{x}})\|\leq$
$\displaystyle
Le_{m,k+1}+LP\|\hat{\cal{N}}_{k}(x)\|E_{m,k+1}+L\sum_{i=0}^{k-1}\Lambda_{k+1+m}^{i}e_{m,k-i}$
$\displaystyle+LP\sum_{i=0}^{k-2}\Lambda_{k+1+m}^{i}\|\hat{\cal{N}}_{k-1-i}(x)\|E_{m,k-i}$
$\displaystyle+LP\Lambda_{k+1+m}^{k-1}\|\hat{\bf{W}}_{m+1}\hat{\cal{N}}_{m}({\bf{x}})-\hat{\bf{W}}_{1}{\bf{x}}\|.$
(3.6)

By replacing $i+1$ with $i$, the third and fourth terms of the right-hand-side
of inequality (3) become

$L\sum_{i=1}^{k}\Lambda_{k+1+m}^{i-1}e_{m,k+1-i}\ \ \mbox{and}\ \
LP\sum_{i=1}^{k-1}\Lambda_{k+1+m}^{i-1}\|\hat{\cal{N}}_{k+1-1-i}(x)\|E_{m,k+1-i},$

respectively. Substituting them into the third and fourth terms of the right-
hand-side of (3) leads to inequality (3.1) with $n=k+1$. Thus, the induction
principle ensures that inequality (3.1) holds for all positive integers $n$
and $m$. $\Box$

Lemma 3.1 has pointed the direction for establishing the analysis framework
for uniform convergence of deep neural networks. Along this line, we now
derive two additional technical lemmas that help estimate the products and
sums appearing in (3.1).

###### Lemma 3.2

If $\alpha_{n}$, $n\in\mathbb{N}$, is a sequence of non-negative numbers
satisfying the condition

$\lim_{n\to\infty}\alpha_{n}=\alpha<1,$ (3.7)

then there exists a positive constant $c$ such that

$\prod_{j=1}^{n}\alpha_{j}\leq c,\ \ \mbox{for all}\ \ n\in\mathbb{N}$ (3.8)

and

$\sum_{j=1}^{n}\left(\prod_{i=j+1}^{n}\alpha_{i}\right)\leq c,\ \ \mbox{for
all}\ \ n\in\mathbb{N}.$ (3.9)

Proof: We introduce the notation

$A_{n}:=\prod_{j=1}^{n}\alpha_{j},$

and

$B_{n}:=\sum_{j=1}^{n}\left(\prod_{i=j+1}^{n}\alpha_{i}\right).$

Since condition (3.7) is satisfied, there exist $N\in\mathbb{N}$ and
$\alpha\leq\gamma<1$ such that

$\alpha_{n}\leq\gamma,\ \ \mbox{for all}\ \ n\geq N.$ (3.10)

The sequence $A_{n}$, $n\in\mathbb{N}$, is decreasing if $n$ is sufficiently
large. Therefore, it must be bounded and thus, (3.8) holds true.

It remains to show that there exists a positive constant $c$ such that
$B_{n}\leq c$ for all $n\in\mathbb{N}$. Let $N\in\mathbb{N}$ be the integer
chosen so that inequality (3.10) holds. For all $n\geq N$, we write

$B_{n}=\sum_{j=1}^{N-1}\left(\prod_{i=j+1}^{n}\alpha_{i}\right)+\sum_{j=N}^{n}\left(\prod_{i=j+1}^{n}\alpha_{i}\right).$
(3.11)

For the first sum of the right-hand-side of equation (3.11), by (3.10) we have
that

$\displaystyle\sum_{j=1}^{N-1}\left(\prod_{i=j+1}^{n}\alpha_{i}\right)$
$\displaystyle=\sum_{j=1}^{N-1}\left(\prod_{i=j+1}^{N-1}\alpha_{i}\right)\left(\prod_{i=N}^{n}\alpha_{i}\right)$
$\displaystyle\leq\gamma^{n-N+1}\sum_{j=1}^{N-1}\left(\prod_{i=j+1}^{N-1}\alpha_{i}\right).$

The inequality above together with $0<\gamma<1$ leads to

$\sum_{j=1}^{N-1}\left(\prod_{i=j+1}^{n}\alpha_{i}\right)<B_{N-1}.$ (3.12)

For the second sum of the right-hand-side of equation (3.11), once again,
according to (3.10), we have that

$\displaystyle\sum_{j=N}^{n}\left(\prod_{i=j+1}^{n}\alpha_{i}\right)$
$\displaystyle=1+\sum_{j=N}^{n-1}\left(\prod_{i=j+1}^{n}\alpha_{i}\right)$
$\displaystyle\leq\sum_{j=0}^{n-N}\gamma^{j}<\frac{1}{1-\gamma}.$

Substituting this inequality and the estimate (3.12) into the right-hand-side
of equation (3.11) leads to

$B_{n}<B_{N-1}+\frac{1}{1-\gamma},\ \ \mbox{for all}\ \ n\geq N.$

Defining

$c:=\max\left\\{B_{1},B_{2},\dots,B_{N-2},B_{N-1}+\frac{1}{1-\gamma}\right\\},$

we conclude that $B_{n}\leq c$, for all $n\in\mathbb{N}$. $\Box$

Here comes the second technical lemma.

###### Lemma 3.3

Let $\alpha_{n}$, $\beta_{n}$, $n\in\mathbb{N}$, be two sequences of non-
negative numbers. If

$\lim_{n\to\infty}\alpha_{n}<1,\ \ \mbox{and}\ \
\lim_{n\to\infty}\beta_{n}=0,$ (3.13)

then for any positive $\epsilon>0$, there exits $N\in\mathbb{N}$ such that

$\sum_{i=0}^{n-1}\left(\prod_{j=0}^{i-1}\alpha_{n-j}\right)\beta_{n-i}<\epsilon,\
\ \mbox{for all}\ \ n>N.$ (3.14)

Proof: For $n\in\mathbb{N}$, we define the sequence

$A_{n}:=\sum_{i=0}^{n-1}\left(\prod_{j=0}^{i-1}\alpha_{n-j}\right)\beta_{n-i}.$

Let $\epsilon>0$ be arbitrary. By the second limit of (3.13), there exists
$N_{1}\in\mathbb{N}$ such that $\alpha_{n}\leq 1$ for all $n\geq N_{1}$ and

$\beta_{n}<\epsilon,\ \ \mbox{for all}\ \ n\geq N_{1}.$ (3.15)

We split $A_{n}$ into two terms according to $N_{1}$. That is,

$A_{n}=\sum_{i=0}^{n-N_{1}}\left(\prod_{j=0}^{i-1}\alpha_{n-j}\right)\beta_{n-i}+\sum_{i=n-N_{1}+1}^{n-1}\left(\prod_{j=0}^{i-1}\alpha_{n-j}\right)\beta_{n-i}.$
(3.16)

We denote by $A^{1}_{n}$ and $A_{n}^{2}$ the first and second sums of (3.16),
respectively. By using the inequality (3.15), we obtain for all $n\geq N_{1}$
that

$A_{n}^{1}<\epsilon\sum_{i=0}^{n-N_{1}}\left(\prod_{j=0}^{i-1}\alpha_{n-j}\right).$

By changing the index $k:=n-j$, we have that

$\left(\prod_{j=0}^{i-1}\alpha_{n-j}\right)=\left(\prod_{k=n-i+1}^{n}\alpha_{k}\right).$

Hence, we get that

$A_{n}^{1}<\epsilon\sum_{i=0}^{n-N_{1}}\left(\prod_{k=n-i+1}^{n}\alpha_{k}\right).$
(3.17)

Letting $j:=n-i$, using the non-negativity of the numbers $\alpha_{k}$ and
employing (3.9) of Lemma 3.2, there exists a positive constant $c_{1}$ such
that for all $n\geq N_{1}$,

$\displaystyle\sum_{i=0}^{n-N_{1}}\left(\prod_{k=n-i+1}^{n}\alpha_{k}\right)$
$\displaystyle=\sum_{j=N_{1}}^{n}\left(\prod_{k=j+1}^{n}\alpha_{k}\right)$
$\displaystyle\leq\sum_{j=1}^{n}\left(\prod_{i=j+1}^{n}\alpha_{i}\right)\leq
c_{1}.$

This together with (3.17) yields that $A_{n}^{1}<c_{1}\epsilon$ for all $n\geq
N_{1}$.

We next estimate $A_{n}^{2}$. Clearly, there exists a positive constant
$c_{2}$ such that $\beta_{n}\leq c_{2}$ for all $n\in\mathbb{N}$. Using this
fact and the change of indices $k:=n-j$, we obtain that

$\displaystyle A_{n}^{2}$ $\displaystyle\leq
c_{2}\sum_{i=n-N_{1}+1}^{n-1}\left(\prod_{k=n-i+1}^{n}\alpha_{k}\right).$

We then perform an additional change of indices $j:=n-i+1$ and find that

$\displaystyle A_{n}^{2}$ $\displaystyle\leq
c_{2}\sum_{j=2}^{N_{1}}\left(\prod_{k=j}^{n}\alpha_{k}\right).$ (3.18)

The first limit of (3.13) implies that there exist $0<\gamma<1$ and
$N_{2}\in\mathbb{N}$ with $N_{2}\geq N_{1}$ such that $\alpha_{n}\leq\gamma$
for all $n>N_{2}$. It follows from (3.18) that

$\displaystyle A_{n}^{2}$ $\displaystyle\leq
c_{2}\sum_{j=2}^{N_{1}}\left(\prod_{k=j}^{N_{2}-1}\alpha_{k}\right)\left(\prod_{k=N_{2}}^{n}\alpha_{k}\right)$
$\displaystyle\leq
c_{2}\gamma^{n-N_{2}+1}\sum_{j=2}^{N_{1}}\left(\prod_{k=j}^{N_{2}-1}\alpha_{k}\right)$
$\displaystyle=c_{2}\gamma^{n-N_{2}+1}\sum_{j=1}^{N_{1}-1}\left(\prod_{k=j+1}^{N_{2}-1}\alpha_{k}\right).$

Again, by the non-negativity of the numbers $\alpha_{k}$ and estimate (3.9) of
Lemma 3.2, we observe for all $n>N_{2}$ that

$\sum_{j=1}^{N_{1}-1}\left(\prod_{k=j+1}^{N_{2}-1}\alpha_{k}\right)\leq\sum_{j=1}^{n}\left(\prod_{k=j+1}^{n}\alpha_{k}\right)\leq
c_{1}.$

This implies for all $n>N_{2}$ that

$A_{n}^{2}\leq c_{1}c_{2}\gamma^{n-N_{2}+1}.$

Since $0<\gamma<1$, there exists $N_{3}\in\mathbb{N}$ such that for all
$n>N_{3}$, $\gamma^{n}<\epsilon$. Therefore, we conclude that
$A_{n}^{2}<c_{1}c_{2}\epsilon$, for all $n>N_{2}+N_{3}$. Adding this estimate
to the estimate for $A_{n}^{1}$, we have that

$A_{n}<(c_{1}+c_{1}c_{2})\epsilon,\ \ \mbox{for all}\ \ n>N_{2}+N_{3},$

proving (3.14). $\Box$

With the help of Lemmas 3.1 and 3.3, we establish the following result on
uniform convergence of the sequence $\hat{\cal{N}}_{n}({\bf{x}})$ generated by
the recursion (3.1) and (3.2) in a general setting. We say that
$\hat{\bf{b}}_{n}$, $n\in\mathbb{N}$, converges if there exists
${\bf{b}}^{*}\in\mathbb{R}^{l}$ (resp. ${\bf{b}}^{*}\in\ell_{p}(\mathbb{N})$)
such that $\lim_{n\to\infty}\|\hat{\bf{b}}_{n}-{\bf{b}}^{*}\|=0$ when
$\hat{\bf{b}}_{n}\in\mathbb{R}^{l}$ (resp.
$\hat{\bf{b}}_{n}\in\ell_{p}(\mathbb{N})$), $n\in\mathbb{N}$, and $\|\cdot\|$
denotes a norm in $\mathbb{R}^{l}$ (resp. in $\ell_{p}(\mathbb{N})$).
Likewise, we say that $\hat{\bf{W}}_{n}$, $n\in\mathbb{N}$, converges if there
exists ${\bf{W}}^{*}\in\mathbb{R}^{l\times l}$ (resp.
${\bf{W}}^{*}\in\ell_{p}(\mathbb{N}^{2})$) such that
$\lim_{n\to\infty}\|\hat{\bf{W}}_{n}-{\bf{W}}^{*}\|=0$ when
$\hat{\bf{W}}_{n}\in\mathbb{R}^{l\times l}$ (resp.
$\hat{\bf{W}}_{n}\in\ell_{p}(\mathbb{N}^{2})$), $n\in\mathbb{N}$, and
$\|\cdot\|$ denotes a norm in $\mathbb{R}^{l\times l}$ (resp. in
$\ell_{p}(\mathbb{N}^{2})$). Here, we do not distinguish the notation of a
vector norm from a matrix norm since they can be clearly differentiated from
the context. It is known that the sequence $\hat{\bf{b}}_{n}$,
$n\in\mathbb{N}$, converges if and only if it is Cauchy in $\mathbb{R}^{l}$,
and the sequence $\hat{\bf{W}}_{n}$, $n\in\mathbb{N}$, converges if and only
if it is Cauchy in $\ell_{p}(\mathbb{N}^{2})$.

###### Theorem 3.4

Suppose that the activation function $\sigma$ and the pooling operator
${\cal{P}}_{\mu}$ are Lipschitz continuous with the Lipschitz constants $L$
and $P$, respectively, and that $\mathbb{D}\subset\mathbb{R}^{s}$ is bounded.
If the sequences $\hat{\bf{b}}_{n}$, $\hat{\bf{W}}_{n}$, $n\in\mathbb{N}$,
converge, the sequence $\|\hat{\bf{W}}_{n}\|$, $n\in\mathbb{N}$, satisfies the
condition

$\omega:=\lim_{n\to\infty}LP\|\hat{\bf{W}}_{n}\|<1,$ (3.19)

and there exists a positive constant $\rho$ such that

$\sup_{{\bf{x}}\in\mathbb{D}}\|\hat{\cal{N}}_{n}({\bf{x}})\|\leq\rho,\ \
\mbox{for all}\ \ n\in\mathbb{N},$ (3.20)

then the sequence $\hat{\cal{N}}_{n}$ converges uniformly to ${\cal{N}}\in
C_{l}(\mathbb{D})$ (resp. ${\cal{N}}\in C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$)
if $\hat{\bf{W}}_{n}\in\mathbb{R}^{l\times l}$ (resp.
$\hat{\bf{W}}_{n}\in\ell_{p}(\mathbb{N}^{2})$).

Proof: It suffices to show that the sequence ${\cal{N}}_{n}$,
$n\in\mathbb{N}$, is Cauchy in space $C_{l}(\mathbb{D})$ or
$C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$. Let $\epsilon>0$ be arbitrary. We wish
to prove that there exist $N\in\mathbb{N}$ and $c>0$ such that

$\sup_{{\bf{x}}\in\mathbb{D}}\|\hat{\cal{N}}_{n+m}({\bf{x}})-\hat{\cal{N}}_{n}({\bf{x}})\|<c\epsilon,\
\ \mbox{for all}\ \ n\geq N,m\in\mathbb{N}.$ (3.21)

Motivated by Lemma 3.1, we introduce

$J_{n,1}:=L\sum_{i=0}^{n-1}\Lambda_{n+m}^{i-1}e_{m,n-i},\ \
J_{n,2}({\bf{x}}):=LP\sum_{i=0}^{n-2}\Lambda_{n+m}^{i-1}\|\hat{\cal{N}}_{n-1-i}({\bf{x}})\|E_{m,n-i},\
\ {\bf{x}}\in\mathbb{D},$

and

$J_{n,3}({\bf{x}}):=LP\Lambda_{n+m}^{n-2}\|\hat{\bf{W}}_{m+1}\hat{\cal{N}}_{m}({\bf{x}})-\hat{\bf{W}}_{1}{\bf{x}}\|,\
\ {\bf{x}}\in\mathbb{D}.$

By Lemma 3.1, for all $n,m\in\mathbb{N}$ we have that

$\sup_{{\bf{x}}\in\mathbb{D}}\|\hat{\cal{N}}_{n+m}({\bf{x}})-\hat{\cal{N}}_{n}({\bf{x}})\|\leq
J_{n,1}+\sup_{{\bf{x}}\in\mathbb{D}}J_{n,2}({\bf{x}})+\sup_{{\bf{x}}\in\mathbb{D}}J_{n,3}({\bf{x}}).$
(3.22)

We next show that the three terms on the right-hand-side of inequality (3.22)
tend to zero as $n\to\infty$ for all $m\in\mathbb{N}$.

We first consider $J_{n,1}$. For $n\in\mathbb{N}$, we let

$\alpha_{n}:=LP\|\hat{\bf{W}}_{n+m}\|\ \ \mbox{and}\ \ \beta_{n}:=e_{m,n},\ \
m\in\mathbb{N}.$

By hypothesis (3.19), we have that $\lim_{n\to\infty}\alpha_{n}=\omega<1$.
Since the sequence $\hat{\bf{b}}_{n}$, $n\in\mathbb{N}$, converges, we note
that $\lim_{n\to\infty}\beta_{n}=0$. Applying Lemma 3.3 and noting the
definition (3.4) of $\Lambda_{n+m}^{i-1}$, we conclude that there exists
$N_{1}\in\mathbb{N}$ such that

$J_{n,1}<L\epsilon,\ \ \mbox{for all}\ \ n>N_{1},\ m\in\mathbb{N}.$ (3.23)

For the second term on the right-hand-side of (3.22), by the uniform
boundedness assumption (3.20) on $\hat{\cal{N}}_{n}({\bf{x}})$, we see that

$\sup_{{\bf{x}}\in\mathbb{D}}J_{n,2}({\bf{x}})\leq\rho
LP\sum_{i=0}^{n-2}\Lambda_{n+m}^{i-1}E_{m,n-i},\ \ m\in\mathbb{N}.$

For $n\in\mathbb{N}$, we define

$\alpha_{n}:=LP\|\hat{\bf{W}}_{n+m}\|\ \ \mbox{and}\ \ \beta_{n}:=E_{m,n},\ \
m\in\mathbb{N}.$

By hypothesis (3.19), we observe that $\lim_{n\to\infty}\alpha_{n}<1$ and by
the convergence of sequence $\hat{\bf{W}}_{n}$, $n\in\mathbb{N}$, we find that
$\lim_{n\to\infty}\beta_{n}=0$. Again, by employing Lemma 3.3, there exists
$N_{2}\in\mathbb{N}$ such that

$\sup_{{\bf{x}}\in\mathbb{D}}J_{n,2}({\bf{x}})<\rho LP\epsilon,\ \ \mbox{for
all}\ \ n>N_{2}.$ (3.24)

It remains to estimate the third term on the right-hand-side of (3.22). To
this end, we assume that $\mathbb{D}\subset\mathbb{R}^{s}$ is bounded by
$D>0$. Again, due to the uniform boundedness of
$\|\hat{\cal{N}}_{n}({\bf{x}})\|$, $n\in\mathbb{N}$, and convergence of
$\hat{\bf{W}}_{n}$, $n\in\mathbb{N}$, which implies that
$\|\hat{\bf{W}}_{n}\|\leq w$ for all $n\in\mathbb{N}$ and for some $w>0$, we
reach that

$\sup_{{\bf{x}}\in\mathbb{D}}J_{n,3}({\bf{x}})\leq
LPw(\rho+D)\Lambda_{n+m}^{n-2}.$ (3.25)

By the definition (3.4) of $\Lambda_{n+m}^{n-2}$ with the change of indices
$k:=n+m-j$, we note that

$\Lambda_{n+m}^{n-2}=\prod_{k=m+2}^{n+m}LP\|\hat{\bf{W}}_{k}\|.$ (3.26)

By virtue of hypothesis (3.19), there exist $N_{3}\in\mathbb{N}$ and
$\omega_{0}\in[\omega,1)$ such that $LP\|\hat{\bf{W}}_{k}\|<\omega_{0}$, for
all $k>N_{3}$. Therefore, by (3.25) and (3.26), for all $m\geq 1$ and $n\geq
2$,

$\sup_{{\bf{x}}\in\mathbb{D}}J_{n,3}({\bf{x}})\leq
LP\omega(\rho+D)\omega_{0}^{n+m-N_{3}}\omega^{N_{3}-m-1}..$ (3.27)

In view of $0<\omega_{0}<1$, there exists $N_{4}\in\mathbb{N}$ such that
$\omega_{0}^{n-1}<\epsilon$, for all $n>N_{4}$. Upon substituting this result
into the right-hand-side of (3.27) yields

$\sup_{{\bf{x}}\in\mathbb{D}}J_{n,3}({\bf{x}})\leq LPw(\rho+D)\epsilon,\ \
\mbox{for all}\ \ m\geq 1,\ n>N_{4}.$ (3.28)

Now, we choose $N:=\max\\{N_{i}:i=1,2,3,4\\}$. Upon substituting estimates
(3.23), (3.24) and (3.28) into the right-hand-side of (3.22) gives rise to
(3.21) with $c:=L+\rho LP+LPw(\rho+D)$. $\Box$

It was indicated in [16] that when the activation function is contractive, the
neural networks converge exponentially as the layer number tends to infinity
if both the sequence of weight matrices and that of bias vectors converge
exponentially. Our next task is to establish an exponential convergence
theorem for the deep neural network with general Lipschitz continuous
activation functions, not necessarily contractive, and general Lipschitz
continuous poolings.

###### Lemma 3.5

Suppose that the activation function $\sigma$ and the pooling operator
${\cal{P}}_{\mu}$ are Lipschitz continuous with the Lipschitz constants $L$
and $P$, respectively, and that $\mathbb{D}\subset\mathbb{R}^{s}$ is bounded
by $D>0$. If $\lim_{n\to\infty}\hat{\bf{b}}_{n}={\bf{b}}^{*}$,
$\lim_{n\to\infty}\hat{\bf{W}}_{n}={\bf{W}}^{*}$, and satisfies (3.19), and
there exists a positive constant $\rho$ such that (3.20) holds, then there
exists a function ${\cal{N}}\in C_{l}(\mathbb{D})$ (resp. ${\cal{N}}\in
C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$) if
$\hat{\bf{W}}_{n}\in\mathbb{R}^{l\times l}$ (resp.
$\hat{\bf{W}}_{n}\in\ell_{p}(\mathbb{N}^{2})$) such that for all
$n\in\mathbb{N}$,

$\sup_{{\bf{x}}\in\mathbb{D}}\|\hat{\cal{N}}_{n}({\bf{x}})-{\cal{N}}({\bf{x}})\|\leq
L\left[\sum_{i=0}^{n-1}\omega_{0}^{i}e_{n-i}+\rho
LP\sum_{i=0}^{n-2}\omega_{0}^{i}E_{n-i}+LPw(\rho+D)\omega_{0}^{n-1}\right],$
(3.29)

where $e_{n}:=\|\hat{\bf{b}}_{n}-{\bf{b}}^{*}\|$,
$E_{n}:=\|\hat{\bf{W}}_{n}-{\bf{W}}^{*}\|$, $w$ is an upper bound of the
sequence $\|\hat{\bf{W}}_{n}\|$, $n\in\mathbb{N}$, and
$\omega\leq\omega_{0}<1$.

Proof: Theorem 3.4 ensures that the deep neural networks
$\hat{\cal{N}}_{n}({\bf{x}})$ converges to a function ${\cal{N}}$. By Lemma
3.1, for all positive integers $n$, $m$, inequality (3.1) holds. Due to
hypothesis (3.19), there exist $N\in\mathbb{N}$ and $\omega\leq\omega_{0}<1$
such that $LP\|\hat{\bf{W}}_{m}\|\leq\omega_{0}$, for all $m\geq N$. It
follows that $\Lambda_{n+m}^{i-1}\leq\omega_{0}^{i}$, $i=0,1,\dots,n-1$, for
all $m\geq N,n\in\mathbb{N}$ and $\Lambda_{n+m}^{n-2}\leq\omega_{0}^{n-1}$,
for all $m\geq N,n\in\mathbb{N}$. Substituting these bounds into the right-
hand-side of the inequality (3.1), letting $m\to\infty$ in the both sides of
the resulting inequality, and using the limits of $\hat{\bf{b}}_{n}$ and
$\hat{\bf{W}}_{n}$, and the uniform boundedness of
$\hat{\cal{N}}_{n}({\bf{x}})$, we obtain for ${\bf{x}}\in\mathbb{D}$ the error
bound (3.29). $\Box$

Lemma 3.5 allows us to establish the following exponential convergence result.

###### Theorem 3.6

Suppose that the activation function $\sigma$ and the pooling operator
${\cal{P}}_{\mu}$ are Lipschitz continuous with the Lipschitz constants $L$
and $P$, respectively, and $\mathbb{D}\subseteq\mathbb{R}^{s}$ is bounded. If
$\hat{\bf{b}}_{n}$ and $\hat{\bf{W}}_{n}$ converge, respectively, to
${\bf{b}}^{*}$ and ${\bf{W}}^{*}$ exponentially with (3.19), and there exists
a positive constant $\rho$ such that
$\|\hat{\cal{N}}_{n}({\bf{x}})\|\leq\rho$, for all ${\bf{x}}\in\mathbb{D}$ and
all $n\in\mathbb{N}$, then the neural networks $\hat{\cal{N}}_{n}$ converge to
a function ${\cal{N}}\in C_{l}(\mathbb{D})$ (resp. ${\cal{N}}\in
C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$) if
$\hat{\bf{W}}_{n}\in\mathbb{R}^{l\times l}$ (resp.
$\hat{\bf{W}}_{n}\in\ell_{p}(\mathbb{N}^{2})$) exponentially and uniformly in
$\mathbb{D}$.

Proof: First of all, Theorem 3.4 ensures that the neural networks
$\hat{\cal{N}}_{n}$ converge to a function ${\cal{N}}$ uniformly in
$\mathbb{D}$. It remains to prove that the convergence rate of
$\hat{\cal{N}}_{n}$ to ${\cal{N}}$ is exponential. For this purpose, by the
hypothesis of this theorem, we conclude that there exist $0<r<1$ and $c>0$
such that $e_{n}\leq cr^{n}$ and $E_{n}\leq cr^{n}.$ Substituting these
estimates into the right-hand-side of (3.29) in Lemma 3.5, we find that

$\sup_{{\bf{x}}\in\mathbb{D}}\|\hat{\cal{N}}_{n}({\bf{x}})-{\cal{N}}({\bf{x}})\|\leq(cL+c\rho
L^{2}P)nr_{0}^{n}+\frac{L^{2}Pw(\rho+D)}{\omega_{0}}r_{0}^{n},$

where $r_{0}:=\max\\{r,\omega_{0}\\}$ and $D>0$ is a bound of $\mathbb{D}$.
Thus, there exist $N\in\mathbb{N}$ and $r_{1}\in(r_{0},1)$ such that for all
$n>N$,

$\sup_{{\bf{x}}\in\mathbb{D}}\|\hat{\cal{N}}_{n}({\bf{x}})-{\cal{N}}({\bf{x}})\|\leq\left[cL+c\rho
L^{2}P+\frac{L^{2}Pw(\rho+D)}{\omega_{0}}\right]r_{1}^{n}.$

That is, the neural networks $\hat{\cal{N}}_{n}$ converge to the function
${\cal{N}}$ exponentially and uniformly in $\mathbb{D}$. $\Box$

In Theorems 3.4 and 3.6, we assume that the sequence
$\|\hat{\cal{N}}_{n}({\bf{x}})\|$ is uniformly bounded in a bounded set
$\mathbb{D}$. This can be derived from the next lemma and an additional
hypothesis on the vector/sequence norm. In the next lemma, we assume that
either ${\bf{W}}_{n}\in\mathbb{R}^{(m_{n}+\mu)\times m_{n-1}}$ with
$m_{0}:=s$, and ${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$, $n\in\mathbb{N}$, or
${\bf{W}}_{1}\in\ell_{p}(\mathbb{N}\times\mathbb{N}_{s})$,
${\bf{W}}_{n}\in\ell_{p}(\mathbb{N}^{2})$, $n>1$, and
${\bf{b}}_{n}\in\ell_{p}(\mathbb{N})$, $n\in\mathbb{N}$.

###### Lemma 3.7

If the activation function $\sigma$ and the pooling operator ${\cal{P}}_{\mu}$
are Lipschitz continuous with the Lipschitz constants $L$ and $P$,
respectively, then for all $n\in\mathbb{N}$

$\|\hat{\cal{N}}_{n}({\bf{x}})\|\leq\left(\prod_{j=1}^{n}LP\|\hat{\bf{W}}_{j}\|\right)\|{\bf{x}}\|+\sum_{j=1}^{n}\left(\prod_{i=j+1}^{n}LP\|\hat{\bf{W}}_{i}\|\right)(L\|\hat{\bf{b}}_{j}\|+\|(\sigma\circ{\cal{P}}_{\mu})({\bf
0}_{j})\|),$ (3.30)

where ${\bf 0}_{j}\in\mathbb{R}^{m_{j}+\mu}$ are zero vectors for the finite
dimensional case and ${\bf 0}_{j}\in\ell_{p}(\mathbb{N})$ are zero vectors for
the infinite dimensional case.

Proof: We establish inequality (3.30) by induction on $n$. When $n=1$, by
(2.4), for all ${\bf{x}}\in\mathbb{R}^{s}$ we have that

$\displaystyle\|\hat{\cal{N}}_{1}({\bf{x}})\|$
$\displaystyle\leq\|\sigma({\cal{P}}_{\mu}(\hat{\bf{W}}_{1}{\bf{x}})+\hat{\bf{b}}_{1})-\sigma({\cal{P}}_{\mu}({\bf
0}_{1}))\|+\|\sigma({\cal{P}}_{\mu}({\bf 0}_{1}))\|$ $\displaystyle\leq
LP\|\hat{\bf{W}}_{1}\|\|{\bf{x}}\|+L\|\hat{\bf{b}}_{1}\|+\|\sigma({\cal{P}}_{\mu}({\bf
0}_{1}))\|.$

Thus, inequality (3.30) holds for $n=1$. We assume that inequality (3.30)
holds for $n=k$ and proceed for the case $n=k+1$. Using the recursion formula
(3.2) and the induction hypothesis, for all ${\bf{x}}\in\mathbb{R}^{d}$ we
obtain that

$\|\hat{\cal{N}}_{k+1}({\bf{x}})\|\leq\|\sigma({\cal{P}}_{\mu}(\hat{\bf{W}}_{k+1}\hat{\cal{N}}_{k}({\bf{x}}))+\hat{\bf{b}}_{k+1})-\sigma({\cal{P}}_{\mu}({\bf
0}_{k+1}))\|+\|\sigma({\cal{P}}_{\mu}({\bf 0}_{k+1}))\|.$

Using the Lipschitz continuity of $\sigma$ and ${\cal{P}}_{\mu}$, we find that

$\|\hat{\cal{N}}_{k+1}({\bf{x}})\|\leq
LP\|\hat{\bf{W}}_{k+1}\|\|\hat{\cal{N}}_{k}({\bf{x}})\|+L\|\hat{\bf{b}}_{k+1}\|+\|\sigma({\cal{P}}_{\mu}({\bf
0}_{k+1}))\|.$

Invoking the induction hypothesis in the right-hand-side of the last
inequality yields

$\displaystyle\|\hat{\cal{N}}_{k+1}({\bf{x}})\|\leq$ $\displaystyle
LP\|\hat{\bf{W}}_{k+1}\|\left[\left(\prod_{j=1}^{k}LP\|\hat{\bf{W}}_{j}\|\right)\|{\bf{x}}\|+\sum_{j=1}^{k}\left(\prod_{i=j+1}^{k}LP\|\hat{\bf{W}}_{i}\|\right)(L\|\hat{\bf{b}}_{j}\|+\|\sigma({\cal{P}}_{\mu}({\bf
0}_{j}))\|)\right]$
$\displaystyle+L\|\hat{\bf{b}}_{k+1}\|+\|\sigma({\cal{P}}_{\mu}({\bf
0}_{k+1}))\|$ $\displaystyle=$
$\displaystyle\left(\prod_{j=1}^{k+1}LP\|\hat{\bf{W}}_{j}\|\right)\|{\bf{x}}\|+\sum_{j=1}^{k+1}\left(\prod_{i=j+1}^{k+1}LP\|\hat{\bf{W}}_{i}\|\right)(L\|\hat{\bf{b}}_{j}\|+\|\sigma({\cal{P}}_{\mu}({\bf
0}_{j}))\|).$

Thus, (3.30) holds for $n=k+1$ and the induction principle ensures that
inequality (3.30) holds for all $n\in\mathbb{N}$. $\Box$

Lemma 3.7 with an additional hypothesis on the norm guarantees that the
sequence $\|\hat{\cal{N}}_{n}({\bf{x}})\|$, $n\in\mathbb{N}$, is uniformly
bounded in a bounded set $\mathbb{D}$. Such a hypothesis will be made clear in
later sections in specific contexts.

##  4 Uniform Convergence of Deep Neural Networks with a Fixed Width

In this section, we establish uniform convergence of deep neural networks with
a fixed width.

Throughout this section, we let $l\in\mathbb{N}$ be fixed and suppose that the
sequence of weight matrices ${\bf{W}}_{1}\in\mathbb{R}^{l\times s}$,
${\bf{W}}_{n}\in\mathbb{R}^{l\times l}$, $n\geq 2$, and the sequence of bias
vectors ${\bf{b}}_{n}\in\mathbb{R}^{l}$, $n\in\mathbb{N}$. We then define
neural networks ${\cal{N}}_{n}({\bf{x}})\in\mathbb{R}^{l}$ by (2.3) without
pooling and they satisfy the recursions (2.4) and (2.5). We assume that the
vector norm used in this section satisfies (2.7) and (2.8).

The next lemma that follows directly from Lemma 3.7 provides a bound of the
deep neural network by the norms of the weight matrices and bias vectors.
Here, the widths of the weight matrices and bias vectors are assumed to be
variable, which is somewhat more general than what we need in this section.
Since the neural networks are constructed without pooling, in this case we
have that ${\cal{P}}_{\mu}$ is the identity operator and $P=1$.

###### Lemma 4.1

If the activation function $\sigma$ is Lipschitz continuous with a Lipschitz
constant $L$, the weight matrices ${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times
m_{n-1}}$, for $n\in\mathbb{N}$ with $m_{0}:=s$, and the bias vectors
${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$ for $n\in\mathbb{N}$, then for all
$n\in\mathbb{N}$

$\|{\cal{N}}_{n}({\bf{x}})\|\leq\left(\prod_{j=1}^{n}L\|{\bf{W}}_{j}\|\right)\|{\bf{x}}\|+\sum_{j=1}^{n}\left(\prod_{i=j+1}^{n}L\|{\bf{W}}_{i}\|\right)(L\|{\bf{b}}_{j}\|+\|\sigma({\bf
0}_{m_{j}})\|),$ (4.1)

where ${\bf 0}_{j}\in\mathbb{R}^{j}$ is a zero vector.

Inequality (4.1) in Lemma 4.1 may be translated to the boundedness of the deep
neural network sequence in a bounded domain.

###### Lemma 4.2

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with a
Lipschitz constant $L$, $\mathbb{D}\subseteq\mathbb{R}^{s}$ is bounded by
$D>0$, the weight matrices ${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times m_{n-1}}$,
for $n\in\mathbb{N}$ with $m_{0}:=s$, and the bias vectors
${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$ for $n\in\mathbb{N}$. If there exists a
constant $c>0$ such that $\|{\bf{b}}_{n}\|\leq c$, for all $n\in\mathbb{N}$
and ${\bf{W}}_{n}$, $n\in\mathbb{N}$, satisfy the condition

$\lim_{n\to\infty}L\|{\bf{W}}_{n}\|<1,$ (4.2)

then there exists a positive constant $c_{1}$ such that

$\sup_{{\bf{x}}\in\mathbb{D}}\|{\cal{N}}_{n}({\bf{x}})\|\leq
c_{1}D+Lcc_{1}+c_{1}\max\\{\|\sigma({\bf 0}_{m_{j}})\|:j\in\mathbb{N}_{n}\\},\
\ \mbox{for all}\ \ n\in\mathbb{N},$ (4.3)

where ${\bf 0}_{j}\in\mathbb{R}^{j}$ is a zero vector. If the widths $m_{n}$
are bounded, then there exists a constant $\rho>0$ such that
$\sup_{{\bf{x}}\in\mathbb{D}}\|{\cal{N}}_{n}({\bf{x}})\|\leq\rho$ for all
$n\in\mathbb{N}$.

Proof: By condition (4.2), according to Lemma 3.2 with
$\alpha_{n}:=L\|{\bf{W}}_{n}\|$, there exists a positive constant $c_{1}$ such
that for all $n\in\mathbb{N}$,

$\prod_{j=1}^{n}L\|{\bf{W}}_{j}\|\leq c_{1}$

and

$\sum_{j=1}^{n}\left(\prod_{i=j+1}^{n}L\|{\bf{W}}_{i}\|\right)\leq c_{1}.$

The hypothesis of this lemma ensures that inequality (4.1) in Lemma 4.1 holds
for all $n\in\mathbb{N}$. Using inequality (4.1) together with the bounds
given above, we obtain the estimate (4.3).

When the widths $m_{n}$ are bounded, there is a positive constant $c_{2}$ such
that

$\max\\{\|\sigma({\bf 0}_{m_{j}})\|:j\in\mathbb{N}_{n}\\}\leq c_{2}\ \
\mbox{for all}\ \ n\in\mathbb{N}.$

Therefore, (4.3) ensures that

$\sup_{{\bf{x}}\in\mathbb{D}}\|{\cal{N}}_{n}({\bf{x}})\|\leq\rho:=c_{1}D+Lcc_{1}+c_{1}c_{2},\
\ \mbox{for all}\ \ n\in\mathbb{N},$

proving the lemma. $\Box$

We are now ready to derive the uniform convergence of deep neural networks
with a fixed width from Theorem 3.4.

###### Theorem 4.3

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with a
Lipschitz constant $L$ and $\mathbb{D}\subset\mathbb{R}^{s}$ is bounded. If
the sequences ${\bf{b}}_{n}$, ${\bf{W}}_{n}$, $n\in\mathbb{N}$, converge with
(4.2), then the neural networks ${\cal{N}}_{n}$ converge uniformly in
$\mathbb{D}$.

Proof: Note that the neural networks ${\cal{N}}_{n}({\bf{x}})$ satisfy the
recursions (2.4) and (2.5). Since the sequence ${\bf{b}}_{n}$,
$n\in\mathbb{N}$, converges, it is bounded. By hypothesis, the sequence
$\|{\bf{W}}_{n}\|$, $n\in\mathbb{N}$, satisfies (4.2). Moreover, the neural
networks have a fixed width. By Lemma 4.2, we know that
${\cal{N}}_{n}({\bf{x}})$ is uniformly bounded by a constant $\rho$ for all
$n\in\mathbb{N}$ and ${\bf{x}}\in\mathbb{D}$. Therefore, by Theorem 3.4, the
neural networks ${\cal{N}}_{n}({\bf{x}})$, $n\in\mathbb{N}$, converge
uniformly to a function ${\cal{N}}\in C_{l}(\mathbb{D})$ in $\mathbb{D}$.
$\Box$

When we restrict the activation functions to contractions, Theorem 4.3 is
specialized to the uniform convergence theorem established in [16], since in
the special case of having a contractive activation function the assumption
(4.2) becomes

$\lim_{n\to\infty}\|{\bf{W}}_{n}\|<\gamma,$ (4.4)

which was assumed in [16]. When $L=1$, the activation functions are non-
expansive. Typical examples of non-expansive activation functions include ReLU
and Leaky ReLU. In this case, the assumption (4.2) becomes

$\lim_{n\to\infty}\|{\bf{W}}_{n}\|<1,$ (4.5)

which ensures the uniform convergence of deep neural networks with a non-
expansive activation function, according to Theorem 4.3. To our best
knowledge, uniform convergence of deep neural networks even with non-expansive
activation functions, and a fixed width is not available in the literature.
Theorem 4.3 guarantees uniform convergence of neural networks with not only
contractive but also expansive (Lipschitz continuous) activation functions.

The next result on the rate of uniform convergence of neural networks follows
directly from Theorem 3.6.

###### Theorem 4.4

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with a
Lipschitz constant $L$ and $\mathbb{D}\subseteq\mathbb{R}^{s}$ is bounded. If
${\bf{b}}_{n}$ and ${\bf{W}}_{n}$ converge to ${\bf{b}}^{*}$ and
${\bf{W}}^{*}$ exponentially with (4.2), then the neural networks
${\cal{N}}_{n}$ converge to a function ${\cal{N}}\in C_{l}(\mathbb{D})$
exponentially and uniformly in $\mathbb{D}$.

To close this section, we present a convergence result for deep neural
networks with pooling. In building a neural network, pooling is used to
enhance features and reduce the dimension of features. Uniform convergence of
deep neural networks with the average pooling and a fixed width in the vector
norms $\|\cdot\|_{p}$, for $1\leq p\leq+\infty$, and that of deep neural
networks with the max pooling in the vector norm $\|\cdot\|_{\infty}$ were
established in [16]. Theorems 4.3 and 4.4 may be modified for uniform
convergence of deep neural networks with pooling. The average pooling and the
max pooling are most popular in application. For an integer $\mu\geq 2$, the
average pooling ${\cal{P}}_{\mu}^{A}$ is the linear operator from
$\mathbb{R}^{l+\mu}$ to $\mathbb{R}^{l}$ defined by

$({\cal{P}}_{\mu}^{A}{\bf{x}})_{i}:=\frac{1}{\mu+1}\sum_{j=0}^{\mu}x_{i+j},\ \
i\in\mathbb{N}_{l},\ \ {\bf{x}}\in\mathbb{R}^{l+\mu}$ (4.6)

and the max pooling ${\cal{P}}_{\mu}^{M}$ is the nonlinear map from
$\mathbb{R}^{l+\mu}$ to $\mathbb{R}^{l}$ defined by

$({\cal{P}}_{\mu}^{M}{\bf{x}})_{i}:=\max\\{x_{i+j}:j=0,1,\dots,\mu\\},\ \
i\in\mathbb{N}_{l},\ \ {\bf{x}}\in\mathbb{R}^{l+\mu}.$ (4.7)

It is known that if $1\leq p\leq+\infty$, then for all
${\bf{x}}\in\mathbb{R}^{l+\mu}$

$\|{\cal{P}}_{\mu}^{A}{\bf{x}}\|_{p}\leq\|{\bf{x}}\|_{p},\ \ \mbox{for all}\ \
{\bf{x}}\in\mathbb{R}^{l+\mu},\ p\in[1,+\infty]$ (4.8)

and

$\|{\cal{P}}_{\mu}^{M}{\bf{x}}-{\cal{P}}_{\mu}^{M}{\bf{y}}\|_{p}\leq(\mu+1)^{1/p}\|{\bf{x}}-{\bf{y}}\|_{p},\
\ \mbox{for all}\ \ {\bf{x}},{\bf{y}}\in\mathbb{R}^{l+\mu}.$ (4.9)

It can be seen from (4.8) that ${\cal{P}}^{A}_{\mu}$ is a non-expansive linear
operator, and from (4.9) that ${\cal{P}}^{M}_{\mu}$ is Lipschitz continuous
with the Lipschitz constant $P:=(\mu+1)^{1/p}$ with respect to the vector norm
$\|\cdot\|_{p}$, for $1\leq p\leq\infty$. Clearly, when $1\leq p<\infty$,
${\cal{P}}^{M}_{\mu}$ are expanding and when $p=\infty$, ${\cal{P}}^{M}_{\mu}$
is non-expansive. In general, by ${\cal{P}}$ we denote the pooling map from
$\mathbb{R}^{l+\mu}$ to $\mathbb{R}^{l}$. Suppose that
${\bf{W}}_{1}\in\mathbb{R}^{(l+\mu)\times s}$,
${\bf{W}}_{n}\in\mathbb{R}^{(l+\mu)\times l}$ for $n\geq 2$,
${\bf{b}}_{n}\in\mathbb{R}^{l}$ for $n\in\mathbb{N}$. Deep neural networks
with pooling have the form

${{\cal{N}}}^{\cal{P}}_{n}({\bf{x}}):=\left(\bigodot_{i=1}^{n}\sigma({\cal{P}}({\bf{W}}_{i}\cdot)+{\bf{b}}_{i})\right)({\bf{x}}),\
\ {\bf{x}}\in\mathbb{R}^{s}.$ (4.10)

Clearly, the sequence ${{\cal{N}}}^{\cal{P}}_{n}({\bf{x}})$, $n\in\mathbb{N}$,
satisfies the recursions (3.1) and (3.2). The next theorem follows directly
from Theorems 3.4 and 3.6.

###### Theorem 4.5

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with the
Lipschitz constant $L$, ${\cal{P}}:\mathbb{R}^{l+\mu}\to\mathbb{R}^{l}$ is a
Lipchitz continuous pooling operator with the Lipchitz constant $P$,
$\mathbb{D}\subset\mathbb{R}^{l}$ is bounded, and $1\leq p\leq+\infty$. If the
sequences ${\bf{b}}_{n}\in\mathbb{R}^{l}$,
${\bf{W}}_{n}\in\mathbb{R}^{(l+\mu)\times l}$, $n\in\mathbb{N}$, converge in
the vector norms $\|\cdot\|_{p}$ and matrix norm $\|\cdot\|_{p}$,
respectively, with

$\lim_{n\to\infty}LP\|{\bf{W}}_{n}\|_{p}<1,$ (4.11)

then the neural networks ${\cal{N}}_{n}^{\cal{P}}$ converge uniformly in
$\mathbb{D}$.

Furthermore, if ${\bf{b}}_{n}$ and ${\bf{W}}_{n}$ converge to ${\bf{b}}^{*}$
and ${\bf{W}}^{*}$ exponentially, then the neural networks
${\cal{N}}_{n}^{\cal{P}}$ converge to a function ${\cal{N}}\in
C_{l}(\mathbb{D})$ exponentially and uniformly in $\mathbb{D}$.

Note that condition (4.11) for the average pooling and the max pooling is
simplified to

$\lim_{n\to\infty}L\|{\bf{W}}_{n}\|_{p}<1$

and

$\lim_{n\to\infty}L(\mu+1)^{1/p}\|{\bf{W}}_{n}\|_{p}<1,$

respectively, in light of (4.8) and (4.9). Theorem 4.5 extends the uniform
convergence result with pooling in [16] from contractive activation functions
to Lipschitz continuous activation functions. Moreover, the result for the max
pooling in [16] is for $p=+\infty$ only. While Theorem 4.5 for the max pooing
holds for all norms $\|\cdot\|_{p}$, $1\leq p\leq+\infty$.

##  5 Uniform Convergence of Deep Neural Networks with Bounded Widths

In this section, we extend the uniform convergence theorems established in the
last section for deep neural networks with weight matrices of a fixed width to
those with weight matrices of bounded widths.

We adopt the setting described in the beginning of section 2 and assume in
this section that the matrix widths $m_{n}$, $n\in\mathbb{N}$, of the deep
neural networks are bounded. Specifically, we assume
$l:=\max\\{m_{n}:n\in\mathbb{N}\\}<+\infty$. Thus, $1\leq m_{n}\leq l$ for all
$n\in\mathbb{N}$ and $m_{n_{0}}=l$ for some $n_{0}\in\mathbb{N}$. We define
the deep neural network ${\cal{N}}_{n}({\bf{x}})$ by (2.3). Clearly,
${\cal{N}}_{n}({\bf{x}})$ is a vector-valued function in $\mathbb{R}^{m_{n}}$.
The size of ${\cal{N}}_{n}({\bf{x}})$ varies according to $n$. The goal of
this section is to establish uniform convergence theorems of the sequence
${\cal{N}}_{n}({\bf{x}})$, $n\in\mathbb{N}$. The approach used in the last
section for convergence analysis is limited to deep neural networks with
weight matrices of a fixed width. To address this issue, we augment the matrix
${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times m_{n-1}}$ to
$\tilde{{\bf{W}}}_{n}\in\mathbb{R}^{l\times l}$ and the vector
${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$ to $\tilde{{\bf{b}}}_{n}\in\mathbb{R}^{l}$
by the zero-padding. That is, we let

$\tilde{{\bf{W}}}_{1}:=\left[\begin{array}[]{l}{\bf{W}}_{1}\\\ {\bf
0}_{(l-m_{1})\times s}\end{array}\right],$ (5.1)
$\tilde{{\bf{W}}}_{n}:=\left[\begin{array}[]{ll}{\bf{W}}_{n}&{\bf
0}_{m_{n}\times(l-m_{n-1})}\\\ {\bf 0}_{(l-m_{n})\times m_{n-1}}&{\bf
0}_{(l-m_{n})\times(l-m_{n-1})}\end{array}\right],\ \ n>1,$ (5.2)

and

$\tilde{{\bf{b}}}_{n}:=\left[\begin{array}[]{l}{\bf{b}}_{n}\\\ {\bf
0}_{l-m_{n}}\end{array}\right].$ (5.3)

We then define the deep neural network $\tilde{{\cal{N}}}_{n}({\bf{x}})$ by

$\tilde{{\cal{N}}}_{n}({\bf{x}}):=\left(\bigodot_{i=1}^{n}\sigma(\tilde{{\bf{W}}}_{i}\cdot+\tilde{{\bf{b}}}_{i})\right)({\bf{x}}),\
\ {\bf{x}}\in\mathbb{R}^{s}.$ (5.4)

Clearly, $\tilde{{\cal{N}}}_{n}({\bf{x}})$ is a vector-valued function in
$\mathbb{R}^{l}$, that is, for all $n\in\mathbb{N}$,
$\tilde{{\cal{N}}}_{n}({\bf{x}})$ have the same size $l$. Moreover,
$\tilde{{\cal{N}}}_{n}({\bf{x}})$ has the recursion

$\tilde{\cal{N}}_{1}({\bf{x}}):=\sigma(\tilde{\bf{W}}_{1}{\bf{x}}+\tilde{\bf{b}}_{1})$
(5.5)

and

$\tilde{{\cal{N}}}_{n+1}({\bf{x}})=\sigma(\tilde{{\bf{W}}}_{n+1}\tilde{{\cal{N}}}_{n}({\bf{x}})+\tilde{{\bf{b}}}_{n+1}),\
\ {\bf{x}}\in\mathbb{R}^{s},\ \ \mbox{for all}\ \ n\in\mathbb{N}.$ (5.6)

We need to define the uniform convergence of ${\cal{N}}_{n}({\bf{x}})$,
$n\in\mathbb{N}$.

###### Definition 5.1

We say that a sequence of vector-valued functions
${\cal{N}}_{n}:\mathbb{R}^{s}\to\mathbb{R}^{m_{n}}$, $n\in\mathbb{N}$,
converges uniformly to a function ${\cal{N}}\in C_{l}(\mathbb{D})$ if the
sequence of the augmented vector-valued functions
$\tilde{{\cal{N}}}_{n}({\bf{x}})$, $n\in\mathbb{N}$, converges uniformly to
${\cal{N}}$ in the space $C_{l}(\mathbb{D})$ in a norm on $\mathbb{R}^{l}$.

Likewise, we need to define the convergence of ${\bf{W}}_{n}$,
$n\in\mathbb{N}$.

###### Definition 5.2

We say that a sequence of matrices ${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times
m_{n-1}}$, $n\in\mathbb{N}$ converges to ${\bf{W}}\in\mathbb{R}^{l\times l}$
as $n\to\infty$ if

$\lim_{n\to\infty}\|\tilde{{\bf{W}}}_{n}-{\bf{W}}\|=0,$

where $\|\cdot\|$ is the induced matrix norm of a vector norm on
$\mathbb{R}^{l}$.

Convergence of a sequence of vectors ${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$,
$n\in\mathbb{N}$, can be understood as convergence of a sequence of
$m_{n}\times 1$ matrices.

We need to understand the relation between the norm of a matrix and that of
its augmented matrix with zero blocks. For $\mu,\nu\in\mathbb{N}$, suppose
that ${\bf{W}}\in\mathbb{R}^{\mu\times\nu}$ and for $l\geq\max\\{\mu,\nu\\}$
we let

$\tilde{{\bf{W}}}:=\left[\begin{array}[]{ll}{\bf{W}}&{\bf
0}_{\mu\times(l-\nu)}\\\ {\bf 0}_{(l-\mu)\times\nu}&{\bf
0}_{(l-\mu)\times(l-\nu)}\end{array}\right].$ (5.7)

In the next lemma, we show that the matrix augmentation process described
above preserves certain matrix norms.

###### Lemma 5.3

For $\mu,\nu\in\mathbb{N}$, suppose that $l\geq\max\\{\mu,\nu\\}$. If
$\tilde{{\bf{W}}}\in\mathbb{R}^{l\times l}$ is an augmented matrix of
${\bf{W}}\in\mathbb{R}^{\mu\times\nu}$ defined by (5.7) and $\|\cdot\|$ is the
matrix norm induced from a vector norm on $\mathbb{R}^{l}$ satisfying the
extension invariant (2.7) and the monotonicity condition (2.8), then

$\|\tilde{{\bf{W}}}\|=\|{\bf{W}}\|.$ (5.8)

Proof: By the definition (5.7) of $\tilde{{\bf{W}}}$, we have that

$\tilde{{\bf{W}}}\tilde{{\bf{x}}}=\left[\begin{array}[]{c}{\bf{W}}{\bf{x}}\\\
{\bf 0}_{l-\nu}\end{array}\right],\ \ \mbox{for all}\ \
\tilde{{\bf{x}}}:=\left[\begin{array}[]{c}{\bf{x}}\\\
{\bf{x}}^{\prime}\end{array}\right]\in\mathbb{R}^{l}\ \ \mbox{with}\ \
{\bf{x}}\in\mathbb{R}^{\nu},\ {\bf{x}}^{\prime}\in\mathbb{R}^{l-\nu}.$

Since the vector norm $\|\cdot\|$ satisfies (2.7), we obtain that

$\|\tilde{{\bf{W}}}\tilde{{\bf{x}}}\|=\|{\bf{W}}{\bf{x}}\|,\ \ \mbox{for all}\
\ \tilde{{\bf{x}}}:=\left[\begin{array}[]{c}{\bf{x}}\\\
{\bf{x}}^{\prime}\end{array}\right]\in\mathbb{R}^{l}\ \ \mbox{with}\ \
{\bf{x}}\in\mathbb{R}^{\nu},\ {\bf{x}}^{\prime}\in\mathbb{R}^{l-\nu}.$ (5.9)

The definition of the matrix norm induced from the vector norm together with
equation (5.9) ensures that

$\displaystyle\|\tilde{{\bf{W}}}\|$
$\displaystyle=\sup\left\\{\frac{\|{\bf{W}}{\bf{x}}\|}{\|\tilde{{\bf{x}}}\|}:\
\ \mbox{for all}\ \ \tilde{{\bf{x}}}:=\left[\begin{array}[]{c}{\bf{x}}\\\
{\bf{x}}^{\prime}\end{array}\right]\in\mathbb{R}^{l}\ \ \mbox{with}\ \
{\bf{x}}\in\mathbb{R}^{\nu},\
{\bf{x}}^{\prime}\in\mathbb{R}^{l-\nu}\right\\}.$ (5.12)

In the right-hand-side of equation (5.12), we restrict ${\bf{x}}^{\prime}={\bf
0}$ and since the vector norm satisfies the extension invariant property
(2.7), we note that $\|\tilde{{\bf{x}}}\|=\|{\bf{x}}\|$ with this restriction.
Therefore, we find that

$\|\tilde{{\bf{W}}}\|\geq\sup\left\\{\frac{\|{\bf{W}}{\bf{x}}\|}{\|{\bf{x}}\|}:{\bf{x}}\in\mathbb{R}^{\nu}\right\\}=\|{\bf{W}}\|.$
(5.13)

On the other hand, in light of the monotonicity condition (2.8) that the
vector norm satisfies, we observe for all
$\tilde{{\bf{x}}}:=\left[\begin{array}[]{c}{\bf{x}}\\\
{\bf{x}}^{\prime}\end{array}\right]\in\mathbb{R}^{l}$ with
${\bf{x}}\in\mathbb{R}^{\nu}$ and ${\bf{x}}^{\prime}\in\mathbb{R}^{l-\nu}$
that

$\|\tilde{{\bf{x}}}\|\geq\left\|\left[\begin{array}[]{c}{\bf{x}}\\\ {\bf
0}\end{array}\right]\right\|=\|{\bf{x}}\|,\ \ \mbox{where}\ \ {\bf
0}\in\mathbb{R}^{l-\nu}.$

This inequality together with (5.12) ensures that

$\|\tilde{{\bf{W}}}\|\leq\sup\left\\{\frac{\|{\bf{W}}{\bf{x}}\|}{\|{\bf{x}}\|}:{\bf{x}}\in\mathbb{R}^{\nu}\right\\}=\|{\bf{W}}\|.$
(5.14)

Combining inequalities (5.13) and (5.14) yields equation (5.8). $\Box$

Next, we present the following uniform convergence theorem for deep neural
networks with a Lipschitz continuous activation function and with weight
matrices of bounded widths.

###### Theorem 5.4

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with the
Lipschitz constant $L$, $\mathbb{D}\subset\mathbb{R}^{s}$ is bounded,
$l:=\sup\\{m_{i}:i\in\mathbb{N}\\}<+\infty$ and $\|\cdot\|$ is the matrix norm
induced from a vector norm satisfying the extension invariant condition (2.7)
and the monotonicity condition (2.8). If the sequences
${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$, ${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times
m_{n-1}}$, $n\in\mathbb{N}$, converge, and the sequence $\|{\bf{W}}_{n}\|$,
$n\in\mathbb{N}$, satisfies the condition (4.2), then the sequence
${\cal{N}}_{n}$, $n\in\mathbb{N}$, converges uniformly in $\mathbb{D}$.

Proof: According to Definition 5.1, it suffices to prove that
$\tilde{{\cal{N}}}_{n}:\mathbb{R}^{s}\to\mathbb{R}^{l}$, $n\in\mathbb{N}$,
converges uniformly. This is done by employing Theorem 4.3. By the hypothesis
of this theorem and Definition 5.2, we see that the sequences
$\tilde{{\bf{b}}}_{n}\in\mathbb{R}^{l}$,
$\tilde{{\bf{W}}}_{n}\in\mathbb{R}^{l\times l}$, $n\in\mathbb{N}$, converge.
It remains to prove that

$\lim_{n\to\infty}L\|\tilde{{\bf{W}}}_{n}\|<1.$ (5.15)

Because $\|\cdot\|$ is the matrix norm induced from a vector norm that
satisfies the extension invariant condition (2.7) and the monotonicity
condition (2.8), by equation (5.8) of Lemma 5.3, we conclude that

$\|\tilde{{\bf{W}}}_{n}\|=\|{\bf{W}}_{n}\|,\ \ \mbox{for all}\ \
n\in\mathbb{N}.$

This equation with the hypothesis that the sequence $\|{\bf{W}}_{n}\|$,
$n\in\mathbb{N}$, satisfies the condition (4.2) implies that inequality (5.15)
holds. Hence, the conclusion of this theorem follows directly from Theorem
4.3. $\Box$

We can obtain uniform convergence results for neural networks with Lipschitz
continuous activation functions and pooling, and with bounded matrix widths.
Moreover, the exponential convergence result for neural networks of this type
can also be obtained.

##  6 Deep Neural Networks with Unbounded Widths

We consider in this section deep neural networks with weight matrices of
unbounded widths. In this case, we suppose that
${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times m_{n-1}}$,
${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$, $n\in\mathbb{N}$, and a sequence of deep
neural networks ${\cal{N}}_{n}({\bf{x}})$, $n\in\mathbb{N}$, are defined by
(2.3). Then, the sequence ${\cal{N}}_{n}({\bf{x}})$, $n\in\mathbb{N}$,
satisfies (2.5). We further assume that the matrix widths $m_{n}$,
$n\in\mathbb{N}$, are unbounded. That is, there exists a subsequence
$m_{n_{i}}$, $i\in\mathbb{N}$, with $\lim_{i\to\infty}m_{n_{i}}=+\infty$. Due
to the unboundedness of the widths, the approach used in the last section is
not applicable. We will extend all vectors and matrices to elements in
sequence spaces $\ell_{p}(\mathbb{N})$ and $\ell_{p}(\mathbb{N}^{2})$,
respectively, and consider convergence in the sequence spaces.

We extend matrices ${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times m_{n-1}}$ to
$\tilde{{\bf{W}}}_{n}\in\ell_{p}(\mathbb{N}\times\mathbb{N}_{s})$ for $n=1$
and $\tilde{{\bf{W}}}_{n}\in\ell_{p}(\mathbb{N}^{2})$ for $n>1$, by the zero-
padding. Specifically, we set

$\tilde{{\bf{W}}}_{1}:=\left[\begin{array}[]{l}{\bf{W}}_{1}\\\ {\bf
0}_{\infty\times s}\end{array}\right]$ (6.1)

where ${\bf 0}_{\infty\times s}\in\ell_{p}(\mathbb{N}\times\mathbb{N}_{s})$ is
the zero semi-infinite matrix and

$\tilde{{\bf{W}}}_{n}:=\left[\begin{array}[]{ll}{\bf{W}}_{n}&{\bf
0}_{m_{n}\times\infty}\\\ {\bf 0}_{\infty\times m_{n-1}}&{\bf
0}_{\infty\times\infty}\end{array}\right],\ \ n>1,$ (6.2)

where ${\bf 0}_{\infty\times
m_{n-1}}\in\ell_{p}(\mathbb{N}\times\mathbb{N}_{m_{n-1}})$, ${\bf
0}_{{m_{n}}\times\infty}\in\ell_{p}(\mathbb{N}_{m_{n}}\times\mathbb{N})$, and
${\bf 0}_{\infty\times\infty}\in\ell_{p}(\mathbb{N}^{2})$ are the zero semi-
infinite matrices. We extend vectors ${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$ to
$\tilde{{\bf{b}}}_{n}\in\ell_{p}(\mathbb{N})$ by

$\tilde{{\bf{b}}}_{n}:=\left[\begin{array}[]{c}{\bf{b}}_{n}\\\ {\bf
0}_{\infty}\end{array}\right],$ (6.3)

where ${\bf 0}_{\infty}\in\ell_{p}(\mathbb{N})$ is the zero vector. We define
the deep neural network $\tilde{{\cal{N}}}_{n}({\bf{x}})$ by

$\tilde{{\cal{N}}}_{n}({\bf{x}}):=\left(\bigodot_{i=1}^{n}\sigma(\tilde{{\bf{W}}}_{i}\cdot+\tilde{{\bf{b}}}_{i})\right)({\bf{x}}),\
\ {\bf{x}}\in\mathbb{R}^{s}.$ (6.4)

Clearly, we have the recursion

$\tilde{\cal{N}}_{1}({\bf{x}}):=\sigma(\tilde{\bf{W}}_{1}{\bf{x}}+\tilde{\bf{b}}_{1})$
(6.5)

and

$\tilde{{\cal{N}}}_{n+1}({\bf{x}})=\sigma(\tilde{{\bf{W}}}_{n+1}\tilde{{\cal{N}}}_{n}({\bf{x}})+\tilde{{\bf{b}}}_{n+1}),\
\ {\bf{x}}\in\mathbb{R}^{s},\ \ \mbox{for all}\ \ n\in\mathbb{N}.$ (6.6)

It is important to understand the relation between
$\tilde{{\cal{N}}}_{n}({\bf{x}})$ and ${\cal{N}}_{n}({\bf{x}})$ for each
$n\in\mathbb{N}$. In this regard, we have the following fact.

###### Lemma 6.1

There holds the relation

$\tilde{{\cal{N}}}_{n}({\bf{x}})=\left[\begin{array}[]{c}{\cal{N}}_{n}({\bf{x}})\\\
\sigma({\bf 0}_{\infty})\end{array}\right].$ (6.7)

Proof: We prove equation (6.7) by induction on $n$. When $n=1$, by (6.1) and
by the definition of ${\cal{N}}_{1}({\bf{x}})$ and
$\tilde{{\cal{N}}}_{1}({\bf{x}})$, we observe for ${\bf{x}}\in\mathbb{R}^{s}$
that

$\tilde{{\cal{N}}}_{1}({\bf{x}})=\sigma\left(\left[\begin{array}[]{l}{\bf{W}}_{1}{\bf{x}}+{\bf{b}}_{1}\\\
{\bf
0}_{\infty}\end{array}\right]\right)=\left[\begin{array}[]{l}\sigma({\bf{W}}_{1}{\bf{x}}+{\bf{b}}_{1})\\\
\sigma({\bf
0}_{\infty})\end{array}\right]=\left[\begin{array}[]{l}{\cal{N}}_{1}({\bf{x}})\\\
\sigma({\bf 0}_{\infty})\end{array}\right].$

That is, equation (6.7) holds for $n=1$.

We assume that equation (6.7) holds for $n=k$ and proceed for the case
$n=k+1$. By recursion (6.6), the induction hypothesis, the definition (6.2) of
matrix $\tilde{{\bf{W}}}_{k+1}$, the definition (6.3) of vector
$\tilde{{\bf{b}}}_{k+1}$, and the recursion (2.5), we obtain that

$\displaystyle\tilde{{\cal{N}}}_{k+1}({\bf{x}})$
$\displaystyle=\sigma\left(\tilde{{\bf{W}}}_{k+1}\left[\begin{array}[]{c}{\cal{N}}_{k}({\bf{x}})\\\
{\bf 0}_{\infty}\end{array}\right]+\tilde{{\bf{b}}}_{k+1}\right)$
$\displaystyle=\sigma\left(\left[\begin{array}[]{l}{\bf{W}}_{k+1}{\cal{N}}_{k}({\bf{x}})\\\
{\bf 0}_{\infty}\end{array}\right]+\left[\begin{array}[]{l}{\bf{b}}_{k+1}\\\
{\bf 0}_{\infty}\end{array}\right]\right)$
$\displaystyle=\left[\begin{array}[]{l}\sigma({\bf{W}}_{k+1}{\cal{N}}_{k}({\bf{x}})+{\bf{b}}_{k+1})\\\
\sigma({\bf 0}_{\infty})\end{array}\right]$
$\displaystyle=\left[\begin{array}[]{l}{\cal{N}}_{k+1}({\bf{x}})\\\
\sigma({\bf 0}_{\infty})\end{array}\right].$

Thus, equation (6.7) holds for $n=k+1$. The induction principle ensures that
equation (6.7) holds for all $n\in\mathbb{N}$. $\Box$

Lemma 6.1 reveals that when $\sigma(0)=0$, which is satisfied by many
activation functions such as ReLU, hyperbolic tangent, the Gaussian error
linear unit, the exponential linear unit, the scaled exponential linear unit
and the sigmoid linear unit, we have that

$\tilde{{\cal{N}}}_{n}({\bf{x}})=\left[\begin{array}[]{l}{\cal{N}}_{n}({\bf{x}})\\\
{\bf 0}_{\infty}\end{array}\right],$ (6.8)

which is in $\ell_{p}(\mathbb{N})$ for all $1\leq p\leq+\infty$. However, when
$\sigma(0)\neq 0$, which includes sigmoid, softplus, and Gaussian,
$\tilde{{\cal{N}}}_{n}({\bf{x}})\notin\ell_{p}(\mathbb{N})$ for any
$p\in[1,+\infty)$, but
$\tilde{{\cal{N}}}_{n}({\bf{x}})\in\ell_{\infty}(\mathbb{N})$.

We now define the notion of uniform convergence of
${\cal{N}}_{n}\in\mathbb{R}^{m_{n}}$, $n\in\mathbb{N}$.

###### Definition 6.2

Let $\mathbb{D}\subset\mathbb{R}^{s}$ be bounded. We say that a sequence of
vector-valued functions ${\cal{N}}_{n}:\mathbb{R}^{s}\to\mathbb{R}^{m_{n}}$,
$n\in\mathbb{N}$, converges uniformly in $\mathbb{D}$ to a function
${\cal{N}}:\mathbb{R}^{s}\to\ell_{p}(\mathbb{N})$, for $1\leq p\leq\infty$, if
the sequence of the extended functions
$\tilde{{\cal{N}}}_{n}:\mathbb{R}^{s}\to\ell_{p}(\mathbb{N})$,
$n\in\mathbb{N}$, converges uniformly to ${\cal{N}}\in
C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$.

We need to define the convergence of ${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times
m_{n-1}}$, $n\in\mathbb{N}$.

###### Definition 6.3

We say that a sequence of matrices ${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times
m_{n-1}}$, $n\in\mathbb{N}$, converges to
${\bf{W}}\in\ell_{p}(\mathbb{N}^{2})$ as $n\to\infty$ if

$\lim_{n\to\infty}\|\tilde{{\bf{W}}}_{n}-{\bf{W}}\|_{\ell_{p}(\mathbb{N}^{2})}=0,$

where $\tilde{\bf{W}}_{n}$ is the extension of ${\bf{W}}_{n}$ by (6.1) and
(6.2).

Likewise, we need to define the convergence of
${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$, $n\in\mathbb{N}$.

###### Definition 6.4

We say that a sequence of vectors ${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$,
$n\in\mathbb{N}$, converges to ${\bf{b}}\in\ell_{p}(\mathbb{N})$ as
$n\to\infty$ if

$\lim_{n\to\infty}\|\tilde{{\bf{b}}}_{n}-{\bf{b}}\|_{\ell_{p}(\mathbb{N})}=0,$

where $\tilde{{\bf{b}}}_{n}$ is the extension of ${\bf{b}}_{n}$ by (6.3).

The next lemma concerns the uniform boundedness of the sequence of the deep
neural networks $\tilde{\cal{N}}_{n}({\bf{x}})$, $n\in\mathbb{N}$.

###### Lemma 6.5

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with a
Lipschitz constant $L$, the weight matrices
${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times m_{n-1}}$, for $n\in\mathbb{N}_{n}$
with $m_{0}:=s$, and the bias vectors ${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$ for
$n\in\mathbb{N}_{n}$. Suppose that $p\in[1,+\infty]$ if $\sigma(0)=0$ and
$p=+\infty$ if $\sigma(0)\neq 0$. If there exists a constant $c>0$ such that
$\|{\bf{b}}_{n}\|_{p}\leq c$, for all $n\in\mathbb{N}$ and the sequence
$\|{\bf{W}}_{n}\|$, $n\in\mathbb{N}$, satisfies the condition (4.2) and
$\mathbb{D}\subset\mathbb{R}^{s}$ is bounded, then there exists a positive
constant $\rho$ such that

$\|\tilde{{\cal{N}}}_{n}({\bf{x}})\|_{\ell_{p}(\mathbb{N})}\leq\rho,\ \
\mbox{for all}\ \ n\in\mathbb{N},\ {\bf{x}}\in\mathbb{D}.$

Proof: When $\sigma(0)=0$, by equation (6.7) of Lemma 6.1, we obtain that

$\|\tilde{{\cal{N}}}_{n}({\bf{x}})\|_{\ell_{p}(\mathbb{N})}=\|{\cal{N}}_{n}({\bf{x}})\|_{p},$
(6.9)

for all $p\in[1,+\infty]$. In this case, estimate (4.3) of Lemma 4.2 reduces
to

$\sup_{{\bf{x}}\in\mathbb{D}}\|{\cal{N}}_{n}({\bf{x}})\|_{p}\leq
c_{1}D+Lcc_{1},\ \ \mbox{for all}\ \ n\in\mathbb{N},$

where $D>0$ is an upper bound of $\mathbb{D}$. This estimate combined with
(6.9) ensures that
$\|\tilde{{\cal{N}}}_{n}({\bf{x}})\|_{\ell_{p}(\mathbb{N})}$ are uniformly
bounded.

When $\sigma(0)\neq 0$, by noticing that $\|\sigma({\bf
0}_{\infty})\|_{\ell_{\infty}(\mathbb{N})}=|\sigma(0)|$ and again by equation
(6.7), we find that

$\|\tilde{{\cal{N}}}_{n}({\bf{x}})\|_{\ell_{\infty}(\mathbb{N})}=\max\\{\|{\cal{N}}_{n}({\bf{x}})\|_{\infty},|\sigma(0)|\\}.$
(6.10)

Estimate (4.3) ensures that

$\sup_{{\bf{x}}\in\mathbb{D}}\|{\cal{N}}_{n}({\bf{x}})\|_{\infty}\leq
c_{1}D+Lcc_{2}+c_{1}|\sigma(0)|,\ \ \mbox{for all}\ \ n\in\mathbb{N}.$

Substituting this bound into the right-hand-side of equation (6.10) leads to
the boundedness of
$\sup_{{\bf{x}}\in\mathbb{D}}\|\tilde{{\cal{N}}}_{n}({\bf{x}})\|_{\ell_{\infty}(\mathbb{N})}$.
$\Box$

We need to understand the relation between the norm of a matrix and that of
its extension by the zero-padding. To this end, for $\mu,\nu\in\mathbb{N}$, we
suppose that ${\bf{W}}\in\mathbb{R}^{\mu\times\nu}$and we define the extension
by

$\tilde{{\bf{W}}}:=\left[\begin{array}[]{ll}{\bf{W}}&{\bf
0}_{\mu\times\infty}\\\ {\bf 0}_{\infty\times\nu}&{\bf
0}_{\infty\times\infty}\end{array}\right].$ (6.11)

In the next lemma, we show that the extension (6.11) preserves matrix norms.

###### Lemma 6.6

Let $p\in[1,+\infty]$. If $\tilde{{\bf{W}}}\in\ell_{p}(\mathbb{N}^{2})$ is the
extension of matrix ${\bf{W}}\in\mathbb{R}^{\mu\times\nu}$ defined by equation
(6.11), then

$\|\tilde{{\bf{W}}}\|_{\ell_{p}(\mathbb{N}^{2})}=\|{\bf{W}}\|_{p}.$ (6.12)

Proof: By definition (6.11) of the extension $\tilde{{\bf{W}}}$, we have that

$\tilde{{\bf{W}}}\tilde{{\bf{x}}}=\left[\begin{array}[]{c}{\bf{W}}{\bf{x}}\\\
{\bf 0}_{\infty}\end{array}\right],\ \ \mbox{for all}\ \
\tilde{{\bf{x}}}:=\left[\begin{array}[]{c}{\bf{x}}\\\
{\bf{x}}^{\prime}\end{array}\right]\in\ell_{p}(\mathbb{N})\ \ \mbox{with}\ \
{\bf{x}}\in\mathbb{R}^{\nu},\ {\bf{x}}^{\prime}\in\ell_{p}(\mathbb{N}).$

We obtain that

$\|\tilde{{\bf{W}}}\tilde{{\bf{x}}}\|_{\ell_{p}(\mathbb{N})}=\|{\bf{W}}{\bf{x}}\|_{p},\
\ \mbox{for all}\ \ \tilde{{\bf{x}}}:=\left[\begin{array}[]{c}{\bf{x}}\\\
{\bf{x}}^{\prime}\end{array}\right]\in\ell_{p}(\mathbb{N})\ \ \mbox{with}\ \
{\bf{x}}\in\mathbb{R}^{\nu},\ {\bf{x}}^{\prime}\in\ell_{p}(\mathbb{N}).$
(6.13)

The definition of the norm $\|\tilde{\bf{W}}\|_{\ell_{p}(\mathbb{N}^{2})}$
together with (6.13) ensures that

$\displaystyle\|\tilde{{\bf{W}}}\|_{\ell_{p}(\mathbb{N}^{2})}$
$\displaystyle=\sup\left\\{\frac{\|{\bf{W}}{\bf{x}}\|_{p}}{\|\tilde{{\bf{x}}}\|_{\ell_{p}(\mathbb{N})}}:\
\ \mbox{for all}\ \ \tilde{{\bf{x}}}:=\left[\begin{array}[]{c}{\bf{x}}\\\
{\bf{x}}^{\prime}\end{array}\right]\in\ell_{p}(\mathbb{N})\ \ \mbox{with}\ \
{\bf{x}}\in\mathbb{R}^{\nu},\
{\bf{x}}^{\prime}\in\ell_{p}(\mathbb{N})\right\\}.$ (6.16)

In the right-hand-side of equation (6.16), we restrict ${\bf{x}}^{\prime}={\bf
0}$ and note that
$\|\tilde{{\bf{x}}}\|_{\ell_{p}(\mathbb{N})}=\|{\bf{x}}\|_{p}$ with this
restriction. Thus, we find that

$\|\tilde{{\bf{W}}}\|_{\ell_{p}(\mathbb{N}^{2})}\geq\sup\left\\{\frac{\|{\bf{W}}{\bf{x}}\|_{p}}{\|{\bf{x}}\|_{p}}:{\bf{x}}\in\mathbb{R}^{\nu}\right\\}=\|{\bf{W}}\|_{p}.$
(6.17)

On the other hand, we observe for all
$\tilde{{\bf{x}}}:=\left[\begin{array}[]{l}{\bf{x}}\\\
{\bf{x}}^{\prime}\end{array}\right]\in\ell_{p}(\mathbb{N})$ with
${\bf{x}}\in\mathbb{R}^{\nu},\ {\bf{x}}^{\prime}\in\ell_{p}(\mathbb{N})$ that

$\|\tilde{{\bf{x}}}\|_{\ell_{p}(\mathbb{N})}\geq\left\|\left[\begin{array}[]{l}{\bf{x}}\\\
{\bf
0}_{\infty}\end{array}\right]\right\|_{\ell_{p}(\mathbb{N})}=\|{\bf{x}}\|_{p}.$

This inequality together with (6.16) ensures that

$\|\tilde{{\bf{W}}}\|_{\ell_{p}(\mathbb{N}^{2})}\leq\sup\left\\{\frac{\|{\bf{W}}{\bf{x}}\|_{p}}{\|{\bf{x}}\|_{p}}:{\bf{x}}\in\mathbb{R}^{\nu}\right\\}=\|{\bf{W}}\|_{p}.$
(6.18)

Combining inequalities (6.17) and (6.18) yields equation (6.12). $\Box$

With the help of Lemmas 6.5 and 6.6, we have the following uniform convergence
result for neural networks with weight matrices of unbounded widths.

###### Theorem 6.7

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with the
Lipschitz constant $L$, and $p\in[1,+\infty]$ if $\sigma(0)=0$ and $p=+\infty$
if $\sigma(0)\neq 0$. If the sequences ${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$,
$n\in\mathbb{N}$, and ${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times m_{n-1}}$,
$n\in\mathbb{N}$, converge, the sequence $\|{\bf{W}}_{n}\|$, $n\in\mathbb{N}$,
satisfies the condition (4.2), and $\mathbb{D}\subset\mathbb{R}^{s}$ is
bounded, then the sequence of neural networks ${\cal{N}}_{n}$ converges
uniformly in $\mathbb{D}$ to a function in
$C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$.

In addition, if ${\bf{b}}_{n}$ and ${\bf{W}}_{n}$ converge to ${\bf{b}}^{*}$
and ${\bf{W}}^{*}$ exponentially, then the sequence of neural networks
${\cal{N}}_{n}$ converges to a function ${\cal{N}}\in
C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$ exponentially and uniformly in
$\mathbb{D}$.

Proof: By Definition 6.2, it suffices to prove that the sequence
$\tilde{{\cal{N}}}_{n}({\bf{x}})$, $n\in\mathbb{N}$, converges uniformly in
$C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$. When $p$ satisfies the condition of
this theorem according to $\sigma(0)$, by Lemma 6.5,
$\sup_{{\bf{x}}\in\mathbb{D}}\|\tilde{\cal{N}}_{n}({\bf{x}})\|_{\ell_{p}(\mathbb{N})}$
is bounded for all $n\in\mathbb{N}$. Because the sequence $\|{\bf{W}}_{n}\|$,
$n\in\mathbb{N}$, satisfies the condition (4.2), according to Lemma 6.6, we
have that

$\lim_{n\to\infty}L\|\tilde{\bf{W}}_{n}\|_{\ell_{p}(\mathbb{N})}<1.$

Thus, Theorems 3.4 with ${\cal{P}}_{\mu}$ being the identity operator ensures
that the sequence $\tilde{\cal{N}}_{n}({\bf{x}})$, $n\in\mathbb{N}$, converges
uniformly in $C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$.

The second part of this theorem follows from Theorem 3.6 with
${\cal{P}}_{\mu}$ being the identity operator. $\Box$

Theorem 6.7 may be extended to neural networks with Lipschitz continuous
poolings and unbounded matrix widths.

##  7 Uniform Convergence of Convolutional Neural Networks

In this section, we establish uniform convergence results of convolutional
neural networks (CNNs). Pointwise convergence of CNNs with the ReLU activation
function was investigated in [36] by considering CNNs as deep neural networks
with weight matrices of increasing widths. We study uniform convergence of
CNNs with Lipschitz continuous activation functions by considering two types
of matrix extensions. We first consider the matrix extension by the zero-
padding which is described in the last section, and apply Theorem 6.7 to CNNs
with Lipschitz continuous activation functions to obtain a uniform convergence
result of CNNs. We then consider the extension of the weight matrices to semi-
infinite matrices by the constant-padding along the diagonals, which is
natural for CNNs, and present a uniform convergence theorem for CNNs, with a
weaker hypothesis on the filter mask.

We now recall the construction of CNNs. Given a vector
${\bf{x}}:=[x_{1},x_{2},\dots,x_{l}]^{\top}\in\mathbb{R}^{l}$ and a filter
mask ${\bf{w}}:=[w_{0},w_{1},\dots,w_{\tau}]^{\top}$, the convolution
${\bf{x}}*{\bf{w}}$ of ${\bf{x}}$ with ${\bf{w}}$ is a vector in
$\mathbb{R}^{l+\tau}$ defined by

$({\bf{x}}*{\bf{w}})_{i}:=\sum_{j=\max(0,i-l)}^{\min(i-1,\tau)}{\bf{w}}_{j}{\bf{x}}_{i-j},\
\ i\in\mathbb{N}_{l+\tau}.$

For $n\in\mathbb{N}$, by $\tau_{n}$ we denote a sequence of positive integers.
For each $n\in\mathbb{N}$, given a filter mask
${\bf{w}}^{(n)}:=[w^{(n)}_{0},w^{(n)}_{1},\dots,w^{(n)}_{\tau_{n}}]^{\top}\in\mathbb{R}^{\tau_{n}+1}$
and a bias vector ${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$, we construct a CNN by

${\cal{N}}_{n}({\bf{x}}):=\sigma({\cal{N}}_{n-1}({\bf{x}})*{\bf{w}}^{(n)}+{\bf{b}}_{n}),$
(7.1)

with ${\cal{N}}_{0}({\bf{x}}):={\bf{x}}$. In (7.1),
$\sigma:\mathbb{R}\to\mathbb{R}$ is a Lipschitz continuous activation function
with the Lipschitz constant $L$.

One can express the convolution ${\bf{x}}*{\bf{w}}$ via multiplication of
${\bf{x}}$ with an $(l+\tau)\times l$ Toeplitz matrix

$T_{ij}:=\left\\{\begin{aligned} w_{i-j},&\quad 1\leq j\leq i\leq j+\tau\leq
l+\tau,\\\ 0,&\quad\mbox{otherwise},\end{aligned}\right.$

in the form

${\bf{x}}*{\bf{w}}=T{\bf{x}}.$

Clearly, matrix $T$ has a specific form

$T=\left[\begin{array}[]{ccccccc}w_{0}&0&0&0&\cdots&\cdots&0\\\
w_{1}&w_{0}&0&0&\cdots&\cdots&0\\\
\vdots&\ddots&\ddots&\ddots&\ddots&\ddots&\vdots\\\
w_{\tau}&w_{\tau-1}&\cdots&w_{0}&0&\cdots&0\\\
0&w_{\tau}&\cdots&w_{1}&w_{0}&0\cdots&0\\\
\vdots&\ddots&\ddots&\ddots&\ddots&\ddots&\vdots\\\
\cdots&\cdots&0&w_{\tau}&w_{\tau-1}&\cdots&w_{0}\\\
\cdots&\cdots&\cdots&0&w_{\tau}\cdots&\cdots&w_{1}\\\
\vdots&\ddots&\ddots&\ddots&\ddots&\ddots&\vdots\\\ 0&\cdots&\cdots&\cdots
0&\cdots&w_{\tau}&w_{\tau-1}\\\ 0&\cdots&\cdots&\cdots&\cdots&\cdots
0&w_{\tau}\end{array}\right].$

That is, we map the filter mask ${\bf{w}}$ to the Toeplitz matrix $T$ whose
diagonal and the first to the $\tau$th sub-diagonals are the components of
${\bf{w}}$. The CNNs ${\cal{N}}_{n}$ may be written in a form of the DNN with
increasing widths:

$m_{0}:=s,\ \ m_{n}:=m_{n-1}+\tau_{n},\ \ n\in\mathbb{N}.$ (7.2)

Introducing weight matrices ${\bf{W}}_{n}\in\mathbb{R}^{m_{n}\times m_{n-1}}$
defined by

$({\bf{W}}_{n})_{ij}:=\left\\{\begin{aligned} w^{(n)}_{i-j},&\quad 1\leq j\leq
i\leq j+\tau_{n}\leq m_{n-1}+\tau_{n},\\\
0,&\quad\mbox{otherwise},\end{aligned}\right.$ (7.3)

we have for $n\in\mathbb{N}$ that

${\cal{N}}_{n}({\bf{x}})=\sigma({\bf{W}}_{n}{\cal{N}}_{n-1}({\bf{x}})+{\bf{b}}_{n}),\
\ {\bf{x}}\in\mathbb{R}^{s}.$ (7.4)

Clearly, from (7.2), the CNNs ${\cal{N}}_{n}$ are deep neural networks with
unbounded widths.

We first consider uniform convergence of CNNs ${\cal{N}}_{n}$ in the sense of
Definition 6.2. To this end, we extend ${\bf{W}}_{1}$, ${\bf{W}}_{n}$, $n\geq
2$, and ${\bf{b}}_{n}$, $n\in\mathbb{N}$, to
$\tilde{{\bf{W}}}_{1}\in\ell_{p}(\mathbb{N}\times\mathbb{N}_{s})$,
$\tilde{{\bf{W}}}_{n}\in\ell_{p}(\mathbb{N}\times\mathbb{N})$, $n\geq 2$, and
$\tilde{{\bf{b}}}_{n}\in\ell_{p}(\mathbb{N})$, $n\in\mathbb{N}$, respectively,
by (6.1), (6.2) and (6.3). We then define $\tilde{{\cal{N}}}_{n}({\bf{x}})$ by
(6.6) and note that $\tilde{{\cal{N}}}_{n}({\bf{x}})$ is an extension of
${\cal{N}}_{n}({\bf{x}})$, that results from the zero-padding. We confine
ourselves to filter masks of a fixed length. Uniform convergence of CNNs in
this sense follows directly from Theorem 6.7.

We need the following preliminary result.

###### Lemma 7.1

Suppose that $\tau$ is a positive integer and $\tau_{n}=\tau$, for all
$n\in\mathbb{N}$. If $w_{i}^{(n)}$ satisfies the condition

$w_{i}^{(n)}=o\left(1\right),\ \ \mbox{for}\ \ i=0,1,\dots,\tau,$ (7.5)

then there hold

(i) $\tilde{{\bf{W}}}_{n}$, $n\in\mathbb{N}$, is a Cauchy sequence;

(ii) $\lim_{n\to+\infty}L\|{\bf{W}}_{n}\|_{p}<1$ for all $p\in[1,+\infty]$,
where $L$ is the Lipschitz constant of $\sigma$.

Proof: Suppose that $n$ and $m$ are arbitrary positive integers with $n>1$. A
direct computation leads to

$\|\tilde{{\bf{W}}}_{n+m}-\tilde{{\bf{W}}}_{n}\|_{\ell_{1}(\mathbb{N})}\leq\sum_{k=0}^{\tau}|w_{k}^{(n+m)}-w_{k}^{(n)}|+\sum_{k=0}^{\tau}|w_{k}^{(n+m)}|$
(7.6)

and

$\|\tilde{{\bf{W}}}_{n+m}-\tilde{{\bf{W}}}_{n}\|_{\ell_{\infty}(\mathbb{N})}\leq\sum_{k=0}^{\tau}|w_{k}^{(n+m)}-w_{k}^{(n)}|+\sum_{k=0}^{\tau}|w_{k}^{(n+m)}|.$
(7.7)

By the interpolation theorem of Mitjagin [21], it follows from inequalities
(7.6) and (7.7) for all $p\in[1,\infty]$ that

$\|\tilde{{\bf{W}}}_{n+m}-\tilde{{\bf{W}}}_{n}\|_{\ell_{p}(\mathbb{N})}\leq\sum_{k=0}^{\tau}|w_{k}^{(n+m)}-w_{k}^{(n)}|+\sum_{k=0}^{\tau}|w_{k}^{(n+m)}|.$
(7.8)

Let $\epsilon>0$ be arbitrarily small. According to hypothesis (7.5), there
exists $N\in\mathbb{N}$ such that for all $n,m>N$, the right-hand-side of
(7.8) is bounded by $\epsilon$, which ensures that the sequence
$\tilde{\bf{W}}_{n}$, $n\in\mathbb{N}$, is Cauchy in spaces
$\ell_{p}(\mathbb{N})$, for $p\in[1,+\infty]$.

It remains to prove (ii). According to the Riesz-Thorin interpolation theorem,
we observe for all $p\in[1,+\infty]$ that

$\|{\bf{W}}_{n}\|_{p}\leq\sum_{k=0}^{\tau}|w_{k}^{(n)}|.$

Invoking hypothesis (7.5) in the right-hand-side of the above estimate, we
conclude for all $p\in[1,+\infty]$ that for $0<\epsilon_{0}<1$, there exists
$N\in\mathbb{N}$ with $L\leq N$ such that for all $n>N$

$L\|{\bf{W}}_{n}\|_{p}\leq L\sum_{k=0}^{\tau}|w_{k}^{(n)}|<\epsilon_{0}<1,$

which implies (ii). $\Box$

Lemma 7.1 enables us to derive the next theorem.

###### Theorem 7.2

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with a
Lipschitz constant $L$, $\tau_{n}=\tau$, for all $n\in\mathbb{N}$, with $\tau$
being a positive integer, and $\mathbb{D}$ is a bounded set in
$\mathbb{R}^{s}$. If the sequences ${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$,
$n\in\mathbb{N}$, converges and $w_{i}^{(n)}$ satisfies the condition (7.5),
then the CNNs ${\cal{N}}_{n}$ converge uniformly on $\mathbb{D}$ to a function
in $C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$, for $p\in[1,+\infty]$ if
$\sigma(0)=0$ and $p=+\infty$ if $\sigma(0)\neq 0$.

Proof: The hypothesis of this theorem together with Lemma 7.1 ensures that the
assumption of Theorem 6.7 is satisfied. Hence, the result of this theorem
follows directly from Theorem 6.7. $\Box$

The exponential convergence of the CNNs that we present next follows directly
from the second part of Theorem 6.7.

###### Theorem 7.3

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with a
Lipschitz constant $L$, $\tau_{n}=\tau$, for all $n\in\mathbb{N}$, with $\tau$
being a positive integer, and $\mathbb{D}$ is a bounded set in
$\mathbb{R}^{s}$. If the sequences ${\bf{b}}_{n}\in\mathbb{R}^{m_{n}}$,
$n\in\mathbb{N}$, converges exponentially and $w_{i}^{(n)}$ satisfies the
condition

$w_{i}^{(n)}={\cal O}\left(r^{n}\right),\ \ \mbox{for some }\ \ 0<r<1\mbox{
and for all}\ \ i=0,1,\dots,\tau,$ (7.9)

then the neural networks ${\cal{N}}_{n}$ converge uniformly and exponentially
on $\mathbb{D}$ to a function in $C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$, for
$p\in[1,+\infty]$ if $\sigma(0)=0$ and $p=+\infty$ if $\sigma(0)\neq 0$.

Proof: Condition (7.9) implies that hypothesis (7.5) is satisfied, which in
turn ensures that Lemma 7.1 holds true. Moreover, it implies that the sequence
${\bf{W}}_{n}$, $n\in\mathbb{N}$, converges exponentially. The result of this
theorem follows directly from the second part of Theorem 6.7. $\Box$

Hypothesis (7.5) on the filter mask may be relaxed if a different extension of
matrices ${\bf{W}}_{n}$, $n\geq 2$, is adopted. We next present a relaxation
on the hypothesis (7.5) of the filter mask. To this end, instead of using the
zero-padding adopted previously for matrix extension, we adopt the constant
padding along each of the diagonal and sub-diagonals. Specifically, we extend
${\bf{W}}_{n}$, $n\geq 2$, to $\bar{{\bf{W}}}_{n}\in\ell_{p}(\mathbb{N}^{2})$
by

$(\bar{{\bf{W}}}_{n})_{ij}:=\begin{cases}w^{(n)}_{i-j},&0\leq
i-j\leq\tau_{n}\\\ 0,&\mbox{otherwise}.\end{cases}$ (7.10)

Note that the matrix $\bar{\bf{W}}_{n}$ is a semi-infinite Toeplitz matrix
whose diagonal and the first to the $\tau_{n}$-th sub-diagonals are the
components of
${\bf{w}}^{(n)}:=[w^{(n)}_{0},w^{(n)}_{1},\dots,w^{(n)}_{\tau_{n}}]^{\top}$.

We define $\bar{{\cal{N}}}_{n}({\bf{x}})$ by (6.4) with $\tilde{{\bf{W}}}_{n}$
being replaced by $\bar{{\bf{W}}}_{n}$ and $\bar{{\cal{N}}}_{n}({\bf{x}})$ is
an extension of ${\cal{N}}_{n}({\bf{x}})$, different from
$\tilde{{\cal{N}}}_{n}({\bf{x}})$. Then, $\bar{{\cal{N}}}_{n}({\bf{x}})$,
$n\in\mathbb{N}$, satisfy the recursions (3.1) and (3.2).

###### Definition 7.4

We say that a sequence of vector-valued functions
${\cal{N}}_{n}:\mathbb{R}^{s}\to\mathbb{R}^{m_{n}}$, $n\in\mathbb{N}$,
converges uniformly in $\mathbb{D}$ to a function
${\cal{N}}:\mathbb{R}^{s}\to\ell_{p}(\mathbb{N})$ if the functions
$\bar{{\cal{N}}}_{n}:\mathbb{R}^{s}\to\ell_{p}(\mathbb{N})$, $n\in\mathbb{N}$,
converges uniformly to ${\cal{N}}$ in the space
$C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$.

We next study the uniform convergence of
${\cal{N}}_{n}:\mathbb{R}^{s}\to\mathbb{R}^{m_{n}}$, $n\in\mathbb{N}$. We
first transfer a condition on the masks to the weight matrices.

###### Lemma 7.5

If a sequence of masks
$[w^{(n)}_{0},w^{(n)}_{1},\dots,w^{(n)}_{k}]\in\mathbb{R}^{k+1}$,
$n\in\mathbb{N}$, satisfies the conditions that for each $k=0,1,\dots,\tau$,
the sequence $w_{k}^{(n)}$, $n\in\mathbb{N}$, converges in $\mathbb{R}$ and

$\lim_{n\to\infty}L\sum_{k=0}^{\tau}|w^{(n)}_{k}|<1,$ (7.11)

then $\lim_{n\to\infty}\|\bar{\bf{W}}_{n}\|_{\ell_{p}(\mathbb{N}^{2})}$ exists
and

$\lim_{n\to\infty}L\|\bar{\bf{W}}_{n}\|_{\ell_{p}(\mathbb{N}^{2})}<1.$ (7.12)

Proof: First, we prove that
$\lim_{n\to\infty}\|\bar{\bf{W}}_{n}\|_{\ell_{p}(\mathbb{N}^{2})}$ exists.
Since for each $k=0,1,\dots,\tau$, the sequence $w_{k}^{(n)}$,
$n\in\mathbb{N}$, converges in $\mathbb{R}$, it is a Cauchy sequence in
$\mathbb{R}$. We next show that the sequence $\bar{\bf{W}}_{n}$,
$n\in\mathbb{N}$, is Cauchy in $\ell_{p}(\mathbb{N}^{2})$. To this end, we
suppose that $n,m\in\mathbb{N}$ are arbitrary. A direct computation leads to

$\|\bar{{\bf{W}}}_{n+m}-\bar{{\bf{W}}}_{n}\|_{\ell_{1}(\mathbb{N})}=\sum_{k=0}^{\tau}|w_{k}^{(n+m)}-w_{k}^{(n)}|$
(7.13)

and

$\|\bar{{\bf{W}}}_{n+m}-\bar{{\bf{W}}}_{n}\|_{\ell_{\infty}(\mathbb{N})}=\sum_{k=0}^{\tau}|w_{k}^{(n+m)}-w_{k}^{(n)}|.$
(7.14)

Again, by the interpolation theorem of Mitjagin [21], it follows from
equations (7.13) and (7.14) for all $p\in[1,\infty]$ that

$\|\bar{{\bf{W}}}_{n+m}-\bar{{\bf{W}}}_{n}\|_{\ell_{p}(\mathbb{N})}\leq\sum_{k=0}^{\tau}|w_{k}^{(n+m)}-w_{k}^{(n)}|.$
(7.15)

Let $\epsilon>0$ be arbitrarily small. Since for each $k=0,1,\dots,\tau$, the
sequence $w^{(n)}_{k}$ converges, there exists $N\in\mathbb{N}$ such that for
all $n,m>N$, the right-hand-side of (7.15) is bounded by $\epsilon$, which
ensures that $\bar{\bf{W}}_{n}$, $n\in\mathbb{N}$, is Cauchy. Moreover, since
for all $n,m\in\mathbb{N}$

$\left|\|\bar{\bf{W}}_{n}\|_{\ell_{p}(\mathbb{N}^{2})}-\|\bar{\bf{W}}_{n+m}\|_{\ell_{p}(\mathbb{N}^{2})}\right|\leq\|\bar{\bf{W}}_{n}-\bar{\bf{W}}_{n+m}\|_{\ell_{p}(\mathbb{N}^{2})},$

the sequence $\|\bar{\bf{W}}_{n}\|_{\ell_{p}(\mathbb{N}^{2})}$,
$n\in\mathbb{N}$, is Cauchy in $\mathbb{R}$. Therefore, this sequence has a
limit in $\mathbb{R}$.

Likewise, by direct computation, we find that

$\|\bar{{\bf{W}}}_{n}\|_{\ell_{1}(\mathbb{N}^{2})}=\sum_{k=0}^{\tau}|w_{k}^{(n)}|$
(7.16)

and

$\|\bar{{\bf{W}}}_{n}\|_{\ell_{\infty}(\mathbb{N}^{2})}=\sum_{k=0}^{\tau}|w_{k}^{(n)}|.$
(7.17)

Again, by the interpolation theorem of Mitjagin, it follows from equations
(7.16) and (7.17) for all $p\in[1,\infty]$ that

$\|\bar{{\bf{W}}}_{n}\|_{\ell_{p}(\mathbb{N}^{2})}\leq\sum_{k=0}^{\tau}|w_{k}^{(n)}|.$
(7.18)

Convergence of the sequence
$\|\bar{{\bf{W}}}_{n}\|_{\ell_{p}(\mathbb{N}^{2})}$, $n\in\mathbb{N}$,
together with inequality (7.18) and hypothesis (7.11) leads to (7.12). $\Box$

We next establish the uniform boundedness of $\bar{\cal{N}}_{n}({\bf{x}})$,
$n\in\mathbb{N}$.

###### Lemma 7.6

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with a
Lipschitz constant $L$ and $\mathbb{D}\subseteq\mathbb{R}^{s}$ is bounded. If
there exists a constant $c>0$ such that
$\|\tilde{\bf{b}}_{n}\|_{\ell_{p}(\mathbb{N})}\leq c$, for all
$n\in\mathbb{N}$ and the sequence of masks
$[w^{(n)}_{0},w^{(n)}_{1},\dots,w^{(n)}_{k}]\in\mathbb{R}^{k+1}$,
$n\in\mathbb{N}$, satisfies condition that for each $k=0,1,\dots,\tau$, the
sequence $w_{k}^{(n)}$, $n\in\mathbb{N}$, converges in $\mathbb{R}$ and
(7.11), then there exist positive constants $c_{1},c_{2}$ such that

$\|\bar{\cal{N}}_{n}({\bf{x}})\|_{\ell_{p}(\mathbb{N})}\leq
c_{1}c_{2}+Lcc_{2}+c_{2}\max\\{\|\sigma({\bf
0}_{\infty})\|_{\ell_{p}(\mathbb{N})}:j\in\mathbb{N}_{n}\\},\ \ \mbox{for
all}\ \ n\in\mathbb{N},\ {\bf{x}}\in\mathbb{D}.$ (7.19)

Moreover, there exists a constant $\rho>0$ such that
$\|\bar{\cal{N}}_{n}({\bf{x}})\|_{\ell_{p}(\mathbb{N})}\leq\rho$, for all
$n\in\mathbb{N}$ and ${\bf{x}}\in\mathbb{D}$, for $p\in[1,+\infty]$ if
$\sigma(0)=0$ and for $p=+\infty$ if $\sigma(0)\neq 0$.

Proof: By Lemma 7.5, we have that inequality (7.12) holds true. This together
with Lemma 3.7 with ${\cal{P}}_{\mu}$ being the identity operator ensures that
estimate (7.19) is satisfied. It follows from (7.19) that
$\|\bar{\cal{N}}_{n}({\bf{x}})\|_{\ell_{p}(\mathbb{N})}$ is bounded for all
$n\in\mathbb{N}$ and ${\bf{x}}\in\mathbb{D}$, for $p\in[1,+\infty]$ if
$\sigma(0)=0$ and for $p=+\infty$ if $\sigma(0)\neq 0$. $\Box$

With the above preparation, we are ready to present our second uniform
convergence theorem for CNNs.

###### Theorem 7.7

Suppose that $\sigma:\mathbb{R}\to\mathbb{R}$ is Lipschitz continuous with a
Lipschitz constant $L$ and $\mathbb{D}\subseteq\mathbb{R}^{s}$ is an arbitrary
bounded set. If the sequence ${\bf{b}}_{n}$, $n\in\mathbb{N}$, converges, and
for each $k=0,1,\dots,\tau$, $\\{w^{(n)}_{k}\\}$ is convergent and satisfies
hypothesis (7.11) then the neural networks ${\cal{N}}_{n}$ converge uniformly
to a function in $C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$ for all
$p\in[1,\infty]$ if $\sigma(0)=0$ and for $p=+\infty$ if $\sigma(0)\neq 0$.

Moreover, if the sequences ${\bf{b}}_{n}$, and $w^{(n)}_{k}$,
$n\in\mathbb{N}$, for all $k=0,1,\dots,\tau$, converge exponentially, then the
uniform convergence of ${\cal{N}}_{n}$ to a function in
$C_{\ell_{p}(\mathbb{N})}(\mathbb{D})$ is exponential, for all
$p\in[1,\infty]$ if $\sigma(0)=0$ and for $p=+\infty$ if $\sigma(0)\neq 0$.

Proof: Since for all $k=0,1,\dots,\tau$, the sequence $w^{(n)}_{k}$,
$n\in\mathbb{N}$, satisfies hypothesis (7.11), we see that Lemmas 7.5 and 7.6
hold true. This theorem follows from Theorems 3.4 and 3.6 with
${\cal{P}}_{\mu}$ being the identity operator. $\Box$

Theorem 7.7 may be further extended to CNNs with pooling and we leave this to
the interested reader.

## References

  * [1] M. Artzrouni, On the convergence of infinite products of matrices, Linear Algebra Appl. 74 (1986), 11–21.
  * [2] D.-A. Clevert, T. Unterthiner and S. Hochreiter, Fast and accurate deep network learning by exponential linear units (elus), International Conference on Learning Representations (ICLR), 2015/11/23.
  * [3] P. L. Combettes and J.-C. Pesquet, Lipschitz certificates for layered network structures driven by averaged activation operators, SIAM J. Math. Data Sci. 2 (2020), no. 2, 529–557.
  * [4] I. Daubechies, Ten Lectures on Wavelets, SIAM, Philadelphia, 1992.
  * [5] I. Daubechies, R. DeVore, S. Foucart, B. Hanin, and G. Petrova, Nonlinear approximation and (deep) ReLU networks, Constr. Approx. 55 (2022), 127–172.
  * [6] R. DeVore, B. Hanin, and G. Petrova, Neural network approximation, Acta Numer. 30 (2021), 327–444.
  * [7] W. E and Q. Wang, Exponential convergence of the deep neural network approximation for analytic functions, Sci. China Math. 61 (2018), no. 10, 1733–-1740.
  * [8] D. Elbrächter, D. Perekrestenko, P. Grohs, and H. Bölcskei, Deep neural network approximation theory, arXiv:1901.02220.
  * [9] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, Cambridge, 2016.
  * [10] B. Hanin and M. Nica, Finite depth and width corrections to the neural tangent kernel, arXiv:1909.05989, 2019.
  * [11] M. Hasannasab, J. Hertrich, S. Neumayer, G. Plonka, S. Setzer, and G. Steidl, Parseval proximal neural networks, J. Fourier Anal. Appl. 26 (2020), no. 4, Paper No. 59, 31 pp.
  * [12] K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification, Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1026-1034.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 770–778.
  * [14] K. He, X. Zhang, S. Ren, and J. Sun, Identity mappings in deep residual networks, In: B. Leibe, J. Matas, N. Sebe, M. Welling (eds) Computer Vision – ECCV 2016, Lecture Notes in Computer Science, vol. 9908, Springer, Cham.
  * [15] J. Huang, Y. Jiao, Z. Li, S. Liu, Y. Wang, and Y. Yang, An error analysis of generative adversarial networks for learning distributions, J. Mach. Learn. Res. 23 (2022), Paper No. 116, 43 pp.
  * [16] W. Huang, Y. Xu and H. Zhang, Uniform convergence of deep neural networks with contractive activation functions and poolings, preprint, 2022, (arXiv preprint arXiv:2205.06570).
  * [17] A. Jacot, F. Gabriel, and C. Hongler, Neural tangent kernel: convergence and generalization in neural networks, 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.
  * [18] G. Klambauer, T. Unterthiner, A. Mayr, S. Hochreiter, Self-normalizing neural networks, Advances in Neural Information Processing Systems 30 (2017).
  * [19] P. D. Lax, Functional Analysis, Wiley-Interscience, New York, 2002.
  * [20] Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature 521 (2015), no. 7553, 436–-444, 2015.
  * [21] B. S. Mitjagin, An interpolation theorem for modular spaces (Russian), Mat. Sb., New Series, 66 (108) (1965), 473–482.
  * [22] H. Montanelli and Q. Du, New error bounds for deep ReLU networks using sparse grids, SIAM J. Math. Data Sci. 1 (2019), 10.1137/18M1189336.
  * [23] H. Montanelli and H. Yang, Error bounds for deep ReLU networks using the Kolmogorov–Arnold superposition theorem, Neural Networks 129 (2020), 1–6.
  * [24] Q. Nguyen, M. Mondelli, and G. F. Montufar, Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep ReLU networks, Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 8119–8129, 2021.
  * [25] T. Poggio, H. Mhaskar, L. Rosasco, B. Miranda, and Q. Liao, Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review, Internat. J. Automat. Comput. 14 (2017), 503–519.
  * [26] K. Scaman and A. Virmaux, Lipschitz regularity of deep neural networks: analysis and efficient estimation, 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.
  * [27] Z. Shen, H. Yang, and S. Zhang, Deep network approximation characterized by number of neurons, Commun. Comput. Phys. 28 (2020), no. 5, 1768–1811.
  * [28] Z. Shen, H. Yang, and S. Zhang, Deep network with approximation error being reciprocal of width to power of square root of depth, Neural Comput. 33 (2021), no. 4, 1005–1036.
  * [29] Z. Shen, H. Yang, and S. Zhang, Optimal approximation rate of ReLU networks in terms of width and depth, J. Math. Pures Appl. 157 (2022), 101–135.
  * [30] E. Stein and R. Shakarchi, Fourier Analysis. An introduction, Princeton University Press, Princeton, NJ, 2003.
  * [31] Y. Wang, A mathematical introduction to generative adversarial nets (GAN), arXiv:2009.00169, 2020.
  * [32] J. H. M. Wedderburn, Lectures on Matrices, Dover, New York, 1964.
  * [33] Y. Xu, Multi-grade deep learning, arXiv preprint arXiv:2302.00150.
  * [34] Y. Xu, Successive affine learning for deep neural networks, arXiv preprint arXiv:2305.07996.
  * [35] Y. Xu and H. Zhang, Convergence of deep ReLU networks, arXiv preprint arXiv:2107.12530.
  * [36] Y. Xu and H. Zhang, Convergence of deep convolutional neural networks, Neural Networks, 153 (2022), 553–563.
  * [37] D. Yarotsky, Error bounds for approximations with deep relu networks, Neural Networks 94 (2017), 103–114.
  * [38] T. Zaslavsky, Facing up to arrangements: face-count formulas for partitions of space by hyperplanes, Mem. Amer. Math. Soc. 1 (1975), issue 1, no. 154.
  * [39] D. X. Zhou, Universality of deep convolutional neural networks, Appl. Comput. Harmon. Anal. 48 (2020), no. 2, 787–794.
  * [40] D. Zou, R. Balan, and M. Singh, On Lipschitz bounds of general convolutional neural networks, IEEE Trans. Inform. Theory 66 (2020), no. 3, 1738–-1759.