# Asymptotic Analysis of Deep Residual Networks111Alain Rossier’s research was
supported through EPSRC Centre for Doctoral Training in Mathematics of Random
Systems: Analysis, Modelling and Simulation (EP/S023925/1).

Rama Cont1, Alain Rossier1,2 and Renyuan Xu3

1 Mathematical Institute, University of Oxford 2 Instadeep Ltd
3 Department of Industrial and Systems Engineering, University of Southern
California
Email addresses<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We investigate the asymptotic properties of deep Residual networks (ResNets)
as the number of layers increases. We first show the existence of scaling
regimes for trained weights markedly different from those implicitly assumed
in the neural ODE literature. We study the convergence of the hidden state
dynamics in these scaling regimes, showing that one may obtain an ODE, a
stochastic differential equation (SDE) or neither of these. In particular, our
findings point to the existence of a diffusive regime in which the deep
network limit is described by a class of stochastic differential equations
(SDEs). Finally, we derive the corresponding scaling limits for the
backpropagation dynamics.

###### Contents

  1. 1 Introduction
  2. 2 Scaling regimes
    1. 2.1 Scaling regimes for trained network weights
    2. 2.2 Smoothness of weights with respect to the layer
  3. 3 Scaling behavior of trained weights: numerical experiments
    1. 3.1 Methodology
    2. 3.2 Results for fully-connected layers
    3. 3.3 Results for convolutional networks
    4. 3.4 Summary: three scaling regimes
  4. 4 Deep network limit
    1. 4.1 Scaling regime 1: ODE limit
    2. 4.2 Scaling regime 2
    3. 4.3 Link with numerical experiments
    4. 4.4 Detailed proofs
  5. 5 Asymptotic analysis of the backpropagation dynamics
    1. 5.1 Backpropagation in supervised learning
    2. 5.2 Backward equation for the Jacobian under Scaling regime 1
    3. 5.3 Backward equation for the Jacobian under Scaling regime 2
    4. 5.4 Proofs
  6. A Hyperparameters
  7. B Auxiliary lemma

##  1 Introduction

Residual networks, or ResNets, are multilayer neural network architectures in
which a skip connection is introduced at every layer ([13]). This allows very
deep networks to be trained by circumventing vanishing and exploding
gradients, mentioned in [3]. The increased depth in ResNets has lead to
commensurate performance gains in applications ranging from speech recognition
[14, 33] to computer vision [13, 16].

A residual network with $L$ layers may be represented as

$h^{(L)}_{k+1}=h^{(L)}_{k}+\delta^{(L)}_{k}\sigma_{d}\left(A^{(L)}_{k}h^{(L)}_{k}+b^{(L)}_{k}\right),$
(1.1)

where $h^{(L)}_{k}$ is the hidden state at layer $k=0,\ldots,L$,
$h^{(L)}_{0}=x\in\mathbb{R}^{d}$ the input, $h^{(L)}_{L}\in\mathbb{R}^{d}$ the
output, $\sigma\colon\mathbb{R}\to\mathbb{R}$ a nonlinear activation function,
$\sigma_{d}(x)=(\sigma(x_{1}),\ldots,\sigma(x_{d}))^{\top}$ its component-wise
extension to $x\in\mathbb{R}^{d}$, and $A_{k}^{(L)}$, $b_{k}^{(L)}$, and
$\delta_{k}^{(L)}$ trainable network weights for $k=0,\ldots,L-1$.

ResNets have been the focus of several theoretical studies due to a perceived
link with a class of differential equations. The idea, put forth in [12] and
[4], is to view (1.1) as a discretization of a system of ordinary differential
equations

$\frac{\mathrm{d}H_{t}}{\mathrm{d}t}=\sigma_{d}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}H_{t}+\overline{b}_{t}\right),$ (1.2)

where $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu\colon[0,1]\to\mathbb{R}^{d\times d}$ and
$\overline{b}\colon[0,1]\to\mathbb{R}^{d}$ are appropriate smooth functions
and $H(0)=x$. This may be justified ([31]) by assuming that

$\delta^{(L)}\sim 1/L,\quad A_{k}^{(L)}\sim\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{k/L},\quad
b_{k}^{(L)}\sim\overline{b}_{k/L}$ (1.3)

as $L$ increases. Such models, named neural ordinary differential equations or
neural ODEs [4, 7], have motivated the use of optimal control methods to train
ResNets [8].

However, the precise link between deep ResNets and the neural ODE (1.2) is
unclear: in practice, the weights $A^{(L)}$ and $b^{(L)}$ result from
training, yet the validity of the scaling assumptions (1.3) for trained
weights is far from obvious. As a matter of fact, there is empirical evidence
showing that using a scaling factor $\delta^{(L)}\sim 1/L$ can deteriorate the
network accuracy [2]. Also, there is no guarantee that weights obtained
through training have a non-zero limit which depends smoothly on the layer, as
(1.3) would require. In fact, for ResNet architectures used in practice,
empirical evidence points to the contrary [5]. These observations motivate an
in-depth examination of the actual scaling behavior of weights with network
depth in ResNets and of its impact on the asymptotic behavior of those
networks.

Contributions. We systematically investigate the scaling behavior of trained
networks weights and examine in detail the consequence of this scaling for the
asymptotic properties of ResNets as the number of layers increases. We first
show, through detailed numerical experiments, the existence of scaling regimes
for trained weights markedly different from those implicitly assumed in the
neural ODE literature. We study the convergence of the hidden state dynamics
in these scaling regimes, showing that one may obtain an ODE, a stochastic
differential equation (SDE) or neither of these. More precisely, we show
strong convergence of the hidden state dynamics to a limiting ODE or SDE, by
viewing the discrete hidden state dynamics as a “nonlinear Euler scheme” of
the limiting equation. At a mathematical level, we extend the convergence
analysis of Higham et al. [15] for discretization schemes of time-homogeneous
(Markov) diffusions to a class of nonlinear approximations for Itô processes
with bounded coefficients.

In particular, our findings point to the existence of a “diffusive regime” in
which the deep network limit is described by a class of stochastic
differential equations (SDEs). These novel findings on the relation between
ResNets and neural ODEs complement previous work [31, 9, 11, 25, 29]. Finally,
we derive the corresponding scaling limit for the backpropagation dynamics.
The results we obtain are different from previous ones on asymptotics of
ResNets [4, 12, 23], and correspond to a different scaling regime which is
relevant for trained weights in practical settings.

In particular, in the diffusive regime we find a limit different from the
“Neural SDE” literature [21]. Indeed, we observe that the Jacobian of the
output with respect to the hidden states depends on hidden states across all
levels, so may not be directly expressed as the solution of a forward or
backward stochastic differential equation, as proposed in [21]. However, in
Section 5 we obtain a representation for the asymptotics of the
backpropagation dynamics in terms of an auxiliary forward SDE.

Outline. Section 2 describes the various scaling regimes for trained weights
evidenced in [5] and the methodology for studying this scaling behaviour in
the deep network limit. In Section 3, we report detailed numerical experiments
on the scaling of trained network weights across a range of ResNet
architectures and datasets, showing the existence of at least three different
scaling regimes, none of which correspond to (1.3). In Section 4, we show that
under these scaling regimes, the dynamics of the the hidden state may be
described in terms of a class of ordinary or stochastic differential
equations, different from the neural ODEs studied in [4, 12, 23]. In Section
5, we derive the large depth limit of the backpropagation dynamics under each
scaling regime.

Notations. Let $\|v\|$ denote the Euclidean norm of a vector $v$. For a matrix
$M$, denote $M^{\top}$ its transpose, $\mathrm{diag}(M)$ its diagonal vector,
$\mathrm{tr}(M)$ its trace and $\left\lVert
M\right\rVert_{F}=\sqrt{\mathrm{tr}(M^{\top}M)}$ its Frobenius norm. Denote
$\left\lfloor x\right\rfloor$ the integer part of a real number $x$. Let
$\mathcal{N}(m,\Sigma)$ denote the Gaussian distribution with mean $m$ and
(co)variance $\Sigma$, $\otimes$ denote the tensor product, and
$\mathbb{R}^{d,\,\otimes n}=\mathbb{R}^{d}\times\cdots\times\mathbb{R}^{d}$
($n$ times). Define the vectorisation operator by ${\rm
vec}\colon\mathbb{R}^{d_{1}{\times\cdots\times}d_{n}}\to\mathbb{R}^{d_{1}\cdots
d_{n}}$, and let $\mathbbm{1}_{S}$ be the indicator function of a set $S$.
$\mathcal{C}^{0}$ is the space of continuous functions, for $\nu\geq 0$,
$\mathcal{C}^{\nu}$ is the space of $\nu$-Hölder continuous functions, and
$\mathcal{H}^{1}$ is the Sobolev space of order $1$.

##  2 Scaling regimes

We start by providing a framework for describing the scaling regimes for
trained network weights, as identified in numerical experiments on deep
ResNets [5].

###  2.1 Scaling regimes for trained network weights

As described in Section 1, the neural ODE limit assumes

$\delta^{(L)}\sim\frac{1}{L}\quad{\rm and}\quad A^{(L)}_{\left\lfloor
Lt\right\rfloor}\overset{L\to\infty}{\longrightarrow}\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{t},\quad
b^{(L)}_{\left\lfloor
Lt\right\rfloor}\overset{L\to\infty}{\longrightarrow}\overline{b}_{t}$ (2.1)

for $t\in[0,1]$, where $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu\colon[0,1]\to\mathbb{R}^{d\times d}$ and
$\overline{b}\colon[0,1]\to\mathbb{R}^{d}$ are smooth functions [31]. Our
numerical experiments, detailed in Section 3, show that the norm of the
weights generally shrinks as $L$ increases (see for example Figures 2 and 4),
so one cannot expect the above assumption to hold, unless weights are
renormalized in some way. We consider here a more general assumption which
includes (2.1) but allows for shrinking weights.

######  Scaling regime 1.

There exist $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu\in\mathcal{C}^{0}\left([0,1],\mathbb{R}^{d\times d}\right)$ and
$\beta\in\left[0,1\right]$ such that

$\forall s\in[0,1],\qquad\mkern 1.0mu\overline{\mkern-1.5muA\mkern
0.0mu}\mkern 0.0mu_{s}=\lim_{L\to\infty}L^{\beta}\,A^{(L)}_{\left\lfloor
Ls\right\rfloor}.$ (2.2)

Figure 1: Trained weights as a function of $k/L$ for $k=0,\ldots,L$ and
$L=9100$. Left: rescaled weights $L^{\beta}A^{(L)}_{k,(0,0)}$ for a $\tanh$
network with $\beta=0.2$. Right: cumulative sum
$\sum_{j=0}^{k-1}A^{(L)}_{j,(0,0)}$for a $\mathrm{ReLU}$ network. Note that
each $A^{(L)}_{k,(0,0)}\in\mathbb{R}$.

These renormalized weights do converge to a continuous function of the layer
in some cases, as shown in Figure 1 (top) which displays a ResNet (1.1) with
fully connected layers and $\tanh$ activation function, without explicit
regularization (see Section 3.2).

Yet, it is not the case that network weights always converge to a smooth
function of the layer, even after rescaling. Indeed, network weights
$A_{k}^{(L)}$ are usually initialized to random, independent and identically
distributed (i.i.d.) values, whose scaling limit would then correspond to a
white noise, which cannot be represented as a function of the layer. In this
case, the cumulative sum $\sum_{j=0}^{k-1}A^{(L)}_{j}$ of the weights behaves
like a random walk, which does have a well-defined scaling limit
$W\in\mathcal{C}^{0}\left(\left[0,1\right],\mathbb{R}^{d\times d}\right)$.
Figure 1 (bottom) shows that, for a $\mathrm{ReLU}$ ResNet with fully-
connected layers, this cumulative sum of trained weights converges to an
irregular, that is, non-smooth function of the layer.

This observation motivates the consideration of a different scaling regime
where the weights $A^{(L)}_{k}$ are represented as the increments of a
continuous function $W^{A}$, i.e. the cumulative sum of the weights may
converges to a limit but not the weight themselves. We also allow for a trend
term as in Scaling regime 1.

######  Scaling regime 2.

There exist $\beta\in\left[0,1\right)$, $\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu\in\mathcal{C}^{0}\left([0,1],\mathbb{R}^{d\times d}\right)$, and
$W^{A}\in\mathcal{C}^{0}([0,1],\mathbb{R}^{d\times d})$ non-zero such that
$W^{A}_{0}=0$ and

$A_{k}^{(L)}=L^{-\beta}\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{k/L}+W^{A}_{(k+1)/L}-W^{A}_{k/L}.$ (2.3)

The above decomposition is unique. Indeed, for $s\in[0,1]$,

$\displaystyle L^{\beta-1}\sum_{k=0}^{\left\lfloor
Ls\right\rfloor-1}A_{k}^{(L)}$ $\displaystyle=L^{-1}\sum_{k=0}^{\left\lfloor
Ls\right\rfloor-1}\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{k/L}+L^{\beta-1}W^{A}_{\left\lfloor
Ls\right\rfloor/L}\mathop{\to}^{L\to\infty}\int_{0}^{s}\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{r}\mathrm{d}r.$ (2.4)

The integral of $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu$
is thus uniquely determined by the weights $A_{k}^{(L)}$, so $\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu$ can be obtained by
discretization and $W^{A}$ by fitting the residual error in (2.4). In
addition, Scaling Regimes 1 and 2 are mutually exclusive since Scaling regime
2 requires $W^{A}$ to be non-zero.

######  Remark 1.

In the case of independent Gaussian weights

$A_{k,mn}^{(L)}\overset{{\tiny\text{i.i.d}}}{\sim}\mathcal{N}\left(0,L^{-1}d^{-2}\right)\quad\text{and}\quad
b_{k,n}^{(L)}\overset{{\tiny\text{i.i.d}}}{\sim}\mathcal{N}\left(0,L^{-1}d^{-1}\right),$

where $A_{k,mn}^{(L)}$ is the $(m,n)$-th entry of
$A_{k}^{(L)}\in\mathbb{R}^{d\times d}$ and $b_{k,n}^{(L)}$ is the $n$-th entry
of $b_{k}^{(L)}\in\mathbb{R}^{d}$, we can represent the weights
$\\{A^{(L)},b^{(L)}\\}$ as the increments of a matrix-valued Brownian motion

$A_{k}^{(L)}=d^{-1}\left(W^{A}_{(k+1)/L}-W^{A}_{k/L}\right),$

which is a special case of Scaling regime 2.

This remark shows that Scaling regime 2 corresponds to a ’diffusive’ regime.

###  2.2 Smoothness of weights with respect to the layer

A question related to the existence of a scaling limit is the degree of
smoothness of the limits $\mkern 1.0mu\overline{\mkern-1.5muA\mkern
0.0mu}\mkern 0.0mu$ or $W^{A}$, if they exist. To quantify the smoothness of
the function mapping the layer number to the corresponding network weight, we
define in Table 2.2 several quantities which may be viewed as discrete
versions of various (semi-)norms used to measure the smoothness of functions.

Table 1: Quantities associated to a tensor $A^{(L)}\in\mathbb{R}^{L\times
d\times d}$.

Quantity |  Definition
---|---
Maximum norm |  $\max_{k}\left\lVert A^{(L)}_{k}\right\rVert_{F}$
Cumulative sum norm |  $\left\lVert\sum_{k=1}^{L}A_{k}^{(L)}\right\rVert_{F}$
$\beta$-scaled norm of increments |  $L^{\beta}\max_{k}\left\lVert A^{(L)}_{k+1}-A^{(L)}_{k}\right\rVert_{F}$
Root sum of squares |  $\left(\sum_{k}\left\lVert A^{(L)}_{k}\right\rVert_{F}^{2}\right)^{1/2}$

##  3 Scaling behavior of trained weights: numerical experiments

We now report on detailed numerical experiments to investigate the scaling
properties and asymptotic behavior of trained weights for residual networks as
the number of layers increases. We focus on two types of architectures: fully-
connected and convolutional networks.

###  3.1 Methodology

We underline that Scaling Regimes 1 and 2 are mutually exclusive since Scaling
regime 2 requires $W^{A}$ to be non-zero. In order to examine whether one of
these scaling regimes, or neither, holds for the trained weights $A^{(L)}$ and
$b^{(L)}$, we proceed as follows.

Step 1: First, to obtain the scaling exponent $\beta\in[0,1)$, note that under
Scaling regime 2,

$L^{\beta-1}\sum_{k=1}^{L}A_{k}^{(L)}=\frac{1}{L}\sum_{k=1}^{L}\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{k/L}+L^{\beta-1}W^{A}_{1}\qquad\mathop{\to}^{L\to\infty}\int_{0}^{1}\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{s}\mathrm{d}s.$

Hence, we perform a logarithmic regression of the cumulative sum norm of
$A^{(L)}$ with respect to $L$, and the rate of increase of
$\sum_{k=1}^{L}A_{k}^{(L)}$ as $L\to\infty$ is $1-\beta$.

Step 2: After identifying the correct scale $L^{-\beta}$ for the weights, we
compute the $\beta$-scaled norm of increments of $A^{(L)}$ to check whether
they satisfy Scaling regime 1 and measure the smoothness of the trained
weights. On one hand, if the $\beta$-scaled norm of increments of $A^{(L)}$
does not vanish as $L\to\infty$, it means that the rescaled weights cannot be
represented as a continuous function of the layer, as in Scaling regime 1. On
the other hand, if the $\beta$-scaled norm of increments of $A^{(L)}$ vanishes
(say, as $L^{-\nu}$) when $L$ increases, it supports Scaling regime 1 with a
Hölder-continuous limit function $\mkern 1.0mu\overline{\mkern-1.5muA\mkern
0.0mu}\mkern 0.0mu\in\mathcal{C}^{\nu}([0,1],\mathbb{R}^{d\times d})$.

Step 3: To discriminate between Scaling regimes 1 and 2, we decompose the
cumulative sum $\sum_{j=0}^{k-1}A_{j}^{(L)}$ of the trained weights into a
trend component $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu$
and a noise component $W^{A}$, as shown in (2.4). The presence of non-
negligible noise term $W^{A}$ favors Scaling regime 2.

Step 4: Finally, we estimate the regularity of the term $W^{A}$ under Scaling
regime 2. If $W^{A}$ has diffusive behavior, as in the example of i.i.d.
random weights, then its quadratic variation tensor defined by

$\displaystyle\left[W^{A}\right]_{s}$
$\displaystyle=\lim_{L\to\infty}\sum_{k=0}^{\left\lfloor
Ls\right\rfloor-1}\left(W^{A}_{\frac{k+1}{L}}-W^{A}_{\frac{k}{L}}\right)\otimes\left(W^{A}_{\frac{k+1}{L}}-W^{A}_{\frac{k}{L}}\right)^{\top}$

has a finite limit as $L\to\infty$. Hence, using (2.3) and Cauchy-Schwarz, we
obtain

${\left|\kern-1.07639pt\left|\kern-1.07639pt\left|\left[W^{A}\right]_{s}\right|\kern-1.07639pt\right|\kern-1.07639pt\right|}\leq
2\cdot\lim_{L\to\infty}\sum_{k=0}^{\left\lfloor Ls\right\rfloor-1}\left\lVert
A^{(L)}_{k}\right\rVert_{F}^{2}+L^{1-2\beta}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu\right\rVert^{2}_{L^{2}}$
(3.1)

where
${\left|\kern-1.07639pt\left|\kern-1.07639pt\left|\cdot\right|\kern-1.07639pt\right|\kern-1.07639pt\right|}$
is the Hilbert-Schmidt norm. As $\mkern 1.0mu\overline{\mkern-1.5muA\mkern
0.0mu}\mkern 0.0mu$ is continuous on a compact domain, its $L^{2}$ norm is
finite. Hence, if we have $\beta\geq\frac{1}{2}$, the fact that the root sum
of squares of $A^{(L)}$ is upper bounded as $L\to\infty$ implies that the
quadratic variation of $W^{A}$ is finite.

We follow all of the above steps for $b^{(L)}$ as well. Note that the scaling
exponent $\beta$ may not be the same for $A^{(L)}$ and $b^{(L)}$.

######  Remark 2.

Note that $\sigma=\mathrm{ReLU}$ is homogeneous of degree $1$, so we can write

$\delta\cdot\sigma_{d}\left(Ah+b\right)=\mathrm{sign}(\delta)\cdot\sigma_{d}\left(\left|\delta\right|Ah+\left|\delta\right|b\right).$

Hence, when analyzing the scaling of trained weights in the case of a
$\mathrm{ReLU}$ activation with fully-connected layers, we look at the
quantities $\left|\delta^{(L)}\right|A^{(L)}$ and
$\left|\delta^{(L)}\right|b^{(L)}$, as they represent the total scaling of the
residual connection.

###  3.2 Results for fully-connected layers

We first consider the case where the network layers are fully-connected. We
consider the network architecture (1.1) for two different setups:

  1. (i)

$\sigma=\tanh$, $\delta^{(L)}_{k}=\delta^{(L)}\in\mathbb{R}_{+}$ trainable,

  2. (ii)

$\sigma=\mathrm{ReLU}$, $\delta^{(L)}_{k}\in\mathbb{R}$ trainable.

We choose to present these two cases for the following reasons. First, both
$\tanh$ and $\mathrm{ReLU}$ are widely used in practice. Further, having
$\delta^{(L)}$ scalar makes the derivation of the limiting behavior simpler.
Also, since $\tanh$ is an odd function, the sign of $\delta^{(L)}$ can be
absorbed into the activation. Therefore, we can assume that $\delta^{(L)}$ is
non-negative for $\tanh$. Regarding $\mathrm{ReLU}$, having a shared
$\delta^{(L)}$ would hinder the expressiveness of the network. Indeed, if for
instance $\delta^{(L)}>0$, we would get $h^{(L)}_{k+1}\geq h^{(L)}_{k}$
element-wise since $\mathrm{ReLU}$ is non-negative. This would imply that
$h^{(L)}_{L}\geq x$, which is not desirable. The same argument applies to the
case $\delta^{(L)}<0$. Thus, we let $\delta^{(L)}_{k}\in\mathbb{R}$ depend on
the layer number for $\mathrm{ReLU}$ networks.

We consider two data sets. The first one is synthetic: fix $d=10$ and generate
$N$ i.i.d samples $x_{i}$ coming from the $d-$dimensional uniform distribution
in $[-1,1]^{d}$. Let $K=100$ and simulate the following dynamical system:

$\begin{cases}z_{0}^{x_{i}}&=x_{i}\\\
z_{k}^{x_{i}}&=z_{k-1}^{x_{i}}+K^{-1/2}\tanh_{d}\left(g_{d}\left(z_{k-1}^{x_{i}},k,K\right)\right),\,\,\,k=1,\ldots,K,\end{cases}$

where $g_{d}(z,k,K)\coloneqq\sin(5k\pi/K)z+\cos(5k\pi/K)\mathbbm{1}_{d}$. The
targets $y_{i}$ are defined as $y_{i}=z_{K}^{x_{i}}/\left\lVert
z_{K}^{x_{i}}\right\rVert$. The motivation behind this low-dimensional dataset
is to be able to train very deep residual networks and to be sure that there
exists at least a (sparse) optimal solution.

The second dataset is a low-dimensional embedding of the MNIST handwritten
digits dataset [20]. Let $\left(\widetilde{x},c\right)\in\mathbb{R}^{28\times
28}\times\left\\{0,\ldots,9\right\\}$ be an input image and its corresponding
class. We transform $\widetilde{x}$ into a lower dimensional embedding
$x\in\mathbb{R}^{d}$ using an untrained convolutional projection, where
$d=25$. More precisely, we stack two convolutional layers initialized
randomly, we apply them to the input and we flatten the downsized image into a
$d-$dimensional vector. Doing so reduces the dimensionality of the problem
while allowing very deep networks to reach at least $99\%$ training accuracy.
The target $y\in\mathbb{R}^{d}$ is the one-hot encoding of the corresponding
class.

The weights are updated by stochastic gradient descent (SGD) using batches of
size $B$ on the mean-square loss and a constant learning rate $\eta$, until
the loss falls below $\epsilon$, or when the maximum number of updates
$T_{\max}$ is reached. We repeat the experiments for depths $L$ varying from
$L_{\min}$ to $L_{\max}$. Details are given in Appendix A.

Results. For the case of a $\tanh$ activation (i), we observe in Figure 2 that
for both datasets, $\delta^{(L)}\sim L^{-0.7}$ clearly decreases as $L$
increases, and $A^{(L)}$ decreases slightly when $L$ increases. We deduce that
$\beta=0.3$ for the MNIST dataset and $\beta=0.2$ for the synthetic dataset.

Figure 2: Scaling for $\tanh$ activation and $\delta^{(L)}\in\mathbb{R}$.
Left: Maximum norm of $\delta^{(L)}$ with respect to $L$. Right: Cumulative
sum norm of $A^{(L)}$ with respect to $L$. The dashed lines are for the
synthetic data and the solid lines are for MNIST. The plots are in log-log
scale.

We use these results to identify the scaling behavior of $A^{(L)}$. We observe
in Figure 3 (left) that the $\beta$-scaled norm of increments of $A^{(L)}$
decreases like $L^{-1/2}$, suggesting that Scaling regime 1 holds, with
$\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu$ being
$1/2-$Hölder continuous. This is confirmed in Figure 3 (right), as the trend
part $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu$ is visibly
continuous and even of class $\mathcal{C}^{1}$. The noise part $W^{A}$ is
negligible. This observation is even more striking given that the weights are
trained without explicit regularization.

Figure 3: Identification of scaling behavior in the case of $\tanh$ activation
and $\delta^{(L)}\in\mathbb{R}$. Left: log-log plot of root sum of squares of
$A^{(L)}$ (pink) and the $\beta$-scaled norm of increments of $A^{(L)}$
(orange). Dashed lines are for the synthetic data and the solid lines are for
MNIST. Right: Decomposition of the trained weights $A^{(L)}_{k,(9,7)}$ with
the trend part $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu$
and the noise part $W^{A}$ for $L=10321$, as defined in (2.3), for the
synthetic dataset.

Regarding the case of a $\mathrm{ReLU}$ activation function (ii), we observe
in Figure 4 (left) that the trend part of the residual connection
$\left|\delta^{(L)}\right|A^{(L)}$ scales like $L^{-0.8}$ for the synthetic
dataset and like $L^{-0.9}$ for the MNIST dataset. We see in Figure 4 (right)
that keeping the sign of $\delta^{(L)}_{k}$ is important, as the sign
oscillates considerably throughout the network depth $k=0,\ldots,L-1$.

Figure 4: Scaling for $\mathrm{ReLU}$ activation and
$\delta_{k}^{(L)}\in\mathbb{R}$. Left: Cumulative sum norm of
$|\delta^{(L)}|A^{(L)}$ with respect to $L$, in log-log scale. Right: trained
values of $\delta^{(L)}_{k}$ as a function of $k$, for $L=9100$ and for the
synthetic dataset.

Figure 5 (left) shows that the $\beta$-scaled norm of increments diverges as
the depth increases. This suggests that there exists a noise part $W^{A}$.
Following (3.1), the fact that the root sum of squares of
$\left|\delta^{(L)}\right|A^{(L)}$ is upper bounded as $L\to\infty$ and
$\beta\geq 1/2$ implies that $W^{A}$ has finite quadratic variation. These
claims are also supported by Figure 5 (right): there is a non-zero trend part
$\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu$, and a non-
negligible noise part $W^{A}$.

Figure 5: $\mathrm{ReLU}$ activation and scalar $\delta_{k}^{(L)}$. Left: in
pink we plot in log-log scale the root sum of squares of
$|\delta^{(L)}|A^{(L)}$, and in orange the $\beta$-scaled norm of increments
of $|\delta^{(L)}|A^{(L)}$. The dashed lines are for the synthetic data and
the solid lines for MNIST. Right: Decomposition of the trained weights
$|\delta^{(L)}|\,A^{(L)}_{k,(7,7)}$ with the trend part $\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu$ and the noise part
$W^{A}$ for $L=10321$, as defined in (2.3), for the synthetic dataset.

Given the scaling behavior of the trained weights, we conclude that Scaling
regime 1 seems to be a plausible description for the $\tanh$ case (i), but
Scaling regime 2 provides a better description for the $\mathrm{ReLU}$ case
(ii).

Scaling behavior of $b^{(L)}$ are shown for the $\tanh$ case in Figure 6 and
for the $\mathrm{ReLU}$ case in Figure 7. We observe that the cumulative sum
norm, the scaled norm of the increments and the root sum of squares of
$b^{(L)}$ scales in the same way as $A^{(L)}$ as the depth $L$ increases. In
particular, the scaling exponent $\beta$ for $b^{(L)}$ is equal to the scaling
exponent of $A^{(L)}$, justifying the setup considered in Section 2.

Figure 6: Scaling behavior for $b^{(L)}$ with $\tanh$ activation and scalar
$\delta^{(L)}$. Left: cumulative sum norm of $b^{(L)}$ with respect to $L$, in
log-log scale. Middle: the root sum of squares of $b^{(L)}$ in pink and the
$\beta-$scaled norm of increments of $b^{(L)}$ in orange, in log-log scale.
The dashed lines are for the synthetic data and the solid lines are for MNIST.
Right: Decomposition of the trained weights $b^{(L)}_{k,5}$ with the trend
part $\overline{b}$ and the noise part $W^{b}$ for $L=10321$, as defined in
(2.3), for the synthetic dataset.

Figure 7: Scaling and hypothesis verification for $b^{(L)}$ with
$\mathrm{ReLU}$ activation and $\delta_{k}^{(L)}\in\mathbb{R}$. Left:
cumulative sum norm of $|\delta^{(L)}|b^{(L)}$ with respect to $L$, in log-log
scale. Middle: the root sum of squares of $|\delta^{(L)}|b^{(L)}$ in pink and
the $\beta-$scaled norm of increments of $|\delta^{(L)}|b^{(L)}$ in orange, in
log-log scale. The dashed lines are for the synthetic data and the solid lines
for MNIST. Right: Decomposition of the trained weights
$|\delta^{(L)}|\,b^{(L)}_{k,6}$ with the trend part $\overline{b}$ and the
noise part $W^{b}$ for $L=10321$, as defined in (2.3), for the synthetic
dataset.

Importance of the stochastic term $W^{A}$. It is legitimate to ask whether the
noise term $W^{A}$ plays a significant role in the output accuracy of the
network. To test this, we create a residual network with denoised weights
$\widetilde{A}^{(L)}_{k}\coloneqq L^{-\beta}\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{k/L}$, compute its
training error and we compare it to the original training error. We observe in
Figure 8 (left) that for $\tanh$, the noise part $W^{A}$ is negligible and
does not influence the loss. However, for $\mathrm{ReLU}$, the loss with
denoised weights is one order of magnitude above the original training loss,
meaning that the noise part $W^{A}$ plays a significant role in the accuracy
of the trained network.

Figure 8: Loss value, as a function of $L$, in black for the trained weights
$A_{k}^{(L)}$ and in green for the denoised weights
$\widetilde{A}^{(L)}_{k}=L^{-\beta}\mkern 1.0mu\overline{\mkern-1.5muA\mkern
0.0mu}\mkern 0.0mu_{k/L}$. Left: $\tanh$ activation and
$\delta^{(L)}\in\mathbb{R}$. Right: $\mathrm{ReLU}$ activation and
$\delta_{k}^{(L)}\in\mathbb{R}$. Note that these curves are for the synthetic
dataset and that we plot them in log-log scale. Also, we show in off-white the
loss value range in which we consider our networks to have converged.

Sensitivity of $\alpha$ and $\beta$ with respect to the hyperparameters. The
values of $\alpha$ and $\beta$ stem from the trained weights, which are
themselves a function of the initialization and the training algorithm. We are
using stochastic gradient descent, and the most significant hyperparameters of
SGD are the learning rate $\eta$ and the batch size $B$.
Hence, we report the value $\alpha$ and $\beta$ found for the $\tanh$ and
trainable $\delta$ architecture on the synthetic data with different batch
sizes $B\in\left\\{8,32,128\right\\}$ and learning rates
$\eta\in\left\\{0.01,0.003,0.001\right\\}$, with $5$ different realizations
for the initialization. We report the average values of $\alpha$ and $\beta$
for $5$ different seeds in Table 2 below.

Table 2: Average value of $\alpha$ (left) and $\beta$ (centre) for the trained
weights, over $5$ random initialization. $\eta$ is the learning rate, $B$ the
batch size.

$\alpha$ | $B=8$ | $B=32$ | $B=128$
---|---|---|---
$\eta=.01$ | $.69\pm.02$ | $.73\pm.02$ | $.67\pm.02$
$\eta=.003$ | $.59\pm.05$ | $.60\pm.01$ | $.58\pm.01$
$\eta=.001$ | $.58\pm.01$ | $.55\pm.01$ | $.53\pm.01$

$\beta$ | $B=8$ | $B=32$ | $B=128$
---|---|---|---
$\eta=.01$ | $.24\pm.02$ | $.29\pm.05$ | $.22\pm.02$
$\eta=.003$ | $.33\pm.01$ | $.41\pm.06$ | $.40\pm.02$
$\eta=.001$ | $.39\pm.02$ | $.43\pm.02$ | $.41\pm.01$

We observe that the learning rate does affect $\alpha$ and $\beta$ while
keeping $\alpha+\beta$ around $1$, and the batch size does not affect $\alpha$
or $\beta$. A plausible explanation for these observations is that a higher
batch size means a more precise descent direction at the cost of efficiency,
but the shape of the solution is not supposed to change.

###  3.3 Results for convolutional networks

We now consider the original ResNet with convolutional layers introduced in
[13]. This architecture is close to the state-of-the-art methods used for
image recognition tasks. We do not include batch normalization [17] since it
only slightly improves the performance of the network while making the
analysis significantly more complicated. The architecture is displayed in
Figures 9 and 10.

Figure 9: Residual architecture. There are 4 blocks that are respectively
repeated $n_{1}$, $n_{2}$, $n_{3}$ and $n_{4}$ times. The network depth is
$L=n_{1}+n_{2}+n_{3}+n_{4}$. The Basic Block architecture is detailed in
Figure 10. Figure 10: Basic Block from Figure 9. See (3.2) for details.

Our network still possesses the skip connections from (1.1): the dynamics of
the hidden state reads

$h_{k+1}=\sigma\left(h_{k}+\Delta_{k}*\sigma\left(A_{k}*h_{k}\right)+F_{k}*h_{k}\right)$
(3.2)

for $k=0,\ldots,L-1$, where $\sigma=\mathrm{ReLU}$. Here, $\Delta_{k}$,
$A_{k}$, and $F_{k}$ are kernels and $*$ denotes convolution. Note that
$\Delta_{k}$ plays the same role as $\delta_{k}^{(L)}$ from (1.1). To lighten
the notation, we omit the superscripts $x$ (the input) and $L$ (the number of
layers).

We train our residual networks at depths ranging from $L_{\min}=8$ to
$L_{\max}=121$ on the CIFAR-10 [19] dataset with the unregularized relative
entropy loss. Here, ’depth’ is the number of residual connections. We note
that a network with $L_{\max}=121$ is already very deep. As a comparison, a
standard ResNet-152 [13] has depth $L=50$ in our framework.

Figure 11: Scaling of $\Delta^{(L)}$ (left) and $A^{(L)}$ (right) against the
network depth $L$ for convolutional architectures on CIFAR-10. In blue:
spectral norm of the kernels $\Delta^{(L)}_{k}$, resp. $A^{(L)}_{k}$, for
$k=0,\ldots,L-1$. In red: maximum norm, defined in Table 2.2. The plots are in
log-log scale.

Results. Table 3.3 shows the accuracy of our convolutional residual networks
trained on an NVIDIA GeForce RTX 2080 GPU on the CIFAR-10 dataset. The results
are in line with those of traditional ResNet architectures [13], even though
our networks do not have batch normalization layers [17]. It is also
noteworthy to add that our concept of depth is not that of traditional
ResNets. We define the number of layers $L$ as the number of skip connections
in the network, that is the number of $\Delta_{k}$ kernels in (3.2).

Table 3: Learning error in $\%$ on CIFAR-10 for each network depth $L$.

$L$ | 8 | 11 | 12 | 14 | 16 | 20 | 24 | 28
---|---|---|---|---|---|---|---|---
Test error | 6.64 | 6.37 | 6.32 | 5.98 | 6.25 | 5.98 | 6.24 | 7.03
$L$ | 33 | 42 | 50 | 65 | 80 | 100 | 121 |
Test error | 6.13 | 6.21 | 6.32 | 6.19 | 6.30 | 6.20 | 6.37 |

As in Section 3.2, we investigate how the weights scale with depth and whether
Scaling regime 1 or Scaling regime 2 holds true for convolutional layers. To
that end, we follow the steps of [30] to get the singular values, and
therefore the spectral norms, of the linear operators defined by the
convolutional kernels $\Delta_{k}^{(L)}$ and $A_{k}^{(L)}$. Figure 11 shows
the maximum norm, and hence the scaling of $\Delta^{(L)}$ and $A^{(L)}$
against the network depth $L$. We observe that $\Delta^{(L)}\sim L^{-\alpha}$
and $A^{(L)}\sim L^{-\beta}$ with $\alpha=0.1$ and $\beta=0$.

Figure 12: Scaling behavior of $\Delta^{(L)}$ (left) and $A^{(L)}$ (right). We
plot in pink the root sum of squares and in orange the $\alpha$-scaled norm of
increments of $\Delta^{(L)}$ (left) and the $\beta$-scaled norm of increments
of $A^{(L)}$ (right). Plots are in log-log scale. The root sum of squares and
the scaled norm of increments are defined in Table 2.2. We obtain $\alpha$ and
$\beta$ from Figure 11.

We then use the values obtained for $\alpha$ and $\beta$ to verify which
Scaling regime holds. Figure 12 shows that both the $\alpha$-scaled norm of
increments of $\Delta^{(L)}$ and the $\beta$-scaled norm of increments of
$A^{(L)}$ seem to have lower bounds as the depth grows. This suggests that
Scaling regime 1 does not hold for convolutional layers.

We also observe that the root sum of squares stays in the same order as the
depth increases. Coupled with the fact that the maximum norms of
$\Delta^{(L)}$ and $A^{(L)}$ are close to constant order as the depth
increases, this suggests that the scaling limit is sparse with a finite number
of weights being of constant order in $L$.

###  3.4 Summary: three scaling regimes

Our experiments show different scaling regimes for trained weights based on
the network architecture.F or fully-connected layers with $\tanh$ activation
and a shared $\delta^{(L)}\in\mathbb{R}$, we observe a behavior consistent
with Scaling regime 1 for both the synthetic dataset and MNIST. For fully-
connected layers with $\mathrm{ReLU}$ activation and
$\delta_{k}^{(L)}\in\mathbb{R}$, we observe that Scaling regime 2 holds for
the synthetic dataset and MNIST. We deduce that the results for fully-
connected layers are consistent with our findings in Figure 1.

In the case of convolutional architectures trained on CIFAR-10 and presented
in Section 3.3, we observe that the maximum norm of the trained weights does
not decrease with the network depth and the trained weights display a sparse
structure, indicating a third scaling regime corresponding to sparse scaling
limits for both $\Delta^{(L)}$ and $A^{(L)}$. These results are consistent
with previous evidence on the existence of sparse CNN representations for
image recognition [24]. We stress that the setup for our CIFAR-10 experiments
has been chosen to approach state-of-the-art performance with our generic
architecture, as shown in Figures 9 and 10.

##  4 Deep network limit

In this section, we study the scaling limit of the hiddent state dynamics
(1.1) under scaling regimes 1 and 2.

###  4.1 Scaling regime 1: ODE limit

First, we show that the scaling regime 1 together with a smooth and Lipschitz-
continuous activation function lead to two ODE limits under different
parameter regimes, including the neural ODE described in [4, 31, 12] as a
special case.

We consider a setup which is consistent with Scaling regime 1 and
$\delta^{(L)}=L^{-\alpha}$ for some $\alpha\geq 0$:

$\displaystyle h_{0}^{(L)}$ $\displaystyle=x,$ (4.1) $\displaystyle
h^{(L)}_{k+1}$
$\displaystyle=h^{(L)}_{k}+L^{-\alpha}\,\sigma_{d}\left(A^{(L)}_{k}h^{(L)}_{k}+b^{(L)}_{k}\right),$

with

$\displaystyle A^{(L)}_{k}=L^{-\beta}\mkern 1.0mu\overline{\mkern-1.5muA\mkern
0.0mu}\mkern 0.0mu_{k/L},\quad b^{(L)}_{k}=L^{-\beta}\overline{b}_{k/L}.$

We focus our analysis on smooth activation functions.

######  Assumption 1 (Activation function).

The activation function $\sigma$ satisfies
$\sigma\in\mathcal{C}^{3}(\mathbb{R},\mathbb{R})$, $\sigma(0)=0$,
$\sigma^{\prime}(0)=1$, and has a bounded third derivative
$\sigma^{\prime\prime\prime}$.

Most smooth activation functions, including $\tanh$, satisfy this condition.
The boundedness of the third derivative $\sigma^{\prime\prime\prime}$ may be
relaxed to an exponential growth condition [26].

As observed in the numerical experiments, non-smooth activation functions such
as $\mathrm{ReLU}$ lead to a different scaling regime to that of smooth
functions.

We now describe ODE limits under Scaling regime 1. Let $\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}:[0,1]\to\mathbb{R}^{d}$ be a continuous-time
extension of the hidden states $h^{x,(L)}_{k}$:

$\displaystyle\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{t}\coloneqq
h_{k}^{x,(L)}\mathbbm{1}_{\frac{k}{L}\leq t<\frac{k+1}{L}},\quad
k=0,1,\ldots,L.$ (4.2)

######  Theorem 1 (ODE limits under Scaling regime 1).

Under Assumption 1 and $\sigma$ Lipschitz,

  * •

Neural ODE limit [31, Lemma 4.6]: If $\alpha=1$ and $\beta=0$ and we further
assume that $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu\in\mathcal{H}^{1}\left([0,1],\mathbb{R}^{d\times d}\right)$ and
$\overline{b}\in\mathcal{H}^{1}\left([0,1],\mathbb{R}^{d}\right)$, then the
interpolated hidden state dynamics (4.2) converge to the solution of the
neural ODE

$\displaystyle\frac{\mathrm{d}H_{t}}{\mathrm{d}t}=\sigma(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}H_{t}+\overline{b}_{t}),\qquad H_{0}=x,$ (4.3)

in the sense that $\lim_{L\rightarrow\infty}\sup_{0\leq t\leq 1}\|H_{t}-\mkern
1.0mu\overline{\mkern-1.5muH\mkern 0.0mu}\mkern-1.0mu_{t}\|=0$.

  * •

A different ODE limit: If $\alpha+\beta=1$ and $\beta>0$, and there exist
$M>0$ and $\kappa>0$ such that $\forall s,t\in[0,1]$, $\left\lVert\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{t}-\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{s}\right\rVert+\left\lVert\overline{b}_{t}-\overline{b}_{s}\right\rVert\leq
M|t-s|^{\kappa/2}$, then the interpolated hidden state dynamics (4.2) converge
to the solution of the following ODE

$\displaystyle\frac{\mathrm{d}H_{t}}{\mathrm{d}t}=\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}H_{t}+\overline{b}_{t},\qquad H_{0}=x,$ (4.4)

in the sense that $\lim_{L\rightarrow\infty}\sup_{0\leq t\leq 1}\|H_{t}-\mkern
1.0mu\overline{\mkern-1.5muH\mkern 0.0mu}\mkern-1.0mu_{t}\|=0$.

###  4.2 Scaling regime 2

Let $(\Omega,\mathcal{F},\mathbb{F},\mathbb{P})$ be a probability space with a
$\mathbb{P}$-complete filtration $\mathbb{F}=(\mathcal{F}_{t})_{t\geq 0}$. Let
$(B^{A}_{t})_{t\geq 0}$, resp. $(B^{b}_{t})_{t\geq 0}$, be $d\times
d$-dimensional, resp. $d$-dimensional, $\mathbb{F}$-standard uncorrelated
Brownian motions. We consider a setup which is consistent with Section 3 where
the noise part comes from the increment of some stochastic process and
$\delta^{(L)}=L^{-\alpha}$ for some $\alpha\geq 0$:

$\displaystyle h_{0}^{(L)}$ $\displaystyle=x,$ (4.5) $\displaystyle
h^{(L)}_{k+1}$
$\displaystyle=h^{(L)}_{k}+L^{-\alpha}\,\sigma_{d}\left(A^{(L)}_{k}h^{(L)}_{k}+b^{(L)}_{k}\right),$

with

$\displaystyle A^{(L)}_{k}$ $\displaystyle=$ $\displaystyle L^{-\beta}\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{k/L}+\left(W^{A}_{(k+1)/L}-W^{A}_{k/L}\right),$ $\displaystyle
b^{(L)}_{k}$ $\displaystyle=$ $\displaystyle
L^{-\beta}\overline{b}_{k/L}+\left(W^{b}_{(k+1)/L}-W^{b}_{k/L}\right),$

where $(W^{A}_{t})_{t\in[0,1]}$ and $(W^{b}_{t})_{t\in[0,1]}$ are Itô
processes [28] adapted to $\mathbb{F}$ and can be written in the form:

$\displaystyle\left(\mathrm{d}W_{t}^{A}\right)_{ij}$
$\displaystyle=\left(U^{A}_{t}\right)_{ij}\mathrm{d}t+\sum_{k,l=1}^{d}\left(q_{t}^{A}\right)_{ijkl}\left(\mathrm{d}B^{A}_{t}\right)_{kl}\quad\mbox{for
}\,i,j=1,\ldots,d,$ (4.6) $\displaystyle\mathrm{d}W_{t}^{b}$
$\displaystyle=U^{b}_{t}\mathrm{d}t+q^{b}_{t}\,\mathrm{d}B^{b}_{t},$

with $W_{0}^{A}=0$, $W_{0}^{b}=0$, $q_{t}^{A}\in\mathbb{R}^{d,\otimes 4}$ and
$q_{t}^{b}\in\mathbb{R}^{d\times d}$ for $t\in[0,1]$. We use the following
notation for the quadratic variation of $W^{A}$ and $W^{b}$:

$\displaystyle\big{[}W^{A}\big{]}_{t}=\int_{0}^{t}\Sigma^{A}_{u}\,\mathrm{d}u,\quad\big{[}W^{b}\big{]}_{t}=\int_{0}^{t}\Sigma^{b}_{u}\,\mathrm{d}u,$
(4.7)

where $\Sigma^{A}$ and $\Sigma^{b}$ are bounded processes with values
respectively in $\mathbb{R}^{d,\,\otimes 4}$ and $\mathbb{R}^{d\times d}$.
From (4.6) and (4.7), we have the quadratic variation process as follows:

$\left(\Sigma_{t}^{A}\right)_{i_{1}j_{1}i_{2}j_{2}}\coloneqq\sum_{k,l=1}^{d}\left(q_{t}^{A}\right)_{i_{1}j_{1}kl}\,\left(q_{t}^{A}\right)_{i_{2}j_{2}kl},\quad\mbox{for
}i_{1},j_{1},i_{2},j_{2}=1,\ldots,d,\quad\Sigma_{t}^{b}\coloneqq
q_{t}^{b}\left(q_{t}^{b}\right)^{\top}.$ (4.8)

Here $(U_{t}^{A})_{t\geq 0}$, $(U_{t}^{b})_{t\geq 0}$,
$(\Sigma_{t}^{A})_{t\geq 0}$ and $(\Sigma_{t}^{b})_{t\geq 0}$ are
progressively measurable processes that satisfy the following conditions.

######  Assumption 2 (Regularity of the Ito processes $(W^{A},W^{b})$ and
continuous functions $(\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu,\overline{b})$).

We assume:

  * (i)

There exists a constant $C_{1}>0$ such that almost surely

$\displaystyle\sup_{0\leq t\leq 1}\left\lVert
U_{t}^{A}\right\rVert+\sup_{0\leq t\leq 1}\left\lVert
U_{t}^{b}\right\rVert+\sup_{0\leq t\leq
1}\left\lVert\Sigma_{t}^{A}\right\rVert+\sup_{0\leq t\leq
1}\left\lVert\Sigma_{t}^{b}\right\rVert\leq C_{1}.$ (4.9)

  * (ii)

There exist $M>0$ and $\kappa>0$ such that $\forall s,t\in[0,1]$ almost surely

$\displaystyle\left\lVert U_{t}^{A}-U^{A}_{s}\right\rVert^{2}+\left\lVert
U_{t}^{b}-U^{b}_{s}\right\rVert^{2}+\left\lVert\Sigma_{t}^{A}-\Sigma^{A}_{s}\right\rVert^{2}+\left\lVert\Sigma_{t}^{b}-\Sigma^{b}_{s}\right\rVert^{2}\leq
M|t-s|^{\kappa},$ (4.10)

and

$\displaystyle\left\lVert\mkern 1.0mu\overline{\mkern-1.5muA\mkern
0.0mu}\mkern 0.0mu_{t}-\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{s}\right\rVert^{2}+\left\lVert\overline{b}_{t}-\overline{b}_{s}\right\rVert^{2}\leq
M|t-s|^{\kappa}.$ (4.11)

Note that (4.9) implies that $(U^{A},U^{B},\Sigma^{A},\Sigma^{B})$ are almost
surely uniformly bounded and (4.10) implies that
$(U^{A},U^{B},\Sigma^{A},\Sigma^{B})$ are almost surely Hölder continuous with
exponent $\kappa/2$.

######  Lemma 1 (Uniform integrability).

Under Assumption 2 (i), we have

$\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert
W^{A}_{s}\right\rVert^{p_{0}}\right]\,\lor\,\mathbb{E}\left[\sup_{0\leq s\leq
1}\left\lVert W^{b}_{s}\right\rVert^{p_{0}}\right]<\infty,$ (4.12)

for any $p_{0}>1$.

###### Proof.

By Minkowski’s inequality and Assumption 2-(i),

$\displaystyle\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert
W^{A}_{s}\right\rVert^{p_{0}}\right]$ $\displaystyle\leq$ $\displaystyle
2^{p_{0}-1}\mathbb{E}\left[\sup_{0\leq s\leq
1}\left\lVert\int_{0}^{s}U_{t}^{A}\mathrm{d}t\right\rVert^{p_{0}}\right]$
$\displaystyle+$ $\displaystyle 2^{p_{0}-1}\mathbb{E}\left[\sup_{0\leq s\leq
1}\left\lVert\left(\int_{0}^{s}\sum_{k,l=1}^{d}\left(q_{t}^{A}\right)_{ijkl}\left(\mathrm{d}B^{A}_{t}\right)_{kl}\right)_{i,j}\right\rVert^{p_{0}}\right]$
$\displaystyle\leq$ $\displaystyle
2^{p_{0}-1}C_{1}^{p_{0}}+2^{p_{0}-1}\mathbb{E}\left[\sup_{0\leq s\leq
1}\left\lVert\left(\int_{0}^{s}\sum_{k,l=1}^{d}\left(q_{t}^{A}\right)_{ijkl}\left(\mathrm{d}B^{A}_{t}\right)_{kl}\right)_{i,j}\right\rVert^{p_{0}}\right]$

By the Burkholder-Davis-Gundy inequality and Assumption 2 $(i)$,

$\displaystyle\mathbb{E}\left[\sup_{0\leq s\leq
1}\left\lVert\left(\int_{0}^{s}\sum_{k,l=1}^{d}\left(q_{t}^{A}\right)_{ijkl}\left(\mathrm{d}B^{A}_{t}\right)_{kl}\right)_{i,j}\right\rVert^{p_{0}}\right]\leq
C_{p_{0}}\mathbb{E}\left[\left(\int_{0}^{1}\Sigma_{u}^{A}\mathrm{d}u\right)^{p_{0}/2}\right]\leq
C_{p_{0}}\,C_{1}^{p_{0}/2}.$

Combining the two inequalities above, we get $\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert W^{A}_{s}\right\rVert^{p_{0}}\right]<\infty$. Similarly
$\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert
W^{b}_{s}\right\rVert^{p_{0}}\right]<\infty$ holds. ∎

Write $Q:[0,1]\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$, where each
component $Q_{i}$ is defined, for $i=1,\ldots,d$, as

$Q_{i}(t,x)\coloneqq\sum_{j,k=1}^{d}x_{j}x_{k}\left(\Sigma^{A}_{t}\right)_{ijik}+\Sigma^{b}_{t,ii}.$
(4.13)

Let $\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}:[0,1]\to\mathbb{R}^{d}$ be a continuous-time
extension of the hidden states $h^{(L)}_{k}$:

$\displaystyle\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{t}\coloneqq
h_{k}^{x,(L)}\mathbbm{1}_{\frac{k}{L}\leq t<\frac{k+1}{L}},\quad
k=0,1,\ldots,L.$ (4.14)

######  Assumption 3 (Uniform integrability).

There exist $p_{1}>4$ and a constant $C_{0}$ such that for all $L$,

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{t}\right\rVert^{p_{1}}\right]\leq C_{0}.$ (4.15)

Assumption 3 is standard in the convergence of approximation schemes for SDEs
[15]. In practice, condition (4.15) is guaranteed throughout the training as
both the inputs and the outputs of the network are bounded.

Let us now describe the intuition behind the deep network limit when
$\beta>0$. Denote $t_{k}=k/L$ and define for $s\in[t_{k},t_{k+1})$:

$\displaystyle\widetilde{M}^{(L)}_{k,s}\coloneqq\left(W^{A}_{s}-W^{A}_{t_{k}}\right)h^{(L)}_{k}+\left(W^{b}_{s}-W^{b}_{t_{k}}\right)+L^{1-\beta}\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k}}h^{(L)}_{k}(s-t_{k})+L^{1-\beta}\overline{b}_{t_{k}}(s-t_{k}).$

Using Itô’s formula [18] to $\sigma\big{(}\widetilde{M}^{(L)}_{k,s}\big{)}$
for $s\in[t_{k},t_{k+1})$, we obtain the following approximation

$h^{(L)}_{k+1}-h^{(L)}_{k}=\delta^{(L)}\sigma\left(\widetilde{M}_{k,t_{k+1}}^{(L)}\right)\simeq
D_{1}+D_{2}+D_{3},$ (4.16)

where

$\displaystyle D_{1}$ $\displaystyle\coloneqq
L^{-\alpha}\left(\left(W^{A}_{t_{k+1}}-W^{A}_{t_{k}}\right)h_{k}^{(L)}+\left(W^{b}_{t_{k+1}}-W^{b}_{t_{k}}\right)\right),$
$\displaystyle D_{2}$
$\displaystyle\coloneqq\frac{1}{2}L^{-\alpha}\sigma^{\prime\prime}(0)\,Q\left(t_{k},{h^{(L)}_{k}}\right)({t_{k+1}-t_{k}}),$
$\displaystyle D_{3}$ $\displaystyle\coloneqq L^{1-\beta-\alpha}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k}}h_{k}^{(L)}({t_{k+1}-t_{k}})+\overline{b}_{t_{k}}({t_{k+1}-t_{k}})\right).$

We observe from $D_{1}$ that (4.16) admits a diffusive limit only when
$\alpha=0$. In this case, we see that $D_{2}$ and $D_{3}$ do not explode only
when $\beta\geq 1$, corresponding to a stochastic differential equation (SDE)
limit that is diffusive. Another case where we obtain a non-trivial limit is
when $\alpha>0$ and $\alpha+\beta=1$, which leads to an ODE limit.

We now provide a precise mathematical description of the different scaling
limits of $\mkern 1.0mu\overline{\mkern-1.5muH\mkern 0.0mu}\mkern-1.0mu^{(L)}$
for various values of $\alpha$ and $\beta$, using the concept of uniform
convergence in $L^{2}$, also known as strong convergence. For a general
exponent $p\geq 1$, we have the following definition.

######  Definition 1 (Uniform convergence in $L^{p}$).

Let $p\geq 1$ and $\mathcal{M}$ be the class of random functions
$X:[0,1]\times\Omega\to\mathbb{R}^{d}$ such that

$\mathbb{E}\left[\mathop{\sup}_{t\in[0,1]}\left\lVert
X(t)\right\rVert^{p}\right]<\infty.$

We say that a sequence $(X^{(L)})_{L\in\mathbb{N}}\subset\mathcal{M}$
_converges uniformly in $L^{p}$_ to $X^{*}\in\mathcal{M}$ if

$\lim_{L\to\infty}\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert
X^{(L)}_{t}-{X}^{*}_{t}\right\rVert^{p}\right]=0.$ (4.17)

We now show that Scaling regime 2 together with a smooth activation function
lead to an ODE limit (which is different from the neural ODE) or a stochastic
differential equation (SDE) depending on the values of $\alpha$ and $\beta$.

######  Theorem 2 (ODE limit under Scaling regime 2).

Under Assumptions 1, 2, and 3, if $\alpha>0$, $\beta>0$ and $\alpha+\beta=1$,
then the interpolated hidden state dynamics (4.14) converge uniformly in
$L^{2}$ to the solution to the ODE

$\displaystyle\frac{\mathrm{d}H_{t}}{\mathrm{d}t}=\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}H_{t}+\overline{b}_{t},\qquad H_{0}=x.$ (4.18)

In particular, this implies the convergence of the hidden state process for
any typical initialization (i.e almost surely with respect to the
initialization). Note that in Theorem 2, the limit (4.18) defines a linear
input-output map behaving like a linear network [1]. This is different from
the neural ODE (1.2), where the activation function $\sigma$ appears in the
limit.

######  Theorem 3 (SDE limit under Scaling regime 2).

Let Assumptions 1, 2 and 3 hold and let $\alpha=0$ and $\beta\geq 1$. Denote
$H$ as the solution to the SDE

$\mathrm{d}H_{t}=\mathrm{d}W^{A}_{t}H_{t}+\mathrm{d}W^{b}_{t}+\frac{1}{2}\sigma^{\prime\prime}(0)Q(t,H_{t})\,\mathrm{d}t+\mathbbm{1}_{\beta=1}(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}H_{t}+\overline{b}_{t})\,\mathrm{d}t,$ (4.19)

with initial condition $H_{0}=x$. If there exist $p_{2}>2$ such that
$\mathbb{E}\left[\sup_{0\leq t\leq 1}\|H_{t}\|^{p_{2}}\right]<\infty$, then
the interpolated hidden state dynamics (4.14) converge uniformly in $L^{2}$ to
the solution of (4.19).

The proofs of Theorem 3 is given in Section 4.4. And the proof of Theorem 2
follows similar ideas. In particular, $D_{1}$ and $D_{2}$ vanish in the limit
when $\alpha>0$ in (4.16).

Interestingly, when the activation function $\sigma$ is smooth, all limits in
both Theorems 2 and 3 depend on the activation only through
$\sigma^{\prime}(0)$ (assumed to be $1$ for simplicity) and
$\sigma^{\prime\prime}(0)$. In contrast to the behavior of the neural ODE
limit (1.2), the characteristics of $\sigma$ away from $0$ are not relevant to
the limit. In addition, our proofs rely on the smoothness of $\sigma$ at $0$.
If the activation function is not differentiable at $0$, then a different
limit should be expected.

The case $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu\equiv
0$, $\overline{b}\equiv 0$, $\alpha=0$, and $\beta=1$ in Theorem 3 is
considered in [26], under the additional assumption that $W^{A}$ and $W^{b}$
are Brownian motions with constant drift. We consider a more general setup,
where we introduce nonzero terms $\mkern 1.0mu\overline{\mkern-1.5muA\mkern
0.0mu}\mkern 0.0mu$ and $\overline{b}$ and we allow $W^{A}$ and $W^{b}$ to be
arbitrary Itô processes. Moreover, [26] prove weak convergence, which
corresponds to convergence of quantities averaged across many trained networks
with random independent initializations, whereas in practice, the training is
done only once. Thus, the strong convergence, shown in Theorems 1 and 3, is a
more relevant notion for studying the asymptotic behavior of deep neural
networks.

Although the ResNet dynamics (4.5) is not expressed as an Euler scheme of a
(ordinary or stochastic) differential equation, we nevertheless show strong
convergence to a limitng ODE (in the case of Theorem 2) or SDE (in the case of
Theorem 3), using techniques inspired by [15]. The challenge is to bound the
difference between the ResNet dynamics and the Euler scheme of the limiting
SDE. It is worth mentioning that the results in [15] hold for a class of time-
homogeneous (Markov) diffusion processes whereas our result holds for Itô
processes with bounded coefficients. This distinction is important for
training neural networks since the “diffusion” assumption involves the
distribution of the hidden state dynamics which can never be tested in
practice. We can only verify the smoothness of the hidden state dynamics as
detailed in Section 3. In addition, we also relax one technical condition
assumed in [15], which is difficult to verify in practice. See Remark 3.

Note that we assume that the Ito processes $W^{A}$ and $W^{b}$ are driven by
uncorrelated Brownian motions $B^{A}$ and $B^{b}$. This assumption might look
strong, but we pose it for ease of exposition: assuming a generic correlation
structure between $B^{A}$ and $B^{b}$ would only a cross-term in the
definition of $Q$.

###  4.3 Link with numerical experiments

Let us now discuss how the analysis above sheds light on the numerical results
in Section 3.2 and Section 3.3.

Figure 2 shows that $\beta=0.2$ and $\alpha=0.7$ for the synthetic dataset
with fully-connected layers and $\tanh$ activation function. This corresponds
to the assumptions of Theorem 1 with the ODE limit (4.18). This is also
consistent with the estimated decomposition in Figure 3 (right) where the
noise part is negligible.

Regarding $\mathrm{ReLU}$ activation with fully-connected layers, we observe
that $\beta+\alpha=0.9$ from Figure 4 (left). Since ReLU is homogeneous of
degree 1 (see Remark 2), $|\delta^{(L)}|$ can be moved inside $\sigma$, so
without loss of generality we can assume $\alpha=0$ and $\beta=0.9$. If we
replace the $\mathrm{ReLU}$ function by a smooth version $\sigma^{\epsilon}$,
then the limit is described by the stochastic differential equation (4.19).
The $\mathrm{ReLU}$ case would then correspond to a limit of this equation as
$\epsilon\to 0$. The existence of such a limit is, however, nontrivial and
left for future work.

From the experiments with convolutional architectures, we observe that the
maximum norm (Figure 11), the scaled norm of the increments, and the root sum
of squares (Figure 12) are upper bounded as the number of layers $L$
increases. This indicates that the weights fall into a sparse regime when $L$
is large. In this case, there is no continuous ODE or SDE limit and Scaling
regimes 1 and 2 both fail.

###  4.4 Detailed proofs

####  4.4.1 Proof of Theorem 1

It suffices to prove the second case with limit (4.4). First we show that
there exists $C_{\infty}>0$ such that

$\displaystyle\sup_{L}\max_{0\leq k\leq L}\left\lVert
h_{k}^{(L)}\right\rVert\leq C_{\infty}.$ (4.20)

Indeed, denote $C_{\sigma}$ as the Lipschitz constant of $\sigma$. Then

$\displaystyle\left\lVert
h_{k+1}^{(L)}-h_{k}^{(L)}\right\rVert\leq\frac{C_{\sigma}}{L}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k}}h_{k}^{(L)}+\overline{b}_{t_{k}}\right\rVert\leq\frac{C_{\sigma}}{L}\left(A_{\max}+b_{\max}\right)\left(\left\lVert
h_{k}^{(L)}\right\rVert+1\right),$

where $A_{\max}:=\sup_{0\leq t\leq 1}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{t}\right\rVert<\infty$,
$b_{\max}:=\sup_{0\leq t\leq
1}\left\lVert\overline{b}_{t}\right\rVert<\infty$, and $C_{\max}\coloneqq
A_{\max}+b_{\max}$. Hence

$\displaystyle\left\lVert
h_{k+1}^{(L)}\right\rVert\leq\left(\frac{C_{\sigma}C_{\max}}{L}+1\right)\left\lVert
h_{k}^{(L)}\right\rVert+\frac{C_{\sigma}C_{\max}}{L}.$

By induction:

$\displaystyle\left\lVert h_{j}^{(L)}\right\rVert$ $\displaystyle\leq$
$\displaystyle\left\lVert
x\right\rVert\left(\frac{C_{\sigma}C_{\max}}{L}+1\right)^{j}+\frac{C_{\sigma}C_{\max}}{L}\sum_{i=1}^{j}\left(\frac{C_{\sigma}C_{\max}}{L}+1\right)^{i-1}$
$\displaystyle\leq$ $\displaystyle\left(\left\lVert
x\right\rVert+C_{\sigma}C_{\max}\right)\left(\frac{C_{\sigma}C_{\max}}{L}+1\right)^{L}$
$\displaystyle\rightarrow$ $\displaystyle\left(\left\lVert
x\right\rVert+C_{\sigma}C_{\max}\right)\exp\left(C_{\sigma}C_{\max}\right)\quad{\rm
as}\,\,L\rightarrow\infty.$

Hence (4.20) holds.

Denote $\Delta h_{k}^{(L)}:=h_{k+1}^{(L)}-h_{k}^{(L)}$ and
$M^{(L)}_{k}(h):=\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k}}h+\overline{b}_{t_{k}}$. From (4.16) we have

$\Delta
h_{k}^{(L)}:=h_{k+1}^{(L)}-h_{k}^{(L)}=L^{-\alpha}\sigma\left(L^{-\beta}M_{k}^{(L)}\left(h_{k}^{(L)}\right)\right).$

Denote as well $\Delta h_{k,i}^{(L)}$ and $M_{k,i}^{(L)}$ the $i$-th element
of $\Delta h_{k}^{(L)}$ and $M_{k}^{(L)}$, respectively. Applying a third-
order Taylor expansion of $\sigma$ around $0$ with the help of Assumption 1,
for $i=1,2,\ldots,d$, we get

$\displaystyle\Delta
h_{k,i}^{(L)}=L^{-\alpha}\sigma_{d}\left(L^{-\beta}M_{k,i}^{(L)}\left(h_{k}^{(L)}\right)\right)$
$\displaystyle=L^{-\beta-\alpha}M_{k,i}^{(L)}\left(h_{k}^{(L)}\right)+\frac{1}{2}\sigma^{\prime\prime}(0)L^{-2\beta-\alpha}\left(M_{k,i}^{(L)}\left(h_{k}^{(L)}\right)\right)^{2}+\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)L^{-3\beta-\alpha}\left(M_{k,i}^{(L)}\left(h_{k}^{(L)}\right)\right)^{3}$
$\displaystyle=L^{-1}M_{k,i}^{(L)}\left(h_{k}^{(L)}\right)+\frac{1}{2}\sigma^{\prime\prime}(0)L^{-\beta-1}\left(M_{k,i}^{(L)}(h_{k}^{(L)})\right)^{2}+\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)L^{-2\beta-1}\left(M_{k,i}^{(L)}\left(h_{k}^{(L)}\right)\right)^{3}$
(4.21)

with $\left|\xi^{(L)}_{k,i}\right|\leq L^{-\beta}\left|\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k}}h_{k}^{(L)}+\overline{b}_{t_{k}}\right|_{i}$. The last equation
holds since $\alpha+\beta=1$. Denote $t_{k}=k/L$ for $k=0,1,\ldots,L$ as the
uniform partition of the interval $[0,1]$. For $t\in(t_{k},t_{k+1}]$, define
$\widetilde{H}_{0}^{(L)}\coloneqq x=h_{0}^{(L)}$ and

$\displaystyle\widetilde{H}_{t}^{(L)}$ $\displaystyle\coloneqq
h_{k}^{(L)}+\left(t-t_{k}\right)M_{k,i}^{(L)}\left(h_{k}^{(L)}\right)+\frac{1}{2}\sigma^{\prime\prime}(0)L^{-\beta-1}\left(M_{k,i}^{(L)}\left(h_{k}^{(L)}\right)\right)^{2}$
$\displaystyle\quad+\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)L^{-2\beta-1}\left(M_{k,i}^{(L)}\left(h_{k}^{(L)}\right)\right)^{3}.$

Then we have $\widetilde{H}_{t_{k+1}}^{(L)}=h_{k}^{(L)}+\Delta
h_{k}^{(L)}=h_{k+1}^{(L)}$ for all $k=0,1,\ldots,L-1$. Recall
$(H_{t})_{t\in\left[0,1\right]}$ the solution to the ODE (4.4). Denote
$d^{(L)}_{k}(t)\coloneqq H_{t}-\widetilde{H}_{t}^{(L)}$ for
$t\in(t_{k},t_{k+1}]$ and define the errors

$e^{(L),1}_{k}\coloneqq\sup_{t_{k}<t\leq
t_{k+1}}\left\lVert\widetilde{H}_{t}^{(L)}-h_{k}^{(L)}\right\rVert\quad\mbox{and}\quad
e^{(L),2}_{k}\coloneqq\sup_{t_{k}<t\leq t_{k+1}}\left\lVert
d^{(L)}_{k}(t)\right\rVert.$

We first bound $e^{(L),1}_{k}$. Note that by definition:

$\displaystyle e^{(L),1}_{k}$ $\displaystyle\leq\left\lVert
M_{k}^{(L),i}\left(h_{k}^{(L)}\right)\right\rVert
L^{-1}+\frac{1}{2}\sigma^{\prime\prime}(0)L^{-\beta-1}\left(M_{k}^{(L),i}\left(h_{k}^{(L)}\right)\right)^{2}+\frac{1}{6}c_{0}L^{-2\beta-1}\left|M_{k}^{(L),i}\left(h_{k}^{(L)}\right)\right|^{3}$
$\displaystyle\leq D_{\infty}L^{-1},$ (4.22)

where
$D_{\infty}:=A_{\max}C_{\infty}+b_{\max}+\frac{1}{2}\sigma^{\prime\prime}(0)(A_{\max}C_{\infty}+b_{\max})^{2}+\frac{1}{6}c_{0}(A_{\max}C_{\infty}+b_{\max})^{2}$.
Therefore we have

$\lim_{L\rightarrow\infty}\sup_{0\leq k<L}e^{(L),1}_{k}=0.$

Next, we bound $e^{(L),2}_{k}$. For $t\in(t_{k+1},t_{k+2}]$,

$\displaystyle d^{(L)}_{k+1}(t)$
$\displaystyle=d^{(L)}_{k}(t_{k+1})-\left(t-t_{k+1}\right)M_{k+1}^{(L),i}\left(h_{k+1}^{(L)}\right)+\int_{t_{k+1}}^{t}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{s}H_{s}+\overline{b}_{s}\right)\mathrm{d}s$
$\displaystyle\quad-\frac{1}{2}\sigma^{\prime\prime}(0)L^{-\beta-1}\left(M_{k+1}^{(L),i}\left(h_{k+1}^{(L)}\right)\right)^{2}-\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi_{k+1}^{i}\right)L^{-2\beta-1}\left(M_{k+1}^{(L),i}\left(h_{k+1}^{(L)}\right)\right)^{3}.$
(4.23)

Denote $c_{0}:=\sup_{x\in\mathbb{R}}|\sigma^{\prime\prime\prime}(x)|<\infty$,
hence from (4.21) and (4.23),

$\displaystyle e^{(L),2}_{k+1}$ $\displaystyle\leq$ $\displaystyle
e^{(L),2}_{k}+\sup_{t_{k+1}<t\leq
t_{k+2}}\left\lVert\int_{t_{k+1}}^{t}\left(\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{s}H_{s}+\overline{b}_{s}\right)-\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k+1}}h_{k+1}^{(L)}+\overline{b}_{t_{k+1}}\right)\right)\mathrm{d}s\right\rVert$
$\displaystyle+\frac{1}{2}\left|\sigma^{\prime\prime}(0)\right|L^{-\beta-1}\left\lVert
M_{k+1}^{(L)}(h_{k+1}^{(L)})\right\rVert^{2}+\frac{1}{6}c_{0}L^{-2\beta-1}\left\lVert
M_{k+1}^{(L)}(h_{k+1}^{(L)})\right\rVert^{3}.$

Denote $H_{\max}\coloneqq\sup_{0\leq t\leq 1}\left\lVert
H_{t}\right\rVert<\infty$. Then,

$\displaystyle\mathcal{E}_{k}^{(L)}$ $\displaystyle\coloneqq$
$\displaystyle\sup_{t_{k+1}<t\leq
t_{k+2}}\left\lVert\int_{t_{k+1}}^{t}\left(\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{s}H_{s}+\overline{b}_{s}\right)-\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k+1}}h_{k+1}^{(L)}+\overline{b}_{t_{k+1}}\right)\right)\mathrm{d}s\right\rVert$
$\displaystyle\leq$ $\displaystyle\sup_{t_{k+1}<t\leq
t_{k+2}}\left\lVert\int_{t_{k+1}}^{t}\left(\overline{b}_{s}-\overline{b}_{t_{k+1}}\right)\mathrm{d}s\right\rVert+\sup_{t_{k+1}<t\leq
t_{k+2}}\left\lVert\int_{t_{k+1}}^{t}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{s}-\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k+1}}\right)H_{s}\mathrm{d}s\right\rVert$
$\displaystyle+\sup_{t_{k+1}<t\leq t_{k+2}}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k+1}}\int_{t_{k+1}}^{t}\left(H_{s}-h_{k+1}^{(L)}\right)\mathrm{d}s\right\rVert.$

Hence, we deduce

$\displaystyle\mathcal{E}_{k}^{(L)}$ $\displaystyle\leq$
$\displaystyle\int_{t_{k+1}}^{t_{k+2}}\left\lVert\overline{b}_{s}-\overline{b}_{t_{k+1}}\right\rVert\mathrm{d}s+H_{\max}\int_{t_{k+1}}^{t_{k+2}}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{s}-\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k+1}}\right\rVert\mathrm{d}s+A_{\max}\int_{t_{k+1}}^{t_{k+2}}\left\lVert
H_{s}-h_{k+1}^{(L)}\right\rVert\mathrm{d}s$ $\displaystyle\leq$ $\displaystyle
M(1+H_{\max})\int_{t_{k+1}}^{t_{k+2}}|s-t_{t_{k+1}}|^{\kappa/2}\mathrm{d}s+A_{\max}\int_{t_{k+1}}^{t_{k+2}}\left\lVert
H_{s}-h_{k+1}^{(L)}\right\rVert\mathrm{d}s$ $\displaystyle\leq$
$\displaystyle\frac{M}{1+\kappa/2}\left(1+H_{\max}\right)L^{-(1+\kappa/2)}+\sup_{0\leq
t\leq 1}\|\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}\|L^{-1}\left(D_{\infty}L^{-1}+e_{k+1}^{(L),2}\right).$

The last equation holds by (4.22). Then, we have for $L>A_{\max}$,

$\displaystyle\left(1-A_{\max}L^{-1}\right)e^{(L),2}_{k+1}$ $\displaystyle\leq
e^{(L),2}_{k}+\frac{M}{1+\kappa}\left(1+H_{\max}\right)L^{-(1+\kappa)}+\frac{1}{2}\sigma^{\prime\prime}(0)L^{-(\beta+1)}\left(A_{\max}C_{\infty}+b_{\max}\right)^{2}$
$\displaystyle\quad+\frac{1}{6}c_{0}L^{-(2\beta+1)}\left(A_{\max}C_{\infty}+b_{\max}\right)^{3}+A_{\max}D_{\infty}L^{-2}$
$\displaystyle\leq e_{k}^{(L),2}+L^{-(1+\nu)}C_{2},$ (4.24)

with $\nu:=\min\\{\kappa,\beta,1\\}>0$ and $C_{2}$ a constant independent of
$k$ and $L$. Finally, when $L\geq G^{1/\gamma}+2A_{\max}$ we have from (4.24):

$\displaystyle e_{0}^{(L),2}$ $\displaystyle\leq$
$\displaystyle\frac{L^{-(1+\gamma)}G}{1-A_{\max}L^{-1}}\leq\frac{1}{L-A_{\max}},$
(4.25)

and for $k=0,\ldots,L-1$,

$\displaystyle e^{(L),2}_{k+1}$ $\displaystyle\leq$
$\displaystyle\frac{1}{1-A_{\max}L^{-1}}\left(e^{(L),2}_{k}+L^{-(1+\gamma)}G\right)$
(4.26) $\displaystyle\leq$
$\displaystyle\left(\frac{1}{1-A_{\max}L^{-1}}\right)^{k+1}e^{(L),2}_{0}+L^{-(1+\gamma)}G\frac{\left(\frac{1}{1-A_{\max}L^{-1}}\right)^{k+2}-1}{\left(\frac{1}{1-A_{\max}L^{-1}}\right)-1}$
$\displaystyle\leq$
$\displaystyle\exp\left(2A_{\max}\frac{k+1}{L}\right)\frac{1}{L-A_{\max}}+L^{-\gamma}\frac{G}{A_{\max}}\exp\left(2A_{\max}\frac{k+2}{L}\right).$

(4.26) holds since $e_{0}^{(L),2}\leq\frac{1}{L-A_{\max}}$ and
$\frac{1}{1-A_{\max}L^{-1}}<1+2A_{\max}L^{-1}\leq\exp(2A_{\max}L^{-1})$ when
$L>2A_{\max}$. Therefore, we conclude

$\lim_{L\rightarrow\infty}\sup_{0\leq k<L}e^{(L),2}_{k}=0.$

####  4.4.2 Proof of Theorem 2

We provide a complete proof of Theorem 2 for the case $\alpha=0$ and
$\beta=1$. Other cases follow similarly. When $\alpha=0$ and $\beta=1$, we
define the targeted SDE limit for the discrete scheme (4.5) as follows:

$\displaystyle\mathrm{d}H_{t}=\mu(t,H_{t})\mathrm{d}t+\mathrm{d}V_{t}^{A}\,H_{t}+\mathrm{d}V_{t}^{b}\,\,\,\,\,\mbox{for}\,\,t\in\left[0,1\right],\,\,\,\,H_{0}=x,$
(4.27)

in which

$\displaystyle\mu\left(t,h\right)$ $\displaystyle\coloneqq
U_{t}^{A}\,h+\,U_{t}^{b}+\mkern 1.0mu\overline{\mkern-1.5muA\mkern
0.0mu}\mkern
0.0mu_{t}h+\overline{b}_{t}+\frac{1}{2}\sigma^{\prime\prime}(0)Q(t,h),$ (4.28)
$\displaystyle\mathrm{d}V_{t}^{A}$
$\displaystyle\coloneqq\sum_{k,l=1}^{d}\left(q_{t}^{A}\right)_{ijkl}\left(\mathrm{d}B_{t}^{A}\right)_{kl},\quad\mathrm{d}V_{t}^{b}\coloneqq
q_{t}^{b}\,\mathrm{d}B_{t}^{b},$

with $V_{0}^{A}=0$ and $V_{0}^{b}=0$. Here the quadratic variation process
$\frac{1}{2}\sigma^{\prime\prime}(0)Q(t,h)$ is the Itô correction term for the
drift. On the one hand this correction term introduces non-linearity into the
drift and makes the proof challenging. On the other hand, this term is the key
for the convergence analysis. See (4.55) and (4.56).

Euler-Maruyama scheme of the limiting SDE. Denote $\Delta_{L}=1/L$ as the sub-
interval length and $t_{k}=k/L$, $k=0,1,\ldots,L$ as the uniform partition of
the interval $[0,1]$. Further denote $\Delta
V^{A}_{k}=V^{A}_{t_{k+1}}-V^{A}_{t_{k}}$ and $\Delta
V^{b}_{k}=V^{b}_{t_{k+1}}-V^{b}_{t_{k}}$ as the increment of the stochastic
processes. Define the Euler-Maruyama discretization scheme of the SDE (4.27)
as:

$\displaystyle\widehat{h}_{k+1}^{(L)}-\widehat{h}_{k}^{(L)}\coloneqq\mu\left(t_{k},\widehat{h}_{k}^{(L)}\right)\Delta_{L}+\Delta
V_{k}^{A}\,\widehat{h}_{k}^{(L)}+\Delta V_{k}^{b},$ (4.29)

and the one-step forward increment follows:

$\displaystyle f^{(L)}(k,h)\coloneqq\mu\left(t_{k},h\right)\Delta_{L}+\Delta
V_{k}^{A}\,h+\Delta V_{k}^{b}.$ (4.30)

Therefore (4.29) can be rewritten as
$\widehat{h}_{k+1}^{(L)}=\widehat{h}_{k}^{(L)}+f^{(L)}\left(k,\widehat{h}_{k}^{(L)}\right)$.

Continuous-time extension. Recall that we extend the scheme
$\big{\\{}h^{(L)}_{k}:k=0,\ldots,L\big{\\}}$ to a continuous-time process
$\mkern 1.0mu\overline{\mkern-1.5muH\mkern 0.0mu}\mkern-1.0mu^{(L)}_{t}$ on
$t\in[0,1]$ by a piecewise constant and right-continuous interpolation of
$\\{h^{(L)}_{k}:k=0,\ldots,L-1\\}$:

$\displaystyle\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{t}\coloneqq\sum_{k=0}^{L}h_{k}^{(L)}{\bf
1}_{t_{k}\leq t<t_{k+1}}.$ (4.31)

We call $\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{t}$ the continuous-time extension (CTE) of
$\\{h^{(L)}_{k}:k=0,\ldots,L-1\\}$.

Continuous-time approximation. Denote

$\displaystyle M^{(L)}_{k}(h)$ $\displaystyle\coloneqq$
$\displaystyle\left(\mu\left(t_{k},h\right)-\frac{1}{2}\sigma^{\prime\prime}(0)Q(t_{k},h)\right)\Delta_{L}+\Delta
V_{k}^{A}\,h+\Delta V_{k}^{b}$ (4.32) $\displaystyle=$
$\displaystyle\left(U^{A}_{t_{k}}\,h+\,U^{b}_{t_{k}}+\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k}}h+\overline{b}_{t_{k}}\right)\Delta_{L}+\Delta
V_{k}^{A}\,h+\Delta V_{k}^{b}$ $\displaystyle=:$
$\displaystyle\widetilde{\mu}\left(t_{k},h\right)\Delta_{L}+\Delta
V_{k}^{A}\,h+\Delta V_{k}^{b},$

and from (4.16) we thus have

$\Delta
h_{k}^{(L)}:=h_{k+1}^{(L)}-h_{k}^{(L)}=\sigma\left(M_{k}^{(L)}\left(h_{k}^{(L)}\right)\right).$

Denote $\Delta h_{k,i}^{(L)}$ and $M_{k,i}^{(L)}$ the $i$-th element of
$\Delta h_{k}^{(L)}$ and $M_{k}^{(L)}$, respectively. Applying a third-order
Taylor expansion of $\sigma$ around $0$ with the help of Assumption 1, for
$i=1,2,\ldots,d$, we get

$\displaystyle\Delta
h_{k,i}^{(L)}=\sigma\left(M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)\right)$
$\displaystyle=M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)+\frac{1}{2}\sigma^{\prime\prime}(0)M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)^{2}+\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)^{3}$
$\displaystyle=\underbrace{{\mu}_{i}\left(t_{k},h_{k}^{(L)}\right)\Delta_{L}+\left(\Delta
V_{k}^{A}\,h_{k}^{(L)}\right)_{i}+\left(\Delta
V_{k}^{b}\right)_{i}}_{f_{i}^{(L)}\left(k,h_{k}^{(L)}\right)}+\underbrace{\frac{1}{2}\sigma^{\prime\prime}(0)\left(M_{k,i}^{(L)}\left(h_{k}^{(L)}\right)^{2}-Q_{i}\left(t_{k},h_{k}^{(L)}\right)\right)}_{N_{k,i}^{(L)}\left(h_{k}^{(L)}\right)}$
$\displaystyle\quad\,+\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)^{3}$
$\displaystyle=f^{(L)}_{i}\left(k,h_{k}^{(L)}\right)+N_{k,i}^{(L)}\left(h_{k}^{(L)}\right)+\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)^{3},$

with
$\left|\xi^{(L)}_{k,i}\right|<\left|M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)\right|$.
The increment of the ResNet $\Delta h_{k,i}^{(L)}$ has two parts: the
increment of the Euler-Maruyama scheme $f^{(L)}_{i}\left(k,h_{k}^{(L)}\right)$
and the residual

$D_{k,i}^{(L)}\left(h_{k}^{(L)}\right)\coloneqq\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)^{3}+N_{k,i}^{(L)}\left(h_{k}^{(L)}\right).$
(4.33)

It is clear from here that the Euler-Maruyama scheme of the limiting SDE is
different from the ResNet dynamics. Hence classical results on the convergence
of discrete SDE schemes cannot be applied directly.

In our analysis it will be more natural to work with the following continuous-
time approximation (CTA), defined as

$\displaystyle\widetilde{H}^{(L)}_{t}\coloneqq
h_{0}^{(L)}+\int_{0}^{t}\mu\left({{t_{k_{s}}}},\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\mathrm{d}s+\int_{0}^{t}\left(\mathrm{d}V^{A}_{s}\,\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}+\mathrm{d}V^{b}_{s}\right)+\sum_{k<Lt}D_{k}^{(L)}\left(h_{k}^{(L)}\right),$
(4.34)

where
$D_{k}^{(L)}(h)=\left(D_{k,1}^{(L)}(h),\ldots,D_{k,d}^{(L)}(h)\right)^{\top}$
and $k_{s}$ is the integer for which $s\in[t_{k_{s}},t_{k_{s}+1})$ for a given
$s\in[0,1)$.

Here $\widetilde{H}^{(L)}_{t}$ approximates the CTE (4.31) with a continuous
version, with interpolations both in time and in space, of the $f^{(L)}(k,h)$
part while the residual term $D_{k}^{(L)}(h)$ remains the same. By design we
have $\widetilde{H}^{(L)}_{t_{k}}=\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{L}_{t_{k}}=h^{(L)}_{k}$, that is,
$\widetilde{H}^{(L)}_{t}$ and $\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{t}$ coincide with the discrete solution at grid
points $t_{k}$, $k=0,1,\ldots,L-1$. This relationship is instrumental in order
to control the error.

We will first study the difference between $\widetilde{H}$ and $h^{(L)}$, and
then the difference between $\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu$ and $h^{(L)}$, in the supremum norm. The sum of the two
will give a bound for the error of the discrete approximation.

####  4.4.3 Preliminary result

######  Lemma 2 (Local Lipschitz condition and uniform integrability).

Under the assumptions from Theorem 3, we have the folllowing results:

  1. (i)

For each $R>0$, there exists a constant $C_{R}$, depending only on $R$, such
that almost surely we have

$\displaystyle\left\lVert\mu(t,x)-\mu(t,y)\right\rVert^{2}\leq
C_{R}\left\lVert x-y\right\rVert^{2},\,\,\forall t\in[0,1]\,\,\forall
x,y\in\mathbb{R}^{d}\textit{ with }\left\lVert x\right\rVert\lor\left\lVert
y\right\rVert\leq R,$ (4.35)

where $\mu$ is defined in (4.28).

  2. (ii)

There exist some constants $p>2$ and $C>0$ such that

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq
1}\left\lVert\widetilde{H}^{(L)}_{t}\right\rVert^{p}\right]\lor\mathbb{E}\left[\sup_{0\leq
t\leq 1}\left\lVert H_{t}\right\rVert^{p}\right]\leq C.$ (4.36)

######  Remark 3.

Note that [15] assumes the uniform integrability condition for
$\widetilde{H}_{t}^{(L)}$ which is difficult to verify in practice. Here we
relax this condition by only assuming the uniform integrability condition for
the ResNet dynamics $\\{h_{k}^{(L)}:k=0,\ldots,L\\}$, see Assumption 3. We can
then prove (4.36) under Assumption 3 and some properties of the Itô processes.

###### Proof of Lemma 2.

First, there exists $C_{2}>0$ such that

$\displaystyle\|Q(t,x)-Q(t,y)\|\leq C_{2}\left\lVert
x-y\right\rVert\,\left\lVert x+y\right\rVert\leq 2C_{2}R\left\lVert
x-y\right\rVert,$ (4.37)

since $Q(t,x)$ is quadratic in $x$ and $\sup_{0\leq t\leq
1}\|\Sigma_{t}^{A}\|\leq C_{1}$. Then,

$\displaystyle\|\mu\left(t,x\right)-\mu\left(t,y\right)\|^{2}$
$\displaystyle=$ $\displaystyle\left\|U_{t}^{A}\,(x-y)+\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}(x-y)+\frac{1}{2}\sigma^{\prime\prime}(0)\left(Q(t,x)-Q(t,y)\right)\right\|^{2}$
$\displaystyle\leq$ $\displaystyle\left(3\,\max_{t\in[0,1]}\left\lVert
U^{A}_{t}\right\rVert+3\,\max_{t\in[0,1]}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}\right\rVert+3\left|\sigma^{\prime\prime}(0)\right|C_{2}^{2}R^{2}\right)\left\lVert
x-y\right\rVert^{2}.$

Note that $\max_{t\in[0,1]}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{t}\right\rVert<\infty$
since $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu\in\mathcal{C}^{0}\left([0,1],\mathbb{R}^{d}\right)$ and
$\max_{t\in[0,1]}\left\lVert U^{A}_{t}\right\rVert<C_{1}$ almost surely
according to (4.9), respectively. Therefore (4.35) holds by taking
$C_{R}=3\,\max_{t\in[0,1]}\left\lVert
U^{A}_{t}\right\rVert+3\,\max_{t\in[0,1]}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}\right\rVert+3\left|\sigma^{\prime\prime}(0)\right|C_{2}^{2}R^{2}$.

Thanks to the assumption in Theorem 3, there exists a constant $C_{3}>0$ such
that $\mathbb{E}\left[\sup_{0\leq t\leq 1}\|H_{t}\|^{p_{1}}\right]\leq C_{3},$
then we only need to show that (4.36) holds for $\widetilde{H}$ for some
$p>2$. To see this, let $k_{s}$ be the integer for which
$s\in[t_{k_{s}},t_{k_{s}+1})$ for a given $s\in[0,1)$. Then

$\displaystyle\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}$ $\displaystyle=$
$\displaystyle
h_{k_{s}}^{(L)}-\left(h_{k_{s}}^{(L)}+\int_{t_{k_{s}}}^{s}\mu\left(t_{k_{r}},\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{r}\right)\mathrm{d}r+\int_{t_{k_{s}}}^{s}\left(\mathrm{d}V_{r}^{A}\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu_{r}^{(L)}+\mathrm{d}V_{r}^{b}\right)\right)$
$\displaystyle=$
$\displaystyle-\mu\left({{t_{k_{s}}}},h_{k_{s}}^{(L)}\right)(s-t_{k_{s}})-\left(V^{A}_{s}-V^{A}_{t_{k_{s}}}\right)\,h_{k_{s}}^{(L)}-\left(V^{b}_{s}-V^{b}_{t_{k_{s}}}\right).$

Hence, by the Minkowski inequality,

$\displaystyle\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}\right\rVert^{p}$
$\displaystyle\leq
3^{p-1}\left(\left\lVert\mu\left({{t_{k_{s}}}},h_{k_{s}}^{(L)}\right)\right\rVert^{p}\left(\Delta_{L}\right)^{p}+\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{p}\,\left\lVert
V^{A}_{s}-V^{A}_{t_{k_{s}}}\right\rVert^{p}+\left\lVert
V^{b}_{s}-V^{b}_{t_{k_{s}}}\right\rVert^{p}\right)$ $\displaystyle\leq
C_{4}\left(\left\lVert h_{k_{s}}^{(L)}\right\rVert^{2p}+\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{p}+1+\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{p}\,\left\lVert
V^{A}_{s}-V^{A}_{t_{k_{s}}}\right\rVert^{p}+\left\lVert
V^{b}_{s}-V^{b}_{t_{k_{s}}}\right\rVert^{p}\right)$ (4.38)

for some $C_{4}>0$, as $\mu(t,h)$ is quadratic in $h$. The value of $p>2$ will
be determined later. From (4.38), we get

$\displaystyle\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}\right\rVert^{p}\right]$
(4.39) $\displaystyle\leq$ $\displaystyle
C_{4}\left(\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu_{s}^{(L)}\right\rVert^{2p}\right]+\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu_{s}^{(L)}\right\rVert^{p}\right]+1\right)$
$\displaystyle+\,\,C_{5}\left(\left(\mathbb{E}\left[\sup_{0\leq s\leq
1}\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu_{s}^{(L)}\right\rVert^{2p}\right]\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert
V^{A}_{s}-V^{A}_{t_{k_{s}}}\right\rVert^{2p}\right]\right)^{1/2}+\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert V^{b}_{s}-V^{b}_{t_{k_{s}}}\right\rVert^{p}\right]\right)$
$\displaystyle\leq$ $\displaystyle C_{4}\left(\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu_{s}^{(L)}\right\rVert^{2p}\right]+\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu_{s}^{(L)}\right\rVert^{p}\right]+1\right)$
$\displaystyle+\,\,C_{6}\left(\left(\mathbb{E}\left[\sup_{0\leq s\leq
1}\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu_{s}^{(L)}\right\rVert^{2p}\right]\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert
V^{A}_{s}\right\rVert^{2p}\right]\right)^{1/2}+\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert V^{b}_{s}\right\rVert^{p}\right]\right),$

for some constants $C_{4},C_{5},C_{6}>0$ independent of $L$, $R$ and $\delta$.
The first inequality holds by the Hölder and (4.39) holds by the Minkowski
inequality. Take $p=\frac{1}{2}p_{1}>2$. Then, (4.39) is bounded thanks to
Assumption 3 for $\mathbb{E}\left[\sup_{0\leq t\leq 1}\|\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu_{t}^{(L)}\|^{2p}\right]<\infty$, and we have
$\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert
W^{A}_{s}\right\rVert^{p}\right]<\infty$ and $\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert W^{b}_{s}\right\rVert^{p}\right]<\infty$ by (4.12). Hence,
by the Minkowski inequality, we have

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq
1}\left\lVert\widetilde{H}^{(L)}_{t}\right\rVert^{p}\right]\leq
2^{p-1}\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}\right\rVert^{p}\right]+2^{p-1}\mathbb{E}\left[\sup_{0\leq
t\leq 1}\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{t}\right\rVert^{p}\right]<\infty.$

∎

####  4.4.4 Proof of Theorem 3

We are now ready to show the proof of Theorem 3.

###### Proof.

Let us define two stopping times to utilize the local Lipschitz property of
$\mu$:

$\displaystyle\tau_{R}:=\inf\left\\{t\geq
0:\left\lVert\widetilde{H}^{(L)}_{t}\right\rVert\geq
R\right\\},\quad\rho_{R}:=\inf\left\\{t\geq 0:\left\lVert
H_{t}\right\rVert\geq R\right\\},\quad\theta_{R}:=\tau_{R}\land\rho_{R},$
(4.40)

and define the approximation errors

$\displaystyle e_{1}(t):=\widetilde{H}^{(L)}_{t}-H_{t},\,\,\text{ and
}\,\,e_{2}(t):=\widetilde{H}^{(L)}_{t}-\mkern
1.0mu\overline{\mkern-1.5muH\mkern 0.0mu}\mkern-1.0mu^{(L)}_{t}.$ (4.41)

The proof contains two steps. The first step is to show
$\lim_{L\rightarrow\infty}\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert
e_{1}(t)\right\rVert^{2}\right]=0$ and the second step is to show
$\lim_{L\rightarrow\infty}\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert
e_{2}(t)\right\rVert^{2}\right]=0$.

Following the idea in [15], we first show that for any $\delta>0$ (to be
determined later):

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert
e_{1}(t)\right\rVert^{2}\right]\leq\mathbb{E}\left[\sup_{0\leq t\leq
1}\left\lVert\widetilde{H}^{(L)}_{t\wedge\theta_{R}}-H_{t\wedge\theta_{R}}\right\rVert^{2}\right]+\frac{2^{p+1}\delta
C}{p}+\frac{2(p-2)C}{p\delta^{2/(p-2)}R^{p}},$ (4.42)

where $C$ and $p$ are defined in (4.36). To see this, recall that by Young’s
inequality, for $r^{-1}+q^{-1}=1$, we have

$ab=\delta^{1/r}a\cdot\delta^{1/q-1}b\leq\frac{\delta}{r}a^{r}+\frac{1}{q\delta^{q/r}}b^{q},\quad\forall
a,b,\delta>0.$ (4.43)

First decompose the left-hand side of (4.42) to obtain, for all $\delta>0$,

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq 1}\|e_{1}(t)\|^{2}\right]$
$\displaystyle=$ $\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq
1}\|e_{1}(t)\|^{2}\mathbbm{1}_{\\{\tau_{R}>1,\rho_{R}>1\\}}\right]+\mathbb{E}\left[\sup_{0\leq
t\leq 1}\|e_{1}(t)\|^{2}\mathbbm{1}_{\\{\tau_{R}\leq 1\text{ or }\rho_{R}\leq
1\\}}\right]$ (4.44) $\displaystyle\leq$
$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq
1}\|e_{1}(t\wedge\theta_{R})\|^{2}\mathbbm{1}_{\\{\theta_{R}>1\\}}\right]+\frac{2\delta}{p}\mathbb{E}\left[\sup_{0\leq
t\leq 1}\|e_{1}(t)\|^{p}\right]$
$\displaystyle\quad+\,\frac{1-2/p}{\delta^{2/(p-2)}}\mathbb{P}\left(\tau_{R}\leq
1\text{ or }\rho_{R}\leq 1\right).$

where we apply (4.43) with $r=p/2$ to the second term. Now

$\displaystyle\mathbb{P}(\tau_{R}\leq
1)=\mathbb{E}\left[\mathbbm{1}_{\\{\tau_{R}\leq
1\\}}\frac{\|\widetilde{H}^{(L)}_{\tau_{R}}\|^{p}}{R^{p}}\right]\leq\frac{1}{R^{p}}\mathbb{E}\left[\sup_{0\leq
t\leq
1}\left\lVert\widetilde{H}^{(L)}_{t}\right\rVert^{p}\right]\leq\frac{C}{R^{p}}.$
(4.45)

A similar result can be derived for $\rho_{R}$, so that we have

$\displaystyle\mathbb{P}(\tau_{R}\leq 1\text{ or }\rho_{R}\leq
1)\leq\mathbb{P}(\tau_{R}\leq 1)+\mathbb{P}(\rho_{R}\leq
1)\leq\frac{2C}{R^{p}}.$ (4.46)

Using the inequalities in (4.45)–(4.46), along with

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert
e_{1}(t)\right\rVert^{p}\right]\leq 2^{p-1}\mathbb{E}\left[\sup_{0\leq t\leq
1}\left(\left\lVert\widetilde{H}^{(L)}_{t}\right\rVert^{p}+\left\lVert
H_{t}\right\rVert^{p}\right)\right]\leq 2^{p}\,C$ (4.47)

in (4.44), we show the desired result (4.42).

To obtain a uniform bound on $\widetilde{H}-H$ , we bound the first term on
the right-hand side of (4.42). Using the definition of the targeted SDE limit
in (4.27):

$H_{t\wedge\theta_{R}}:=H_{0}+\int_{0}^{t\wedge\theta_{R}}\mu(s,H_{s})\mathrm{d}s+\int_{0}^{t\wedge\theta_{R}}\left(\mathrm{d}W_{s}^{A}\,H_{s}+\mathrm{d}W_{s}^{b}\right),$

and the continuous-time approximation (4.34), we get

$\displaystyle\left\lVert\widetilde{H}^{(L)}_{t\wedge\theta_{R}}-H_{t\wedge\theta_{R}}\right\rVert^{2}$
$\displaystyle=$
$\displaystyle\Bigg{\lVert}\int_{0}^{t\wedge\theta_{R}}\left(\mu\left({{t}_{k_{s}}},\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\mathrm{d}s-\mu\left(s,H_{s}\right)\right)\mathrm{d}s$
$\displaystyle\quad+\int_{0}^{t\wedge\theta_{R}}\mathrm{d}W_{s}^{A}\,\left(\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-H_{s}\right)+\sum_{k<L(t\wedge\theta_{R})}D_{k}^{(L)}\left({h}^{(L)}_{k}\right)\Bigg{\rVert}^{2}$
$\displaystyle=$
$\displaystyle\Bigg{\lVert}\int_{0}^{t\wedge\theta_{R}}\left(\mu\left(s,\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)-\mu(s,H_{s})+\mu\left({{t}_{k_{s}}},\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)-\mu\left(s,\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\right)\mathrm{d}s$
$\displaystyle\quad+\int_{0}^{t\wedge\theta_{R}}\mathrm{d}W_{s}^{A}\,\left(\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-H_{s}\right)+\sum_{k<L(t\wedge\theta_{R})}D_{k}^{(L)}\left({h}^{(L)}_{k}\right)\Bigg{\lVert}^{2}.$

We first bound the above using Cauchy-Schwarz inequality:

$\displaystyle\left\lVert\widetilde{H}^{(L)}_{t\wedge\theta_{R}}-H_{t\wedge\theta_{R}}\right\rVert^{2}$
$\displaystyle\leq$ $\displaystyle
4\left[\int_{0}^{t\wedge\theta_{R}}\left\lVert\mu\left(s,\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\mathrm{d}s-\mu\left(s,H_{s}\right)\right\rVert^{2}\mathrm{d}s\right]+4\,\left\lVert\int_{0}^{t\wedge\theta_{R}}\mathrm{d}W_{s}^{A}\,\left(\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-H_{s}\right)\right\rVert^{2}$
$\displaystyle+\,\,4\,\left[\int_{0}^{t\wedge\theta_{R}}\left\lVert\mu\left({{t}_{k_{s}}},\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\mathrm{d}s-\mu\left(s,\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\right\rVert^{2}\mathrm{d}s\right]+4\,\left\lVert\sum_{k<L(t\wedge\theta_{R})}D_{k}^{(L)}\left({h}^{(L)}_{k}\right)\right\rVert^{2}.$

Now, from the local Lipschitz condition (4.35) and Doob’s martingale
inequality [28], we have for any $\tau\leq 1$,

$\displaystyle\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left\lVert\widetilde{H}^{(L)}_{t\wedge\theta_{R}}-H_{t\wedge\theta_{R}}\right\rVert^{2}\right]$
$\displaystyle\leq
4\left(C_{R}+4C^{2}_{1}\right)\mathbb{E}\int_{0}^{\tau\wedge\theta_{R}}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-H_{s}\right\rVert^{2}\mathrm{d}s$
$\displaystyle\,\,+\,4\,\mathbb{E}\left[\int_{0}^{t\wedge\theta_{R}}\left\lVert\mu\left(t_{k_{s}},\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\mathrm{d}s-\mu\left(s,\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\right\rVert^{2}\mathrm{d}s\right]\,+\,4\sum_{k\leq
L\tau}\mathbb{E}\left\lVert
D_{k}^{(L)}\left({h}^{(L)}_{k}\right)\mathbbm{1}_{\|{h}^{(L)}_{k}\|\leq
R}\right\rVert^{2}$ $\displaystyle\leq
C_{R}^{\prime}\int_{0}^{\tau}\mathbb{E}\left[\sup_{0\leq r\leq
s}\left\lVert\widetilde{H}^{(L)}_{r\wedge\theta_{R}}-H_{r\wedge\theta_{R}}\right\rVert^{2}\right]\mathrm{d}s+C_{R}^{\prime}\underbrace{\mathbb{E}\int_{0}^{\tau\wedge\theta_{R}}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}\right\rVert^{2}\mathrm{d}s}_{\raisebox{0.5pt}{\fontsize{8.5pt}{0}\fontfamily{and}\selectfont1⃝}}$
$\displaystyle\,\,+\,4\,\underbrace{\mathbb{E}\left[\int_{0}^{t\wedge\theta_{R}}\left\lVert\mu\left({{t}_{k_{s}}},\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)-\mu\left(s,\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\right\rVert^{2}\mathrm{d}s\right]}_{\raisebox{0.5pt}{\fontsize{8.5pt}{0}\fontfamily{and}\selectfont2⃝}}+\,4\underbrace{\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left\lVert\sum_{k<L(t\wedge\theta_{R})}D_{k}^{(L)}\left({h}^{(L)}_{k}\right)\right\rVert^{2}\right]}_{\raisebox{0.5pt}{\fontsize{8.5pt}{0}\fontfamily{and}\selectfont3⃝}}$
(4.48)

where $C_{R}^{\prime}\coloneqq 8\left(C_{R}+4C^{2}_{1}\right)$. First, we give
an upper bound for \fontsize{8.5pt}{0}\fontfamily{and}\selectfont2⃝. By the
Cauchy–Schwarz inequality,

$\displaystyle\left\lVert\mu(t,h)-\mu(s,h)\right\rVert^{2}\leq 5\,\,\Big{(}$
$\displaystyle\left\lVert U_{t}^{A}-U_{s}^{A}\right\rVert^{2}\,\left\lVert
h\right\rVert^{2}+\,\left\lVert
U_{t}^{b}-U_{s}^{b}\right\rVert^{2}+\left\lVert\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{s}-\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}\right\rVert\left\lVert
h\right\rVert^{2}+\left\lVert\overline{b}_{t}-\overline{b}_{s}\right\rVert^{2}$
$\displaystyle+\frac{1}{2}\sigma^{\prime\prime}(0)\left\lVert
Q(t,h)-Q(s,h)\right\rVert^{2}\Big{)}.$

Hence, for $h\in\mathbb{R}^{d}$, the following holds almost surely by (4.10):

$\displaystyle\left\lVert\mu(t,h)-\mu(s,h)\right\rVert^{2}\leq
C_{M}\left|t-s\right|^{\kappa}\left(1+\left\lVert
h\right\rVert^{2}+\left\lVert h\right\rVert^{4}\right).$ (4.49)

Under Assumption 3, there exists a constant $\widetilde{C}_{0}>0$ such that

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq 1}\left(\left\lVert\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{t}\right\rVert^{4}+\left\lVert\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{t}\right\rVert^{2}\right)\right]\leq\widetilde{C}_{0}.$

Hence by Tonelli’s theorem,

$\displaystyle\mathbb{E}\left[\int_{0}^{t\wedge\theta_{R}}\left\lVert\mu\left({{t}_{k_{s}}},\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)-\mu\left(s,\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\right\rVert^{2}\mathrm{d}s\right]$
$\displaystyle\leq\int_{0}^{1}\mathbb{E}\left[\left\lVert\mu\left({{t}_{k_{s}}},\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)-\mu\left(s,\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}\right)\right\rVert^{2}\right]\mathrm{d}s$
$\displaystyle\leq(\widetilde{C}_{0}+1)C_{M}L\left(\int_{0}^{1/L}r^{\kappa}\mathrm{d}r\right)$
$\displaystyle=\frac{(\widetilde{C}_{0}+1)C_{M}}{1+\kappa}L^{-\kappa}.$ (4.50)

Upper bound on \fontsize{8.5pt}{0}\fontfamily{and}\selectfont3⃝. Define the
following discrete filtration

$\mathcal{G}_{k}\coloneqq\sigma\Big{(}U_{s}^{A},U_{s}^{A},q_{s}^{A},q_{s}^{b},B_{s}^{A},B_{s}^{b}\,:\,\,s\leq
t_{k+1}\Big{)}.$ (4.51)

Note that $h_{k}^{(L)}$ is $\mathcal{G}_{k-1}$-measurable but not
$\mathcal{G}_{k}$-measurable. Define for $k=0,\ldots,L-1$ and for
$i=1,\ldots,d$:

$\displaystyle X^{(L)}_{k,i}$ $\displaystyle:=$
$\displaystyle\left(\left(\Delta
V_{k}^{A}\,h_{k}^{(L)}\right)_{i}+\left(\Delta
V_{k}^{b}\right)_{i}\right)^{2}-\mathbb{E}\left[\left.\left(\left(\Delta
V_{k}^{A}\,h_{k}^{(L)}\right)_{i}+\left(\Delta
V_{k}^{b}\right)_{i}\right)^{2}\right|\mathcal{G}_{k-1}\right]$ $\displaystyle
Y^{(L)}_{k,i}$ $\displaystyle:=$
$\displaystyle\mathbb{E}\left[\left.\left(\left(\Delta
V_{k}^{A}\,h_{k}^{(L)}\right)_{i}+\left(\Delta
V_{k}^{b}\right)_{i}\right)^{2}\right|\mathcal{G}_{k-1}\right]-Q_{i}\left(t_{k},h_{k}^{(L)}\right)\Delta_{L}$
$\displaystyle J^{(L)}_{k,i}$ $\displaystyle:=$
$\displaystyle\widetilde{\mu}_{i}(t,h)^{2}(\Delta_{L})^{2}+2\widetilde{\mu}_{i}(t,h)\Delta_{L}\left(\left(\Delta
V_{k}^{A}\,h\right)_{i}+\left(\Delta V_{k}^{b}\right)_{i}\right).$

We can then decompose

$\displaystyle D_{k,i}^{(L)}\left(h_{k}^{(L)}\right)$
$\displaystyle=\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)^{3}+N_{k,i}^{(L)}\left(h_{k}^{(L)}\right)$
$\displaystyle=\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)^{3}+\frac{1}{2}\sigma^{\prime\prime}(0)\left(X^{(L)}_{k,i}+Y^{(L)}_{k,i}+J^{(L)}_{k,i}\right).$

Hence, we deduce the following bound on
\fontsize{8.5pt}{0}\fontfamily{and}\selectfont3⃝ by Cauchy-Schwarz.

$\displaystyle\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left|\sum_{k<L(t\wedge\theta_{R})}D_{k,i}^{(L)}\left({h}^{(L)}_{k}\right)\right|^{2}\right]$
$\displaystyle\leq\sigma^{\prime\prime}(0)^{2}\,\,\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left|\sum_{k<L(t\wedge\theta_{R})}X^{(L)}_{k,i}\,\right|^{2}+\left|\sum_{k<L(t\wedge\theta_{R})}Y^{(L)}_{k,i}\,\right|^{2}+L\sum_{k<L(t\wedge\theta_{R})}\left|J^{(L)}_{k,i}\right|^{2}\right]$
$\displaystyle\quad+\,4\,\,\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left|\sum_{k<L(t\wedge\theta_{R})}\frac{1}{6}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)^{3}\right|^{2}\right]$
$\displaystyle\leq\sigma^{\prime\prime}(0)^{2}\,\,\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left|\sum_{k<Lt}X^{(L)}_{k,i}\mathbbm{1}_{\left\lVert{h}^{(L)}_{k}\right\rVert\leq
R}\right|^{2}\right]+\sigma^{\prime\prime}(0)^{2}\,\,\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left|\sum_{k<Lt}Y^{(L)}_{k,i}\mathbbm{1}_{\left\lVert{h}^{(L)}_{k}\right\rVert\leq
R}\right|^{2}\right]$
$\displaystyle\quad+\,\sigma^{\prime\prime}(0)^{2}L\sum_{k=0}^{L-1}\mathbb{E}\left[\left|J^{(L)}_{k,i}\right|^{2}\mathbbm{1}_{\left\lVert{h}^{(L)}_{k}\right\rVert\leq
R}\right]+\frac{1}{9}\sigma^{\prime\prime\prime}\left(\xi^{(L)}_{k,i}\right)^{2}L\sum_{k=0}^{L-1}\mathbb{E}\left[M^{(L)}_{k,i}\left(h_{k}^{(L)}\right)^{6}\mathbbm{1}_{\left\lVert{h}^{(L)}_{k}\right\rVert\leq
R}\right].$ (4.52)

We provide an upper bound for each of the four terms in (4.52). For the first
term, denote $\widetilde{X}^{(L)}_{k,i}\coloneqq
X^{(L)}_{k,i}\mathbbm{1}\left(\big{\lVert}{h}^{(L)}_{k}\big{\rVert}\leq
R\right)$ and
$S^{(L)}_{k,i}\coloneqq\sum_{k^{\prime}=0}^{k}\widetilde{X}^{(L)}_{k^{\prime},i}$
so that $\big{\\{}S^{(L)}_{k,i}:k=-1,0,\ldots,L-1\big{\\}}$ is a
$\left(\mathcal{G}_{k}\right)$–martingale. Hence, by Doob’s martingale
inequality, we have

$\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left|\sum_{k<Lt}X^{(L)}_{k,i}\mathbbm{1}_{\left\lVert{h}^{(L)}_{k}\right\rVert\leq
R}\right|^{2}\right]=\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left|S^{(L)}_{\left\lfloor Lt\right\rfloor,i}\right|^{2}\right]\leq
4\,\mathbb{E}\left[\left|S^{(L)}_{\left\lfloor
L\tau\right\rfloor,i}\right|^{2}\right].$ (4.53)

Fix $k=0,\ldots,L-1$. For $i=1,\ldots,d$, we compute the following conditional
expectation.

$\displaystyle\mathbb{E}\left[\left(S^{(L)}_{k,i}\right)^{2}\,\Big{|}\,\,\mathcal{G}_{k-1}\right]$
$\displaystyle=\mathbb{E}\left[\left(S^{(L)}_{k-1,i}\right)^{2}+2\widetilde{X}^{(L)}_{k,i}\sum_{k^{\prime}=0}^{k-1}\widetilde{X}^{(L)}_{k^{\prime},i}+\left(\widetilde{X}^{(L)}_{k,i}\right)^{2}\,\Bigg{|}\,\,\mathcal{G}_{k-1}\right]$
$\displaystyle=\left(S^{(L)}_{k-1,i}\right)^{2}+\mathbb{E}\left[\left(\widetilde{X}^{(L)}_{k,i}\right)^{2}\,\Big{|}\,\,\mathcal{G}_{k-1}\right].$
(4.54)

The cross-term disappear as
$\mathbb{E}\left[\left.\widetilde{X}^{(L)}_{k,i}\,\right|\,\mathcal{G}_{k-1}\right]=\mathbb{E}\left[\left.X^{(L)}_{k,i}\,\right|\,\mathcal{G}_{k-1}\right]\mathbbm{1}\left(\big{\lVert}{h}^{(L)}_{k}\big{\rVert}\leq
R\right)=0$ by definition of $X^{(L)}_{k,i}$. Furthermore, conditionally on
$\mathcal{G}_{k-1}$ and on $\big{\\{}\big{\lVert}{h}^{(L)}_{k}\big{\rVert}\leq
R\big{\\}}$, observe that $X^{(L)}_{k,i}$ is the centered square of a normal
random variable whose variance is $\mathcal{O}(L^{-1})$ uniformly in $k$ by
(4.9), so there exist $C_{R,1}>0$ depending only on $R$ such that

$\sup_{0\leq
k<L}\mathbb{E}\left[\left(\widetilde{X}^{(L)}_{k,i}\right)^{2}\,\Big{|}\,\,\mathcal{G}_{k-1}\right]\leq
C_{R,1}L^{-2}.$

Hence, plugging back into (4.53), we obtain

$\mathbb{E}\left[\sup_{0\leq t\leq\tau}\left|\sum_{k\leq
Lt}X_{k}^{i}\mathbbm{1}_{\left\lVert{h}^{(L)}_{k}\right\rVert\leq
R}\right|^{2}\right]\leq 4C_{R,1}L^{-1}.$ (4.55)

For the second term involving $Y^{(L)}_{k,i}$, we explicitly compute the
conditional expectation using the definition of $V$ in (4.28) and the
definition of $Q$ in (4.13).

$\displaystyle Y^{(L)}_{k,i}$
$\displaystyle=\mathbb{E}\left.\left[\left(\Delta
V^{b}_{k}\right)_{i}^{2}+\sum_{j,l=1}^{d}h^{(L)}_{k,j}h^{(L)}_{k,l}\left(\Delta
V_{k}^{A}\right)_{ij}\left(\Delta
V_{k}^{A}\right)_{il}\,\right|\,\mathcal{G}_{k-1}\right]-Q_{i}\left(t_{k},h_{k}^{(L)}\right)\Delta_{L}$
$\displaystyle=\int_{t_{k}}^{t_{k+1}}\left(\mathbb{E}\left[\left.\Sigma^{b}_{s,ii}\,\right|\,\mathcal{G}_{k-1}\right]+\sum_{j,l=1}^{d}h^{(L)}_{k,j}h^{(L)}_{k,l}\mathbb{E}\left[\left.\Sigma^{A}_{s,ijil}\,\right|\,\mathcal{G}_{k-1}\right]\right)\mathrm{d}s-Q_{i}\left(t_{k},h_{k}^{(L)}\right)\Delta_{L}$
$\displaystyle=\int_{t_{k}}^{t_{k+1}}\left(\mathbb{E}\left[\left.\Sigma^{b}_{s,ii}-\Sigma^{b}_{t_{k},ii}\,\right|\,\mathcal{G}_{k-1}\right]+\sum_{j,l=1}^{d}h^{(L)}_{k,j}h^{(L)}_{k,l}\mathbb{E}\left[\left.\Sigma^{A}_{s,ijil}-\Sigma^{A}_{t_{k},ijil}\,\right|\,\mathcal{G}_{k-1}\right]\right)\mathrm{d}s.$

By Cauchy-Schwarz, Tonelli and (4.10) in Assumption 2 $(ii)$ we obtain:

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq\tau}\left|\sum_{k\leq
L(t\wedge\theta_{R})}Y^{(L)}_{k,i}\mathbbm{1}_{\left\lVert{h}^{(L)}_{k}\right\rVert\leq
R}\right|^{2}\right]\leq\mathbb{E}\left[\left(\sum_{k=0}^{L-1}\left|Y^{(L)}_{k,i}\right|\mathbbm{1}_{\left\lVert{h}^{(L)}_{k}\right\rVert\leq
R}\right)^{2}\right]$
$\displaystyle\leq\mathbb{E}\left[\left(\sum_{k=0}^{L-1}\int_{t_{k}}^{t_{k+1}}\left(\mathbb{E}\left[\left|\Sigma^{b}_{s,ii}-\Sigma^{b}_{t_{k},ii}\right|\nonscript\;\middle|\nonscript\;\mathcal{G}_{k-1}\right]+R^{2}\sum_{j,l=1}^{d}\mathbb{E}\left[\left|\Sigma^{A}_{s,ijil}-\Sigma^{A}_{t_{k},ijil}\right|\nonscript\;\middle|\nonscript\;\mathcal{G}_{k-1}\right]\right)\mathrm{d}s\right)^{2}\right]$
$\displaystyle\leq\left(\sum_{k=0}^{L-1}\int_{t_{k}}^{t_{k+1}}(1+R^{2})M^{1/2}\left|s-t_{k_{s}}\right|^{\kappa/2}\mathrm{d}s\right)^{2}$
$\displaystyle=M(1+R^{2})^{2}\left(L\int_{0}^{1/L}r^{\kappa/2}\mathrm{d}r\right)^{2}=\frac{M(1+R^{2})^{2}}{(1+\kappa/2)^{2}}L^{-\kappa}\eqqcolon
C_{R,2}L^{-\kappa},$ (4.56)

where $C_{R,2}>0$ depends only on $R$. Moving to the third term of (4.52)
involving $J^{(L)}_{k,i}$, observe that

$\displaystyle\sup_{\left\lVert h\right\rVert\leq
R}\mathbb{E}\left[\left|\sum_{j=1}^{d}\int_{t_{k}}^{t_{k+1}}(\mathrm{d}V_{t}^{A})_{ij}h_{j}\right|^{2}\right]$
$\displaystyle\leq
R^{2}d\sum_{j=1}^{d}\mathbb{E}\left[\left|\sum_{l,m=1}^{d}\int_{t_{k}}^{t_{k+1}}\left(q_{s}^{A}\right)_{ijlm}\left(\mathrm{d}B_{s}^{A}\right)_{lm}\right|^{2}\right]\leq
C_{7}R^{2}\Delta_{L},$
$\displaystyle\mathbb{E}\left[\left|\int_{t_{k}}^{t_{k+1}}(\mathrm{d}V_{t}^{b})_{i}\right|^{2}\right]$
$\displaystyle=\mathbb{E}\left[\left|\sum_{l=1}^{d}\int_{t_{k}}^{t_{k+1}}\left(q_{r}^{b}\right)_{il}\left(\mathrm{d}B_{s}^{b}\right)_{l}\right|^{2}\right]\leq
C_{8}\Delta_{L},$

for some $C_{7},C_{8}>0$ independent of $R$ since $\sup_{0\leq t\leq
1}\left\lVert\Sigma^{A}_{t}\right\rVert\leq C_{1}$ and $\sup_{0\leq t\leq
1}\left\lVert\Sigma^{b}_{t}\right\rVert\leq C_{1}$ almost surely. Then there
exists $C_{R,3}>0$ depending only on $R$ such that

$\displaystyle\sup_{\|h\|\leq
R}\mathbb{E}\left[\left|J^{(L)}_{k,i}\right|^{2}\mathbbm{1}_{\left\lVert
h_{k}^{(L)}\right\rVert\leq R}\right]\leq C_{R,3}L^{-3}.$ (4.57)

Finally, we bound the fourth term of (4.52) using Cauchy-Schwarz, Assumption 1
and property (4.9) of the Itô processes:

$\displaystyle\sigma^{\prime\prime\prime}(\xi_{i})^{2}\sup_{\|h\|\leq
R}\mathbb{E}\left[M^{(L)}_{k,i}\big{(}h\big{)}^{6}\right]\leq
m^{2}\,C_{R,4}L^{-3},$ (4.58)

for some constant $C_{R,4}>0$ depending only on $R$. Combining the results in
(4.55), (4.56), (4.57) and (4.58), there exists constants $C_{R,5},C_{R,6}>0$
depending only on $R$ such that

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq\tau}\left|\sum_{k\leq
L(t\wedge\theta_{R})}D_{k}^{(L),i}\left({h}^{(L)}_{k}\right)\right|^{2}\right]\leq\frac{C_{R,5}}{4d}L^{-\kappa}+\frac{C_{R,6}}{4d}L^{-1}.$
(4.59)

Upper bound on \fontsize{8.5pt}{0}\fontfamily{and}\selectfont1⃝. Given
$s\in[0,T\wedge\theta_{R})$, we have

$\displaystyle\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}$ $\displaystyle=$
$\displaystyle
h_{k_{s}}^{(L)}-\left(h_{k_{s}}^{(L)}+\int_{t_{k_{s}}}^{s}\mu(s,\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s})\mathrm{d}s+\int_{t_{k_{s}}}^{s}\left(\mathrm{d}V_{s}^{A}\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu_{s}^{(L)}+\mathrm{d}V_{s}^{b}\right)\right)$ (4.60)
$\displaystyle=$
$\displaystyle-\mu\left(t_{k_{s}},h_{k_{s}}^{(L)}\right)(s-t_{k_{s}})-\left(V^{A}_{s}-V^{A}_{t_{k_{s}}}\right)\,h_{k_{s}}^{(L)}-\left(V^{b}_{s}-V^{b}_{t_{k_{s}}}\right)$

by continuity of $\mu$. Hence

$\displaystyle\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}\right\rVert^{2}\leq
3\left\lVert\mu\left(t_{k_{s}},h_{k_{s}}^{(L)}\right)\right\rVert^{2}(\Delta_{L})^{2}+3\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{2}\left\lVert
V^{A}_{s}-V^{A}_{t_{k_{s}}}\right\rVert^{2}+3\left\lVert
V^{b}_{s}-V^{b}_{t_{k_{s}}}\right\rVert^{2}.$ (4.61)

Now, from the local Lipschitz condition (4.35), for $\left\lVert
h\right\rVert\leq R$ we have almost surely

$\displaystyle\left\lVert\mu(s,h)\right\rVert^{2}\leq
2\left(\left\lVert\mu(s,h)-\mu(s,0)\right\rVert^{2}+\left\lVert\mu(s,0)\right\rVert^{2}\right)\leq
2\left(C_{R}\left\lVert
h\right\rVert^{2}+\left\lVert\mu(s,0)\right\rVert^{2}\right).$

Combining the two previous inequalities we obtain

$\displaystyle\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}\right\rVert^{2}\leq
4\left(C_{R}\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{2}+\left\lVert\mu(s,0)\right\rVert^{2}+1\right)\left(\Delta_{L}^{2}+\left\lVert
V^{A}_{s}-V^{A}_{t_{k_{s}}}\right\rVert^{2}+\left\lVert
V^{b}_{s}-V^{b}_{t_{k_{s}}}\right\rVert^{2}\right).$

Hence, using (4.36) and the Lyapunov inequality [27], we get

$\displaystyle\mathbb{E}\int_{0}^{\tau\wedge\theta_{R}}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}\right\rVert^{2}\mathrm{d}s$
(4.62) $\displaystyle\leq$
$\displaystyle\mathbb{E}\int_{0}^{\tau\wedge\theta_{R}}4\left(C_{R}\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{2}+\left\lVert\mu(s,0)\right\rVert^{2}+1\right)\left(\Delta_{L}^{2}+\left\lVert
V^{A}_{s}-V^{A}_{t_{k_{s}}}\right\rVert^{2}+\left\lVert
V^{b}_{s}-V^{b}_{t_{k_{s}}}\right\rVert^{2}\right)\,\mathrm{d}s$
$\displaystyle\leq$
$\displaystyle\int_{0}^{\tau}4\,\,\mathbb{E}\left[\left(C_{R}\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{2}+\left\lVert\mu(s,0)\right\rVert^{2}+1\right)\left(\Delta_{L}^{2}+\left\lVert
V^{A}_{s}-V^{A}_{t_{k_{s}}}\right\rVert^{2}+\left\lVert
V^{b}_{s}-V^{b}_{t_{k_{s}}}\right\rVert^{2}\right)\right]\,\mathrm{d}s$
$\displaystyle\leq$
$\displaystyle\int_{0}^{1}4\left(C_{R}\,\mathbb{E}\left[\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{2}\right]+\left\lVert\mu(s,0)\right\rVert^{2}+1\right)\left(\Delta_{L}^{2}+2C_{1}\Delta_{L}+2C_{1}\Delta_{L}\right)\,\mathrm{d}s$
$\displaystyle\leq$ $\displaystyle
4\left(C_{R}\,C_{0}^{2/p}+1+\int_{0}^{1}\left\lVert\mu(s,0)\right\rVert^{2}\mathrm{d}s\right)\left(\Delta_{L}^{2}+4C_{1}\Delta_{L}\right).$

Combining the results in (4.50), (4.59) and (4.62), we have in (4.48) that

$\displaystyle\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left\lVert\widetilde{H}^{(L)}_{\tau\wedge\theta_{R}}-H_{t\wedge\theta_{R}}\right\rVert^{2}\right]\leq
C_{R}^{\prime}\left(C_{R}\,C_{0}^{2/p}+1+\int_{0}^{1}\left\lVert\mu(s,0)\right\rVert^{2}\mathrm{d}s\right)\left(L^{-2}+4C_{1}L^{-1}\right)$
$\displaystyle+\,\frac{(\widetilde{C}_{0}+1)C_{M}}{1+\kappa}L^{-\kappa}+\left(C_{R,5}L^{-\kappa}+C_{R,6}L^{-1}\right)+C_{R}^{\prime}\int_{0}^{\tau}\mathbb{E}\left[\sup_{0\leq
r\leq
s}\left\lVert\widetilde{H}^{(L)}_{r\wedge\theta_{R}}-H_{r\wedge\theta_{R}}\right\rVert^{2}\right]\mathrm{d}s.$

Applying the Grönwall inequality,

$\displaystyle\mathbb{E}\left[\sup_{0\leq
t\leq\tau}\left\lVert\widetilde{H}^{(L)}_{\tau\wedge\theta_{R}}-H_{t\wedge\theta_{R}}\right\rVert^{2}\right]\leq
C_{9}C_{R,7}{L^{-\min\\{1,\kappa\\}}}\exp(C_{R}^{\prime}),$ (4.63)

where $C_{9}$ is a universal constant independent of $L$, $R$ and $\delta$ and
$C_{R,7}$ is a constant only depending on $R$. Combining (4.63) with (4.42),
we have

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert
e_{1}(t)\right\rVert^{2}\right]\leq
C_{9}C_{R,7}L^{-\min\\{1,\kappa\\}}\exp(C_{R}^{\prime})+\frac{2^{p+1}\delta
C}{p}+\frac{2(p-2)C}{p\delta^{2/(p-2)}R^{p}}.$ (4.64)

Given any $\epsilon>0$, we can choose $\delta>0$ so that $\frac{2^{p+1}\delta
C}{p}<\frac{\epsilon}{3}$, then choose $R$ so that
$\frac{2(p-2)C}{p\delta^{2/(p-2)}R^{p}}<\frac{\epsilon}{3}$, and finally
choose $L$ sufficiently large so that

$C_{9}C_{R,7}L^{-\min\\{1,\kappa\\}}\exp(C_{R}^{\prime})\leq\frac{\epsilon}{3}.$

Therefore in (4.64), we have,

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert
e_{1}(t)\right\rVert^{2}\right]\leq\epsilon.$ (4.65)

It remains to provide a uniform bound for $\mkern
1.0mu\overline{\mkern-1.5muH\mkern 0.0mu}\mkern-1.0mu-\widetilde{H}$. Recall
the relationship between $\widetilde{H}$ and $\mkern
1.0mu\overline{\mkern-1.5muH\mkern 0.0mu}\mkern-1.0mu$ defined in (4.60): by
(4.9) we have almost surely that

$\displaystyle\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}\right\rVert^{2}$
$\displaystyle\leq$ $\displaystyle
3\left(\left\lVert\mu\left({{t_{k_{s}}}},h_{k_{s}}^{(L)}\right)\right\rVert^{2}\left(\Delta_{L}\right)^{2}+\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{2}\,\left\lVert
V^{A}_{s}-V^{A}_{t_{k_{s}}}\right\rVert^{2}+\left\lVert
V^{b}_{s}-V^{b}_{t_{k_{s}}}\right\rVert^{2}\right)$ $\displaystyle\leq$
$\displaystyle C_{10}\left(\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{4}+\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{2}+1\right)\,(\Delta_{L})^{2}$
$\displaystyle+\,\,3\left(\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{2}\,\left\lVert
V^{A}_{s}-V^{A}_{t_{k_{s}}}\right\rVert^{2}+\left\lVert
V^{b}_{s}-V^{b}_{t_{k_{s}}}\right\rVert^{2}\right).$

Therefore,

$\displaystyle\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{s}-\widetilde{H}^{(L)}_{s}\right\rVert^{2}\right]\leq
C_{10}\left(\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{4}\right]+\mathbb{E}\left[\sup_{0\leq s\leq
1}\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{2}\right]+1\right)\,(\Delta_{L})^{2}$
$\displaystyle+3\,\left(\left(\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{4}\right]\,\mathbb{E}\left[\sup_{0\leq s\leq
1}\left\lVert
V^{A}_{s}-V^{A}_{t_{k_{s}}}\right\rVert^{4}\right]\right)^{1/2}+\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert
V^{b}_{s}-V^{b}_{t_{k_{s}}}\right\rVert^{2}\right]\right).$ (4.66)

First, by Assumption 3,

$\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert
h_{k_{s}}^{(L)}\right\rVert^{n}\right]=\mathbb{E}\left[\sup_{0\leq s\leq
1}\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu_{s}^{(L)}\right\rVert^{n}\right]<\infty,\quad
n\in\left\\{2,4\right\\}.$ (4.67)

Second, by the Power Mean inequality and Doob’s martingale inequality,

$\displaystyle\mathbb{E}\left[\sup_{t_{k}\leq s<t_{k+1}}\left\lVert
V_{s}^{A}-V_{t_{k}}^{A}\right\rVert^{4}\right]$
$\displaystyle=\mathbb{E}\left[\sup_{t_{k}\leq
s<t_{k+1}}\left(\sum_{i,j=1}^{d}\left|\sum_{k,l=1}^{d}\int_{t_{k}}^{s}\left(q_{r}^{A}\right)_{ijkl}\left(\mathrm{d}B_{r}^{A}\right)_{kl}\right|^{2}\right)^{2}\right]$
$\displaystyle\leq d^{8}\sum_{i,j,k,l=1}^{d}\mathbb{E}\left[\sup_{t_{k}\leq
s<t_{k+1}}\left|\int_{t_{k}}^{s}\left(q_{r}^{A}\right)_{ijkl}\left(\mathrm{d}B_{r}^{A}\right)_{kl}\right|^{4}\right]$
$\displaystyle\leq\Big{(}\frac{4}{3}\Big{)}^{4}d^{8}\sum_{i,j,k,l=1}^{d}\mathbb{E}\left[\left|\int_{t_{k}}^{t_{k+1}}\left(q_{s}^{A}\right)_{ijkl}\left(\mathrm{d}B_{s}^{A}\right)_{kl}\right|^{4}\right]\leq
C_{11}\Delta^{2}_{L},$

Hence

$\displaystyle\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert
V_{s}^{A}-V_{t_{k_{s}}}^{A}\right\rVert^{4}\right]\leq\mathbb{E}\left[\sum_{k=0}^{L-1}\left(\sup_{t_{k}\leq
s<t_{k+1}}\left\lVert
V_{s}^{A}-V_{t_{k_{s}}}^{A}\right\rVert^{4}\right)\right]\leq
C_{11}\Delta_{L}.$ (4.68)

By Hölder inequality,

$\displaystyle\mathbb{E}\left[\sup_{0\leq s\leq 1}\left\lVert
V_{s}^{A}-V_{t_{k_{s}}}^{A}\right\rVert^{2}\right]\leq\left(\mathbb{E}\left[\sup_{0\leq
s\leq 1}\left\lVert
V_{s}^{A}-V_{t_{k_{s}}}^{A}\right\rVert^{4}\right]\right)^{1/2}\leq\sqrt{C_{11}}\Delta^{1/2}_{L}.$
(4.69)

Combining (4.67), (4.68), and (4.69) in (4.66), we obtain

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert
e_{2}(t)\right\rVert^{2}\right]=\mathbb{E}\left[\sup_{0\leq t\leq
1}\left\lVert\mkern 1.0mu\overline{\mkern-1.5muH\mkern
0.0mu}\mkern-1.0mu^{(L)}_{t}-\widetilde{H}^{(L)}_{t}\right\rVert^{2}\right]\leq
C_{12}\Delta_{L}^{1/2},$

for some constant $C_{12}>0$. By choosing $L>(C_{12}/\epsilon)^{2}$, we have

$\displaystyle\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert
e_{2}(t)\right\rVert^{2}\right]\leq{\epsilon}.$ (4.70)

Finally, combining (4.65) and (4.70) leads to the desired result.

∎

##  5 Asymptotic analysis of the backpropagation dynamics

The most widely used method to train neural networks is the pairing of

  * •

the backpropagation algorithm to find the exact gradient (or a stochastic
approximation) of the loss function with respect to the network weights, and

  * •

a variant of the gradient descent algorithm to iteratively update the network
weights.

We are interested to study the behaviour of the former in residual networks,
under our Scaling regimes 1 and 2. To do so, we will first formalize the
objective function and the discrete backward equation linking the gradient of
the loss function across layers.

###  5.1 Backpropagation in supervised learning

Suppose we want to learn the mapping
$f_{\mathrm{true}}\in\mathcal{C}^{1}(\mathbb{R}^{d},\mathbb{R}^{d})$ through a
dataset of input-target pairs
$\mathcal{D}\coloneqq\left\\{(x_{i},y_{i}):i=1,\ldots,N\right\\}\subset\mathbb{R}^{d}\times\mathbb{R}^{d}$,
where $x_{i}\in B$ for some $B\subset\mathbb{R}^{d}$ compact and
$y_{i}=f_{\mathrm{true}}(x_{i})$. The goal of any parametric supervised
learning is to find, given a class of mappings
$\phi_{\theta}:\mathbb{R}^{d}\to\mathbb{R}^{d}$, the parameter
$\theta\in\Theta$ that minimizes the average training error:

$J_{\mathcal{D}}(\theta)\coloneqq\frac{1}{N}\sum_{i=1}^{N}\ell(\phi_{\theta}(x_{i}),y_{i})=\frac{1}{N}\sum_{i=1}^{N}\ell(\phi_{\theta}(x_{i}),f_{\mathrm{true}}(x_{i})).$
(5.1)

Here, $\ell:\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R}_{+}$ is a loss
function, for example the squared error $\ell(\widehat{y},y)=\left\lVert
y-\widehat{y}\right\rVert^{2}$. In the following, we omit the dependence in
$\mathcal{D}$. Fix $L\in\mathbb{N}$ and define

$\theta^{(L)}\coloneqq\left(A^{(L)}_{k},b^{(L)}_{k}\right)_{k=1}^{L}\in\left(\mathbb{R}^{d\times
d}\times\mathbb{R}^{d}\right)^{L}.$

For an input $x\in\mathbb{R}^{d}$, recall the following forward dynamics for
the residual network

$\displaystyle h_{0}^{(L),x}$ $\displaystyle=x,$ (5.2) $\displaystyle
h^{(L),x}_{k+1}$
$\displaystyle=h^{(L),x}_{k}+L^{-\alpha}\,\sigma_{d}\left(A^{(L)}_{k}h^{(L),x}_{k}+b^{(L)}_{k}\right).$

We define $\phi_{\theta^{(L)}}(x)\coloneqq h_{L}^{(L),x}$. Our goal is to
compute $\nabla_{\theta^{(L)}}J\left(\theta^{(L)}\right)$. Observe from the
definition (5.1) and the chain rule that

$\nabla_{\theta_{k}^{(L)}}J\left(\theta^{(L)}\right)=\frac{1}{N}\sum_{i=1}^{N}\nabla_{\theta_{k}^{(L)}}h^{(L),\,x_{i}}_{k+1}\left(\frac{\partial
h_{L}^{(L),\,x_{i}}}{\partial
h^{(L),\,x_{i}}_{k+1}}\frac{\partial\ell}{\partial\widehat{y}}\left(h_{L}^{(L),\,x_{i}},y_{i}\right)\right).$

The terms $\partial\ell/\partial\widehat{y}$ and $\nabla_{\theta_{k}}h_{k+1}$
are straightforward to obtain, so the crux of the challenge lies in computing
$\partial h_{L}/\partial h_{k+1}$. Using (5.2), for $x\in\mathbb{R}^{d}$, we
get

$\displaystyle g_{k}^{(L),x}\coloneqq\frac{\partial h_{L}^{(L),\,x}}{\partial
h^{(L),x}_{k}}$ $\displaystyle=\frac{\partial h_{L}^{(L),\,x}}{\partial
h^{(L),x}_{k+1}}\frac{\partial h^{(L),x}_{k+1}}{\partial h^{(L),x}_{k}}$
$\displaystyle=g_{k+1}^{(L),x}\left(I_{d}+L^{-\alpha}\,\mathrm{diag}\left(\sigma^{\prime}_{d}\left(A_{k}^{(L)}h_{k}^{(L),x}+b_{k}^{(L)}\right)\right)A^{(L)}_{k}\right),$
(5.3)

where
$\sigma^{\prime}_{d}(z)=\left(\sigma^{\prime}(z_{i})\right)_{i=1}^{d}\in\mathbb{R}^{d}$
for $z\in\mathbb{R}^{d}$. The terminal condition is given by
$g_{L}^{(L),x}=I_{d}$. We now obtain the asymptotic dynamics of $g$ under
three different cases. In particular, we derive (backward) ODE limits for any
set of weights under Scaling regime 1 and the asymptotic limit derived from an
SDE under Scaling regime 2. For clarity, we omit the dependence in the input
$x$ for $g_{k}^{(L)}$.

###  5.2 Backward equation for the Jacobian under Scaling regime 1

Let $\mkern 1.0mu\overline{\mkern-1.5muG\mkern 0.0mu}\mkern
0.0mu^{(L)}:[0,1]\to\mathbb{R}^{d\times d}$ be a continuous-time extension of
the Jacobians $g^{(L)}_{k}$ defined in (5.3):

$\displaystyle\mkern 1.0mu\overline{\mkern-1.5muG\mkern 0.0mu}\mkern
0.0mu^{(L)}_{t}=g_{k+1}^{(L)}\mathbbm{1}_{\frac{k}{L}<t\leq\frac{k+1}{L}},\quad
k=0,1,\ldots,L-1.$ (5.4)

######  Theorem 4 (Backpropagation limits under Scaling regime 1).

Under the same assumptions as Theorem 1,

  * •

Neural ODE regime: If $\alpha=1$, $\beta=0$, and
$(H_{t})_{t\in\left[0,1\right]}$ is the solution to the neural ODE (4.3), then
the backpropagation dynamics converge uniformly to the solution to the linear
(backward) ODE

$\displaystyle\frac{\mathrm{d}G_{t}}{\mathrm{d}t}=-G_{t}\mathrm{diag}\left(\sigma^{\prime}_{d}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}H_{t}+\overline{b}_{t}\right)\right)\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{t},\quad G_{1}=I_{d}$
(5.5)

in the sense that $\lim_{L\rightarrow\infty}\sup_{0\leq t\leq 1}\|G_{t}-\mkern
1.0mu\overline{\mkern-1.5muG\mkern 0.0mu}\mkern 0.0mu^{(L)}_{t}\|=0$.

  * •

Linear ODE regime: If $\alpha+\beta=1$, $\beta>0$, and
$(H_{t})_{t\in\left[0,1\right]}$ is the solution to the linear ODE (4.4), then
the backpropagation dynamics converge uniformly to the solution to the linear
(backward) ODE

$\displaystyle\frac{\mathrm{d}G_{t}}{\mathrm{d}t}=-G_{t}\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{t},\quad G_{1}=I_{d}$
(5.6)

in the sense that $\lim_{L\rightarrow\infty}\sup_{0\leq t\leq 1}\|G_{t}-\mkern
1.0mu\overline{\mkern-1.5muG\mkern 0.0mu}\mkern 0.0mu^{(L)}_{t}\|=0$.

The ideas of the proof follow closely those of Theorem 1 and the complete
proof is given in Section 5.4.1. We readily see that under Scaling regime 1,
the backward dynamics of the gradient become linear. When $\beta>0$, which is
the case observed in practice, the dependence on the activation function
disappears in the large depth limit, exactly as for the forward dynamics.

###  5.3 Backward equation for the Jacobian under Scaling regime 2

Recall the set-up of Theorem 3. Let
$(\Omega,\mathcal{F},\mathbb{F},\mathbb{P})$ be a probability space with a
$\mathbb{P}$-complete filtration $\mathbb{F}=(\mathcal{F}_{t})_{t\geq 0}$. Let
$(B^{A}_{t})_{t\geq 0}$, resp. $(B^{b}_{t})_{t\geq 0}$, be $d\times
d$-dimensional, resp. $d$-dimensional, independent $\mathbb{F}$-Brownian
motions. Recall that for Scaling regime 2,

$A_{k}^{(L)}=\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{k/L}L^{-1}+W_{(k+1)/L}^{A}-W_{k/L}^{A}\qquad
b^{(L)}_{k}=\overline{b}_{k/L}L^{-1}+W^{b}_{(k+1)/L}-W^{b}_{k/L},$ (5.7)

where $(W^{A}_{t})_{t\in[0,1]}$ and $(W^{b}_{t})_{t\in[0,1]}$ are Itô
processes [28] adapted to $\mathbb{F}$ and can be written in the form:

$\displaystyle\left(\mathrm{d}W_{t}^{A}\right)_{ij}$
$\displaystyle=\sum_{k,l=1}^{d}\left(q_{t}^{A}\right)_{ijkl}\left(\mathrm{d}B^{A}_{t}\right)_{kl}\quad\mbox{for
}\,i,j=1,\ldots,d,$ (5.8) $\displaystyle\mathrm{d}W_{t}^{b}$
$\displaystyle=q^{b}_{t}\mathrm{d}B^{b}_{t},$

with $W_{0}^{A}=0$, $W_{0}^{b}=0$, $q_{t}^{A}\in\mathbb{R}^{d,\otimes 4}$ and
$q_{t}^{b}\in\mathbb{R}^{d\times d}$ for $t\in[0,1]$. We use the notation in
(4.7) and (4.8) for the quadratic variation of $W^{A}$ and $W^{b}$.

Define

$\nu(t,h)\coloneqq\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}\mathbbm{1}_{\beta=1}+\frac{1}{2}\sigma^{\prime\prime}(0)\nabla_{h}Q(t,h).$
(5.9)

We will use the following assumption for the results in this section:

######  Assumption 4.

$\sup_{L}\mathbb{E}\left[\sup_{0\leq t\leq 1}\left\lVert\mkern
1.0mu\overline{\mkern-1.5muG\mkern 0.0mu}\mkern
0.0mu_{t}^{(L)}\right\rVert^{4}\right]<\infty,\qquad\mathbb{E}\left[\exp\left(8\int_{0}^{1}\left|\mathrm{tr}\left(\nu(s,H_{s})\right)\right|\mathrm{d}s\right)\right]<\infty.$

The boundedness of the fourth moment of the Jacobians $g_{k}^{(L)}$ in $L$ is
similar to Assumption (3) and is standard in the convergence of approximation
schemes for SDE. The second part of Assumption 4 is a technical condition: we
need the fourth moment of the $\nabla_{x}H_{t}^{x}$ to be bounded. Theorem 5
proves that the process $t\mapsto\nabla_{x}H_{t}^{x}$ satisfy a linear SDE
with drift $\nu(t,H_{t})$ linear in $H_{t}$, so we need finiteness of the
$L^{8}$ norm of the exponential of the drift, see Lemma 3 for more details. In
practice, $g_{k}^{(L)}$ and $h_{k}^{(L),\,x}$ stay bounded during training, so
Assumption (4) is satisfied.

######  Theorem 5 (Backpropagation dynamics under Scaling regime 2).

Let Assumptions 1, 2, 3, and 4 hold and let $\alpha=0$ and $\beta=1$. Let
$(H_{t})_{t\in\left[0,1\right]}$ be a solution to the SDE (4.19) and
$(J_{t})_{t\in\left[0,1\right]}\subset\mathbb{R}^{d\times d}$ be the unique
solution to the linear matrix-valued SDE

$\mathrm{d}J_{t}=\big{(}\nu(t,H_{t})\mathrm{d}t+\mathrm{d}W^{A}_{t}\big{)}J_{t},\quad
J_{0}=I_{d},$ (5.10)

where $\nu$ is defined in (5.9). Then, $\mathbb{P}-$a.s., $J_{t}$ is
invertible for all $t\in\left[0,1\right]$ and

$\mkern 1.0mu\overline{\mkern-1.5muG\mkern 0.0mu}\mkern
0.0mu^{(L)}=\sum_{k=0}^{L-1}g_{k}^{(L)}\mathbbm{1}_{\left[t_{k},t_{k+1}\right)}\mathop{\longrightarrow}^{L\to\infty}G_{t}\coloneqq
J_{1}J_{t}^{-1}$ (5.11)

uniformly in $L^{1}(\mathbb{P})$ in the sense of Def. 1.

The steps to prove Theorem 5 are similar to those of Theorem 3 but the details
are technically more involved. Indeed, terms that depend on $g_{k}^{(L),\,x}$
are not a priori adapted to the filtration generated by the Ito processes
$W^{A}$ and $W^{b}$. To overcome this challenge, we denote

$\displaystyle J_{k}^{(L),\,x}:=\nabla_{x}h_{k}^{(L),\,x},$ (5.12)

and we can rewrite $g^{(L),\,x}_{0}=g^{(L),\,x}_{k}J_{k}^{(L),\,x}$. This
leads to a new perspective to understand $g^{(L),\,x}_{k}$ through two
components $J_{k}^{(L),\,x}$ and $g^{(L),\,x}_{0}$. The first term
$J_{k}^{(L),\,x}$ is adapted to the filtration generated by the Ito processes,
and $g^{(L),\,x}_{0}$ is the Jacobian of the output with respect to the input,
and does not depend on the layer. The complete proof is provided in Section
5.4.2.

####  5.3.1 Connection with Neural SDE

In a recent work, [21] show that, when the hidden state $H$ satisfies a
continuous-time ’neural SDE’ dynamics, the Jacobian of the output with respect
to the hidden states satisfies a backward SDE:

$\mathrm{d}\widehat{G}_{t}=\widehat{G}_{t}\left(-\nu(t,\widehat{H}_{t})\mathrm{d}t-\mathrm{d}\widehat{W}^{A}_{t}\right),\quad\widehat{G}_{1}=I_{d},$
(5.13)

where $\widehat{W}^{A}$ is the time-reversed Brownian motion defined by
$\widehat{W}^{A}_{t}\coloneqq W^{A}_{t}-W^{A}_{1}$, and $\widehat{H}_{t}$ is
the solution of the backward flow of diffeomorphisms generated by the forward
SDE (4.19).

It is clear that the limit $G_{t}$ in (5.11) differs from the adjoint process
(5.13). Our limit $G_{t}=J_{1}J_{t}^{-1}$ does not satisfy any forward nor
backward SDE, as its solution is a function of $H_{1}$ which depends on
weights across all layers i.e. the entire path of $W^{A}$. Indeed, Theorem 3.1
in [32] states that $t\mapsto J_{t}^{-1}$ solves the following SDE.

$\mathrm{d}(J_{t}^{-1})=J_{t}^{-1}\left(-\nu(t,H_{t})\mathrm{d}t-\mathrm{d}W_{t}^{A}+\mathrm{d}\left[W^{A}\right]_{t}\right),\quad
J_{0}^{-1}=I_{d}.$

Therefore, one can write

$\displaystyle G_{t}=J_{1}J_{t}^{-1}$
$\displaystyle=J_{1}\left(J_{1}^{-1}+\int_{t}^{1}J_{s}^{-1}\left(-\nu(s,H_{s})\mathrm{d}s-\mathrm{d}W_{s}^{A}+\mathrm{d}\left[W^{A}\right]_{s}\right)\right)$
$\displaystyle=I_{d}+\int_{t}^{1}G_{s}\left(-\nu(s,H_{s})\mathrm{d}s-\mathrm{d}W_{s}^{A}+\mathrm{d}\left[W^{A}\right]_{s}\right).$
(5.14)

One can readily see that $G_{t}$ depends on $H_{1}$ for all $t\in[0,1]$. Note
that the quadratic variation drift correction term stems from using Ito
integrals instead of Stratonovitch integrals.
In contrast to (5.13), (5.14) is the exact large-depth limit of gradients
computed by backpropagation in finite depth residual networks, as stated in
Theorem 5.

###  5.4 Proofs

####  5.4.1 Proof of Theorem 4

The ideas of the proof follow closely those of Theorem 1, and we will provide
here the main arguments to the Neural ODE case. The other case is very
similar.
Denote $t_{k}=k/L$, $k=0,1,\ldots,L$ as the uniform partition of the interval
$[0,1]$. For $t\in(t_{k},t_{k+1}]$, define

$\widetilde{G}^{(L)}_{t}\coloneqq
g_{k+1}^{(L)}\left(I_{d}+(t_{k+1}-t)\,\mathrm{diag}\left(\sigma^{\prime}_{d}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k}}H_{t_{k}}+\overline{b}_{t_{k}}\right)\right)\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{t_{k}}\right),$

where $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu$ and
$\overline{b}$ are specified in Theorem 1. Hence, we can directly deduce that

$\displaystyle\sup_{t\in\left[0,1\right]}\left\lVert\widetilde{G}^{(L)}_{t}-\mkern
1.0mu\overline{\mkern-1.5muG\mkern 0.0mu}\mkern 0.0mu^{(L)}_{t}\right\rVert$
$\displaystyle\leq L^{-1}\sup_{0\leq
k<L}\sup_{t\in\left(t_{k},t_{k+1}\right]}\left\lVert
g_{k+1}^{(L)}\mathrm{diag}\left(\sigma^{\prime}_{d}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k}}H_{t_{k}}+\overline{b}_{t_{k}}\right)\right)\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{t_{k}}\right\rVert$
$\displaystyle\leq L^{-1}\sup_{0\leq k<L}\left\lVert
g_{k+1}^{(L)}\right\rVert\,\,\sup_{t\in\left[0,1\right]}\left\lVert\mathrm{diag}\left(\sigma^{\prime}_{d}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t}H_{t}+\overline{b}_{t}\right)\right)\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{t}\right\rVert$

By continuity of $\mkern 1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu$, $\overline{b}$ and $H$, the first supremum is finite and by a similar
argument as in the proof of Theorem 1, the second supremum is also finite.
Thus, there exists a constant $G_{\infty}>0$ such that
$\sup_{t\in\left[0,1\right]}\left\lVert\widetilde{G}^{(L)}_{t}-\mkern
1.0mu\overline{\mkern-1.5muG\mkern 0.0mu}\mkern
0.0mu^{(L)}_{t}\right\rVert\leq G_{\infty}L^{-1}$. Now, we also have, for
$t\in(t_{k},t_{k+1}]$,

$\displaystyle\widetilde{G}^{(L)}_{t}-G_{t}$
$\displaystyle=\widetilde{G}^{(L)}_{t_{k+1}}-G_{t_{k+1}}+(t_{k+1}-t)\,g_{k+1}^{(L)}\,\mathrm{diag}\left(\sigma^{\prime}_{d}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{t_{k}}H_{t_{k}}+\overline{b}_{t_{k}}\right)\right)\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{t_{k}}$
$\displaystyle-\int_{t}^{t_{k+1}}G_{s}\mathrm{diag}\left(\sigma^{\prime}_{d}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{s}H_{s}+\overline{b}_{s}\right)\right)\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{s}\mathrm{d}s.$

Hence, for $e^{(L)}_{k}\coloneqq\sup_{t_{k}<t\leq
t_{k+1}}\left\lVert\widetilde{G}^{(L)}_{t}-G_{t}\right\rVert$, we can estimate

$\displaystyle e^{(L)}_{k}$ $\displaystyle\leq
e^{(L)}_{k+1}+\int_{t}^{t_{k+1}}\left\lVert
G_{s}J_{s}-g_{k+1}^{(L)}J_{t_{k}}\right\rVert\mathrm{d}s.$ $\displaystyle\leq
e^{(L)}_{k+1}+\int_{t}^{t_{k+1}}\left(\left\lVert
G_{s}-g_{k+1}^{(L)}\right\rVert\left\lVert J_{s}\right\rVert+\left\lVert
g_{k+1}^{(L)}\right\rVert\left\lVert
J_{s}-J_{t_{k}}\right\rVert\right)\mathrm{d}s.$ $\displaystyle\leq
e^{(L)}_{k+1}+J_{\infty}L^{-1}\left(e_{k}^{(L)}+G_{\infty}L^{-1}\right)+\int_{t}^{t_{k+1}}\left\lVert
g_{k+1}^{(L)}\right\rVert\left\lVert J_{s}-J_{t_{k}}\right\rVert\mathrm{d}s,$

where $J_{s}\coloneqq\mathrm{diag}\left(\sigma^{\prime}_{d}\left(\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu_{s}H_{s}+\overline{b}_{s}\right)\right)\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern 0.0mu_{s}$ and
$J_{\infty}\coloneqq\sup_{s\in\left[0,1\right]}\left\lVert
J_{s}\right\rVert<\infty$. Now, recall that $\left\lVert
g_{k+1}^{(L)}\right\rVert$ is uniformly bounded in $k,L$, and we have $\mkern
1.0mu\overline{\mkern-1.5muA\mkern 0.0mu}\mkern
0.0mu,\overline{b}\in\mathcal{H}^{1}$ and $H\in\mathcal{C}^{1}$, so there
exists a constant $J^{\prime}_{\infty}<\infty$ such that
$\|g_{k+1}^{(L)}\|\int_{t}^{t_{k+1}}\left\lVert
J_{s}-J_{t_{k}}\right\rVert\mathrm{d}s<J^{\prime}_{\infty}L^{-2}$. Thus,

$\left(1-J_{\infty}L^{-1}\right)e^{(L)}_{k}\leq
e^{(L)}_{k+1}+\left(J_{\infty}G_{\infty}+J^{\prime}_{\infty}\right)L^{-2}.$

By Gronwall’s lemma and the fact that $e_{L}^{(L)}=\mathcal{O}(L^{-1})$, we
deduce that $\max_{k}e_{k}^{(L)}=\mathcal{O}(L^{-1})$ and conclude

$\lim_{L\to\infty}\sup_{t\in\left[0,1\right]}\left\lVert\widetilde{G}^{(L)}_{t}-G_{t}\right\rVert\leq\lim_{L\to\infty}\left(\sup_{t\in\left[0,1\right]}\left\lVert\widetilde{G}^{(L)}_{t}-\mkern
1.0mu\overline{\mkern-1.5muG\mkern 0.0mu}\mkern
0.0mu^{(L)}_{t}\right\rVert+\max_{k}e_{k}^{(L)}\right)=0.$

####  5.4.2 Proof of Theorem 5

The ideas of the proof follow closely those of Theorem 3, and we will provide
here the main arguments for the case $\alpha=0$ and $\beta=1$. Other cases
follow similarly. For the ease of notation exposition, we consider $U=0$ and
we use $C$ to denote some generic constant (independent from $L$ and other
parameters, such as $\varepsilon$, $\delta$, and $R$, to be defined later)
that may vary from step to step.

The proof consists of 11 steps that can be summarized as follows. Step 1
decomposes the discrete gradient $g_{k}^{(L)}$ into two terms: the Jacobian of
the output with respect to the input, and the Jacobian of the hidden state
$h_{k}^{(L)}$ with respect to the input, which we denote by $J_{k}^{(L)}$. We
then write a forward equation for $J_{k}^{(L)}$. Step 2 defines a continuous-
time approximation $\widetilde{J}_{k}^{(L)}$ and a continuous-time
interpolation $\mkern 1.0mu\overline{\mkern-1.5muJ\mkern 0.0mu}\mkern
0.0mu_{k}^{(L)}$. Step 3 establishes a uniform bound $\mathcal{O}(L^{-1})$
between $\widetilde{J}_{k}^{(L)}$ and $\mkern
1.0mu\overline{\mkern-1.5muJ\mkern 0.0mu}\mkern 0.0mu_{k}^{(L)}$. Step 4
defines high-probability events under which the hidden states $h^{(L)}_{k}$
and the continuous-time limit $J_{t}$ are uniformly bounded. Step 5 decomposes
the difference between $\widetilde{J}^{(L)}$ and $J$ with a drift term and an
error term $D^{(L)}$, which can be further decomposed into a variance term
$N^{(L)}$ and a Taylor remainder term $R^{(L)}$. Step 6 proves that $R^{(L)}$
uniformly vanishes as $\mathcal{O}(L^{-1})$. Step 7 decomposes $N^{(L)}$ into
three terms. Step 8 and 9 prove that these terms uniformly vanishes as
$\mathcal{O}(L^{-\min(1,\kappa)})$. Step 10 wraps everything together to show
a uniform $L^{2}$ bound between $\widetilde{J}^{(L)}$ and $J^{(L)}$. Step 11
uses it to prove a uniform $L^{1}$ bound between the discrete gradients
$g^{(L)}_{k}$ and their limit $G_{t}=J_{1}J_{t}^{-1}$.

Step 0: Well-posedness of the statement. The matrix-valued linear stochastic
differential equation (5.10) has a continuous and adapted solution, and this
solution is unique in the sense that almost all sample processes of any two
solutions coincide, see for example [10]. Furthermore, $\mathbb{P}-$a.s.,
$J_{t}$ is invertible for all $t\in\left[0,1\right]$, see Corollary 2.1 in
[6]. Also, Theorem 3.1 in [32] states that $K_{t}\coloneqq J_{t}^{-1}$ solves
the following SDE.

$\mathrm{d}K_{t}=K_{t}\left(-\nu(t,H_{t})\mathrm{d}t-\mathrm{d}W_{t}^{A}+\mathrm{d}\left[W^{A}\right]_{t}\right),\quad
K_{0}=I_{d}.$

Recall from Assumption 2 that the quadratic variation of $W^{A}$ is uniformly
continuous with resepect to the Lesbegue measure. Therefore, by Lemma 3 and
Assumption 4, we conclude that the fourth moments of the supremum of $J$ and