# On the infinite-depth limit of finite-width neural networks

ufiane Hayou<EMAIL_ADDRESS>
Department of Mathematics
National University of Singapore

###### Abstract

In this paper, we study the infinite-depth limit of finite-width residual
neural networks with random Gaussian weights. With proper scaling, we show
that by fixing the width and taking the depth to infinity, the pre-activations
converge in distribution to a zero-drift diffusion process. Unlike the
infinite-width limit where the pre-activation converge weakly to a Gaussian
random variable, we show that the infinite-depth limit yields different
distributions depending on the choice of the activation function. We document
two cases where these distributions have closed-form (different) expressions.
We further show an intriguing change of regime phenomenon of the post-
activation norms when the width increases from $3$ to $4$. Lastly, we study
the sequential limit infinite-depth-then-infinite-width and compare it with
the more commonly studied infinite-width-then-infinite-depth limit.

##

###  1 Introduction

The empirical success of over-parameterized neural networks has sparked a
growing interest in the theoretical understanding of these models. The large
number of parameters – millions if not billions – and the complex (non-linear)
nature of the neural computations (presence of non-linearities) make this
hypothesis space highly non-trivial. However, in certain situations,
increasing the number of parameters has the effect of ‘placing’ the network in
some ‘average’ regime that simplifies the theoretical analysis. This is the
case with the infinite-width asymptotics of random neural networks. The
infinite-width limit of neural network architectures has been extensively
studied in the literature, and has led to many interesting theoretical and
algorithmic innovations. We summarize these results below.

  * •

_Initialization schemes_ : the infinite-width limit of different neural
architectures has been extensively studied in the literature. In particular,
for multi-layer perceptrons (MLP), a new initialization scheme that stabilizes
forward and backward propagation (in the infinite-width limit) was derived in
[1, 2]. This initialization scheme is known as the Edge of Chaos, and
empirical results show that it significantly improves performance. In [3, 4],
the authors derived similar results for the ResNet architecture, and showed
that this architecture is _placed_ by-default on the Edge of Chaos for any
choice of the variances of the initialization weights (Gaussian weights). In
[5], the authors showed that an MLP that is initialized on the Edge of Chaos
exhibits similar properties to ResNets, which might partially explain the
benefits of the Edge of Chaos initialization.

  * •

_Gaussian process behaviour_ : Multiple papers (e.g. [6, 7, 8, 9, 10]) studied
the weak limit of neural networks when the width goes to infinity. The results
show that a randomly initialized neural network (with Gaussian weights) has a
similar behaviour to that of a Gaussian process, for a wide range of neural
architectures, and under mild conditions on the activation function. In [7],
the authors leveraged this result and introduced the neural network Gaussian
process (NNGP), which is a Gaussian process model with a neural kernel that
depends on the architecture and the activation function. Bayesian regression
with the NNGP showed that NNGP surprisingly achieves performance close to the
one achieved by an SGD-trained finite-width neural network.

The large depth limit of this Gaussian process was studied in [4], where the
authors showed that with proper scaling, the infinite-depth (weak) limit is a
Gaussian process with a universal kernel111A kernel is called universal when
any continuous function on some compact set can be approximated arbitrarily
well with kernel features..

  * •

_Neural Tangent Kernel (NTK)_ : the infinite-width limit of the NTK is the so-
called NTK regime or Lazy-training regime. This topic has been extensively
studied in the literature. The optimization and generalization properties (and
some other aspects) of the NTK have been studied in [11, 12, 13, 14]. The
large depth asymptotics of the NTK have been studied in [15, 16, 17, 18]. We
refer the reader to [19] for a comprehensive discussion on the NTK.

  * •

_Others_ : the theory of infinite-width neural networks has also been utilized
for network pruning [20, 21], regularization [22], feature learning [23], and
ensembling methods [24] (this is by no means an exhaustive list).

The theoretical analysis of infinite-width neural networks has certainly led
to many interesting (theoretical and practical) discoveries. However, most
works on this limit consider a fixed depth network. _What about infinite-
depth_? Existing works on the infinite-depth limit can generally be divided
into three categories:

  * •

_Infinite-width-then-infinite-depth limit_ : in this case, the width is taken
to infinity first, then the depth is take to infinity. This is the infinite-
depth limit of infinite-width neural networks. This limit was particularly
used to derive the Edge of Chaos initialization scheme [2, 1], study the
impact of the activation function [5], the behaviour of the NTK [15, 18],
kernel shaping [25, 26] etc.

  * •

_The joint infinite-width-and-depth limit_ : in this case, the depth-to-width
ratio is fixed, and therefore, the width and depth are jointly taken to
infinity at the same time. There are few works that study the joint width-
depth limit. For instance, in [27], the authors showed that for a special form
of residual neural networks (ResNet), the network output exhibits a (scaled)
log-normal behaviour in this joint limit. This is different from the
sequential limit where width is taken to infinity first, followed by the
depth, in which case the distribution of the network output is asymptotically
normal ([2, 5]). In [28], the authors studied the covariance kernel of an MLP
in the joint limit, and showed that it converges weakly to the solution of
Stochastic Differential Equation (SDE). In [29], the authors showed that in
the joint limit case, the NTK of an MLP remains random when the width and
depth jointly go to infinity. This is different from the deterministic limit
of the NTK where the width is taken to infinity before depth [15]. More
recently, in [30], the author explored the impact of the depth-to-width ratio
on the correlation kernel and the gradient norms in the case of an MLP
architecture, and showed that this ratio can be interpreted as an effective
network depth.

  * •

_Infinite-depth limit of finite-width neural networks_ : in both previous
limits (infinite-width-then-infinite-depth limit, and the joint infinite-
width-depth limit), the width goes to infinity. Naturally, one might ask what
happens if width is fixed and depth goes to infinity? What is the limiting
distribution of the network output at initialization? In [31], the author
showed that neural networks with bounded width are still universal
approximators, which motivates the study of finite-width large depth neural
networks. In [32], the authors showed that the pre-activations of a particular
ResNet architecture converge weakly to a diffusion process in the infinite-
depth limit. This is the result of the fact that ResNet can be seen as
discretizations of SDEs (see Section 2).

In the present paper, we study the infinite-depth limit of finite-width ResNet
with random Gaussian weights. We are particularly interested in the
_asymptotic behaviour of the pre/post-activation values_. Our contributions
are four-fold:

  1. 1.

Unlike the infinite-width limit, we show that the resulting distribution of
the pre-activations in the infinite-depth limit is not necessarily Gaussian.
In the simple case of networks of width $1$, we study two cases where we
obtain known but completely different distributions by carefully choosing the
activation function.

  2. 2.

For ReLU activation function, we introduce and discuss the phenomenon of
_network collapse_. This phenomenon occurs when the pre-activations in some
hidden layer have all non-positive values which results in zero post-
activations. This leads to a stagnant network where increasing the depth
beyond a certain level has no effect on the network output. For any fixed
width, we show that in the infinite-depth limit, network collapse is a zero-
probability event, meaning that almost surely, all post-activations in the
network are non-zero.

  3. 3.

For networks with general width, where the distribution of the pre-activations
is generally intractable, we focus on the norm of the post-activations with
ReLU activation function, and show that this norm has approximately a
Geometric Bronwian Motion (GBM) dynamics. We call this Quasi-GBM. We also shed
light on a regime change phenomenon that occurs when the width $n$ increases
from $3$ to $4$. For width $n\leq 3$, resp. $n\geq 4$, the logarithmic growth
factor of the post-activations is , resp. positive.

  4. 4.

We study the sequential limit infinite-depth-then-infinite-width, which is the
converse of the more commonly studied infinite-width-then-infinite-depth
limit, and show some key differences between these limits. We particularly
show that the pre-activations converge to the solution of a Mckean-Vlasov
process, which has marginal Gaussian distributions, and thus we recover the
Gaussian behaviour in this limit. We compare the two sequential limits and
discuss some differences.

The proofs of the theoretical results are provided in the appendix and
referenced after each result. Empirical evaluations of these theoretical
findings are also provided.

###  2 The infinite-depth limit

Hereafter, we denote the width, resp. depth, of the network by $n$, resp. $L$.
We also denote the input dimension by $d$. Let $d,n,L\geq 1$, and consider the
following ResNet architecture of width $n$ and depth $L$

$\displaystyle Y_{0}$ $\displaystyle=W_{in}x,\quad x\in\mathbb{R}^{d}$ (1)
$\displaystyle Y_{l}$
$\displaystyle=Y_{l-1}+\frac{1}{\sqrt{L}}W_{l}\phi(Y_{l-1}),\quad
l=1,\dots,L,$

where $\phi:\mathbb{R}\to\mathbb{R}$ is the activation function, $L\geq 1$ is
the network depth, $W_{in}\in\mathbb{R}^{n\times d}$, and
$W_{l}\in\mathbb{R}^{n\times n}$ is the weight matrix in the $l^{th}$ layer.
We assume that the weights are randomly initialized with iid Gaussian
variables $W_{l}^{ij}\sim\mathcal{N}(0,\frac{1}{n})$,
$W_{in}^{ij}\sim\mathcal{N}(0,\frac{1}{d})$. For the sake of simplification,
we only consider networks with no bias, and we omit the dependence of $Y_{l}$
on $n$ in the notation. While the activation function is only defined for real
numbers, we will abuse the notation and write
$\phi(z)=(\phi(z^{1}),\dots,\phi(z^{k}))$ for any $k$-dimensional vector
$z=(z^{1},\dots,z^{k})\in\mathbb{R}^{k}$ for any $k\geq 1$. We refer to the
vectors $\\{Y_{l},l=0,\dots,L\\}$ by the _pre-activations_ and the vectors
$\\{\phi(Y_{l}),l=0,\dots,L\\}$ by the _post-activations_. Hereafter,
$x\in\mathbb{R}^{d}$ is fixed, and we assume that $x\neq 0$.

The $1/\sqrt{L}$ scaling in Eq. 1 is not arbitrary. This specific scaling was
shown to stabilize the norm of $Y_{l}$ as well as gradient norms in the large
depth limit (e.g. [4, 20, 33]). In the next result, we show that the infinite
depth limit of Eq. 1 (in the sens of the distribution) exists and has the same
distribution of the solution of a stochastic differential equation. In the
case of a single input, this has been already shown in [32]. The details are
provided in Section A. We also generalize this result in the case of multiple
inputs and obtain similar SDE dynamics (see 5 in the Appendix).

###### Proposition 1

Assume that the activation function $\phi$ is Lipschitz on $\mathbb{R}^{n}$.
Then, in the limit $L\to\infty$, the process $X^{L}_{t}=Y_{\lfloor
tL\rfloor}$, $t\in[0,1]$, converges in distribution to the solution of the
following SDE

$dX_{t}=\frac{1}{\sqrt{n}}\|\phi(X_{t})\|dB_{t},\quad X_{0}=W_{in}x,$ (2)

where $(B_{t})_{t\geq 0}$ is a Brownian motion (Wiener process), independent
from $W_{in}$. Moreover, we have that for any $t\in[0,1]$ and any Lipschitz
function $\Psi:\mathbb{R}^{n}\to\mathbb{R}$,

$\mathbb{E}\Psi(Y_{\lfloor
tL\rfloor})=\mathbb{E}\Psi(X_{t})+\mathcal{O}(L^{-1/2}),$

where the constant in $\mathcal{O}$ does not depend on $t$.
Moreover, if the activation function $\phi$ is only locally Lipschitz, then
$X^{L}_{t}$ converges locally to $X_{t}$. More precisely, for any fixed $r>0$,
we consider the stopping times

$\tau^{L}=\inf\\{t\geq 0:\|X^{L}_{t}\|\geq r\\},\quad\tau=\inf\\{t\geq
0:\|X_{t}\|\geq r\\},$

then the stopped process $X^{L}_{t\land\tau^{L}}$ converges in distribution to
the stopped solution $X_{t\land\tau}$ of the above SDE.

The proof of 1 is provided in Section A.6. We use classical results on the
numerical approximations of SDEs. 1 shows that the infinite-depth limit of
finite-width ResNet (Eq. 1) has a similar behaviour to the solution of the SDE
given in Eq. 7. In this limit, $Y_{\lfloor tL\rfloor}$ converges in
distribution to $X_{t}$. Hence, properties of the solutions of Eq. 7 should
theoretically be ‘shared’ by the pre-activations $Y_{\lfloor tL\rfloor}$ when
the depth is large. For the rest of the paper, we study some properties of the
solutions of Eq. 7. This requires the definition of filtered probability
spaces which we omit here. All the technical details are provided in Section
A. We compare the theoretical findings with empirical results obtained by
simulating the pre/post-activations of the original network Eq. 1. We refer to
$X_{t}$, the solution of Eq. 7, by the _infinite-depth network_.

The distribution of $X_{1}$ (the last layer in the infinite-depth limit) is
generally intractable, unlike in the infinite-width-then-infinite-depth limit
(Gaussian, [4]) or joint infinite-depth-and-width limit (involves a log-normal
distribution in the case of an MLP architecture, [27]). Intuitively, one
should not expect a universal behaviour (e.g. the Gaussian behaviour in the
infinite-width case) of the solution of Eq. 7 as this latter is highly
sensitive to the choice of the activation function, and different activation
functions might yield completely different distributions of $X_{1}$. We
demonstrate this in the next section by showing that we can recover closed-
form distributions by carefully choosing the activation function. The main
ingredient is the use of It$\hat{o}$ ’s lemma. See Section A for more details.

###  3 Different behaviours depending on the activation function

In this section, we restrict our analysis to a width-$1$ ResNet with one-
dimensional inputs, where each layer consists of a single neuron, i.e.
$d=n=1$. In this case, the process $(X_{t})_{0\leq t\leq 1}$ is one-
dimensional and is solution of the following SDE

$dX_{t}=|\phi(X_{t})|dB_{t},\quad X_{0}=W_{in}x.$

We can get rid of the absolute value in the equation above since the process
$X_{t}$ has the same distribution as $\tilde{X}_{t}$, the solution of the SDE
$d\tilde{X}_{t}=\phi(\tilde{X}_{t})dB_{t}$. The intuition behind this is that
the infinitesimal random variable ‘$dB_{t}$’ is Gaussian distributed with zero
mean and variance $dt$. Hence, it is a symmetric random variable and can
absorb the sign of $\phi(X_{t})$. The rigorous justification of this fact is
provided in 7 in the Appendix. Hereafter in this section, we consider the
process $X$, solution of the SDE

$dX_{t}=\phi(X_{t})dB_{t},\quad X_{0}=W_{in}x.$

Given a function $g\in\mathcal{C}^{2}(\mathbb{R})$222Here
$\mathcal{C}^{2}(\mathbb{R})$ refers to the vector space of functions
$g:\mathbb{R}\to\mathbb{R}$ that are twice differentiable and their second
derivatives are continuous., we use It$\hat{o}$ ’s lemma (Lemma 4 in the
appendix) to derive the dynamics of the process $g(X_{t})$. We obtain,

$\displaystyle
dg(X_{t})=\underbrace{\phi(X_{t})g^{\prime}(X_{t})}_{{\color[rgb]{0.10,0.14,0.76}\definecolor[named]{pgfstrokecolor}{rgb}{0.10,0.14,0.76}\sigma(X_{t})}}dB_{t}+\underbrace{\frac{1}{2}\phi(X_{t})^{2}g^{\prime\prime}(X_{t})}_{\,{\color[rgb]{0.70,0.0,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{0.70,0.0,0.0}\mu(X_{t})}}dt.$
(3)

In financial mathematics nomenclature, the function $\mu$ is called the
_drift_ and $\sigma$ is called the _volatility_ of the diffusion process.
It$\hat{o}$ ’s lemma is a valuable tool in stochastic calculus and is often
used to transform and simplify SDEs to better understand their properties. It
can also be used to find candidate functions $g$ and activation functions
$\phi$ such that the SDE Eq. 3 admits solutions with known distributions,
which yields a closed-form distribution for $X_{t}$. We consecrate the rest of
this section to this purpose.

####  3.1 ReLU activation

ReLU is a piece-wise linear activation function. Let us first deal with the
simpler case of linear activation functions. In the next result, we show that
linear activation functions yield log-normal distributions. In this case, the
process $X_{t}$ follows the Geometric Brownian motion dynamics. Later in this
section, we show that this result can be adapted to the case of the ReLU
activation function given by $\phi(x)=\max(x,0)$.

###### Proposition 2

Let $x\in\mathbb{R}$ such that $x\neq 0$. Consider a linear activation
function $\phi(y)=\alpha y+\beta,$ where $\alpha>0,\beta\in\mathbb{R}$ are
constants. Let $\sigma>0$ and define the function $g$ by $g(y)=(\alpha
y+\beta)^{\gamma},$ where $\gamma=\sigma\alpha^{-1}$. Consider the stochastic
process $X_{t}$ defined by

$dX_{t}=|\phi(X_{t})|dB_{t},\quad X_{0}=W_{in}x.$

Then, the process $g(X_{t})$ is a solution of the SDE

$dg(X_{t})=ag(X_{t})dt+\sigma g(X_{t})dB_{t},$

where $a=\frac{1}{2}\sigma^{2}\gamma^{-1}(\gamma-1)$. As a result, we have
that for all $t\in[0,1]$,

$g(X_{t})\sim g(X_{0})\exp\left(\left(a-\frac{1}{2}\sigma^{2}\right)t+\sigma
B_{t}\right).$

The proof of 2 is provided in Section D, and consists of using It$\hat{o}$
lemma and solving a differential equation. When the activation function is
ReLU, we still obtain a log-normal distribution conditionally on the event
that the initial value $X_{0}$ is positive.

###### Proposition 3

Let $x\in\mathbb{R}$ such that $x\neq 0$, and let $\phi$ be the ReLU
activation function given by $\phi(z)=\max(z,0)$ for all $z\in\mathbb{R}$.
Consider the stochastic process $X_{t}$ defined by

$dX_{t}=\phi(X_{t})dB_{t},\quad X_{0}=W_{in}x.$

Then, the process $X$ is a mixture of a Geometric Brownian motion and a
constant process. More precisely, we have for all $t\in[0,1]$

$X_{t}\sim\mathbbm{1}_{\\{X_{0}>0\\}}\,X_{0}\exp\left(-\frac{1}{2}t+B_{t}\right)+\mathbbm{1}_{\\{X_{0}\leq
0\\}}X_{0}.$

Hence, given a fixed $X_{0}>0$, the process $X$ is a Geometric Brownian
motion.

(a) Distribution of $\log(Y_{L})$ with $L=5$

(b) Distribution of $\log(Y_{L})$ with $L=50$

(c) Neural path $(\log(Y_{l})_{1\leq l\leq L}$ with $L=100$

(d) Distribution of $\log(Y_{L})$ with $L=100$

(e) Neural path $(Y_{l})_{1\leq l\leq L}$ with $L=100$

(f) Distribution of $Y_{L}$ with $L=100$

Figure 1: Empirical verification of 2. (a), (b), (d) Histograms of
$\log(Y_{L})$ and based on $N=5000$ simulations for depths
$L\in\\{5,50,100\\}$ with $Y_{0}=1$. Estimated density (Gaussian kernel
estimate) and theoretical density (Gaussian) are illustrated on the same
graphs. (c), (e) 30 Simulations of the sequence $(\log(Y_{l}))_{l\leq L}$ (c)
and the sequence $(Y_{l})_{l\leq L}$ (e). We call such sequences Neural paths.
The results are reported for depth $L=100$, with $Y_{0}=1$, $\phi$ being the
ReLU activation. The theoretical mean of $\log(Y_{l})$ is given by
$m(l)=-\frac{l}{2L}$ and that of $Y_{l}$ is equal to $Y_{0}=1$. We also
illustrate the $99\%$ confidence intervals, based on the theoretical
prediction for $\log(Y_{l})$ (2), and the empirical Quantiles for $Y_{l}$.
(f) Histogram of $Y_{L}$ based on $N=5000$ simulations for depth $L=100$.

The proof of 3 is provided in Section E. We show that conditionally on
$X_{0}>0$, with probability $1$, the process $X_{t}$ is positive for all
$t\in[0,1]$333In Section E, we show that the stopping $\tau=\inf\\{t\geq
0:\textrm{ s.t. }X_{t}\leq 0\\}$ is infinite almost surely, which is stronger
that what we need. This is a classic result in stochastic calculus.. When
$X_{t}>0$, the ReLU activation is just the identity function, which justifies
the similarity between this result and the one obtained with linear
activations (2). Conversely, if $X_{0}<0$, the process is constant equal to
$X_{0}$ since the updates ‘$dX_{t}$’ are equal to zero in this case. A
rigorous justification of this is given for general width $n$ later in the
paper (Lemma 1). An empirical verification of 2 is provided in Fig. 1 where we
compare the theoretical results to simulations of the _neural paths_
$(Y_{l})_{1\leq l\leq L}$ and $(\log(Y_{l}))_{1\leq l\leq L}$ from the
original (finite-depth) ResNet given by Eq. 1. We observe an excellent match
with theoretical predictions for depths $L=50$ and $L=100$. In the case of a
small depth ($L=5$), the theoretical distribution does not fit well the
empirical one (obtained by simulations), which is expected since the dynamics
of $X$ describe (only) the infinite-depth limit of the ResNet. More figures
are provided in Section K.
Remark: notice that the log-normal behaviour is a result of the fact that we
only consider the case $n=1$ (width one). Indeed, the single neuron case
forces ReLU to act like a linear activation when $X_{0}>0$, and like a ‘zero’
activation when $X_{0}\leq 0$. For general width $n\geq 1$, such behaviour
does not hold in general, and usually some coordinates of $X_{t}$ will be
negative while others are non-negative, which implies that the volatility term
$\|\phi(X_{t})\|$ has non-trivial dependence on $X_{t}$. We discuss this in
more details in Section 4. In the next section, we illustrate a case of an
exotic (non-standard) activation function that yields a completely different
closed-form distribution of $X_{t}$.

####  3.2 Exotic activation

The next result shows that with a particular choice of the activation function
$\phi$ and mapping $g$, the stochastic process $g(X_{t})$ is the solution of
well-known type of SDEs known as the Ornstein-Uhlenbeck SDEs. In this case,
the activation function is non-standard and involves the inverse of the
imaginary error function, a variant of the error function.

Figure 2: Exotic Activation.

######  Proposition 4 (Ornstein-Uhlenbeck neural networks)

Let $x\in\mathbb{R}$ such that $x\neq 0$. Consider the following activation
function $\phi$

$\phi(y)=\exp(h^{-1}(\alpha y+\beta)^{2}),$

where $\alpha,\beta\in\mathbb{R}$ are constants and $h^{-1}$ is the inverse
function of the imaginary error function given by
$h(z)=\frac{2}{\sqrt{\pi}}\int_{0}^{z}e^{t^{2}}dt$444Although the name might
be misleading, the imaginary error function is real valued, and it has a an
inverse $h^{-1}$ that is continuous and increasing.. Let $g$ be the function
defined by

$g(y)=\alpha\sqrt{\pi}h^{-1}(\alpha y+\beta).$

Consider the stochastic process $X_{t}$ defined by555in Section C, we show
that the activation function $\phi$ is only locally Lipschitz. Hence, the
solution of this SDE exists only in the local sense and the convergence in
distribution of $Y_{\lfloor tL\rfloor}$ to $X_{t}$ is also in the local sense
(1). However, by continuity of the Brownian path, the stopping times
$\tau^{L}$ and $\tau$ diverge almost surely when $r$ goes to infinity.
Therefore, the conclusion of 4 remains true for all $t\in[0,1]$. Technical
details are provided in Section C.

$dX_{t}=|\phi(X_{t})|dB_{t},\quad X_{0}=W_{in}x.$

Then, the stochastic process $g(X_{t})$ follows the Ornstein-Uhlenbeck
dynamics on $(0,1]$ given by

$dg(X_{t})=ag(X_{t})dt+2adB_{t},\quad g(X_{0})=g(W_{in}x),$

where $a=\frac{\pi\alpha^{2}}{4}$. As a result, conditionally on $X_{0}$
(fixed $X_{0}$), we have that for all $t\in[0,1]$,

$g(X_{t})\sim\mathcal{N}\left(g(X_{0})e^{-at},\frac{\pi}{2}(1-e^{-2at})\right),$

and the process $X_{t}$ is distributed as
$X_{t}\sim\alpha^{-1}(h(\alpha^{-1}\pi^{-1/2}\mathcal{N}\left(g(X_{0})e^{-at},\frac{\pi}{2}(1-e^{-2at})\right))-\beta)$.

Fig. 2 shows the graph of the activation function
$\phi(y)=\exp(h^{-1}(y)^{2})$ mentioned in 4 with $\alpha=1$ and $\beta=0$.
With this choice of the activation function, the infinite-depth network output
$X_{1}$ has the distribution
$g^{-1}\left(\mathcal{N}\left(g(X_{0})e^{-at},2(1+e^{-2at})\right)\right)$
(conditionally on $X_{0}$), where $g$ is given in the statement of the
proposition. This distribution, although easy to simulate, is different from
both the Gaussian distribution that we obtain in the infinite-width limit and
the log-normal distribution associated with ReLU activation. This confirms
that not only do neural networks exhibit completely different behaviours when
the ratio depth-to-width is large, but in this case, that their behaviour is
very sensitive to the choice of the activation function.

(a) Distribution of $g(Y_{L})$ with $L=5$

(b) Distribution of $g(Y_{L})$ with $L=50$

(c) Neural path $(g(Y_{l})_{1\leq l\leq L}$ with $L=100$

(d) Distribution of $g(Y_{L})$ with $L=100$

(e) Neural path $(Y_{l})_{1\leq l\leq L}$ with $L=100$

(f) Distribution of $Y_{L}$ with $L=100$

Figure 3: Empirical verification of 4. (a), (b), (d) Histograms of $g(Y_{L})$
based on $N=5000$ simulations for depths $L\in\\{5,50,100\\}$ with $Y_{0}=1$.
Estimated density (Gaussian kernel estimate) and theoretical density
(Gaussian) are illustrated on the same graphs. (c), (e) 30 Simulations of the
neural paths $(g(Y_{l}))_{l\leq L}$ (c) and $(Y_{l})_{l\leq L}$ (e). The
results are reported for depth $L=100$, with $Y_{0}=1$, $\phi$ is given un 4.
The theoretical mean of $g(Y_{l})$ (conditionally on $Y_{0}$) is approximated
by $m(l)=g(Y_{0})e^{-\frac{\pi l}{3L}}$ and that of $Y_{l}$ is equal to
$Y_{0}=1$. We also illustrate the $99\%$ confidence intervals, based on the
theoretical prediction for $g(Y_{l})$ (2), and the empirical Quantiles for
$Y_{l}$.  (f) Histogram of $Y_{L}$ based on $N=5000$ simulations for depth
$L=100$.

The results of 4 are empirically confirmed in Fig. 3. The original ResNet
given by Eq. 7 with depth $L=100$ exhibit very similar behaviour to that of
the SDE.

###  4 General width $n\geq 1$

Let $n\geq 1$ and $x\in\mathbb{R}^{d}$ such that $x\neq 0$. Consider the
process $X$ given by the SDE

$dX_{t}=\frac{1}{\sqrt{n}}\|\phi(X_{t})\|dB_{t},\quad X_{0}=W_{in}x,$ (4)

where $\phi$ is the activation function, and $B$ is an $n$-dimensional
Brownian motion, independent from $W_{in}$. Intuitively, if for some $s$,
$\|\phi(X_{s})\|=0$, then for all $t\geq s$, $X_{t}=X_{s}$ since the
increments ’$dX_{t}$’ are all zero for $t\geq s$. This holds for any choice of
the activation function $\phi$, provided that the process $X$ exists, i.e. the
SDE has a unique solution. We summarize this in the next lemma.

######  Lemma 1 (Collapse)

Let $x\in\mathbb{R}^{d}$ such that $x\neq 0$, and
$\phi:\mathbb{R}\to\mathbb{R}$ be a Lipschitz function. Let $X$ be the
solution of the SDE given by Eq. 4. Assume that for some $s\geq 0$,
$\phi(X_{s})=0$. Then, for all $t\geq s$, $X_{t}=X_{s}$, almost surely.

Lemma 1 is a particular case of Lemma 7 in the Appendix. The proof consists of
using the uniqueness of the solution of Eq. 4 when the volatility term is
Lipschitz. This result is trivial in the finite depth case (Eq. 1). When there
exists $s$ such that $\phi(X_{s})=0$, the process $X$ becomes constant (equal
to $X_{s}$) for all $t\geq s$ (almost surely). We call this phenomenon
_process collapse_. In the case of finite-depth networks (Eq. 1), we call the
same phenomenon _network collapse_. Understanding when, and whether, such
event occurs is useful since it has significant implications on the the large
depth behaviour of neural networks. Indeed, if such event occurs, it would
mean that increasing depth has no effect on the network output after some time
$s$ (or approximately, after layer index $\lfloor sL\rfloor$). In the next
result, we show that under mild conditions on the activation function, process
collapse is a zero-probability event.

####  4.1 Network collapse

The next result gives (mild) sufficient conditions on the activation function
so that the process $X$ almost surely does not collapse. In the proof, we use
It$\hat{o}$ ’s lemma in the multi-dimensional case, which states that for any
function $g:\mathbb{R}^{n}\to\mathbb{R}$ that is
$\mathcal{C}^{2}(\mathbb{R}^{n})$, we have that

$\displaystyle dg(X_{t})=\nabla
g(X_{t})^{\top}dX_{t}+\frac{1}{2n}\|\phi(X_{t})\|^{2}\textup{Tr}\left[\nabla^{2}g(X_{t})\right].$

###### Lemma 2

Let $x\in\mathbb{R}^{d}$ such that $x\neq 0$, and consider the stochastic
process $X$ given by the following SDE

$dX_{t}=\frac{1}{\sqrt{n}}\|\phi(X_{t})\|dB_{t}\,,\quad t\in[0,\infty),\quad
X_{0}=W_{in}x,$

where $\phi(z):\mathbb{R}\to\mathbb{R}$ is Lipschitz, injective,
$\mathcal{C}^{2}(\mathbb{R})$ and satisfies $\phi(0)=0$, and $\phi^{\prime}$
and $\phi^{\prime\prime}\phi$ are bounded on $\mathbb{R}$, and $(B_{t})_{t\geq
0}$ is an $n$-dimensional Brownian motion independent from
$W_{in}\sim\mathcal{N}(0,d^{-1}I)$. Let $\tau$ be the stopping time given by

$\tau=\min\\{t\geq 0:\phi(X_{t})\ =0\\}.$

Then, we have that

$\mathbb{P}\left(\tau=\infty\right)=1.$

The proof of Lemma 2 is provided in Section F. Many standard activation
functions satisfy the conditions of Lemma 2. Examples include Hyperbolic
Tangent $\textrm{Tanh}(z)=\frac{e^{2z}-1}{e^{2z}+1}$, and smooth versions of
ReLU activation such as GeLU given by $\phi_{GeLU}(z)=z\Psi(z)$ where $\Psi$
is the cumulative distribution function of the standard Gaussian variable, and
Swish (or SiLU) given by $\phi_{Swish}(z)=zh(z)$ where $h(z)=(1+e^{-z})^{-1}$
is the Sigmoid function. The result of Lemma 2 can be extended to the case
when $\phi$ is the ReLU function with miner changes.

###### Lemma 3

Consider the stochastic process (7) given by the SDE

$dX_{t}=\frac{1}{\sqrt{n}}\|\phi(X_{t})\|dB_{t}\,,\quad t\in[0,\infty),\quad
X_{0}=W_{in}x,$

where $\phi$ is the ReLU activation function, and $(B_{t})_{t\geq 0}$ is an
$n$-dimensional Brownian motion independent from
$W_{in}\sim\mathcal{N}(0,d^{-1}I)$. Let $\tau$ be the stopping time given by

$\tau=\min\\{t\geq 0:\|\phi(X_{t})\|=0\\}=\min\\{t\geq 0:\,\forall
i\in[n],\,X^{i}_{t}\leq 0\\}.$

Then, we have that

$\mathbb{P}\left(\tau=\infty\,\huge|\,\|\phi(X_{0})\|>0\right)=1.$

As a result, we have that

$\mathbb{P}(\tau=\infty)=1-2^{-n}.$

The proof of Lemma 3 relies on a particular choice of a sequence of functions
$(\phi_{m})_{m\geq 1}$ that approximate the ReLU activation $\phi$. Details
are provided in Section F.

The result of Lemma 3 shows that for all $T>0$, with probability $1$, if there
exists $j\in[n]$ such that $X_{0}^{j}>0$, then for all $t\in[0,T]$, there
exists a coordinate $i$ such that $X^{i}_{t}>0$, which implies that the
volatility of the process $X$ given by $\frac{1}{\sqrt{n}}\|\phi(X_{t})\|$
does not vanish in finite time $t$. Notably, this implies that for any
$t\in[0,1]$, the norm of post-activations given by $\|\phi(X_{t})\|$ does not
vanish (with probability 1). This is important as it ensures that the vector
$\phi(X_{t})$, which represents the post-activations in the infinite-depth
network, does not vanish, and therefore the process $X_{t}$ does not get stuck
in

Figure 4: Probability of the event $\\{\exists l\in[L]\textrm{ such that
}\phi(Y_{l})=0\\}$ (collapse) for varying widths and depths. The probability
and the $95\%$ confidence intervals are estimated using $N=5000$ samples.

an absorbent point. The dependence between the coordinates of the process
$X_{t}$ is crucial in this result. In the opposite case where $X_{t}$ are
independent, the event $\\{\|\phi(X_{t})\|=0\\}$ has probability $2^{-n}$.
Notice also that this result holds only in the infinite-depth limit. With
finite-depth ResNet (Eq. 1) with ReLU activation, it is not hard to show that
the network collapse event $\\{\exists l\in[L],\textrm{ s.t.
}\|\phi(Y_{\lfloor tL\rfloor})\|=0\\}$ has non-zero probability. However, as
the depth increases, the probability of network collapse goes to zero. Fig. 4
shows the probability of network collapse for a finite-width and depth ResNet
(Eq. 1). As the depth $L$ increases, it becomes unlikely that the network
collapses. This is in agreement with our theoretical prediction that the
infinite-depth network represented by the process $X_{t}$ has zero-probability
collapse event, conditionally on the fact that $\|\phi(X_{0})\|>0$. The
probability of neural collapse also decreases with width, which is expected,
since it becomes less likely to have all pre-activations non-positive as the
width increases.

####  4.2 Post-activation norm

As a result of Lemma 3, conditionally on $\|\phi(X_{0})\|>0$, we can safely
consider manipulating functions that require positiveness such as the
logarithm of the norm of the post-activations. In the next result, we show
that the norm of the post-activations has a distribution that

(a) $n=2$

(b) $n=3$

(c) $n=4$

(d) $n=6$

(e) $n=20$

(f) $n=100$

Figure 5: Histogram of $\sqrt{n}\log(\|\phi(Y_{L})\|/\|\phi(Y_{0})\|)$ for
depth $L=100$ and different widths $n\in\\{2,3,4,6,20,100\\}$ based on
$N=5000$ simulations. Gaussian density estimate and (Gaussian) kernel density
estimates are shown. We observe a great match between the best Gaussian
estimate and the empirical distribution, which confirms the quasi-log-normal
theoretical predictions from 1.

resembles the log-normal distribution. We call this Quasi Geometric Brownian
Motion distribution (Quasi-GBM).

######  Theorem 1 (Quasi-GBM behaviour of the post-activations norm)

We have that for all $t\in[0,1]$,

$\|\phi(X_{t})\|=\|\phi(X_{0})\|\exp\left(\frac{1}{\sqrt{n}}\hat{B}_{t}+\frac{1}{n}\int_{0}^{t}\mu_{s}ds\right),\quad\textrm{almost
surely,}$

where $\mu_{s}=\frac{1}{2}\|\phi^{\prime}(X_{s})\|^{2}-1$, and
$(\hat{B})_{t\geq 0}$ is a one-dimensional Brownian motion. As a result, for
all $0\leq s\leq t\leq 1$

$\mathbb{E}\left[\log\left(\frac{\|\phi(X_{t})\|}{\|\phi(X_{s})\|}\right)\huge|\,\|\phi(X_{0})\|>0\right]=\left(\frac{(1-2^{-n})^{-1}}{4}-\frac{1}{n}\right)(t-s).$

Moreover, for $n\geq 2$, we have

$\textup{Var}\left[\log\left(\frac{\|\phi(X_{t})\|}{\|\phi(X_{s})\|}\right)\huge|\,\|\phi(X_{0})\|>0\right]\leq\left(n^{-1/2}+\Gamma_{s,t}^{1/2}\right)^{2}(t-s),$

where
$\Gamma_{s,t}=\frac{1}{4}\int_{s}^{t}\,\left(\left(\mathbb{E}\phi^{\prime}(X^{1}_{u})\phi^{\prime}(X^{2}_{u})-\frac{(1-2^{-n})^{2}}{4}\right)+n^{-1}\left(\frac{1-2^{-n}}{2}-\mathbb{E}\phi^{\prime}(X^{1}_{u})\phi^{\prime}(X^{2}_{u})\right)\right)du$.

(a) $n=2$

(b) $n=3$

(c) $n=4$

(d) $n=6$

(e) $n=20$

(f) $n=100$

Figure 6: 30 simulations of the sequence
$\left(\log(\|\phi(Y_{l})\|/\|\phi(Y_{0})\|)\right)_{1\leq l\leq L}$ for depth
$L=100$ and different widths $n\in\\{2,3,4,6,20,100\\}$. Theoretical means
from 1 are shown in red dashed lines and compared to their empirical
counterparts. We observe that when the ratio $L/n$ increases (especially for
$n=100$), the empirical mean also increases and becomes significantly
different from the theoretical prediction.

Different tools from stochastic calculus and probability theory are used in
the proof of 1. Technical details are provided in Section G. The first result
in the theorem suggests that the norm of the post-activations has a quasi-log-
normal distribution (conditionally on $X_{0}$). The first term in the
exponential is Gaussian ($n^{-1/2}\hat{B}_{t}$) and the second term depends on
$n^{-1}\mu_{s}$, which involves an average over
$(\phi^{\prime}(X_{s}^{i}))_{1\leq i\leq n}$. In the large width limit, this
average concentrates around its mean as we will see in 2. In Fig. 5, we show
the histogram of $\sqrt{n}\log(\|\phi(Y_{L})\|/\|\phi(X_{0})\|)$ for depth
$L=100$ and varying widths $n$. Surprisingly, the log-normal approximation
fits the empirical distribution very well even for small widths
$n\in\\{2,3,4,6\\}$ for which the term $n^{-1}\mu_{s}$ is not necessarily
close to its mean666We currently do not have a rigorous explanation for this
effect. A possible explanation for this empirical result is that the integral
over $\mu_{s}$ has some ‘averaging’ effect.. More interestingly, the result of
1 sheds light on an intriguing regime change that occurs between widths $n=3$
and $n=4$. Indeed, for $n\leq 3$, the logarithmic growth factor of the norm of
the post-activations $\|\phi(X_{t})\|$ tends to decrease with depth on
average, while it increases for $n\geq 4$. When $n=4$, the average growth is
positive although very small. This regime change phenomenon suggests that for
$n\leq 3$, the random variable $\|\phi(X_{t})\|/\|\phi(X_{s})\|$ has
significant probability mass in the region $(0,1)$. This probability mass
tends to $0$ as $n$ increases since $\|\phi(X_{t})\|/\|\phi(X_{s})\|$
converges to a deterministic constant (we will see this in the next theorem),
and the variance upperbound in 1 converges to $0$ when $n$ goes to infinity,
which can be explained by the fact that
$\mathbb{E}\phi^{\prime}(X^{1}_{u})\phi^{\prime}(X^{2}_{u})\overset{n\to\infty}{\longrightarrow}1/4$
(the coordinates become independent in the large width limit, see next
theorem). Experiments showing this concentration are provided in Section K.5.
In Fig. 6, we simulate 30 neural paths (i.e. $(Y_{l})_{1\leq l\leq L}$) for
depth $L=100$ and compute the logarithmic factor
$\log(\|\phi(Y_{l})\|/\|\phi(Y_{0})\|)$. An excellent match with the
theoretical results is observed for widths $n\in\\{2,3,4,6,20\\}$. A mismatch
between theory and empirical results appears when $n=50$, which is expected,
since the theoretical results of 1 yield good approximations only when $n\ll
L$.

Notice that the case of $n=1$ matches the result of 3. Indeed, the latter
implies that conditionally on $\phi(X_{0})>0$, we have
$\log(\phi(X_{t})/\phi(X_{0}))=\log(X_{t}/X_{0})\sim-t/2+B_{t}$ where $B$ is a
one-dimensional Brownian motion, and where we have used the fact that
$X_{t}>0$ for all $t$. This result can be readily obtained from 1 by setting
$n=1$.

An interesting question is that of the infinite-width limit of the process
$X_{t}$, which corresponds to the sequential limit infinite-depth-then-
infinite-width of the ResNet $Y_{\lfloor tL\rfloor}$ (Eq. 1). We discuss this
in the next section.

####  4.3 Infinite-width limit of infinite-depth networks

In the next result, we show that when the width goes to infinity, the ratio
$\|\phi(X_{t})\|/\|\phi(X_{0})\|$ concentrates around a layer dependent
($t$-dependent) constant. In this limit, the coordinates of $X_{t}$ converge
in $L_{2}$ to a Mckean-Vlasov process, which allows us to recover the Gaussian
behaviour of the pre-activations of the ResNet. We later compare this with the
converse sequential limit infinite-width-then-infinite-depth where the pre-
activations are also normally distributed, and show a key difference in the
variance of the Gaussian distribution.

######  Theorem 2 (Infinite-depth-then-infinite-width limit)

For $0\leq s\leq t\leq 1$, we have

$\log\left(\frac{\|\phi(X_{t})\|}{\|\phi(X_{s})\|}\right)\,\mathbbm{1}_{\\{\|\phi(X_{0})\|>0\\}}\underset{n\to\infty}{\longrightarrow}\frac{t-s}{4},\quad\textrm{and,
}\quad\frac{\|\phi(X_{t})\|}{\|\phi(X_{s})\|}\,\mathbbm{1}_{\\{\|\phi(X_{0})\|>0\\}}\underset{n\to\infty}{\longrightarrow}\exp\left(\frac{t-s}{4}\right).$

where the convergence holds in $L_{1}$.

Moreover, we have that

$\sup_{i\in[n]}\mathbb{E}\left(\sup_{t\in[0,1]}|X^{i}_{t}-\tilde{X}^{i}_{t}|^{2}\right)=\mathcal{O}(n^{-1}),$

where $X^{i}_{t}$ is the solution of the following (Mckean-Vlasov) SDE

$d\tilde{X}^{i}_{t}=\left(\mathbb{E}\phi(\tilde{X}^{i}_{t})^{2}\right)^{1/2}dB^{i}_{t},\quad\tilde{X}^{i}_{0}=X^{i}_{0}.$

As a result, the pre-activations $Y^{i}_{\lfloor tL\rfloor}$ (Eq. 1) converge
in distribution to a Gaussian distribution in the limit infinite-depth-then-
infinite-width

$\forall i\in[n],\,\,\,\,Y^{i}_{\lfloor
tL\rfloor}\xrightarrow{L\to\infty\textrm{ then
}n\to\infty}\mathcal{N}(0,d^{-1}\|x\|^{2}\exp(t/2)).$

The proof of 2 requires the use of a special variant of the Law of large
numbers for non iid random variables, and a convergence result of particle
systems from the theory of Mckean-Vlasov processes. Details are provided in
Section H. In neural network terms, 2 shows that the logarithmic growth factor
of the norm of the post-activations, given by $\log\left(\|\phi(Y_{\lfloor
tL\rfloor})\|/\|\phi(Y_{\lfloor sL\rfloor})\|\right)$, converges to $(t-s)/4$
in the sequential limit $L\to\infty$, then $n\to\infty$. More importantly, the
pre-activations $Y_{\lfloor tL\rfloor}^{i}$ converge in distribution to a
zero-mean Gaussian distribution in this limit, with a layer-dependent
variance. In the converse sequential limit, i.e. $n\to\infty$, then
$L\to\infty$, the limiting distribution of the pre-activations $Y_{\lfloor
tL\rfloor}^{i}$ is also Gaussian with the same variance. We show this in the
following result, which uses Lemma 5 in [4].

######  Theorem 3 (Infinite-width-then-infinite-depth limit)

Let $t\in[0,1]$. Then, in the limit $\lim_{L\to\infty}\lim_{n\to\infty}$
(infinite width, then infinite depth), we have that

$\frac{\|\phi(Y_{\lfloor
tL\rfloor})\|}{\|\phi(Y_{0})\|}\,\mathbbm{1}_{\\{\|\phi(Y_{0})\|>0\\}}\longrightarrow\exp\left(\frac{t}{2}\right),$

where the convergence holds in probability.

Moreover, the pre-activations $Y^{i}_{\lfloor tL\rfloor}$ (Eq. 1) converge in
distribution to a Gaussian distribution in the limit infinite-width-then-
infinite-depth

$\forall i\in[n],\,\,\,\,Y^{i}_{\lfloor
tL\rfloor}\xrightarrow{n\to\infty\textrm{ then
}L\to\infty}\mathcal{N}(0,d^{-1}\|x\|^{2}\exp(t)).$

The proof of 3 is provided in Section I. We use existing results from [4] on
the infinite-depth asymptotics of the neural network Gaussian process (NNGP).
It turns out that the order to the sequential limit (taking the width to
infinity first, then taking the depth to infinity, or the converse) does not
affect the limiting distribution, which is a Gaussian with variance
$\propto\exp(t/2)$). Intuitively, by taking the width to infinity first, we
make the coordinates independent from each other, and the processes
$(Y^{i}_{l})_{1\leq L}$ become iid Markov chains. Taking the infinite-depth
limit after the infinite-width limit consists of taking the infinite-depth
limit of one-dimensional Markov chains. On the other hand, when we take depth
to infinity first, the coordinates $(X^{i}_{t})_{1\leq i\leq n}$ remain
dependent (through the volatility term $n^{-1/2}\|\phi(X_{t})\|$), which
results in the Quasi-log-normal behaviour of the norm of the post-activations
(1). Taking the width to infinity then yields an asymptotic norm of the post-
activations equal to $\|\phi(X_{0})\|\exp(t/2)$ (2) which is the same norm in
the converse limit (3). It remains to take the width to infinity to decouple
the coordinates and obtain the Gaussian distribution (through the Mckean-
Vlasov dynamics). Knowing that the variance of the pre-activations is mainly
determined by the norm of the post-activations (Eq. 4), we can see why the
variance is similar in both sequential limits.

###  5 Discussion on the case of multiple inputs

The result of 1 can be easily generalized to the multiple input case, and the
resulting dynamics is still an SDE. The generalization to the multiple inputs
case is given by 5 in the Appendix.

An important question in the literature on infinite-width neural networks is
the behaviour of the correlation of the pre-activations (or the post-
activations) for different inputs $a$ and $b$, which is given by
$\frac{\langle Y_{\lfloor tL\rfloor}(a),Y_{\lfloor
tL\rfloor}(b)\rangle}{\|Y_{\lfloor tL\rfloor}(a)\|\|Y_{\lfloor
tL\rfloor}(b)\|}$. This correlation can be as a geometric measure of the
information as it propagates through the network. In the infinite-width-then-
depth limit, this correlation (generally) converges to a degenerate limit (a
constant value) which results in either a constant or a sharp landscape of the
network output and causes gradient exploding/vanishing issues [2, 3, 5].
Techniques such block scaling [4], or kernel shaping [26, 25] solve this
problem and ensure that the correlation is well-behaved in the large depth
limit.

Figure 7: 10 Simulations of the correlation path $\left(\frac{\langle
Y_{\lfloor tL\rfloor}(a),Y_{\lfloor tL\rfloor}(b)\rangle}{\|Y_{\lfloor
tL\rfloor}(a)\|\|Y_{\lfloor tL\rfloor}(b)\|}\right)_{1\leq l\leq L}$ for depth
$L=200$, width $n=20$, and different $(a,b)$ (different initial correlations
$c_{0}$). The color code depends only on the initial correlation value $c_{0}$
(red for the largest correlations values)

In our case, when the width $n$ is finite and the depth $L$ is taken to
infinity, we can define the correlation for two inputs $a\neq b$ and time
$t\in[0,1]$ by

$c_{t}(a,b)\overset{def}{=}\frac{\langle
X_{t}(a),X_{t}(b)\rangle}{\|X_{t}(a)\|\|X_{t}(b)\|}.$

Using It$\hat{o}$ ’s lemma, $c_{t}$ has dynamics of the form

$dc_{t}(a,b)=\Psi(X_{t}(a),X_{t}(b))dB_{t},$ (5)

for some non-trivial mapping $\Psi$. Unfortunately, this kind of dynamics
(which is not an SDE) is generally intractable, and we are currently
investigating these dynamics for future work. However, since we scale the
ResNet blocks with the factor $1/\sqrt{L}$ (Eq. 1), which is the same scaling
that solves the degeneracy issue in the infinite-width-then-depth limit [4],
it should be expected that the correlation kernel $c_{t}$ does not converge to
a degenerate limit. In Fig. 7, we simulate the correlation path in a ResNet of
depth $L=200$ and width $n=20$. The paths exhibits some level of stochasticity
but no degeneracy can be observed. Understanding the correlation dynamics (Eq.
5) in the infinite-depth limit of finite-width networks is an interesting open
question. The infinite-width limit777The infinite-width limit of infinite-
depth correlations of these dynamics is also an interesting open question. We
leave this for future work.

###  6 Practical implications

Our theoretical analysis has many interesting implications from a practical
standpoint. Here we summarize some key insights form our results.

Figure 8: 10 Simulations of the gradient norm for scaled ResNet (Eq. 1) and
non-scaled ResNet ($Y_{l}=Y_{l-1}+W_{l}\phi(Y_{l-1})$) for depth $L=100$ and
width $n=10$. We normalize the gradients norms by the gradient norm of the
last layer. The color code depends only on the ratio of the graident norm at
the first layer to that of the last layer (dark red for the largest values).
Without scaling, the gradient norm explodes (highly likely). The $1/\sqrt{L}$
stabilizes the gradients as they back-propagate through the network.

##### Initialization and stability in the large depth limit.

An important factor pertaining to the trainability of neural networks is the
behaviour of the neurons (pre/post-activations). Ensuring that the neurons are
well-behaved at initialization is crucial for training since the first step of
any gradient-based training algorithm depends on the values of the neurons at
initialization. This has led to interesting developments in initialization
schemes for MLPs such as the Edge of Chaos [1, 2] which ensures that the
variance of the pre-activation does not (exponentially) vanish or explode in
the large depth limit. In the case of ResNet, we know from the existing theory
on the infinite-width limit of neural networks that scaling the residual
blocks with $1/\sqrt{L}$ stabilizes the pre/post-activations in the large
depth limit [4]. Hence, we do not need a special initialization scheme with
this scaling. However, one could argue that this (approximately) ensures
stability _only_ when the width is much larger than the depth. What about the
other cases when $n\approx L$ or $n\ll L$? the last case can be studied by
fixing the width and taking the depth to infinity. In our paper, we not _only_
show that the neurons remain stable in fixed-width large-depth networks, but
we fully characterize their behaviour when the depth is infinite and show that
it follows an SDE in this limit. To summarize, we show that initializing
ResNet Eq. 1 with standard Gaussian random variables and scaling the blocks
with $1/\sqrt{L}$ ensures stability inside the network in large-depth (fixed-
width) networks (notice that this is actually equivalent to scaling the
variance of the initialization weights with $1/L$, which can be seen as an
initialization scheme). Intuitively, by stabilizing the pre-activations, we
also stabilize the gradients. To confirm this intuition, we show in Fig. 8 the
evolution of gradient norms as they back-propagate through the network. This
experiment was conducted by fixing the last layer’s gradient to a constant
value and back-propagating the gradient from there. The result shows that the
$1/\sqrt{L}$ scaling, along with standard Gaussian initialization, ensure
well-behaved gradients which is a desirable property for gradient-based
training. Another interesting property of the Edge of Chaos initialization
scheme for MLPs is that it ensures that correlation kernel (correlation
between the pre-activations for different inputs) does not exponentially
converge to a degenerate value (constant value)888The correlation still
converges to 1 with an EOC initialization. The benefit of the EOC lies in the
fact that the convergence rate is much slower (polynomial Vs exponential) [2,
5]. We discussed some aspects of the correlation kernel in Section 5 and
showed empirically that with the $1/\sqrt{L}$ scaling, the correlation is
well-behaved and does not converge to degenerate values (Fig. 7).

##### Network collapse.

Another issue that could occur in finite-width networks is that of network
collapse, i.e. when the pre-activations in a hidden layer are all negative,
which causes the post-activations to be all zero. In ResNet (Eq. 1), this
implies that increasing depth beyond some level has no effect on the network
output. This is problematic since the weights in those ‘inactive’ layers have
zero gradient and thus will not be updated when such event occurs. A simple
way to understand network collapse is to see what happens at initialization.
When the width $n$ is sufficiently large, one can expect that such event is
unlikely to occur. What about small-width neural networks? we offer a simple
answer to this question: for finite-width neural networks, increasing the
depth $L$ ensures that such event is unlikely to happen. This is true even for
extremely small widths, e.g. $n=2,3$, which is counter-intuitive. Empirical
results in Fig. 4 support this theoretical prediction.

##### No universal kernel regime.

An interesting application of fixed-depth infinite-width neural network is the
so-called Neural Network Gaussian Process (NNGP). This is the Gaussian process
limit of neural networks, that can be used to perform posterior inference and
obtain uncertainty estimates [7]. The converse case, i.e. fixed-width
infinite-depth, has been however poorly understood, and the question of
whether the infinite-depth limit of finite-width networks has some universal
behaviour has been an open question since. We addressed this question in this
work and showed that the limit (in the case of the ResNet architecture Eq. 1)
does not admit a universal distribution (e.g. Gaussian process in the
infinite-width limit). More precisely, this limit is highly sensitive to the
choice of the activation function.

##### What about infinite-depth-then-width?

the infinite-depth limit of infinite-width neural networks has been studied in
the literature [5, 15]. It is known that in this limit, the network behaves as
a Gaussian process with a well-defined kernel. What about the converse limit,
i.e. infinite-width limit of infinite-depth networks? this has been so far an
open question, and our work addresses one part of it. We show that the
marginal distributions are zero-mean Gaussians with the same variance as in
the infinite-width-then-depth limit. Characterizing the full covariance kernel
is still however an open question (see Section 5 for a discussion on this
topic).

###  7 Conclusion, discussion, and limitations

Understanding the limiting laws of randomly initialized neural networks is
important on many levels. Primarily, understanding these limiting laws allows
us to derive new designs that are immune to exploding/vanishing pre-
activations/gradients phenomena. Next, they also enable a deeper understanding
of overparameterized neural networks, and (often) yield many interesting (and
simple) justifications to the apparent advantage of overparameterization. So
far, the focus has been mainly on the infinite-width limit (and infinite-
width-then-infinite-depth limit) with few developments on the joint limit. Our
work adds to this stream of papers by studying the infinite-depth limit of
finite-width neural networks. We showed that unlike the infinite-width limit,
where we always obtain (under some mild conditions on the activation function)
a Gaussian distribution, the infinite-depth limit is highly sensitive to the
choice of the activation function; using the It$\hat{o}$ ’s lemma, we showed
how we can obtain certain known distributions by carefully tuning the
activation function. In the general width limit, we showed an important
characteristic of infinite-depth neural networks with general activation
functions (including ReLU, conditionally on $\|\phi(X_{0})\|>0$): the
probability of process collapse is zero, meaning that with probability one,
the process $X_{t}$ does not get stuck at any absorbent point. This is not
true for finite-depth ResNets as we can see in Fig. 4, which highlights the
fact that as we increase depth, the collapse probability tends to decrease,
and eventually converges to zero in the infinite-depth limit, which is in
agreement with our results.

This work, although novel in many aspects, is still far from depicting a
complete picture of the infinite-depth limit of finite-width networks. There
are still numerous interesting open questions in this research direction.
Indeed, one of these is the dynamics of the gradient, and more specifically
the behaviour of the NTK in the infinite-depth limit of finite-width neural
networks. For instance, we already know that in the joint infinite-width-depth
limit of MLPs, the NTK is random [31]; but what happens when the width is
fixed and the depth goes to infinity? In the MLP case, a degenerate NTK should
be expected. Henceforth, questions remain as to whether a suitable scaling
leads to interesting (non-degenerate) infinite-depth limit of the NTK as is
the case of the infinite-depth limit of infinite-width NTK [4].

### References

  * [1] B. Poole et al.  “Exponential expressivity in deep neural networks through transient chaos”  In _30th Conference on Neural Information Processing Systems_ , 2016
  * [2] S.S. Schoenholz, J. Gilmer, S. Ganguli and J. Sohl-Dickstein  “Deep Information Propagation”  In _International Conference on Learning Representations_ , 2017
  * [3] G. Yang and S. Schoenholz  “Mean field residual networks: On the edge of chaos”  In _Advances in neural information processing systems_ , 2017, pp. 7103–7114
  * [4] Soufiane Hayou et al.  “Stable ResNet”  In _Proceedings of The 24th International Conference on Artificial Intelligence and Statistics_ 130, Proceedings of Machine Learning Research  PMLR, 2021, pp. 1324–1332
  * [5] S. Hayou, A. Doucet and J. Rousseau  “On the Impact of the Activation Function on Deep Neural Networks Training”  In _International Conference on Machine Learning_ , 2019
  * [6] R.M. Neal  “Bayesian Learning for Neural Networks”  Springer Science & Business Media, 1995
  * [7] J. Lee et al.  “Deep Neural Networks as Gaussian Processes”  In _International Conference on Learning Representations_ , 2018
  * [8] G. Yang  “Tensor Programs III: Neural Matrix Laws”  In _arXiv preprint arXiv:2009.10685_ , 2020
  * [9] A.G. Matthews et al.  “Gaussian Process Behaviour in Wide Deep Neural Networks”  In _International Conference on Learning Representations_ , 2018
  * [10] Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein and Roman Novak  “Infinite attention: NNGP and NTK for deep attention networks”  In _Proceedings of the 37th International Conference on Machine Learning_ 119, Proceedings of Machine Learning Research  PMLR, 2020, pp. 4376–4386
  * [11] Fusheng Liu, Haizhao Yang, Soufiane Hayou and Qianxiao Li  “Connecting Optimization and Generalization via Gradient Flow Path Length”  arXiv, 2022
  * [12] Sanjeev Arora et al.  “Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks”  In _Proceedings of the 36th International Conference on Machine Learning_ 97, Proceedings of Machine Learning Research  PMLR, 2019, pp. 322–332
  * [13] Mariia Seleznova and Gitta Kutyniok  “Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?”  In _Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference_ 145, Proceedings of Machine Learning Research  PMLR, 2022, pp. 868–895
  * [14] Soufiane Hayou, Arnaud Doucet and Judith Rousseau  “ETraining dynamics of deep networks using stochastic gradient descent via neural tangent kernel”  arXiv, 2019
  * [15] S. Hayou, A. Doucet and J. Rousseau  “Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks”  In _arXiv preprint arXiv:1905.13654_ , 2020
  * [16] Soufiane Hayou, Arnaud Doucet and Judith Rousseau  “The Curse of Depth in Kernel Regime”  In _Proceedings on ”I (Still) Can’t Believe It’s Not Better!” at NeurIPS 2021 Workshops_ 163, Proceedings of Machine Learning Research  PMLR, 2022, pp. 41–47
  * [17] Arthur Jacot, Franck Gabriel, Francois Ged and Clement Hongler  “Freeze and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts”  In _Proceedings of Mathematical and Scientific Machine Learning_ 190, Proceedings of Machine Learning Research  PMLR, 2022, pp. 257–270
  * [18] Lechao Xiao, Jeffrey Pennington and Samuel Schoenholz  “Disentangling Trainability and Generalization in Deep Neural Networks”  In _Proceedings of the 37th International Conference on Machine Learning_ 119, Proceedings of Machine Learning Research  PMLR, 2020, pp. 10462–10472
  * [19] Arthur Jacot  “Theory of Deep Learning: Neural Tangent Kernel and Beyond”  In _PhD Thesis, Ecole Polytechnique Federale de Lausanne_ , 2022  URL: https://infoscience.epfl.ch/record/295831/files/EPFL_TH9825.pdf
  * [20] S. Hayou, J.F. Ton, A. Doucet and Y.W. Teh  “Robust Pruning at Initialization”  In _International Conference on Learning Representations_ , 2021
  * [21] Soufiane Hayou, Jean-Francois Ton, Arnaud Doucet and Yee Whye Teh  “Pruning untrained neural networks: Principles and analysis”  In _ArXiv_ , 2020
  * [22] S. Hayou and F. Ayed  “Regularization in ResNet with Stochastic Depth”  In _Proceedings of Thirty-fifth Neural Information Processing Systems (NeurIPS)_ , 2021
  * [23] Yizhang Lou, Chris E Mingard and Soufiane Hayou  “Feature Learning and Signal Propagation in Deep Neural Networks”  In _Proceedings of the 39th International Conference on Machine Learning_ , 2022, pp. 14248–14282
  * [24] Bobby He, Balaji Lakshminarayanan and Yee Whye Teh  “Bayesian Deep Ensembles via the Neural Tangent Kernel”  In _Advances in Neural Information Processing Systems_ 33 Curran Associates, Inc., 2020, pp. 1010–1022
  * [25] James Martens et al.  “Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping”  In _arXiv, preprint 2110.01765_ , 2021
  * [26] Guodong Zhang, Aleksandar Botev and James Martens  “Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers”  In _International Conference on Learning Representations_ , 2022
  * [27] Mufan Li, Mihai Nica and Dan Roy  “The future is log-Gaussian: ResNets and their infinite-depth-and-width limit at initialization”  In _Advances in Neural Information Processing Systems_ 34 Curran Associates, Inc., 2021, pp. 7852–7864
  * [28] Mufan Bill Li, Mihai Nica and Daniel M. Roy  “The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization”  In _arXiv_ , 2022
  * [29] Boris Hanin and Mihai Nica  “Finite Depth and Width Corrections to the Neural Tangent Kernel”  In _International Conference on Learning Representations_ , 2020
  * [30] Boris Hanin  “Correlation Functions in Random Fully Connected Neural Networks at Finite Width”  arXiv, 2022
  * [31] Boris Hanin  “Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations”  In _Mathematics_ 7.10, 2019
  * [32] Stefano Peluchetti and Stefano Favaro  “Infinitely deep neural networks as diffusion processes”  In _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ 108, Proceedings of Machine Learning Research  PMLR, 2020, pp. 1126–1136
  * [33] Pierre Marion, Adeline Fermanian, Gérard Biau and Jean-Philippe Vert  “Scaling ResNets in the Large-depth Regime”  In _arXiv_ , 2022
  * [34] Peter Tankov and Nizar Touzi  “CALCUL STOCHASTIQUE ET FINANCE”, 2018  URL: http://www.cmap.polytechnique.fr/~touzi/Poly-MAP552.pdf
  * [35] Jonathan E. Ingersoll  “Theory of Financial Decision Making”  In _Rowman and Littlefield_ , 1987
  * [36] Bernt Øksendal  “Stochastic Differential Equations”  In _Springer_ , 2003
  * [37] Peter Kloeden and Eckhard Platen  “Numerical Solution of Stochastic Differential Equations”  In _Stochastic Modelling and Applied Probability_ Springer Berlin, Heidelberg, 1995, pp. 342–343
  * [38] Benjamin Jourdain, Sylvie Meleard and Wojbor Woyczynski  “Nonlinear SDEs driven by Lévy processes and related PDEs”  In _Latin American journal of probability and mathematical statistics_ 4, 2007
  * [39] Soo Sung, Supranee Lisawadi and Andrei Volodin  “Weak laws of large numbers for arrays under a condition of uniform integrability”  In _The Korean Mathematical Society J. Korean Math. Soc_ 45, 2008, pp. 289–300

## Appendix

###  A Review of Stochastic Calculus

In this section, we introduce the required mathematical framework and tools to
handle stochastic differential equations (SDEs). We suppose that we have a
probability space $(\Omega,\mathcal{F},\mathbb{P})$, where $\Omega$ is the
event space, $\mathbb{P}$ is the probability measure, and $\mathcal{F}$ is the
sigma-algebra associated with $\Omega$. For $n\geq 1$, we denote by $B$ the
standard $n$-dimensional Brownian motion, and $\mathcal{F}_{t}$ its natural
filtration. Equipped with $(\mathcal{F}_{t})_{t\geq 0}$, we say that the
probability space $(\Omega,\mathcal{F},(\mathcal{F}_{t})_{t\geq
0},\mathbb{P})$ is a filtered probability space. $\mathcal{F}_{t}$ is the
collection of events that are measurable up to time $t$, i.e. can be verified
if we have knowledge of the Brownian motion $B$ (and potentially some other
independent source such as the initial condition of a process $X$ defined by a
$B$-driven stochastic differential equation) up to time $t$. We are now ready
to define a special type of stochastic processes known as It$\hat{o}$
processes.

####  A.1 Existence and uniqueness

######  Definition 1 (It$\hat{o}$ diffusion process)

A stochastic process $(X_{t})_{t\in[0,T]}$ valued in $\mathbb{R}^{n}$ is
called an It$\hat{o}$ diffusion process if it can be expressed as

$X_{t}=X_{0}+\int_{0}^{t}\mu_{s}ds+\int_{0}^{t}\sigma_{s}dB_{s},$

where $B$ is a $n$-dimensional Brownian motion and
$\sigma_{t}\in\mathbb{R}^{n\times n},\mu\in\mathbb{R}^{n}$ are predictable
processes satisfying
$\int_{0}^{T}(\|\mu_{s}\|_{2}+\|\sigma_{s}\sigma_{s}^{\top}\|_{2})ds<\infty$
almost surely.

The following result gives conditions under which a strong solution of a given
SDE exists, and is unique.

######  Theorem 4 (Thm 8.3 in [34])

Let $n\geq 1$, and consider the following SDE

$dX_{t}=\mu(t,X_{t})dt+\sigma(t,X_{t})dB_{t},\quad X_{0}\in L_{2},$

where $B$ is a $m$-dimensional Brownian process for some $m\geq 1$, and
$\mu:\mathbb{R}^{+}\times\mathbb{R}^{n}\to\mathbb{R}^{n}$ and
$\sigma:\mathbb{R}^{+}\times\mathbb{R}^{n}\to\mathbb{R}^{n\times m}$ are
measurable functions satisfying

  1. 1.

there exists a constant $K>0$ such that for all $t\geq 0$,
$x,x^{\prime}\in\mathbb{R}^{n}$

$\|\mu(t,x)-\mu(t,x^{\prime})\|+\|\sigma(t,x)-\sigma(t,x^{\prime})\|\leq
k\|x-x^{\prime}\|.$

  2. 2.

the functions $\|\mu(.,0)\|$ and $\|\sigma(.,0)\|$ are $L_{2}(\mathbb{R}^{+})$
with respect to the Lebesgue measure on $\mathbb{R}^{+}$.

Then, for all $T\geq 0$, there exists a unique strong solution of the SDE
above.

####  A.2 It$\hat{o}$ ’s lemma

The following result, known as It$\hat{o}$ ’s lemma, is a classic result in
stochastic calculus. We state a version of this result from [34]. Other
versions and extensions exist in the literature (e.g. [35, 36, 37]).

######  Lemma 4 (It$\hat{o}$ ’s lemma, Thm 6.7 in [34])

Let $X_{t}$ be an It$\hat{o}$ diffusion process (Definition 1) of the form

$dX_{t}=\mu_{t}dt+\sigma_{t}dB_{t},t\in[0,T],X_{0}\sim\nu$

where $\nu$ is some given distribution. Let
$g:\mathbb{R}^{+}\times\mathbb{R}^{n}\to\mathbb{R}$ be
$\mathcal{C}^{1,2}([0,T],\mathbb{R}^{n})$ (i.e. $\mathcal{C}^{1}$ in the first
variable $t$ and $\mathcal{C}^{2}$ in the second variable $x$). Then, with
probability $1$, we have that

$f(t,X_{t})=f(0,X_{0})+\int_{0}^{t}\nabla_{x}f(s,X_{s})\cdot
dX_{s}+\int_{0}^{t}\left(\partial_{t}f(s,X_{s})+\frac{1}{2}\textup{Tr}\left[\sigma_{s}^{\top}\nabla_{x}^{2}f(s,X_{s})\sigma_{s}\right]\right)ds,$

where $\nabla_{x}f$ and $\nabla_{x}^{2}f$ refer to the gradient and the
Hessian, respectively. This can also be expressed as an SDE

$df(t,X_{t})=\nabla_{x}f(t,X_{t})\cdot
dX_{t}+\left(\partial_{t}f(t,X_{t})+\frac{1}{2}\textup{Tr}\left[\sigma_{t}^{\top}\nabla_{x}^{2}f(t,X_{t})\sigma_{t}\right]\right)dt.$

####  A.3 Convergence of Euler’s scheme to the SDE solution

The following result gives a convergence rate of the Euler discretization
scheme to the solution of the SDE.

######  Theorem 5 ( Corollary of Thm 10.2.2 in [37])

Let $d\geq 1$ and consider the $\mathbb{R}^{d}$-valued ito process $X$
(Definition 1) given by

$X_{t}=X_{0}+\int_{0}^{t}\mu(s,X_{s})ds+\int_{0}^{t}\sigma(s,X_{s})dB_{s},$

where $B$ is a $m$-dimensional Brownian motion for some $m\geq 1$, $X_{0}$
satisfies $\mathbb{E}\|X_{0}\|^{2}<\infty$, and
$\mu:\mathbb{R}^{+}\times\mathbb{R}^{d}\to\mathbb{R}^{d}$ are
$\sigma:\mathbb{R}^{+}\times\mathbb{R}^{d}\to\mathbb{R}^{d\times m}$ are
measurable functions satisfying the following conditions:

  1. 1.

There exists a constant $K>0$ such that for all
$t\in\mathbb{R},x,x^{\prime}\in\mathbb{R}^{d}$,

$\|\mu(t,x)-\mu(t,x^{\prime})\|+\|\sigma(t,x)-\sigma(t,x^{\prime})\|\leq
K\|x-x^{\prime}\|.$

  2. 2.

There exists a constant $K^{\prime}>0$ such that for all
$t\in\mathbb{R},x\in\mathbb{R}^{d}$

$\|\mu(t,x)\|+\|\sigma(t,x)\|\leq K^{\prime}(1+\|x\|).$

  3. 3.

There exists a constant $K^{\prime\prime}>0$ such that for all
$t,s\in\mathbb{R},x\in\mathbb{R}^{d}$,

$\|\mu(t,x)-\mu(s,x)\|+\|\sigma(t,x)-\sigma(s,x)\|\leq
K^{\prime\prime}(1+\|x\|)|t-s|^{1/2}.$

Let $\delta\in(0,1)$ such that $\delta^{-1}\in\mathbb{N}$ (integer), and
consider the times $t_{k}=k\delta$ for $k\in\\{1,\dots,\delta^{-1}\\}$.
Consider the Euler scheme given by

$Y^{i}_{k+1}=Y^{i}_{k}+\mu^{i}(t_{k},Y^{k}_{n})\delta+\sum_{j=1}^{m}\sigma^{i,j}(t_{k},Y^{k}_{n})\Delta
B^{j}_{k},\quad Y^{i}_{0}=X^{i}_{0},$

where $Y^{i},\mu^{i},\sigma^{i,j}$ denote the coordinates of these vectors for
$i\in[d],j\in[m]$, and $\Delta B^{j}_{k}\sim\mathcal{N}(0,\delta)$. Then, we
have that

$\mathbb{E}\sup_{t\in[0,1]}\|X_{t}-Y_{\lfloor
t\delta^{-1}\rfloor}\|^{2}=\mathcal{O}(\delta).$

We can extend the result of 5 to the case of locally Lipschitz drift and
volatility functions $\mu$ and $\sigma$. For this purpose, let us first define
local convergence.

###### Definition 2

Let $(X^{L})_{L\geq 1}$ be a sequence of processes and $X$ be a stochastic
process. For $r>0$, define the following stopping times

$\tau^{L}=\\{t\geq 0:|X^{L}_{t}|\geq r\\},\tau=\\{t\geq 0:|X_{t}|\geq r\\}.$

We say that $X^{L}$ converges locally to $X$ if for any $r>0$,
$X^{L}_{t\land\tau^{L}}$ converge to $X_{t\land\tau}$.
This definition is general for any type of convergence, we will specify
clearly the type of convergence when we use this notion of local convergence.

######  Lemma 5 (Locally-Lipschitz coefficients)

Consider the same setting of 5 with the following conditions instead

  1. 1.

For any $r>0$, there exists a constant $K>0$ such that for all
$t\in\mathbb{R},x,x^{\prime}\in\mathbb{R}^{d}$ with $\|x\|,\|x^{\prime}\|\leq
r$,

$\|\mu(t,x)-\mu(t,x^{\prime})\|+\|\sigma(t,x)-\sigma(t,x^{\prime})\|\leq
K\|x-x^{\prime}\|.$

  2. 2.

For any $r>0$, there exists a constant $K^{\prime}>0$ such that for all
$t\in\mathbb{R},x\in\mathbb{R}^{d}$ satisfying $\|x\|\leq r$

$\|\mu(t,x)\|+\|\sigma(t,x)\|\leq K^{\prime}(1+\|x\|).$

  3. 3.

For any $r>0$, there exists a constant $K^{\prime\prime}>0$ such that for all
$t,s\in\mathbb{R},x\in\mathbb{R}^{d}$ satisfying $\|x\|\leq r$,

$\|\mu(t,x)-\mu(s,x)\|+\|\sigma(t,x)-\sigma(s,x)\|\leq
K^{\prime\prime}(1+\|x\|)|t-s|^{1/2}.$

Then, for any $r>0$, we have that

$\mathbb{E}\sup_{t\in[0,1]}\|X_{t\land\tau}-Y_{\lfloor(t\land\tau_{\delta})\delta^{-1}\rfloor}\|^{2}=\mathcal{O}(\delta),$

where $\tau_{\delta}=\inf\\{t\geq 0:\|Y_{\lfloor t\delta^{-1}\rfloor}\|>r\\}$,
and $\tau=\inf\\{t\geq 0:\|X_{t}\|>r\\}$.

We omit the proof here as it consists of the same techniques used in [37],
with the only difference consisting of considering the stopped process
$X^{\tau}$. By stopping the process, we force the process to stay in a region
where the coefficients are Lipschitz.

####  A.4 Convergence of Particles to the solution of Mckean-Vlasov process

The next result gives sufficient conditions for the system of particles to
converge to its mean-field limit, known as the Mckean-Vlasov process.

######  Theorem 6 ( Mckean-Vlasov process, Corollary of Thm 3 in [38])

et $d\geq 1$ and consider the $\mathbb{R}^{d}$-valued ito process $X$
(Definition 1) given by

$dX_{t}=\sigma(X_{t},\nu^{n}_{t})dB_{t},\quad X_{0}\textrm{ has
{iid}~{}components},$

where $B$ is a $d$-dimensional Brownian motion,
$\nu^{n}_{t}\overset{def}{=}\frac{1}{d}\sum_{i=1}^{d}\delta_{\\{X^{i}_{t}\\}}$
the empirical distribution of the coordinates of $X_{t}$, and $\sigma$ is
real-valued given by $\sigma(x,\nu)=\int\zeta(x,y)d\nu(y)$ for all
$x\in\mathbb{R}^{d}$ and distribution $\nu$. Assume that the function $\zeta$
is Lipschitz continuous. Then, we have that for all $T\in\mathbb{R}^{+}$,

$\sup_{i\in[n]}\mathbb{E}\left(\sup_{t\leq
T}|X^{i}_{t}-\tilde{X}^{i}_{t}|^{2}\right)=\mathcal{O}(n^{-1}),$

where $\tilde{X}^{i}$ is the solution of the following Mckean-Vlasov equation

$d\tilde{X}^{i}_{t}=\sigma(\tilde{X}^{i}_{t},\nu^{i}_{t})dB^{i}_{t},\quad\tilde{X}^{i}_{0}=X^{i}_{0},$

where $\nu^{i}_{t}$ is the distribution of $\tilde{X}^{i}$.

####  A.5 Other results from probability and stochastic calculus

The next trivial lemma has been opportunely used in [27] to derive the
limiting distribution of the network output (multi-layer perceptron) in the
joint infinite width-depth limit. This simple result will also prove useful in
our case of the finite-width-infinite-depth limit.

###### Lemma 6

Let $W\in\mathbb{R}^{n\times n}$ be a matrix of standard Gaussian random
variables $W_{ij}\sim\mathcal{N}(0,1)$. Let $v\in\mathbb{R}^{n}$ be a random
vector independent from $W$ and satisfies $\|v\|_{2}=1$ . Then,
$Wv\sim\mathcal{N}(0,I)$.

Proof  The proof follows a simple characteristic function argument. Indeed, by
conditioning on $v$, we observe that $Wv\sim\mathcal{N}(0,I)$. Let
$u\in\mathbb{R}^{n}$, we have that

$\displaystyle\mathbb{E}_{W,v}[e^{i\langle u,Wv\rangle}]$
$\displaystyle=\mathbb{E}_{v}[\mathbb{E}_{W}[e^{i\langle u,Wv\rangle}|v]]$
$\displaystyle=\mathbb{E}_{v}[e^{-\frac{\|u\|^{2}}{2}}]$
$\displaystyle=e^{-\frac{\|u\|^{2}}{2}}.$

This concludes the proof as the latter is the characteristic function of a
random Gaussian vector with Identity covariance matrix.

The next theorem shows when a stochastic process (ito)

######  Theorem 7 (Variation of Thm 8.4.3 in [36])

Let $(X_{t})_{t\in[0,T]}$ and $(Y_{t})_{t\in[0,T]}$ be two stochastic
processes given by

$\begin{cases}dX_{t}=b(X_{t})dt+\sigma(X_{t})dB_{t},\quad
X_{0}=x\in\mathbb{R}.\\\ dY_{t}=b_{t}dt+v_{t}d\hat{B}_{t},\quad
Y_{0}=X_{0},\end{cases}$

where $\sigma:\mathbb{R}\to\mathbb{R}^{1\times k}$, $(b_{t})_{t\geq 0}$ and
$(v_{t})_{t\geq 0}$ are real valued adapted stochastic processes, and $v$ is
adapted to the filtration of the Brownian motion $(\hat{B}_{t})_{t\geq 0}$,
$(B_{t})_{t\geq 0}$ is an $k$-dimensional Brownian motion and
$(\hat{B}_{t})_{t\geq 0}$ is a $1$-dimensional Brownian motion. Assume that
$\mathbb{E}[b_{t}|\mathcal{N}_{t}]=b(Y_{t})$ where
$\mathcal{N}_{t}=\sigma((Y_{s})_{s\leq t})$ is the $\sigma$-Algebra generated
by $\\{Y_{s}:s\leq t\\}$, and $v_{t}^{2}=\sigma(Y_{t})\sigma(Y_{t})^{\top}$
almost surely (in terms of $dt\times dP$ measure where $dt$ is the natural
Borel measure on $[0,T]$ and $dP$ is the probability measure associated with
the probability space). Then, $X_{t}$ and $Y_{t}$ have the same distribution
for all $t\in[0,T]$.

Proof  The proof of this theorem is the same as that of Thm 8.4.3 in [36] with
small differences. Indeed, our result is slightly different from that of [36]
in the sense that here we consider Brownian motions with different dimensions,
while in their theorem, the author considers the case where the Brownian
motions involved in $(X_{t})$ and $(Y_{t})$ are of the same dimension.
However, both results make use of the so-called Martingale problem, which
characterizes the weak uniqueness and hence the distribution of Ito
processes999We omit the details on the Martingale problem here. We invite the
curious reader to check Chapter 8 in [36] for further details.. The generator
of $X_{t}$ is given for $f\in\mathcal{C}^{2}(\mathbb{R})$ by

$\mathcal{G}(f)(x)=b(x)\frac{\partial f}{\partial
x}+\frac{1}{2}\sigma(x)\sigma(x)^{\top}\frac{\partial^{2}f}{\partial x^{2}}.$

Now define the process $\mathcal{H}(f)$ for $f\in\mathcal{C}^{2}(\mathbb{R})$
by

$\mathcal{H}(f)(t)=b_{t}\frac{\partial f}{\partial
x}(Y_{t})+\frac{1}{2}v^{2}_{t}\frac{\partial^{2}f}{\partial x^{2}}(Y_{t}).$

Let $\mathcal{N}_{t}=\sigma((Y_{s})_{s\leq t})$ be the $\sigma$-Algebra
generated by $\\{Y_{s}:s\leq t\\}$. Using It$\hat{o}$ lemma, we have that for
$s<t$,

$\displaystyle\mathbb{E}[f(Y_{s})|\mathcal{N}_{t}]$
$\displaystyle=f(Y_{t})+\mathbb{E}[\int_{t}^{s}\mathcal{H}(f)(r)dr|\mathcal{N}_{t}]$
$\displaystyle=f(Y_{t})+\mathbb{E}\left[\int_{t}^{s}\mathbb{E}[\mathcal{H}(f)(r)|\mathcal{N}_{r}]dr|\mathcal{N}_{t}\right]$
$\displaystyle=f(Y_{t})+\mathbb{E}\left[\int_{t}^{s}\mathcal{G}(f)(Y_{r})dr|\mathcal{N}_{t}\right],$

where we have used the fact that $\mathbb{E}[b_{r}|\mathcal{N}_{r}]=b(Y_{r})$.
Now define the process $M$ by

$M_{t}=f(Y_{t})-\int_{0}^{t}\mathcal{G}(f)(Y_{r})dr.$

For $s>t$, we have that

$\displaystyle\mathbb{E}[M_{s}|\mathcal{N}_{t}]$
$\displaystyle=f(Y_{t})+\mathbb{E}\left[\int_{t}^{s}\mathcal{G}(f)(Y_{r})dr|\mathcal{N}_{t}\right]-\mathbb{E}\left[\int_{0}^{s}\mathcal{G}(f)(Y_{r})dr|\mathcal{N}_{t}\right]\quad\textup{(by
It$\hat{o}$ lemma),}$
$\displaystyle=f(Y_{t})-\mathbb{E}\left[\int_{0}^{t}\mathcal{G}(f)(Y_{r})dr|\mathcal{N}_{t}\right]=M_{t}.$

Hence, $M_{t}$ is a martingale (w.r.t to $\mathcal{N}_{t}$). We conclude that
$Y_{t}$ has the same law as $X_{t}$ by the uniqueness of the solution of the
martingale problem (see 8.3.6 in [36]).

The next result is a simple corollary of the existence and uniqueness of the
strong solution of an SDE under the Lipschitz conditions on the drift and the
volatility. It basically shows that a zero-drift process collapses (becomes
constant) once the volatility is zero.

###### Lemma 7

Let $g:\mathbb{R}^{n}\to\mathbb{R}$ be a Lipschitz function. Let $Z$ be the
solution of the stochastic differential equation

$dZ_{t}=g(Z_{t})dB_{t},\quad Z_{0}\in\mathbb{R}^{n}.$

If $g(Z_{0})=0$, then $Z_{t}=Z_{0}$ almost surely.

Proof  This follows for the uniqueness of the strong solution of an SDE(4).

####  A.6 Proof of 1

We are now ready to prove the following result.

Proposition 1. _Assume that the activation function $\phi$ is Lipschitz on
$\mathbb{R}^{n}$. Then, in the limit $L\to\infty$, the process
$X^{L}_{t}=Y_{\lfloor tL\rfloor}$, $t\in[0,1]$, converges in distribution to
the solution of the following SDE_

$dX_{t}=\frac{1}{\sqrt{n}}\|\phi(X_{t})\|dB_{t},\quad X_{0}=W_{in}x,$ (6)

_where $(B_{t})_{t\geq 0}$ is a Brownian motion (Wiener process). Moreover, we
have that for any $t\in[0,1]$ Lipschitz function
$\Psi:\mathbb{R}^{n}\to\mathbb{R}$,_

$\mathbb{E}\Psi(Y_{\lfloor
tL\rfloor})=\mathbb{E}\Psi(X_{t})+\mathcal{O}(L^{-1/2}),$

_where the constant in $\mathcal{O}$ does not depend on $t$.
Moreover, if the activation function $\phi$ is only locally Lipschitz, then
$X^{L}_{t}$ converges locally to $X_{t}$. More precisely, for any fixed $r>0$,
we consider the stopping times_

$\tau^{L}=\inf\\{t\geq 0:\|X^{L}_{t}\|\geq r\\},\quad\tau=\inf\\{t\geq
0:\|X_{t}\|\geq r\\},$

_then the stopped process $X^{L}_{t\land\tau^{L}}$ converges in distribution
to the stopped solution $X_{t\land\tau}$ of the above SDE._

Proof  The proof is based on 5 in the appendix. It remains to express Eq. 1 in
the required form and make sure all the conditions are satisfied for the
result to hold. Using Lemma 6, we can write Eq. 1 as

$Y_{l}=Y_{l-1}+\frac{1}{\sqrt{L}}\sigma(Y_{l-1})\zeta^{L}_{l-1},$

where $\sigma(y)\overset{def}{=}\frac{1}{\sqrt{n}}\|\phi(y)\|$ for all
$y\in\mathbb{R}^{n}$ and $\zeta^{L}_{l}$ are iid random Gaussian vectors with
distribution $\mathcal{N}(0,I)$. This is equal in distribution to the Euler
scheme of SDE Eq. 7. Since $\sigma$ trivially inherits the Lipschitz or local
Lipschitz properties of $\phi$, we conclude for the convergence using 5 and
Lemma 5.

Now let $\Psi$ be $K$-Lipschitz for some constant $K>0$. We have that

$|\mathbb{E}\Psi(Y_{\lfloor tL\rfloor})-\mathbb{E}\Psi(X_{t})|\leq
K\mathbb{E}\sup_{t\in[0,1]}\|\bar{Y}_{\lfloor
tL\rfloor}-X_{t}\|=\mathcal{O}(L^{-1/2}),$

where $\bar{Y}$ is the Euler scheme as in 5, and where we have used the fact
that $Y_{\lfloor tL\rfloor}$ and $\bar{Y}_{\lfloor tL\rfloor}$ have the same
distribution.

The result of 1 can be generalized to the case with multiple inputs with
minimal changes in the proof. We summarize this result in the next
proposition.

###### Proposition 5

Let $x_{1},x_{2},\dots,x_{k}\in\mathbb{R}^{d}$ be non-zero inputs, and denote
by $Y_{l}(x_{i})$ the pre-activation vector in layer $l$ for the input
$x_{i}$. Consider the vector
$\bm{Y}_{l}^{k}=(Y_{l}(x_{1})^{\top},Y_{l}(x_{2})^{\top},\dots,Y_{l}(x_{k})^{\top})^{\top}\in\mathbb{R}^{k\cdot
n}$ consisting of the concatenation of the pre-activations vectors for all
inputs $x_{i}$. Assume that the activation function $\phi$ is Lipschitz on
$\mathbb{R}^{n}$. Then, in the limit $L\to\infty$, the process
$\bm{X}^{L,k}_{t}=\bm{Y}^{k}_{\lfloor tL\rfloor}$, $t\in[0,1]$, converges in
distribution to the solution of the following SDE

$d\bm{X}^{k}_{t}=\frac{1}{\sqrt{n}}\Sigma(\bm{X}^{k}_{t})^{1/2}d\bm{B}_{t},\quad\bm{X}^{k}_{0}=((W_{in}x_{1})^{\top},\dots,(W_{in}x_{k})^{\top})^{\top},$
(7)

where $(\bm{B}_{t})_{t\geq 0}$ is an $kn$-dimensional Brownian motion (Wiener
process), independent from $W_{in}$, and $\Sigma(\bm{X}^{k}_{t})$ is the
covariance matrix given by

$\Sigma(\bm{X}^{k}_{t})=\left[\begin{array}[]{ c | c | c | c}\alpha_{1,1}I_{n}&\alpha_{1,2}I_{n}&\dots&\alpha_{1,k}I_{n}\\\ \hline\cr\alpha_{2,1}I_{n}&\alpha_{2,2}I_{n}&\dots&\alpha_{2,k}I_{n}\\\ \hline\cr\vdots&\vdots&\vdots&\vdots\\\ \alpha_{k,1}I_{n}&\dots&\dots&\alpha_{k,k}I_{n}\\\ \end{array}\right],$

where
$\alpha_{i,j}=\langle\phi(\bm{X}_{t}^{k,i}),\phi(\bm{X}_{t}^{k,j})\rangle$,
with
$((X_{t}^{k,1})^{\top},\dots,(X_{t}^{k,k})^{\top})^{\top}\overset{def}{=}\bm{X}_{t}^{k}$.
Moreover, if the activation function $\phi$ is only locally Lipschitz, then
$\bm{X}^{L,k}_{t}$ converges locally to $\bm{X}_{t}^{k}$. More precisely, for
any fixed $r>0$, we consider the stopping times $\tau^{L}=\inf\\{t\geq
0:\|\bm{X}^{L,k}_{t}\|\geq r\\}$, and $\quad\tau=\inf\\{t\geq
0:\|\bm{X}^{k}_{t}\|\geq r\\},$ then the stopped process
$\bm{X}^{L,k}_{t\land\tau^{L}}$ converges in distribution to the stopped
solution $\bm{X}^{k}_{t\land\tau}$ of the above SDE.

Proof  The proof is similar to that of 1. The only difference lies the
definition of the Gaussian vector $\zeta^{L}_{l}$. In this case, we have for
all $x_{i}$

$Y_{l}(x_{i})=Y_{l-1}(x_{i})+\frac{1}{\sqrt{L}}\frac{1}{\sqrt{n}}\zeta^{L}_{l-1}(Y_{l-1}(x_{i})),$

where
$\zeta^{L}_{l-1}(Y_{l-1}(x_{i}))\overset{def}{=}\sqrt{n}W_{l}\phi(Y_{l-1}(x_{i}))$.
Concatenating these identities yield

$\bm{Y}^{k}_{l}=\bm{Y}^{k}_{l-1}+\frac{1}{\sqrt{L}}\frac{1}{\sqrt{n}}\bm{\zeta}^{L}_{l-1},$

where $\bm{\zeta}^{L}_{l-1}$ is the concatenation of the vector
$\zeta^{L}_{l-1}(Y_{l-1}(x_{i}))$ for $i=1,\dots,k$. It is straightforward
that the covariance matrix of the Gaussian vector $\bm{\zeta}^{L}_{l-1}$ is
given by the matrix $\Sigma$ above (with $X$ replaced by $Y$). We conclude
using 5.

###  B Some technical results for the proofs

####  B.1 Approximation of $X$

In the next lemma, we provide an approximate stochastic process $X^{m}$ to
$X$, that differs from $X$ by the volatility term. The upper-bound on the
$L_{2}$ norm of the difference between $X^{m}$ and $X$ will prove useful in
the proofs of other results. The proof of this lemma requires the use of
Gronwall’s lemma, a tool that is often used in stochastic calculus.

###### Lemma 8

Let $x\in\mathbb{R}^{d}$ such that $x\neq 0$, $m\geq 1$ be an integer, and
consider the two stochastic processes $X^{m}$ and $X$ given by

$\begin{cases}dX^{m}_{t}=\frac{1}{\sqrt{n}}\|\phi_{m}(X^{m}_{t})\|dB_{t}\,,\quad
t\in[0,\infty),\quad X^{m}_{0}=W_{in}x,\\\
dX_{t}=\frac{1}{\sqrt{n}}\|\phi(X_{t})\|dB_{t}\,,\quad t\in[0,\infty),\quad
X_{0}=W_{in}x,\end{cases}$

where $\phi_{m}(z)=\int_{0}^{z}h(mu)du$ where $h$ is the Sigmoid function
given by $h(u)=(1+e^{-u})^{-1}$, $\phi$ is the ReLU activation function, and
$(B_{t})_{t\geq 0}$ is an $n$-dimensional Brownian motion. We have the
following

$\forall t\geq
0,\,\,\mathbb{E}\left\|X^{m}_{t}-X_{t}\right\|^{2}\leq\frac{2nt}{m^{2}}e^{2t}.$

Proof  Let $t\geq 0$. We have that

$\displaystyle\mathbb{E}\left\|X^{m}_{t}-X_{t}\right\|^{2}=\frac{1}{n}\mathbb{E}\left\|\int_{0}^{t}(\|\phi_{m}(X^{m}_{s})\|-\|\phi(X_{s})\|)dB_{s}.\right\|^{2}$

Using It$\hat{o}$ isometry and the fact that
$(\|\phi_{m}(X^{m}_{s})\|-\|\phi(X_{s})\|)^{2}\leq\|\phi_{m}(X^{m}_{s})-\phi(X_{s})\|^{2}$,
we obtain

$\displaystyle\mathbb{E}\left\|X^{m}_{t}-X_{t}\right\|^{2}$
$\displaystyle\leq\int_{0}^{t}\mathbb{E}\left\|\phi_{m}(X^{m}_{s})-\phi(X_{s})\right\|^{2}ds$
$\displaystyle\leq
2\int_{0}^{t}\mathbb{E}\left\|\phi_{m}(X^{m}_{s})-\phi(X^{m}_{s})\right\|^{2}ds+2\int_{0}^{t}\mathbb{E}\left\|\phi(X^{m}_{s})-\phi(X_{s})\right\|^{2}ds$
$\displaystyle\leq\frac{2nt}{m^{2}}+2\int_{0}^{t}\mathbb{E}\left\|X^{m}_{s}-X_{s}\right\|^{2}ds,$

where we have used Lemma 9 and the fact that ReLU is $1$-Lipschitz. We
concldue using Gronwall’s lemma.

####  B.2 Approximation of $\phi$

The next lemma provides a simple upper-bound on the distance between the ReLU
activation $\phi$ and an approximate function $\phi_{m}$ that converges to
$\phi$ in the limit of large $m$.

###### Lemma 9

Consider the function $\phi_{m}(z)=\int_{0}^{z}h(mu)du$ where $z\in\mathbb{R}$
where $m\geq 1$. We have that

$\sup_{z\in\mathbb{R}}|\phi_{m}(z)-\phi(z)|\leq\frac{1}{m}.$

Proof  Let $m\geq 1$ and $z\in\mathbb{R}$. Assume that $z>0$. We have that

$\displaystyle|\phi_{m}(z)-\phi(z)|$
$\displaystyle=\int_{0}^{z}\frac{e^{-mu}}{1+e^{-mu}}du$
$\displaystyle\leq\int_{0}^{z}e^{-mu}du$
$\displaystyle=\frac{1}{m}(1-e^{-mz})\leq\frac{1}{m}.$

For the case where $z\leq 0$, the proof is the same. We have that

$\displaystyle|\phi_{m}(z)-\phi(z)|$
$\displaystyle=\int_{0}^{z}\frac{e^{mu}}{1+e^{mu}}du$
$\displaystyle\leq\int_{0}^{z}e^{mu}du$
$\displaystyle=\frac{1}{m}(1-e^{mz})\leq\frac{1}{m},$

which concludes the proof.

####  B.3 Other lemmas

The next lemma shows that the logarithmic growth factor
$\log\left(\frac{\|\phi_{m}(X^{m}_{t})\|}{\|\phi_{m}(X^{m}_{0})\|}\right)$
converges to $\log\left(\frac{\|\phi(X_{t})\|}{\|\phi(X_{0})\|}\right)$ when
$m$ goes to infinity, where the convergence holds in $L_{1}$. The key
ingredient in the use of uniform integrability coupled with convergence in
probability, which is sufficient to conclude on the $L_{1}$ convergence. This
result will help us conclude in the proof of 1.

###### Lemma 10

Let $x\in\mathbb{R}^{d}$ such that $x\neq 0$, $m\geq 1$ be an integer, and
consider the two stochastic processes $X^{m}$ and $X$ given by

$\begin{cases}dX^{m}_{t}=\frac{1}{\sqrt{n}}\|\phi_{m}(X^{m}_{t})\|dB_{t}\,,\quad
t\in[0,\infty),\quad X^{m}_{0}=W_{in}x,\\\
dX_{t}=\frac{1}{\sqrt{n}}\|\phi(X_{t})\|dB_{t}\,,\quad t\in[0,\infty),\quad
X_{0}=W_{in}x,\end{cases}$

where $\phi_{m}(z)=\int_{0}^{z}h(mu)du$ where $h$ is the Sigmoid function
given by $h(u)=(1+e^{-u})^{-1}$, $\phi$ is the ReLU activation function, and
$(B_{t})_{t\geq 0}$ is an $n$-dimensional Brownian motion. Then, conditionally
on the fact that $\|\phi(X_{0})\|>0$, we have that

$\forall t\geq
0,\,\,\log\left(\frac{\|\phi_{m}(X^{m}_{t})\|}{\|\phi_{m}(X^{m}_{0})\|}\right)\overset{L^{1}}{\longrightarrow}\log\left(\frac{\|\phi(X_{t})\|}{\|\phi(X_{0})\|}\right).$

Proof

Let $t>0$. From Lemma 8, we know that $X^{m}$ converges in $L^{2}$ to $X$.
Using Lemma 9 and the fact that ReLU is $1$-Lipschitz, we obtain

$\mathbb{E}\|\phi_{m}(X^{m}_{t})-\phi(X_{t})\|^{2}\leq\frac{2n}{m^{2}}+2\mathbb{E}\|X^{m}_{t}-X_{t}\|^{2},$

which implies that $\phi_{m}(X^{m}_{t})$ converges in $L^{2}$ to
$\phi(X_{t})$. In particular, the convergence holds in probability. Using this
fact with the Continuous mapping theorem, we obtain that

$\forall t\geq
0,\,\,\log\left(\|\phi_{m}(X^{m}_{t})\|\right)\overset{\mathbb{P}}{\longrightarrow}\log\left(\|\phi(X_{t})\|\right).$
(8)

Let us show the following,

$\forall t\geq
0,\,\,\log\left(\frac{\|\phi_{m}(X^{m}_{t})\|}{\|\phi_{m}(X^{m}_{0})\|}\right)\overset{\mathbb{P}}{\longrightarrow}\log\left(\frac{\|\phi(X_{t})\|}{\|\phi(X_{0})\|}\right).$

Let $\epsilon>0$ and $t>0$. We have

$\displaystyle\mathbb{P}\left(\left|\log\left(\frac{\|\phi_{m}(X^{m}_{t})\|}{\|\phi_{m}(X^{m}_{0})\|}\right)-\log\left(\frac{\|\phi(X_{t})\|}{\|\phi(X_{0})\|}\right)\right|\geq\epsilon\right)$
$\displaystyle\leq\mathbb{P}\left(\left|\log\|\phi_{m}(X^{m}_{t})\|-\log\left(\|\phi(X_{t})\|\right)\right|\geq\epsilon/2\right)$
$\displaystyle+\mathbb{P}\left(\left|\log\|\phi_{m}(X^{m}_{0})\|-\log\left(\|\phi(X_{0})\|\right)\right|\geq\epsilon/2\right),$

where the first term converges to zero by Eq. 8, and the second term converges
to zero by Lemma 9. Hence, the convergence in probability holds.

To conclude, it suffices to show that the sequence of random variables
$\left(Y^{m}_{t}=\log\left(\frac{\|\phi_{m}(X^{m}_{t})\|}{\|\phi_{m}(X^{m}_{0})\|}\right)\right)_{m\geq
1}$ is uniformly integrable.
Let $K>0$. From the proof of Lemma 2, with $\zeta=\phi_{m}$, we have that

$Y^{m}_{t}=\frac{1}{\sqrt{n}}\int_{0}^{t}\mu(X^{m}_{s})ds+\frac{1}{2n}\int_{0}^{t}\sum_{i=1}^{n}\sigma_{i}(X^{m}_{s})dB^{i}_{s},$

where
$\sigma_{i}(X^{m}_{s})=\frac{|\phi_{m}^{\prime}(X^{m,i}_{s})\phi_{m}(X^{m,i}_{s})|}{\|\phi_{m}(X^{m}_{s})\|}$,
and
$\mu(X^{m}_{s})=\frac{1}{2}\sum_{i=1}^{n}\left(\phi_{m}^{\prime\prime}(X^{m,i}_{s})\phi_{m}(X^{m,i}_{s})+\phi_{m}^{\prime}(X^{m,i}_{s})^{2}\right)-\frac{\|\phi_{m}^{\prime}(X^{m}_{s})\circ\phi_{m}(X^{m}_{s})\|^{2}}{\|\phi_{m}(X^{m}_{s})\|^{2}}$.
Therefore,

$\displaystyle\mathbb{E}|Y^{m}_{t}|^{2}$
$\displaystyle=\frac{1}{n}\mathbb{E}\left(\int_{0}^{t}\mu(X^{m}_{s})ds\right)^{2}+\frac{1}{4n^{2}}\mathbb{E}\int_{0}^{t}\sum_{i=1}^{m}\sigma_{i}(X^{m}_{s})^{2}ds$
(9)
$\displaystyle\leq\frac{t}{n}\int_{0}^{t}\mathbb{E}\mu(X^{m}_{s})^{2}ds+\frac{1}{4n^{2}}\mathbb{E}\int_{0}^{t}\sum_{i=1}^{m}\sigma_{i}(X^{m}_{s})^{2}ds,$

where we have used the It$\hat{o}$ isometry and Cauchy-Schwartz inequality.
Using the conditions on $\phi_{m}$, it is straightforward that term
$\frac{1}{4n^{2}}\mathbb{E}\int_{0}^{t}\sum_{i=1}^{m}\sigma_{i}(X^{m}_{s})^{2}ds$
is uniformly bounded. It remains to bound the first term. Similarly to the
proof of 1, we condition on the regions of $|X^{m,i}_{s}|$ and obtain that the
terms $\mathbb{E}\mu(X^{m}_{s})^{2}$ are uniformly bounded over $m$ (we omit
the proof here as it is just a repetition of the techniques used in the proof
of 1). Therefore, we have that $\sup_{m\geq
1}\mathbb{E}|Y^{m}_{t}|^{2}<\infty$, which implies uniform integrability. This
concludes the proof.

###### Lemma 11

Let $x\in\mathbb{R}^{d}$ such that $x\neq 0$, $m\geq 1$ be an integer, and
consider the stochastic processes $X$ given by

$dX_{t}=\frac{1}{\sqrt{n}}\|\phi(X_{t})\|dB_{t}\,,\quad t\in[0,\infty),\quad
X_{0}=W_{in}x,$

where $\phi$ is the ReLU activation function, and $(B_{t})_{t\geq 0}$ is an
$n$-dimensional Brownian motion independent from $X_{0}$. Then, conditionally
on the fact that $\|\phi(X_{0})\|>0$, we have that for all $s\in[0,1],i\in[n]$

$\mathbb{P}(|X^{i}_{s}|\leq\delta)=\mathcal{O}_{\delta\to 0}(\delta),$

where the bound holds uniformly over $s\in[0,1]$.

Proof  We have that

$X^{i}_{s}=X^{i}_{0}+\frac{1}{\sqrt{n}}\int_{0}^{s}\|\phi(X_{u})\|dB^{i}_{u}.$

Since $\phi(X_{u})>0$ for all $u\geq 0$ almost surely, and by the independence
of $B$ and $X_{0}$, we can easily see that $X^{i}_{s}$ has no Dirac mass and
is the sum of two continuous random variables (not independent) $X^{i}_{0}$
and $\frac{1}{\sqrt{n}}\int_{0}^{s}\|\phi(X_{u})\|dB^{i}_{u}$ that have
bounded density functions, and thus $X^{i}_{s}$ has a bounded density function
$h_{s}$. Hence, writing
$\mathbb{P}(|X^{i}_{s}|\leq\delta)=\int_{-\delta}^{\delta}h_{s}(t)dt=\mathcal{O}(\delta)$
concludes the proof. The bound can be taken uniformly over $s\in[0,1]$ by
taking $\sup_{s\in[0,1]|h_{s}(0)|}$.

###  C The Ornstein-Uhlenbeck (OU) process

The OU process is the (unique) strong solution to the following diffusion

$dX_{t}=a(b-X_{t})dt+\sigma dB_{t},$ (10)

where $a,b,\sigma\in\mathbb{R}$ are constants, and $B$ is a one dimensional
Brownian motion. In financial mathematics, the OU process is used as a model
of short-term interest rate under the name of the Vasicek model. The OU
process has a closed-form expression and its marginal distribution is
Gaussian. The next lemma gives a full characterization of the marginal
distributions of an OU process.

###### Lemma 12

Eq. 10 admits the following solution

$X_{t}=X_{0}e^{-at}+b(1-e^{-at})+\sigma\int_{0}^{t}e^{-a(t-s)}dB_{s}.$

As a result, we have the following

  * •

$X_{t}$ is Gaussian.

  * •

$\mathbb{E}[X_{t}]=X_{0}e^{-at}+b(1-e^{-at}).$

  * •

$\textup{Cov}(X_{t},X_{s})=\frac{\sigma^{2}}{2a}\left(e^{-a|t-s|}-e^{-a(t+s)}\right)$.

Proof  Consider the process $Z_{t}=e^{at}X_{t}$, using It$\hat{o}$ lemma, we
have that

$\displaystyle dZ_{t}$ $\displaystyle=aZ_{t}dt+e^{at}dX_{t}$
$\displaystyle=abe^{at}dt+\sigma e^{at}dW_{t}.$

Integrating between $0$ and $t$ yields

$Z_{t}=Z_{0}+b(e^{at}-1)+\sigma\int_{0}^{t}e^{as}dW_{s}.$

We conclude by multiplying both sides with $e^{-at}$.

The result for $\mathbb{E}[X_{t}]$ is straightforward since
$\mathbb{E}\left[\int_{0}^{t}e^{-a(t-s)}dW_{s}\right]=0$ by the properties of
It$\hat{o}$ integral and the Brownian motion. For the covariance, without loss
of generality assume that $t>s\geq 0$. We have that

$\displaystyle\textup{Cov}(X_{t},X_{s})$
$\displaystyle=\sigma^{2}\mathbb{E}\left[\int_{0}^{t}e^{-a(t-u)}dW_{u}\int_{0}^{s}e^{-a(s-u)}dW_{u}\right]$
$\displaystyle=\sigma^{2}\mathbb{E}\left[\int_{0}^{s}e^{-a(t-u)}dW_{u}\int_{0}^{s}e^{-a(s-u)}dW_{u}\right]$
$\displaystyle=\sigma^{2}\int_{0}^{s}e^{-2a(\frac{s+t}{2}-u)}du$
$\displaystyle=\frac{\sigma^{2}}{2a}\left(e^{-a(t-s)}-e^{-a(t+s)}\right),$

which completes the proof.

We would like to find sufficient conditions on the activation function $\phi$
and a function $g$ such that the process $g(X_{t})$ (Eq. 7) follows an the OU
dynamics. For this purpose, we proceed by reverse-engineering the problem;
Using It$\hat{o}$ ’s lemma (Eq. 3), this is satisfied when there exist
constants $a,b,\sigma$ such that

$\begin{cases}\mu(y)=\frac{1}{2n}\phi(y)^{2}g^{\prime\prime}(y)=a(b-g(y))\\\
\sigma(y)=\frac{1}{\sqrt{n}}\phi(y)g^{\prime}(y)=\sigma.\\\ \end{cases}$

This implies that $\frac{g^{\prime\prime}(y)}{g^{\prime
2}(y)}=2a\sigma^{-2}(b-g(y))$. Letting $G=\int g$ be the primitive function of
$g$, we obtain that $G$ satisfies a differential equation of the form

$\frac{1}{G^{\prime\prime}(y)}=\alpha y+\beta G(y)+\zeta,$

where $\alpha,\beta,\zeta\in\mathbb{R}$ are constants.

Let us consider the case where $\alpha=\zeta=0$ and $\beta\neq 0$, i.e.
$\frac{1}{G^{\prime\prime}(y)}=\beta G(y)$. Equivalently, we solve the
differential equation $G^{\prime\prime}(y)=\frac{\beta}{G}$ where
$\beta\in\mathbb{R}$. Multiplying both sides by $G^{\prime}$ and integrating
we obtain $\frac{1}{2}G^{\prime}(y)^{2}=\beta\log(|G|)+\gamma$. A sufficient
condition for this to hold is to have $G>0$ and $G$ satisfies

$\frac{G^{\prime}(y)}{\sqrt{\log(G)+\gamma}}=\zeta$

for some constants $\zeta,\gamma$. Integrating the left-hand side yields

$\int^{y}\frac{G^{\prime}(u)}{\sqrt{\log(G(u))+\gamma}}du=\int^{G(y)}\frac{1}{\sqrt{\log(u)+\gamma}}du=\alpha\,\textrm{Erfi}(\sqrt{\log(G(y))+\gamma})+\beta.$

where Erfi is the imaginary error function101010Although the name might be
misleading, the imaginary error function is real when the input is real. given
by

$\textup{Erfi}(z)=\frac{2}{\sqrt{\pi}}\int_{0}^{z}e^{t^{2}}dt.$

To alleviate the notation, we denote $h:=\textrm{Erfi}$ in the rest of this
section. From the above, $G$ should have the form

$G(y)=\exp\left(\zeta+\left(h^{-1}(\alpha y+\beta)\right)^{2}\right),$

where $\alpha,\beta,\zeta$ are all constants, and $h^{-1}$ is the inverse
function of the imaginary error function. We conclude that the activation
function $\phi$ should have the form

$\phi(y)=\frac{2\sigma}{\alpha^{2}\pi}\exp(-\zeta+h^{-1}(\alpha
y+\beta)^{2}).$

In this case, the coefficients $a$ and $b$ are given by

$b=0,a=\frac{\sigma^{2}}{\alpha^{2}\pi}\exp(-2\zeta).$

Letting $g=G^{\prime}$, the process $g(X_{t})$ has the following dynamics

$dg(X_{t})=-ag(X_{t})dt+\sigma dB_{t},$

Hence $g(X_{t})$ is an OU process, and we can conclude that the network output
in the infinite-depth limit $X_{1}$ satisfies

$g(X_{1})\sim\mathcal{N}\left(g(X_{0})e^{-a},\frac{\sigma^{2}}{2a}\left(1-e^{-2a}\right)\right).$

We can then infer the distribution of $X_{1}$ by a simple change of variable.
Note that this distribution is non-trivial, and unlike the infinite-width
limit of the same ResNet ([4]) where the distribution is Gaussian, here the
distribution of the pre-activations is directly impacted by the choice of the
activation function $\phi$.

However, with this particular choice of the activation function $\phi$, the
existence of the process $X$ can only be proven in the local sense, because
$\phi$ is only locally Lipschitz. Let us first show this in the next lemma. We
will see how we can mitigate this issue later.

###### Lemma 13

Let $\phi:\mathbb{R}\to\mathbb{R}$ defined by

$\phi(y)=\exp(h^{-1}(\alpha y+\beta)^{2}),$

where $\alpha,\beta\in\mathbb{R}$ are two constants.

We have that $\phi$ is locally Lipschitz, meaning that for any compact set
$K\subset\mathbb{R}$, there exists $C_{K}$ such that

$\forall x,x^{\prime}\in K,|\phi(x^{\prime})-\phi(x)|\leq
C_{K}|x^{\prime}-x|.$

Proof  It suffices to show that the derivative of $\phi$ is locally bounded to
conclude. We have that

$\displaystyle\phi^{\prime}(y)$ $\displaystyle=2\alpha(h^{-1})^{\prime}(\alpha
y+\beta)h^{-1}(\alpha y+\beta)\exp(h^{-1}(\alpha y+\beta)^{2})$
$\displaystyle=\alpha\sqrt{\pi}\,h^{-1}(\alpha y+\beta).$

Since $h^{-1}$ is continuous on $\mathbb{R}$, then $\phi^{\prime}$ is bounded
on any compact set of $\mathbb{R}$, which concludes the proof.

Now we can rigorously prove the following result.

###### Proposition 6

Let $x\in\mathbb{R}$ such that $x\neq 0$. Consider the following activation
function $\phi$

$\phi(y)=\frac{2\sigma}{\alpha^{2}\pi}\exp(-\zeta+h^{-1}(\alpha
y+\beta)^{2}),$

where $\alpha,\beta\in\mathbb{R}$ and $\sigma,\zeta>0$ are constants. Let $g$
be the function defined by

$g(y)=\alpha\sqrt{\pi}\exp(\zeta)h^{-1}(\alpha y+\beta).$

Consider the stochastic process $X_{t}$ defined by

$dX_{t}=|\phi(X_{t})|dB_{t},\quad X_{0}=W_{in}x.$

Then, we have that for all $t\in[0,1]$,

$g(X_{t})\sim\mathcal{N}\left(g(X_{0})e^{-at},\frac{\sigma^{2}}{2a}(1-e^{-2at})\right),$

where $a=\frac{\sigma^{2}}{\alpha^{2}\pi}\exp(-2\zeta)$.

Proof  For $N>0$, consider the stopping time $\tau_{N}$ defined by

$\tau_{N}=\inf\\{t\geq 0:|X_{t}|\geq N\\}.$

Using the continuity of paths of $X$, it is straightforward that
$\lim_{N\to\infty}\tau_{N}=\infty$ almost surely. Let $N>0$ be large enough.
The SDE satisfied by the process $X$ has a unique strong solution for
$t\in[0,\tau_{N})$ since the activation function $\phi$ is Lipschitz on the
interval $(-N,N)$. By applying It$\hat{o}$ lemma for $t\in(0,\tau_{N})$, we
have that

$dg(X_{t})=-ag(X_{t})dt+\sigma dB_{t},$

(from previous results). Using the fact that
$\lim_{N\to\infty}\tau_{N}=\infty$ almost surely, and taking $N$ large enough,
we obtain that for all $t\in(0,1]$, we have that

$dg(X_{t})=-ag(X_{t})dt+\sigma dB_{t},$

we conclude using Lemma 12.

###  D The Geometric Brownian Motion (GBM)

The GBM dynamics refers to stochastic differential equations of the form

$dX_{t}=aX_{t}dt+\sigma X_{t}dB_{t},$ (11)

where $a,\sigma$ are constants and $B$ is a one dimensional Brownian motion.
This SDE played a crucial role in financial mathematics and is often used as a
model of stock prices. It admits a closed-form solution given in the next
lemma.

###### Lemma 14

Eq. 11 admits the following solution

$X_{t}=X_{0}\exp\left(\left(a-\frac{1}{2}\sigma^{2}\right)t+\sigma
B_{t}\right).$

The distribution of $X_{t}$ is known as a log-Gaussian distribution. Moreover,
the solution is unique.

Proof  The existence and uniqueness of the solution follows from 4. Indeed, it
suffices to have the drift and the volatility both Lipschitz to obtain the
result. This is satisfied in the case of GBM. Now consider the process
$Z_{t}=\log(X_{t})$. Using It$\hat{o}$ lemma111111Notice that here, $X_{t}$
should be positive in order to consider $\log(X_{t})$. This is easy to show
and the proof is similar to that of Lemma 15., it is easy to verify that

$dZ_{t}=\left(a-\frac{1}{2}\sigma^{2}\right)dt+\sigma dB_{t},$

we conclude by integrating both sides.

Now let us find sufficient conditions under which the infinite-depth network
represented by the process $X$ has a GBM behaviour. In order for this to hold,
it suffices to have

$\begin{cases}\mu(y)=\frac{1}{2n}\phi(y)^{2}g^{\prime\prime}(y)=ag(y)\\\
\sigma(y)=\frac{1}{\sqrt{n}}\phi(y)g^{\prime}(y)=\sigma g(y).\\\ \end{cases}$

This implies $\frac{g^{\prime\prime}}{g^{\prime 2}}\propto\frac{1}{g}$, or
equivalently $\frac{g^{\prime\prime}}{g^{\prime}}\propto\frac{g^{\prime}}{g}$,
which in turn yields $\log(|g^{\prime}|)=\alpha\log(|g|)+\beta$, and therefore
$|g^{\prime}|\propto|g|^{\zeta}$. Assuming that $g^{\prime},g>0$, we can
easily verify that functions of the form $g(y)=\alpha(y+\beta)^{\gamma}$ where
$\alpha,\beta,\gamma>0$ satisfy the requirements. Hence, the activation
function should satisfy $\phi(y)=\sigma\gamma^{-1}(y+\beta)$, i.e. the
activation should be linear. In this case, we have
$a=\frac{1}{2}\sigma^{2}\gamma^{-1}(\gamma-1)$ and the process $g(X_{t})$ has
the following GBM dynamics

$dg(X_{t})=ag(X_{t})dt+\sigma g(X_{t})dB_{t}.$

From Lemma 14, we conclude that

$g(X_{1})\sim g(X_{0})\exp\left(\left(a-\frac{1}{2}\sigma^{2}\right)t+\sigma
B_{1}\right).$

Observe that in the special case of $\gamma=1,\beta=0,\alpha=1$, we have
$g(y)=y$ and $a=0$. In this case, we obtain $Y_{1}\sim
Y_{0}\exp\left(-\frac{1}{2}\sigma^{2}t+\sigma B_{1}\right)$.

We summarize the previous results in following proposition.

###### Proposition 7

Let $x\in\mathbb{R}$ such that $x\neq 0$. Consider the following activation
function $\phi$

$\phi(y)=\alpha y+\beta,$

where $\alpha>0,\beta\in\mathbb{R}$ are constants. Let $\sigma>0$ and define
the function $g$ by

$g(y)=(\alpha y+\beta)^{\gamma}.$

where $\gamma=\sigma\alpha^{-1}$. Consider the stochastic process $X_{t}$
defined by

$dX_{t}=|\phi(X_{t})|dB_{t},\quad X_{0}=W_{in}x.$

Then, the process $g(X_{t})$ satisfies the following GBM dynamics

$dg(X_{t})=ag(X_{t})dt+\sigma g(X_{t})dB_{t},$

where $a=\frac{1}{2}\sigma^{2}\gamma^{-1}(\gamma-1)$. As a result, we have
that for all $t\in[0,1]$,

$g(X_{t})\sim g(X_{0})\exp\left(\left(a-\frac{1}{2}\sigma^{2}\right)t+\sigma
B_{t}\right).$

###  E ReLU in the case $n=d=1$

Consider the process $X$ given by the SDE

$dX_{t}=\phi(Y_{t})dB_{t},\quad t\in[0,1],X_{0}>0.$

where $\phi(z)=\max(z,0)$ for $z\in\mathbb{R}$ is the ReLU activation
function. Note that we assume $X_{0}>0$ in this case. We will deal with the
general case later in this section.

It is straightforward that if $X_{s}\leq 0$ for some $s\in[0,1]$, then for all
$t\geq s,X_{t}=X_{s}$. This is because $dX_{t}=0\times dB_{t}$ whenever
$X_{t}\leq 0$. A rigorous justification is provided in Lemma 1. Hence, the
event $\\{X_{s}\leq 0\\}$ constitutes a stopping event where the process
becomes constant. We also say that $0$ is an absorbent point of the process
$X$. A classic tool in stochastic calculus to deal with such situations is the
notion of _stopping time_ which is a random variable that depend on the
trajectory of $X$ (or equivalently on the natural filtration $\mathcal{F}_{t}$
associated with the Brownian motion $B$). Consider the following stopping time

$\tau=\inf\\{t\in[0,1],\textup{ s.t. }X_{s}\leq 0\\}.$ (12)

Observe that we have for all $t\in[0,\tau]$

$dX_{t}=X_{t}dB_{t},$

which implies that $Y_{t}$ is a Geometric Brownian motion in the interval
$[0,\tau]$. Hence, if $\tau>1$ (a.s.), the network output has also a log-
normal distribution in the infinite-depth limit. In the next lemma, we show
that $\tau=\infty$ with probability $1$ which confirms the above.

###### Lemma 15

Let $\tau$ be the stopping time defined by Eq. 12. We have that

$\mathbb{P}(\tau=\infty)=1.$

Proof  By continuity of the Brownian path and the ReLU function $\phi$, the
paths of the process $X$ are also continuous121212This is a classic result in
stochastic calculus. More rigorously, $X$ can be chosen to have continuous
paths with probability 1.. we have that $\tau>0$ almost surely. From the
observation above, taking the limit $t\to\tau^{-}$ and using the continuity,
we obtain

$X_{\tau}=X_{0}\exp\left(-\frac{1}{2}\tau+B_{\tau}\right).$

For some $\omega\in\\{\tau<\infty\\}$, we have that $X_{\tau}(\omega)=0$ (by
continuity). Hence $-\frac{1}{2}\tau(w)+X_{\tau}(\omega)=-\infty$. This
happens with probability zero, which means that the event $\\{\tau<\infty\\}$
has probability zero. This concludes the proof.

Hence, with the ReLU activation function, given $X_{0}>0$, the network output
is distributed as

$X_{1}\sim X_{0}\exp\left(-\frac{1}{2}+B_{1}\right).$

Now let us go back to the original setup for $X_{0}$. Recall that
$X_{0}=W_{in}x$ for some $x\neq 0$ and $W_{in}\sim\mathcal{N}(0,1)$. By
conditioning on $X_{0}$ and observing that $0$ is an absorbent point of the
process $X$, we obtain that

$X_{1}\sim\mathbbm{1}_{\\{X_{0}>0\\}}X_{0}\exp\left(-\frac{1}{2}+B_{1}\right)+\mathbbm{1}_{\\{X_{0}\leq
0\\}}X_{0}.$

We summarize these results in the next proposition.

###### Proposition 8

Let $x\in\mathbb{R}$ such that $x\neq 0$, and let $\phi$ be the ReLU
activation function given by $\phi(z)=\max(z,0)$ for all $z\in\mathbb{R}$.
Consider the stochastic process $X_{t}$ defined by

$dX_{t}=\phi(X_{t})dB_{t},\quad X_{0}=W_{in}x.$

Then, the process $X$ is a mixture of a Geometric Brownian motion and a
constant process. More precisely, we have for all $t\in[0,1]$

$X_{t}\sim\mathbbm{1}_{\\{X_{0}>0\\}}\,X_{0}\exp\left(-\frac{1}{2}t+B_{t}\right)+\mathbbm{1}_{\\{X_{0}\leq
0\\}}X_{0}.$

Hence, conditionally on $X_{0}>0$, the process $X$ is a Geometric Bronwian
motion.

###  F Proof of Lemma 2 and Lemma 3

Lemma 2. _Let $x\in\mathbb{R}^{d}$ such that $x\neq 0$, and consider the
stochastic process $X$ given by the following SDE_

$dX_{t}=\frac{1}{\sqrt{n}}\|\phi(X_{t})\|dB_{t}\,,\quad t\in[0,\infty),\quad
X_{0}=W_{in}x,$

_where $\phi(z):\mathbb{R}\to\mathbb{R}$ is Lipschitz, injective,
$\mathcal{C}^{2}(\mathbb{R})$ and satisfies $\phi(0)=0$, and $\phi^{\prime}$
and $\phi^{\prime\prime}\phi$ are bounded on $\mathbb{R}$, and $(B_{t})_{t\geq
0}$ is an $n$-dimensional Brownian motion independent from
$W_{in}\sim\mathcal{N}(0,d^{-1}I)$. Let $\tau$ be the stopping time given by_

$\tau=\min\\{t\geq 0:\phi(X_{t})\ =0\\}.$

_Then, we have that_

$\mathbb{P}\left(\tau=\infty\right)=1.$

Proof  It is straightforward that with probability $1$ we have
$\|\phi(X_{0})\|>0$, which implies that with probability $1$, $\tau>0$. Let
$t<\tau$. Using It$\hat{o}$ ’s lemma with the function
$g(z)=\frac{1}{2}\log(\|\zeta(x)\|^{2})$, we obtain

$dg(X_{t})=\nabla
g(X_{t})^{\top}dX_{t}+\frac{1}{2n}\|\zeta(X_{t})\|^{2}\textup{Tr}(\nabla^{2}g(X_{t}))dt.$

Therefore,

$g(X_{t})-g(X_{0})=\frac{1}{\sqrt{n}}\int_{0}^{t}\mu(X_{s})ds+\frac{1}{2n}\int_{0}^{t}\sum_{i=1}^{n}\sigma_{i}(X_{s})dB^{i}_{s},$

where
$\sigma_{i}(X_{s})=\frac{|\phi^{\prime}(X^{i}_{s})\phi(X^{i}_{s})|}{\|\phi(X_{s})\|}$,
and
$\mu(X_{s})=\frac{1}{2}\sum_{i=1}^{n}\left(\phi^{\prime\prime}(X^{i}_{s})\phi(X^{i}_{s})+\phi^{\prime}(X^{i}_{s})^{2}\right)-\frac{\|\phi^{\prime}(X_{s})\circ\phi(X_{s})\|^{2}}{\|\phi(X_{s})\|^{2}}$,
and $\circ$ refers to the Hadamard product of vectors, i.e. coordinate-wise
product.

For some $\omega\in\\{\tau<\infty\\}$, using the path continuity of the
process $X$ and the continuity of $g$, we have that
$\lim_{t\to\tau(\omega)^{-}}g(X_{\tau(\omega)}(\omega))=-\infty$. Therefore,
we should also have

$\frac{1}{\sqrt{n}}\int_{0}^{\tau(\omega)}\mu(X_{s}(\omega))ds+\frac{1}{2n}\int_{0}^{\tau(\omega)}\sum_{i=1}^{n}\sigma_{i}(X_{s}(\omega))dB^{i}_{s}(\omega)=-\infty.$

Hence, we have that

$\displaystyle\mathbb{P}\left(\tau<\infty\right)$
$\displaystyle\leq\mathbb{P}\left(\frac{1}{\sqrt{n}}\int_{0}^{t}\mu(X_{s})ds+\frac{1}{2n}\int_{0}^{t}\sum_{i=1}^{n}\sigma_{i}(X_{s})dB^{i}_{s}=-\infty\right)$
$\displaystyle=\lim_{A\to\infty}\mathbb{P}\left(\frac{1}{\sqrt{n}}\int_{0}^{t}\mu(X_{s})ds+\frac{1}{2n}\int_{0}^{t}\sum_{i=1}^{n}\sigma_{i}(X_{s})dB^{i}_{s}\leq-A\right)$
$\displaystyle=\lim_{A\to\infty}\mathbb{P}\left(\frac{1}{\sqrt{n}}\int_{0}^{t}\mu(X_{s})ds+\frac{1}{2n}\int_{0}^{t}\sigma(X_{s})d\hat{B}_{s}\leq-A\right),$

where $\hat{B}$ is a one-dimensional Brownian motion, and where we use 7, and
$\sigma(X_{s})=(\sum_{i=1}^{n}\sigma_{i}(X_{s})^{2})^{1/2}=\frac{\|\phi^{\prime}(X_{s})\circ\phi(X_{s})\|}{\|\phi(X_{s})\|}$.
Using the conditions on $\phi$, there exists a constant $K>0$ such
$|\phi^{\prime}|\leq K,|\phi^{\prime\prime}\phi|\leq K$. With this we obtain
for all $Z\in\mathbb{R}^{n}$

$|\sigma(Z)|=\frac{\|\phi^{\prime}(Z)\circ\phi(Z)\|}{\|\phi(Z)\|}\leq K,$

and

$|\mu(Z)|=\left|\frac{1}{2}\sum_{i=1}^{n}\left(\phi^{\prime\prime}(Z^{i})\phi(Z^{i})+\phi^{\prime}(Z^{i})^{2}\right)-\frac{\|\phi^{\prime}(Z)\circ\phi(Z)\|^{2}}{\|\phi(Z)\|^{2}}\right|\leq\frac{1}{2}nK+\left(\frac{1}{2}n+1\right)K^{2}.$

Hence, the random variable
$\frac{1}{\sqrt{n}}\int_{0}^{t}\mu(X_{s})ds+\frac{1}{2n}\int_{0}^{t}\sigma(X_{s})d\hat{B}_{s}$
is finite with probability $1$. We conclude that

$\mathbb{P}\left(\tau=\infty\right)=1.$

Lemma 3. _Consider the stochastic process ( 7) given by the SDE_

$dX_{t}=\frac{1}{\sqrt{n}}\|\phi(X_{t})\|dB_{t}\,,\quad t\in[0,\infty),\quad
X_{0}=W_{in}x,$

_where $\phi$ is the ReLU activation function, and $(B_{t})_{t\geq 0}$ is an
$n$-dimensional Brownian motion. Let $\tau$ be the stopping time given by_

$\tau=\min\\{t\geq 0:\|\phi(X_{t})\|=0\\}=\min\\{t\geq 0:\,\forall
i\in[n],\,X^{i}_{t}\leq 0\\}.$

_Then, we have that_

$\mathbb{P}\left(\tau=\infty\,\huge|\,\|\phi(X_{0})\|>0\right)=1.$

_As a result, we have that_

$\mathbb{P}(\tau=\infty)=1-2^{-n}.$

Proof  Let $t_{0}>0$. Using Lemma 7, we know that if for some $t_{1}$,
$\|\phi(X_{t_{1}})\|=0$, then for all $t\geq t_{1}$, we have that
$X_{t}=X_{t_{1}}$ and $\|\phi(X_{t})\|=0$. Hence, we have that

$\mathbb{P}\left(\tau\leq
t_{0}\,\huge|\,\|\phi(X_{0})\|>0\right)=\mathbb{P}\left(\|\phi(X_{t_{0}})\|=0\,\huge|\,\|\phi(X_{0})\|>0\right).$

Let $m\geq 1$ and consider the function $\phi_{m}(z)=\int_{0}^{z}h(m\,u)du$
and $h(t)=(1+e^{-t})^{-1}$ is the Sigmoid function131313Note that $\phi_{m}$
has a closed-form formula given by
$\phi_{m}(z)=m^{-1}(\log(1+e^{mz})-\log(2))$, which can be seen as a shifted
and scaled version of the Softplus function. However, we do not need the
closed-form formula in our analysis.. It is straightforward that $\phi_{m}$
satisfies the conditions of Lemma 2. Let $X^{m}$ be the solution of the
following SDE (the solution exists and is unique since $\phi_{m}$ is trivially
Lipschitz)

$dX^{m}_{t}=\frac{1}{\sqrt{n}}\|\phi_{m}(X^{m}_{t})\|dB_{t}\,,\quad
t\in[0,\infty),\quad X_{0}=W_{in}x.$

We know from Lemma 8 that $X^{m}_{t}$ converges in $L^{2}$ to $X_{t}$
(uniformly over $t\in[0,T]$ for any $T>0$). In particular, this implies
convergence in distribution. Moreover, observe that for all $t$

$\mathbb{E}\|\phi_{m}(X^{m}_{t})-\phi(X_{t})\|^{2}\leq\frac{2n}{m^{2}}+2\mathbb{E}\|X^{m}_{t}-X_{t}\|^{2},$

where we used triangular inequality and the upperbound from Lemma 9. Thus, we
have that $\phi_{m}(X^{m}_{t})$ converges in $L^{2}$ (and in distribution) to
$\phi(X_{t})$.

Let $\delta_{k}=[1/(k+1),1/{k})$ for $k\geq 1$, and define
$\delta_{0}=[1,\infty)$. For $m\geq 1$, using Lemma 9, we have that

$\mathbb{P}\left(\|\phi(X_{t_{0}})\|=0\,\huge\cap\,\|\phi(X_{0})\|>0\right)\leq\sum_{k=0}^{\infty}\mathbb{P}\left(\|\phi_{m}(X_{t_{0}})\|\leq
1/m\,\huge\cap\,\|\phi(X_{0})\|\in\delta_{k}\right).\\\ $

Given $k\geq 0$, we have that for $m>n^{1/2}(k+1)$,

$\displaystyle\mathbb{P}\left(\|\phi_{m}(X_{t_{0}})\|\leq
1/m\,\huge\cap\,\|\phi(X_{0})\|\in\delta_{k}\right)$
$\displaystyle\leq\mathbb{P}\left(\|\phi_{m}(X^{m}_{t_{0}})\|\leq
1/m+\log(m)/m\,\huge\cap\,\|\phi(X_{0})\|\in\delta_{k}\right)$ (13)
$\displaystyle+\mathbb{P}\left(\|\phi_{m}(X^{m}_{t_{0}})-\phi(X_{t_{0}})\|>\log(m)/m\,\huge\cap\,\|\phi(X_{0})\|\in\delta_{k}\right)$

Let us deal with the first term. Using Lemma 9, we have that

$\displaystyle\mathbb{P}\left(\|\phi_{m}(X^{m}_{t_{0}})\|\leq
1/m+\log(m)/m\,\huge\cap\,\|\phi(X_{0})\|\in\delta_{k}\right)$
$\displaystyle\leq\mathbb{P}\left(\|\phi_{m}(X^{m}_{t_{0}})\|\leq
1/m+\log(m)/m\,\huge\cap\,\|\phi_{m}(X^{m}_{0})\|\geq
1/(k+1)-n^{1/2}m^{-1}\huge\cap\,\|\phi(X_{0})\|\in\delta_{k}\right)$
$\displaystyle\leq\mathbb{P}\left(\log\left(\frac{\|\phi_{m}(X^{m}_{t_{0}})\|}{\|\phi_{m}(X^{m}_{0})\|}\right)\leq-\log(m/(1+\log(m))+\log((k+1)^{-1}-m^{-1}n^{1/2})\,\huge\cap\,\|\phi(X_{0})\|\in\delta_{k}\right)$

From Lemma 10, we know that the random variable
$\log\left(\frac{\|\phi_{m}(X^{m}_{t_{0}})\|}{\|\phi_{m}(X^{m}_{0})\|}\right)$
converges in $L^{1}$ and thus it is bounded in $L^{1}$ norm (over $m$).
Therefore, a simple application of Markov’s inequality yields that the
probability above goes to $0$ when $m$ goes to $\infty$.

The second term in Eq. 13 also converges to $0$ using the $L^{2}$ convergence
of $\phi_{m}(X^{m}_{t_{0}})$ to $\phi(X_{t_{0}})$ coupled with a simple
application of Markov’s inequality. We therefore obtain that for all $k\geq
0$, $\lim_{m\to\infty}\mathbb{P}\left(\|\phi_{m}(X_{t_{0}})\|\leq
1/m\,\huge\cap\,\|\phi(X_{0})\|\in\delta_{k}\right)=0$. Using the Dominated
convergence theorem, we obtain that for all $t_{0}$,

$\mathbb{P}\left(\|\phi(X_{t_{0}})\|=0\,\huge\cap\,\|\phi(X_{0})\|>0\right)=0,$

which implies that
$\mathbb{E}\left[\mathbbm{1}_{\\{\tau>t\\}}\huge|\|\phi(X_{0})\|>0\right]=1$
for all $t>0$. Another application of the Dominated convergence theorem yields
the result. The second part is straightforward by observing that
$\mathbb{P}\left(\|\phi(X_{0})\|=0\right)=2^{-n}$.

###  G Proof of 1

Theorem 1.  _We have that for all $t\in[0,1]$,_

$\|\phi(X_{t})\|=\|\phi(X_{0})\|\exp\left(\frac{1}{\sqrt{n}}\hat{B}_{t}+\frac{1}{n}\int_{0}^{t}\mu_{s}ds\right),\quad\textrm{almost
surely,}$

_where $\mu_{s}=\frac{1}{2}\|\phi^{\prime}(X_{s})\|^{2}-1$, and
$(\hat{B})_{t\geq 0}$ is a one-dimensional Brownian motion. As a result, we
have that for all $0\leq s\leq t\leq 1$_

$\mathbb{E}\left[\log\left(\frac{\|\phi(X_{t})\|}{\|\phi(X_{s})\|}\right)\huge|\,\|\phi(X_{0})\|>0\right]=\left(\frac{1-2^{-n}}{4}-\frac{1}{n}\right)(t-s),$

_Moreover, for $n\geq 2$, we have_

$\textup{Var}\left[\log\left(\frac{\|\phi(X_{t})\|}{\|\phi(X_{s})\|}\right)\huge|\,\|\phi(X_{0})\|>0\right]\leq\left(n^{-1/2}+\Gamma_{s,t}^{1/2}\right)^{2}(t-s),$

_where
$\Gamma_{s,t}=\frac{1}{4}\int_{s}^{t}\,\left(\left(\mathbb{E}\phi^{\prime}(X^{1}_{u})\phi^{\prime}(X^{2}_{u})-\frac{(1-2^{-n})^{2}}{4}\right)+n^{-1}\left(\frac{1-2^{-n}}{2}-\mathbb{E}\phi^{\prime}(X^{1}_{u})\phi^{\prime}(X^{2}_{u})\right)\right)du$.
_

Proof  Let $t\in[0,1]$. Let us firs consider the case where
$\|\phi(X_{0})\|=0$. For all $t$ we have $\|\phi(X_{t})\|=0$ and the result is
trivial.

We consider the case where $\|\phi(X_{0})\|>0$ (happens with probability
$1-2^{-n}$), and all the expectations in this proof are conditionally on this
event. Consider the function $g:\mathbb{R}^{n}\to\mathbb{R}$ given by

$g(x)=\log(\|\phi(x)\|)=\frac{1}{2}\log(\|\phi(x)\|^{2}).$

Ideally, we would like to use It$\hat{o}$ ’s lemma and Lemma 3, which ensures
that $\|\phi(X_{t})\|$ remains positive on $[0,1]$, and obtain for all
$t\in[0,1]$

$dg(X_{t})\overset{dist}{=}\frac{1}{\sqrt{n}}d\hat{B}_{t}+\frac{1}{n}\mu_{t}dt,$

where $\mu_{t}$ is some well defined quantity. This would let us conclude.
However, It$\hat{o}$ ’s lemma requires that the function be
$\mathcal{C}^{2}(\mathbb{R}^{n})$, which is violated by our choice of $g$. To
mitigate this issue, we consider a sequence of function $(g_{m})_{m\geq m}$
that approximates the function $g$ when $m$ goes to infinity. For $m\geq 1$,
let $g_{m}$ be defined by

$g_{m}(x)=\frac{1}{2}\log(\|\phi_{m}(x)\|^{2}),$

where $\phi_{m}(t)=\int_{0}^{t}h(m\,u)du$ and $h(t)=(1+e^{-t})^{-1}$ is the
Sigmoid function. We have that

$\begin{cases}\frac{\partial g_{m}}{\partial
x_{i}}(x)=\frac{h(mx_{i})\phi_{m}(x_{i})}{\|\phi_{m}(x)\|^{2}}\\\
\frac{\partial^{2}g_{m}}{\partial
x_{i}^{2}}(x)=\frac{mh(mx_{i})(1-h(mx_{i}))\phi_{m}(x_{i})+h(mx_{i})^{2}}{\|\phi_{m}(x)\|^{2}}-2\frac{h(mx_{i})^{2}\phi_{m}(x_{i})^{2}}{\|\phi_{m}(x)\|^{4}}\end{cases}$

Let $X^{m}$ be the solution of the following SDE

$dX^{m}_{t}=\frac{1}{\sqrt{n}}\|\phi_{m}(X^{m}_{t})\|dB_{t}\,,\quad
t\in[0,\infty),\quad X_{0}=W_{in}x,$

Using It$\hat{o}$ ’s lemma, we have that

$dg_{m}(X^{m}_{t})=\frac{1}{n}\mu_{s}^{m}ds+\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\sigma_{s}^{m,i}dB_{s}^{i},$

where

$\mu^{m}_{s}=\frac{1}{2}\sum_{i=1}^{n}\left(mh(mX^{m,i}_{s})(1-h(mX^{m,i}_{s}))\phi_{m}(X^{m,i}_{s})+h(mX^{m,i}_{s})^{2}\right)-\frac{\|h(mX^{m}_{s})\circ\phi_{m}(X^{m}_{s})\|^{2}}{\|\phi_{m}(X^{m}_{s})\|^{2}},$

and
$\sigma^{m,i}_{s}=\frac{|h(mX^{m,i}_{s})\circ\phi_{m}(X^{m,i}_{s})|}{\|\phi_{m}(X^{m}_{s})\|}$.
By Lemma 10, we know that $g_{m}(X^{m}_{s})-g_{m}(X^{m}_{0})$ converges in
$L_{1}$ to $g(X_{s})-g(X_{0})$. Let us now compute the limit of
$g_{m}(X^{m}_{s})-g_{m}(X^{m}_{0})$ from the equation above to conclude. More
precisely, let us show that for all $s\in(0,1)$

$\lim_{m\to\infty}\mathbb{E}\left|g_{m}(X^{m}_{s})-g_{m}(X^{m}_{0})-(Z_{s}-Z_{0})\right|=0,$

where $Z$ is the process given by

$Z_{t}=g(X_{0})+\frac{1}{\sqrt{n}}\left(\sum_{i=1}^{n}\int_{0}^{t}\sigma^{i}_{s}dB^{i}_{t}\right)+\frac{1}{n}\int_{0}^{t}\mu_{s}ds,$

where $\mu_{s}=\frac{1}{2}\|\phi^{\prime}(X^{i}_{s})\|^{2}-1$, and
$\sigma_{s}^{i}=\frac{\phi(X_{s}^{i})}{\|\phi(X_{s})\|}$.

Let $t\in(0,1]$. Using triangular inequality, It$\hat{o}$ isometry, and
Cauchy-Schwartz inequality, we have that

$\displaystyle\mathbb{E}\left|g_{m}(X^{m}_{t})-g_{m}(X^{m}_{0})-(Z_{t}-Z_{0})\right|$
$\displaystyle\leq\frac{1}{n}\int_{0}^{t}\mathbb{E}\left|\mu^{m}_{s}-\mu_{s}\right|ds+\frac{1}{\sqrt{n}}\mathbb{E}\left|\int_{0}^{t}\sum_{i=1}^{n}(\sigma_{s}^{i}-\sigma^{m,i}_{s})dB^{i}_{s}\right|$
(14)
$\displaystyle\leq\frac{1}{n}\int_{0}^{t}\mathbb{E}\left|\mu^{m}_{s}-\mu_{s}\right|ds+\frac{1}{\sqrt{n}}\left(\mathbb{E}\int_{0}^{t}\sum_{i=1}^{n}(\sigma_{s}^{i}-\sigma^{m,i}_{s})^{2}ds\right)^{1/2}$

We first deal with the the term
$\int_{0}^{t}\mathbb{E}\left|\mu^{m}_{s}-\mu_{s}\right|ds$. Let us show that
this term converges to $0$. Let us show the following,

$\forall s>0,\lim_{m\to\infty}\mathbb{E}\,|\mu^{m}_{s}-\mu_{s}|=0$.

Let $s\in[0,1]$. We have that

$\mu^{m}_{s}=\frac{1}{2}\sum_{i=1}^{n}J^{m}_{i}-G^{m},$

where
$J^{m}_{i}=mh(mX^{m,i}_{s})(1-h(mX^{m,i}_{s}))\phi_{m}(X^{m,i}_{s})+h(mX^{m,i}_{s})^{2}$,
and
$G^{m}=\frac{\|h(mX^{m}_{s})\circ\phi_{m}(X^{m}_{s})\|^{2}}{\|\phi_{m}(X^{m}_{s})\|^{2}}$.
Let us start with the term $G^{m}$. Observe that $G^{m}\leq 1$ almost surely.
We have that

$\displaystyle\mathbb{E}|1-G^{m}|$
$\displaystyle=\mathbb{E}\left[\frac{\|(1-h(mX^{m}_{s})^{2})^{1/2}\circ\phi_{m}(X^{m}_{s})\|^{2}}{\|\phi_{m}(X^{m}_{s})\|^{2}}\right]$
$\displaystyle=\mathbb{E}\left[\frac{\|(1-h(mX^{m}_{s})^{2})^{1/2}\circ\phi_{m}(X^{m}_{s})\|^{2}}{\|\phi_{m}(X^{m}_{s})\|^{2}}\mathbbm{1}_{\\{\min_{i}|X^{m,i}_{s}|\geq\log(m)/m\\}}\right]$
$\displaystyle+\mathbb{E}\left[\frac{\|(1-h(mX^{m}_{s})^{2})^{1/2}\circ\phi_{m}(X^{m}_{s})\|^{2}}{\|\phi_{m}(X^{m}_{s})\|^{2}}\mathbbm{1}_{\\{\min_{i}|X^{m,i}_{s}|<\log(m)/m\\}}\right]$

When $\min_{i}|x^{i}|\geq\log(m)/m$, we have that for all $i\in[n]$

$(1-h(mx^{i})^{2})\leq 2(1-h(mx^{i}))\leq 2\exp(-m\times\log(m)/m)=2m^{-1}.$

Therefore,

$\mathbb{E}\left[\frac{\|(1-h(mX^{m}_{s})^{2})^{1/2}\circ\phi_{m}(X^{m}_{s})\|^{2}}{\|\phi_{m}(X^{m}_{s})\|^{2}}\mathbbm{1}_{\\{\min_{i}|X^{m,i}_{s}|\geq\log(m)/m\\}}\right]\leq
2m^{-1}.$

For the remaining term, using the fact that $1-h^{2}\leq 1$, we have that

$\displaystyle\mathbb{E}\left[\frac{\|(1-h(mX^{m}_{s})^{2})^{1/2}\circ\phi_{m}(X^{m}_{s})\|^{2}}{\|\phi_{m}(X^{m}_{s})\|^{2}}\mathbbm{1}_{\\{\min_{i}|X^{m,i}_{s}|<\log(m)/m\\}}\right]$
$\displaystyle\leq\mathbb{P}\left(\min_{i}|X^{m,i}_{s}|<\log(m)/m\right)$
$\displaystyle\leq\sum_{i=1}^{n}\mathbb{P}\left(|X^{m,i}_{s}|<\log(m)/m\right).$

Now using Lemma 8, we have that

$\mathbb{P}(\|X^{m}_{s}-X_{s}\|\geq\log(m)/m)\leq\frac{C_{n}}{\log(m)^{2}},$

for some constant $C_{n}$ that depends on $n$. Therefore, for all $i$, we have

$\displaystyle\mathbb{P}(|X^{m,i}_{s}|<\log(m)/m)$
$\displaystyle\leq\mathbb{P}(|X^{i}_{s}|<2\log(m)/m)+\mathbb{P}(|X^{m,i}_{s}-X_{s}^{i}|\geq\log(m)/m)$
$\displaystyle\leq\mathbb{P}(|X^{i}_{s}|<2\log(m)/m)+\frac{C_{n}}{\log(m)^{2}}.$

Recall that
$X^{i}_{s}=X^{i}_{0}+\frac{1}{\sqrt{n}}\int_{0}^{s}\|\phi(X_{u})\|dB_{u}$. By
Lemma 11, we know that

$\displaystyle\mathbb{P}(|X^{i}_{s}|<2\log(m)/m)=\mathcal{O}(\log(m)/m),$

We conclude that $\lim_{m\to\infty}\mathbb{E}|1-G^{m}|=1.$

Now let us show that for all $i$,
$\lim_{m\to\infty}\mathbb{E}|J^{m}_{i}-\phi^{\prime}(X^{i}_{s})|=0$. Let
$i\in[n]$ and
$A^{m,i}_{s}=mh(mX^{m,i}_{s})(1-h(mX^{m,i}_{s}))\phi_{m}(X^{m,i}_{s})$. We
have that

$\displaystyle\mathbb{E}\,|A^{m,i}_{s}|$
$\displaystyle=\mathbb{E}\,|A^{m,i}_{s}|\mathbbm{1}_{\\{|X^{m,i}_{s}|\leq
2\log(m)/m\\}}+\mathbb{E}\,|A^{m,i}_{s}|\mathbbm{1}_{\\{|X^{m,i}_{s}|\leq
2\log(m)/m\\}}$ $\displaystyle\leq m\times
2\log(m)/m\times\mathbb{P}(|X^{m,i}_{s}|\leq
2\log(m)/m)+m^{-1}\mathbb{E}|\phi_{m}(X^{m,i}_{s})|$ $\displaystyle\leq
2\log(m)\times\left(\mathbb{P}(|X^{i}_{s}|\leq
3\log(m)/m)+\mathbb{P}(|X^{m,i}_{s}-X^{i}_{s}|\geq\log(m)/m)\right)+m^{-1}\mathbb{E}|\phi_{m}(X^{m,i}_{s})|$
$\displaystyle\leq 2\log(m)\times\mathbb{P}(|X^{i}_{s}|\leq
3\log(m)/m)+\frac{2C_{n}}{\log(m)}+m^{-1}\mathbb{E}|\phi_{m}(X^{m,i}_{s})|,$

where we have used Lemma 8 and Markov’s inequality. Using Lemma 9 and the fact
that the absolute value function is Lipschitz, we know that
$\lim_{m\to\infty}\mathbb{E}\,|\phi_{m}(X^{m,i}_{s})|=\mathbb{E}\,|\phi(X^{i}_{s})|<\infty$.
Therefore, the third term vanishes in the limit $m\to\infty$. The second term
$2C_{n}/\log(m)$ also vanishes. The first term also vanished using Lemma 11.
Therefore, $\lim_{m\to\infty}\mathbb{E}|A^{m,i}_{s}|=0$.

Let us now deal with the last term in $J^{m}_{i}$. We have that

$\displaystyle\mathbb{E}\,\left|h(mX^{m,i}_{s})^{2}-\phi^{\prime}(X^{i}_{s})\right|$
$\displaystyle=\mathbb{E}\,\left|h(mX^{m,i}_{s})^{2}-\phi^{\prime}(X^{i}_{s})\right|\mathbbm{1}_{\\{|X^{m,i}_{s}|\geq\log(m)/m\\}}$
$\displaystyle+\mathbb{E}\,\left|h(mX^{m,i}_{s})^{2}-\phi^{\prime}(X^{i}_{s})\right|\mathbbm{1}_{\\{|X^{m,i}_{s}|<\log(m)/m\\}}$
$\displaystyle\leq 3m^{-1}+2\mathbb{P}(|X^{m,i}_{s}|<\log(m)/m)$
$\displaystyle\leq
3m^{-1}+2\mathbb{P}(|X^{i}_{s}|<2\log(m)/m)+2\frac{C_{n}}{\log(m)^{2}}$

Using Lemma 11, we obtain that
$\lim_{m\to\infty}\mathbb{E}|h^{2}(mX^{m,i}_{s})-\phi^{\prime}(X^{i}_{s})|=0$.
Hence, we obtain that
$\lim_{m\to\infty}\mathbb{E}|J^{m}_{i}-\phi^{\prime}(X_{s})|=0$. We conclude
that $\lim_{m\to\infty}\mathbb{E}|\mu^{m}_{s}-\mu_{s}|=0$. Moreover, from the
analysis above, it is easy to see that $\sup_{m\geq
1,\,s\in(0,1]}\mathbb{E}|\mu^{m}_{s}-\mu_{s}|<\infty$.

We now deal with the second term
$\left(\mathbb{E}\int_{0}^{t}\sum_{i=1}^{n}(\sigma_{s}^{i}-\sigma^{m,i}_{s})^{2}ds\right)^{1/2}$
from Eq. 14. For this part only, we define the stopping time $\tau_{\epsilon}$
for $\epsilon\in(0,\|\phi(X_{0})\|\wedge\|\phi(X_{0})\|^{-1})$ (Recall that
the analysis is conducted conditionally on the fact that $\|\phi(X_{0})\|>0$)
by

$\tau_{\epsilon}=\inf\\{t\geq 0,\textrm{ s.t.
}\|\phi(X_{t})\|\in[0,\epsilon]\cup[\epsilon^{-1},\infty)\\}.$

Notice that $\tau_{\epsilon}>0$ almost surely since
$\|\phi(X_{0})\|\in(\epsilon,\epsilon^{-1})$.

Let $s\in(0,1]$. We have that

$\displaystyle\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\sum_{i=1}^{n}(\sigma_{s}^{i}-\sigma^{m,i}_{s})^{2}$
$\displaystyle\leq
2\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\sum_{i=1}^{n}\left(\frac{\phi(X^{i}_{s})}{\|\phi(X_{s})\|}-\frac{\phi_{m}(X^{m,i})}{\|\phi_{m}(X^{m}_{s})\|}\right)^{2}ds$
$\displaystyle+2\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\frac{\|(1-h(mX^{m}_{s})^{2})^{1/2}\circ\phi_{m}(X^{m}_{s}))\|}{\|\phi(X^{m}_{s})\|^{2}}ds.$

The second term can be upperbounded in the following fashion

$\displaystyle\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\frac{\|(1-h(mX^{m}_{s})^{2})^{1/2}\circ\phi_{m}(X^{m}_{s}))\|}{\|\phi(X^{m}_{s})\|^{2}}ds$
$\displaystyle\leq\int_{0}^{t}\mathbb{E}\frac{\|(1-h(mX^{m}_{s})^{2})^{1/2}\circ\phi_{m}(X^{m}_{s}))\|}{\|\phi(X^{m}_{s})\|^{2}}ds$
$\displaystyle=\int_{0}^{t}\mathbb{E}|1-G_{m}|,$

where $G_{m}$ is defined above. We know that $\int_{0}^{t}\mathbb{E}|1-G_{m}|$
converges to $0$ in the limit $m\to\infty$ by the Dominated convergence
theorem (the integrand is bounded). Let us show that the first term also
vanishes. We have that

$\displaystyle\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\sum_{i=1}^{n}\left(\frac{\phi(X^{i}_{s})}{\|\phi(X_{s})\|}-\frac{\phi_{m}(X^{m,i})}{\|\phi_{m}(X^{m}_{s})\|}\right)^{2}ds$
$\displaystyle\leq
2\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\sum_{i=1}^{n}\left(\frac{\phi(X^{i}_{s})-\phi_{m}(X^{m,i})}{\|\phi(X_{s})\|}\right)^{2}ds$
$\displaystyle+2\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\sum_{i=1}^{n}\phi_{m}(X^{m,i}_{s})^{2}\left(\frac{1}{\|\phi(X_{s})\|}-\frac{1}{\|\phi(X^{m}_{s})\|}\right)^{2}ds$
$\displaystyle\leq
2\epsilon^{-2}\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\|\phi(X^{i}_{s})-\phi_{m}(X^{m,i}_{s})\|^{2}ds$
$\displaystyle+2\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\sum_{i=1}^{n}\phi_{m}(X^{m,i}_{s})^{2}\left(\frac{1}{\|\phi(X_{s})\|}-\frac{1}{\|\phi(X^{m}_{s})\|}\right)^{2}ds.$

The first term
$2\epsilon^{-2}\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\|\phi(X^{i}_{s})-\phi_{m}(X^{m,i}_{s})\|^{2}ds$
converges to 0 in the limit $m\to\infty$ by Lemma 9 and Lemma 8. Let us deal
with the second term. Define the event
$E=\\{\sup_{s\in(0,1]}\|X^{m}_{s}-X_{s}\|\leq\log(m)/m\\}$ for $m$ large
enough such that $\log(m)/m<\epsilon$. Observe that on the event $E$, we have
that for all $s\in(0,1]$,
$\|\phi(X^{m}_{s})-\phi(X_{s})\|\leq(\sqrt{n}+\log(m))m^{-1}$. Hence,

$\displaystyle\mathbb{E}\,\mathbbm{1}_{E}\,\int_{0}^{t\wedge\tau_{\epsilon}}\sum_{i=1}^{n}\phi_{m}(X^{m,i}_{s})^{2}\left(\frac{1}{\|\phi(X_{s})\|}-\frac{1}{\|\phi(X^{m}_{s})\|}\right)^{2}ds\leq\epsilon^{-2}(\sqrt{n}+\log(m))^{2}m^{-2}\to_{m\to\infty}0.$

Moreover, letting $E^{c}$ be the complementary event of $E$, we have

$\displaystyle\mathbb{E}\,\mathbbm{1}_{E^{c}}\,\int_{0}^{t\wedge\tau_{\epsilon}}\sum_{i=1}^{n}\phi_{m}(X^{m,i}_{s})^{2}$
$\displaystyle\left(\frac{1}{\|\phi(X_{s})\|}-\frac{1}{\|\phi(X^{m}_{s})\|}\right)^{2}ds$
$\displaystyle\leq\mathbb{E}\,\mathbbm{1}_{E^{c}}\,\int_{0}^{t\wedge\tau_{\epsilon}}\sum_{i=1}^{n}\phi_{m}(X^{m,i}_{s})^{2}\left(\frac{2}{\|\phi(X_{s})\|^{2}}+\frac{2}{\|\phi(X^{m}_{s})\|^{2}}\right)ds$
$\displaystyle\leq
2\epsilon^{-2}\mathbb{E}\,\mathbbm{1}_{E^{c}}\,\int_{0}^{t\wedge\tau_{\epsilon}}\|\phi_{m}(X^{m}_{s})\|^{2}ds+2\mathbb{P}(E^{c}).$

Using the fact that
$\|\phi_{m}(X^{m}_{s})\|\leq\frac{\sqrt{n}}{m}+\|\phi(X_{s})\|+\|X^{m}_{s}-X_{s}\|$
(by Lemma 9 and the fact that ReLU is Lipschitz), we obtain

$\displaystyle\mathbb{E}\,\mathbbm{1}_{E^{c}}\,\int_{0}^{t\wedge\tau_{\epsilon}}\|\phi_{m}(X^{m}_{s})\|^{2}ds$
$\displaystyle\leq 3nm^{-2}+3\sup_{s\leq
1}\mathbb{E}\|X^{m}_{s}-X_{s}\|^{2}+3\epsilon^{-2}\mathbb{P}(E^{c}).$

The term $\sup_{s\leq 1}\mathbb{E}\|X^{m}_{s}-X_{s}\|^{2}$ converges to $0$ by
Lemma 8. Using Doob’s martingale inequality on the submartingale
$\|X^{m}_{s}-X_{s}\|$ (with respect to the natural filtration generated by the
Brownian motion $B$)141414The submartingale behaviour is a result of the
convexity of the norm function and the fact that $X^{m}_{s}X_{s}$ is a
martingale since
$d(X^{m}_{s}X_{s})=\frac{1}{\sqrt{n}}(\phi_{m}(X^{m}_{s})-\phi(X_{s}))dB_{s}$.
A simple application of Jensen’s inequality yields the result. we obtain that

$\mathbb{P}(E^{c})\leq\frac{\mathbb{E}\|X^{m}_{1}-X_{1}\|^{2}}{m^{-2}\log(m)^{2}}=\frac{C_{n}}{\log(m)^{2}},$

where we have used Lemma 8. We conclude that
$\lim_{m\to\infty}\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}\sum_{i=1}^{n}(\sigma_{s}^{i}-\sigma^{m,i}_{s})^{2}=0$.

By observing that
$\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}|\mu^{m}_{s}-\mu_{s}|ds\leq\mathbb{E}\int_{0}^{t}\mathbb{E}|\mu^{m}_{s}-\mu_{s}|ds$,
a simple application of the Dominated convergence theorem yields
$\lim_{m\to\infty}\mathbb{E}\int_{0}^{t\wedge\tau_{\epsilon}}|\mu^{m}_{s}-\mu_{s}|ds=0$.
Hence, we proved that

$\lim_{m\to\infty}\mathbb{E}\left|g_{m}(X^{m}_{t\wedge\tau_{\epsilon}})-g_{m}(X^{m}_{0})-(Z_{t\wedge\tau_{\epsilon}}-Z_{0})\right|=0.$

From Lemma 10, we know that
$g_{m}(X^{m}_{t\wedge\tau_{\epsilon}})-g_{m}(X^{m}_{0})$ converges in $L_{1}$
to $g(X_{t\wedge\tau_{\epsilon}})-g(X_{0})$151515the result of Lemma 10 holds