# Spiked matrix

Tengyuan Liang, Subhabrata Sen, Pragya Sur  Tengyuan Liang
<EMAIL_ADDRESS>University of Chicago  Subhabrata Sen
<EMAIL_ADDRESS>Harvard University  Pragya Sur
<EMAIL_ADDRESS>Harvard University

# High-dimensional Asymptotics of Langevin Dynamics in Spiked Matrix Models

Tengyuan Liang, Subhabrata Sen, Pragya Sur  Tengyuan Liang
<EMAIL_ADDRESS>University of Chicago  Subhabrata Sen
<EMAIL_ADDRESS>Harvard University  Pragya Sur
<EMAIL_ADDRESS>Harvard University

###### Abstract

We study Langevin dynamics for recovering the planted signal in the spiked
matrix model. We provide a “path-wise” characterization of the overlap between
the output of the Langevin algorithm and the planted signal. This overlap is
characterized in terms of a self-consistent system of integro-differential
equations, usually referred to as the Crisanti-Horner-Sommers-Cugliandolo-
Kurchan (CHSCK) equations in the spin glass literature. As a second
contribution, we derive an explicit formula for the limiting overlap in terms
of the signal-to-noise ratio and the injected noise in the diffusion. This
uncovers a sharp phase transition—in one regime, the limiting overlap is
strictly positive, while in the other, the injected noise overcomes the
signal, and the limiting overlap is zero.

Keywords— Langevin dynamics, high-dimensional asymptotics, spiked matrices,
spin-glasses

##  1 Introduction

Gradient descent based methods and their noisy counterparts are routinely used
in modern Machine Learning. For a host of learning problems, it has now been
established that gradient based methods converge to special estimators with
attractive generalization properties ([45], [17], [30], [29], [42], [37],
[34], [44], [2], [33], [49], [22]). Thus the limiting performance of gradient
descent and its variants can often be characterized via careful analyses of
these special limiting estimators (c.f. [41], [16], [26], [35], , [36] , [21]
and the references cited therein). However, an understanding of “path-wise”
properties of these algorithms still lies in its infancy. In this paper, we
consider Langevin dynamics as a proxy for stochastic gradient descent, and a
simple recovery problem with a non-convex objective function—that of
recovering a planted rank 1 matrix under additive Gaussian noise.

Formally, we observe a symmetric matrix $\mathbf{J}\in\mathbb{R}^{N\times N}$,
given by

$\displaystyle\mathbf{J}=VV^{\top}+\mathbf{Z},$ (1.1)

where $V\in\mathbb{R}^{N}$. We assume
$\mathbf{Z}=\mathbf{G}^{\top}\mathbf{D}\mathbf{G}$ where $\mathbf{D}$ is
diagonal and $\mathbf{G}$ is Haar distributed. Denote $\mathbf{D}={\rm
Diag}(\sigma^{1},\ldots,\sigma^{N})$ to be a sequence of deterministic
diagonal matrices. As $N$ increases, assume that
$\mu_{D_{N}}=\frac{1}{N}\sum_{i=1}^{N}\delta_{\sigma^{i}}$ converges weakly to
a probability measure $\mu_{D}$ with compact support. Let $d_{+},d_{-}$ denote
the upper and lower edges respectively of $\mu_{D}$. We assume throughout that
$\max_{1\leq i\leq N}\sigma^{i}$ and $\min_{1\leq i\leq N}\sigma^{i}$ converge
to $d_{+}$ and $d_{-}$ respectively. We assume that the entries of $V=(V^{i})$
are iid $\mathcal{N}(0,\frac{\lambda}{N})$ for some $\lambda>0$. We seek to
recover the planted truth $V$, given the observation $\mathbf{J}$. The natural
estimator in this setting is derived from PCA—one computes the eigenvector
$\hat{v}$ corresponding to the largest eigenvalue of $\mathbf{J}$, and uses
$\hat{v}\hat{v}^{\top}$ as an estimator for the latent subspace $VV^{\top}$.
The performance of this estimator can be precisely characterized using recent
advances in Random Matrix Theory ([12, 13]). In particular, assume that the
empirical spectral measure $\mu_{D_{N}}$ converges weakly to a limiting
measure $\mu_{D}$ supported on $[d_{-},d_{+}]$. Define
$G_{\mu_{D}}:\mathbb{R}\backslash[d_{-},d_{+}]\to\mathbb{R}$,

$\displaystyle
G_{\mu_{D}}(z)=\mathbb{E}_{X\sim\mu_{D}}\Big{[}\frac{1}{z-X}\Big{]}$

as the Cauchy transform of $\mu_{D}$. Further, define

$\displaystyle G_{\mu_{D}}(d^{+})=\lim_{z\downarrow
d_{+}}G_{\mu_{D}}(z),\,\,\,\,\,G_{\mu_{D}}(d^{-})=\lim_{z\uparrow
d_{-}}G_{\mu_{D}}(z).$

The BBP phase transition establishes that if
$\lambda>\frac{1}{G_{\mu_{D}}(d^{+})}$, there exists an “outlier” eigenvalue
at $G_{\mu_{D}}^{-1}(1/\lambda)$; otherwise, the largest sample eigenvalue
sticks to the spectral edge $d_{+}$. In the first case, the eigenvector
$\hat{v}$ corresponding to the outlying eigenvalue has a non-trivial overlap
with the planted signal $V$, i.e. with high probability,
$|V^{\top}\hat{v}|>0$. In the latter case,
$V^{\top}\hat{v}\stackrel{{\scriptstyle P}}{{\to}}0$.

To study the Langevin dynamics in this setting, we introduce the following
system of Stochastic Differential Equations

$\displaystyle\mathrm{d}{X^{i}_{t}}\,=\sum_{j=1}^{N}\mathbf{J}_{ij}X_{t}^{j}\mathrm{d}{t}\,-f^{\prime}\big{(}\frac{1}{N}\sum_{j=1}^{N}(X_{t}^{j})^{2}\big{)}X_{t}^{i}\mathrm{d}{t}\,+\beta^{-1/2}\mathrm{d}{W_{t}^{i}}\,\enspace.$
(1.2)

We assume that $f:[0,\infty)\to\mathbb{R}$ satisfies $f^{\prime}$ to be non-
negative and Lipschitz. Eventually, we will apply our results to the special
case $f(x)=x^{2}/2$. Note that this SDE can be looked upon as a penalized
version of the PCA problem on taking $\beta\rightarrow\infty$. The function
$f$ acts as a “confining” potential, so that we can work without a norm
constraint on the diffusion. For any $N\geq 1$ and $T\geq 0$, the SDE (1.2)
has a strong solution, which we denote by $\bm{X}_{t}\equiv\\{X_{t}^{i}:1\leq
i\leq N,t\in[0,T]\\}$ (see [3, Lemma 6.7] for a proof).

The strong solution to the Langevin diffusion (1.2) defines a natural
collection of estimators indexed by $t\in[0,\infty)$. We track the statistical
performance of these estimators via the normalized “overlap”
$R^{N}(t)=\frac{1}{N}\sum_{i=1}^{N}\sqrt{N}V^{i}X_{t}^{i}$. We note that the
entries of $V$ are typically of order $1/\sqrt{N}$; the multiplicative factor
$\sqrt{N}$ normalizes these entries so that the resultant is of order 1. Armed
with these notations, we can pose a natural question of interest:

_How does the overlap $R^{N}(t)$ evolve over time? _

In this paper, we provide sharp asymptotics for the evolution of $R^{N}(t)$
under a high-dimensional asymptotic limit where we send $N\to\infty$ with
$t\geq 0$ _fixed_. In practice, one should interpret these limits as
approximate characterizers of the overlap when the dimension $N$ is large,
while the diffusion has been run for a “short time”. In particular, we
emphasize that the time scales involved are significantly shorter than those
involved for “mixing” of these diffusion processes. We believe that these
asymptotics are particularly relevant for Statistics and Machine Learning. In
particular, it allows one to explicitly characterize the statistical effect of
“early stopping” in this problem.

Our first result characterizes the limiting behavior of $R^{N}(t)$. We will
see that the behavior of $R^{N}$ is intricately tied to that of the “auto-
correlation” function
$K^{N}(t,s)=\frac{1}{N}\sum_{i=1}^{N}X_{t}^{i}X_{s}^{i}$. Note that for any
$T\geq 0$, $R^{N}$ and $K^{N}$ are sequences of (random) continuous functions
on $[0,T]$ and $[0,T]^{2}$ respectively. In subsequent discussions, we equip
these metric spaces with the sup-norm topology.

In addition, we need to specify the initial conditions for the Langevin
diffusion $\\{\mathbf{X}_{t}:t\geq 0\\}$. We work under the following two sets
of initial conditions.

Initial Conditions:

  * (i)

(I.I.D. initial conditions) Assume that $\\{(\sqrt{N}V^{i},X_{0}^{i}):1\leq
i\leq N\\}$ are i.i.d., independent of $\mathbf{G}$ and
$\\{\mathbf{W}_{t}:t\geq 0\\}$. Assume furthermore that
$\mathbb{E}[(X_{0}^{i})^{2}]=1$,
$\mathbb{E}[V^{i}X_{0}^{i}]=\frac{\rho\sqrt{\lambda}}{\sqrt{N}}$ for some
$\rho\in[0,1]$.

  * (ii)

(I.I.D. under rotated basis) Define $\mathbf{Y}_{t}=\mathbf{G}\mathbf{X}_{t}$,
$U=\mathbf{G}V$, $\mathbf{B}_{t}=\mathbf{G}\mathbf{W}_{t}$. Assume that
$\\{(Y_{0}^{i},\sqrt{N}U^{i}):1\leq i\leq N\\}$ are i.i.d., independent of
$\\{\mathbf{B}_{t}:t\geq 0\\}$. Assume in addition that, $Y_{0}^{i}\sim F$
i.i.d, $\mathbb{E}[(Y_{0}^{i})^{2}]=1$,
$\mathbb{E}[U^{i}Y_{0}^{i}]=\frac{\rho\sqrt{\lambda}}{\sqrt{N}}$ for some
$\rho\in[0,1]$.

######  Remark 1.

Note that the parameter $\rho$ governs the correlation between the
initialization and the spike direction. For concreteness, we assume that
$\rho\geq 0$. The results generalize directly to $\rho<0$ if we replace $V$ by
$-V$. In addition, we require the second moments of $X_{0}$ and $Y_{0}$ to be
finite. The precise value 1 is chosen merely for convenience.

######  Remark 2.

The I.I.D. initial condition is arguably the most natural initialization in
Statistics and Machine Learning. The I.I.D. under rotated basis condition,
although a bit less natural from this perspective, is simpler to analyze with
the same theoretical tools. We think of the second initialization throughout
as a warm-up, with the first initialization being of principal interest.

######  Theorem 1.1.

Assume one of the Initial Conditions specified above. Fix $T\geq 0$. As
$N\to\infty$, $R^{N}$ and $K^{N}$ converge almost surely to deterministic
limits $R$ and $K$ respectively. Furthermore, these limits are the unique
solutions to the following system of integro-differential equations:

$\displaystyle R(t)$
$\displaystyle=\exp\left\\{-\int_{0}^{t}f^{\prime}(K(s))\mathrm{d}{s}\,\right\\}\cdot\mathop{\mathbb{E}}\limits_{(\mathbf{u},\bm{\sigma},\mathbf{Y}_{0})\sim\pi^{\infty}}\left[\exp(\bm{\sigma}t)\mathbf{Y}_{0}\mathbf{u}\right]$
$\displaystyle\qquad+\int_{0}^{t}\exp\left\\{-\int_{s}^{t}f^{\prime}(K(r))\mathrm{d}{r}\,\right\\}R(s)\mathop{\mathbb{E}}\limits_{(\mathbf{u},\bm{\sigma},\mathbf{Y}_{0})\sim\pi^{\infty}}\left[\exp(\bm{\sigma}(t-s))\mathbf{u}^{2}\right]\mathrm{d}{s}\,.$
(1.3)

$\displaystyle K(t,s)=\beta^{-1}\int_{0}^{s\wedge
t}\exp\left\\{-\int_{r}^{t}f^{\prime}(K(w))\mathrm{d}{w}\,-\int_{r}^{s}f^{\prime}(K(w))\mathrm{d}{w}\,\right\\}\mathop{\mathbb{E}}\limits_{(\mathbf{u},\bm{\sigma},\mathbf{Y}_{0})\sim\pi^{\infty}}\left[\exp(\bm{\sigma}(t+s-2r))\right]\mathrm{d}{r}\,$
$\displaystyle\qquad+\exp\left\\{-\int_{0}^{t}f^{\prime}(K(r))\mathrm{d}{r}\,-\int_{0}^{s}f^{\prime}(K(r))\mathrm{d}{r}\,\right\\}\cdot\mathop{\mathbb{E}}\limits_{(\mathbf{u},\bm{\sigma},\mathbf{Y}_{0})\sim\pi^{\infty}}\left[\exp(\bm{\sigma}(t+s))\mathbf{Y}_{0}^{2}\right]$
$\displaystyle\qquad+\exp\left\\{-\int_{0}^{t}f^{\prime}(K(r))\mathrm{d}{r}\,\right\\}\cdot\int_{0}^{s}\exp\left\\{-\int_{r}^{s}f^{\prime}(K(w))\mathrm{d}{w}\,\right\\}R(r)\mathop{\mathbb{E}}\limits_{(\mathbf{u},\bm{\sigma},\mathbf{Y}_{0})\sim\pi^{\infty}}\left[\exp(\bm{\sigma}(t+s-r))\mathbf{Y}_{0}\mathbf{u}\right]\mathrm{d}{r}\,$
$\displaystyle\qquad+\exp\left\\{-\int_{0}^{s}f^{\prime}(K(r))\mathrm{d}{r}\,\right\\}\cdot\int_{0}^{t}\exp\left\\{-\int_{r}^{t}f^{\prime}(K(w))\mathrm{d}{w}\,\right\\}R(r)\mathop{\mathbb{E}}\limits_{(\mathbf{u},\bm{\sigma},\mathbf{Y}_{0})\sim\pi^{\infty}}\left[\exp(\bm{\sigma}(t+s-r))\mathbf{Y}_{0}\mathbf{u}\right]\mathrm{d}{r}\,$
$\displaystyle+\int_{0}^{t}\mathrm{d}{r_{1}}\,\int_{0}^{s}\mathrm{d}{r_{2}}\,\left\\{\exp\left\\{-\int_{r_{1}}^{t}f^{\prime}(K(w))\mathrm{d}{w}\,-\int_{r_{2}}^{s}f^{\prime}(K(w))\mathrm{d}{w}\,\right\\}R(r_{1})R(r_{2})\mathop{\mathbb{E}}\limits_{(\mathbf{u},\bm{\sigma},\mathbf{Y}_{0})\sim\pi^{\infty}}\left[\exp(\bm{\sigma}(t+s-r_{1}-r_{2}))\mathbf{u}^{2}\right]\right\\}.$
(1.4)

Here we use the abbreviated notation $K(t):=K(t,t)$. It remains to specify the
distribution $\pi^{\infty}$; this limit depends on the initial condition:

  * (i)

Under I.I.D. initial conditions,
$\pi^{\infty}=\mathcal{N}(0,1)\otimes\mu_{D}\otimes\mathcal{N}(0,1)$.

  * (ii)

Under I.I.D. under rotated basis initialization,
$\pi^{\infty}=\mathcal{N}(0,1)\otimes\mu_{D}\otimes F$.

Theorem (1.1) holds for any $t\in[0,T]$. This allows one to characterize the
overlap at any time $t$, for sufficiently large $N$. In this light, Theorem
(1.1) may be interpreted as a quantification of the effects of early stopping
on Langevin dynamics. However, though the theorem provides a precise
characterization of $R^{N}(t)$ under a high-dimensional limit, it is a priori
unclear whether this description yields an explicit understanding regarding
the behavior of $R^{N}(t)$. This is primarily due to the mathematically
involved nature of the fixed point equations (1.1)-(1.1). To gain a better
understanding of the CHSCK equations, our next theorem illustrates that
Theorem (1.1) can yield explicit results on the correlation between the output
$\mathbf{X}_{t}$ and the latent vector $\sqrt{N}V$ under the double limit
$t\to\infty$, following $N\to\infty$. To this end, first note that this
correlation can be captured through the ratio $R^{2}(t)/(\lambda K(t,t))$,
since

$\displaystyle\frac{\Big{(}\sum_{i=1}^{N}\sqrt{N}V^{i}X_{t}^{i}\Big{)}^{2}}{[\sum_{i=1}^{N}(X_{t}^{i})^{2}][N\sum_{i=1}^{N}(V^{i})^{2}]}=\frac{(R^{N}(t))^{2}}{[K^{N}(t,t)][\sum_{i=1}^{N}(V^{i})^{2}]}.$

and $\|V\|_{2}^{2}\to\lambda$ almost surely under our initial conditions.

To precisely characterize the limiting correlation, we specialize to the
following setting: consider $f(x)=x^{2}/2$ and $\mu_{D}$ to be the scaled
semi-circle distribution, supported on $[-\sigma_{\star},\sigma_{\star}]$, for
some $\sigma_{\star}>0$. This corresponds to a setting where the additive
noise $\mathbf{Z}$ in (1.1) is a symmetric Gaussian matrix. Note that
$f(x)=x^{2}/2$ satisfies the regularity conditions required in Theorem 1.1.
Formally, the semi-circle law on $[-\sigma_{\star},\sigma_{\star}]$ has a
density

$\displaystyle\frac{\mathrm{d}{\mu_{D}}\,}{\mathrm{d}{x}\,}=\frac{2}{\pi\sigma_{\star}^{2}}\sqrt{\sigma_{\star}^{2}-x^{2}},\,\,\,\,-\sigma_{\star}\leq
x\leq\sigma_{\star}.$ (1.5)

Let
$\mathcal{S}:\mathbb{R}\backslash[-\sigma_{\star},\sigma_{\star}]\to\mathbb{R}$
denote the Stieljes transform of $\mu_{D}$, i.e.,

$\displaystyle\mathcal{S}(z)=\mathop{\mathbb{E}}_{\bm{\sigma}\sim\mu_{D}}\big{[}\frac{1}{z-\bm{\sigma}}\big{]}=\frac{2}{z+\sqrt{z^{2}-\sigma_{\star}^{2}}}.$
(1.6)

######  Theorem 1.2.

Assume $\lambda>\sigma_{\star}/2$, and set
$\tilde{\lambda}=\lambda+\frac{\sigma_{\star}^{2}}{4\lambda}$. If
$\beta<\frac{1}{\sigma_{\star}^{2}}$, the equation
$z=\beta^{-1}\mathcal{S}(z/2)$ has two real roots. Set $s_{\beta}$ to be the
largest real root of this equation if $\beta<\frac{1}{\sigma_{\star}^{2}}$,
otherwise set $s_{\beta}=2\sigma_{\star}$.

  * (i)

If $2\tilde{\lambda}<s_{\beta}$ or $\rho=0$,
$\lim_{t\to\infty}\frac{R(t)^{2}}{\lambda K(t,t)}=0$.

  * (ii)

If $2\tilde{\lambda}>s_{\beta}$ and $\rho>0$,

$\displaystyle\lim_{t\to\infty}\frac{R(t)^{2}}{\lambda
K(t,t)}=\left(1-\frac{\beta^{-1}}{2\lambda(\lambda+\sigma_{\star}^{2}/4\lambda)}\right)\left(1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\right).$
(1.7)

Several remarks are in order regarding Theorem 1.2. First, note that
$R^{2}(t)/(\lambda K(t,t))$ captures the limiting _correlation_ between the
output $\mathbf{X}_{t}$ and the latent vector $\sqrt{N}V$, since

$\displaystyle\frac{\Big{(}\sum_{i=1}^{N}\sqrt{N}V^{i}X_{t}^{i}\Big{)}^{2}}{[\sum_{i=1}^{N}(X_{t}^{i})^{2}][N\sum_{i=1}^{N}(V^{i})^{2}]}=\frac{(R^{N}(t))^{2}}{[K^{N}(t,t)][\sum_{i=1}^{N}(V^{i})^{2}]}.$

and $\|V\|_{2}^{2}\to\lambda$ almost surely under our initial conditions.
Further, it is information theoretically impossible to recover the planted
vector $V$ below the BBP threshold (i.e., when $\lambda<\sigma_{\star}/2$).
Thus we focus on the interesting regime $\lambda>\sigma_{\star}/2$. In this
setting, Theorem 1.2 can be interpreted as follows: first, if $\rho=0$ i.e.,
if the initialization has $o(1)$ correlation, then the limiting correlation
stays at zero. We note that our asymptotics is _different_ from non-asymptotic
results (in $N$) which allow vanishing (in $N$) overlap of the initialization
with the truth. This is specifically the notion considered in [8], [7]. We
defer a detailed comparison of our results with this recent line of work to
Section 1.1. Our result establishes that if the diffusive noise is large
($2\tilde{\lambda}<s_{\beta}$), the initial correlation washes off, and the
correlation converges to zero in the limit $t\to\infty$. On the other hand, if
the diffusive noise is small ($2\tilde{\lambda}>s_{\beta}$), we obtain a non-
trivial correlation in the limit. While our setup is quite simple, we can
precisely quantify the tradeoff between the SNR (captured by $\lambda$) and
the strength of the injected noise (captured by $\beta^{-1}$) in the limiting
correlation—we consider this to be one of the main contributions of our work.
In particular, the limiting correlation increases as a function of $\lambda$,
as well as $\beta^{-1}$, as one would naturally expect.

###  1.1 Related literature

  * (i)

Dynamical Mean Field Theory and Crisanti-Horner-Sommers-Cugliandolo-Kurchan
(CHSCK) equations: Dynamical Mean-Field Theory originated in the theory of
spin glasses in the 80’s [48, 47]. In this approach, dynamics are
characterized in terms of the “correlation” and “response” functions. In
special cases, these functions satisfy a system of integro-differential
equations [20, 23] (we refer to such systems as CHSCK equations henceforth for
simplicity). In general settings, the effective dynamics is described in terms
of a non-Markovian stochastic process with long-term memory.

Recently, this framework has been employed in the statistical physics
literature to study high-dimensional inference problems such as Gaussian
mixture models, max-margin classification, and tensor PCA [39, 38, 18, 43, 40,
46, 1]. In these papers, versions of CHSCK equations have been proposed, and
analyzed numerically to track the performance of specific algorithms. Our work
differs from these earlier inquiries in some crucial ways—first, the
derivations of the CHSCK equations in these works are non-rigorous, and the
subsequent analysis is also numerical. In sharp contrast, our results are
fully rigorous, furthermore, we characterize the precise tradeoff between the
SNR and the injected noise in the Langevin algorithm. However, it should be
noted that our model is, in some sense simpler than the other models described
above. The main technical difference is that the spiked matrix model does not
require a “response function”—this considerably simplifies the CHSCK system,
and the subsequent analysis.

  * (ii)

Prior rigorous results: Dynamical Mean-field Theory for mean-field spin
glasses was established on rigorous footing in the works of [3], [6, 5], [31],
[32], among others. The CHSCK equations were formally derived in [10]. While
some useful information could be extracted from these equations under special
settings (e.g. Langevin dynamics for matrix models [3] and at high-temperature
[25]), a general analysis of these equations has been quite challenging.

Recently, there has been renewed interest in this area. [28] derived the CHSCK
equations for spherical spin glasses starting from disorder dependent initial
conditions. [27] examined the universality of these equations to the law of
the disorder variables. [24] also introduced alternative techniques for
establishing universality of such dynamical algorithms.

We note that in sharp contrast to our setting, the models analyzed in this
line of work correspond to “null” models, i.e. without any planted signal. As
a result, our analysis is not directly comparable to the aforementioned
papers. Despite this difference, our derivation of the CHSCK equations and its
subsequent analysis relies partially on techniques from [3], which studies the
Langevin dynamics in matrix models in the absence of a spike. That said, we
emphasize that planted models are closer to models typically observed in
Statistics and Machine Learning problems. Thus, despite the simplicity of
(1.1), Theorem (1.2) provides useful insights regarding the interplay between
the SNR and the noise magnitude in determining the exact value of the limiting
correlation. To the best of our knowledge, our work is the first to
characterize this tradeoff for a planted model. We hope that this precise
analysis would spark further investigations into Langevin-type dynamics for
more complex planted models, as seen in contemporary Statistics and Machine
Learning problems.

  * (iii)

Recent flow-based analyses: There has been considerable interest in
understanding gradient descent algorithms (i.e. $\beta=\infty$ dynamics) in
Statistics and Machine Learning. [14], [15] derive novel systems of integro-
differential equations for gradient descent dynamics for certain spiked
models. Although the equations are similar in spirit, there are crucial
differences between these results, and the ones proved in this paper. First,
for the spiked matrix model, the result in [15] applies only to the
$\beta=\infty$ case, and recovers the BBP phase transition. In contrast, we
discover additional phase transitions depending on the strength of the
injected noise. Further, the proof techniques are also completely
different—[14, 15] use recent advances in the random matrix literature on
Green functions to derive their results. In a different line of work, [9], [4]
have also used random matrix asymptotics to study the role of “early stopping”
for gradient descent algorithms for linear regression. In this problem, the
solution at any fixed time is available in closed form, which considerably
simplifies the analysis.

Recently, [19] derived the CHSCK equations for gradient descent (i.e.
$\beta=\infty$) for empirical risk minimization. The approaches in these two
papers are completely different—[19] first use Approximate Message Passing
(AMP) style algorithms to study first order methods, and subsequently take a
continuous time limit. Our approach, on the other hand, is grounded in random
matrix theory.

  * (iv)

We note that our results assume a “warm-start”, i.e. an $O(1)$-correlation
between the initialization and the planted signal. This situation is different
from a random initialization, where the initialization has $O(1/\sqrt{N})$
correlation with the planted signal. The approach pursued in this paper does
not seem suited to analyze the evolution of the correlation in this regime.
Some answers are provided by the recent theory of “bounding flows” [11] and
the subsequent applications of this machinery to planted models [8], [7]. The
main restriction of this approach is that it yields “non-sharp” answers, in
constrast to the approach outlined in our paper.

Outline: The rest of the paper is structured as follows: we prove Theorem 1.2
in Section 2, while the proof of the CHSCK equations Theorem 1.1 is deferred
to Section 3.

Acknowledgements Liang was supported by NSF CAREER Grant DMS-2042473. Sur was
supported by NSF DMS-2113426. Sen was supported by a Harvard Dean’s
Competitive Fund Award.

##  2 Proof of Theorem 1.2

Throughout this section, for notational convenience, we use the shorthand
$K(t)=K(t,t)$. Let
$E(t):=\exp\\{\int_{0}^{t}f^{\prime}(K(s))\mathrm{d}{s}\,\\}$. For our
subsequent analysis, it will be convenient to transform the functions $R(t)$,
$K(t)$ into a new set of functions $g(t)$, $h(t)$, defined as follows:

$\displaystyle g(t)$ $\displaystyle:=E(t)R(t)$ $\displaystyle h(t)$
$\displaystyle:=E^{2}(t)K(t).$ (2.1)

Observe that $(R(t))^{2}/K(t)=(g(t))^{2}/h(t)$, and thus it suffices to track
$g(t)$, $h(t)$. In turn, we observe that (1.1),(1.1) imply that $g(t)$, $h(t)$
are uniquely specified as the solutions to the following fixed-point system.

$\displaystyle g(t)$
$\displaystyle=\mathbb{E}[\mathbf{Y}_{0}\mathbf{u}]\cdot\mathbb{E}[e^{t\bm{\sigma}}]+\mathbb{E}[\mathbf{u}^{2}]\cdot\int_{0}^{t}g(s)\mathbb{E}[e^{(t-s)\bm{\sigma}}]\mathrm{d}{s}\,,$
(2.2) $\displaystyle h(t)$
$\displaystyle=\beta^{-1}\int_{0}^{t}E^{2}(s)\mathbb{E}[e^{(2t-2s)\bm{\sigma}}]\mathrm{d}{s}\,+\mathbb{E}[\mathbf{Y}_{0}^{2}]\cdot\mathbb{E}[e^{2t\bm{\sigma}}]$
(2.3)
$\displaystyle+2\mathbb{E}[\mathbf{Y}_{0}\mathbf{u}]\cdot\int_{0}^{t}g(s)\mathbb{E}[e^{(2t-s)\bm{\sigma}}]\mathrm{d}{s}\,+\mathbb{E}[\mathbf{u}^{2}]\cdot\int_{0}^{t}\int_{0}^{t}g(s_{1})g(s_{2})\mathbb{E}[e^{(2t-s_{1}-s_{2})\bm{\sigma}}]\mathrm{d}{s_{1}}\,\mathrm{d}{s_{2}}\,.$
(2.4)

Our first lemma characterizes the behavior of $g(\cdot)$.

######  Lemma 2.1.

With $\bm{\sigma}$ drawn from the semi-circle distribution with parameter
$\sigma_{\star}$, the function $g(\cdot)$ defined in (2.1) satisfies the
following

$\displaystyle
g(t)=\sqrt{\lambda}\rho\left\\{\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}\exp\big{\\{}(\lambda+\frac{\sigma_{\star}^{2}}{4\lambda})t\big{\\}}+\frac{1}{2\pi\lambda}\int_{-\sigma_{\star}}^{\sigma_{\star}}\frac{e^{xt}\sqrt{\sigma_{\star}^{2}-x^{2}}}{(\lambda+\frac{\sigma_{\star}^{2}}{4\lambda})-x}\mathop{}\\!\mathrm{d}{x}\right\\}\;,$
(2.5)

where $x_{+}=\max\\{x,0\\}$.

###### Proof of Lemma 2.1 .

Note that the last term in the RHS of (2.2) contains a convolution, suggesting
that the Laplace transform of $g(\cdot)$ should provide useful information.
For ${\rm Re}(z)>\sigma_{\star}$, recall that the Laplace transform of $g$ is
given by

$\displaystyle\mathcal{L}_{g}(z):=\int_{0}^{\infty}e^{-zt}g(t)\mathrm{d}{t}\,,$
(2.6)

and the Stieltjes transform of $\mu_{D}$ is given by

$\displaystyle\mathcal{S}(z)=\mathbb{E}\big{[}\frac{1}{z-\bm{\sigma}}\big{]}.$
(2.7)

Evaluating Laplace transforms, Eqn. (2.6) can be transformed as

$\displaystyle\mathcal{L}_{g}(z)=\frac{\mathop{\mathbb{E}}[\mathbf{Y}_{0}\mathbf{u}]\cdot\mathcal{S}(z)}{1-\mathop{\mathbb{E}}[\mathbf{u}^{2}]\cdot\mathcal{S}(z)}.$
(2.8)

By the Fourier-Mellin formula (inverse Laplace transform), we know that

$\displaystyle g(t)=\frac{1}{2\pi i}\lim_{T\rightarrow+\infty}\int_{\gamma-
iT}^{\gamma+iT}e^{zt}\mathcal{L}_{g}(z)dz.$ (2.9)

When $\mu_{D}$ follows a semi-circle law, using (1.6), we have

$\displaystyle\mathcal{L}_{g}(z)=\frac{2\sqrt{\lambda}\rho}{z+\sqrt{z^{2}-\sigma^{2}_{\star}}-2\lambda}\;.$
(2.10)

Figure 1: The contour integral for evaluating the Fourier-Mellin formula.

Fix $\gamma\in\mathbb{R}$ with
$\gamma>\lambda+\frac{\sigma_{\star}^{2}}{4\lambda}$, we need to evaluate the
Fourier-Mellin integral of the form

$\displaystyle g_{R}(t)$ $\displaystyle:=\frac{1}{2\pi{i\mkern
1.0mu}}\int_{\gamma-{i\mkern 1.0mu}R}^{\gamma+{i\mkern
1.0mu}R}e^{zt}\mathcal{L}_{g}(z)\mathop{}\\!\mathrm{d}{z}$
$\displaystyle=\frac{1}{2\pi{i\mkern
1.0mu}}\int_{S_{1}}e^{zt}\mathcal{L}_{g}(z)\mathop{}\\!\mathrm{d}{z}\;.$

In other words, we need to know the integral along the line segment $S_{1}$ in
Fig 1. First, we observe that the function

$\displaystyle f_{t}(z)$
$\displaystyle:=e^{zt}\mathcal{L}_{g}(z)=e^{zt}\frac{2\sqrt{\lambda}\rho}{z+\sqrt{z^{2}-\sigma^{2}_{\star}}-2\lambda}$
(2.11)
$\displaystyle=e^{zt}\frac{2\sqrt{\lambda}\rho}{4\lambda}\frac{\sqrt{z^{2}-\sigma^{2}_{\star}}-(z-2\lambda)}{z-(\lambda+\frac{\sigma_{\star}^{2}}{4\lambda})}$
(2.12)

has a simple pole at $z=\lambda+\frac{\sigma_{\star}^{2}}{4\lambda}$.
Therefore using the Cauchy Residue Theorem, we have,

$\displaystyle\frac{1}{2\pi{i\mkern
1.0mu}}\oint_{S_{1}+S_{2}+S_{3}}f_{t}(z)\mathop{}\\!\mathrm{d}{z}=\sqrt{\lambda}\rho\cdot\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}e^{(\lambda+\frac{\sigma_{\star}^{2}}{4\lambda})t}\;.$
(2.13)

Second, we evaluate the integral over the arc $S_{2}^{\epsilon}$. Parametrize
the $z(\theta)\in S_{2}$ by $z(\theta)=\gamma-(R\cos\theta+{i\mkern
1.0mu}R\sin\theta)$ with
$\theta\in[\frac{\pi}{2}+\epsilon,\frac{3\pi}{2}-\epsilon]$, therefore

$\displaystyle\left|\frac{1}{2\pi{i\mkern
1.0mu}}\int_{S_{2}^{\epsilon}}f_{t}(z)\mathop{}\\!\mathrm{d}{z}\right|\leq\frac{R}{2}e^{t(\gamma-R\cos(\epsilon))}\sup_{\theta\in[\frac{\pi}{2}+\epsilon,\frac{3\pi}{2}-\epsilon]}\frac{2\sqrt{\lambda}\rho}{|\gamma+Re^{{i\mkern
1.0mu}\theta}+\sqrt{(\gamma+Re^{{i\mkern
1.0mu}\theta})^{2}-\sigma_{\star}^{2}}-2\lambda|}$ (2.14)

and thus for any fixed $\epsilon$

$\displaystyle\lim_{R\rightarrow\infty}\frac{1}{2\pi{i\mkern
1.0mu}}\int_{S_{2}^{\epsilon}}f_{t}(z)\mathop{}\\!\mathrm{d}{z}=0\;.$ (2.15)

For the part of the integral $S_{2}\backslash S_{2}^{\epsilon}$, one can show
that

$\displaystyle\left|\frac{1}{2\pi{i\mkern 1.0mu}}\int_{S_{2}\backslash
S_{2}^{\epsilon}}f_{t}(z)\mathop{}\\!\mathrm{d}{z}\right|\leq\epsilon\frac{R}{\pi}e^{t\gamma}\sup_{\theta\in[\frac{\pi}{2},\frac{\pi}{2}+\epsilon]}\frac{2\sqrt{\lambda}\rho}{|\gamma+Re^{{i\mkern
1.0mu}\theta}+\sqrt{(\gamma+Re^{{i\mkern
1.0mu}\theta})^{2}-\sigma_{\star}^{2}}-2\lambda|}$ (2.16)

and thus with some universal constant $C(\lambda,\sigma_{\star},\rho,\gamma)$

$\displaystyle\lim_{R\rightarrow\infty}\frac{1}{2\pi{i\mkern
1.0mu}}\int_{S_{2}^{\epsilon}}f_{t}(z)\mathop{}\\!\mathrm{d}{z}=C(\lambda,\sigma_{\star},\rho,\gamma)\cdot\epsilon\;.$
(2.17)

Putting these estimates together and sending $\epsilon\rightarrow 0$, we have
shown

$\displaystyle\lim_{R\rightarrow\infty}\frac{1}{2\pi{i\mkern
1.0mu}}\int_{S_{2}}f_{t}(z)\mathop{}\\!\mathrm{d}{z}=0$ (2.18)

Third, we evaluate the integral over the segment $S_{3}$ (corresponding to the
branch point of $\sqrt{z^{2}-\sigma_{\star}^{2}}$), with $z=x+{i\mkern
1.0mu}\epsilon$ where $x\in[-\sigma_{\star},\sigma_{\star}]$. We consider this
since the function $z\mapsto\sqrt{z}$ is a well-defined single-valued function
only on $z\in\mathbb{C}\backslash(-\infty,0)$. One can show that

$\displaystyle\lim_{\epsilon\rightarrow 0}\frac{1}{2\pi{i\mkern
1.0mu}}\int_{S_{3}^{\epsilon}}f_{t}(z)\mathop{}\\!\mathrm{d}{z}$ (2.19)
$\displaystyle=\frac{1}{2\pi{i\mkern
1.0mu}}\int_{-\sigma_{\star}}^{\sigma_{\star}}e^{xt}\frac{2\sqrt{\lambda}\rho}{(x-2\lambda)+{i\mkern
1.0mu}\sqrt{\sigma_{\star}^{2}-x^{2}}}\mathop{}\\!\mathrm{d}{x}-\frac{1}{2\pi{i\mkern
1.0mu}}\int_{-\sigma_{\star}}^{\sigma_{\star}}e^{xt}\frac{2\sqrt{\lambda}\rho}{(x-2\lambda)-{i\mkern
1.0mu}\sqrt{\sigma_{\star}^{2}-x^{2}}}\mathop{}\\!\mathrm{d}{x}$ (2.20)
$\displaystyle=\frac{2\sqrt{\lambda}\rho}{2\pi{i\mkern
1.0mu}}\int_{-\sigma_{\star}}^{\sigma_{\star}}e^{xt}\frac{-2i\sqrt{\sigma_{\star}^{2}-x^{2}}}{-4\lambda
x+4\lambda^{2}+\sigma_{\star}^{2}}\mathop{}\\!\mathrm{d}{x}$ (2.21)
$\displaystyle=-\sqrt{\lambda}\rho\frac{1}{2\pi\lambda}\int_{-\sigma_{\star}}^{\sigma_{\star}}e^{xt}\frac{\sqrt{\sigma_{\star}^{2}-x^{2}}}{(\lambda+\frac{\sigma_{\star}^{2}}{4\lambda})-x}\mathop{}\\!\mathrm{d}{x}\;.$
(2.22)

Finally, we integrate over the semi-circular arc in $S_{3}$. Parametrizing
$z=\sigma_{\star}+\epsilon e^{i\theta}$, we have, $\mathrm{d}{z}\,=\epsilon
ie^{i\theta}\mathrm{d}{\theta}\,$. In turn, the integral reduces to

$\displaystyle\int_{\pi/2}^{-\pi/2}e^{(\sigma_{\star}+\epsilon
e^{i\theta})t}\frac{2\sqrt{\lambda}\rho}{\sigma_{\star}+\epsilon
e^{i\theta}+\sqrt{(\sigma_{\star}+\epsilon
e^{i\theta})^{2}-\sigma_{\star}^{2}}-2\lambda}\epsilon
ie^{i\theta}\mathrm{d}{\theta}\,.$

We note that this integral converges to zero as $\epsilon\to 0$. Putting three
pieces together, we conclude the proof. ∎

We next turn our attention to $h$. Recall
$E(t)=\exp\\{\int_{0}^{t}K(s)\mathrm{d}s\\}$, and set $F(t)=E^{2}(t)$.
Differentiating, we obtain that $F^{\prime}(t)=2K(t)F(t)=2h(t)$, defined in
(2.1). Coupled with (2.4), we have, $F$ satisfies the integro-differential
equation,

$\displaystyle F^{\prime}(t)$
$\displaystyle=2\beta^{-1}\int_{0}^{t}F(s)\mathbb{E}\Big{[}\exp(2(t-s)\bm{\sigma})\Big{]}\mathrm{d}s+\Phi(t),$
$\displaystyle\Phi(t)$
$\displaystyle=2\mathbb{E}[\bm{Y_{0}}^{2}]\mathbb{E}[\exp(2t\bm{\sigma})]+4\mathbb{E}[\bm{Y_{0}}\bm{u}]\int_{0}^{t}g(s)\mathbb{E}[\exp(\bm{\sigma}(2t-s))]\mathrm{d}s$
(2.23) $\displaystyle+$ $\displaystyle
2\mathbb{E}[\bm{u}^{2}]\int_{0}^{t}\int_{0}^{t}g(s_{1})g(s_{2})\mathbb{E}\Big{[}\exp\Big{(}\bm{\sigma}(2t-s_{1}-s_{2})\Big{)}\Big{]}\mathrm{d}s_{1}\mathrm{d}s_{2}.$

######  Lemma 2.2.

The function $h$ has Laplace transform

$\displaystyle\mathcal{L}_{h}(z)=\frac{1}{2}\Big{[}\frac{z\mathcal{L}_{\Phi}(z)+\beta^{-1}\mathcal{S}(z/2)}{z-\beta^{-1}\mathcal{S}(z/2)}\Big{]}.$

###### Proof of Lemma 2.2.

Taking Laplace transforms in (2.23), we obtain,

$\displaystyle
z\mathcal{L}_{F}(z)-1=2\beta^{-1}\mathcal{L}_{F}(z)\mathcal{L}_{M}(z)+\mathcal{L}_{\Phi}(z),$

where we set $M(t)=\mathbb{E}[\exp(2t\sigma)]$. Transposing, we obtain,

$\displaystyle\mathcal{L}_{F}(z)=\frac{1+\mathcal{L}_{\Phi}(z)}{z-2\beta^{-1}\mathcal{L}_{M}(z)}.$

Recall that $F^{\prime}(t)=2h(t)$, and thus
$\mathcal{L}_{h}(z)=\frac{1}{2}[z\mathcal{L}_{F}(z)-1]$. This allows us to
obtain the Laplace transform of $h$. Finally we note that by direct
computation, $\mathcal{L}_{M}(z)=\frac{1}{2}\mathcal{S}(z/2)$. This completes
the calculation.

∎

We set $\tilde{\lambda}=\lambda+\frac{\sigma_{\star}^{2}}{4\lambda}$. Let
$s_{\beta}$ denote the largest real solution to the equation
$z=\beta^{-1}\mathcal{S}(z/2)$.

######  Lemma 2.3.

If $2\tilde{\lambda}>s_{\beta}$, we have that

$\displaystyle\lim_{z\to
0}\Big{|}z\mathcal{L}_{\Phi}(2\tilde{\lambda}+z)-2\lambda^{2}\rho^{2}\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}^{2}\mathbb{E}\left[\frac{1}{(\tilde{\lambda}-\bm{\sigma})^{2}}\right]\Big{|}=0.$

###### Proof.

By definition,

$\displaystyle\mathcal{L}_{\Phi}(2\tilde{\lambda}+z)$
$\displaystyle=\int_{0}^{\infty}\exp(-(2\tilde{\lambda}+z)t)\Phi(t)\mathrm{d}t:=T_{1}+T_{2}+T_{3},$
$\displaystyle T_{1}$
$\displaystyle:=2\mathbb{E}[\bm{Y_{0}}^{2}]\int_{0}^{\infty}\exp(-(2\tilde{\lambda}+z)t)\mathbb{E}[\exp(2t\bm{\sigma})]\mathrm{d}t,$
$\displaystyle T_{2}$
$\displaystyle:=4\mathbb{E}[\bm{Y_{0}}\bm{u}]\int_{0}^{\infty}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z)t)g(s)\mathbb{E}[\exp(\bm{\sigma}(2t-s))]\mathrm{d}s\mathrm{d}t,$
$\displaystyle T_{3}$
$\displaystyle:=2\mathbb{E}[\bm{u}^{2}]\int_{0}^{\infty}\int_{0}^{t}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z)t)g(s_{1})g(s_{2})\mathbb{E}\Big{[}\exp\Big{(}\bm{\sigma}(2t-s_{1}-s_{2})\Big{)}\Big{]}\mathrm{d}s_{1}\mathrm{d}s_{2}\mathrm{d}t.$

We first claim that there exists a constant $C>0$ such that

$\displaystyle\Big{|}\int_{0}^{\infty}\exp(-(2\tilde{\lambda}+z)t)\mathbb{E}[\exp(2t\bm{\sigma})]\mathrm{d}t\Big{|}\leq
C,\,\,\,\,\Big{|}\int_{0}^{\infty}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z)t)g(s)\mathbb{E}[\exp(\bm{\sigma}(2t-s))]\mathrm{d}s\mathrm{d}t\Big{|}\leq
C.$

We establish each bound in turn. Set $z=z_{1}+iz_{2}$ with
$z_{1},z_{2}\in\mathbb{R}$, and note that $z_{1},z_{2}\to 0$. Choose $z_{1}$
small enough so that $2\tilde{\lambda}+z_{1}>4\sigma_{\star}$ , since
$2\tilde{\lambda}>4\sigma_{\star}$ by assumption. Thus we have,

$\displaystyle\Big{|}\int_{0}^{\infty}\exp(-(2\tilde{\lambda}+z)t)\mathbb{E}[\exp(2t\bm{\sigma})]\mathrm{d}t\Big{|}$
$\displaystyle\leq\int_{0}^{\infty}\exp(-(2\tilde{\lambda}+z_{1})t)\mathbb{E}[\exp(2t\bm{\sigma})]\mathrm{d}t$
$\displaystyle\leq\int_{0}^{\infty}\exp\Big{(}-(2\tilde{\lambda}-2\sigma_{\star}+z_{1})t\Big{)}\mathrm{d}t$
$\displaystyle\leq\frac{1}{2\tilde{\lambda}-2\sigma_{\star}+z_{1}}.$

This remains bounded as $z_{1}\to 0$. To analyze the second term, first
observe from Lemma 2.1 that

$\displaystyle\lim_{t\to\infty}\exp(-\tilde{\lambda}t)g(t)=\sqrt{\lambda}\rho\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}.$

Thus for any $\varepsilon>0$, there exists $t_{0}$ large enough such that for
all $t\geq t_{0}$,

$\displaystyle(1-\varepsilon)\sqrt{\lambda}\rho\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}\exp(\tilde{\lambda}t)\leq
g(t)\leq(1+\varepsilon)\sqrt{\lambda}\rho\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}\exp(\tilde{\lambda}t).$

Then we have, setting $z=z_{1}+iz_{2}$ with $z_{1}$ small enough,

$\displaystyle\Big{|}\int_{0}^{\infty}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z)t)g(s)\mathbb{E}[\exp(\bm{\sigma}(2t-s))]\mathrm{d}s\mathrm{d}t\Big{|}$
$\displaystyle\leq\int_{0}^{\infty}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z_{1})t)g(s)\mathbb{E}[\exp(\bm{\sigma}(2t-s))]\mathrm{d}s\mathrm{d}t$
$\displaystyle\leq
C_{1}+\int_{t_{0}}^{\infty}\Big{[}\int_{0}^{t_{0}}\exp(-(2\tilde{\lambda}+z_{1})t)g(s)\mathbb{E}[\exp(\bm{\sigma}(2t-s))]\mathrm{d}s+\int_{t_{0}}^{t}\exp(-(2\tilde{\lambda}+z_{1})t)g(s)\mathbb{E}[\exp(\bm{\sigma}(2t-s))]\mathrm{d}s\Big{]}\mathrm{d}t.$

The control of this term is complete once we control the two integrals above.
To this end,

$\displaystyle\int_{t_{0}}^{\infty}\int_{0}^{t_{0}}\exp\Big{(}-(2\tilde{\lambda}+z_{1})t\Big{)}g(s)\mathbb{E}[\exp(\bm{\sigma}(2t-s))]\mathrm{d}s\mathrm{d}t$
$\displaystyle\leq\Big{(}\max_{0\leq s\leq
t_{0}}g(s)\Big{)}\int_{t_{0}}^{\infty}\exp\Big{(}-(2\tilde{\lambda}+z_{1}-2\sigma_{\star})t\Big{)}\mathrm{d}t$
$\displaystyle\leq\Big{(}\max_{0\leq s\leq
t_{0}}g(s)\Big{)}\frac{\exp(-t_{0}(2\tilde{\lambda}+z_{1}-2\sigma_{\star}))}{2\tilde{\lambda}+z_{1}-2\sigma_{\star}}$

which remains bounded as $z_{1}\to 0$. Finally,

$\displaystyle\int_{t_{0}}^{\infty}\int_{t_{0}}^{t}\exp\Big{(}-(2\tilde{\lambda}+z_{1})t)g(s)\mathbb{E}[\exp(\bm{\sigma}(2t-s))]\Big{)}\mathrm{d}s\mathrm{d}t$
$\displaystyle\leq(1+\varepsilon)\sqrt{\lambda}\rho\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}\int_{t_{0}}^{\infty}\int_{t_{0}}^{t}\exp\Big{(}-(2\tilde{\lambda}+z_{1})t)\Big{)}\exp(\tilde{\lambda}s+\sigma_{\star}(2t-s))\mathrm{d}s\mathrm{d}t$
$\displaystyle\leq\frac{(1+\varepsilon)}{\tilde{\lambda}-\sigma_{\star}}\sqrt{\lambda}\rho\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}\int_{t_{0}}^{\infty}\exp\Big{(}-(\tilde{\lambda}+z_{1}-\sigma_{\star})t)\mathrm{d}t$

which remains bounded as $z_{1}\to 0$.

Finally, we turn to the term $T_{3}$. Recall that, using (2.5),
$g(s)=\sqrt{\lambda}\rho(1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}})_{+}\exp(\tilde{\lambda}s)+g_{1}(s)$,
where

$g_{1}(s)=\frac{\sigma_{\star}^{2}}{4\lambda}\mathbb{E}_{x\sim\mu_{D}}\left[\frac{e^{xs}}{\tilde{\lambda}-x}\right].$

Further recalling that $\mathbb{E}[\bm{u}^{2}]=\lambda$, we have

$\displaystyle T_{3}=$ $\displaystyle
2\mathbb{E}[\bm{u}^{2}]\int_{0}^{\infty}\int_{0}^{t}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z)t)g(s_{1})g(s_{2})\mathbb{E}\Big{[}\exp\Big{(}\bm{\sigma}(2t-s_{1}-s_{2})\Big{)}\Big{]}\mathrm{d}s_{1}\mathrm{d}s_{2}\mathrm{d}t$
$\displaystyle=2\lambda^{2}\rho^{2}\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}^{2}\int_{0}^{\infty}\int_{0}^{t}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z)t+\tilde{\lambda}(s_{1}+s_{2}))\mathbb{E}\Big{[}\exp\Big{(}\bm{\sigma}(2t-s_{1}-s_{2})\Big{)}\Big{]}\mathrm{d}s_{1}\mathrm{d}s_{2}\mathrm{d}t+\mathrm{Rem}$
(2.24)

where $\mathrm{Rem}$ is the remainder term which we will later show to be
bounded as $z\to 0$. Computing the first term, we obtain,

$\displaystyle\int_{0}^{\infty}\int_{0}^{t}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z)t+\tilde{\lambda}(s_{1}+s_{2}))\mathbb{E}\Big{[}\exp\Big{(}\bm{\sigma}(2t-s_{1}-s_{2})\Big{)}\Big{]}\mathrm{d}s_{1}\mathrm{d}s_{2}\mathrm{d}t$
$\displaystyle=\mathbb{E}\Big{[}\int_{0}^{\infty}\int_{0}^{t}\int_{0}^{t}\exp\Big{(}-(2\tilde{\lambda}+z-2\bm{\sigma})t+(\tilde{\lambda}-\bm{\sigma})s_{1}+(\tilde{\lambda}-\bm{\sigma})s_{2}\Big{)}\mathrm{d}s_{1}\mathrm{d}s_{2}\mathrm{d}t\Big{]}$
$\displaystyle=\mathbb{E}\Big{[}\int_{0}^{\infty}\exp(-(2\tilde{\lambda}+z-2\bm{\sigma})t)\frac{\Big{(}\exp(t(\tilde{\lambda}-\bm{\sigma}))-1\Big{)}^{2}}{(\tilde{\lambda}-\bm{\sigma})^{2}}\mathrm{d}t\Big{]}$
$\displaystyle=\mathbb{E}\Big{[}\int_{0}^{\infty}\exp(-(2\tilde{\lambda}+z-2\bm{\sigma})t)\frac{\Big{(}\exp(2t(\tilde{\lambda}-\bm{\sigma}))-2\exp(t(\tilde{\lambda}-\bm{\sigma}))+1\Big{)}}{(\tilde{\lambda}-\bm{\sigma})^{2}}\mathrm{d}t\Big{]}$
$\displaystyle=\frac{1}{z}\mathbb{E}\left[\frac{1}{(\tilde{\lambda}-\bm{\sigma})^{2}}\right]+\mathrm{Rem}_{1}.$
(2.25)

Note that $\mathrm{Rem}_{1}$ involves two terms and remains bounded when $z\to
0$ since

$\displaystyle\mathrm{Rem}_{1}$
$\displaystyle=-2\mathbb{E}\left[\frac{1}{(\tilde{\lambda}-\bm{\sigma})^{2}}\int_{0}^{\infty}\exp((-\tilde{\lambda}-z+\bm{\sigma})t)\right]+\mathbb{E}\left[\frac{1}{(\tilde{\lambda}-\bm{\sigma})^{2}}\int_{0}^{\infty}\exp((-2\tilde{\lambda}-z+2\bm{\sigma})t)\right]$
$\displaystyle=-2\mathbb{E}\left[\frac{1}{(\tilde{\lambda}-\bm{\sigma})^{2}(\tilde{\lambda}+z-\bm{\sigma})}\right]+\mathbb{E}\left[\frac{1}{(\tilde{\lambda}-\bm{\sigma})^{2}(2\tilde{\lambda}+z-2\bm{\sigma})}\right]$

Thus, the proof is complete on establishing that $\mathrm{Rem}$ remains
bounded as $z\rightarrow 0$.

We note that $\mathrm{Rem}$ may be expressed as the sum of three terms,
denoted as $\mathrm{Rem}_{T_{j}},j=1,\ldots 3$, where

$\displaystyle\mathrm{Rem}_{T_{1}}$
$\displaystyle=2\sqrt{\lambda}\rho(1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}})_{+}\frac{\sigma_{\star}^{2}}{4\lambda}\int_{0}^{\infty}\int_{0}^{t}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z)t+\tilde{\lambda}s_{1})\mathbb{E}_{x\sim\mu_{D}}\left[\frac{\exp(xs_{2})}{\tilde{\lambda}-x}\right]\mathbb{E}\Big{[}\exp\Big{(}\bm{\sigma}(2t-s_{1}-s_{2})\Big{)}\Big{]}\mathrm{d}s_{1}\mathrm{d}s_{2}\mathrm{d}t,$

$\mathrm{Rem}_{T_{2}}$ is the same as above except the roles of $s_{1}$ and
$s_{2}$ are reversed, and the third term is defined as below:

$\displaystyle\mathrm{Rem}_{T_{3}}=\frac{2\sigma_{\star}^{4}}{16\lambda}\int_{0}^{\infty}\int_{0}^{t}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z)t\mathbb{E}_{x_{1}\sim\mu_{D}}\left[\frac{\exp(x_{1}s_{1})}{(\tilde{\lambda}-x_{1})}\right]\mathbb{E}_{x_{2}\sim\mu_{D}}\left[\frac{\exp(x_{2}s_{2})}{(\tilde{\lambda}-x_{2})}\right]\mathbb{E}\Big{[}\exp\Big{(}\bm{\sigma}(2t-s_{1}-s_{2})\Big{)}\Big{]}\mathrm{d}s_{1}\mathrm{d}s_{2}\mathrm{d}t$

Now the support of $x,\bm{\sigma}$ is upper bounded by $\sigma_{\star}$ and
$2t-s_{1}-s_{2}\geq 0,s_{1},s_{2}\leq t$ in the range of integration. Thus, we
may upper bound $\mathrm{Rem}_{T_{1}}$ by

$\displaystyle Rem_{T_{1}}$ $\displaystyle\leq
2\sqrt{\lambda}\rho(1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}})_{+}\mathbb{E}_{(x,\bm{\sigma})\sim\mu_{D}^{\otimes
2}}\left[\frac{1}{\tilde{\lambda}-x}\int_{0}^{\infty}\int_{0}^{t}\int_{0}^{t}\exp(-(2\tilde{\lambda}+z-2\sigma_{\star})t+(\tilde{\lambda}-\sigma_{\star})t)ds_{1}ds_{2}dt\right]$
$\displaystyle=2\sqrt{\lambda}\rho(1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}})_{+}\mathbb{E}_{x\sim\mu_{D}}\left[\frac{1}{(\tilde{\lambda}-x)}\int_{0}^{\infty}t^{2}\exp(-(\tilde{\lambda}+z-\sigma_{\star})t)\right]$
$\displaystyle=2\sqrt{\lambda}\rho(1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}})_{+}\frac{\Gamma(3)}{(\tilde{\lambda}+z-\sigma_{\star})^{3}}\mathbb{E}_{x\sim\mu_{D}}\left[\frac{1}{(\tilde{\lambda}-x)}\right],$

which remains bounded when $z\rightarrow 0$. Similarly, $\mathrm{Rem}_{T_{2}}$
can be bounded. Applying a similar trick, $\mathrm{Rem}_{T_{3}}$ can be
bounded by

$\displaystyle\frac{2\sigma_{\star}^{4}}{16\lambda}\mathbb{E}_{(x_{1},x_{2})\sim\mu_{D}^{\otimes
2}}\left[\frac{1}{(\tilde{\lambda}-x_{1})(\tilde{\lambda}-x_{2})}\right]\int_{0}^{\infty}t^{2}\exp(-(2\tilde{\lambda}+z-2\sigma_{\star})t)dt=$
$\displaystyle\frac{2\sigma_{\star}^{4}}{16\lambda}\frac{\Gamma(3)}{(2\tilde{\lambda}+z-2\sigma_{\star})^{3}}\mathbb{E}_{(x_{1},x_{2})\sim\mu_{D}^{\otimes
2}}\left[\frac{1}{(\tilde{\lambda}-x_{1})(\tilde{\lambda}-x_{2})}\right],$

and this once again remains bounded as $z\rightarrow 0$. Putting everything
together and from (2), we have the desired result.

∎

We now turn to the proof of Theorem 1.2.

###### Proof of Theorem 1.2.

First, using (1.6), we have,

$\displaystyle\lim_{z\downarrow\sigma_{\star}}\mathcal{S}(z)=\frac{2}{\sigma_{\star}},\,\,\,\,\,\lim_{z\uparrow-\sigma_{\star}}\mathcal{S}(z)=-\frac{2}{\sigma_{\star}}.$

If $z_{*}$ is a real root of $z=\beta^{-1}\mathcal{S}(z/2)$, $y=2z_{*}$ is a
real root of the fixed point equation $2\beta y=\mathcal{S}(y)$. In turn, such
a root exists if and only if $2\beta\sigma_{*}<2/\sigma_{*}$, which
immediately gives us the desired conclusion.

Consider first the regime $2\tilde{\lambda}>s_{\beta}$. Observe that (2.5)
implies,

$\displaystyle\lim_{t\to\infty}\exp(-2\tilde{\lambda}t)g^{2}(t)=\lambda\rho^{2}\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}^{2}.$

Further, Lemma 2.3 implies that as $z\to 0$ along a sector,

$\displaystyle z\mathcal{L}_{\Phi}(2\tilde{\lambda}+z)\to
2\lambda^{2}\rho^{2}\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}^{2}\mathbb{E}\left[\frac{1}{(\tilde{\lambda}-\bm{\sigma})^{2}}\right]$

Thus Lemma 2.2 implies

$\displaystyle\lim_{z\to
0}z\mathcal{L}_{h}(2\tilde{\lambda}+z)=\frac{2\tilde{\lambda}}{2\tilde{\lambda}-\beta^{-1}\mathcal{S}(\tilde{\lambda})}\mathbb{E}\left[\frac{1}{(\tilde{\lambda}-\bm{\sigma})^{2}}\right]\lambda^{2}\rho^{2}\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}^{2}.$

Thus using [3, Lemma 7.2],

$\displaystyle\lim_{t\uparrow\infty}\exp(-2\tilde{\lambda}t)h(t)=\frac{2\tilde{\lambda}}{2\tilde{\lambda}-\beta^{-1}\mathcal{S}(\tilde{\lambda})}\mathbb{E}\left[\frac{1}{(\tilde{\lambda}-\bm{\sigma})^{2}}\right]\lambda^{2}\rho^{2}\Big{(}1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\Big{)}_{+}^{2}.$

Noting that $\rho>0$, we have,

$\displaystyle\lim_{t\uparrow\infty}\frac{g^{2}(t)}{h(t)}=\frac{2\tilde{\lambda}-\beta^{-1}\mathcal{S}(\tilde{\lambda})}{2\tilde{\lambda}\lambda}\left[\mathbb{E}\left(\frac{1}{(\tilde{\lambda}-\bm{\sigma})^{2}}\right)\right]^{-1}.$

To simplify this, first note that
$\mathcal{S}^{\prime}(z)=-\mathbb{E}\left(\frac{1}{(z-\bm{\sigma})^{2}}\right)$.
Calculating the exact value of this derivative we obtain that

$-\mathbb{E}\left(\frac{1}{(z-\bm{\sigma})^{2}}\right)=\frac{2\sqrt{z^{2}-\sigma_{\star}^{2}}-z}{\sigma_{\star}^{2}\sqrt{z^{2}-\sigma_{\star}^{2}}}.$

Equating these for $z=\tilde{\lambda}$ yields that

$\left[\mathbb{E}\left(\frac{1}{(\tilde{\lambda}-\sigma)^{2}}\right)\right]^{-1}=\lambda^{2}(1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}).$

Putting things together,

$\lim_{t\uparrow\infty}\frac{g^{2}(t)}{\lambda
h(t)}=\left(1-\frac{\beta^{-1}\mathcal{S}(\tilde{\lambda})}{2\tilde{\lambda}}\right)\left(1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\right).$

Now note that $\mathcal{S}(\tilde{\lambda})=1/\lambda$, so the final limit
becomes

$\lim_{t\rightarrow\infty}\frac{g^{2}(t)}{\lambda
h(t)}=\left(1-\frac{\beta^{-1}}{2\lambda(\lambda+\sigma_{\star}^{2}/4\lambda)}\right)\left(1-\frac{\sigma_{\star}^{2}}{4\lambda^{2}}\right).$

It remains to analyze the sub-critical regime $2\tilde{\lambda}<s_{\beta}$.
Using Lemma 2.2, we note that $\mathcal{L}_{h}(\cdot)$ has a simple pole at
$s_{\beta}$. Thus there exists $C_{1}>0$ such that

$\displaystyle\lim_{z\to 0}z\mathcal{L}_{h}(s_{\beta}+z)=C_{1}\neq 0.$ (2.26)

In turn, this implies

$\displaystyle\lim_{t\uparrow\infty}\exp(-s_{\beta}t)h(t)=C_{1}.$

This completes the proof in this sub-case, as
$\lim_{t\to\infty}g^{2}(t)/h(t)=0$ immediately in this case.

Finally, we focus on the case $\rho=0$. Lemma 2.1 implies that $g=0$ in this
case. On the other hand, in this case
$\Phi(t)=2\mathbb{E}[\bm{Y_{0}}^{2}]\mathbb{E}[\exp(2t\bm{\sigma})]$. Thus
using Lemma 2.2, we note that the leading asymptotics of $h(\cdot)$ is
determined by the pole at $s_{\beta}$. Specifically,
$\lim_{t\uparrow\infty}h(t)\neq 0$, which concludes the proof.

∎

##  3 Proof of Theorem 1.1

Setting $U=\mathbf{G}V\in\mathbb{R}^{N}$, the dynamics under the rotated
coordinate system $\mathbf{Y}_{t}:=\mathbf{G}\mathbf{X}_{t}$ can be expressed
as

$\displaystyle\mathrm{d}{Y_{t}^{i}}\,=U^{i}\langle
U,Y_{t}\rangle\mathrm{d}{t}\,+\sigma^{i}Y_{t}^{i}\mathrm{d}{t}\,-f^{\prime}\big{(}\|Y_{t}\|^{2}/N\big{)}Y_{t}^{i}\mathrm{d}{t}\,+\beta^{-1/2}\mathrm{d}{B_{t}^{i}}\,.$
(3.1)

Setting $u^{i}=\sqrt{N}U^{i}$, we note that Eqn. (3.1) reduces to

$\displaystyle\mathrm{d}{Y_{t}^{i}}\,=u^{i}R^{N}(t)\mathrm{d}{t}\,+\big{(}\sigma^{i}-f^{\prime}(K^{N}(u))\big{)}Y_{t}^{i}\mathrm{d}{t}\,+\beta^{-1/2}\mathrm{d}{B_{t}^{i}}\,.$
(3.2)

Utilizing this expression, one can verify that $Y_{t}^{i}$ takes the following
integral form

$\displaystyle Y_{t}^{i}$
$\displaystyle=\exp\left\\{\int_{0}^{t}\big{[}\sigma^{i}-f^{\prime}(K^{N}(s))\big{]}\mathrm{d}{s}\,\right\\}\left\\{Y_{0}^{i}+u^{i}\int_{0}^{t}\exp\left\\{-\int_{0}^{s}\big{[}\sigma^{i}-f^{\prime}(K^{N}(r))\big{]}\mathrm{d}{r}\,\right\\}R^{N}(s)\mathrm{d}{s}\,\right\\}$
(3.3)
$\displaystyle\quad+\beta^{-1/2}\int_{0}^{t}\exp\left\\{\int_{s}^{t}\big{[}\sigma^{i}-f^{\prime}(K^{N}(r))\big{]}\mathrm{d}{r}\,\right\\}\mathrm{d}{B^{i}_{s}}\,.$
(3.4)

We first re-express the solution of our SDE.

######  Lemma 3.1.

Define $F_{t}(K,\lambda)=f^{\prime}(K(t,t))-\lambda$. Then we have,

$\displaystyle Y_{t}^{i}=$ $\displaystyle
Y_{0}^{i}\exp\Big{[}-\int_{0}^{t}F_{s}(K^{N},\sigma_{i})\mathrm{d}{s}\,\Big{]}+\int_{0}^{t}\exp\Big{[}-\int_{s}^{t}F_{s_{1}}(K^{N},\sigma_{i})\mathrm{d}{s_{1}}\,\Big{]}u_{i}R^{N}(s)\mathrm{d}{s}\,+\beta^{-1/2}B_{t}^{i}-$
$\displaystyle\beta^{-1/2}\int_{0}^{t}B_{s}^{i}F_{s}(K^{N},\sigma_{i})\exp\Big{[}-\int_{s}^{t}F_{s_{1}}(K^{N},\sigma_{i})\mathrm{d}{s_{1}}\,\Big{]}\mathrm{d}{s}\,.$
(3.5)

###### Proof.

The proof follows by an application of the integration by parts formula
$\int_{0}^{t}f_{s}\mathrm{d}{B_{s}}\,=f_{t}B_{t}-\int_{0}^{t}B_{s}f_{s}^{\prime}\mathrm{d}{s}\,$
on the solution (3.3). ∎

######  Definition 3.2.

We define the empirical measure

$\displaystyle\nu=\frac{1}{N}\sum_{i}\delta_{Y_{0}^{i},u^{i},\sigma^{i},B_{\bullet}^{i}},$
(3.6)

where the fourth marginal is an empirical distribution on the path space
$\mathcal{C}[0,T]$.

Consider the following collections of functions, with domain space
$\mathbb{R}^{3}\times\mathcal{C}[0,T]$ and range space one of
$\mathcal{C}[0,T]^{j}$ for $j=1,2,3$:

$\mathcal{F}_{j}\subset\\{f:\mathbb{R}^{3}\times\mathcal{C}[0,T]\rightarrow\mathcal{C}([0,T]^{j})\\},\,\,j=1,2,3,\,\,\text{with}$
$\displaystyle\mathcal{F}_{1}=\\{f_{j},j=1\ldots
5:f_{1}(Y_{0},u,\sigma,B_{\bullet})(w)=uY_{0}\exp(w\sigma),f_{2}(\cdot)(w)=u^{2}\exp(w\sigma),f_{3}(\cdot)(t)=uB_{t},$
$\displaystyle f_{4}(\cdot)(w)=Y_{0}^{2}\exp(w\sigma),f_{5}(\cdot)(w)=\sigma
Y_{0}\exp(w\sigma)\\},$
$\displaystyle\mathcal{F}_{2}=\\{f_{6},\ldots,f_{9}:f_{6}(Y_{0},u,\sigma,B_{\bullet})(t,w)=uB_{t}\exp(w\sigma),f_{7}(\cdot)(t,w)=uB_{t}\sigma\exp(w\sigma),$
$\displaystyle
f_{8}(\cdot)(t,s)=B_{t}B_{s},f_{9}(\cdot)(s,w)=Y_{0}B_{s}\exp(w\sigma)\\},$
$\displaystyle\mathcal{F}_{3}=\\{f_{10},f_{11},f_{12}:f_{10}(Y,u,\sigma,B_{\bullet})(s_{1},s_{2},w)=B_{s_{1}}B_{s_{2}}\exp(w\sigma),$
$\displaystyle f_{11}(\cdot)(s_{1},s_{2},w)=\sigma
B_{s_{1}}B_{s_{2}}\exp(w\sigma),f_{12}(\cdot)(s_{1},s_{2},w)=\sigma^{2}B_{s_{1}}B_{s_{2}}\exp(w\sigma)\\}$

Finally, define

$\displaystyle\mathcal{F}=\mathcal{F}_{1}\cup\mathcal{F}_{2}\cup\mathcal{F}_{3}.$
(3.7)

All functions in $\mathcal{F}$ have the same domain. To simplify notation, for
elements in $\mathcal{F}_{1}$, we use $f(\cdot)(w)$ to mean
$f(Y_{0},u,\sigma,B_{\bullet})$ evaluated at $w$ for a generic point
$(Y_{0},u,\sigma,B_{\bullet})\in\mathbb{R}^{3}\times\mathcal{C}[0,T]$.
Similarly for elements in $\mathcal{F}_{2}$ and $\mathcal{F}_{3}$.

Define $\mathcal{C}_{N}=\\{\int f\mathrm{d}{\nu}\,:f\in\mathcal{F}\\}$ and
note that for $f\in\mathcal{F}_{j}$, $\int
fd\nu\in\mathcal{C}[0,T]^{j},j=1,2,3.$ We introduce the following convention:
for a discrete set $S=\\{s_{1},\ldots,s_{k}\\}$ and a function $f(\cdot)$,
$f(S)=f(s_{1},\ldots,s_{k})$. The next result establishes that for all $N\geq
1$, $(R_{N},K_{N})$ depend on $\nu$ through the statistics $\mathcal{C}_{N}$.

######  Lemma 3.3.

There exist functions

$\displaystyle\Phi^{(1)}:\mathcal{C}[0,T]\times\mathcal{C}([0,T]^{2})\times\mathcal{C}[0,T]^{|\mathcal{F}_{1}|}\times\mathcal{C}([0,T]^{2})^{|\mathcal{F}_{2}|}\times\mathcal{C}([0,T]^{3})^{|\mathcal{F}_{3}|}\to\mathcal{C}[0,T],$
$\displaystyle\Phi^{(2)}:\mathcal{C}[0,T]\times\mathcal{C}([0,T]^{2})\times\mathcal{C}[0,T]^{|\mathcal{F}_{1}|}\times\mathcal{C}([0,T]^{2})^{|\mathcal{F}_{2}|}\times\mathcal{C}([0,T]^{3})^{|\mathcal{F}_{3}|}\to\mathcal{C}([0,T]^{2}),$

such that

$\displaystyle
R^{N}=\Phi^{(1)}(R^{N},K^{N},\mathcal{C}_{N}),\,\,K^{N}=\Phi^{(2)}(R^{N},K^{N},\mathcal{C}_{N}).$
(3.8)

###### Proof.

To this end, we combine (3.5) with the definition of $R_{N}$ to get a fixed
point equation

$\displaystyle R^{N}(t)=$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}u^{i}Y_{0}^{i}\exp\Big{[}-\int_{0}^{t}(f^{\prime}(K^{N}(s))-\sigma^{i})\mathrm{d}{s}\,\Big{]}$
$\displaystyle+\frac{1}{N}\sum_{i=1}^{N}(u^{i})^{2}\int_{0}^{t}\exp\Big{[}-\int_{s}^{t}(f^{\prime}(K^{N}(s_{1}))-\sigma^{i})\mathrm{d}{s_{1}}\,\Big{]}R^{N}(s)\mathrm{d}{s}\,$
$\displaystyle+\frac{\beta^{-1/2}}{N}\sum_{i=1}^{N}u^{i}B_{t}^{i}-\frac{1}{N}\sum_{i=1}^{N}u^{i}\int_{0}^{t}B_{s}^{i}(f^{\prime}(K^{N}(s))-\sigma^{i})\exp\Big{[}-\int_{s}^{t}(f^{\prime}(K^{N}(s_{1}))-\sigma^{i})\mathrm{d}{s_{1}}\,\Big{]}\mathrm{d}{s}\,.$
(3.9)

This implicitly specifies the function $\Phi^{(1)}$. The corresponding
equation for $K^{N}$ is more involved. To track this representation
systematically, we recall the representation of the solution $Y_{t}^{i}$ from
(3.5), and denote $Y_{t}^{i}:=\sum_{j=1}^{5}T_{j}^{i}(t)$, where the
$T_{j}^{i}(t)$ represent the respective terms in the RHS of (3.5). Now, recall
that $K^{N}(t,s)=\frac{1}{N}\sum_{i=1}^{N}Y_{t}^{i}Y_{s}^{i}$, and therefore

$\displaystyle
K^{N}(t,s)=\sum_{j=1}^{5}\frac{1}{N}\sum_{i=1}^{N}T_{j}^{i}(t)T_{j}^{i}(s)+\sum_{1\leq
j_{1}<j_{2}\leq
5}\Big{[}\frac{1}{N}\sum_{i=1}^{N}T_{j_{1}}^{i}(t)T_{j_{2}}^{i}(s)+\frac{1}{N}\sum_{i=1}^{N}T_{j_{1}}^{i}(s)T_{j_{2}}^{i}(t)\Big{]}.$
(3.10)

We argue that the RHS above is a continuous function of $R^{N}$, $K^{N}$ and
$\nu$. To this end, we record these terms explicitly. Define

$\displaystyle
H_{\tau}^{\theta}(K)=\exp\Big{[}-\int_{\tau}^{\theta}f^{\prime}(K(\xi))\mathrm{d}{\xi}\,\Big{]},\,\,\,DH_{\tau}^{\theta}=\frac{\mathrm{d}{H_{\tau}^{\theta}(K)}\,}{\mathrm{d}{\tau}\,}=f^{\prime}(K(\tau))\exp\Big{[}-\int_{\tau}^{\theta}f^{\prime}(K(\xi)\mathrm{d}{\xi}\,\Big{]}.$
(3.11)

First, we present the “diagonal” terms.

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}T_{1}^{i}(t)T_{1}^{i}(s)$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}(Y_{0}^{i})^{2}\exp(\sigma^{i}(t+s))H_{0}^{t}(K^{N})H_{0}^{s}(K^{N}),$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}T_{2}^{i}(t)T_{2}^{i}(s)$
$\displaystyle=\int_{0}^{t}\int_{0}^{s}\frac{1}{N}\sum_{i=1}^{N}(u^{i})^{2}\exp(\sigma^{i}(t+s-s_{1}-s_{2}))H_{s_{1}}^{t}(K^{N})H_{s_{2}}^{s}(K^{N})R^{N}(s_{1})R^{N}(s_{2})\mathrm{d}{s_{1}}\,\mathrm{d}{s_{2}}\,,$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}T_{3}^{i}(t)T_{3}^{i}(s)$
$\displaystyle=\frac{\beta^{-1}}{N}\sum_{i=1}^{N}B_{t}^{i}B_{s}^{i},$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}T_{4}^{i}(t)T_{4}^{i}(s)$
$\displaystyle=\frac{\beta^{-1}}{N}\sum_{i=1}^{N}\int_{0}^{t}\int_{0}^{s}B_{s_{1}}^{i}B_{s_{2}}^{i}DH_{s_{1}}^{t}(K^{N})DH_{s_{2}}^{t}(K^{N})\exp(\sigma^{i}(t+s-s_{1}-s_{2}))\mathrm{d}{s_{1}}\,\mathrm{d}{s_{2}}\,,$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}T_{5}^{i}(t)T_{5}^{i}(s)$
$\displaystyle=\frac{\beta^{-1}}{N}\sum_{i=1}^{N}(\sigma^{i})^{2}\int_{0}^{t}\int_{0}^{s}B_{s_{1}}^{i}B_{s_{2}}^{i}\exp(\sigma^{i}(t+s-s_{1}-s_{2}))\mathrm{d}{s_{1}}\,\mathrm{d}{s_{2}}\,.$

Next, we present the “off-diagonal” terms.

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}[T_{1}^{i}(t)T_{2}^{i}(s)+T_{1}^{i}(s)T_{2}^{i}(t)]=$
$\displaystyle
H_{0}^{t}(K^{N})\int_{0}^{s}\frac{1}{N}\sum_{i=1}^{N}u^{i}Y_{0}^{i}\exp(\sigma^{i}(t+s-s_{1}))H_{s_{1}}^{s}(K^{N})R^{N}(s_{1})\mathrm{d}{s_{1}}\,$
$\displaystyle+$ $\displaystyle
H_{0}^{s}(K^{N})\int_{0}^{t}\frac{1}{N}\sum_{i=1}^{N}u^{i}Y_{0}^{i}\exp(\sigma^{i}(t+s-s_{1}))H_{s_{1}}^{t}(K^{N})R^{N}(s_{1})\mathrm{d}{s_{1}}\,,$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}[T_{1}^{i}(t)T_{3}^{i}(s)+T_{1}^{i}(s)T_{3}^{i}(t)]=$
$\displaystyle
H_{0}^{t}(K^{N})\frac{\beta^{-1/2}}{N}\sum_{i=1}^{N}Y_{0}^{i}B_{s}^{i}\exp(\sigma^{i}t)+H_{0}^{s}(K^{N})\frac{\beta^{-1/2}}{N}\sum_{i=1}^{N}Y_{0}^{i}B_{t}^{i}\exp(\sigma^{i}s),$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}[T_{1}^{i}(t)T_{4}^{i}(s)+T_{1}^{i}(s)T_{4}^{i}(t)]=$
$\displaystyle-\beta^{-1/2}H_{0}^{t}(K^{N})\int_{0}^{s}\frac{1}{N}\sum_{i=1}^{N}Y_{0}^{i}B_{s_{1}}^{i}\exp(\sigma^{i}(t+s-s_{1}))DH_{s_{1}}^{s}(K^{N})\mathrm{d}{s_{1}}\,$
$\displaystyle-\beta^{-1/2}H_{0}^{s}(K^{N})\int_{0}^{t}\frac{1}{N}\sum_{i=1}^{N}Y_{0}^{i}B^{i}_{s_{1}}\exp(\sigma^{i}(t+s-s_{1}))DH_{s_{1}}^{t}(K^{N})\mathrm{d}{s_{1}}\,$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}[T_{1}^{i}(t)T_{5}^{i}(s)+T_{1}^{i}(s)T_{5}^{i}(t)]=$
$\displaystyle\beta^{-1/2}H_{0}^{t}(K^{N})\int_{0}^{s}\frac{1}{N}\sum_{i=1}^{N}\sigma^{i}Y_{0}^{i}\exp(\sigma^{i}(t+s-s_{1}))\mathrm{d}{s_{1}}\,$
$\displaystyle+\beta^{-1/2}H_{0}^{s}(K^{N})\int_{0}^{t}\frac{1}{N}\sum_{i=1}^{N}\sigma^{i}Y_{0}^{i}\exp(\sigma^{i}(t+s-s_{1}))\mathrm{d}{s_{1}}\,,$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}[T_{2}^{i}(t)T_{3}^{i}(s)+T_{2}^{i}(s)T_{3}^{i}(t)]=$
$\displaystyle\beta^{-1/2}\int_{0}^{t}\frac{1}{N}\sum_{i=1}^{N}u^{i}B^{i}_{s}\exp(\sigma^{i}(t-s_{1}))H_{s_{1}}^{t}(K^{N})R^{N}(s_{1})\mathrm{d}{s_{1}}\,$
$\displaystyle+\beta^{-1/2}\int_{0}^{s}\frac{1}{N}\sum_{i=1}^{N}u^{i}B^{i}_{t}\exp(\sigma^{i}(s-s_{1}))H_{s_{1}}^{t}(K^{N})R^{N}(s_{1})\mathrm{d}{s_{1}}\,$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}[T_{2}^{i}(t)T_{4}^{i}(s)+T_{2}^{i}(s)T_{4}^{i}(t)]=$
$\displaystyle-\beta^{-1/2}\int_{0}^{t}\int_{0}^{s}\frac{1}{N}\sum_{i=1}^{N}u^{i}B^{i}_{s_{2}}\exp(\sigma^{i}(t+s-s_{1}-s_{2}))H^{t}_{s_{1}}(K^{N})R^{N}(s_{1})DH_{s_{2}}^{s}(K^{N})\mathrm{d}{s_{1}}\,\mathrm{d}{s_{2}}\,$
$\displaystyle-$
$\displaystyle\beta^{-1/2}\int_{0}^{t}\int_{0}^{s}\frac{1}{N}\sum_{i=1}^{N}u^{i}B^{i}_{s_{1}}\exp(\sigma^{i}(t+s-s_{1}-s_{2}))H^{s}_{s_{2}}(K^{N})R^{N}(s_{2})DH_{s_{1}}^{t}(K^{N})\mathrm{d}{s_{1}}\,\mathrm{d}{s_{2}}\,,$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}[T_{2}^{i}(t)T_{5}^{i}(s)+T_{2}^{i}(s)T_{5}^{i}(t)]=$
$\displaystyle\beta^{-1/2}\int_{0}^{t}\int_{0}^{s}\frac{1}{N}\sum_{i=1}^{N}u^{i}\sigma^{i}B^{i}_{s_{2}}\exp(\sigma^{i}(t+s-s_{1}-s_{2}))H^{t}_{s_{1}}(K^{N})R^{N}(s_{1})\mathrm{d}{s_{1}}\,\mathrm{d}{s_{2}}\,$
$\displaystyle+$
$\displaystyle\beta^{-1/2}\int_{0}^{t}\int_{0}^{s}\frac{1}{N}\sum_{i=1}^{N}u^{i}\sigma^{i}B^{i}_{s_{1}}\exp(\sigma^{i}(t+s-s_{1}-s_{2}))H^{s}_{s_{2}}(K^{N})R^{N}(s_{2})\mathrm{d}{s_{1}}\,\mathrm{d}{s_{2}}\,,$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}[T_{3}^{i}(t)T_{4}^{i}(s)+T_{3}^{i}(s)T_{4}^{i}(t)]=$
$\displaystyle-\frac{\beta^{-1}}{N}\sum_{i=1}^{N}\int_{0}^{s}B_{t}^{i}B_{s_{1}}^{i}\exp(\sigma^{i}(s-s_{1}))DH_{s_{1}}^{s}(K^{N})\mathrm{d}{s_{1}}\,$
$\displaystyle-\frac{\beta^{-1}}{N}\sum_{i=1}^{N}\int_{0}^{t}B_{s}^{i}B_{s_{1}}^{i}\exp(\sigma^{i}(t-s_{1}))DH_{s_{1}}^{t}(K^{N})\mathrm{d}{s_{1}}\,$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}[T_{3}^{i}(t)T_{5}^{i}(s)+T_{3}^{i}(s)T_{5}^{i}(t)]=$
$\displaystyle\beta^{-1}\int_{0}^{s}\frac{1}{N}\sum_{i=1}^{N}\sigma^{i}B_{t}^{i}B_{s_{1}}^{i}\exp(\sigma^{i}(s-s_{1}))\mathrm{d}{s_{1}}\,+\beta^{-1}\int_{0}^{t}\frac{1}{N}\sum_{i=1}^{N}\sigma^{i}B_{s}^{i}B^{i}_{s_{1}}\exp(\sigma^{i}(t-s_{1}))\mathrm{d}{s_{1}}\,,$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}[T_{4}^{i}(t)T_{5}^{i}(s)+T_{4}^{i}(s)T_{5}^{i}(t)]=$
$\displaystyle-\frac{\beta^{-1}}{N}\sum_{i=1}^{N}\sigma^{i}\int_{0}^{t}\int_{0}^{s}B_{s_{1}}^{i}B_{s_{2}}^{i}\exp(\sigma^{i}(t+s-s_{1}-s_{2}))DH^{t}_{s_{1}}(K^{N})\mathrm{d}{s_{1}}\,\mathrm{d}{s_{2}}\,$
$\displaystyle-\frac{\beta^{-1}}{N}\sum_{i=1}^{N}\sigma^{i}\int_{0}^{t}\int_{0}^{s}B_{s_{1}}^{i}B_{s_{2}}^{i}\exp(\sigma^{i}(t+s-s_{1}-s_{2}))DH^{s}_{s_{2}}(K^{N})\mathrm{d}{s_{1}}\,\mathrm{d}{s_{2}}\,.$

This specifies the function $\Phi^{(2)}$ implicitly.

∎

Given Lemma 3.3, we next establish that $R^{N}$, $K^{N}$ are, in fact,
functions of the low-dimensional statistics $\mathcal{C}_{N}$.

######  Lemma 3.4.

There exist functions

$\displaystyle\Psi^{(1)}:\mathcal{C}[0,T]^{|\mathcal{F}_{1}|}\times\mathcal{C}([0,T]^{2})^{|\mathcal{F}_{2}|}\times\mathcal{C}([0,T]^{3})^{|\mathcal{F}_{3}|}\to\mathcal{C}[0,T],$
$\displaystyle\Psi^{(2)}:\mathcal{C}[0,T]^{|\mathcal{F}_{1}|}\times\mathcal{C}([0,T]^{2})^{|\mathcal{F}_{2}|}\times\mathcal{C}([0,T]^{3})^{|\mathcal{F}_{3}|}\to\mathcal{C}([0,T]^{2})$

such that

$R^{N}=\Psi^{(1)}(\mathcal{C}_{N}),\,\,K^{N}=\Psi^{(2)}(\mathcal{C}_{N}).$

To this end, our main strategy is to apply a Picard iteration scheme on the
fixed point equations (3.8). We start with some initial guess $R_{0}^{N}$,
$K_{0}^{N}$, and carry out the iterative updates

$\displaystyle
R_{m+1}^{N}=\Phi^{(1)}(R_{m}^{N},K_{m}^{N},\mathcal{C}_{N}),\,\,\,\,K_{m+1}^{N}=\Phi^{(2)}(R_{m}^{N},K_{m}^{N},\mathcal{C}_{N}).$
(3.12)

We will show that this iteration system is contractive, and thus identify
$R^{N}$, $K^{N}$ as the unique fixed points of this system. In this endeavor,
we will utilize the precise form of the functions $\Phi^{(1)}$, $\Phi^{(2)}$
as described in the proof of Lemma 3.3. First, we need some preliminary
estimates.

######  Lemma 3.5.

Recall that $f^{\prime}$, defined in (1.2), is non-negative and Lipschitz.
Then we have that

  * (i)

for any $m,N\geq 1$, $0\leq H^{\theta}_{\tau}(K_{m}^{N})\leq 1$.

  * (ii)

For any $0\leq t\leq T$, $N,m\geq 1$,
$\int_{0}^{t}|DH_{u}^{t}(K_{m}^{N})|\mathrm{d}{u}\,\leq 1$.

  * (iii)

For any $\theta\leq T$,

$\displaystyle\sup_{\tau\leq\theta}|H^{\theta}_{\tau}(K_{m+1}^{N})-H^{\theta}_{\tau}(K_{m}^{N})|\leq\|f^{\prime}\|_{L}\int_{0}^{\theta}|K_{m+1}^{N}(s,s)-K_{m}^{N}(s,s)|\mathrm{d}{s}\,.$
(3.13)

  * (iv)

For any $N,m\geq 1$, $0\leq\tau\leq\theta\leq T$,

$\displaystyle|DH^{\theta}_{\tau}(K_{m+1}^{N})-DH^{\theta}_{\tau}(K_{m}^{N})|$
$\displaystyle\leq\|f^{\prime}\|_{L}\Big{[}|K_{m+1}^{N}(\tau,\tau)-K_{m}^{N}(\tau,\tau)|+(DH^{\theta}_{\tau}(K_{m+1}^{N})$
$\displaystyle+DH^{\theta}_{\tau}(K_{m}^{N}))\int_{0}^{\theta}|K_{m+1}^{N}(s,s)-K_{m}^{N}(s,s)|\mathrm{d}{s}\,\Big{]}.$

  * (v)

With probability 1, there exist $C_{0}$, $C_{1}$ (possibly random), depending
on $T$ such that

$\displaystyle\sup_{m\geq 1}\|R_{m}^{N}\|_{\infty}\leq C_{0}\exp{(C_{1}T)}.$
(3.14)

###### Proof of Lemma 3.5.

The parts (i)-(iv) are directly adapted from [3], and are just collected here
for the convenience of the reader. We prove (v). Note that

$\displaystyle
R_{m+1}^{N}(t)=H_{0}^{t}(K_{m}^{N})\frac{1}{N}\sum_{i=1}^{N}u^{i}Y_{0}^{i}\exp{(\sigma^{i}t)}+\frac{1}{N}\sum_{i=1}^{N}(u^{i})^{2}\int_{0}^{t}\exp{(\sigma^{i}(t-s))}R_{m}^{N}(s)H_{s}^{t}(K_{m}^{N})\mathrm{d}{s}\,$
$\displaystyle+\frac{\beta^{-1/2}}{N}\sum_{i=1}^{N}u^{i}B^{i}_{t}-\frac{1}{N}\sum_{i=1}^{N}u^{i}\int_{0}^{t}B^{i}_{s}\exp{(\sigma^{i}(t-s))}DH_{s}^{t}(K_{m}^{N})\mathrm{d}{s}\,+\frac{1}{N}\sum_{i=1}^{N}u^{i}\sigma^{i}\int_{0}^{t}B_{s}^{i}\exp(\sigma^{i}(t-s))H_{s}^{t}(K_{m}^{N})\mathrm{d}{s}\,.$

This implies

$\displaystyle|R_{m+1}^{N}(t)|\leq$
$\displaystyle|H_{0}^{t}(K_{m}^{N})|\cdot|\frac{1}{N}\sum_{i=1}^{N}u^{i}Y_{0}^{i}\exp(\sigma^{i}t)|+\int_{0}^{t}|\frac{1}{N}\sum_{i=1}^{N}(u^{i})^{2}\exp(\sigma^{i}(t-s))|\cdot|H_{s}^{t}(K_{(m)}^{N}|\cdot|R_{(m)}^{N}(s)|\mathrm{d}{s}\,+\frac{\beta^{-1/2}}{N}|\sum_{i=1}^{N}u^{i}B_{t}^{i}|$
$\displaystyle+\int_{0}^{t}|\frac{1}{N}\sum_{i=1}^{N}u^{i}B_{s}^{i}\exp(\sigma^{i}(t-s))|\cdot|DH_{s}^{t}(K_{(m)}^{N})|\mathrm{d}{s}\,+\int_{0}^{t}|\frac{1}{N}\sum_{i=1}^{N}u^{i}\sigma^{i}B_{s}^{i}\exp(\sigma^{i}(t-s))|\cdot|H_{s}^{t}(K_{(m)}^{N})|\mathrm{d}{s}\,.$

Thus there exists $C_{0},C_{1}>0$ (possibly random), independent of $m$, such
that

$\displaystyle\sup_{0\leq t\leq T}|R_{m+1}^{N}(t)|\leq
C_{0}+C_{1}\int_{0}^{T}|R_{m}^{N}(s)|\mathrm{d}{s}\,.$

Iterating this bound in $m$, we obtain

$\displaystyle\sup_{0\leq t\leq T}|R_{m+1}^{N}(t)|\leq C_{0}\exp(C_{1}T).$

This completes the proof. ∎

Armed with Lemma 3.5, we turn to a proof of Lemma 3.4.

###### Proof of Lemma 3.4.

First observe that for any $m\geq 1$,(3.9) implies

$\displaystyle R_{m+1}^{N}(t)-R_{m}^{N}(t)=$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}u^{i}Y_{0}^{i}\exp(\sigma^{i}t)(H_{0}^{t}(K_{m}^{N})-H_{0}^{t}(K_{m-1}^{N}))$
$\displaystyle+$
$\displaystyle\int_{0}^{t}\frac{1}{N}\sum_{i=1}^{N}(u^{i})^{2}\exp(\sigma^{i}(t-s))(R_{m}^{N}(s)H^{t}_{s}(K_{m}^{N})-R_{m-1}^{N}(s)H^{t}_{s}(K_{m-1}^{N}))\mathrm{d}{s}\,$
$\displaystyle-$
$\displaystyle\int_{0}^{t}\frac{1}{N}\sum_{i=1}^{N}u^{i}B_{s}^{i}\exp(\sigma^{i}(t-s))(DH_{s}^{t}(K_{m}^{N})-DH_{s}^{t}(K_{m-1}^{N}))\mathrm{d}{s}\,$
$\displaystyle+$
$\displaystyle\int_{0}^{t}\frac{1}{N}\sum_{i=1}^{N}u^{i}\sigma^{i}B_{s}^{i}\exp(\sigma^{i}(t-s))(H_{s}^{t}(K_{m}^{N})-H_{s}^{t}(K_{m-1}^{N}))\mathrm{d}{s}\,.$

This implies, using Lemma 3.5, there exists a constant $C>0$ such that

$\displaystyle\sup_{0\leq t\leq T}|R_{m+1}^{N}(t)-R_{m}^{N}(t)|$
$\displaystyle\leq
C\Big{[}\int_{0}^{T}|K_{m}^{N}(\tau,\tau)-K_{m-1}^{N}(\tau,\tau)|\mathrm{d}{\tau}\,+\sup_{[0,T]}\int_{0}^{t}|R_{m}^{N}(s)H_{s}^{t}(K_{m}^{N})-R_{m-1}^{N}(s)H_{s}^{t}(K_{m-1}^{N})|\mathrm{d}{s}\,\Big{]}.$
(3.15)

To control the second term, we observe,

$\displaystyle|R_{m}^{N}(s)H_{s}^{t}(K_{m}^{N})-R_{m-1}^{N}(s)H_{s}^{t}(K_{m-1}^{N})|$
$\displaystyle\leq|R_{m}^{N}(s)|\cdot|H_{s}^{t}(K_{m}^{N})-H_{s}^{t}(K_{m-1}^{N})|+|H_{s}^{t}(K_{m-1}^{N})|\cdot|R_{m}^{N}(s)-R_{m-1}^{N}(s)|,$

which directly implies

$\displaystyle\sup_{[0,T]}\int_{0}^{t}|R_{m}^{N}(s)H_{s}^{t}(K_{m}^{N})-R_{m-1}^{N}(s)H_{s}^{t}(K_{m-1}^{N})|\mathrm{d}{s}\,$
$\displaystyle\lesssim\int_{0}^{T}|K_{m}^{N}(s,s)-K_{m-1}^{N}(s,s)|\mathrm{d}{s}\,+\int_{0}^{T}|R_{m}^{N}(s)-R_{m-1}^{N}(s)|\mathrm{d}{s}\,.$

Plugging this back into (3.15), there exists $C>0$ (independent of $m$) such
that

$\displaystyle\|R_{m+1}^{N}-R_{m}^{N}\|_{\infty}$
$\displaystyle:=\sup_{t\in[0,T]}|R_{m+1}^{N}(t)-R_{m}^{N}(t)|$
$\displaystyle\leq
C\Big{[}\int_{0}^{T}|K_{m}^{N}(s,s)-K_{m-1}^{N}(s,s)|\mathrm{d}{s}\,+\int_{0}^{T}|R_{m}^{N}(s)-R_{m-1}^{N}(s)|\mathrm{d}{s}\,\Big{]}.$

Analyzing the iterative update equation for $K_{m}^{N}$, one can derive a
similar bound.

$\displaystyle\|K_{m+1}^{N}-K_{m}^{N}\|_{\infty}$
$\displaystyle:=\sup_{s,t\in[0,T]^{2}}|K_{m}^{N}(s,t)-K_{m-1}^{N}(s,t)|$
$\displaystyle\leq
C\Big{[}\int_{0}^{T}|K_{m}^{N}(s,s)-K_{m-1}^{N}(s,s)|\mathrm{d}{s}\,+\int_{0}^{T}|R_{m}^{N}(s)-R_{m-1}^{N}(s)|\mathrm{d}{s}\,\Big{]}.$
(3.16)

Iterating these bounds, it follows that there exists $C_{1}>0$ such that

$\displaystyle\max\\{\|R_{m+1}^{N}-R_{m}^{N}\|_{\infty},\|K_{m+1}^{N}-K_{m}^{N}\|_{\infty}\\}\leq
2^{m}C^{m}C_{1}\frac{T^{m}}{m!}.$

Thus the iteration is contractive, and the fixed point system (3.8) has a
unique fixed point for each $N\geq 1$. This completes the proof. ∎

Finally, we will establish that $\Psi^{(1)},\Psi^{(2)}$ derived in Lemma 3.4
are continuous functions. To this end, recall that

$\displaystyle\Psi^{(1)}:\mathcal{C}[0,T]^{|\mathcal{F}_{1}|}\times\mathcal{C}([0,T]^{2})^{|\mathcal{F}_{2}|}\times\mathcal{C}([0,T]^{3})^{|\mathcal{F}_{3}|}\to\mathcal{C}[0,T],$
$\displaystyle\Psi^{(2)}:\mathcal{C}[0,T]^{|\mathcal{F}_{1}|}\times\mathcal{C}([0,T]^{2})^{|\mathcal{F}_{2}|}\times\mathcal{C}([0,T]^{3})^{|\mathcal{F}_{3}|}\to\mathcal{C}([0,T]^{2})$

We equip $\mathcal{C}([0,T]^{j})$, $j=1,2,3$, with the sup-norm topology.
Further, we equip
$\mathcal{C}[0,T]^{|\mathcal{F}_{1}|}\times\mathcal{C}([0,T]^{2})^{|\mathcal{F}_{2}|}\times\mathcal{C}([0,T]^{3})^{|\mathcal{F}_{3}|}$
with the product topology. With this notion of convergence, we can establish
the following continuity properties of $\Psi^{(1)}$ and $\Psi^{(2)}$.

######  Lemma 3.6.

The maps $\mathscr{C}\mapsto\Psi^{(1)}(\mathscr{C})$ and
$\mathscr{C}\mapsto\Psi^{(2)}(\mathscr{C})$ are continuous.

###### Proof.

We measure the discrepancy between $\mathscr{C}$ and $\tilde{\mathscr{C}}$
using the uniform topology on this space, and denote
$\mathrm{d}(\mathscr{C},\tilde{\mathscr{C}})=\|\mathscr{C}-\tilde{\mathscr{C}}\|_{\infty}$.
Define $R,K,\tilde{R}$, $\tilde{K}$ via the fixed point equations

$\displaystyle
R=\Phi^{(1)}(R,K,\mathscr{C}),\,\,\,K=\Phi^{(2)}(R,K,\mathscr{C}),$
$\displaystyle\tilde{R}=\Phi^{(1)}(\tilde{R},\tilde{K},\tilde{\mathscr{C}}),\,\,\,\tilde{K}=\Phi^{(2)}(\tilde{R},\tilde{K},\tilde{\mathscr{C}}).$

Observe that

$\displaystyle\|R-\tilde{R}\|_{\infty}$
$\displaystyle=\|\Phi^{(1)}(R,K,\mathscr{C})-\Phi^{(1)}(\tilde{R},\tilde{K},\tilde{\mathscr{C}})\|_{\infty}$
$\displaystyle\leq\|\Phi^{(1)}(R,K,\mathscr{C})-\Phi^{(1)}(\tilde{R},\tilde{K},\mathscr{C})\|_{\infty}+\|\Phi^{(1)}(\tilde{R},\tilde{K},\mathscr{C})-\Phi^{(1)}(\tilde{R},\tilde{K},\tilde{\mathscr{C}})\|_{\infty}.$

Controlling the second term, we note from (3.9) that

$\displaystyle\|\Phi^{(1)}(\tilde{R},\tilde{K},\mathscr{C})-\Phi^{(1)}(\tilde{R},\tilde{K},\tilde{\mathscr{C}})\|_{\infty}\leq\|\mathscr{C}-\tilde{\mathscr{C}}\|_{\infty}.$

On the other hand, using the same argument as in the proof of Lemma 3.4, we
have that

$\displaystyle\|\Phi^{(1)}(R,K,\mathscr{C})-\Phi^{(1)}(\tilde{R},\tilde{K},\mathscr{C})\|_{\infty}\leq
C\Big{[}\int_{0}^{T}|K(s,s)-\tilde{K}(s,s)|\mathrm{d}{s}\,+\int_{0}^{T}|R(s)-\tilde{R}(s)|\mathrm{d}{s}\,\Big{]}.$

Combining, we obtain,

$\displaystyle\|R-\tilde{R}\|_{\infty}\leq\|\mathscr{C}-\tilde{\mathscr{C}}\|_{\infty}+C\Big{[}\int_{0}^{T}|K(s,s)-\tilde{K}(s,s)|\mathrm{d}{s}\,+\int_{0}^{T}|R(s)-\tilde{R}(s)|\mathrm{d}{s}\,\Big{]}.$

Thus there exists $C^{\prime}>0$ such that

$\displaystyle\|R-\tilde{R}\|_{\infty}\leq\|\mathscr{C}-\tilde{\mathscr{C}}\|_{\infty}\exp(C^{\prime}T).$

Thus $\mathscr{C}\to\Psi^{(1)}(\mathscr{C})$ is Lipschitz. The argument for
$\Psi^{(2)}$ is analogous, and thus omitted. This completes the proof. ∎

Our next lemma establishes that under the two initial conditions introduced in
Section 1, the low-dimensional statistics $\mathcal{C}_{N}$ converge almost
surely to deterministic limits. We defer the proof of this lemma to Section
3.1.

######  Lemma 3.7.

Under the i.i.d. and i.i.d. under rotated basis initial conditions, each
element in $\mathcal{C}_{N}$ converges to deterministic limits almost surely.

Finally, we complete the proof of Theorem 1.1, assuming Lemma 3.4, 3.6 and
Lemma 3.7.

###### Proof of Theorem 1.1.

Fix $T>0$. Lemma 3.4, 3.6 and 3.7 together imply that $R^{N}$, $K^{N}$
converge to deterministic functions $R$ and $K$ respectively. It remains to
characterize these limits. Note that from (3.3),

$\displaystyle Y_{t}^{i}$
$\displaystyle=\exp\left\\{\int_{0}^{t}\big{[}\sigma^{i}-f^{\prime}(K^{N}(s))\big{]}\mathrm{d}{s}\,\right\\}\left\\{Y_{0}^{i}+u^{i}\int_{0}^{t}\exp\left\\{-\int_{0}^{s}\big{[}\sigma^{i}-f^{\prime}(K^{N}(r))\big{]}\mathrm{d}{r}\,\right\\}R^{N}(s)\mathrm{d}{s}\,\right\\}$
(3.17)
$\displaystyle\quad+\beta^{-1/2}\int_{0}^{t}\exp\left\\{\int_{s}^{t}\big{[}\sigma^{i}-f^{\prime}(K^{N}(r))\big{]}\mathrm{d}{r}\,\right\\}\mathrm{d}{B^{i}_{s}}\,.$
(3.18)

We recall that $R^{N}(t)=\frac{1}{N}\sum_{i=1}^{N}u^{i}Y^{i}_{t}$ and
$K^{N}(t,s)=\frac{1}{N}\sum_{i=1}^{N}Y_{t}^{i}Y_{s}^{i}$, and that $R^{N}\to
R$, $K^{N}\to K$ uniformly. Thus calculating $R^{N}$, $K^{N}$ and setting
$N\to\infty$, we obtain the desired fixed point equations in the limit. Note
that the final limit operation requires that
$\nu=\frac{1}{N}\sum_{i=1}^{N}\delta_{Y_{0}^{i},u^{i},\sigma^{i},B_{\bullet}^{i}}$
converges to the claimed limits under the i.i.d. and rotated i.i.d. initial
conditions. This completes the proof. ∎

###  3.1 Proof of Lemma 3.7

We prove Lemma 3.7 in this section. Note that for $f\in\mathcal{F}_{j}$,
$j=1,2,3$, $\int f\mathrm{d}{\nu}\,\in\mathcal{C}([0,T]^{j})$. Thus in the
lemma above, we mean specifically that $\int f\mathrm{d}{\nu}\,$ converges
almost surely as a $\mathcal{C}([0,T]^{j})$-valued random variable.
Throughout, we equip $\mathcal{C}([0,T]^{j})$ with a sup-norm.

###### Proof.

Before embarking on a formal proof, we summarize our general proof strategy.
The proof follows in two stages–first, we establish that for any fixed
$x\in[0,T]^{j}$, $\int f\mathrm{d}{\nu}\,(x)$ converges almost surely. We next
prove an additional Holder continuity property of $\int f\mathrm{d}{\nu}\,$,
uniformly in $n$, that allows us to bootstrap the above pointwise a.s.
convergence to a functional a.s. convergence statement.

We first consider the case of the initial condition (ii). Here, almost sure
convergence of $\int f\mathrm{d}{\nu}\,(x)$ for any fixed $x\in[0,T]^{j}$
follows immediately from the Strong Law of Large Numbers. We next establish
that for sufficiently large $n$, the following holds for each element of
$\mathcal{C}_{N}$: there exists $C,\alpha>0$, independent of $n$, such that
almost surely, for any $x,y\in[0,1]^{j}$,

$\displaystyle|\int f\mathrm{d}{\nu}\,(x)-\int f\mathrm{d}{\nu}\,(y)|\leq
C\|x-y\|^{\alpha}.$ (3.19)

Before we delve into the proof, we argue that (3.19) suffices to establish
convergence of $\int f\mathrm{d}{\nu}\,$ to $\mathbb{E}\int
f\mathrm{d}{\nu}\,$ in sup-norm.

To see this, first observe that for every $f\in\mathcal{F}_{j},j=1,2,3$,
$\mathbb{E}[\int f\mathrm{d}{\nu}\,]\in\mathcal{C}([0,T]^{j})$, by the
explicit form of these functions. Thus $\mathbb{E}[\int
f\mathrm{d}{\nu}\,](\cdot)$ is uniformly continuous on $[0,T]^{j}$. This
implies that if we fix $\epsilon>0$, there exists $\delta_{f}>0$ such that
whenever $|t_{1}-t_{2}|\leq\delta_{f}$, we have that

$\Big{|}\mathbb{E}\Big{[}\int
f\mathrm{d}{\nu}\,\Big{]}(t_{1})-\mathbb{E}\Big{[}\int
f\mathrm{d}{\nu}\,\Big{]}(t_{2})\Big{|}\leq\epsilon.$

Define $\delta=\min\\{\delta_{f}:f\in\mathcal{F}\\}$ and fix any
$0<\delta^{\prime}<\delta$. Let $\\{x_{1},\cdots,x_{\ell}\\}$ denote a
$\delta^{\prime}$-net of $[0,1]^{j}$. For any $y\in[0,T]^{j}$, if we denote
$x_{t}$ to be the nearest point in the net, we know that for sufficiently
large $n$,

$\displaystyle\Big{|}\Big{[}\int
f\mathrm{d}{\nu}\,\Big{]}(y)-\mathbb{E}\Big{[}\int
f\mathrm{d}{\nu}\,\Big{]}(y)\Big{|}$ $\displaystyle\leq\Big{|}\Big{[}\int
f\mathrm{d}{\nu}\,\Big{]}(y)-\Big{[}\int
f\mathrm{d}{\nu}\,\Big{]}(x_{t})\Big{|}$ (3.20)
$\displaystyle+\Big{|}\Big{[}\int
f\mathrm{d}{\nu}\,\Big{]}(x_{t})-\mathbb{E}\Big{[}\int
f\mathrm{d}{\nu}\,\Big{]}(x_{t})\Big{|}+\Big{|}\mathbb{E}\Big{[}\int
f\mathrm{d}{\nu}\,\Big{]}(x_{t})-\mathbb{E}\Big{[}\int
f\mathrm{d}{\nu}\,\Big{]}(y)\Big{|}$ $\displaystyle\leq
C(\delta^{\prime})^{\alpha}+2\epsilon.$

Since the RHS does not depend on $y$, and $\epsilon,\delta^{\prime}>0$ can be
arbitrary as long as $\delta^{\prime}<\delta$, we have the required sup-norm
convergence of $\int f\mathrm{d}{\nu}\,$ as a function on
$\mathcal{C}([0,T]^{j})$. We shall next establish Holder continuity of $\int
f\mathrm{d}{\nu}\,$ (uniformly in $n$), that is, property (3.19) for each
$f\in\mathcal{F}$.

To begin, let us consider $f_{1}$. Observe that

$\displaystyle\Big{|}\frac{1}{N}\sum_{i=1}^{N}u^{i}Y_{0}^{i}\exp(w_{1}\sigma^{i})-\frac{1}{N}\sum_{i=1}^{N}u^{i}Y_{0}^{i}\exp(w_{2}\sigma^{i})\Big{|}$
$\displaystyle\leq\exp(\|\sigma\|_{\infty}T)\Big{(}\frac{1}{N}\sum_{i=1}^{N}|u^{i}||Y_{0}^{i}|\Big{)}|w_{1}-w_{2}|$
(3.21) $\displaystyle\leq C|w_{1}-w_{2}|,$

where the last inequality is true a.s. with $C$ independent of N only for
sufficiently large $n$ on using the SLLN. Thus $f_{1}$ is Lipschitz almost
surely for sufficiently large $n$. For the above upper bound, recall that we
always have $\|\sigma_{i}\|_{\infty}\leq 2\max\\{|d_{+}|,|d_{-}|\\}$ for $n$
sufficiently large. Similar arguments work for $f_{2}$, $f_{4}$, $f_{5}$, so
we skip presenting those details here.

We next turn to $f_{3}$. Recall that

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}u^{i}B_{t}^{i}=\frac{1}{\sqrt{N}}\sum_{i=1}^{N}U^{i}B_{t}^{i}=\frac{1}{\sqrt{N}}\sum_{i=1}^{N}V^{i}W_{t}^{i},$

where the last equality follows from the orthogonality of the matrix
$\mathbf{G}$. Recall that $V^{i}\sim\mathcal{N}(0,\frac{\lambda}{N})$. This
implies

$\displaystyle\mathbb{P}\Big{[}\sup_{0\leq t\leq
T}\Big{|}\sum_{i=1}^{N}V^{i}W_{t}^{i}\Big{|}>\sqrt{N}\varepsilon\Big{]}$
$\displaystyle\leq\mathbb{P}\Big{[}\sup_{0\leq t\leq
T}\Big{|}\sum_{i=1}^{N}V^{i}W_{t}^{i}\Big{|}>\sqrt{N}\varepsilon,\sum_{i=1}^{N}(V^{i})^{2}\leq
C^{\prime}\Big{]}+\mathbb{P}\Big{[}\sum_{i=1}^{N}(V^{i})^{2}\geq
C^{\prime}\Big{]}$ $\displaystyle\leq 10\exp(-CN)$ (3.22)

using standard deviation bounds for the supremum of a Brownian motion and a
Chernoff bound on $\sum_{i=1}^{N}(V^{i})^{2}$. This completes the proof for
$f_{3}$.

Next, we move to functions in $\mathcal{F}_{2}$. First note that for $f_{8}$,
we have that $\int f_{8}\mathrm{d}{\nu}\,\rightarrow\mathbb{E}\int
f_{8}\mathrm{d}{\nu}\,$ on $\mathcal{C}([0,T]^{2})$, by the Uniform Law of
Large Numbers and properties of Brownian Motion. Next, we present the proof
for $f_{6}$—the proofs for $f_{7}$ and $f_{9}$ are similar. To show Holder
continuity of $f_{6}$, set
$\bm{D}_{w}=\textrm{diag}(\exp(w\sigma^{1})\cdots,\exp(w\sigma^{N}))$. Then we
have,

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}u^{i}B_{t}^{i}\exp(w\sigma^{i})=\frac{1}{\sqrt{N}}V^{\top}\mathbf{G}^{\top}\bm{D}_{w}\mathbf{G}\mathbf{W}_{t}.$

Thus for any $(t_{1},w_{1}),(t_{2},w_{2})\in[0,T]^{2}$,

$\displaystyle\Big{|}\frac{1}{\sqrt{N}}V^{\top}\mathbf{G}^{\top}\bm{D}_{w_{1}}\mathbf{G}\mathbf{W}_{t_{1}}-\frac{1}{\sqrt{N}}V^{\top}\mathbf{G}^{\top}\bm{D}_{w_{2}}\mathbf{G}\mathbf{W}_{t_{2}}\Big{|}$
$\displaystyle\leq\frac{1}{\sqrt{N}}\Big{|}V^{\top}\mathbf{G}^{\top}\bm{D}_{w_{1}}\mathbf{G}(\mathbf{W}_{t_{1}}-\mathbf{W}_{t_{2}})\Big{|}+\frac{1}{\sqrt{N}}|V^{\top}\mathbf{G}^{\top}(\bm{D}_{w_{1}}-\bm{D}_{w_{2}})\mathbf{G}\mathbf{W}_{t_{2}}|$
$\displaystyle\leq\|\mathbf{G}V\|_{2}\|\bm{D}_{w_{1}}\|_{2}\frac{1}{\sqrt{N}}\|\mathbf{W}_{t_{1}}-\mathbf{W}_{t_{2}}\|_{2}+\|\mathbf{G}V\|_{2}\|\bm{D}_{w_{1}}-\bm{D}_{w_{2}}\|_{2}\frac{1}{\sqrt{N}}\|\mathbf{W}_{t_{2}}\|_{2}.$

There exists $C>0$ such that almost surely, $\|\mathbf{G}V\|_{2}=\|V\|_{2}<C$
, $\sup_{w\in[0,T]}\|\bm{D}_{w_{1}}\|_{2}\leq\exp(\|\sigma\|_{\infty}T)$.
Further, $\|\bm{D}_{w_{1}}-\bm{D}_{w_{2}}\|_{2}\leq C|w_{1}-w_{2}|$, and

$\displaystyle\sup_{t_{1},t_{2}\in[0,T]}\Big{|}\frac{1}{N}\sum_{i=1}^{N}(W_{t_{1}}^{i}-W_{t_{2}}^{i})^{2}-|t_{1}-t_{2}|\Big{|}\stackrel{{\scriptstyle
as}}{{\to}}0.$

Thus there exists constants $C,C^{\prime}>0$ such that almost surely,

$\displaystyle\Big{|}\frac{1}{\sqrt{N}}V^{\top}\mathbf{G}^{\top}\bm{D}_{w_{1}}\mathbf{G}\mathbf{W}_{t_{1}}-\frac{1}{\sqrt{N}}V^{\top}\mathbf{G}^{\top}\bm{D}_{w_{2}}\mathbf{G}\mathbf{W}_{t_{2}}\Big{|}$
$\displaystyle\leq C(\sqrt{|t_{1}-t_{2}|}+|w_{1}-w_{2}|)\leq
C^{\prime}(\sqrt{|t_{1}-t_{2}|}+\sqrt{|w_{1}-w_{2}|}),$

where the last inequality follows from the fact that $w_{1},w_{2}\in[0,T]$, so
that we always have $\sqrt{|w_{1}-w_{2}|}\leq\sqrt{2T}$. The other functions
in $\mathcal{F}_{2}$ may be controlled using analogous arguments.

Finally, we move onto the functions in $\mathcal{F}_{3}$. We sketch the proof
for $f_{12}$; the same proof works for $f_{10}$, $f_{11}$. Set
$\tilde{\bm{D}}_{w}=\mathrm{diag}((\sigma^{i})^{2}\exp(w\sigma_{i}))$. Thus we
have, for
$(w,s_{1},s_{2}),(w^{\prime},s_{1}^{\prime},s_{2}^{\prime})\in[0,T]^{3}$,

$\displaystyle\Big{|}\frac{1}{N}\sum_{i=1}^{N}(\sigma^{i})^{2}\exp(w\sigma^{i})B_{s_{1}}^{i}B_{s_{2}}^{i}-\frac{1}{N}\sum_{i=1}^{N}(\sigma^{i})^{2}\exp(w^{\prime}\sigma^{i})B_{s_{1}^{\prime}}^{i}B_{s_{2}^{\prime}}^{i}\Big{|}$
$\displaystyle=\frac{1}{N}\Big{|}\mathbf{B}_{s_{1}}^{\top}\tilde{\bm{D}}_{w}\mathbf{B}_{s_{2}}-\mathbf{B}_{s_{1}^{\prime}}^{\top}\tilde{\bm{D}}_{w^{\prime}}\mathbf{B}_{s_{2}^{\prime}}\Big{|}\leq
C(\sqrt{|s_{1}-s_{1}^{\prime}|}+|w-w^{\prime}|+\sqrt{|s_{2}-s_{2}^{\prime}|})$
(3.23) $\displaystyle\leq
C^{\prime}(\sqrt{|s_{1}-s_{1}^{\prime}|}+\sqrt{|w-w^{\prime}|}+\sqrt{|s_{2}-s_{2}^{\prime}|})$

almost surely for some universal constants $C,C^{\prime}>0$. This completes
the proof for initial condition (ii).

Next, we turn to the initial condition (i). The main difference now lies in
the fact that the pointwise almost sure convergence no longer follows directly
from SLLN for all choices of $\mathcal{F}$. First, we observe that the
difference in initial conditions only affects $f_{1}$, $f_{4}$, $f_{5}$ and
$f_{9}$—thus we can restrict to these specific functions. We will use the same
strategy to establish functional almost sure convergence, starting from the
pointwise a.s. convergence, thus we omit those details. We present the proof
for $f_{1}$–the proofs for $f_{4},f_{5},f_{9}$ are similar. Fix $w\in[0,T]$.
We have,

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}u^{i}Y_{0}^{i}\exp(w\sigma^{i})=\frac{1}{\sqrt{N}}V^{\top}\mathbf{G}^{\top}\bm{D}_{w}\mathbf{G}\mathbf{X}_{0},$
(3.24)

where we use $\bm{D}_{w}=\mathrm{diag}(\exp(w\sigma^{i}))$. We observe that

$\displaystyle\mathbb{E}\Big{[}\frac{1}{N}\sum_{i=1}^{N}u^{i}Y_{0}^{i}\exp(w\sigma^{i})\Big{]}=\mathbb{E}\Big{[}\mathbb{E}[\frac{1}{\sqrt{N}}V^{\top}\mathbf{G}^{\top}\bm{D}_{w}\mathbf{G}\mathbf{X}_{0}|\mathbf{G},\mathbf{X}_{0}]\Big{]}=0.$

since entries of $V$ are i.i.d. with mean $0$. Now, there exists $M_{0}>0$
such that $\mathcal{E}_{n}=\\{\sum_{i=1}^{N}(X_{0}^{i})^{2}<N\cdot M_{0}\\}$
occurs eventually almost surely. Given $\mathbf{G}$ and $\mathbf{X}_{0}$,
$V^{\top}\mathbf{G}^{\top}\bm{D}_{w}\mathbf{G}\mathbf{X}_{0}\sim\mathcal{N}(0,\lambda(\mathbf{X}_{0}^{\top}\mathbf{G}^{\top}\bm{D}_{w}\mathbf{G}\mathbf{X}_{0})/N)$.
Moreover,

$\displaystyle\frac{\mathbf{X}_{0}^{\top}\mathbf{G}^{\top}\bm{D}_{w}\mathbf{G}\mathbf{X}_{0}}{N}\leq\|\bm{D}_{w}\|_{2}\frac{\sum_{i=1}^{N}(X_{0}^{i})^{2}}{N}.$

Thus on the event $\mathcal{E}_{n}$, for any $\varepsilon>0$,

$\displaystyle\mathbb{P}\Big{[}\Big{|}\frac{1}{\sqrt{N}}V^{\top}\mathbf{G}^{\top}\bm{D}_{w}\mathbf{G}\mathbf{X}_{0}\Big{|}>\varepsilon|\mathbf{G},\mathbf{X}_{0}\Big{]}\leq
2\exp(-\frac{N\varepsilon^{2}}{2M^{\prime}})$

for some universal constant $M^{\prime}>0$. The proof is complete on using the
Borel Cantelli lemma.

∎

## References

  * ABUZ [18] Elisabeth Agoritsas, Giulio Biroli, Pierfrancesco Urbani, and Francesco Zamponi.  Out-of-equilibrium dynamical mean-field equations for the perceptron model.  Journal of Physics A: Mathematical and Theoretical, 51(8):085002, 2018.
  * ACHL [19] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo.  Implicit regularization in deep matrix factorization.  Advances in Neural Information Processing Systems, 32, 2019.
  * ADG [01] G Ben Arous, Amir Dembo, and Alice Guionnet.  Aging of spherical spin glasses.  Probability theory and related fields, 120(1):1–67, 2001.
  * ADT [20] Alnur Ali, Edgar Dobriban, and Ryan Tibshirani.  The implicit regularization of stochastic gradient flow for least squares.  In International conference on machine learning, pages 233–244. PMLR, 2020.
  * AG [97] G Ben Arous and Alice Guionnet.  Symmetric langevin spin glass dynamics.  The Annals of Probability, 25(3):1367–1422, 1997.
  * AG [98] Gerard Ben Arous and Alice Guionnet.  Langevin dynamics for sherrington-kirkpatrick spin glasses.  In Mathematical aspects of spin glasses and neural networks, pages 323–353. Springer, 1998.
  * AGJ [20] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath.  Algorithmic thresholds for tensor pca.  The Annals of Probability, 48(4):2052–2087, 2020.
  * AGJ [21] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath.  Online stochastic gradient descent on non-convex losses from high-dimensional inference.  Journal of Machine Learning Research, 22(106):1–51, 2021.
  * AKT [19] Alnur Ali, J Zico Kolter, and Ryan J Tibshirani.  A continuous-time view of early stopping for least squares regression.  In The 22nd international conference on artificial intelligence and statistics, pages 1370–1378. PMLR, 2019.
  * BADG [06] Gérard Ben Arous, Amir Dembo, and Alice Guionnet.  Cugliandolo-kurchan equations for dynamics of spin-glasses.  Probability theory and related fields, 136(4):619–660, 2006.
  * BAGJ [20] Gérard Ben Arous, Reza Gheissari, and Aukosh Jagannath.  Bounding flows for spherical spin glass dynamics.  Communications in Mathematical Physics, 373(3):1011–1048, 2020\.
  * BAP [05] Jinho Baik, Gérard Ben Arous, and Sandrine Péché.  Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices.  The Annals of Probability, 33(5):1643–1697, 2005.
  * BGN [11] Florent Benaych-Georges and Raj Rao Nadakuditi.  The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices.  Advances in Mathematics, 227(1):494–521, 2011.
  * [14] Antoine Bodin and Nicolas Macris.  Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model.  Advances in Neural Information Processing Systems, 34, 2021.
  * [15] Antoine Bodin and Nicolas Macris.  Rank-one matrix estimation: analytic time evolution of gradient descent dynamics.  In Conference on Learning Theory, pages 635–678. PMLR, 2021.
  * BRT [19] Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov.  Does data interpolation contradict statistical optimality?  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1611–1619. PMLR, 2019.
  * CB [20] Lenaic Chizat and Francis Bach.  Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss.  In Conference on Learning Theory, pages 1305–1338. PMLR, 2020.
  * CBS+ [20] Chiara Cammarota, Giulio Biroli, Stefano Sarao, Lenka Zdeborova, and Florent Krzakala.  Who is afraid of big bad minima? analysis of gradient-flow in a spiked matrix-tensor model.  In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, volume 33, 2020.
  * CCM [21] Michael Celentano, Chen Cheng, and Andrea Montanari.  The high-dimensional asymptotics of first order methods with random data.  arXiv preprint arXiv:2112.07572, 2021.
  * CK [93] Leticia F Cugliandolo and Jorge Kurchan.  Analytical solution of the off-equilibrium dynamics of a long-range spin-glass model.  Physical Review Letters, 71(1):173, 1993.
  * CL [21] Niladri S Chatterji and Philip M Long.  Finite-sample analysis of interpolating linear classifiers in the overparameterized regime.  Journal of Machine Learning Research, 22(129):1–30, 2021.
  * CLB [21] Niladri S Chatterji, Philip M Long, and Peter L Bartlett.  When does gradient descent with logistic loss find interpolating two-layer networks?  Journal of Machine Learning Research, 22(159):1–48, 2021.
  * CS [92] Andrea Crisanti and H-J Sommers.  The sphericalp-spin interaction spin glass model: the statics.  Zeitschrift für Physik B Condensed Matter, 87(3):341–354, 1992\.
  * DG [21] Amir Dembo and Reza Gheissari.  Diffusions interacting through a random matrix: universality via stochastic taylor expansion.  Probability Theory and Related Fields, 180(3):1057–1097, 2021.
  * DGM [07] Amir Dembo, Alice Guionnet, and Christian Mazza.  Limiting dynamics for spherical models of spin glasses at high temperature.  Journal of Statistical Physics, 126(4):781–815, 2007.
  * DKT [19] Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis.  A model of double descent for high-dimensional binary linear classification.  arXiv preprint arXiv:1911.05822, 2019.
  * DLZ [21] Amir Dembo, Eyal Lubetzky, and Ofer Zeitouni.  Universality for langevin-like spin glass dynamics.  The Annals of Applied Probability, 31(6):2864–2880, 2021.
  * DS [20] Amir Dembo and Eliran Subag.  Dynamics for spherical spin glasses: disorder dependent initial conditions.  Journal of Statistical Physics, 181(2):465–514, 2020.
  * [29] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro.  Characterizing implicit bias in terms of optimization geometry.  In International Conference on Machine Learning, pages 1832–1841. PMLR, 2018.
  * [30] Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro.  Implicit bias of gradient descent on linear convolutional networks.  Advances in Neural Information Processing Systems, 31, 2018.
  * Gru [96] Malte Grunwald.  Sanov results for glauber spin-glass dynamics.  Probability theory and related fields, 106(2):187–232, 1996.
  * Gru [98] Malte Grunwald.  Sherrington-kirkpatrick spin-glass dynamics.  In Mathematical Aspects of Spin Glasses and Neural Networks, pages 355–382. Springer, 1998.
  * JM [20] Hui Jin and Guido Montúfar.  Implicit bias of gradient descent for mean squared error regression with wide neural networks.  arXiv preprint arXiv:2006.07356, 2020.
  * JT [19] Ziwei Ji and Matus Telgarsky.  The implicit bias of gradient descent on nonseparable data.  In Conference on Learning Theory, pages 1772–1798. PMLR, 2019.
  * LRZ [20] Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai.  On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels.  In Conference on Learning Theory, pages 2683–2711. PMLR, 2020.
  * LS [20] Tengyuan Liang and Pragya Sur.  A precise high-dimensional asymptotic theory for boosting and minimum-$\ell_{1}$-norm interpolated classifiers.  arXiv preprint arXiv:2002.01586, 2020.
  * MBB [18] Siyuan Ma, Raef Bassily, and Mikhail Belkin.  The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning.  In International Conference on Machine Learning, pages 3325–3334. PMLR, 2018.
  * MBC+ [20] Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová.  Marvels and pitfalls of the langevin algorithm in noisy high-dimensional inference.  Physical Review X, 10(1):011057, 2020.
  * MKUZ [19] Stefano Sarao Mannelli, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborova.  Passed & spurious: Descent algorithms and local minima in spiked matrix-tensor models.  In international conference on machine learning, pages 4333–4342. PMLR, 2019.
  * MKUZ [20] Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová.  Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification.  Advances in Neural Information Processing Systems, 33:9540–9550, 2020.
  * MRSY [19] Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan.  The generalization error of max-margin linear classifiers: High-dimensional asymptotics in the overparametrized regime.  arXiv preprint arXiv:1911.01544, 2019.
  * MWCC [18] Cong Ma, Kaizheng Wang, Yuejie Chi, and Yuxin Chen.  Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval and matrix completion.  In International Conference on Machine Learning, pages 3345–3354. PMLR, 2018.
  * MZ [20] Stefano Sarao Mannelli and Lenka Zdeborová.  Thresholds of descending algorithms in inference problems.  Journal of Statistical Mechanics: Theory and Experiment, 2020(3):034004, 2020.
  * NLG+ [19] Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry.  Convergence of gradient descent on separable data.  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3420–3428. PMLR, 2019.
  * SHN+ [18] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro.  The implicit bias of gradient descent on separable data.  The Journal of Machine Learning Research, 19(1):2822–2878, 2018\.
  * SMU [21] Stefano Sarao Mannelli and Pierfrancesco Urbani.  Analytical study of momentum-based acceleration methods in paradigmatic high-dimensional non-convex problems.  Advances in Neural Information Processing Systems, 34, 2021.
  * SZ [81] Haim Sompolinsky and Annette Zippelius.  Dynamic theory of the spin-glass phase.  Physical Review Letters, 47(5):359, 1981.
  * SZ [82] Haim Sompolinsky and Annette Zippelius.  Relaxational dynamics of the edwards-anderson model and the mean-field theory of spin-glasses.  Physical Review B, 25(11):6860, 1982.
  * ZBH+ [21] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  Communications of the ACM, 64(3):107–115, 2021.