# Performance of Bayesian linear regression
in a model with mismatch

Jean Barbierlabel=e1 [<EMAIL_ADDRESS><EMAIL_ADDRESS>Wei-Kuo
Chenlabel=e2 [<EMAIL_ADDRESS>Dmitry Panchenkolabel=e3 [
<EMAIL_ADDRESS>Manuel Sáenzlabel=e1 [<EMAIL_ADDRESS><EMAIL_ADDRESS>International Center for Theoretical Physics, Trieste, Italy,
School of Mathematics, University of Minnesota, Minneapolis, United States,
Department of Mathematics, University of Toronto, Canada,

###### Abstract

In this paper we analyze, for a model of linear regression with gaussian
covariates, the performance of a Bayesian estimator given by the mean of a
log-concave posterior distribution with gaussian prior, in the high-
dimensional limit where the number of samples and the covariates’ dimension
are large and proportional. Although the high-dimensional analysis of Bayesian
estimators has been previously studied for Bayesian-optimal linear regression
where the correct posterior is used for inference, much less is known when
there is a mismatch. Here we consider a model in which the responses are
corrupted by gaussian noise and are known to be generated as linear
combinations of the covariates, but the distributions of the ground-truth
regression coefficients and of the noise are unknown. This regression task can
be rephrased as a statistical mechanics model known as the Gardner spin glass,
an analogy which we exploit. Using a leave-one-out approach we characterize
the mean-square error for the regression coefficients. We also derive the log-
normalizing constant of the posterior. Similar models have been studied by
Shcherbina and Tirozzi and by Talagrand, but our arguments are much more
straightforward. An interesting consequence of our analysis is that in the
quadratic loss case, the performance of the Bayesian estimator is independent
of a global “temperature” hyperparameter and matches the ridge estimator:
sampling and optimizing are equally good.

,
and

##  1 Introduction and set-up

Linear regression with random covariates in the high-dimensional regime, where
both the number of data points and their dimension scale proportionally, is a
paradigmatic problem for modern inference. It has been studied mostly from two
complementary perspectives: $(i)$ In Bayesian statistics where, due to
important simplifications that occur in this setting, analyses have focused on
the case of the estimator obtained as the expectation of the “true” posterior
distribution of the regression coefficients [9, 7, 8, 40]. This is usually
referred to as the _Bayesian-optimal estimator_. But also on _sparse_ Bayesian
regression, see [16] and the section below on related works. $(ii)$ Another
large body of literature has studied instead the performance of optimization
procedures in the context of robust statistics and M-estimation, where an
estimator is obtained as the mode of a log-concave posterior measure or,
equivalently, as the minimizer of a convex cost function [47, 12, 21, 2, 3,
14, 45, 17, 20, 26, 27, 5, 50, 49, 46].

Our work focuses on Bayesian inference outside the optimal setting, for which
much less is known. The lack of perfect knowledge of the model generating the
data forces one to make certain inaccurate prior assumptions on the
distribution of the regression coefficients as well as on the form of the
model. This is, in a sense, more realistic and of practical interest. One
natural setting for this, is to consider Bayesian linear regression with some
sources of mismatch. Due to the lack of certain simplifying symmetries
inherent to the optimal setting of inference, known as the “Nishimori
identities” in physics [18, 37, 11], even a small mismatch can result in
several theoretical challenges that are highly amenable to mathematical and
statistical studies.

###  1.1 Our model

In the present work, we aim to study a mismatched linear regression model
which is as follows. Let $x^{*}=(x_{i}^{*})_{i\leqslant N}$ be an unknown
ground-truth of regression coefficients. Assume that

$\displaystyle\lim_{N\to\infty}{\mathbb{E}}\Bigl{|}\frac{\|x^{*}\|}{\sqrt{N}}-\sqrt{\gamma}\Bigr{|}^{2}=0\quad\mbox{and}\quad{\mathbb{E}}\Big{(}\frac{\|x^{*}\|}{\sqrt{N}}\Big{)}^{4}\leqslant
C<\infty$ (1)

for some deterministic $\gamma\geqslant 0$ and fixed $C$ independent of $N$.
Let $\alpha>0$ be a fixed parameter and let $M=\lfloor\alpha N\rfloor$111To be
more precise, we might take $M=\alpha N+o(N)$. But tracking this correction
makes no difference.; $\|\,\cdot\,\|$ is the usual $L_{2}$ norm. Consider a
matrix $G=(g_{ki})_{k\leqslant M,i\leqslant N}$ with i.i.d. standard normal
entries. The columns in $G$ are understood as random covariate vectors, while
the rows correspond to independent samples. The responses
$y=(y_{k})_{k\leqslant M}$ are obtained according to the linear model

$\displaystyle y_{k}=S_{k}(x^{*})+z_{k}$ (2)

for $S_{k}(x^{*}):=(\bar{G}x^{*})_{k},$ where $\bar{G}:=N^{-1/2}G$ and the
variables $z=(z_{k})_{k\leqslant
M}\stackrel{{\scriptstyle\rm{i.i.d.}}}{{\sim}}\mathcal{N}(0,\Delta_{*})$ for
some $\Delta_{*}>0$ which gives the strength of the noise;
$\mathcal{N}(m,\sigma^{2})$ is the normal law of mean $m$ and variance
$\sigma^{2}$. Here we will consider the regression task of recovering $x^{*}$
from the data $\mathcal{D}:=(y,\bar{G})$ with two sources of mismatch:

  1. 1.

The distribution of $x^{*}$ is assumed to be unknown. Thus, we take the priori
to be that of a gaussian vector with independent coordinates of mean $0$ and
variance $1/(2\kappa)>0$.

  2. 2.

In the true model generating the data (2), the noise is given by i.i.d.
gaussian random variables of variance $\Delta_{*}$. But its distribution is
also assumed to be unknown. We therefore consider a factorized log-concave
noise distribution with density proportional to $\prod_{k\leqslant M}\exp
u(S_{k}(x)-y_{k})$. Here $u$ is a non-positive concave function on
$\mathbb{R}$ satisfying some further technical conditions given in Assumption
1 below.

Under these sources of mismatch, the Bayesian posterior distribution is given
by

$\displaystyle P(x\in
A\mid\mathcal{D})=\frac{1}{{Z}(\mathcal{D})}\int_{A}\exp\Big{(}\sum_{k\leqslant
M}u\big{(}S_{k}(x)-y_{k}\big{)}-\kappa\|x\|^{2}\Big{)}\,dx$ (3)

for any measurable subset $A$ of $\mathbb{R}^{N}$, and ${{Z}(\mathcal{D})}$ is
the normalization constant, also called the partition function in the
terminology of statistical physics. Our aim is to study the performance of the
Bayesian estimator $\widehat{x}$ defined through the mean of this mismatched
Bayesian posterior distribution:

$\displaystyle\widehat{x}(\mathcal{D}):=\int_{\mathbb{R}^{N}}x\,P(dx\mid\mathcal{D}).$
(4)

###  1.2 Main results and contributions

Our main contributions are establishing the asymptotics of the log-normalizing
constant and the performance of estimator (4). Informally, we prove that,
whenever the four-dimensional system (17)–(20) presented below admits a unique
solution, the log-normalizing constant converges to some non-random quantity
$F(q,\rho)$, that is

$\displaystyle\lim_{N\to\infty}{\mathbb{E}}\Bigl{|}\frac{1}{N}\ln{Z}(\mathcal{D})-(-\kappa\gamma+F(q,\rho))\Bigr{|}^{2}=0,$

and the mean-square error converges to some non-random quantity $q$, that is

$\displaystyle\lim_{N\to\infty}\frac{1}{N}\mathbb{E}\|\widehat{x}-x^{*}\|_{2}^{2}=q.$
(5)

Here $(q,\rho)$ is the unique critical point of the real-valued function $F$
with domain over the set $\\{(q,\rho):0\leqslant q\leqslant\rho<\infty\\}$
defined in (21) below.

There are two major findings that follow from our results. First of all,
somewhat surprisingly (5) implies that the performance of the Bayesian
estimator with gaussian prior is insensitive to the statistical properties of
the coefficients $x^{*}$ other than its empirical second moment. The second
major finding regards our main running example for our results which will play
a special role given its tractability in the analysis and its importance in
statistics. This example will correspond to the model where the function
$u(s)$ is of the form $-s^{2}/(2\Delta)$ for some $\Delta>0$. Notice that this
model is still mismatched since $\Delta$ is not necessarily equal to the
$\Delta_{*}$ appearing in the model generating the data. When we convert the
posterior distribution into a _Gibbs measure_ parametrized by the inverse
temperature $\beta$ we discover that, for quadratic $u$, the mean-square error
is independent of the inverse temperature $\beta$ in the high-dimensional
limit. This means that when the performance is evaluated using a square loss,
Bayesian regression with gaussian prior and ridge regression have the same
performance. In other words, sampling and optimizing are equivalent in this
setting. One may argue that the closedness between the mean and the mode of
the posterior distribution with gaussian prior is mainly due to the isotropy
of the covariates. This intuition makes perfect sense in the classical regime,
where a lot of data is accessible $M\gg N$ when maximum likelihood estimation
is optimal [53]. But in the high-dimensional regime where both $M$ and $N$ are
large and comparable, this feature becomes much less evident. To the best of
our knowledge, this fact was not rigorously obtained prior to our work.
Exploring the generality of this equivalence between optimization and sampling
procedures is an interesting open direction for future research. See [2, 5]
for related connections between M-estimation and Bayesian inference from
statistical mechanics heuristics.

The approach in this paper relies primarily on a connection between our
regression task and a generalized version of the Shcherbina-Tirozzi (ST) mean-
field spin glass model [44]. We study the limits of the corresponding log-
partition function and some critical quantities called the overlaps. By
utilizing the so-called smart path method, analogous results were intensively
explored by Talagrand [48] assuming that $u$ grows at most linearly. Whereas
the machinery therein relies on quite heavy analysis, the advantage of our
approach is methodological. In addition to being able to relax the growth
condition for $u$ from linear to quadratic, our analysis is also rather simple
and self-contained. It is based on methods coming from the mathematical
physics of mean-field spin glasses: mainly a rigorous version of the cavity
method [48, 38] also called the leave-one-out method in statistics (see
Sections 5 and 6 for details). We expect for the simplicity of our approach to
allow us to extend the analysis to generalized linear models in mismatched
settings, for which results similar to the present ones exist only in the
Bayesian-optimal setting of inference [8]. This is left for future work.

###  1.3 Related works

In recent years, many works on mismatched regression have appeared: on non-
rigorous statistical mechanics techniques such as the replica method [32, 31]
like, e.g., in [2]; on the analysis of approximate message-passing algorithms,
like in [14, 45, 20, 27]; and on Gordon’s convex min-max theorem [50, 49]. All
these approaches strongly rely on the fact that the estimator is obtained as a
unique minimizer of a convex function and therefore do not generalize easily
to mismatched Bayesian settings. The leave-one-out method was also used in
[22, 21] but, like in the aforementioned references, only for the analysis of
(non-Bayesian) M-estimators.

Complementary to our setup is a recent work [34] on the mismatched Bayesian
linear regression that derives conditions under which the “naive mean-field
approximation” to the log-partition function is valid, and provides an
infinite-dimensional variational formula for it (see [39] for a related
setting). Last but not least, Bayesian linear regression has also been
considered in some different high-dimensional scaling regimes than the one
presented here: in settings with sparse ground-truth vectors of regression
coefficients with a possibly vanishing fraction of non-zero entries. Reference
[16] studies in detail the properties of the posterior distribution with
sparsity inducing priors in this case, see also [13, 25, 24, 28, 33, 43, 54,
30, 1, 4] for related prior works on sparse linear regression. It has recently
been understood that in the very sparse regime, and with the number of sampled
responses much smaller than the covariates’ dimension, intriguing “all-or-
nothing” phase transitions occur where the recovery error jumps from almost
zero to its maximum value at a certain threshold [23, 41, 42, 35, 29, 10, 36].
For a recent review on the Bayesian approach to high-dimensional inference,
see [6].

###  1.4 Organization

The paper is organized as follows. In Section 2, we explain the link between
the high-dimensional regression problem presented here and the ST spin glass
model. In Section 3, we first state two results on the main quantities of
interest in spin glass models, namely, the free energy (i.e., log-partition
function) and overlaps (that will later be related to the mean-square error)
in the ST model. We then state our main results on the regression task with a
particular emphasis on the classical example of the Bayesian regression with
quadratic loss and gaussian prior. Sections 4–6 are devoted to establishing
results concerning the ST model. More explicitly, Section 4 establishes a
number of concentration properties for the overlaps and free energy. Section 5
shows how to obtain certain fixed-point equations for the “order parameters”
of the model based on the concentration results in Section 4 by utilizing the
cavity method and simple gaussian integration by parts. Section 6 proves a
closed-form expression for the limiting log-partition function of the
posterior in a straightforward manner thanks to the convergence results for
the order parameters. Finally, our results on the regression task are
established in Section 7.

##  2 Connection to the Shcherbina-Tirozzi model

In this section, we explain how our regression task can be related to a
generalization of the ST model [44], a connection mentioned (but not
exploited) in [20]. For $h\in\mathbb{R},$ consider the _Hamiltonian_ (i.e.,
energy function) $H_{N}=(H_{N}(\sigma))_{\sigma\in\mathbb{R}^{N}}$ defined by

$\displaystyle H_{N}(\sigma):=\sum_{k\leqslant
M}u\big{(}S_{k}(\sigma)+z_{k}\big{)}-h\sum_{i\leqslant
N}\sigma_{i}-\kappa\|\sigma\|^{2}$ (6)

and the corresponding log-partition function, also called _free energy_ , by

$\displaystyle F_{N}$ $\displaystyle:=\frac{1}{N}\ln\int_{\mathbb{R}^{N}}\exp
H_{N}(\sigma)\,d\sigma.$ (7)

In the terminology of the large deviation theory [19, 51], it is the scaled
cumulant log-moment generating function associated with the energy per
variable $H_{N}(\sigma)/N$. When $z_{k}=0$ and $u$ grows at most linearly, the
Hamiltonian $H_{N}$ is known as the original ST model extensively studied in
[48]. In the setting of the present paper, we allow $u$ to grow quadratically
(see Assumption 1 below for details).

For a given realization of the randomness $(G,z)$ defining an instance of the
model, we will refer to each element of a sequence
$(\sigma^{\ell})_{\ell\geqslant 1}$ of independently drawn samples from the
_Gibbs measure_

$\displaystyle G_{N}(d\sigma):=\frac{\exp H_{N}(\sigma)\,d\sigma}{\int\exp
H_{N}(\sigma^{\prime})\,d\sigma^{\prime}}$

as _replicas_.

Denote by $\langle\,\cdot\,\rangle$ the Gibbs expectation of replicas
$(\sigma^{\ell})_{\ell\geqslant 1}$:

$\displaystyle\big{\langle}f((\sigma^{\ell})_{\ell\leqslant
L})\big{\rangle}:=\int_{\mathbb{R}^{NL}}f((\sigma^{\ell})_{\ell\leqslant
L})\prod_{\ell\leqslant L}G_{N}(d\sigma^{\ell}).$ (8)

Two key quantities for our analysis will be the _overlaps_ between replicas
defined according to

$R_{1,1}:=\frac{\|\sigma^{1}\|^{2}}{N}\quad\mbox{ and }\quad
R_{1,2}:=\frac{\sigma^{1}\cdot\sigma^{2}}{N}.$ (9)

Additionally, we also define a pair of “conjugate overlaps” in terms of the
following two auxiliary quantities

$\displaystyle
A(\sigma^{\ell}):=\Big{(}u^{\prime}\big{(}S_{k}(\sigma^{\ell})+z_{k}\big{)}\Big{)}_{k\leqslant
M}\quad\mbox{ and }\quad
B(\sigma^{\ell}):=\Big{(}u^{\prime\prime}\big{(}S_{k}(\sigma^{\ell})+z_{k}\big{)}\Big{)}_{k\leqslant
M}.$ (10)

For convenience, we will sometimes write $A^{\ell}=(A_{k}^{\ell})_{k\leqslant
M}$, $B^{\ell}=(B_{k}^{\ell})_{k\leqslant M}$, $S_{k}^{\ell}$ instead of
$A(\sigma^{\ell})$, $B(\sigma^{\ell})$, or $S_{k}(\sigma^{\ell})$. Whenever
the replica index $\ell$ is omitted, it is assumed to be equal to $1$. The
conjugate overlaps are then defined by

$Q_{1,1}:=\frac{1}{N}\sum_{k\leqslant
M}\Big{(}(A_{k})^{2}+B_{k}\Big{)}\quad\mbox{ and }\quad
Q_{1,2}:=\frac{1}{N}\langle A^{1},A^{2}\rangle.$ (11)

As one shall see, the vector $(R_{1,1},R_{1,2},Q_{1,1},Q_{1,2})$ of overlaps
order parameters is the “correct” one, in the sense that all the statistical
analysis can be asymptotically expressed solely in terms of the limits of
these quantities and the model at the “macroscopic level” is completely
determined by them.

In view of our regression task, the posterior measure (3) can be thought of as
a Gibbs measure of the spin glass model with Hamiltonian

$\displaystyle\widetilde{H}_{N}(x):=\sum_{k\leqslant
M}u\Big{(}\frac{1}{\sqrt{N}}\sum_{i\leqslant
N}g_{ki}(x_{i}-x_{i}^{*})-z_{k}\Big{)}-\kappa\|x\|^{2},$

that is,

$\displaystyle
P(dx\mid\mathcal{D})=\widetilde{G}_{N}(dx):=\frac{\exp{\widetilde{H}_{N}(x)}\,dx}{\int\exp{\widetilde{H}_{N}(x^{\prime})}\,dx^{\prime}}.$
(12)

As in the ST model, we define the free energy by

$\widetilde{F}_{N}:=\frac{1}{N}\ln\int_{\mathbb{R}^{N}}\exp\widetilde{H}_{N}(x)\,dx.$

Denote by $(x^{\ell})_{\ell\geqslant 1}$ the replicas drawn i.i.d. from
measure (12) and by $\langle\,\cdot\,\rangle^{\sim}$ the Gibbs expectation
associated to the Hamiltonian $\widetilde{H}_{N}$, defined similarly as (8)
but instead with $\widetilde{G}_{N}$.

In particular, from (4), we readily see that the Bayesian estimator

$\widehat{x}=\langle x\rangle^{\sim}.$

The connection between the ST and our models relies on a number of change of
variables. First of all, by letting $w:=x-x^{*}$, we may write

$\displaystyle\int_{\mathbb{R}^{N}}\exp\widetilde{H}_{N}(x)\,dx$
$\displaystyle=\int_{\mathbb{R}^{N}}\exp\Big{(}\sum_{k\leqslant
M}u\Big{(}\frac{1}{\sqrt{N}}\sum_{i\leqslant
N}g_{ki}w_{i}-z_{k}\Big{)}-\kappa\|w+x^{*}\|^{2}\Big{)}\,dw$
$\displaystyle=\exp(-\kappa\|x^{*}\|^{2})\int_{\mathbb{R}^{N}}\exp\Big{(}\sum_{k\leqslant
M}u\Big{(}\frac{1}{\sqrt{N}}\sum_{i\leqslant N}g_{ki}w_{i}-z_{k}\Big{)}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad-2\kappa\sum_{i\leqslant
N}x_{i}^{*}w_{i}-\kappa\|w\|^{2}\Big{)}\,dw.$

To simplify this integral, take an orthogonal matrix $O\in\mathcal{O}(N)$ such
that $Ox^{*}=N^{-1/2}{\|x^{*}\|}\bm{1}_{N}$. Note that conditionally on
$x^{*}$,

$\displaystyle{\mathbb{E}}(GO^{\intercal})_{k\ell}(GO^{\intercal})_{k^{\prime}\ell^{\prime}}$
$\displaystyle={\mathbb{E}}\Big{(}\sum_{i\leqslant N}g_{ki}O_{\ell
i}\Big{)}\Big{(}\sum_{i\leqslant N}g_{k^{\prime}i}O_{\ell^{\prime}i}\Big{)}$
$\displaystyle=\delta_{kk^{\prime}}\sum_{i\leqslant N}O_{\ell
i}O_{\ell^{\prime}i}=\delta_{kk^{\prime}}(OO^{\intercal})_{\ell\ell^{\prime}}=\delta_{kk^{\prime}}\delta_{\ell\ell^{\prime}},$

which implies that $GO^{\intercal}\sim G$ conditionally on $x^{*}$. Note also
that $\langle
x^{*},O^{\intercal}a\rangle=N^{-1/2}\|x^{*}\|\langle\bm{1}_{N},a\rangle$ and
$\|O^{\intercal}a\|=\|a\|.$ From these and using the change of variable
$\sigma:=Ow$,

$\displaystyle\begin{split}&\int_{\mathbb{R}^{N}}\exp\Big{(}\sum_{k\leqslant
M}u\Big{(}\frac{1}{\sqrt{N}}\sum_{i\leqslant
N}g_{ki}w_{i}-z_{k}\Big{)}-2\kappa\sum_{i\leqslant
N}x_{i}^{*}w_{i}-\kappa\|w\|^{2}\Big{)}\,dw\\\
&\qquad=\int_{\mathbb{R}^{N}}\exp\Big{(}\sum_{k\leqslant
M}u\Big{(}(\bar{G}O^{\intercal}\sigma)_{k}-z_{k}\Big{)}-2\kappa\langle
x^{*},O^{\intercal}\sigma\rangle-\kappa\|O^{\intercal}\sigma\|^{2}\Big{)}\,d\sigma\\\
&\qquad=\int_{\mathbb{R}^{N}}\exp\Big{(}\sum_{k\leqslant
M}u\Big{(}(\bar{G}O^{\intercal}\sigma)_{k}-z_{k}\Big{)}-\frac{2\kappa\|x^{*}\|}{\sqrt{N}}\sum_{i\leqslant
N}\sigma_{i}-\kappa\|\sigma\|^{2}\Big{)}\,d\sigma,\end{split}$ (13)

which implies, from the rotational invariance of $G$ mentioned above and the
symmetry of $z_{k},$ that

$\displaystyle{\mathbb{E}}\ln\int_{\mathbb{R}^{N}}\exp\Big{(}\sum_{k\leqslant
M}u\Big{(}\frac{1}{\sqrt{N}}\sum_{i\leqslant
N}g_{ki}w_{i}-z_{k}\Big{)}-2\kappa\sum_{i\leqslant
N}x_{i}^{*}w_{i}-\kappa\|w\|^{2}\Big{)}\,dw$
$\displaystyle\qquad={\mathbb{E}}\ln\int_{\mathbb{R}^{N}}\exp\Big{(}\sum_{k\leqslant
M}u\Big{(}(\bar{G}\sigma)_{k}+z_{k}\Big{)}-\frac{2\kappa\|x^{*}\|}{\sqrt{N}}\sum_{i\leqslant
N}\sigma_{i}-\kappa\|\sigma\|^{2}\Big{)}\,d\sigma.$

To sum up,

$\displaystyle{\mathbb{E}}\widetilde{F}_{N}=-\kappa{\mathbb{E}}\gamma_{N}+\frac{1}{N}{\mathbb{E}}\ln\int_{\mathbb{R}^{N}}\exp\Bigl{(}\sum_{k\leqslant
M}u\big{(}S_{k}(\sigma)+z_{k}\big{)}-2\kappa\sqrt{\gamma_{N}}\sum_{i\leqslant
N}\sigma_{i}-\kappa\|\sigma\|^{2}\Bigr{)}\,d\sigma$

for $\gamma_{N}:=\|x^{*}\|^{2}/N.$ If now we randomize
$h=h_{N}=2\kappa\sqrt{\gamma_{N}}$ in the Hamiltonian (6) of the ST model
independently of all other randomness, then we readily get

$\displaystyle{\mathbb{E}}\widetilde{F}_{N}=-\kappa{\mathbb{E}}\gamma_{N}+{\mathbb{E}}F_{N}$
(14)

and similarly,

$\displaystyle{\mathbb{E}}\|\widehat{x}-x^{*}\|^{2}={\mathbb{E}}\|\langle
x\rangle^{\sim}-x^{*}\|^{2}={\mathbb{E}}\|\langle\sigma\rangle\|^{2}.$ (15)

Consequently, to understand our regression problem, we may study the limiting
free energy and overlaps corresponding to the generalized ST model first. This
characterization is part of our main results and is stated in the next
section.

##  3 Main results

The first part of our main results consists of an expression for the limiting
free energy and the convergence of the overlaps in the generalized ST model
defined in (6). Throughout the rest of this paper, the following assumption on
the function $u$ is in force.

######  Assumption 1 (Growth condition).

The function $u(s)$ is concave, non-positive, and there exists a constant
$d>0$ such that

$\displaystyle\max\bigl{(}|u(0)|,|u^{\prime}(0)|,\|u^{\prime\prime}\|_{\infty},\|u^{\prime\prime\prime}\|_{\infty},\|u^{(4)}\|_{\infty}\bigr{)}\leqslant
d$

and

$\displaystyle|u^{\prime}(s)|\leqslant d(1+\sqrt{|u(s)|}).$

######  Remark 3.1.

The concavity of $u$ is conceptually an important assumption, which ensures
that our posterior distribution is a log-concave measure and, as a result, the
overlaps are concentrated under the Gibbs measure $G_{N}$. While the other
assumptions are purely technical, overall they weaken the settings considered
in [48]. In particular, in this reference $u$ is allowed to grow at most
linearly and thus its results do not apply to our main example, namely, a
quadratic $u$.

To formulate our limiting free energy, we introduce
$\bar{F}:\mathbb{R}^{4}\mapsto\mathbb{R}$ such that

$\displaystyle\bar{F}(q,\rho,r,\bar{r})$
$\displaystyle:=\alpha{\mathbb{E}}\ln{\mathbb{E}}_{\xi}\exp
u\big{(}\widetilde{z}\sqrt{\Delta_{*}+q}+\xi\sqrt{\rho-q}\big{)}$
$\displaystyle\qquad+\frac{1}{2}\Big{(}\frac{r+h^{2}}{2\kappa+r-\bar{r}}-\ln(2\kappa+r-\bar{r})+rq-\bar{r}\rho+\ln
2\pi\Big{)}$ (16)

for $0\leqslant q<\rho$ and $0\leqslant\bar{r}\leqslant r$, where
$\widetilde{z},\xi$ are independent random variables with normal law
$\mathcal{N}(0,1)$. If we let
$\theta=\widetilde{z}\sqrt{\Delta_{*}+q}+\xi\sqrt{\rho-q}$, basic gaussian
integration by parts with respect to $\widetilde{z}$ and $\xi$ implies that
the critical points of the function $\bar{F}$ must satisfy the following
system of equations:

$\displaystyle q$
$\displaystyle=\Psi(r,\bar{r}):=\frac{r+h^{2}}{(2\kappa+r-\bar{r})^{2}},$ (17)
$\displaystyle\rho$
$\displaystyle=\bar{\Psi}(r,\bar{r}):=\frac{1}{2\kappa+r-\bar{r}}+\frac{r+h^{2}}{(2\kappa+r-\bar{r})^{2}},$
(18) $\displaystyle r$
$\displaystyle=\Phi(q,\rho):=\alpha{\mathbb{E}}\Bigl{(}\frac{{\mathbb{E}}_{\xi}u^{\prime}(\theta)\exp{u(\theta)}}{{\mathbb{E}}_{\xi}\exp{u(\theta)}}\Bigr{)}^{2},$
(19) $\displaystyle\bar{r}$ $\displaystyle=\hbox to0.0pt{\hskip
1.1111pt\leavevmode\hbox{$\overline{\hbox{}}$}\hss}{\leavevmode\hbox{$\Phi$}}(q,\rho):=\alpha{\mathbb{E}}\frac{{\mathbb{E}}_{\xi}(u^{\prime\prime}(\theta)+u^{\prime}(\theta)^{2})\exp{u(\theta)}}{{\mathbb{E}}_{\xi}\exp{u(\theta)}}.$
(20)

In addition, for any $0\leqslant q<\rho,$ we consider the function
$F:\mathbb{R}^{2}\mapsto\mathbb{R}$ defined as

$\displaystyle F(q,\rho)$
$\displaystyle:=\alpha{\mathbb{E}}\ln{\mathbb{E}}_{\xi}\exp
u\big{(}\widetilde{z}\sqrt{\Delta_{*}+q}+\xi\sqrt{\rho-q}\big{)}$
$\displaystyle\qquad+\frac{1}{2}\Big{(}\ln(\rho-q)+h^{2}(\rho-q)+\frac{\rho}{\rho-q}-2\kappa\rho+\ln
2\pi\Big{)}.$ (21)

Observe that if $(q,\rho,r,\bar{r})$ satisfies (17)–(20), then
$\bar{F}(q,\rho,r,\bar{r})$ and $F(q,\rho)$ match each other as can be seen by
rewriting (17) and (18) as

$2\kappa+r-\bar{r}=\frac{1}{\rho-q}\quad\mbox{and}\quad
r+h^{2}=\frac{q}{(\rho-q)^{2}}$

and plugging these into the second line in (16) after writing
$rq-\bar{r}\rho=-r(\rho-q)+(r-\bar{r})\rho$. Here, if we express $r,\bar{r}$
in terms of $q,\rho$ in (19)–(20), the pair $(q,\rho)$ is also a critical
point of $F$. Our first main result states that the limiting free energy in
the ST model is equal to $F(q,\rho)$ as long as (17)–(20) have a unique
solution.

######  Theorem 3.2 (Free energy of the ST spin glass).

For any $\kappa>0$, $h\in\mathbb{R}$, and $\Delta_{*}\geqslant 0$, if
(17)–(20) have a unique solution $(q,\rho,r,\bar{r})$, then

$\lim_{N\to\infty}{\mathbb{E}}\bigl{|}F_{N}-F(q,\rho)\bigr{|}^{2}=0.$

The next result establishes the convergence of the overlaps (9) and (11).

######  Theorem 3.3 (Convergence of overlaps).

For any $\kappa>0$, $h\in\mathbb{R}$, and $\Delta_{*}\geqslant 0$, if
(17)–(20) have a unique solution $(q,\rho,r,\bar{r})$, then under
${\mathbb{E}}\langle\,\cdot\,\rangle$ and as $N$ diverges,

$R_{1,1}\xrightarrow{L^{2}}\rho,\quad R_{1,2}\xrightarrow{L^{2}}q,\quad
Q_{1,1}\xrightarrow{L^{2}}\bar{r},\quad\mbox{and}\quad
Q_{1,2}\xrightarrow{L^{2}}r.$

From these and with the help of the identities (14) and (15), we can now
compute the limiting free energy and the mean-square error associated to our
regression task.

######  Theorem 3.4 (Free energy and mean-square error in mismatched
regression).

Assume the convergence (1) for the norm of the ground-truth coefficients
towards some deterministic $\sqrt{\gamma}\geqslant 0$. For $\kappa>0$,
$h=2\kappa\sqrt{\gamma}$, and $\Delta_{*}\geqslant 0$, if (17)–(20) have a
unique solution $(q,\rho,r,\bar{r})$, then

$\displaystyle\lim_{N\to\infty}{\mathbb{E}}\bigl{|}\widetilde{F}_{N}-(-\kappa\gamma+F(q,\rho))\bigr{|}=0$
(22)

and

$\displaystyle\lim_{N\to\infty}\frac{1}{N}{\mathbb{E}}\|\widehat{x}-x^{*}\|^{2}=\lim_{N\to\infty}\frac{1}{N}{\mathbb{E}}\|\langle
x\rangle^{\sim}-x^{*}\|^{2}=q.$ (23)

######  Example 1 (Quadratic loss).

Let $u(s)=-{s^{2}}/(2\Delta),$ where $\Delta>0$ is not necessarily equal to
the true gaussian noise variance $\Delta_{*}.$ Obviously $u$ satisfies
Assumption 1. More importantly, it can be checked that (17)–(20) have a unique
solution given by

$\begin{cases}q=\frac{c^{2}(h^{2}(\Delta+c)^{2}+\Delta_{*}\alpha)}{(\Delta+c)^{2}-\alpha
c^{2}},\\\ \rho=q+c,\\\ r=\frac{\alpha(\Delta_{*}+q)}{(\Delta+\rho-q)^{2}},\\\
\bar{r}=r-\frac{\alpha}{\Delta+\rho-q},\end{cases}$ (24)

where the constant

$\displaystyle
c:=\frac{\sqrt{(2\kappa\Delta+\alpha-1)^{2}+8\kappa\Delta}-(2\kappa\Delta+\alpha-1)}{4\kappa}.$

To see this, note that (19) and (20) can be written as

$\displaystyle r$
$\displaystyle=\frac{\alpha(\Delta_{*}+q)}{(\Delta+\rho-q)^{2}}\quad\mbox{and}\quad\bar{r}=r-\frac{\alpha}{\Delta+\rho-q}.$
(25)

Take $a=\rho-q$ and $a^{\prime}=r-\bar{r}$. From the difference between (17)
and (18) and that of (19) and (20), we arrive at

$\begin{cases}aa^{\prime}+2\kappa a=1,\\\ aa^{\prime}+\Delta
a^{\prime}=\alpha,\end{cases}$ (26)

from which we can eliminate $a^{\prime}$ to get a single equation for $a$,

$2\kappa a^{2}+\left(2\kappa\Delta+\alpha-1\right)a-\Delta=0.$

It is easy to see that the only non-negative solution to this equation is
$a=c$ and thus, $a^{\prime}=\alpha/(\Delta+c).$ Finally, using these values of
$a$ and $a^{\prime}$ and the second equation of (25) leads to

$q=\frac{c^{2}(h^{2}(\Delta+c)^{2}+\Delta_{*}\alpha)}{(\Delta+c)^{2}-\alpha
c^{2}}.$ (27)

Note that $q$ is positive since (26) implies that $aa^{\prime}<1$ and
$aa^{\prime}<\alpha$, which ensures that
$(\Delta+c)^{2}/c^{2}=(aa^{\prime})^{2}<\alpha$. From this $q$,
$(\rho,r,\bar{r})$ can then be uniquely determined through the second to the
fourth equations in (24).

Surprisingly, if we parametrize the posterior distribution by an “inverse
temperature” hyperparameter $\beta>0$, namely, the original Gibbs measure
$\widetilde{G}_{N}$ in (12) becomes

$\displaystyle\widetilde{G}_{N,\beta}(dx)=\frac{\exp(\beta\widetilde{H}_{N}(x))\,dx}{\int\exp(\beta\widetilde{H}_{N}(x^{\prime}))\,dx^{\prime}},$

meaning that $\Delta$ and $\kappa$ are replaced by $\Delta/\beta$ and
$\beta\kappa$ in the original model, then the resulting constant $q_{\beta}$,
i.e., the mean-square error, remains the same as the original $q$ for any
inverse temperature $\beta>0$. This can be easily seen from the above
equations (for quadratic $u$). At the same time, formula (27) matches the one
for ridge regression at zero temperature (see, e.g., equation (26) in [49]).
This non-trivial fact is due to the rotational invariance of the model
following from the isotropy of the covariates. But as mentioned already in the
introduction, this is not a priori evident in the high-dimensional regime we
consider here, where $M$ and $N$ are comparable. It would be interesting to
explore in future research for which settings posterior mean and mode (i.e.,
sampling and optimizing) yield the same reconstruction performance.

##  4 Concentration

In this section we will prove a number of useful concentration results for
overlaps and the free energy. For this, it will be convenient to restate the
growth conditions in Assumption 1 in a more general way, more suitable for the
proof. Take $n\geqslant 1$ and consider two functions
$\phi,\psi:\mathbb{R}^{n}\mapsto\mathbb{R}$ which together with the function
$u:\mathbb{R}\mapsto\mathbb{R}$ satisfy the following set of conditions.
Namely, we suppose that there exist $d>0$ and $\eta_{0}>0$ such that the
following holds:

  1. (G-1)

The function $u(s)$ satisfies

$\displaystyle 0\geqslant
u(s)\geqslant-d(1+s^{2})\quad\mbox{and}\quad|u^{\prime}(s)|\leqslant
d(1+|s|).$ (28)

  2. (G-2)

The functions $\phi,\psi:\mathbb{R}^{n}\mapsto\mathbb{R}$ satisfy

$\displaystyle\max\bigl{(}|\phi(s)-\phi(s^{\prime})|,|\psi(s)-\psi(s^{\prime})|\bigr{)}$
$\displaystyle\leqslant
d\bigl{(}1+\|s\|+\|s^{\prime}\|\bigr{)}\|s-s^{\prime}\|,$ (29)
$\displaystyle\max\bigl{(}|\phi(s)|,|\psi(s)|\bigr{)}$ $\displaystyle\leqslant
d\bigl{(}1+\|s\|^{2}\bigr{)},$ (30)
$\displaystyle\max_{1\leqslant\ell,\ell^{\prime}\leqslant
n}\bigl{(}|\partial_{s_{\ell}s_{\ell^{\prime}}}\phi(s)|,|\partial_{s_{\ell}s_{\ell^{\prime}}}\psi(s)|\bigr{)}$
$\displaystyle\leqslant d.$ (31)

  3. (G-3)

The following inequality holds:

$\displaystyle\sup_{y\in\mathbb{R}^{n}}\Bigl{(}\sum_{\ell\leqslant
n}u(y^{\ell})+\eta_{0}|\psi(y^{1},\ldots,y^{n})|\Bigr{)}\leqslant\alpha d.$

The last assumption (G-3) will be used in an equivalent form,

$\displaystyle\max_{y^{1},\ldots,y^{n}\in\mathbb{R}^{M}}\sum_{k\leqslant
M}\Bigl{(}\sum_{\ell\leqslant
n}u(y_{k}^{\ell})+\eta_{0}|\psi(y_{k}^{1},\ldots,y_{k}^{n})|\Bigr{)}\leqslant
Nd.$ (32)

Also, the assumptions (29), (30) may look redundant given (31), but we state
them for convenience.

We will later specialize these assumptions for the following two specific
choices:

  1. $\bullet$

For $n=1$: $\phi(s)=s^{2}$ and
$\psi(s)=u^{\prime\prime}(s)+u^{\prime}(s)^{2}$,

  2. $\bullet$

For $n=2$: $\phi(s_{1},s_{2})=s_{1}s_{2}$ and
$\psi(s_{1},s_{2})=u^{\prime}(s_{1})u^{\prime}(s_{2})$.

One can easily see that Assumption 1 on $u$ implies (G-1)–(G-3) in both cases,
by possibly increasing the value of $d$.

For $\sigma^{1},\ldots,\sigma^{n}\in\mathbb{R}^{N}$ denote
$\vec{\sigma}=(\sigma^{1},\ldots,\sigma^{n})\in\mathbb{R}^{Nn}$ and set
$\|\vec{\sigma}\|=(\|\sigma^{1}\|^{2}+\cdots+\|\sigma^{n}\|^{2})^{1/2}$.
Consider any $\lambda$ and $\eta$ satisfying
$|\lambda|,|\eta|\leqslant\min(\eta_{0},\kappa/(2d))$. For any
$\sigma\in\mathbb{R}^{N}$ recall $S_{k}(\sigma):=(\bar{G}\sigma)_{k}$ and
define

$\displaystyle X_{N}(\sigma):=\sum_{k\leqslant
M}u\bigl{(}S_{k}(\sigma)+z_{k}\bigr{)}.$

For $\vec{\sigma}\in\mathbb{R}^{Nn}$, we define an Hamiltonian between
“coupled replicas” $(\sigma^{\ell})_{l\leqslant n}$:

$\displaystyle X_{N,\lambda,\eta}(\vec{\sigma}):=\sum_{\ell\leqslant
n}X_{N}(\sigma^{\ell})+h\sum_{\ell\leqslant n}\sum_{i\leqslant
N}\sigma_{i}^{\ell}-\kappa\|\vec{\sigma}\|^{2}+\lambda NR(\vec{\sigma})+\eta
NQ(\vec{\sigma}),$ (33)

where the generalized overlaps are

$\displaystyle R(\vec{\sigma})$ $\displaystyle:=\frac{1}{N}\sum_{i\leqslant
N}\phi(\sigma_{i}^{1},\ldots,\sigma_{i}^{n}),$ (34) $\displaystyle
Q(\vec{\sigma})$ $\displaystyle:=\frac{1}{N}\sum_{k\leqslant
M}\psi\bigl{(}S_{k}(\sigma^{1})+z_{k},\ldots,S_{k}(\sigma^{n})+z_{k}\bigr{)},$
(35)

for functions $\phi$ and $\psi$ verifying assumptions (G-1)–(G-3). Define the
(non-averaged) free energy associated to $X_{N,\lambda,\eta}$ by

$\displaystyle F_{N}(\lambda,\eta)$ $\displaystyle:=\frac{1}{N}\ln
Z_{N,\lambda,\eta}\quad\mbox{where}\quad
Z_{N,\lambda,\eta}:=\int_{\mathbb{R}^{Nn}}\exp{X_{N,\lambda,\eta}(\vec{\sigma})}\,d\vec{\sigma}.$

Note that assumptions (28), (29), and (32) as well as
$|\lambda|,|\eta|\leqslant\min(\eta_{0},\kappa/(2d))$ ensure that this
quantity is a.s. finite in $G$ and $z=(z_{k})_{k\leqslant M}$ Denote by
$\langle\,\cdot\,\rangle_{\lambda,\eta}$ the expectation with respect to the
Gibbs measure proportional to $\exp X_{N,\lambda,\eta}(\vec{\sigma})$:

$\displaystyle\langle
g(\vec{\sigma})\rangle_{\lambda,\eta}:=\frac{1}{Z_{N,\lambda,\eta}}\int_{\mathbb{R}^{Nn}}g(\vec{\sigma})\exp{X_{N,\lambda,\eta}(\vec{\sigma})}\,d\vec{\sigma}.$

From now on, when we say that “there exist constants
$\varepsilon,K,K^{\prime},D,L$, etc.”, these constants are functions of all
the parameters of the model $(d,n,\alpha,\Delta_{*},\kappa,\eta_{0},h)$, which
are bounded uniformly over compacts. As before, $\mathbb{E}$ is the
expectation with respect to the gaussians $(G,z)$.

######  Proposition 4.1.

Under the above assumptions, there exists some $K>0$ such that

$\displaystyle{\mathbb{E}}\Bigl{\langle}\Bigl{|}R(\vec{\sigma})-{\mathbb{E}}\langle
R(\vec{\sigma})\rangle_{0,0}\Bigr{|}^{2}\Bigr{\rangle}_{0,0}$
$\displaystyle\leqslant\frac{K}{\sqrt{N}}\quad\mbox{and}\quad{\mathbb{E}}\Bigl{\langle}\Bigl{|}Q(\vec{\sigma})-{\mathbb{E}}\langle
Q(\vec{\sigma})\rangle_{0,0}\Bigr{|}^{2}\Bigr{\rangle}_{0,0}\leqslant\frac{K}{\sqrt{N}}.$
(36)

Moreover, for any $k\geqslant 1$, there exists some $C_{k}>0$ such that

$\displaystyle{\mathbb{E}}\bigl{\langle}|R(\vec{\sigma})|^{k}\bigr{\rangle}_{0,0}\leqslant
C_{k}\quad\mbox{ and
}\quad{\mathbb{E}}\bigl{\langle}|Q(\vec{\sigma})|^{k}\bigr{\rangle}_{0,0}\leqslant
C_{k}.$

In the rest of this section, we will establish Proposition 4.1. To fix our
notation, for any real $M\times N$-matrix $A$, define the Frobenius and
operator norms, respectively, as

$\displaystyle\|A\|:=\Bigl{(}\sum_{k\leqslant M}\sum_{i\leqslant
N}A_{ki}^{2}\Bigr{)}^{1/2}\quad\mbox{and}\quad\|A\|_{op}:=\sup_{y\in\mathbb{R}^{N}:\|y\|\leqslant
1}\|Ay\|.$

Recall $\|\cdot\|$ applied to a vector is the $L_{2}$ norm and note that
$\|A\|_{op}\leqslant\|A\|$.

###  4.1 Control of the self-overlap

Let $\bar{z}:=z/\sqrt{N}$ and recall also that $\bar{G}:=G/\sqrt{N}$.

######  Lemma 4.2.

There exist $\varepsilon>0$ and $K\geqslant 1$ such that if
$t,|\lambda|,|\eta|\leqslant\varepsilon$ then

$\displaystyle\bigl{\langle}\exp{t\|\vec{\sigma}\|^{2}}\bigr{\rangle}_{\lambda,\eta}\leqslant\exp{KN(\|\bar{z}\|^{2}+\|\bar{G}\|_{op}^{2}+1+h^{2})}.$

_Proof._ From (30),

$\displaystyle|R(\vec{\sigma})|$
$\displaystyle\leqslant\frac{d}{N}\sum_{i\leqslant
N}\Bigl{(}1+\sum_{\ell\leqslant
n}(\sigma_{i}^{\ell})^{2}\Bigr{)}=\frac{d}{N}(N+\|\vec{\sigma}\|^{2})$ (37)

and

$\displaystyle|Q(\vec{\sigma})|$
$\displaystyle\leqslant\frac{d}{N}\Bigl{(}M+\sum_{\ell\leqslant
n}\sum_{k\leqslant
M}(S_{k}(\sigma^{\ell})+z_{k})^{2}\Bigr{)}\leqslant\frac{d}{N}\Bigl{(}M+2\|\bar{G}\|_{op}^{2}\|\vec{\sigma}\|^{2}+2n\|z\|^{2}\Bigr{)}.$
(38)

By (37), if $t+d|\lambda|\leqslant\kappa/2$, then

$\displaystyle t\|\vec{\sigma}\|^{2}-\kappa\|\vec{\sigma}\|^{2}+\lambda
NR(\vec{\sigma})\leqslant|\lambda|dN-\frac{\kappa}{2}\|\vec{\sigma}\|^{2},$

which, together with (32) for $|\eta|\leqslant\eta_{0}$ implies that,

$\displaystyle\int\exp\big{(}{t\|\vec{\sigma}\|^{2}+X_{N,\lambda,\eta}(\vec{\sigma})}\big{)}\,d\vec{\sigma}$
$\displaystyle\leqslant\exp
N\Bigl{(}d+d|\lambda|+\frac{nh^{2}}{2\kappa}+\frac{n}{2}\ln\frac{2\pi}{\kappa}\Bigr{)},$
(39)

where we used that for any $a>0$ and $b\in\mathbb{R}$,

$\displaystyle\int\exp({-ax^{2}+bx})\,dx$
$\displaystyle=\Bigl{(}\frac{\pi}{a}\Bigr{)}^{1/2}\exp\frac{b^{2}}{4a}.$ (40)

Next, since by (28)

$\displaystyle u(S_{k}(\sigma)+z_{k})$
$\displaystyle\geqslant-d(1+2S_{k}(\sigma)^{2}+2z_{k}^{2}),$

we have

$\displaystyle\sum_{k\leqslant M}u(S_{k}(\sigma)+z_{k})$
$\displaystyle\geqslant-d\bigl{(}M+2\|\bar{G}\|_{op}^{2}\|\sigma\|^{2}+2\|z\|^{2}\bigr{)}.$

Combining this with (37) and (38), we obtain that

$\displaystyle X_{N,\lambda,\eta}(\vec{\sigma})$
$\displaystyle\geqslant-\widetilde{T}\|\vec{\sigma}\|^{2}-N\widetilde{T}^{\prime}+h\sum_{\ell\leqslant
n}\sum_{i\leqslant N}\sigma_{i}^{\ell},$

where

$\displaystyle\widetilde{T}$
$\displaystyle:=\kappa+{2d(1+|\eta|)}\|\bar{G}\|_{op}^{2}+d|\lambda|>0\quad\mbox{and}\quad\widetilde{T}^{\prime}:=d\bigl{(}\alpha
n+|\lambda|+\alpha|\eta|\bigr{)}+2nd(1+|\eta|)\|\bar{z}\|^{2}>0.$

Consequently, from (40),

$\displaystyle Z_{N,\lambda,\eta}\geqslant\exp
nN\Bigl{(}-\frac{\widetilde{T}^{\prime}}{n}+\frac{h^{2}}{4\widetilde{T}}+\frac{1}{2}\ln\frac{\pi}{\widetilde{T}}\Bigr{)}\geqslant\exp
nN\Bigl{(}-\frac{\widetilde{T}^{\prime}}{n}-\frac{1}{2}\ln{\widetilde{T}}\Bigr{)},$

which together with the fact that $\ln x\leqslant x+1$ for all $x>0$ implies
that

$\displaystyle\frac{1}{Z_{N,\lambda,\eta}}\leqslant\exp
nN\Bigl{(}\frac{\widetilde{T}^{\prime}}{n}+\frac{1}{2}({\widetilde{T}}+1)\Bigr{)}\leqslant\exp
K^{\prime}N\bigl{(}\|\bar{z}\|^{2}+\|\bar{G}\|_{op}^{2}+1\bigr{)}.$

Our assertion follows by putting this and (39) together.

$\square$

We let $I(A)$ be the indicator function for event $A$.

######  Lemma 4.3.

There exist constants $\varepsilon>0$ and $K,K^{\prime}>0$ such that, for any
$|\lambda|,|\eta|\leqslant\varepsilon$,

$\displaystyle\Bigl{\langle}I\Bigl{(}\|\vec{\sigma}\|\geqslant
K^{\prime}\sqrt{N(\|\bar{z}\|^{2}+\|\bar{G}\|_{op}^{2}+1+h^{2})}\Bigr{)}\Bigr{\rangle}_{\lambda,\eta}$
$\displaystyle\leqslant\exp({-KN}).$

_Proof._ Recall the constants $\varepsilon$ and $K$ from the statement of
Lemma 4.2. If $t\leqslant\varepsilon$, then for any $a>0$,

$\bigl{\langle}I(\|\vec{\sigma}\|\geqslant
a)\bigr{\rangle}_{\lambda,\eta}\leqslant\bigl{\langle}\exp
t(\|\vec{\sigma}\|^{2}-a^{2})\bigr{\rangle}_{\lambda,\eta}\leqslant\exp(-{a^{2}t}+{KN(\|\bar{z}\|^{2}+\|\bar{G}\|_{op}^{2}+1+h^{2})}).$

Our proof is completed by taking
$a=\sqrt{{2t^{-1}KN(\|\bar{z}\|^{2}+\|\bar{G}\|_{op}^{2}+1+h^{2})}}.$

$\square$

######  Lemma 4.4.

Let $k\geqslant 1.$ There exist $0<\varepsilon<1$ and $K\geqslant 1$ such
that, for any $|\lambda|,|\eta|\leqslant\varepsilon$,

$\displaystyle\bigl{\langle}\|\vec{\sigma}\|^{2k}\bigr{\rangle}_{\lambda,\eta}$
$\displaystyle\leqslant
KN^{k}\bigl{(}\|\bar{z}\|^{2k}+\|\bar{G}\|_{op}^{2k}+1+h^{2k}\bigr{)}.$ (41)

_Proof._ From Lemma 4.2,

$\displaystyle\ln\bigl{\langle}\exp{t\|\vec{\sigma}\|^{2}}\bigr{\rangle}_{\lambda,\eta}\leqslant{KN(\|\bar{z}\|^{2}+\|\bar{G}\|_{op}^{2}+1+h^{2})}.$

For any non-negative random variable $X$, by Lemma 3.1.8 in [48],

${\mathbb{E}}X^{k}\leqslant 2^{k}\bigl{(}k^{k}+\bigl{(}\ln{\mathbb{E}}\exp
X\bigr{)}^{k}\bigr{)}.$

Using this inequality with $X=t\|\vec{\sigma}\|^{2}$, we have

$\displaystyle
t^{k}\bigl{\langle}\|\vec{\sigma}\|^{2k}\bigr{\rangle}_{\lambda,\eta}$
$\displaystyle\leqslant
2^{k}\bigl{(}k^{k}+K^{k}N^{k}(\|\bar{z}\|^{2}+\|\bar{G}\|_{op}^{2}+1+h^{2})^{k}\bigr{)}$
$\displaystyle\leqslant
2^{k}\bigl{(}k^{k}+4^{k}K^{k}N^{k}(\|\bar{z}\|^{2k}+\|\bar{G}\|_{op}^{2k}+1+h^{2k})\bigr{)}$
$\displaystyle\leqslant
C_{k}N^{k}\bigl{(}\|\bar{z}\|^{2k}+\|\bar{G}\|_{op}^{2k}+1+h^{2k}\bigr{)}.$

This finishes the proof.

$\square$

###  4.2 Convexity and pseudo-Lipschitz property of the overlaps

######  Lemma 4.5.

If $|\lambda|+\|\bar{G}\|_{op}^{2}|\eta|\leqslant\kappa/(dn)$ then the
following function is convex:

$\displaystyle\vec{\sigma}\mapsto\frac{\kappa}{2}\|\vec{\sigma}\|^{2}-\lambda
NR(\vec{\sigma})-\eta NQ(\vec{\sigma}).$ (42)

_Proof._ Recall the generalized overlap definitions (34) and (35). Note that
for any
$\vec{y}=(y^{1},\ldots,y^{n})\in\mathbb{R}^{N}\times\cdots\times\mathbb{R}^{N}$,

$\displaystyle\sum_{\ell,\ell^{\prime}\leqslant n}\sum_{i,i^{\prime}\leqslant
N}\partial_{\sigma_{i}^{\ell}\sigma_{i^{\prime}}^{\ell^{\prime}}}\Bigl{(}\sum_{j\leqslant
N}\phi(\sigma_{j}^{1},\ldots,\sigma_{j}^{n})\Bigr{)}y_{i}^{\ell}y_{i^{\prime}}^{\ell^{\prime}}=\sum_{\ell,\ell^{\prime}\leqslant
n}\Bigl{(}\sum_{i\leqslant
N}\partial_{s_{\ell}s_{\ell^{\prime}}}\phi(\sigma_{i}^{1},\ldots,\sigma_{i}^{n})y_{i}^{\ell}y_{i}^{\ell^{\prime}}\Bigr{)}.$

Since, by (31), $|\partial_{s_{\ell}s_{\ell^{\prime}}}\phi(s)|\leqslant d$,
this is bounded in absolute value by

$\displaystyle d\sum_{i\leqslant N}\Bigl{(}\sum_{\ell\leqslant
n}|y_{i}^{\ell}|\Bigr{)}^{2}\leqslant dn\|\vec{y}\|^{2}.$

Similarly,

$\displaystyle\sum_{\ell,\ell^{\prime}\leqslant n}\sum_{i,i^{\prime}\leqslant
N}\partial_{\sigma_{i}^{\ell}\sigma_{i^{\prime}}^{\ell^{\prime}}}\Bigl{(}\sum_{k\leqslant
M}\psi\bigl{(}S_{k}(\sigma^{1})+z_{k},\ldots,S_{k}(\sigma^{n})+z_{k}\bigr{)}\Bigr{)}y_{i}^{\ell}y_{i^{\prime}}^{\ell^{\prime}}$
$\displaystyle\qquad=\sum_{\ell,\ell^{\prime}\leqslant n}\sum_{k\leqslant
M}\partial_{s_{\ell}s_{\ell^{\prime}}}\psi\bigl{(}S_{k}(\sigma^{1})+z_{k},\ldots,S_{k}(\sigma^{n})+z_{k}\bigr{)}\Bigl{(}\sum_{i\leqslant
N}\bar{g}_{ki}y_{i}^{\ell}\Bigr{)}\Bigl{(}\sum_{i^{\prime}\leqslant
N}\bar{g}_{ki^{\prime}}y_{i^{\prime}}^{\ell^{\prime}}\Bigr{)},$

and since, by (31), $|\partial_{s_{\ell}s_{\ell^{\prime}}}\psi(s)|\leqslant
d$, this is bounded in absolute value by

$\displaystyle
d\sum_{\ell,\ell^{\prime}=1}^{n}|(\bar{G}y^{\ell},\bar{G}y^{\ell^{\prime}})|\leqslant
d\|\bar{G}\|_{op}^{2}\Bigl{(}\sum_{\ell\leqslant
n}\|y^{\ell}\|\Bigr{)}^{2}\leqslant d{n}\|\bar{G}\|_{op}^{2}\|\vec{y}\|^{2}.$

These complete our proof.

$\square$

######  Lemma 4.6.

We have that

$\displaystyle\bigl{|}R(\vec{\sigma})-R(\vec{\sigma}^{\prime})\bigr{|}$
$\displaystyle\leqslant\frac{d}{N}\Bigl{(}\sqrt{N}+\|\vec{\sigma}\|+\|\vec{\sigma}^{\prime}\|\Bigr{)}\|\vec{\sigma}-\vec{\sigma}^{\prime}\|$

and

$\displaystyle\bigl{|}Q(\vec{\sigma})-Q(\vec{\sigma}^{\prime})\bigr{|}\leqslant\frac{d}{N}\Bigl{(}\sqrt{M}+2\sqrt{n}\|z\|+\|\bar{G}\|_{op}\|\vec{\sigma}\|+\|\bar{G}\|_{op}\|\vec{\sigma}^{\prime}\|\Bigr{)}\|\bar{G}\|_{op}\|\vec{\sigma}-\vec{\sigma}^{\prime}\|.$

_Proof._ For simplicity, for $\vec{\sigma}=(\sigma^{1},\ldots,\sigma^{n})$, we
denote $\sigma(i)=(\sigma_{i}^{1},\ldots,\sigma_{i}^{n}).$ From (29),

$\displaystyle\bigl{|}R(\vec{\sigma})-R(\vec{\sigma}^{\prime})\bigr{|}$
$\displaystyle\leqslant\frac{d}{N}\sum_{i\leqslant
N}\bigl{(}1+\|\sigma(i)\|+\|\sigma^{\prime}(i)\|\bigr{)}\|\sigma(i)-\sigma^{\prime}(i)\|$
$\displaystyle\leqslant\frac{d}{N}\Bigl{(}\sum_{i\leqslant
N}\bigl{(}1+\|\sigma(i)\|+\|\sigma^{\prime}(i)\|\bigr{)}^{2}\Bigr{)}^{1/2}\Bigl{(}\sum_{i\leqslant
N}\|\sigma(i)-\sigma^{\prime}(i)\|^{2}\Bigr{)}^{1/2}$
$\displaystyle\leqslant\frac{d}{N}\Bigl{(}\Bigl{(}\sum_{i\leqslant
N}1\Bigr{)}^{1/2}+\Bigl{(}\sum_{i\leqslant
N}\|\sigma(i)\|^{2}\Bigr{)}^{1/2}+\Bigl{(}\sum_{i\leqslant
N}\|\sigma^{\prime}(i)\|^{2}\Bigr{)}^{1/2}\Bigr{)}\|\vec{\sigma}-\vec{\sigma}^{\prime}\|$
$\displaystyle\leqslant\frac{d}{N}\Bigl{(}\sqrt{N}+\|\vec{\sigma}\|+\|\vec{\sigma}^{\prime}\|\Bigr{)}\|\vec{\sigma}-\vec{\sigma}^{\prime}\|,$

where in the third line we used the Minkowski inequality. Again, by (29),

$\displaystyle\bigl{|}Q(\vec{\sigma})-Q(\vec{\sigma}^{\prime})\bigr{|}$
$\displaystyle\quad\leqslant\frac{d}{N}\sum_{k\leqslant
M}\Bigl{(}1+\Bigl{(}\sum_{\ell\leqslant
n}|S_{k}(\sigma^{\ell})+z_{k}|^{2}\Bigr{)}^{1/2}+\Bigl{(}\sum_{\ell\leqslant
n}|S_{k}({\sigma^{\prime}}^{\ell})+z_{k}|^{2}\Bigr{)}^{1/2}\Bigr{)}$
$\displaystyle\qquad\qquad\qquad\qquad\times\Bigl{(}\sum_{\ell\leqslant
n}S_{k}(\sigma^{\ell}-{\sigma^{\prime}}^{\ell})^{2}\Bigr{)}^{1/2}$
$\displaystyle\quad\leqslant\frac{d}{N}\Bigl{(}\sum_{k\leqslant
M}\Bigl{(}1+\Bigl{(}\sum_{\ell\leqslant
n}|S_{k}(\sigma^{\ell})+z_{k}|^{2}\Bigr{)}^{1/2}+\Bigl{(}\sum_{\ell\leqslant
n}|S_{k}({\sigma^{\prime}}^{\ell})+z_{k}|^{2}\Bigr{)}^{1/2}\Bigr{)}^{2}\Bigr{)}^{1/2}$
$\displaystyle\qquad\qquad\qquad\qquad\times\Bigl{(}\sum_{k\leqslant
M}\sum_{\ell\leqslant
n}S_{k}(\sigma^{\ell}-{\sigma^{\prime}}^{\ell})^{2}\Bigr{)}^{1/2}.$

Here, since

$\Bigl{(}\sum_{k\leqslant M}\sum_{\ell\leqslant
n}S_{k}(\sigma^{\ell}-{\sigma^{\prime}}^{\ell})^{2}\Bigr{)}^{1/2}=\Bigl{(}\sum_{\ell\leqslant
n}\|\bar{G}(\sigma^{\ell}-{\sigma^{\prime}}^{\ell})\|^{2}\Bigr{)}^{1/2}\leqslant\|\bar{G}\|_{op}\|\vec{\sigma}-\vec{\sigma}^{\prime}\|,$

and, similarly,

$\displaystyle\Bigl{(}\sum_{k\leqslant M}\sum_{\ell\leqslant
n}|S_{k}(\sigma^{\ell})+z_{k}|^{2}\Bigr{)}^{1/2}$
$\displaystyle\leqslant\Bigl{(}\sum_{k\leqslant M}\sum_{\ell\leqslant
n}S_{k}(\sigma^{\ell})^{2}\Bigr{)}^{1/2}+\Bigl{(}\sum_{k\leqslant
M}\sum_{\ell\leqslant
n}z_{k}^{2}\Bigr{)}^{1/2}\leqslant\|\bar{G}\|_{op}\|\vec{\sigma}\|+\sqrt{n}\|z\|,$

we can again use the Minkowski inequality to obtain

$\displaystyle\bigl{|}Q(\vec{\sigma})-Q(\vec{\sigma}^{\prime})\bigr{|}\leqslant\frac{d}{N}\Bigl{(}\sqrt{M}+2\sqrt{n}\|z\|+\|\bar{G}\|_{op}\|\vec{\sigma}\|+\|\bar{G}\|_{op}\|\vec{\sigma}^{\prime}\|\Bigr{)}\|\bar{G}\|_{op}\|\vec{\sigma}-\vec{\sigma}^{\prime}\|.$

$\square$

######  Lemma 4.7.

For any $k\geqslant 1$ there exists some $C_{k}>0$ such that

$\displaystyle{\mathbb{E}}\Bigl{\langle}|R(\vec{\sigma})|^{k}\Bigr{\rangle}_{0,0}\leqslant
C_{k}(1+h^{2k})\quad\mbox{ and
}\quad{\mathbb{E}}\Bigl{\langle}|Q(\vec{\sigma})|^{k}\Bigr{\rangle}_{0,0}\leqslant
C_{k}(1+h^{2k}).$

_Proof._ Let us denote by $\vec{s}:=\vec{\sigma}/\sqrt{N}.$ Note that from
Lemma 4.6,

$\displaystyle\bigl{|}R(\vec{\sigma})\bigr{|}$
$\displaystyle\leqslant{|R(0)|}+\frac{d}{N}\bigl{(}\sqrt{N}+\|\vec{\sigma}\|\bigr{)}\|\vec{\sigma}\|\leqslant
d+d\bigl{(}1+\|\vec{s}\|\bigr{)}\|\vec{s}\|,$
$\displaystyle\bigl{|}Q(\vec{\sigma})\bigr{|}$
$\displaystyle\leqslant{|Q(0)|}+\frac{d}{N}\bigl{(}\sqrt{M}+2\sqrt{n}\|z\|+\|\bar{G}\|_{op}\|\vec{\sigma}\|\bigr{)}\|\bar{G}\|_{op}\|\vec{\sigma}\|$
$\displaystyle\leqslant{d(\alpha+n\|\bar{z}\|^{2})}+d\bigl{(}\sqrt{\alpha}+2\sqrt{n}\|\bar{z}\|+\|\bar{G}\|_{op}\|\vec{s}\|\bigr{)}\|\bar{G}\|_{op}\|\vec{s}\|,$

where we used the assumption (30). Using Lemma 4.4 with $\lambda=\eta=0$, we
see that there exists some $C$ such that

$\big{\langle}\|\vec{s}\|^{2k}\big{\rangle}_{0,0}\leqslant
C\bigl{(}\|\bar{z}\|^{2k}+\|\bar{G}\|_{op}^{2k}+1+h^{2k}\bigr{)}.$

By Corollary 5.35 in [52],

${\mathbb{P}}(\|\bar{G}\|_{op}\geqslant 1+\sqrt{\alpha}+t)\leqslant
2\exp({-Nt^{2}/2}),$ (43)

which implies that ${\mathbb{E}}\|\bar{G}\|_{op}^{m}\leqslant C_{m}<\infty$
for any $m\geqslant 1$ and some constant $C_{m}$ independent of $N$.
Similarly, by gaussian concentration for gaussian vectors,
${\mathbb{E}}\|\bar{z}\|^{m}\leqslant C_{m}<\infty$. Together with the above
inequalities, this finishes the proof.

$\square$

###  4.3 Concentration of the free energy

For clarity, here we will now denote $F_{N}(\lambda,\eta)$ by $F(G,z)$. Let us
start with the following result.

######  Lemma 4.8.

There exist constants $\varepsilon>0$ and $K>0$ such that for all
$|\lambda|,|\eta|\leqslant\varepsilon$,

$\displaystyle\bigl{|}F(G,z)-F(G^{\prime},z^{\prime})\bigr{|}$
$\displaystyle\quad\leqslant
K\Bigl{(}\|\bar{z}\|^{3}+\|\bar{G}\|_{op}^{3}+\|\bar{z}^{\prime}\|^{3}+\|\bar{G}^{\prime}\|_{op}^{3}+1+\|\bar{G}\|_{op}h^{2}+\|\bar{G}^{\prime}\|_{op}h^{2}+h^{2}\Bigr{)}\bigl{\|}(\bar{G}-\bar{G}^{\prime},\bar{z}-\bar{z}^{\prime})\bigr{\|}.$

_Proof._ First of all,

$\displaystyle\partial_{g_{ki}}F(G,z)$
$\displaystyle=\frac{1}{N^{3/2}}\sum_{\ell\leqslant
n}\Bigl{\langle}\Bigl{(}u^{\prime}\big{(}S_{k}(\sigma^{\ell})+z_{k}\big{)}+{\eta}\partial_{x_{\ell}}\psi\bigl{(}S_{k}(\sigma^{1})+z_{k},\ldots,S_{k}(\sigma^{n})+z_{k}\bigr{)}\Bigr{)}\sigma_{i}^{\ell}\Bigr{\rangle}_{\lambda,\eta}.$

Using that $|u^{\prime}(s)|\leqslant d(1+|s|)$ by (28) (below the constant $K$
amy change from place to place),

$\displaystyle\sum_{k\leqslant M}\sum_{i\leqslant
N}\Bigl{\langle}\sum_{\ell\leqslant
n}u^{\prime}\big{(}S_{k}(\sigma^{\ell})+z_{k}\big{)}\sigma_{i}^{\ell}\Bigr{\rangle}_{\lambda,\eta}^{2}$
$\displaystyle\leqslant\Bigl{\langle}\Bigl{(}\sum_{k\leqslant
M}\sum_{\ell\leqslant
n}u^{\prime}\big{(}S_{k}(\sigma^{\ell})+z_{k}\big{)}^{2}\Bigr{)}\sum_{i\leqslant
N}\sum_{\ell\leqslant n}(\sigma_{i}^{\ell})^{2}\Bigr{\rangle}_{\lambda,\eta}$
$\displaystyle\leqslant 3d\Bigl{\langle}\Big{(}\sum_{k\leqslant
M}\sum_{\ell\leqslant
n}\bigl{(}1+S_{k}(\sigma^{\ell})^{2}+z_{k}^{2}\bigr{)}\Big{)}\|\vec{\sigma}\|^{2}\Big{\rangle}_{\lambda,\eta}$
$\displaystyle\leqslant
3d\Bigl{\langle}\bigl{(}nM+\|\bar{G}\|_{op}^{2}\|\vec{\sigma}\|^{2}+n\|z\|^{2}\bigr{)}\|\vec{\sigma}\|^{2}\Big{\rangle}_{\lambda,\eta}$
$\displaystyle\leqslant
KN^{2}\bigl{(}\|\bar{z}\|^{6}+\|\bar{G}\|_{op}^{6}+1+\|\bar{G}\|_{op}^{2}h^{4}+h^{4}\bigr{)}$

where the last inequality used (41). Note that from (31), there exists a
constant $d^{\prime}$ such that

$\displaystyle\bigl{|}\partial_{s_{\ell}}\psi(s)\bigr{|}$
$\displaystyle\leqslant d^{\prime}\bigl{(}1+\|s\|\bigr{)}.$

Using this inequality, we can argue similarly that

$\displaystyle\sum_{k\leqslant M}\sum_{i\leqslant
N}\Bigl{\langle}\sum_{\ell\leqslant
n}\partial_{x_{\ell}}\psi\bigl{(}S_{k}(\sigma^{1})+z_{k},\ldots,S_{k}(\sigma^{n})+z_{k}\bigr{)}\sigma_{i}^{\ell}\Bigr{\rangle}_{\lambda,\eta}^{2}$
$\displaystyle\qquad\leqslant{d^{\prime}}^{2}\Bigl{\langle}\Bigl{(}\sum_{k\leqslant
M}\sum_{\ell\leqslant
n}\Bigl{[}1+\Bigl{(}\sum_{\ell^{\prime}=1}^{n}|S_{k}(\sigma^{\ell^{\prime}})+z_{k}|^{2}\Bigr{)}^{1/2}\Bigr{]}^{2}\Bigr{)}\sum_{i\leqslant
N}\sum_{\ell\leqslant n}(\sigma_{i}^{\ell})^{2}\Bigr{\rangle}_{\lambda,\eta}$
$\displaystyle\qquad\leqslant
4{d^{\prime}}^{2}n\Bigl{\langle}\Bigl{(}\sum_{k\leqslant
M}\Bigl{[}1+\sum_{\ell^{\prime}=1}^{n}(S_{k}(\sigma^{\ell^{\prime}})^{2}+z_{k}^{2})\Bigr{]}\Bigr{)}\|\vec{\sigma}\|^{2}\Bigr{\rangle}_{\lambda,\eta}$
$\displaystyle\qquad\leqslant
4{d^{\prime}}^{2}n\Bigl{\langle}\Bigl{(}M+\|\bar{G}\|_{op}^{2}\|\vec{\sigma}\|^{2}+n\|z\|^{2}\Bigr{)}\|\vec{\sigma}\|^{2}\Bigr{\rangle}_{\lambda,\eta}$
$\displaystyle\qquad\leqslant
K^{\prime}N^{2}\bigl{(}\|\bar{z}\|^{6}+\|\bar{G}\|_{op}^{6}+1+\|\bar{G}\|_{op}^{2}h^{4}+h^{4}\bigr{)}.$

Putting these together yields that

$\displaystyle\sum_{k\leqslant M}\sum_{i\leqslant
N}|\partial_{g_{ki}}F(G,z)|^{2}$
$\displaystyle\leqslant\frac{K^{\prime\prime}}{N}\bigl{(}\|\bar{z}\|^{6}+\|\bar{G}\|_{op}^{6}+1+\|\bar{G}\|_{op}^{2}h^{4}+h^{4}\bigr{)}.$

In similar manner, we readily compute that

$\displaystyle\partial_{z_{k}}F(G,z)$
$\displaystyle=\frac{1}{N}\sum_{\ell\leqslant
n}\Bigl{\langle}u^{\prime}\big{(}S_{k}(\sigma^{\ell})+z_{k}\big{)}+{\eta}\partial_{x_{\ell}}\psi\bigl{(}S_{k}(\sigma^{1})+z_{k},\ldots,S_{k}(\sigma^{n})+z_{k}\bigr{)}\Bigr{\rangle}_{\lambda,\eta}$

and

$\displaystyle\sum_{k\leqslant M}|\partial_{z_{k}}F(G,z)|^{2}$
$\displaystyle\leqslant\frac{K^{\prime\prime\prime}}{N}\bigl{(}\|\bar{z}\|^{4}+1+h^{4}\bigr{)}.$

All together, we get that

$\|\nabla
F(G,z)\|\leqslant\frac{K_{4}}{\sqrt{N}}\bigl{(}\|\bar{z}\|^{3}+\|\bar{G}\|_{op}^{3}+1+\|\bar{G}\|_{op}h^{2}+h^{2}\bigr{)},$

for some $\delta>0$, which implies Lemma 4.8.

$\square$

######  Lemma 4.9.

There exist absolute constants $\varepsilon>0$ and $K>0$ such that, for all
$|\lambda|,|\eta|\leqslant\varepsilon$,

$\displaystyle\mathrm{Var}F_{N}(\lambda,\eta)\leqslant\frac{K(1+h^{4})}{N}.$

_Proof._ For simplicity of notation, denote

$y:=(G,z),\quad f(y):=F(G,z),\quad\mbox{ and }\quad
r(y):=\|\bar{G}\|_{op}^{3}+\|\bar{z}\|^{3}+\|\bar{G}\|_{op}h^{2}.$

If we define $\bar{y}:=y/\sqrt{N}$, then the above Lemma proves that

$\displaystyle\big{|}f(y)-f(y^{\prime})\big{|}\leqslant
K\bigl{(}r(y)+r(y^{\prime})+1\bigr{)}\|\bar{y}-\bar{y}^{\prime}\|,$

Let $L>0$ be a constant to be chosen later and $L_{h}:=L(1+h^{2})$. Since, for
$r(y),r(y^{\prime})\leqslant L_{h}$,

$\displaystyle f(y)$ $\displaystyle\leqslant
f(y^{\prime})+K(2L_{h}+1)\|\bar{y}-\bar{y}^{\prime}\|$

and we have equality for $y^{\prime}=y$, if we define

$\displaystyle\Phi(y)$
$\displaystyle:=\inf\bigl{\\{}f(y^{\prime})+K(2L_{h}+1)\|\bar{y}-\bar{y}^{\prime}\|:r(y^{\prime})\leqslant
L_{h}\bigr{\\}},$

then $\Phi(y)=f(y)$ for all $y$ such that $r(y)\leqslant L_{h}$. From this
definition, it can be seen from the reverse triangle inequality that

$\displaystyle\big{|}\Phi(y)-\Phi(y^{\prime})\big{|}$ $\displaystyle\leqslant
K(2L_{h}+1)\|\bar{y}-\bar{y}^{\prime}\|=\frac{K(2L_{h}+1)}{\sqrt{N}}\|y-y^{\prime}\|$
(44)

and

$\displaystyle\big{|}\Phi(y)-f(y)\big{|}$
$\displaystyle\leqslant\bigl{(}|\Phi(y)|+|f(y)|\bigr{)}I(r(y)>L_{h}).$ (45)

By (44) and the Gaussian-Poincaré concentration inequality,

$\displaystyle{\mathbb{E}}\bigl{|}\Phi(y)-{\mathbb{E}}\Phi(y)\bigr{|}^{2}\leqslant{\mathbb{E}}\|\nabla\Phi(y)\|^{2}\leqslant\frac{K^{2}(2L_{h}+1)^{2}}{N}\leqslant\frac{K^{\prime}(1+h^{4})}{N}.$
(46)

Next, using (45) and the Cauchy-Schwarz inequality, we control

$\displaystyle\big{|}{\mathbb{E}}f(y)-{\mathbb{E}}\Phi(y)\big{|}$
$\displaystyle\leqslant{\mathbb{E}}\big{|}f(y)-\Phi(y)\big{|}$
$\displaystyle\leqslant{\mathbb{E}}\bigl{(}|\Phi(y)|+|f(y)|\bigr{)}I(r(y)>L_{h})$
$\displaystyle\leqslant\sqrt{2\bigl{(}{\mathbb{E}}f(y)^{2}+{\mathbb{E}}\Phi(y)^{2}\bigr{)}{\mathbb{P}}(r(y)\geqslant
L_{h})}.$

Recalling that
$r(y)=\|\bar{G}\|_{op}^{3}+\|\bar{z}\|^{3}+\|\bar{G}\|_{op}h^{2}$, by (43) and
gaussian concentration for $\|\bar{z}\|$, for large enough $L>0$, we have
${\mathbb{P}}(r(y)\geqslant L_{h})\leqslant 4\exp(-N)$. On the other hand, by
Lemma 4.8 with $(G,z)=y$ and $(G^{\prime},z^{\prime})=0$ we have

$\displaystyle f(y)^{2}\,$ $\displaystyle{\leqslant
2f(0)^{2}+2K^{2}\Bigl{(}\|\bar{z}\|^{3}+\|\bar{G}\|_{op}^{3}+1+\|\bar{G}\|_{op}h^{2}+h^{2}\Bigr{)}^{2}\frac{\|y\|^{2}}{N}}$
$\displaystyle{\leqslant
2f(0)^{2}+2K^{2}N^{4}\Bigl{(}\frac{\|\bar{z}\|^{3}}{N^{3/2}}+\frac{\|\bar{G}\|^{3}}{N^{3/2}}+\frac{1+h^{2}}{N^{3/2}}+\frac{\|\bar{G}\|h^{2}}{N^{3/2}}\Bigr{)}^{2}\Big{(}\frac{\|\bar{G}\|^{2}}{N}+\frac{\|\bar{z}\|^{2}}{N}\Big{)},}$

and since $|f(0)|$ does not depend on $N$ and all the moments of $\|\bar{z}\|$
and $\|\bar{G}\|/\sqrt{N}$ are uniformly bounded, we have that
${\mathbb{E}}f(y)^{2}\leqslant K^{\prime}(1+h^{4})N^{4}$. Similarly,

$\displaystyle{\Phi(y)^{2}\leqslant
2f(0)^{2}+2K^{2}N(2L_{h}+1)^{2}\Big{(}\frac{\|\bar{G}\|^{2}}{N}+\frac{\|\bar{z}\|^{2}}{N}\Big{)},}$

from which we conclude that ${\mathbb{E}}\Phi(y)^{2}\leqslant
K^{\prime\prime}(1+h^{4})N$. Finally, using the bound

$\displaystyle{\mathbb{E}}\bigl{|}f(y)-{\mathbb{E}}f(y)\bigr{|}^{2}\leqslant
3\bigl{(}{\mathbb{E}}\bigl{|}f(y)-\Phi(y)\bigr{|}^{2}+{\mathbb{E}}\bigl{|}\Phi(y)-{\mathbb{E}}\Phi(y)\bigr{|}^{2}+\bigl{|}{\mathbb{E}}\Phi(y)-{\mathbb{E}}f(y)\bigr{|}^{2}\bigr{)}$

finishes the proof.

$\square$

###  4.4 Proof of Proposition 4.1

We will only consider the case of the overlap $R(\vec{\sigma})$, since the
proof for $Q(\vec{\sigma})$ is almost identical. In this section we will
denote by $K$ a constant that depends only on the parameters of the model, but
can vary from one occurrence to the next. Let us consider the event

$E_{N}:=\Bigl{\\{}\|\bar{G}\|_{op}\leqslant
K\Bigr{\\}}\bigcap\,\Bigl{\\{}\|\bar{z}\|\leqslant K\Bigr{\\}}.$

By (43), we can and will choose $K\geqslant 1$ so that the probability of the
event $E_{N}$ is at least $1-4e^{-N}$. From Lemma 4.7,

$\displaystyle{\mathbb{E}}\Bigl{[}\Bigl{\langle}\Bigl{|}R(\vec{\sigma})-{\mathbb{E}}\langle
R(\vec{\sigma})\rangle_{0,0}\Bigr{|}^{2}\Bigr{\rangle}_{0,0}I({E_{N}^{c}})\Bigr{]}$
$\displaystyle\leqslant 4\Bigl{(}{\mathbb{E}}\langle
R(\vec{\sigma})^{4}\rangle_{0,0}\,{\mathbb{P}}(E_{N}^{c})\Bigr{)}^{1/2}\leqslant
K\exp(-N/2).$ (47)

Let $F_{N}^{\prime}$ be the $\lambda$-derivative of $F_{N}$. In the remainder
we will control the same expectation on the event $E_{N}$, and we will use the
convexity of $F_{N}(\lambda,\eta)$ in $\lambda$, which implies that

$\displaystyle\Bigl{|}\langle R(\vec{\sigma})\rangle_{0,0}-{\mathbb{E}}\langle
R(\vec{\sigma})\rangle_{0,0}\Bigr{|}=\Bigl{|}F_{N}^{\prime}(0,0)-{\mathbb{E}}F_{N}^{\prime}(0,0)\Bigr{|}$
$\displaystyle\qquad\leqslant
F_{N}^{\prime}(\lambda,0)-F_{N}^{\prime}(-\lambda,0)+\frac{1}{\lambda}\Bigl{(}|F_{N}(\lambda,0)-{\mathbb{E}}F_{N}(\lambda,0)|$
$\displaystyle\qquad\qquad+|F_{N}(0,0)-{\mathbb{E}}F_{N}(0,0)|+|F_{N}(-\lambda,0)-{\mathbb{E}}F_{N}(-\lambda,0)|\Bigr{)}.$
(48)

Take $\varepsilon>0$ such that, for $|\lambda|,|\eta|\leqslant\varepsilon$,
the inequalities in Lemmas 4.3, 4.4 and 4.9 hold. By Lemma 4.9, the
expectation of the square of the term between the parentheses above can be
bounded by $K/(N\lambda^{2})$.

In order to control the first term
$F_{N}^{\prime}(\lambda,0)-F_{N}^{\prime}(-\lambda,0)$ above, our goal will be
to control

$\displaystyle\partial_{\lambda\lambda}F_{N}(\lambda,\eta)$
$\displaystyle=N\Bigl{\langle}\bigl{|}R(\vec{\sigma})-\langle
R(\vec{\sigma})\rangle_{\lambda,\eta}\bigr{|}^{2}\Bigr{\rangle}_{\lambda,\eta}.$

For simplicity of notation we denote $\vec{s}:=\vec{\sigma}/\sqrt{N}$. Since,
for $|\lambda|,|\eta|\leqslant\varepsilon$, the inequalities in Lemmas 4.3 and
4.4 hold, on the event $E_{N}$, for some $L\geqslant 1$,

$\displaystyle\big{\langle}I(\|\vec{s}\|\geqslant
L)\big{\rangle}_{\lambda,\eta}\leqslant\exp(-KN)\quad\mbox{and}\quad\big{\langle}\|\vec{s}\|^{10}\big{\rangle}_{\lambda,\eta}\leqslant
K.$ (49)

Also, by Lemma 4.6,

$\displaystyle\big{|}R(\vec{\sigma})-R(\vec{\sigma}^{\prime})\big{|}$
$\displaystyle\leqslant
D\Bigl{(}1+\|\vec{s}\|+\|\vec{s}^{\prime}\|\Bigr{)}\|\vec{s}-\vec{s}^{\prime}\|,$
(50)

where we also denoted $\vec{s}^{\prime}:=\vec{\sigma}^{\prime}/\sqrt{N}$.

We now proceed as in the proof of Lemma 4.9. Take $L$ as in (49) and $D$ as in
(50), denote $r(\vec{\sigma}):=\|\vec{s}\|$ and let

$\displaystyle\rho(\vec{\sigma})$
$\displaystyle:=\inf\bigl{\\{}R(\vec{\sigma}^{\prime})+D(2L+1)\|\vec{s}-\vec{s}^{\prime}\|:r(\vec{s}^{\prime})\leqslant
L\bigr{\\}}.$

Then, from (50), $\rho(\vec{\sigma})=R(\vec{\sigma})$ for all
$r(\vec{\sigma})\leqslant L$,

$\displaystyle|\rho(\vec{\sigma})-\rho(\vec{\sigma}^{\prime})|$
$\displaystyle\leqslant D\bigl{(}2L+1\bigr{)}\|\vec{s}-\vec{s}^{\prime}\|,$
(51) $\displaystyle|\rho(\vec{\sigma})-R(\vec{\sigma})|$
$\displaystyle\leqslant\big{(}|\rho(\vec{\sigma})|+|R(\vec{\sigma})|\big{)}I(r(\vec{\sigma})>L).$
(52)

Using (52) and the Cauchy-Schwarz and Minkowski inequalities,

$\displaystyle\Bigl{\langle}|\rho(\vec{\sigma})-R(\vec{\sigma})|^{2}\Bigr{\rangle}_{\lambda,\eta}$
$\displaystyle\leqslant\Bigl{\langle}\bigl{(}|\rho(\vec{\sigma})|+|R(\vec{\sigma})|\bigr{)}^{2}I(r(\vec{\sigma})>L)\Bigr{\rangle}_{\lambda,\eta}$
$\displaystyle\leqslant
2\Bigl{(}\big{\langle}\rho(\vec{\sigma})^{4}\big{\rangle}_{\lambda,\eta}^{1/2}+\big{\langle}R(\vec{\sigma})^{4}\big{\rangle}_{\lambda,\eta}^{1/2}\Bigr{)}\big{\langle}I(r(\vec{\sigma})>L)\big{\rangle}_{\lambda,\eta}^{1/2}$
$\displaystyle\leqslant
2\Bigl{(}\big{\langle}\rho(\vec{\sigma})^{4}\big{\rangle}_{\lambda,\eta}^{1/2}+\big{\langle}R(\vec{\sigma})^{4}\big{\rangle}_{\lambda,\eta}^{1/2}\Bigr{)}\exp(-KN),$

where in the last line we used the first equation in (49). Since
$R(0)=\phi(0)$, by (50)

$|R(\vec{\sigma})|\leqslant|\phi(0)|+D\bigl{(}1+\|\vec{s}\|\bigr{)}\|\vec{s}\|.$

Since $\rho(0)=R(0)=\phi(0)$, by (51),

$|\rho(\vec{\sigma})|\leqslant|\phi(0)|+D\big{(}2L+1\big{)}\|\vec{s}\|.$

Therefore, the second equation in (49) implies that

$\bigl{\langle}\rho(\vec{\sigma})^{4}\bigr{\rangle}_{\lambda,\eta}^{1/2}+\bigl{\langle}R(\vec{\sigma})^{4}\bigr{\rangle}_{\lambda,\eta}^{1/2}\leqslant
K$

and we get that

$\Bigl{\langle}|\rho(\vec{\sigma})-R(\vec{\sigma})|^{2}\Bigr{\rangle}_{\lambda,\eta}\leqslant
K\exp(-KN).$ (53)

By Lemma 4.5, on the event $E_{N}$, if $|\lambda|+K|\eta|\leqslant\kappa/(dn)$
then function (42) is convex, and thus the Hamiltonian
$X_{N,\lambda,\eta}(\vec{\sigma})$ in (33) is strictly concave on
$\mathbb{R}^{Nn}$ and its Hessian is bounded by
$-\frac{\kappa}{2}\mathrm{Id}$. In this case, the Brascamp-Lieb inequality
(see, e.g., [15, Theorem 5.1]) implies that

$\Bigl{\langle}\bigl{|}\rho(\vec{\sigma})-\langle\rho(\vec{\sigma})\rangle_{\lambda,\eta}\bigr{|}^{2}\Bigr{\rangle}_{\lambda,\eta}\leqslant\frac{2}{\kappa}\Bigl{\langle}\|\nabla\rho(\vec{\sigma})\|^{2}\Bigr{\rangle}_{\lambda,\eta}\leqslant\frac{K}{N},$

where the last inequality used (51). Together with (53), this implies that, on
the event $E_{N}$,

$\displaystyle\partial_{\lambda\lambda}F_{N}(\lambda,\eta)$
$\displaystyle=N\Bigl{\langle}\bigl{|}R(\vec{\sigma})-\langle
R(\vec{\sigma})\rangle_{\lambda,\eta}\bigr{|}^{2}\Bigr{\rangle}_{\lambda,\eta}\leqslant
K,$

assuming as above that $|\lambda|,|\eta|\leqslant\varepsilon$ and
$|\lambda|+K|\eta|\leqslant\kappa/(dn)$. In particular,

$\Bigl{\langle}\bigl{|}R(\vec{\sigma})-\langle
R(\vec{\sigma})\rangle_{0,0}\bigr{|}^{2}\Bigr{\rangle}_{0,0}\leqslant\frac{K}{N}$
(54)

and
$\bigl{|}F_{N}^{\prime}(\lambda,0)-F_{N}^{\prime}(-\lambda,0)\bigr{|}\leqslant
K\lambda.$ Plugging this into (48) and, as we mentioned above, using Lemma 4.9
for the second term,

$\displaystyle{\mathbb{E}}\Bigl{[}\bigl{|}\langle
R(\vec{\sigma})\rangle_{0,0}-{\mathbb{E}}\langle
R(\vec{\sigma})\rangle_{0,0}\bigr{|}^{2}I({E_{N}})\Bigr{]}$
$\displaystyle\leqslant
K\Bigl{(}\lambda^{2}+\frac{1}{N\lambda^{2}}\Bigr{)}\leqslant\frac{K}{\sqrt{N}},$

by optimizing over $\lambda$. Using (54) on the event $E_{N}$ and then
combining with (47) gives

$\displaystyle{\mathbb{E}}\Bigl{\langle}\Bigl{|}R(\vec{\sigma})-{\mathbb{E}}\langle
R(\vec{\sigma})\rangle_{0,0}\Bigr{|}^{2}\Bigr{\rangle}_{0,0}\leqslant\frac{K}{\sqrt{N}}.$

This proves the first inequality in (36). The proof of the second inequality
is identical.

$\square$

##  5 Fixed point equations

This section establishes the proof of Theorem 3.3. The key ingredient here is
to show that equations (17)–(20) are satisfied by the subsequential limits of
the overlaps, based on a cavity/leave-one-out argument and gaussian
integration by parts.

###  5.1 M-cavity

For every measurable function $f:\mathbb{R}^{Nn}\mapsto\mathbb{R}$, define the
Gibbs expectation associated to the “cavity in $M$” by

$\langle
f\rangle_{c}:=\frac{1}{Z_{c}^{n}}\int_{\mathbb{R}^{Nn}}\big{(}\prod_{\ell\leqslant
n}d\sigma^{\ell}\big{)}f\exp{\sum_{\ell\leqslant n}H_{N,c}(\sigma^{\ell})},$

where the M-cavity Hamiltonian is

$H_{N,c}(\sigma^{\ell}):=\sum_{k=2}^{M}u\bigl{(}S_{k}(\sigma^{\ell})+z_{k}\bigr{)}+h\sum_{i\leqslant
N}\sigma_{i}^{\ell}-\kappa\|\sigma^{\ell}\|^{2}$

and $Z_{c}$ is its corresponding partition function. In other words, the
Hamiltonian here singles out the component $u(S_{M}(\sigma)+z_{M})$ from
$H_{N}(\sigma).$ It is easy to check that for every measurable
$f:\mathbb{R}^{Nn}\mapsto\mathbb{R}$,

$\langle f(\sigma^{1},\ldots,\sigma^{n})\rangle=\frac{\langle
f(\sigma^{1},\ldots,\sigma^{n})\exp{\sum_{\ell\leqslant
n}u(S_{1}(\sigma^{\ell})+z_{1})}\rangle_{c}}{\langle\exp{\sum_{\ell\leqslant
n}u(S_{1}(\sigma^{\ell})+z_{1})}\rangle_{c}},$ (55)

where $\langle\,\cdot\,\rangle$ is the expectation with respect to the Gibbs
measure proportional to $\exp\sum_{\ell\leqslant n}H_{N}(\sigma^{\ell})$ of
the “replicated system”. The following two lemmas will play an essential role
in establishing the fixed point equations in the next subsection.

######  Lemma 5.1.

For every $k\geqslant 1$, there exists some $C_{k}>0$ such that, uniformly on
$N$,

${\mathbb{E}}\big{\langle}|u(S_{1}(\sigma)+z_{1})|^{k}\big{\rangle}\leqslant
C_{k}.$

_Proof._ Here we will denote $u(S_{1}(\sigma)+z_{1})$ by $u_{1}\leqslant 0$.
Because $|u_{1}|^{k}$ is an increasing function of $|u_{1}|$ and
$\exp(-|u_{1}|)$ is decreasing, the Harris inequality implies
$\langle|u_{1}|^{k}\exp(-|u_{1}|)\rangle_{c}\leqslant\langle|u_{1}|^{k}\rangle_{c}\langle\exp(-|u_{1}|)\rangle_{c}$.
As a consequence we have that

${\mathbb{E}}\langle|u_{1}|^{k}\rangle={\mathbb{E}}\frac{\langle|u_{1}|^{k}\exp(-|u_{1}|)\rangle_{c}}{\langle\exp(-|u_{1}|)\rangle_{c}}\leqslant{\mathbb{E}}\langle|u_{1}|^{k}\rangle_{c}.$

Furthermore, because the standard gaussian vector
$g_{1}:=(g_{11},\dots,g_{1N})$ is independent of $\sigma$ under the cavity
measure, by the rotational invariance of $g_{1}$ we have that $\langle
g_{1},\sigma\rangle\sim g\|\sigma\|$ where $g$ is an independent standard
gaussian variable, so under the cavity measure $S_{1}(\sigma)\sim
g\|\sigma\|/\sqrt{N}=g\sqrt{R_{1,1}}$. Thus, by Assumption 1

${\mathbb{E}}\langle|u_{1}|^{k}\rangle_{c}\leqslant
d^{k}{\mathbb{E}}\langle(1+2g^{2}R_{1,1}+2z_{1}^{2})^{k}\rangle_{c}.$

Finally, by the independence of $R_{1,1}$ from $g$ and $z_{1}$ and Proposition
4.1, the right-hand side of this last equation is easily seen to be smaller
than some constant that does not depend on $N$.

$\square$

######  Lemma 5.2.

There exist constants $K,K^{\prime}>0$ such that

$\Big{|}{\mathbb{E}}\langle R_{1,1}\rangle_{c}-{\mathbb{E}}\langle
R_{1,1}\rangle\Big{|}\leqslant\frac{K}{\sqrt{N}}\quad\mbox{and}\quad\Big{|}{\mathbb{E}}\langle
R_{1,2}\rangle_{c}-{\mathbb{E}}\langle
R_{1,2}\rangle\Big{|}\leqslant\frac{K^{\prime}}{\sqrt{N}}.$

_Proof._ As in the previous lemma, we will denote $u(S_{1}(\sigma)+z_{1})$ by
$u_{1}$. Let

$H_{N,t}(\sigma):=H_{N,c}(\sigma)+tu_{1}(\sigma)$

for $t\in[0,1]$, and denote by $\langle\,\cdot\,\rangle_{t}$ the expectation
with respect to the associated Gibbs measure. We then clearly have that

${\mathbb{E}}\langle R_{1,1}\rangle_{c}={\mathbb{E}}\langle
R_{1,1}\rangle_{t=0}\quad\mbox{and}\quad{\mathbb{E}}\langle
R_{1,1}\rangle={\mathbb{E}}\langle R_{1,1}\rangle_{t=1}.$

Along this interpolation,

$\begin{split}\Big{|}\frac{d}{dt}{\mathbb{E}}\langle
R_{1,1}\rangle_{t}\Big{|}&=\Big{|}{\mathbb{E}}\langle(R_{1,1}-\langle
R_{1,1}\rangle_{t})u_{1}\rangle_{t}\Big{|}\leqslant\sqrt{{\mathbb{E}}\langle(R_{1,1}-\langle
R_{1,1}\rangle_{t})^{2}\rangle_{t}}\times\sqrt{{\mathbb{E}}\langle
u_{1}^{2}\rangle_{t}}\leqslant\frac{K}{\sqrt{N}}\times K^{\prime},\end{split}$

for some constants $K,K^{\prime}>0$ that do not depend on $t\in[0,1]$. Here,
the bound for the first factor follows from the Brascamp-Lieb inequality,
while the bound on the second term is a generalization of Lemma 5.1. The proof
for $R_{1,2}$ is similar.

$\square$

###  5.2 Consistency equations for the overlaps

Throughout this subsection, we will repeatedly use the concentration results
for the generalized overlaps $Q$ and $R$ from Section 4 that are applicable to
the overlaps $Q_{1,1}$ using $\psi(s)=u^{\prime\prime}(s)+u^{\prime}(s)^{2}$,
$Q_{1,2}$ using $\psi(s_{1},s_{2})=u^{\prime}(s_{1})u^{\prime}(s_{2})$,
$R_{1,1}$ using $\phi(s)=s^{2}$, and $R_{1,2}$ using
$\phi(s_{1},s_{2})=s_{1}s_{2}$. Denote

$\rho_{N}:={\mathbb{E}}\langle R_{1,1}\rangle,\quad q_{N}:={\mathbb{E}}\langle
R_{1,2}\rangle,\quad\bar{r}_{N}:={\mathbb{E}}\langle
Q_{1,1}\rangle\quad\mbox{and}\quad r_{N}:={\mathbb{E}}\langle Q_{1,2}\rangle.$

By Proposition 4.1, there exists some compact set $K$ such that for all
$N\geqslant 1$, $(\rho_{N},q_{N},\bar{r}_{N},r_{N})$ belongs to $K$. Thus,
there exists a subsequence $(N_{m})_{m\geqslant 1}$ of system sizes along
which

$(\rho_{m},q_{m},\bar{r}_{m},r_{m}):=(\rho_{N_{m}},q_{N_{m}},\bar{r}_{N_{m}},r_{N_{m}})\xrightarrow{m\to+\infty}(\rho,q,\bar{r},r)$

for some $(\rho,q,\bar{r},r)$ in $K$. For the rest of this section, we work
with this convergent subsequence and we aim to show that $(\rho,q,\bar{r},r)$
satisfies (17)–(20) in Propositions 5.4 and 5.5. To begin with, we establish

######  Lemma 5.3.

Let, for $n\geqslant 1$, $f:\mathbb{R}^{n}\mapsto\mathbb{R}$ be continuous and
bounded and

$\displaystyle\theta_{\ell}:=\sqrt{\rho-q}\,\xi_{\ell}+\sqrt{q+\Delta_{*}}\,\widetilde{z}\quad\mbox{for}\quad\ell\leqslant
n,$ (56)

with $\widetilde{z},(\xi_{\ell})_{\ell\leqslant n}$ i.i.d. standard gaussian
variables. Under Assumption 1, we have that

${\mathbb{E}}\big{\langle}f(S_{1}(\sigma^{1})+z_{1},\dots,S_{1}(\sigma^{n})+z_{1})\big{\rangle}_{c}\xrightarrow{m\to+\infty}{\mathbb{E}}f(\theta_{1},\dots,\theta_{n}).$

_Proof._ Along the proof $\ell$ and $\ell^{\prime}$ always belong to
$\\{1,\ldots,n\\}$. Recall $S_{1}^{\ell}:=S_{1}(\sigma^{\ell})$. Because $f$
is continuous and bounded and because for fixed values of
$\sigma^{1},\dots,\sigma^{\ell}$ the quantity
$(S_{1}^{1}+z_{1},\dots,S_{1}^{n}+z_{1})$ is a gaussian vector with covariance
matrix $\Sigma_{m}$ with entries
$\Sigma_{m,\ell\ell^{\prime}}=R_{\ell,\ell^{\prime}}+\Delta_{*}$, we have that
there exists some continuous and bounded $g:\mathbb{R}^{n\times
n}\mapsto\mathbb{R}$ such that

${\mathbb{E}}\big{\langle}f(S_{1}^{1}+z_{1},\dots,S_{1}^{n}+z_{1})\big{\rangle}_{c}={\mathbb{E}}\langle
g(\Sigma_{m})\rangle_{c}.$ (57)

By Proposition 4.1 and Lemma 5.2, $\Sigma_{m}\xrightarrow{L^{2}}\Sigma$ as
$m\to+\infty$ where

$\Sigma_{\ell\ell^{\prime}}=(\rho+\Delta_{*}){I}(\ell=\ell^{\prime})+(q+\Delta_{*})I(\ell\neq\ell^{\prime}).$

Then, by the continuous mapping theorem, ${\mathbb{E}}\langle
g(\Sigma_{m})\rangle_{c}\to g(\Sigma)$. Observe that $\Sigma=DD^{\intercal}$,
where

$D_{kk^{\prime}}=\sqrt{q+\Delta_{*}}\ {I}(k^{\prime}=1)+\sqrt{\rho-q}\
{I}(k^{\prime}=k+1)\in\mathbb{R}^{n\times n+1}.$

We can then define i.i.d. standard gaussian variables
$v:=(\widetilde{z},\xi_{1},\dots,\xi_{n})$ so that
$\theta_{\ell}:=(Dv)_{\ell}$ is (56), and then by reversing relation (57) we
get $g(\Sigma)={\mathbb{E}}f(\theta_{1},\dots,\theta_{n})$.

$\square$

######  Proposition 5.4.

The vector $(\rho,q,\bar{r},r)$ satisfies (19) and (20).

_Proof._ Let $(\theta_{\ell})_{\ell\leqslant n}$ as in (56),
$\theta=\theta_{1}$, $\xi=\xi_{1}$ and $a_{\ell}:=S_{1}(\sigma^{\ell})+z_{1}$.
Let $K\geqslant 1$. For each $k\leqslant K$ define (we let
$\prod_{j=2}^{1}(\cdots)=1$)

$f_{k}(a_{1},\dots,a_{k}):=({u^{\prime}}(a_{1})^{2}+{u^{\prime\prime}}(a_{1}))\exp{u(a_{1})}\prod_{j=2}^{k}(1-\exp{u(a_{j})}).$

By equation (55), the overlap definition (11), the spin symmetry and the
geometric series (recall $u(s)\leqslant 0$) we have that

$\displaystyle{\mathbb{E}}\langle{Q_{1,1}}\rangle$
$\displaystyle=\alpha{\mathbb{E}}\langle{u^{\prime}}(a_{1})^{2}+{u^{\prime\prime}}(a_{1})\rangle$
$\displaystyle=\alpha{\mathbb{E}}\frac{\langle({u^{\prime}}(a_{1})^{2}+{u^{\prime\prime}}(a_{1}))\exp
u(a_{1})\rangle_{c}}{\langle\exp u(a_{1})\rangle_{c}}$
$\displaystyle=\alpha\sum_{k\leqslant K}{\mathbb{E}}\langle
f_{k}(a_{1},\dots,a_{k})\rangle_{c}+\alpha{\mathbb{E}}\big{(}\langle{u^{\prime}}(a_{1})^{2}+{u^{\prime\prime}}(a_{1})\rangle(1-\langle\exp{u(a_{1})}\rangle_{c})^{{K}}\big{)}.$
(58)

Note that by Assumption 1 the first term is the mean of a continuous and
bounded function of $a_{1},\dots,a_{K}$. Then by Lemma 5.3 we have that

$\sum_{k\leqslant K}{\mathbb{E}}\langle
f_{k}(a_{1},\dots,a_{k})\rangle_{c}\xrightarrow{m\to+\infty}\sum_{k\leqslant
K}{\mathbb{E}}f_{k}(\theta_{1},\dots,\theta_{k}).$

The second term can be bounded in the following way

$\displaystyle\Big{|}{\mathbb{E}}\left(\langle{u^{\prime}}(a_{1})^{2}+{u^{\prime\prime}}(a_{1})\rangle(1-\langle\exp{u(a_{1})}\rangle_{c})^{K}\right)\Big{|}$
$\displaystyle\qquad\leqslant{\mathbb{E}}\left(\langle
d+d^{2}(1+{\sqrt{|u(a_{1})|}})^{2}\rangle(1-\langle\exp{u(a_{1})}\rangle_{c})^{{K}}\right)$
$\displaystyle\qquad\leqslant\sqrt{{\mathbb{E}}\langle[d+d^{2}(1+{\sqrt{|u(a_{1})|}})^{2}]^{2}\rangle{\mathbb{E}}\langle(1-\exp{u(a_{1})})^{{2K}}\rangle_{c}}$
$\displaystyle\qquad\leqslant\sqrt{C\,{\mathbb{E}}\langle(1-\exp{u(a_{1})})^{{2K}}\rangle_{c}},$

where in the first line we used Assumption 1, in the second Cauchy-Schwarz and
Jensen’s inequalities, and in the last one that by Lemma 5.1 there exists a
constant $C>0$ that bounds the first factor in the square root. Now, because
$(1-\exp{u(a_{1})})^{{2K}}$ is a continuous and bounded function of $a_{1}$,
by Lemma 5.3 we have that

${\mathbb{E}}\langle(1-\exp{u(a_{1})})^{2K}\rangle_{c}\xrightarrow{m\to+\infty}{\mathbb{E}}(1-\exp{u(\theta)})^{2K}.$

By taking the limit $m\to+\infty$ in equation (58) and using the convergence
of the overlap $Q_{1,1}$ along the subsequence $(N_{m})_{m\geqslant 1}$, this
implies that

$\bar{r}=\alpha\sum_{k\leqslant
K}{\mathbb{E}}f_{k}(\theta_{1},\dots,\theta_{k})+\mathcal{O}\Big{(}\sqrt{{\mathbb{E}}(1-\exp{u(\theta)})^{2K}}\Big{)}.$

Consider the two terms in $f_{k}$ separately. Taking the limit $K\to+\infty$,
by the monotone convergence theorem,

$\displaystyle\sum_{k\leqslant
K}{\mathbb{E}}{u^{\prime}}(\theta)^{2}\exp{u(\theta)}\prod_{j=2}^{k}(1-\exp{u(\theta_{j})})$
$\displaystyle={\mathbb{E}}_{\widetilde{z}}\,\Big{[}{\mathbb{E}}_{\xi}{u^{\prime}}(\theta)^{2}\exp{u(\theta)}\sum_{k\leqslant
K}(1-{\mathbb{E}}_{\xi}\exp{u(\theta)})^{k-1}\Big{]}$
$\displaystyle\qquad\xrightarrow{K\to+\infty}{\mathbb{E}}_{\widetilde{z}}\,\frac{{\mathbb{E}}_{\xi}{u^{\prime}}(\theta)^{2}\exp
u(\theta)}{{\mathbb{E}}_{\xi}\exp u(\theta)},$

and since $|u^{\prime\prime}(a_{1})|\leqslant d$ by Assumption 1, by the
dominated convergence theorem, we also have

$\sum_{k\leqslant
K}{\mathbb{E}}{u^{\prime\prime}}(\theta)\exp{u(\theta)}\prod_{j=2}^{k}(1-\exp{u(\theta_{j})})\xrightarrow{K\to+\infty}{\mathbb{E}}_{\widetilde{z}}\,\frac{{\mathbb{E}}_{\xi}{u^{\prime\prime}}(\theta)\exp
u(\theta)}{{\mathbb{E}}_{\xi}\exp u(\theta)}.$

To see that the second term goes to $0$, note that $1-\exp{u(\theta)}<1$. The
limit then follows by the dominated convergence theorem. This proves
$\bar{r}=\bar{\Phi}(q,\rho)$, i.e., equation (20).

The proof that $r=\Phi(q,\rho)$ (equation (19)) follows in a similar way. Let
$K\geqslant 1$. For each $k\leqslant K$ define

$g_{k}(a_{1},\dots,a_{2k}):={u^{\prime}}(a_{1}){u^{\prime}}(a_{2})\exp(u(a_{1})u(a_{2}))\prod_{j=2}^{k}\big{(}1-\exp(u(a_{2j-1})+u(a_{2j}))\big{)}.$

By Assumption 1, all these are continuous and bounded functions of
$a_{1},\dots,a_{2k}$. As before, by the cavity formula and the geometric
series we have that

${\mathbb{E}}\langle{Q_{1,2}}\rangle=\alpha\sum_{k\leqslant
K}{\mathbb{E}}\langle
g_{k}(a_{1},\dots,a_{2k})\rangle_{c}+\alpha{\mathbb{E}}\big{(}\langle{u^{\prime}}(a_{1}){u^{\prime}}(a_{2})\rangle(1-\langle\exp{(u(a_{1})+u(a_{2}))}\rangle_{c})^{{K}}\big{)}.$

Finally, the conclusion follows in a similar manner as for the first equation:
Lemma 5.3 is used for the sum while the second term can be seen to go to $0$
when $K\to+\infty$.

$\square$

To establish the second pair of fixed point equations, a feasible approach
would be to perform a cavity argument over the variables indexed by
$i=1,\ldots,N$ analogous to the proof of Proposition 5.4 (this “cavity in $N$”
is in general more involved than the “cavity in $M$”). Nevertheless, by the
virtue of concentration of the overlaps, we show that they can also be
obtained through an elementary derivation via gaussian integration by parts.

######  Proposition 5.5.

The vector $(\rho,q,\bar{r},r)$ satisfies (17) and (18).

_Proof._ To see this, note that by the term $-\kappa(\sigma_{1}^{1})^{2}$ in
the Hamiltonian (6) we can use gaussian integration by parts with respect to
$\sigma_{1}^{1}$ and the gaussian variables $(g_{k1})_{k\leqslant M}$. In
particular

$\displaystyle{\mathbb{E}}\langle
f((\sigma^{\ell}),g_{ki})g_{ki}\rangle={\mathbb{E}}\langle\partial_{g_{ki}}f\rangle+\sum_{\ell}\big{(}{\mathbb{E}}\langle
f\partial_{g_{ki}}H_{N}(\sigma^{\ell})\rangle-{\mathbb{E}}\langle
f\rangle\langle\partial_{g_{ki}}H_{N}(\sigma^{\ell})\rangle\big{)},$

and, denoting the modified Hamiltonian
$\widehat{H}_{N}(\sigma):=\sum_{k\leqslant
M}u(S_{k}(\sigma)+z_{k})-h\sum_{i\leqslant N}\sigma_{i}$,

$\displaystyle 2\kappa{\mathbb{E}}\langle
f(\sigma_{i})\sigma_{i}\rangle={\mathbb{E}}\langle\partial_{\sigma_{i}}f\rangle+{\mathbb{E}}\langle
f\partial_{\sigma_{i}}\widehat{H}_{N}(\sigma)\rangle.$

These identities imply (recall the definitions (10))

$\displaystyle 2\kappa{\mathbb{E}}\langle{\sigma_{1}}\rangle$
$\displaystyle={\mathbb{E}}\Big{\langle}{\frac{1}{\sqrt{N}}\sum_{k\leqslant
M}A_{k}g_{k1}}-h\Big{\rangle}$
$\displaystyle={\mathbb{E}}\Big{\langle}\frac{\sigma_{1}^{1}}{N}\sum_{k\leqslant
M}\big{(}(A_{k}^{1})^{2}+B_{k}^{1}-A_{k}^{1}A_{k}^{2}\big{)}-h\Big{\rangle},$
$\displaystyle 2\kappa{\mathbb{E}}\langle(\sigma_{1})^{2}\rangle$
$\displaystyle={\mathbb{E}}\Big{\langle}1{+\sigma_{1}\Big{(}\frac{1}{\sqrt{N}}\sum_{k\leqslant
M}A_{k}g_{k1}-h\Big{)}}\Big{\rangle}$
$\displaystyle={\mathbb{E}}\Big{\langle}1+\frac{(\sigma_{1}^{1})^{2}}{N}\sum_{k\leqslant
M}\big{(}(A_{k}^{1})^{2}+B_{k}^{1}\big{)}-\frac{\sigma_{1}^{1}\sigma_{1}^{2}}{N}\sum_{k\leqslant
M}A_{k}^{1}A_{k}^{2}-h\sigma_{1}\Big{\rangle},$ $\displaystyle
2\kappa{\mathbb{E}}\langle\sigma^{1}_{1}\sigma^{2}_{1}\rangle$
$\displaystyle={\mathbb{E}}\Big{\langle}\sigma_{1}^{2}{\Big{(}\frac{1}{\sqrt{N}}\sum_{k\leqslant
M}A^{1}_{k}g_{k1}-h\Big{)}}\Big{\rangle}$
$\displaystyle={\mathbb{E}}\Big{\langle}\frac{\sigma_{1}^{1}\sigma_{1}^{2}}{N}\sum_{k\leqslant
M}\big{(}(A_{k}^{1})^{2}{+B_{k}^{1}}\big{)}+\frac{(\sigma_{1}^{2})^{2}}{N}\sum_{k\leqslant
M}A_{k}^{1}A_{k}^{2}-2\frac{\sigma_{1}^{2}\sigma_{1}^{3}}{N}\sum_{k\leqslant
M}A_{k}^{1}A_{k}^{3}-h\sigma_{1}^{2}\Big{\rangle}.$

The concentrations of $R_{1,1}$, $R_{1,2}$, $Q_{1,1}$, and $Q_{1,2}$ (see
Proposition 4.1) then implies that

$\displaystyle{\mathbb{E}}\langle\sigma_{1}\rangle$
$\displaystyle=-\frac{h}{2\kappa+r_{m}-\bar{r}_{m}}+o_{m}(1),$ $\displaystyle
2\kappa\rho_{m}$
$\displaystyle=1+\rho_{m}\bar{r}_{m}-q_{m}r_{m}+\frac{h^{2}}{2\kappa+r_{m}-\bar{r}_{m}}+o_{m}(1),$
$\displaystyle 2\kappa q_{m}$
$\displaystyle=-q_{m}(r_{m}-\bar{r}_{m})+(\rho_{m}-q_{m})r_{m}+\frac{h^{2}}{2\kappa+r_{m}-\bar{r}_{m}}+o_{m}(1).$

Sending $m\to+\infty$, these lead to $q=\Psi(r,\bar{r})$ and
$\rho=\bar{\Psi}(r,\bar{r})$ (equations (17) and (18)) and complete our proof.

$\square$

Propositions 5.4 and 5.5 together conclude that $(\rho,q,\bar{r},r)$ satisfies
(17)–(20). We are now in position to prove Theorem 3.3 for the ST spin glass
model.

###  5.3 Proof of Theorem 3.3.

Consider any subsequence of system sizes. As explained above Lemma 5.3, there
exists some further subsequence $(N_{m})_{n\geqslant 1}$ such that

$(\rho_{N_{m}},q_{N_{m}},\bar{r}_{N_{m}},r_{N_{m}})\xrightarrow{m\to+\infty}(\rho,q,\bar{r},r)$

for some $(\rho,q,\bar{r},r)$ in a compact set $K$. By Propositions 5.4 and
5.5 and the continuity of the functions $\bar{\Phi}$, $\Phi$, $\bar{\Psi}$,
and $\Psi$, we see that $(\rho,q,\bar{r},r)$ is a solution of (17)–(20). Since
this system of equations has only one unique solution as ensured by the given
assumption, it follows that any convergent subsequence will share the same
limit and this implies the assertion.

##  6 Proof of Theorem 3.2

In this section, we establish the convergence of $F_{N}$ using a basic
interpolation in this section. Let $\alpha,\Delta_{*},\Delta,\kappa>0$ be
fixed. For $t\in[0,1]$, consider an auxiliary free energy

$\phi_{N}(t):=\frac{1}{N}{\mathbb{E}}\ln\int\exp\Big{(}\sum_{k\leqslant
M}u\big{(}\sqrt{t}((\bar{G}\sigma)_{k}+z_{k})\big{)}-h\sum_{i\leqslant
N}\sigma_{i}-\kappa\|\sigma\|^{2}\Big{)}\,d\sigma.$

Let ${\mathbb{E}}\langle\,\cdot\,\rangle_{t}$ be the joint mean with respect
to the gaussian $(G,z)$ and Gibbs measure associated with the Hamiltonian (6)
after replacing $u(x)$ by $u(\sqrt{t}x)$. We need the following crucial lemma.

######  Lemma 6.1.

For any $t_{0}\in(0,1],$ there exist constants $K>0$ and $\delta>0$ such that
for all $t\in[t_{0}-\delta,t_{0}+\delta]\cap(0,1]$ and $N\geqslant 1$ we have

$\displaystyle\max\Bigl{(}\Bigl{|}\frac{d}{dt}{\mathbb{E}}\langle
R_{1,1}\rangle_{t}\Big{|},\Bigl{|}\frac{d}{dt}{\mathbb{E}}\langle
R_{1,2}\rangle_{t}\Big{|},\Bigl{|}\frac{d}{dt}{\mathbb{E}}\langle
Q_{1,1}\rangle_{t}\Big{|},\Bigl{|}\frac{d}{dt}{\mathbb{E}}\langle
Q_{1,2}\rangle_{t}\Big{|}\Bigr{)}\leqslant K.$

###### Proof.

We establish this inequality only for $\frac{d}{dt}{\mathbb{E}}\langle
R_{1,1}\rangle_{t}$ as the others can be treated similarly. Let

$L(\sigma):=\frac{1}{N}\sum_{k\leqslant
M}\big{(}S_{k}(\sigma)+z_{k}\big{)}u^{\prime}\big{(}\sqrt{t}(S_{k}(\sigma)+z_{k})\big{)}.$

Write

$\displaystyle\frac{d}{dt}\mathbb{E}\langle R_{1,1}\rangle_{t}$
$\displaystyle=\frac{N}{2\sqrt{t}}\,\mathbb{E}\big{\langle}R_{1,1}(L(\sigma^{1})-L(\sigma^{2}))\big{\rangle}_{t}$
$\displaystyle=\frac{N}{2\sqrt{t}}\,\mathbb{E}\big{\langle}(R_{1,1}-\langle
R_{1,1}\rangle_{t})(L(\sigma^{1})-\langle
L(\sigma^{1})\rangle_{t})\big{\rangle}_{t}.$

As in Section 4.4,

$\mathbb{E}\big{\langle}(R_{1,1}-\langle
R_{1,1}\rangle_{t})^{2}\big{\rangle}_{t}\leqslant\frac{K}{N}\quad\mbox{and}\quad\mathbb{E}\big{\langle}(L(\sigma^{1})-\langle
L(\sigma^{1})\rangle_{t})^{2}\big{\rangle}_{t}\leqslant\frac{K}{N},$

so, by the Cauchy-Schwarz inequality, $\frac{d}{dt}\mathbb{E}\langle
R_{1,1}\rangle_{t}\leqslant K.$ Since this bound is valid locally uniformly in
$t$, the assertion follows. ∎

We now turn to the proof of Theorem 3.2. Let $(q,\rho,\bar{r},r)$ be the
unique solution to (17)–(20). As explained in Section 3,
$F(q,\rho,r,\bar{r})=F(q,\rho).$ Also, from Lemma 4.9,
$\lim_{N\to\infty}\mbox{Var}(F_{N})=0.$ Our proof will be complete if we can
show that all subsequential limits of ${\mathbb{E}}F_{N}$ are equal to
$F(q,\rho,r,\bar{r})$. We establish this statement as follows.

First of all, from Lemma 6.1, we can use the Arzela-Ascoli theorem and a
diagonalization process to further pass to a subsequence along which
${\mathbb{E}}\langle R_{1,1}\rangle_{t}$, ${\mathbb{E}}\langle
R_{1,2}\rangle_{t}$, ${\mathbb{E}}\langle Q_{1,1}\rangle_{t}$ and
${\mathbb{E}}\langle Q_{1,2}\rangle_{t}$ converge to some
$\rho_{t},q_{t},\bar{r}_{t},r_{t}$ uniformly with respect to $t\in[0,1]$,
respectively. For the rest of the proof, to lighten our notation, we shall
simply assume that the convergences of these overlaps and ${\mathbb{E}}F_{N}$
are valid along the original sequence $N.$ Here,
$\rho_{t},q_{t},\bar{r}_{t},r_{t}$ are defined on $t\in[0,1]$ and are locally
Lipschitz on $(0,1].$ Furthermore, from Theorem 3.3, they satisfy (17)–(20)
for $u(x)$ being replaced by $u(\sqrt{t}x)$ for all $t\in[0,1]$. Now, from the
uniqueness of the solution to the fixed point equations (17)–(20)
(corresponding to $t=1$),

$\displaystyle(\rho_{1},q_{1},\bar{r}_{1},r_{1})=(\rho,q,\bar{r},r).$ (59)

When $t=0,$

$\displaystyle\rho_{0}=\frac{1}{2\kappa}+\frac{h^{2}}{(2\kappa)^{2}},\quad
q_{0}=\frac{h^{2}}{2\kappa},\quad\bar{r}_{0}=0,\quad r_{0}=0.$ (60)

Sending $t\downarrow 0$ in (17)–(20) also yields that $\lim_{t\downarrow
0}(\rho_{t},q_{t},\bar{r}_{t},r_{t})=(\rho_{0},q_{0},\bar{r}_{0},r_{0})$.
Hence, $\rho_{t},q_{t},\bar{r}_{t},r_{t}$ are continuous on $[0,1]$.

Next, set

${f(t):=\alpha{\mathbb{E}}\ln{\mathbb{E}}_{\xi}\exp{u(\sqrt{t}\theta_{t})}+\frac{1}{2}\Big{(}\frac{r_{t}+h^{2}}{2\kappa+r_{t}-\bar{r}_{t}}-\ln(2\kappa+r_{t}-\bar{r}_{t})+r_{t}q_{t}-\bar{r}_{t}\rho_{t}+\ln
2\pi\Big{)},}$

where
$\theta_{t}:=\widetilde{z}\sqrt{\Delta_{*}+q_{t}}+\xi\sqrt{\rho_{t}-q_{t}}$
with i.i.d. standard gaussian $\widetilde{z},\xi$. Because the fixed point
equations satisfy the critical point conditions

$\frac{\partial f}{\partial\rho_{t}}=\frac{\partial f}{\partial
q_{t}}=\frac{\partial f}{\partial\bar{r}_{t}}=\frac{\partial f}{\partial
r_{t}}=0$

and $\rho_{t},q_{t},\bar{r}_{t},r_{t}$ are locally Lipschitz on $(0,1),$ we
conclude that $df/dt=\partial f/\partial t$. To compute this partial
derivative, let
$\theta_{t}^{\prime}:=\widetilde{z}\sqrt{\Delta_{*}+q_{t}}+\xi^{\prime}\sqrt{\rho_{t}-q_{t}}$
for $\xi^{\prime}$ an independent copy of $\xi$. Using gaussian integration by
parts with respect to $\widetilde{z}$ and $\xi$ yields

$f^{\prime}(t)=\frac{\partial f}{\partial
t}=\frac{\alpha}{2\sqrt{t}}\,{\mathbb{E}}_{\widetilde{z}}\,\frac{{\mathbb{E}}_{\xi}u^{\prime}(\sqrt{t}\theta_{t})\exp
u(\sqrt{t}\theta_{t})\theta_{t}}{{\mathbb{E}}_{\xi^{\prime}}u^{\prime}(\sqrt{t}\theta_{t}^{\prime})\exp
u(\sqrt{t}\theta_{t}^{\prime})}={\frac{1}{2}}(\bar{r}_{t}(\rho_{t}+\Delta_{*})-r_{t}(q_{t}+\Delta_{*})).$

On the other hand, note that

$\phi_{N}^{\prime}(t)={\frac{1}{2}}{\mathbb{E}}\big{\langle}Q_{1,1}(R_{1,1}+\Delta_{*})-Q_{1,2}(R_{1,2}+\Delta_{*})\big{\rangle}_{t},$
(61)

where we used gaussian integration by parts with respect to the elements of
$\bar{G}$ and $z$ similar to the proof of Proposition 5.5. By Theorem 3.3, for
every $t\in[0,1]$, $(R_{1,1},R_{1,2},Q_{1,1},Q_{1,2})$ converges to
$(\rho_{t},q_{t},\bar{r}_{t},r_{t})$ under
${\mathbb{E}}\langle\,\cdot\,\rangle_{t}$ and, hence,

$\lim_{N\to+\infty}\phi_{N}^{\prime}(t)=\frac{1}{2}(\bar{r}_{t}(\rho_{t}+\Delta_{*})-r_{t}(q_{t}+\Delta_{*})).$

To complete the proof, note that $\phi_{N}(1)={\mathbb{E}}F_{N}$ and that from
(59) and (60),

$\displaystyle f(0)$ $\displaystyle=\alpha
u(0)+\frac{h^{2}}{4\kappa}+\frac{1}{2}\ln\frac{\pi}{\kappa}=\phi_{N}(0),$
$\displaystyle f(1)$ $\displaystyle=F(q,\rho,\bar{r},r).$

Consequently, from (61) and noting that ${\mathbb{E}}\langle
R_{1,1}\rangle_{t},$ ${\mathbb{E}}\langle R_{1,2}\rangle_{t},$
${\mathbb{E}}\langle Q_{1,1}\rangle_{t},$ and ${\mathbb{E}}\langle
Q_{1,2}\rangle_{t}$ are uniformly bounded on $[0,1]$ due to Lemma 4.7, the
dominated convergence theorem implies

$\displaystyle|{\mathbb{E}}F_{N}-F(q,\rho,r,\bar{r})|=|\phi_{N}(1)-f(1)|\leqslant\int_{0}^{1}|\phi^{\prime}_{N}(t)-f^{\prime}(t)|dt\to
0,$

completing our proof.

$\square$

##  7 Proof of Theorem 3.4

We now provide the proof of the main theorem for the regression task, from the
results obtained for the ST model. Throughout the entire section, the
assumptions in Theorem 3.4 are in force. Recall
$h_{N}:=2\kappa\sqrt{\gamma_{N}}$ and $h:=2\kappa\sqrt{\gamma}.$ From the
convergence (1) we have ${\mathbb{E}}|h_{N}-h|^{2}\to 0$. Throughout this
section, we denote by $F_{N}(h),F_{N}(h_{N})$ and
$\langle\,\cdot\,\rangle_{h},\langle\,\cdot\,\rangle_{h_{N}}$ the free
energies $F_{N}$ and Gibbs expectations $\langle\,\cdot\,\rangle$, defined by
(7), (8), associated to the deterministic $h$ and randomized $h_{N},$
respectively. In particular $\langle\,\cdot\,\rangle_{h}$ is the same as the
plain $\langle\,\cdot\,\rangle$, while when defining $F_{N}(h_{N})$ and
$\langle\,\cdot\,\rangle_{h_{N}}$ the constant $h$ is replaced in the
Hamiltonian (6) by the random $h_{N}$. In the latter case, we also assume that
$h_{N}$ is independent of all other randomness. First of all, we establish the
following lemma:

######  Lemma 7.1 (Controlling the randomness $h_{N}$).

The free energy and overlap for the ST model with a random $h_{N}$ and fixed
$h$ are asymptotically the same:

$\displaystyle\lim_{N\to\infty}{\mathbb{E}}\bigl{|}F_{N}(h_{N})-F_{N}(h)\bigr{|}=0$
(62)

and

$\displaystyle\lim_{N\to\infty}{\mathbb{E}}\langle
R_{1,2}\rangle_{h_{N}}=\lim_{N\to\infty}{\mathbb{E}}\langle
R_{1,2}\rangle_{h}=q.$ (63)

###### Proof.

For $0\leqslant t\leqslant 1,$ consider the interpolated free energy,

$F_{N,t}:=\frac{1}{N}\ln\int\exp\Bigl{(}\sum_{k\leqslant
M}u\big{(}S_{k}(\sigma)+z_{k}\big{)}-\big{(}th_{N}+(1-t)h\big{)}\sum_{i\leqslant
N}\sigma_{i}-\kappa\|\sigma\|^{2}\Bigr{)}\,d\sigma.$

Let $H_{N,t}(\sigma)$ be the Hamiltonian defined by the above exponent. The
associated Gibbs mean is denoted by $\langle\,\cdot\,\rangle_{t}$ and is
defined by

$\displaystyle\big{\langle}f((\sigma^{\ell})_{\ell\leqslant
L})\big{\rangle}_{t}:=\frac{1}{(Z_{N,t})^{L}}\int
f((\sigma^{\ell})_{\ell\leqslant L})\prod_{\ell\leqslant L}\exp
H_{N,t}(\sigma^{\ell})\,d\sigma^{\ell}$ (64)

with $Z_{N,t}:=\int\exp H_{N,t}(\sigma)\,d\sigma$ the corresponding
normalization. Clearly $F_{N,1}=F_{N}(h_{N})$ and $F_{N,0}=F_{N}(h)$, as well
as $\langle R_{1,2}\rangle_{t=1}=\langle R_{1,2}\rangle_{h_{N}}$ and $\langle
R_{1,2}\rangle_{t=0}=\langle R_{1,2}\rangle_{h}$. Let
$M_{\ell}:=N^{-1}\sum_{i\leqslant N}\sigma_{i}^{\ell}.$ A direct
differentiation gives that

$\displaystyle\frac{d}{dt}F_{N,t}=(h-h_{N})\langle M_{1}\rangle_{t}$

and

$\displaystyle\frac{d}{dt}\langle R_{1,2}\rangle_{t}$
$\displaystyle=2N(h-h_{N})\langle R_{1,2}(M_{1}-M_{3})\rangle_{t}$
$\displaystyle=2N(h-h_{N})\langle(R_{1,2}-\langle
R_{1,2}\rangle_{t})(M_{1}-\langle M_{1}\rangle_{t})\rangle_{t}.$

From these, the mean value theorem and the Hölder inequality, there exists
$t\in[0,1]$ such that

$\displaystyle{\mathbb{E}}\big{|}F_{N}(h_{N})-F_{N}(h)\big{|}$
$\displaystyle\leqslant\bigl{(}{\mathbb{E}}|h_{N}-h|^{2}\bigr{)}^{1/2}\bigl{(}{\mathbb{E}}\langle
M_{1}^{2}\rangle_{t}\bigr{)}^{1/2}$
$\displaystyle\leqslant\bigl{(}{\mathbb{E}}|h_{N}-h|^{2}\bigr{)}^{1/2}\Bigl{(}\frac{{\mathbb{E}}\langle\|\sigma^{1}\|^{2}\rangle_{t}}{N}\Bigr{)}^{1/2}$
(65)

and some $t\in[0,1]$ such that

$\displaystyle\bigl{|}{\mathbb{E}}\langle
R_{1,2}\rangle_{h_{N}}-{\mathbb{E}}\langle R_{1,2}\rangle_{h}\bigr{|}$
$\displaystyle\qquad\leqslant
2N\bigl{(}{\mathbb{E}}|h_{N}-h|^{2}\bigr{)}^{1/2}\Bigl{(}{\mathbb{E}}\bigl{\langle}\bigl{|}R_{1,2}-\langle
R_{1,2}\rangle_{t}\bigr{|}^{4}\bigr{\rangle}_{t}\Bigr{)}^{1/4}\Bigl{(}{\mathbb{E}}\bigl{\langle}\bigl{|}M_{1}-\langle
M_{1}\rangle_{t}\bigr{|}^{4}\bigr{\rangle}_{t}\Bigr{)}^{1/4}.$ (66)

The same proof as of Lemma 4.4 yields that for any $k\geqslant 1$ there exists
some $K>0$ such that for all $t\in[0,1]$ and $N\geqslant 1,$

$\displaystyle\langle\|\sigma\|^{2k}\rangle_{t}$ $\displaystyle\leqslant
KN^{k}\bigl{(}\|\bar{z}\|^{2k}+\|\bar{G}\|_{op}^{2k}+h^{2k}+h_{N}^{2k}+1\bigr{)}.$
(67)

Since the expectation of every term in the bracket is bounded by a universal
constant, we obtain (62) by using (7) and taking $k=1.$

As for (63), note that for any fixed $w\in\mathbb{R}^{N},$
$f(\sigma):=N^{-1}w\cdot\sigma$ is Lipschitz with Lipschitz constant $\|w\|/N$
and that the Hessian of the interpolated Hamiltonian $H_{N,t}(\sigma)$ in
$F_{N,t}$ is bounded from above by $-\kappa{\rm Id}/2.$ From the Brascamp-Lieb
inequality [15, Theorem 5.1],

$\displaystyle\Bigl{\langle}\Bigl{|}\frac{w\cdot\sigma^{1}}{N}-\frac{w\cdot\langle\sigma^{1}\rangle_{t}}{N}\Bigr{|}^{4}\Bigr{\rangle}_{t}$
$\displaystyle\leqslant\frac{K\|w\|^{4}}{N^{4}},$ (68)

where $K$ is a constant independent of $w,N,t.$ Therefore, letting
$w=(1,\ldots,1)$ implies

$\displaystyle\bigl{\langle}\bigl{|}M_{1}-\langle
M_{1}\rangle_{t}\bigr{|}^{4}\bigr{\rangle}_{t}$
$\displaystyle\leqslant\frac{K\|w\|^{4}}{N^{4}}=\frac{K}{N^{2}}.$ (69)

In addition, writing

$\displaystyle R_{1,2}-\langle R_{1,2}\rangle_{t}$
$\displaystyle=\frac{\sigma^{1}\cdot\sigma^{2}}{N}-\frac{\langle\sigma^{1}\rangle_{t}\cdot\sigma^{2}}{N}+\frac{\langle\sigma^{1}\rangle_{t}\cdot\sigma^{2}}{N}-\frac{\langle\sigma^{1}\rangle_{t}\cdot\langle\sigma^{2}\rangle_{t}}{N},$

we can apply (68) with the choices $w=\sigma^{2}$ and
$w=\langle\sigma^{1}\rangle_{t}$ to get

$\displaystyle\Bigl{\langle}\Bigl{|}\frac{\sigma^{1}\cdot\sigma^{2}}{N}-\frac{\langle\sigma^{1}\rangle_{t}\cdot\sigma^{2}}{N}\Bigr{|}^{4}\Bigr{\rangle}_{t}$
$\displaystyle\leqslant\frac{K\langle\|\sigma^{2}\|^{4}\rangle_{t}}{N^{4}}=\frac{K\langle\|\sigma^{1}\|^{4}\rangle_{t}}{N^{4}}$

and

$\displaystyle\Bigl{\langle}\Bigl{|}\frac{\langle\sigma^{1}\rangle_{t}\cdot\sigma^{2}}{N}-\frac{\langle\sigma^{1}\rangle_{t}\cdot\langle\sigma^{2}\rangle_{t}}{N}\Bigr{|}^{4}\Bigr{\rangle}_{t}$
$\displaystyle\leqslant\frac{K\|\langle\sigma^{1}\rangle_{t}\|^{4}}{N^{4}}\leqslant\frac{K\langle\|\sigma^{1}\|^{4}\rangle_{t}}{N^{4}}.$

Putting these together and applying (67) with $k=2$, we obtain that for all
$0\leqslant t\leqslant 1$ and $N\geqslant 1,$

$\displaystyle{\mathbb{E}}\bigl{\langle}\bigl{|}R_{1,2}-\langle
R_{1,2}\rangle_{t}\bigr{|}^{4}\bigr{\rangle}_{t}$
$\displaystyle\leqslant\frac{32K{\mathbb{E}}\langle\|\sigma^{1}\|^{4}\rangle_{t}}{N^{4}}\leqslant\frac{K^{\prime}}{N^{2}}$
(70)

for some constant $K^{\prime}$ independent of $t,N.$ Plugging (69) and (70)
into (7) validates (63). ∎

Now we proceed to establish (22) and (23). From Theorem 3.3 and (15),

$\displaystyle\lim_{N\to\infty}{\mathbb{E}}\langle R_{1,2}\rangle_{h}$
$\displaystyle=q$

and

$\displaystyle\frac{1}{N}{\mathbb{E}}\|\langle
x\rangle^{\sim}-x^{*}\|^{2}=\frac{1}{N}{\mathbb{E}}\|\langle\sigma\rangle_{h_{N}}\|^{2}={\mathbb{E}}\langle
R_{1,2}\rangle_{h_{N}}.$

These and (63) together yields (23).

As for (22), from (LABEL:add:eq-5.1), we can write

$\displaystyle\widetilde{F}_{N}$
$\displaystyle=-\kappa\gamma_{N}+F_{N}(x^{*}),$

where

$\displaystyle
F_{N}(x^{*}):=\frac{1}{N}\ln\int_{\mathbb{R}^{N}}\exp\Big{(}\sum_{k\leqslant
M}u\Big{(}(\bar{G}O^{\intercal}\sigma)_{k}-z_{k}\Big{)}-h_{N}\sum_{i\leqslant
N}\sigma_{i}-\kappa\|\sigma\|^{2}\Big{)}\,d\sigma$

and $O$ is an orthonormal matrix satisfying
$Ox^{*}=N^{-1/2}{\|x^{*}\|}\bm{1}_{N}.$ Write

$\displaystyle\begin{split}\widetilde{F}_{N}-{\mathbb{E}}\widetilde{F}_{N}&=-\kappa(\gamma_{N}-{\mathbb{E}}\gamma_{N})\\\
&\qquad+F_{N}(x^{*})-{\mathbb{E}}[F_{N}(x^{*})\mid x^{*}]\\\
&\qquad+{\mathbb{E}}[F_{N}(x^{*})\mid
x^{*}]-{\mathbb{E}}F_{N}(x^{*}).\end{split}$ (71)

Here the first term on the right-hand side vanishes in $L^{2}$ norm by
assumption. For the second term, observe that conditionally on $x^{*}$,
$F_{N}(x^{*})$ is the same as $F_{N}(h_{N})$ in distribution. Lemma 4.9 (with
$n=1$ and $\lambda=\eta=0$) allows us to write that with probability one,

$\displaystyle\mbox{Var}(F_{N}(x^{*})\mid x^{*})=\mbox{Var}(F_{N}(h_{N})\mid
h_{N})$ $\displaystyle\leqslant\frac{K(1+h_{N}^{4})}{N},$

where $K$ is independent of $N.$ Hence, we can bound the second term by

$\displaystyle{\mathbb{E}}\bigl{|}F_{N}(x^{*})-{\mathbb{E}}[F_{N}(x^{*})\mid
x^{*}]\bigr{|}$
$\displaystyle\leqslant\big{(}{\mathbb{E}}\bigl{|}F_{N}(x^{*})-{\mathbb{E}}[F_{N}(x^{*})\mid
x^{*}]\bigr{|}^{2}\big{)}^{1/2}$
$\displaystyle=\big{(}{\mathbb{E}}\,\mbox{Var}(F_{N}(x^{*})\mid
x^{*})\big{)}^{1/2}\leqslant\sqrt{\frac{K(1+{\mathbb{E}}h_{N}^{4})}{N}}.$

Lastly, write

$\displaystyle{\mathbb{E}}[F_{N}(x^{*})\mid x^{*}]-{\mathbb{E}}F_{N}(x^{*})$
$\displaystyle={\mathbb{E}}[F_{N}(h_{N})\mid h_{N}]-{\mathbb{E}}F_{N}(h_{N})$
$\displaystyle={\mathbb{E}}[F_{N}(h_{N})-F_{N}(h)\mid
h_{N}]+\bigl{(}{\mathbb{E}}F_{N}(h)-{\mathbb{E}}F_{N}(h_{N})\bigr{)}.$

We bound the third term in (71) by using the Jensen inequality,

$\displaystyle{\mathbb{E}}\bigl{|}{\mathbb{E}}[F_{N}(x^{*})\mid
x^{*}]-{\mathbb{E}}F_{N}(x^{*})\bigr{|}$
$\displaystyle\leqslant{\mathbb{E}}\bigl{|}{\mathbb{E}}[F_{N}(h_{N})-F_{N}(h)\mid
h_{N}]\bigr{|}+\bigl{|}{\mathbb{E}}F_{N}(h)-{\mathbb{E}}F_{N}(h_{N})\bigr{|}$
$\displaystyle\leqslant{\mathbb{E}}\bigl{|}F_{N}(h_{N})-F_{N}(h)\bigr{|}\to
0,$

where the last limit used (62). Combined with Theorem 3.2 and the fact that
${\mathbb{E}}h_{N}^{4}$ is bounded by assumption (1), this completes our proof
for (22).

W.-K.C. was partly supported by NSF DMS-17-52184. D.P. was partially supported
by NSERC and Simons Fellowship.

## References

  * [1] [author] Abramovich, FelixF. and Grinshtein, VadimV. (2010). MAP model selection in Gaussian regression. Electronic Journal of Statistics 4 932–949.
  * [2] [author] Advani, M.M. and Ganguli, S.S. (2016). Statistical mechanics of optimal convex inference in high dimensions. Physical Review X 6 031034\.
  * [3] [author] Advani, M.M. and Ganguli, S.S. (2016). An equivalence between high dimensional Bayes optimal inference and M-estimation. In Advances in Neural Information Processing Systems 3378–3386.
  * [4] [author] Arias-Castro, EryE. and Lounici, KarimK. (2014). Estimation and variable selection with exponential weights. Electronic Journal of Statistics 8 328–354.
  * [5] [author] Aubin, B.B., Krzakala, F.F., Lu, Y.Y. and Zdeborová, L.L. (2020). Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization. In Advances in Neural Information Processing Systems 33\.
  * [6] [author] Banerjee, SayantanS., Castillo, IsmaëlI. and Ghosal, SubhashisS. (2021). Bayesian inference in high-dimensional models. arXiv preprint arXiv:2101.04491.
  * [7] [author] Barbier, J.J., Dia, M.M., Macris, N.N. and Krzakala, F.F. (2016). The mutual information in random linear estimation. In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton) 625–632. IEEE.
  * [8] [author] Barbier, J.J., Krzakala, F.F., Macris, N.N., Miolane, L.L. and Zdeborová, L.L. (2019). Optimal errors and phase transitions in high-dimensional generalized linear models. Proceedings of the National Academy of Sciences 116 5451–5460.
  * [9] [author] Barbier, J.J., Macris, N.N., Dia, M.M. and Krzakala, F.F. (2020). Mutual information and optimality of approximate message-passing in random linear estimation. IEEE Transactions on Information Theory 66 4270–4303.
  * [10] [author] Barbier, J.J., Macris, N.N. and Rush, C.C. (2020). All-or-nothing statistical and computational phase transitions in sparse spiked matrix estimation. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  * [11] [author] Barbier, JeanJ. and Panchenko, DmitryD. (2020). Strong replica symmetry in high-dimensional optimal Bayesian inference. arXiv preprint arXiv:2005.03115.
  * [12] [author] Bean, D.D., Bickel, P.P., El Karoui, N.N. and Yu, B.B. (2013). Optimal M-estimation in high-dimensional regression. Proceedings of the National Academy of Sciences 110 14563–14568.
  * [13] [author] Bottolo, LeonardL. and Richardson, SylviaS. (2010). Evolutionary stochastic search for Bayesian model exploration. Bayesian Analysis 5 583–618.
  * [14] [author] Bradic, J.J. (2016). Robustness in sparse high-dimensional linear models: relative efficiency and robust approximate message passing. Electronic Journal of Statistics 10 3894–3944.
  * [15] [author] Brascamp, H. J.H. J. and Lieb, E. H.E. H. (1976). On extensions of the Brunn-Minkowski and Prékopa-Leindler theorems, including inequalities for log concave functions, and with an application to the diffusion equation. Journal of functional Analysis 22 366–389.
  * [16] [author] Castillo, IsmaëlI., Schmidt-Hieber, JohannesJ. and Van der Vaart, AadA. (2015). Bayesian linear regression with sparse priors. The Annals of Statistics 43 1986–2018.
  * [17] [author] Celentano, M.M. and Montanari, A.A. (2019). Fundamental barriers to high-dimensional regression with convex penalties. arXiv preprint arXiv:1903.10603.
  * [18] [author] Contucci, P.P., Giardina, C.C. and Nishimori, H.H. (2009). Spin glass identities and the Nishimori line. In Spin Glasses: Statics and Dynamics 103–121. Springer.
  * [19] [author] Dembo, AmirA. and Zeitouni, OferO. (1998). Large Deviations Techniques and Applications, 2nd edition. Springer.
  * [20] [author] Donoho, D.D. and Montanari, A.A. (2016). High dimensional robust m-estimation: asymptotic variance via approximate message passing. Probability Theory and Related Fields 166 935–969.
  * [21] [author] El Karoui, N.N. (2018). On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators. Probability Theory and Related Fields 170 95–175.
  * [22] [author] El Karoui, N.N., Bean, D.D., Bickel, P. J.P. J., Lim, C.C. and Yu, B.B. (2013). On robust regression with high-dimensional predictors. Proceedings of the National Academy of Sciences 110 14557–14562.
  * [23] [author] Gamarnik, D.D. and Zadik, I.I. (2017). High dimensional regression with binary coefficients. estimating squared error and a phase transtition. In Conference on Learning Theory 948–953.
  * [24] [author] George, EdwardIE. and Foster, Dean PD. P. (2000). Calibration and empirical Bayes variable selection. Biometrika 87 731–747.
  * [25] [author] George, Edward IE. I. (2000). The variable selection problem. Journal of the American Statistical Association 95 1304–1308.
  * [26] [author] Gerbelot, C.C., Abbara, A.A. and Krzakala, F.F. (2020). Asymptotic errors for convex penalized linear regression beyond gaussian matrices. In Proceedings of Machine Learning Research: Conference on Learning Theory , 125 1682-1713.
  * [27] [author] Gerbelot, C.C., Abbara, A.A. and Krzakala, F.F. (2020). Asymptotic errors for teacher-Student Convex Generalized Linear Models (or: how to Prove Kabashima’s Replica Formula). arXiv preprint arXiv:2006.06581.
  * [28] [author] Ishwaran, HemantH. and Rao, J SunilJ. S. (2005). Spike and slab variable selection: frequentist and Bayesian strategies. The Annals of Statistics 33 730–773.
  * [29] [author] Luneau, C.C., Barbier, J.J. and Macris, N.N. (2020). Information theoretic limits of learning a sparse rule. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  * [30] [author] Martin, RyanR., Mess, RaymondR. and Walker, Stephen GS. G. (2017). Empirical Bayes posterior concentration in sparse high-dimensional linear models. Bernoulli 23 1822–1847.
  * [31] [author] Mézard, M.M. and Montanari, A.A. (2009). Information, Physics and Computation. Oxford University Press.
  * [32] [author] Mézard, M.M., Parisi, G.G. and Virasoro, M. A.M. A. (1987). Spin-Glass Theory and Beyond 9\. World Scientific, Singapore.
  * [33] [author] Mitchell, Toby JT. J. and Beauchamp, John JJ. J. (1988). Bayesian variable selection in linear regression. Journal of the american statistical association 83 1023–1032.
  * [34] [author] Mukherjee, S.S. and Sen, S.S. (2021). Variational Inference in high-dimensional linear regression. arXiv preprint arXiv:2104.12232.
  * [35] [author] Niles-Weed, J.J. and Zadik, I.I. (2020). The All-or-Nothing Phenomenon in Sparse Tensor PCA. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  * [36] [author] Niles-Weed, J.J. and Zadik, I.I. (2021). It was ”all” for ”nothing”: sharp phase transitions for noiseless discrete channels. arXiv preprint arXiv:2102.12422.
  * [37] [author] Nishimori, H.H. (2001). Statistical Physics of Spin Glasses and Information Processing: An Introduction. Oxford University Press.
  * [38] [author] Panchenko, D.D. (2013). The Sherrington-Kirkpatrick model. Springer Science & Business Media.
  * [39] [author] Ray, KolyanK. and Szabó, BotondB. (2021). Variational Bayes for high-dimensional linear regression with sparse priors. Journal of the American Statistical Association 1–12.
  * [40] [author] Reeves, G.G. and Pfister, H. D.H. D. (2016). The replica-symmetric prediction for compressed sensing with Gaussian matrices is exact. In 2016 IEEE International Symposium on Information Theory (ISIT) 665–669. IEEE.
  * [41] [author] Reeves, G.G., Xu, J.J. and Zadik, I.I. (2019). The all-or-nothing phenomenon in sparse linear regression. In Conference on Learning Theory 2652–2663. PMLR.
  * [42] [author] Reeves, G.G., Xu, J.J. and Zadik, I.I. (2019). All-or-nothing phenomena: From single-letter to high dimensions. In 2019 IEEE 8th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP) 654–658. IEEE.
  * [43] [author] Scott, James GJ. G. and Berger, James OJ. O. (2010). Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem. The Annals of Statistics 2587–2619.
  * [44] [author] Shcherbina, M.M. and Tirozzi, B.B. (2003). Rigorous solution of the Gardner problem. Communications in mathematical physics 234 383–422.
  * [45] [author] Sur, P.P. and Candès, E. J.E. J. (2019). A modern maximum-likelihood theory for high-dimensional logistic regression. Proceedings of the National Academy of Sciences 116 14516–14525.
  * [46] [author] Taheri, H.H., Pedarsani, R.R. and Thrampoulidis, C.C. (2020). Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensions. arXiv preprint arXiv:2006.08917.
  * [47] [author] Takahashi, T.T. and Kabashima, Y.Y. (2020). Macroscopic Analysis of Vector Approximate Message Passing in a Model Mismatch Setting. In 2020 IEEE International Symposium on Information Theory (ISIT) 1403-1408. 10.1109/ISIT44484.2020.9173999
  * [48] [author] Talagrand, M.M. (2011). Mean Field Models for Spin Glasses. Volume I: Basic Examples. Springer Verlag.
  * [49] [author] Thrampoulidis, C.C., Abbasi, E.E. and Hassibi, B.B. (2018). Precise Error Analysis of Regularized $M$-Estimators in High Dimensions. IEEE Transactions on Information Theory 64 5592–5628.
  * [50] [author] Thrampoulidis, C.C., Oymak, S.S. and Hassibi, B.B. (2015). Regularized Linear Regression: A Precise Analysis of the Estimation Error. In Proceedings of The 28th Conference on Learning Theory. Proceedings of Machine Learning Research 40 1683–1709.
  * [51] [author] Touchette, HugoH. (2009). The large deviation approach to statistical mechanics. Physics Reports 478 1–69.
  * [52] [author] Vershynin, R.R. (2010). Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027.
  * [53] [author] Wasserman, LarryL. (2004). All of statistics: a concise course in statistical inference 26\. Springer.
  * [54] [author] Yuan, MingM. and Lin, YiY. (2005). Efficient empirical Bayes variable selection and estimation in linear models. Journal of the American Statistical Association 100 1215–1225.