# Studying Hopfield models via fully lifted random duality theory

Mihailo Stojnic 111e-mail<EMAIL_ADDRESS>
Abstract

Relying on a recent progress made in studying bilinearly indexed (bli) random
processes in [85, 84], the main foundational principles of fully lifted random
duality theory (fl RDT) were established in [86]. We here study famous
Hopfield models and show that their statistical behavior can be characterized
via the fl RDT. Due to a nestedly lifted nature, the resulting
characterizations and, therefore, the whole analytical machinery that produces
them, become fully operational only if one can successfully conduct underlying
numerical evaluations. After conducting such evaluations for both positive and
negative Hopfield models, we observe a remarkably fast convergence of the fl
RDT mechanism. Namely, for the so-called square case, the fourth decimal
precision is achieved already on the third (second non-trivial) level of
lifting (3-sfl RDT) for the positive and on the fourth (third non-trivial)
level of lifting (4-sfl RDT) for the corresponding negative model. In
particular, we obtain the scaled ground state free energy $\approx 1.7788$ for
the positive and $\approx 0.3279$ for the negative model.

Index Terms: Hopfield models; Fully lifted random duality theory.

##  1 Introduction

Many natural phenomena, typically uncovered and explained through the
postulates of statistical and experimental physics, have become very popular
subjects of intensive studies in a host of mathematical fields over the last
several decades. Development of a synergistic approach, where key physical
insights are used to trace paths for mathematically rigorous considerations,
turned out to be very fruitful. Among the most successful examples of such a
synergy are certainly the considerations related to the physics of the matter
structure and, in particular, those related to the so called Sherrington-
Kirkpatrick (SK) models (see, e.g., [58, 60, 61, 59, 68] and [92, 91, 53, 54,
55, 56, 57, 37, 90]).

As was expected (or even predicted), such synergistic approach evolved even
further over time. Understanding the accompanying mathematics became more
complete and, consequently, one reached the level where many of the relevant
random structures, not even known to have any physical meaning, could be
rigorously handled by following the statistical physics ideas as key insights.
The number of such “ _artificial_ ” examples (where the direct connection to
the corresponding natural phenomena is not immediately apparent) quickly grew
over the last several decades and nowadays, quite likely, dominates the number
of the “ _natural_ ” ones. Moreover, given the pace at which the new ones are
appearing, one can be assured that many more are yet to come. What often makes
them particularly attractive, is that they typically span a variety of
scientific fields ranging from theoretical and applied mathematics and
probability (see, e.g., [92, 91, 53, 54, 55, 56, 57, 37, 90, 80, 82, 83, 67]),
to signal processing and information theory (see, e.g., [13, 12, 72, 70]), and
reaching to theoretical/algorithmic computer science (see, e.g., [50, 19, 25,
49, 47, 34, 1, 5, 96, 45, 92, 91, 51, 21, 4]) and many others. Moreover, they
also include a host of different topics within each of these areas. Examples
are again rather diverse, ranging from spherical/binary perceptrons in neural
networks and machine learning (see, e.g., [67, 73, 89, 92, 91, 97, 52, 98, 22,
28]), to compressed sensing and statistical regression in signal processing
and information theory (see, e.g., [71, 72, 70, 13, 12, 27]), and
satisfiability/assignment/matching problems in theoretical computer science
and optimization and algorithms theory (see, e.g., [19, 26, 25, 26, 49, 47,
34, 1]) and [5, 94, 95, 96, 45, 88, 92, 91, 51, 21, 20, 4, 3, 48]).

Here, we are interested in yet another particular type of random structure,
the so-called Hopfied models, that have gained a lot interest and,
consequently, have been extensively studied in many different scientific
fields (see, e.g., [40, 62, 39, 63, 65, 9, 11, 87, 78, 14, 99, 92, 91]). These
models ware popularized in [40] (or, if viewed in different contexts, one can
say even in [62, 39]). To provide their a mathematical description, one starts
with the so-called Hamiltonian of the following type

${\cal H}(G)=\sum_{i\neq j}^{n}A_{ij}{\bf x}_{i}{\bf x}_{j},$ (1)

where

$A_{ij}(G)=\sum_{l=1}^{m}G_{il}G_{lj},$ (2)

are the so-called quenched interactions and $G$ is an $m\times n$ matrix. For
the mathematical convenience, we then combine (1) and (2) to obtain a more
compact form

${\cal H}(G)={\bf x}^{T}G^{T}G{\bf x},$ (3)

where we include the diagonal elements as well (as it will become clear
throughout the presentation, such an inclusion contributes to the easiness of
writing and the overall clarity while affecting, exactly, in no way the
ensuing analytical considerations). Depending on the context where the models
are used, the components of $G$ play more or less prominent role on their own
(in other words, they are relevant by themselves rather than just as
constitutive components of $A$). For example, within the neural networks
contexts, one can often view $G$ as the matrix of the so-called stored
patterns. On the other hand, we, in this paper view $G$ as a purely
mathematical object. While the overall treatment is on mathematical objects
level, we, often, refer to many relevant quantities utilizing the terminology
frequently seen in statistical physics. Along the same lines, our prevalent
interest is in the so-called thermodynamic limit, or, as it is called in
mathematics, large $n$ linear regime. In other words, we are interested in
scenarios where $n\rightarrow\infty$, and all other dimensions are linearly
proportional to $n$ with proportionalities remaining constant as $n$ grows.
For example, we assume that $m$ is also large and
$\alpha=\lim_{n\rightarrow\infty}\frac{m}{n}$ with $\alpha$ remaining constant
as $n$ grows. The nature of the components of $G$ also fluctuates throughout
the literature, again, depending on the context where the models are used. For
example, in neural nets, the elements are typically binary spins, whereas in
other fields their various continuous statistical forms are often seen as
well. Here, we view them as i.i.d. standard normals, but the results hold for
pretty much any statistics that can be pushed through the central limit
theorem.

To characterize the behavior of the physical interpretations described through
the above Hamiltonian, one usually looks at the partition function

$Z^{\pm}(\beta,G)=\sum_{{\bf x}\in{\mathcal{X}}}e^{\pm\beta{\cal H}(G)},$ (4)

where $\beta>0$ is a parameter, typically called the _inverse temperature_ ,
and ${\mathcal{X}}$ is a set typically consisting of the corners of the unit
norm $n$-dimensional hypercube, i.e.
${\mathcal{X}}=\\{-\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}}\\}^{n}$ (from the
mathematical point of view though, ${\mathcal{X}}$ can be tretated as actually
any set). We adopt the convention of [24, 78] and refer to the sceanario with
“$+$” sign as the _positive_ Hopfield model, and to the one with “$-$” sign as
the _negative_ Hopfiled model. Instead of directly handling the partition
function, $Z^{\pm}(\beta,G)$, one often considers the following thermodynamic
limit, scaled $\log$, version called (average) free energy

$f^{\pm}(\beta)=\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\log{(Z^{\pm}(\beta,G)})}{\beta
n}=\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\log{(\sum_{{\bf
x}\in{\mathcal{X}}}e^{\pm\beta{\cal H}(G)})}}{\beta n},$ (5)

where ${\mathbb{E}}_{G}$ stands for the expectation with respect to $G$
(throughout the paper, ${\mathbb{E}}$ always stands for the expectation and
its subscript indicates the type of randomness with respect to which the
expectation is taken). Of particular interest is often the thermodynamic limit
ground state regime, where, in addition to $n\rightarrow\infty$, one also has
$\beta\rightarrow\infty$ and

$\displaystyle
f^{\pm}(\infty)\triangleq\lim_{\beta\rightarrow\infty}f^{\pm}(\beta)$
$\displaystyle=$
$\displaystyle\lim_{\beta,n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\log{(Z^{\pm}(\beta,G)})}{\beta
n}=\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}\pm{\cal H}(G)}{n}$ (6) $\displaystyle=$
$\displaystyle\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}\pm{\bf x}^{T}G^{T}G{\bf x}}{n}.$

The above corresponds to the standard Hopfield model treatment. As mentioned
earlier, these models have a long history and have been studied extensively
throughout the literature in a variety of fields (see, e.g., excellent
references [63, 65, 9, 11, 87, 23, 69, 7]). Here, we find it convenient to
study their so-called _square-root_ counterparts (see, e.g., [78]). For such
variants all of the above remains in place with the exception that, instead of
${\cal H}$ from (3), one now considers

${\cal H}_{sq}(G)={\bf y}^{T}G{\bf x},$ (7)

and the corresponding partition function

$Z_{sq}^{\pm}(\beta,G)=\sum_{{\bf x}\in{\mathcal{X}}}\left(\sum_{{\bf
y}\in{\mathcal{Y}}}e^{\beta{\cal H}_{sq}(G)}\right)^{\pm 1}.$ (8)

where ${\mathcal{Y}}={\mathbb{S}}^{m}$ (with ${\mathbb{S}}^{m}$ being the
$m$-dimensional unit sphere). The thermodynamic limit (average) free energy
becomes

$\displaystyle f_{sq}^{\pm}(\beta)$ $\displaystyle=$
$\displaystyle\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\log{(Z_{sq}^{\pm}(\beta,G)})}{\beta\sqrt{n}}=\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\log\left(\sum_{{\bf
x}\in{\mathcal{X}}}\left(\sum_{{\bf y}\in{\mathcal{Y}}}e^{\beta{\cal
H}_{sq}(G)}\right)^{\pm 1}\right)}{\beta\sqrt{n}}$ (9) $\displaystyle=$
$\displaystyle\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\log\left(\sum_{{\bf
x}\in{\mathcal{X}}}\left(\sum_{{\bf y}\in{\mathcal{Y}}}e^{\beta{\bf
y}^{T}G{\bf x})}\right)^{\pm 1}\right)}{\beta\sqrt{n}},$

and its corresponding ground state special case

$\displaystyle
f_{sq}^{\pm}(\infty)\triangleq\lim_{\beta\rightarrow\infty}f_{sq}^{\pm}(\beta)$
$\displaystyle=$
$\displaystyle\lim_{\beta,n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\log{(Z_{sq}^{\pm}(\beta,G)})}{\beta\sqrt{n}}=\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}\pm\max_{{\bf y}\in{\mathcal{Y}}}{\bf y}^{T}G{\bf
x}}{\sqrt{n}}$ (10) $\displaystyle=$
$\displaystyle\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}\pm\sqrt{{\bf x}^{T}G^{T}G{\bf x}}}{\sqrt{n}}.$

Our main focus throughput the paper is precisely the free energy given in (9).
However, as the prevalent interest is to properly understand the,
mathematically most challenging, ground state regime, we use the studying of
the free energy, defined for general $\beta$, to eventually deduce particular
ground state behavior (10). Along the same lines, while the analysis presented
below accounts for any $\beta$, in the interest of easing the exposition, we,
on occasion, neglect some of the terms with no significant role in the ground
state regime.

While we study the square-root variant of the model, the free energy
concentrations ensure that the obtained results immediately translate to the
standard model as well. It is also important to note that, despite the fact
that Hopfield models have been studied for several decades, analytical results
particularly related to the problems of our interests here are rather scarce.
For example, [23, 69, 7] cover the range of all $\beta$’s and offer a
statistical physics type of treatment with replica analysis based estimates.
On the other hand, mathematically rigorous treatments (see, e.g., [63, 65, 9,
87, 99, 10]), are usually restricted to certain $\alpha$ and/or $\beta$
regimes. Most of them typically relate to the high-temperature ($T$) regime
(low inverse temperature $\beta=\frac{1}{T}$), where the so-called replica
symmetric behavior is in place. The regimes, where the replica symmetry is
expected not to hold, are, on the other hand, much harder. Moreover, the
hardest ones among them are precisely the ground state ones of interest here.
They assume zero temperature scenarios ($\beta\rightarrow\infty$) and the
available results for such regimes are even scarcer. In particular, utilizing
the random duality theory (RDT) machinery, [75] established free energy upper
bounds which also turned out to precisely match the replica symmetry based
predictions. [78] then went a bit further, used a _lifted_ RDT variant, and
lowered the bounds of [75], thereby showing that the replica symmetry
predictions are _not_ tight and signaling that the assumed symmetry _must_ be
broken.

We here first recognize the connection of the underlying statistical Hopfield
models and bilinearly indexed (bli) random processes. Utilizing a recent
progress made in studying such processes in [84, 85], in [86] a _fully lifted_
random duality theory (fl RDT) was established. Keeping that in mind, we first
observe that the square-root Hopfield models can be characterized via the fl
RDT and, then, use a particular _stationarized_ fl RDT variant (called sfl
RDT) to obtain desired characterization.

##  2 Connecting Hopfield models and sfl RDT

We start by observing that the free energy from (9),

$\displaystyle f_{sq}^{\pm}(\beta)$ $\displaystyle=$
$\displaystyle\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\log\left(\sum_{{\bf
x}\in{\mathcal{X}}}\left(\sum_{{\bf y}\in{\mathcal{Y}}}e^{\beta{\bf
y}^{T}G{\bf x})}\right)^{\pm 1}\right)}{\beta\sqrt{n}},$ (11)

is essentially a function of bli ${\bf y}^{T}G{\bf x}$. To make a connection
with the bli related results of [84, 85], we need a few definitions. For
$r\in{\mathbb{N}}$, $k\in\\{1,2,\dots,r+1\\}$, real scalars $s$, $x$, and $y$
such that $s^{2}=1$, $x>0$, and $y>0$, sets
${\mathcal{X}}\subseteq{\mathbb{R}}^{n}$ and
${\mathcal{Y}}\subseteq{\mathbb{R}}^{m}$, function
$f_{S}(\cdot):{\mathbb{R}}^{n}\rightarrow R$, vectors ${\bf p}=[{\bf
p}_{0},{\bf p}_{1},\dots,{\bf p}_{r+1}]$, ${\bf q}=[{\bf q}_{0},{\bf
q}_{1},\dots,{\bf q}_{r+1}]$, and ${\bf c}=[{\bf c}_{0},{\bf c}_{1},\dots,{\bf
c}_{r+1}]$ such that

$\displaystyle 1={\bf p}_{0}\geq{\bf p}_{1}\geq{\bf p}_{2}\geq\dots\geq{\bf
p}_{r}\geq{\bf p}_{r+1}$ $\displaystyle=$ $\displaystyle 0$ $\displaystyle
1={\bf q}_{0}\geq{\bf q}_{1}\geq{\bf q}_{2}\geq\dots\geq{\bf q}_{r}\geq{\bf
q}_{r+1}$ $\displaystyle=$ $\displaystyle 0,$ (12)

${\bf c}_{0}=1$, ${\bf c}_{r+1}=0$, and
${\mathcal{U}}_{k}\triangleq[u^{(4,k)},{\bf u}^{(2,k)},{\bf h}^{(k)}]$ such
that the components of $u^{(4,k)}\in{\mathbb{R}}$, ${\bf
u}^{(2,k)}\in{\mathbb{R}}^{m}$, and ${\bf h}^{(k)}\in{\mathbb{R}}^{n}$ are
i.i.d. standard normals, we set

$\displaystyle\psi_{S,\infty}(f_{S},{\cal X},{\cal Y},{\bf p},{\bf q},{\bf
c},x,y,s)={\mathbb{E}}_{G,{\mathcal{U}}_{r+1}}\frac{1}{n{\bf
c}_{r}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{2}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{1}}\left(\left(Z_{S,\infty}\right)^{{\bf
c}_{2}}\right)\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)\right)^{\frac{{\bf c}_{4}}{{\bf
c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}}\right),$

where

$\displaystyle Z_{S,\infty}$ $\displaystyle\triangleq$ $\displaystyle
e^{D_{0,S,\infty}}$ $\displaystyle D_{0,S,\infty}$ $\displaystyle\triangleq$
$\displaystyle\max_{{\bf x}\in{\mathcal{X}},\|{\bf x}\|_{2}=x}s\max_{{\bf
y}\in{\mathcal{Y}},\|{\bf
y}\|_{2}=y}\left(f_{S}+\sqrt{n}y\left(\sum_{k=2}^{r+1}c_{k}{\bf
h}^{(k)}\right)^{T}{\bf x}+\sqrt{n}x{\bf y}^{T}\left(\sum_{k=2}^{r+1}b_{k}{\bf
u}^{(2,k)}\right)\right)$ $\displaystyle b_{k}$ $\displaystyle\triangleq$
$\displaystyle b_{k}({\bf p},{\bf q})=\sqrt{{\bf p}_{k-1}-{\bf p}_{k}}$
$\displaystyle c_{k}$ $\displaystyle\triangleq$ $\displaystyle c_{k}({\bf
p},{\bf q})=\sqrt{{\bf q}_{k-1}-{\bf q}_{k}}.$ (14)

The following theorem is a fundamental element of sfl RDT.

######  Theorem 1 ([86]).

Assume that $n$ is large and that
$\alpha=\lim_{n\rightarrow\infty}\frac{m}{n}$, where $\alpha$ remains constant
as $n$ grows. Let matrix $G\in{\mathbb{R}}^{m\times n}$ be comprised of i.i.d.
standard normal elements and let ${\mathcal{X}}\subseteq{\mathbb{R}}^{n}$ and
${\mathcal{Y}}\subseteq{\mathbb{R}}^{m}$ be two given sets. Assume the
complete sfl RDT frame from [84] and consider a given function $f({\bf
x}):R^{n}\rightarrow R$. Set

$\displaystyle\psi_{rp}$ $\displaystyle\triangleq-\max_{{\bf
x}\in{\mathcal{X}}}s\max_{{\bf y}\in{\mathcal{Y}}}\left(f({\bf x})+{\bf
y}^{T}G{\bf
x}\right)\qquad\mbox{({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{random
primal}})}$ $\displaystyle\psi_{rd}({\bf p},{\bf q},{\bf c},x,y,s)$
$\displaystyle\triangleq\frac{x^{2}y^{2}}{2}\sum_{k=2}^{r+1}\Bigg{(}\Bigg{.}{\bf
p}_{k-1}{\bf q}_{k-1}-{\bf p}_{k}{\bf q}_{k}\Bigg{.}\Bigg{)}{\bf
c}_{k}-\psi_{S,\infty}(f({\bf x}),{\cal X},{\cal Y},{\bf p},{\bf q},{\bf
c},x,y,s)\hskip
2.8903pt\mbox{({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{fl
random dual}})}.$

Let $\hat{{\bf p}_{0}}\rightarrow 1$, $\hat{{\bf q}_{0}}\rightarrow 1$, and
$\hat{{\bf c}_{0}}\rightarrow 1$, $\hat{{\bf p}}_{r+1}=\hat{{\bf
q}}_{r+1}=\hat{{\bf c}}_{r+1}=0$, and let the non-fixed parts of $\hat{{\bf
p}}\triangleq\hat{{\bf p}}(x,y)$, $\hat{{\bf q}}\triangleq\hat{{\bf q}}(x,y)$,
and $\hat{{\bf c}}\triangleq\hat{{\bf c}}(x,y)$ be the solutions of the
following system

$\displaystyle\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},x,y,s)}{d{\bf
p}}=0,\quad\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},x,y,s)}{d{\bf
q}}=0,\quad\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},x,y,s)}{d{\bf c}}=0.$ (16)

Then,

$\displaystyle\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\psi_{rp}}{\sqrt{n}}$
$\displaystyle=$
$\displaystyle\min_{x>0}\max_{y>0}\lim_{n\rightarrow\infty}\psi_{rd}(\hat{{\bf
p}}(x,y),\hat{{\bf q}}(x,y),\hat{{\bf
c}}(x,y),x,y,s)\qquad\mbox{({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{strong
sfl random duality}})},$

where $\psi_{S,\infty}(\cdot)$ is as in (2)-(2).

###### Proof.

The $s=-1$ scenario was proven in [86]. The $s=1$ follows through a line-by-
line repetition and trivial adjustment of the arguments of Section 3 of [86]
with $s=-1$ replaced by $s=1$. ∎

The following corollary is the operational version of the above theorem
directly applicable to the Hopfield models of interest here.

######  Corollary 1.

Assume the setup of Theorem 1. Let all elements of
${\mathcal{X}}\subseteq{\mathbb{R}}^{n}$ and
${\mathcal{Y}}\subseteq{\mathbb{R}}^{m}$ have unit norm. Set

$\displaystyle\psi_{rp}$ $\displaystyle\triangleq-\max_{{\bf
x}\in{\mathcal{X}}}s\max_{{\bf y}\in{\mathcal{Y}}}\left({\bf y}^{T}G{\bf
x}\right)\qquad\mbox{({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{random
primal}})}$ $\displaystyle\psi_{rd}({\bf p},{\bf q},{\bf c},x,y,s)$
$\displaystyle\triangleq\frac{1}{2}\sum_{k=2}^{r+1}\Bigg{(}\Bigg{.}{\bf
p}_{k-1}{\bf q}_{k-1}-{\bf p}_{k}{\bf q}_{k}\Bigg{.}\Bigg{)}{\bf
c}_{k}-\psi_{S,\infty}(0,{\cal X},{\cal Y},{\bf p},{\bf q},{\bf
c},1,1,s)\qquad\mbox{({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{fl
random dual}})}.$

Let the non-fixed parts of $\hat{{\bf p}}$, $\hat{{\bf q}}$, and $\hat{{\bf
c}}$ be the solutions of the following system

$\displaystyle\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},1,1,s)}{d{\bf
p}}=0,\quad\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},1,1,s)}{d{\bf
q}}=0,\quad\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},1,1,s)}{d{\bf c}}=0.$ (19)

Then,

$\displaystyle\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\psi_{rp}}{\sqrt{n}}$
$\displaystyle=$ $\displaystyle\lim_{n\rightarrow\infty}\psi_{rd}(\hat{{\bf
p}},\hat{{\bf q}},\hat{{\bf
c}},1,1,s)\qquad\mbox{({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{strong
sfl random duality}})},$

where $\psi_{S,\infty}(\cdot)$ is as in (2)-(2).

###### Proof.

A trivial direct consequence of Theorem 1, after taking $f({\bf x})=0$ and
recognizing that sets ${\mathcal{X}}$ and ${\mathcal{Y}}$ now have unit norm
elements. ∎

As noted in [86], the above random primal problems’ trivial concentrations
enable various corresponding probabilistic variants of (1) and (1). We,
however, skip stating such trivialities.

##  3 Practical evaluations

The results of Theorem 1 and Corollary 1 have very elegant mathematical forms.
Still, to see any formal utilization, one should be able to practically
evaluate them. There are two main obstacles that one might face when trying to
do so: (i) It is not, a priori, clear what should be the correct value for
$r$. (ii) Set ${\mathcal{Y}}$ lacks a component-wise structure
characterization and the decoupling over ${\bf y}$ is not necessarily overly
obvious. For example, if the optimal $r$ is large (and the $r$-convergence of
obtained results rather slow), or if the decoupling over ${\bf y}$ is
technically complicated, the importance of the characterizations given in
Theorem 1 and Corollary 1 would remain on a purely theoretical level and their
elegance would prove as irrelevant regarding the practical usefulness. Below,
we show that neither of these two obstacles poses a serious detriment to the
overall success of the introduced methodology.

We start by recognizing that, based on the results of Corollary 1, the key
object of practical interest is the so-called _random dual_

$\displaystyle\psi_{rd}({\bf p},{\bf q},{\bf c},1,1,s)$
$\displaystyle\triangleq\frac{1}{2}\sum_{k=2}^{r+1}\Bigg{(}\Bigg{.}{\bf
p}_{k-1}{\bf q}_{k-1}-{\bf p}_{k}{\bf q}_{k}\Bigg{.}\Bigg{)}{\bf
c}_{k}-\psi_{S,\infty}(0,{\cal X},{\cal Y},{\bf p},{\bf q},{\bf c},1,1,s).$
$\displaystyle=\frac{1}{2}\sum_{k=2}^{r+1}\Bigg{(}\Bigg{.}{\bf p}_{k-1}{\bf
q}_{k-1}-{\bf p}_{k}{\bf q}_{k}\Bigg{.}\Bigg{)}{\bf
c}_{k}-\frac{1}{n}\varphi(D^{(bin)}(s))-\frac{1}{n}\varphi(D^{(sph)}(s)),$

where analogously to (2)-(2)

$\displaystyle\varphi(D,{\bf c})$ $\displaystyle=$
$\displaystyle{\mathbb{E}}_{G,{\mathcal{U}}_{r+1}}\frac{1}{{\bf
c}_{r}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{2}}\left(\left(e^{D}\right)^{{\bf
c}_{2}}\right)\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)\right)^{\frac{{\bf c}_{4}}{{\bf
c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}}\right),$

and

$\displaystyle D^{(bin)}(s)$ $\displaystyle=$ $\displaystyle\max_{{\bf
x}\in{\mathcal{X}}}\left(s\sqrt{n}\left(\sum_{k=2}^{r+1}c_{k}{\bf
h}^{(k)}\right)^{T}{\bf x}\right)$ $\displaystyle D^{(sph)}(s)$
$\displaystyle\triangleq$ $\displaystyle s\max_{{\bf
y}\in{\mathcal{Y}}}\left(\sqrt{n}{\bf y}^{T}\left(\sum_{k=2}^{r+1}b_{k}{\bf
u}^{(2,k)}\right)\right).$ (23)

One then trivially has

$\displaystyle D^{(bin)}(s)$ $\displaystyle=$ $\displaystyle\max_{{\bf
x}\in{\mathcal{X}}}\left(s\sqrt{n}\left(\sum_{k=2}^{r+1}c_{k}{\bf
h}^{(k)}\right)^{T}{\bf
x}\right)=\sum_{i=1}^{n}\left|s\left(\sum_{k=2}^{r+1}c_{k}{\bf
h}_{i}^{(k)}\right)\right|=\sum_{i=1}^{n}\left|\left(\sum_{k=2}^{r+1}c_{k}{\bf
h}_{i}^{(k)}\right)\right|=\sum_{i=1}^{n}D^{(bin)}_{i},$

with

$\displaystyle D^{(bin)}_{i}(c_{k})=\left|\left(\sum_{k=2}^{r+1}c_{k}{\bf
h}_{i}^{(k)}\right)\right|.$ (25)

Consequently,

$\displaystyle\varphi(D^{(bin)}(s),{\bf c})$ $\displaystyle=$ $\displaystyle
n{\mathbb{E}}_{G,{\mathcal{U}}_{r+1}}\frac{1}{{\bf
c}_{r}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{2}}\left(e^{{\bf
c}_{2}D_{1}^{(bin)}}\right)\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)\right)^{\frac{{\bf c}_{4}}{{\bf
c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf
c}_{r-1}}}\right)=n\varphi(D_{1}^{(bin)}).$

On the other hand, we also have

$\displaystyle D^{(sph)}(s)$ $\displaystyle\triangleq$ $\displaystyle
s\max_{{\bf y}\in{\mathcal{Y}}}\left(\sqrt{n}{\bf
y}^{T}\left(\sum_{k=2}^{r+1}b_{k}{\bf
u}^{(2,k)}\right)\right)=s\sqrt{n}\left\|\sum_{k=2}^{r+1}b_{k}{\bf
u}^{(2,k)}\right\|_{2}.$ (27)

We now utilize the following _square root trick_ introduced on numerous
occasions in [78, 77, 81, 79]

$\displaystyle D^{(sph)}(s)$ $\displaystyle=$ $\displaystyle
s\sqrt{n}\left\|\sum_{k=2}^{r+1}b_{k}{\bf
u}^{(2,k)}\right\|_{2}=s\sqrt{n}\min_{\gamma}\left(\frac{\left\|\sum_{k=2}^{r+1}b_{k}{\bf
u}^{(2,k)}\right\|_{2}^{2}}{4\gamma}+\gamma\right)$ (28) $\displaystyle=$
$\displaystyle
s\sqrt{n}\min_{\gamma}\left(\frac{\sum_{i=1}^{m}\left(\sum_{k=2}^{r+1}b_{k}{\bf
u}_{i}^{(2,k)}\right)^{2}}{4\gamma}+\gamma\right).$

After scaling $\gamma=\gamma_{sq}\sqrt{n}$, (28) becomes

$\displaystyle D^{(sph)}(s)$ $\displaystyle=$ $\displaystyle
s\sqrt{n}\min_{\gamma_{sq}}\left(\frac{\sum_{i=1}^{m}\left(\sum_{k=2}^{r+1}b_{k}{\bf
u}_{i}^{(2,k)}\right)^{2}}{4\gamma_{sq}\sqrt{n}}+\gamma_{sq}\sqrt{n}\right)=s\min_{\gamma_{sq}}\left(\frac{\sum_{i=1}^{m}\left(\sum_{k=2}^{r+1}b_{k}{\bf
u}_{i}^{(2,k)}\right)^{2}}{4\gamma_{sq}}+\gamma_{sq}n\right)$ $\displaystyle=$
$\displaystyle
s\min_{\gamma_{sq}}\left(\sum_{i=1}^{m}D_{i}^{(sph)}+\gamma_{sq}n\right),$

with

$\displaystyle D_{i}^{(sph)}(b_{k})=\frac{\left(\sum_{k=2}^{r+1}b_{k}{\bf
u}_{i}^{(2,k)}\right)^{2}}{4\gamma_{sq}}.$ (30)

Below, we distinguish two types of Hopfield models: (i) the positive ones
(obtained for $s=1$); and (ii) the negative ones (obtained for $s=-1$).

###  3.1 Positive square root Hopfield models ($s=1$)

Connecting the ground state energy of the positive square root Hopfield
models, $f^{+}_{sq}$ given in (10), and the random primal, $\psi_{rp}(\cdot)$
given in Corollary 1, we have

$\displaystyle f_{sq}^{+}(\infty)$ $\displaystyle=$
$\displaystyle\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}\sqrt{{\bf x}^{T}G^{T}G{\bf
x}}}{\sqrt{n}}=\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}\max_{{\bf y}\in{\mathcal{Y}}}{\bf y}^{T}G{\bf
x}}{\sqrt{n}}$ (31) $\displaystyle=$
$\displaystyle-\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\psi_{rp}}{\sqrt{n}}=-\lim_{n\rightarrow\infty}\psi_{rd}(\hat{{\bf
p}},\hat{{\bf q}},\hat{{\bf c}},1,1,1),$

where the non-fixed parts of $\hat{{\bf p}}$, $\hat{{\bf q}}$, and $\hat{{\bf
c}}$ are the solutions of the following system

$\displaystyle\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},1,1,1)}{d{\bf
p}}=0,\quad\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},1,1,1)}{d{\bf
q}}=0,\quad\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},1,1,1)}{d{\bf c}}=0.$ (32)

Moreover, relying on (3)-(30), we have

$\displaystyle\lim_{n\rightarrow\infty}\psi_{rd}(\hat{{\bf p}},\hat{{\bf
q}},\hat{{\bf c}},1,1,1)=\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf q}},\hat{{\bf
c}},\hat{\gamma}_{sq},1,1,1),$ (33)

where

$\displaystyle\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,1)$
$\displaystyle=$ $\displaystyle\frac{1}{2}\sum_{k=2}^{r+1}\Bigg{(}\Bigg{.}{\bf
p}_{k-1}{\bf q}_{k-1}-{\bf p}_{k}{\bf q}_{k}\Bigg{.}\Bigg{)}{\bf c}_{k}$ (34)
$\displaystyle-\varphi(D_{1}^{(bin)}(c_{k}({\bf p},{\bf q})),{\bf
c})-\gamma_{sq}-\alpha\varphi(D_{1}^{(sph)}(b_{k}({\bf p},{\bf q})),{\bf c}),$

with $\varphi(D_{1}^{(bin)}(c_{k}({\bf p},{\bf q})),{\bf c})$ (based on (3),
(25), and (3)) given by

$\displaystyle\varphi(D_{1}^{(bin)}(c_{k}({\bf p},{\bf q})),{\bf c})$
$\displaystyle={\mathbb{E}}_{{\mathcal{U}}_{r+1}}\frac{1}{{\bf
c}_{r}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{2}}\left(e^{{\bf
c}_{2}\left|\sum_{k=2}^{r+1}c_{k}({\bf p},{\bf q}){\bf
h}_{i}^{(k)}\right|}\right)\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)\right)^{\frac{{\bf c}_{4}}{{\bf
c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}}\right),$

$\varphi(D_{1}^{(sph)}(b_{k}({\bf p},{\bf q})),{\bf c})$ (based on (3), (3),
and (30)) given by

$\displaystyle\varphi(D_{1}^{(sph)}(b_{k}({\bf p},{\bf q})),{\bf c})$
$\displaystyle={\mathbb{E}}_{{\mathcal{U}}_{r+1}}\frac{1}{{\bf
c}_{r}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{2}}\left(e^{{\bf
c}_{2}\frac{\left(\sum_{k=2}^{r+1}b_{k}({\bf p},{\bf q}){\bf
u}_{i}^{(2,k)}\right)^{2}}{4\gamma_{sq}}}\right)\right)^{\frac{{\bf
c}_{3}}{{\bf c}_{2}}}\right)\right)^{\frac{{\bf c}_{4}}{{\bf
c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}}\right),$

and with $\hat{\gamma}_{sq}$ and the non-fixed parts of $\hat{{\bf p}}$,
$\hat{{\bf q}}$, and $\hat{{\bf c}}$ being the solutions of the following
system

$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,1)}{d{\bf p}}$ $\displaystyle=$ $\displaystyle 0$
$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,1)}{d{\bf q}}$ $\displaystyle=$ $\displaystyle 0$
$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,1)}{d{\bf c}}$ $\displaystyle=$ $\displaystyle 0$
$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,1)}{d\gamma_{sq}}$ $\displaystyle=$ $\displaystyle 0,$ (37)

and consequently

$\displaystyle c_{k}(\hat{{\bf p}},\hat{{\bf q}})$ $\displaystyle=$
$\displaystyle\sqrt{\hat{{\bf q}}_{k-1}-\hat{{\bf q}}_{k}}$ $\displaystyle
b_{k}(\hat{{\bf p}},\hat{{\bf q}})$ $\displaystyle=$
$\displaystyle\sqrt{\hat{{\bf p}}_{k-1}-\hat{{\bf p}}_{k}}.$ (38)

Connecting (31) and (33), we then find

$\displaystyle f_{sq}^{+}(\infty)$ $\displaystyle=$
$\displaystyle\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}\pm\sqrt{{\bf x}^{T}G^{T}G{\bf
x}}}{\sqrt{n}}=\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}\max_{{\bf y}\in{\mathcal{Y}}}{\bf y}^{T}G{\bf
x}}{\sqrt{n}}$ $\displaystyle=$
$\displaystyle-\lim_{n\rightarrow\infty}\psi_{rd}(\hat{{\bf p}},\hat{{\bf
q}},\hat{{\bf c}},1,1,1)=-\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf
q}},\hat{{\bf c}},\hat{\gamma}_{sq},1,1,1)$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\sum_{k=2}^{r+1}\Bigg{(}\Bigg{.}\hat{{\bf
p}}_{k-1}\hat{{\bf q}}_{k-1}-\hat{{\bf p}}_{k}\hat{{\bf
q}}_{k}\Bigg{.}\Bigg{)}\hat{{\bf c}}_{k}+\varphi(D_{1}^{(bin)}(c_{k}(\hat{{\bf
p}},\hat{{\bf q}})),{\bf
c})+\hat{\gamma}_{sq}+\alpha\varphi(D_{1}^{(sph)}(b_{k}(\hat{{\bf
p}},\hat{{\bf q}})),{\bf c}).$

The following theorem summarizes the above results.

######  Theorem 2.

Assume the complete sfl RDT setup of [84]. Consider large $n$ linear regime
with $\alpha=\lim_{n\rightarrow\infty}\frac{m}{n}$ and let $\varphi(\cdot)$
and $\bar{\psi}(\cdot)$ be as given in (3) and (34), respectively. Also, let
the “fixed” parts of $\hat{{\bf p}}$, $\hat{{\bf q}}$, and $\hat{{\bf c}}$
satisfy $\hat{{\bf p}}_{1}\rightarrow 1$, $\hat{{\bf q}}_{1}\rightarrow 1$,
$\hat{{\bf c}}_{1}\rightarrow 1$, $\hat{{\bf p}}_{r+1}=\hat{{\bf
q}}_{r+1}=\hat{{\bf c}}_{r+1}=0$, and let the “non-fixed” parts of $\hat{{\bf
p}}_{k}$, $\hat{{\bf q}}_{k}$, and $\hat{{\bf c}}_{k}$
($k\in\\{2,3,\dots,r\\}$) be the solutions of (3.1). Moreover, let
$c_{k}(\hat{{\bf p}},\hat{{\bf q}})$ and $b_{k}(\hat{{\bf p}},\hat{{\bf q}})$
be as in (3.1). Then

$\displaystyle f_{sq}^{+}(\infty)$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\sum_{k=2}^{r+1}\Bigg{(}\Bigg{.}\hat{{\bf
p}}_{k-1}\hat{{\bf q}}_{k-1}-\hat{{\bf p}}_{k}\hat{{\bf
q}}_{k}\Bigg{.}\Bigg{)}\hat{{\bf c}}_{k}+\varphi(D_{1}^{(bin)}(c_{k}(\hat{{\bf
p}},\hat{{\bf q}})),\hat{{\bf
c}})+\hat{\gamma}_{sq}+\alpha\varphi(D_{1}^{(sph)}(b_{k}(\hat{{\bf
p}},\hat{{\bf q}})),\hat{{\bf c}}).$

###### Proof.

Follows from the previous discussion, Theorem 1, Corollary 1, and the sfl RDT
machinery presented in [85, 84, 86]. ∎

####  3.1.1 Numerical evaluations

Theorem 2 is sufficient to conduct numerical evaluations and obtain concrete
values for the studied free energy. There are several analytical results that
can be helpful in that regard and that we present below. As it will be soon
clear, they mostly relate to the evaluation of the so-called _spherical_ part
$\varphi(D_{i}^{(sph)}(c_{k}({\bf p},{\bf q})),{\bf c})$, where one can obtain
convenient closed form expressions. For the easiness of the exposition, we
start with the smallest possible $r$ and proceed by incrementally increasing
it by $1$. Such a path eventually enables seeing in a neat and systematic way
how the whole lifting mechanism _astonishingly quickly_ converges. For the
numerical values concreteness, we, on occasion, specialize the considerations
to the most famous, so-called, square case, i.e., to the case $\alpha=1$.

1) _$r=1$ – first level of lifting:_ For $r=1$ one has that $\hat{{\bf
p}}_{1}\rightarrow 1$ and $\hat{{\bf q}}_{1}\rightarrow 1$ which together with
$\hat{{\bf p}}_{r+1}=\hat{{\bf p}}_{2}=\hat{{\bf q}}_{r+1}=\hat{{\bf
q}}_{2}=0$, and $\hat{{\bf c}}_{2}\rightarrow 0$ gives

$\displaystyle-\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf q}},\hat{{\bf
c}},\gamma_{sq},1,1,1)$ $\displaystyle=-\frac{1}{2}{\bf c}_{2}+\frac{1}{{\bf
c}_{2}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf c}_{2}|\sqrt{1-0}{\bf
h}_{1}^{(2)}|}\right)+\gamma_{sq}+\alpha\frac{1}{{\bf
c}_{2}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}\frac{(\sqrt{1-0}{\bf u}_{1}^{(2,2)})^{2}}{4\gamma_{sq}}}\right)$
$\displaystyle\rightarrow\frac{1}{{\bf
c}_{2}}\log\left(1+{\mathbb{E}}_{{\mathcal{U}}_{2}}{\bf c}_{2}|\sqrt{1-0}{\bf
h}_{1}^{(2)}|\right)+\gamma_{sq}+\alpha\frac{1}{{\bf
c}_{2}}\log\left(1+{\mathbb{E}}_{{\mathcal{U}}_{2}}{\bf
c}_{2}\frac{(\sqrt{1-0}{\bf u}_{1}^{(2,2)})^{2}}{4\gamma_{sq}}\right)$
$\displaystyle\rightarrow\frac{1}{{\bf c}_{2}}\log\left(1+{\bf
c}_{2}\sqrt{\frac{2}{\pi}}\right)+\gamma_{sq}+\alpha\frac{1}{{\bf
c}_{2}}\log\left(1+\frac{{\bf c}_{2}}{4\gamma_{sq}}\right)$
$\displaystyle\rightarrow\sqrt{\frac{2}{\pi}}+\gamma_{sq}+\frac{\alpha}{4\gamma_{sq}}.$
(41)

One then easily finds $\hat{\gamma}_{sq}=\frac{\sqrt{\alpha}}{2}$ and

$\displaystyle f^{+,1}_{sq}(\infty)=-\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf
q}},\hat{{\bf c}},\hat{\gamma}_{sq},1,1,1)$
$\displaystyle=\sqrt{\frac{2}{\pi}}+\sqrt{\alpha},$ (42)

which for $\alpha=1$ becomes

$\displaystyle
f^{+,1}_{sq}(\infty)=\sqrt{\frac{2}{\pi}}+1\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{1.7979}}.$
(43)

2) _$r=2$ – second level of lifting:_ To make the presentation smoother, we
split the second level of lifting into two subparts: (i) _partial_ second
level of lifting; and (ii) _full_ second level of lifting.

For the _partial_ part, we have, when $r=2$, that $\hat{{\bf
p}}_{1}\rightarrow 1$ and $\hat{{\bf q}}_{1}\rightarrow 1$, $\hat{{\bf
p}}_{2}=\hat{{\bf q}}_{2}=0$, and $\hat{{\bf p}}_{r+1}=\hat{{\bf
p}}_{3}=\hat{{\bf q}}_{r+1}=\hat{{\bf q}}_{3}=0$ but in general $\hat{{\bf
c}}_{2}\neq 0$. As above, one again has

$\displaystyle-\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf q}},{\bf
c},\gamma_{sq},1,1,1)$ $\displaystyle=-\frac{1}{2}{\bf c}_{2}+\frac{1}{{\bf
c}_{2}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf c}_{2}|\sqrt{1-0}{\bf
h}_{1}^{(2)}|}\right)+\gamma_{sq}+\alpha\frac{1}{{\bf
c}_{2}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}\frac{(\sqrt{1-0}{\bf u}_{1}^{(2,2)})^{2}}{4\gamma_{sq}}}\right).$ (44)

After taking the derivatives with respect to ${\bf c}_{2}$ and $\gamma_{sq}$,
one, for $\alpha=1$, finds $\hat{\gamma}_{sq}=0.6173$ and $\hat{{\bf
c}}_{2}=0.4246$, which gives

$\displaystyle(\mbox{\emph{partial} second level:})\qquad\qquad
f^{+,2}_{sq}(\infty)\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{1.7832}}.$
(45)

For the _full_ part, we have the same setup as above with the exception that
now, in general, (in addition to $\hat{{\bf c}}_{2}\neq 0$) one has ${\bf
p}_{2}\neq 0$ and ${\bf q}_{2}\neq 0$. As above, we write

$\displaystyle-\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,1)$
$\displaystyle=-\frac{1}{2}(1-{\bf p}_{2}{\bf q}_{2}){\bf c}_{2}+\frac{1}{{\bf
c}_{2}}{\mathbb{E}}_{{\mathcal{U}}_{3}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sqrt{1-{\bf q}_{2}}{\bf h}_{1}^{(2)}+\sqrt{{\bf q}_{2}}{\bf
h}_{1}^{(3)}|}\right)$ $\displaystyle\quad+\gamma_{sq}+\alpha\frac{1}{{\bf
c}_{2}}{\mathbb{E}}_{{\mathcal{U}}_{3}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}\frac{(\sqrt{1-{\bf p}_{2}}{\bf u}_{i}^{(2,2)}+\sqrt{{\bf p}_{2}}{\bf
u}_{i}^{(2,3)})^{2}}{4\gamma_{sq}}}\right)$ $\displaystyle=-\frac{1}{2}(1-{\bf
p}_{2}{\bf q}_{2}){\bf c}_{2}+\frac{1}{{\bf
c}_{2}}{\mathbb{E}}_{{\mathcal{U}}_{3}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sqrt{1-{\bf q}_{2}}{\bf h}_{1}^{(2)}+\sqrt{{\bf q}_{2}}{\bf
h}_{1}^{(3)}|}\right)$
$\displaystyle\quad+\gamma_{sq}+\alpha\left(-\frac{1}{2{\bf
c}_{2}}\log\left(\frac{2\gamma_{sq}-{\bf c}_{2}(1-{\bf
p}_{2})}{2\gamma_{sq}}\right)+\frac{{\bf p}_{2}}{2(2\gamma_{sq}-{\bf
c}_{2}(1-{\bf p}_{2}))}\right)$ (46)

After taking the derivatives one, for $\alpha=1$, finds
$\hat{\gamma}_{sq}=0.6941$, ${\bf p}_{2}=0.5315$, ${\bf q}_{2}=0.6320$, and
$\hat{{\bf c}}_{2}=1.0056$, which gives

$\displaystyle(\mbox{\emph{full} second level:})\qquad\qquad
f^{+,2}_{sq}(\infty)\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{1.7801}}.$
(47)

3) _$r=3$ – third level of lifting:_ Since the above already showed the idea
behind the partial lifting, we now immediately consider _full_ third level of
lifting and at the end discuss the specialization to the partial part. For
$r=3$, we have that $\hat{{\bf p}}_{1}\rightarrow 1$ and $\hat{{\bf
q}}_{1}\rightarrow 1$ as well as $\hat{{\bf p}}_{r+1}=\hat{{\bf
p}}_{4}=\hat{{\bf q}}_{r+1}=\hat{{\bf q}}_{4}=0$. As above, one again has

$\displaystyle-\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,1)$
$\displaystyle=-\frac{1}{2}(1-{\bf p}_{2}{\bf q}_{2}){\bf
c}_{2}-\frac{1}{2}({\bf p}_{2}{\bf q}_{2}-{\bf p}_{3}{\bf q}_{3}){\bf c}_{3}$
$\displaystyle\quad+\frac{1}{{\bf
c}_{3}}{\mathbb{E}}_{{\mathcal{U}}_{4}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sqrt{1-{\bf q}_{2}}{\bf h}_{1}^{(2)}+\sqrt{{\bf q}_{2}-{\bf
q}_{3}}{\bf h}_{1}^{(3)}+\sqrt{{\bf q}_{3}}{\bf
h}_{1}^{(4)}|}\right)^{\frac{{\bf c}_{3}}{{\bf c}_{2}}}\right)$
$\displaystyle\quad+\gamma_{sq}+\alpha\frac{1}{{\bf
c}_{3}}{\mathbb{E}}_{{\mathcal{U}}_{4}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{4}}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}\frac{(\sqrt{1-{\bf p}_{2}}{\bf u}_{i}^{(2,2)}+\sqrt{{\bf p}_{2}-{\bf
p}_{3}}{\bf u}_{i}^{(2,3)}+\sqrt{{\bf p}_{3}}{\bf
u}_{i}^{(2,4)})^{2}}{4\gamma_{sq}}}\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)$ $\displaystyle=-\frac{1}{2}(1-{\bf p}_{2}{\bf q}_{2}){\bf
c}_{2}-\frac{1}{2}({\bf p}_{2}{\bf q}_{2}-{\bf p}_{3}{\bf q}_{3}){\bf c}_{3}$
$\displaystyle\quad+\frac{1}{{\bf
c}_{3}}{\mathbb{E}}_{{\mathcal{U}}_{4}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sqrt{1-{\bf q}_{2}}{\bf h}_{1}^{(2)}+\sqrt{{\bf q}_{2}-{\bf
q}_{3}}{\bf h}_{1}^{(3)}+\sqrt{{\bf q}_{3}}{\bf
h}_{1}^{(4)}|}\right)^{\frac{{\bf c}_{3}}{{\bf c}_{2}}}\right)$
$\displaystyle\quad+\gamma_{sq}+\alpha\Bigg{(}\Bigg{.}-\frac{1}{2{\bf
c}_{2}}\log\left(\frac{2\gamma_{sq}-{\bf c}_{2}(1-{\bf
p}_{2})}{2\gamma_{sq}}\right)$ $\displaystyle\quad-\frac{1}{2{\bf
c}_{3}}\log\left(\frac{2\gamma_{sq}-{\bf c}_{2}(1-{\bf p}_{2})-{\bf
c}_{3}({\bf p}_{2}-{\bf p}_{3})}{2\gamma_{sq}-{\bf c}_{2}(1-{\bf
p}_{2})}\right)$ $\displaystyle\quad+\frac{{\bf p}_{3}}{2(2\gamma_{sq}-{\bf
c}_{2}(1-{\bf p}_{2})-{\bf c}_{3}({\bf p}_{2}-{\bf p}_{3}))}\Bigg{.}\Bigg{)}$
(48)

Taking the derivatives while fixing ${\bf p}_{3}={\bf q}_{3}=0$ one obtains
the _partial_ third level lifting and in particular for $\alpha=1$

$\displaystyle(\mbox{\emph{partial} third level:})\qquad\qquad
f^{+,3}_{sq}(\infty)\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{1.7791}}.$
(49)

On the other hand, taking the derivatives without fixing ${\bf p}_{3}={\bf
q}_{3}=0$ one obtains the _full_ third level lifting and in particular for
$\alpha=1$

$\displaystyle(\mbox{\emph{full} third level:})\qquad\qquad
f^{+,3}_{sq}(\infty)\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{1.7788}}.$
(50)

All relevant quantities for the third level of lifting (for both partial and
full scenarios), together with the first and second level corresponding ones
are systematically shown in Table 1.

Table 1: $r$-sfl RDT parameters; positive _square-root_ Hopfield model; $\alpha=1$; $\hat{{\bf c}}_{1}\rightarrow 1$; $n,\beta\rightarrow\infty$ $r$-sfl RDT | $\hat{\gamma}_{sq}$ | $\hat{{\bf p}}_{3}$ | $\hat{{\bf p}}_{2}$ | $\hat{{\bf p}}_{1}`$ | $\hat{{\bf q}}_{3}$ | $\hat{{\bf q}}_{2}$ | $\hat{{\bf q}}_{1}$ | $\hat{{\bf c}}_{3}$ | $\hat{{\bf c}}_{2}$ | $f_{sq}^{+,r}$
---|---|---|---|---|---|---|---|---|---|---
$\mathbf{1}$ (full) | $0.5$ | $0$ | $0$ | $\rightarrow 1$ | $0$ | $0$ | $\rightarrow 1$ | $\rightarrow 0$ | $\rightarrow 0$ | $\mathbf{1.7979}$
$\mathbf{2}$ (partial) | $0.6173$ | $0$ | $0$ | $\rightarrow 1$ | $0$ | $0$ | $\rightarrow 1$ | $\rightarrow 0$ | $0.4246$ | $\mathbf{1.7832}$
$\mathbf{2}$ (full) | $0.6941$ | $0$ | $0.5315$ | $\rightarrow 1$ | $0$ | $0.6320$ | $\rightarrow 1$ | $\rightarrow 0$ | $1.0056$ | $\mathbf{1.7801}$
$\mathbf{3}$ (partial) | $0.7434$ | $0$ | $0.7510$ | $\rightarrow 1$ | $0$ | $0.8397$ | $\rightarrow 1$ | $0.2508$ | $1.7762$ | $\mathbf{1.7791}$
$\mathbf{3}$ (full) | $0.7752$ | $0.36$ | $0.8504$ | $\rightarrow 1$ | $0.4427$ | $0.9160$ | $\rightarrow 1$ | $0.5044$ | $2.6825$ | $\mathbf{1.7788}$

4) General _$r$ -th level of lifting:_ For the completeness, we state the form
of $\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,1)$ obtained for
general $r$. Following the above procedures, we basically have

$\displaystyle-\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,1)$
$\displaystyle=-\frac{1}{2}\sum_{k=2}^{r+1}({\bf p}_{k-1}{\bf q}_{k-1}-{\bf
p}_{k}{\bf q}_{k}){\bf c}_{k}$ $\displaystyle\quad+\frac{1}{{\bf
c}_{r}}{\mathbb{E}}_{{\mathcal{U}}_{r+1}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sum_{k=2}^{r+1}\sqrt{{\bf q}_{k-1}-{\bf q}_{k}}{\bf
h}_{1}^{(k)}|}\right)^{\frac{{\bf c}_{3}}{{\bf c}_{2}}}\right)^{\frac{{\bf
c}_{4}}{{\bf c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}}\right)$
$\displaystyle\quad+\gamma_{sq}+\alpha\Bigg{(}\Bigg{.}-\sum_{k=2}^{r}\frac{1}{2{\bf
c}_{k}}\log\left(\frac{\Theta_{k}}{\Theta_{k-1}}\right)+\frac{{\bf
p}_{r}}{2\Theta_{r}}\Bigg{.}\Bigg{)},$ (51)

where

$\displaystyle\Theta_{1}$ $\displaystyle=$ $\displaystyle 2\gamma_{sq}$
$\displaystyle\Theta_{k}$ $\displaystyle=$ $\displaystyle\Theta_{k-1}-{\bf
c}_{k}({\bf p}_{k-1}-{\bf p}_{k}),k\in\\{2,3,\dots,r\\}.$ (52)

The above characterization of $\bar{\psi}(\cdot)$ gives explicit dependence on
all key parameters ${\bf p}$, ${\bf q}$, ${\bf m}$, and $\gamma_{sq}$ and
consequently enables direct computations of all the needed derivatives for any
$r\in{\mathbb{N}}$. Several additional helpful relations can be deduced from
[84]. We show them nest.

####  3.1.2 Helpful relations

We first need a few definitions and to that end start with

$\displaystyle Z_{(s)}$ $\displaystyle\triangleq$
$\displaystyle\sum_{i_{1}=1}^{l}\left(\sum_{i_{2}=1}^{l}e^{\beta
D_{0}^{(i_{1},i_{2})}}\right)^{s}=\sum_{i_{1}=1}^{l}\left(\sum_{i_{2}=1}^{l}A^{(i_{1},i_{2})}\right)^{s}=\sum_{i_{1}=1}^{l}\left(C^{(i_{1})}\right)^{s}$
$\displaystyle D_{0}^{(i_{1},i_{2})}$ $\displaystyle\triangleq$
$\displaystyle({\bf y}^{(i_{2})})^{T}\left(\sum_{k=1}^{r+1}b_{k}{\bf
u}^{(2,k)}\right)+\left(\sum_{k=1}^{r+1}c_{k}{\bf h}^{(k)}\right)^{T}{\bf
x}^{(i_{1})}$ (53)

With ${\bf m}_{0}=1$, we set

$\displaystyle\zeta_{r}\triangleq{\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{2}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{1}}\left(Z^{\frac{{\bf
m}_{1}}{{\bf m}_{0}}}\right)\right)^{\frac{{\bf m}_{2}}{{\bf
m}_{1}}}\right)\right)^{\frac{{\bf m}_{3}}{{\bf
m}_{2}}}\dots\right)^{\frac{{\bf m}_{r}}{{\bf m}_{r-1}}},r\geq 1.$ (54)

One can then write

$\displaystyle\zeta_{k}={\mathbb{E}}_{{\mathcal{U}}_{k}}\left(\zeta_{k-1}\right)^{\frac{{\bf
m}_{k}}{{\bf m}_{k-1}}},k\geq
2,\quad\mbox{and}\quad\zeta_{0}=Z,\qquad\zeta_{1}={\mathbb{E}}_{{\mathcal{U}}_{1}}\left(Z^{\frac{{\bf
m}_{1}}{{\bf m}_{0}}}\right).$ (55)

Also, consider the operators

$\displaystyle\Phi_{{\mathcal{U}}_{k}}$ $\displaystyle\triangleq$
$\displaystyle{\mathbb{E}}_{{\mathcal{U}}_{k}}\frac{\zeta_{k-1}^{\frac{{\bf
m}_{k}}{{\bf m}_{k-1}}}}{\zeta_{k}},$ (56)

and set

$\displaystyle\gamma_{0}(i_{1},i_{2})$ $\displaystyle=$
$\displaystyle\frac{(C^{(i_{1})})^{s}}{Z_{(s)}}\frac{A^{(i_{1},i_{2})}}{C^{(i_{1})}}$
$\displaystyle\gamma_{01}^{(r)}$ $\displaystyle=$
$\displaystyle\prod_{k=r}^{1}\Phi_{{\mathcal{U}}_{k}}(\gamma_{0}(i_{1},i_{2}))$
$\displaystyle\gamma_{02}^{(r)}$ $\displaystyle=$
$\displaystyle\prod_{k=r}^{1}\Phi_{{\mathcal{U}}_{k}}(\gamma_{0}(i_{1},i_{2})\times\gamma_{0}(i_{1},p_{2}))$
$\displaystyle\gamma_{k_{1}+1}^{(r)}$ $\displaystyle=$
$\displaystyle\prod_{k=r}^{k_{1}+1}\Phi_{{\mathcal{U}}_{k}}\left(\prod_{k=k_{1}}^{1}\Phi_{{\mathcal{U}}_{k}}\gamma_{0}(i_{1},i_{2})\times\prod_{k=k_{1}}^{1}\Phi_{{\mathcal{U}}_{k}}\gamma_{0}(p_{1},p_{2})\right).$
(57)

Then,[84] gives

$\displaystyle\hat{{\bf p}}_{k-1}$ $\displaystyle=$ $\displaystyle
E_{{\mathcal{U}}_{r+1}}\langle({\bf x}^{(i_{1})})^{T}{\bf
x}^{(p_{1})}\rangle_{\gamma_{k}^{(r)}}$ $\displaystyle\hat{{\bf q}}_{k-1}$
$\displaystyle=$ $\displaystyle E_{{\mathcal{U}}_{r+1}}\langle({\bf
y}^{(i_{2})})^{T}{\bf y}^{(p_{2})}\rangle_{\gamma_{k}^{(r)}},$ (58)

where $\langle\cdot\rangle_{\gamma_{k}^{(r)}}$ stands for the average with
respect to $\gamma_{k}^{(r)}$. For example, to make everything concrete, one
can take $r=3$ and for $\beta=\bar{\beta}\sqrt{n}$,
$\bar{\beta}\rightarrow\infty$ and $b_{k}=\hat{b}_{k}$ and
$c_{k}=\hat{c}_{k}$, observe that

$\displaystyle Z_{(s)}$ $\displaystyle\rightarrow$ $\displaystyle
e^{\max_{{\bf x}}s\max_{{\bf y}}\beta D_{0}^{(i_{1},i_{2})}}\rightarrow
e^{\bar{\beta}\left(\sqrt{n}\|\sum_{k=1}^{r+1}\hat{b}_{k}{\bf
u}^{(2,k)}\|_{2}+\left(\sum_{k=1}^{r+1}\hat{c}_{k}{\bf
h}^{(k)}\right)^{T}\bar{{\bf x}}\right)}$ $\displaystyle\rightarrow$
$\displaystyle e^{\bar{\beta}\left(\frac{\|\sum_{k=1}^{r+1}\hat{b}_{k}{\bf
u}^{(2,k)}\|_{2}^{2}}{4\hat{\gamma}_{sq}}+n\hat{\gamma}_{sq}+\left(\sum_{k=1}^{r+1}\hat{c}_{k}{\bf
h}^{(k)}\right)^{T}\bar{{\bf x}}\right)}$ $\displaystyle\rightarrow$
$\displaystyle e^{\bar{\beta}\left(\frac{\bar{{\bf
y}}^{T}\left(\sum_{k=1}^{r+1}\hat{b}_{k}{\bf
u}^{(2,k)}\right)}{2}+n\hat{\gamma}_{sq}+\left(\sum_{k=1}^{r+1}\hat{c}_{k}{\bf
h}^{(k)}\right)^{T}\bar{{\bf x}}\right)}$

where (since $\hat{{\bf p}}_{1}\rightarrow 1$ and $\hat{{\bf
q}}_{1}\rightarrow 1$) the optimal $\bar{{\bf x}}_{i}$ and $\bar{{\bf y}}_{i}$
are given as

$\displaystyle\bar{{\bf x}}_{i}$ $\displaystyle=$
$\displaystyle\mbox{sign}\left(\sqrt{1-\hat{{\bf q}}_{2}}{\bf
h}_{i}^{(2)}+\sqrt{\hat{{\bf q}}_{2}-\hat{{\bf q}}_{3}}{\bf
h}_{i}^{(3)}+\sqrt{\hat{{\bf q}}_{3}}{\bf h}_{i}^{(4)}\right)$
$\displaystyle\bar{{\bf y}}_{i}$ $\displaystyle=$
$\displaystyle\frac{\left(\sqrt{1-\hat{{\bf p}}_{2}}{\bf
u}_{i}^{(2,2)}+\sqrt{\hat{{\bf p}}_{2}-\hat{{\bf p}}_{3}}{\bf
u}_{i}^{(2,3)}+\sqrt{\hat{{\bf p}}_{3}}{\bf
u}_{i}^{(2,4)}\right)}{2\hat{\gamma}_{sq}}.$ (60)

Relying on (25), one further has

$\displaystyle
D^{(bin)}_{1}(\hat{c}_{k})=\left|\left(\sum_{k=2}^{r+1}\hat{c}_{k}{\bf
h}_{i}^{(k)}\right)\right|=\left|\left(\sum_{k=2}^{r+1}\sqrt{\hat{{\bf
q}}_{k-1}-\hat{{\bf q}}_{k}}{\bf h}_{i}^{(k)}\right)\right|=\left|\sqrt{1-{\bf
q}_{2}}{\bf h}_{1}^{(2)}+\sqrt{{\bf q}_{2}-{\bf q}_{3}}{\bf
h}_{1}^{(3)}+\sqrt{{\bf q}_{3}}{\bf h}_{1}^{(4)}\right|,$ (61)

and, analogously, relying on (30)

$\displaystyle
D_{1}^{(sph)}(\hat{b}_{k})=\frac{\left(\sum_{k=2}^{r+1}\hat{b}_{k}{\bf
u}_{i}^{(2,k)}\right)^{2}}{4\gamma_{sq}}=\frac{\left(\sqrt{1-\hat{{\bf
p}}_{2}}{\bf u}_{1}^{(2,2)}+\sqrt{\hat{{\bf p}}_{2}-\hat{{\bf p}}_{3}}{\bf
u}_{1}^{(2,3)}+\sqrt{\hat{{\bf p}}_{3}}{\bf
u}_{1}^{(2,4)}\right)^{2}}{4\gamma_{sq}}.$ (62)

We recall that triplets ${\bf u}^{(2,k)}$ and ${\bf h}^{(k)}$ are independent
of each other and that each of these vectors is comprised of i.i.d.
components, which allows a decoupling over $i$. Moreover, due to symmetry of
indices $i$ we also have from (3.1.2)

$\displaystyle\hat{{\bf p}}_{k-1}$ $\displaystyle=$ $\displaystyle
E_{{\mathcal{U}}_{r+1}}\langle({\bf x}^{(i_{1})})^{T}{\bf
x}^{(p_{1})}\rangle_{\gamma_{k}^{(r)}}=nE_{{\mathcal{U}}_{r+1}}\left\langle\frac{\bar{{\bf
x}}_{1}^{(i_{1})}}{\sqrt{n}}\frac{\bar{{\bf
x}}_{1}^{(p_{1})}}{\sqrt{n}}\right\rangle_{\gamma_{k}^{(r)}}=E_{{\mathcal{U}}_{r+1}}\left\langle\bar{{\bf
x}}_{1}^{(i_{1})}\bar{{\bf x}}_{1}^{(p_{1})}\right\rangle_{\gamma_{k}^{(r)}}$
$\displaystyle\hat{{\bf q}}_{k-1}$ $\displaystyle=$ $\displaystyle
E_{{\mathcal{U}}_{r+1}}\langle({\bf y}^{(i_{2})})^{T}{\bf
y}^{(p_{2})}\rangle_{\gamma_{k}^{(r)}}=nE_{{\mathcal{U}}_{r+1}}\left\langle\frac{\bar{{\bf
y}}_{1}^{(i_{2})}}{\sqrt{n}}\frac{\bar{{\bf
y}}_{1}^{(p_{2})}}{\sqrt{n}}\right\rangle_{\gamma_{k}^{(r)}}=E_{{\mathcal{U}}_{r+1}}\left\langle\bar{{\bf
y}}_{1}^{(i_{2})}\bar{{\bf y}}_{1}^{(p_{2})}\right\rangle_{\gamma_{k}^{(r)}}$
(63)

where $\bar{{\bf x}}^{(i_{1})}$ is optimal over $i_{1}$ indexing system and
$\bar{{\bf x}}^{(p_{1})}$ is optimal over $p_{1}$ indexing system and,
analogously, $\bar{{\bf y}}^{(i_{2})}$ is optimal over $i_{2}$ indexing system
and $\bar{{\bf y}}^{(p_{2})}$ is optimal over $p_{2}$ indexing system. From
(3.1.2), we then have

$\hat{{\bf
p}}_{2}={\mathbb{E}}_{{\mathcal{U}}_{4}}\left(\frac{{\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left(\frac{{\mathbb{E}}_{{\mathcal{U}}_{2}}\bar{{\bf
x}}_{1}e^{\hat{{\bf
c}}_{2}D^{(bin)}_{1}(\hat{c}_{k})}}{{\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}D^{(bin)}_{1}(\hat{c}_{k})}}\right)^{2}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}D^{(bin)}_{1}(\hat{c}_{k})}\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)}{{\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}D^{(bin)}_{1}(\hat{c}_{k})}\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)}\right),$ (64)

and

$\hat{{\bf
p}}_{3}={\mathbb{E}}_{{\mathcal{U}}_{4}}\left(\frac{{\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left(\frac{{\mathbb{E}}_{{\mathcal{U}}_{2}}\bar{{\bf
x}}_{1}e^{\hat{{\bf
c}}_{2}D^{(bin)}_{1}(\hat{c}_{k})}}{{\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}D^{(bin)}_{1}(\hat{c}_{k})}}\right)\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}D^{(bin)}_{1}(\hat{c}_{k})}\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)}{{\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}D^{(bin)}_{1}(\hat{c}_{k})}\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)}\right)^{2}.$ (65)

Analogously to (64) and (65), we also have

$\hat{{\bf
q}}_{2}=\alpha{\mathbb{E}}_{{\mathcal{U}}_{4}}\left(\frac{{\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left(\frac{{\mathbb{E}}_{{\mathcal{U}}_{2}}\bar{{\bf
y}}_{1}e^{\hat{{\bf
c}}_{2}\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)}}{{\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)}}\right)^{2}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)}\right)^{\frac{{\bf
c}_{3}}{{\bf
c}_{2}}}\right)}{{\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)}\right)^{\frac{{\bf
c}_{3}}{{\bf c}_{2}}}\right)}\right),$ (66)

and

$\hat{{\bf
q}}_{3}=\alpha{\mathbb{E}}_{{\mathcal{U}}_{4}}\left(\frac{{\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left(\frac{{\mathbb{E}}_{{\mathcal{U}}_{2}}\bar{{\bf
y}}_{1}e^{\hat{{\bf
c}}_{2}\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)}}{{\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)}}\right)\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)}\right)^{\frac{{\bf
c}_{3}}{{\bf
c}_{2}}}\right)}{{\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{\hat{{\bf
c}}_{2}\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)}\right)^{\frac{{\bf
c}_{3}}{{\bf c}_{2}}}\right)}\right)^{2},$ (67)

It is now simple to write the above equations for general $r$. First, with
${\bf c}_{1}\rightarrow 1$, we set

$\displaystyle\zeta_{r}^{(bin)}$ $\displaystyle\triangleq$
$\displaystyle{\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{2}}\left(\left(e^{D^{(bin)}_{1}(\hat{c}_{k})}\right)^{\frac{{\bf
c}_{2}}{{\bf c}_{1}}}\right)\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)\right)^{\frac{{\bf c}_{4}}{{\bf
c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}},r\geq 2$
$\displaystyle\zeta_{r}^{(sph)}$ $\displaystyle\triangleq$
$\displaystyle{\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left(\left({\mathbb{E}}_{{\mathcal{U}}_{2}}\left(\left(e^{\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)}\right)^{\frac{{\bf
c}_{2}}{{\bf c}_{1}}}\right)\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)\right)^{\frac{{\bf c}_{4}}{{\bf
c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}},r\geq 2$

One can then write

$\displaystyle\zeta_{k}^{(bin)}$
$\displaystyle={\mathbb{E}}_{{\mathcal{U}}_{k}}\left(\zeta_{k-1}^{(bin)}\right)^{\frac{{\bf
c}_{k}}{{\bf c}_{k-1}}},k\geq
3,\quad\zeta_{1}^{(bin)}=e^{D^{(bin)}_{1}(\hat{c}_{k})},\hskip
3.61371pt\zeta_{2}^{(bin)}={\mathbb{E}}_{{\mathcal{U}}_{2}}\left(\left(e^{D^{(bin)}_{1}(\hat{c}_{k})}\right)^{\frac{{\bf
c}_{2}}{{\bf c}_{1}}}\right)$ $\displaystyle\zeta_{k}^{(sph)}$
$\displaystyle={\mathbb{E}}_{{\mathcal{U}}_{k}}\left(\zeta_{k-1}^{(sph)}\right)^{\frac{{\bf
c}_{k}}{{\bf c}_{k-1}}},k\geq
3,\quad\zeta_{1}^{(sph)}=e^{\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)},\hskip
3.61371pt\zeta_{2}^{(sph)}={\mathbb{E}}_{{\mathcal{U}}_{2}}\left(\left(e^{\left(D^{(sph)}_{1}(\hat{b}_{k})+\hat{\gamma}_{sq}\right)}\right)^{\frac{{\bf
c}_{2}}{{\bf c}_{1}}}\right).$

Also, for $k\geq 2$, let

$\displaystyle\Phi_{{\mathcal{U}}_{k}}^{(bin)}(\cdot)$
$\displaystyle\triangleq$
$\displaystyle{\mathbb{E}}_{{\mathcal{U}}_{k}}\frac{\left(\zeta_{k-1}^{(bin)}\right)^{\frac{{\bf
c}_{k}}{{\bf c}_{k-1}}}}{\zeta_{k}^{(bin)}}\left(\cdot\right)$
$\displaystyle\Phi_{{\mathcal{U}}_{k}}^{(sph)}(\cdot)$
$\displaystyle\triangleq$
$\displaystyle{\mathbb{E}}_{{\mathcal{U}}_{k}}\frac{\left(\zeta_{k-1}^{(sph)}\right)^{\frac{{\bf
c}_{k}}{{\bf c}_{k-1}}}}{\zeta_{k}^{(sph)}}\left(\cdot\right).$ (70)

Then

$\displaystyle\hat{{\bf p}}_{2}$ $\displaystyle=$
$\displaystyle\prod_{k=r}^{3}\Phi_{{\mathcal{U}}_{k}}^{(bin)}\left(\left(\Phi_{{\mathcal{U}}_{2}}^{(bin)}(\bar{{\bf
x}}_{1})\right)^{2}\right)$ $\displaystyle\hat{{\bf p}}_{3}$ $\displaystyle=$
$\displaystyle\prod_{k=r}^{4}\Phi_{{\mathcal{U}}_{k}}^{(bin)}\left(\left(\Phi_{{\mathcal{U}}_{3}}^{(bin)}\left(\Phi_{{\mathcal{U}}_{2}}^{(bin)}(\bar{{\bf
x}}_{1})\right)\right)^{2}\right)$ $\displaystyle\vdots$ $\displaystyle=$
$\displaystyle\vdots$ $\displaystyle\hat{{\bf p}}_{k_{1}}$ $\displaystyle=$
$\displaystyle\prod_{k=r}^{k_{1}+1}\Phi_{{\mathcal{U}}_{k}}^{(bin)}\left(\left(\prod_{k=k_{1}}^{2}\Phi_{{\mathcal{U}}_{k}}^{(bin)}(\bar{{\bf
x}}_{1})\right)^{2}\right)$ $\displaystyle\vdots$ $\displaystyle=$
$\displaystyle\vdots$ $\displaystyle\hat{{\bf p}}_{r}$ $\displaystyle=$
$\displaystyle\left(\prod_{k=r}^{2}\Phi_{{\mathcal{U}}_{k}}^{(bin)}(\bar{{\bf
x}}_{1})\right)^{2},$

and

$\displaystyle\hat{{\bf q}}_{2}$ $\displaystyle=$
$\displaystyle\alpha\prod_{k=r}^{3}\Phi_{{\mathcal{U}}_{k}}^{(bin)}\left(\left(\Phi_{{\mathcal{U}}_{2}}^{(bin)}(\bar{{\bf
y}}_{1})\right)^{2}\right)$ $\displaystyle\hat{{\bf q}}_{3}$ $\displaystyle=$
$\displaystyle\alpha\prod_{k=r}^{4}\Phi_{{\mathcal{U}}_{k}}^{(bin)}\left(\left(\Phi_{{\mathcal{U}}_{3}}^{(bin)}\left(\Phi_{{\mathcal{U}}_{2}}^{(bin)}(\bar{{\bf
y}}_{1})\right)\right)^{2}\right)$ $\displaystyle\vdots$ $\displaystyle=$
$\displaystyle\vdots$ $\displaystyle\hat{{\bf q}}_{k_{1}}$ $\displaystyle=$
$\displaystyle\alpha\prod_{k=r}^{k_{1}+1}\Phi_{{\mathcal{U}}_{k}}^{(bin)}\left(\left(\prod_{k=k_{1}}^{2}\Phi_{{\mathcal{U}}_{k}}^{(bin)}(\bar{{\bf
y}}_{1})\right)^{2}\right)$ $\displaystyle\vdots$ $\displaystyle=$
$\displaystyle\vdots$ $\displaystyle\hat{{\bf q}}_{r}$ $\displaystyle=$
$\displaystyle\alpha\left(\prod_{k=r}^{2}\Phi_{{\mathcal{U}}_{k}}^{(bin)}(\bar{{\bf
y}}_{1})\right)^{2},$ (72)

and the products are running in the _index decreasing order_.

In (3.1.1), the integrals containing $\hat{{\bf q}}$ are not easily solvable
in closed form. Having that in mind, one observes that the above relations,
(64), (65), and ultimately (3.1.2), offer an advantageous avenue for numerical
evaluations. On the other hand, as is clear from (3.1.1), the integrals
containing $\hat{{\bf p}}$ are already solved in closed form. It is therefore
numerically more advantageous to directly differentiate (3.1.1) over ${\bf p}$
than to rely on (66), (67), and ultimately (3.1.2). Nonetheless, we included
(66), (67), and (3.1.2) for the completeness. Finally, one of the key
properties of the above mechanism that is of incredible practical relevance
should not go unnoticed. Namely, in a rather remarkable fashion, the entire
lifting mechanism exhibits a rapid convergence. In particular, looking at the
values given in Table 1, one observes that the convergence of the ground state
energy on the the fourth decimal is astonishingly achieved already on the
third level of full lifting.

####  3.1.3 Modulo-${\bf m}$ sfl RDT

It is interesting to note that the above can be repeated if one relies on the
so-called modulo-$m$ sfl RDT frame of [84]. Instead of Theorem 2, we then
basically have the following theorem.

######  Theorem 3.

Assume the setup of Theorem 2 and instead of the complete, assume the
modulo-${\bf m}$ sfl RDT setup of [84]. Let

$\displaystyle-\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,1)$
$\displaystyle=-\frac{1}{2}\sum_{k=2}^{r+1}({\bf p}_{k-1}{\bf q}_{k-1}-{\bf
p}_{k}{\bf q}_{k}){\bf c}_{k}$ $\displaystyle\quad+\frac{1}{{\bf
c}_{r}}{\mathbb{E}}_{{\mathcal{U}}_{r+1}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sum_{k=2}^{r+1}\sqrt{{\bf q}_{k-1}-{\bf q}_{k}}{\bf
h}_{1}^{(k)}|}\right)^{\frac{{\bf c}_{3}}{{\bf c}_{2}}}\right)^{\frac{{\bf
c}_{4}}{{\bf c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}}\right)$
$\displaystyle\quad+\gamma_{sq}+\alpha\Bigg{(}\Bigg{.}-\sum_{k=2}^{r}\frac{1}{2{\bf
c}_{k}}\log\left(\frac{\Theta_{k}}{\Theta_{k-1}}\right)+\frac{{\bf
p}_{r}}{2\Theta_{r}}\Bigg{.}\Bigg{)}$ $\displaystyle\Theta_{1}$
$\displaystyle=2\gamma_{sq}$ $\displaystyle\Theta_{k}$
$\displaystyle=\Theta_{k-1}-{\bf c}_{k}({\bf p}_{k-1}-{\bf
p}_{k}),k\in\\{2,3,\dots,r\\}.$ (73)

Let $\hat{\gamma}_{sq}$ and the “non-fixed" parts of $\hat{{\bf p}}$ and
$\hat{{\bf q}}$ be the solutions of the following system of equations

$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,1)}{d{\bf p}}=0$ $\displaystyle\frac{d\bar{\psi}_{rd}({\bf
p},{\bf q},{\bf c},\gamma_{sq},1,1,1)}{d{\bf q}}=0$
$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,1)}{d\gamma_{sq}}=0.$ (74)

Then

$\displaystyle f_{sq,{\bf m}}^{+}(\infty)$ $\displaystyle\geq$
$\displaystyle\min_{{\bf c}}\left(-\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf
q}},{\bf c},\hat{\gamma}_{sq},1,1,1)\right).$ (75)

###### Proof.

Follows from the previous discussion, Theorem 1, Corollary 1, and the sfl RDT
machinery presented in [85, 84, 86]. ∎

We conducted the numerical evaluations using the modulo-${\bf m}$ results of
the above theorem for all $r$ from Table 1, i.e., for all $r\leq 3$. Exploring
a very wide range of ${\bf c}$ parameters, we have found that the inequality
in (75) is in fact tight, i.e., that $f_{sq}^{+,r}(\infty)=f_{sq,{\bf
m}}^{+,r}(\infty)$. This indicates that the _stationarity_ over ${\bf c}$ is
actually of the _minimization_ type.

###  3.2 Negative square root Hopfield model ($s=-1$)

Following what was done above when considering the positive vaiant of the
model, we start by connecting the ground state energy of the negative square
root Hopfield model, $f^{-}_{sq}$ given in (10), and the random primal,
$\psi_{rp}(\cdot)$, given in Corollary 1. To that end, we have

$\displaystyle-f_{sq}^{-}(\infty)$ $\displaystyle=$
$\displaystyle\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\min_{{\bf
x}\in{\mathcal{X}}}\sqrt{{\bf x}^{T}G^{T}G{\bf
x}}}{\sqrt{n}}=-\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}-\sqrt{{\bf x}^{T}G^{T}G{\bf x}}}{\sqrt{n}}$ (76)
$\displaystyle=$
$\displaystyle-\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}-\max_{{\bf y}\in{\mathcal{Y}}}{\bf y}^{T}G{\bf
x}}{\sqrt{n}}=\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\psi_{rp}}{\sqrt{n}}=\lim_{n\rightarrow\infty}\psi_{rd}(\hat{{\bf
p}},\hat{{\bf q}},\hat{{\bf c}},1,1,-1),$

where the non-fixed parts of $\hat{{\bf p}}$, $\hat{{\bf q}}$, and $\hat{{\bf
c}}$ are the solutions of the following system

$\displaystyle\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},1,1,-1)}{d{\bf
p}}=0,\quad\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},1,1,-1)}{d{\bf
q}}=0,\quad\frac{d\psi_{rd}({\bf p},{\bf q},{\bf c},1,1,-1)}{d{\bf c}}=0.$
(77)

Moreover, relying on (3)-(30), we have

$\displaystyle\lim_{n\rightarrow\infty}\psi_{rd}(\hat{{\bf p}},\hat{{\bf
q}},\hat{{\bf c}},1,1,-1)=\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf
q}},\hat{{\bf c}},\hat{\gamma}_{sq},1,1,-1),$ (78)

where

$\displaystyle\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,-1)$
$\displaystyle=$ $\displaystyle\frac{1}{2}\sum_{k=2}^{r+1}\Bigg{(}\Bigg{.}{\bf
p}_{k-1}{\bf q}_{k-1}-{\bf p}_{k}{\bf q}_{k}\Bigg{.}\Bigg{)}{\bf c}_{k}$ (79)
$\displaystyle-\varphi(D_{1}^{(bin)}(c_{k}({\bf p},{\bf q})),{\bf
c})+\gamma_{sq}-\alpha\varphi(-D_{1}^{(sph)}(b_{k}({\bf p},{\bf q})),{\bf
c}).$

Connecting (76), (78), and (79), we then find

$\displaystyle-f_{sq}^{-}(\infty)$ $\displaystyle=$
$\displaystyle-\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}-\sqrt{{\bf x}^{T}G^{T}G{\bf
x}}}{\sqrt{n}}=-\lim_{n\rightarrow\infty}\frac{{\mathbb{E}}_{G}\max_{{\bf
x}\in{\mathcal{X}}}-\max_{{\bf y}\in{\mathcal{Y}}}{\bf y}^{T}G{\bf
x}}{\sqrt{n}}$ $\displaystyle=$
$\displaystyle\lim_{n\rightarrow\infty}\psi_{rd}(\hat{{\bf p}},\hat{{\bf
q}},\hat{{\bf c}},1,1,-1)=\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf
q}},\hat{{\bf c}},\hat{\gamma}_{sq},1,1,-1)$ $\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{k=2}^{r+1}\Bigg{(}\Bigg{.}\hat{{\bf
p}}_{k-1}\hat{{\bf q}}_{k-1}-\hat{{\bf p}}_{k}\hat{{\bf
q}}_{k}\Bigg{.}\Bigg{)}\hat{{\bf c}}_{k}-\varphi(D_{1}^{(bin)}(c_{k}(\hat{{\bf
p}},\hat{{\bf q}})),{\bf
c})+\hat{\gamma}_{sq}-\alpha\varphi(-D_{1}^{(sph)}(b_{k}(\hat{{\bf
p}},\hat{{\bf q}})),{\bf c}).$

The following theorem summarizes the above results.

######  Theorem 4.

Assume the complete sfl RDT setup of [84]. Consider large $n$ linear regime
with $\alpha=\lim_{n\rightarrow\infty}\frac{m}{n}$ and let $\varphi(\cdot)$
and $\bar{\psi}(\cdot)$ be as given in (3) and (34), respectively. Also, let
the “fixed” parts of $\hat{{\bf p}}$, $\hat{{\bf q}}$, and $\hat{{\bf c}}$
satisfy $\hat{{\bf p}}_{1}\rightarrow 1$, $\hat{{\bf q}}_{1}\rightarrow 1$,
$\hat{{\bf c}}_{1}\rightarrow 1$, $\hat{{\bf p}}_{r+1}=\hat{{\bf
q}}_{r+1}=\hat{{\bf c}}_{r+1}=0$, and let the “non-fixed” parts of $\hat{{\bf
p}}_{k}$, $\hat{{\bf q}}_{k}$, and $\hat{{\bf c}}_{k}$
($k\in\\{2,3,\dots,r\\}$) be the solutions of the following system of
equations

$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,-1)}{d{\bf p}}=0$ $\displaystyle\frac{d\bar{\psi}_{rd}({\bf
p},{\bf q},{\bf c},\gamma_{sq},1,1,-1)}{d{\bf q}}=0$
$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,-1)}{d{\bf c}}=0$ $\displaystyle\frac{d\bar{\psi}_{rd}({\bf
p},{\bf q},{\bf c},\gamma_{sq},1,1,-1)}{d\gamma_{sq}}=0,$ (81)

Moreover, let $c_{k}(\hat{{\bf p}},\hat{{\bf q}})$ and $b_{k}(\hat{{\bf
p}},\hat{{\bf q}})$ be as in (3.1). Then

$\displaystyle-f_{sq}^{-}(\infty)$ $\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{k=2}^{r+1}\Bigg{(}\Bigg{.}\hat{{\bf
p}}_{k-1}\hat{{\bf q}}_{k-1}-\hat{{\bf p}}_{k}\hat{{\bf
q}}_{k}\Bigg{.}\Bigg{)}\hat{{\bf c}}_{k}-\varphi(D_{1}^{(bin)}(c_{k}(\hat{{\bf
p}},\hat{{\bf q}})),\hat{{\bf
c}})+\hat{\gamma}_{sq}-\alpha\varphi(-D_{1}^{(sph)}(b_{k}(\hat{{\bf
p}},\hat{{\bf q}})),\hat{{\bf c}}).$

###### Proof.

Analogously to Theorem 2, follows from the previous discussion, Theorem 1,
Corollary 1, and the sfl RDT machinery presented in [85, 84, 86]. ∎

####  3.2.1 Numerical evaluations

The same way results of Theorem 2 were utilized in the previous section for
the numerical evaluations regarding the positive model, we, here, utilize
results of Theorem 4 to conduct the corresponding numerical evaluations for
the negative model. To ease the exposition, we parallel the positive model
evaluations as much as possible. Several analytical results that were obtained
earlier for the positive model have their negative model counterparts and we
state them below as well. As earlier, we start with $r=1$ and proceed
inductively while, on occasion, specializing the considerations to the square
case ($\alpha=1$) to enable the numerical values concreteness.

1) _$r=1$ – first level of lifting:_ For $r=1$ one has that $\hat{{\bf
p}}_{1}\rightarrow 1$ and $\hat{{\bf q}}_{1}\rightarrow 1$ which together with
$\hat{{\bf p}}_{r+1}=\hat{{\bf p}}_{2}=\hat{{\bf q}}_{r+1}=\hat{{\bf
q}}_{2}=0$, and $\hat{{\bf c}}_{2}\rightarrow 0$ gives

$\displaystyle\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf q}},\hat{{\bf
c}},\gamma_{sq},1,1,-1)$ $\displaystyle=\frac{1}{2}{\bf c}_{2}-\frac{1}{{\bf
c}_{2}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf c}_{2}|\sqrt{1-0}{\bf
h}_{1}^{(2)}|}\right)+\gamma_{sq}-\alpha\frac{1}{{\bf
c}_{2}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{-{\bf
c}_{2}\frac{(\sqrt{1-0}{\bf u}_{1}^{(2,2)})^{2}}{4\gamma_{sq}}}\right)$
$\displaystyle\rightarrow-\frac{1}{{\bf
c}_{2}}\log\left(1+{\mathbb{E}}_{{\mathcal{U}}_{2}}{\bf c}_{2}|\sqrt{1-0}{\bf
h}_{1}^{(2)}|\right)+\gamma_{sq}-\alpha\frac{1}{{\bf
c}_{2}}\log\left(1-{\mathbb{E}}_{{\mathcal{U}}_{2}}{\bf
c}_{2}\frac{(\sqrt{1-0}{\bf u}_{1}^{(2,2)})^{2}}{4\gamma_{sq}}\right)$
$\displaystyle\rightarrow-\frac{1}{{\bf c}_{2}}\log\left(1+{\bf
c}_{2}\sqrt{\frac{2}{\pi}}\right)+\gamma_{sq}-\alpha\frac{1}{{\bf
c}_{2}}\log\left(1-\frac{{\bf c}_{2}}{4\gamma_{sq}}\right)$
$\displaystyle\rightarrow-\sqrt{\frac{2}{\pi}}+\gamma_{sq}+\frac{\alpha}{4\gamma_{sq}}.$
(83)

One then easily finds $\hat{\gamma}_{sq}=\frac{\sqrt{\alpha}}{2}$ and

$\displaystyle-f^{-,1}_{sq}(\infty)=\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf
q}},\hat{{\bf c}},\hat{\gamma}_{sq},1,1,-1)$
$\displaystyle=-\sqrt{\frac{2}{\pi}}+\sqrt{\alpha},$ (84)

which for $\alpha=1$ becomes

$\displaystyle-f^{-,1}_{sq}(\infty)=-\sqrt{\frac{2}{\pi}}+1\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{0.2021}}.$
(85)

2) _$r=2$ – second level of lifting:_ Paralleling again the presentation of
the positive model, we split the second level of lifting into two subparts:
(i) _partial_ second level of lifting; and (ii) _full_ second level of
lifting.

For the _partial_ part, we have, when $r=2$, that $\hat{{\bf
p}}_{1}\rightarrow 1$ and $\hat{{\bf q}}_{1}\rightarrow 1$, $\hat{{\bf
p}}_{2}=\hat{{\bf q}}_{2}=0$, and $\hat{{\bf p}}_{r+1}=\hat{{\bf
p}}_{3}=\hat{{\bf q}}_{r+1}=\hat{{\bf q}}_{3}=0$ but in general $\hat{{\bf
c}}_{2}\neq 0$. As above, one again has

$\displaystyle\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf q}},{\bf
c},\gamma_{sq},1,1,-1)$ $\displaystyle=\frac{1}{2}{\bf c}_{2}-\frac{1}{{\bf
c}_{2}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf c}_{2}|\sqrt{1-0}{\bf
h}_{1}^{(2)}|}\right)+\gamma_{sq}-\alpha\frac{1}{{\bf
c}_{2}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{-{\bf
c}_{2}\frac{(\sqrt{1-0}{\bf u}_{1}^{(2,2)})^{2}}{4\gamma_{sq}}}\right).$ (86)

After taking the derivatives with respect to ${\bf c}_{2}$ and $\gamma_{sq}$,
one, for $\alpha=1$, finds $\hat{\gamma}_{sq}=0.1654$ and $\hat{{\bf
c}}_{2}=2.6916$, which gives

$\displaystyle(\mbox{\emph{partial} second
level:})\qquad\qquad-f^{-,2}_{sq}(\infty)\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{0.3202}}.$
(87)

For the _full_ part, the setup is basically the same as above with the sole
exception that now, in general, (in addition to $\hat{{\bf c}}_{2}\neq 0$) one
also has ${\bf p}_{2}\neq 0$ and ${\bf q}_{2}\neq 0$. Writing as above, we
then find

$\displaystyle\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,-1)$
$\displaystyle=\frac{1}{2}(1-{\bf p}_{2}{\bf q}_{2}){\bf c}_{2}-\frac{1}{{\bf
c}_{2}}{\mathbb{E}}_{{\mathcal{U}}_{3}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sqrt{1-{\bf q}_{2}}{\bf h}_{1}^{(2)}+\sqrt{{\bf q}_{2}}{\bf
h}_{1}^{(3)}|}\right)$ $\displaystyle\quad+\gamma_{sq}-\alpha\frac{1}{{\bf
c}_{2}}{\mathbb{E}}_{{\mathcal{U}}_{3}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{-{\bf
c}_{2}\frac{(\sqrt{1-{\bf p}_{2}}{\bf u}_{i}^{(2,2)}+\sqrt{{\bf p}_{2}}{\bf
u}_{i}^{(2,3)})^{2}}{4\gamma_{sq}}}\right)$ $\displaystyle=\frac{1}{2}(1-{\bf
p}_{2}{\bf q}_{2}){\bf c}_{2}-\frac{1}{{\bf
c}_{2}}{\mathbb{E}}_{{\mathcal{U}}_{3}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sqrt{1-{\bf q}_{2}}{\bf h}_{1}^{(2)}+\sqrt{{\bf q}_{2}}{\bf
h}_{1}^{(3)}|}\right)$
$\displaystyle\quad-(-\gamma_{sq})-\alpha\left(-\frac{1}{2{\bf
c}_{2}}\log\left(\frac{-2\gamma_{sq}-{\bf c}_{2}(1-{\bf
p}_{2})}{-2\gamma_{sq}}\right)+\frac{{\bf p}_{2}}{2(-2\gamma_{sq}-{\bf
c}_{2}(1-{\bf p}_{2}))}\right)$ $\displaystyle=-\bar{\psi}_{rd}({\bf p},{\bf
q},{\bf c},-\gamma_{sq},1,1,1).$ (88)

After taking the derivatives one, for $\alpha=1$, finds
$\hat{\gamma}_{sq}=0.1654$, ${\bf p}_{2}=0$, ${\bf q}_{2}=0$, and $\hat{{\bf
c}}_{2}=2.6916$, which gives

$\displaystyle(\mbox{\emph{full} second
level:})\qquad\qquad-f^{-,2}_{sq}(\infty)\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{0.3202}}.$
(89)

In other words, we observe a remarkable _indistinguishability between partial
and full lifting_ phenomenon.

3) _$r=3$ – third level of lifting:_ As for the positive model, we immediately
consider _full_ third level of lifting and, for $r=3$, have that $\hat{{\bf
p}}_{1}\rightarrow 1$ and $\hat{{\bf q}}_{1}\rightarrow 1$ as well as
$\hat{{\bf p}}_{r+1}=\hat{{\bf p}}_{4}=\hat{{\bf q}}_{r+1}=\hat{{\bf
q}}_{4}=0$. As above, one again has

$\displaystyle\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,-1)$
$\displaystyle=\frac{1}{2}(1-{\bf p}_{2}{\bf q}_{2}){\bf
c}_{2}+\frac{1}{2}({\bf p}_{2}{\bf q}_{2}-{\bf p}_{3}{\bf q}_{3}){\bf c}_{3}$
$\displaystyle\quad-\frac{1}{{\bf
c}_{3}}{\mathbb{E}}_{{\mathcal{U}}_{4}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sqrt{1-{\bf q}_{2}}{\bf h}_{1}^{(2)}+\sqrt{{\bf q}_{2}-{\bf
q}_{3}}{\bf h}_{1}^{(3)}+\sqrt{{\bf q}_{3}}{\bf
h}_{1}^{(4)}|}\right)^{\frac{{\bf c}_{3}}{{\bf c}_{2}}}\right)$
$\displaystyle\quad+\gamma_{sq}-\alpha\frac{1}{{\bf
c}_{3}}{\mathbb{E}}_{{\mathcal{U}}_{4}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{4}}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{-{\bf
c}_{2}\frac{(\sqrt{1-{\bf p}_{2}}{\bf u}_{i}^{(2,2)}+\sqrt{{\bf p}_{2}-{\bf
p}_{3}}{\bf u}_{i}^{(2,3)}+\sqrt{{\bf p}_{3}}{\bf
u}_{i}^{(2,4)})^{2}}{4\gamma_{sq}}}\right)^{\frac{{\bf c}_{3}}{{\bf
c}_{2}}}\right)$ $\displaystyle=\frac{1}{2}(1-{\bf p}_{2}{\bf q}_{2}){\bf
c}_{2}+\frac{1}{2}({\bf p}_{2}{\bf q}_{2}-{\bf p}_{3}{\bf q}_{3}){\bf c}_{3}$
$\displaystyle\quad-\frac{1}{{\bf
c}_{3}}{\mathbb{E}}_{{\mathcal{U}}_{4}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sqrt{1-{\bf q}_{2}}{\bf h}_{1}^{(2)}+\sqrt{{\bf q}_{2}-{\bf
q}_{3}}{\bf h}_{1}^{(3)}+\sqrt{{\bf q}_{3}}{\bf
h}_{1}^{(4)}|}\right)^{\frac{{\bf c}_{3}}{{\bf c}_{2}}}\right)$
$\displaystyle\quad-(-\gamma_{sq})-\alpha\Bigg{(}\Bigg{.}-\frac{1}{2{\bf
c}_{2}}\log\left(\frac{-2\gamma_{sq}-{\bf c}_{2}(1-{\bf
p}_{2})}{-2\gamma_{sq}}\right)$ $\displaystyle\quad-\frac{1}{2{\bf
c}_{3}}\log\left(\frac{-2\gamma_{sq}-{\bf c}_{2}(1-{\bf p}_{2})-{\bf
c}_{3}({\bf p}_{2}-{\bf p}_{3})}{-2\gamma_{sq}-{\bf c}_{2}(1-{\bf
p}_{2})}\right)$ $\displaystyle\quad+\frac{{\bf p}_{3}}{2(-2\gamma_{sq}-{\bf
c}_{2}(1-{\bf p}_{2})-{\bf c}_{3}({\bf p}_{2}-{\bf p}_{3}))}\Bigg{.}\Bigg{)}$
$\displaystyle=-\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},-\gamma_{sq},1,1,1).$
(90)

Taking the derivatives we obtain (for $\alpha=1$) $\hat{\gamma}_{sq}=0.1748$,
$\hat{{\bf p}}_{3}=\hat{{\bf q}}_{3}=0$, $\hat{{\bf p}}_{2}=0.5722$,
$\hat{q1}_{2}=0.0599$, $\hat{{\bf c}}_{2}=10.54$, and $\hat{{\bf
c}}_{1}=2.2264$ and

$\displaystyle(\mbox{\emph{partial/full} third
level:})\qquad\qquad-f^{-,3}_{sq}(\infty)\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{0.3272}}.$
(91)

We again observe the remarkable _indistinguishability between partial and full
lifting_ phenomenon.

4) General _$r$ -th level of lifting:_ For the completeness, we state the form
of $\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,1)$ obtained for
general $r$. Following the above procedures, we basically have

$\displaystyle\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,-1)$
$\displaystyle=\frac{1}{2}\sum_{k=2}^{r+1}({\bf p}_{k-1}{\bf q}_{k-1}-{\bf
p}_{k}{\bf q}_{k}){\bf c}_{k}$ $\displaystyle\quad-\frac{1}{{\bf
c}_{r}}{\mathbb{E}}_{{\mathcal{U}}_{r+1}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sum_{k=2}^{r+1}\sqrt{{\bf q}_{k-1}-{\bf q}_{k}}{\bf
h}_{1}^{(k)}|}\right)^{\frac{{\bf c}_{3}}{{\bf c}_{2}}}\right)^{\frac{{\bf
c}_{4}}{{\bf c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}}\right)$
$\displaystyle\quad-(-\gamma_{sq})-\alpha\Bigg{(}\Bigg{.}-\sum_{k=2}^{r}\frac{1}{2{\bf
c}_{k}}\log\left(\frac{\Theta_{k}}{\Theta_{k-1}}\right)+\frac{{\bf
p}_{r}}{2\Theta_{r}}\Bigg{.}\Bigg{)}$ $\displaystyle=-\bar{\psi}_{rd}({\bf
p},{\bf q},{\bf c},-\gamma_{sq},1,1,1),$ (92)

where

$\displaystyle\Theta_{1}$ $\displaystyle=$ $\displaystyle-2\gamma_{sq}$
$\displaystyle\Theta_{k}$ $\displaystyle=$ $\displaystyle\Theta_{k-1}-{\bf
c}_{k}({\bf p}_{k-1}-{\bf p}_{k}),k\in\\{2,3,\dots,r\\}.$ (93)

The above characterization of $\bar{\psi}(\cdot)$ gives explicit dependence on
all key parameters ${\bf p}$, ${\bf q}$, ${\bf m}$, and $\gamma_{sq}$ and
consequently enables direct computations of all needed derivatives for any
$r\in{\mathbb{N}}$. In addition to the above presented numerical evaluations
for the first three levels, we also conducted the numerical evaluations for
the fourth (partial) level $r=4$ and obtained

$\displaystyle(\mbox{\emph{partial} fourth
level:})\qquad\qquad-f^{-,4}_{sq}(\infty)\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbf{0.3279}}.$
(94)

All the relevant quantities for the fourth (partial) level of lifting,
together with the first, second, and third level corresponding ones are
systematically shown in Table 2. Observing the results from Table 2 more
closely, we once again note the rapid convergence of the entire lifting
mechanism with the fourth decimal convergence of the ground state energy
already achieved on the fourth level of lifting.

Table 2: $r$-sfl RDT parameters; negative _square-root_ Hopfield model; $\alpha=1$; $\hat{{\bf p}}_{1},\hat{{\bf q}}_{1},\hat{{\bf c}}_{1}\rightarrow 1$; $n,\beta\rightarrow\infty$ $r$-sfl RDT | $\hat{\gamma}_{sq}$ | $\hat{{\bf p}}_{3}$ | $\hat{{\bf p}}_{2}$ | $\hat{{\bf q}}_{3}$ | $\hat{{\bf q}}_{2}$ | $\hat{{\bf c}}_{4}$ | $\hat{{\bf c}}_{3}$ | $\hat{{\bf c}}_{2}$ | $-f_{sq}^{-,r}$
---|---|---|---|---|---|---|---|---|---
$\mathbf{1}$ (partial/full) | $0.5$ | $0$ | $0$ | $0$ | $0$ | $\rightarrow 0$ | $\rightarrow 0$ | $\rightarrow 0$ | $\mathbf{0.2021}$
$\mathbf{2}$ (partial/full) | $0.1654$ | $0$ | $0$ | $0$ | $0$ | $\rightarrow 0$ | $\rightarrow 0$ | $2.6916$ | $\mathbf{0.3202}$
$\mathbf{3}$ (partial/full) | $0.1748$ | $0$ | $0.5722$ | $0$ | $0.0599$ | $\rightarrow 0$ | $10.54$ | $2.2264$ | $\mathbf{0.3272}$
$\mathbf{4}$ (partial) | $0.1766$ | $.3639$ | $0.6836$ | $0.0107$ | $0.1282$ | $27.98$ | $5.07$ | $2.1306$ | $\mathbf{0.3279}$

We also add that all considerations regarding more efficient numerical
evaluations between (3.1.2)-(67) are trivial to repeat. The only thing that
needs to be adjusted is $s$. Instead of $s=1$ now $s=-1$. The final results
are exactly as those stated in (64)-(67). The only tiny difference is that now
$\hat{\gamma}_{sq}$ from Table 2 should be taken with the negative sign in
these equations.

The above mentioned _indistinguishability between partial and full lifting_
phenomenon clearly visible in Table 2 disappears as $\alpha$ increases. For
the completeness, we provide in Table 3 a scenario ($\alpha=3.5$) where one
observes the behavior that fully resembles the one seen in the positive model
with a clear distinction between partial and full lifting on both second and
third level. The convergence on the fourth decimal is achieved now on the
third level which also parallels the behavior of the positive models.

Table 3: $r$-sfl RDT parameters; negative _square-root_ Hopfield model; $\alpha=3.5$; $\hat{{\bf c}}_{1}\rightarrow 1$; $n,\beta\rightarrow\infty$ $r$-sfl RDT | $\hat{\gamma}_{sq}$ | $\hat{{\bf p}}_{3}$ | $\hat{{\bf p}}_{2}$ | $\hat{{\bf p}}_{1}`$ | $\hat{{\bf q}}_{3}$ | $\hat{{\bf q}}_{2}$ | $\hat{{\bf q}}_{1}$ | $\hat{{\bf c}}_{3}$ | $\hat{{\bf c}}_{2}$ | $f_{sq}^{-,r}$
---|---|---|---|---|---|---|---|---|---|---
$\mathbf{1}$ (full) | $0.9354$ | $0$ | $0$ | $\rightarrow 1$ | $0$ | $0$ | $\rightarrow 1$ | $\rightarrow 0$ | $\rightarrow 0$ | $\mathbf{1.0729}$
$\mathbf{2}$ (partial) | $0.6366$ | $0$ | $0$ | $\rightarrow 1$ | $0$ | $0$ | $\rightarrow 1$ | $\rightarrow 0$ | $1.4759$ | $\mathbf{1.1288}$
$\mathbf{2}$ (full) | $0.6234$ | $0$ | $0.3416$ | $\rightarrow 1$ | $0$ | $0.2154$ | $\rightarrow 1$ | $\rightarrow 0$ | $1.6843$ | $\mathbf{1.1301}$
$\mathbf{3}$ (partial) | $0.6030$ | $0$ | $0.7921$ | $\rightarrow 1$ | $0$ | $0.6399$ | $\rightarrow 1$ | $1.15$ | $2.26$ | $\mathbf{1.1309}$
$\mathbf{3}$ (full) | $0.5940$ | $0.2464$ | $0.8935$ | $\rightarrow 1$ | $0.1508$ | $0.7887$ | $\rightarrow 1$ | $1.40$ | $2.79$ | $\mathbf{1.1312}$

####  3.2.2 Modulo-${\bf m}$ sfl RDT

As in the case of the positive model, one can rely on the modulo-$m$ sfl RDT
frame of [84] for the negative model as well. Instead of Theorem 4, we then
have the following theorem.

######  Theorem 5.

Assume the setup of Theorem 4 and instead of the complete, assume the
modulo-${\bf m}$ sfl RDT setup of [84]. Let

$\displaystyle\bar{\psi}_{rd}({\bf p},{\bf q},{\bf c},\gamma_{sq},1,1,-1)$
$\displaystyle=\frac{1}{2}\sum_{k=2}^{r+1}({\bf p}_{k-1}{\bf q}_{k-1}-{\bf
p}_{k}{\bf q}_{k}){\bf c}_{k}$ $\displaystyle\quad-\frac{1}{{\bf
c}_{r}}{\mathbb{E}}_{{\mathcal{U}}_{r+1}}\log\left({\mathbb{E}}_{{\mathcal{U}}_{r}}\left(\dots\left({\mathbb{E}}_{{\mathcal{U}}_{3}}\left({\mathbb{E}}_{{\mathcal{U}}_{2}}e^{{\bf
c}_{2}|\sum_{k=2}^{r+1}\sqrt{{\bf q}_{k-1}-{\bf q}_{k}}{\bf
h}_{1}^{(k)}|}\right)^{\frac{{\bf c}_{3}}{{\bf c}_{2}}}\right)^{\frac{{\bf
c}_{4}}{{\bf c}_{3}}}\dots\right)^{\frac{{\bf c}_{r}}{{\bf c}_{r-1}}}\right)$
$\displaystyle\quad-(-\gamma_{sq})-\alpha\Bigg{(}\Bigg{.}-\sum_{k=2}^{r}\frac{1}{2{\bf
c}_{k}}\log\left(\frac{\Theta_{k}}{\Theta_{k-1}}\right)+\frac{{\bf
p}_{r}}{2\Theta_{r}}\Bigg{.}\Bigg{)}$ $\displaystyle=-\bar{\psi}_{rd}({\bf
p},{\bf q},{\bf c},-\gamma_{sq},1,1,1)$ $\displaystyle\Theta_{1}$
$\displaystyle=-2\gamma_{sq}$ $\displaystyle\Theta_{k}$
$\displaystyle=\Theta_{k-1}-{\bf c}_{k}({\bf p}_{k-1}-{\bf
p}_{k}),k\in\\{2,3,\dots,r\\}.$ (95)

Let $\hat{\gamma}_{sq}$ the “non-fixed" parts of $\hat{{\bf p}}$ and
$\hat{{\bf q}}$ be the solutions of the following systems of equations

$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,-1)}{d{\bf p}}$ $\displaystyle=-\frac{d\bar{\psi}_{rd}({\bf
p},{\bf q},{\bf c},-\gamma_{sq},1,1,1)}{d{\bf p}}=\frac{d\bar{\psi}_{rd}({\bf
p},{\bf q},{\bf c},\gamma_{sq},1,1,1)}{d{\bf p}}=0$
$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,-1)}{d{\bf q}}$ $\displaystyle=-\frac{d\bar{\psi}_{rd}({\bf
p},{\bf q},{\bf c},-\gamma_{sq},1,1,1)}{d{\bf q}}=\frac{d\bar{\psi}_{rd}({\bf
p},{\bf q},{\bf c},\gamma_{sq},1,1,1)}{d{\bf q}}=0$
$\displaystyle\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},\gamma_{sq},1,1,-1)}{d\gamma_{sq}}$
$\displaystyle=-\frac{d\bar{\psi}_{rd}({\bf p},{\bf q},{\bf
c},-\gamma_{sq},1,1,1)}{d\gamma_{sq}}=\frac{d\bar{\psi}_{rd}({\bf p},{\bf
q},{\bf c},\gamma_{sq},1,1,1)}{d\gamma_{sq}}=0.$ (96)

Then

$\displaystyle-f_{sq,{\bf m}}^{-}(\infty)$ $\displaystyle\leq$
$\displaystyle\max_{{\bf c}}\left(\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf
q}},{\bf c},\hat{\gamma}_{sq},1,1,-1)\right)=\min_{{\bf
c}}\left(-\bar{\psi}_{rd}(\hat{{\bf p}},\hat{{\bf q}},{\bf
c},-\hat{\gamma}_{sq},1,1,1)\right).$ (97)

###### Proof.

Follows from the previous discussion, Theorem 1, Corollary 1, and the sfl RDT
machinery presented in [85, 84, 86]. ∎

We again conducted the numerical evaluations (albeit on a smaller range of
${\bf c}$ parameters) using the modulo-${\bf m}$ results of the above theorem
and obtained $-f_{sq}^{-,r}(\infty)=-f_{sq,{\bf m}}^{-,r}(\infty)$.

##  4 Conclusion

The famous Hopfield models and their behavior in statistical mediums are the
main subject of study in this paper. While structurally somewhat similar to
the celebrated Sherrington-Kirkpatrick (SK) models, these models are a much
harder mathematical challenge. Early statistical physics considerations [23,
69, 7] traced a possible path for mathematically rigorous considerations.
However, despite the existence of such a, potentially helpful, tool, the
progress on the rigourous front has been pretty much stagnant over the last
several decades. Namely, the mathematically rigorous treatments, (see, e.g.,
[63, 65, 9, 87, 99]), typically relate to the so-called high-temperature ($T$)
regime (low inverse temperature $\beta=\frac{1}{T}$), where the so-called
replica symmetric behavior is in place. On the other hand, of a prevalent
interest in many mathematical areas (including optimization, algorithms,
machine learning and neural networks) are precisely their counterparts, i.e.,
the regimes where the symmetric behavior is not in place. Moreover, the
hardest of them all, the so-called ground state, zero temperature, regime is
typically the most sought after one.

We start by recognizing that, due to measure concentrations, one can, instead
of the standard Hopfield models and associated ground state energies, study
their so-called _square-root_ variants. We then distinguish two important
classes of the models: (i) the _positive_ ; and (ii) the _negative_ ones.
Utilizing the random duality theory (RDT) principles, [75] established the
upper bounds for the positive and the lower bounds for the negative models
which both turned out to precisely match the corresponding replica symmetry
based predictions. On the other hand, [78] then went a bit further, used a
_lifted_ RDT variant, and lowered the [75]’s bounds on the positive and lifted
the [75]’s bounds on the negative models, thereby showing that the replica
symmetry produces a strict (but _not_ tight) bounds and ensuring that the
symmetry _must_ be broken.

We, here, continue the line of study of [75, 78], where the Hopfield models
are connected to the behaviour of the bilinearly indexed (bli) random
processes. However, we, now, rely on a strong recent progress made in studying
bli processes in [85, 84] and on the main foundational principles of the
_stationarized fully lifted_ (sfl) RDT established in [86] based on that
progress. After recognizing that the statistical variants of the Hopfield
models are indeed directly related to the bli processes, we show that the sfl
RDT principles can be used to characterize their statistical behavior. As the
obtained analytical characterizations rely on a nested concept of lifting,
they are fully operational only if one can successfully conduct underlying
numerical evaluations. After conducting such evaluations for the ground state
energies of both positive and negative models, we observe an incredible level
of convergence of the employed sfl RDT mechanism. Namely, for the so-called
square case ($m=n$), the fourth decimal precision is achieved already on the
third (second non-trivial) level (3-sfl RDT) for the positive case and on the
fourth (third non-trivial) level (4-sfl RDT) for the corresponding negative
one. In particular, we obtain the scaled ground state free energy $\approx
1.7788$ for the positive and $\approx 0.3279$ for the negative model. We also
observe a remarkable structural difference in the lifting form of both models.
It turns out that the positive model exhibits the so-called _full_ lifting for
any of the evaluated levels. On the other hand, the negative model exhibits
exclusively the _partial_ lifting. In other words, for the negative models, we
uncover the existence of the “ _indistinguishability between the partial and
full lifting_ ” phenomenon. Even more remarkably, this phenomenon disappears
as $\alpha$ increases. While we uncover both the existence and the
disappearance of this phenomenon as completely mathematical objects, it will
be more than interesting to see if a statistical physics explanation can be
associated with it.

It is not that difficult to see that the introduced methodology easily allows
for various extensions and generalizations. For example, all those discussed
in [85, 84, 86], including those related to the asymmetric Little models (see,
e.g., [15, 46, 10, 16, 6, 74]), various _random feasibility problems_ (rfps)
such as _spherical_ perceptrons (see, e.g., [31, 29, 30, 32, 33, 2, 73, 81,
79, 36, 35, 64, 22, 52, 98, 97, 17, 41, 8, 93, 66, 67]), _binary_ perceptrons
(see, e.g., [73, 36, 35, 76, 43, 38, 42]), the _sectional_ compressed sensing
$\ell_{1}$ phase transitions (see, e.g., [72, 77]), and many others are
immediately possible. Similarly to the problems that we considered here, such
extensions require a bit of technical juggling that is problem specific and we
discuss them, together with all underlying intricacies, in separate papers.

As already mentioned in [86], the sfl RDT considerations, and therefore the
ones presented here, do not require the standard Gaussianity assumption of the
random primals. Relying on the central limit theorem and, in particular, its
Lindeberg variant [44], the obtained results can quickly be extended to a wide
range of different statistics. Results of [18] are a particularly elegant
approach that enables such a quick extension.

## References

  * [1] D. Achlioptas and Y. Peres.  The threshold for random k-SAT is $2klnk-{O}(k)$.  Journal of the AMS, 17:947–973, 2004.
  * [2] A. E. Alaoui and M. Sellke.  Algorithmic pure states for the negative spherical perceptron.  2020\.  available online at http://arxiv.org/abs/2010.15811.
  * [3] D. J. Aldous.  Asymptotics in the random assignment problem.  Probab Theory Related Fields, 93:507–534, 1992.
  * [4] D. J. Aldous.  The zeta(2) limit in the random assignment problem.  Random Structures Algorithms, 18:381–418, 2001.
  * [5] S. E. Alm and G. B. Sorkin.  Exact expectations and distributions for the random assignment problem.  Combinat. Probab. Comput., 11(3):217–248, 2002.
  * [6] D. J. Amit, H. Gutfreund, and H. Sompolinsky.  Phys. Rev. A, 32:1007, 1985.
  * [7] D. J. Amit, H. Gutfreund, and H. Sompolinsky.  Storing infinite number of patterns in a spin glass model of neural networks.  Phys. Rev. Letters, 55:1530, 1987.
  * [8] P. Baldi and S. Venkatesh.  Number od stable points for spin-glasses and neural networks of higher orders.  Phys. Rev. Letters, 58(9):913–916, Mar. 1987.
  * [9] A. Barra, G. Genovese, and F. Guerra.  The replica symmetric approximation of the analogical neural network.  J. Stat. Physics, July 2010.
  * [10] A. Barra, G. Genovese, and F. Guerra.  Equlibrium statistical mechanics of bipartite spin systems.  Journal of Physics A: Mathematical and Theoeretical, 44(245002), 2011.
  * [11] A. Barra, G. Genovese, F. Guerra, and D. Tantari.  How glassy are neural networks.  J. Stat. Mechanics: Thery and Experiment, July 2012.
  * [12] M. Bayati and A. Montanari.  The lasso risk of gaussian matrices.  2010\.  available online at http://arxiv.org/abs/1008.2581.
  * [13] M. Bayati and A. Montanari.  The dynamics of message passing on dense graphs, with applications to compressed sensing.  IEEE Transactions on Information Theory, 57(2), 2011.
  * [14] A. Bovier and V. Gayrard.  Hopfield models as generalized random mean field models.  In mathematical aspects of spin glasses and neural networks, Progr. Prob., 41:3–89, 1998.
  * [15] R. Brunetti, G. Parisi, and F. Ritort.  Asymmetric Little spin glas model.  Physical Review B, 46(9), September 1992.
  * [16] S. Cabasino, E. Marinari, P. Paolucci, and G. Parisi.  Eigenstates and limit cycles in the SK model.  Journal of Physics A: Mathematical and General, 21:4201, 1988.
  * [17] S. H. Cameron.  Tech-report 60-600.  Proceedings of the bionics symposium, pages 197–212, 1960.  Wright air development division, Dayton, Ohio.
  * [18] S. Chatterjee.  A generalization of the Lindenberg principle.  The Annals of Probability, 34(6):2061–2076.
  * [19] A. Coja-Oghlan.  The asymptotic k-SAT threshold.  Proceedings of the forty-fifth annual ACM symposium on theory of computing (STOC), pages 804–813, 2014.
  * [20] D. Coppersmith and G. Sorkin.  Constructive bounds and exact expectations for the random assignment problem.  Random Structures Algorithms, 15:113–144, 1999.
  * [21] D. Coppersmith and G. Sorkin.  On the expected incremental cost of a minimum assignment.  Contemporary combinatories, Bolyai Society Mathematical Studies, 10(8), 2002.  B. Bollobas (Editor), Springer, New York.
  * [22] T. Cover.  Geomretrical and statistical properties of systems of linear inequalities with applications in pattern recognition.  IEEE Transactions on Electronic Computers, (EC-14):326–334, 1965\.
  * [23] A. Crisamti, D. J. Amit, and H. Gutfreund.  Saturation level of the Hopfield model for neural network.  Europhys. Lett., (2):337, 1986.
  * [24] D. Dean and F. Ritort.  Squared interaction matrix Sherrington-Kirkpatrick model for a spin glass.  Phys. Rev. B, 65:224209, 2002.
  * [25] J. Ding, A. Sly, and N. Sun.  Satisfiability Threshold for Random Regular nae-sat.  Communications in Mathematical Physics, 341(2):435–489, 2015.
  * [26] J. Ding, A. Sly, and N. Sun.  Satisfiability threshold for random regular NAE-SAT.  Proceedings of the forty-sixth annual ACM symposium on theory of computing (STOC), pages 814–822, 2015.
  * [27] D. Donoho.  High-dimensional centrally symmetric polytopes with neighborlines proportional to dimension.  Disc. Comput. Geometry, 35(4):617–652, 2006.
  * [28] D. Donoho and J. Tanner.  Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing.  Preprint, 2009.  available at arXiv:0906.2530.
  * [29] S. Franz, S. Hwang, and P. Urbani.  Jamming in multilayer supervised learning models.  Phys. Rev. Lett., 123(16):160602, 2019.
  * [30] S. Franz and G. Parisi.  The simplest model of jamming.  Journal of Physics A: Mathematical and Theoretical, 49(14):145001, 2016.
  * [31] S. Franz, G. Parisi, M. Sevelev, P. Urbani, and F. Zamponi.  Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems.  SciPost Physics, 2:019, 2017.
  * [32] S. Franz, A. Sclocchi, and P. Urbani.  Critical jammed phase of the linear perceptron.  Phys. Rev. Lett., 123(11):115702, 2019.
  * [33] S. Franz, A. Sclocchi, and P. Urbani.  Surfing on minima of isostatic landscapes: avalanches and unjamming transition.  SciPost Physics, 9:12, 2020.
  * [34] A. Frieze and N. Wormald.  Random k-Sat: a tight threshold for moderately growing k.  Combinatorica, 28:297–305, 2005.
  * [35] E. Gardner.  The space of interactions in neural networks models.  J. Phys. A: Math. Gen., 21:257–270, 1988.
  * [36] E. Gardner and B. Derrida.  Optimal storage properties of neural networks models.  J. Phys. A: Math. Gen., 21:271–284, 1988.
  * [37] F. Guerra.  Broken replica symmetry bounds in the mean field spin glass model.  Comm. Math. Physics, 233:1–12, 2003.
  * [38] H. Gutfreund and Y. Stein.  Capacity of neural networks with discrete synaptic couplings.  J. Physics A: Math. Gen, 23:2613, 1990.
  * [39] D. O. Hebb.  Organization of behavior.  New York: Wiley, 1949.
  * [40] J. J. Hopfield.  Neural networks and physical systems with emergent collective computational abilities.  Proc. Nat. Acad. Science, 79:2554, 1982.
  * [41] R. D. Joseph.  The number of orthants in $n$-space instersected by an $s$-dimensional subspace.  Tech. memo 8, project PARA, 1960.  Cornel aeronautical lab., Buffalo, N.Y.
  * [42] J. H. Kim and J. R. Roche.  Covering cubes by random half cubes with applications to biniary neural networks.  Journal of Computer and System Sciences, 56:223–252, 1998.
  * [43] W. Krauth and M. Mezard.  Storage capacity of memory networks with binary couplings.  J. Phys. France, 50:3057–3066, 1989.
  * [44] J. W. Lindeberg.  Eine neue herleitung des exponentialgesetzes in der wahrscheinlichkeitsrechnung.  Math. Z., 15:211–225, 1922.
  * [45] S. Linusson and J. Wastlund.  A proof of Parisi’s conjecture on the random assignment problem.  Probabil Theory Related Fields, 128(3):419–440, 2004.
  * [46] W. A. Little.  The existence of persistent states in the brain.  Math. Biosci., 19(1-2):101–120, 1974.
  * [47] S. Mertens, M. Mezard, and R. Zecchina.  Threshold values of random K-SAT from the cavity method.  Random Struct. Alg., 28:340–373, 2006.
  * [48] M. Mezard and G. Parisi.  On the solution of the random link matching problem.  J Physique, 48:1451–1459, 1987.
  * [49] M. Mezard, G. Parisi, and R. Zecchina.  Analytic and algorithmic solution of random satisfiability problems.  Science, 297:812–815, 2002.
  * [50] M. Molloy.  The freezing threshold for k-colourings of a random graph.  Proceedings of the forty-third annual ACM symposium on theory of computing (STOC), pages 921–930, 2012.
  * [51] C. Nair, B. Prabhakar, and M. Sharma.  Proofs of the Parisi and Coppersmith-Sorkin random assignment conjectures.  Random Structures and Algorithms, 27(4):413–444, 2005.
  * [52] R. O.Winder.  Threshold logic.  Ph. D. dissertation, Princetoin University, 1962.
  * [53] D. Panchenko.  A connection between the Ghirlanda-Guerra identities and ultrametricity.  The Annals of Probability, 38(1):327–347, 2010.
  * [54] D. Panchenko.  The Ghirlanda-Guerra identities for mixed p-spin model.  Comptes Rendus Mathematique, 348(3-4):189–192, 2010.
  * [55] D. Panchenko.  The Parisi ultrametricity conjecture.  Ann. Math., 77(1):383–393, 2013.
  * [56] D. Panchenko.  The Sherrington-Kirkpatrick model.  Springer Science & Business Media, 2013.
  * [57] D. Panchenko.  On the replica symmetric solution of the l-sat model.  Electronic Journal of Probability, 19, 2014.
  * [58] G. Parisi.  Infnite number of order parameters for spin-glasses.  Phys. Rev. Lett., 43:1754–1756, 1979.
  * [59] G. Parisi.  Breaking the symmetry in SK model.  J. Physics, A13:1101, 1980.
  * [60] G. Parisi.  A sequence of approximated solutions to the SK model for spin glasses.  Journal of Physics A: Mathematical and General, 13(4):L115, 1980\.
  * [61] G. Parisi.  Order parameter for spin glasses.  Phys. Rev. Lett., 50:1946, 1983.
  * [62] L. Pastur and A. Figotin.  On the theory of disordered spin systems.  Theory Math. Phys., 35(403-414), 1978.
  * [63] L. Pastur, M. Shcherbina, and B. Tirozzi.  The replica-symmetric solution without the replica trick for the Hopfield model.  Journal of Statistical Physics, 74(5/6), 1994.
  * [64] L. Schlafli.  Gesammelte Mathematische AbhandLungen I.  Basel, Switzerland: Verlag Birkhauser, 1950.
  * [65] M. Shcherbina and B. Tirozzi.  The free energy of a class of Hopfield models.  Journal of Statistical Physics, 72(1/2), 1993.
  * [66] M. Shcherbina and B. Tirozzi.  On the volume of the intrersection of a sphere with random half spaces.  C. R. Acad. Sci. Paris. Ser I, (334):803–806, 2002.
  * [67] M. Shcherbina and B. Tirozzi.  Rigorous solution of the Gardner problem.  Comm. on Math. Physics, (234):383–422, 2003.
  * [68] D. Sherrington and S. Kirkpatrick.  Solvable model of a spin glass.  Phys. Rev. Letters, 35:1792–1796, 1972.
  * [69] H. Steffan and R. Kuhn.  Replica symmetry breaking in attractor neural network models.  Z. Phys. B, 95:249–260, 1994.
  * [70] M. Stojnic.  A framework for perfromance characterization of _LASSO_ algortihms.  available online at http://arxiv.org/abs/1303.7291.
  * [71] M. Stojnic.  Upper-bounding $\ell_{1}$-optimization weak thresholds.  available online at http://arxiv.org/abs/1303.7289.
  * [72] M. Stojnic.  Various thresholds for $\ell_{1}$-optimization in compressed sensing.  available online at http://arxiv.org/abs/0907.3666.
  * [73] M. Stojnic.  Another look at the Gardner problem.  2013\.  available online at http://arxiv.org/abs/1306.3979.
  * [74] M. Stojnic.  Asymmetric Little model and its ground state energies.  2013\.  available online at http://arxiv.org/abs/1306.3978.
  * [75] M. Stojnic.  Bounding ground state energy of Hopfield models.  2013\.  available online at http://arxiv.org/abs/1306.3764.
  * [76] M. Stojnic.  Discrete perceptrons.  2013\.  available online at http://arxiv.org/abs/1303.4375.
  * [77] M. Stojnic.  Lifting $\ell_{1}$-optimization strong and sectional thresholds.  2013\.  available online at http://arxiv.org/abs/1306.3770.
  * [78] M. Stojnic.  Lifting/lowering Hopfield models ground state energies.  2013\.  available online at http://arxiv.org/abs/1306.3975.
  * [79] M. Stojnic.  Negative spherical perceptron.  2013\.  available online at http://arxiv.org/abs/1306.3980.
  * [80] M. Stojnic.  Regularly random duality.  2013\.  available online at http://arxiv.org/abs/1303.7295.
  * [81] M. Stojnic.  Spherical perceptron as a storage memory with limited errors.  2013\.  available online at http://arxiv.org/abs/1306.3809.
  * [82] M. Stojnic.  Fully bilinear generic and lifted random processes comparisons.  2016\.  available online at http://arxiv.org/abs/1612.08516.
  * [83] M. Stojnic.  Generic and lifted probabilistic comparisons – max replaces minmax.  2016\.  available online at http://arxiv.org/abs/1612.08506.
  * [84] M. Stojnic.  Bilinearly indexed random processes – _stationarization_ of fully lifted interpolation.  2023\.  available online at arxiv.
  * [85] M. Stojnic.  Fully lifted interpolating comparisons of bilinearly indexed random processes.  2023\.  available online at arxiv.
  * [86] M. Stojnic.  Fully lifted random duality theory.  2023\.  available online at arxiv.
  * [87] M. Talagrand.  Rigorous results for the Hopfield model with many patterns.  Prob. theory and related fields, 110:177–276, 1998.
  * [88] M. Talagrand.  An assignment problem at high temperature.  Ann. Probab., 31(2):818–848, 2003.
  * [89] M. Talagrand.  The Generic Chaining.  Springer-Verlag, 2005.
  * [90] M. Talagrand.  The Parisi formula.  Annals of mathematics, 163(2):221–263, 2006.
  * [91] M. Talagrand.  Mean field models and spin glasse: Volume II.  A series of modern surveys in mathematics 55, Springer-Verlag, Berlin Heidelberg, 2011.
  * [92] M. Talagrand.  Mean field models and spin glasses: Volume I.  A series of modern surveys in mathematics 54, Springer-Verlag, Berlin Heidelberg, 2011.
  * [93] S. Venkatesh.  Epsilon capacity of neural networks.  Proc. Conf. on Neural Networks for Computing, Snowbird, UT, 1986\.
  * [94] J. Wastlund.  A proof of a conjecture of Buck, Chan, and Robbins on the expected value of the minimum assignment.  Random Structures Algorithms, 26(1-2):237–251, 2004.
  * [95] J. Wastlund.  An easy proof of the $\zeta(2)$ limit in the random assignment problem.  Electronic Communications in Probability, 14:1475, 2009.
  * [96] J. Wastlund.  Replica symmetry of the minimum matching.  Annals of Mathematics, 175(3):1061–1091, 2012.
  * [97] J. G. Wendel.  A problem in geometric probablity.  Mathematics Scandinavia, 11:109–111, 1962.
  * [98] R. O. Winder.  Single stage threshold logic.  Switching circuit theory and logical design, pages 321–332, Sep. 1961.  AIEE Special publications S-134.
  * [99] J. Y. Zhao.  The Hopfield model with superlinearly many patterns.  2011\.  available online at http://arxiv.org/abs/1108.4771.