11institutetext: Dipartimento di Matematica e Fisica “Ennio De Giorgi”,
Università del Salento, Lecce, Italy.22institutetext: Department of
Mathematics, King’s College London, The Strand, London, UK.33institutetext:
Scuola Superiore ISUFI, Università del Salento, Lecce, Italy.44institutetext:
Dipartimento di Informatica, Università di Pisa, Pisa Italy.55institutetext:
Istituto Nazionale di Fisica Nucleare, Sezione di Lecce, Italy.

# Unsupervised and Supervised learning by Dense Associative Memory under
replica symmetry breaking

Linda Albanese 4,5 Andrea Alessandrelli 2 Alessia Annibale 1,5 Adriano Barra

###### Abstract

Statistical mechanics of spin glasses is one of the main strands toward a
comprehension of information processing by neural networks and learning
machines. Tackling this approach, at the fairly standard replica symmetric
level of description, recently Hebbian attractor networks with multi-node
interactions (often called Dense Associative Memories) have been shown to
outperform their classical pairwise counterparts in a number of tasks, from
their robustness against adversarial attacks and their capability to work with
prohibitively weak signals to their supra-linear storage capacities. Focusing
on mathematical techniques more than computational aspects, in this paper we
relax the replica symmetric assumption and we derive the one-step broken-
replica-symmetry picture of supervised and unsupervised learning protocols for
these Dense Associative Memories: a phase diagram in the space of the control
parameters is achieved, independently, both via the Parisi’s hierarchy within
then replica trick as well as via the Guerra’s telescope within the broken-
replica interpolation. Further, an explicit analytical investigation is
provided to deepen both the big-data and ground state limits of these networks
as well as a proof that replica symmetry breaking does not alter the
thresholds for learning and slightly increases the maximal storage capacity.
Finally the De Almeida and Thouless line, depicting the onset of instability
of a replica symmetric description, is also analytically derived highlighting
how, crossed this boundary, the broken replica description should be
preferred.

###### Abstract

Hebbian neural networks with multi-node interactions, often called Dense
Associative Memories, have recently attracted considerable interest in the
statistical mechanics community, as they have been shown to outperform their
pairwise counterparts in a number of features, including resilience against
adversarial attacks, pattern retrieval with extremely weak signals and supra-
linear storage capacities. However, their analysis has so far been carried out
within a replica-symmetric theory. In this manuscript, we relax the assumption
of replica symmetry and analyse these systems at one step of replica-symmetry
breaking, focusing on two different prescriptions for the patterns that we
will refer to as supervised and unsupervised learning. We derive the phase
diagram of the model using two different approaches, namely Parisi’s
hierarchical ansatz for the relationship between different replicas within the
replica approach, and the so-called telescope ansatz within Guerra’s
interpolation method: our results show that replica-symmetry breaking does not
alter the threshold for learning and slightly increases the maximal storage
capacity. Further, we also derive analytically the instability line of the
replica-symmetric theory, using a generalization of the De Almeida and
Thouless approach.

##  1 Introduction

Since Hopfield’s seminal work on the use of biologically inspired neural
networks for associative memory and pattern recognition tasks Hopfield , there
have been many important contributions applying concepts from statistical
mechanics, in particular spin-glass theory, to the study of neural networks
with pairwise Hebbian interactions (for an overview of the vast literature,
the reader is referred to the books MPV ; TalaBook ; Amit ; Dotsenko ; Engel ;
Nishimori ; CoolenKuhnSollich ; Huang  and to the recent reviews Lenka ;
Carleo ). This connection was first pointed out by Hopfield himself in
Hopfield , where he showed that neural networks with pairwise Hebbian
interactions were particular realizations of spin glasses, and it was further
developed by Amit, Gutfreund, and Sompolinsky AGS  who showed that the free-
energy landscape of such systems is characterized by a large number of local
minima, corresponding to different patterns of information, retrieved by the
network from different initial conditions. In addition to neural networks with
pairwise Hebbian interactions, a large body of work has focused on extending
Hebbian learning to multi-node interactions, both in the early days (see e.g.
kanter1988asymmetric ; Gardner ; Baldi ) and more recently (see e.g. HopKro1 ;
ramsauer2020hopfield ). These models are often referred to as dense
associative memories HopKro1 , as they can store many more patterns than the
number of neurons in the network. In particular, they can perform pattern
recognition with a supra-linear storage load Baldi  and can work at very low
signal-to-noise ratio, when compared to their pairwise counterparts Barra-
PRLdetective . In addition, they have been shown to be resilient to
adversarial attacks Krotov2018  and combinations of such models can lead to
exponential storage capacity Lucibello2023 .

In recent years, a duality between Hebbian neural networks and machine
learning models has been pointed out. For example, the Hopfield model has been
shown to be equivalent to a restricted Boltzmann machine Contucci , an
archetypical model for machine learning hinton1984boltzmann , and sparse
restricted Boltzmann machines have been mapped to Hopfield models with diluted
patterns parallel1 ; parallel2 . Furthermore, restricted Boltzmann machines
with generic priors have led to the definition of generalized Hopfield models
Remi ; DaniPRE2017 ; DaniPRE2018  and neural networks with multi-node Hebbian
interactions have recently been shown to be equivalent to higher-order
Boltzmann machines AAAF-JPA2021 ; agliariAlone  and deep Boltzmann machines
alberici2020annealing ; alberici2021deep . As a result, multi-node Hebbian
learning is receiving a second wave of interest since its foundation in the
eighties kanter1988asymmetric ; Gardner  as a paradigm to understand deep
learning ramsauer2020hopfield ; Metha2014 .

In order to make this connection clearer, we should stress that neither the
Hopfield model nor its generalized versions with multi-node Hebbian
interactions, are machine learning models, in that they are not trained from
data and their couplings do not evolve in time according to a learning rule.
Instead, their couplings are fixed from the outset to the values that would
have been attained by a neural network trained according to the Hebbian rule,
which adjusts the strengths of the connections between the neurons based on
the input patterns that the network is exposed to. In this work, we consider
two different routes for exposing the network to data, as considered in
EmergencySN ; prlmiriam . In particular, we analyze the scenario where the
input patterns are noisy or corrupted versions of pattern archetypes. In the
first route, that we will call ‘unsupervised’, we expose the network to all of
the input patterns together, without specifying the archetype that each
pattern is meant to represent barlow1989unsupervised . In the second route,
that we will call ‘supervised’, we split the training dataset in different
classes, corresponding to the archetypes, and the network is exposed to the
examples in each class, class by class, in the same way that a teacher helps a
student to learn one topic at time, from different examples
cunningham2008supervised . Beyond being more realistic than traditional
Hebbian storing, this generalization allows to pose a number of new questions,
such as: is the network able, in either scenario, to retrieve the archetypes
by itself, i.e. find hidden and intrinsic structures in the training dataset?
which is the minimal amount of examples per archetype that the network has to
experience, given the noise in the examples and the number of archetypes, in
the two scenarios?

These questions have been addressed in neural networks with pairwise as well
as multinode Hebbian interactions, for both random uncorrelated datasets and
structured datasets (including the well-known MNist or Fashion MNist)
prlmiriam ; unsup ; super , within a replica-symmetric (RS) analysis, which is
based on the assumption that different replicas (or copies) of the systems are
invariant under permutations. However, since neural networks are particular
realization of spin-glasses, such symmetry is expected to be broken in certain
regions of the parameters space, where the system develops a multitude of
degenerate states. In a recent work albanese2023almeida , we devised a simple
and rigorous method to detect the onset of the instability of RS theories in
neural networks with pairwise and multinode Hebbian interactions. Building on
this work, we derive the RS instability lines for the two different protocols
defined here, supervised and unsupervised, and we re-investigate the problem
defined above, previously analysed within an RS assumption, by carrying out
the analysis at one step of replica-symmetry breaking ($1$RSB). Since Parisi’s
seminal work on replica symmetry breaking MPV , several alternative
mathematical techniques have been proposed to investigate replica-symmetry
broken phases bovier2012mathematical ; gradenigo2020solving ; barra2010replica
. In this work, we will use interpolations techniques, that were pioneered by
Guerra in the context of his work on spin glasses guerra_broken  and were
later applied to neural networks (e.g. AABO-JPA2020 ). One advantage of this
approach is that it avoids certain heuristics required by the replica method
and it leads to simpler calculations. For completeness, we will also derive
the same results using the replica approach, with the pedagogical aim of
creating a bridge between the two methods, one being a cornerstone of the
statistical physics community (Parisi’s), the other constituting a golden
niche in the mathematical physics community (Guerra’s).

The paper is structured as follows. In Section 2 we define a first model for
dense associative memories that we will shall refer to as ’unsupervised’ and
we analyse it at one step of replica symmetry breaking via Guerra’s
interpolation method and replica techniques. In Section 3 we define a second
model for dense associative memories, that we will shall refer to as
’supervised’ and we carry out the same analysis as in Sec. 2. Finally, in Sec.
4 we summarize and discuss our results. We relegate technical details to the
Appendices as well as the derivation of the instability line of the RS theory,
which shows the importance of analysing these systems under the assumption of
replica symmetry breaking.

##  2 ‘Unsupervised’ dense associative memories

In this section we analyse the information processing capabilities of Dense
Associative Memories (DAM) in the so-called ‘unsupervised’ setting, as
introduced in Sec. 2.1, via two different approaches, namely Guerra’s
interpolation techniques (Sec. 2.2) and Parisi’s replica approach (Sec. 2.3).
Both analyses are carried out at the first step of replica-symmetry breaking.
Results are discussed in Sec. 2.4. The instability line of the RS theory
(equivalent to the de Almeida and Thouless line in the context of spin-
glasses), showing the importance of working under the assumption of broken
replica symmetry, is derived in Appendix A.1.

###  2.1 Model and definitions

We consider a system of $N$ neurons, modelled via Ising spins
$\sigma_{i}\in\\{-1,1\\}$, with $i={1,\ldots,N}$, where neurons interact via
$P$-node interactions. We assume to have $K$ Rademacher archetypes, defined as
$N$-dimensional vectors $\bm{\xi}^{\mu}$, with $\mu=1,\ldots,K$, where each
entry is drawn randomly and independently from the distribution

$\displaystyle\mathbb{P}(\xi_{i}^{\mu})=\dfrac{1}{2}\left(\delta_{\xi_{i}^{\mu},+1}+\delta_{\xi_{i}^{\mu},-1}\right).$
(1)

Such archetypes are not provided to the network, instead the network will have
to infer them by experiencing solely noisy or corrupted versions of them. In
particular, we assume that for each archetype $\mu$, $M$ examples
$\bm{\eta}^{\mu,a}$, with $a=1,\ldots,M$, are available, which are corrupted
versions of the archetype, such that

$\mathbb{P}(\eta_{i}^{\mu,a}|\xi_{i}^{\mu})=\frac{1-r}{2}\delta_{\eta_{i}^{\mu,a},-\xi_{i}^{\mu}}+\frac{1+r}{2}\delta_{\chi_{i}^{\mu,a},+\xi_{i}^{\mu}},$
(2)

where $r\in[0,1]$ controls the _quality_ of the dataset, i.e. for $r=1$ the
example matches perfectly the archetype, while for $r=0$ it is purely
stochastic. To quantify the information content of the dataset it is useful to
introduce the variable

$\rho=\frac{1-r^{2}}{Mr^{2}}$ (3)

that we shall refer to as the dataset entropy.111It was shown in prlmiriam
that the conditional entropy which quantifies the amount of information needed
to reconstruct the original pattern $\bm{\xi}^{\mu}$ given the set of related
examples $\\{\eta^{\mu,a}\\}_{a=1,\ldots,M}$, is a monotonically increasing
function of $\rho$. We note that $\rho$ vanishes either when the examples are
identical to the archetypes (i.e. $r\to 1$) or when the number of examples is
infinite (i.e. $M\to\infty$), or both. If we regard the traditional DAM model
as storing patterns and the unsupervised DAM model as learning patterns from
stored examples, then $\rho$ can be thought of as the parameter that
quantifies the difference between storing and learning (which is larger, the
larger $\rho$).

######  Definition 1.

The cost function (or Hamiltonian) of the ‘unsupervised’ DAM model is

$\displaystyle\mathcal{H}_{N}^{(P)}(\bm{\sigma}|\bm{\eta})=$
$\displaystyle-\dfrac{1}{\,\mathcal{R}^{P/2}\,M\,N^{P-1}}\displaystyle\sum\limits_{\mu=1}^{K}\displaystyle\sum\limits_{a=1}^{M}\left(\displaystyle\sum\limits_{i_{1}<\cdots<i_{P}}^{N,\cdots,N}\eta_{i_{1}}^{\mu,a}\cdots\eta_{i_{P}}^{\mu,a}\sigma_{{}_{i_{1}}}\cdots\sigma_{{}_{i_{P}}}\right),$
(4)

where the constant $\mathcal{R}^{P/2}$, with $\mathcal{R}=r^{2}(1+\rho)$ and
$\rho$ defined in (3), is included in the definition for mathematical
convenience, while the factor $N^{P-1}$ ensures that the Hamiltonian is
extensive in the network size $N$. On the left hand side (LHS), the label
$(P)$ denotes the order of the multi-node interactions and $\bm{\eta}$ is a
short-hand for the collection of all the examples, for all the archetypes
$\\{\bm{\eta}^{\mu,a}\\}_{\mu=1,a=1}^{K,M}$.

######  Remark 1.

For $P=2$, i.e. pairwise interactions, the unsupervised DAM model reduces to
the Hopfield neural network in the unsupervised setting EmergencySN . In
addition, in the absence of dataset corruption, i.e. $r=1$, $\mathcal{R}=1$
and the model reduces to the standard Hopfield model, as analysed in AGS ;
Amit .

######  Definition 2.

The partition function associated to the Hamiltonian (4) at inverse noise
level $\beta\in\mathcal{R}^{+}$ is defined as

$\displaystyle\mathcal{Z}^{(P)}_{N}(\bm{\eta})=\sum_{\bm{\sigma}}^{2^{N}}\exp\left(-\beta\mathcal{H}^{(P)}_{N}(\bm{\sigma}|\bm{\eta})\right)=:\sum_{\bm{\sigma}}^{2^{N}}\mathcal{B}^{(P)}_{N}(\bm{\sigma}|\bm{\eta}),$
(5)

where $\mathcal{B}^{(P)}_{N}(\bm{\sigma}|\bm{\eta})$ is referred to as the
Boltzmann factor. At finite network size $N$, the free energy of the model
$\mathcal{F}^{(P)}_{N}$ is given by

$\displaystyle\mathcal{-}\beta\mathcal{F}^{(P)}_{N}=\frac{1}{N}\mathbb{E}\log\mathcal{Z}^{(P)}_{N}(\bm{\eta})$
(6)

where $\mathbb{E}[.]$ denotes the average over the realizations of examples
$\bm{\eta}$, regarded as quenched.

We are interested in the behaviour of the system in the limit of large system
size $N\to\infty$ and finite network load $\alpha$, as specified by the
following

######  Definition 3.

In the thermodynamic limit $N\to\infty$, the load of the unsupervised DAM
model is defined as

$\lim_{N\to+\infty}\frac{K}{N^{P-1}}=:\alpha<\infty$ (7)

We will mostly focus on the so-called ‘saturated’ regime, where $\alpha>0$.
The regime where the system is away from saturation can be inspected by taking
the limit $\alpha\to 0$. For convenience, we also introduce the parameter
$\gamma$, defined by

$\alpha=\gamma\dfrac{2}{P!}.$ (8)

The free energy in the thermodynamic limit is denoted as

$\mathcal{F}^{(P)}=\lim_{N\to\infty}\mathcal{F}^{(P)}_{N}.$ (9)

In the following, we focus on the ability of the network to retrieve a single
archetype, say $\nu$. Given the invariance of the Hamiltonian w.r.t.
permutations of the archetypes, we will set without loss of generality
$\nu=1$. Then, following standard procedures Amit , we split the sum over
$\mu$ in the Hamiltonian into the contribution from $\mu=1$ (regarded as the
signal) and the contributions from the other archetypes $\mu>1$ (regarded as
slow noise). Furthermore, we add in the argument of the exponent of the
Boltzmann factor a term $J\sum_{i}\xi_{i}^{1}\sigma_{i}$, so that the free
energy can serve as a moment generating functional of the so-called Mattis
magnetization $m_{1}$ (vide infra) by taking derivatives of
$\ln\mathcal{Z}_{N}^{(P)}$ w.r.t. $J$. As this term is not part of the
original Hamiltonian, $J$ will be set to zero later on.
Starting from (5), the resulting partition function is

$\displaystyle\mathcal{Z}^{(P)}_{N}(\bm{\eta})=$
$\displaystyle\lim_{J\rightarrow
0}\sum_{\bm{\sigma}}\exp\left[J\sum_{i=1}^{N}\xi_{i}^{1}\sigma_{i}+\dfrac{\beta^{\prime}}{2\,\mathcal{R}^{P/2}M\,N^{P-1}}\displaystyle\sum\limits_{a=1}^{M}\left(\sum_{i=1}^{N}\eta_{{}_{i}}^{a,1}\sigma_{{}_{i}}\right)^{{}^{P}}\right.$
$\displaystyle\left.+\dfrac{\beta^{\prime}P!}{2\mathcal{R}^{P/2}M\,N^{P-1}}\,\displaystyle\sum\limits_{\mu>1}^{K}\displaystyle\sum\limits_{a=1}^{M}\displaystyle\sum\limits_{\
{i_{1}<\cdots<i_{P}}}^{N,\cdots,N}\left(\eta^{\mu\,,a}_{i_{1}}\cdots\eta^{\mu\,,a}_{i_{P}}\>\sigma_{{}_{i_{1}}}\cdots\sigma_{{}_{i_{{}_{P}}}}\right)\,\right]$
(10)

where we have set $\beta^{\prime}=2\beta/P!$ and for the first pattern
($\mu=1$) we used $P!\sum_{i_{1}<\cdots<i_{P}}=\sum_{i_{1}\neq\cdots\neq
i_{P}}$ and neglected terms which vanish in the thermodynamic limit. Focusing
only on the last term in (2.1) we note that the product
$\eta_{i_{1}}^{\mu,a}\ldots\eta_{i_{P}}^{\mu,a}$ is a random i.i.d. Binomial
variable for each $\mu,a$, and the sum over $a$ converges, by the central
limit theorem (CLT), to a Gaussian variable with suitably defined mean and
variance, i.e.

$\left(\dfrac{1}{M}\displaystyle\sum\limits_{a=1}^{M}\eta^{\mu\,,a}_{i_{1}}\cdots\eta^{\mu\,,a}_{i_{P}}\right)\sim\mu+\lambda^{\mu}_{i_{1},\ldots,i_{P}}\frac{\sigma}{\sqrt{M}},\quad\quad\quad\lambda_{i_{1}\ldots
i_{P}}^{\mu}\sim{\mathcal{N}}(0,1),$ (11)

where

$\displaystyle\mu=\langle\eta^{\mu}_{i_{1}}\cdots\eta^{\mu}_{i_{P}}\rangle_{\bm{\xi}}=\left(r\xi_{i_{1}}^{\mu}\right)\ldots\left(r\xi_{i_{P}}^{\mu}\right)$
(12)

and

$\displaystyle\sigma=\sqrt{\langle(\eta^{\mu}_{i_{1}}\cdots\eta^{\mu}_{i_{P}})^{2}\rangle_{\bm{\xi}}-\langle\eta^{\mu}_{i_{1}}\cdots\eta^{\mu}_{i_{P}}\rangle_{\bm{\xi}}^{2}}=\sqrt{(\xi_{i_{1}}^{\mu})^{2}\ldots(\xi_{i_{P}}^{\mu})^{2}-r^{2P}(\xi_{i_{1}}^{\mu}\ldots\xi_{i_{P}}^{\mu})^{2}}$
(13)

due to the independence of different sites. This enables us to write

$\displaystyle\left(\dfrac{1}{M}\displaystyle\sum\limits_{a=1}^{M}\eta^{\mu\,,a}_{i_{1}}\cdots\eta^{\mu\,,a}_{i_{P}}\right)$
$\displaystyle\sim$ $\displaystyle r^{P}\left(1+\lambda_{i_{1}\ldots
i_{P}}^{\mu}\sqrt{\rho_{P}}\right)\xi_{i_{1}}^{\mu}\ldots\xi_{i_{P}}^{\mu}$
(14)

with

$\rho_{P}=\frac{1-r^{2P}}{Mr^{2P}}.$

Similarly, applying the CLT to the sum over $\mu$ we get

$\left(\dfrac{1}{MN^{P-1}}\sum_{a}\sum_{\mu}\eta^{\mu\,,a}_{i_{1}}\cdots\eta^{\mu\,,a}_{i_{P}}\right)=\alpha\sqrt{\frac{r^{2P}}{K}}(1+\sqrt{\rho_{P}})\lambda_{i_{1}\ldots
i_{P}},$

having used

$\frac{1}{K}\sum_{\mu}\xi_{i_{1}}^{\mu}\ldots\xi_{i_{P}}^{\mu}=\frac{1}{\sqrt{K}}\lambda_{i_{1}\ldots
i_{P}},\quad\quad\quad\lambda_{i_{1}\ldots i_{P}}\sim{\mathcal{N}}(0,1)$

and

$\frac{1}{K}\sum_{\mu}\lambda_{i_{1}\ldots
i_{P}}^{\mu}\xi_{i_{1}}^{\mu}\ldots\xi_{i_{P}}^{\mu}=\frac{1}{\sqrt{K}}\lambda_{i_{1}\ldots
i_{P}},\quad\quad\quad\lambda_{i_{1}\ldots i_{P}}\sim{\mathcal{N}}(0,1).$

Thus the partition function of the unsupervised DAM model reads as

$\displaystyle\mathcal{Z}^{(P)}_{N}(\bm{\eta}^{1},\bm{\lambda};J)=$
$\displaystyle\lim_{J\rightarrow
0}\sum_{\bm{\sigma}}\exp\left[J\sum_{i=1}^{N}\xi_{i}^{1}\sigma_{i}+\dfrac{\beta^{\prime}}{2\mathcal{R}^{P/2}M\,N^{P-1}}\displaystyle\sum\limits_{a=1}^{M}\left(\displaystyle\sum\limits_{i}^{N}\eta_{{}_{i}}^{1,a}\sigma_{{}_{i}}\right)^{{}^{P}}\right.$
$\displaystyle\left.+\frac{\alpha}{\sqrt{K}}\dfrac{\beta^{\prime}P!(1+\sqrt{\rho_{P}})}{2(1+\rho)^{P/2}}\left(\displaystyle\sum\limits_{{i_{1}<\cdots<i_{P}}}^{N,\cdots,N}\lambda_{i_{1},\ldots,i_{P}}\>\sigma_{{}_{i_{1}}}\cdots\sigma_{{}_{i_{{}_{P}}}}\right)\right].$
(15)

To make analytical progress, it is convenient to introduce the order
parameters of the model.

######  Definition 4.

The order parameters of the unsupervised DAM model are the Mattis
magnetization $m_{1}$ of the archetype $\bm{\xi}^{1}$, the Mattis
magnetizations $n_{a,1}$ of each example $a=1\ldots M$ of the archetype
$\bm{\xi}^{1}$ and the two-replica overlaps $q_{lm}$, which quantifies the
correlations between the variables ${\bm{\sigma}}$ in two copies $l$ and $m$
of the system, sharing the same disorder (i.e. two replicas):

$\displaystyle m_{1}(\bm{\sigma})$ $\displaystyle:=$
$\displaystyle\dfrac{1}{N}\displaystyle\sum\limits_{i=1}^{N}\xi_{i}^{1}\sigma_{i}$
(16) $\displaystyle n_{1,a}(\bm{\sigma})$ $\displaystyle:=$
$\displaystyle\dfrac{r}{\mathcal{R}}\dfrac{1}{N}\displaystyle\sum\limits_{i=1}^{N}\eta_{i}^{1\,,a}\sigma_{i},$
(17) $\displaystyle q_{lm}(\bm{\sigma}^{(l)},\bm{\sigma}^{(m)})$
$\displaystyle:=$
$\displaystyle\dfrac{1}{N}\displaystyle\sum\limits_{i=1}^{N}\sigma_{i}^{(l)}\sigma_{i}^{(m)}\,.$
(18)

In the next subsection we calculate the free energy of the system in the
thermodynamic limit using Guerra’s interpolation technique, assuming one step
of replica symmetry breaking (1RSB).

###  2.2 1RSB analysis via Guerra’s interpolation technique

The key idea of Guerra’s interpolation method is to define an auxiliary free
energy, $\mathcal{F}^{(P)}(t)$, which is function of a parameter $t\in[0,1]$
that interpolates between the free energy of the original DAM model (obtained
at $t=1$) and the free energy of an exactly solvable one-body model (obtained
at $t=0$), whose effective fields mimic those of the original model. As the
direct calculation of $\mathcal{F}^{(P)}(t=1)$ is cumbersome, this is obtained
by using the fundamental theorem of calculus, namely we first evaluate
$\mathcal{F}^{(P)}(t=0)$ and then we obtain $\mathcal{F}^{(P)}(t=1)$ as

$\mathcal{F}^{(P)}(t=1)=\mathcal{F}^{(P)}(t=0)+\int_{0}^{1}ds\,\frac{d\mathcal{F}^{(P)}}{dt}|_{t=s}.$
(19)

To perform the analysis within the first step of replica symmetry breaking, we
make the standard $1$RSB assumption MPV , stated below:

######  Assumption 1.

In $1$RSB assumption, the distribution of the two-replica overlap $q_{12}$, in
the thermodynamic limit, displays two delta-peaks at the values $\bar{q}_{1}$
and $\bar{q}_{2}$, and the concentration on these two values is ruled by the
parameter $\theta\in[0,1]$, namely

$\displaystyle\lim_{N\rightarrow+\infty}\mathbb{P}_{N}(q_{12})=\theta\delta(q_{12}-\bar{q}_{1})+(1-\theta)\delta(q_{12}-\bar{q}_{2}),$
(20)

where

$\mathbb{P}_{N}(q_{12})=\sum_{\bm{\sigma}^{(1)},\bm{\sigma}^{(2)}}\sum_{\bm{\eta}}\mathbb{P}(\bm{\eta})\mathbb{P}_{N}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)}|\bm{\eta})\delta(q_{12}-q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)}))$

and
$\mathbb{P}_{N}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)}|\bm{\eta})=\mathcal{B}_{N}(\bm{\sigma}_{1}|\bm{\eta})\mathcal{B}_{N}(\bm{\sigma}_{2}|\bm{\eta})$
as replicas are conditionally independent, given the disorder $\bm{\eta}$.
The Mattis magnetizations $m_{1}$ and $n_{1,a}$ are assumed to be self-
averaging at their equilibrium values $\bar{m}$ and $\bar{n}$, respectively,
namely

$\displaystyle\lim_{N\rightarrow+\infty}\mathbb{P}_{N}(m_{1})$
$\displaystyle=\delta(m_{1}-\bar{m}),$ (21)
$\displaystyle\lim_{N\rightarrow+\infty}\mathbb{P}_{N}(n_{1,a})$
$\displaystyle=\delta(n_{1,a}-\bar{n}).$ (22)

where
$\mathbb{P}_{N}(m_{1})=\mathbb{E}_{\bm{\eta}}\sum_{\bm{\sigma}}\mathcal{B}_{N}(\bm{\sigma}|\bm{\eta})\delta(m_{1}-m_{1}(\bm{\sigma}))$
and
$\mathbb{P}_{N}(n_{1,a})=\mathbb{E}_{\bm{\eta}}\sum_{\bm{\sigma}}\mathcal{B}_{N}(\bm{\sigma}|\bm{\eta})\delta(n_{1,a}-n_{1,a}(\bm{\sigma}))$,
with
$\mathbb{E}_{\bm{\eta}}f(\bm{\eta})=\sum_{\bm{\eta}}f(\bm{\eta})\mathbb{P}({\bm{\eta}})$.

######  Definition 5.

Given the interpolating parameter $t\in[0,1]$, the constant $A_{1},\ A_{2},\
\psi\in\mathbb{R}$ to be fixed later on, and the i.i.d. standard Gaussian
variables $Y_{i}^{(b)}\sim\mathcal{N}(0,1)$ for $i=1,\ldots,N$ and $b=1,2$
(that must be averaged over as explained below), the Guerra’s $1$RSB
interpolating partition function for the unsupervised DAM model is given by

$\displaystyle\mathcal{Z}^{(P)}_{2}(\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)$
$\displaystyle\coloneqq$
$\displaystyle\sum_{\bm{\sigma}}\mathcal{B}^{(P)}_{2}(\bm{\sigma}|\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)$
$\displaystyle=$
$\displaystyle\sum_{\bm{\sigma}}\exp{\Bigg{[}}J\sum_{i=1}^{N}\xi_{i}^{1}\sigma_{i}+\dfrac{t\beta^{\prime}N}{2M}\left(1+\rho\right)^{P/2}\displaystyle\sum\limits_{a=1}^{M}n_{1,a}^{{}^{P}}(\bm{\sigma})+\psi(1-t)N\displaystyle\sum\limits_{a=1}^{M}n_{1,a}(\bm{\sigma})$
$\displaystyle+$
$\displaystyle\sqrt{1-t}\displaystyle\sum\limits_{b=1}^{2}\left(A_{b}\displaystyle\sum\limits_{i=1}^{N}Y_{i}^{(b)}\sigma_{i}\right)+\sqrt{t}\dfrac{\beta^{\prime}P!\alpha(1+\sqrt{\rho_{P}})}{2(1+\rho)^{P/2}\sqrt{K}}\displaystyle\sum\limits_{{i_{1}<\cdots<i_{P}}}^{N,\cdots,N}\lambda_{i_{1},\ldots,i_{P}}\sigma_{{}_{i_{1}}}\cdots\sigma_{{}_{i_{{}_{P}}}}\Bigg{]}.$

The index $2$ on the LHS stands for the number of vectors ${\bm{Y}}^{(b)}$
that must be averaged over. Their number is equal to $k\\!+\\!1$ where $k$ is
the number of steps of replica-symmetry breaking (here $k=1$).

The average over the generalised Boltzmann factor associated to the
interpolating partition function
$\mathcal{Z}^{(P)}_{2}(\bm{\eta},\bm{\eta},\bm{Y};J,t)$, can be defined as

$\omega_{t}(\cdot)\coloneqq\frac{1}{\mathcal{Z}^{(P)}_{2}(\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)}\,\sum_{\bm{\sigma}}\leavevmode\nobreak\
\cdot\leavevmode\nobreak\
\mathcal{B}_{2}^{(P)}(\bm{\sigma}|\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t).$
(25)

######  Remark 2.

We note that for $t=1$, (LABEL:def:partfunct_GuerraRS) recovers the original
partition function (2.1), whereas for $t=0$ it reduces to the partition
function of a system of $N$ non-interacting neurons, described by the
Hamiltonian
$-\beta^{{}^{\prime}}\mathcal{H}_{N}^{(1)}(\bm{\sigma}|\bm{\eta},{\bf
Y};J)=\sum_{i}h_{i}\sigma_{i}$, with local fields $h_{i}=J\xi_{i}^{1}+\psi
r\mathcal{R}^{-1}\sum_{a=1}^{M}\eta_{i}^{1,a}+\sum_{b=1}^{2}A_{b}J_{i}^{(b)}$,
that is readily evaluated. The parameters of the one-body model $\psi$,
$A_{1}$ and $A_{2}$ must be chosen in such a way that the $t$-dependent terms
in $d\mathcal{F}^{(P)}/dt$ cancel out in the thermodynamic limit, under the
1RSB assumption.

In what follows, we will average over the fields ${\bm{Y}}^{(b)}$ for $b=1,2$
recursively, as explained by Guerra in guerra_broken  and in the statements
below. To this purpose, we define

$\displaystyle\mathcal{Z}_{1}^{(P)}(\bm{\eta}^{1},\bm{\lambda},\bm{Y}^{(1)};J,t)\coloneqq$
$\displaystyle\left[\mathbb{E}_{\bm{Y}^{(2)}}\mathcal{Z}_{2}^{(P)}(\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)^{\theta}\right]^{1/\theta}$
(26)
$\displaystyle\mathcal{Z}_{0}^{(P)}(\bm{\eta}^{1},\bm{\lambda};J,t)\coloneqq$
$\displaystyle\exp\left\\{\mathbb{E}_{\bm{Y}^{(1)}}\left[\ln\mathcal{Z}_{1}^{(P)}(\bm{\eta}^{1},\bm{\lambda},\bm{Y}^{(1)};J,t)\right]\right\\}$
(27)
$\displaystyle\mathcal{Z}^{(P)}_{N}(\bm{\eta}^{1},\bm{\lambda};J,t)\coloneqq$
$\displaystyle\mathcal{Z}_{0}^{(P)}(\bm{\eta}^{1},\bm{\lambda};J,t),$ (28)

where with $\mathbb{E}_{\bm{Y}^{(b)}}[.]$ we mean the average over the vectors
${\bm{Y}}^{(b)}$ for $b=1,2$. The interpolating quenched free energy related
to the partition function (LABEL:def:partfunct_GuerraRS) is introduced as

$-\beta^{{}^{\prime}}\mathcal{F}^{(P)}_{N}(J,t)\coloneqq\frac{1}{N}\mathbb{E}_{0}\left[\ln\mathcal{Z}^{(P)}_{N}(\bm{\eta}^{1},\bm{\lambda};J,t)\right],$
(29)

where $\mathbb{E}_{0}[.]$ denotes the average over the variables
$\lambda^{\mu}_{i_{1},\ldots,i_{P}}$’s and $\eta_{i}^{1,a}$’s. In the
thermodynamic limit, assuming the limit exists, we write

$\mathcal{F}^{(P)}(J,t)\coloneqq\lim_{N\to\infty}\mathcal{F}^{(P)}_{N}(J,t).$
(30)

######  Remark 3.

We note that in the RS case, where
$\mathcal{Z}_{2}^{(P)}(\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)\equiv\mathcal{Z}_{1}^{(P)}(\bm{\eta}^{1},\bm{\lambda},\bm{Y}^{(1)};J,t)$
and $\bm{Y}^{(2)}={0}$, we have

$-\beta^{{}^{\prime}}\mathcal{F}^{(P)}(J,t)\coloneqq\lim_{N\to\infty}\frac{1}{N}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\left[\ln\mathcal{Z}^{(P)}_{1}(\bm{\eta}^{1},\bm{\lambda},\bm{Y}^{(1)};J,t)\right],$
(31)

hence the average $\mathbb{E}_{\bm{Y}^{(1)}}$ acts on the same level as
$\mathbb{E}_{0}$ i.e. outside the logarithm. When moving from the RS to the
$1$RSB scenario, an extra average is introduced, $\mathbb{E}_{\bm{Y}^{(2)}}$,
which acts on a different level than $\mathbb{E}_{0}$ and
$\mathbb{E}_{\bm{Y}^{(1)}}$ i.e. inside the logarithm. This reflects the
presence of hierarchical valleys in the free energy landscape Rammal  which
leads to the well-known multi-scale thermalization in spin glasses. The
internal average $\mathbb{E}_{Y^{(2)}}$ is related to thermalization within a
valley at the lower level of the (two-level) hierarchy, whereas the external
averages account for thermalization across valleys (i.e. at the higher level
of the hierarchy). The structure of the hierarchy is captured by the parameter
$\theta$, which controls the amplitudes of the valleys at the lower level of
the hierarchy and can be related to effective temperatures Leticia , as
discussed in VanMourik  for Parisi’s replica approach and in Mingione  for
Guerra’s interpolation techniques. Note that the distinction between the two
levels of the hierarchy is not made for $\psi$ which accounts for the signal
contribution and is kept RS (as usually done for the magnetization in spin-
glass models).

Now, following Guerra’s prescription guerra_broken , given two copies (or
replicas) of the system, we define the following averages, corresponding to
thermalization within the two different levels of the hierarchy

$\displaystyle\langle\cdot\rangle_{t,1}\coloneqq\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}[\mathbb{E}_{\bm{Y}^{(2)}}\mathcal{W}_{2,t}\omega_{t}(\cdot)]^{2},$
(32)
$\displaystyle\langle\cdot\rangle_{t,2}\coloneqq\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\mathbb{E}_{\bm{Y}^{(2)}}\mathcal{W}_{2,t}[\omega_{t}(\cdot)]^{2},$
(33)

where

$\mathcal{W}_{2,t}\coloneqq\dfrac{\mathcal{Z}_{2}^{\theta}(\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)}{\mathbb{E}_{Y^{(2)}}\mathcal{Z}^{\theta}_{2}(\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)}.$
(34)

At this point, we are able to state our second assumption, that is at each
level $a=1,2$ of the hierarchy, the two-replica overlap
$q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})$ self-averages around the value
$\bar{q}_{a}$ of the corresponding peak in the overlap distribution
$\mathbb{P}_{N}(q)$, as given in Assumption 1:

######  Assumption 2.

For any $t\in[0;1]$

$\displaystyle\langle[q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})-\bar{q}_{a}]^{k}\rangle_{t,a}$
$\displaystyle\xrightarrow[]{N\to\infty}$ $\displaystyle
0\;\;\;\mathrm{for}\;a=1,2\;\;\mathrm{and}\;k\geq 2$

Finally, we provide the explicit expression of the quenched free energy in
terms of the control parameters in the next

######  Proposition 1.

In the thermodynamic limit $N\to\infty$, within the $1$RSB Assumption and
under the Assumption 2, the quenched free energy for the unsupervised DAM
model, reads as

$\begin{array}[]{lll}-\beta^{{}^{\prime}}\mathcal{F}^{(P)}(J=0)=&\
\dfrac{1}{\theta}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\ln\mathbb{E}_{Y^{(2)}}\left\\{\cosh^{\theta}\left[\beta^{\prime}\dfrac{P}{2}\bar{n}^{P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}\right.\right.\\\
\\\
&\left.\left.+Y^{(1)}\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\bar{q}_{1}^{{}^{P-1}}}+Y^{(2)}\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\left(\bar{q}_{2}^{P-1}-\bar{q}_{1}^{{}^{P-1}}\right)}\right]\right\\}\\\
\\\
&-\beta^{{}^{\prime}}\dfrac{1}{2}(P-1)(1+\rho)^{P/2}\bar{n}^{P}+\dfrac{{\beta^{{}^{\prime}}}^{2}}{4}\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\left[1-P\bar{q}_{2}^{P-1}+(P-1)\bar{q}_{2}^{P}\right]\\\
\\\
&-\dfrac{{\beta^{{}^{\prime}}}^{2}}{4}\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}(P-1)\theta(\bar{q}_{2}^{P}-\bar{q}_{1}^{P})+\ln
2.\end{array}$ (35)

where

$\hat{\eta}_{\mbox{\tiny$M$}}:=\dfrac{1}{rM}\sum_{a=1}^{M}\eta^{1,a},$ (36)

$\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}=\prod\limits_{a=1}^{M}\mathbb{E}_{(\eta^{1,a}|\xi^{1})}$
and $\mathbb{E}_{\bm{\xi}^{1}}$ and $\mathbb{E}_{(\eta^{1,a}|\xi^{1})}$ denote
the expectations over the Bernoulli distributions (1) and (2) respectively. In
the above, $\bar{n}$, $\bar{q}_{1},\ \bar{q}_{2}$ fulfill the following self-
consistency equations

$\begin{array}[]{lll}\bar{n}=\dfrac{1}{1+\rho}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})\tanh
g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}\hat{\eta}_{\mbox{\tiny$M$}}\right],\\\
\\\
\bar{q}_{1}=\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})\tanh
g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}\right]^{2},\\\
\\\
\bar{q}_{2}=\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})\tanh^{2}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}\right],\end{array}$
(37)

with

$\displaystyle g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})=$
$\displaystyle\beta^{\prime}\dfrac{P}{2}\bar{n}^{P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}+Y^{(1)}\beta^{\prime}\sqrt{\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\dfrac{P}{2}\bar{q}_{1}^{{}^{P-1}}}$
$\displaystyle+Y^{(2)}\beta^{\prime}\sqrt{\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\dfrac{P}{2}\left(\bar{q}_{2}^{{}^{P-1}}-\bar{q}_{1}^{{}^{P-1}}\right)}$
(38)

and $\beta^{{}^{\prime}}=2\beta/P!$. Furthermore, as
$\bar{m}=-\beta^{{}^{\prime}}\nabla_{J}\mathcal{F}^{(P)}(J)|_{J=0}$, we have

$\displaystyle\bar{m}=\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})\tanh
g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}\xi^{1}\right].$
(39)

In order to prove the aforementioned proposition, we need to premise the
following

######  Lemma 1.

In the thermodynamic limit, the $t$ derivative of the interpolating quenched
free energy (29) under the 1RSB assumption and under Assumption 2 is given by

$\begin{array}[]{lll}\dfrac{d\mathcal{F}^{(P)}(J,t)}{dt}=&\dfrac{1}{2}(P-1)(1+\rho)^{P/2}\bar{n}^{P}-\dfrac{\beta^{{}^{\prime}}}{4}\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\left[1-P\bar{q}_{2}^{P-1}+(P-1)\bar{q}_{2}^{P}\right]\\\
\\\
&+\dfrac{\beta^{{}^{\prime}}}{4}\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}(P-1)\theta(\bar{q}_{2}^{P}-\bar{q}_{1}^{P}).\end{array}$
(40)

The proof of this Lemma is shown in Appendix B.1. Now let us prove Proposition
1:

###### Proof.

Exploiting the fundamental theorem of calculus, we can relate the free energy
of the original model $\mathcal{F}^{(P)}(J,t=1)$ and the one stemming from the
one body terms $\mathcal{F}^{(P)}(J,t=0)$ via (19). Since, the $t$-derivative
(40) does not depend on $t$, all we need is to add the expression above to the
free energy $\mathcal{F}^{(P)}(J,t=0)$, which only contains one body terms.
Let us start with the computation of the latter at finite size $N$:

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}_{N}^{(P)}(J,t=0)=$
$\displaystyle\dfrac{1}{\theta
N}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\ln\mathbb{E}_{Y^{(2)}}\sum_{\bm{\sigma}}\exp\left[-\beta\mathcal{H}_{N}^{(1)}(\bm{\sigma}|\bm{\eta}^{1},{\bf
Y};J)\right]$ $\displaystyle=$ $\displaystyle\ln
2+\dfrac{1}{\theta}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\ln\mathbb{E}_{Y^{(2)}}\cosh^{\theta}\left(J\xi^{1}+\dfrac{\psi}{(1+\rho)}\hat{\eta}_{M}+Y^{(1)}A_{1}+Y^{(2)}A_{2}\right),$
(41)

with the definition of $-\beta\mathcal{H}_{N}^{(1)}(\bm{\sigma}|\bm{\eta};J)$
given in Remark 2.

Upon setting the constants to the values (148) stated in the proof of Lemma 1,
we have

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}_{N}^{(P)}(J,t=0)=$
$\displaystyle\ln
2+\dfrac{1}{\theta}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\ln\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta^{{}^{\prime}},K,N,\bm{Y}),$
(42)

where $g(\beta,K,N,\bm{Y})$ reads as

$\displaystyle g(\beta,K,N,\bm{Y})=$ $\displaystyle
J\xi^{1}+\beta^{\prime}\dfrac{P}{2}\bar{n}^{P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}$
$\displaystyle+Y^{(1)}\beta^{\prime}\sqrt{\dfrac{P(1+\sqrt{\rho_{P}})^{2}}{2(1+\rho)^{P}}\dfrac{P!K}{2N^{P-1}}\bar{q}_{1}^{{}^{P-1}}}+Y^{(2)}\beta^{\prime}\sqrt{\dfrac{P(1+\sqrt{\rho_{P}})^{2}}{2(1+\rho)^{P}}\dfrac{P!K}{2N^{P-1}}\left(\bar{q}_{2}^{{}^{P-1}}-\bar{q}_{1}^{{}^{P-1}}\right)}.$
(43)

Recalling that $\lim\limits_{N\to\infty}K/N^{P-1}=2\gamma/P!$, taking the
thermodynamic limit of the above equations and inserting them in the
fundamental theorem of calculus (19), we reach (35). Finally, by maximizing
(35) w.r.t. to the order parameters $\bar{n},\bar{q}_{1},\bar{q}_{2}$, we
obtain the self-consistency equations (37), hence we reach the thesis. ∎

###  2.3 1RSB analysis via Parisi’s replica trick

In this Section, we derive the expression of the quenched free energy of the
unsupervised DAM model provided in Proposition 1 by using the replica method
SK1975 ; MPV , at the first step of replica-symmetry breaking GiorgioRSB ; MPV
. The core of this approach consists in writing the quenched free energy, as
defined in (6), as

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}_{N}^{(P)}(J)=\dfrac{1}{N}\lim_{n\to
0}\dfrac{\mathbb{E}\mathcal{Z}_{N}^{n}(J)-1}{n},$ (44)

with $\beta^{{}^{\prime}}=2\beta/P!$ and where we have used the shorthand
$\mathcal{Z}_{N}(J)=\mathcal{Z}_{N}^{(P)}(\bm{\eta};J)$ for the partition
function, defined as

$\displaystyle\mathcal{Z}_{N}(J)$
$\displaystyle=\sum_{\bm{\sigma}}\exp\left(-\beta^{{}^{\prime}}\mathcal{H}^{(P)}_{N}(\bm{\sigma}|\bm{\eta})+J\displaystyle\sum\limits_{i=1}^{N}\xi_{i}^{1}\sigma_{i}\right)\,,$
(45)

$\mathbb{E}[.]$ denotes, as in (6), the average over the quenched disorder
$\bm{\eta}$ and in accordance with the explanation provided in Sec. 2.1, the
inclusion of the last term in the round brackets of (45) is intended to make
the free energy a moment-generating function for the Mattis magnetization
$m_{1}(\bm{\sigma})$.

For integer $n$ the function $\mathcal{Z}_{N}^{n}$ is the product of a system
of $n$ identical replicas of the original system

$\displaystyle\mathbb{E}\mathcal{Z}_{N}^{n}(J)$
$\displaystyle=\mathbb{E}\sum_{\bm{\sigma}^{1}\ldots\bm{\sigma}^{n}}\exp\left(-\beta^{{}^{\prime}}\sum_{a=1}^{N}\mathcal{H}^{(P)}_{N}(\bm{\sigma}^{(a)}|\bm{\eta})+J\displaystyle\sum\limits_{a=1}^{n}\displaystyle\sum\limits_{i=1}^{N}\xi_{i}^{1}\sigma_{i}^{(a)}\right)$
(46)

namely

$\displaystyle\mathbb{E}\mathcal{Z}_{N}^{n}(J)$
$\displaystyle=\mathbb{E}\sum_{\bm{\sigma}^{1}\ldots\bm{\sigma}^{n}}\exp\left[\dfrac{\beta^{{}^{\prime}}P!}{2\mathcal{R}^{P/2}\,M\,N^{P-1}}\displaystyle\sum\limits_{a=1}^{n}\displaystyle\sum\limits_{\mu=1}^{K}\displaystyle\sum\limits_{A=1}^{M}\left(\displaystyle\sum\limits_{i_{1}<\cdots<i_{P}}^{N,\ldots,N}\eta^{\mu\,,A}_{i}\cdots\eta^{\mu\,,A}_{i_{P}}\sigma^{(a)}_{{}_{i_{1}}}\cdots\sigma^{(a)}_{{}_{i_{P}}}\right)+J\displaystyle\sum\limits_{a,i}^{n,N}\xi_{i}^{1}\sigma_{i}^{(a)}\right]\,.$
(47)

Splitting, as before, the signal term ($\mu=1$) from the remaining terms
$\mu>1$, which act as a quenched noise on the learning and retrieval of
pattern $\bm{\xi}^{1}$, we can write

$\displaystyle\mathbb{E}\mathcal{Z}_{N}^{n}(J)=$
$\displaystyle\sum_{\bm{\sigma}^{1}\ldots\bm{\sigma}^{n}}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\exp\left[\dfrac{\beta^{{}^{\prime}}(1+\rho)^{P/2}}{2MN}\sum_{a=1}^{n}\sum_{A=1}^{M}\left(\dfrac{r}{N\mathcal{R}}\sum_{i}\eta^{1\,,A}_{i}\sigma_{i}^{(a)}\right)^{P}+J\displaystyle\sum\limits_{a,i}^{n,N}\xi_{i}^{1}\sigma_{i}^{(a)}\right]\cdot$
$\displaystyle\mathbb{E}_{\bm{\xi}^{\mu>1}}\mathbb{E}_{(\bm{\eta}^{\mu>1}|\bm{\xi}^{\mu>1})}\exp\left[\dfrac{\beta^{{}^{\prime}}P!}{2M\mathcal{R}^{P/2}N^{P-1}}\sum_{\mu>1,a,A}\sum_{i_{1}<\cdots<i_{P}}\eta^{\mu\,,A}_{i_{1}}\ldots\eta^{\mu\,,A}_{i_{P}}\sigma_{i_{1}}^{(a)}\ldots\sigma_{i_{P}}^{(a)}\right].$
(48)

where $\mathbb{E}_{\bm{\xi}^{\mu}}[.]=\prod_{i}\mathbb{E}_{\xi_{i}^{\mu}}[.]$
denotes the average over the pattern $\bm{\xi}^{\mu}$, whose entries
$\xi_{i}^{\mu}$ are drawn from the distribution (1) and
$\mathbb{E}_{(\bm{\eta}^{\mu}|\bm{\xi}^{\mu})}=\prod_{i,a}\mathbb{E}_{(\eta_{i}^{\mu,a}|\xi_{i}^{\mu})}$
with $\mathbb{E}_{\eta_{i}^{\mu,A}}$ denoting the expectation over the
conditional distribution (2). As before, exploiting the CLT to rewrite the
last term in (48) via (14), we obtain

$\displaystyle\mathbb{E}\mathcal{Z}_{N}^{n}(J)=$
$\displaystyle\sum_{\bm{\sigma}^{1}\ldots\bm{\sigma}^{n}}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\exp\left[\dfrac{\beta^{{}^{\prime}}(1+\rho)^{P/2}N}{2M}\sum_{a=1}^{n}\sum_{A=1}^{M}\left(\dfrac{r}{N\mathcal{R}}\sum_{i}\eta^{1\,,A}_{i}\sigma_{i}^{(a)}\right)^{P}+J\displaystyle\sum\limits_{a,i}^{n,N}\xi_{i}^{1}\sigma_{i}^{(a)}\right]\cdot$
$\displaystyle\mathbb{E}_{\bm{\lambda}}\exp\left[\dfrac{\beta^{{}^{\prime}}P!\alpha(1+\sqrt{\rho_{P}})}{2(1+\rho)^{P/2}\sqrt{K}}\sum_{a}\sum_{i_{1}<\cdots<i_{P}}\lambda_{i_{1},\ldots,i_{P}}\sigma_{i_{1}}^{(a)}\ldots\sigma_{i_{P}}^{(a)}\right]$
$\displaystyle=$
$\displaystyle\sum_{\bm{\sigma}^{1}\ldots\bm{\sigma}^{n}}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathcal{Z}_{signal}^{n}(J)\mathbb{E}_{\bm{\lambda}}\mathcal{Z}_{noise}^{n},$
(49)

Treating the two terms
$\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{\mbox{\tiny$1$}})}\mathcal{Z}_{signal}^{n}(J)$
and $\mathbb{E}_{\bm{\lambda}}\mathcal{Z}_{noise}^{n}$ separately, we insert
in the former the identity

$1=\prod\limits_{a,A=1,1}^{n,M}\int
dn_{1,A}^{(a)}\,\delta\left(n_{1,A}^{(a)}-n_{1,A}^{(a)}(\bm{\sigma})\right)$

and use the Fourier representation of the Dirac delta, obtaining

$\displaystyle\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathcal{Z}_{signal}^{n}(J)=$
$\displaystyle\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\int\prod_{a,A}dn_{1,A}^{(a)}\prod_{a,A}\frac{dz_{1,A}^{(a)}}{2\pi}\exp\left[\dfrac{\beta^{{}^{\prime}}N(1+\rho)^{P/2}}{2M}\sum_{A,a}(n_{1,A}^{(a)})^{P}\right.$
$\displaystyle\left.+i\sum_{a,A}\left(n_{1,A}^{(a)}-\dfrac{r}{\mathcal{R}N}\sum_{i}\eta_{i}^{1,A}\sigma_{i}^{(a)}\right)z_{1,A}^{(a)}+J\displaystyle\sum\limits_{a,i}^{n,N}\xi_{i}^{1}\sigma_{i}^{(a)}\right].$
(50)

For the noise term, performing first the expectation over $\bm{\lambda}$

$\displaystyle\mathbb{E}_{\bm{\lambda}}\mathcal{Z}_{noise}^{n}=$
$\displaystyle\int\prod_{\bm{i}}\dfrac{d\lambda_{i_{1},\ldots,i_{P}}}{\sqrt{2\pi}}\exp\left[-\dfrac{(\lambda_{i_{1},\ldots,i_{P}})^{2}}{2}+\dfrac{\beta^{{}^{\prime}}\alpha(1+\sqrt{\rho_{P}})}{2(1+\rho)^{P/2}\sqrt{K}}P!\displaystyle\sum\limits_{a}^{n}\displaystyle\sum\limits_{i_{1}<\ldots<i_{P}}^{N,\ldots,N}\lambda_{i_{1},\ldots,i_{P}}\sigma_{i_{1}}^{(a)}\cdots\sigma_{i_{P}}^{(a)}\right]$
$\displaystyle=$
$\displaystyle\exp\left[\dfrac{1}{2}\left(\dfrac{\beta^{{}^{\prime}}(1+\sqrt{\rho_{P}})}{2(1+\rho)^{P/2}N^{P-1}}\right)^{2}K(P!)^{2}\displaystyle\sum\limits_{i_{1}<\ldots<i_{P}}^{N,\ldots,N}\displaystyle\sum\limits_{a,b}^{n,n}\sigma_{i_{1}}^{(a)}\sigma_{i_{1}}^{(b)}\cdots\sigma_{i_{P}}^{(a)}\sigma_{i_{P}}^{(b)}\right]$
$\displaystyle=$
$\displaystyle\exp\left[\dfrac{1}{2}\left(\dfrac{\beta^{{}^{\prime}}(1+\sqrt{\rho_{P}})}{2(1+\rho)^{P/2}N^{P/2-1}}\right)^{2}KP!\displaystyle\sum\limits_{a,b}^{n,n}\left(\dfrac{1}{N}\displaystyle\sum\limits_{i}^{N}\sigma_{i_{1}}^{(a)}\sigma_{i_{1}}^{(b)}\right)^{P}\>\right]$

inserting

$1=\prod_{a,b}\int dq_{ab}\,\delta(q_{ab}-q_{ab}(\bm{\sigma}))$

and using the Fourier representation of the Dirac delta, we have

$\displaystyle\mathbb{E}_{\bm{\lambda}}\mathcal{Z}_{noise}^{n}=$
$\displaystyle\int\prod_{a,b}dq_{ab}\int\prod_{a,b}\dfrac{dp_{ab}}{2\pi}\exp\left[\dfrac{1}{2}\left(\dfrac{\beta^{{}^{\prime}}(1+\sqrt{\rho_{P}})^{2}}{2(1+\rho)^{P/2}N^{P/2-1}}\right)^{2}KP!\displaystyle\sum\limits_{a,b}^{n,n}q_{ab}^{P}\right.$
$\displaystyle\left.+i\sum_{a,b}\left(q_{ab}-\dfrac{1}{N}\sum_{i}\sigma_{i}^{(a)}\sigma_{i}^{(b)}\right)p_{ab}\right]$
(52)

Inserting Eqs. (50) and (52) in (49), we obtain

$\displaystyle\mathbb{E}\mathcal{Z}_{N}^{n}(J)=\int\prod_{a,A}dn_{1,A}^{(a)}\prod_{a,A}\dfrac{dz_{1,A}^{(a)}}{2\pi}\int\prod_{a,b}dq_{ab}\int\prod_{a,b}\dfrac{dp_{a,b}}{2\pi}\exp\left(-NA[\bm{Q},\bm{P},\bm{Z},\bm{N}]\right)$
(53)

where

$\displaystyle
A[\bm{Q},\bm{P},\bm{Z},\bm{N}]=-\dfrac{i}{N}\sum_{a,b}p_{ab}q_{ab}-\dfrac{1}{4}\dfrac{\beta^{{}^{\prime}}\,{}^{2}(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\gamma\displaystyle\sum\limits_{a,b}^{n,n}q_{ab}^{P}-\dfrac{\beta^{{}^{\prime}}(1+\rho)^{P/2}}{2M}\sum_{A,a}(n_{1,A}^{(a)})^{P}$
$\displaystyle-\dfrac{i}{N}\sum_{a,A}n_{1,A}^{(a)}z_{1,A}^{(a)}-\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\log\left[\sum_{\bm{\sigma}}\exp\left(-\dfrac{i}{N}\sum_{a,b,i}p_{ab}\sigma^{(a)}\sigma^{(b)}-\dfrac{ir}{\mathcal{R}N}\sum_{a,A}\eta^{1,A}z_{1,A}^{(a)}\sigma^{(a)}+J\displaystyle\sum\limits_{a}\xi^{1}\sigma^{(a)}\right)\right].$
(54)

and we have used the shorthand notation ${\bm{Q}}=\\{q_{ab}\\}_{a,b=1}^{n}$
and similarly for ${\bm{P}}$, ${\bm{N}}$ and ${\bm{Z}}$. Next, we assume that
the limit $n\to 0$ in

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}^{(P)}(\beta)=\lim_{N\to\infty}\dfrac{1}{N}\lim_{n\to
0}\dfrac{\mathbb{E}\mathcal{Z}_{N}^{n}-1}{n},$ (55)

can be taken by analytic continuation and that the two limits $N\to\infty$ and
$n\to 0$ can be interchanged (provided they exist), so that the integrals in
(53) can be performed by steepest descent. This leads to

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}^{(P)}(\beta)=-\lim_{n\to
0}\dfrac{1}{n}A[\bm{Q}^{\star},\bm{P}^{\star},\bm{Z}^{\star},\bm{N}^{\star}]$
(56)

where $\bm{Q}^{\star},\bm{P}^{\star},\bm{Z}^{\star},\bm{N}^{\star}$, are
solution of the saddle point equations

$\displaystyle\dfrac{\partial A}{\partial p_{ab}}=0$
$\displaystyle\quad\Rightarrow\quad
q_{ab}^{*}=\langle\sigma^{a}\sigma^{b}\rangle_{\rm eff}$ (57)
$\displaystyle\dfrac{\partial A}{\partial z_{1,A}^{(a)}}=0$
$\displaystyle\quad\Rightarrow\quad(n_{1,A}^{(a)})^{*}=\dfrac{r}{\mathcal{R}}\displaystyle\sum\limits_{A=1}^{M}\mathbb{E}\langle\eta^{1,A}\sigma^{a}\rangle_{\rm
eff}$ (58) $\displaystyle\dfrac{\partial A}{\partial q_{ab}}=0$
$\displaystyle\quad\Rightarrow\quad
p_{ab}^{*}=\dfrac{i\beta^{{}^{\prime}}\,{}^{2}}{2}\dfrac{P}{2}\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)}N(q_{ab}^{*})^{P-1}$
(59) $\displaystyle\dfrac{\partial A}{\partial n_{1,A}^{(a)}}=0$
$\displaystyle\quad\Rightarrow\quad(z_{1,A}^{(a)})^{*}=i\beta^{{}^{\prime}}\dfrac{P}{2}\dfrac{(1+\rho)^{P/2}}{M}N[(n_{1,A}^{(a)})^{*}]^{P-1}$
(60)

where the average $\langle\ldots\rangle_{\rm eff}$ is computed over the
distribution $p_{\rm eff}(\bm{\sigma})=e^{-\beta^{{}^{\prime}}H_{\rm
eff}(\bm{\sigma})}/Z_{\rm eff}$ where $H_{\rm eff}(\bm{\sigma})$ is equal to
the terms in the round brackets in the second line of (54). Inserting (59) and
(60) in $H_{\rm eff}(\bm{\sigma})$, we obtain

$-\beta^{{}^{\prime}}H_{\rm
eff}(\bm{\sigma})=\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma}{2}\frac{P(1+\sqrt{\rho_{P}})^{2}}{2(1+\rho)^{P}}\sum_{a,b}[q_{ab}^{*}]^{P-1}\sigma^{(a)}\sigma^{(b)}+\beta^{{}^{\prime}}\dfrac{P}{2}(1+\rho)^{P/2-1}\sum_{a}\dfrac{1}{rM}\sum_{A}\eta^{1,A}[(n_{1,A}^{(a)})^{*}]^{P-1}\sigma^{(a)}$
(61)

and, substituting in (56)

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}^{(P)}(J)=$
$\displaystyle\lim_{n\to
0}\dfrac{1}{n}\left\\{\dfrac{{\beta^{{}^{\prime}}}^{2}(1+\sqrt{\rho_{P}})^{2}\gamma}{4(1+\rho)^{P}}\sum_{a,b}(P-1)[q_{ab}^{*}]^{P}+\frac{\beta^{{}^{\prime}}(1+\rho)^{P/2}(P-1)}{2M}\sum_{a,A}[(n_{1,A}^{(a)})^{*}]^{P}\right.$
$\displaystyle+\left.\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\log\left[\sum_{\bm{\sigma}}\exp\left(-\beta^{{}^{\prime}}H_{\rm
eff}+J\displaystyle\sum\limits_{a}\xi_{i}^{1}\sigma_{i}^{(a)}\right)\right]\right\\}\,.$
(62)

where $q_{ab}^{\star}$ and $[n_{1,A}^{(a)}]^{*}$ must be determined from (57),
(58). Since $p_{\rm eff}(\bm{\sigma})$ depends on $\bm{Q}$, $\bm{N}$ via (61),
Eqs. (57) and (58) denote a set of self-consistency equations. Now, in order
to proceed, we need to find the form of ${\bm{Q}}$ and $\bm{N}$ in the limit
$n\to 0$. We will make the $1$RSB ansatz :

$\displaystyle q_{ab}$
$\displaystyle=\delta_{ab}+(1-\delta_{ab})(\theta\bar{q}_{1}+(1-\theta)\bar{q}_{2}),$
(63) $\displaystyle n_{1,A}^{(a)}$ $\displaystyle=\bar{n},$ (64)

Using (63) to evaluate the first term in (62) and the first term in (61), we
get

$\displaystyle\sum_{ab}q_{ab}^{P}=$ $\displaystyle
n\left(1+n\theta\bar{q}_{1}^{P}-\theta\bar{q}_{1}^{P}-\bar{q}_{2}^{P}+n\bar{q}_{2}^{P}+\theta\bar{q}_{2}^{P}-n\theta\bar{q}_{2}^{P}\right)$
(65) $\displaystyle\sum_{ab}q_{ab}^{P-1}\sigma^{(a)}\sigma^{(b)}=$
$\displaystyle\frac{1}{2}\left[2(1-\bar{q}_{2}^{P-1})n+(\bar{q}_{2}^{P-1}-\bar{q}_{1}^{P-1})\sum_{k=1}^{n/\theta}\left(\sum_{a=1}^{\theta}\sigma^{((k-1)\theta+a)}\right)^{2}+\bar{q}_{1}^{P-1}\left(\sum_{a}\sigma^{(a)}\right)^{2}\right].$
(66)

Upon inserting in (62) we obtain

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}^{(P)}(J)=\dfrac{{\beta^{{}^{\prime}}}^{2}(1+\sqrt{\rho_{P}})^{2}\gamma}{4(1+\rho)^{P}}[1-(P-1)\theta\left(\bar{q}_{2}^{P}-\bar{q}_{1}^{P}\right)+(P-1)\bar{q}_{2}^{P}]-\frac{\beta^{{}^{\prime}}(1+\rho)^{P/2}(P-1)}{2}\bar{n}^{P}$
$\displaystyle-\frac{(\beta^{{}^{\prime}})^{2}(1+\sqrt{\rho_{P}})^{2}\gamma
P}{4(1+\rho)^{P}}\bar{q}_{2}^{P-1}+\lim_{n\to
0}\frac{1}{n}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\log\left[\sum_{\bm{\sigma}}\exp\left(-\beta^{{}^{\prime}}H_{\rm
eff}(\bm{\sigma})+J\displaystyle\sum\limits_{a}\xi^{1}\sigma^{(a)}\right)\right]\,$
(67)

with

$\displaystyle-\beta^{{}^{\prime}}H_{\rm eff}(\bm{\sigma})$ $\displaystyle=$
$\displaystyle{\beta^{{}^{\prime}}}^{2}\gamma\frac{P(1+\sqrt{\rho_{P}})^{2}}{4(1+\rho)^{P}}\left[(\bar{q}_{2}^{P-1}-\bar{q}_{1}^{P-1})\sum_{k=1}^{n/\theta}\left(\sum_{b=1}^{\theta}\sigma^{((k-1)\theta+b)}\right)^{2}+\bar{q}_{1}^{P-1}\left(\sum_{a=1}^{n}\sigma^{(a)}\right)^{2}\right]$
(68)
$\displaystyle+\beta^{{}^{\prime}}\dfrac{P}{2}(1+\rho)^{P/2-1}\hat{\eta}_{M}\left(\sum_{a=1}^{n}\sigma^{(a)}\right)\bar{n}^{P-1}$

where we have used the definition (36).

Next, we apply a Gaussian transformation to the last term in (67) to linearize
the squared terms in the Hamiltonian (68)

$\displaystyle\lim_{n\to
0}\dfrac{1}{n}\ln\left[\sum_{\bm{\sigma}}\exp\Big{(}-\beta^{{}^{\prime}}H_{\rm
eff}(\bm{\sigma})+J\displaystyle\sum\limits_{a}\xi^{1}\sigma^{(a)}\Big{)}\right]$
$\displaystyle=\lim_{n\to
0}\dfrac{1}{n}\ln\left\\{\int{\mathcal{D}}u\,\int\left[\prod_{k=1}^{n/\theta}{\mathcal{D}}v_{k}\right]\sum_{\bm{\sigma}}\exp\left[\frac{\beta^{{}^{\prime}}P(1+\rho)^{P/2-1}}{2}\hat{\eta}_{M}\bar{n}^{P-1}\,\displaystyle\sum\limits_{a=1}^{n}\sigma^{(a)}+J\displaystyle\sum\limits_{a=1}^{n}\xi^{1}\sigma^{(a)}\,\right.\right.$
$\displaystyle\left.\left.+\beta^{{}^{\prime}}u\,\displaystyle\sum\limits_{a=1}^{n}\sigma^{(a)}\sqrt{\gamma\frac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\frac{P}{2}\bar{q}_{1}^{P-1}}+\beta^{{}^{\prime}}\sum_{k=1}^{n/\theta}v_{k}\displaystyle\sum\limits_{b=1}^{\theta}\sigma^{((k-1)\theta+b)}\sqrt{\gamma\frac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\frac{P}{2}(\bar{q}_{2}^{P-1}-\bar{q}_{1}^{P-1})}\,\right]\right\\}$
(69)

where we have set ${\mathcal{D}}u=du\,e^{-u^{2}/2}/\sqrt{2\pi}$ and
${\mathcal{D}}v_{k}=dv_{k}\,e^{-v_{k}^{2}/2}/\sqrt{2\pi}$ for
$k=1,\ldots,n/\theta$. Now, writing

$\displaystyle\sum\limits_{a=1}^{n}\sigma^{(a)}=\displaystyle\sum\limits_{k=1}^{n/\theta}\displaystyle\sum\limits_{b=1}^{\theta}\sigma^{((k-1)\theta+b)},$

the sum over the spin configuration can be done explicitly, finding

$\displaystyle=\lim_{n\to
0}\dfrac{1}{n}\ln\left\\{\int{\mathcal{D}}u\,\prod_{k=1}^{n/\theta}\int{\mathcal{D}}v_{k}\,\prod_{b=1}^{\theta}\sum_{\sigma^{((k-1)\theta+b)}}\exp\left(\frac{\beta^{{}^{\prime}}P(1+\rho)^{P/2-1}\hat{\eta}_{M}\bar{n}^{P-1}\sigma^{((k-1)\theta+b)}}{2}\,\right.\right.$
$\displaystyle\hskip
28.45274pt+J\xi^{1}\sigma^{((k-1)\theta+b)}+\beta^{{}^{\prime}}u\,\sigma^{((k-1)\theta+b)}\sqrt{\gamma\frac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\frac{P}{2}\bar{q}_{1}^{P-1}}$
$\displaystyle\hskip
28.45274pt\left.\left.+\beta^{{}^{\prime}}v_{k}\sigma^{((k-1)\theta+b)}\sqrt{\gamma\frac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\frac{P}{2}(\bar{q}_{2}^{P-1}-\bar{q}_{1}^{P-1})}\,\right)\right\\}$
$\displaystyle=\lim_{n\to
0}\dfrac{1}{n}\ln\left\\{\int{\mathcal{D}}u\,\prod_{k=1}^{n/\theta}\left[\int{\mathcal{D}}v_{k}2^{\theta}\cosh^{\theta}\left(\frac{\beta^{{}^{\prime}}P(1+\rho)^{P/2-1}}{2}\hat{\eta}_{M}\bar{n}^{P-1}+J\xi^{1}\,\right.\right.\right.$
$\displaystyle\hskip
28.45274pt\left.\left.\left.+\beta^{{}^{\prime}}u\,\sqrt{\gamma\frac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\frac{P}{2}\bar{q}_{1}^{P-1}}+\beta^{{}^{\prime}}v_{k}\sqrt{\gamma\frac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\frac{P}{2}\left(\bar{q}_{2}^{P-1}-\bar{q}_{1}^{P-1}\right)}\,\right)\right]\right\\}$

Now applying the $n\to 0$ limit SK1975  and exploiting the relation
$\lim\limits_{n\to
0}\dfrac{1}{n}\ln\mathbb{E}_{Y^{(1)}}[f^{n}(Y^{(1)})]=\mathbb{E}_{Y^{(1)}}[\ln
f(Y^{(1)})]$, we get

$\displaystyle=\lim_{n\to
0}\dfrac{1}{n}\ln\left\\{\mathbb{E}_{u}\left[\mathbb{E}_{v}\,2^{\theta}\cosh^{\theta}\left(\beta^{{}^{\prime}}u\,\sqrt{\gamma\frac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\frac{P}{2}\bar{q}_{1}^{P-1}}\,\right.\right.\right.$
$\displaystyle\left.\left.\left.{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}+\mathbb{E}_{\bm{\xi}^{1}}}+\beta^{{}^{\prime}}v\sqrt{\gamma\frac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\frac{P}{2}(\bar{q}_{2}^{P-1}-\bar{q}_{1}^{P-1})}\,+\frac{\beta^{{}^{\prime}}P(1+\rho)^{P/2-1}}{2}\hat{\eta}_{M}\bar{n}^{P-1}+J\xi^{1}\,\right)\right]^{n/\theta}\right\\}$
$\displaystyle=\dfrac{1}{\theta}\mathbb{E}_{u}\ln\left\\{\mathbb{E}_{v}\,2^{\theta}\cosh^{\theta}\left[\beta^{{}^{\prime}}\left(u\,\sqrt{\gamma\frac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\frac{P}{2}\bar{q}_{1}^{P-1}}+v\sqrt{\gamma\frac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\frac{P}{2}(\bar{q}_{2}^{P-1}-\bar{q}_{1}^{P-1})}\right.\right.\right.$
$\displaystyle\,\left.\left.\left.+\frac{P(1+\rho)^{P/2-1}\hat{\eta}_{M}}{2}\bar{n}^{P-1}+J\xi^{1}\,\right)\right]\right\\}$

Finally, inserting this expression in (67) and denoting with
$\mathbb{E}_{Y^{(1)}}[.]$ and $\mathbb{E}_{Y^{(2)}}[.]$ the Gaussian averages
w.r.t. $Y^{(1)}$ and $Y^{(2)}$ respectively, we reach the following expression
for the free energy

$\begin{array}[]{lll}-\beta^{{}^{\prime}}\mathcal{F}^{(P)}(J)=&\ln
2+\dfrac{1}{\theta}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\ln\mathbb{E}_{Y^{(2)}}\left\\{\cosh^{\theta}\left[J\xi^{1}+\beta^{\prime}\dfrac{P}{2}\bar{n}^{P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}\right.\right.\\\
\\\
&\left.\left.+Y^{(1)}\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\bar{q}_{1}^{{}^{P-1}}}+Y^{(2)}\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}(\bar{q}_{2}^{{}^{P-1}}-\bar{q}_{1}^{{}^{P-1}})}\right]\right\\}\\\
&-\dfrac{{\beta^{{}^{\prime}}}}{2}(P-1)(1+\rho)^{P/2}\bar{n}^{P}+\dfrac{{\beta^{{}^{\prime}}}^{2}}{4}\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\left[1-P\bar{q}_{2}^{P-1}+(P-1)\bar{q}_{2}^{P}\right]\\\
\\\
&-\dfrac{{\beta^{{}^{\prime}}}^{2}}{4}\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}(P-1)\theta(\bar{q}_{2}^{P}-\bar{q}_{1}^{P}),\end{array}$

where $\bar{n},\bar{q}_{1}$ and $\bar{q}_{2}$ must fulfill the same self-
consistency equations (37) obtained by using Guerra’s interpolation technique,
as reported in Proposition 1.

###  2.4 Results

Figure 1: Phase diagram of the unsupervised DAM model with $P=4$, in the
parameters space ($\gamma$, $\tilde{\beta}$), where $\gamma$ is the storage
load defined in (8) and $\tilde{\beta}=\beta^{{}^{\prime}}/(1+\rho)^{P/2}$ is
the scaled inverse noise, where $\rho$ is the dataset entropy defined in (3).
Different panels refer to different sizes $M$ of the training set, as shown in
the legend, where $\tilde{M}(r,P)=r^{-2P}$ and $r$ has been set to $r=0.2$.
For high values of the noise $\tilde{\beta}^{-1}$, the system is ergodic,
while lowering the noise two phases appear: a retrieval phase at small
$\gamma$ (below the black line), and a spin glass phase (below the grey line)
at high values of $\gamma$. The spin-glass phase within the retrieval region
(delimited by the dashed line) is unstable. As $M$ increases, the spin glass
region shrinks and the retrieval region expands. Figure 2: Phase diagrams of
the unsupervised DAMs model for $P=6,8,10$, and different values of $M$, as
shown in the legends. $\tilde{M}(r,P)$ and $r$ are as in Figure 1. As $P$
increases, the spin glass region shrinks and the retrieval region expands.

Solving numerically the self consistency equations (37), we obtain the phase
diagram shown in Figure 1, in the parameters space ($\gamma$,
$\tilde{\beta}$), where $\gamma$ is the storage load defined in (8) and
$\tilde{\beta}=\beta^{{}^{\prime}}/(1+\rho)^{P/2}$ is a scaled inverse
temperature, where $\rho$ is the dataset entropy defined in (3). Panels show
results for $P=4$ and three different values of $M$, as shown in the legend.
The result for $M\to\infty$ follows from the analysis carried out in Sec
2.4.1. In each panel, the grey curve marks the transition from the ergodic
phase with $\bar{m}=\bar{q}_{1}=\bar{q}_{2}=0$ (above) to the spin glass
phase, where either $\bar{q}_{1}$ or $\bar{q}_{2}$ becomes non-zero (below).
The black curve marks the transition from the retrieval phase with
$\bar{m}\neq 0$ (left) to $\bar{m}=0$ (right). The spin glass solution within
the retrieval region is always unstable and it is delimited by the dotted
curve (we refer to this as the instability region). In Figure 2 we show the
results for $P=6,8,10$, as shown in the legend. As $P$ and $M$ grow, the spin
glass region shrinks and the retrieval region expands. For $M=+\infty$ the
critical storage lines become equal to those of traditional DAM models where
archetypes are encoded in the interactions, rather than their noisy examples
Albanese2021 . This is as expected: when an infinite number of examples is
provided, the system reaches the same performance as a neural network where
archetypes are stored in the interactions directly. For finite $M$, however,
the network’s ability to retrieve the archetypes degrades at values of the
storage load which are lower than the storage capacity in traditional DAM
models. Both Figures 1 and 2 have been obtained for the value of
$\theta=\theta_{\otimes}$ which maximizes the retrieval region, as shown in
Figure 3 (left panel). In the latter, we show the line that separates the
retrieval region from the spin-glass or the ergodic region, for different
values of $\theta$. For $\theta=0$ (which corresponds to the RS theory), the
line exhibits a re-entrance, as commonly observed in spin-glasses and
associative memories SK1975 ; AGS-1987 ; Crisanti-RSB ; Albanese2021 . As
$\theta$ is increased, the retrieval region expands, reaching its maximum at
$\theta=\theta_{\otimes}$, where the re-entrance disappears completely,
showing the same qualitative behaviour as in the Hopfield model AGS  and
traditional dense associative memories Albanese2021 . Increasing $\theta$
above $\theta_{\otimes}$ the re-entrance appears again and gets more
pronounced as $\theta$ approaches $1$, where the transition line becomes
identical to the RS (and the $\theta=0$) case, as expected. In Figure 3 (right
panel) we show the dependence of $\theta_{\otimes}$ on $P$.

Figure 3: Left: Critical line separating the retrieval (left) from the spin-
glass or ergodic region (right) for different values of $\theta$. For
$\theta=0$, where the theory is RS, the retrieval region exhibits a re-
entrance.As $\theta$ is increased the retrieval region expands, reaching its
maximum at $\theta=\theta_{\otimes}$, where the re-entrance disappears
completely. Right: $\theta_{\otimes}$ versus $P$. As $P$ increases the value
of $\theta_{\otimes}$ decreases, indicating that the larger $P$ the better the
RS approximation is expected to be.

As $P$ increases, the value of $\theta_{\otimes}$ decreases. This is as
expected from spin-glass theory, as for large $P$, $P$-spin models are known
to converge to the REM model derrida1981random , which is RS. Interestingly,
we find that $\theta_{\otimes}$ takes the same value as in the DAM model
considered in Albanese2021 , for all values of $M$, hence the optimal $1$RSB
breaking parameter is not influenced by the use of corrupted examples rather
than archetypes.

####  2.4.1 Limiting cases: $M\to\infty$ and $\beta\to\infty$

Finally, we consider two instructive limiting cases. One is the limit
$M\to\infty$, where the number of available examples is large. Although
idealised, this scenario is becoming less utopian nowadays in a number of
applications, and it is instructive as an explicit relation between the
archetype magnetization and the mean magnetization of the examples naturally
emerges. The second scenario is the zero noise limit $\beta\to\infty$, where
the information processing capabilities of the network are expected to be
maximal. We now state the following

######  Corollary 1.

In the limit $M\to\infty$, the 1RSB self consistent equations for the order
parameters of the unsupervised DAM model are

$\displaystyle\bar{n}$
$\displaystyle=\dfrac{\bar{m}}{1+\rho}+\beta^{\prime}\dfrac{P}{2}\rho(1+\rho)^{P/2-2}\left[1+(\theta-1)\bar{q}_{2}-\theta\bar{q}_{1}\right]\bar{n}^{{}^{P-1}},$
$\displaystyle\bar{q}_{1}$
$\displaystyle=\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})\tanh
g(\beta,\gamma,\bm{Y})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})}\right]^{2},$
$\displaystyle\bar{q}_{2}$
$\displaystyle=\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})\tanh^{2}g(\beta,\gamma,\bm{Y})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})}\right],$
(70) $\displaystyle\bar{m}$
$\displaystyle=\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})\tanh
g(\beta,\gamma,\bm{Y})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})}\right].$

where

$\displaystyle g(\beta,\gamma,\bm{Y})=$
$\displaystyle\beta^{\prime}\dfrac{P}{2}\bar{n}^{{}^{P-1}}(1+\rho)^{P/2-1}+Y^{(2)}\beta^{{}^{\prime}}\sqrt{\gamma\dfrac{P}{2}\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}(\bar{q}_{2}^{{}^{P-1}}-\bar{q}_{1}^{{}^{P-1}})}$
$\displaystyle+\beta^{\prime}Y^{(1)}\sqrt{\rho\left(\dfrac{P}{2}\bar{n}^{{}^{P-1}}(1+\rho)^{P/2-1}\right)^{2}+\gamma\dfrac{P}{2}\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\,\bar{q}_{1}^{{}^{P-1}}\;}$
(71)

and $\beta^{{}^{\prime}}=2\beta/P!$.

###### Proof.

In the limit $M\to\infty$, we can apply the CLT to the sum of the examples
defined in (36), to write

$\hat{\eta}_{\mbox{\tiny$M$}}\sim\xi^{1}(1+\lambda\sqrt{\rho}),$ (72)

where $\lambda$ is a standard Gaussian variable $\lambda\sim\mathcal{N}(0,1)$.

Inserting (72) in (38) and in the expression for $\bar{n}$ provided in (37),
we get

$\displaystyle g(\beta,\gamma,\bm{Y},\lambda,\xi^{1})=$
$\displaystyle\beta^{\prime}\dfrac{P}{2}\bar{n}^{P-1}(1+\rho)^{P/2-1}\xi^{1}(1+\sqrt{\rho}\lambda)+Y^{(1)}\beta^{\prime}\sqrt{\dfrac{\gamma
P(1+\sqrt{\rho_{P}})^{2}}{2(1+\rho)^{P}}\bar{q}_{1}^{{}^{P-1}}}$
$\displaystyle+Y^{(2)}\beta^{\prime}\sqrt{\dfrac{\gamma(1+\sqrt{\rho_{P}})^{2}P}{2(1+\rho)^{P}}(\bar{q}_{2}^{{}^{P-1}}-\bar{q}_{1}^{{}^{P-1}})}\,.$
(73)

and

$\bar{n}=\frac{1}{(1+\rho)}\mathbb{E}_{\xi^{1}}\mathbb{E}_{\lambda}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\lambda,\xi^{1})\tanh^{\theta}g(\beta,\gamma,\bm{Y},\lambda,\xi^{1})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\lambda,\xi^{1})}\cdot{\xi^{1}(1+\lambda\sqrt{\rho})}\right]$

Hence, by applying to the standard Gaussian variable $\lambda$ the Stein’s
lemma, which states that for a standard Gaussian variable $J\sim N(0,1)$ and a
generic function $f(J)$, for which the two expectations
$\mathbb{E}\left(Jf(J)\right)$ and $\mathbb{E}\left(\partial_{J}f(J)\right)$
both exist, one has

$\mathbb{E}\left[Jf(J)\right]=\mathbb{E}\left[\frac{\partial f(J)}{\partial
J}\right],$ (74)

and explicitly averaging over $\xi$ we get the self-consistency equations in
(70).

∎

Finally, it has been shown in super , within a RS analysis, that neglecting
the second term in the equation for $\bar{n}$, given in (70), has negligible
impact on the solutions of the self-consistency equations, in the relevant
regime of low noise, where the network works as an associative memory. As the
equation for $\bar{n}$ is the same in the RS theory and in the $1$RSB theory
that we are considering here, the same truncation of $\bar{n}$ can be adopted
here

$\bar{n}=\dfrac{\bar{m}}{1+\rho}$ (75)

This leads to a simplified expression for $g$ that is

$\displaystyle g(\beta,\gamma,\bm{Y})=$
$\displaystyle\tilde{\beta}\dfrac{P}{2}\bar{m}^{{}^{P-1}}(1+\rho)^{P/2-1}+\tilde{\beta}Y^{(2)}\sqrt{\gamma\dfrac{P}{2}(1+\sqrt{\rho_{P}})^{2}(\bar{q}_{2}^{{}^{P-1}}-\bar{q}_{1}^{{}^{P-1}})}$
$\displaystyle+\tilde{\beta}Y^{(1)}\sqrt{\rho\left(\dfrac{P}{2}\bar{m}^{{}^{P-1}}\right)^{2}+\gamma\dfrac{P}{2}(1+\sqrt{\rho_{P}})^{2}\,\bar{q}_{1}^{{}^{P-1}}\;}\;.$
(76)

where we re-scaled the noise
$\tilde{\beta}=\beta^{{}^{\prime}}/(1+\rho)^{P/2}$.  Solving numerically the
self consistency equations (70) where the equation for $\bar{n}$ is replaced
with its truncated version (75), leads to the phase diagram shown by the lines
$M=+\infty$ in Figs. 1 and 2.
Next, we turn to the analysis of the ground state, i.e. to the limit
$\beta\to\infty$, and we state the following

######  Corollary 2.

In the limit $\beta\to\infty$ and for $M\gg 1$, the $1$RSB self consistency
equations for the order parameters of the unsupervised DAM model are

$\displaystyle\bar{m}$
$\displaystyle=1-2\mathbb{E}_{Y^{(1)}}\left[\dfrac{1}{1+\exp\left(2D(A_{1}+A_{2}Y^{(1)})\right)\mathcal{Q}(A)}\right]$
(77) $\displaystyle\Delta\bar{q}$
$\displaystyle=\bar{q}_{2}-\bar{q}_{1}=4\mathbb{E}_{Y^{(1)}}\left\\{\dfrac{\exp\left(2D(A_{1}+A_{2}Y^{(1)})\right)\mathcal{Q}(A)}{\left[1+\exp\left(2D(A_{1}+A_{2}Y^{(1)})\right)\mathcal{Q}(A)\right]^{2}}\right\\}$
(78)

where $D=\tilde{\beta}\theta$ and

$\begin{array}[]{lll}&\mathcal{Q}(A)=\dfrac{1+\mathrm{erf}(K^{+})}{1+\mathrm{erf}(K^{-})},&K^{\pm}=\dfrac{DA_{3}^{2}\pm(A_{1}+A_{2}Y^{(1)})}{A_{3}\sqrt{2}},\\\
\\\ &A_{1}=\dfrac{P}{2}\bar{m}^{P-1},&A_{2}=\sqrt{\rho
A_{1}^{2}+\gamma(1+\sqrt{\rho_{P}})^{2}\dfrac{P}{2}},\\\ \\\
&A_{3}=\sqrt{\gamma(1+\sqrt{\rho_{P}})^{2}\dfrac{P(P-1)}{2}\Delta\bar{q}}.\end{array}$
(79)

We report the proof of this Corollary in Appendix B.2.

Figure 4: Critical storage capacity $\gamma_{c}$ in the ground state for the
unsupervised DAM model, for example noise $r=0.2$, as a function of the ratio
$M/M_{\otimes}^{unsup}$ both for the RS (red line) and $1$RSB (blue line)
assumptions. We notice that $\gamma_{c}$ increases with $M$ and, for $M\gg
M_{\otimes}^{unsup}$, it saturates to the critical load ($\hat{\gamma}$) found
numerically for the RS case in EmergencySN .

Next we compute, by numerically solving (77)-(78), the ground-state critical
storage capacity $\gamma_{C}$ beyond which a black-out scenario emerges,
namely $\bar{m}\neq 0$ for $\gamma<\gamma_{C}$ and $\bar{m}=0$ for
$\gamma>\gamma_{C}$. In Fig. 4 we plot $\gamma_{C}$ as a function of the ratio
between the number $M$ of experienced examples and the minimum number
$M_{\otimes}$ of examples required by the unsupervised DAM model (with $P>2$)
to correctly learn and retrieve an archetype, within the RS theory, which has
been proved to be

$\displaystyle
M^{unsup}_{\otimes}(r,P,\gamma)=\gamma\dfrac{1}{P}\dfrac{1}{r^{2P}}.$ (80)

in unsup . In order to ascertain whether replica symmetry breaking alters such
threshold, we plot $\gamma_{C}$ within both the $1$RSB assumption (blue line)
and the RS theory (red line). We see that $\gamma_{C}$ becomes non-zero
(meaning that retrieval can occur) at the same value of $M=M_{\otimes}$ for
the RS and the $1$RSB theory. This shows that the minimum number of examples
that a network needs to accomplish retrieval of the archetype is the same
within the RS or the $1$RSB assumption.

Results show that, as in the standard Hebbian storage AABO-JPA2020 ; Crisanti-
RSB ; Steffan-RSB , the phenomenon of replica symmetry breaking induces a mild
improvement in terms of the maximal storage.

##  3 ‘Supervised’ dense associative memories

In this section we analyse the information processing capabilities of the DAM
model in the so-called ‘supervised’ setting, where the dataset given to the
network is now split into different categories, one for each archetype
$\bm{\xi}^{\mu}$, with $\mu=1\ldots P$. Again, we analyze the model, defined
in Sec. 3.1, via both Guerra’s interpolation techniques (Sec. 3.2) and
Parisi’s replica approach (Sec. 3.3), at the first step of replica-symmetry
breaking. In Appendix A we derive the instability line of the RS theory.

###  3.1 Model and definitions

As in the unsupervised DAM model, we consider a network of $N$ Ising neurons
$\sigma_{i}\in\\{-1,+1\\}$, $i=1,\ldots,N$, interacting via $P$-node
interactions. We assume to have $K$ Rademacher archetypes
$\bm{\xi}^{\mu}\in\\{-1,+1\\}^{N}$, with $\mu=1,\ldots,K$, defined as
$N$-dimensional vectors with entries drawn randomly and independently from the
distribution (1). In addition, we assume to have $M$ examples
$\eta^{\mu,a}\in\\{-1,+1\\}^{N}$ for each archetype, which are corrupted
version of the archetypes, with entries distributed according to (2). We shall
refer to this model as the supervised DAM model.

######  Definition 6.

The Hamiltonian of the supervised DAM model is

$\displaystyle\mathcal{H}^{(P)}_{N}(\bm{\sigma}|\bm{\eta})=$
$\displaystyle-\dfrac{1}{\mathcal{R}^{P/2}N^{P-1}M^{P}}\displaystyle\sum\limits_{\mu=1}^{K}\displaystyle\sum\limits_{a_{1},\cdots,a_{P}=1}^{M,\cdots,M}\left(\displaystyle\sum\limits_{i_{1}<\cdots<i_{P}}^{N,\cdots,N}\eta^{\mu,a_{1}}_{i_{1}}\cdots\eta^{\mu,a_{P}}_{i_{P}}\sigma_{i_{1}}\cdots\sigma_{i_{P}}\right)$
(81)

where the constant $\mathcal{R}:=r^{2}+\frac{1-r^{2}}{M}$ in the denominator
of the r.h.s. is included for mathematical convenience and the factor
$N^{P-1}$ ensures the Hamiltonian to be ${\mathcal{O}}(N)$, as explained
previously for the unsupervised model, see Eq. (4).

We highlight the hidden role of a teacher that, before providing the dataset
to the network, has grouped examples pertaining to the same archetype together
(hence the proliferation of summations in the cost function (81) with respect
to (4)).

As before, we will focus (without loss of generality) on the ability of the
network to store and retrieve the first archetype $\bm{\xi}^{1}$, hence the
Mattis magnetization $m_{1}(\bm{\sigma})$ provided in (16) remains a relevant
order parameter. However, the set of order parameters for the examples,
previously given by $n_{1,a}(\mathbb{\sigma})$ with $a=1\ldots M$ (see (17))
have now to be substituted with a single order parameter

$n_{1}(\bm{\sigma})=\dfrac{r}{\mathcal{R}}\dfrac{1}{NM}\displaystyle\sum\limits_{i,a=1}^{N,M}\eta_{i}^{1,a}\sigma_{i}.$
(82)

Its probability distribution, in the thermodynamic limit, is assumed self-
averaging, namely

$\lim_{N\rightarrow+\infty}\mathbb{P}_{N}(n_{1}|\bm{\sigma})=\delta(n_{1}(\bm{\sigma})-\bar{n}),$
(83)

as for the Mattis magnetization, while the distribution for the overlap
defined in (18) is still assumed bimodal as in Assumption (1) (see (20)). As
for the unsupervised model, we add an extra term in the cost function, namely
$J\sum_{i}\xi_{i}^{1}\sigma_{i}$, in order to generate the moments of the
Mattis magnetization $m_{1}$ by taking the derivatives of the quenched free
energy w.r.t. $J$ and, as this term is not part of the original Hamiltonian,
it will be set to zero at the end of the calculations. Therefore, we write the
partition function as

$\displaystyle\mathcal{Z}^{(P)}_{N}(\bm{\eta})$
$\displaystyle=\lim_{J\rightarrow
0}\mathcal{Z}^{(P)}_{N,K,M,\beta}(\bm{\eta};J)$
$\displaystyle=\lim_{J\rightarrow
0}\sum_{\bm{\sigma}}\exp\left\\{J\sum_{i}\xi_{i}^{1}\sigma_{i}+\beta^{\prime}\dfrac{N}{2}(1+\rho)^{P/2}n_{1}^{P}(\bm{\sigma})\right.$
$\displaystyle\left.+\dfrac{\beta^{\prime}P!}{2N^{P-1}\mathcal{R}^{P/2}}\displaystyle\sum\limits_{\mu>1}^{K}\displaystyle\sum\limits_{i_{1}<\cdots<i_{P}}^{N,\cdots,N}\left[\left(\dfrac{1}{M}\displaystyle\sum\limits_{a_{1}=1}^{M}\eta_{i_{1}}^{\mu,a_{1}}\right)\ldots\left(\dfrac{1}{M}\displaystyle\sum\limits_{a_{P}=1}^{M}\eta_{i_{P}}^{\mu,a_{P}}\right)\right]\sigma_{i_{1}}\cdots\sigma_{i_{P}}\right\\}$
(84)

where $\rho$ is the dataset entropy defined in (3),
$\beta^{\prime}:=2\beta/P!$ and for the first pattern, $\mu=1$, we have used
the relation $P!\sum_{i_{1}<\cdots<i_{P}}=\sum_{i_{1},\cdots,i_{P}}$ and
neglected terms which vanish in the thermodynamic limit. Next, we apply the
CLT to the variables in the round brackets in (84), and rewrite each term as

$\dfrac{1}{M}\displaystyle\sum\limits_{a=1}^{M}\eta^{\mu,a}_{i}=r\xi_{i}^{\mu}\Big{(}1+\phi_{i}^{\mu}\sqrt{\rho}\Big{)}\;\;\;\;\mathrm{with}\;\;\;\;\phi_{i}^{\mu}\sim\mathcal{N}(0,1)\,.$
(85)

Replaicing the previous in the expression inside the square brackets in (84)
we obtain

$\displaystyle\sum\limits_{\mu>1}^{K}\left[\left(\dfrac{1}{M}\displaystyle\sum\limits_{a_{1}=1}^{M}\eta_{i_{1}}^{\mu,a_{1}}\right)\ldots\left(\dfrac{1}{M}\displaystyle\sum\limits_{a_{P}=1}^{M}\eta_{i_{P}}^{\mu,a_{P}}\right)\right]=r^{P}\displaystyle\sum\limits_{\mu>1}^{K}\xi_{i_{1}}^{\mu}\ldots\xi_{i_{P}}^{\mu}\left(1+\phi_{i_{1}}^{\mu}\sqrt{\rho}\right)\ldots\left(1+\phi_{i_{P}}^{\mu}\sqrt{\rho}\right)$

and apply again the CLT now over the $\mu$-summation, we can write

$\frac{1}{K}\displaystyle\sum\limits_{\mu>1}^{K}\left[\left(\dfrac{1}{M}\displaystyle\sum\limits_{a_{1}=1}^{M}\eta_{i_{1}}^{\mu,a_{1}}\right)\ldots\left(\dfrac{1}{M}\displaystyle\sum\limits_{a_{P}=1}^{M}\eta_{i_{P}}^{\mu,a_{P}}\right)\right]=\lambda_{i_{1},\ldots,i_{P}}\sqrt{\dfrac{r^{2P}(1+\rho)^{P}}{K}}$

where $\lambda_{i_{1},\ldots,i_{P}}\sim\mathcal{N}(0,1)$.
Inserting all back in (84) we get

$\displaystyle\mathcal{Z}^{(P)}_{N,K,M,\beta}(\bm{\eta}^{1},\bm{\lambda};J)$
$\displaystyle=\exp\left\\{J\sum_{i}\xi_{i}^{1}\sigma_{i}+\beta^{\prime}\dfrac{N}{2}(1+\rho)^{P/2}n_{1}^{P}(\bm{\sigma})+\dfrac{\beta^{\prime}P!\sqrt{K}}{2N^{P-1}}\displaystyle\sum\limits_{i_{1}<\cdots<i_{P}}^{N,\cdots,N}\lambda_{i_{1},\ldots,i_{P}}\sigma_{i_{1}}\cdots\sigma_{i_{P}}\right\\}\,.$
(86)

In the next subsection we calculate the free energy of the system in the
thermodynamic limit using Guerra’s interpolation technique, assuming one step
of replica symmetry breaking (1RSB).

###  3.2 1RSB analysis via Guerra’s interpolation technique

As for the unsupervised case, the plan is to construct an interpolation
between the original model and a simpler one-body model, whose statistical
features are as close as possible to the original one, then solve the one body
model and finally obtain the solution of the original model via the
fundamental theorem of calculus. Thus we define the following interpolating
partition function

######  Definition 7.

Given the interpolating parameter $t\in[0,1]$, $A_{1},\ A_{2},\
\psi\in\mathbb{R}$ constants to be set a posteriori, and the i.i.d. standard
Gaussian variables $Y_{i}^{(b)}\sim\mathcal{N}(0,1)$ for $i=1,\ldots,N$ and
$b=1,2$, the Guerra’s 1-RSB interpolating partition function for the DAM
model, trained by a teacher, is given by

$\begin{array}[]{lll}\mathcal{Z}^{(P)}_{2}(\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)&\coloneqq\displaystyle\sum\limits_{\\{\bm{\sigma}\\}}\mathcal{B}_{2}^{(P)}(\bm{\sigma}|\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)\\\
\\\
&=\displaystyle\sum\limits_{\\{\bm{\sigma}\\}}\exp{\Bigg{[}}J\sum_{i=1}^{N}\xi_{i}^{1}\sigma_{i}+\dfrac{t\beta^{\prime}N}{2M}\left(1+\rho\right)^{P/2}n_{1}^{{}^{P}}(\bm{\sigma})+\psi(1-t)Nn_{1}(\bm{\sigma})\\\
\\\
&+\sqrt{t}\dfrac{\beta^{\prime}\sqrt{K}}{N^{P-1}}\displaystyle\sum\limits_{{i_{1}<\cdots<i_{P}}}^{N,\cdots,N}\lambda_{i_{1},\ldots,i_{P}}\sigma_{{}_{i_{1}}}\cdots\sigma_{{}_{i_{{}_{P}}}}+\sqrt{1-t}\displaystyle\sum\limits_{b=1}^{2}A_{b}\displaystyle\sum\limits_{i=1}^{N}Y_{i}^{(b)}\sigma_{i}\Bigg{]}.\end{array}$
(87)

where
$\mathcal{B}_{2}^{(P)}(\bm{\sigma}|\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)$ is
denoted as Boltzmann factor.

As before, the introduction of the interpolating partition function gives rise
to a generalized measure, average and interpolating quenched free energy (that
we do not repeat here). All these generalizations retrieve the standard
definitions when evaluated at $t=1$. Following Guerra’s method guerra_broken ,
we must average out the fields ${\bf Y}^{(1)}$ and ${\bf Y}^{(2)}$
recursively, in the interpolating free energy resulting from the interpolating
partition function (87), as already done in the previous Section for the
unsupervised case, see Eqs. (26) (27) and (28). Proceeding in the same way and
omitting obvious details due to the similarity of the proofs in the supervised
and unsupervised cases, we state directly the next

######  Proposition 2.

In the thermodynamic limit $N\to\infty$, within the $1$RSB assumption and
under the Assumption 2, the quenched free energy for the supervised DAM model,
reads as

$\begin{array}[]{lll}-\beta^{{}^{\prime}}\mathcal{F}^{(P)}(J)&=\ln
2+\dfrac{1}{\theta}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\ln\mathbb{E}_{Y^{(2)}}\left\\{\cosh^{\theta}\left[J\xi^{1}+\beta^{\prime}\dfrac{P}{2}\bar{n}^{P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}\right.\right.\\\
\\\
&\left.\left.+Y^{(1)}\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\bar{q}_{1}^{{}^{P-1}}}+Y^{(2)}\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\left(\bar{q}_{2}^{{}^{P-1}}-\bar{q}_{1}^{{}^{P-1}}\right)}\right]\right\\}+\dfrac{{\beta^{{}^{\prime}}}^{2}}{4}\gamma\left[1-P\bar{q}_{2}^{P-1}+(P-1)\bar{q}_{2}^{P}\right]\\\
\\\
&-\dfrac{{\beta^{{}^{\prime}}}}{2}(P-1)(1+\rho)^{P/2}\bar{n}^{P}-\dfrac{{\beta^{{}^{\prime}}}^{2}}{4}\gamma(P-1)\theta(\bar{q}_{2}^{P}-\bar{q}_{1}^{P}),\end{array}$
(88)

with $\bar{n}$, $\bar{q}_{1},\ \bar{q}_{2}$ fulfilling the following self-
consistency equations

$\begin{array}[]{lll}\bar{n}=\dfrac{1}{1+\rho}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})\tanh
g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}\dfrac{1}{r}\hat{\chi}_{\mbox{\tiny$M$}}\right],\\\
\\\
\bar{q}_{1}=\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})\tanh
g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}\right]^{2},\\\
\\\
\bar{q}_{2}=\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})\tanh^{2}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}\right],\end{array}$
(89)

where

$\displaystyle g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})=$
$\displaystyle\beta^{\prime}\dfrac{P}{2}\bar{n}^{P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}+Y^{(1)}\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\bar{q}_{1}^{{}^{P-1}}}+Y^{(2)}\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}(\bar{q}_{2}^{{}^{P-1}}-\bar{q}_{1}^{{}^{P-1}})}$
(90)

and $\beta^{{}^{\prime}}=2\beta/P!$. Furthermore, as
$\bar{m}=-\beta^{{}^{\prime}}\nabla_{J}\mathcal{F}^{(P)}(J)|_{J=0}$, we have

$\displaystyle\bar{m}=\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})\tanh
g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y},\bm{\eta}^{1})}\xi^{1}\right].$
(91)

The proof of the aforementioned proposition is lengthy but the steps to follow
are identical to the ones provided for the unsupervised case.

###  3.3 1RSB analysis via Parisi’s replica trick

As stated in the previous Section, the core of this approach consists in
writing the logarithm of the partition function as $\lim\limits_{n\to
0}(\mathbb{E}\mathcal{Z}(J)^{n}-1)/n$ where

$\displaystyle\mathbb{E}\mathcal{Z}_{N}^{n}(J)=$
$\displaystyle\mathbb{E}\sum_{\bm{\sigma}^{1}\ldots\bm{\sigma}^{n}}\exp\left[\dfrac{\beta^{{}^{\prime}}P!}{2\mathcal{R}^{P/2}N^{P-1}M^{P}}\displaystyle\sum\limits_{a=1}^{n}\displaystyle\sum\limits_{\mu=1}^{K}\left(\displaystyle\sum\limits_{i_{1}<\cdots<i_{P}}^{N,\ldots,N}\displaystyle\sum\limits_{A_{1},\ldots,A_{P}}^{M,\ldots,M}\eta^{\mu,A_{1}}_{i_{1}}\cdots\eta^{\mu,A_{P}}_{i_{P}}\sigma_{i_{1}}^{(a)}\cdots\sigma_{i_{P}}^{(a)}\right)\right.$
$\displaystyle\left.+J\displaystyle\sum\limits_{a,i}^{n,N}\xi_{i}^{1}\sigma_{i}^{(a)}\right].$
(92)

Proceeding as in the unsupervised case, we compute separately the signal term
and the noise term. The former, accounting for the examples pertaining to the
first archetype, reads as

$\displaystyle\mathbb{E}\mathcal{Z}_{signal}^{n}(J)=$
$\displaystyle\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\sum_{\bm{\sigma}^{1}\ldots\bm{\sigma}^{n}}\exp\left[\dfrac{\beta^{{}^{\prime}}N(1+\rho)^{P/2}}{2}\displaystyle\sum\limits_{a=1}^{n}\left(\dfrac{r}{NM\mathcal{R}}\displaystyle\sum\limits_{i}^{N}\displaystyle\sum\limits_{A}^{M}\eta^{1,A}_{i}\sigma_{i}^{(a)}\right)^{P}\right.$
$\displaystyle\left.+J\displaystyle\sum\limits_{a=1}^{n}\displaystyle\sum\limits_{i=1}^{N}\xi_{i}^{1}\sigma_{i}^{(a)}\right],$
(93)

while the latter accounting for the examples pertaining to all the other
archetypes but the first one, can be written as

$\displaystyle\mathbb{E}\mathcal{Z}_{noise}^{n}$
$\displaystyle=\mathbb{E}_{\bm{\lambda}}\sum_{\bm{\sigma}^{1}\ldots\bm{\sigma}^{n}}\exp\left[\dfrac{\beta^{{}^{\prime}}P!\sqrt{K}}{2N^{P-1}}\displaystyle\sum\limits_{a=1}^{n}\left(\displaystyle\sum\limits_{i_{1}<\cdots<i_{P}}^{N,\ldots,N}\lambda_{i_{1},\ldots,i_{P}}\sigma_{i_{1}}^{(a)}\cdots\sigma_{i_{P}}^{(a)}\right)\right]$
(94)

where now $\mathbb{E}_{\bm{\lambda}}$ is the Gaussian average w.r.t.
$\bm{\lambda}$. Performing the integration over $\bm{\lambda}$ and inserting
the definitions of the order parameters as done for the unsupervised setting,
we can write

$\displaystyle\mathbb{E}\mathcal{Z}_{N}^{n}(J)=\int\prod_{a}dn_{1}^{(a)}\prod_{a}\dfrac{dz_{1}^{(a)}}{2\pi}\int\prod_{a,b}dq_{ab}\int\prod_{a,b}\dfrac{dp_{ab}}{2\pi}\exp\left(-NA[\bm{Q},\bm{P},\bm{N},\bm{Z}]\right)$
(95)

where

$\displaystyle A[\bm{Q},\bm{P},\bm{N},\bm{Z}]$
$\displaystyle=-\dfrac{i}{N}\sum_{a,b}p_{ab}q_{ab}-\dfrac{1}{4}{\beta^{{}^{\prime}}}^{2}\gamma\displaystyle\sum\limits_{a,b}^{n,n}q_{ab}^{P}-\dfrac{\beta^{{}^{\prime}}(1+\rho)^{P/2}}{2}\sum_{a}(n_{1}^{(a)})^{P}-\dfrac{i}{N}\sum_{a}n_{1}^{(a)}z_{1}^{(a)}$
$\displaystyle-\mathbb{E}\log\left[\sum_{\bm{\sigma}}\exp\left(-\dfrac{i}{N}\sum_{a,b}p_{ab}\sigma^{(a)}\sigma^{(b)}-\dfrac{i}{N(1+\rho)}\sum_{a}\hat{\eta}_{{}_{M}}z_{1}^{(a)}\sigma^{(a)}+J\displaystyle\sum\limits_{a}\xi^{1}\sigma^{(a)}\right)\right].$
(96)

and $\hat{\eta}_{M}=(rM)^{-1}\sum_{A=1}^{M}\eta^{1,A}$. Assuming (as usual
within the replica method) that the limits $N\to\infty$ and $n\to 0$ can be
interchanged, the integrals can be performed by steepest descent. This leads
to

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}^{(P)}(J)=\lim_{n\to
0}\dfrac{1}{n}A[\bm{Q}^{\star},\bm{P}^{\star},\bm{Z}^{\star},\bm{N}^{\star}]$
(97)

where $\bm{Q}^{\star},\bm{P}^{\star},\bm{Z}^{\star},\bm{N}^{\star}$, are
solution of the saddle point equations

$\displaystyle\dfrac{\partial A}{\partial p_{ab}}=0$
$\displaystyle\quad\Rightarrow\quad
q_{ab}^{*}=\langle\sigma^{a}\sigma^{b}\rangle_{\rm eff}$ (98)
$\displaystyle\dfrac{\partial A}{\partial z_{1}^{(a)}}=0$
$\displaystyle\quad\Rightarrow\quad(n_{1}^{(a)})^{*}=\dfrac{r}{\mathcal{R}}\displaystyle\sum\limits_{A=1}^{M}\mathbb{E}\langle\eta^{1,A}\sigma^{a}\rangle_{\rm
eff}$ (99) $\displaystyle\dfrac{\partial A}{\partial q_{ab}}=0$
$\displaystyle\quad\Rightarrow\quad
p_{ab}^{*}=i\beta^{{}^{\prime}}\,{}^{2}\dfrac{P}{4}\gamma N(q_{ab}^{*})^{P-1}$
(100) $\displaystyle\dfrac{\partial A}{\partial n_{1}^{(a)}}=0$
$\displaystyle\quad\Rightarrow\quad(z_{1}^{(a)})^{*}=i\beta^{{}^{\prime}}\dfrac{P}{2}(1+\rho)^{P/2}N[(n_{1}^{(a)})^{*}]^{P-1}$
(101)

which provides a set of self-consistency equations for $\bm{Q}$, $\bm{N}$,
$\bm{P}$, and $\bm{Z}$. Using the last two equations to eliminate $\bm{P}$ and
$\bm{Z}$ from the description, we finally obtain

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}(J)$ $\displaystyle=\lim_{n\to
0}\dfrac{1}{n}\left\\{\dfrac{1}{4}{\beta^{{}^{\prime}}}^{2}\gamma(P-1)\displaystyle\sum\limits_{a,b}^{n,n}[q_{ab}^{*}]^{P}+\dfrac{\beta^{{}^{\prime}}(1+\rho)^{P/2}}{2}(P-1)\sum_{a}[(n_{1}^{(a)})^{*}]^{P}\right.$
$\displaystyle\left.-\mathbb{E}\log\left[\sum_{\bm{\sigma}}\exp\left(\beta^{{}^{\prime}}\,{}^{2}\dfrac{P}{4}\gamma\sum_{a,b}[q_{ab}^{*}]^{P-1}\sigma^{(a)}\sigma^{(b)}+\beta^{{}^{\prime}}\dfrac{P}{2}(1+\rho)^{P/2-1}\sum_{a}\hat{\eta}_{{}_{M}}[(n_{1}^{(a)})^{*}]^{P-1}\sigma^{(a)}\right.\right.\right.$
$\displaystyle\left.\left.\left.+J\displaystyle\sum\limits_{a}\xi^{1}\sigma^{(a)}\right)\right]\right\\}\,.$
(102)

To make progress, we need to find the form of ${\bm{Q}}$ and $\bm{N}$ in the
limit $n\to 0$. Again, we use the $1$RSB ansatz for the two-replica overlap
provided in (63), while we assume that $n_{1,a}$ is self-averaging, i.e.
$n_{1}^{(a)}=\bar{n}$. Proceeding analogously to the unsupervised case, after
taking the limit $n\to 0$ we find the same expression for the quenched free
energy (see (88)) and order parameters (see 89) previously obtained by using
Guerra’s interpolation technique.

###  3.4 Limiting cases: $M\to\infty$ and $\beta\to\infty$

Now we state the next corollaries concerning the large datasets $M\to\infty$
limit and the ground state $\beta\to\infty$ limit. We omit their proofs since
these are trivial variations of those provided for the unsupervised case (cfr.
Corollaries 1 and 2).

######  Corollary 3.

In the large dataset limit $M\to\infty$, the 1RSB self-consistency equations
for the order parameters of the supervised DAM model can be expressed as

$\displaystyle\bar{n}$
$\displaystyle=\dfrac{\bar{m}}{1+\rho}+\beta^{\prime}\dfrac{P}{2}\rho(1+\rho)^{P/2-2}\left[1+(\theta-1)\bar{q}_{2}-\theta\bar{q}_{1}\right]\bar{n}^{{}^{P-1}},$
$\displaystyle\bar{q}_{1}$
$\displaystyle=\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})\tanh
g(\beta,\gamma,\bm{Y})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})}\right]^{2},$
$\displaystyle\bar{q}_{2}$
$\displaystyle=\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})\tanh^{2}g(\beta,\gamma,\bm{Y})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})}\right],$
(103) $\displaystyle\bar{m}$
$\displaystyle=\mathbb{E}_{{}_{1}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta,\gamma,\bm{Y})\tanh
g(\beta,\gamma,\bm{Y})}{\mathbb{E}_{Y^{(2)}}\cosh^{\theta}g(\beta^{{}^{\prime}},\gamma,\bm{Y})}\right].$

where the expression of $g$ reads as

$\displaystyle g(\beta,\gamma,\bm{Y})=$
$\displaystyle\beta^{\prime}\dfrac{P}{2}\bar{n}^{{}^{P-1}}(1+\rho)^{P/2-1}+Y^{(2)}\beta^{{}^{\prime}}\sqrt{\gamma\dfrac{P}{2}(\bar{q}_{2}^{{}^{P-1}}-\bar{q}_{1}^{{}^{P-1}})}$
$\displaystyle+Y^{(1)}\beta^{{}^{\prime}}\sqrt{\rho\left(\dfrac{P}{2}\bar{n}^{{}^{P-1}}(1+\rho)^{P/2-1}\right)^{2}+\gamma\dfrac{P}{2}\,\bar{q}_{1}^{{}^{P-1}}\;}\;.$
(104)

Moreover, if we use the truncated expression for $\bar{n}$, namely

$\bar{n}=\dfrac{\bar{m}}{1+\rho}$ (105)

we get the simplified expression of (104) that is

$\displaystyle g(\beta,\gamma,\bm{Y})=$
$\displaystyle\tilde{\beta}\dfrac{P}{2}\bar{m}^{{}^{P-1}}+Y^{(2)}\tilde{\beta}\sqrt{\gamma\dfrac{P}{2}(1+\rho)^{P}(\bar{q}_{2}^{{}^{P-1}}-\bar{q}_{1}^{{}^{P-1}})}$
$\displaystyle+Y^{(1)}\tilde{\beta}\sqrt{\rho\left(\dfrac{P}{2}\bar{m}^{{}^{P-1}}\right)^{2}+\gamma(1+\rho)^{P}\dfrac{P}{2}\,\bar{q}_{1}^{{}^{P-1}}\;}\;,$
(106)

where $\tilde{\beta}=\beta^{{}^{\prime}}/(1+\rho)^{P/2}$.

Figure 5: Phase Diagram of the supervised DAM model for different values of
the dataset entropy $\rho$, ranging from $\rho=0.1$ (light grey stars), $0.05$
(grey triangles) up to $0$ (black dots). The interpretation of the different
regions is the same as in the unsupervised model. As $\rho$ decrease (i.e. as
more information is provided to the network), the instability region shrinks
and the retrieval region expands, similarly to what was observed earlier upon
increasing $M$.

######  Corollary 4.

The ground-state (i.e. $\beta\to\infty$) self-consistency equations for the
order parameters of the theory in the large dataset limit (i.e. $M\gg 1$) and
under the $1$RSB assumption are

$\displaystyle\bar{n}$ $\displaystyle=\dfrac{\bar{m}}{(1+\rho)}$ (107)
$\displaystyle\bar{m}$
$\displaystyle=1-2\mathbb{E}_{Y^{(1)}}\left[\dfrac{1}{1+\exp\left(2D(A_{1}+A_{2}Y^{(1)})\right)\mathcal{Q}(A)}\right]$
(108) $\displaystyle\Delta\bar{q}$
$\displaystyle=\bar{q}_{2}-\bar{q}_{1}=4\mathbb{E}_{Y^{(1)}}\left\\{\dfrac{\exp\left(2D(A_{1}+A_{2}Y^{(1)})\right)\mathcal{Q}(A)}{\left[1+\exp\left(2D(A_{1}+A_{2}Y^{(1)})\right)\mathcal{Q}(A)\right]^{2}}\right\\}$
(109)

where $D=\tilde{\beta}\theta$ and

$\begin{array}[]{lll}&\mathcal{Q}(A)=\dfrac{1+\mathrm{erf}(K^{+})}{1+\mathrm{erf}(K^{-})},&K^{\pm}=\dfrac{DA_{3}^{2}\pm(A_{1}+A_{2}Y^{(1)})}{A_{3}\sqrt{2}},\\\
\\\
&A_{1}=\dfrac{P}{2}\bar{m}^{P-1},&A_{2}=\sqrt{\rho\left(\dfrac{P\bar{m}^{P-1}}{2}\right)^{2}+\gamma(1+\rho)^{P}\dfrac{P}{2}},\\\
\\\
&A_{3}=\sqrt{\gamma(1+\rho)^{P}\dfrac{P(P-1)}{2}\Delta\bar{q}}.\end{array}$
(110)

By numerically solving the self-consistency equations 103, we obtain the phase
diagram shown in Fig. 5, for different values of $P$ and different values of
the dataset entropy $\rho$, as shown in the legend.

Figure 6: Critical storage capacity $\gamma_{c}$ in the ground state for the
supervised DAM model for example noise $r=0.2$, as a function of the ratio
$M/M_{\otimes}^{sup}$, both for the RS (red line) and $1$RSB (blue line)
assumptions. Different panels show results for different values of $P=4,\ 6,\
8,\ 10$.

In super  it has been proven that the minimum number of examples
$M_{\otimes}(r,\gamma)$ required by the supervised DAM model to correctly
learn and retrieve an archetype, is given by

$\displaystyle
M_{\otimes}(r,\gamma)=\left(1+2\gamma\right)\dfrac{1}{r^{\mbox{\tiny$2$}}}$
(111)

where $\gamma\neq 0$. In Fig. 6 we plot the critical storage $\gamma_{C}$ in
the ground state versus the ratio $M/M_{\otimes}$. As previously observed for
the unsupervised setting, the phenomenon of replica symmetry breaking slightly
increases the critical storage and it does not alter the minimum number of
examples required for learning.

##  4 Conclusions and outlooks

This manuscript analyses the equilibrium behaviour of Dense Associative
Memories (DAM) trained with or without the supervision of a teacher, within a
$1$RSB assumption, thus extending previous analysis carried out at a RS level
super ; unsup . The unsupervised and supervised settings differ in the choice
of the couplings (which involve $P$ nodes) and are given by

$\displaystyle J^{unsup}_{i_{1}i_{2}...i_{P}}$
$\displaystyle\sim\frac{1}{MN^{P-1}}\sum_{\mu=1}^{K}\sum_{a=1}^{M}\eta_{i_{1}}^{\mu,a}...\>\eta_{i_{P}}^{\mu,a},$
(112) $\displaystyle J^{sup}_{i_{1}i_{2}...i_{P}}$
$\displaystyle\sim\frac{1}{M^{P}N^{P-1}}\sum_{\mu=1}^{K}\sum_{a_{1},\ldots,a_{P}=1}^{M,\ldots,M}\eta_{i_{1}}^{\mu,a_{1}}...\>\eta_{i_{P}}^{\mu,a_{P}}$
(113)

respectively, where
$\left\\{\bm{\eta}^{\mu}_{a}\right\\}_{a=1,...,M}^{\mu=1,...,K}$ are perturbed
versions of the unknown archetypes
$\left\\{\bm{\xi}^{\mu}\right\\}^{\mu=1,\ldots,K}$. The network does not
experience the archetypes directly, instead it has to infer them from the
supplied examples. For both the settings, we obtained explicit expressions for
the quenched free energy and derived full phase diagrams. In doing so, we
proved a full equivalence, at $1$RSB level, between two different approaches,
namely Guerra’s telescopic interpolation guerra_broken  and Parisi’s RSB
theory MPV . In addition, we derived (in Appendix A) the De Almeida-Thouless
line, which marks the onset of the instability of the replica symmetric
description, and below which the $1$RSB description should be preferred.

The main differences brought about by the RSB description, with respect to the
RS one, consist in the disappearance of the instability region within the
retrieval zone of the phase diagram close to saturation (as standard in glassy
statistical mechanics Amit ) and in a slight improvement of the value of the
critical storage. Importantly, the threshold for learning, both in the
supervised and unsupervised settings, is not influenced by replica symmetry
breaking, i.e. the minimum number of examples required to infer the archetypes
is the same in the RS and 1RSB description. Interestingly, the optimal value
of the Parisi’s parameter $\theta$, that controls the distribution of the
overlaps in the $1$RSB scenario, is not influenced by the dataset entropy and
is equal to that of the classical Hopfield model. From the mathematical
viewpoint, possible future developments would be relaxing the constraints of a
self-averaging Mattis magnetization and inspecting how the learning and
retrieval properties of these networks change with different kind of noise: in
this work we have focused on multiplicative noise, however the use of additive
noise has lately gained large popularity in generative models for machine
learning (see for instance MarcGiulio2023 ). Furthermore, within the framework
of multiplicative noise, an interesting outlook would be considering
corruptions of archetypes which also consist of blank (in addition to
inverted) entries, as done recently in AdriFede2023  for networks away from
saturation. Their operation in the saturated regime and the effects of replica
symmetry breaking have not yet been investigated: we plan to report soon on
these topics.

## Acknowledgements

All the authors acknowledge the stimulating research environment provided by
the Alan Turing Institute’s Theory and Methods Challenge Fortnights event
Physics-informed Machine Learning. Albanese acknowledges Ermenegildo Zegna
Founder’s Scholarship, UMI (Unione Matematica Italiana), INdAM – GNFM Project
(CUP E53C22001930001) and PRIN grant Stochastic Methods for Complex Systems n.
2017JFFHS for financial support and King’s College London for kind
hospitality. Alessandrelli acknowledges INdAM (Istituto Nazionale d’Alta
Matematica) and Unisalento for support via PhD-AI. Barra’s research is
supported by MAECI via the BULBUL grant (Italy-Israel collaboration), Project
n. F85F21006230001 Brain-inspired Ultra-fast and Ultra-sharp machines for
assisted healthcare and by MUR via the PRIN-2022 grant Statistical Mechanics
of Learning Machines: from algorithmic and information-theoretical limits to
new biologically inspired paradigms, Project n. 20229T9EAT that are gratefully
acknowledged.

##  Appendix A Instability of the RS solution: AT lines

In the main text we have analysed the equilibrium behaviour of supervised and
unsupervised DAM models under the assumption of replica symmetry breaking.
While such phenomenon is expected in these models, a formal proof that the
replica symmetric theory becomes unstable in certain ranges of the control
parameters has not been provided in the literature. In this section we provide
such a proof and we derive the critical line of the RS instability in the
phase diagram, for both the unsupervised and the supervised DAM models,
separately. We will use the method recently introduced in albanese2023almeida
, which provides a simple alternative to the method originally introduced by
de Almeida and Thouless in de1978stability , as it does not require to compute
the so-called replicon (i.e. the smallest eigenvalue of the spectrum) of the
Hessian of the quadratic fluctuations of the free-energy around its RS value
and it does not rely on the availability of an “ansatz-free” expression for
the free-energy.

###  A.1 AT line for DAM in unsupervised setting

Following the method introduced in albanese2023almeida , we aim to determine
the region in the phase diagram where the quenched free energy evaluated
within the $1$RSB approximation, that from now on we denote for convenience as
$\mathcal{F}^{(P)}_{1RSB}(\bar{n},\bar{q}_{2},\bar{q}_{1}|\theta)$, is smaller
than the free energy evaluated within the RS assumption,
$\mathcal{F}^{(P)}_{RS}(\bar{n}^{\prime},\bar{q})$, in the limit $\theta\to
1$, where the transition from RS to RSB is expected to occur.

We start by recalling the expression for
$\mathcal{F}^{(P)}_{1RSB}(\bar{n},\bar{q}_{2},\bar{q}_{1}|\theta)$, given in
(35), with $\bar{n}$, $\bar{q}_{1}$ and $\bar{q}_{2}$ determined from the
self-consistency equations (37), and by providing the expression for the
quenched free energy within the RS assumption, as derived in unsup

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}^{(P)}_{RS}(\bar{n}^{\prime},\bar{q})=$
$\displaystyle-\dfrac{{\beta^{{}^{\prime}}}^{2}(1+\rho)^{P/2}}{2}(P-1)\bar{n}^{\prime
P}+\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma(1+\sqrt{\rho_{P}})^{2}}{4(1+\rho)^{P}}(1-\bar{q}^{P})$
$\displaystyle-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma(1+\sqrt{\rho_{P}})^{2}}{4(1+\rho)^{P}}\bar{q}^{P-1}(1-\bar{q})+\ln
2$
$\displaystyle+\mathbb{E}\ln\cosh\left(\beta^{{}^{\prime}}(1+\rho)^{P/2-1}\dfrac{P}{2}\hat{\eta}_{\mbox{\tiny$M$}}\bar{n}^{\prime
P-1}+\beta^{{}^{\prime}}z\sqrt{\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\dfrac{P}{2}\bar{q}^{P-1}}\right)$
(114)

where
$\mathbb{E}=\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{z}$,
$\hat{\eta}_{\mbox{\tiny$M$}}:=\frac{1}{rM}\displaystyle\sum\limits_{a=1}^{M}\eta^{1,a}$,
and $\bar{n}^{\prime}$ and $\bar{q}$ fulfill the following self-consistency
equations

$\begin{array}[]{lll}\bar{n}^{\prime}=\dfrac{1}{1+\rho}\mathbb{E}\left\\{\tanh{\left[\beta^{\prime}\dfrac{P}{2}\bar{n}^{\prime
P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}+z\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\bar{q}^{{}^{P-1}}}\;\right]}\hat{\eta}_{M}\right\\}\end{array}$
(115)
$\begin{array}[]{lll}\bar{q}=\mathbb{E}\left\\{\tanh^{\mbox{\tiny$2$}}{\left[\beta^{\prime}\dfrac{P}{2}\bar{n}^{\prime
P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}+z\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\bar{q}^{{}^{P-1}}}\;\right]}\right\\}.\end{array}$
(116)

We note that for $\theta=1$, $\bar{q}_{1}=\bar{q}$, $\bar{n}=\bar{n}^{\prime}$
and the 1RSB expression for the quenched free- energy reduces to the RS one.
Now, we expand the $1$RSB expression for the quenched free energy
$\mathcal{F}^{(P)}_{1RSB}(\bar{n},\bar{q}_{2},\bar{q}_{1}|\theta)$, as given
in (35), around $\theta=1$, using
$\mathcal{F}^{(P)}_{1RSB}(\bar{n},\bar{q}_{2},\bar{q}_{1}|\theta)|_{\theta=1}=\mathcal{F}^{(P)}_{RS}(\bar{n}^{\prime},\bar{q})$

$\displaystyle\mathcal{F}_{1RSB}(\bar{n},\bar{q}_{2},\bar{q}_{1}|\theta)=\mathcal{F}_{RS}(\bar{n}^{\prime},\bar{q})+(\theta-1)\partial_{\theta}\mathcal{F}_{1RSB}(\bar{n},\bar{q}_{2},\bar{q}_{1}|\theta)|_{\theta=1}.$
(117)

Since $\bar{q}_{1}$ and $\bar{q}_{2}$ are determined through the self-
consistency equations (37), they depend on $\theta$ as well, so we have to
expand (37) around $\theta=1$ too. Following Ref. albanese2023almeida
closely, we obtain

$\displaystyle\bar{q}_{1}$ $\displaystyle=$
$\displaystyle\bar{q}+(\theta-1)A(\bar{n}^{\prime},\,\bar{q},\tilde{q}_{2}(\bar{n}^{\prime},\,\bar{q}))$
(118) $\displaystyle\bar{q}_{2}$ $\displaystyle=$
$\displaystyle\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})+(\theta-1)B(\bar{n}^{\prime},\,\bar{q},\tilde{q}_{2}(\bar{n}^{\prime},\,\bar{q}))$
(119)

where $\bar{q}$ is the solution of (116),
$\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})$ is the solution of the following
self-consistency equation

$\displaystyle\bar{q}_{2}=\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh{g(\bar{n}^{\prime},\bar{q},\tilde{q}_{2}(\bar{n}^{\prime},\,\bar{q}))}\tanh^{2}{g(\bar{n}^{\prime},\bar{q},\tilde{q}_{2}(\bar{n}^{\prime},\,\bar{q}))}}{\mathbb{E}_{Y^{(2)}}\cosh{g(\bar{n}^{\prime},\bar{q},\tilde{q}_{2}(\bar{n}^{\prime},\,\bar{q}))}}\right]$
(120)

where

$\displaystyle
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})=$
$\displaystyle\beta^{\prime}\dfrac{P}{2}\bar{n}^{\prime
P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}+Y^{(1)}\beta^{\prime}\sqrt{\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\dfrac{P}{2}\bar{q}^{{}^{P-1}}}$
$\displaystyle+Y^{(2)}\beta^{\prime}\sqrt{\gamma\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\dfrac{P}{2}\left((\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}))^{{}^{P-1}}-\bar{q}^{{}^{P-1}}\right)}.$
(121)

and the functions
$A(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})$ and
$B(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})$ are
given by

$\displaystyle
A(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})=$
$\displaystyle
2\mathbb{E}_{Y^{(1)}}\left\\{\dfrac{\mathbb{E}_{Y^{(2)}}\ln\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\sinh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\tanh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}\right\\}$
$\displaystyle-2\mathbb{E}_{Y^{(1)}}\left\\{\dfrac{\mathbb{E}_{Y^{(2)}}\ln\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\left(\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\right)^{2}}\right.$
$\displaystyle\left.\hskip 42.67912pt\cdot\dfrac{\mathbb{E}_{Y^{(2)}}\sinh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\tanh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\left(\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\right)^{2}}\right\\}\,,$
(122) $\displaystyle
B(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})=$
$\displaystyle 2\mathbb{E}_{Y^{(1)}}\left\\{\dfrac{\mathbb{E}_{Y^{(2)}}\sinh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\tanh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\left(\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\right)^{2}}\right.$
$\displaystyle\hskip 42.67912pt\left.\cdot\dfrac{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\log\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\left(\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\right)^{2}}\right\\}$
$\displaystyle-2\mathbb{E}_{Y^{(1)}}\left\\{\dfrac{\left(\mathbb{E}_{Y^{(2)}}\sinh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\tanh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\right)^{2}}{\left(\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\right)^{3}}\right.$
$\displaystyle\hskip 42.67912pt\left.\cdot\dfrac{\mathbb{E}_{Y^{(2)}}\sinh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\log\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\left(\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\right)^{3}}\right\\},$
(123)

respectively. Similarly, we can expand $\bar{n}$ as

$\bar{n}=\bar{n}^{\prime}+(\theta-1)C(\bar{n}^{\prime},\,\bar{q},\tilde{q}_{2}(\bar{n}^{\prime},\,\bar{q}))$
(124)

where

$\displaystyle\begin{array}[]{lll}&C(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})=\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\sinh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\log\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}\right]\\\
\\\ &-\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\sinh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\log\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\left(\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\right)^{2}}\right]\end{array}$
(128)

In the limit $\theta\to 1$, Eq. (117) implies that
$\mathcal{F}^{(P)}_{1RSB}(\bar{n}^{\prime},\bar{q}_{2}(\bar{q}),\bar{q}|\theta)<\mathcal{F}^{(P)}_{RS}(\bar{n}^{\prime},\bar{q})$
when we have that
$\partial_{\theta}\mathcal{F}_{1RSB}(\bar{n}^{\prime},\bar{q}_{2}(\bar{q}),\bar{q}|\theta)|_{\theta=1}>0$.
Next, we evaluate

$\displaystyle
K(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})$
$\displaystyle\coloneqq\partial_{\theta}(-\beta^{{}^{\prime}}\mathcal{F}^{(P)}_{\rm
1RSB}(\bar{n}^{\prime},\bar{q}_{2}(\bar{q}),\bar{q}|\theta))|_{\theta=1}$
$\displaystyle=$
$\displaystyle-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma(1+\sqrt{\rho_{P}})^{2}}{4(1+\rho)^{P}}(P-1)[(\tilde{q}_{2}(\bar{q}))^{P}-\bar{q}^{P}]-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma(1+\sqrt{\rho_{P}})^{2}}{4(1+\rho)^{P}}P[(\tilde{q}_{2}(\bar{q}))^{P-1}-\bar{q}^{P-1}]$
$\displaystyle-\mathbb{E}\ln\cosh\left(\beta^{\prime}\dfrac{P}{2}\bar{n}^{\prime
P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}+Y\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\dfrac{(1+\sqrt{\rho_{P}})^{2}}{(1+\rho)^{P}}\bar{q}^{{}^{P-1}}}\right)$
$\displaystyle+\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\log\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}\right].$
(129)

In order to determine the sign of the expression above, it is useful to note
that $K(\bar{n}^{\prime},\bar{q},\bar{q})=0$, as the last term of (121)
vanishes so, the last two addends in (A.1) elide each other. This is as
expected, as $\bar{q}$ is an extremum of the RS free-energy, which is
retrieved for $\theta=1$. Next, we study $K(\bar{n}^{\prime},x,\bar{q})$ as a
function of $x\in[0,\bar{q}]$ and locate its extrema. These are found from

$\displaystyle\partial_{x}K(\bar{n}^{\prime},x,\bar{q})=$
$\displaystyle-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma(1+\sqrt{\rho_{P}})^{2}(P-1)Px^{P-2}}{4(1+\rho)^{P}}$
$\displaystyle\hskip
14.22636pt\cdot\left[x-\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\sinh
g(\bar{n}^{\prime},\bar{q},x)\tanh
g(\bar{n}^{\prime},\bar{q},x)}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\bar{q},x)}\right]\right]=0$ (130)

as

$\displaystyle x=\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\sinh
g(\bar{n}^{\prime},\bar{q},x)\tanh
g(\bar{n}^{\prime},\bar{q},x)}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\bar{q},x)}\right]\equiv\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}).$
(131)

where the last equality follows from algebraic manipulations of trigonometric
functions.

Given that $K(\bar{n}^{\prime},x,\bar{q})$ vanishes for $x=\bar{q}$, if the
extremum $x=\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})$ is global in the domain
considered, we must have that
$K(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})>0$ if
$x=\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})$ is a maximum and
$K(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})<0$ if
$x=\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})$ is a minimum. Evaluating

$\displaystyle\partial^{2}_{x}K(\bar{n}^{\prime},x,\bar{q})|_{x=\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})}=$
$\displaystyle-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma(1+\sqrt{\rho_{P}})^{2}(P-1)P}{4(1+\rho)^{P}}(\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}))^{P-2}$
$\displaystyle\cdot\left[1-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma(1+\sqrt{\rho_{P}})^{2}}{2(1+\rho)^{P}}(P-1)P(\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}))^{P-2}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\textnormal{sech}^{3}g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}\right]\right]$
(132)

we have that if the expression in the square brackets is positive, namely if
the parameter $\gamma{\beta^{{}^{\prime}}}^{2}$ satisfies

$\displaystyle\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma(1+\sqrt{\rho_{P}})^{2}}{2(1+\rho)^{P}}(P-1)P(\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}))^{P-2}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\textnormal{sech}^{3}g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}\right]<1,$
(133)

$K(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})<0$ and
$\mathcal{F}^{(P)}_{1RSB}(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q}|\theta)-\mathcal{F}^{(P)}_{RS}(\bar{n}^{\prime},\bar{q})=-(\theta-1)K(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})/\beta^{{}^{\prime}}<0$,
hence the RS theory is unstable. We plot the line (133) in Figure 7, for
different values of $M$, in the parameter space $(\gamma,\tilde{\beta})$,
where $\tilde{\beta}=\beta^{\prime}/(1+\rho)^{P/2}$, together with the
critical lines delimiting the retrieval region (top plots) and the spin-glass
region (bottom plots), within the RS and the RSB theory, respectively. We note
that the above recovers the expression for the RS instability line found in
DAM models albanese2023almeida  in the limits $r\to 0$ or $M\to\infty$, where
the values of $\rho_{P}$ and $\rho$ vanish.

Figure 7: The AT line (marking the onset of instability of the RS theory) is
shown as a dashed line in the parameter space of storage load $\gamma$ and
scaled noise $\tilde{\beta}=\beta^{\prime}/(1+\rho)^{P/2}$ for the
unsupervised DAM model, with $P=4$. For comparison, we also show the critical
lines delimiting the retrieval region (top plots) and the spin-glass region
(bottom plots), within the RS and the RSB theory, respectively, as obtained in
the phase diagram in Fig. 1. Different columns correspond to different numbers
of examples, $M$, presented to the network, namely $M=M_{\otimes}^{unsup}$
(left), $M=2M_{\otimes}^{unsup}$ (middle) and $M=+\infty$ (right). We note
that the AT line coincides with the line delimiting the SG region withing the
RS theory.

###  A.2 AT line for DAM in supervised setting

In this section we derive the RS instability line for the supervised DAM
model. We start by providing the expression for the quenched free-energy in RS
assumption as derived in Gardner  for the standard Dense Hopfield Model.

$\displaystyle-\beta^{{}^{\prime}}\mathcal{F}^{(P)}_{RS}(\bar{n}^{\prime},\bar{q})=$
$\displaystyle\ln
2-\dfrac{{\beta^{{}^{\prime}}}^{2}}{2}(P-1)(1+\rho)^{P/2}\bar{n}^{\prime
P}+\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma}{4}(1-\bar{q}^{P})-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma
P}{4}\bar{q}^{P-1}(1-\bar{q})$
$\displaystyle+\mathbb{E}_{\xi}\mathbb{E}_{(\eta|\xi)}\mathbb{E}_{z}\ln\cosh\left(\beta^{{}^{\prime}}\dfrac{P}{2}(1+\rho)^{P/2-1}\hat{\eta}_{M}\bar{n}^{P-1}+\beta^{{}^{\prime}}z\sqrt{\gamma\dfrac{P}{2}\bar{q}^{P-1}}\right)$
(134)

with $\beta^{{}^{\prime}}:=2\beta/P!$ and where $\bar{q}$ and $\bar{n}$
satisfy the self-consistency equations

$\displaystyle\bar{n}^{\prime}=\dfrac{1}{1+\rho}\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{z}\left\\{\tanh\left[\beta^{{}^{\prime}}\dfrac{P}{2}(1+\rho)^{P/2-1}\hat{\eta}_{M}\bar{n}^{\prime
P-1}+\beta^{{}^{\prime}}z\sqrt{\gamma\dfrac{P}{2}\bar{q}^{P-1}}\right]\hat{\eta}_{M}\right\\},$
$\displaystyle\bar{q}=\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{z}\left\\{\tanh^{2}\left[\beta^{{}^{\prime}}\dfrac{P}{2}(1+\rho)^{P/2-1}\hat{\eta}_{M}\bar{n}^{\prime
P-1}+\beta^{{}^{\prime}}z\sqrt{\gamma\dfrac{P}{2}\bar{q}^{P-1}}\right]\right\\}.$
(135)

We also recall that the quenched free-energy within the 1RSB approximation,
$\mathcal{F}^{(P)}_{1RSB}(\bar{n},\bar{q}_{1},\bar{q}_{2}|\theta)$, is as
given in (88), where the order parameters $\bar{q}_{2}$, $\bar{q}_{1}$ and
$\bar{n}$ satisfy the set of self-consistency equations provided in (89). We
note that for $\theta=1$, $\bar{q}_{1}=\bar{q}$, $\bar{n}=\bar{n}^{\prime}$
and the 1RSB expression for the quenched free-energy reduces to the RS one.
Now, we expand, to the leading order in $\theta-1$, the 1RSB quenched free-
energy around its RS expression, as shown in (117). Since the self-consistency
equations also depend on $\theta$, we need to expand them too. We can write
$\bar{q}_{1}$ as in (118), with
$A(\bar{n}^{\prime},\bar{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})$ given in
(122), and $\bar{q}_{2}$ as given in (119), where
$\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})$ is the solution of (120) and
$B(\bar{n}^{\prime},\bar{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})$ is as
given in (123) (we recall that now the expression of
$g(\bar{n}^{\prime},\bar{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})$ is as
given in (90)). With these expressions in hand, we can now compute the
derivative of $\mathcal{F}^{(P)}_{\rm 1RSB}$ w.r.t. $\theta$ when $\theta=1$,
as needed in (117)

$\displaystyle
K(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q}):=$
$\displaystyle\partial_{\theta}(-\beta^{{}^{\prime}}\mathcal{F}^{(P)}_{\rm
1RSB}(\bar{n}^{\prime},\bar{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q}|\theta))|_{\theta=1}$
$\displaystyle=-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma}{4}(P-1)[(\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}))^{P}-\bar{q}^{P}]-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma}{4}P[(\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}))^{P-1}-\bar{q}^{P-1}]$
$\displaystyle-\mathbb{E}_{\xi}\mathbb{E}_{(\eta|\xi)}\mathbb{E}_{Y}\ln\cosh\left(\beta^{\prime}\dfrac{P}{2}\bar{n}^{\prime
P-1}(1+\rho)^{P/2-1}\hat{\eta}_{\mbox{\tiny$M$}}+Y\beta^{\prime}\sqrt{\gamma\dfrac{P}{2}\bar{q}^{{}^{P-1}}}\right)$
$\displaystyle+\mathbb{E}_{\xi}\mathbb{E}_{(\eta|\xi)}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})\log\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}\right].$
(136)

Again, we have that $K(\bar{n}^{\prime},\bar{q},\bar{q})=0$ (as $\bar{q}$ is
the extremum of the RS free-energy). Next, we inspect the sign of
$K(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})$. To this
purpose, we study $K(\bar{n}^{\prime},x,\bar{q})$ for $x\in[0,\bar{q}]$ and
locate its extrema, which are found from

$\displaystyle\partial_{x}K(\bar{n}^{\prime},x,\bar{q})=$
$\displaystyle-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma(P-1)Px^{P-2}}{4(1+\rho)^{P}}\left[x-\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\sinh
g(\bar{n}^{\prime},\bar{q},x)\tanh
g(\bar{n}^{\prime},\bar{q},x)}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\bar{q},x)}\right]\right]=0$ (137)

as

$\displaystyle
x=\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\cosh{g(\bar{n}^{\prime},x,\bar{q})}\tanh^{2}{g(\bar{n}^{\prime},x,\bar{q})}}{\mathbb{E}_{Y^{(2)}}\cosh{g(\bar{n}^{\prime},x,\bar{q})}}\right]\equiv\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})$
(138)

where the last equality follows from (120). Under the assumption that the
extremum $x=\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})$ is global in the domain
considered, we have that
$K(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})>0$ if
$x=\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})$ is a maximum and
$K(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})<0$ if it
is a minimum. In particular, if

$\displaystyle\partial^{2}_{x}K(\bar{n}^{\prime},x,\bar{q})|_{x=\tilde{q}_{2}(\bar{n}^{\prime},\bar{q})}=$
$\displaystyle-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma(P-1)P(\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}))^{P-2}}{4}$
$\displaystyle\cdot\left[1-\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma}{2}(P-1)P(\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}))^{P-2}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\textnormal{sech}^{3}g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}\right]\right]$
(139)

is positive,
$K(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})<0$ and
$\mathcal{F}^{(P)}_{1RSB}(\bar{n}^{\prime},\bar{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q}|\theta)<\mathcal{F}^{(P)}_{RS}(\bar{n}^{\prime},\bar{q})$.
This happens when the expression in the curly brackets of the equation above
is negative, i.e. when the parameter $\gamma{\beta^{{}^{\prime}}}^{2}$
satisfies the inequality

$\dfrac{{\beta^{{}^{\prime}}}^{2}\gamma}{2}(P-1)P(\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}))^{P-2}\mathbb{E}_{Y^{(1)}}\left[\dfrac{\mathbb{E}_{Y^{(2)}}\textnormal{sech}^{3}g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}{\mathbb{E}_{Y^{(2)}}\cosh
g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})}\right]>1.$
(140)

We note that (140) is functionally identical to the expression found in
albanese2023almeida  for the RS instability line of the standard DAM model,
the only difference being encoded in the term
$g(\bar{n}^{\prime},\tilde{q}_{2}(\bar{n}^{\prime},\bar{q}),\bar{q})$, which
is here defined differently, as it reflects the supervised protocol.  In
particular, in the limit of $r\to 0$ or $M\to\infty$, where $\rho$ and
$\rho_{P}$ vanish, (140) retrieves the RS instability line of standard DAM
models, as obtained in albanese2023almeida .

##  Appendix B Proofs

###  B.1 Proof of Lemma 1

Defining the shorthand
$\mathcal{Z}_{2}(J,t)=\mathcal{Z}_{2}^{(P)}(\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)$,
we start by computing the derivative of Eq. (29) w.r.t. $t$

$\begin{array}[]{lll}-\beta^{{}^{\prime}}\partial_{t}\mathcal{F}^{(P)}_{N}(J,t)&=&\dfrac{1}{N}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\partial_{t}\left[\ln\left[\mathbb{E}_{\bm{Y}^{(2)}}\mathcal{Z}_{2}^{\theta}(J,t)\right]^{1/\theta}\right]=\dfrac{1}{N}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\left[\dfrac{\mathbb{E}_{\bm{Y}^{(2)}}\partial_{t}\mathcal{Z}_{2}(J,t)^{\theta-1}\partial_{t}\mathcal{Z}_{2}(J,t)}{\mathbb{E}_{\bm{Y}^{(2)}}\mathcal{Z}_{2}(J,t)^{\theta}}\right]\\\
\\\
&=&\dfrac{1}{N}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\mathbb{E}_{\bm{Y}^{(2)}}\left[\mathcal{W}_{2,t}\dfrac{\partial_{t}\mathcal{Z}_{2}(J,t)}{\mathcal{Z}_{2}(J,t)}\right]\end{array}$
(141)

where
$\mathbb{E}_{0}=\mathbb{E}_{\bm{\xi}^{1}}\mathbb{E}_{(\bm{\eta}^{1}|\bm{\xi}^{1})}\mathbb{E}_{\bm{\lambda}}$
and $\mathcal{W}_{2,t}$ is defined in (34). Recalling the definition of
$\mathcal{Z}_{2}(J,t)$ in (5) we have:

$\begin{array}[]{lll}\partial_{t}\mathcal{Z}_{2}(J,t)&=&\displaystyle\sum\limits_{\bm{\sigma}}\mathcal{B}_{2}(\bm{\sigma};J,t)\Bigg{[}\left(\dfrac{\beta^{\prime}N}{2M}\left(\dfrac{\mathcal{R}}{r^{2}}\right)^{P/2}\displaystyle\sum\limits_{a=1}^{M}n_{1,a}^{{}^{P}}(\bm{\sigma})\right)-\psi
N\displaystyle\sum\limits_{a=1}^{M}n_{1,a}(\bm{\sigma})\\\ \\\
&&+\dfrac{1}{2}\dfrac{\beta^{\prime}P!(1+\sqrt{\rho_{P}})\sqrt{K}}{2\sqrt{t}(1+\rho)^{P/2}\,N^{P-1}}\displaystyle\sum\limits_{{i_{1}<\cdots<i_{P}}}^{N,\cdots,N}\lambda_{i_{1},\ldots,i_{P}}\sigma_{{}_{i_{1}}}\cdots\sigma_{{}_{i_{{}_{P}}}}-\dfrac{1}{2\sqrt{1-t}}\displaystyle\sum\limits_{b=1}^{2}\left(A_{b}\displaystyle\sum\limits_{i=1}^{N}Y_{i}^{(b)}\sigma_{i}\right)\Bigg{]}.\end{array}$
(142)

where, in order to lighten the notation we have set
$\mathcal{B}^{(P)}_{2}(\bm{\sigma}|\bm{\eta}^{1},\bm{\lambda},\bm{Y};J,t)=\mathcal{B}_{2}(\bm{\sigma};J,t)$.
Inserting this expression in (141) and using the definition (25) for the
average of a single replica of the system over the generalised Boltzmann
factor $\mathcal{B}_{2}(\bm{\sigma};J,t)$, we obtain

$\displaystyle-\beta^{{}^{\prime}}\partial_{t}\mathcal{F}^{(P)}_{N}(J,t)=\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\mathbb{E}_{\bm{Y}^{(2)}}\Big{\\{}\dfrac{\beta^{\prime}}{2M}\left(\dfrac{\mathcal{R}}{r^{2}}\right)^{P/2}\mathcal{W}_{2,t}\omega_{t}\left(\displaystyle\sum\limits_{a=1}^{M}n_{1,a}^{{}^{P}}(\bm{\sigma})\right)-\psi\mathcal{W}_{2,t}\omega_{t}\left(\displaystyle\sum\limits_{a=1}^{M}n_{1,a}(\bm{\sigma})\right)\Big{\\}}$
$\displaystyle+\dfrac{1}{2}\dfrac{\beta^{{}^{\prime}}P!(1+\sqrt{\rho_{P}})\sqrt{K}}{2\sqrt{t}(1+\rho)^{P/2}N^{P}}\displaystyle\sum\limits_{{i_{1}<\cdots<i_{P}}}^{N,\cdots,N}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\mathbb{E}_{\bm{Y}^{(2)}}\left\\{\mathcal{W}_{2,t}\dfrac{1}{\mathcal{Z}_{2}(J,t)}\sum_{\bm{\sigma}}\Big{(}\lambda_{i_{1},\ldots,i_{P}}\sigma_{{}_{i_{1}}}\cdots\sigma_{{}_{i_{{}_{P}}}}\Big{)}\mathcal{B}_{2}(\bm{\sigma};J,t)\right\\}$
$\displaystyle-\dfrac{1}{2N\sqrt{1-t}}\displaystyle\sum\limits_{b=1}^{2}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\mathbb{E}_{\bm{Y}^{(2)}}\left\\{\mathcal{W}_{2,t}\dfrac{1}{\mathcal{Z}_{2}(J,t)}\sum_{\bm{\sigma}}\left(A_{b}\displaystyle\sum\limits_{i=1}^{N}Y_{i}^{(b)}\sigma_{i}\right)\mathcal{B}_{2}(\bm{\sigma};J,t)\right\\}.$
(143)

Next, denoting the combined average over the quenched disorder and the
Boltzmann distribution as

$\langle\cdot\rangle=\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\mathbb{E}_{\bm{Y}^{(2)}}\left[\mathcal{W}_{2,t}\omega_{t}(\cdot)\right]$
(144)

and applying the Stein’s lemma (74) to the standard Gaussian variables
$Y_{i}^{(b)}$ and $\lambda^{\mu}_{i_{1},\ldots,i_{P}}$, we get

$\displaystyle-\beta^{{}^{\prime}}\partial_{t}\mathcal{F}^{(P)}_{N}=$
$\displaystyle\dfrac{\beta^{\prime}}{2M}\left(\dfrac{\mathcal{R}}{r^{2}}\right)^{P/2}\displaystyle\sum\limits_{a=1}^{M}\langle
n_{1,a}^{P}(\bm{\sigma})\rangle_{t}-\psi\displaystyle\sum\limits_{a=1}^{M}\langle
n_{1,a}(\bm{\sigma})\rangle_{t}$
$\displaystyle+{\dfrac{{\beta^{{}^{\prime}}}^{2}K(1+\sqrt{\rho_{P}})^{2}}{8(1+\rho)^{P}\,N^{2P-1}}}\displaystyle\sum\limits_{i_{{}_{1}},\cdots,i_{{}_{P}}}^{N,\cdots,N}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\mathbb{E}_{\bm{Y}^{(2)}}\left(\mathcal{W}_{2,t}\omega_{t}(1)+(\theta-1)\mathcal{W}_{2,t}\omega_{t}^{2}(\sigma_{{}_{i_{1}}}\cdots\sigma_{{}_{i_{{}_{P}}}})\right)$
$\displaystyle-{\dfrac{{\beta^{{}^{\prime}}}^{2}P!K(1+\sqrt{\rho_{P}})^{2}}{8(1+\rho)^{P}\,N^{2P-1}}}\displaystyle\sum\limits_{i_{{}_{1}}<\cdots<i_{{}_{P}}}^{N,\cdots,N}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\theta\left[\mathbb{E}_{\bm{Y}^{(2)}}\left(\mathcal{W}_{2,t}\omega_{t}(\sigma_{{}_{i_{1}}}\cdots\sigma_{{}_{i_{{}_{P}}}})\right)\right]^{2}$
$\displaystyle-\dfrac{(A_{1})^{2}}{2}\displaystyle\sum\limits_{i=1}^{N}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\mathbb{E}_{\bm{Y}^{(2)}}\left(1+(\theta-1)\mathcal{W}_{2,t}\omega_{t}^{2}(\sigma_{i})\right)-\dfrac{(A_{1})^{2}}{2}\theta\displaystyle\sum\limits_{i=1}^{N}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\left[\mathcal{W}_{2,t}\omega_{t}(\sigma_{i})\right]^{2}$
$\displaystyle-\dfrac{(A_{2})^{2}}{2}\displaystyle\sum\limits_{i=1}^{N}\mathbb{E}_{0}\mathbb{E}_{\bm{Y}^{(1)}}\mathbb{E}_{\bm{Y}^{(2)}}\left(1+(\theta-1)\mathcal{W}_{2,t}\omega_{t}^{2}(\sigma_{i})\right).$
(145)

Finally, we use the definitions (32) and (33) and we arrive at the compact
expression:

$\displaystyle-\beta^{{}^{\prime}}\partial_{t}\mathcal{F}^{(P)}_{N}=$
$\displaystyle\dfrac{\beta^{\prime}}{2M}\left(\dfrac{\mathcal{R}}{r^{2}}\right)^{P/2}\displaystyle\sum\limits_{a=1}^{M}\langle
n_{1,a}^{P}(\bm{\sigma})\rangle_{t}-\psi\displaystyle\sum\limits_{a=1}^{M}\langle
n_{1,a}(\bm{\sigma})\rangle_{t}$
$\displaystyle+{\dfrac{{\beta^{{}^{\prime}}}^{2}(1+\sqrt{\rho_{P}})^{2}}{4(1+\rho)^{P}}}\dfrac{P!K}{2N^{P-1}}\left[1+(\theta-1)\langle
q_{12}^{P}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})\rangle_{t,2}-\theta\langle
q_{12}^{P}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})\rangle_{t,1}\right]$
$\displaystyle-\dfrac{A_{1}^{2}}{2}\left[1+(\theta-1)\langle
q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})\rangle_{t,2}-\theta\langle
q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})\rangle_{t,1}\right]$
$\displaystyle-\dfrac{A_{2}^{2}}{2}\left[1+(\theta-1)\langle
q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})\rangle_{t,2}\right].$ (146)

Next, we take the thermodynamic limit $N\to\infty$, using assumption 2. By
manipulating the expression for the moments $\langle
q_{12}^{P}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})\rangle_{t,a}$, with $a=1,2$,
using Newton’s binomial theorem

$\begin{array}[]{lll}\langle
q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})^{P}\rangle_{t,a}&=&\langle[(q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})-\bar{q}_{a}+\bar{q}_{a}]^{P}\rangle_{t,a}=\displaystyle\sum\limits_{k=0}^{P}\begin{pmatrix}P\\\
k\end{pmatrix}\langle(q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})-\bar{q}_{a})^{k}\rangle_{t,a}\bar{q}_{a}^{P-k}\\\
&&=P\,\bar{q}_{a}^{P-1}\langle
q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})\rangle_{t,a}+\displaystyle\sum\limits_{k=2}^{P}\begin{pmatrix}P\\\
k\end{pmatrix}\langle(q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})-\bar{q}_{a})^{k}\rangle_{t,a}\bar{q}_{a}^{P-k}+(1-P)\bar{q}_{a}^{P},\end{array}$

we obtain, in the thermodynamic limit, under Assumption 2,

$\begin{array}[]{lll}\langle
q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})^{P}\rangle_{t,a}&=&P\,\bar{q}_{a}^{P-1}\langle
q_{12}(\bm{\sigma}^{(1)},\bm{\sigma}^{(2)})\rangle_{t,a}-(P-1)\bar{q}_{a}^{P}\end{array}$
(147)

for $a=1,2$. Finally, we insert the above in (146) and we choose the constants