# Low-resolution descriptions of model neural activity
reveal hidden features and underlying system properties

Riccardo Aldrigo  Physics Department, University of Trento, via Sommarive, 14
I-38123 Trento, Italy  Roberto Menichetti  Physics Department, University of
Trento, via Sommarive, 14 I-38123 Trento, Italy  INFN-TIFPA, Trento Institute
for Fundamental Physics and Applications, I-38123 Trento, Italy  Raffaello
Potestio<EMAIL_ADDRESS>Physics Department, University of
Trento, via Sommarive, 14 I-38123 Trento, Italy  INFN-TIFPA, Trento Institute
for Fundamental Physics and Applications, I-38123 Trento, Italy

(August 28, 2024)

###### Abstract

The analysis of complex systems such as neural networks is made particularly
difficult by the overwhelming number of their interacting components. In the
absence of prior knowledge, identifying a small but informative subset of
network nodes on which the analysis should focus is a rather difficult task.
In this work, we address this problem in the context of a Hopfield model, for
which we seek representations given in terms of a subgroup of its neurons with
the aim of identifying those that can reveal the largest amount of information
about the system. Using a methodology based on concepts from information
theory, we reveal that such optimised low-resolution representations are not
only informative _per se_ , but rather they also serve as probes of the neural
network properties; these results show a tight and potentially fruitful
relation between the level of detail at which the network is inspected and the
type and amount of information that can be extracted from it.

##  I Introduction

Neural systems owe their complex emergent phenomena to the winding, nonlinear
interplay of a large number of fundamental units – neurons rolls1998neural ;
dayan2005theoretical ; zamora2011characterizing ; lynn2019physics . The
intricacy and sheer size of biological neural networks (the human brain is
made of tens of billions of nerve cells herculano2009human ; joudaki2012eeg )
is however such that a comprehensive understanding of their behaviour is still
largely out of reach; several approaches have thus been proposed that aim,
_via_ the introduction of rather essential and computationally manageable
models, at reproducing and/or capturing specific aspects of neural activity.
These effective representations of the brain architecture and function range,
e.g., from generalised Potts models kropff2005storage ; russo2008free  or
restricted Boltzmann machines ackley1985learning ; aarts1989simulated ;
marullo2020boltzmann ; goel2020boltzmann  to graph-theory methods
sporns2014network ; bassett2018nature  and the most recent, large-scale deep
learning techniques vogels2005annurev . In this context, a classical example
of in silico neural network is provided by the Hopfield model
hopfield1982neural ; hopfield1984pnas ; stat_mech_H_model ;
ramsauer2020hopfield , in which a neuron is represented as a two-state
variable, or binary spin, whose values are associated with the biological
_firing_ or _rest_ conditions; albeit their relatively simple mathematical
formulation, Hopfield models—or extensions thereof—showcase a rich
phenomenology, and are still widely employed to investigate the process of
memory retrieval barra2010replica ; agliari2018non ; agliari2020replica ;
alemanno2023hopfield .

Irrespective of their specific details and purpose, the aforementioned models
exhibit several advantageous features: on the one hand, the overall
“simplicity” of their elemental constituents and associated interactions, as
well as the controllable size of the network—where the latter can be either
sufficiently small to carry out numerical simulations crisanti1986saturation ;
bruck1990convergence ; stat_mech_H_model ; ramsauer2020hopfield , or
infinitely large to allow an exact mathematical treatment huang2021statistical
; on the other hand, one has that the results of an analysis of the emergent
properties of the system can be interpreted in light of the structure of the
underlying model, which is known by definition. Critically, this does not
generally hold in the study of biological neural networks: here, in fact, it
is rarely—if not outright never—the case that one can acquire data (e.g. time
series) about the state of each individual neuron in the system; as a
particularly evident example, think of an electroencephalogram (EEG) exam
where only a few tens of signal streams are recorded out of the billions of
neurons that compose a biological brain herculano2009human ; joudaki2012eeg .
Additionally, the detailed characteristics of the network that are responsible
for the _generation_ of those states are usually unknown: taking once again
the example of an EEG, the neurons that contribute to the experimentally
observed patterns are too complex, too many, and their connections too tightly
intertwined for all these ingredients to be deconstructed in sufficient
detail.

In the framework of an empirical analysis of a subset of neurons, whereby only
the emergent behaviour of the system is observed while its generative
mechanism is unknown, it would be thus desirable to develop strategies that
allow one to distinguish between particularly “important” units, whose states
reveal relevant information about the system, and “irrelevant” ones that can
be safely ignored in the study. In this work, we employ an information-
theoretic analysis method developed by some of us, namely the mapping entropy
optimisation workflow (MEOW) JCTC ; errica2021deep ; HGP ; giulini2024excogito
, to identify and characterise maximally informative subsets of neurons in a
network only given the time series of their states; we thus aim to identify
low-resolution representations of the system in terms of small(er) numbers of
elements that can be almost as useful as a fine-grained description that
accounts for all the network constituents. The MEOW strategy relies on the
idea that a subgroup of elements is “important” if observing them provides
(almost) as much statistical information as one gathers by analysing the whole
system; more specifically, we take the empirical probability distribution of
the observed network states as the high-resolution reference, and attempt at
reconstructing it from the distribution of low-resolution configurations of a
subset of neurons. The discrepancy between the original, high-resolution
empirical probability distribution and the reconstructed one is quantified by
the _mapping entropy_ , a Kullback-Leibler divergence between them
kullback1951information ; Shell_Scott ; rudzinski2011coarse ;
noid2023perspective . The MEOW strategy searches for those subgroups of
elements that minimise the mapping entropy, constituting the maximally
informative reduced representations of the network that can be designed to
investigate its behaviour.

We here apply this idea to the case of a Hopfield model. While the MEOW
approach only takes the time series of the neuron states as an input, applying
the protocol in such a context enables us to directly relate the outcomes of
the analysis to the structure and interactions of the model. Hence, this
allows us to validate the workflow in a controlled case where the details of
the system are known, with the perspective of employing MEOW in more complex
scenarios in which only the observations of the network states are available,
while the underlying generative process is not.

The MEOW analysis enables us to highlight a number of system properties
related to the level of detail at which the Hopfield network is observed: in
particular, we identify three distinct regimes in which qualitatively
different groups of neurons are pinpointed as informative depending on the
resolution level of the coarse description, the latter being roughly related
to the number of constituents retained in the simplified picture. These
results highlight the tight connection between the level of detail at which a
system is described and the amount and quality of information that its
analysis can reveal.

The paper is organized as follows. In Sect. II we recap the fundamentals about
the Hopfield model and the mapping entropy minimisation workflow. In Sect. III
we illustrate and comment on the results of the MEOW analysis of various types
of Hopfield networks of different sizes. Lastly, in Sect. IV we provide our
concluding thoughts and discuss possible future developments and applications
of this work.

##  II Materials and Methods

In this section we provide an overview of the fundamental ingredients of the
models and analysis methods employed in this manuscript. More specifically, in
Sec. II.1 we briefly introduce the Hopfield model, summarise its main
properties, and discuss the technical aspects of the numerical simulations
carried out in this work. Subsequently, in Sec. II.2 we recap the strategy and
the constitutive information-theoretic quantities underlying the mapping
entropy optimization workflow recently developed by some of us JCTC ;
errica2021deep ; HGP ; giulini2024excogito , focusing on those aspects that
are specifically associated with the application of MEOW to the analysis of a
Hopfield network.

###  II.1 The Hopfield model

The neural network model analysed in this work was originally developed by
J.J. Hopfield in 1982 with the aim of exploiting the collective properties of
the system as content-addressable memories hopfield1982neural . Starting from
previous studies in the field McCulloch ; Rosenblatt ; Minsky_Papert , in his
seminal manuscript Hopfield formulated the first example of an _attractor_
neural network (ANN) amit1985prl ; MBF , namely a network capable, in
appropriate conditions, of retrieving a set of stored memory patterns encoded
in the system’s interaction matrix, leveraging a nonlinear dynamic evolution
of its constituents.

A Hopfield network consists of a single layer of $N$ coupled perceptrons, or
model neurons, each of which can be represented as a two-state spin
$\sigma_{i},\ i=1,...,N$ that can take on $+1$ and $-1$ values, respectively
associated to the neuron being in a firing (active) and silent (inactive)
condition. The system dynamics is based on a recurrent architecture, meaning
that the output states of the perceptrons are employed as inputs of the same
layer that generates them. More specifically, the state $\\{\sigma_{i}(t)\\}$
of the network at time $t$ is processed by the neurons to form the set of
output signals $\\{h_{i}(t)\\}$; such _local fields_ constitute, at the
subsequent time cycle and after being subjected to a nonlinear transformation,
the new input state $\\{\sigma_{i}(t+\Delta t)\\}$ of the network, see Fig. 1
for a schematic representation of this overall workflow. As a consequence of
this dynamic evolution, starting from an initial state
$\\{\sigma_{i}(t_{0})\\}$ the system over time wanders throughout the space of
the $2^{N}$ possible configurations available to its neurons, eventually
converging towards one of the stored memories.

Critically, the memories consist of a set of $p$ states, or _patterns_
$\\{\xi^{\mu}_{i}\\}$,

$\displaystyle\\{\xi^{\mu}_{i}\\}\equiv(\xi_{1}^{\mu},\xi_{2}^{\mu},\cdots\
\xi_{N}^{\mu}),$ $\displaystyle\xi^{\mu}_{i}=\pm 1,$
$\displaystyle\mu=1,...,p,$ (1)

which are usually assumed to be independently distributed random variables,
with

$P(\\{\xi^{\mu}_{i}\\})=\prod_{\mu=1}^{p}\prod_{i=1}^{N}\left(\frac{1}{2}\delta(\xi^{\mu}_{i},1)+\frac{1}{2}\delta(\xi^{\mu}_{i},-1)\right),$
(2)

where $\delta(\cdot,\cdot)$ represents a Kronecker delta.

Figure 1: Schematic illustration of the recurrent architecture underlying the
dynamics of the Hopfield model. A multi-perceptron system is closed onto
itself to form an attractor neural network, with the states $\sigma_{i},\
i=1,...,N$ of all neurons at a given time being combined via the synaptic
coefficients $J_{ij}$ to provide the set of local fields $h_{i}$. Such fields,
after undergoing a nonlinear transformation, constitute the state of the
network at the subsequent timestep.

From the preceding discussion, it follows that the behaviour of a Hopfield
network is dictated by three main ingredients, namely (_i_) how the state of
the network at time $t$ is processed to return the set of local fields; (_ii_)
how the latter relate to the configuration of the system at the following time
cycle; and (_iii_) how the memory patterns enter in the definition of the
model. Let us briefly analyse these three aspects.

As for the first ingredient, the Hopfield model resorts to a superposition
principle for which the total signal received by a specific perceptron is
given by a linear combination of those that are transmitted to it by the
remaining $N-1$ neurons in the system. The local field $h_{i}(t)$ experienced
at time $t$ by the $i$-th perceptron thus reads

$h_{i}(t)=\sum_{j=1}^{N}J_{ij}\sigma_{j}(t),$ (3)

see Fig. 1, where the time-independent couplings $J_{ij}$ in Eq. 3, with
$J_{ii}=0$, characterise the interaction between each pair of neurons in the
system and are called _synaptic coefficients_ of the network.

Different prescriptions can then be employed to transform the local fields
into the new state of the network, which can be however divided into two main
categories depending on whether the nature of the nonlinear relation between
$\\{h_{i}(t)\\}$ and $\\{\sigma_{i}(t+\Delta t)\\}$ is deterministic or
stochastic MBF . In this work, the stochastic time evolution of the system is
implemented through a noisy Glauber dynamics Glauber  in which the probability
$P(\sigma_{i}(t+\Delta t)=\sigma_{i}\mid h_{i}(t)=h_{i})=P(\sigma_{i}\mid
h_{i})$ that the $i$-th perceptron turns into the firing state $\sigma_{i}=\pm
1$ when subject to a field $h_{i}$ reads

$P(\sigma_{i}\mid h_{i})=\frac{\exp(\beta h_{i}\sigma_{i})}{\exp(\beta
h_{i})+\exp(-\beta h_{i})},$ (4)

where $\beta^{-1}=T$ is an effective temperature parameter quantifying the
influence of the noisy environment on the synaptic transmission. For $T=0$ the
system is driven towards the lowest-energy state.

Neurons are evolved asynchronously Amit_1 ; Amit_2_saturation ; MBF ;
hopfield1982neural , so that each time cycle $\Delta t$ consists of a series
of $N$ updates of a single, randomly chosen perceptron carried out according
to Eq. 4.

Finally, for the system to work as a content-addressable memory, the $p$
patterns $\\{\xi^{\mu}_{i}\\}$ in Eq. II.1 should be rendered (meta)stable
states of the network. To achieve this, the Hopfield model resorts to the
Hebbian learning rule Hebb , storing the patterns in the synaptic coefficients
$J_{ij}$ between neurons by setting

$J_{ij}=\frac{1}{N}\sum_{\mu=1}^{p}\xi_{i}^{\mu}\xi_{j}^{\mu}.$ (5)

Despite its relatively simple mathematical formulation, the Hopfield model
exhibits an extremely rich phenomenology. Indeed, it has been proven
Amit_2_saturation  that the efficacy of the system in retrieving the encoded
memories along its dynamics is critically dependent on the amount of
noise—that is, the temperature $T$ in Eq. 4—and on the ratio $\alpha=p/N$
between the number of stored patterns and the size of the network. The phase
diagram of the system as a function of these parameters, sketched in Fig. 2,
is characterised by a paramagnetic and a spin glass phase (above the
temperature curves labelled with $T_{M}$ and $T_{g}$, respectively) in which
the network is not able to retrieve any memory, as well as a retrieval phase
(below $T_{M}$) where the embedded patterns appear as thermodynamic metastable
(below $T_{M}$ and above $T_{C}$) or stable (below $T_{C}$) states.

Figure 2: Sketch of the phase diagram of the Hopfield model in the
thermodynamic limit $N\gg 1$ as a function of the parameter $\alpha=p/N$ and
the temperature $T$. Starting from the high-temperature disordered phase,
$T_{g}$ marks the transition at which the system enters a spin glass phase; in
both cases, the system is unable to recover the stored memory patterns. By
lowering the temperature, memory retrieval states appear at $T_{M}$ as
metastable states, and become minima of the free energy below $T_{C}$. Figure
adapted from Amit_2_saturation .

Given this general summary, we now describe the technical details associated
with the numerical simulations of the Hopfield model carried out in this work.
As previously discussed, the time evolution of the system was obtained by
relying on a stochastic asynchronous Glauber dynamics with a random updating
sequence of the single perceptrons Amit_1 ; Amit_2_saturation ; MBF ;
hopfield1982neural . The three main simulation parameters are then the network
size $N$, the number of stored memory patterns $p$, and the system temperature
$T$. In this work, we first investigated a small system consisting of $N=10$
neurons at $T=0$, for which an exhaustive exploration of all the possible
reduced representations of the system, see Sec. II.2, can be performed to
extract the maximally informative ones. Subsequently, we addressed the
analysis of larger networks with $N=100$ at finite temperature. For fixed $N$
we performed simulations with $p$ ranging from $2$ to a maximum of $10$ memory
patterns depending on the number of neurons in the ANN, See sec. III.
Furthermore, the temperature of the larger system was set to $T=0.2$, which,
by assuming a finite-size phase diagram akin to the one presented in Fig. 2,
enables the network to fluctuate in its configurational space while remaining
in the retrieval phase even at relatively high values of $\alpha$. Finally,
for the system to explore all the memorised patterns more than once with
nonzero probability, the pool of configurations employed in the analysis
consisted of $n_{dyn}$ independent realisations of the dynamics of the
network, in which each one started from a random assignment of the neuron
states, and then eventually relaxed into an attractor basin. The number
$n_{dyn}$ of independent trajectories was set to $1000$ for all the
investigated values of $N$ and $p$, where the number of time cycles of each
dynamics was chosen approximately equal to the retrieval period of the
network.

###  II.2 Mapping entropy optimisation workflow (MEOW)

As discussed in the introduction, in this work we aim to identify simplified
representations of a Hopfield network in terms of few “important” elemental
units selected from a pool of variables that potentially contains also noisy,
irrelevant ones. This is achieved only starting from a series of empiric
observations of system states, without relying on previous knowledge of their
underlying generative process. The simplification strategy that we employ in
this endeavour is _decimation_ , that is, the analysis of the system in terms
of a subset of its constituents and the implicit marginalisation on the
discarded ones. Such a protocol has its origins in the process of coarse-
graining (CG’ing) that lies at the heart of the renormalisation group in
statistical and quantum physics, as well as of several modelling strategies of
soft matter systems ref_8_HGP ; ref_9_HGP ; ref_10_HGP . In the context of the
Hopfield model, decimation consists of describing the network only in terms of
a reduced number of selected neurons, a procedure that generally results in a
loss of statistical information on the system. What we then look for, and
identify with the aforementioned optimal simplified representations of the
network, are the decimation mapping operators that, for a fixed number of
retained neurons, provide a description of the ANN whose information content
is _as close as possible_ to the original high-resolution reference. The
protocol implementing this strategy constitutes the mapping entropy
optimization workflow (MEOW) recently developed by some of us JCTC ; HGP ;
giulini2024excogito ; we now briefly summarise the strategy underlying the
MEOW and the associated fundamental information-theoretical ingredients,
focussing our attention on the technical details related to the application of
this workflow to the resolution reduction of a Hopfield network.

Consider a dynamic trajectory of a Hopfield ANN as obtained via the simulation
protocol described in Sec. II.1: at each time $t$, the high-resolution or
“fine-grained” configuration of the system is given by the state of all of its
constituent neurons, $\phi(t)=(\sigma_{1}(t),\sigma_{2}(t),...,\sigma_{N}(t))$
with $\sigma_{i}(t)\in\\{-1,1\\}$. From such trajectory (or an ensemble of
them), one can construct the _empirical_ probability
$p(\phi)=p({\sigma_{1},\sigma_{2},...,\sigma_{N}})$ of observing a specific
microstate $\phi$, which is nothing but the frequency with which the selected
configuration appears in the time series, namely

$p(\phi)=\frac{1}{T}\sum_{t=1}^{T}\prod_{i=1}^{N}\delta(\sigma_{i},\sigma_{i}(t)),$
(6)

where $T$ is the total number of simulation steps.111We stress that the
empirical nature of $p(\phi)$ is such that, although in principle the possible
microstates of the network are $2^{N}$, some of these configurations will not
be visited along the trajectory, either as a mere consequence of the
finiteness of the sample or due to the additional presence of ergodicity-
breaking phenomena, see Fig. 2. In the following, such missing configurations
will be _excluded_ from the high-resolution state space rather than being
endowed with a vanishing empirical probability. This procedure is akin to the
one of taking restricted equilibrium averages in systems displaying ergodicity
breaking; furthermore, it is particularly suited in the analysis of time
series for which the properties of the configurational space and/or the
mechanism underlying the generation of the high-resolution samples are not
known Multiscale_relevance_Marsili .

By explicitly accounting for the state of all constituent neurons, the
empirical fine-grained probability $p(\phi)$ provides a complete
characterisation of the (observed) statistical properties of the network. The
complexity inherent to such a high-dimensional description, however, can
hinder the process of distilling the relevant information on the system out of
a potentially noisy background; it is thus natural to investigate whether
simplified representations of the network can be constructed that are capable
of enhancing the signal-to-noise ratio. As previously introduced, the
elemental ingredient lying at the core of MEOW to tackle this problem is a
coarse-graining procedure that decimates the $N$ high-resolution degrees of
freedom of the network, describing the latter only in terms of a subset of
$n_{cg}<N$ neurons. Let us first discuss how such an operation is implemented
in practice for a _specific_ selection of the sites, and what are its
implications on our knowledge of the global statistical properties of the
system. Starting from this, we will then describe the identification of the
aforementioned maximally informative reduced representations of the network.

We perform an analysis of the system in which only a specific subset of
neurons out of the $N$ constituent ones is explicitly accounted for.
Practically, this amounts to introducing a mapping operator $M(\phi)$ that
projects each high-resolution configuration
$\phi=({\sigma_{1},\sigma_{2},...,\sigma_{N}})$ of the ANN onto its low-
resolution counterpart
$\Psi_{\phi}=M(\phi)\equiv(\sigma_{i_{1}},\sigma_{i_{2}},...,\sigma_{i_{n_{cg}}})$;
the latter consists of only the states of the $n_{cg}$ neurons
($i_{1},...,i_{n_{cg}}),\ i_{\nu}\in\\{1,..,N\\}$, which were selected to be
retained at the CG level. Critically, this procedure reverberates on the
statistical properties of the network _as inspected in its decimated form_ :
the mapping operator induces a _low resolution_ empirical probability of
observing a specific CG configuration $\Psi$, $P(\Psi)$, that can be obtained
from $p(\phi)$ as

$P(\Psi)=\sum_{\phi}p(\phi)\delta(\Psi,\Psi_{\phi}),$ (7)

namely as the marginalised high-resolution probability of all the fine-grained
states that map on the selected CG configuration. From $P(\Psi)$ all the
properties of the reduced network can be obtained; at the same time, the
effect of the projection is to conceal the detailed features that pertain to
the set of integrated neurons, and only a partial description of the system
remains available. One could then ask to what extent an observer provided with
such limited knowledge could succeed in deducing from it the same features
encoded in the high-resolution reference.

The goal is hence to reconstruct the statistical properties of the full
network, namely the fine-grained empirical probability $p(\phi)$, _only given_
its reduced counterpart $P(\Psi)$. In doing so, we note that, after mapping,
no additional information on each microstate is readily accessible but for its
association with the corresponding CG label; a reversal of the coarse-graining
procedure should thus be compatible with a maximum entropy principle in which
all the microstates that map onto the same CG configuration are attributed
equal likelihood to occur. The resulting reconstructed or _backmapped_ high-
resolution probability distribution, $\bar{p}(\phi)$, accordingly reads

$\bar{p}(\phi)=\frac{P(\Psi_{\phi})}{\Omega(\Psi_{\phi})},$ (8)

where

$\Omega(\Psi)=\sum_{\phi^{\prime}}\delta(\Psi,\Psi_{\phi^{\prime}})$ (9)

is the degeneracy of CG configuration $\Psi$, that is, the observed number of
unique fine-grained states of the network that map onto $\Psi$. From Eq. 8 it
follows that describing the network in terms of a selected subset of its
neurons has generated a _loss of statistical information on the system_ , in
that, in contrast to the high-resolution reference $p(\phi)$, upon backmapping
all the microstates of the ANN that compose each CG configuration $\Psi$ have
become statistically equivalent; additionally, we observe that the
reconstructed probability $\bar{p}(\phi)$, which is common to all high-
resolution microstates mapping onto $\Psi$, is given by the average of their
original probabilities.

The loss of information generated by coarsening the representation of the
network can be quantified, in information-theoretic terms, _via_ the _mapping
entropy_ $S_{map}$ JCTC ; HGP ; Shell_Scott ; rudzinski2011coarse ;
noid2023perspective ; giulini2024excogito , with

$\displaystyle
S_{map}=\sum_{\phi}p(\phi)\ln\left[\frac{p(\phi)}{\bar{p}(\phi)}\right]=$
$\displaystyle=\sum_{\phi}p(\phi)\ln\left[p(\phi)\frac{\Omega(\Psi_{\phi})}{P(\Psi_{\phi})}\right].$
(10)

Being a Kullback-Leibler divergence between the original and the backmapped
probability distributions kullback1951information , the mapping entropy is
nonnegative because of Gibbs’ inequality, with
$S_{map}=0\Longleftrightarrow\bar{p}(\phi)=p(\phi)\ \forall\phi$. Most
importantly, since the detailed form of $\bar{p}(\phi)$—and consequently the
resulting mapping entropy—only depends on the specific subset of neurons
chosen to represent the network at low resolution, see Eqs. 7, 8 and 9, it
follows that different mapping operators can be associated with different
amounts of information loss on the statistical properties of the reference
system. One is then naturally led to look, in the space of possible selections
of neurons, for those that _minimise_ $S_{map}$; a low value of mapping
entropy implies that, in spite of the coarsening of the system representation,
the information content available from the retained neurons is close to the
one encoded in the original distribution. These mappings constitute the so-
called _maximally informative_ reduced representations that can be designed to
investigate the behaviour of the system, and their identification and analysis
is at the core of the strategy implemented in MEOW JCTC ; errica2021deep ; HGP
; giulini2024excogito .

To examine the properties of a Hopfield network through the lenses of the
mapping entropy, we rely on a code recently developed by some of us, the
extensible coarse-graining toolbox, or EXCOGITO giulini2024excogito .
Specifically, the program takes as input the fine-grained probabilities of all
the configurations explored in the course of a series of simulations of the
Hopfield model, where the probability of a given microstate $p(\phi)$ is
calculated as the empirical frequency of its occurrence in the dataset, see
Eq. 6. The software then proceeds to identify the maximally informative
reduced representations of the network by determining, among the possible
selections of a fixed number $n_{cg}$ of neurons, the ones that minimise
$S_{map}$; this procedure is then iterated for various levels of coarse-
graining $n_{cg}$. For each analysed mapping, its value of $S_{map}$ is
calculated by clustering all the fine-grained configurations in which the
selected neurons are in the same state, enabling the reconstruction of the
empirical coarse-grained probability distribution $P(\Psi)$ as well as of the
degeneracy factor $\Omega(\Psi)$ (Eqs. 7-9) that enter in the definition of
the mapping entropy (Eq. II.2). Critically, we note that the overall network
size $N$ plays a crucial role in fulfilling the optimisation of $S_{map}$.
Indeed, for small values of $N$ all the possible $n_{map}=2^{N}-1$ mappings of
the system can be exhaustively probed (and ranked) to detect the maximally
informative ones at each degree of coarse-graining $n_{cg}$; on the contrary,
such an extensive exploration becomes rapidly unfeasible as the number of
constituent neurons increases ref_25_HGP . In this latter case, the software
minimises $S_{map}$ in the space of the possible reduced representations of
the network that can be constructed at a given $n_{cg}$ via a Monte Carlo
simulated annealing (SA) procedure ref_62_JCTC ; ref_63_JCTC ; the algorithm
proposes a transition from one neuron subset to another other, the two
differing by one single retained unit, and the new subset is accepted or
rejected according to a Metropolis-like criterion that employs $S_{map}$ as
cost function. By exponentially decreasing the SA effective temperature
parameter along the course of the simulation, the latter is gradually pushed
to identify (local) minima of the mapping entropy. For each $n_{cg}$ a series
of $K_{\text{SA}}=48$ independent SA simulations were performed, thus
resulting in the detection of _a pool_ of maximally informative reduced
representations of the $N=100$ network at each degree of coarse-graining.

Finally, in the following the mapping entropy is studied as a function of the
_resolution_ of the CG representations Marsili_1 , which is defined as:

$H_{S}=-\sum_{\Psi}P(\Psi)\ln P(\Psi).$ (11)

This quantity, which technically is the Shannon entropy of the empirical CG
probability distribution, was introduced by Marsili and coworkers as a measure
of the degree of detail with which an empirical dataset is described; a
qualitative understanding of this quantity can in fact be gathered by
observing that, if all elements of a dataset of size $\mathcal{M}$ are
labelled differently, their empirical probability is $1/\mathcal{M}$, which
returns the largest value of the resolution ($\ln\mathcal{M}$); by grouping
elements e.g. through some clustering procedure, the empirical probability of
each group, defined as the fraction of elements in it, is associated to a
lower value of the entropy. The lowest value is attained when all elements are
included in the same cluster, which corresponds to a null entropy. We redirect
the reader interested in this topic to the available literature
Criticality_Informative_samples_Marsili ; Marsili_1 ;
Multiscale_relevance_Marsili ; On_sampling_and_modeling_Marsili ;
Mele_Potestio ; HGP .

(a) $p=1$

(b) $p=2$

(c) $p=3$

(d) $p=4$

(e) $p=5$

(f)  Figure 3: Mapping entropy of the $N=10$ Hopfield model plotted as a
function of resolution for five values of $p$. All possible coarse-grained
representations of the system are covered for numbers of retained neurons
$n_{CG}$ ranging from $1$ to $10$. Simulations have been carried out with
parameters $n_{dyn}=1000$, $\tau=3$, $T=0$ (see main text for further
details).

##  III Results and Discussion

###  III.1 Matrix reconstruction and bias detection for an $N=10$ Hopfield
model

We begin to examine the properties of the Hopfield model through the lenses of
mapping entropy considering a small network of $N=10$ at $T=0$, with an amount
of stored patterns in the range $p\in\\{1,..,5\\}$. The relatively small total
number of reduced representations (irrespectively of the resolution level
$n_{cg}$) that can be designed to describe a network of this size,
$n_{map}=2^{10}-1=1023$, allows us to perform an exhaustive exploration of all
of them for every value of $p$ investigated; the obtained results will thus
constitute a benchmark for the following analysis on a larger system discussed
in Sec. III.2.

The mapping entropy and resolution associated with each CG representation of
the system were calculated as detailed in Sec. II.2, providing, at fixed
number of stored patterns, a point in the corresponding $S_{map}\ vs.\ H_{S}$
plane; combined together, these results fully characterise the landscape of
information loss attained while observing a simple, yet nontrivial Hopfield
network in terms of a subset of its constituent units.

The $S_{map}\ vs.\ H_{S}$ graphs arising from this analysis are presented in
Fig. 3, where we report results separately for each value of $p$ and further
label CG mappings according to their degree of coarse-graining $n_{cg}$.
Overall, two features can be readily appreciated: first, and as expected, the
mapping entropy on average decreases as the number of retained neurons
increases, vanishing for $n_{cg}=N$; this is due to the fact that the larger
the number of elements considered in the reduced description, the smaller the
amount of information that is lost on the statistical properties of the high-
resolution reference.

Second, it appears that a linear relation holds between $S_{map}$ and $H_{S}$
at a fixed $n_{cg}$. This behaviour can be rationalised through the definition
of the mapping entropy provided in Eq. II.2, which can be rewritten as

$\begin{split}S_{map}=-H_{S}^{\phi}+H_{S}+\sum_{\Psi}P(\Psi)\ln(\Omega(\Psi)).\end{split}$
(12)

In Eq. 12, $H_{S}^{\phi}=-\sum_{\phi}p(\phi)\ln p(\phi)$ is the fine-grained
resolution of the reference model, a constant quantity that is independent of
the CG mapping. On the contrary, $H_{S}$ is strictly related to the choice of
the reduced representation, hence contributing to variations in $S_{map}$. We
note that this in principle also holds for the third factor in Eq. 12, in
which the logarithm of the degeneracy $\Omega(\Psi)$ of the CG labels is
averaged over the CG probability distribution; for a sufficiently large sample
of high-resolution configurations, however, this term only depends on the
mapping via the amount of retained sites $n_{cg}$, and is thus constant at a
fixed degree of coarse-graining HGP . In such limit, Eq. 12 thus clarifies the
linear dependence of $S_{map}$ on $H_{S}$ observed in Fig. 3, according to
which the lower the coarse-grained resolution of the reduced representation,
the lower its information loss.

At the same time, by further analysing the behaviour of the mapping entropy as
a function of the resolution at a fixed degree of coarse-graining, a closer
inspection of Fig. 3 reveals that, despite the overall linearity of $S_{map}$
in $H_{S}$, for several values of $n_{cg}$ the reduced representations of the
system split into distinct clusters separated by gaps in mapping entropy. This
feature is particularly evident in the case of $p=2$ and $p=3$ (resp. Fig. 3b
and 3c), while it is almost absent for the other amounts of stored patterns.
One is then naturally led to ask why such a “classification” of CG mappings
appears; critically, the reason for this is to be found in the structure of
the synaptic weight matrix $J_{ij}$ characterising the interactions among the
network’s constituents. To better clarify this point, we focus on the simplest
case with $p=2$ retaining $n_{cg}=3$ sites—for which a zoomed version of the
results presented in Fig. 3b is reported in Fig. 4; furthermore, we first
consider a Hopfield network with two “biased” patterns, namely

$\\{\xi^{1}\\}$ = | -1 | -1 | -1 | -1 | -1 | -1 | -1 | 1 | -1 | 1
---|---|---|---|---|---|---|---|---|---|---
$\\{\xi^{2}\\}$ = | -1 | -1 | -1 | -1 | -1 | 1 | 1 | -1 | 1 | -1
(13)

In Eq. 13 the first five neurons have equal memory values in both patterns,
while the memories of the last five are opposite in sign. As a consequence of
the Hebbian rule in Eq. 5, if the product $\xi^{\mu}_{i}\xi^{\mu}_{j}$ of the
memories of a pair of spins $(\sigma_{i},\sigma_{j})$ is equal in all the
stored patterns, the modulus of their coupling achieves its highest possible
value ($|J_{ij}|=p/N$ in the general case of $p$ patterns), thus rendering
these neurons maximally interacting. If this does not hold, cancellations
occur that can also result in a complete decoupling of the pair. In
particular, the memory patterns in Eq. 13 generate the block-diagonal synaptic
weight matrix reported in Eq. 14, in which neurons end up being partitioned
into two groups: units in each group maximally interact with each other (with
$|J_{ij}|=2/10$ for $p=2$), while units from different groups do not. The
first block consists of the first five spins in Eq. 13 that have equal
memories in both patterns, while the other block pertains to the remaining
ones.

$S=\begin{pmatrix}0&0.2&0.2&0.2&0.2&0&0&0&0&0\\\
0.2&0&0.2&0.2&0.2&0&0&0&0&0\\\ 0.2&0.2&0&0.2&0.2&0&0&0&0&0\\\
0.2&0.2&0.2&0&0.2&0&0&0&0&0\\\ 0.2&0.2&0.2&0.2&0&0&0&0&0&0\\\
0&0&0&0&0&0&0.2&$-0.2$&0.2&$-0.2$\\\ 0&0&0&0&0&0.2&0&$-0.2$&0.2&$-0.2$\\\
0&0&0&0&0&$-0.2$&$-0.2$&0&$-0.2$&0.2\\\ 0&0&0&0&0&0.2&0.2&$-0.2$&0&$-0.2$\\\
0&0&0&0&0&$-0.2$&$-0.2$&0.2&$-0.2$&0\end{pmatrix}.$ (14)

The mapping entropy analysis performed on all the possible reduced
representations that can be designed for this biased Hopfield network by
retaining $n_{cg}=3$ neurons is presented Fig. 4c. Also in this case we
observe the appearance of two clusters separated by a gap in $S_{map}$; by
investigating their composition, it emerges that the CG representations that
enter the group of lower $S_{map}$ are the ones that only contain spins coming
both from either the first or the second blocks in Eq. 14, thus consisting of
retained neurons that are maximally interacting among themselves and decoupled
with the reminder. Constructing “mixed” CG mappings that gather interacting as
well as noninteracting spins results in a higher information loss on the
properties of the original network. This result, obtained with a model crafted
_ad hoc_ , is consistently found in the $p=2$, random patterns model of Fig.
3b, see Fig. 4b. In fact, also in this case the mappings belonging to the
lower $S_{map}$ cluster are those retaining a subset of spins whose memory
states are either identical or are opposite in sign between the two patterns.

(a)

(b)

(c)

Figure 4: Panel (a): close-up of Fig. 3b, showing the mapping entropy for
mappings with $n_{cg}=3$. Panel (b): same as panel (a), where mappings are
coloured depending on the properties of the retained neurons. Red dots:
mappings that contain only neurons whose memories have the same value on all
patterns. Orange dots: mappings that contain only neurons whose memories are
opposite. Light blue dots: mappings that contain neurons coming from both
latter groups. Panel (c): mapping entropy for mappings with $n_{cg}=3$ of an
$N=10$ Hopfield model with biased patterns, see Eq. 13. Red dots: mappings
that retain only spins from the first five (identical among the patterns).
Orange dots: mappings that retain only spins from the last five. Light blue
dots: mappings that retain spins from both the first and the second half.

(a) $p=1$

(b) $p=2$

(c) $p=3$

(d) $p=4$

(e) $p=5$

Figure 5: Mapping entropy plotted as a function of resolution for $p=1,\cdots\
5$, for the coarse-grained representations of the system that minimize the
loss of information at fixed $n_{cg}$. The graphs refer to the same model
studied in Fig. 3.  (a)

(b)

(c)

(d)

(e)

(f)

(g)

(h) $p=2$

(i) $p=3$

(j) $p=4$

Figure 6: Heat maps of the synaptic matrices of three $N=10$ Hopfield models
with different numbers of memorized patterns $p$. The darker the shade of
blue, the stronger the coupling (in absolute value). Top row: the three
original matrices (absolute values). Middle row: the matrices reconstructed
with the information derived from the relevant neurons identified by the
mapping entropy. Bottom row: the same matrices of the middle row but with the
addition of the signs obtained from the analysis of time correlations
$C_{ij}$. The red symbols indicate those signs that differ from the real ones.

Let us now investigate more in detail the behaviour of mapping entropy for a
varying number of retained neurons in the low-resolution description of the
network; in particular, we focus on the specific set of mappings that
minimises the mapping entropy at any given value of $n_{cg}$, representing the
maximally informative reduced representations that can be designed for the
$N=10$ Hopfield network at each degree of coarse-graining. The results of this
analysis are presented in Fig. 5, and highlight that although, as previously
discussed, $S_{map}$ decreases for increasing $n_{cg}$, such decrease is
interrupted by some “flattenings” that occur between pairs of consecutive
values $(\bar{n}_{cg},\bar{n}_{cg}+1)$. This behavior is more evident for
$p=2$ and $p=3$, reduces to a softer change of slope for $p=4$ and vanish at
$p=1\ ,5$. Critically, such “steps” in $S_{map}$ suggest that adding a neuron
to a maximally informative mapping with $n_{cg}=\bar{n}_{cg}$ retained neurons
allows little or no further information gain about the system; the retained
spin set minimizing the mapping entropy at fixed $n_{cg}=\bar{n}_{cg}$ thus
stands out from the others. This behaviour is comparable to the one observed
by Giulini and coworkers HGP  for a discrete system of non-interacting spins,
where they observed an entire range of $n_{cg}$ values featuring almost the
same mapping entropy minimum. In that case, this behaviour could be explained
by noting that the mappings minimising $S_{map}$ most frequently included
spins whose probability to be in a given state (e.g. $+1$) was appreciably
different from that of the others, the latter being prone to be treated as
noise. Here we obtained a similar result, despite the crucial difference of
dealing with a strongly interacting system.

Based on the observations discussed insofar, one could expect the group of
relevant neurons $\\{s_{i_{\nu}}^{*}\\}_{\nu=1,...,\bar{n}_{cg}}$ highlighted
by the step in $S_{map}$ to be characterized by strong couplings among them.
We could also argue that the couplings with neurons outside this group should
be weaker, which is another argument in favour of the interpretation of the
discarded neurons’ dynamics as noise. In fact, the presence of a further
neuron strongly coupled with the group of retained ones would have shifted the
flattening to $n_{cg}=\bar{n}_{cg}+1$, and this additional neuron would have
been itself a part of the retained group. In light of these considerations,
what these results suggest is that the minimization of the mapping entropy
highlights which neurons are practically decoupled from the rest of the
network.

In order to test the validity of this hypothesis for the set of relevant
neurons, we reshuffled the order of the spins inside the synaptic matrix,
grouping the retained spins $\\{s_{i_{\nu}}^{*}\\}$ in the first
$\bar{n}_{cg}$ columns/rows; we then calculated then matrix semi-dispersion
as:

$\displaystyle\rho=\frac{\langle\absolutevalue{J}\rangle_{s^{*}s^{*}}-\langle\absolutevalue{J}\rangle_{ss^{*}}}{\langle\absolutevalue{J}\rangle_{s^{*}s^{*}}+\langle\absolutevalue{J}\rangle_{ss^{*}}}$
(15) $\displaystyle J=\left(\begin{array}[]{c|c}s^{*}s^{*}&s^{*}s\\\ \hline\cr
s\,\,s^{*}&s\,\,s\end{array}\right),$ (18)

where $\langle\absolutevalue{J}\rangle_{A}$ stands for the absolute coupling
averaged over the elements of block $A$. $\rho$ quantifies in a single value
the properties of the retained neurons we want to investigate, that is, the
strength of the couplings within the group of retained spins relative to that
of the couplings between retained and discarded ones. If the retained spins
interact much more strongly among themselves than with the discarded ones,
$\rho>0$.

To gain further insight, we identify and rank the previously discussed “steps”
that appear in the plots of $S_{map}$ vs. $H_{S}$, see Fig. 5, by calculating
the increment of the discrete first derivative of $S_{map}(H_{S})$ (something
akin to a finite-difference second order derivative):

$\begin{split}\Delta(n_{cg})=&\frac{S_{map}(n_{cg}-1)-S_{map}(n_{cg})}{H_{s}(n_{cg}-1)-H_{s}(n_{cg})}+\\\
&-\frac{S_{map}(n_{cg})-S_{map}(n_{cg}+1)}{H_{s}(n_{cg})-H_{s}(n_{cg}+1)}.\end{split}$
(19)

With this definition, we have that for $n_{cg}=\bar{n}_{cg}$ the quantity
$\Delta$ is minimized. Table 1 shows the three mappings with lowest values of
$\Delta$ (ordered by $\Delta$ in ascending order) and their related semi-
dispersions $\rho$ for the $p=3$ model; the last column reports the maximum
possible value of semi-dispersion at fixed $n_{cg}$ for the $p=3$ synaptic
matrix.

$n_{cg}$ | Mapping | $\Delta$ | $\rho$ | $\rho_{max}$
---|---|---|---|---
4 | [$s_{2},s_{3},s_{5},s_{6}$] | $-0.85$ | $0.50$ | $0.50$
7 | [$s_{2},s_{3},s_{5},s_{6},s_{8},s_{9},s_{10}$] | $-0.81$ | $0.30$ | $0.30$
2 | [$s_{2},s_{5}$] | $0.14$ | $0.41$ | $0.41$
Table 1: List of the mappings that minimise the quantity $\Delta$ defined in
Eq. 19. $n_{cg}$ is the number of neurons retained in the mapping; the Mapping
column lists the spins contained in the low-resolution representation; the
last three columns report the values of $\Delta$, the semi-dispersion $\rho$,
and the largest value of the latter that can be attained in the system for
that particular coarse-graining level.

The results reported in Tab. 1 confirm our expectations, and so do those for
$p=2$ and $p=4$ (data not shown), but the observed signal weakens for higher
numbers of memorized patterns. This happens for two reasons: first, with
higher numbers of memories the system moves towards the spin glass phase, and
the efficiency of memory retrieval decreases because of the rising internal
noise hopfield1982neural  and due to the growth of non-retrieval states MBF .
Second, increasing $p$ the synaptic matrix will statistically feature fewer
strongly coupled neurons.

(a) $p=2$, biased patterns

(b) $p=5$, random patterns

(c) $p=5$, biased patterns

(d)

Figure 7: Minima of the mapping entropy plotted as a function of resolution
for every number $n_{cg}$ of retained spins. Panel (a): results from a
simulation of an $N=10$, $p=2$ Hopfield model with biased patterns described
by Eq. 13. Panel (b): results from the same simulation of an $N=10$, $p=5$
Hopfield model presented in Fig. 5e. Panel (c): results from a simulation of
an $N=10$, $p=5$ Hopfield model with biased patterns described by Eq. 21.

The information about the coupling among neurons, which the minimisation of
the mapping entropy allowed us to infer, can be leveraged to reconstruct,
although approximately, the entire synaptic matrix $J$. To this end, we employ
the data pertaining to the first two groups of informative neurons, identified
thanks to Eq. 19. More specifically, we first calculate the value of
$\Delta(n_{cg})$ for every CG mapping with minimum value of mapping entropy at
fixed $n_{cg}$; second, we rank these groups of neurons by increasing value of
$\Delta$; third, we focus on the first two groups of this rank. We then make
use of the fact that these groups of neurons feature high values of synaptic
matrix semi-dispersion, and assume that this is due to strong couplings inside
each group and weak couplings with discarded neurons.

In practice, we initialise our reconstructed $S$ matrix as a null matrix, so
that all neurons are initially decoupled. We then leverage our knowledge about
the matrix semi-dispersion and increase by $1$ 222We chose to increase and
decrease the couplings by an arbitrary value since we are not interested in
the absolute magnitude of the coupling (the global order of magnitude of the
couplings is irrelevant for the dynamics of the Hopfield model
hopfield1982neural ), but rather on the relative magnitude among couplings.
the value of all couplings within the first group of most informative (high
semi-dispersion) neurons. Conversely, we decrease by $1$ all couplings between
the aforementioned neurons and the other ones (we recall that high semi-
dispersion implies the presence of a group of strongly coupled neurons with
week interactions with the others). We repeat the same procedure with the
second group of informative neurons. What we get at the end of this heuristic
procedure is a reconstructed $S$ matrix with positive and negative couplings.
The elements of this matrix are not an estimate of the real couplings of the
model, but rather an approximated ranking of their absolute strength. This
means that the higher the value given to a “reconstructed” coupling, the
stronger the real coupling (in absolute value) should be compared to the other
ones.

To give an example, we can look at Tab.1 and try to reconstruct the synaptic
matrix of the model in absolute value. The group of neurons corresponding to
the lowest value of $\Delta$ is $\\{s_{2},s_{3},s_{5},s_{6}\\}$. Thus we
proceed by increasing by $1$ all couplings between these neurons and
decreasing by $-1$ all couplings between them and the remaining group
$\\{s_{1},s_{4},s_{7},s_{8},s_{9},s_{10}\\}$. Then we consider the second
lowest value of $\Delta$ and its related mapping
$\\{s_{2},s_{3},s_{5},s_{6},s_{8},s_{9},s_{10}\\}$ and perform the same
modifications to the matrix as before. What we get at the end is the following
matrix:

$S=\begin{pmatrix}$0$&$-2$&$-2$&$0$&$-2$&$-2$&$0$&$-1$&$-1$&$-1$\\\
$-2$&$0$&$2$&$-2$&$2$&$2$&$-2$&$0$&$0$&$0$\\\
$-2$&$2$&$0$&$-2$&$2$&$2$&$-2$&$0$&$0$&$0$\\\
$0$&$-2$&$-2$&$0$&$-2$&$-2$&$0$&$-1$&$-1$&$-1$\\\
$-2$&$2$&$2$&$-2$&$0$&$2$&$-2$&$0$&$0$&$0$\\\
$-2$&$2$&$2$&$-2$&$2$&$0$&$-2$&$0$&$0$&$0$\\\
$0$&$-2$&$-2$&$0$&$-2$&$-2$&$0$&$-1$&$-1$&$-1$\\\
$-1$&$0$&$0$&$-1$&$0$&$0$&$-1$&$0$&$1$&$1$\\\
$-1$&$0$&$0$&$-1$&$0$&$0$&$-1$&$1$&$0$&$1$\\\
$-1$&$0$&$0$&$-1$&$0$&$0$&$-1$&$1$&$1$&$0$\end{pmatrix}.$ (20)

Fig. 6 shows some of the reconstructed $S$ matrices (second row) we obtained
with this method compared with the real ones (first row). The last row of the
figure shows the results of a tentative derivation of the coupling signs,
which would complete the synaptic matrix reconstruction. In order to determine
the sign of a coupling $J_{ij}$, we compute the correlation
$C_{ij}\equiv\langle s_{i}s_{j}\rangle$ between the corresponding spin pair,
averaging over the whole configuration sample. The coupling $J_{ij}$ is
attributed the sign of the associated correlation; in contrast, the coupling
is set to $0$ if $|C_{ij}|<0.1C^{\star}$, where
$C^{\star}\equiv\sup_{ij}|C_{ij}|$ is the largest absolute value of the
correlation between all spin pairs. The results mirror the original signs with
the exception of a few errors for the $p=4$ model.

Another way to leverage the information given by $S_{map}$ is to identify
possible biases introduced in the definition of the model. If we take, for
example, the $N=10$, $p=2$ Hopfield model with patterns given by Eq. 13, the
mapping entropy curve in Fig. 7a highlights that increasing from $5$ to $6$
the number of retained neurons does not correspond to an appreciable increase
in the information content of the reduced representation (that is, $S_{map}$
remains practically the same). Notably, the particular value $n_{cg}=5$ is a
consequence of the fact that the first five spins have identical states in the
memory patterns.

We then apply the same process to a $p=5$ Hopfield model; we thus impose the
values of the first $m$ spins of each one of the five patterns so that they
are equal or opposite. Consider for example the following case:

$p_{1}$: | -1 | -1 | 1 | 1 | -1 | -1 | 1 | -1 | 1 | -1,
---|---|---|---|---|---|---|---|---|---|---
$p_{2}$: | -1 | -1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1,
$p_{3}$: | 1 | 1 | -1 | -1 | -1 | -1 | 1 | -1 | 1 | -1,
$p_{4}$: | 1 | 1 | -1 | -1 | 1 | 1 | 1 | -1 | -1 | 1,
$p_{5}$: | -1 | -1 | 1 | 1 | 1 | -1 | -1 | 1 | -1 | -1
(21)

Here we imposed that the first $m=4$ spins have either equal memories in
patterns $p_{1}$, $p_{2}$, $p_{5}$ and opposite ones in $p_{3}$, $p_{4}$. In
this way, the couplings $J_{ij}$ for $i\neq j\in\\{1,2,3,4\\}$ will all have
maximum absolute value equal to $p/N$. Figures 7b and 7c show a comparison
between the minima of the mapping entropy for simulations with random and
biased patterns, respectively. As we previously argued, in the absence of a
bias the model with $p=5$ doesn’t present any clear decrease of the
information loss rate, as quantified by the quantity $\Delta$ in Eq. 19, while
increasing $H_{S}$, see Fig. 5; on the contrary, in Fig. 7b we can see that
this rate actually increases, that is, the $S_{map}$-vs.-$H_{S}$ curve becomes
steeper. This does not happen in the case of biased patterns: Fig. 7c shows
that, after the $S_{map}$ minimum related to $n_{cg}=4$, the information loss
rate actually decreases slightly. The group of relevant neurons highlighted by
the mapping entropy is thus composed by four spins and these correspond
exactly to the first four biased spins $\\{s_{1},s_{2},s_{3},s_{4}\\}$.

###  III.2 CG’ing regimes for an $N=100$ Hopfield model

(a) $p=4$

(b) $p=5$

(c)

Figure 8: Mapping entropy minima for a Hopfield model with $N=100$ neurons,
for $n_{cg}=1,\ \cdots\ 20$ plotted as a function of resolution for $p=4$
(panel a) and $p=5$ (panel b). The simulated Hopfield model had $N=100$
neurons.

The larger the size of a neural network, the richer and more complex its
behaviour, which is at least in part the reason behind the emergent properties
of our brain MBF . Dealing with a finite but large number of neurons $N$,
however, makes the network investigation through the MEOW approach more
difficult; in fact, the number of possible mappings that we have to take into
account when performing decimation grows like $2^{N}$, preventing us from
carrying out an exhaustive enumeration of all possible coarse-grained
representations of the system. Nonetheless, resuming the discussion in the
previous section, what we are interested in is not an extensive analysis of
the whole CG mapping space, but rather the properties of the mappings that
minimise the mapping entropy as a function of resolution.

To this end, we implemented the mapping entropy optimisation workflow relying
on a simulated annealing (SA) minimisation strategy ref_62_JCTC ; ref_63_JCTC
, as it was originally done by Giulini _et al._ in Ref. JCTC . We performed
simulations of an $N=100$ Hopfield network at $T=0.2$ for different values of
$p\in$ {2, 3, …, 10}; these sets of parameters were chosen to fall within the
retrieval phase of the infinite-size model, see Sec. II.1. We then looked for
the maximally informative CG representations of the system, i.e. the ones that
minimise the mapping entropy at fixed $n_{cg}\in$ {1, 2, …, 20}. The plots for
a subset of the different results obtained, in particular those that pertain
to the lowest values of mapping entropy for $p=4,\ 5$, are shown in Fig. 8.

For $2\leq p\leq 8$, starting from the coarser representations, the
information loss decreases with an approximately constant rate until it
reaches a sort of “inflection region”. Here, the steepness of the $S_{map}\
vs.\ H_{S}$ curve decreases considerably before it grows again, and the
mapping entropy continues its descent. The inflection region includes CG
mappings relative to two to four values of $n_{cg}$, and shows a smearing of
these mappings on a broad interval of resolutions. We thus observe a
resolution gap between the two regions where the mapping entropy decreases at
a constant rate.

(a) $p=4$

(b) $p=5$

(c)

Figure 9: Resolutions of the CG representations visited during SA runs at
different values of $n_{cg}$ and $p=4$ (panel a) and $p=5$ (panel b). The
various resolutions explored in one SA process at given $n_{cg}$ and $p$ have
the same color, and the ones corresponding to the $S_{map}$ minima found by
the algorithm are indicated by black circles. The results for the different
runs at fixed $p$ are plotted side by side to make it easier to compare the
resolutions at which information loss is minimized for varying numbers of
retained neurons.

To rationalise these observations we focus on the role played by resolution;
more specifically, we exploit the SA algorithm by keeping track of all the
$H_{S}$ values of the mappings visited during the minimization procedure. The
results of this analysis for $p=4$ and $p=5$ are shown in Fig. 9.

For values of $n_{cg}$ smaller than the ones that constitute the inflection
region ($n_{cg}<5$ for $p=4,\ 5$), the CG representations that minimize the
mapping entropy are also associated with low values of the resolution; in
particular, in the course of the optimisation, the tentative mappings
typically have larger values of $H_{S}$ than those that are, eventually,
selected for their low $S_{map}$. This behaviour is consistent with the
analysis carried out on the $N=10$ model, for which the mapping entropy
minimum at fixed $n_{cg}$ had the lowest resolution because of the linear
relationship between the two measures, see Eq. 12.

This picture is challenged by the optimal mappings retaining a number of
neurons located after the inflection region ($n_{cg}>7$ for $p=4,\ 5$); these
attain _high_ resolution values, see Fig. 9, that is, the SA minimization of
the mapping entropy converges towards low-compressed, higher-resolution CG
representations. Mappings that decrease the resolution $H_{S}$, in fact,
contribute to lowering $S_{map}$ if and only if the term
$\sum_{\phi}p(\phi)\text{log}(\Omega(\Psi_{\phi}))$ (Eq. 12) does not
increase. Here, instead, we have that low values of mapping entropy correspond
to a high $H_{S}$ counterbalanced by a decrease in the configuration space
term, which is in turn due to the limited number of resolved CG configurations
HGP .

Finally, the inflection region, where we see a smearing of the resolution
values for fixed $n_{cg}$ and almost constant $S_{map}$, corresponds to a
changeover situation in which informative mappings cover almost the entire
range of the available values of resolution. This means that both high coding
cost and highly compressed representations can be found, that lose the same
amount of information.

(a) $p=2$

(b) $p=3$

(c) $p=4$

(d) $p=5$

(e) $p=6$

(f) $p=7$

(g) $p=8$

(h) $p=9$

(i) $p=10$

Figure 10: Probability densities of the semi-dispersion for the three
different resolution regimes. In light gray: $n_{cg}$’s lower than the
inflection region. In dark gray: $n_{cg}$’s higher than the inflection region.
In green: $n_{cg}$’s that correspond to the inflection region. Each panel
corresponds to a different value of $p$, ranging from $2$ to $10$.

Another informative way of reading the $S_{map}-H_{S}$ graphs is to inspect
the low $S_{map}$ mappings starting from those with low $n_{cg}$ (on the left
of the inflection region) and studying their properties as the resolution is
increased. As we pointed out for the model with $N=10$, the presence of a
sudden decrease in the information loss rate suggests that increasing the
detail of our CG description does not provide any significant gain in
information about the reference system. The minimisation of the mapping
entropy is then calling our attention to these optimal mappings, which we can
investigate by looking at their values of synaptic matrix semi-dispersion, see
Eq. 15.

Fig. 10 displays, for each value of $p$ under examination, the semi-dispersion
distributions of the CG mappings that minimize the mapping entropy. We
intentionally distinguished between mappings relative to resolutions lower
than, included in, or higher than the inflection region. This highlights once
again a gap that separates mappings before and after this region: low-
resolution mappings (in light grey) present high values of semi-dispersion,
while the latter is low or even negative for high-resolution mappings (in dark
grey). The CG representations that form the inflection region (shown in green
in the plots) take instead intermediate semi-dispersion values, signaling a
transition between these two regimes.

We thus have two distinct regimes of CG representations, corresponding to
different values of the resolution as well as the semi-dispersion, separated
by a third, crossover interval. CG mappings tend to fall either in one regime
or the other depending on their number of retained neurons, with the exception
of a few specific values of $n_{cg}$ that constitute the intermediate phase
where the distribution is wider and covers a large range of semi-dispersion
values.

The first regime includes all the mappings with lower numbers of retained
neurons: in this case, the mapping entropy minimization favours CG
representations with high values of semi-dispersion and high compression,
which means that the most informative way of describing the reference model is
to divide it into almost disconnected groups. Every highly compressed
informative mapping constitutes a block of strongly interacting neurons that
are effectively decoupled from the rest of the network, the dynamics of the
latter being seen as an effective noise. The more neurons we retain in our
description, the fewer strongly coupled spins are present and the more their
dynamics will depend on the discarded neurons, until this strategy eventually
becomes inadequate.

The second regime shows us an alternative strategy, which includes all the
mappings with higher numbers of retained neurons: when their number becomes
too large for a neat decoupling of their group from the rest, the most
informative description of the model features null or negative values of semi-
dispersion; this indicates that the optimal representation involves weakly
interacting neurons, which are often tightly coupled with the discarded ones.
When this external interactions are strong, the statistical behaviour of the
discarded spins is tightly bound to that of the retained ones; hence, high-
resolution, informative mappings are seemingly able to describe the whole
network state with the greatest detail, enabling the distinction between the
different retrieved patterns.

The third class of mappings to discuss is that of the inflection region, which
covers almost entirely the range of semi-dispersion separating the two
aforementioned regimes. In this case, both high- and low-resolution
descriptions, retaining strongly and weakly coupled groups of retained neurons
respectively, can be equally informative low-dimensional representations of
the reference system.

The behaviour discussed insofar occurs for all values of $p\leq 9$. For
$p=10$, instead, we have a substantial collapse of all semi-dispersion
distributions onto the same range, which indicates the onset of the spin glass
phase and the loss of a well-defined crossover region between compressed and
low-compression mappings.

(a) $p=4$

(b) $p=5$

(c) $p=6$

Figure 11: Heat maps of the probability $P_{cons}$ that a given neuron is
retained in an optimal mapping, plotted as a function of the total number of
retained neurons $n_{cg}$ for a model with $N=100$ and $p=4,\ 5,\ 6$. Each row
corresponds to a specific neuron $i$; neurons on the $y$ axis are sorted in
descending order of the mean probability $P_{cons}$ to be retained in the
high-resolution regime. The latter corresponds to the interval of $n_{cg}$
located of the right of the rightmost white vertical line; correspondingly,
the low-resolution and inflection regimes correspond to the first and middle
intervals, respectively.

Lastly, we focus on the probability $P_{cons}$ for an individual neuron to be
retained in an optimal mapping. $P_{cons}$ is a function of $n_{cg}$, and it
is simply obtained as the empirical occurrence of a given neuron in the pool
of least-$S_{map}$ mappings obtained by MEOW. Fig. 11 illustrates this
probability as a function of the number of retained neurons $n_{cg}$. To
organise the data, we computed the average $P_{cons}$ for each neuron over the
entire high-resolution regime, and sorted them accordingly in decreasing order
along the $y$ axis. In the graphs, two vertical lines mark the three regimes
previously discussed: it is quite evident that the neurons showing high
conservation probabilities in the two main regimes (high and low semi-
dispersion) are rarely the same. This is consistent with our understanding of
the various $S_{map}$ optimisation strategies: in fact, if a neuron is
retained in an optimal mapping ascribed to the high-resolution regime, it
would likely present relatively strong couplings with a broad number of
neurons. On the other hand, a neuron typically retained in a low-resolution
regime mapping will present very strong couplings only with few other neurons,
and it is likely to be almost decoupled from the rest. In light of this, we
can rationalise why the groups of neurons with high conservation probability
in the two sectors relative to these regimes tend to be distinct.

This analysis thus illustrates that the answer to the search for the most
informative neurons of a network will critically depend on the chosen
resolution scale, that is, the value of $n_{cg}$ in relation to the size of
the configuration sample.

##  IV Conclusions

In this work we carried out a study of the Hopfield model through the lenses
of information-theoretic measures, expanding on the work done by Giulini and
coworkers HGP  and Roudi and coworkers On_sampling_and_modeling_Marsili ;
Marsili_1 . Specifically, we investigated the properties of low-resolution
representations of the model making use of the mapping entropy optimization
workflow (MEOW) and an analysis based on resolution, a measure of the level of
detail inherent in a dataset of coarse representations of a system’s states.
Applying these tools to the configurations sampled in a number of simulations
of different Hopfield networks, we observed that the low-resolution mappings
with the lowest values of mapping entropy reveal information about intrinsic
properties of the reference system.

We first addressed the study of a Hopfield model with $N=10$ neurons. Focusing
on the behaviour of the mapping entropy minima at varying resolution, we
observed the presence of sudden decreases of the information loss rate that
highlighted specific groups of neurons; these proved to have strong
interactions among themselves and weak couplings with the discarded ones. As a
simple, quantitative measure of this partitioning into groups we employed the
semi-dispersion of the average strength of the synaptic matrix elements, which
governs the interaction between neurons in the various groups; when looking at
the mappings that minimise the mapping entropy in correspondence of these
decrease rate variations, we observe that the semi-dispersion is maximized.
This is suggestive of the fact that mapping entropy minima are in
correspondence of mappings that retain strongly interacting neurons, while the
reminder is weakly coupled with the first group as well as within itself. This
result is in line with the one discussed in HGP  for a discrete, non-
interacting model. The information obtained from mapping entropy and
resolution, combined with the study of the correlation between neurons,
allowed us to approximately reconstruct the synaptic matrix of Hopfield models
with $p\leq 4$; furthermore, by inspecting the properties of mappings in
correspondence of decreases of the information loss rate we were able to
“detect” the presence of biases in the patterns employed to build the synaptic
matrix according to the Hebbian rule.

Moving to Hopfield models with $N=100$ neurons, we showed that the minima of
the mapping entropy feature a regular, linearly decreasing trend as a function
of resolution, with the exception of a relatively flat region where the value
of $S_{map}$ remains constant for a given number $n_{cg}$ of retained neurons.
This behaviour allowed us to distinguish three regions and, correspondingly,
three qualitatively distinct behaviours.

The first regime pertains to the most informative CG mappings with the lowest
resolutions, or, equivalently, with the lowest $n_{cg}$. These mappings
correspond to some of the most compressed representations that can be obtained
at those values of $n_{cg}$ and their related semi-dispersion takes positive
values. The conclusion is that, for low numbers of retained neurons, the
mapping entropy selects those representations that describe specific clusters
of neurons with strong internal interactions; the latter can thus be
effectively decoupled from the rest of the network, which is treated as
effective noise. This is the same coarse-graining mechanism encountered for
the $N=10$ model.

A rather opposite outcome is obtained for the second CG’ing regime, where we
find the most informative CG mappings with higher resolutions and higher
$n_{cg}$. These mappings correspond to some of the least compressed
representations that can be obtained at those $n_{cg}$, and their related
semi-dispersion values are low or even negative. For higher numbers of
retained neurons, the mapping entropy optimisation workflow selects those
representations that contain spins having weak couplings between each other,
and strong couplings with the discarded ones; the optimal description of the
entire network state is thus obtained retaining those that strongly correlate
with the neglected ones.

The third CG regime, whereby the mapping entropy remains constant, is made up
of equally informative mappings that, however, vary appreciably in terms of
resolution as well as semi-dispersion. These representations entail the same
amount of information in spite of the fact that their “coarse-graining
strategy” shifts continuously from the first to the second; such a degeneracy
indicates the existence of an “information plateau”, a buffer region where
varying the representations of the system does not appreciably increases or
decreases its informativeness.

In conclusion, this work has shown that it is possible to gather useful
knowledge about a model neural network through the analysis of the information
content of its low-resolution representations; in particular, we have seen
that the elements of the network that are pinpointed as the most informative
ones depend on the specific coarse-grained resolution level at which the
system is described: this result can serve as a guide in the study of complex
systems as well as in the construction of effective models of the latter, and
paves the way to the development of a semi-automated protocol for the analysis
of limited data gathered from systems composed by a large number of
constituents.

##  V Acknowledgments

The authors are indebted with Matteo Marsili and Margherita Mele for an
insightful reading of the manuscript and useful comments. RP acknowledges
support from ICSC - Centro Nazionale di Ricerca in HPC, Big Data and Quantum
Computing, funded by the European Union under NextGenerationEU. Views and
opinions expressed are however those of the author(s) only and do not
necessarily reflect those of the European Union or The European Research
Executive Agency. Neither the European Union nor the granting authority can be
held responsible for them. Funded by the European Union under
NextGenerationEU. PRIN 2022 PNRR Prot. n. P2022MTB7E.

##  VI Author contributions

RP conceived the study. RP and RM proposed the method. RA carried out the
simulations and the preliminary data analyses. All authors contributed to the
analysis and interpretation of the data. All authors drafted the paper,
reviewed the results, and approved the final version of the manuscript.

## References

  * (1) E. Rolls, A. Treves, et al., Neural networks and brain function.  Oxford University press, 1998.
  * (2) P. Dayan and L. F. Abbott, Theoretical neuroscience: computational and mathematical modeling of neural systems.  MIT press, 2005.
  * (3) G. Zamora-López, E. Russo, P. M. Gleiser, C. Zhou, and J. Kurths, “Characterizing the complexity of brain and mind networks,” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 369, no. 1952, pp. 3730–3747, 2011.
  * (4) C. W. Lynn and D. S. Bassett, “The physics of brain network structure, function and control,” Nature Reviews Physics, vol. 1, no. 5, pp. 318–332, 2019.
  * (5) S. Herculano-Houzel, “The human brain in numbers: a linearly scaled-up primate brain,” Frontiers in human neuroscience, p. 31, 2009.
  * (6) A. Joudaki, N. Salehi, M. Jalili, and M. G. Knyazeva, “Eeg-based functional brain networks: does the network size matter?,” PloS one, vol. 7, no. 4, p. e35673, 2012.
  * (7) E. Kropff and A. Treves, “The storage capacity of potts models for semantic memory retrieval,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2005, no. 08, p. P08010, 2005.
  * (8) E. Russo, V. M. Namboodiri, A. Treves, and E. Kropff, “Free association transitions in models of cortical latching dynamics,” New Journal of Physics, vol. 10, no. 1, p. 015008, 2008.
  * (9) D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, “A learning algorithm for boltzmann machines,” Cognitive science, vol. 9, no. 1, pp. 147–169, 1985\.
  * (10) E. Aarts and J. Korst, Simulated annealing and Boltzmann machines: a stochastic approach to combinatorial optimization and neural computing.  John Wiley & Sons, Inc., 1989.
  * (11) C. Marullo and E. Agliari, “Boltzmann machines as generalized hopfield networks: a review of recent results and outlooks,” Entropy, vol. 23, no. 1, p. 34, 2020.
  * (12) S. Goel, A. Klivans, and F. Koehler, “From boltzmann machines to neural networks and back again,” Advances in Neural Information Processing Systems, vol. 33, pp. 6354–6365, 2020.
  * (13) O. Sporns, “Network neuroscience,” The Future of the Brain: Essays by the World’s Leading Neuroscientists, p. 90, 2014.
  * (14) D. S. Bassett, P. Zurn, and J. I. Gold, “On the nature and use of models in network neuroscience,” Nature Reviews Neuroscience, vol. 19, no. 9, pp. 566–578, 2018.
  * (15) T. P. Vogels, K. Rajan, and L. Abbott, “Neural network dynamics,” Annual Review of Neuroscience, vol. 28, no. 1, pp. 357–376, 2005.
  * (16) J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities,” Proceedings of the National Academy of Sciences, vol. 79, no. 8, pp. 2554–2558, 1982.
  * (17) J. J. Hopfield, “Neurons with graded response have collective computational properties like those of two-state neurons.,” Proceedings of the National Academy of Sciences, vol. 81, no. 10, pp. 3088–3092, 1984.
  * (18) A. D. Bruce, E. J. Gardner, and D. J. Wallace, “Dynamics and statistical mechanics of the hopfield model,” Journal of Physics A: Mathematical and General, vol. 20, no. 10, p. 2909, 1987.
  * (19) H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, et al., “Hopfield networks is all you need,” arXiv preprint arXiv:2008.02217, 2020\.
  * (20) A. Barra, G. Genovese, and F. Guerra, “The replica symmetric approximation of the analogical neural network,” Journal of Statistical Physics, vol. 140, pp. 784–796, 2010.
  * (21) E. Agliari, D. Migliozzi, and D. Tantari, “Non-convex multi-species hopfield models,” Journal of Statistical Physics, vol. 172, no. 5, pp. 1247–1269, 2018.
  * (22) E. Agliari, L. Albanese, A. Barra, and G. Ottaviani, “Replica symmetry breaking in neural networks: a few steps toward rigorous results,” Journal of Physics A: Mathematical and Theoretical, vol. 53, no. 41, p. 415005, 2020.
  * (23) F. Alemanno, L. Camanzi, G. Manzan, and D. Tantari, “Hopfield model with planted patterns: a teacher-student self-supervised learning model,” arXiv preprint arXiv:2304.13710, 2023.
  * (24) A. Crisanti, D. J. Amit, and H. Gutfreund, “Saturation level of the hopfield model for neural network,” Europhysics Letters, vol. 2, no. 4, p. 337, 1986\.
  * (25) J. Bruck, “On the convergence properties of the hopfield model,” Proceedings of the IEEE, vol. 78, no. 10, pp. 1579–1585, 1990.
  * (26) H. Huang, Statistical mechanics of neural networks.  Springer, 2021.
  * (27) M. Giulini, R. Menichetti, M. S. Shell, and R. Potestio, “An information-theory-based approach for optimal model reduction of biomolecules,” Journal of Chemical Theory and Computation, vol. 16, no. 11, pp. 6795–6813, 2020.
  * (28) F. Errica, M. Giulini, D. Bacciu, R. Menichetti, A. Micheli, and R. Potestio, “A deep graph network–enhanced sampling approach to efficiently explore the space of reduced representations of proteins,” Frontiers in Molecular Biosciences, vol. 8, p. 637396, 2021.
  * (29) R. Holtzman, M. Giulini, and R. Potestio, “Making sense of complex systems through resolution, relevance, and mapping entropy,” Phys. Rev. E, vol. 106, 2022.
  * (30) M. Giulini, R. Fiorentini, L. Tubiana, R. Potestio, and R. Menichetti, “Excogito, an extensible coarse-graining toolbox for the investigation of biomolecules by means of low-resolution representation,” 2024.
  * (31) S. Kullback and R. A. Leibler, “On information and sufficiency,” The annals of mathematical statistics, vol. 22, no. 1, pp. 79–86, 1951.
  * (32) M. S. Shell, “The relative entropy is fundamental to multiscale and inverse thermodynamic problems,” The Journal of Chemical Physics, vol. 129, no. 14, 2008.
  * (33) J. F. Rudzinski and W. Noid, “Coarse-graining entropy, forces, and structures,” The Journal of chemical physics, vol. 135, no. 21, 2011.
  * (34) W. Noid, “Perspective: Advances, challenges, and insight for predictive coarse-grained models,” The Journal of Physical Chemistry B, 2023.
  * (35) W. McCulloch and W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” Bulletin of Mathematical Biophysics, no. 2, p. 115–133, 1943.
  * (36) F. Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms.  Spartan Books, 1962.
  * (37) M. Minsky and S. Papert, Perceptrons: an introduction to computational geometry.  MIT Press, 1969.
  * (38) D. J. Amit, H. Gutfreund, and H. Sompolinsky, “Storing infinite numbers of patterns in a spin-glass model of neural networks,” Phys. Rev. Lett., vol. 55, pp. 1530–1533, Sep 1985.
  * (39) D. J. Amit, Modeling Brain Function: The World of Attractor Neural Networks.  Cambridge University Press, 1989.
  * (40) R. J. Glauber, “Time-dependent statistics of the ising model,” Journal of Mathematical Physics, vol. 4, no. 2, pp. 294–307, 1963.
  * (41) D. J. Amit, H. Gutfreund, and H. Sompolinsky, “Spin-glass models of neural networks,” Phys. Rev. A, vol. 32, pp. 1007–1018, 1985.
  * (42) D. J. Amit, H. Gutfreund, and H. Sompolinsky, “Statistical mechanics of neural networks near saturation,” Annals of Physics, vol. 173, no. 1, pp. 30–67, 1987.
  * (43) D. O. Hebb, The organization of behavior: a neuropsychological theory.  Wiley, 1949.
  * (44) W. G. Noid, “Perspective: Coarse-grained models for biomolecular systems,” The Journal of Chemical Physics, vol. 139, no. 9, 2013.
  * (45) R. Potestio, C. Peter, and K. Kremer, “Computer simulations of soft matter : Linking the scales,” Entropy, vol. 16, no. 8, pp. 4199–4245, 2014.
  * (46) S. Kmiecik, D. Gront, M. Kolinski, L. Wieteska, A. E. Dawid, and A. Kolinski, “Coarse-grained protein models and their applications,” Chemical Reviews, vol. 116, no. 14, pp. 7898–7936, 2016.
  * (47) R. J. Cubero, M. Marsili, and Y. Roudi, “Multiscale relevance and informative encoding in neuronal spike trains,” J. Comput. Neurosci., vol. 48, no. 1, pp. 85–102, 2020.
  * (48) M. Giulini, R. Menichetti, and R. Potestio, “A journey through mapping space: characterising the statistical and metric properties of reduced representations of macromolecules,” The European Physical Journal B, vol. 94, no. 10, 2021.
  * (49) S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by simulated annealing,” Science, vol. 220, no. 4598, pp. 671–680, 1983.
  * (50) V. Černý, “Thermodynamical approach to the traveling salesman problem: An efficient simulation algorithm,” J. Optim. Theory Appl., vol. 45, no. 1, p. 41–51, 1985.
  * (51) M. Marsili and Y. Roudi, “Quantifying relevance in learning and inference,” Physics Reports, vol. 963, pp. 1–43, 2022.
  * (52) A. Haimovici and M. Marsili, “Criticality of mostly informative samples: a bayesian model selection approach,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2015, no. 10, 2015.
  * (53) M. Marsili, I. Mastromatteo, and Y. Roudi, “On sampling and modeling complex systems,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2013, no. 09, 2013.
  * (54) M. Mele, R. Covino, and R. Potestio, “Information-theoretical measures identify accurate low-resolution representations of protein configurational space,” Soft Matter, vol. 18, pp. 7064–7074, 2022.
  * (55) W. A. Little, “The existence of persistent states in the brain,” Mathematical biosciences, vol. 19, no. 1-2, pp. 101–120, 1974.
  * (56) W. A. Little and G. L. Shaw, “Analytic study of the memory storage capacity of a neural network,” Mathematical biosciences, vol. 39, no. 3-4, pp. 281–290, 1978.
  * (57) V. Dichio and F. D. V. Fallani, “Statistical models of complex brain networks: a maximum entropy approach,” Reports on Progress in Physics, 2023.
  * (58) J. Feldman and D. Ballard, “Connectionist models and their properties,” Cognitive Science, vol. 6, no. 3, pp. 205–254, 1982.
  * (59) K. Pagiamtzis and A. Sheikholeslami, “Content-addressable memory (cam) circuits and architectures: a tutorial and survey,” IEEE Journal of Solid-State Circuits, vol. 41, no. 3, pp. 712–727, 2006.
  * (60) B. Katz, Nerve, muscle and synapse.  McGraw-Hill, 1966.
  * (61) L. P. Kadanoff, “Notes on migdal’s recursion formulas,” Annals of Physics, vol. 100, no. 1, pp. 359–394, 1976.
  * (62) J. V. José, L. P. Kadanoff, S. Kirkpatrick, and D. R. Nelson, “Renormalization, vortices, and symmetry-breaking perturbations in the two-dimensional planar model,” Phys. Rev. B, vol. 16, pp. 1217–1241, 1977\.
  * (63) M. Giulini, M. Rigoli, G. Mattiotti, R. Menichetti, T. Tarenzi, R. Fiorentini, and R. Potestio, “From system modeling to system analysis: The impact of resolution level and resolution distribution in the computer-aided investigation of biomolecules,” Frontiers in Molecular Biosciences, vol. 8, 2021.
  * (64) J. Jin, A. J. Pak, A. E. P. Durumeric, T. D. Loose, and G. A. Voth, “Bottom-up coarse-graining: Principles and perspectives,” Journal of Chemical Theory and Computation, vol. 18, no. 10, pp. 5759–5791, 2022.
  * (65) M. Giulini, R. Menichetti, M. S. Shell, and R. Potestio, “An information-theory-based approach for optimal model reduction of biomolecules,” Journal of Chemical Theory and Computation, vol. 16, no. 11, pp. 6795–6813, 2020.
  * (66) C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, pp. 379–423, 1948.
  * (67) S.-I. Amari, “Learning patterns and pattern sequences by self-organizing nets of threshold elements,” IEEE Transactions on Computers, vol. C-21, no. 11, pp. 1197–1206, 1972.
  * (68) W. Little, “The existence of persistent states in the brain,” Mathematical Biosciences, vol. 19, no. 1, pp. 101–120, 1974.
  * (69) E. Caianiello, “Outline of a theory of thought-processes and thinking machines,” Journal of Theoretical Biology, vol. 1, no. 2, pp. 204–235, 1961.
  * (70) R. J. Cubero, J. Jo, M. Marsili, Y. Roudi, and J. Song, “Statistical criticality arises in most informative representations,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2019, no. 6, 2019.
  * (71) D. Durstewitz, G. Koppe, and M. I. Thurm, “Reconstructing computational system dynamics from neural data with recurrent neural networks,” Nat Rev Neurosci, vol. 24, pp. 693–710, 2023.
  * (72) H. Sompolinsky, “Statistical mechanics of neural networks,” Physics Today, vol. 41, no. 12, p. 70, 1988.
  * (73) T. Mora and W. Bialek, “Are biological systems poised at criticality?,” Journal of Statistical Physics, vol. 144, no. 2, pp. 268–302, 2011.
  * (74) K. Binder and P. Virnau, “Phase transitions and phase coexistence: equilibrium systems versus externally driven or active systems - some perspectives,” Soft Materials, vol. 19, no. 3, pp. 267–285, 2021.
  * (75) K. Kawasaki and T. Yamada, “Time-Dependent Ising Model with Long Range Interaction,” Progress of Theoretical Physics, vol. 39, no. 1, pp. 1–25, 1968.
  * (76) H. Sompolinsky and A. Zippelius, “Dynamic theory of the spin-glass phase,” Phys. Rev. Lett., vol. 47, pp. 359–362, 1981.
  * (77) S. Kirkpatrick and D. Sherrington, “Infinite-ranged models of spin-glasses,” Phys. Rev. B, vol. 17, pp. 4384–4403, 1978.