# Simplicial Hopfield networks

Thomas F. Burns
Neural Coding and Brain Computing Unit
OIST Graduate University, Okinawa, Japan
<EMAIL_ADDRESS>
&Tomoki Fukai
Neural Coding and Brain Computing Unit
OIST Graduate University, Okinawa, Japan
<EMAIL_ADDRESS>

###### Abstract

Hopfield networks are artificial neural networks which store memory patterns
on the states of their neurons by choosing recurrent connection weights and
update rules such that the energy landscape of the network forms attractors
around the memories. How many stable, sufficiently-attracting memory patterns
can we store in such a network using $N$ neurons? The answer depends on the
choice of weights and update rule. Inspired by setwise connectivity in
biology, we extend Hopfield networks by adding setwise connections and
embedding these connections in a simplicial complex. Simplicial complexes are
higher dimensional analogues of graphs which naturally represent collections
of pairwise and setwise relationships. We show that our simplicial Hopfield
networks increase memory storage capacity. Surprisingly, even when connections
are limited to a small random subset of equivalent size to an all-pairwise
network, our networks still outperform their pairwise counterparts. Such
scenarios include non-trivial simplicial topology. We also test analogous
modern continuous Hopfield networks, offering a potentially promising avenue
for improving the attention mechanism in Transformer models.

##  1 Introduction

Hopfield networks (Hopfield, 1982)111After the proposal of Marr (1971), many
similar models of associative memory were proposed, e.g., those of Nakano
(1972), Amari (1972), Little (1974), and Stanley (1976) – all before Hopfield
(1982). Nevertheless, much of the research literature refers to and seems more
proximally inspired by Hopfield (1982). Many of these models can also be
considered instances of the Lenz-Ising model (Brush, 1967) with infinite-range
interactions. store memory patterns in the weights of connections between
neurons. In the case of pairwise connections, these weights translate to the
synaptic strength between pairs of neurons in biological neural networks. In
such a Hopfield network with $N$ neurons, there will be $N\choose 2$ of these
pairwise connections, forming a complete graph. Each edge is weighted by a
procedure which considers $P$ memory patterns and which, based on these
patterns, seeks to minimise a defined energy function such that the network’s
dynamics are attracted to and ideally exactly settles in the memory pattern
which is nearest to the current states of the neurons. The network therefore
acts as a content addressable memory – given a partial or noise-corrupted
memory, the network can update its states through recurrent dynamics to
retrieve the full memory. Since its introduction, the Hopfield network has
been extended and studied widely by neuroscientists (Griniasty et al., 1993;
Schneidman et al., 2006; Sridhar et al., 2021; Burns et al., 2022), physicists
(Amit et al., 1985; Agliari et al., 2013; Leonetti et al., 2021), and computer
scientists (Widrich et al., 2020; Millidge et al., 2022). Of particular
interest to the machine learning community is the recent development of modern
Hopfield networks (Krotov & Hopfield, 2016) and their close correspondence
(Ramsauer et al., 2021) to the attention mechanism of Transformers (Vaswani et
al., 2017).

An early (Amit et al., 1985; McEliece et al., 1987) and ongoing (Hillar &
Tran, 2018) theme in the study of Hopfield networks has been their memory
storage capacity, i.e., determining the number of memory patterns which can be
reliably stored and later recalled via the dynamics. As discussed in Appendix
A.1, this theoretical and computational exercise serves two purposes: (i)
improving the memory capacity of such models for theoretical purposes and
computational applications; and (ii) gaining an abstract understanding of
neurobiological mechanisms and their implications for biological memory
systems. Traditional Hopfield networks with binary neuron states, in the limit
of $N\rightarrow\infty$ and $P\rightarrow\infty$, maintain associative
memories for up to approximately $0.14N$ patterns (Amit et al., 1985; McEliece
et al., 1987), and fewer if the patterns are statistically or spatially
correlated (Löwe, 1998). However, by a clever reformulation of the update rule
based on the network energy, this capacity can be improved to $N^{d-1}$, where
$d\geq 2$ (Krotov & Hopfield, 2016), and even further to $2^{N/2}$ (Demircigil
et al., 2017). Networks using these types of energy-based update rules are
called modern Hopfield networks. Krotov & Hopfield (2016) (like Hopfield
(1984)) also investigated neurons which took on continuous states. Upon
generalising this model by using the softmax activation function, Ramsauer et
al. (2021) showed a connection to the attention mechanism of Transformers
(Vaswani et al., 2017). However, to the best of our knowledge, these modern
Hopfield networks have not been extended further to include explicit setwise
connections between neurons, as has been studied and shown to improve memory
capacity in traditional Hopfield networks (Peretto & Niez, 1986; Lee et al.,
1986; Baldi & Venkatesh, 1987; Newman, 1988). Indeed, Krotov & Hopfield
(2016), who introduced modern Hopfield networks, make a mathematical analogy
between their energy-based update rule and setwise connections given their
energy-based update rule can be interpreted as allowing individual pairs of
pre- and post-synaptic neurons to make multiple synapses with each other –
making pairwise connections mathematically as strong as equivalently-ordered
setwise connections222Work by Horn, D. & Usher, M. (1988) study almost the
same system but with an slight modification to the traditional update rule,
whereas Krotov & Hopfield (2016) use their modern, energy-based update rule..
Demircigil et al. (2017) later proved this analogy to be accurate in terms of
theoretical memory capacity. By adding explicit setwise connections to modern
Hopfield networks, we essentially allow all connections (pairwise and higher)
to increase their strength – following the same interpretation, this can be
thought of as allowing both pairwise and setwise connections between all
neurons, any of which may be precisely controlled.

Functionally, setwise connections appear in abundance in biological neural
networks. What’s more, these setwise interactions often modulate and interact
with one another in highly complex and nonlinear fashions, adding to their
potential computational expressiveness. We discuss these biological mechanisms
in Appendix A.2. There are many contemporary models in deep learning which
implicitly model particular types of setwise interactions (Jayakumar et al.,
2020). To explicitly model such interactions, we have multiple options. For
reasons we discuss in Appendix A.3, we choose to model our setwise connections
using a simplicial complex.

We therefore develop and study Simplicial Hopfield Networks. We weight the
simplices of the simplicial complex to store memory patterns and generalise
the energy functions and update rules of traditional and modern Hopfield
networks. Our main contributions are:

  * •

We introduce extensions of various Hopfield networks with setwise connections.
In addition to generalising Hopfield networks to include explicit,
controllable setwise connections based on an underlying simplicial structure,
we also study whether the topological features of the underlying structure
influences performance.

  * •

We prove and discuss higher memory capacity in the general case of simplicial
Hopfield networks. For the fully-connected simplicial Hopfield network, we
prove a larger memory capacity than previously shown by Newman (1988);
Demircigil et al. (2017) for higher-degree Hopfield networks.

  * •

We empirically show improved performance under parameter constraints. By
restricting the total number of connections to that of pairwise Hopfield
networks with a mixture of pairwise and setwise connections, we show
simplicial Hopfield networks retain a surprising amount of improved
performance over pairwise networks but with fewer parameters, and are robust
to topological variability.

##  2 Simplicial Hopfield networks

###  2.1 Simplicial complexes

Simplicial complexes are mathematical objects which naturally represent
collections of setwise relationships. Here we use the combinatorial form,
called an abstract simplicial complex. Although, to build intuition and
visualise the simplicial complex, we also refer to their geometric
realisations.

######  Definition 2.1.

Let $K$ be a subset of $2^{[N]}.$ The subset $K$ is an abstract simplicial
complex if for any $\sigma\in K,$ the condition $\rho\subseteq\sigma$ gives
$\rho\in K,$ for any $\rho\subseteq\sigma.$

In other words, an abstract simplicial complex $K$ is a collection of finite
sets closed under taking subsets. A member of $K$ is called a simplex
$\sigma$. A $k$–dimensional simplex (or $k$–simplex) has cardinality $k+1$ and
$k+1$ faces which are $(k-1)$–simplices (obtained by omitting one element from
$\sigma$). If a simplex $\sigma$ is a face of another simplex $\tau$, we say
that $\tau$ is a coface of $\sigma$. We denote the set of all $k$–simplices in
$K$ as $K_{k}$.

Geometrically, for $k=0,1,2,$ and $3$, a $k$–simplex is, respectively, a
point, line segment, triangle, and tetrahedron. Therefore, one may visualise a
simplicial complex as being constructed by gluing together simplices such that
every finite set of vertices in $K$ form the vertices of at most one simplex.
This structure makes it possible to associate every setwise relationship
uniquely with a $k$–simplex identified by its elements, which in our case are
neurons (see Figure 1A). Simplices in $K$ which are contained in no higher
dimensional simplices, i.e., they have no cofaces, are called the facets of K.
The dimension of $K$, $\text{dim}(K)$, is the dimension of its largest facet.
We call a simplicial complex $K$ a $k$–skeleton when all possible faces of
dimension $k$ exist and $\text{dim}(K)=k$.

###  2.2 Model

Figure 1: A. Comparative illustrations of connections in a pairwise Hopfield
network (left) and a simplicial Hopfield network (right) with $N=4$. In a
simplicial Hopfield network, $\sigma={\\{i,j\\}}$ is an edge ($1$–simplex),
$\sigma={\\{i,j,k\\}}$ is a triangle ($2$–simplex), $\sigma={\\{i,j,k,l\\}}$
is a tetrahedron ($3$–simplex), and so on. B. Connection weight histograms of
$1$–, $2$–, and $3$–simplices in a simplicial Hopfield network. In the binary
case, the x-axis range is $[-P/N,+P/N]$. Here, $N=100$ and $P=10$, thus the
range is $[-0.1,+0.1]$. Note that each dimension shows a similar, Gaussian
distribution of weights (although there are different absolute numbers of
these weights; see ‘Mixed diluted networks’ in Section 2.2). C. Illustration
of the hierarchical relationship between elements in the complex, up to
$3$–simplices, with arrows indicating potential sources of weight modulation
or interaction, e.g., between (co)faces or using Hodge Laplacians within the
same dimension. Such modulations and interactions (including their biological
interpretations) are discussed in Appendices A.2 and A.3.

A network of $N$ neurons is modelled by $N$ spins. Let $K$ be a simplicial
complex on $N$ vertices. In the binary neuron case, $S_{j}^{(t)}=\pm 1$ at
time-step $t$. Given a set of neurons $\sigma$ (which contains the neuron $i$
and is a unique $(|\sigma|-1)$–simplex in $K$), $w(\sigma)$ is the associated
simplicial weight and $S_{\sigma}^{(t)}$ the product of their spins. Spin
configurations correspond to patterns of neural firing, with dynamics governed
by a defined energy. The traditional model is defined by energy and weight
functions

$\displaystyle E=-{\sum_{\sigma\in K}w(\sigma)S_{\sigma}^{(t)}}\hskip
59.62032ptw(\sigma)=\frac{1}{N}\sum_{\mu=1}^{P}\xi_{\sigma}^{\mu},$ (1)

with $\xi_{i}^{\mu}$ ($=\pm 1$) static variables being the $P$ binary memory
patterns stored in the simplicial weights. Similar for spins,
$\xi_{\sigma}^{\mu}$ is the product of the static pattern variables for the
set of neurons $\sigma$ in the pattern $\mu$. Figure 1B shows examples of the
resulting Gaussian distributions of weights at each dimension of the
simplicial complex. We use these weights to update the state of a neuron $i$
by applying the traditional Hopfield update rule

$S_{i}^{(t)}=\Theta\left(\sum_{\sigma\in K}w(\sigma)S_{\sigma\setminus
i}^{(t-1)}\right)\hskip 59.62032pt\Theta(x)=\begin{cases}1&\text{if $x\geq
0$}\\\ -1&\text{if $x<0$}\end{cases}.$ (2)

When $K$ is a $1$-skeleton, this becomes the traditional pairwise Hopfield
network (Hopfield, 1982). In the modern Hopfield case, the energy function and
update rule are

$\displaystyle E$ $\displaystyle=-\sum_{\mu=1}^{P}{\sum_{\sigma\in
K}F(\xi_{\sigma}^{\mu}S_{\sigma}^{(t)})}$ (3) $\displaystyle S_{i}^{(t)}$
$\displaystyle=\textrm{sgn}\left[\sum_{\mu=1}^{P}\left(F(1\cdot\xi_{i}^{\mu}+\sum_{\sigma\in
K}\xi_{\sigma\setminus i}^{\mu}S_{\sigma\setminus
i}^{(t-1)})-F(-1\cdot\xi_{i}^{\mu}+\sum_{\sigma\in K}\xi_{\sigma\setminus
i}^{\mu}S_{\sigma\setminus i}^{(t-1)})\right)\right],$ (4)

where the function $F$ can be chosen, for example, to be of a polynomial
$F(x)=x^{n}$ or exponential $F(x)=e^{x}$ form. When $K$ is a $1$-skeleton,
this becomes the modern pairwise Hopfield network (Krotov & Hopfield, 2016).

In the continuous modern Hopfield case, spins and patterns take real values
$S_{j},\xi_{j}^{\mu}\in\mathbb{R}$. Patterns are arranged in a matrix
$\textbf{$\Xi$}=(\xi^{1},...,\xi^{P})$ and we define the log-sum-exp function
(lse) for $T^{-1}>0$ as

$\text{lse}(T^{-1},\textbf{$\Xi$}^{T}S^{(t)},K)=T\text{
log}\left(\sum_{\mu=1}^{P}\sum_{\sigma\in
K}\text{exp}(T^{-1}\Xi_{\sigma}^{\mu}S_{\sigma}^{(t)})\right).$ (5)

The energy function is

$E=-\text{lse}(T^{-1},\textbf{$\Xi$}^{T}S^{(t)},K)+\dfrac{1}{2}S^{(t)T}S^{(t)}.$
(6)

For each simplex $\sigma\in K$, we denote the submatrix of the patterns stored
on that simplex as $\textbf{$\Xi$}_{\sigma}$ (which has dimensions
$P\times\sigma$). Using the dot product to measure the similarity between the
patterns and spins, the update rule is

$S^{(t)}=\text{softmax}\left(T\sum_{\sigma\in
K}\left(\textbf{$\Xi$}_{\sigma}^{T}\overrightarrow{S_{\sigma}^{(t-1)}}\right)\right)\textbf{$\Xi$}.$
(7)

In practice, however, the dot product has been found to under-perform in
modern continuous Hopfield networks compared to Euclidean or Manhattan
distances (Millidge et al., 2022). Transformer models in natural language
tasks have also seen performance improvements by replacing the dot product
with cosine similarity (Henry et al., 2020), again a measure with a more
geometric flavour. However, these similarity measures generalise distances
between pairs of elements rather than sets of elements.

We therefore use higher-dimensional geometric similarity measures, cumulative
Euclidean distance (ced) and Cayley–Menger distance (cmd). Let $d_{\rho}$ be
the (Euclidean or Manhattan) distance between pattern $\xi_{\rho}^{\mu}$ and
spins $S_{\rho}^{(t)}$ for pattern $\mu$ and spins $\rho\subset\sigma$. Let
$K^{\sigma}_{1}$ be the subset of $K$ such that all elements in
$K^{\sigma}_{1}$ are $1$–simplex faces of $\sigma$. We define the cumulative
Euclidean distance as

$\text{ced}(\xi_{\sigma}^{\mu},S_{\sigma}^{(t)})=\sqrt{\sum_{\rho\in
K^{\sigma}_{1}}(d_{\rho})^{2}}.$ (8)

We define cmd$(\xi_{\sigma}^{\mu},S_{\sigma}^{(t)})$ as the Cayley–Menger
determinant of all $\rho\in K^{\sigma}_{1}$, with distances set as $d_{\rho}$.

##### Mixed diluted networks.

A computational concern in the above models is that the number of unique
possible $k$–simplices is $N\choose k+1$, e.g., with $N=100$ there are
approximately $9.89\times 10^{28}$ possible $50$–simplices, compared to just
$4,950$ edges ($1$–simplices) found in a pairwise Hopfield network. If we
allow all possible simplices for a simplicial Hopfield network with $N$
neurons, the total number of simplices (excluding $0$–simplices, i.e.,
autapses) will be $\sum_{d=2}^{N}{N\choose d}$. Simultaneously, there is also
an open question as to how many setwise connections is biologically-realistic
to model. We also note that setwise connections can be functionally built from
combinations of pairwise connections by introducing additional hidden neurons,
as shown by Krotov & Hopfield (2021). Therefore, we might in fact be under-
estimating the total number of ‘functional’ setwise connections, which may
appear via common network motifs or ‘synapsembles’ (Buzsáki, 2010).

Conservatively, we evaluate classes of simplicial Hopfield networks which are
low-dimensional, i.e., $\text{dim}(K)$ is small, and where the total number of
weighted simplices is not greater than those normally found in a pairwise
Hopfield network, i.e., the number of non-zero weights is $N\choose 2$. We
randomly choose weights to be non-zero, with each weight of a dimension having
an equal probability and according to Table 1. (See Appendix A.4 for a small
worked example.) Such random networks have previously been studied in the
traditional pairwise case as ‘diluted networks’ (Treves & Amit, 1988; Bovier &
Gayrard, 1993a; b; Löwe & Vermet, 2011). Here we study mixed diluted networks,
since we use a mixture of connections of different degrees. We believe we are
also the first to study such networks beyond pairwise connections, as well as
in modern and continuous cases.

##### Topology.

Different collections of simplices in a simplicial complex can result in
different Euler characteristics (a homotopy invariant property). Table 1 shows
this from a parameter perspective via counting only the simplices with non-
zero weights. However, even when using the same proportion of $1$– and
$2$–simplices, the choices of which vertices those simplices contain can be
different due to randomness. Therefore, the topologies of each network may
vary (and so too may their subsequent dynamics and performance). One well-
studied and often important topological property in the context of simplicial
complexes, homology, counts the number of holes in each dimension. In the
$0$th dimension, this is the number of connected components; in the $1$st
dimension, this is the number of triangles formed by edges which don’t also
have a $2$–simplex ‘filling in’ the interior surface of that triangle; in the
$2$nd dimension, this is the number of tetrahedra formed by triangles which
don’t also have a $3$–simplex ‘filling in’ the interior volume of that
tetrahedron; and so on. The exact number of these holes in dimension $k$ can
be calculated by the $k$–th Betti number, $\beta_{k}$ (see Appendix A.5). We
calculate these for our networks to observe the relationship between homology
and memory capacity.

###  2.3 Theoretical memory capacity

##### Mixed networks.

Much is already known about the theoretical memory capacity of various
Hopfield networks, including those with explicit (Newman, 1988) or implicit
(Demircigil et al., 2017) setwise connectivity. However, we wish to point out
a somewhat underappreciated relationship between memory capacity and the
explicit or implicit number of network connections – which, in the fully-
connected network, is determined by the degree of the connections (see
Appendix A.6 for proof).

######  Corollary 2.2 (Memory capacity is proportional to the number of
network connections).

If the connection weights in a Hopfield network are symmetric, then the order
of the network’s memory capacity is proportional to the number of its
connections.

What happens when there are connections between the same neurons at multiple
degrees, i.e., what we call a mixed network? To the best of our knowledge, the
theoretical memory capacity of such networks has not been well-studied.
However, we found one classical study by Dreyfus et al. (1987) which showed,
numerically, adding triplet connections to a pairwise model improved
attractivity and memory capacity. Most prior formal studies have only
considered connections at single higher degrees (Newman, 1988; Bengtsson,
1990). Although, higher order neural networks have historically considered
such mixtures of interactions on different degrees simultaneously (Zhang,
2012), but as regular neural networks (e.g., feed-forward networks), not
Hopfield networks. Higher order Boltzmann machines (HOBMs) (Sejnowski, 1986)
have also been studied with mixed connections (Amari et al., 1992; Leisink &
Kappen, 1999)333HOBMs also suffer the same problem as we face here, one of
having many high-order parameters between the neurons to keep a track of.
Possibly a factoring trick like in Memisevic & Hinton (2010) for HOBMs could
be helpful in simplicial Hopfield networks.. However, HOBMs are unlike
Hopfield networks in that they typically have hidden units, are trained
differently, and have stochastic neural activations 444Despite this, there are
equivalences (Leonelli et al., 2021; Marullo & Agliari, 2021; Smart & Zilman,
2021).. Modern Hopfield networks also include an implicit mixture of
connections of different degrees555Recall that
$\left(\sum_{i}a_{i}\right)^{b}=\sum_{i}a_{i}^{b+1}.$ (but – and see Theorem 1
of Demircigil et al. (2017), which remains unproven – the mixture is
unbalanced and not particularly natural, especially for $F(x)=x^{n}$ when $n$
is odd). Therefore, we include the following result demonstrating fixed
points, large basins of attraction (i.e., convergence) to those fixed points
in mixed networks, and memory capacity which is linear in the number of fully-
connected degrees of connections (a proof is provided in Appendix A.6).

######  Lemma 2.3 (Fully-connected mixed Hopfield networks).

A fully-connected mixed Hopfield network based on a $D$–skeleton with $N$
neurons and $P$ patterns has, when $N\rightarrow\infty$ and $P$ is finite: (i)
fixed point attractors at the memory patterns; and (ii) dynamic convergence
towards the fixed point attractors within a finite Hamming distance $\delta$.
When $P\rightarrow\infty$ with $N\rightarrow\infty$, the network has capacity
to store up to $(\sum_{d=1}^{D}N^{d})/(2\text{ ln }N)$ memory patterns (with
small retrieval errors) or $(\sum_{d=1}^{D}N^{d})/(4\text{ ln }N)$ (without
retrieval errors).

This naturally comports with Theorem 2 from Demircigil et al. (2017), except
here we show an increased capacity in the mixed network, courtesy of Corollary
2.2.

##### Mixed diluted networks.

As mentioned earlier, full setwise connectivity is not necessarily tractable
nor realistic. Löwe & Vermet (2011) show for pairwise diluted networks
constructed as Erdös-Renyi graphs (constructed by including each possible edge
on the vertex set with probability $p$) that the memory capacity is
proportional to $pN$. Crucial for this result is that the random graph must be
asymptotically connected. This makes sense, given that if any vertex was
disconnected its dynamics could never be influenced. Empirically, it does seem
that a certain threshold of mean connectivity in pairwise random networks is
crucial for attractor dynamics (Treves & Amit, 1988).

######  Remark 2.4.

By a straightforward generalisation of Löwe & Vermet (2011)’s result, diluted
networks constructed as pure Erdös-Renyi hypergraphs may store on the order of
$pN^{d-1}$ memory patterns, where $d$ is the degree of the connections.

In the case of an unbounded number of allowable connections, Remark 2.4 would
suggest picking as many higher-degree connections as possible when choosing
between connections of lower or higher degrees in our mixed diluted networks.
However, in the bounded case (our case), we are non-trivially changing the
asymptotic behaviour in terms of connectivity and dynamics when we use a
mixture of connection degrees. We also need to beware of asymmetries which may
arise (Kanter, 1988). This makes the analysis of mixed diluted networks not
particularly straightforward (also see Section 4).

###  2.4 Numerical simulations and performance metrics

Given the large space of possible network settings, in the main text we focus
primarily on conditions listed in Table 1. Additional experiments are also
shown in Appendix A.8.

Table 1: List of network condition keys (top row), their number of non-zero weights for 1– and 2–simplices (second and third rows), and their ‘functional’ Euler characteristic ($\chi$, bottom row). $N$ is the number of neurons. $C=(N-1)N$. For simulation, the number of simplices at each dimension are rounded to the nearest integer. | K1 | R$\overline{1}2$ | R$\overline{12}$ | R1$\overline{2}$ | R2
---|---|---|---|---|---
1–simplices | $N\choose 2$ |  $0.75$$N\choose 2$ |  $0.50$$N\choose 2$ |  $0.25$$N\choose 2$ | 0
2–simplices | 0 |  $0.25$$N\choose 2$ |  $0.50$$N\choose 2$ |  $0.75$$N\choose 2$ | $N\choose 2$
$\chi$ | $N-(1/2)C$ | $N-0.25C$ | $N$ | $N+0.25C$ | $N+(1/2)C$

In our numerical simulations, we perform updates synchronously until $E$ is
non-decreasing or until a maximum number of steps is reached, whichever comes
first. When a simulation concludes we compute the overlap (for binary
patterns) or mean squared error (MSE) (for continuous patterns) of the final
spin configuration with respect to all stored patterns using

$m^{\mu}=\left|\dfrac{1}{N}\sum_{i=1}^{N}{S_{i}^{(t)}\xi_{i}^{\mu}}\right|\hskip
59.62032pt\text{MSE}^{\mu}=\dfrac{1}{N}\sum_{i=1}^{N}{(S_{i}^{(t)}-\xi_{i}^{\mu})^{2}}.$
(9)

We say the network recalls (or attempts to recall) whichever pattern has the
largest overlap (where $m^{\mu}=1$ indicates perfect recall) or smallest MSE
(where $\text{MSE}^{\mu}=0$ indicates perfect recall).

##  3 Numerical simulations

###  3.1 Binary memory patterns

After embedding random binary patterns, we started the network in random
initial states and recorded the final overlap of the closest pattern. Table 2
shows the final overlaps for traditional simplicial Hopfield networks
($N=100$). Our simplicial Hopfield networks significantly outperform the
pairwise Hopfield networks (K1). In fact, the R1$\overline{2}$ model performs
as well at $0.3N$ patterns as the the pairwise network performs on $0.05N$
patterns, a six-fold increase in the number of patterns and more than double
the theoretical capacity of the pairwise network, $\sim 0.14N$ (Amit et al.,
1985). Surprisingly, Table 3 shows homology accounts for very little of the
variance in network performance.

Table 2: Mean $\pm$ standard deviation of overlap distributions ($n=100$) from traditional simplicial Hopfield networks with varying numbers (top row) of random binary patterns. K1 is the traditional pairwise Hopfield network. R1$\overline{2}$ significantly outperforms K1 at all tested levels (one-way t-tests $p<10^{-11}$, $F>50.13$). At all pattern loadings, a one-way ANOVA showed significant variance between the networks ($p<10^{-20}$, $F>26.35$). Box and whisker plots shown in Figure 6. No. patterns | $0.05N$ | $0.1N$ | $0.15N$ | $0.2N$ | $0.3N$
---|---|---|---|---|---
K1 | $0.87\pm 0.18$ | $0.81\pm 0.16$ | $0.66\pm 0.10$ | $0.65\pm 0.10$ | $0.59\pm 0.08$
R$\overline{1}2$ | $0.96\pm 0.10$ | $0.94\pm 0.14$ | $0.82\pm 0.20$ | $0.71\pm 0.17$ | $0.64\pm 0.13$
R$\overline{12}$ | $0.98\pm 0.10$ | $0.99\pm 0.03$ | $0.97\pm 0.10$ | $0.91\pm 0.15$ | $0.76\pm 0.16$
R1$\overline{\textbf{2}}$ | $1\pm 0$ | $0.99\pm 0.04$ | $0.99\pm 0.05$ | $0.98\pm 0.08$ | $0.87\pm 0.16$
R2 | $1\pm 0$ | $0.99\pm 0.18$ | $0.94\pm 0.18$ | $0.74\pm 0.29$ | $0.53\pm 0.23$

###  3.2 Continuous memory patterns

##### Energy landscape.

Using Equation 3 and given a set of patterns, a simplicial complex $K$, and an
inverse temperature $T^{-1}$, we may calculate the energy of network states.
To inspect changes in the energy landscapes of different network conditions,
we set $N=10$ and $P=10$ random patterns. We performed principle component
analysis (PCA) to create a low dimensional projection of the patterns. Then,
we generated network states evenly spaced in a $10\times 10$ grid which
spanned the projected memory patterns in the first two dimensions of PCA
space. We calculated each state’s energy by transforming these points from PCA
space back into the $N$–dimensional space, across the network conditions at
$T^{-1}=1,2,10$ (Figure 7). At $T^{-1}=1$, differences between the network
conditions’ energy landscapes are very subtle. However, at $T^{-1}=2$ and
$T^{-1}=10$, we see a clear change: those with more $2$–simplices possess more
sophisticated, pattern-separating landscapes.

##### Recall as a function of memory loading.

We tested the performance of our simplicial Hopfield networks by embedding
data from the MNIST (LeCun et al., 2010), CIFAR-10 (Krizhevsky & Hinton,
2009), and Tiny ImageNet (Le & Yang, 2015) datasets as memories. We followed
the protocol of Millidge et al. (2022) to test recall under increasing memory
load as an indication of the networks’ memory capacities. To embed the
memories, we normalise the pixel values between $0$ and $1$, and treat them as
continuous-valued neurons, e.g., for MNIST we have $N=28\times 28=784$
neurons. We initialise $S$ as one of the memory patterns corrupted by Gaussian
noise with variance $0.5$. After allowing the network to settle in an energy
minima, we measured the performance as the fraction of correctly recalled
memories (over all tested memories) of the uncorrupted patterns, where
‘correct recall’ was defined as a sum of the squared difference being $<50$.
In all tests, we used $T^{-1}=100$. Also see Appendix A.7 for further
simulation details.

Figure 2 compares a pairwise architecture, K1, with a higher-order
architecture, R1$\overline{2}$. The performance of the K1 networks is
comparable to that shown in Millidge et al. (2022), however, R1$\overline{2}$
significantly outperforms K1 across all datasets. Since the MNIST dataset is
relatively simple and K1 already performs well, the performance improvement is
small, albeit significant. In the CIFAR-10 and Tiny ImageNet datasets, the
performance improvements are more noticeable, with most distance functions
seeing improvements of $\geq 10\%$ in the fraction of correctly retrieved
memories.

Figure 2: Recall (mean $\pm$ S.D. over $10$ trials) as a function of memory
loading using the MNIST, CIFAR-10, and Tiny ImageNet datasets, using different
distance functions (see legend). Here we compare the performance of modern
continuous pairwise networks (top row) and modern continuous simplicial
networks (bottom row). The simplicial networks are R1$\overline{2}$ networks
(see Table 1 for information). R1$\overline{2}$ significantly outperforms the
pairwise network (K1) at all tested levels where there was not already perfect
recall (one-way t-tests $p<10^{-9}$, $F>16.01$). At all memory loadings, a
one-way ANOVA showed significant variance between the networks ($p<10^{-5}$,
$F>11.95$). Tabulated results are shown in Tables 6, 7, and 8.

Also noticeable in the results for CIFAR-10 and Tiny ImageNet (see Figure 2)
is the relatively high performance of the ced and cmd distance measures.
Indeed, cmd performs as well or better than the Manhattan distance in our
experiments. And both ced and cmd (along with the Euclidean and Manhattan
distances) outperform the dot product in CIFAR-10 and Tiny ImageNet at high
memory loadings. This further supports the intuition and results of Millidge
et al. (2022), that more ‘geometric’ distances perform better as similarity
measures in modern Hopfield networks.

##  4 Discussion

We have introduced a new class of Hopfield networks which generalises and
extends traditional, modern, and continuous Hopfield networks. Our main
finding is that mixed diluted networks can improve performance in terms of
memory recall, even when there is no increase in the number of parameters.
This improvement therefore comes from the topology rather than additional
information in the form of parameters. We also show how distance measures of a
more ‘geometric flavour’ can further improve performance in these networks.
This simplicial framework (in diluted or undiluted forms) now opens up new
avenues for researchers in neuroscience and machine learning. In neuroscience,
we can now model how setwise connections, such as those provided by astrocytes
and dendrites, may improve memory function and may interact to form important
topological structures to guide memory dynamics. In machine learning, such
topological structures may now be utilised in improved attention mechanisms or
Transformer models, such as in Ramsauer et al. (2021). At the intersection of
these fields, we may now further study how the topology of networks in
neuroscience and machine learning systems may correspond to one another and
share functional characteristics, such as how the activity of ‘pairwise’
Transformer models have shown similarities to activities in auditory cortex
(Millet et al., 2022). Could ‘setwise’ Transformer models correspond more
closely? Or to a more diverse range of cell types? These and related questions
are now open for exploration, and may lead to improved performance in
applications (Clift et al., 2020).

##### Convolution operations and higher-order neural networks.

From the perspective of modern deep learning, considering higher order
correlations between downstream inputs to a neuron is quite classical. For
example, convolutional neural networks have incorporated specialised setwise
operations since their inception (Fukushima, 1980; Lecun et al., 1998), and
more general setwise relationships have also been introduced in higher-order
neural networks (Pineda, 1987; Reid et al., 1989; Ghosh & Shun, 1992; Zhang,
2012). Although our setwise connections are not explicitly convolutional, they
are in one notable sense conceptually similar: they collect information from a
particular subset of neurons and only become active when those particular
neurons are active in the right way. One of the main differences, however, is
that – unlike typical convolution operations – we don’t restrict the
connection locations to some particular locations or arrangements within the
input space. Our results therefore suggest that, in some cases, replacing
regular feedforward connections with random convolutions may offer improved
performance in some circumstances.

##### Improvements and extensions.

Our study focusses on random choices of weighted simplices. What if we choose
more carefully? Indeed, it seems quite likely biological setwise connections
are not random, and are almost certainly not randomly chosen to replace random
pairwise connections.

It now seems natural to study how online weight modulations (e.g., based on
spectral theories) could generate new connections between Hopfield networks
and, e.g., geometric deep learning. Such modulations may have novel biological
interpretations, e.g., spatial and anti-Hebbian memory structures may be
modelled by strategically inserting inhibitory interactions (Haga & Fukai,
2019; Burns et al., 2022) between higher simplices (and may also model
disinhibition).

##### Further analytic studies.

Our numerical results suggest diluted mixed networks have larger memory
capacities than pairwise networks. In a fairly intuitive sense, this is not
particularly surprising – we are adding degrees of freedom to the energy
landscape, within which we may build more stable and nicely-behaved
attractors. However, we have not yet proven this increased capacity
analytically for the diluted case, only given some theoretical indications as
to why this occurs and proven the undiluted case. We hypothesise it is
possible to do so using generalised methods from replica-symmetric theory
(Amit et al., 1985) or self-consistent signal-to-noise analysis (Shiino &
Fukai, 1993), in combination with methods from structural Ramsey theory. The
capacity for modern simplicial networks may be on the order of a double-
exponential in the number of neurons (since, in the limit of
$N\rightarrow\infty$, there is an exponential relationship in the number of
multispin interactions on top of an exponential relationship in the number of
intra-multispin interactions, i.e., both pair-spins and multi-spins can have
higher degrees of attraction). This capacity, however, will likely scale
nonlinearly with the choice of (random) dilution, e.g., there may be a steep
drop in performance around a critical dilution range, likely where some
important dynamical guarantees are lost due to an intolerably small number of
connections of a particular order.

Even higher orders and diluted mixtures of setwise connections may also be
studied. Such networks, per Lemma 2.3, will likely improve their performance
as higher-degree connections are added (as shown in Appendix A.8). However,
and as implied in Section 2.3, the number and distribution of these
connections may need to be careful chosen in highly diluted settings.

#### Reproducibility Statement

To reproduce our results in the main text and appendices, we provide our
Python code as supplementary material at
https://github.com/tfburns/simplicial-hopfield-networks. We have also provided
a small worked example in Appendix A.4 to help clarify computational steps in
the model construction. Assumptions made in our theoretical results are stated
in Section 2.3 and Appendix A.6.

#### Acknowledgements

The first author thanks Milena Menezes Carvalho for graphic design assistance
with Figure 1, as well as Robert Tang, Tom George, and members of the Neural
Coding and Brain Computing Unit at OIST for helpful discussions. We thank
anonymous reviewers for their feedback and suggestions. The second author
acknowledges support from KAKENHI grants JP19H04994 and JP18H05213.

## References

  * Agliari et al. (2013) Elena Agliari, Adriano Barra, Andrea De Antoni, and Andrea Galluzzi.  Parallel retrieval of correlated patterns: From Hopfield networks to Boltzmann machines.  _Neural Networks_ , 38:52–63, 2013.  ISSN 0893-6080.  doi: https://doi.org/10.1016/j.neunet.2012.11.010.  URL https://www.sciencedirect.com/science/article/pii/S0893608012003073.
  * Amari et al. (1992) S Amari, K Kurata, and H Nagaoka.  Information geometry of Boltzmann machines.  _IEEE Trans Neural Netw_ , 3(2):260–271, 1992\.
  * Amari (1972) S.-I. Amari.  Learning patterns and pattern sequences by self-organizing nets of threshold elements.  _IEEE Transactions on Computers_ , C-21(11):1197–1206, 1972.  doi: 10.1109/T-C.1972.223477.
  * Amit et al. (1985) Daniel J. Amit, Hanoch Gutfreund, and H. Sompolinsky.  Storing infinite numbers of patterns in a spin-glass model of neural networks.  _Phys. Rev. Lett._ , 55:1530–1533, Sep 1985.  doi: 10.1103/PhysRevLett.55.1530.  URL https://link.aps.org/doi/10.1103/PhysRevLett.55.1530.
  * Araque et al. (1999) Alfonso Araque, Vladimir Parpura, Rita P Sanzgiri, and Philip G Haydon.  Tripartite synapses: glia, the unacknowledged partner.  _Trends in Neurosciences_ , 22(5):208–215, 1999\.
  * Arbuthnott & Wickens (2007) Gordon W. Arbuthnott and Jeff Wickens.  Space, time and dopamine.  _Trends in Neurosciences_ , 30(2):62–69, 2007\.  ISSN 0166-2236.  doi: https://doi.org/10.1016/j.tins.2006.12.003.  URL https://www.sciencedirect.com/science/article/pii/S0166223606002748.
  * Aronov et al. (2017) Dmitriy Aronov, Rhino Nevers, and David W. Tank.  Mapping of a non-spatial dimension by the hippocampal–entorhinal circuit.  _Nature_ , 543(7647):719–722, Mar 2017.  ISSN 1476-4687.  doi: 10.1038/nature21692.
  * Baldi & Venkatesh (1987) Pierre Baldi and Santosh S. Venkatesh.  Number of stable points for spin-glasses and neural networks of higher orders.  _Phys. Rev. Lett._ , 58:913–916, Mar 1987.  doi: 10.1103/PhysRevLett.58.913.  URL https://link.aps.org/doi/10.1103/PhysRevLett.58.913.
  * Bao et al. (2019) Xiaojun Bao, Eva Gjorgieva, Laura K. Shanahan, James D. Howard, Thorsten Kahnt, and Jay A. Gottfried.  Grid-like neural representations support olfactory navigation of a two-dimensional odor space.  _Neuron_ , 102(5):1066–1075.e5, Jun 2019.  ISSN 0896-6273.  doi: 10.1016/j.neuron.2019.03.034.  URL https://doi.org/10.1016/j.neuron.2019.03.034.
  * Barth & Poulet (2012) Alison L. Barth and James F A Poulet.  Experimental evidence for sparse firing in the neocortex.  _Trends in Neurosciences_ , 35(6):345–355, 2012\.  ISSN 01662236.  doi: 10.1016/j.tins.2012.03.008.
  * Bellmund et al. (2018) Jacob L. S. Bellmund, Peter Gärdenfors, Edvard I. Moser, and Christian F. Doeller.  Navigating cognition: Spatial codes for human thinking.  _Science_ , 362(6415):eaat6766, 2018.  doi: 10.1126/science.aat6766.  URL https://www.science.org/doi/abs/10.1126/science.aat6766.
  * Bellos (2015) Alex Bellos.  He ate all the pi : Japanese man memorises $\pi$ to 111,700 digits.  _The Guardian_ , 2015.  URL https://www.theguardian.com/science/alexs-adventures-in-numberland/2015/mar/13/pi-day-2015-memory-memorisation-world-record-japanese-akira-haraguchi.
  * Bengtsson (1990) Mats Bengtsson.  _Higher Order Artificial Neural Networks_.  Diane Publishing Company, 1990.  ISBN 9780941375924.  URL https://books.google.co.jp/books?id=FTlFeFg8HH4C.
  * Bliss & Gardner-Medwin (1973) T. V P Bliss and A. R. Gardner-Medwin.  Long-lasting potentiation of synaptic transmission in the dentate area of the unanaesthetized rabbit following stimulation of the perforant path.  _The Journal of Physiology_ , 232(2):357–374, 1973.  ISSN 14697793.  doi: 10.1113/jphysiol.1973.sp010274.
  * Bloss et al. (2018) Erik B Bloss, Mark S Cembrowski, Bill Karsh, Jennifer Colonell, Richard D Fetter, and Nelson Spruston.  Single excitatory axons form clustered synapses onto CA1 pyramidal cell dendrites.  _Nature Neuroscience_ , 21(3):353–363, March 2018\.
  * Bodnar et al. (2021) Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Liò, Guido F Montufar, and Michael Bronstein.  Weisfeiler and Lehman Go Cellular: CW Networks.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , volume 34, pp. 2625–2640. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper/2021/file/157792e4abb490f99dbd738483e0d2d4-Paper.pdf.
  * Bordone et al. (2019) Melina Paula Bordone, Mootaz M. Salman, Haley E. Titus, Elham Amini, Jens V. Andersen, Barnali Chakraborti, Artem V. Diuba, Tatsiana G. Dubouskaya, Eric Ehrke, Andiara Espindola de Freitas, Guilherme Braga de Freitas, Rafaella A. Gonçalves, Deepali Gupta, Richa Gupta, Sharon R. Ha, Isabel A. Hemming, Minal Jaggar, Emil Jakobsen, Punita Kumari, Navya Lakkappa, Ashley P. L. Marsh, Jessica Mitlöhner, Yuki Ogawa, Ramesh Kumar Paidi, Felipe C. Ribeiro, Ahmad Salamian, Suraiya Saleem, Sorabh Sharma, Joana M. Silva, Shripriya Singh, Kunjbihari Sulakhiya, Tesfaye Wolde Tefera, Behnam Vafadari, Anuradha Yadav, Reiji Yamazaki, and Constanze I. Seidenbecher.  The energetic brain – a review from students to students.  _Journal of Neurochemistry_ , 151(2):139–165, 2019.  doi: https://doi.org/10.1111/jnc.14829.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/jnc.14829.
  * Bovier & Gayrard (1993a) Anton Bovier and Véronique Gayrard.  Lower bounds on the memory capacity of the dilute Hopfield model.  In Nino Boccara, Eric Goles, Servet Martinez, and Pierre Picco (eds.), _Cellular Automata and Cooperative Systems_ , pp. 55–66. Springer Netherlands, Dordrecht, 1993a.  ISBN 978-94-011-1691-6.  doi: 10.1007/978-94-011-1691-6_6.  URL https://doi.org/10.1007/978-94-011-1691-6_6.
  * Bovier & Gayrard (1993b) Anton Bovier and Véronique Gayrard.  Rigorous results on the thermodynamics of the dilute Hopfield model.  _Journal of Statistical Physics_ , 72(1):79–112, Jul 1993b.  ISSN 1572-9613.  doi: 10.1007/BF01048041.  URL https://doi.org/10.1007/BF01048041.
  * Brady et al. (2008) Timothy F Brady, Talia Konkle, George A Alvarez, and Aude Oliva.  Visual long-term memory has a massive storage capacity for object details.  _Proceedings of the National Academy of Sciences_ , 105(38):14325–14329, 2008.
  * Brombas et al. (2017) A. Brombas, S. Kalita-de Croft, E. J. Cooper-Williams, and S. R. Williams.  Dendro-dendritic cholinergic excitation controls dendritic spike initiation in retinal ganglion cells.  _Nature Communications_ , 8(1):15683, Jun 2017\.  ISSN 2041-1723.  doi: 10.1038/ncomms15683.  URL https://doi.org/10.1038/ncomms15683.
  * Bruck & Roychowdhury (1990) Jehoshua Bruck and Vwani P. Roychowdhury.  On the Number of Spurious Memories in the Hopfield Model.  _IEEE Transactions on Information Theory_ , 36(2):393–397, 1990.  ISSN 15579654.  doi: 10.1109/18.52486.
  * Brush (1967) Stephen G. Brush.  History of the lenz-ising model.  _Rev. Mod. Phys._ , 39:883–893, Oct 1967.  doi: 10.1103/RevModPhys.39.883.  URL https://link.aps.org/doi/10.1103/RevModPhys.39.883.
  * Brysbaert et al. (2016) Marc Brysbaert, Michaël Stevens, Paweł Mandera, and Emmanuel Keuleers.  How many words do we know? Practical estimates of vocabulary size dependent on word definition, the degree of language input and the participant’s age.  _Frontiers in Psychology_ , 7, 2016.  ISSN 1664-1078.  doi: 10.3389/fpsyg.2016.01116.  URL https://www.frontiersin.org/articles/10.3389/fpsyg.2016.01116.
  * Burns & Tang (2022) Thomas F Burns and Robert Tang.  Detecting danger in gridworlds using Gromov’s link condition.  _arXiv_ , 2022.  doi: 10.48550/ARXIV.2201.06274.  URL https://arxiv.org/abs/2201.06274.
  * Burns et al. (2022) Thomas F Burns, Tatsuya Haga, and Tomoki Fukai.  Multiscale and extended retrieval of associative memory structures in a cortical model of local-global inhibition balance.  _eNeuro_ , 9(3), 2022.  doi: 10.1523/ENEURO.0023-22.2022.  URL https://www.eneuro.org/content/9/3/ENEURO.0023-22.2022.
  * Buzsáki (2010) György Buzsáki.  Neural syntax: Cell assemblies, synapsembles, and readers.  _Neuron_ , 68(3):362–385, 2010.  ISSN 0896-6273.  doi: https://doi.org/10.1016/j.neuron.2010.09.023.  URL https://www.sciencedirect.com/science/article/pii/S0896627310007658.
  * Chaudhuri & Fiete (2016) Rishidev Chaudhuri and Ila Fiete.  Computational principles of memory.  _Nature Neuroscience_ , 19(3):394–403, March 2016\.
  * Chaudhuri & Fiete (2019) Rishidev Chaudhuri and Ila Fiete.  Bipartite expander Hopfield networks as self-decoding high-capacity error correcting codes.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/97008ea27052082be055447be9e85612-Paper.pdf.
  * Chavlis & Poirazi (2021) Spyridon Chavlis and Panayiota Poirazi.  Drawing inspiration from biological dendrites to empower artificial neural networks.  _Current Opinion in Neurobiology_ , 70:1–10, 2021.  ISSN 0959-4388.  doi: https://doi.org/10.1016/j.conb.2021.04.007.  URL https://www.sciencedirect.com/science/article/pii/S0959438821000544.  Computational Neuroscience.
  * Chen et al. (2013) Yuhan Chen, Shengjun Wang, Claus C Hilgetag, and Changsong Zhou.  Trade-off between multiple constraints enables simultaneous formation of modules and hubs in neural systems.  _PLoS Comput. Biol._ , 9(3):e1002937, March 2013\.
  * Chipman et al. (2021) Peter H Chipman, Chi Chung Alan Fung, Alejandra Pazo Fernandez, Abhilash Sawant, Angelo Tedoldi, Atsushi Kawai, Sunita Ghimire Gautam, Mizuki Kurosawa, Manabu Abe, Kenji Sakimura, Tomoki Fukai, and Yukiko Goda.  Astrocyte GluN2C NMDA receptors control basal synaptic strengths of hippocampal CA1 pyramidal neurons in the stratum radiatum.  _Elife_ , 10, October 2021.
  * Christian & Thompson (2003) Kimberly M. Christian and Richard F. Thompson.  Neural Substrates of Eyeblink Conditioning: Acquisition and Retention.  _Learning and Memory_ , 10(6):427–455, 2003.  ISSN 10720502.  doi: 10.1101/lm.59603.
  * Clift et al. (2020) James Clift, Dmitry Doryn, Daniel Murfet, and James Wallbridge.  Logic and the 2-simplicial transformer.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=rkecJ6VFvr.
  * Clopath et al. (2017) Claudia Clopath, Tobias Bonhoeffer, Mark Hübener, and Tobias Rose.  Variance and invariance of neuronal long-term representations.  _Philos Trans R Soc Lond B Biol Sci_ , 372(1715), March 2017\.
  * Constantinescu et al. (2016) Alexandra O. Constantinescu, Jill X. O’Reilly, and Timothy E. J. Behrens.  Organizing conceptual knowledge in humans with a gridlike code.  _Science_ , 352(6292):1464–1468, 2016.  doi: 10.1126/science.aaf0941.  URL https://www.science.org/doi/abs/10.1126/science.aaf0941.
  * Covelo & Araque (2018) Ana Covelo and Alfonso Araque.  Neuronal activity determines distinct gliotransmitter release from a single astrocyte.  _eLife_ , 7:e32237, Jan 2018.  ISSN 2050-084X.  doi: 10.7554/eLife.32237.  URL https://doi.org/10.7554/eLife.32237.
  * Darwin et al. (1972) Christopher J. Darwin, Michael T. Turvey, and Robert G. Crowder.  An auditory analogue of the Sperling partial report procedure: Evidence for brief auditory storage.  _Cognitive Psychology_ , 3(2):255–267, 1972.  ISSN 00100285.  doi: 10.1016/0010-0285(72)90007-2.
  * Daw & Doya (2006) Nathaniel D. Daw and Kenji Doya.  The computational neurobiology of learning and reward.  _Current Opinion in Neurobiology_ , 16(2):199–204, 2006.  ISSN 09594388.  doi: 10.1016/j.conb.2006.03.006.
  * Demircigil et al. (2017) Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet.  On a model of associative memory with huge storage capacity.  _Journal of Statistical Physics_ , 168(2):288–299, Jul 2017.  ISSN 1572-9613.  doi: 10.1007/s10955-017-1806-y.  URL https://doi.org/10.1007/s10955-017-1806-y.
  * Didier et al. (2001) Anne Didier, Alan Carleton, Jan G. Bjaalie, Jean-Didier Vincent, Ole Petter Ottersen, Jon Storm-Mathisen, and Pierre-Marie Lledo.  A dendrodendritic reciprocal synapse provides a recurrent excitatory connection in the olfactory bulb.  _Proceedings of the National Academy of Sciences_ , 98(11):6441–6446, 2001.  doi: 10.1073/pnas.101126398.  URL https://www.pnas.org/doi/abs/10.1073/pnas.101126398.
  * Doron et al. (2022) Adi Doron, Alon Rubin, Aviya Benmelech-Chovav, Netai Benaim, Tom Carmi, Ron Refaeli, Nechama Novick, Tirzah Kreisel, Yaniv Ziv, and Inbal Goshen.  Hippocampal astrocytes encode reward location.  _Nature_ , Aug 2022.  ISSN 1476-4687.  doi: 10.1038/s41586-022-05146-6.  URL https://doi.org/10.1038/s41586-022-05146-6.
  * Dreyfus et al. (1987) Gérard Dreyfus, Isabelle Guyon, Jean-Pierre Nadal, and Léon Personnaz.  High order neural networks for efficient associative memory design.  In D. Anderson (ed.), _Neural Information Processing Systems_. American Institute of Physics, 1987.  URL https://proceedings.neurips.cc/paper/1987/file/c7e1249ffc03eb9ded908c236bd1996d-Paper.pdf.
  * Ebli et al. (2020) Stefania Ebli, Michaël Defferrard, and Gard Spreemann.  Simplicial neural networks.  In _NeurIPS 2020 Workshop on Topological Data Analysis and Beyond_ , 2020.  URL https://openreview.net/forum?id=nPCt39DVIfk.
  * Ercsey-Ravasz et al. (2013) Mária Ercsey-Ravasz, Nikola T. Markov, Camille Lamy, David C. Van Essen, Kenneth Knoblauch, Zoltán Toroczkai, and Henry Kennedy.  A predictive network model of cerebral cortical connectivity based on a distance rule.  _Neuron_ , 80(1):184–197, 2013.  ISSN 0896-6273.  doi: https://doi.org/10.1016/j.neuron.2013.07.036.  URL https://www.sciencedirect.com/science/article/pii/S0896627313006600.
  * Etherington et al. (2010) Sarah J Etherington, Susan E Atkinson, Greg J Stuart, and Stephen R Williams.  _Synaptic Integration_ , chapter 1, pp. 1–15.  Wiley, 2010.  ISBN 9780470015902.  doi: https://doi.org/10.1002/9780470015902.a0000208.pub2.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/9780470015902.a0000208.pub2.
  * Feng et al. (2019) Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao.  Hypergraph neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 3558–3565, 2019.
  * Folli et al. (2017) Viola Folli, Marco Leonetti, and Giancarlo Ruocco.  On the maximum storage capacity of the Hopfield model.  _Frontiers in Computational Neuroscience_ , 10, 2017.  ISSN 1662-5188.  doi: 10.3389/fncom.2016.00144.  URL https://www.frontiersin.org/articles/10.3389/fncom.2016.00144.
  * Fukushima (1980) Kunihiko Fukushima.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position.  _Biological Cybernetics_ , 36(4):193–202, Apr 1980.  ISSN 1432-0770.  doi: 10.1007/BF00344251.  URL https://doi.org/10.1007/BF00344251.
  * Geinisman et al. (2001) Yuri Geinisman, Robert W. Berry, John F. Disterhoft, John M. Power, and Eddy A. Van der Zee.  Associative learning elicits the formation of multiple-synapse boutons.  _Journal of Neuroscience_ , 21(15):5568–5573, 2001.  ISSN 0270-6474.  doi: 10.1523/JNEUROSCI.21-15-05568.2001.  URL https://www.jneurosci.org/content/21/15/5568.
  * Ghosh & Shun (1992) Joydeep Ghosh and Yoan Shun.  Efficient higher-order neural networks for classification and function approximation.  _International Journal of Neural Systems_ , 3(4):323–350, 1992.  doi: 10.1142/S0129065792000255.  URL https://doi.org/10.1142/S0129065792000255.
  * Gilbert & Kesner (2004) Paul E. Gilbert and Raymond P. Kesner.  Memory for objects and their locations: The role of the hippocampus in retention of object-place associations.  _Neurobiology of Learning and Memory_ , 81(1):39–45, 2004.  ISSN 10747427.  doi: 10.1016/S1074-7427(03)00069-8.
  * Golding et al. (2002) Nace L. Golding, Nathan P. Staff, and Nelson Spruston.  Dendritic spikes as a mechanism for cooperative long-term potentiation.  _Nature_ , 418(6895):326–331, Jul 2002.  ISSN 1476-4687.  doi: 10.1038/nature00854.  URL https://doi.org/10.1038/nature00854.
  * Gordon et al. (1993) A. M. Gordon, G. Westling, K. J. Cole, and R. S. Johansson.  Memory representations underlying motor commands used during manipulation of common and novel objects.  _Journal of Neurophysiology_ , 69(6):1789–1797, 1993.  ISSN 00223077.  doi: 10.1152/jn.1993.69.6.1789.
  * Gosti et al. (2019) Giorgio Gosti, Viola Folli, Marco Leonetti, and Giancarlo Ruocco.  Beyond the maximum storage capacity limit in Hopfield recurrent neural networks.  _Entropy_ , 21(8), 2019.  ISSN 1099-4300.  doi: 10.3390/e21080726.  URL https://www.mdpi.com/1099-4300/21/8/726.
  * Gradshteyn & Ryzhik (2007) I. S. Gradshteyn and I. M. Ryzhik.  _Table of integrals, series, and products_.  Elsevier/Academic Press, Amsterdam, seventh edition, 2007.  ISBN 978-0-12-373637-6; 0-12-373637-4.  Translated from the Russian. Translation edited and with a preface by Alan Jeffrey and Daniel Zwillinger.
  * Griesbauer et al. (2022) Eva-Maria Griesbauer, Ed Manley, Jan M. Wiener, and Hugo J. Spiers.  London taxi drivers: A review of neurocognitive studies and an exploration of how they build their cognitive map of London.  _Hippocampus_ , 32(1):3–20, 2022.  doi: https://doi.org/10.1002/hipo.23395.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/hipo.23395.
  * Griniasty et al. (1993) M. Griniasty, M. V. Tsodyks, and Daniel J. Amit.  Conversion of Temporal Correlations Between Stimuli to Spatial Correlations Between Attractors.  _Neural Computation_ , 5(1):1–17, 01 1993.  ISSN 0899-7667.  doi: 10.1162/neco.1993.5.1.1.  URL https://doi.org/10.1162/neco.1993.5.1.1.
  * Gripon & Berrou (2011) Vincent Gripon and Claude Berrou.  Sparse neural networks with large learning diversity.  _IEEE Transactions on Neural Networks_ , 22(7):1087–1096, 2011.  ISSN 10459227.  doi: 10.1109/TNN.2011.2146789.
  * Groschner et al. (2022) Lukas N. Groschner, Jonatan G. Malis, Birte Zuidinga, and Alexander Borst.  A biophysical account of multiplication by a single neuron.  _Nature_ , 603(7899):119–123, Mar 2022.  ISSN 1476-4687.  doi: 10.1038/s41586-022-04428-3.  URL https://doi.org/10.1038/s41586-022-04428-3.
  * Guo et al. (2017) Zengcai V. Guo, Hidehiko K. Inagaki, Kayvon Daie, Shaul Druckmann, Charles R. Gerfen, and Karel Svoboda.  Maintenance of persistent activity in a frontal-thalamocortical loop.  _Nature_ , 545(7653):181–186, May 2017.  ISSN 1476-4687.  doi: 10.1038/nature22324.  URL https://doi.org/10.1038/nature22324.
  * Gustafsson & Wigström (1988) Bengt Gustafsson and Holger Wigström.  Physiological mechanisms underlying long-term potentiation.  _Trends in Neurosciences_ , 11(4):156–162, 1988\.  ISSN 01662236.  doi: 10.1016/0166-2236(88)90142-7.
  * Haga & Fukai (2019) Tatsuya Haga and Tomoki Fukai.  Extended temporal association memory by modulations of inhibitory circuits.  _Phys. Rev. Lett._ , 123:078101, Aug 2019.  doi: 10.1103/PhysRevLett.123.078101.  URL https://link.aps.org/doi/10.1103/PhysRevLett.123.078101.
  * Hajij et al. (2020) Mustafa Hajij, Kyle Istvan, and Ghada Zamzmi.  Cell complex neural networks.  In _NeurIPS 2020 Workshop on Topological Data Analysis and Beyond_ , 2020.  URL https://openreview.net/forum?id=6Tq18ySFpGU.
  * Hart & Huk (2020) Eric Hart and Alexander C. Huk.  Recurrent circuit dynamics underlie persistent activity in the macaque frontoparietal network.  _eLife_ , 9:e52460, May 2020.  ISSN 2050-084X.  doi: 10.7554/eLife.52460.  URL https://doi.org/10.7554/eLife.52460.  32379044[pmid].
  * Hedrick et al. (2022) Nathan G. Hedrick, Zhongmin Lu, Eric Bushong, Surbhi Singhi, Peter Nguyen, Yessenia Magaña, Sayyed Jilani, Byung Kook Lim, Mark Ellisman, and Takaki Komiyama.  Learning binds new inputs into functional synaptic clusters via spinogenesis.  _Nature Neuroscience_ , 25(6):726–737, Jun 2022\.  ISSN 1546-1726.  doi: 10.1038/s41593-022-01086-6.  URL https://doi.org/10.1038/s41593-022-01086-6.
  * Henry et al. (2020) Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen.  Query-Key normalization for Transformers.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 4246–4253, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.379.  URL https://aclanthology.org/2020.findings-emnlp.379.
  * Herculano-Houzel (2009) Suzana Herculano-Houzel.  The human brain in numbers: a linearly scaled-up primate brain.  _Frontiers in Human Neuroscience_ , 3, 2009.  ISSN 1662-5161.  doi: 10.3389/neuro.09.031.2009.  URL https://www.frontiersin.org/articles/10.3389/neuro.09.031.2009.
  * Hillar & Tran (2018) Christopher J. Hillar and Ngoc M. Tran.  Robust exponential memory in Hopfield networks.  _The Journal of Mathematical Neuroscience_ , 8(1):1, Jan 2018.  ISSN 2190-8567.  doi: 10.1186/s13408-017-0056-2.  URL https://doi.org/10.1186/s13408-017-0056-2.
  * Hoffmann (2019) Heiko Hoffmann.  Sparse associative memory.  _Neural Computation_ , 31(5):998–1014, 2019.  ISSN 1530888X.  doi: 10.1162/neco_a_01181.
  * Hoover et al. (2022) Benjamin Hoover, Duen Horng Chau, Hendrik Strobelt, and Dmitry Krotov.  A universal abstraction for hierarchical Hopfield networks.  In _The Symbiosis of Deep Learning and Differential Equations II_ , 2022.  URL https://openreview.net/forum?id=SAv3nhzNWhw.
  * Hopfieid (2008) J J Hopfieid.  Searching for memories, Sudoku, implicit check bits, and the iterative use of not-always-correct rapid neural computation.  _Neural Computation_ , 20(5):1119–1164, 2008\.  ISSN 08997667.  doi: 10.1162/neco.2008.09-06-345.
  * Hopfield (1982) J J Hopfield.  Neural networks and physical systems with emergent collective computational abilities.  _Proceedings of the National Academy of Sciences_ , 79(8):2554–2558, 1982.  doi: 10.1073/pnas.79.8.2554.  URL https://www.pnas.org/doi/abs/10.1073/pnas.79.8.2554.
  * Hopfield (1984) J J Hopfield.  Neurons with graded response have collective computational properties like those of two-state neurons.  _Proceedings of the National Academy of Sciences_ , 81(10):3088–3092, 1984.  doi: 10.1073/pnas.81.10.3088.  URL https://www.pnas.org/doi/abs/10.1073/pnas.81.10.3088.
  * Horn, D. & Usher, M. (1988) Horn, D. and Usher, M.  Capacities of multiconnected memory models.  _J. Phys. France_ , 49(3):389–395, 1988.  doi: 10.1051/jphys:01988004903038900.  URL https://doi.org/10.1051/jphys:01988004903038900.
  * Jayakumar et al. (2020) Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu.  Multiplicative interactions and where to find them.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=rylnK6VtDH.
  * Jenkins et al. (2018) R Jenkins, AJ Dowsett, and AM Burton.  How many faces do people know?  _Proceedings of the Royal Society B_ , 285(1888):20181319, 2018.
  * Jones & Powell (1969) EG Jones and TPS Powell.  Morphological variations in the dendritic spines of the neocortex.  _Journal of cell science_ , 5(2):509–529, 1969\.
  * Kaiser & Hilgetag (2006) Marcus Kaiser and Claus C Hilgetag.  Nonoptimal component placement, but short processing paths, due to long-distance projections in neural systems.  _PLoS Comput. Biol._ , 2(7):e95, July 2006.
  * Kanter (1988) I. Kanter.  Asymmetric neural networks with multispin interactions.  _Phys. Rev. A_ , 38:5972–5975, Dec 1988.  doi: 10.1103/PhysRevA.38.5972.  URL https://link.aps.org/doi/10.1103/PhysRevA.38.5972.
  * Kato et al. (2022) Ayaka Kato, Kazumi Ohta, Kazuo Okanoya, and Hokto Kazama.  Dopaminergic neurons dynamically update sensory values during navigation.  _bioRxiv_ , 2022.  doi: 10.1101/2022.08.17.504092.  URL https://www.biorxiv.org/content/early/2022/08/17/2022.08.17.504092.
  * Kavalali et al. (1999) Ege T. Kavalali, Jürgen Klingauf, and Richard W. Tsien.  Activity-dependent regulation of synaptic clustering in a hippocampal culture system.  _Proceedings of the National Academy of Sciences_ , 96(22):12893–12900, 1999.  doi: 10.1073/pnas.96.22.12893.  URL https://www.pnas.org/doi/abs/10.1073/pnas.96.22.12893.
  * Kim et al. (2017) Do Hyun Kim, Jinha Park, and Byungnam Kahng.  Enhanced storage capacity with errors in scale-free Hopfield neural networks: An analytical study.  _PLoS ONE_ , 12(10), 2017.  ISSN 19326203.  doi: 10.1371/journal.pone.0184683.
  * Kozachkov et al. (2022) Leo Kozachkov, Ksenia V. Kastanenka, and Dmitry Krotov.  Building Transformers from neurons and astrocytes.  _bioRxiv_ , 2022.  doi: 10.1101/2022.10.12.511910.  URL https://www.biorxiv.org/content/early/2022/10/15/2022.10.12.511910.
  * Krizhevsky & Hinton (2009) Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, Toronto, Ontario, 2009.
  * Krotov & Hopfield (2016) Dmitry Krotov and John J. Hopfield.  Dense associative memory for pattern recognition.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , NIPS’16, pp. 1180–1188, Red Hook, NY, USA, 2016. Curran Associates Inc.  ISBN 9781510838819.
  * Krotov & Hopfield (2021) Dmitry Krotov and John J. Hopfield.  Large associative memory problem in neurobiology and machine learning.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=X4y_10OX-hX.
  * Lansner (2009) Anders Lansner.  Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations.  _Trends in Neurosciences_ , 32(3):178–186, 2009\.  ISSN 01662236.  doi: 10.1016/j.tins.2008.12.002.
  * Le & Yang (2015) Ya Le and Xuan Yang.  Tiny ImageNet visual recognition challenge.  _CS 231N_ , 7(7):3, 2015.
  * Lecun et al. (1998) Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.  doi: 10.1109/5.726791.
  * LeCun et al. (2010) Yann LeCun, Corinna Cortes, and Christopher Burges.  MNIST handwritten digit database, 2010.  URL http://yann.lecun.com/exdb/mnist/.
  * Lederman & Klatzky (2009) S. J. Lederman and R. L. Klatzky.  Haptic perception: A tutorial.  _Attention, Perception, and Psychophysics_ , 71(7):1439–1459, 2009.  ISSN 19433921.  doi: 10.3758/APP.71.7.1439.
  * Lee et al. (2013) Kea Joo Lee, In Sung Park, Hyun Kim, William T. Greenough, Daniel T. S. Pak, and Im Joo Rhyu.  Motor skill training induces coordinated strengthening and weakening between neighboring synapses.  _Journal of Neuroscience_ , 33(23):9794–9799, 2013.  ISSN 0270-6474.  doi: 10.1523/JNEUROSCI.0848-12.2013.  URL https://www.jneurosci.org/content/33/23/9794.
  * Lee et al. (1986) Y. C. Lee, Gary D. Doolen, H. H. Chen, G. Z. Sun, T. Maxwell, H. Y. Lee, and C. Lee Giles.  Machine learning using higher order correlation networks.  In _Proceedings of the “Evolution, Games, and Learning” Conference_ , 1986.
  * Leisink & Kappen (1999) Martijn A. R. Leisink and Hilbert J. Kappen.  Learning in higher order Boltzmann machines using linear response.  _Neural Networks_ , 13:2000, 1999.
  * Leonelli et al. (2021) Francesca Elisa Leonelli, Elena Agliari, Linda Albanese, and Adriano Barra.  On the effective initialisation for restricted Boltzmann machines via duality with Hopfield model.  _Neural Networks_ , 143:314–326, 2021.  ISSN 0893-6080.  doi: https://doi.org/10.1016/j.neunet.2021.06.017.  URL https://www.sciencedirect.com/science/article/pii/S0893608021002495.
  * Leonetti et al. (2021) M. Leonetti, E. Hörmann, L. Leuzzi, G. Parisi, and G. Ruocco.  Optical computation of a spin glass dynamics with tunable complexity.  _Proceedings of the National Academy of Sciences_ , 118(21):e2015207118, 2021.  doi: 10.1073/pnas.2015207118.  URL https://www.pnas.org/doi/abs/10.1073/pnas.2015207118.
  * Lim (2020) Lek-Heng Lim.  Hodge Laplacians on graphs.  _SIAM Review_ , 62(3):685–715, 2020.  doi: 10.1137/18M1223101.  URL https://doi.org/10.1137/18M1223101.
  * Little (1974) W.A. Little.  The existence of persistent states in the brain.  _Mathematical Biosciences_ , 19(1):101–120, 1974\.  ISSN 0025-5564.  doi: https://doi.org/10.1016/0025-5564(74)90031-5.  URL https://www.sciencedirect.com/science/article/pii/0025556474900315.
  * Löwe & Vermet (2011) Matthias Löwe and Franck Vermet.  The Hopfield model on a sparse Erdös-Renyi graph.  _Journal of Statistical Physics_ , 143(1):205–214, Apr 2011.  ISSN 1572-9613.  doi: 10.1007/s10955-011-0167-1.  URL https://doi.org/10.1007/s10955-011-0167-1.
  * Löwe (1998) Matthias Löwe.  On the storage capacity of Hopfield models with correlated patterns.  _The Annals of Applied Probability_ , 8(4):1216 – 1250, 1998.  doi: 10.1214/aoap/1028903378.  URL https://doi.org/10.1214/aoap/1028903378.
  * Mackintosh (1983) N. J. Mackintosh.  _Conditioning and associative learning_.  Oxford University Press, Oxford, 1983.
  * Maren (2001) Stephen Maren.  Neurobiology of Pavlovian Fear Conditioning.  _Annual Review of Neuroscience_ , 24(1):897–931, 2001.  ISSN 0147-006X.  doi: 10.1146/annurev.neuro.24.1.897.
  * Marr (1971) D Marr.  Simple memory: a theory for archicortex.  _Philos Trans R Soc Lond B Biol Sci_ , 262(841):23–81, July 1971.
  * Marullo & Agliari (2021) Chiara Marullo and Elena Agliari.  Boltzmann machines as generalized Hopfield networks: A review of recent results and outlooks.  _Entropy_ , 23(1), 2021.  ISSN 1099-4300.  doi: 10.3390/e23010034.  URL https://www.mdpi.com/1099-4300/23/1/34.
  * Mau et al. (2020) William Mau, Michael E Hasselmo, and Denise J Cai.  The brain in motion: How ensemble fluidity drives memory-updating and flexibility.  _eLife_ , 9:e63550, Dec 2020.  ISSN 2050-084X.  doi: 10.7554/eLife.63550.  URL https://doi.org/10.7554/eLife.63550.
  * McEliece et al. (1987) R. McEliece, E. Posner, E. Rodemich, and S. Venkatesh.  The capacity of the Hopfield associative memory.  _IEEE Transactions on Information Theory_ , 33(4):461–482, 1987.  doi: 10.1109/TIT.1987.1057328.
  * McSweeney & Murphy (2014) Frances K. McSweeney and Eric S. Murphy (eds.).  _The Wiley Blackwell Handbook of Operant and Classical Conditioning_.  John Wiley & Sons, Ltd, Oxford, UK, may 2014.  ISBN 9781118468135.  doi: 10.1002/9781118468135.  URL http://doi.wiley.com/10.1002/9781118468135.
  * Memisevic & Hinton (2010) Roland Memisevic and Geoffrey E. Hinton.  Learning to Represent Spatial Transformations with Factored Higher-Order Boltzmann Machines.  _Neural Computation_ , 22(6):1473–1492, 06 2010\.  ISSN 0899-7667.  doi: 10.1162/neco.2010.01-09-953.  URL https://doi.org/10.1162/neco.2010.01-09-953.
  * Millet et al. (2022) Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, and Jean-Remi King.  Toward a realistic model of speech processing in the brain with self-supervised learning, 2022.  URL https://arxiv.org/abs/2206.01685.
  * Millidge et al. (2022) Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz.  Universal Hopfield networks: A general framework for single-shot associative memory models, 2022.  URL https://arxiv.org/abs/2202.04557.
  * Milner (1966) B. Milner.  Amnesia following operation on the temporal lobes.  In C. Whitty and O. Zangwill (eds.), _Amnesia_ , pp. 109–133. Butterworth, London, 1966.
  * Milner (1955) G.A. Milner.  The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information.  _Psychological Review_ , 101(2):343–352, 1955\.  URL http://spider.apa.org/ftdocs/rev/1994/april/rev1012343.html.
  * Min et al. (2012) Rogier Min, Mirko Santello, and Thomas Nevian.  The computational power of astrocyte mediated synaptic plasticity.  _Frontiers in Computational Neuroscience_ , 6, 2012.  ISSN 1662-5188.  doi: 10.3389/fncom.2012.00093.  URL https://www.frontiersin.org/articles/10.3389/fncom.2012.00093.
  * Minai & Levy (1993) Ali A. Minai and William B. Levy.  The dynamics of sparse random networks.  _Biological Cybernetics_ , 70(2):177–187, 1993\.  ISSN 03401200.  doi: 10.1007/BF00200831.
  * Mofrad & Parker (2017) A. A. Mofrad and M.G Parker.  Nested-clique network model of neural associative memory.  _Neural Computation_ , 29:1681–1695, 2017.
  * Mongillo et al. (2008) Gianluigi Mongillo, Omri Barak, and Misha Tsodyks.  Synaptic Theory of Working Memory.  _Science_ , 319(5869):1543–1546, 2008.  ISSN 10959203.  doi: 10.1126/science.1150769.
  * Müller et al. (2020) Michael G Müller, Christos H Papadimitriou, Wolfgang Maass, and Robert Legenstein.  A model for structured information representation in neural networks of the brain.  _eNeuro_ , 7(3):ENEURO.0533–19.2020, May 2020\.
  * Nakano (1972) Kaoru Nakano.  Associatron-a model of associative memory.  _IEEE Transactions on Systems, Man, and Cybernetics_ , SMC-2(3):380–388, 1972.  doi: 10.1109/TSMC.1972.4309133.
  * Newman (1988) Charles M Newman.  Memory capacity in neural network models: Rigorous lower bounds.  _Neural Networks_ , 1(3):223–238, 1988.  ISSN 0893-6080.  doi: https://doi.org/10.1016/0893-6080(88)90028-7.  URL https://www.sciencedirect.com/science/article/pii/0893608088900287.
  * Papadimitriou et al. (2020) Christos H. Papadimitriou, Santosh S. Vempala, Daniel Mitropolsky, Michael Collins, and Wolfgang Maass.  Brain computation by assemblies of neurons.  _Proceedings of the National Academy of Sciences_ , 117(25):14464–14472, 2020.  doi: 10.1073/pnas.2001893117.  URL https://www.pnas.org/doi/abs/10.1073/pnas.2001893117.
  * Park et al. (2021) Seongmin A. Park, Douglas S. Miller, and Erie D. Boorman.  Inferences on a multidimensional social hierarchy use a grid-like code.  _Nature Neuroscience_ , 24(9):1292–1301, Sep 2021\.  ISSN 1546-1726.  doi: 10.1038/s41593-021-00916-3.  URL https://doi.org/10.1038/s41593-021-00916-3.
  * Perea et al. (2009) Gertrudis Perea, Marta Navarrete, and Alfonso Araque.  Tripartite synapses: astrocytes process and control synaptic information.  _Trends in Neurosciences_ , 32(8):421–431, 2009\.
  * Peretto & Niez (1986) P Peretto and J J Niez.  Long term memory storage capacity of multiconnected neural networks.  _Biol. Cybern._ , 54(1):53–64, may 1986.  ISSN 0340-1200.  doi: 10.1007/BF00337115.  URL https://doi.org/10.1007/BF00337115.
  * Petri & Barrat (2018) Giovanni Petri and Alain Barrat.  Simplicial activity driven model.  _Phys. Rev. Lett._ , 121:228301, Nov 2018.  doi: 10.1103/PhysRevLett.121.228301.  URL https://link.aps.org/doi/10.1103/PhysRevLett.121.228301.
  * Pfeiffer & Foster (2015) Brad E. Pfeiffer and David J. Foster.  Autoassociative dynamics in the generation of sequences of hippocampal place cells.  _Science_ , 349(6244):180–183, 2015.  ISSN 10959203.  doi: 10.1126/science.aaa9633.
  * Pinault et al. (1997) Didier Pinault, Yoland Smith, and Martin Deschênes.  Dendrodendritic and axoaxonic synapses in the thalamic reticular nucleus of the adult rat.  _Journal of Neuroscience_ , 17(9):3215–3233, 1997\.  ISSN 0270-6474.  doi: 10.1523/JNEUROSCI.17-09-03215.1997.  URL https://www.jneurosci.org/content/17/9/3215.
  * Pineda (1987) Fernando Pineda.  Generalization of back propagation to recurrent and higher order neural networks.  In D. Anderson (ed.), _Neural Information Processing Systems_. American Institute of Physics, 1987.  URL https://proceedings.neurips.cc/paper/1987/file/735b90b4568125ed6c3f678819b6e058-Paper.pdf.
  * Poirazi & Papoutsi (2020) Panayiota Poirazi and Athanasia Papoutsi.  Illuminating dendritic function with computational models.  _Nature Reviews Neuroscience_ , 21(6):303–321, Jun 2020.  ISSN 1471-0048.  doi: 10.1038/s41583-020-0301-7.  URL https://doi.org/10.1038/s41583-020-0301-7.
  * Poirazi et al. (2003) Panayiota Poirazi, Terrence Brannon, and Bartlett W. Mel.  Pyramidal neuron as two-layer neural network.  _Neuron_ , 37(6):989–999, 2003.  ISSN 0896-6273.  doi: https://doi.org/10.1016/S0896-6273(03)00149-1.  URL https://www.sciencedirect.com/science/article/pii/S0896627303001491.
  * Poleg-Polsky & Diamond (2016) Alon Poleg-Polsky and Jeffrey S Diamond.  NMDA receptors multiplicatively scale visual signals and enhance directional motion discrimination in retinal ganglion cells.  _Neuron_ , 89(6):1277–1290, 2016.
  * Preziosi & Coane (2017) Melissa A. Preziosi and Jennifer H. Coane.  Remembering that big things sound big: Sound symbolism and associative memory.  _Cognitive Research: Principles and Implications_ , 2(1), 2017.  ISSN 2365-7464.  doi: 10.1186/s41235-016-0047-y.
  * Pulikkottil et al. (2021) Vinu Varghese Pulikkottil, Bhanu Priya Somashekar, and Upinder S. Bhalla.  Computation, wiring, and plasticity in synaptic clusters.  _Current Opinion in Neurobiology_ , 70:101–112, 2021.  ISSN 0959-4388.  doi: https://doi.org/10.1016/j.conb.2021.08.001.  URL https://www.sciencedirect.com/science/article/pii/S0959438821000921.  Computational Neuroscience.
  * Ramsauer et al. (2021) Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter.  Hopfield networks is all you need.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=tL89RnzIiCd.
  * Rebola et al. (2017) Nelson Rebola, Mario Carta, and Christophe Mulle.  Operation and plasticity of hippocampal CA3 circuits: Implications for memory encoding.  _Nature Reviews Neuroscience_ , 18(4):209–221, 2017.  ISSN 14710048.  doi: 10.1038/nrn.2017.10.
  * Reid et al. (1989) Max B. Reid, Lilly Spirkovska, and Ellen Ochoa.  Rapid training of higher-order neural networks for invariant pattern recognition.  In _In Proc. International Joint Conference on Neural Nets_ , pp. 689–692, 1989.
  * Rescorla & Wagner (1972) R.A. Rescorla and A.R. Wagner.  A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement.  In A.H. Black and W.F. Prokasy (eds.), _Classical Conditioning II: Current Research and Theory_ , pp. 64–99. Appleton Century Crofts, New York, 1972.
  * Reuveni et al. (2017) Iris Reuveni, Sourav Ghosh, and Edi Barkai.  Real time multiplicative memory amplification mediated by whole-cell scaling of synaptic response in key neurons.  _PLOS Computational Biology_ , 13(1):1–31, 01 2017.  doi: 10.1371/journal.pcbi.1005306.  URL https://doi.org/10.1371/journal.pcbi.1005306.
  * Rigby et al. (2022) Mark Rigby, Federico Grillo, Benjamin Compans, Guilherme Neves, Julia Gallinaro, Sophie Nashashibi, Gema Vizcay-Barrena, Florian Levet, Jean-Baptiste Sibarita, Angus Kirkland, Roland A. Fleck, Claudia Clopath, and Juan Burrone.  Multi-synaptic boutons are a feature of ca1 hippocampal connections that may underlie network synchrony.  _bioRxiv_ , 2022.  doi: 10.1101/2022.05.30.493836.  URL https://www.biorxiv.org/content/early/2022/05/30/2022.05.30.493836.
  * Rizzuto & Kahana (2001) Daniel S. Rizzuto and Michael J. Kahana.  An autoassociative neural network model of paired-associate learning.  _Neural Computation_ , 13(9):2075–2092, 2001\.  ISSN 08997667.  doi: 10.1162/089976601750399317.
  * Rocchi et al. (2017) Jacopo Rocchi, David Saad, and Daniele Tantari.  High storage capacity in the Hopfield model with auto-interactions—stability analysis.  _Journal of Physics A: Mathematical and Theoretical_ , 50(46):465001, 2017.
  * Roddenberry et al. (2021) T. Mitchell Roddenberry, Nicholas Glaze, and Santiago Segarra.  Principled simplicial neural networks for trajectory prediction.  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 9020–9029. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/roddenberry21a.html.
  * Rosas et al. (2022) Fernando E. Rosas, Pedro A. M. Mediano, Andrea I. Luppi, Thomas F. Varley, Joseph T. Lizier, Sebastiano Stramaglia, Henrik J. Jensen, and Daniele Marinazzo.  Disentangling high-order mechanisms and high-order behaviours in complex systems.  _Nature Physics_ , Mar 2022.  ISSN 1745-2481.  doi: 10.1038/s41567-022-01548-5.  URL https://doi.org/10.1038/s41567-022-01548-5.
  * Rusakov & Kullmann (1998) Dmitri A. Rusakov and Dimitri M. Kullmann.  Extrasynaptic glutamate diffusion in the hippocampus: Ultrastructural constraints, uptake, and receptor activation.  _Journal of Neuroscience_ , 18(9):3158–3170, 1998\.  ISSN 0270-6474.  doi: 10.1523/JNEUROSCI.18-09-03158.1998.  URL https://www.jneurosci.org/content/18/9/3158.
  * Santoro et al. (2022) Andrea Santoro, Federico Battiston, Giovanni Petri, and Enrico Amico.  Unveiling the higher-order organization of multivariate time series, 2022\.  URL https://arxiv.org/abs/2203.10702.
  * Sathasivam & Wan Abdullah (2008) Saratha Sathasivam and Wan Ahmad Tajuddin Wan Abdullah.  Logic Learning in Hopfield Networks.  _Modern Applied Science_ , 2(3), 2008.  ISSN 1913-1844.  doi: 10.5539/mas.v2n3p57.
  * Schaub et al. (2020) Michael T Schaub, Austin R Benson, Paul Horn, Gabor Lippner, and Ali Jadbabaie.  Random walks on simplicial complexes and the normalized Hodge 1-Laplacian.  _SIAM Review_ , 62(2):353–391, 2020.
  * Schneidman et al. (2006) Elad Schneidman, Michael J. Berry, Ronen Segev, and William Bialek.  Weak pairwise correlations imply strongly correlated network states in a neural population.  _Nature_ , 440(7087):1007–1012, Apr 2006.  ISSN 1476-4687.  doi: 10.1038/nature04701.  URL https://doi.org/10.1038/nature04701.
  * Scoville & Milner (1957) W. B. Scoville and B. Milner.  Loss of recent memory after bilateral hippocampal lesions.  _Journal of Neurolgy, Neurosurgery, and Psychiatry_ , 20(1):11–21, 1957.
  * Seidl et al. (2022) Philipp Seidl, Philipp Renz, Natalia Dyubankova, Paulo Neves, Jonas Verhoeven, Jörg K. Wegner, Marwin Segler, Sepp Hochreiter, and Günter Klambauer.  Improving few- and zero-shot reaction template prediction using modern Hopfield networks.  _Journal of Chemical Information and Modeling_ , 62(9):2111–2120, 2022.  doi: 10.1021/acs.jcim.1c01065.  URL https://doi.org/10.1021/acs.jcim.1c01065.  PMID: 35034452.
  * Sejnowski (1986) Terrence J. Sejnowski.  Higher-order Boltzmann machines.  _AIP Conference Proceedings_ , 151(1):398–403, 1986.  doi: 10.1063/1.36246.  URL https://aip.scitation.org/doi/abs/10.1063/1.36246.
  * Sharma et al. (2022) Sugandha Sharma, Sarthak Chandra, and Ila Fiete.  Content addressable memory without catastrophic forgetting by heteroassociation with a fixed scaffold.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 19658–19682. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/sharma22b.html.
  * Shiino & Fukai (1993) Masatoshi Shiino and Tomoki Fukai.  Self-consistent signal-to-noise analysis of the statistical behavior of analog neural networks and enhancement of the storage capacity.  _Phys. Rev. E_ , 48:867–897, Aug 1993.  doi: 10.1103/PhysRevE.48.867.  URL https://link.aps.org/doi/10.1103/PhysRevE.48.867.
  * Smart & Zilman (2021) Matthew Smart and Anton Zilman.  On the mapping between Hopfield networks and restricted Boltzmann machines.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=RGJbergVIoO.
  * Sorra & Harris (1993) KE Sorra and KM Harris.  Occurrence and three-dimensional structure of multiple synapses between individual radiatum axons and their target pyramidal cells in hippocampal area ca1.  _Journal of Neuroscience_ , 13(9):3736–3748, 1993\.  ISSN 0270-6474.  doi: 10.1523/JNEUROSCI.13-09-03736.1993.  URL https://www.jneurosci.org/content/13/9/3736.
  * Sperling (1963) George Sperling.  A Model for Visual Memory Tasks.  _Human Factors: The Journal of Human Factors and Ergonomics Society_ , 5(1):19–31, 1963.  ISSN 15478181.  doi: 10.1177/001872086300500103.
  * Sperling et al. (2003) Reisa Sperling, Elizabeth Chua, Andrew Cocchiarella, Erin Rand-Giovannetti, Russell Poldrack, Daniel L. Schacter, and Marilyn Albert.  Putting names to faces: Successful encoding of associative memories activates the anterior hippocampal formation.  _NeuroImage_ , 20(2):1400–1410, 2003.  ISSN 10538119.  doi: 10.1016/S1053-8119(03)00391-4.
  * Spivak (2009) David I Spivak.  Higher-dimensional models of networks.  _arXiv:0909.4314_ , 2009.
  * Squire (1992) Larry R. Squire.  Memory and the Hippocampuss: A Synthesis From Findings With Rats, Monkeys, and Humans.  _Psychological Review_ , 99(2):195–231, 1992\.  ISSN 0033295X.  doi: 10.1037/0033-295X.99.2.195.
  * Squire et al. (1989) Larry R. Squire, Arthur P. Shimamura, and David G. Amaral.  Memory and the Hippocampus.  _Neural Models of Plasticity_ , pp. 208–239, 1989.  doi: 10.1016/b978-0-12-148956-4.50016-4.
  * Sridhar et al. (2021) Vivek H. Sridhar, Liang Li, Dan Gorbonos, Máté Nagy, Bianca R. Schell, Timothy Sorochkin, Nir S. Gov, and Iain D. Couzin.  The geometry of decision-making in individuals and collectives.  _Proceedings of the National Academy of Sciences_ , 118(50):e2102157118, 2021.  doi: 10.1073/pnas.2102157118.  URL https://www.pnas.org/doi/abs/10.1073/pnas.2102157118.
  * Standing (1973) Lionel Standing.  Learning 10000 pictures.  _The Quarterly journal of experimental psychology_ , 25(2):207–222, 1973.
  * Stanley (1976) J. C. Stanley.  Simulation studies of a temporal sequence memory model.  _Biological Cybernetics_ , 24(3):121–137, Sep 1976.  ISSN 1432-0770.  doi: 10.1007/BF00364115.  URL https://doi.org/10.1007/BF00364115.
  * Storkey (1997) Amos Storkey.  Increasing the capacity of a Hopfield network without sacrificing functionality.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , 1327:451–456, 1997.  ISSN 16113349.  doi: 10.1007/bfb0020196.
  * Sutherland & Rudy (1989) R. J. Sutherland and J. W. Rudy.  Configural association theory: The role of the hippocampal formation in learning, memory, and amnesia.  _Psychobiology_ , 17(2):129–144, 1989.  ISSN 08896313.  doi: 10.3758/BF03337828.
  * Treves & Amit (1988) A Treves and D J Amit.  Metastable states in asymmetrically diluted Hopfield networks.  _Journal of Physics A: Mathematical and General_ , 21(14):3155–3169, jul 1988.  doi: 10.1088/0305-4470/21/14/016.  URL https://doi.org/10.1088/0305-4470/21/14/016.
  * Tyulmankov et al. (2021) Danil Tyulmankov, Ching Fang, Annapurna Vadaparty, and Guangyu Robert Yang.  Biological learning in key-value memory networks.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , volume 34, pp. 22247–22258. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper/2021/file/bacadc62d6e67d7897cef027fa2d416c-Paper.pdf.
  * Ullman (2004) Michael T. Ullman.  Contributions of memory circuits to language: The declarative/procedural model.  _Cognition_ , 92(1-2):231–270, 2004.  ISSN 00100277.  doi: 10.1016/j.cognition.2003.10.008.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Vogel et al. (2001) Edward K. Vogel, Geoffrey F. Woodman, and Steven J. Luck.  Storage of features, conjunctions, and objects in visual working memory.  _Journal of Experimental Psychology: Human Perception and Performance_ , 27(1):92–114, 2001.  ISSN 00961523.  doi: 10.1037/0096-1523.27.1.92.
  * Watson et al. (2011a) Richard A. Watson, C. L. Buckley, and Rob Mills.  Optimization in "self-modeling" complex adaptive systems.  _Complexity_ , 16(5):17–26, 2011a.  ISSN 10990526.  doi: 10.1002/cplx.20346.
  * Watson et al. (2011b) Richard A. Watson, Rob Mills, and C. L. Buckley.  Global adaptation in networks of selfish components: Emergent associative memory at the system scale.  _Artificial Life_ , 17(3):147–166, 2011b.  ISSN 10645462.  doi: 10.1162/artl_a_00029.
  * Weber et al. (2017) Melanie Weber, Pedro D. Maia, and J. Nathan Kutz.  Estimating memory deterioration rates following neurodegeneration and traumatic brain injuries in a Hopfield network model.  _Frontiers in Neuroscience_ , 11(NOV), 2017.  ISSN 1662453X.  doi: 10.3389/fnins.2017.00623.
  * Widrich et al. (2020) Michael Widrich, Bernhard Schäfl, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Holzleitner, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, et al.  Modern Hopfield networks and attention for immune repertoire classification.  _Advances in Neural Information Processing Systems_ , 33:18832–18845, 2020.
  * Wills et al. (2005) Tom J. Wills, Colin Lever, Francesca Cacucci, Neil Burgess, and John O’Keefe.  Attractor dynamics in the hippocampal representation of the local environment.  _Science_ , 308(5723):873–876, 2005.  ISSN 00368075.  doi: 10.1126/science.1108905.
  * Winkler & Cowan (2005) István Winkler and Nelson Cowan.  From sensory to long-term memory: Evidence from auditory memory reactivation studies.  _Experimental Psychology_ , 52(1):3–20, 2005\.  ISSN 16183169.  doi: 10.1027/1618-3169.52.1.3.
  * Woodward et al. (2015) Alexander Woodward, Tom Froese, and Takashi Ikegami.  Neural coordination can be enhanced by occasional interruption of normal firing patterns: A self-optimizing spiking neural network model.  _Neural Networks_ , 62:39–46, 2015.  ISSN 18792782.  doi: 10.1016/j.neunet.2014.08.011.
  * Xu et al. (2022) Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen.  Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning, 2022.  URL https://arxiv.org/abs/2204.08770.
  * Zhang (2012) Ming Zhang.  _Artificial higher order neural networks for modeling and simulation_.  IGI Global, 2012.  ISBN 9781466621756.  doi: 10.4018/978-1-4666-2175-6.

##  Appendix A Appendix

###  A.1 Purpose of studying and extending Hopfield networks

In the context of memory more broadly, associative memory is a form of long-
term memory, memory which can potentially last a lifetime. However, memory
also exists on at least two shorter and functionally distinct time-scales:
short-term memory and sensory memory. Short-term or working memory stores
information for on the order of tens of seconds, and performs as a limited,
passive temporary memory reservoir with which to manipulate and use
information (Milner, 1955; Mongillo et al., 2008). Sensory memory operates on
even shorter timescales than short-term memory, typically on the order of only
seconds. It is where visual (Sperling, 1963; Vogel et al., 2001), auditory
(Darwin et al., 1972; Winkler & Cowan, 2005), and other sensory information
(Gordon et al., 1993; Lederman & Klatzky, 2009) is first actively ‘remembered’
– all other information is either immediate sensory information or recalled
information. A general schema for forming a long-term memory is therefore to:
(1) receive external sensory information; (2) store the features of that
sensory information in sensory memory; (3) manage and store one or more
features of the sensory information (and/or combine it with other sensory
information) in short-term memory; and then (4) consolidate this information
into long-term memory. Understanding these processes from a theoretical and
computational perspective has several real-world implications, including: (i)
it may allow us to create more intelligent machines, by gaining inspiration
and insight from biological strategies to store, retrieve, and use long-term
associative memories; and (ii) it may help us understand the neurobiological
mechanisms (and their implications) for biological memory systems, helping to
not only understand related psychological and biological phenomena, but to
potentially help identify therapeutic targets for related dysfunction.

Psychological abilities attributed to associative memory in humans and non-
human animals are typically said to be any form of long-term memorisation
which involves ‘pairing’ or ‘associating’ distinct stimuli such that when
presented with one stimuli, the subject can recall the other stimuli.
Classical examples of this type of associative memory include pairings: name-
face pairs (Sperling et al., 2003), object-sound pairs (Preziosi & Coane,
2017), and object-place pairs (Gilbert & Kesner, 2004). These types of
associative memories are part of explicit or declarative memory (Ullman,
2004), i.e., long-term memory that can be explicitly or voluntarily stated or
declared. This is in contrast to associative memories which are part of
implicit or non-declarative memory, i.e., long-term memory recalled or used
unconsciously or unintentionally. Examples of this type of associative memory
are generated by classical conditioning (Maren, 2001; Christian & Thompson,
2003) and operant conditioning (Mackintosh, 1983; McSweeney & Murphy, 2014).

Computational accounts of implicit associative memory have a long and
successful history starting from examples like the Rescorla–Wagner model
(Rescorla & Wagner, 1972). Today, this research has grown into the
computational field of reinforcement learning (Daw & Doya, 2006). In
comparison to implicit associative memory, explicit associative memory seems
more computationally sophisticated, as suggested by its complex biological
bases (Chaudhuri & Fiete, 2016; Clopath et al., 2017; Mau et al., 2020).

Soon after the proposal of Marr (1971), one of the first and most influential
computational models of (explicit) associative memory was the Hopfield model
(Hopfield, 1982), which we study and extend here. A nice feature of this model
is that in the basic case of embedding a single memory, it is easy to see
there exists a choice of threshold for every neuron whereby any partial
activation of the memory will lead to activation of all its other members.
This therefore generates an attractor dynamic around the fixed point of the
stored memory, which is comparable to the neuronal assembly attractor dynamics
seen in the hippocampus (Wills et al., 2005; Pfeiffer & Foster, 2015; Rebola
et al., 2017).

Hippocampus is probably the foremost brain structure involved in memory. It,
together with the surrounding entorhinal, perirhinal, and parahippocampal
cortices, is especially important for explicit memory (Scoville & Milner,
1957; Milner, 1966; Squire, 1992). It is a necessary structure for the initial
formation and learning of explicit memory, acting as a short-term memory for
later long-term consolidation, thought to occur in cortex (Squire et al.,
1989; Sutherland & Rudy, 1989). Classically, we think these capacities are
mainly achieved via Hebbian learning and long-term potentiation (Bliss &
Gardner-Medwin, 1973; Gustafsson & Wigström, 1988). However, there is now an
increasing literature which shows how other mechanisms may help to achieve
these memory functions (see Appendix A.2 for examples).

As a complete computational account of long-term memory storage, the classical
Hopfield model encounters challenges. As discussed in Section 1 of the main
text, the memory capacity of the classical Hopfield network is linear in the
number of neurons $N$, specifically: approximately $0.14N$ patterns may be
stored before spurious attractors overwhelm workable levels of memory recall
(Amit et al., 1985; McEliece et al., 1987; Bruck & Roychowdhury, 1990), and
this capacity diminishes further when the patterns are statistically or
spatially correlated (Löwe, 1998), in sparse connectivity regimes (Treves &
Amit, 1988; Löwe & Vermet, 2011), and in combination (Burns et al., 2022).
Biological networks typically have very sparse connectivity (Minai & Levy,
1993; Lansner, 2009; Barth & Poulet, 2012) and everyday memory items typical
share many statistical features, may have sophisticated inter-relations, and
are spatially or semantically correlated in non-trivial structures
(Constantinescu et al., 2016; Aronov et al., 2017; Bellmund et al., 2018; Bao
et al., 2019; Park et al., 2021; Griesbauer et al., 2022). Despite this,
humans can remember very high-fidelity information of thousands of
statistically similar images (Standing, 1973; Brady et al., 2008) and human
faces (Jenkins et al., 2018), tens of thousands of linguistic items (Brysbaert
et al., 2016), and more than 100,000 digits of the number $\pi$ (Bellos, 2015)
– all, seemingly, without dramatically sacrificing or over-writing other
memories.

Although modern Hopfield networks have substantially increased theoretical
memory capacity (Krotov & Hopfield, 2016; Demircigil et al., 2017), the
combined biological and psychological evidence mentioned above, along with the
finite (if large) number of brain cells (Herculano-Houzel, 2009) and energetic
demands of maintaining them and their inter-connections (Bordone et al.,
2019), suggest there may be more to the neurophysiological and computational
story. Furthermore, even if we find that the theoretical memory capacity
should still be high according to a Hopfield interpretation, capacity can be
considered as a measure of undesired interferences between memories, and thus
may be maximised for cognitive convenience or speed and accuracy of memory
recall.

Nevertheless, problems and criticisms don’t detract from the usefulness and
importance of the Hopfield model or its modern variations in the study of
memory systems (Sathasivam & Wan Abdullah, 2008; Rizzuto & Kahana, 2001; Weber
et al., 2017), usefulness in machine learning applications (Widrich et al.,
2020; Seidl et al., 2022), contribution to more general machine learning
(Sharma et al., 2022; Hoover et al., 2022), or connection between biology and
machine learning (Chaudhuri & Fiete, 2019; Tyulmankov et al., 2021; Kozachkov
et al., 2022). There are even more opportunities to build upon this
substantial foundation to create more sophisticated computational models of
associative memory. Notably, much work has improved the efficiency and
capacity of the Hopfield network (Storkey, 1997; Hopfieid, 2008; Krotov &
Hopfield, 2016; Gripon & Berrou, 2011; Mofrad & Parker, 2017). Other work has
focused on achieving sparse representations (Kim et al., 2017; Hoffmann, 2019)
or including other forms of biological realism (Watson et al., 2011a; b;
Woodward et al., 2015; Burns et al., 2022). The current work contributes to
developments in sparsity, biological realism, and memory capacity.

###  A.2 Setwise connections and modulations are bountiful in biology

Setwise connections are not limited to the case, as one might expect, of
multiple synaptic contacts between pairs or sets of cells (Jones & Powell,
1969; Sorra & Harris, 1993; Geinisman et al., 2001; Lee et al., 2013; Rigby et
al., 2022), which may result in multiplicative interactions (Poleg-Polsky &
Diamond, 2016; Reuveni et al., 2017; Groschner et al., 2022) or form of
functional synaptic clusters (Kavalali et al., 1999; Bloss et al., 2018;
Pulikkottil et al., 2021; Hedrick et al., 2022). Other examples (some of which
are illustrated in Figure 3) include the wide spatial dispersion of certain
neurotransmitters (Rusakov & Kullmann, 1998; Arbuthnott & Wickens, 2007; Kato
et al., 2022), dendro-dendritic synapses (Pinault et al., 1997; Didier et al.,
2001; Brombas et al., 2017), persistently-connected neuronal assembly
structures or ‘synapsembles’ (Buzsáki, 2010; Papadimitriou et al., 2020),
distributed persistent activity during activities such as motor planning and
working memory (Guo et al., 2017; Hart & Huk, 2020), neuroglial modulations of
neurotransmitter release probabilities across multiple neurons or synapses
(Min et al., 2012; Covelo & Araque, 2018; Chipman et al., 2021), ‘tripartite’
astrocyte–neuron synapses (Araque et al., 1999; Perea et al., 2009),
astrocytic coding (Doron et al., 2022), and during dendritic integration at
the level of individual neurons (Golding et al., 2002; Etherington et al.,
2010). Modulations of and interactions between such connections are
illustrated in Figure 4.

It is also possible to ‘functionally’ construct setwise connections through
only pairwise synapses, as shown in Krotov & Hopfield (2021). In one sense,
this kind of pairwise-based reconstruction of setwise connections could also
be thought of as similar to results from Poirazi et al. (2003), who showed how
multi-layer artificial neural networks can approximate more biologically-
sophisticated model neurons with dendrites. Indeed, many of the known and
suggested computational features of dendritic integration (Poirazi & Papoutsi,
2020; Chavlis & Poirazi, 2021) may be considered as highly specialised and
sophisticated forms of convolutions or setwise interactions.

Figure 3: A. Multi-synaptic bouton. B. Dendritic integration. C. Extra-
synaptic neurotransmitter diffusion. D. Functional connections between neural
assemblies. Figure 4: A. Neurotransmitter depletion at a synapse. B. NMDA-
spikes and unequal synaptic strengths in dendritic integration. C.
Transmission speed plasticity using myelination (top) and axon diameter
(bottom) to affect ‘temporal’/functional setwise influence in a post-synaptic
cell. D. Astrocytic messaging, between both neurons and astrocytes.

###  A.3 Options for modelling setwise connectivity in neural networks and why
we choose simplicial complexes

In geometric and topological artificial intelligence and machine learning,
recent advances have been realised by utilising higher dimensional analogues
of graphs such as simplicial complexes (Ebli et al., 2020; Roddenberry et al.,
2021), cube complexes (Burns & Tang, 2022), cell complexes (Hajij et al.,
2020; Bodnar et al., 2021), and hypergraphs (Feng et al., 2019; Xu et al.,
2022). Unlike graphs, these structures can naturally represent higher-degree,
setwise relationships. However, not all structures are appropriate for all
systems (Spivak, 2009; Rosas et al., 2022).

Why do we choose to model our collections of setwise connections as weighted
simplicial complexes and not use general cell complexes or hypergraphs? There
are three main reasons:

  1. 1.

Simplicial complexes allow all possible setwise relations to exist. Simplices,
by construction, may span any number of vertices. This means any possible
combination of neurons may share a common, setwise weight. This is also
possible in hypergraphs, but not possible in all complexes, e.g., cube
complexes may not include triangles. The ‘natural’ complex for modelling all
possible setwise relationships is therefore a simplicial one. Since we also
wish for our setwise weights to be symmetrical (i.e., have the same value when
updating the spin with respect to each constituent neuron), it is unnecessary
to include any more than one unique object per setwise relationship. This also
makes the choice of a simplex suitable, since there can only be one unique
simplex for a given set of vertices – which is also the case for edges in
undirected hypergraphs (but we find these are less suitable).

  2. 2.

The sub-edge problem of hypergraphs makes them less suitable. Hypergraphs are
graphs where edges may contain any number of unique vertices from the vertex
set. In a sense, these are a more general structure than simplicial complexes
and lack downward closure, e.g., if the edge $\\{1,2,3\\}$ exists, edges such
as $\\{2,3\\}$ or $\\{1\\}$ do not necessarily exist, whereas if $\\{1,2,3\\}$
was a simplex in a simplicial complex, simplices $\\{2,3\\}$ and $\\{1\\}$
exist. However, hypergraphs do not have well-defined ‘sub-edges’ (described as
the ‘sub-edge problem’ in Remark 3.5 of Spivak (2009)). This has the
consequence of defining interactions between ‘levels’ of hyperedges (setwise
relationships) in hypergraphs slightly awkward. In contrast, simplicial
complexes have a well-defined hierarchy of setwise relationships, partly due
to the downward closure condition.

  3. 3.

Downward closure of setwise connections is biologically plausible. Another
benefit of downward closure in simplicial complexes is that it currently seems
better supported from the perspective of biological plausibility (also see
Appendix A.2). For example, although it can happen that a setwise connection
(anatomical or functional) between neurons could exist without any underlying
pairwise connections, the typical machinery used to create such setwise
connections is sufficiently local to assume that, because of the functional
local modularity of connections in the brain (Kaiser & Hilgetag, 2006; Chen et
al., 2013; Müller et al., 2020; Ercsey-Ravasz et al., 2013), there is a high
probability of these neurons having a pairwise connection simply due to
proximity.

As an additional practical benefit, simplicial complexes are currently more
well-studied than structures such as hypergraphs (at least in some areas,
e.g., spectral theories or (co)homology, which are of natural interest here),
meaning that we can also take advantage of the relative maturity of the field
in those areas – admittedly, we use very few advanced methods or properties in
an essential way in this study, although we hope to do so in future studies,
having now introduced an initial interpretation of simplicial Hopfield
networks and begun exploring some of their potential benefits. However, it
will also be interesting to see what differences can be found between
hypergraphic and simplicial Hopfield networks, and perhaps which provides a
closer approximation to biology or which shows improved performance on certain
tasks.

Among other possibilities, the weight of a simplex $w(\sigma)$ could be
modulated by the local energy of its spins $S_{\sigma}$, its coface’s spins,
or of those of simplices in the same dimension as $\sigma$ which are
‘neighbouring’ (in the Hodge Laplacian sense (Lim, 2020; Schaub et al.,
2020)). These interactions could take many different mathematical forms (Petri
& Barrat, 2018; Ebli et al., 2020; Roddenberry et al., 2021; Rosas et al.,
2022; Santoro et al., 2022). Neurobiologically, these interactions could
represent neural–glia interactions, glia–glia interactions, nonlinear
dendritic integration (especially dendritic spikes and shunting),
neurotransmitter–neuromodulator interactions, or hierarchical assembly
operations and dynamics, to name a few (see Appendix A.2 for illustrations and
further information).

The downward closure of simplicial complexes could be seen as a disadvantage.
For example, when including simplices of high dimension, we are also forced to
limit our choices of simplices if we wish to maintain the simplicial
structure. Again, whenever a $k$–simplex exists in $K$, all its faces must
also exist, e.g., if a triangle exists, so must its surrounding edges and
their surrounding vertices. If any constituent simplex is missing, the
structure of the simplicial complex is broken and in our case would become an
undirected, weighted hypergraph. Instead, in our simulations, we prefer to
interpret ‘missing simplices’ as merely functionally ‘missing-in-action’ by
setting their weights to zero if we do not wish to include them in the model.
This has the consequence of having no mathematical effect on our update rules
while retaining the convenience of a simplicial structure.

###  A.4 A small worked example

Consider a small example of just $P=3$ memory patterns embedded in a
simplicial Hopfield network on $N=6$ neurons. First, let’s consider a
3–skeleton, i.e., a network without any ‘dilution’ or ‘missing weights’ up to
$D=3$. Such a network will have functional connections totalling

$\sum_{d=2}^{D+1}{N\choose d}={6\choose 2}+{6\choose 3}+{6\choose
4}=15+20+15=50.$ (10)

We typically do not include functional self-connections (autapses) – although
Hopfield networks with such networks have been studied (Folli et al., 2017;
Rocchi et al., 2017; Gosti et al., 2019). In simplicial Hopfield networks,
such self-connections correspond to 0–simplices, i.e., vertices. While these
vertices do exist in the underlying simplicial complex $K$, we set their
associated weights to 0.

Given $N=6$, the $3$–skeleton in this example is

$\begin{split}{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}K_{0}=}&{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\\{\\{1\\},\\{2\\},\\{3\\},\\{4\\},\\{5\\},\\{6\\}\\}},\\\
K_{1}=&\\{\\{1,2\\},\\{1,3\\},\\{1,4\\},\\{1,5\\},\\{1,6\\},\\{2,3\\},\\{2,4\\},\\{2,5\\},\\{2,6\\},\\{3,4\\},\\{3,5\\},\\\
&\\{3,6\\},\\{4,5\\},\\{4,6\\},\\{5,6\\}\\},\\\
\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{K_{2}=}&{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\\{\\{1,2,3\\},\\{1,2,4\\},\\{1,2,5\\},\\{1,2,6\\},\\{1,3,4\\},\\{1,3,5\\},\\{1,3,6\\},\\{1,4,5\\},\\{1,4,6\\},}\\\
&{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\\{1,5,6\\},\\{2,3,4\\},\\{2,3,5\\},\\{2,3,6\\},\\{2,4,5\\},\\{2,4,6\\},\\{2,5,6\\},\\{3,4,5\\},\\{3,4,6\\},}\\\
&{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\\{3,5,6\\},\\{4,5,6\\}\\}},\\\
{\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}K_{3}=}&{\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}\\{\\{1,2,3,4\\},\\{1,2,3,5\\},\\{1,2,3,6\\},\\{1,2,4,5\\},\\{1,2,4,6\\},\\{1,2,5,6\\},\\{1,3,4,5\\},}\\\
&{\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}\\{1,3,4,6\\},\\{1,3,5,6\\},\\{1,4,5,6\\},\\{2,3,4,5\\},\\{2,3,4,6\\},\\{2,3,5,6\\},\\{2,4,5,6\\},}\\\
&{\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}\\{3,4,5,6\\}\\}},\\\
K=&\emptyset\
\cup{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>K_{1}\cup{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}K_{2}}\cup{\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}K_{3}}.\end{split}$
(11)

Set three patterns as

$\begin{split}\xi^{1}=&(-1,+1,-1,+1,-1,+1),\\\
\xi^{2}=&(+1,-1,+1,-1,-1,+1),\\\ \xi^{3}=&(-1,-1,-1,+1,+1,+1).\end{split}$
(12)

For all $\sigma\in K_{0}$, we set $w(\sigma)=0$. For all higher dimensions, we
set

$w(\sigma)=\frac{1}{N}\sum_{\mu=1}^{P}\xi_{\sigma}^{\mu}.$ (13)

For example,

$\begin{split}w(\\{1,3\\})=&1/6\cdot\left((-1\cdot+1)+(+1\cdot+1)+(-1\cdot-1)\right)=1/6,\\\
w({\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\\{3,5,6\\}})=&1/6\cdot\left((-1\cdot-1\cdot+1)+(+1\cdot-1\cdot+1)+(-1\cdot+1\cdot+1)\right)=-1/6,\\\
w({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}\\{2,4,5,6\\}})=&1/6\cdot\left((+1\cdot+1\cdot-1\cdot+1)+(-1\cdot-1\cdot-1\cdot+1)+(-1\cdot+1\cdot+1\cdot+1)\right)\\\
=&-1/2.\end{split}$ (14)

Given a set of spins $S^{(t)}$ at a time-step $t$, the network will evolve
according to Equation 2, minimising the energy shown in Equation 1. The energy
function consists of a sum of products of the weights with the product of
their respective spins, e.g., if $S^{(t)}=(+1,+1,-1,+1,-1,-1)$,

$\begin{split}E&=-\left(\dots+w(\\{1,3\\})\cdot
S_{\\{1,3\\}}^{(t)}+\dots+{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}w(\\{3,5,6\\})\cdot
S_{\\{3,5,6\\}}^{(t)}}+\dots+{\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}w(\\{2,4,5,6\\})\cdot
S_{\\{2,4,5,6\\}}^{(t)}}+\dots\right)\\\
&=-\left(\dots+1/6\cdot-1+\dots\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}-1/6\cdot-1+\dots\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+1/2\cdot
1+\dots\right).\end{split}$ (15)

In this 3–skeleton case, then, the network’s energy function can also be
written as

$\begin{split}E=-\biggl{[}&\dfrac{1}{2}\sum_{i,j}\biggl{(}\frac{1}{N}\sum_{\mu=1}^{P}\xi_{i}^{\mu}\xi_{j}^{\mu}\biggl{)}S_{i}^{(t)}S_{j}^{(t)}\\\
&+{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\dfrac{1}{3}\sum_{i,j,k}\biggl{(}\frac{1}{N}\sum_{\mu=1}^{P}\xi_{i}^{\mu}\xi_{j}^{\mu}\xi_{k}^{\mu}\biggl{)}S_{i}^{(t)}S_{j}^{(t)}S_{k}^{(t)}}\\\
&+{\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}\dfrac{1}{4}\sum_{i,j,k,l}\biggl{(}\frac{1}{N}\sum_{\mu=1}^{P}\xi_{i}^{\mu}\xi_{j}^{\mu}\xi_{k}^{\mu}\xi_{l}^{\mu}\biggl{)}S_{i}^{(t)}S_{j}^{(t)}S_{k}^{(t)}S_{l}^{(t)}}\biggr{]}.\end{split}$
(16)

Essentially, the energy function is similar to a sum of the energy functions
of Krotov & Hopfield (2016) with all possible $k$–neuron connections, but
where the weights of those connections are independent of each other for each
level of interaction, making each connection and each level more controllable.
The memory capacity of this type of simiplicial Hopfield network is discussed
in Section 2.3 and Appendix A.6. However, one of our main contributions is the
findings related to the ‘diluted’ case, i.e., where more than just the
0–simplices have their weights set to 0. Indeed, these are the cases we mainly
evaluate in Section 2.4.

Following on with the same example as above, we can create a diluted
simplicial Hopfield network based on $K$. For example, if we chose to limit
ourselves to ${6\choose 2}=15$ parameters, we could choose to apportion one
third of these parameters to each dimension, i.e., set weights only for a
subset $K^{\prime}\subset K$, e.g.,

$\begin{split}K_{1}^{\prime}=&\\{\\{1,2\\},\\{1,6\\},\\{2,3\\},\\{2,4\\},\\{5,6\\}\\},\\\
{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}K_{2}^{\prime}=}&{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\\{\\{1,2,3\\},\\{1,2,6\\},\\{1,3,4\\},\\{3,4,5\\},\\{3,4,6\\}\\}},\\\
{\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}K_{3}^{\prime}=}&{\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}\\{\\{1,3,5,6\\},\\{1,4,5,6\\},\\{2,3,4,5\\},\\{2,3,4,6\\},\\{2,3,5,6\\}\\}},\\\
K^{\prime}=&K_{1}^{\prime}\cup{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}K_{2}^{\prime}}\cup{\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}K_{3}^{\prime}}.\end{split}$
(17)

In our numerical simulations, the choice of which connections to keep is
entirely random. By analogy, we can think of this dilution procedure as a
naïve solution to the following (fairly contrived) communications problem:
Imagine we are tasked with increasing the speed at which a deliberative body
of people, e.g., a very large committee, comes to its decisions. Currently,
each committee member has individual channels of communication with every
other member. This is good for high-fidelity, accurate, and nuanced
conversations between members, but not so good for efficiency or speed of
decision-making. For example, if a certain block of members consistently vote
similarly, it would perhaps be quicker for those members to communicate as a
group to check what their majority opinion is rather than all members
individually communicating with every other member one at a time. Conversely,
when two members consistently vote differently or are active members within
distinct voting blocks (and especially if their votes are often tie-breakers),
perhaps those two members ought to regularly discuss matters privately and in
detail. Our naïve solution is to randomly replace some individual channels of
communication with small group communication channels. Possibly by performing
a survey of members or observing patterns in their voting or communications,
we could come up with a better strategy. However, in this analogy, it appears
the naïve solution works reasonably well (see results in Section 3 of the main
text). We think a deserving next step will be determine better strategies,
perhaps based on or accounting for the overall memory structure and
correlations between memory items.

###  A.5 Simplicial homology

Simplicial homology allows us to precisely count the number of ‘holes’ in each
dimension of a simplicial complex by calculating the $k$–dimensional Betti
number, $\beta_{k}$. A related topological property which is particularly
useful when studying low-dimensional objects (e.g., classification of
surfaces) is the Euler characteristic, which for a simplicial complex can be
calculated by $\chi(K)=\sum_{i=0}^{\infty}(-1)^{k}|K_{k}|$, i.e., it is an
alternating sum which ‘balances’ out the number of holes in odd and even
dimensions. It is related to the Betti numbers insofar as the Euler
characteristic is also given by
$\chi(K)=\sum_{i=0}^{\infty}(-1)^{k}\beta_{k}$. Note, however, that
$|K_{k}|\neq\beta_{k}$. As such, although the Euler characteristic can be used
for comparing the topologies of two simplicial complexes, it is not as
informative (with respect to holes) as homology (although the latter is more
costly to compute, as we will now see).

#####  ${k}$-chains and boundaries.

The group of $k$-chains is a free Abelian group with the basis of $K_{k}$,

$C_{k}=C_{k}(K):=\mathbb{Z}K_{k}:=\left\\{\sum_{\sigma\in
K_{k}}\alpha_{\sigma}\sigma\mid\alpha_{\sigma}\in\mathbb{Z}\right\\}.$

The boundary (difference between the ‘end points’) of a face $\sigma$ in
dimension $k$ is

$\partial_{k}(\sigma):=\sum_{j\in\sigma}\operatorname{sign}(j,\sigma)(\sigma\backslash
j).$

where $\operatorname{sign}(j,\sigma)=(-1)^{i-1}$, where $j$ is the $i$-th
element of $\sigma$ (ordered) and $\sigma\backslash
j:=\sigma\backslash\\{j\\}$.

######  Example 1.

Consider the simplicial complex

$K=\\{\\{1,2,3\\},\\{1,2\\},\\{1,3\\},\\{2,3\\},\\{1\\},\\{2\\},\\{3\\},\emptyset\\}.$
(18)

For $k=2$, we have $K_{2}=\\{\\{1,2,3\\}\\}$ and
$C_{2}=\\{\\{1\cdot\\{1,2,3\\}\\}\\}$. Let us calculate the boundary of
$\sigma=\\{1,2,3\\}$. First, we calculate the respective sign functions:

$\displaystyle\operatorname{sign}(1,\sigma)=(-1)^{i-1}=(-1)^{1-1}=(-1)^{0}=1$
$\displaystyle\operatorname{sign}(2,\sigma)=(-1)^{i-1}=(-1)^{2-1}=(-1)^{1}=-1$
$\displaystyle\operatorname{sign}(3,\sigma)=(-1)^{i-1}=(-1)^{3-1}=(-1)^{2}=1.$

Using these values, we may calculate the boundary by

$\displaystyle\partial_{2}(\sigma)$
$\displaystyle=\operatorname{sign}(1,\sigma)(\sigma\backslash 1)$
$\displaystyle+\operatorname{sign}(2,\sigma)(\sigma\backslash 2)$
$\displaystyle+\operatorname{sign}(3,\sigma)(\sigma\backslash 3)$
$\displaystyle=(1)(\sigma\backslash 1)$ $\displaystyle+(-1)(\sigma\backslash
2)$ $\displaystyle+(1)(\sigma\backslash 3)$ $\displaystyle=(1)(\\{2,3\\})$
$\displaystyle+(-1)(\\{1,3\\})$ $\displaystyle+(1)(\\{1,2\\})$
$\displaystyle=\\{2,3\\}$ $\displaystyle-\\{1,3\\}$ $\displaystyle+\\{1,2\\}.$

The boundary of $\\{1,2,3\\}$ is $\\{2,3\\}-\\{1,3\\}+\\{1,2\\}$. Notice, this
is a cycle.

Figure 5: Geometric realisation of the simplicial complex in equation 18 and
the boundary of $\\{1,2,3\\}$.

##### Chain complex.

The $k$-th boundary mapping, $\partial_{k}$, is the map $C_{k}(K)\rightarrow
C_{k-1}(K)$. If $k>m-1$ or $k<-1$, then $C_{k}(K):=0$ and $\partial_{k}:=0$,
where $m$ is the number of $0$–simplices. Based on this, the chain complex of
$K$ is

$0\rightarrow
C_{n-1}(K)\overset{\partial_{n-1}}{\parbox{19.91684pt}{\rightarrowfill}}\ldots\overset{\partial_{2}}{\rightarrow}C_{1}(K)\overset{\partial_{1}}{\rightarrow}C_{0}(K)\overset{\partial_{0}}{\rightarrow}C_{-1}(K)\rightarrow
0.$

We define $\partial^{2}:=\partial\circ\partial=0$. For example,
$\partial_{i-1}\circ\partial_{i}=0$. This has the consequence of making the
boundary of, say, a solid tetrahedron ($3$–simplex) a set of oriented
triangles ($2$–simplices) with a ‘net flow’ of 0 (similar to Stokes’ curl
theorem in calculus).

######  Example 2.

Consider the following simplicial complex and its chain complex:

$\displaystyle K=\\{\\{1,2\\},\\{1\\},\\{2\\},\\{3\\},\\{4\\},\emptyset\\},$
$\displaystyle 0\rightarrow
C_{1}(K)\overset{\partial_{1}}{\rightarrow}C_{0}(K)\overset{\partial_{0}}{\rightarrow}C_{-1}(K)=0.$

The boundary map $\partial_{1}$ is $\\{1,2\\}\mapsto\\{2\\}-\\{1\\}$ and all
faces in $K_{0}$ are mapped to the empty set by $\partial_{0}$.

##### Homology.

We can now see that our $k$-cycles are $Z_{k}=\operatorname{ker}\partial_{k}$
($k$-chains $\alpha$ where $\partial_{k}(\alpha)=0$). Whereas, the
$k$-boundaries are $B_{k}=\operatorname{im}\partial_{k+1}$ ($k$-chains in the
image of $\partial_{k+1}$). Notice, $B_{k}\subset Z_{k}$.

The (reduced) $k$-homology of $K$ is the Abelian group

$\widetilde{H_{k}}(K):={Z_{k}}/{B_{k}},$

and we define $\widetilde{H_{m-1}}(K):=\operatorname{ker}\partial_{m-1}$ for
$k>m-1$ and $\widetilde{H}_{k}(K):=0$ for $k<0$.

The $k$-th Betti number (number of topological holes) is

$\displaystyle\beta_{k}$
$\displaystyle=\operatorname{dim}\left(\widetilde{H_{k}}\right)$
$\displaystyle=\operatorname{dim}\left(Z_{k}\right)-\operatorname{dim}\left(B_{k}\right)$
$\displaystyle=\operatorname{nullity}\left(\partial_{k}\right)-\operatorname{rank}\left(\partial_{k+1}\right).$

Note,
$\operatorname{nullity}\partial_{0}=\operatorname{dim}\left(C_{0}\right)$.

######  Example 3.

Consider the simplicial complex

$K=\\{\\{1,2\\},\\{1,3\\},\\{2,3\\},\\{1\\},\\{2\\},\\{3\\},\emptyset\\},$

which is the same as that depicted in Figure 5 but without the filled-in
triangle. The chain complex is

$\textstyle{0\rightarrow\mathbb{Z}^{3}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\begin{smallmatrix}&\\{1,2\\}&\\{1,3\\}&\\{2,3\\}\\\
\\{1\\}&-1&-1&0\\\ \\{2\\}&1&0&-1\\\
\\{3\\}&0&1&1\end{smallmatrix}}$$\scriptstyle{\partial_{1}}$$\textstyle{\mathbb{Z}^{3}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{(0\text{
map})}$$\scriptstyle{\partial_{0}}$$\textstyle{0.}$

We may compute its Betti numbers by

$\displaystyle\beta_{0}$
$\displaystyle=\operatorname{nullity}\left(\partial_{0}\right)-\operatorname{rank}\left(\partial_{0+1}\right)$
$\displaystyle=3-2=1$ $\displaystyle\beta_{1}$
$\displaystyle=\operatorname{nullity}\left(\partial_{1}\right)-\operatorname{rank}\left(\partial_{1+1}\right)$
$\displaystyle=1-0=1,$

and, by definition, $\beta_{>1}=0$ in this example.

###  A.6 Proofs

The following are our proofs for statements in the main text.

######  Corollary A.1 (Memory capacity is proportional to the number of
network connections).

If the connection weights in a Hopfield network are symmetric, then the order
of the network’s memory capacity is proportional to the number of its
connections.

###### Proof.

Let $d$ be the degree of connections in a Hopfield network with $N$ neurons.
The explicit or implicit number of connections in such a network is $N^{d}$.
By a simple counting argument, the number of repeated connections between any
set of $d$ neurons is interpreted as $d!$. By Newman (1988); Demircigil et al.
(2017), the order of such a network’s memory capacity is $N^{d-1}$. So the
following relationship holds:

$\dfrac{N^{d-1}}{{N^{d}}/{d!}}\cdot\dfrac{N}{d!}=\dfrac{d!}{N}\cdot\dfrac{N}{d!}=1.$

∎

######  Lemma A.2 (Fully-connected mixed Hopfield networks).

A fully-connected mixed Hopfield network based on a $D$–skeleton with $N$
neurons and $P$ patterns has, when $N\rightarrow\infty$ and $P$ is finite:
fixed point attractors at the memory patterns and dynamic convergence towards
the fixed point attractors within a finite Hamming distance $\delta$. When
$P\rightarrow\infty$ with $N\rightarrow\infty$, the network has capacity to
store up to $(\sum_{d=1}^{D}N^{d})/(2\text{ ln }N)$ memory patterns (with
small retrieval errors) or $(\sum_{d=1}^{D}N^{d})/(4\text{ ln }N)$ (without
retrieval errors).

###### Proof.

Let $K$ be a $D$–skeleton. Let $S_{i}^{(t)}$ be the spin of neuron $i$ at
time-step $t$, and let the spin correspond to a stored pattern,
$S_{i}^{(t)}=\xi_{i}^{1}$. To be in a fixed point, the local field $h_{i}$
applied to $i$ must satisfy the inequality $S_{i}^{(t)}h_{i}>0$, meaning the
local field being applied to the neuron must be of the same sign as the
present spin.

In the case of the mixed network, based on the simplicial Hopfield Equations
1–2, the local field is

$\begin{split}h_{i}\\{S^{(t)}\\}&=\dfrac{1}{|K_{1}|}\sum_{\sigma\in
K_{1}}^{|K_{1}|}\sum_{\mu=1}^{P}\xi_{i}^{\mu}\xi_{\sigma\setminus
i}^{\mu}S_{\sigma\setminus i}^{(t-1)}+\dfrac{1}{|K_{2}|}\sum_{\sigma\in
K_{2}}^{|K_{2}|}\sum_{\mu=1}^{P}\xi_{i}^{\mu}\xi_{\sigma\setminus
i}^{\mu}S_{\sigma\setminus i}^{(t-1)}+\dots\\\
&=\sum_{d=1}^{D}\left(\dfrac{1}{|K_{d}|}\sum_{\sigma\in
K_{d}}^{|K_{d}|}\sum_{\mu=1}^{P}\xi_{i}^{\mu}\xi_{\sigma\setminus
i}^{\mu}S_{\sigma\setminus i}^{(t-1)}\right).\end{split}$ (19)

Because $K$ is a $D$–skeleton, each dimension $K_{d}$ will have ${N\choose d}$
elements. Using Stirling’s Approximation for the binomial coefficient,
${N\choose d}\approx N^{d}/d!$ (for $N\gg d$, which holds if $\text{dim}(K)$
is small, which we argue it should normally be for both computational and
biological reasons), we can simplify Equation 19 slightly

$h_{i}\\{S^{(t)}\\}=\sum_{d=1}^{D}\left(\dfrac{d!}{N^{d}}\sum_{\sigma\in
K_{d}}\sum_{\mu=1}^{P}\xi_{i}^{\mu}\xi_{\sigma\setminus
i}^{\mu}S_{\sigma\setminus i}^{(t-1)}\right).$ (20)

To analyse the stability of a pattern, we set $S_{i}^{(t)}=\xi_{i}^{1}$, where
the choice of $1$ is arbitrary (since the weights are symmetric). Substituting
$\xi^{1}$ for $S$ and Equation 20 for $h$, the inequality we must satisfy for
$i=1$ becomes

$\xi_{1}^{1}h_{1}=\sum_{d=1}^{D}\left(\dfrac{d!\prod_{m}^{d}(N-m)}{N^{d}}+\dfrac{d!}{N^{d}}\sum_{\sigma\in
K_{d}}\sum_{\mu=2}^{P}\xi_{1}^{1}\xi_{1}^{\mu}\xi_{\sigma\setminus
i}^{\mu}\xi_{\sigma\setminus i}^{1}\right)>0.$ (21)

Notice in Equation 21 we decomposed the summation over patterns into signal
terms (for the pattern we are analysing) and noise terms (for the contribution
of all other patterns). In the limit of $N\rightarrow\infty$, the signal terms
are fixed numbers (of order $1$) and, by the Central Limit Theorem, since the
noise terms are sums of random numbers (essentially, random walks), they will
have means of $0$ and standard deviations of

$\dfrac{d!}{N^{d}}\sqrt{(P-1)\prod_{m}^{d}(N-m)},$ (22)

which we can approximate as $\sqrt{P/N^{d}}$. We can see that as $d$
increases, the noise terms reduce in variance. However, if $P$ remains fixed
and $N$ is sufficiently large, the noise terms become negligible compared to
the signal terms. This therefore guarantees that every pattern will be a fixed
point.

Furthermore, these fixed points will remain highly stable against random
noise. Suppose we randomly flip a finite number $\delta$ of spins away from a
pattern $\xi=S$ (a fixed point). The signal terms’ strengths are reduced by
$2\delta$ but still of order $1$, whereas the noise terms remain of order
$N^{-1/2}$. Therefore, states within Hamming distance $\delta$ away from $\xi$
will converge to the fixed point.

Now let $P\rightarrow\infty$ with $N$. The total variance of the noise terms
is

$v=\sum_{d=1}^{D}\sqrt{P/N^{d}}.$ (23)

The probability of stability of neuron $i$ in Equation 21 is the probability
that the noise terms are larger than $-1$; at $\leq-1$ they will overcome the
signal terms. This probability is

$\text{Pr}(\xi_{1}^{1}h_{1}>0)=\dfrac{1}{\sqrt{2\pi
v^{2}}}\int_{-1}^{\infty}dx\text{
exp}\left(-\dfrac{x^{2}}{2v^{2}}\right)=\dfrac{1}{2}\left[1+\text{erf}\left(\sqrt{\dfrac{1}{2v^{2}}}\right)\right],$
(24)

where

$\text{erf}(x)=\dfrac{2}{\sqrt{\pi}}\int_{0}^{x}dte^{-t^{2}}.$ (25)

For small $v$ (which this is, especially as $d$ increases and relative to the
signal), the value of the error function in Equation 24 will be large and can
therefore be approximated (Gradshteyn & Ryzhik, 2007) as

$\text{erf}(x)\approx 1-\dfrac{1}{\sqrt{\pi}x}e^{-x^{2}}.$ (26)

We can now approximate Equation 24 as

$\text{Pr}(\xi_{1}^{1}h_{1}>0)\approx
1-\sqrt{\dfrac{z}{2\pi}}\text{exp}\left(-\dfrac{1}{2z}\right),$ (27)

where

$z=\sum_{d=1}^{D}P/N^{d}.$ (28)

We can now calculate the probability of a stable pattern, i.e., that the
inequality $\xi_{i}^{1}h_{i}>0$ is satisfied for all $i$, with

$\begin{split}\text{Pr}(\text{stable
pattern})&\approx\left[1-\sqrt{\dfrac{z}{2\pi}}\text{exp}\left(-\dfrac{1}{2z}\right)\right]^{N}\\\
&\approx
1-N\sqrt{\dfrac{z}{2\pi}}\text{exp}\left(-\dfrac{1}{2z}\right).\end{split}$
(29)

Since $N\rightarrow\infty$, Equation 29 will be close to $1$ as the second
term will be negligible. This will always be true if

$z=\dfrac{1}{2\text{ln}N}.$ (30)

Therefore, since we have $N$ neurons with $\sum_{d=1}^{D}N^{d}$ connections
between them, the maximum number of patterns we may store (while accepting
small errors) is

$p_{c}=\dfrac{\sum_{d=1}^{D}N^{d}}{2\text{ ln }N}.$ (31)

Or, if we cannot accept errors,

$p_{c}=\dfrac{\sum_{d=1}^{D}N^{d}}{4\text{ ln }N}.$ (32)

∎

An additional informal perspective to consider regarding memory capacity is to
notice that if $P$ scales with $N$, the ratio between the signal and noise
terms will be constant. And, since $P\propto N^{d}$, the theoretical memory
capacity scales polynomially with $N$ and linearly with $D$, and so the
theoretical capacity is approximately $\sum_{d=1}^{D}c_{d}\cdot N^{d}$, where
$c_{d}$ is a constant which depends on $d$.

###  A.7 Numerical implementation of similarity measures

As in Millidge et al. (2022), in order to fairly compare similarity functions,
we: (i) normalised similarity scores (separately for each similarity function)
so their sum would be equal to 1 (since different measures had intrinsically
different scales); and (ii) for distance measures, used the normalised
reciprocal (since distance measures return low values for similar inputs, but