# Eight challenges in developing theory of intelligence

Haiping Huang<EMAIL_ADDRESS>PMI Lab, School of Physics, Sun Yat-
sen University, Guangzhou 510275, People’s Republic of China

###### Abstract

A good theory of mathematical beauty is more practical than any current
observation, as new predictions of physical reality can be verified self-
consistently. This belief applies to the current status of understanding deep
neural networks including large language models and even the biological
intelligence. Toy models provide a metaphor of physical reality, allowing
mathematically formulating that reality (i.e., the so-called theory), which
can be updated as more conjectures are justified or refuted. One does not need
to pack all details into a model, but rather, more abstract models are
constructed, as complex systems like brains or deep networks have many sloppy
dimensions but much less stiff dimensions that strongly impact macroscopic
observables. This kind of bottom-up mechanistic modeling is still promising in
the modern era of understanding the natural or artificial intelligence. Here,
we shed light on eight challenges in developing theory of intelligence
following this theoretical paradigm.

##  I Introduction

Brain is one of the most challenging subject to understand. Like artificial
neural networks, the brain has a broad range of different levels of
abstraction, especially for theoretical studies. More abstract models lose the
ability to generate predictions on low level details, but bring the conceptual
benefits of explaining precisely how the system works, and the mathematical
description may be universal, independent of details (or sloppy variables)
[1]. One seminal example is the Hopfield model [2], where the mechanism
underlying the associative memory observed in the brain was precisely isolated
[3, 4]. There is a resurgence of research interest in Hopfield networks in
recent years due to the large language models [5, 6].

In Marr’s viewpoint [7], understanding a neural system can be divided into
three levels: computation (which task the brain solves), algorithms (how the
brain solves the task, i.e., information processing level), and implementation
(neural circuit level). For artificial neural networks, researchers build a
naive mapping of the first two levels into a toy model level (especially for
theoretical studies). Even the first two levels are now turned into ideas to
solve challenging real-world problems, driven by deep learning [8, 9].
However, biological details are also being incorporated into models of neural
networks [10, 11, 12, 13], and even used to design new learning rules [14].
Indeed, neuroscience researches about biological mechanisms of perception,
cognition, memory and action have already provided a variety of fruitful
insights inspiring the empirical/scientific studies of artificial neural
networks, which in turn inspires the neuroscience researchers to design
mechanistic models to understand the brain [15, 16, 17]. Therefore, it is
promising to integrate physics, statistics, computer science, psychology,
neuroscience and engineering to provide theoretical predictions, and reveal
inner workings of deep (biological) networks and even intelligence [18]. The
artificial intelligence may share different principles from the natural
intelligence, but both can inspire each other, which may call for establishing
new mathematical physics foundation.

The goal of providing a unified framework for neural computation is very
challenging. Due to re-boosted interests in neural networks, there appear a
lot of important yet unsolved scientific questions. We shall list some of them
below 111Most of them were roughly provided in the book of statistical
mechanics of neural networks [59]. Here we give an expanded version., and
provide our personal viewpoints towards a statistical mechanics theory of
these fundamental questions.

##  II  Representation learning

From a viewpoint of unsupervised learning aiming at extract statistical
regularities from raw data, one can ask what a good representation is and how
the meaningful representation is achieved. We have not yet satisfied answers
for these questions. A promising argument is that entangled manifolds at
earlier layers of a deep hierarchy are gradually disentangled into linearly
separable features at output layers [19, 20, 21, 22, 23]. The manifold
perspective is also promising in system neuroscience studies of associative
learning by separating overlapping patterns of neural activities [24]. A
coherent theory of the manifold transformation is still lacking, prohibiting
us from a full understanding of which key network parameters control the
geometry of manifold, and even how learning reshapes the manifold. For
example, the correlation among synapses (e.g., arising during learning) will
attenuate the decorrelation process along the network depth, but encourage
dimension reduction compared to their orthogonal counterparts [25, 22]. In
addition, there may exist other biological plausible factors such as
normalization, attention, homeostatic control [26, 27], which can be
incorporated to test the manifold transformation hypothesis.

Another argument from information theoretic viewpoints demonstrates that the
input information is maximally compressed into a hidden representation whose
task-related information should be maximally retrieved at the output layers,
according to the information bottleneck theory [28, 29]. In this sense, an
information theoretical optimal representation must be invariant to nuisance
variability and their components must be maximally independent, which may be
related to causal factors (latent causes) explaining the observation (see the
following fifth challenge). In a physics language, a corse-grained (or more
abstract) representation is formed in deeper layers compared to the fine-
grained representation in shallower layers. How microscopic interactions among
synapses determine this representation transformation remains elusive and thus
deserves future studies [30, 31].

##  III Generalization

Intelligence can be considered to some extent as the ability of
generalization, especially given very few examples for learning. Therefore,
the generalization is also a hot topic in current studies of deep learning.
Traditional statistical learning theory claims that over-fitting effects
should be strong when the number of examples is much less than the number of
parameters to learn, which thereby could not explain the current success of
deep learning. A promising perspective is to study the causal connection
between the loss landscape and the generalization properties [32, 33, 34, 35].
For a single layered perceptron, a statistical mechanics theory can be
systematically derived [36, 37]. In contrast to the classical bias-variance
trade-off (U-shaped curve of the test error versus increasing model
complexity) [38], deep learning achieves the state-of-the-art performance in
the over-parameterized regime [39, 34]. However, for a multi-layered
perceptron model, how to provide an analytic argument about the over-fitting
effects versus different parameterization regimes (e.g., under-, over- and
even super-parameterization) becomes a non-trivial task [40]. A recent study
of one-hidden-layer networks shows that a first transition occurs at the
interpolation point, where perfect fitting becomes possible. This transition
reflects the properties of hard-to-sample typical solutions. Increasing the
overparameterization degree, a second transition occurs with the discontinuous
appearance of atypical solutions. They are wide minima of good generalization
properties. This second transition sets an upper bound for the effectiveness
of learning algorithms [41].

For an infinitely wide neural network, there exists a lazy learning regime,
where the overparameterized neural networks can be well approximated by a
linear model corresponding to a first-order Taylor expansion around the
initialization, and the the complex learning dynamics is simply training a
kernel machine [42]. Furthermore, clarifying which of lazy-learning (or neural
tangent kernel limit) and feature-learning (or mean-field limit) may explain
the success of deep supervised learning remains open and challenging [43, 44,
45].

##  IV  Adversarial vulnerability

Adversarial examples are defined with those inputs with human-imperceptible
modifications yet leading to unexpected errors in a deep learning decision
making system. This adversarial vulnerability of deep neural networks poses a
significant challenge in the practical applications of both real-world
problems and AI4S (artificial intelligence for science) studies. In physics,
systems with a huge number of degrees of freedom are able to be captured by a
low-dimensional macroscopic description. Explaining the layered computation in
terms of geometry may help to crack the mysterious property of the networks’
susceptibility to adversarial examples [46, 47, 31]. In this sense, a low-
dimensional explanation with a few order parameters about the origin of the
adversarial vulnerability is lacking so far. Although some recent efforts were
devote to this direction [48, 47, 49], more exciting results are expected in
near future works.

##  V  Continual learning

A biological brain is good at adapting the acquired knowledge from similar
tasks to domains of new tasks, even if only a handful examples are available
in the new-task domain. However, the stable adaptation to changing
environments, an essence of lifelong learning, remains a significant challenge
for modern artificial intelligence. More precisely, neural networks are in
general poor at the multi-task learning, although impressive progresses have
been achieved in recent years. For example, during learning, a diagonal Fisher
information term is computed to measure importances of weights (then a rapid
change is not allowed for those important weights) for previous tasks [50]. A
later refinement by allowing synapses accumulating task relevant information
over time was also proposed [51]. More machine learning techniques to reduce
the catastrophic forgetting effects were summarized in the review [52].
However, we still do not know the exact mechanisms for mitigating the
catastrophic forgetting effects in a principled way, which calls for
theoretical studies of deep learning in terms of adaptation to domain-shift
training, i.e., connection weights trained in a solution to one task are
transformed to benefit learning on a related task. Furthermore, it remains
unclear how the related knowledge contained in a source task can be
effectively transferred to boost the performance in a target task, suppose
that both tasks share common semantics in the latent space.

A recent theoretical study interpreted the continual learning with a
statistical mechanics framework using Franz-Parisi potential [53], where the
knowledge from the previous task behaves as a reference configuration in the
Franz-Parisi potential formula [54], being an anchor for learning new
knowledge. This framework also connects to elastic weight consolidation [50],
heuristic weight-uncertainty modulation [55], and neuroscience inspired
metaplasticity [56], providing a theory-grounded method for the real-world
multi-task learning with deep networks.

##  VI Causal learning

Deep learning is criticized as being nothing but a fancy curve-fitting tool,
making a naive association between inputs and outputs. In other words, this
tool could not distinguish correlation from causation. What the deep network
learns is not a concept but merely a statistical correlation, prohibiting the
network from counterfactual inference (a hallmark ability of intelligence). A
human-like AI must be good at retrieving causal relationship among feature
components in sensory inputs, thereby carving relevant information from a sea
of irrelevant noise [57, 58]. Therefore, understanding cause and effect in
deep learning systems is particularly important for a next-generation
artificial intelligence. The question whether the current deep learning
algorithm is able to do causal reasoning remains open. Hence, developing
analytic toy models becomes a key to address this question, although it would
be very challenging to identify causes for observed effects in deep learning
by simple physics equations, due to the architecture and learning complexities
[59]. This challenge is now intimately related the astonishing performances of
large language models (see the following seventh challenge).

##  VII Internal model of the brain

The brain is argued to learn to build an internal model of the outside world,
reflected by spontaneous neural activities as a reservoir for computing (e.g.,
sampling) [60]. The agreement between spontaneous activity and stimulus-evoked
one increases during development especially for natural stimuli [61], while
the spontaneous activity outlines the regime of evoked neural responses [62].
The stimuli were shown to carve a clustered neural space [63, 64]. Then, an
interesting question is what the spontaneous neural space looks like, and how
the space dynamically evolves, especially in adaptation to changing
environments. Furthermore, how sensory inputs combined with the ongoing
cortical activity to determine animal behavior remains open and challenging.
On the other hand, reinforcement learning was used to build world models of
structured environments [65]. In reinforcement learning, data are used to
drive actions which are evaluated based on reward signals the agent receives
from the environment. It is thus interesting which kind of internal models the
agent establishes through learning from interactions with the environments.
This can be connected to representation and generalization challenges.
Moreover, a recent work showed a connection between the reinforcement learning
and statistical physics [66], suggesting that a statistical mechanics theory
could be potentially established to understand the reinforcement learning,
with additionally potential impacts on studying reward-based neural
computations in the brain [67].

Another angle to look at the internal model of the brain is through the lens
of neural dynamics [68, 69, 70], which is placed onto a low dimensional
surface, robust to variations in detailed properties of individual neurons or
circuits. The representation of stimuli, tasks or conditions can be retrieved
for deriving experimentally testable hypotheses [71]. A challenging issue is
to address how neural activity and synaptic plasticity interact with each
other to yield a low dimensional internal representation for cognitive
functions. The recent development of synaptic plasticity combining connection
probability, local synaptic noise and neural activity can realize a dynamic
network in adaptation to time-dependent inputs [72]. This work interprets the
learning as a variational inference problem, making optimal learning under
uncertainty possible in a local circuit. Both learning and neural activity are
placed on low-dimensional subspaces. Future works must include more biological
plausible factors to test the hypothesis in neurophysiological experiments.

Brain states can be considered as an ensemble of dynamical attractors [73].
The key challenge is how learning shapes the stable attractor landscape. One
can interpret the learning as a Bayesian inference, e.g., in an unsupervised
way, but not the autoregressive manner (see the next section). The learning
can then be driven by synaptic weight symmetry breaking [74, 75], separating
two phases of recognizing the network itself and the rule hidden in sensory
inputs. It is very interesting to see if this picture still holds in recurrent
learning supporting neural trajectories on dynamical attractors, e.g.,
dynamical mean-field theory for learning should be developed [76].

##  VIII Large language models

The impressive problem-solving capabilities of Chat-GPT where GPT is a short
hand of generative pretrained transformer lead the fourth industrial
revolution. The Chat-GPT is based on large language models (LLMs) [77], which
represents linguistic information as vectors in high dimensional ambient
space, trained with a large text corpus in an autoregressive way (in analogy
to the hypothesis that the brain is a prediction machine), resulting in a
complex statistical model of how the tokens in the training data correlate
[78]. The computational model thus shows strong formal linguistic competence
[79]. The LLM is also a few-shot or even zero-shot learner [80, 81], i.e., the
language model can perform a wide range of computationally challenging tasks
with prompting alone (e..g, chain-of-thought prompting [82]). Remarkably, the
LLMs display a qualitative leap in capability as the model complexity and
sample complexity are both scaled up [83], akin to phase transitions in
thermodynamic systems.

In contrast to the formal linguistic competence, the functional linguistic
competence is argued to be weak [79]. This raises a fundamental question what
the nature of intelligence is, or whether a single next-token context
conditional prediction is a standard model of artificial general intelligence
[84, 85, 86]. Human’s reasoning capabilities in real-world problems rely on
non-linguistic information as well, e.g., it is unpredictable when a creative
idea for a scientist would come to a challenging problem at hand, which relies
on reasoning about the implications along a sequence of thought. In a
biological implementation, the language modules are separated from the other
modules involving high-level cognition [79]. The LLM explains hierarchical
correlations in word pieces in the training corpora, rather than hidden casual
dependencies. In other words, the neural network has not constructed a mental
model of the world, which requires heterogeneous modular networks, thereby
unlike humans. Therefore, the LLM does not know what it generate (as a
generative model). Even if some key patterns of statistical regularities are
absent in the training data, the model can generate perfect texts in terms of
syntax. However, the texts may be far away from the truth. Knowing what they
know is a crucial hallmark of intelligent systems [86]. In this sense, the
inner workings of the LLM are largely opaque, requiring a great effort to
mathematically formulate the formal linguistic competence, and further
identify key elements that must be included to develop a robust model of the
world. Mechanisms behind the currently observed false positive like
hallucination [87] could then be revealed.

Most importantly, we currently do not have any knowledge about how to build an
additional network that is able to connect performance and awareness [88],
which is linked to what makes us conscious (see the last challenge). Following
the Marr’s framework, both computational and neural correlates of
consciousness remain unknown [89, 90, 91]. A current physical way is to
consider a Lyapunov function governing complex neural computation underlying
LLMs [5, 6]. In this way, the Lyapunov function perspective will open the door
of many degrees of freedom to control how information is distilled via not
only the self-attention but also other potential gating mechanisms.

##  IX Theory of consciousness

One of the most controversial questions is the origin of consciousness—whether
the consciousness is an emergent behavior of highly heterogeneous and modular
brain circuits with various carefully-designed regions (e.g., a total of about
$10^{14}$ connections for human brain and many functionally specific modular
structures, such as Prefrontal cortex, Hippocampus, Cerebellum, etc [92, 93]).
The subjectivity of the conscious experience is in contradiction with the
objectivity of a scientific explanation. According to the Damasio’s model
[94], the ability to identify one’s self in the world and its relationship
with the world is considered to be a central characteristic of conscious
state. Whether a machine algorithm can achieve the self-awareness remains
elusive. The self-monitoring ability (or meta-cognition [95]) may endow the
machines (such as LLMs) to know what they generate. It may be important to
clarify how the model of self is related to the internal model of the brain in
a general context. For example, Karl Friston argued that the conscious
processing can be interpreted as a statistical inference problem of inferring
causes of sensory observations. Therefore, minimizing the surprise (negative
log probability of an event) may lead to self-consciousness [96], in
consistent with the hypothesis that the brain is a prediction machine [86].

There are currently two major cognitive theory of consciousness. One is the
global workspace framework [97], which relates consciousness to the widespread
and sustained propagation of cortical neural activities by demonstrating that
consciousness arises from information-processing computations of specialized
modules. The other is the integrated information theory that provides a
quantitative characterization of conscious state by integrated information
[98]. In this second theory, unconscious states have a low information
content, while conscious states bear a high information content. Both theories
follow a top-down approach, which is in stark contrast to the statistical
mechanics approach following a bottom-up manner building the bridge from
microscopic interactions to macroscopic behavior. These hypotheses are still
under intensive criticism despite some cognitive experiments they can explain
[99]. We remark that conscious states may be an emergent property of neural
activities, lying at a higher level than neural activities. It is currently
unknown how to connect these two levels, for which a new statistical mechanics
theory is required. From an information-theoretic argument, the conscious
state may require a diverse range of configurations of interactions between
brain networks, which can be linked to the entropy concept in physics [100].
The large entropy leads to optimal segregation and integration of information
[101]. From an out-of-equilibrium viewpoint, the edge of chaos where
information processing capabilities are maximized was also conjectured to
separate the conscious from unconscious brain states [102].

Taken together, whether the consciousness can be created from an interaction
of local dynamics within complex neural substrate is still unsolved [103]. A
statistical mechanics theory, if possible, is always promising in the sense
that one can make theoretical predictions from just a few physics parameters,
which may be possible from a high degree of abstraction and thus a universal
principle could be expected.

##  X Conclusion

To sum up, in this viewpoint, we provide some naive thoughts about fundamental
important questions related to neural networks, for which building a good
theory is far from being completed. The traditional researches of statistical
physics of neural networks bifurcate to two main streams: one is to the
engineering side, developing theory-grounded algorithms; and the other is to
the neuroscience side, formulating brain computation by mathematical models
solved by physics methods. In physics, we have the principle of least action,
from which we can deduce the classical mechanics or electrodynamics laws. We
are not sure whether in physics of neural networks (and even the brain) there
exists general principles that can be expressed in a concise form of
mathematics. Readers interested in the interplay between physics theory and
neural computations are encouraged to promote advances along these exciting
yet risky research lines. The eight open problems as we demonstrate above
become increasingly challenging in order. Many exciting progresses will be
anticipated in the comping years.

###### Acknowledgements.

We would like to thank all PMI members for discussions lasting for five years.
This perspective also benefits from discussions with students during the on-
line course of statistical mechanics of neural networks (from September 2022
to June 2023). We are also grateful to invited speakers in the INTheory on-
line seminar during the COVID-19 pandemic. We enjoyed a lot of interesting
discussions with Adriano Barra, Yan Fyodorov, Sebastian Goldt, Pulin Gong,
Moritz Helias, Kamesh Krishnamurthy, Yi Ma, Alexander van Meegen, Remi
Monasson, Srdjan Ostojic, Riccardo Zecchina. This research was supported by
the National Natural Science Foundation of China for Grant numbers 12122515.

## References

  * [1] Daniel Levenstein, Veronica A. Alvarez, Asohan Amarasingham, Habiba Azab, Zhe S. Chen, Richard C. Gerkin, Andrea Hasenstaub, Ramakrishnan Iyer, Renaud B. Jolivet, Sarah Marzen, Joseph D. Monaco, Astrid A. Prinz, Salma Quraishi, Fidel Santamaria, Sabyasachi Shivkumar, Matthew F. Singh, Roger Traub, Farzan Nadim, Horacio G. Rotstein, and A. David Redish.  On the role of theory and modeling in neuroscience.  Journal of Neuroscience, 43(7):1074–1088, 2023.
  * [2] J. J. Hopfield.  Neural networks and physical systems with emergent collective computational abilities.  Proc. Natl. Acad. Sci. USA, 79:2554, 1982.
  * [3] Daniel J Amit, Hanoch Gutfreund, and H Sompolinsky.  Statistical mechanics of neural networks near saturation.  Annals of Physics, 173(1):30–67, 1987.
  * [4] M. Griniasty, M. V. Tsodyks, and Daniel J. Amit.  Conversion of temporal correlations between stimuli to spatial correlations between attractors.  Neural Computation, 5(1):1–17, 1993.
  * [5] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter.  Hopfield networks is all you need.  arXiv:2008.02217, 2020.  in ICLR 2021.
  * [6] Dmitry Krotov and John Hopfield.  Large associative memory problem in neurobiology and machine learning.  arXiv:2008.06996, 2020.  in ICLR 2021.
  * [7] D. Marr.  Vision: A Computational Investigation into the Human Representation and Processing of Visual Information.  MIT Press, Cambridge, MA, 1982.
  * [8] Jurgen Schmidhuber.  Deep learning in neural networks: An overview.  Neural Networks, 61:85–117, 2015.
  * [9] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  Nature, 521(7553):436–444, 2015.
  * [10] L F Abbott, Brian DePasquale, and Raoul-Martin Memmesheimer.  Building functional networks of spiking model neurons.  Nature Neuroscience, 19(3):350–355, 2016.
  * [11] Adam H. Marblestone, Greg Wayne, and Konrad P. Kording.  Toward an integration of deep learning and neuroscience.  Frontiers in Computational Neuroscience, 10:94, 2016.
  * [12] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte, Peter Latham, Grace W Lindsay, Kenneth D Miller, Richard Naud, Christopher C Pack, Panayiota Poirazi, Pieter Roelfsema, João Sacramento, Andrew Saxe, Benjamin Scellier, Anna C Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denis Therien, and Konrad P Kording.  A deep learning framework for neuroscience.  Nature Neuroscience, 22(11):1761–1770, 2019.
  * [13] Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton.  Backpropagation and the brain.  Nature Reviews Neuroscience, 21(6):335–346, 2020.
  * [14] Samuel Schmidgall, Jascha Achterberg, Thomas Miconi, Louis Kirsch, Rojin Ziaei, S. Pardis Hajiseyedrazi, and Jason Eshraghian.  Brain-inspired learning in artificial neural networks: a review.  arXiv:2305.11252, 2023.
  * [15] Yamins Daniel L K and DiCarlo James J.  Using goal-driven deep learning models to understand sensory cortex.  Nat Neurosci, 19(3):356–365, 2016.
  * [16] Andrew Saxe, Stephanie Nelli, and Christopher Summerfield.  If deep learning is the answer, then what is the question?  Nat. Rev. Neurosci., 22:55, 2020.
  * [17] Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick.  Neuroscience-inspired artificial intelligence.  Neuron, 95(2):245–258, 2017.
  * [18] Yi Ma, Doris Tsao, and Heung-Yeung Shum.  On the principles of parsimony and self-consistency for the emergence of intelligence.  Frontiers of Information Technology & Electronic Engineering, 23(9):1298–1323, 2022.
  * [19] James J. DiCarlo and David D. Cox.  Untangling invariant object recognition.  Trends in Cognitive Sciences, 11(8):333–341, 2007.
  * [20] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  * [21] P. P. Brahma, D. Wu, and Y. She.  Why deep learning works: A manifold disentanglement perspective.  IEEE Transactions on Neural Networks and Learning Systems, 27(10):1997–2008, 2016.
  * [22] Haiping Huang.  Mechanisms of dimensionality reduction and decorrelation in deep neural networks.  Phys. Rev. E, 98:062313, 2018.
  * [23] Uri Cohen, SueYeon Chung, Daniel D Lee, and Haim Sompolinsky.  Separability and geometry of object manifolds in deep neural networks.  Nature Communications, 11(1):1–13, 2020.
  * [24] N. Alex Cayco-Gajic and R. Angus Silver.  Re-evaluating circuit mechanisms underlying pattern separation.  Neuron, 101(4):584–602, 2019.
  * [25] Jianwen Zhou and Haiping Huang.  Weakly-correlated synapses promote dimension reduction in deep neural networks.  Phys. Rev. E, 103:012315, 2021.
  * [26] Gina G. Turrigiano and Sacha B. Nelson.  Homeostatic plasticity in the developing nervous system.  Nature Reviews Neuroscience, 5(2):97–107, 2004.
  * [27] John H. Reynolds and David J. Heeger.  The normalization model of attention.  Neuron, 61(2):168–185, 2009.
  * [28] Ravid Shwartz-Ziv and Naftali Tishby.  Opening the black box of deep neural networks via information.  arXiv:1703.00810, 2017.
  * [29] Alessandro Achille and Stefano Soatto.  A separation principle for control in the age of deep learning.  arXiv:1711.03321, 2017.
  * [30] Chan Li and Haiping Huang.  Learning credit assignment.  Phys. Rev. Lett., 125:178301, 2020.
  * [31] Chan Li and Haiping Huang.  Emergence of hierarchical modes from deep learning.  Phys. Rev. Res., 5:L022011, 2023.
  * [32] Haiping Huang and Yoshiyuki Kabashima.  Origin of the computational hardness for learning with binary synapses.  Physical Review E, 90:052813, 2014.
  * [33] Carlo Baldassi, Christian Borgs, Jennifer T Chayes, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina.  Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes.  Proceedings of the National Academy of Sciences, 113(48):E7655–E7662, 2016.
  * [34] S Spigler, M Geiger, S d’Ascoli, L Sagun, G Biroli, and M Wyart.  A jamming transition from under- to over-parametrization affects generalization in deep learning.  Journal of Physics A: Mathematical and Theoretical, 52:474001, 2019\.
  * [35] Wenxuan Zou and Haiping Huang.  Data-driven effective model shows a liquid-like deep learning.  Phys. Rev. Res., 3:033290, 2021.
  * [36] G Gyorgyi.  First-order transition to perfect generalization in a neural network with binary synapses.  Physical Review A, 41(12):7097–7100, 1990.
  * [37] H. Sompolinsky, N. Tishby, and H. S. Seung.  Learning from examples in large neural networks.  Physical review letters, 65:1683–1686, 1990.
  * [38] Pankaj Mehta, Marin Bukov, Ching-Hao Wang, Alexandre G.R. Day, Clint Richardson, Charles K. Fisher, and David J. Schwab.  A high-bias, low-variance introduction to machine learning for physicists.  Physics Reports, 810:1–124, 2019.
  * [39] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.  Reconciling modern machine-learning practice and the classical bias-variance trade-off.  Proceedings of the National Academy of Sciences of the United States of America, 116(32):15849–15854, 2019.
  * [40] Ben Adlam and Jeffrey Pennington.  The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization.  In ICML 2020: 37th International Conference on Machine Learning, 2020.
  * [41] Carlo Baldassi, Clarissa Lauditi, Enrico M. Malatesta, Rosalba Pacelli, Gabriele Perugini, and Riccardo Zecchina.  Learning through atypical phase transitions in overparameterized neural networks.  Phys. Rev. E, 106:014116, 2022.
  * [42] Mikhail Belkin.  Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation.  arXiv:2105.14368, 2021.
  * [43] Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  In Advances in Neural Information Processing Systems, volume 31, pages 8571–8580, 2018.
  * [44] Cong Fang, Hanze Dong, and Tong Zhang.  Mathematical models of overparameterized neural networks.  Proceedings of the IEEE, 109(5):683–703, 2021.
  * [45] Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin.  Deep learning: a statistical viewpoint.  arXiv:2103.09177, 2021.
  * [46] Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow.  Adversarial spheres.  arXiv:1801.02774, 2018.
  * [47] Luca Bortolussi and Guido Sanguinetti.  Intrinsic geometric vulnerability of high-dimensional artificial intelligence.  arXiv:1811.03571, 2018.
  * [48] Richard Kenway.  Vulnerability of deep learning.  arXiv:1803.06111, 2018.
  * [49] Zijian Jiang, Jianwen Zhou, and Haiping Huang.  Relationship between manifold smoothness and adversarial vulnerability in deep learning with local errors.  Chin. Phys. B, 30:048702, 2021.
  * [50] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences of the United States of America, 114(13):3521–3526, 2017.
  * [51] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In Proceedings of the 34th International Conference on Machine Learning - Volume 70, volume 70, pages 3987–3995, 2017.
  * [52] German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  Neural Networks, 113:54–71, 2019.
  * [53] Chan Li, Zhenye Huang, Wenxuan Zou, and Haiping Huang.  Statistical mechanics of continual learning: variational principle and mean-field potential.  arXiv:2212.02846, 2022.
  * [54] Silvio Franz and Giorgio Parisi.  Recipes for metastable states in spin glasses.  Journal De Physique I, 5(11):1401–1415, 1995.
  * [55] Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach.  Uncertainty-guided continual learning with bayesian neural networks.  In International Conference on Learning Representations, 2020.
  * [56] Axel Laborieux, Maxence Ernoult, Tifenn Hirtzlin, and Damien Querlioz.  Synaptic metaplasticity in binarized neural networks.  Nature Communications, 12(1):2549, 2021.
  * [57] Bernhard Schölkopf.  Causality for machine learning.  arXiv:1911.10500, 2019.
  * [58] Judea Pearl and Dana Mackenzie.  The Book of Why: The New Science of Cause and Effect.  Basic Books, New York, NY, 2018.
  * [59] Haiping Huang.  Statistical Mechanics of Neural Networks.  Springer, Singapore, 2022.
  * [60] Dario L Ringach.  Spontaneous and driven cortical activity: implications for computation.  Current Opinion in Neurobiology, 19(4):439–444, 2009.
  * [61] Pietro Berkes, Gergo Orban, Mate Lengyel, and Jozsef Fiser.  Spontaneous Cortical Activity Reveals Hallmarks of an Optimal Internal Model of the Environment.  Science, 331:83, 2011.
  * [62] Luczak Artur, Bartho Peter, and Harris Kenneth D.  Spontaneous Events Outline the Realm of Possible Sensory Responses in Neocortical Populations.  Neuron, 62:413, 2009.
  * [63] Haiping Huang and Taro Toyoizumi.  Clustering of neural code words revealed by a first-order phase transition.  Phys. Rev. E, 93:062416, 2016.
  * [64] Michael J Berry and Gašper Tkačik.  Clustering of neural activity: A design principle for population codes.  Frontiers in computational neuroscience, 14:20, 2020.
  * [65] David Ha and Jurgen Schmidhuber.  World models.  arXiv: 1803.10122, 2018.
  * [66] Jad Rahme and Ryan P. Adams.  A theoretical connection between statistical physics and reinforcement learning.  arXiv:1906.10228, 2019.
  * [67] Emre O. Neftci and Bruno B. Averbeck.  Reinforcement learning in artificial and biological systems.  Nature Machine Intelligence, 1(3):133–143, 2019.
  * [68] David Sussillo and L.F. Abbott.  Generating coherent patterns of activity from chaotic neural networks.  Neuron, 63(4):544–557, 2009.
  * [69] Dean V. Buonomano and Wolfgang Maass.  State-dependent computations: spatiotemporal processing in cortical networks.  Nature Reviews Neuroscience, 10(2):113–125, 2009.
  * [70] Saurabh Vyas, Matthew D. Golub, David Sussillo, and Krishna V. Shenoy.  Computation through neural population dynamics.  Annual Review of Neuroscience, 43(1):249–275, 2020.
  * [71] Mehrdad Jazayeri and Srdjan Ostojic.  Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity.  Current Opinion in Neurobiology, 70:113–120, 2021.
  * [72] Wenxuan Zou, Chan Li, and Haiping Huang.  Ensemble perspective for understanding temporal credit assignment.  Physical Review E, 107(2):024307, 2023.
  * [73] Christoph von der Malsburg.  Concerning the neural code.  arXiv:1811.01199, 2018.
  * [74] Tianqi Hou, K Y Michael Wong, and Haiping Huang.  Minimal model of permutation symmetry in unsupervised learning.  Journal of Physics A: Mathematical and Theoretical, 52(41):414001, 2019.
  * [75] Tianqi Hou and Haiping Huang.  Statistical physics of unsupervised learning with prior knowledge in neural networks.  Phys. Rev. Lett., 124:248302, 2020.
  * [76] Wenxuan Zou and Haiping Huang.  Introduction to dynamical mean-field theory of generic random neural networks.  arXiv:2305.08459, 2023.
  * [77] OpenAI.  Gpt-4 technical report.  arXiv:2303.08774, 2023.
  * [78] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * [79] Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko.  Dissociating language and thought in large language models: a cognitive perspective.  arXiv:2301.06627, 2023.
  * [80] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [81] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  arXiv:2205.11916, 2022.  in NeurIPS 2022.
  * [82] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  arXiv:2201.11903, 2022.  in NeurIPS 2022.
  * [83] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  arXiv:2001.08361, 2020.
  * [84] Terrence J. Sejnowski.  Large Language Models and the Reverse Turing Test.  Neural Computation, 35(3):309–342, 2023.
  * [85] Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman.  Building machines that learn and think like people.  Behavioral and Brain Sciences, 40:e253, 2017.
  * [86] Marcel van Gerven.  Computational foundations of natural intelligence.  Frontiers in Computational Neuroscience, 11:112, 2017.
  * [87] Noam Chomsky, Ian Roberts, and Jeffrey Watumull.  Noam chomsky: The false promise of chatgpt.  The New York Times, 8, 2023.
  * [88] Axel Cleeremans.  Connecting conscious and unconscious processing.  Cognitive Science, 38(6):1286–1315, 2014.
  * [89] Francis Crick and Christof Koch.  A framework for consciousness.  Nature Neuroscience, 6(2):119–126, 2003.
  * [90] Lenore Blum and Manuel Blum.  A theory of consciousness from a theoretical computer science perspective: Insights from the conscious turing machine.  Proceedings of the National Academy of Sciences of the United States of America, 119(21):e2115934119, 2022.
  * [91] Abhilash Dwarakanath, Vishal Kapoor, Joachim Werner, Shervin Safavi, Leonid A. Fedorov, Nikos K. Logothetis, and Theofanis I. Panagiotaropoulos.  Bistability of prefrontal states gates access to consciousness.  Neuron, 111(10):1666–1683, 2023.
  * [92] K. Harris and G. Shepherd.  The neocortical circuit: themes and variations.  Nat Neurosci, 18:170–181, 2015.
  * [93] Liqun Luo.  Architectures of neuronal circuits.  Science, 373(6559):eabg7285, 2021.
  * [94] A. Damasio.  Fundamental feelings.  Nature, 413:781, 2001.
  * [95] Stanislas Dehaene, Hakwan Lau, and Sid Kouider.  What is consciousness, and could machines have it?  Science, 358(6362):486–492, 2017.
  * [96] Karl Friston.  Am i self-conscious? (or does self-organization entail self-consciousness?).  Frontiers in Psychology, 9:579, 2018.
  * [97] Stanislas Dehaene, Michel Kerszberg, and Jean-Pierre Changeux.  A neuronal model of a global workspace in effortful cognitive tasks.  Proceedings of the National Academy of Sciences of the United States of America, 95(24):14529–14534, 1998.
  * [98] Giulio Tononi.  An information integration theory of consciousness.  BMC Neuroscience, 5(1):42, 2004.
  * [99] Christof Koch, Marcello Massimini, Melanie Boly, and Giulio Tononi.  Neural correlates of consciousness: progress and problems.  Nature Reviews Neuroscience, 17(5):307–321, 2016.
  * [100] R. Guevara Erra, D. M. Mateos, R. Wennberg, and J. L. Perez Velazquez.  Statistical mechanics of consciousness: Maximization of information content of network is associated with conscious awareness.  Physical Review E, 94(5):52402, 2016.
  * [101] David W. Zhou, David D. Mowrey, Pei Tang, and Yan Xu.  Percolation model of sensory transmission and loss of consciousness under general anesthesia.  Phys. Rev. Lett., 115:108103, 2015.
  * [102] Daniel Toker, Ioannis Pappas, Janna D Lendner, Joel Frohlich, Diego M Mateos, Suresh Muthukumaraswamy, Robin Carhart-Harris, Michelle Paff, Paul M Vespa, Martin M Monti, et al.  Consciousness is supported by near-critical slow cortical electrodynamics.  Proceedings of the National Academy of Sciences, 119(7):e2024455119, 2022.
  * [103] Patrick Krauss and Andreas Maier.  Will we ever have conscious machines?  Frontiers in Computational Neuroscience, 14:556544, 2020.